By Filip Zeman
Last Validated on May 6 2021 · Originally Published on May 6, 2021 · Viewed 5.3k times

Introduction

Logging is an important part of every application life cycle. Having a good logging system becomes a key feature that helps developers, sysadmins, and support teams to understand and solve appearing problems.

The Python standard library already comes with an integrated logging module that provides basic as well as advanced logging features.

In this tutorial, you will learn how to:

  • Use the logging module and create events of various importance.
  • Configure your logs.
  • Capture the stack trace.
  • Log into a file.
  • How to use loggers, handlers, and formatters

Prerequisites


Step 1 — Basic Logging

Python already provides us with a built-in module. You can import these functionalities in your project with this line of code:

import logging

After you have the module imported, you can start creating your logs. You will use "logger" to log messages. Normally, you have five levels indicating the importance of an alert. They have their own methods with the same name that can create them. The levels, in order of increasing severity, are:

  • Debug: used for detailed information, typically of interest only when diagnosing problems.
  • Info: used to get a confirmation that things are working as expected.
  • Warning: used as an indication that something unexpected happened, or indicative of some problem in the near future (e.g. ‘disk space low’). The software is still working as expected.
  • Error: used in cases of more serious problems the software has not been able to perform some function.
  • Critical: used to indicate serious errors which might cause the program to stop running.

You can start creating them without needing to do any configuration. You can create the messages in the following fashion:

logging.debug("debug info.")
logging.info("general info message")
logging.warning("warning")
logging.error("error")
logging.critical("critical message")

Your output will look like this:

output
WARNING:root:warning
ERROR:root:error
CRITICAL:root:critical message

The output shows the level of the message and root, which is the name of the default logger.

Notice that messages written into debug() and info() functions did not appear. Only the logs with severity of WARNING or higher will show up in the console by default.


Basic Configuration

To configure the default logger in Python, you will use the method basicConfig().

Parameters that are commonly used are:

  • level: How severe the message has to be for it to be logged.
  • format: Format of the logged message.
  • filename: Provides a way to log messages into a file.
  • filemode: If the filename is set, you can change how the logs are saved.
  • a: Append, default option. The logs will be added after each other.
  • w: Entry overwrites the entire file.

As you noticed earlier, debug and info messages did not get inputted into the console. You can change that by using the parameter level:

logging.basicConfig(level=logging.DEBUG)

Logging Into A File

If you would prefer to log into a file rather than to the console, you can use the filename and filemode. You can also use the format attribute:

logging.basicConfig(filename="logs.log", filemode="w", format="%(name)s -> %(levelname)s: %(message)s")

After this there won't be any visible output in the console, but a file was created in the project's directory containing the following:

output
root -> WARNING: warning
root -> ERROR: error
root -> CRITICAL: critical message

Note that the basicConfig() function should be only called once.

The message is written in the file, not in the console. You specified the filemode to w (write mode), so each time the program runs, the file will be overwritten.


Step 2 — Formatting The Output

You can include any variables in your logs, but first lets look at logging primary data such as time or ID. To display the process ID, write the following:

logging.basicConfig(format="%(process)d")

You will see the following output:

output
7696

Another useful utility is logging time. You can include it by using asctime:

logging.basicConfig(format="%(asctime)s %(message)s")
logging.critical("critical")

You will get the following output:

output
2021-04-08 22:41:10,081 critical

To change the time format logged use the datefmt attribute:

logging.basicConfig(format="%(asctime)s %(message)s", datefmt="%d-%b %H:%M")
logging.critical("critical")

This change return the following:

output
08-Apr 23:32 critical

You can explore all the date formats you can use here.


Including Variables

Now let's look at how we can include dynamic parts to our logs. Here is an example of simple message insertion:

name = "Alice"
logging.critical("%s raised critical", name)

The output will give the following:

output
CRITICAL:root:Alice raised critical

Best practise for including variables is by using f-strings, as they are newer, faster and more readable. The code using f-string looks like this:

name = "Alice"
logging.critical(f"{name} raised critical")

You can learn more about f-strings in the official documentation.


Step 3 — Capturing Stack Traces

You might want to capture the full stack trace of the moment the message was logged. Let us create a program that will deliberately try to divide by zero and log a critical when it occurs:

import logging

try:
    a = 10 / 0
except Exception as exc:
    logging.critical("Critical error occurred")

The only thing you get in the console is this:

output
CRITICAL:root:Critical error occurred

While in this case, you know what caused the error, in real life this message wouldn't be of much help. To get more information let's set the parameter exc_info to True:

import logging

try:
    a = 10 / 0
except Exception as exc:
    logging.critical("Critical error occurred", **exc_info=True**)

Now you will get an output with significantly more information, including the file location, specific line where the error occurred, and the name of the error:

output
CRITICAL:root:Critical error occurred
Traceback (most recent call last):
  File "c:\\Users\\Alice\\Desktop\\loggingInPython\\loggingInPython.py", line 4, in <module>
    a = 10 / 0
ZeroDivisionError: division by zero

Step 4 — Using Classes

Until this point, you have used the default logger. You can create your own logger by making the object of the Logger class. This will come especially handy when the project gets bigger. These loggers can only be edited by objects of the other classes, not by basicConfig().

Classes we use with the logging module in Python are:

  • Logger:  You will call functions through the objects of this class.
  • Formatter: Using this class, you can specify the output of the logger.
  • LogRecord: Object contain the information related to the logging message.
  • Handler: Sets the destination of the output. (Console or a file) Subclasses include FileHandler, SMTPHandler, StreamHandler, and more.

Loggers

You will mostly deal with the Logger class. This is how you can create a logger:

logger = logging.getLogger("nameOfTheLogger")
logger.error("Error.")

You are using a method getLogger which takes the name of the logger as its parameter. The output in the default formatting disregards the name of the logger:

output
Error.

Instead of a string name, we can pass __name__ as the name parameter to the getLogger() function. The name will then tell you more about where the events are logged.


Handlers

Handlers are helpful when you want to send your records to specific places. You can use them to send your logs to the file via HTTP or SMTP and more. A logger you create can have more than one handler; thus, the logs can be saved to more than one place. We will go through this by creating a handler.

Let us create one handler for printing output in the console with the level WARNING. For more important messages of the level ERROR and higher, handler assigned to the same logger will save it to a file.

logger = logging.getLogger(__name__)

ConsoleOutputHandler = logging.StreamHandler()
FileOutputHandler = logging.FileHandler('errorLogs.log')

ConsoleOutputHandler.setLevel(logging.WARNING)
FileOutputHandler.setLevel(logging.ERROR)

We can then assign the handlers to the loggers:

logger.addHandler(ConsoleOutputHandler)
logger.addHandler(FileOutputHandler)

Now when you try to create a WARNING and an ERROR message, both will be displayed in the console:

logger.warning("Warning.")
logger.error("Error.")

and the second message will appear in the .log file:

output
Error.

Formatters

If you want to change the way output is formatted, you need to use the class Formatter. Let us create two formatters, one for the console output, another one for the file output.

ConsoleOutputFormat = logging.Formatter('%(name)s: %(levelname)s - %(message)s')
FileOutputFormat = logging.Formatter('%(asctime)s  %(name)s: %(levelname)s - %(message)s')

Now assign these formatters to the handlers:

ConsoleOutputHandler.setFormatter(ConsoleOutputFormat)
FileOutputHandler.setFormatter(FileOutputFormat)

The output in the console will be this:

output
__main__: WARNING - Warning.
__main__: ERROR - Error.

And in the file:

output
2021-04-13 19:54:19,553  __main__: ERROR - Error.

Conclusion

You have learned the basics of logging in Python. The logging module is considered to be very good and sufficient, and its design practical. You can use it in a small project or go into an advanced category by creating your classes and levels.

If you have not been using logging into your project, now you have good basics to start. Logging can speed up your development in a big way when done right.