By Filip Zeman
Last Validated on May 7 2021 · Originally Published on May 7, 2021 · Viewed 6.2k times

How to Log Data As JSON With Python?

The simplest way is to use a custom module:

pip install json-logging

To initialize the logger when not using the web framework, use this line of code:

json_logging.init_non_web(enable_json=True)

For web applications, replace _non_web with the name of framework. (_flask, _sanic,...)

More documentation about this module is provided here.


How To Write Logs To A File With Python?

Using Basic Configuration

You can use basic config. If you configure the attribute filename, logs will be automatically saved to the file you specify. You can also configure the attribute filemode. Setting the value to w will overwrite the file after every entry.

import logging

logging.basicConfig(filename="logs.log", filemode="w", format="%(name)s -> %(levelname)s: %(message)s")

logging.warning("warning")
OUTPUT
root -> WARNING: warning

Using Provided Classes

You can also use the provided classes - loggers and handlers:


logger = logging.getLogger(__name__)
FileOutputHandler = logging.FileHandler('logs.log')

logger.addHandler(FileOutputHandler)

logger.warning("Warning.")

You will create a logger and a handler. When creating a handler, assign the class FileHandler and a file name as an attribute. Then set the handler to a logger:

OUTPUT
Warning.

How to Log to Stdout with Python?

Using Basic Configuration

Python, by default, logs to a console. You can call the function on the module:

import logging

logging.warning("Warning.")
OUTPUT
WARNING:root:Warning.

Python already provides default formatting.

Using Provided Classes

You can also use the provided classes:

import logging

logger = logging.getLogger("nameOfTheLogger")
ConsoleOutputHandler = logging.StreamHandler()

logger.addHandler(ConsoleOutputHandler)

logger.warning("Warning.")

Create a new logger and a new handler. Assign the class StreamHandler to the handler and assign the handler to the logger. The output will be the following:

OUTPUT
Warning.

How To Log Uncaught Exceptions In Python?

For this, you can use the sys.excepthook that allows us to attach a handler for any unhandled exception:

#Creating a logger
logger = logging.getLogger(__name__)
logging.basicConfig(filename='example.log', filemode='w', level=logging.DEBUG)

#Creating a handler
def handle_unhandled_exception(exc_type, exc_value, exc_traceback):
    if issubclass(exc_type, KeyboardInterrupt):
				#Will call default excepthook
        sys.__excepthook__(exc_type, exc_value, exc_traceback)
        return
		#Create a critical level log message with info from the except hook.
    logger.critical("Unhandled exception", exc_info=(exc_type, exc_value, exc_traceback))
#Assign the excepthook to the handler
sys.excepthook = handle_unhandled_exception

First, you will create a logger that will take care of your exceptions.

Then create a handler handle_unhandled_exception that you will later attach to the hook. Then call the __excepthook__ provided by the sys module. This method is invoked every time the exception is uncaught.

Use the info from the hook to create a log message and assign this handler to the sys.excepthook.

You can learn more info about sys.excepthook in the offical documentation.


How To Use Logging In Multiple Modules?

It's recommended to have a logger defined in each module like this:

import logging
logger = logging.getLogger(__name__)

Then in your main program, do the following:

import logging.config
logging.config.fileConfig('/path/to/logging.conf')

If you want to use one logger, you can create a variable for the logging module:

import logging

logger = logging
logger.basicConfig()

Then in other files, you will import the logger and use it from now on:

from base_logger import logger

Every logger is a child of the parent's package logger. Meaning that all you need to do is configure the root logger.


How To Log All Requests From The Python Request Library?

You need to use urllib3 logger and set the log level to DEBUG:

log = logging.getLogger('urllib3')

log.setLevel(logging.DEBUG)

To maximise the message you can get, set HTTPConnection.debuglevel to 1:

from http.client import HTTPConnection
HTTPConnection.debuglevel = 1

By using the additional method, that cannot be captured by logging, you need to settle for console output.


How To Color Python Logging Output?

Without External Module

Create a new custom formatter:

class CustomFormatter(logging.Formatter):

then create variables for the colors. They are created as ASCII code for an escape character followed by appropriate code sequence:

grey = "\\x1b[38;21m"
yellow = "\\x1b[33;21m"
red = "\\x1b[31;21m"
bold_red = "\\x1b[31;1m"
reset = "\\x1b[0m"
format = "%(asctime)s - %(name)s - %(levelname)s - %(message)s (%(filename)s:%(lineno)d)"

You can discover all the possible colors here. Also specify the provided variable format.

After that, create a dictionary and specify a format for each of the log levels - color, your format of the log message, and reset the color at the end. Finally, make your formatter return this custom format:

FORMATS = {
        logging.DEBUG: grey + format + reset,
        logging.INFO: grey + format + reset,
        logging.WARNING: yellow + format + reset,
        logging.ERROR: red + format + reset,
        logging.CRITICAL: bold_red + format + reset
    }

    def format(self, record):
        log_fmt = self.FORMATS.get(record.levelno)
        formatter = logging.Formatter(log_fmt)
        return formatter.format(record)

And the whole formatter:

class CustomFormatter(logging.Formatter):
    grey = "\\x1b[38;21m"
    yellow = "\\x1b[33;21m"
    red = "\\x1b[31;21m"
    bold_red = "\\x1b[31;1m"
    reset = "\\x1b[0m"
    format = "%(asctime)s - %(name)s - %(levelname)s - %(message)s (%(filename)s:%(lineno)d)"

    FORMATS = {
        logging.DEBUG: grey + format + reset,
        logging.INFO: grey + format + reset,
        logging.WARNING: yellow + format + reset,
        logging.ERROR: red + format + reset,
        logging.CRITICAL: bold_red + format + reset
    }

    def format(self, record):
        log_fmt = self.FORMATS.get(record.levelno)
        formatter = logging.Formatter(log_fmt)
        return formatter.format(record)

You can later assign this formatter to a handler:

handler.setFormatter(CustomFormatter())

Using External Module

For this, you can use an external module named colorlog. You can download it by running this command in the terminal:

pip install colorlog

You can read more about this module here.

After you installed the module, do not forget to import it:

import colorlog

Now using colorlog, create a handler using the class StreamHandler(). Then create a logger and assign this handler to it:

handler = colorlog.StreamHandler()

logger = colorlog.getLogger(__name__)
logger.addHandler(handler)

Now set a format for the handler. At the start, specify the desired color as an attribute and create some logging message:

handler.setFormatter(colorlog.ColoredFormatter('%(red)s%(levelname)s:%(name)s:%(message)s'))
logger.warning('colors')
OUTPUT
WARNING:__main__:colors

How To Disable Logging While Running Django Unit Tests?

By Disabling Tests At The Start Of The Application

Suppose you want to do it the quick way. In that case, the following line of code will disable any log messages less severe or equal to CRITICAL:

logging.disable(logging.CRITICAL)

In settings.py Globally

You can also paste these lines of code in your settings.py file, so you do not need to paste the first line multiple times:

import sys
import logging

if len(sys.argv) > 1 and sys.argv[1] == 'test':
    logging.disable(logging.CRITICAL)

sys.argv list contains the command-line arguments passed to the script. If some tests are found, logging will be disabled.

Using Filters

While this will not disable the logging globally, you will have more options to go with while developing your application. Create your filter in the settings.py file:

from logging import Filter

class NotInTestingFilter(Filter):

    def filter(self, record):
        from django.conf import settings
        
        return not settings.TESTING_MODE

In the same file, create a variable in which you will say to the program if you are running in the testing mode:

TESTING_MODE = True

Then add this filter to your logging configuration:

'filters': {
        'testing': {
            '()': NotInTestingFilter
        }
    },

And assign it to a handler:

'filters': ['testing']

How To Disable Logging From The Python Request Library?

You can change the log level of the logger taking care of these messages. Setting the level to WARNING will remove the request messages and keep warnings and errors:

import logging

logging.getLogger("requests").setLevel(logging.WARNING)
logging.getLogger("urllib3").setLevel(logging.WARNING)

In settings.py File

While using Django or other platforms, you can also change the logger in the settings provided by the framework. The following example is from Django:

'urllib3': {
        'handlers': ['file'],
        'level': logging.WARNING
    }