By Juraj Holub
Last Validated on June 16 2021 · Originally Published on June 16, 2021 · Viewed 4.4k times

Introduction

Docker is an open platform that maintains application through abstraction so-called container. Docker offers a built-in abstraction for the logging of the containers — logging driver. This tutorial shows you how to start logging your docker containers.

In this tutorial, you will do the following actions:

  • You will understand what is the Docker logging driver, and you will configure it for the local json-file logging. You will instal a hello world web application, generate some logs in it, and view them through the Docker CLI.
  • These days the microservices architecture becomes dominant and the Docker fits exactly to this architecture. This means that your application runs dozens of docker containers, even on different servers. You will configure a centralised Fluentd docker that will gather logs from all your application containers.

Prerequisites

You will need:

  • Ubuntu 20.04 distribution including the non-root user with sudo access, and installed docker.
  • You should understand the basics of docker, syslog, and what is the JSON.

Step 1 — Trying on Web Server Docker

For the demonstration of the Docker logging, we will use a simple Nginx hello world application docker.  Let's use the docker pull command with the image name (karthequian/helloworld) to retrieve an image from Docker Hub (sudo required for access to docker socket daemon):

sudo docker pull karthequian/helloworld

You’ll see the program’s output appear on the screen:

Using default tag: latest
latest: Pulling from karthequian/helloworld
550fe1bea624: Pull complete 
d421ba34525b: Pull complete 
fdcbcb327323: Pull complete 
bfbcec2fc4d5: Pull complete 
0497d4d5654f: Pull complete 
f9518aaa159c: Pull complete 
a70e975849d8: Pull complete 
Digest: sha256:f5a0b2a5fe9af497c4a7c186ef6412bb91ff19d39d6ac24a4997eaed2b0bb334
Status: Downloaded newer image for nginxdemos/hello:latest
dce37dca1bae60a4dbed391594a9f51176457805049186bc11e05521f95b24cc

The output shows that you fetch an image, and Docker stores it locally and makes it available for running containers.

Now, you can start the docker container by executing docker run command (sudo required):

$ sudo docker run -p 80:80/tcp -d nginxdemos/hello

The option -p 80:80/tcp maps port 80 in the container to port 80 on your computer. Option -d runs the container in the background and print container ID (typical usage for web service). If you visit the URI http://localhost:80 in a browser, you'll see the sample Nginx hello world app.


Removing Container

You can view active containers with the docker ps command (sudo required)

$ sudo docker ps

You’ll see the program’s output appear on the screen:

CONTAINER ID   IMAGE              COMMAND                  CREATED         STATUS         PORTS                                     NAMES
a944317493a0   nginxdemos/hello   "nginx -g 'daemon of…"   3 seconds ago   Up 3 seconds   0.0.0.0:49154->80/tcp, :::49154->80/tcp   intelligent_heisenberg

The output shows that the webserver nginxdemos/hello already run in the container, and the container has ID a944317493a0.

Because this container is the background process, you must stop it by executing docker rm --force (sudo required):

sudo docker rm --force a944317493a0

Now, if you execute docker ps then the web server won't be there anymore.

In the next steps, we will use this example of web service, but we won't run it as a background process. However, your production application will be typically executed as a background process.


Step 2 — Configuring JSON File Logging Driver

Docker includes its own logging mechanism called the logging driver. Everything that your process, running in your docker image, write to stdout, and stderr is instantly sent to the docker logging driver. The driver records incoming text messages in a specific format. The format, and behaviour of logging, is defined by the type of logging driver. The following logging drivers are supported:

  • none: No logs are available for the container.
  • local: Logs are stored in a custom format designed for minimal overhead.
  • json-file: The logs are formatted as JSON and stored in the docker. The default logging driver for Docker.
  • syslog: Writes logging messages to the syslog facility. The Syslog daemon must be running on the host machine. You can read our tutorial about Syslog if you are not familiar with it: How to view Linux logs and their configuration on Ubuntu 20.04.
  • journald: Writes log messages to journald. The journald daemon must be running on the host machine. (see tutorial How to Control Journald with Journalctl).
  • fluentd: Writes log messages to the Fluentd (forward input). The Fluentd daemon must be running on the host machine. We will configure the Fluentd container for the centralised Client-Server logging later in this tutorial.
  • And many others (see official documentation).

Configuring JSON File Logging Driver

For demonstration, we will use docker with a web application from the previous step. If you want to configure a json-file logging driver for this container then you must set up a Docker logging driver. Docker stores logging driver configuration in the file /etc/docker/daemon.json (for Linux distributions). If this file does not exist in your OS, you can create it. The file daemon.json has JSON format, and for a json-file logging driver, it could look like this:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "tag":   "{{.Name}}",
    "compress": true
  }
}

The directive log-driver sets the logging driver to json-file, a default option. It is good practice not to change it if you don't have a reason because it is mostly better supported in docker logging addons. Each type of logging driver includes its own options included in the list log-opts. The json-file driver includes the following options:

max-size: The maximum size of the log before it is rotated. An integer plus a modifier represents the measuring unit (k - kilobytes, m - megabytes, or g - gigabytes). Defaults to -1 (unlimited).

max-file: The maximum number of rotating logs. The max-size must be specified.

compress: Enable compression for rotated logs.

tag: The tag log option specifies how to format a tag that identifies the container’s log messages. By default, the system uses the container ID, which is a random number without any human-readable meaning. Docker supports some special template markup you can use when specifying a tag’s value:

  • {{.Name}}: The container name.
  • {{.ImageName}}: The name of the image used by the container.
  • {{.DaemonName}}: The name of the docker program (docker).

Then you can specify, for example tag as a {{.ImageName}}[{{.Name}}] and the log record would looks like:

May  9 18:11:00 alice hello-world/[dummy]: Hello from Docker.

The full list of all options for each logging driver is defined in the official documentation.

If you set up a configuration file, you can save it. However, the configuration doesn't apply to the already running containers. In our case, you have to follow the first step, remove a webserver container and starts it again.


Step 3 — Viewing JSON File Local Logging

The JSON file logger configured in the previous step stores the logs in the local docker, and you can view them through the command-line interface.

If you want to view the log of this specific container, you can execute docker logs with container ID (sudo required):

sudo docker logs a944317493a0

In our example the container ID is a value a944317493a0 (you can determine the ID of your web server container with docker ps, as shown earlier). You’ll see the program’s output appear on the screen:

172.17.0.1 - - [08/Jun/2021:09:19:30 +0000] "GET / HTTP/1.1" 200 7231 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" "-"
172.17.0.1 - - [08/Jun/2021:09:19:44 +0000] "GET / HTTP/1.1" 200 7231 "<http://0.0.0.0:49154/>" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" "-"
172.17.0.1 - - [08/Jun/2021:09:19:45 +0000] "GET / HTTP/1.1" 200 7231 "<http://0.0.0.0:49154/>" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" "-"
172.17.0.1 - - [08/Jun/2021:09:19:46 +0000] "GET / HTTP/1.1" 200 7231 "<http://0.0.0.0:49154/>" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" "-"
172.17.0.1 - - [08/Jun/2021:09:19:48 +0000] "GET / HTTP/1.1" 200 7231 "<http://0.0.0.0:49154/>" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" "-"

The output shows you routing log records generated by the Nginx server (you can reload http://localhost:80 in your web browser to generate more log records).

Also, if you want constantly follow the tail of the log in real-time for the application debugging purposes, then you can re-execute the command docker logs, but with the option -f (sudo required)

sudo docker logs a944317493a0 -f

Now, you can reload a few times http://localhost:80 in your browser to generate some log records. The output will generate the new log records in real-time to the console.

If you want to exit the log tail viewing in the console, you can press CTRL-C.


Step 4 — Configuring Centralised Fluentd Logging

If you administrate multiple dockers then you might want to centralise logs from all dockers. In such a case, you must configure logging client-server environment:

  • Clients are all containers, which generate logs. You must configure them not to store logs locally, but send them to the centralised logger remotely.
  • The server is a centralised logging daemon (for example, rsyslog). We will configure another docker container that will run logger daemon Fluentd.

Configuring Fluentd Docker

Fluentd is an open-source data collector for a unified logging layer. You can download the fluentd docker from Docker Hub by executing docker pull (sudo required):

$ sudo docker pull fluent/fluentd:edge-debian

The value fluent/fluentd specifies to download fluentd image and the value edge-debian means the latest version of Fluentd for the Debian distribution. You’ll see the program’s output appear on the screen:

edge-debian: Pulling from fluent/fluentd
69692152171a: Pull complete 
0a1f774fea41: Pull complete 
3b32b9ebaa42: Pull complete 
697d218c4f9a: Pull complete 
0a75b407d854: Pull complete 
e87327430399: Pull complete 
a0dbcaaf3fa1: Pull complete 
cb72b6c0baf0: Pull complete 
118fabf661ff: Pull complete 
Digest: sha256:88cc008454712ef0ec51013614031c9917c10be3005f79ee143f1d2c322fd9e0
Status: Downloaded newer image for fluent/fluentd:edge-debian
docker.io/fluent/fluentd:edge-debian

After successful download of the image, you should create a Fluentd configuration file. You can create the directory fluentd in your home directory:

$ mkdir /home/alice/fluentd

The directory /home/alice/fluentd will store the Fluentd configuration (however, you can store this configuration anywhere in your system). Now, let's create the Fluentd configuration file fluentd.conf in this directory:

$ nano /home/alice/fluentd.conf

Fill the file with following configuration:

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match docker.*>
  @type stdout
</match>

The directive source defines where all the log data come from. The value forward provides a TCP endpoint to accept TCP packets (port and bind define TCP port and IP address). The match directive looks for events with matching tags and processes them. The value docker.* means that it processes the only log records with the tag docker.<name-of-the-docker>. The suboption @type specifies the output plugin to use (stdout - print received log to stdout, file - write it to the file, and others). Now, you can save the configuration file and close it.


Starting the Logging Server (Fluentd Docker)

If you downloaded the Fluentd docker and create configuration file than you can run the Fluentd container by executing docker run (sudo required):

$ sudo docker run -p 24224:24224 -v /home/alice/fluentd:/fluentd/etc fluent/fluentd:edge-debian -c /fluentd/etc/fluentd.conf

The option -p 24224:24224 maps port 24224 in the container to port 24224 on your computer. The -v flag mounts the directory /home/alice/fluentd into the container /fluentd/etc. Option -c belongs to the fluentd daemon, and it determines the path to the Fluentd configuration file fluentd.conf. You’ll see the program’s output appear on the screen:

fluentd -c /fluentd/etc/fluentd.conf
2021-06-10 07:08:25 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluentd.conf"
2021-06-10 07:08:25 +0000 [info]: gem 'fluentd' version '1.12.4'
2021-06-10 07:08:25 +0000 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2021-06-10 07:08:25 +0000 [info]: using configuration file: <ROOT>
<source>
    @type forward
    port 24224
    bind "0.0.0.0"
  </source>
  <match docker.*>
    @type stdout
  </match>
</ROOT>
2021-06-10 07:08:25 +0000 [info]: starting fluentd-1.12.4 pid=7 ruby="2.6.7"
2021-06-10 07:08:25 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/local/bundle/bin/fluentd", "-c", "/fluentd/etc/fluentd.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
2021-06-10 07:08:25 +0000 [info]: adding match pattern="*.*" type="stdout"
2021-06-10 07:08:25 +0000 [info]: adding source type="forward"
2021-06-10 07:08:25 +0000 [warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2021-06-10 07:08:25 +0000 [info]: #0 starting fluentd worker pid=16 ppid=7 worker=0
2021-06-10 07:08:25 +0000 [info]: #0 listening port port=24224 bind="0.0.0.0"

The output shows that Fluentd read the configuration file. Now the logging container is ready to receive logs from any client containers. The Fluentd is running in this terminal, you can open a new terminal where we start client docker. At last, we will return to the Fluentd terminal, and check whether it receives the logs from the client docker.


Starting the Logging Client (Web Server)

Next, open the new terminal, and let's start the Nginx web server, that we use with the json-file logger, but this time we will configure it as a Fluentd client container:

$ sudo docker run --log-driver=fluentd --log-opt fluentd-address=tcp://0.0.0.0:24224 -p 80:80/tcp nginxdemos/hello

We use following parameters:

  • --log-driver=fluentd: We set up the logging driver of this docket to the fluentd. In the previous step, we set up the logging driver in the configuration file /etc/docker/daemon.conf, but you can do the same with the command line parameters. In such a case, the command line parameter will override the configuration file with the higher priority.
  • --log-opt fluentd-address=tcp://0.0.0.0:24224: We configure the IP addresses and port of the Fluentd server. All logs captured in this docker will be sent to the TCP port 24224 to the IP address 0.0.0.0.
  • -p 80:80/tcp nginxdemos/hello: We map port 80 in the container to port 80 on your computer.

Now, the webserver is running. You can reload a few times http://localhost:80 in your browser to generate some log records. The output will generate the new log records in real-time to the console where you start the webserver:

Output
172.17.0.1 - - [10/Jun/2021:11:11:10 +0000] "GET / HTTP/1.1" 200 7231 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" "-"
172.17.0.1 - - [10/Jun/2021:11:11:12 +0000] "GET / HTTP/1.1" 200 7231 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" "-"
172.17.0.1 - - [10/Jun/2021:11:11:13 +0000] "GET / HTTP/1.1" 200 7231 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" "-"

The output shows the log records of web server routing. However, we configure this docker to send these log records to the Fluentd docker, soo let's look to the logger server-side.


Viewing Received Logs in Logging Server

At last, you can switch back to the terminal where you run the Fluentd docker. You will see that the tail of the log shows the following three lines:

Output
...
2021-06-10 11:11:10.000000000 +0000 docker.crazy_johnson: {"log":"172.17.0.1 - - [10/Jun/2021:11:11:10 +0000] \\"GET / HTTP/1.1\\" 200 7231 \\"-\\" \\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36\\" \\"-\\"","container_id":"b2564203be04c6c0b4d2e3150e41d227a219a2075846688db1778f550d2c73d1","container_name":"/crazy_johnson","source":"stdout"}
2021-06-10 11:11:12.000000000 +0000 docker.crazy_johnson: {"container_id":"b2564203be04c6c0b4d2e3150e41d227a219a2075846688db1778f550d2c73d1","container_name":"/crazy_johnson","source":"stdout","log":"172.17.0.1 - - [10/Jun/2021:11:11:12 +0000] \\"GET / HTTP/1.1\\" 200 7231 \\"-\\" \\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36\\" \\"-\\""}
2021-06-10 11:11:13.000000000 +0000 docker.crazy_johnson: {"source":"stdout","log":"172.17.0.1 - - [10/Jun/2021:11:11:13 +0000] \\"GET / HTTP/1.1\\" 200 7231 \\"-\\" \\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36\\" \\"-\\"","container_id":"b2564203be04c6c0b4d2e3150e41d227a219a2075846688db1778f550d2c73d1","container_name":"/crazy_johnson"}

The output shows that the server receives the log records from the client docker. You can see that they are wrapped by the metadata header generated by the Fluentd service. In addition to the log message itself, the fluentd log driver sends the following metadata in the structured log message:

  • container_id: The full 64-character container ID.
  • container_name: The client container name at the time it was started.
  • source: Could be the stdout or stderr.
  • log: The message generated by the client container.

Conclusion

In this tutorial, you configured the Docker logging driver for the local json-file logging. You installed a simple web application, generated some logs, and viewed them through the Docker CLI. Next, you configured a centralised Fluentd logging Docker.