By Juraj Holub
Last Validated on May 3 2021 · Originally Published on May 3, 2021 · Viewed 1k times

Introduction

This tutorial shows you how to control the daemon journald with utility journalctl. The journald is a service included in systems that controls all logs.

In this tutorials you will query journald with journalctl by following categories:

  • By boot session
  • By time period
  • By priority
  • By systemd unit service

Prerequisites

You will need:


Step 1 — Quering Journal With Journalctl

The systemd uses a centralised logging approach called a journal. The journal manages daemon journald. This daemon controls all logs generated by the kernel, or any system unit service. You can access, view, or filter journal with journactl utility.

Let's view the last 10 records in the journal with journalctl:

journalctl -n 10

Option -n 10 shows only the last 10 records. You’ll see the program’s output appear on the screen:

Output
-- Logs begin at Thu 2021-03-25 18:44:20 CET, end at Thu 2021-04-15 17:30:14 CEST. --
apr 15 17:27:14 alice gnome-shell[2083]: Window manager warning: Overwriting existing binding of keysym 35 with keysym 35 (keyc>
apr 15 17:27:14 alice gnome-shell[2083]: Window manager warning: Overwriting existing binding of keysym 36 with keysym 36 (keyc>
apr 15 17:27:14 alice gnome-shell[2083]: Window manager warning: Overwriting existing binding of keysym 37 with keysym 37 (keyc>
apr 15 17:30:01 alice CRON[15590]: pam_unix(cron:session): session opened for user root by (uid=0)
apr 15 17:30:01 alice CRON[15591]: (root) CMD ([ -x /etc/init.d/anacron ] && if [ ! -d /run/systemd/system ]; then /usr/sbin/in>
apr 15 17:30:01 alice CRON[15590]: pam_unix(cron:session): session closed for user root
apr 15 17:30:14 alice systemd[1]: Started Run anacron jobs.
apr 15 17:30:14 alice anacron[15595]: Anacron 2.3 started on 2021-04-15
apr 15 17:30:14 alice systemd[1]: anacron.service: Succeeded.
apr 15 17:30:14 alice anacron[15595]: Normal exit (0 jobs run)

The output shows the last 10 lines of file /var/log/syslog.

You can view the same 10 records with tail (if you execute this command immediately after journalctl -n 10):

tail -n 10 /var/log/syslog

The utility tail with option -n 10 shows last 10 lines of file /var/log/syslog:

Output
-- Logs begin at Thu 2021-03-25 18:44:20 CET, end at Thu 2021-04-15 17:39:14 CEST. --
apr 15 17:30:14 alice systemd[1]: Started Run anacron jobs.
apr 15 17:30:14 alice anacron[15595]: Anacron 2.3 started on 2021-04-15
apr 15 17:30:14 alice systemd[1]: anacron.service: Succeeded.
apr 15 17:30:14 alice anacron[15595]: Normal exit (0 jobs run)
apr 15 17:39:01 alice CRON[15782]: pam_unix(cron:session): session opened for user root by (uid=0)
apr 15 17:39:01 alice CRON[15783]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr>
apr 15 17:39:01 alice CRON[15782]: pam_unix(cron:session): session closed for user root
apr 15 17:39:14 alice systemd[1]: Starting Clean php session files...
apr 15 17:39:14 alice systemd[1]: phpsessionclean.service: Succeeded.
apr 15 17:39:14 alice systemd[1]: Finished Clean php session files.

The output is the same as with journalctl -n 10. However, the journactl is a much more advanced tool. Journalctl offers the possibility of complex querying that filters log records by multiple criteria.


Step 2 — Filtering By Boot Session

One of the most common journalctl querying is filtering messages from a specific boot session.

You can view all the logs collected since the last boot by executing journalctl with -b flag:

journalctl -b

The programs shows the output:

Output
-- Logs begin at Thu 2021-03-25 18:44:20 CET, end at Thu 2021-04-15 17:55:52 CEST. --
apr 15 14:02:32 alice kernel: Linux version 5.8.0-44-generic (buildd@lgw01-amd64-054) (gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0>
apr 15 14:02:32 alice kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-5.8.0-44-generic root=UUID=ed37bb52-d7d8-41eb-887b-38ab33c>
apr 15 14:02:32 alice kernel: KERNEL supported cpus:
apr 15 14:02:32 alice kernel:   Intel GenuineIntel
apr 15 14:02:32 alice kernel:   AMD AuthenticAMD
apr 15 14:02:32 alice kernel:   Hygon HygonGenuine
apr 15 14:02:32 alice kernel:   Centaur CentaurHauls
apr 15 14:02:32 alice kernel:   zhaoxin   Shanghai  
apr 15 14:02:32 alice kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
apr 15 14:02:32 alice kernel: x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
apr 15 14:02:32 alice kernel: x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
apr 15 14:02:32 alice kernel: x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers'
apr 15 14:02:32 alice kernel: x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR'
apr 15 14:02:32 alice kernel: x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
apr 15 14:02:32 alice kernel: x86/fpu: xstate_offset[3]:  832, xstate_sizes[3]:   64
apr 15 14:02:32 alice kernel: x86/fpu: xstate_offset[4]:  896, xstate_sizes[4]:   64
apr 15 14:02:32 alice kernel: x86/fpu: Enabled xstate features 0x1f, context size is 960 bytes, using 'compacted' format.
apr 15 14:02:32 alice kernel: BIOS-provided physical RAM map:
apr 15 14:02:32 alice kernel: BIOS-e820: [mem 0x0000000000000000-0x0000000000057fff] usable
apr 15 14:02:32 alice kernel: BIOS-e820: [mem 0x0000000000058000-0x0000000000058fff] reserved
apr 15 14:02:32 alice kernel: BIOS-e820: [mem 0x0000000000059000-0x000000000009dfff] usable
...

The output shows all records in chronological order. You can see that the first records show very low-level kernel information about system booting.

If you want to search for previous boot sessions, you can list all boots that journald knows about by executing journalctl with parameter --list-boots:

journalctl --list-boots

You’ll see the program’s output appear on the screen:

-3 9a8ebc63800e4b488b8b0fe90991600c Thu 2021-03-25 18:44:20 CET—Thu 2021-03-25 18:57:41 CET
-2 8a0a7c2d722d49f3ba3f411cf2344bb8 Thu 2021-03-25 18:58:11 CET—Wed 2021-04-14 16:00:31 CEST
-1 0f419686d8744067acd4e7ab962a280b Wed 2021-04-14 16:01:14 CEST—Thu 2021-04-15 14:02:09 CEST
 0 dbf7a43ac05f45e39be23091acf434bc Thu 2021-04-15 14:02:32 CEST—Thu 2021-04-15 18:00:31 CEST

The output shows all recorded boot with their time stamp and boot ID.

When you find a boot with a timestamp you are interested in, you can filter the journal by giving boot ID as a parameter to journalctl:

journalctl -b 0f419686d8744067acd4e7ab962a280b

The journalctl will show all logs within the boot session from 2021-04-14 16:01:14 to 2021-04-15 14:02:09.


Step 3 — Filtering By Message Priority

Journal records have a well-defined structure that also includes message priority. The journalctl allows filtering by these message priority. Each syslog message has one of the following eight priority levels (list is ordered in descending priority):

  1. emerg: System is unusable.
  2. alert: Action must be taken immediately.
  3. crit: Critical conditions.
  4. err: Error conditions.
  5. warning: Warning conditions.
  6. notice: Normal, but significant, condition.
  7. info: Informational message.
  8. debug: Debug-level message.

Let's take a simple example of filtering by priority. Our scenario is to check if someone attempts to take root access since the last reboot.

Firstly, let's simulate that user alice executes ls with root permissions:

sudo ls

When system ask you for password, pass three incorrect password attempts:

[sudo] password for alice: 
Sorry, try again.
[sudo] password for alice: 
Sorry, try again.
[sudo] password for alice: 
sudo: 3 incorrect password attempts

The journald records this incident with crit priority.

Now, you can find this incident by querying journalctl:

journalctl -b -p crit

Option -b show messages from the last boot, the option -p crit shows only messages with priority crit. You’ll see the program’s output appear on the screen:

Output
-- Logs begin at Thu 2021-03-25 18:44:20 CET, end at Thu 2021-04-15 12:17:23 CEST. --
apr 15 12:17:23 alice sudo[36679]:    alice : 3 incorrect password attempts ; TTY=pts/0 ; PWD=/home/alice ; USER=root ; COMMAND=/usr/bin/ls

The output show that syslog recorded our sudo incident.


Step 4 — Filtering By Systemd Service

If you are interested only in journal entries related to specific systemd unit service, you can use option -u with the name of the service.

Let's view the last 10 log records for rsyslog service by executing journalctl with -u:

journalctl -u rsyslog.service -n 10

Option -n 10 shows only the last 10 records. You’ll see the program’s output appear on the screen:

Output
-- Logs begin at Thu 2021-03-25 18:44:20 CET, end at Thu 2021-04-15 18:44:58 CEST. --
apr 15 14:02:07 alice systemd[1]: Stopping System Logging Service...
apr 15 14:02:07 alice rsyslogd[28793]: [origin software="rsyslogd" swVersion="8.2001.0" x-pid="28793" x-info="<https://www.rsyslog.com>"] exiting on signal 15.
apr 15 14:02:07 alice systemd[1]: rsyslog.service: Succeeded.
apr 15 14:02:07 alice systemd[1]: Stopped System Logging Service.
-- Reboot --
apr 15 14:02:36 alice systemd[1]: Starting System Logging Service...
apr 15 14:02:36 alice rsyslogd[1060]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.  [v8.2001.0]
apr 15 14:02:36 alice rsyslogd[1060]: rsyslogd's groupid changed to 110
apr 15 14:02:36 alice systemd[1]: Started System Logging Service.
apr 15 14:02:36 alice rsyslogd[1060]: rsyslogd's userid changed to 104
apr 15 14:02:36 alice rsyslogd[1060]: [origin software="rsyslogd" swVersion="8.2001.0" x-pid="1060" x-info="<https://www.rsyslog.com>"] start

You can see that the output shows only records related rsyslog.


Step 5 — Filtering By Time Period

Another typical use case is querying journal by specific time period. You can set a lower time limit with flag --since and an upper limit with --until, both followed by a timestamp. The timestamp can have a format that follows the systemd.time specification. For example:

  • 2021-11-23 23:02:15
  • 2021-05-04
  • 12:00
  • 5 hour ago, or 32 min ago
  • yesterday, today, now

Let's view today journal records since 6 pm until now:

journalctl --since 18:00 --until "now"

You’ll see the program’s output appear on the screen:

Output
-- Logs begin at Thu 2021-03-25 18:44:20 CET, end at Thu 2021-04-15 19:09:14 CEST. --
apr 15 18:00:01 alice CRON[16430]: pam_unix(cron:session): session opened for user alice by (uid=0)
apr 15 18:00:01 alice CRON[16431]: (alice) CMD (/usr/sbin/logrotate /home/alice/logrotate.conf --state /home/alice/custom-state)
apr 15 18:00:01 alice CRON[16430]: pam_unix(cron:session): session closed for user alice
apr 15 18:00:01 alice dbus-daemon[1814]: [session uid=1000 pid=1814] Activating via systemd: service name='org.freedesktop.Tracker1' unit='tracker-store.service' requested by ':1.1' (uid=1000 pid=1811 comm="/usr/libexec/tracker-miner-fs " label="unconfined")
apr 15 18:00:01 alice systemd[1800]: Starting Tracker metadata database store and lookup manager...
apr 15 18:00:01 alice dbus-daemon[1814]: [session uid=1000 pid=1814] Successfully activated service 'org.freedesktop.Tracker1'
apr 15 18:00:01 alice systemd[1800]: Started Tracker metadata database store and lookup manager.

...

apr 15 19:09:01 alice CRON[19653]: pam_unix(cron:session): session closed for user root
apr 15 19:09:14 alice systemd[1]: Starting Clean php session files...
apr 15 19:09:14 alice systemd[1]: phpsessionclean.service: Succeeded.
apr 15 19:09:14 alice systemd[1]: Finished Clean php session files.

The output shows a lot of journal records but you can see that they are ordered in time, and all of them are in the queried time period.


Conclusion

In this tutorial, you worked with system daemon journald. You filter the journal with the journalctl tool by various attributes (boot session, time period, priority, and systemd unit service).