By Juraj Holub
Last Validated on April 30 2021 · Originally Published on April 30, 2021 · Viewed 1.1k times

Introduction

This tutorial explains the administration of the Linux system logs. A system log is a file that contains information about events that happened on the system during runtime. After reading this tutorial you will be confident in the following Linux logging basics:

  • Where are the Linux log files stored, how are they formatted, and how to read them.
  • How to read the most important logs including syslog.
  • How to view Ubuntu syslog daemon configuration.
  • How to configure Linux rotation log files, and execute log rotation.

Prerequisites

You will need:

  • Basic knowledge of how to work with Linux command line.
  • Ubuntu 20.04 distribution including the non-root user with sudo access.

Step 1 — Finding Linux System Logs

The Ubuntu system stores all system logs into directory /var/log. Let's look at the most essential log files, their meaning, and what they store:

  • /var/log/syslog stores all records about any global activity in the system. Basically, every event that happens goes into this log.
  • /var/log/auth.log stores all security-related actions. Those include login, logout, and root user activity.
  • /var/log/kern.log stores events of the kernel like info, warning, and errors. Those logs are especially useful for custom kernels.
  • /var/log/boot.log stores system startup info.

The /var/log directory also contains various application logs. For example, if your distribution contains Apache or MySQL then their logs are also stored there.

You can view content of this directory with the following command:

ls /var/log

The ls utility list contents of the /var/log directory into your terminal:

Output
alternatives.log    cups                    openvpn
alternatives.log.1  dist-upgrade            private
apache2             dmesg                   speech-dispatcher
apport.log          dmesg.0                 syslog
apport.log.1        dmesg.1.gz              syslog.1
apt                 dpkg.log                syslog.2.gz
auth.log            dpkg.log.1              syslog.3.gz
auth.log.1          faillog                 syslog.4.gz
auth.log.2.gz       fontconfig.log          syslog.5.gz
boot.log            gdm3                    syslog.6.gz
boot.log.1          gpu-manager.log         teamviewer15
boot.log.2          gpu-manager-switch.log  ubuntu-advantage.log
boot.log.3          hp                      unattended-upgrades
boot.log.4          installer               wtmp
boot.log.5          journal                 Xorg.0.log
boot.log.6          kern.log                Xorg.0.log.old
boot.log.7          kern.log.1              Xorg.1.log
bootstrap.log       kern.log.2.gz           Xorg.1.log.old
btmp                lastlog
btmp.1              letsencrypt

The output shows that there is a lot of various log files with several different suffixes. We will examine their meaning later in the tutorial. For now you should recognize files syslog, auth.log, boot.log and kern.log because we already know them. These system logs that are on every Ubuntu distribution.

Since the command was executed on Ubuntu with installed Apache application, you can see that there is also a directory called apache2 which holds log files for this application.


Step 2 — Viewing Log File Contents

The log files contain a large amount of various information that can be helpful for analysing activities like finding vulnerabilities, system optimisation, monitoring of the system, network, or server, and others.

The log files content must be formatted in a well-known structure for portability reasons. Let’s review the two basic log file formats: plain text and binary.


Plaintext Log Files

These logs are plain text files with the standardized content format. Ubuntu uses the log template called RSYSLOG_TraditionalFileFormat (see templates documentation). This log format consists of four main fields with a space delimiter:

  1. Timestamp: Indicates the time formatted as MMM dd HH:mm:ss (e.g. Sep 28 19:00:00). Note that this format does not include a year.
  2. Hostname: Is the name of the host or system that originally created the message.
  3. Application: Is the name of the application that created the message.
  4. Message: Contains the information. It can also be formatted.

Now, let's review some log file with this format. Print the contents of file /var/log/syslog with utility tail:

tail /var/syslog

The command prints the last 10 lines of this log file:

Output
Mar 23 12:38:09 peter dbus-daemon[1757]: [session uid=1000 pid=1757] Activating via systemd: service name='org.freedesktop.Tracker1' unit='tracker-store.service' requested by ':1.1' (uid=1000 pid=1754 comm="/usr/libexec/tracker-miner-fs " label="unconfined")
Mar 23 12:38:09 peter systemd[1743]: Starting Tracker metadata database store and lookup manager...
Mar 23 12:38:09 peter dbus-daemon[1757]: [session uid=1000 pid=1757] Successfully activated service 'org.freedesktop.Tracker1'
Mar 23 12:38:09 peter systemd[1743]: Started Tracker metadata database store and lookup manager.
Mar 23 12:38:40 peter tracker-store[359847]: OK
Mar 23 12:38:40 peter systemd[1743]: tracker-store.service: Succeeded.
Mar 23 12:39:01 peter CRON[359873]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Mar 23 12:39:23 peter systemd[1]: Starting Clean php session files...
Mar 23 12:39:23 peter systemd[1]: phpsessionclean.service: Succeeded.
Mar 23 12:39:23 peter systemd[1]: Finished Clean php session files.

The output shows that records of this file are formatted as described earlier. For example, the last record contains timestamp Mar 23 12:39:23, hostname peter, application systemd[1] and message Finished Clean php session files.

Binary Log Files

Those are another type of log files that cannot be read with a regular text editor. The directory /var/log contains multiple binary files that are related to users authorization:

  • /var/log/utmp stores users currently logged in the system.
  • /var/log/wtmp stores users previously logged in. It contains the past data from utmp.
  • /var/log/btmp stores failed login attempts.

For these binary logs, the system offers a special command-line tools that show you the relevant information in human-readable form.

To review the /var/log/utmp, run the utility who with option -H (this option shows output as a table and includes column names):

who -H

You’ll see the program’s output appear on the screen:

Output
NAME   LINE         TIME             COMMENT
george pts/0        2021-03-21 15:29 (2001:67c:1220:80c:b1:a84e:69ee:f530)
willie pts/1        2021-03-21 07:20 (adsl-dyn22.78-98-29.t-com.sk)
bonnie pts/2        2021-03-21 10:31 (2001:67c:1220:80c:b1:a84e:69ee:f530)
peter  pts/6        2021-03-21 14:37 (100.64.97.50)
...

The output shows system users that are currently logged in as well as their timestamp and IP address.

To review the /var/log/wtmp, run the utility last with option -R (this option will suppress the hostname column that is not important for us):

last -R

You’ll see the program’s output appear on the screen:

Output
peter    :1           Sat Mar 13 08:06   still logged in
reboot   system boot  Sat Mar 13 08:06   still running
peter    :1           Fri Mar 12 07:42 - down  (1+00:22)
reboot   system boot  Fri Mar 12 07:42 - 08:05 (1+00:23)
peter    :1           Sun Mar  7 11:20 - down  (4+20:21)
reboot   system boot  Sun Mar  7 11:20 - 07:41 (4+20:21)
peter    :1           Fri Mar  5 08:02 - crash (2+03:17)
reboot   system boot  Fri Mar  5 08:01 - 07:41 (6+23:39)
peter    :0           Tue Mar  2 08:38 - crash (2+23:23)
reboot   system boot  Tue Mar  2 08:38 - 07:41 (9+23:03)
peter    :1           Thu Feb 25 11:44 - down  (4+20:53)
reboot   system boot  Thu Feb 25 11:44 - 08:37 (4+20:53)

wtmp begins Thu Feb 25 11:43:23 2021

The output shows a table where the first column refers to the user name (the pseudo-user reboot is recorded each time when the system is rebooted). The third field refers to the login timestamp, and the last column shows the entire session duration.

To review the /var/log/btmp, run the utility lastb with a sudo privilege:

sudo lastb

You’ll see the program’s output appear on the screen:

Output
falcon   tty3                  Thu Feb 12 07:10 - 07:10  (00:00)    
ruby     tty1                  Thu Feb 12 07:09 - 07:09  (00:00)    
sergio   tty1                  Thu Feb 12 07:09 - 07:09  (00:00)    

btmp begins Thu Feb 25 11:43:32 2021

The output shows users that failed to login with the corresponding timestamp.


Step 3 — Viewing Syslog Deamon Configuration

All logs are created and maintained by a background program called daemon. The traditional Linux daemon for logging is syslogd. However, Ubuntu 20.04 distribution uses a daemon called rsyslogd. Rsyslogd means rocket-fast system for log processing, and it is a superset of basic syslogd. This daemon uses a special configuration file /etc/rsyslog.conf that specifies the logging rules.

Print the content of file /etc/rsyslog.conf with utility cat:

cat /etc/rsyslog.conf

The command prints entire content of the configuration file:

Output
...

#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Filter duplicated messages
$RepeatedMsgReduction on

#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog

#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf

The output shows a lot of information, but we focus on two configuration details. Firstly, there is a variable $ActionFileDefaultTemplate that defines syslog record format, and we already know this format from Step 2. We can change this variable if this format is not suitable for us.

Secondly, the output contains a special variable $IncludeConfig that specifies the directory for all configuration files.

On Ubuntu, all rsyslog rules are by default in the /etc/rsyslog.d/50-default.conf file. Print the content of this file with the utility head (option -n 15 specifies to print first 15 lines of a given file):

head -n 15 /etc/rsyslog.conf/50-default.conf

You’ll see the program’s output appear on the screen:

Output
#  Default rules for rsyslog.
#
#			For more information see rsyslog.conf(5) and /etc/rsyslog.conf

#
# First some standard log files.  Log by facility.
#
auth,authpriv.*			/var/log/auth.log
*.*;auth,authpriv.none		-/var/log/syslog
#cron.*				/var/log/cron.log
#daemon.*			-/var/log/daemon.log
kern.*				-/var/log/kern.log
#lpr.*				-/var/log/lpr.log
mail.*				-/var/log/mail.log
#user.*				-/var/log/user.log

The output contains the rsyslogd configuration rules. Each non-empty line (or line that does not start with char #) defines a rule. The rule definition starts with a selector followed by one or more spaces and an action field:

  • The selector specifies the facility with corresponding priority. For example, in our output, the * refers to all facilities or all priorities.
  • The action usually refers to a log file. For example, in our output, the selector * refers to the /var/log/syslog file that we already know.

Step 4 — Viewing Rotating Log Files

The size of log files must be controlled because it keeps growing over time. Each system has limited resources, and too large logs lead to performance and memory problems. Linux solves this problem with a concept called rotating logs. Rotation continuously repeats the following actions:

  1. Instead of deleting the log file, Linux changes the file name to a name with a version suffix and creates a new one. The log file has multiple old back-ups. Optionally, these backups are compressed.
  2. When the count of backup files reaches a specified number, the system deletes the oldest.

Let's view an example of Linux rotating files. Execute ls command with following options:

ls -l -h -t /var/log/syslog*

The -l option formats the output in a long listing including various metadata. The -h option prints file in a human-readable form, and the -t option sorts the output listing by modification time (newest first). The parameter /var/log/syslog* specifies to view all files at directory /var/log that starts with prefix syslog.

You’ll see the program’s output appear on the screen:

Output
-rw-r----- 1 syslog adm  47K mar 30 09:49 /var/log/syslog
-rw-r----- 1 syslog adm 3,5G mar 30 07:45 /var/log/syslog.1.gz
-rw-r----- 1 syslog adm 1,6M mar 29 10:06 /var/log/syslog.2.gz
-rw-r----- 1 syslog adm  29K mar 28 07:49 /var/log/syslog.3.gz
-rw-r----- 1 syslog adm  54K mar 27 08:08 /var/log/syslog.4.gz
-rw-r----- 1 syslog adm 6,4M mar 26 07:35 /var/log/syslog.5.gz
-rw-r----- 1 syslog adm  31K mar 25 08:01 /var/log/syslog.6.gz

The output shows all version of syslog. Typically, it is the biggest log file because as explained earlier, this log stores basically every event of the system. The older versions are labelled with version suffix (e.g. syslog.6.gz is the oldest syslog backup).

You can see that the backups are compressed (the gz suffix means that the file is compressed by the standard GNU zip compression algorithm). As you can observe, these files grow to the size of gigabytes (in our example the biggest is 3.5 GB large). At last, you can view that these files cover only a time interval of six days.


Step 5 — Running Logrotate Daemon

The log rotation is maintained by the system daemon logrotate. Similarly to rsyslogd, the daemon uses a special configuration file /etc/logrotate.conf.

Print the content of file /etc/logrotate.conf with utility cat:

cat /etc/logrotate.conf

The command prints the entire content of this configuration file:

Output
# see "man logrotate" for details
# rotate log files weekly
weekly

# use the adm group by default, since this is the owning group
# of /var/log/syslog.
su root adm

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
#dateext

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# system-specific logs may be also be configured here.

The output shows the global configuration. The contents are quite self-explaining. In our example, the log files are rotated weekly, and the system keeps four rotation backlogs, and the compression is turned off. However, this is the most general configuration for any log, and we can set up a more specific configuration for a specific log.

In Ubuntu, the specific logs are by default configured in directory /etc/logrotate.d (you can see that this directory is included in logrotate.conf).

Let's view content of /etc/logrotate.d by executing ls:

ls /etc/logrotate.d/

The command lists all files into output:

Output
alternatives  bootlog      dpkg               ubuntu-advantage-tools
apache2       btmp         ppp                ufw
apport        certbot      rsyslog            unattended-upgrades
apt           cups-daemon  speech-dispatcher  wtmp

You can see that rsyslog daemon contains its own logrotate configuration file.

Print first 15 lines of rsyslog configuration file with utility head:

head -n 15 /etc/logrotate.d/rsyslog

You’ll see the program’s output appear on the screen:

Output
/var/log/syslog
{
	rotate 7
	daily
	missingok
	notifempty
	delaycompress
	compress
	postrotate
		/usr/lib/rsyslog/rsyslog-rotate
	endscript
}

/var/log/mail.info
/var/log/mail.warn

The output shows that syslog file is rotated daily, keeps seven backups, and backups are compressed.

You can force logrotate to execute rotation immediately by following command (sudo access required):

sudo logrotate -fv /etc/logrotate.conf

The option f force immediate rotation, and the option v turns on verbose mode (it will display messages during rotation). The execution of this command shows the following output:

Output
reading config file /etc/logrotate.conf
including /etc/logrotate.d
reading config file alternatives
reading config file apache2
reading config file apport
reading config file apt
reading config file bootlog
reading config file btmp
reading config file certbot
reading config file cups-daemon
reading config file dpkg
reading config file ppp
reading config file rsyslog
reading config file speech-dispatcher
reading config file ubuntu-advantage-tools
reading config file ufw
reading config file unattended-upgrades
reading config file wtmp
...

The beginning of the output shows that logrotate daemon firstly reads all configuration files. The entire output is very long because it informs about every detail of the rotation progress.


Conclusion

In this tutorial, you viewed the most important Linux logs with examples on syslog. You explored the structure of plain text log files, binary log file, and you opened and checked system configuration for daemon rsyslogd that maintains logs on Ubuntu. Apart from that you also saw a configuration of the logrotate daemon, and you executed log rotation.