By Juraj Holub
Last Validated on May 3 2021 · Originally Published on May 3, 2021 · Viewed 2.5k times

Introduction

The log files size must be controlled because their size always grows over time. Each system has limited resources, and too many logs lead to performance and memory problems. Linux solves this problem with a concept called logrotate. Logrotate is the system daemon that rotates, compresses, and mails system logs. This tutorial shows you:

  • Where are the general and application-specific log configuration files stored.
  • How to create standard logrotate configuration for fictional log of a custom app.
  • How to create system independent logrotate configuration for same log.

Prerequisites

You will need:


Step 1 — Viewing Logrotate Configuration

System logs on Ubuntu 20.04 with a default configuration are maintained by the logrotate daemon. This daemon uses configuration files that specify all rotation details for each application. The configuration consist of the following parts:

  • /etc/logrotate.conf is the most general configuration file with default setup.
  • /etc/logrotate.d is a directory that includes files for specific application rotation. Each such application has its own configuration file in this directory. Otherwise, application logs use general configuration.

We will view both configuration possibilities.


General Logrotate Configuration

As first, let's view more general configuration file /etc/logrotate.conf.

Print the content of file /etc/logrotate.conf with utility cat:

cat /etc/rsyslog.conf

The command prints the entire content of this configuration file:

Output
# see "man logrotate" for details
# rotate log files weekly
weekly

# use the adm group by default, since this is the owning group
# of /var/log/syslog.
su root adm

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
#dateext

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# system-specific logs may be also be configured here.

The output shows the global configuration that specifies following default setup:

  • weekly: Logs are rotated every week. Alternatively, you can specify another time interval (daily, weekly, monthly, or yearly). You can also specify to rotate every hour (hourly), but note that the logrotate daemon runs daily. In such a case, you have to change the logrotate cron job interval to be able to really rotate logs hourly.
  • su root adm: Log rotation is performed with root user and admin group.
  • rotate 4: Log files are rotated 4 times before being removed. If rotate is set to 0 then old versions are removed rather than rotated.
  • create: The new log file is created immediately after the rotation (with the same name as the log file just rotated).
  • compress: This rule determines if old log files are compressed. Since this rule is by default commented, the compression is turned off.
  • include: This specifies the directory for specific application rotation configurations. You can see that it is /etc/logrotate.d (as we already know).

Application Specific Logrotate Configuration

Now, let's view the directory /etc/logrotate.d that holds application specific logrotate configurations.

Let's view content of directory /etc/logrotate.d by executing ls:

ls /etc/logrotate.d/

The command lists all files in this directory into output:

Output
alternatives  bootlog      dpkg               ubuntu-advantage-tools
apache2       btmp         ppp                ufw
apport        certbot      rsyslog            unattended-upgrades
apt           cups-daemon  speech-dispatcher  wtmp

You can see that the rsyslog daemon contains its own logrotate configuration file.

Print first 15 lines of rsyslog configuration file with utility head:

head -n 15 /etc/logrotate.d/rsyslog

You’ll see the program’s output appear on the screen:

Output
/var/log/syslog
{
	rotate 7
	daily
	missingok
	notifempty
	delaycompress
	compress
	postrotate
		/usr/lib/rsyslog/rsyslog-rotate
	endscript
}

/var/log/mail.info
/var/log/mail.warn

The output shows that syslog file is rotated daily, keeps seven backups, and backups are compressed. There are also directives that we do not know yet:

  • missingok: Don't report any error if the log file is missing.
  • notifempty: Don't rotate the log if it is empty.
  • delaycompress: Postpone compression of the previous log file to the next rotation cycle, because it is still used by some program.
  • postrotate/endscript: The lines between postrotate and endscript are executed after the log file is rotated.

The logrotate configuration may include various other directives. For example, you can specify that it is mailed to a specific address if the log is rotated out of existence. Or you can set up that log files are rotated when they grow bigger than the specified size of bytes. All possible directives are described in logrotate manual pages (you can also see them by executing man logrotate).


Step 2 — Creating Fictional Log File For Logrotate

In this step, we will create a log file of a fictional custom application, which we will call my-custom-app. In the following steps, we will set up a logrotate configuration for this app's logs.

Create a new directory in /var/log with mkdir:

mkdir /var/log/my-custom-app

The mkdir creates new directory where we will create the custom log file.

Create the log file /var/log/my-custom-app/backup.log:

nano /var/log/my-custom-app/backup.log

Fill this file with some fictional logging data:

File /var/log/my-custom-app/backup.log
Apr  7 10:00:00 Fictional application start.
Apr  7 11:00:00 Fictional application error.
Apr  7 12:00:00 Fictional application exit.

The content of the file is not important for us, it serves as an example only.

Now, you are ready to configure logrotate for this log.


Step 3 — Creating Standard Logrotate Configuration

In this step, we will create standard logrotate job for some custom application that is not included in the default system services. After we set up a new logrotate configuration file, it will be executed by root every day together with all other standard system logrotate jobs. We will use a my-custom-app example from the previous step.

Create the file my-custom-app as a root in directory /etc/logrotate.d with your text editor:

sudo nano /etc/logrotate.d/my-custom-app

Add the following code to the file to setup logrotate configuration for this custom application:

File /etc/logrotate.d/my-custom-app
/var/log/my-custom-app/*.log {
    daily
    missingok
    rotate 7
    compress
    notifempty
}

We already know all directives in this logrotate configuration file, except the regular expression *.log. It determines that this configuration applies to all log files with the .log suffix. This is just an example, and you can set up your own values and directives.

Now, you can test new configuration by executing logrotate (sudo required):

sudo logrotate /etc/logrotate.conf --debug

The option --debug means that no changes are made to the logs and only debug messages are printed. The logrotate reads the configuration file /etc/logrotate.conf and prints the debug messages:

Output
...

rotating pattern: /var/log/my-custom-app/*.log  after 1 days (7 rotations)
empty log files are not rotated, old logs are removed
switching euid to 0 and egid to 4
considering log /var/log/my-custom-app/backup.log
Creating new state
  Now: 2021-04-07 12:27
  Last rotated at 2021-04-07 12:00
  log does not need rotating (log has been already rotated)
switching euid to 0 and egid to 0

...

This sub-message means that the configuration also includes our new custom configuration file /etc/logrotate.d/my-custom-app.

At this point, you set up new logrotate configuration. The logrotate will rotate also file /var/log/my-custom-app/backup.log.


Step 4 — Creating System Independent Logrotate Configuration

Alternatively, to the previous step, we can create a configuration that is system independent. Such a configuration is not included in the file /etc/logrotate.conf, so it won't be executed together with all the system logrotations every day. Instead, we will create a cron job that will execute our logrotate configuration file within a custom time period.

There are two reasons to configure the system independent configuration:

  • By default, logrotate executes rotation once per day. If you require to rotate more frequently, for example hourly, it cannot be done with the standard configuration.
  • The standard configuration cannot rotate logs as a non-root user.

Similarly to previous step, we will use a my-custom-app example.

Firstly, let's see who is owner of our fictional log with ls (option -l shows also access rights):

ls -l /var/log/my-custom-app/backup.log

You’ll see the program’s output appear on the screen:

Output
rw-r--r-- 1 alice alice 134 apr  7 12:24 my-custom-app/backup.log

The output shows that the owner of this log file is user alice. So we will create a logrotate configuration executable by this user instead of root (just to show that it is possible).

Our custom configuration will be placed in the home directory of user alice. Let's create it with your text editor:

nano /home/alice/logrotate.conf

Add the following code to the file to serve as the logrotate configuration setup for this custom application:

File /home/alice/logrotate.conf
/var/log/my-custom-app/*.log {
    hourly
    missingok
    rotate 7
    compress
    notifempty
}

We define that this configuration is applied to all files in directory /var/log/my-custom-app that end with suffix .log. However, such a custom log can be stored in any directory if it is accessible by alice user. Note that the file set up to logrotate every hour, which is not possible in standard configuration. The rest of file is just an example, and you can set up your own values and directives.

The logrotate still doesn't know about this new configuration because it is not included in /etc/logrotate.conf. Instead of including configuration there, we will use a logrotate state file. This file records what the logrotate did last time when it ran. Logrotate uses it to know what to do next time. By default, logrotate uses the state file /var/lib/logrotate/status but we will create our custom state file.

Create a custom log file in the home directory of user alice by executing logrotate:

logrotate /home/alice/logrotate.conf --state /home/alice/custom-state

The --state option tells logrotate to use an alternate state file /home/alice/custom-state. The logrotate creates this file and you can view its contents with cat:

cat /home/alice/custom-state

You’ll see the program’s output appear on the screen:

Output
logrotate state -- version 2
"/var/log/my-custom-app/backup.log" 2021-4-7-15:0:0

The output shows that logrotate saw log /var/log/my-custom-app/backup.log. Notice that we don't need the root access to run logrotate. However, it won't run again unless we run it by hand. So at last, we will create a cron job that will execute it periodically.

Open cron jobs configuration file by executing crontab:

crontab -e

The -e option is used to edit the  current cron jobs using the editor specified by the VISUAL or EDITOR  environment variables. The command opens a configuration file in your text editor specified by these variables. Go to the bottom of the file add a new line with the following content:

File crontab

...
0 * * * * /usr/sbin/logrotate /home/alice/logrotate.conf --state /home/alice/custom-state

The new line specifies that this cron job will be executed every whole hour (at minute 0). The cron job will execute logrotate with our custom state file. Save the modified file, and you will see the following output:

Output
crontab: installing new crontab

Now, your custom system independent logrotate configuration is set up. You can view the log file after an hour that it rotates. For more details about cron jobs see the following tutorial.


Conclusion

In this tutorial, you viewed the logrotate configuration files. You created a standard logrotate configuration for the custom application. Also, you created a system-independent logrotate configuration for a custom application.