By Juraj Holub
Last Validated on May 4 2021 · Originally Published on May 4, 2021 · Viewed 1.9k times

Introduction

This tutorial shows you how to configure rsyslog as a remote Client-Server service. The remote rsyslog configuration is beneficial when you administrate logs from multiple servers. In such case, you can configure a centralised rsyslog server that will receive logs from all client servers. Each client will send logs to this centralised server. This approach makes logs administration easier.

In this tutorial you will learn to do the following:

  • How to configure rsyslog for centralised server (rsyslog server) that will receive all logs.
  • How to setup rsyslog for another server (rsyslog client) to send all logs to centralised server.

Prerequisites

You will need:

  • You must own two Ubuntu 20.04 servers that are in a single private IP network. The traffic between server won't be encrypted, and everyone with access to the Internet network will be able to read your logs. However, if your servers are in the private network then you encapsulate this traffic from the rest of the Internet.
  • Both servers must be configured.
  • You will require the non-root user with sudo access for both servers.

Step 1 — Finding Private IP Addresses Of Your Servers

First of all, you need to find out the private IPv4 addresses of your servers. In this tutorial we will demonstrate the entire configuration with the following servers:

  • Rsyslog server: Centralised rsyslog server with IP address 0.0.0.0.
  • Rsyslog client: Service server with IP address 1.1.1.1.

Firstly, you must connect to each server through ssh:

ssh alice@rsyslog_server

When you connect to your server, you can determine real IP address by executing ip:

ip a

The parameter a shows IP addresses on device:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 96:00:00:a3:a2:b1 brd ff:ff:ff:ff:ff:ff
    inet 157.90.246.56/32 scope global dynamic eth0
       valid_lft 85961sec preferred_lft 85961sec
    inet6 2a01:4f8:c010:5f07::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::9400:ff:fea3:a2b1/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 96:00:00:a3:a2:b1 brd ff:ff:ff:ff:ff:ff
    inet 0.0.0.0/28 scope global dynamic eth0
       valid_lft 85961sec preferred_lft 85961sec
    inet6 2a01:4f8:c010:5f07::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::9400:ff:fea3:a2b1/64 scope link 
       valid_lft forever preferred_lft forever

The output is divided into sections, where each section defines one interface. Usually, the interface eth0 is the default connection to the Internet. However, we will use eth1, which is an interface to our private network. The line with the directive inet defines IP address. You can see that in our case, it is filled with IP address 0.0.0.0 as an example.

You must connect to both servers and find out their IP addresses before you start the actual rsyslog configuration.


Step 2 — Configuring Centralised Server To Receive Logs

In this section, we will configure the rsyslog server to receive logs from clients. First of all, you must connect to this server through ssh. Ubuntu 20.04 has the rsyslog daemon preinstalled.


Opening TCP Port

Let's open file /etc/rsyslog.conf where the rsyslog configuration is stored (you need sudo edit permission):

sudo nano /etc/rsyslog.conf

The file has following content:

# /etc/rsyslog.conf configuration file for rsyslog
#
# For more information install rsyslog-doc and see
# /usr/share/doc/rsyslog-doc/html/configuration/index.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf

#################
#### MODULES ####
#################

module(load="imuxsock") # provides support for local system logging
#module(load="immark")  # provides --MARK-- message capability

# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")

# provides kernel logging support and enable non-kernel klog messages
module(load="imklog" permitnonkernelfacility="on")

###########################
#### GLOBAL DIRECTIVES ####
###########################

...

The file is pre-prepared for rsyslog centralised server configuration. Just find the following lines that are commented out:

# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")

This lines of code loads modules imudp and imtcp (both at the port 514):

  • Input Model TCP: Rsyslog starts listening at TCP port 514.
  • Input Model UDP: Rsyslog starts listening at TCP port 514.

You can define to listen to UDP, TCP, or both ports. Also, you can change the port number. In our example, we will use a TCP connection. Let's enable it by uncommenting the corresponding two lines of code:

# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

Save changes in file /etc/rsyslog.conf.

If your system includes firewall configuration, you must also allow new TCP service with ufw:

sudo ufw allow 514/tcp

The ufw require sudo because it changes firewall rules. Now, firewall won't filter rsyslog traffic.

Now, you can restart rsyslog to apply the new configuration:

sudo systemctl restart rsyslog.service

Lastly, you can check that the TCP port 514 is open with utilities ss (sudo required because this utility read sockets):

sudo ss -tulnp | grep "rsyslog"

The ss parameters change output:

  • -t: Display TCP sockets.
  • -u: Display UDP sockets.
  • -l: Display only listening sockets.
  • -n: Don't convert port number into human readable service name.
  • -p: Display process that uses socket.

The output of ss is filtered by grep, which prints only lines with the rsyslog service name:

tcp    LISTEN  0       25                  0.0.0.0:514            0.0.0.0:*      users:(("rsyslogd",pid=440497,fd=5))                                           
tcp    LISTEN  0       25                     [::]:514               [::]:*      users:(("rsyslogd",pid=440497,fd=6))

The output shows that rsyslog already listens at the TCP port 514.


Creating Remote Logs Template

By default, all logs received from the TCP port 512 will be merged in directory /var/log with the file of the same name. This is typically unwanted behaviour because it messes all the logs together. To prevent this behaviour, you can edit the rsyslog configuration.

Once again, let's open file /etc/rsyslog.conf where the rsyslog configuration is stored (you need sudo for edit permission), and add there following lines at the beginning of the file:

template RemoteLogs,"/var/log/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs
& ~

The directive template RemoteLogs sets up rsyslog to store all received remote logs in the format that is defined by the third parameter. In our case, the remote logs will be stored in /var/log directory, but each remote server will have its own subdirectory with name equivalent to the server hostname. This subdirectory will store each log in the file with the name equivalent to the client program name.

The directive *.* ?RemoteLogs defines to apply the RemoteLogs configuration rule to all facilities with all priority levels (in other words, to all logs).

The directive & ~ defines that rsyslog stops processing input log after it is stored to a file defined in the previous lines. Without this line, the previous new rule will be overwritten by the default configuration.

The file /etc/rsyslog.conf now looks like this:


# /etc/rsyslog.conf configuration file for rsyslog
#
# For more information install rsyslog-doc and see
# /usr/share/doc/rsyslog-doc/html/configuration/index.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf

$template RemoteLogs,"/var/log/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs
& ~

#################
#### MODULES ####
#################

module(load="imuxsock") # provides support for local system logging
#module(load="immark")  # provides --MARK-- message capability

# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

# provides kernel logging support and enable non-kernel klog messages
module(load="imklog" permitnonkernelfacility="on")

...

Save the file and restart the rsyslog service with systemctl (sudo required):

sudo systemctl restart rsyslog.service

The rsyslog server is now fully configured to receive remote logs from rsyslog clients.


Step 3 — Configuring Client To Send Logs

In this section, we will configure the rsyslog client to send logs to servers. First of all, you must connect to your client server through ssh.

Open file /etc/rsyslog.d/50-default.conf with default rsyslog configuration (sudo required for write permission):

sudo nano /etc/rsyslog.d/50-default.conf

At directive *.* @@0.0.0.0:514 at the new line to the beginning of this file:

#  Default rules for rsyslog.
#
#                       For more information see rsyslog.conf(5) and /etc/rsyslog.conf

*.* @@0.0.0.0:514

#
# First some standard log files.  Log by facility.
#
auth,authpriv.*                 /var/log/auth.log
*.*;auth,authpriv.none          -/var/log/syslog
#cron.*                         /var/log/cron.log
#daemon.*                       -/var/log/daemon.log
kern.*                          -/var/log/kern.log
#lpr.*                          -/var/log/lpr.log
mail.*                          -/var/log/mail.log
#user.*                         -/var/log/user.log

The directive *.* @@0.0.0.0:514 defines that rsyslog sends all the facilities with all the priority levels (in other words, all logs) to IP address 0.0.0.0 (IP address of the centralised server) to the TCP port 514. If you set up singe @ instead of @@ then it is UDP port.

Save changes in file /etc/rsyslog.d/50-default.conf and restart rsyslog service to apply new configuration (restart requires sudo permission):

sudo systemctl restart rsyslog.service

The rsyslog client is now fully configured to send remote logs to the rsyslog server.


Step 4 (Optional) — Viewing Centralised Remote Logs

If you configure both sides of the Client-Server rsyslog service then you can view all remote logs at the rsyslog server.

Let's view content of /var/log directory:

ls /var/log

You’ll see the program’s output appear on the screen:

alternatives.log       boot.log.7              kern.log.3.gz
alternatives.log.1     bootstrap.log           kern.log.4.gz
alternatives.log.2.gz  btmp                    lastlog
alternatives.log.3.gz  btmp.1                  letsencrypt
apache2                cups                    my-custom-app
apport.log             dist-upgrade            openvpn
apport.log.1           dmesg                   private
apport.log.2.gz        dmesg.0                 speech-dispatcher
apport.log.3.gz        dmesg.1.gz              syslog
apport.log.4.gz        dmesg.2.gz              syslog.1
apport.log.5.gz        dmesg.3.gz              syslog.2.gz
apport.log.6.gz        dpkg.log                syslog.3.gz
apport.log.7.gz        dpkg.log.1              syslog.4.gz
apt                    dpkg.log.2.gz           syslog.5.gz
auth.log               dpkg.log.3.gz           syslog.6.gz
auth.log.1             faillog                 syslog.7.gz
auth.log.2.gz          fontconfig.log          teamviewer15
auth.log.3.gz          gdm3                    test.log
auth.log.4.gz          gpu-manager.log         ubuntu-advantage.log
boot.log               gpu-manager-switch.log  unattended-upgrades
boot.log.1             hp                      wtmp
boot.log.2             ip-0.0.0.1              wtmp.1
boot.log.3             journal                 Xorg.0.log
boot.log.4             kern.log                Xorg.0.log.old
boot.log.5             kern.log.1              Xorg.1.log
boot.log.6             kern.log.2.gz           Xorg.1.log.old

The output shows directory ip-0.0.0.1, which is the directory with a remote log revived from a client with this IP address.

You can list content of this directory (sudo required because logs are maintained by root):

sudo ls /var/log/ip-0.0.0.1

You’ll see the program’s output appear on the screen:

kernel.log  rsyslogd.log  sshd.log  sudo.log  systemd-logind.log  systemd.log

The output shows that there are basic system logs. Now you can administrate all logs from rsyslog clients at one centralised rsyslog server.


Conclusion

In this tutorial, you configured the rsyslog Client-Server service. You opened the TCP service at the server-side that receives remote logs. Also, you created a rsyslog rule that records remote logs in a more readable structure. Next, you configured the rsyslog client to send logs to the server. At last, you check that the new service is working properly.