INFO-3182- Operating systems - Linux

Module 18a Logging

Introduction

Operating systems and applications need to record information on their operations for security, activity,
and error monitoring and troubleshooting purposes.

There are two logging services available on modern Linux systems, systemd journal and syslog. As its
name implies the systemd journal is available on systems running systemd. The syslog service is
available on all Linux systems. CentOS Stream 9 and Ubuntu 24.04 have both logging services.

The systemd journal log functionality is available much earlier in the system boot process than syslog.
Therefore, systemd journal captures more information about a system’s startup and low-level system
operations than syslog does. Messages sent to the syslog service are also sent to the systemd journal.

One key difference between systemd journal and syslog is that syslog logs are always stored on disk, but
systemd journal data is often only cached in a temporary file and is lost on system reboot, and possibly
sooner if the cache fills, in which case the oldest entries are removed to make room for newer ones.

Not all logged information passes through systemd journal or syslog. For example, web server services
usually have their own logging functionality for web activity and use systemd journal and/or syslog for
other logging. In the case of web activity, the World Wide Web Consortium (W3C) has defined standard
formats for web activity logs that do not match the formats used by either systemd journal or syslog.
These standard formats from W3C are designed to help in traffic analysis for statistical purposes.

Linux log message priorities

All log messages sent to systemd journal or syslog have priorities (severity levels) assigned to them. Both
systemd journal and syslog make use of these priorities. The priority of each log message is determined
by the originator of the message. Log priorities are often referred to as log levels or priority levels.

There are ten priority levels define as shown in the table below. The last two listed are only used in the
kernel.

Page 1 of 14
 INFO-3182- Operating systems - Linux

Priority Label Name Description Kernel

Level Constant
Name
0 emerg Emergency
Messages related to emergency conditions. KERN_EMERG
Messages of this priority usually indicate
imminent system failure 9(crash)
1 alert Alert Messages of this priority indicate error KERN_ALERT
conditions that require immediate attention.
2 crit Critical Messages of this priority indicate serious KERN_CRIT
software or hardware errors.
3 err Error Messages of this priority indicate an error has KERN_ERR
occurred. These are usually software related
errors and may include software failure due to
misconfiguration.
4 warn Warning Messages of this priority usually indicate error KERN_WARNING
conditions that do not cause software failure
but do impact the software’s performance.
5 notice Notice Messages of this priority indicate significant KERN_NOTICE
non-error conditions. For example, failed
authentication attempts.
6 Info Information Messages of this priority provide details related KERN_INFO
to normal operations. For example, successful
login attempts.
7 debug Debug Messages of this priority provide debugging KERN_DEBUG
information for applications. Most applications
do not generate debug messages by default,
configuration settings usually need to be
adjusted to have them generated. These
messages are not normally saved unless an
administrator is troubleshooting a problem. A
large amount of disk space is consumed very
rapidly when these messages are saved.
<none> <none> <none> Default priority when no priority is specified KERN_DEFAULT
with a message
<none c <none> Continuation of the previous log message KERN_CONt

NOTE: The above table shows the log levels in descending priority order. That is, the lower the number,
the higher the priority for the log levels with numbers. This ordering is important, as shown in the log
filtering examples below.

The way these log levels are used by systemd journal and syslog are discussed below in the sections on
each logging service.

Page 2 of 14
 INFO-3182- Operating systems - Linux

systemd journal

CentOS Stream 9 and Ubuntu 24.04, along with all other modern Linux systems, use the systemd journal
logging functionality. However, the configuration is slightly different on the two distributions.

On CentOS Stream 9, the systemd journal is configured to write to a temporary file in /run/log/journal
only. Since /run is a temporary file system that is recreated every time the system boots, the systemd
journal starts fresh every time. It is possible to make the systemd journal more persistent by creating
the /var/log/journal directory. If this directory exists at boot time, systemd journal will automatically
create a file there which survives system reboots. The file in /var/log/journal is periodically synchronized
with the temporary file in /run/log/journal.

The Ubuntu 24.04 operating system installation process creates the /var/log/journal directory as part of
system installation. This means that the systemd journal is saved persistently as soon as the system
boots the first time. Journal entries are first placed in a file in /run/log/journal and then synchronized
with the file in /var/log/journal periodically.

To ensure that the systemd journal does not cause problems by growing too large, the sizes of the files
in /run/log/journal, and if it is present /var/log/journal are limited. By default, the size of these files is
never allowed to grow to more than 10% of available space on the file system where they reside. Nor
are they allowed to grow large enough to cause total free space on disk to fall below 15% of the file
system capacity where the file is stored.

In the case where file sizes become limited, the oldest records in the files are automatically removed
when new records are added to ensure that the most current information is available.

On both CentOS Stream 9 and Ubuntu 24.04 these default values can be modified by editing the
appropriate options in the /etc/systemd/journald.conf file. By default, the configuration settings in the
/etc/systemd/journald.conf are all commented out with the default values shown as examples. On most
Linux systems the default values never need to be changed.

The files in /run/log/journal and /var/log/journal are stored in an indexed binary format that is not
human readable. This is done to allow the maximum possible amount of data to be saved in a way that
is quickly accessible. The journalctl command is used to access the log data stored in systemd
journal. This command uses the files in /run/log/journal and /var/log/journal automatically, so users do
not need to be concerned about whether they are working with all available data or not, they are using
all data.

Running the journalctl command without parameters displays the contents of the systemd journal
one page at a time, with the oldest available entries at the top. Figure 1 shows the first page of output
from the command used this way.

This form of the output of the journalctl command provides the same search and navigation
functionality as vim. Lines that are too long to display in the terminal window are only partially
displayed. In Figure 1 there are some lines that are too long to display, these are indicated by a ‘>’

Page 3 of 14
 INFO-3182- Operating systems - Linux

symbol at the end of the line in reverse video. The left and right arrow keys can be used to display the
remainder of the content of these lines.

To help identify the priority of messages, journalctl includes some colour and typeface helpers:

• A line with the text in red indicates a priority of Error, Critical, Alert or Emergency.
• A line with the text in yellow indicates a priority of Warning.
• A line with bold text is of priority Notice.
• A line with normal text is of priority Information.
• A line with grey text is of priority Debug.

Figure 1 Screenshot showing the first page of output from the journalctl command without parameters. The lines in the
screenshot with a reverse video > character at the right end of the line are longer than can be displayed on the screen.

The output of the journalctl command as shown in Figure 1 is broken into several columns. These
columns are:

Page 4 of 14
 INFO-3182- Operating systems - Linux

• Date and time – The date and time the logged event occurred.
• The name or IP address of the host where the message originated. In this course, this will always
be localhost. However, it is possible to configure systemd journal to send log entries to a remote
logging server. On the remote system, this column identifies the system the message came
from.
• Originating program or service – The name of the program or service that generated the log
message. In Figure 1, this column always contains kernel because the information show is all for
the early part of the system startup.
• Log message – The text of the message as received from the message originator.

One important piece of information that is missing from the output of the journalctl command is
the process ID (PID) of the process that sent the message. While it is not displayed, the PID of the
sending process is part of the information saved by systemd journal and can be used to filter the output
of journalctl, as shown below. It is also possible to display the PID associated with a log entry along
with other values that are not displayed by default by using the -o verbose parameter.

Figure 2 Screenshot of the output of the journalctl command filtered by log level. This output shows messages of log level notice
and higher.

Page 5 of 14
 INFO-3182- Operating systems - Linux

Figure 2 shows the output of the journalctl command filtered by log level. The command used is:
journalctl -p notice

This command causes the output to contain messages of log level notice and higher (warning, crit, alert,
emerg).

If the command
sudo journalctl -p error

Is used instead, messages with log levels of err, crit, alter and emerg are displayed as shown in Figure 3.

Figure 3 Screenshot showing the journalctl command run on the same system as in Figure 2, but with filtering to show messages
with a log level of error or higher.

NOTE: Newer versions of journalctl, including the one on CentOS Stream 9 and Ubuntu 24.04
support both ‘err’ and ‘error’ for the -p parameter when referring to the error log level. Other versions
of journalctl only recognize ‘err’ with the -p parameter.

Sometimes it is useful to display entries to the systemd journal as they are added to the log. This
functionality is useful when testing software. The journalctl command has the -f parameter
specifically for this purpose. The -f parameter causes the last ten log messages to be output immediately
and then waits for more messages to be added to the systemd journal, displaying each as it is added.
This continues until the user terminates journalctl by entering CTRL+C. Figure 4 shows the -f
parameter being used along with the -p parameter so that only messages of log level notice and higher
are output. Notice that there is no prompt at the end of the output, this is because journalctl is
waiting to output the next message.

Page 6 of 14
 INFO-3182- Operating systems - Linux

Figure 4 Screenshot showing the journalctl command being used to display log messages as they are added to systemd journal.
The command only outputs messages with log level notice or higher because the -p parameter is used specifying that level.

It is possible to filter the output of journalctl based on dates and times. There are two parameters
used for this:

• --since -Used to specify the date and/or time after which log messages are to be displayed. The
date and time are specified as a quote enclosed string of the format “YYYY-MM_DD HH:MM:SS”.
If the date is omitted, the current date is assumed. If the time is omitted, 00:00:00 is assumed.
• --until -Used to specify the date and/or time before which log messages are to be displayed. The
date and time are specified as a quote enclosed string of the format “YYYY-MM_DD HH:MM:SS”.
If the date is omitted, the current date is assumed. If the time is omitted, 23:59:59 is assumed.

Times must be specified in 24-hour format for the --since and --until parameters.

The keywords ‘today’, ‘tomorrow’ and ‘yesterday’ can be used instead of dates.

For example:

• sudo journalctl --–since “2024-11-01” displays all log messages added to the
journal beginning 1st November 2024 at 00:00:00.
• sudo journalctl --until “2024-11-08 12:00:00” displays all log messages
added to the journal prior to noon on 8th November 2024.
• sudo journalctl --since “2024-11-08 00:00:00” --until “2024-11-09
23:59:59” displays all messes added to systemd journal on the 8th or 9th of Number 2024.

It is also possible to combine date filtering with other filters. For example, the command:
sudo journalctl --–since “2024-11-01” -p warning

Page 7 of 14
 INFO-3182- Operating systems - Linux

displays all messages of log level warning or higher since the start of day on 1st November 2024. Figure 5
shows this command being run on a CentOS Stream 9 system. The log entries shown in the output start
on November 7th because persistent logging was not enabled until then.

Figure 5 Screenshot showing the output of the journalctl command filter to only show messages of log level warning or higher
since November 1st.

As noted earlier, there are other pieces of information available in systemd journal for a log message
besides those that are included in the default output format. Many of these can be used for filtering and
all of them are included in the output when the verbose output option is selected. Figure 6 shows the
command:
sudo journalctl -o verbose _PID=881

used to display all the available information about the OpenSSH server service that is running as process
881. Any of the additional items shown in the verbose output can be used for filtering journalctl
output.

Page 8 of 14
 INFO-3182- Operating systems - Linux

Figure 6 Screenshot showing the journalctl command output filtered by processed 881 using the verbose output format. This
shows all the information available for a single journal entry.

While any of this additional information in the log entries in systemd journal can be used for filtering,
some of the more useful items are:

• _COMM The name of the command (program) running in the process.

• _EXE The path to the executable (program) for the process
• _PID The PID of the process
• _UID The UID of the user running the process.
• _SYSTEMD_UNIT The systemd unit that started the process.

Filtering by PID is a very useful technique when attempting to follow what has or is happening within a
process.

Any of the filtering techniques shown above can be used in combination to create complex filters, as
necessary.

Page 9 of 14
 INFO-3182- Operating systems - Linux

Syslog

The syslog logging service available on Linux systems originated in Unix. Even though the basic design of
syslog dates from the 1970s, its concepts are sound, and it has been constantly updated since then.

The log data produced by syslog can be saved in files on the local system’s disk or sent to a centralized
logging server via syslog’s own network protocol. Log files produced by syslog are normally placed in
/var/log or its subdirectories. The network protocol uses the port 514/UDP.

The name of the syslog service is rsyslogd.

The syslog service directs log messages to the appropriate file based on two pieces of information that
are always included in every log message, a log priority (log level), as described earlier in this document,
and a log facility. Log facilities identify the source of the message. The available log facilities are listed in
the table below.

Number Name Description

0 kern Log facility for messages originating from the kernel.
1 user Log facility for user generated messages.
2 mail Log facility for mail services
3 daemon Log facility for services that do not have their own log facility.
4 auth Security and authorization messages from services or programs that prompt
users for usernames and/or passwords. E. g. ssh
5 syslog Log facility for messages originating from within syslog itself.
6 lpr Log facility for messages originating from line printer services
7 news Log facility for messages originating from the Network News Transport
Protocol (NNTP) service. These may include news content.
8 uucp Log facility for Unix-to-Unix Copy (UUCP) service. This is a basic networking
service that predates services such as FTP and SMTP. Rarely used anymore.
9 cron Log facility for job schedulers such as cron and at.
10 authpriv Log facility for security and authorization messages
11 ftp Log facility for the FTP service.
12 ntp Log facility for the NTP service.
13 security Log facility for system audit logs.
14 console Log facility for system console messages.
15 solaris- Log facility for the scheduling service on Solaris operating systems.
cron
16 to 23 local0 to These eight facilities are intended for use by locally defined logging
local7 operations. One common use of local6 is for the logging of spam filter
messages on mail servers.

Some of the facilities listed in the above table are seldom used because the services they were assigned
to are not often used anymore. Two of these seldom used services are UUCP and NNTP.

Page 10 of 14
 INFO-3182- Operating systems - Linux

Log messages are directed to the proper log files by using a combination of log facility and log level
values. Some examples are:

• *.* - Messages from all log facilities and all log levels are saved in the file, or sent to the central
log server, associated with this combination. This is commonly used when sending log messages
to a centralized log server.
• *.info – Messages from all log facilities with a log level of info or higher are saved to the
associated file.
• authpriv.* - All messages with the authpriv facility are saved in the associated file.
• *.info;mail.none;authpriv.none;cron.none- All messages from any log facility except mail,
authpriv and cron with a log level of info or higher are sent to the associated file. Notice that the
facility/level pairs are separated by commas without spaces.

On CentOS Stream 9, syslog configuration is done using the /etc/rsyslog.conf file and on Ubuntu 24.04
the files in the /etc/rsyslog.d directory are used.

CentOS stream 9 and Ubuntu 24.04 both store the syslog log files in /var/log. However, the names of
files for the same purpose are often different. For example, the general log file on CentOS Stream 9 is
/var/log/messages (it is associated with the last facility/level example above). On Ubuntu 24.04, this file
is /var/log/syslog. Figures 7 and 8 show partial directory listings of the /var/log directories on a CentOS
Stream 9 system and an Ubuntu24.04 system, respectively.

Figure 7 Screenshot of part of the directory listing of the /var/log directory on a CentOS Stream 9 system.

Page 11 of 14
 INFO-3182- Operating systems - Linux

Figure 8 Screenshot of part of the directory listing of the /var/log directory on an Ubuntu 24.04 system.

In Figure 7, there are several messages* files. The file named messages is the one currently being
written to by syslog, the remainder are older versions of the file. The date at the end of these older files
indicates the date the file was last written to. The logrotate service is used to automatically manage
these files as discussed below.

Figure 8 shows a similar group of files for the syslog file on Ubuntu 24.04. However, this time instead of
a date, a sequential number is added to the end of older files’ names as shown by the file syslog.1. Other
older syslog files also have the .gz extension. On Ubuntu 24.04, older log files are compressed to save
space. The .gz extension indicates that the files are compressed with the gzip program.

When syslog is installed on a Linux system, the logrotate service is also installed. The logrotate service is
specifically designed to periodically replace log files with new ones. The logrotate service can:

• Rotate log files on a fixed period or by the size of the log file. Old files are renamed as necessary
based on a user-configured method.
• Keep a specified number of log files on disk. When the specified number of log files is reached,
the oldest one is deleted when a new one is created.
• Compress old files to save space if configured to do so.

The logrotate service performs the log rotation process as follows:

• Rename/compress old files, as necessary.

• Rename the current log file while it is in use. This does not interrupt normal logging operations
because open files are referred to by their inode, not by name.
• Create the new log file.
• Send a signal to the process using the current log file causing it to close the old file and open the
new one. The close operation is done using the inode of the old file and the open uses the name

Page 12 of 14
 INFO-3182- Operating systems - Linux

of the new file. This operation is so fast that there is no noticeable interruption in logging
operations. See the documentation of the kill or killall commands for information on signals.

All syslog log entries are in plain text and have the same format. A few sample lines from the
/var/log/secure file on a CentOS Stream 9 system are shown below. Notice that the lines vary in length,
but this does not vary the basic format of the lines.

The lines of a syslog log file are split into parts:

• Date and time – The date and time the event being logged occurred. CentOS does not include
the year, but Ubuntu does.
• Hostname – The hostname of the system where the event occurred. This may be the actual
name of a host or localhost, except on a central logging server where it will always be the actual
name of the host where the event occurred.
• Program - The name and PID of the program that originated the message.
• : - Start of message text.
• Message text – The text of the message as sent by the originating program. The format and
content of the message varies, as necessary.

Nov 10 07:47:56 cs9-swg-1 sshd[2611]: Accepted password for thall from

192.168.252.218 port 49708 ssh2

• The SSH server received a login attempt for user thall using password authentication from host
192.168.252.218 on port 49708/TCP using SSH version 2.
Nov 10 07:47:56 cs9-swg-1 systemd[2616]: pam_unix(systemd-
user:session): session opened for user thall(uid=1000) by thall(uid=0)

• Pluggable Authentication Module (PAM) authenticated user thall with UID 1001. The
authentication process was run by user thall with effective UID 0 (root).
Nov 10 07:47:56 cs9-swg-1 sshd[2611]: pam_unix(sshd:session): session
opened for user thall(uid=1000) by thall(uid=0)

• SSH server opens interactive shell session for user thall UID 1001, this starts with effective ID 0
(root) then switches to the normal privileges for thall.
Nov 10 07:48:06 cs9-swg-1 sudo[2659]: thall : TTY=pts/0 ; PWD=/root
; USER=root ; COMMAND=/bin/bash

• User thall calls sudo to gain root privileges to run the /bin/bash shell and setting the current
working directory to /root. This is a call to sudo -i.
Nov 10 07:48:06 cs9-swg-1 sudo[2659]: pam_unix(sudo-i:session):
session opened for user root(uid=0) by thall(uid=1000)

• Pam authorizes user thall to the root shell requested.

Page 13 of 14
 INFO-3182- Operating systems - Linux

Nov 10 07:57:35 cs9-swg-1 sudo[2659]: pam_unix(sudo-i:session):

session closed for user root

• Sudo session ended. The PID is the only way to associate this close action with the start of the
session.

The above log lines appear one after the other in the log file of the sample system because there is very
little activity on the system used. On a production system there would probably be other log entries in
the file mixed with these lines. It is often necessary to use grep or other string search tools to find the
log entries related to a particular series of events. Creating scripts for searches that are performed
regularly or that are complex is a good idea.

Page 14 of 14