Syslog and Log files

Outline
• Log files
– What need to be logged
– Logging policies
– Finding log files
• Syslog: the system event logger
– how syslog works
– its configuration file
– the software that uses syslog
– debugging syslog
 What to be logged?

• The accounting system

• The kernel
• Various utilities
– all produce data that need to be logged
– most of the data has a limited useful lifetime,
and needs to be summarized, compressed,
archived and eventually thrown away
 Logging policies

• Throw away all data immediately

• Reset log files at periodic intervals
• Rotate log files, keeping data for a fixed
time
• Compress and archive to tape or other
permanent media
 Which one to choose

• Depends on :
– how much disk space you have
– how security-conscious you are
• Whatever scheme you select, regular
maintenance of log files should be
automated using cron (chap 10, periodic process)
 Throwing away log files
• not recommend
– security problems ( accounting data and log
files provide important evidence of break-ins)
– helpful for alerting you to hardware and
software problems.
• In general, keep one or two months
– in a real world, it may take one or two weeks
for SA to realize that site has been
compromised by a hacker and need to review
the logs
 Throwing away (cont.)

• Most sites store each day’s log info on disk,

sometimes in a compressed format
• These daily files are kept for a specific
period of time and then deleted
• One common way to implement this policy
is called “rotation”
 Rotating log files

• Keep backup files that are one day old, two

days old, and so on.
– logfile, logfile.1 , logfile.2, … logfile.7

• Each day rename the files to push older data

toward the end of the chain
– script to archive three days files
#! /bin/sh
cd /var/log
mv logfile.2 logfile.3
mv logfile.1 logfile.2
mv logfile logfile.1
cat /dev/null > logfile

Some daemons keep their log files open all the time,
this script can’t be used with them. To install a new
log file, you must either signal the daemon, or kill
and restart it.
#! /bin/sh
cd /var/log
mv logfile.2.Z logfile.3.Z
mv logfile.1.Z logfile.2.Z
mv logfile logfile.1
cat /dev/null > logfile
kill -signal pid
compress logfile.1

signal - appropriate signal for the program

writing the log file
pid - process id
 Archiving log files

• Some sites must archive all accounting

data and log files as a matter of policy, to
provide data for a potential audit
• Log files should be first rotate on disk,
then written to tape or other permanent
media
– see chap 11, Backups
 Finding log files

• To locate log files, read the system startup

scripts : /etc/rc* or /etc/init.d/*
– if logging is turned on when daemons are run
– where messages are sent
• Some programs handle logging via syslog
– check /etc/syslog.conf to find out where this
data goes
 Finding log files

• Different operating systems put log files in

different places:
– /var/log/*
– /var/cron/log
– /usr/adm
– /var/adm …
• On linux, all the log files are in /var/log
directory.
 Outline
• Log files
– What need to be logged
– Logging policies
– Finding log files
• Syslog: the system event logger
– how syslog works
– its configuration file
– debugging syslog
– the software that uses syslog
 What is syslog

• A comprehensive logging system, used to

manage information generated by the
kernel and system utilities.
• Allow messages to be sorted by their
sources and importance, and routed to a
variety of destinations:
– log files, users’ terminals, or even other
machines.
 Syslog: three parts

• Syslogd and /etc/syslog.conf

– the daemon that does the actual logging
– its configuration file
• openlog, syslog, closelog
– library routines that programs use to send data
to syslogd
• logger
– user-level command for submitting log entries
 syslog-aware programs

Using syslog lib. Routines

write log entries to a special file
/dev/log /dev/klog

reads consults
syslogd /etc/syslog.conf

dispatches

Log Users’s Other

files terminals machines
 Configuring syslogd

• The configuration file /etc/syslog.conf

controls syslogd’s behavior.
• It is a text file with simple format, blank
lines and lines beginning with ‘#’ are
ignored.
– Selector <TAB> action
– eg. mail.info /var/log/maillog
 Configuration file
selector

• Identify
– source -- the program (‘facility’) that is sending
a log message
– importance -- the messages’s severity level
– eg. mail.info /var/log/maillog
• Syntax
– facility.level
– facility names and severity levels must chosen
from a list of defined values
 Configuration file
Facility names

Facility Programs that use it

kern the kernel
user User process, default if not specified
mail The mail system
daemon System daemons
auth Security and authorization related
commands
lpr the BSD line printer spooling system
news The Usenet news system
 Configuration file
Facility names

Facility Programs that use it

uucp Reserved for UUCP
cron the cron daemon
mark Timestamps generated at regular intervals
local0-7 Eight flavors of local message
syslog syslog internal messages
authpriv Private or system authorization messages
ftp the ftp daemon, ftpd
* All facilities except “mark”
 Configuration file
Facility names

• Timestamps can be used to log time at

regular intervals (by default, every 20
minutes), so you can figure out that your
machine crashed between 3:00 and 3:20 am,
not just “sometime last night”. This can be a
big help if debugging problems occur on a
regular basis.
 Configuration file
severity level

Level Approximate meaning

emerg (panic) Panic situation
alert Urgent situation
crit Critical condition
err Other error conditions
warning Warning messages
notice Unusual things that may need
investigation
info Informational messages
debug For debugging
 Configuration file
selector

• Can include multiple facilities separated with ‘,’

commas
– daemon,auth,mail.level action
• Multiple selector can be combined with ‘;’
– daemon.level1; mail.level2 action
• Selector are ‘|’ --ORed together, a message
matching any selector will be subject to the action.
• Can contain ‘*’ or ‘none’, meaning all or nothing.
 Configuration file
selector

• Levels indicate the minimum importance that a

message must have in order to be logged
– mail.warning, would match all the messages
from mail system, at the minimum level of
warning
• Level of ‘none’ will excludes the listed facilities
regardless of what other selectors on the same line
may say.
– *.level1;mail.none action
• all the facilities, except mail, at the minimum level 1 will
subject to action
 Configuration file
action
(Tells what to do with a message)
Action Meaning
filename Write message to a file on the
local machine
@hostname Forward message to the syslogd on
hostname
@ipaddress Forward message to the host at IP address
user1, user2,… Write message to users’ screens if they
are logged in
* Write message to all users logged in
 Configuration file
action

• If a filename action used, the filename must be

absolute path. The file must exist, syslogd will not
create it.
– /var/log/messages
• If a hostname is used, it must be resolved via a
translation mechanism such as DNS or NIS
• While multiple facilities and levels are allowed in
a selector, multiple actions are not allowed.
 Config file examples
# Small network or stand-alone syslog.conf file
# emergencies: tell everyone who is logged on
*.emerg *

# important messages
*.warning;daemon,auth.info /var/adm/messages

# printer errors
lpr.debug /var/adm/lpd-errs
# network client, typically forwards serious messages to
# a central logging machine
# emergencies: tell everyone who is logged on
*.emerg;user.none *

#important messages, forward to central logger

*.warning;lpr,local1.none @netloghost
daemon,auth.info @netloghost

# local stuff to central logger too

local0,local2,local7.debug @netloghost

# card syslogs to local1 - to boulder

local1.debug @boulder.colorado.edu

# printer errors, keep them local

lpr.debug /var/adm/lpd-errs

# sudo logs to local2 - keep a copy here

local2.info /var/adm/sudolog
 Sample syslog output

Dec 27 02:45:00 x-wing netinfod [71]: cann’t lookup child

Dec 27 02:50:00 bruno ftpd [27876]: open of pid file
failed: not a directory
Dec 27 02:50:47 anchor vmunix: spurious VME interrupt
at processor level 5
Dec 27 02:52:17 bruno pingem[107]: moose.cs.colorado.edu
has not answered 34 times
Dec 27 02:55:33 bruno sendmail [28040] : host name/address
mismatch: 192.93.110.26 != bull.bull..fr
 Syslog ‘s functions

• Liberate programmers from the tedious

mechanics of writing log files
• Put SA in control of logging
– before syslog, SA had no control over what info
was kept or where it was stored.
• Can centralize the logging for a network
system
 Syslogd (cont.)

• A hangup signal (HUP, signal 1) cause

syslogd to close its log files, reread its
configuration file, and start logging again.
• If you modify the syslog.conf file, you must
HUP syslogd to make your changes take
effect.
– Kill -1 pid
 Debugging syslog -- logger

• Useful for submitting log entries from shell

scripts

• Can also use it to test changes in syslogd’s

configuration file.
– For example..
Add line to syslog.conf:
local5.warning /tmp/test.log

verify it is working, run

logger -p local5.warning “test messages”

a line containing “test messages” should be written to /tmp/test.log

If this doesn’t happen:

forgot to create the test.log file
forgot to send syslogd a hangup signal
 Software that uses syslog

Program Facility Levels Description

amd auth err-info NFS automounter
date auth notice Display and set date
ftpd daemon err-debug ftp daemon
gated daemon alert-info Routing daemon
gopher daemon err Internet info server
halt/reboot auth crit Shutdown programs
login/rlogind auth crit-info Login programs
lpd lpr err-info BSD line printer daemon
 Software that uses syslog

Program Facility Levels Description

named daemon err-info Name sever (DNS)
passwd auth err Password setting
programs
sendmail mail debug-alert Mail transport system
rwho daemon err-notice romote who daemon
su auth crit, notice substitute UID prog.
sudo local2 notice, alert Limited su program
syslogd syslog, mark err-info internet errors,
timestamps
 Using syslog in programs

• openlog ( ident, logopt, facility);

– messages are logged with the options specified
by logopt begin with the identification string
ident.
• Syslog ( priority, messge, parameters…);
– send message to syslogd, which logs it at the
sepecified priority level
• close ( );
 / * c program: syslog using openlog and closelog */

#include <syslog.h>
main ( )
{
openlog ( “SA-BOOK”, LOG_PID, LOG_USER);
syslog ( LOG_WARNING, “Testing …. “);
closelog ( );
}

On the host, this code produce the following log entry:

Dec 28 17:23:49 moet.colorado.edu SA-BOOK [84]: Testing...
 Final words
• On linux, check following files:
– /etc/syslog.conf : syslog configuration file
– /etc/logrotate.conf : logging policy, rotate
– /etc/logrotate.d/*
– /var/log/* : log files
• try following commands to find out more...
– man logrotate
– man syslogd