In the last post I talked about how to setup a logging server that will accept remote log entries. In this installment I’ll talk about how to sort log entries into specific files.
Facility, severity and priority
Let’s talk about how logging servers figure out which file to store a log entry in when it gets received. Log messages contain two descriptive parameters, facility and severity. When these two parameters are combined, the value is referred to as the priority of the log message.
Facility
Facility defines the type of process that generated the log entry. For example all mail servers are expected to identify that their log entries are part of the “mail” facility. FTP processes should use the FTP facility, NTP processes should use the NTP facility, and so on. RFC 3164 defines the valid facilities, but here’s the list:
Numerical Facility
Code
0 kernel messages (kern)
1 user-level messages (user) – default if not specified
2 mail system (mail)
3 system daemons (daemon)
4 security/authorization messages (auth)
5 internal syslogd (syslog)
6 line printer subsystem (lpr)
7 network news subsystem (news)
8 UUCP subsystem (uucp)
9 clock daemon
10 security/authorization messages (authpriv)
11 FTP daemon (ftp)
12 NTP subsystem (ntp)
13 log audit
14 log alert
15 clock daemon (cron)
16 local use 0 (local0)
17 local use 1 (local1)
18 local use 2 (local2)
19 local use 3 (local3)
20 local use 4 (local4)
21 local use 5 (local5)
22 local use 6 (local6)
23 local use 7 (local7)
The “local use” facilities are similar to private addresses in the IP world. These facilities are not reserved, and are available for anyone to use as they see fit.
Facility problems
There are a couple of problems here. To start, where is the Web server facility? This list was generated back in 1987 before Web servers (or Gopher for that matter) existed. So some of the services we use today (VoIP, SQL, etc.) are missing. Also, some of the listed services typically go unused in a corporate environment. UUCP and Network News (NNTP) are excellent examples.
The lack of current services has caused many vendors to rely heavily of the local use facilities. This can cause potential conflicts when we get into sorting our log entries. For example Linux uses local use 7 to identify its boot time log entries. Apache also uses local use 7 for Web server errors. So down the road it may be difficult for us to sort Web errors and boot messages into different log files.
Another problem is that there is no verbose description about each of these facilities. This can make it a bit difficult for a programmer to identify which one to use. For example, let’s say we’ve written a program that authenticates a user for network access. Which facility should we use? Facility 4 and 10 seem the most likely, but their descriptions are identical. How do we choose? If our program runs as a background process should we actually choose facility 3 instead?
You get the idea. The list is not as clear-cut as it could be. It is not uncommon to see vendors use a different facility than you would expect. For example I’ve seen VPN vendors undecided as to the differences between facilities 4 and 10, so they simply send some percentage of log entries to each.
Severity
Severity defines the importance of the log entry. The same RFC 3164 defines the severity levels as:
Numerical Severity
Code
0 Emergency: system is unusable (emerg)
1 Alert: action must be taken immediately (alert)
2 Critical: critical conditions (crit)
3 Error: error conditions (error)
4 Warning: warning conditions (warn)
5 Notice: normal but significant condition (notice)
6 Informational: informational messages (info)
7 Debug: debug-level messages (debug)
Luckily the severity levels are far less vague than the facility descriptions. This means they are much less confusing to work with. The higher numbered severity levels tend to be very verbose. This means saying you want to send debug level messages to your logging server could easily flood the network. Use the higher numbered severity levels with caution.
Priority
When a log entry gets transmitted to a log server, the first value contained within it is the priority of the message. The priority is the facility and severity values combined per the following math formula:
( Facility x 8 ) + Severity = Priority
So lets say our mail server needs to send a warning message. What would the priority be? The mail facility has a value of 2, while warnings have a severity of 4. So the math would be:
( 2 x 8 ) + 4 = 20
If a print server (facility 6) needed to send a log entry saying it is currently on fire (severity 0), the priority value in the message would be:
( 6 x 8 ) + 0 = 48
When a log entry gets transmitted, the priority value needs to be encapsulated in less than and greater than signs. So the priority value in the above mail server message would be “<20>” while the print server would use “<48>”. Again, this needs to be the first piece of information transmitted in the log message.
Sorting log entries
The priority value is used by logging servers to sort the incoming messages. For example if we wanted all mail messages to go to the same file, we would tell our logging server that all messages with a priority of 16 (2×8+0) through 23 (2×8+7) should go to the “maillog” file. Most logging servers (like rsyslog) will let you do this numerically or by using the short description names.
rsyslog.conf example
Here are two lines out of the rsyslog.conf file that ships with Fedora. Let’s talk about what they are actually doing:
authpriv.* /var/log/secure
*.info;mail.none;authpriv.none;cron.none /var/log/messages
These lines define two of the rules for determining which log entries should go to which log files. The syntax for sorting is:
facility.severity
So the first line says all facility 10 (authpriv) log entries, regardless of severity (“*” is a wild card match) should be sent to the file /var/log/secure.
The second line is a bit more complex as it has multiple conditions separated by semi-colons. These conditions state:
- *.info = All facilities, so long as the severity level is info
- mail.none = No mail facility log entries, regardless of severity
- authpriv.none = No authprive facility log entries, regardless of severity
- cron.none = No cron facility log entries, , regardless of severity
Or, to translate this to English, the line says “Send all severity “info” messages to /var/log/messages, except those that contain a facility of “mail”, “authpriv” or “cron”.
So with these rules we can define any combination of facility and severity values and which log file we would like to direct it. When you first set this up, stick with the defaults. As you start collecting log entries you can tweak the rules as you see fit.
Bending the RFCs
In an ideal world, the RFCs would be a perfect fit for everyone’s needs. Unfortunately this is not always the case. A good example is the logging facilities. As mentioned we are missing facilitates for modern day services, while at the same time have facilitates that we will never use. An obvious answer is to recycle the outdated facilities in order to support modern services.
For example, UUCP ( facility 8 ) is not even supported by modern operating systems. With this in mind, I like to use it as my Windows facility. That way I can sort all Windows log entries into their own file. For network hardware, I use the network news facility (facility 7). If you are unsure if a facility is currently in use, modify your logging server’s configuration file to send all log entries for that facility to a unique file:
ftp.* /var/log/facility-test
If no entries arrive, you are in good shape. Just keep in mind that a legitimate service may use it at a later date. For example if three months from now someone sets up an FTP server, we may have problems if we are already using the FTP facility (facility 11). If you are unsure you can always stick with the local use facilities, as that is what they are intended for. Local use 0 and 7 seem to be the most heavily used, so avoid them when possible.
Other sorting options
While its not part of the RFC, some logging servers give you the ability to sort log entries based on patterns within the message. A good example is Syslog-NG. Syslog-NG will sort based on facility and severity, but you can also sort based on source IP, the application that generated the log entry, etc. This gives you far more flexible sorting options and it may be something to consider if facility/severity is not granular enough for your needs.
Exec Summary
In this installment we talked about how facility and severity is used to sort log entries. In my next post I’ll talk about how to get each of our systems to submit log messages to our centralized server.
Related posts:
