Chapter 19: Now What's It Doing?

Every systems administrator must be able to answer this basic question at the drop of a hat: "How is the server doing?" You must know what your systems are doing to be able to make good planning and capacity decisions. Your manager will want nice reports on how things are running, and even if you haven't been asked for these reports yet, you will be. Your best bet is to have this information ready before it's needed.

FreeBSD supports report generation, pretty graphs, daily status checks, and more. Let's take a look.

Status Mails

If you look in /etc/crontab, under the command periodic(8), you'll see that FreeBSD systems run maintenance jobs every day, week, and month. These jobs perform some basic system checks and notify the administrators of changes, items requiring attention, and potential security issues. The output of the scheduled jobs is emailed daily to the root account on the local system. The simplest way to find out what your system is doing is to read these mails; many very busy systems administrators, like you, have collaborated to make these messages useful and necessary.

The configuration of the daily, weekly, and monthly reports is controlled in /etc/periodic.conf, /etc/defaults/periodic.conf, and /etc/periodic. See Chapter 9 for details, or just jump straight to the scripts in /etc/periodic.

Forwarding Reports

Because you probably don't want to log in as root to all of your servers every day, send your mail to a centralized mailbox by editing /etc/mail/aliases (see Chapter 14) to point mail to some other account. If you have many servers, you may end up with a lot of mail, but with experience you'll quickly learn how to skim the reports looking only for critical or unusual changes.

Logging with Syslogd

UNIX's logging system is one of its most delightful features. Unlike some operating systems that force you to use a small range of limited logs, UNIX allows you to log almost anything, at almost any level of detail. While you'll find system logging hooks for the most common UNIX resources, administrators can choose a logging configuration that meets their needs. Almost all programs integrate with the logging system, syslogd(8).

Syslogd handles messages according to their facility and level, both of which are included in messages to syslogd. You'll need to understand both facility and level to manage your system logs, so they are described next.

Facilities

A facility is a log−entry source (a program) that sends messages to syslogd. This is an arbitrary label—it is essentially just a text string that's used to sort out one program from another. In most cases, each program that needs a unique log needs a unique facility. Many sorts of programs or protocols have facilities dedicated to them—for example, FTP is such a common protocol that syslogd has a special facility just for it. Syslogd also supports a variety of generic facilities that can

424

be used by any program. Programs can lie about their facility, but you'll be able to track them down by finding their name with their message.

The standard facilities and the type of information they provide are listed here.

auth	Publicly accessible information pertaining to user authorization, such as login and
	su.
authpriv	Private information pertaining to user authorization, accessible only by root and
	other selected users.
console	Messages normally printed to the system console can be captured with the
	console facility.
cron	Messages from the system scheduler go through the cron facility.
daemon	A catch−all for all system daemons without other explicit handlers, either in
	syslogd or in their program.
ftp	FTP daemons will automatically log to this facility. (See Chapter 15.)
kern	Messages from the kernel.
lpr	Catches messages from the printing system.
mail	Catches messages from the mail system.
mark	Not an actual log from a system; instead, the mark facility puts an entry in your
	log every 20 minutes. This is useful when combined with some other log.
news	Catches messages from the Internet News daemons.
ntp	Collects messages from Network Time Protocol.
security	Includes messages from various security programs, such as ipf(8) and ipfw(8).
syslog	The log service can log to itself. Just don't log when you log logs from syslogd, or
	you'll make yourself dizzy.
user	The catch−all message facility. If you don't specify a logging facility for user
	programs, they'll use this.
uucp	Logs from the UNIX−to−UNIX Copy Protocol. (This is a piece of pre−Internet
	UNIX history you'll probably never encounter.)
local0	These are reserved for administrator use. Many programs have an option to set a
through	logging facility; choose one of these if at all possible. For example, you might tell
local7	your customer service system to log to local0.

Levels

A log message's level represents its relative importance. While programs send all their logging data to syslogd, most systems only record the important stuff that syslogd receives and discard the trivial messages. Of course, one person's trivia is another's vital data, and that's where the levels come in.

FreeBSD offers eight levels of syslogd importance. You can use these levels to tell syslogd what to record and what to discard. The levels follow, in order from most to least important.

emerg System panic. Messages are flashed on every terminal. The system is basically hosed. You don't even have to reboot—the system is doing it for you.[1]

alert This is bad, but not as bad as the emerg level. The system can continue to operate, but this error should be attended to immediately.

crit These are critical errors, such as hardware problems (like bad blocks or a failing SCSI cable) or serious software issues. You can continue running, if you're brave.

err

425

Miscellaneous errors. These are bad and should be fixed, but they won't destroy the system.

warning Miscellaneous warnings.

notice General information that should be logged, in case you need it, but that probably doesn't really require action on your part.

info General program information, such as individual transactions in a mail server.

debug This level is usually only of use to programmers, and occasionally to sysadmins who are trying to figure out just why some program behaves the way it does. Debugging logs can contain whatever information the programmer felt necessary to debug the code, which may include information that will violate user privacy.

none This means "don't log anything from this facility here." It's most commonly used when excluding information from wildcard entries, as we'll see later.

[1]You might think this is funny now, but you won't if it ever happens to you.

Syslog.conf

The /etc/syslog.conf file has two columns. The first describes the information to be logged, by facility and level. The second tells the action to be taken when a log message matches the description. Syslogd compares each submission to the entries in /etc/syslog.conf and, when it finds a matching entry, processes the log entry in the manner described.

Information sources include both a facility and a level, separated by a period. When you specify a level, the system defaults to recording messages of that level or greater. For example, look at this entry from /etc/syslog.conf:

...............................................................................................

mail.info /var/log/maillog

...............................................................................................

This tells the system to log messages from the mail system to /var/log/maillog if they have a severity level equal to or above "info".

Wildcards

You can also use wildcards in your information source. For example, use this line to log absolutely all messages from the mail system:

...............................................................................................

mail.* /var/log/maillog

...............................................................................................

To log everything from everywhere, uncomment the all.log entry:

...............................................................................................

*.* /var/log/all.log

...............................................................................................

This works, but it's got too much information to be of any real use; if you use it you'll find yourself building complex grep commands just to find what you want. Also, this would include all sorts of private information, thanks to the debug level.

426

Excluding Information

Use the authpriv facility and the none level to exclude authentication information. The semicolon allows you to combine entries on a single line:

...............................................................................................

*.*;authpriv.none /var/log/all.log

...............................................................................................

Comparison

The comparison operators < (less than), = (equals), and > (greater than) can also be used in /etc/syslog.conf. You can use these with levels to log data above a certain level in one file and data below a certain level in another. While syslogd defaults to recording all messages of the specified level or above, you might want to include only a range of information.

For example, suppose you want one log for mail traffic and another for mail debugging information:

...............................................................................................

mail.info	/var/log/maillog
mail.=debug	/var/log/maillog.debug

...............................................................................................

The preceding mail.info entry captures all log messages sent to the mail facility of info level and above. The second line only captures messages that have a level no higher than debug. You can't use a message source of mail.debug, or the debugging log will contain everything in the previous log! This way you don't have to sort through debugging information to learn what your mail server thinks it's doing, and you don't have to sort through mail−transmission information to get to your mail server's debugging output.

Local Facilities

Many programs expect to be able to use syslogd to handle their logging. Most of these can be set to use a facility of your choice. The various local facilities are reserved for these programs.

For example, you might tell a program to log to the facility local3. Just how you set the facility varies from program to program. Once you get the program to mark messages with a facility and send them to syslogd, you have to tell syslogd to catch those messages and to do something with them.

...............................................................................................

local3.* /var/log/programname

...............................................................................................

In general, if a program supports logging to a facility, use a local facility.

427

Logging by Program Name

If you're out of facilities, or if your program simply doesn't support syslogd, you can use the program's name to handle logging. An entry for a name requires at least two lines: the program name with a leading exclamation point and then a line with the logging information.

For example, to log ppp, you could do this:

...............................................................................................

!ppp

*.* /var/log/ppp.log

...............................................................................................

This entry first specifies the program name, and then uses wildcards to tell syslogd to append absolutely everything to a file. (You can't be certain that a random third−party program will have reasonable log levels available, so it's safest to record everything until you know otherwise.)

Logging Host

My networks habitually have a single logging host that handles not only the FreeBSD boxes, but also Cisco routers, 3Com switches, every other UNIX box, and any other syslogd−speaking systems. Since you have only one host whose logs need handling, this saves a lot of maintenance.

Use the at symbol (@) to can send log messages to another host. For example, the following line would dump everything your syslogd receives to the logging host on my network:[2]

...............................................................................................

*.* @loghost.absolutebsd.com

...............................................................................................

The /etc/syslog.conf on the log host determines the final destination for the messages it receives. Fortunately, each log message includes the hostname.

Logging to User Sessions

To log user sessions, list usernames separated by commas. Then, if those users are logged in when the log message arrives, the system will write the message on their terminal.

To write the messages to all users' terminals, use an asterisk (*) for the destination. For example, the default syslog.conf includes this line:

...............................................................................................

*.emerg *

...............................................................................................

This says that any message of emergency level will appear on all users’ terminals.

428