Chapter 13 Process Management 419

Figure 13-1:The output of the top command

Monitoring Log Files

5.8 Monitor system log files regularly for errors, logins, and unusual activity

Regularly monitoring log files is one of the most important tasks that a Linux system administrator can perform. Through this type of proactive maintenance, the administrator can take care of small problems before they become too serious. For example, a “low disk space” warning can give the administrator ample time to fix the problem before the file system becomes full. The administrator should check some logs on a daily basis, such as system and kernel logs. Other logs, such as boot messages or login information, can be examined with less frequency.

Linux log files are located in the /var/log directory. This directory contains several log files for both system functions and applications. The following list includes some of the more important logs that should be monitored on a regular basis.

/var/log/messages: syslog is configured to write kernel and system errors in this log. This log should be checked daily for any warning messages.

/var/log/wtmp: This log file keeps track of the time and date of user logins. This file can grow very quickly. The last command uses this log to show a list of users who have logged-in. This is helpful in tracking when certain users have logged in to the system at times that may indicate suspicious activity.

/var/log/dmesg: This log file contains the results of your last system boot. It displays the various kernel messages that appeared on the monitor during the boot phase. This log file is helpful when debugging boot time problems.

420 Part V Maintaining the Linux System

Know the different types of log files and what information they contain.

Because many of these log files can be very large, you may want to pipe it through the more command, or use grep to find only the keywords in the log file that you are looking for. For example, the following command searches the messages file for any reference to “mail:”

cat /var/log/messages | grep mail

Sometimes you may want to monitor a log file in real time in order to see the results of an application or process that you are testing. For this, you can use the tail command with the “follow” argument:

tail –f /var/log/messages

This command shows the messages file on the monitor screen. As entries are added to the log, they are immediately displayed.

Most log files are not set to roll over automatically, and you will find that these logs can grow very quickly, thus eating valuable disk space. Therefore, it’s a good idea to use some sort of scheduling program, such as cron, to automate rolling over log files.

The easiest way to automate file rollover is to create a simple batch file that renames the current log file so that the system can start writing a new log file with the original name. Most administrators name the files by date, so they can easily track the files when looking for historical data.

Enter a log rotation into your cron schedule to run daily or weekly as required.

Unattended log files can quickly fill your disk file systems. Pay close attention to their sizes, or use the cron process to automate log retention.

Maintaining Documentation

5.9 Document work performed on a system

Linux system administrators often overlook the task of documenting their systems. This can be a major oversight, because without proper documentation, you won’t have immediate access to the information that you need to restore a system in the event of a system crash or hardware failure.

Chapter 13 Process Management 421

You should keep accurate records of all the hardware and software on your system. Here are some of the more important items that you should be tracking:

Hardware

•Vendor serial and model number

•Processor speed and type, and available multi-CPU slots

•Amount of RAM, including number of chips and empty slots

•Disk space and configuration, including any RAID or fault tolerance settings

•Network card type, configuration, and network address

•Peripheral cards, such as SCSI, and video

Software

•Operating system version number, including kernel version and patches

•Versions of all other installed software

As part of your system documentation, you should also keep track of any hardware maintenance or software upgrades. You will greatly benefit from this information during troubleshooting, because you will always have an accurate record of what is currently running on the system, and what parts have been replaced or upgraded. This information is especially important when keeping track of kernel or software upgrades. By upgrading one component, you may create an incompatibility with another component. Only by keeping track of these changes can you go back to find which upgrade caused the problem.

You should routinely keep track of CPU, RAM, and disk usage so that you can proactively plan any resource upgrades that your system may need in the future. By plotting your usage trends, you can tell whether you need to order extra hard drives to correct any future disk space problems. Table 13-1 is an example of the type of information that you should be tracking.

Table 13-1

Linux System Maintenance Log

Date	Action Performed	Name
May 13, 2001	Upgraded Apache Web Server to 1.3.2	J. Smith
June 22, 2001	Upgraded kernel to 2.4	T. Green
July 5, 2001	Upgraded Squid Proxy Server to 2.4	J. Smith
July 15, 2001	Replaced failed hard drive in RAID 5 array — bay 2	J. Smith

422 Part V Maintaining the Linux System

Summary

This chapter details many commands that you can use for process management. Commands like ps and kill have many different options that the Linux system administrator can utilize. The administrator should also regularly check the various system log files for errors, warnings, and information items that can help to proactively monitor the system. This chapter also emphasizes the importance of maintaining accurate hardware, software, and maintenance information. Here are some key points to remember from this chapter:

ps: Lists current users’ processes

ps –e: Lists all processes

ps –f: Uses full listing

ps –u: Lists processes of a particular user

kill: Terminates a process

killall: Terminates several instances of the same process by name

kill –9: Sends a kill signal if a conventional kill command does not work

kill –HUP: Kills the process and restarts it

top: Real-time list of all processes and resource usage

System and kernel messages are written to /var/log/messages