exact setup that you have on a panicking system. While many of them would be happy to set up such a system in exchange for lavish amounts of hard currency, it's a bit much to expect for free.

[1]How did I know this? I exchanged several emails with a kernel developer about this dump, that's how.

Serial Consoles

Being able to gather debugging information is nice, but what if you need to do so remotely? Or what if you need to reset the machine remotely when it isn't responding to the network? This is best done with a serial console. If I had had a serial console on my first panicking system, I wouldn't have been juggling pen, paper, and monitor.

Real UNIX hardware (such as Alpha and SPARC) has a hardware serial console capability. On these systems, you can attach a serial cable to the serial console port and have unfettered access to the BIOS, boot messages, and startup controls. (Most x86 hardware does not allow this; you must be at the console to look at the BIOS or to press the space bar to interrupt the boot. A very few Intel motherboards, such as the L440GX, do have this functionality, but this is a special feature you must hunt for.)

This can be a problem when your FreeBSD system is in a colocation facility on the other side of the country. You have a couple of options here: a hardware serial console or FreeBSD's built−in software serial console. Either console requires an accessible serial device nearby. If you have two FreeBSD boxes in one location, you can plug them into each other. If you have a huge array of FreeBSD boxes, companies such as Lucent make network−accessible "terminal servers" that do nothing but handle gobs of serial connections.

Hardware Serial Console

Nothing any operating system can do will give you access to the PC−compatible BIOS messages across a serial port. This stuff happens before the operating system starts and before the hard drive is read.

Some hardware solutions do exist to work around this. The best I've seen is the PC Weasel (http://www.realweasel.com/). It's a video board with a serial port instead of a video port. By connecting a serial cable to the Weasel, you can manipulate the BIOS remotely, interrupt the boot to come up in single−user mode, and generally do whatever you like with the system as if you were at the console. (There are other manufacturers of similar devices, but they either require proprietary client software or are far more expensive.)

Software Serial Console

FreeBSD includes a software serial console. As FreeBSD boots, it decides where to put its console—on the monitor and keyboard by default. With a few tweaks, though, you can have the console come up on a serial port, but your system must have a serial port. (Some hardware is increasingly arriving "legacy−free," which means that it lacks serial ports and ISA slots; you may need to buy a PCI serial card.)

This serial console has a couple of disadvantages: It does not kick in until FreeBSD's boot loader starts and you will not see BIOS messages, though you will get a chance to interact with the boot

465

process. Still, it's good enough for most uses.

Software Serial Console Physical Setup

You must have a null modem cable to use a serial console (available at any computer store or from online vendors—check pricewathc.com). Get the best cable you can find; if you have an emergency and need to use the serial console, you're probably not in the mood to deal with line noise.

Plug one end of the null modem cable into the serial console port on your FreeBSD server—by default the first serial port (COM1 or sio0, depending on what operating system you're used to). You can change this with a kernel recompile, but it's generally simpler to just use the default on a server.

Plug the other end of your null modem cable into an open serial port on another system. I recommend either another FreeBSD or other UNIX system, or a terminal server. You can use a Windows system, but that won't give you any remote−control functionality. (Yes, you can use VNC or PC Anywhere on the Windows system, but you're starting to look at a complicated setup when a simple FreeBSD box would suffice.)

If you have two FreeBSD machines at a remote location, and want to use serial consoles for both of them, simply attach the console cable to the second serial port on the other server. If you have three machines, you can daisy−chain them into a loop. By combining twos and threes, you should be able to get serial consoles on any number of FreeBSD systems. (I've worked in areas with 30 or 40 FreeBSD machines in one room, where installing monitors was simply not practical, and serial consoles were used to great effect.)

Note If tracking which machine is attached to which port becomes a problem, invest in a terminal server.

Configuring the Software Serial Console

If you're using the serial console exclusively, tell the system to use it by adding an entry in /boot/loader.conf:

...............................................................................................

console="comconsole"

...............................................................................................

To switch back to the default video console, remove the line or comment it out. You could also set this explicitly in /boot/loader.conf with this line:

...............................................................................................

console="vidconsole"

...............................................................................................

Changing the Configuration

If you're in a server−room situation, you may find that you want to switch back and forth from a standard console to a serial console. I generally manage large arrays of FreeBSD boxes via the serial console. If one particular machine is exceptionally troublesome, I might go and put a real console on it; I don't want to reconfigure /boot/loader.conf to make the physical console work—I want it to "just happen." You can do this easily with FreeBSD.

466

One of the first things any x86 system does upon boot is check for the presence of a keyboard. You can set up your FreeBSD system so that it will use a regular console if it detects a keyboard, and a serial console if it doesn't. (It's the best solution we have with inflexible x86 hardware.) To make this work, edit the file /boot.config by putting only this in it:

...............................................................................................

−h

...............................................................................................

This is the argument passed to the system boot, much as if you interrupted the boot and typed boot −h. (Of course, to really do that, you would need to have a keyboard plugged in, so that would render the whole thing kind of moot!)

If you have −h in /boot.config, and you don't have a keyboard, the system will use the serial console. If there is a keyboard, the system will use the video console. This method of doing the check is rather kludgy, but it's good enough if you know the rules.

Using a Serial Console

You can access the serial console in a variety of ways, all of which require a second computer. If your second computer is running a Microsoft operating system, the Hyperterm program gives you access to your serial ports (just set your terminal preferences to 9600 baud, 8 bits, no parity, and 1 stop bit). To use a handheld with a serial port (I frequently use a Handspring Visor with a serial cradle), run one of the several free vt100 terminal programs on it. If your second computer also runs FreeBSD (as it should if you want the maximum bang for your buck), you can use FreeBSD's terminal program. Since this is a FreeBSD book, this is the solution we'll discuss.

FreeBSD accesses serial lines with tip(1), a program that allows you to connect to a remote system in a manner similar to telnet. To run tip, do this:

...............................................................................................

# tip portname

...............................................................................................

A port name is shorthand for specifying the number and speed to be used on a serial port. The file /etc/remote contains a list of port names. Most of the entries in this file are relics of the day when UUCP was the major data−transfer protocol and serial terminals were the norm instead of the exception. At the end of this file, however, you'll see a few entries like this:

...............................................................................................

#Hardwired line cuaa0b|cua0b:dv=/dev/cuaa0:br#2400:pa=none: cuaa0c|cua0c:dv=/dev/cuaa0:br#9600:pa=none:

#Finger friendly shortcuts com1:dv=/dev/cuaa0:br#9600:pa=none: com2:dv=/dev/cuaa1:br#9600:pa=none: com3:dv=/dev/cuaa2:br#9600:pa=none: com4:dv=/dev/cuaa3:br#9600:pa=none:

...............................................................................................

Older UNIX hands will recognize cuaa0b and cuaa0c. (The "com" entries were added for the convenience of people who have grown up with Windows.) Assume that you have two FreeBSD

467

boxes wired back−to−back, with each one's serial port 1 null−modemed into serial port 2. You'll want to connect to your local serial port 2 to talk to the other system's serial console:

...............................................................................................

# tip com2

connected

...............................................................................................

And you won't see anything else, no matter what you type.

If you log into the other system and reboot it, you'll abruptly see action in your tip window:

...............................................................................................

boot() called on cpu#1
Waiting (max 60 seconds) for system process `bufdaemon` to stop		...stopped
Waiting (max 60 seconds) for system process `syncer` to stop...		stopped
syncing disks...	8 8 2 2
done
Uptime: 21m20s
Rebooting...

cpu_reset called on cpu#1 cpu_reset: Stopping other CPUs cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 1

...............................................................................................

There will be a long pause while the system boots. If you're near the system, you'll see the standard BIOS messages flash by. Eventually, you'll see something like this:

...............................................................................................

Console: serial port BIOS drive A: is disk0 BIOS drive C: is disk1

BIOS 639kB/523200kB available memory

FreeBSD/i386 bootstrap loader, Revision 1.0 ([e-mail], Sat Jul 7 18:40:47 EDT 2001) Loading /boot/defaults/loader.conf

/boot/kernel/kernel text=0x1d7b58 data=0x3b28c+0x5132c syms=[0x4+0x30420+0x4+0x3a852]

/

Hit [Enter] to boot immediately, or any other key for command prompt. Booting [/boot/kernel/kernel] in 10 seconds...

...............................................................................................

Hit the space bar to interrupt the boot.

At this point, it's just like you're at the keyboard. It doesn't matter if the system is 1,000 miles away; you can change your booting kernel, get a verbose boot, bring it up in single−user mode and manually fsck the hard drive, whatever. The serial console can't save you from a BIOS or hardware failure, but it will make other things much simpler to diagnose.

Type boot to continue loading the kernel. Eventually you'll see this:

468