certain interactions. If you use an even number you might very well find that you have all of your inodes on one drive. You might even have all of your data on only one drive! This would utterly eradicate any advantages of striping. You want to be absolutely certain that Vinum will use all of your drives, and this number works.

The subdisk labels in our output look slightly different than in the earlier example, however. Since we're striping, we need to have subdisks of identical size. If one disk is larger than the other, that space will be useless because both disks will use the amount of space available on the smaller drive, wasting whatever space remains on the larger drive. If you intend to use Vinum striping, plan your drive partitions accordingly.

Once you have your new vinum.conf set up, run vinum create. The output of vinum create will look almost exactly like our previous example, but it will say "striped" instead of "concat." Again, examine it carefully to be sure it says what you want it to! Then run newfs and mount the new volume, and see what you have.

...............................................................................................

#mount /dev/vinum/test /mnt

#df

Filesystem	1K−blocks	Used	Avail Capacity		Mounted on
/dev/ad4s1a	248111	65146	163117	29%	/
devfs	1	1	0	100%	/dev
/dev/ad4s1f	2032839	133492	1736720	7%	/test1
/dev/ad4s1g	2032839	1266476	603736	68%	/test2
/dev/ad4s1h	29497862	4193767	22944267	15%	/usr
/dev/ad4s1e	3048830	123222	2681702	4%	/var
procfs	4	4	0	100%	/proc
/dev/vinum/test 6286319		1	5783413	0%	/mnt
#

...............................................................................................

This new /mnt partition is smaller than the concatenated volume, since we lost the extra space on one drive, but accesses will be faster since we're sharing the load across multiple disks. The faster access speed will be especially obvious with multiple processes writing to the volume simultaneously.

Mirrored Volumes

Finally, let's mirror our drives. To begin, first unmount the partition, eliminate your old Vinum configuration with vinum destroy, and create a new /etc/vinum.conf.

...............................................................................................

drive alpha device /dev/da0s1e drive beta device /dev/da1s1e volume mirrortest setupstate plex org concat

sd length 3243037k drive alpha plex org concat

sd length 3243037k drive beta

...............................................................................................

This configuration is a bit different from our previous ones. Our drives are the same, but everything else looks a little different. For one, our volume has a different name, mirrortest. The setupstate keyword tells vinum(8) to assume that the existing contents of the disks are identical—we're lying to

400

Vinum. Lying to your computer is not good. The computer will usually catch on quite quickly and panic. (This leads directly to Chapter 20.) Before we use the volume, however, we will newfs it and overwrite everything, so Vinum will never have an opportunity to catch on.[3]

This time we have two plexes, each of the concatenated type. Each plex is concatenated, but has only one subdisk, each of which lives on a separate disk. Our first plex has one subdisk on drive alpha, while the other has one subdisk on drive beta. Again, they are the same size.

We know that a plex is one copy of the data. Since we have two plexes, each on a separate disk, we know that the data is mirrored across the disks. Now that you have a configuration, use the vinum command to implement it on the actual hard drives.

...............................................................................................

#vinum create vinum.conf
2	drives:
D alpha		State: up	/dev/da0s1e		A: 384/3551 MB (10%)
D beta		State: up	/dev/da1s1e		A: 619/3786 MB (16%)
1	volumes:
V mirrortest		State: up	Plexes:	2	Size:	3167	MB
2	plexes:
P mirrortest.p0		C State: up	Subdisks:	1	Size:	3167	MB
P mirrortest.p1		C State: up	Subdisks:	1	Size:	3167	MB
2	subdisks:
S mirrortest.p0.s0		State: up	D: alpha		Size:	3167	MB
S mirrortest.p1.s0		State: up	D: beta		Size:	3167	MB
#

...............................................................................................

You have a mirrored volume, as you can see by the two plexes listed. Newfs, mount, and you're ready to go!

Starting Vinum at Boot

To start the Vinum system at boot, edit /etc/rc.conf and /etc/fstab. The rc.conf option is quite simple:

...............................................................................................

start_vinum="YES"

...............................................................................................

We looked at /etc/fstab in Chapter 16. Follow the examples already in that file, and add a line something like this to the filesystem table:

...............................................................................................

/dev/vinum/mailvol /var/mail ufs rw 2 2

...............................................................................................

401

Other Vinum Commands

The vinum(8) program includes a variety of simple commands to allow you to view its condition.

Status Check

The simplest thing of all is to ask Vinum how it's doing. The vinum ls command shows each plex you have, its size, the name of its drive, and its state:

...............................................................................................

# vinum ls
S mirrortest.p0.s0	State: I	43%	D: alpha	Size:	3167	MB
S mirrortest.p1.s0	State: I	41%	D: beta	Size:	3167	MB
#

...............................................................................................

Everything should all be in an "I" state. If it isn't, check vinum(8) and the Vinum Web site (http://www.vinumvm.org/) and see what your error means. If you want more information, you can do vinum list −r. This will print out detailed status information on every volume, plex, and subdisk in your system.

Configuration Check

To double−check your Vinum system configuration, use vinum printconfig. This shows us the drive names, followed by the volume name, and then the plexes and subdisks.

...............................................................................................

#vinum printconfig

# Vinum configuration of magpire.blackhelicopters.org, saved at Fri Aug 17 19:24:43 2001

drive alpha device /dev/da0s1e drive beta device /dev/da1s1e volume mirrortest

plex name mirrortest.p0 org concat vol mirrortest plex name mirrortest.p1 org concat vol mirrortest

sd name mirrortest.p0.s0 drive alpha plex mirrortest.p0 len 6486074s driveoffset 265s plexoffset 0s

sd name mirrortest.p1.s0 drive beta plex mirrortest.p1 len 6486074s driveoffset 265s plexoffset 0s

#

...............................................................................................

Replacing a Failed Mirrored Plex

The purpose of a mirror is redundancy in case of a drive failure. So, if a drive fails, how do you replace it?[4]

First, if at all possible, be sure you have a good backup. You'll be working directly on the Vinum device, and any mistake can destroy your data. If you don't have a current backup, take a backup before proceeding. Your remaining plex(es) should still be up and serving data, after all!

402

Next, you need to replace the failed hard drive, which can be difficult. For one thing, you've got to identify the failed drive. A vinum list will tell you which of your drives has failed, identifying the drive by device number. You can then compare the device name to those listed in /varx/run/dmesg.boot and get the SCSI ID, LUN, and bus number. Once you identify the failed drive, then shut down the computer, find the matching hard drive, replace it with a drive with the same settings, and boot up again.

Once you reboot, then check to be certain that the new drive came up with the same drive device name as the old one. (This will make later configuration simpler.)

Now, ask Vinum how it's doing:

...............................................................................................

# vinum list
1	drives:
D alpha		State: up	/dev/da0s1e		A: 384/3551 MB (10%)
D beta		State: referenced			A: 0/0	MB
1	volumes:
V mirrortest		State: up	Plexes:	2	Size:	3167	MB
2	plexes:
P mirrortest.p0		C State: up	Subdisks:	1	Size:	3167	MB
P mirrortest.p1		C State: faulty	Subdisks:	1	Size:	3167	MB
2	subdisks:
S mirrortest.p0.s0		State: up	D: alpha		Size:	3167	MB
S mirrortest.p1.s0		State: stale	D: beta		Size:	3167	MB
#

...............................................................................................

The beta drive that was on /dev/da1s1 shows up as "referenced," which means that it's included in the configuration but isn't working correctly. In a cascading chain of errors, the plex on this particular drive is "faulty" because the drive isn't working. The underlying subdisk is "stale," as the data on it does not match the mirror.

None of this is good, but it's all to be expected. As a virgin replacement, the disk partition we've called "beta" has nothing whatsoever on it. If you see any of these errors on an existing Vinum volume, however, it means that the drive in question is having trouble and should be replaced.

Preparing the Replacement Drive

Before we can use the new drive, we need to partition and label it. You did that earlier in this chapter with sysinstall(8), and this will be almost exactly the same. To begin, start sysinstall with the fdisk menu option. Next, delete any disk partitions shown, and dedicate the whole drive to FreeBSD. Then go into the Label editor.

From our previous setup in /etc/vinum.conf, we know that our Vinum partition was 3243037KB. We have a slightly more convenient measurement in the earlier vinum ls, where we can see that one subdisk is 3167MB. We must leave at least that much space for our Vinum partition on our new drive. (I recommend that you allocate a couple extra meg, just in case, if possible.) Now, write your changes to the disk, as if you were adding a disk to your system, and then run disklabel to change the slice type of your Vinum partition, as if you were first setting up Vinum.

403

Note At the rate disk drive size grows these days, it's quite likely that a replacement disk will be much larger than the original, and probably even faster. Feel free to allocate the remainder of the disk. When I have extra space left over, I frequently move a little−used chunk of data—say, /usr/src or /usr/ports/distfiles—onto a large new partition. It's accessed infrequently enough that it won't inflict a great performance hit upon my Vinum system, and yet frees disk space elsewhere. Move data that doesn't need the resilience and speed that Vinum provides, and that is easily recoverable.

Plex Recovery

Finally, you're ready to recover your volume. First off, you need to tell Vinum that it has a new disk. Remember, Vinum's configuration is cumulative. If you just recreate the whole volume, you'll add the old information to itself. This will confuse Vinum and make you lose your data. For example, our vinum.conf shows drives named alpha and beta. Vinum already knows about those drives. If I were to use that same vinum.conf and run vinum create a second time, we would be telling Vinum that it had two alpha drives and two beta drives, both on the same physical device! This would be vastly confusing to us, let alone a poor, dumb machine. Similarly, you don't want to obliterate the entire configuration, and make it forget about the drives it already has. All you want to do is change a bit of the existing setup.

In this example, we want to recover drive beta. Since we don't care about what's on the shiny new replacement drive beta, we can overwrite that part of Vinum's configuration. To do so, create a Vinum configuration file, /etc/vinum.conf.recover. List only the one drive in there, without any additional information on plexes or subdisks.

...............................................................................................

drive beta device /dev/da1s1e

...............................................................................................

That's the drive name and the partition it's on. Now, we add this information to the existing Vinum configuration by running vinum create:

...............................................................................................

# vinum create /etc/vinum.conf.recover
2	drives:
D alpha		State: up	/dev/da0s1e		A: 384/3551 MB (10%)
D beta		State: up	/dev/da1s1e		A: 2/3170 MB (0%)
1	volumes:
V mirrortest		State: up	Plexes:	2	Size:	3167	MB
2	plexes:
P mirrortest.p0		C State: up	Subdisks:	1	Size:	3167	MB
P mirrortest.p1		C State: faulty	Subdisks:	1	Size:	3167	MB
2	subdisks:
S mirrortest.p0.s0		State: up	D: alpha		Size:	3167	MB
S mirrortest.p1.s0		State: stale	D: beta		Size:	3167	MB
#

...............................................................................................

Notice that the state on our beta drive is now "up," instead of "referenced." The plex is still "faulty," mind you, but we're making progress.

404

Now that you have a drive, you can actually tell Vinum to recover the disk. Vinum is fairly smart; it doesn't start drives it cannot talk to. All you have to do is tell Vinum to start the replaced plex, and it will recover it for you:

...............................................................................................

#vinum start mirrortest.p1

Reviving mirrortest.p1.s0 in the background

# vinum[685]: reviving mirrortest.p1.s0

#

...............................................................................................

You can watch its progress with vinum list:

...............................................................................................

# vinum list
2	drives:
D alpha		State: up	/dev/da0s1e		A: 384/3551 MB (10%)
D beta		State: up	/dev/da1s1e		A: 2/3170 MB (0%)
1	volumes:
V mirrortest		State: up	Plexes:	2	Size:	3167	MB
2	plexes:
P mirrortest.p0		C State: up	Subdisks:	1	Size:	3167	MB
P mirrortest.p1		C State: faulty	Subdisks:	1	Size:	3167	MB
2	subdisks:
S mirrortest.p0.s0		State: up	D: alpha		Size:	3167	MB
S mirrortest.p1.s0		State: R 11%	D: beta		Size:	3167	MB
#

...............................................................................................

Note the last line: mirrortest.p1.s0. Our replacement disk is 11 percent recovered. If your mirror is under heavy load, this will take a while. Eventually, however, it will recover, and you'll be back in business with fully mirrored disks.

Pretty cool, huh?

[2]You might notice that disklabel claims that there are eight partitions, but we only see three. The slice can hold up to eight partitions, but five aren't actually in use.

[3]This is the computing equivalent of writing your rent check on the night before payday.

[4]As I was writing this chapter, I noticed a high−pitched noise coming from my computer. Upon investigation, one of my two SCSI hard drives was warm enough to make me jerk my hand back quickly. Since the computer room is kept at about 65 degrees Fahrenheit, it seemed to be time to replace the drive. Sometimes, Fate works with technical folks. Other times, it just picks a decent time to abuse us.

405