86 Technique 13: Moving Made Easy with Archives

Archiving complex search results

Use the pipe character ( | ) to combine programs like find and rpm with the tar command to create archives that contain the results of complex searches.

Using tar with find can seem complex, but it is very useful. One example of a combined command is as follows:

$find / -user freddie | tar -zcvf fredfiles -T -

This command finds all the files owned by the user Freddie and sends the output (the list of filenames) to the tar command. The -T - portion of the command instructs tar to read the list of filenames from its standard input (which, in this case, is the output of the find command) rather than from the command line.

For more in-depth information about using the find command, see Technique 12.

Backing up an installed package

Use tar with rpm to create a backup of an installed package. To back up an installed copy of the webmin package, use this command:

$rpm -ql webmin | tar -cvf webminbackup -T -

For more information about using rpm queries, see Technique 16.

Uprooting Entire Directory Trees with scp

Sometimes, you need to move more than a single file — you need to move an entire directory tree (a directory and all the files and subdirectories

underneath it). When that’s the case, use scp to get the job done quickly and easily. For example:

If you carry your work to and from the office on a laptop, use the scp to copy files from your laptop to your home computer (and back again).

You can also use the scp -r command to quickly move a user from one machine to another.

If you’re upgrading to a new system, scp -r is an easy way to quickly transfer your work with no disruptions.

scp was designed to copy files from one computer to another. You can also use scp to copy a file from one place to another within your computer, just like you would use cp. We find scp to be much more intuitive when it comes to copying directory trees.

To move a directory tree with scp, open your terminal window and enter this command:

$scp -r user@host:source user@host: destination

That’s all there is to it. The -r flag tells scp to copy source and everything underneath it.

Table 13-2 highlights two options worthy of mention.

TABLE 13-2: WORTHWHILE SCP OPTIONS

Option	What It Does
-C	Compresses the data stream for faster
	transfers.
-l limit	Throttles file transfers to no more than
	limitK bits per second. (Use this option if
	you’re sharing a network connection and you
	don’t want to hog all the bandwidth).

Getting familiar with scp (and its secure shell cousin, ssh) is definitely worth the time. scp is a fast, secure, and easy way to move files and archives from one location to another. scp and ssh share many command-line options because scp is built from ssh. For more information about ssh, see Technique 33.

Splitting Big Files into Manageable Chunks

87

Splitting Big Files into

Manageable Chunks

While you’re working across the Web or across a network, the inevitable happens: You lose the network connection mid-upload. You have to go all the way back to the beginning and start the transfer over.

ISPs are known to place limits on the size of incoming files. E-mails with oversized attachments are returned undelivered and unseen by the recipients. How can you get around that?

To transfer a large file to a user with limited access (or over a questionable connection), use the split command. split doesn’t actually speed up the transfer, but it does speed up the recovery if a connection drops.

split breaks a file (any file — archives, pictures, data . . . you name it) into segments that you can reassemble on the other end.

To reassemble the split file accurately, all the pieces must be included. split can’t tell if they’re all there or not — it just re-assembles what it has. If great-aunt Gertrude’s nose looks a bit off, you may have lost a segment.

Use the following command to break a file into 1-megabyte segments for transfer:

$split --bytes=1m filetosplit segmentprefix

split appends the segmentprefix with a unique suffix. When it’s finished, you still have the original file, but you also have a set of 1 megabyte segments. If you started with a 2.5 megabyte file, you end up with three segments: The first two contain 1 megabyte each, and the third file contains the leftovers.

It’s a good idea to calculate an MD5 checksum on the original file to compare it to the reassembled result. Save the number generated by the following command — you’ll need it later:

$ md5sum filetosplit

md5 stands for message digest #5. It’s a cryptographic program that’s good at detecting differences between files. It’s kind of like a fingerprint for a file.

Send the checksum with the attachments or save them to compare to the checksum of the reassembled file. If the checksums match, you can be sure that the entire file was received and reassembled.

It’s easy to move all the segments securely with one scp command:

$ scp segmentprefix.* user@host:directory

To rebuild the file after the upload, use ssh to log in to the remote machine, and use cd to move to the directory containing the segments.

To reassemble the segments, enter this command:

$ cat segmentprefix.* > filename

cat rebuilds the file into its original structure.

After the file is rebuilt, run a new MD5 checksum and compare it to the fingerprint of the original file. The two fingerprints should be identical.

$ md5sum originalfilename

If you’ve sent the split file to a friend running Windows, the type command will concatenate split files on Windows.