Versatile Downloading with wget

91

To run Liquid Weather, click Open on the SuperKaramba menu page. Browse to ~/Desktop/themes/liquid_ weather_plus and choose liquid_weather.theme from the files listed. Double-click the icon to open a weather report on your desktop (see the following figure).

To customize Liquid Weather, right-click on the weather screen and choose Configure Theme Enter Your Location’s Code. Enter your zip code (or weather code if you’re in Britain) and choose OK, and the weather forecast is updated to your region.

Versatile Downloading with wget

wget uses the HTTP, HTTPS, and FTP protocols from the command line to retrieve files or Web sites. wget is handy if you have a slow or undependable Internet connection. If a connection drops partway through a download, wget keeps trying. If the server allows it, wget will continue the download where it left off.

wget has a lot of options that combine to make it a versatile download tool. You can use it in the following ways:

For recursive Web site downloads

For updating Web site mirrors

As a spider to verify links

For executing scheduled downloads

As a persistent agent to download large files

wget works quietly in the background with no further input from you. You can schedule wget (with Task Scheduler) to start downloads when network use is at its lowest so that you don’t interfere with other users. For more information about scheduling tasks with Task Scheduler, check out Technique 20.

Mirroring sites with wget

You might wonder why anyone would mirror a Web site. Many generous people who support the open source movement help provide the world with extra information and closer, quicker downloads by setting up servers and creating sites that mirror and distribute open source software and information.

You can also use site mirrors for quicker access to sites that you use often. Not only do you have quicker access to the site, but you can also take it with you anywhere you go — even without a Web connection.

Download an entire Web site recursively with the following command:

$ wget -r -k http://www.website.com

The -r in this command stands for recursive (meaning that wget copies the directory you name and all the files and subdirectories underneath it) — by default, wget copies five levels of subdirectories to your local system. The -k (or --convert-links if you want to type it out) redirects the links on those five levels to refer back to your local system. If you leave

92 Technique 14: Downloading and Uploading Files in a Snap

out the -k option, the documents that you download will still point back to the original Web site.

If you find yourself setting up a site mirror, either for internal use or for the world, you’ll want to keep it up-to-date. Schedule a job (with Task Scheduler) to run every night:

$wget -r --mirror -k http://www.website.com

The --mirror option checks your copy of the site against the version published on the Web, and downloads only those files that have changed.

Verifying your bookmarks with wget

If you’re anything like us, your bookmark collection is, well, a mess. Bookmarks accumulate over time and pages that you may have been interested in a few months (or years) ago might not be there any more. It’s a good idea to weed out obsolete links now and then just to keep your bookmark collection under control. Use wget to check all the links on your bookmarks or links page, with one easy command. To make wget impersonate a spider and investigate links, use this command:

$wget --spider --force-html -i bookmarks.html

wget visits each link, and reports successful or unsuccessful connections for each entry in your bookmarks page.

Forget where you left your bookmarks? Use the command locate -i bookmark to generate a list of all the files with the word bookmark in their name.

Downloading files with wget

To use wget to download a file from an FTP server, enter the following command:

$ wget ftp://www.sitename.com/filename

If the download is interrupted, resume the download with this command:

$ wget -c ftp://www.sitename.com/filename

The -c option instructs wget to resume the download where it left off.

Downloading and unpacking in one quick step

You can redirect the output of a wget download to a tar command to download and unpack in one easy step:

$ wget -O - http://tarball | tar -zxvf -

The -O option redirects the output to the tar command. tar then unpacks it to a subdirectory in your current directory.

wget’s optional flags

Dozens of flags work with wget — we’ve noted a few in Table 14-1. For a complete list, type man wget at the command line.

TABLE 14-1: HANDY WGET OPTIONS

Option	What It Does
-b	Goes to background after starting.
-q	Turns off the output of wget.
-v	Displays long debugging messages.
-nv	Displays errors or basic info only.
-t count	Tries count times before giving up.
-nc	Doesn’t overwrite files.

One other option worthy of mention is --limit-rate=bandwidth. Use this flag to limit the download speed so you don’t steal all the bandwidth away from other users on your system.

$ wget --limit-rate=20k