Chapter 12. External Filters, Programs and
Commands
Standard UNIX commands make shell scripts more versatile. The power of scripts comes from coupling system commands and shell directives with simple programming constructs.
12.1. Basic Commands
Command Listing
ls
The basic file "list" command. It is all too easy to underestimate the power of this humble command. For example, using the −R, recursive option, ls provides a tree−like listing of a directory structure. Other interesting options are −S, sort listing by file size, −t, sort by file modification time, and −i, show file inodes (see Example 12−3).
Example 12−1. Using ls to create a table of contents for burning a CDR disk
#!/bin/bash |
|
SPEED=2 |
# May use higher speed if supported. |
IMAGEFILE=cdimage.iso |
|
CONTENTSFILE=contents
DEFAULTDIR=/opt
#Script to automate burning a CDR.
#Uses Joerg Schilling's "cdrecord" package.
#(http://www.fokus.gmd.de/nthp/employees/schilling/cdrecord.html)
#If this script invoked as an ordinary user, need to suid cdrecord
#(chmod u+s /usr/bin/cdrecord, as root).
if [ −z "$1" ] then
IMAGE_DIRECTORY=$DEFAULTDIR
# Default directory, if not specified on command line. else
IMAGE_DIRECTORY=$1
fi
ls −lRF $IMAGE_DIRECTORY > $IMAGE_DIRECTORY/$CONTENTSFILE
#The "l" option gives a "long" file listing.
#The "R" option makes the listing recursive.
#The "F" option marks the file types (directories get a trailing /). echo "Creating table of contents."
mkisofs −r −o $IMAGFILE $IMAGE_DIRECTORY
echo "Creating ISO9660 file system image ($IMAGEFILE)."
cdrecord −v −isosize speed=$SPEED dev=0,0 $IMAGEFILE
Chapter 12. External Filters, Programs and Commands |
136 |
Advanced Bash−Scripting Guide
echo "Burning the disk."
echo "Please be patient, this will take a while."
exit 0
cat, tac
cat, an acronym for concatenate, lists a file to stdout. When combined with redirection (> or >>), it is commonly used to concatenate files.
cat filename cat file.1 file.2 file.3 > file.123
The −n option to cat inserts consecutive numbers before all lines of the target file(s). The −b option numbers only the non−blank lines. The −v option echoes nonprintable characters, using ^ notation.
See also Example 12−21 and Example 12−17.
tac, is the inverse of cat, listing a file backwards from its end.
rev
reverses each line of a file, and outputs to stdout. This is not the same effect as tac, as it preserves the order of the lines, but flips each one around.
bash$ cat file1.txt
This is line 1. This is line 2.
bash$ tac file1.txt
This is line 2. This is line 1.
bash$ rev file1.txt
.1 enil si sihT
.2 enil si sihT
cp
This is the file copy command. cp file1 file2 copies file1 to file2, overwriting file2 if it already exists (see Example 12−5).
Particularly useful are the −a archive flag (for copying an entire directory tree) and the −r and −R recursive flags.
mv
This is the file move command. It is equivalent to a combination of cp and rm. It may be used to move multiple files to a directory, or even to rename a directory. For some examples of using mv in a script, see Example 9−14 and Example A−3.
rm
Chapter 12. External Filters, Programs and Commands |
137 |
Advanced Bash−Scripting Guide
Delete (remove) a file or files. The −f forces removal of even readonly files.
When used with the recursive flag −r, this command removes files all the way down the directory tree.
rmdir
Remove directory. The directory must be empty of all files, including invisible "dotfiles", [28] for this command to succeed.
mkdir
Make directory, creates a new directory. mkdir −p project/programs/December creates the named directory. The −p option automatically creates any necessary parent directories.
chmod
Changes the attributes of an existing file (see Example 11−8).
chmod +x filename
# Makes "filename" executable for all users.
chmod u+s filename
#Sets "suid" bit on "filename" permissions.
#An ordinary user may execute "filename" with same privileges as the file's owner.
#(This does not apply to shell scripts.)
chmod 644 filename
#Makes "filename" readable/writable to owner, readable to
#others
#(octal mode).
chmod 1777 directory−name
#Gives everyone read, write, and execute permission in directory,
#however also sets the "sticky bit".
#This means that only the owner of the directory,
#owner of the file, and, of course, root
#can delete any particular file in that directory.
chattr
Change file attributes. This has the same effect as chmod above, but with a different invocation syntax, and it works only on an ext2 filesystem.
ln
Creates links to pre−existings files. Most often used with the −s, symbolic or "soft" link flag. This permits referencing the linked file by more than one name and is a superior alternative to aliasing (see Example 5−6).
ln −s oldfile newfile links the previously existing oldfile to the newly created link, newfile.
Chapter 12. External Filters, Programs and Commands |
138 |
Advanced Bash−Scripting Guide
12.2. Complex Commands
Command Listing
find
−exec COMMAND \;
Carries out COMMAND on each file that find scores a hit on. COMMAND terminates with \; (the ; is escaped to make certain the shell passes it to find literally, which concludes the command sequence). If COMMAND contains {}, then find substitutes the full path name of the selected file.
bash$ find ~/ −name '*.txt'
/home/bozo/.kde/share/apps/karm/karmdata.txt
/home/bozo/misc/irmeyc.txt
/home/bozo/test−scripts/1.txt
find /home/bozo/projects −mtime 1
#Lists all files in /home/bozo/projects directory tree
#that were modified within the last day.
find /etc −exec grep '[0−9][0−9]*[.][0−9][0−9]*[.][0−9][0−9]*[.][0−9][0−9]*' {} \;
#Finds all IP addresses (xxx.xxx.xxx.xxx) in /etc directory files.
#There a few extraneous hits − how can they be filtered out?
#Perhaps by:
find /etc −type f −exec cat '{}' \; | tr −c '.[:digit:]' '\n' \ | grep '^[^.][^.]*\.[^.][^.]*\.[^.][^.]*\.[^.][^.]*$'
#[:digit:] is one of the character classes
#introduced with the POSIX 1003.2 standard.
#Thanks, S.C.
The −exec option to find should not be confused with the exec shell builtin.
Example 12−2. Badname, eliminate file names in current directory containing bad characters and whitespace.
#!/bin/bash
# Delete filenames in current directory containing bad characters.
for filename in * do
badname=`echo "$filename" | sed −n /[\+\{\;\"\\\=\?~\(\)\<\>\&\*\|\$]/p`
# Files containing those nasties: |
+ { ; " \ = ? ~ ( ) < > & * | $ |
|
rm $badname 2>/dev/null |
# So error |
messages deep−sixed. |
done |
|
|
# Now, take care of files containing all manner of whitespace.
12.2. Complex Commands |
139 |
Advanced Bash−Scripting Guide
find . −name "* *" −exec rm −f {} \;
#The path name of the file that "find" finds replaces the "{}".
#The '\' ensures that the ';' is interpreted literally, as end of command.
exit 0
#−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−
#Commands below this line will not execute because of "exit" command.
#An alternative to the above script:
find . −name '*[+{;"\\=?~()<>&*|$ ]*' −exec rm −f '{}' \; exit 0
# (Thanks, S.C.)
Example 12−3. Deleting a file by its inode number
#!/bin/bash
#idelete.sh: Deleting a file by its inode number.
#This is useful when a filename starts with an illegal character, #+ such as ? or −.
ARGCOUNT=1 E_WRONGARGS=70 E_FILE_NOT_EXIST=71 E_CHANGED_MIND=72
# Filename arg must be passed to script.
if [ $# −ne "$ARGCOUNT" ] then
echo "Usage: `basename $0` filename" exit $E_WRONGARGS
fi
if [ ! −e "$1" ] then
echo "File \""$1"\" does not exist." exit $E_FILE_NOT_EXIST
fi
inum=`ls −i | grep "$1" | awk '{print $1}'`
#inum = inode (index node) number of file
#Every file has an inode, a record that hold its physical address info.
echo; echo −n "Are you absolutely sure you want to delete \"$1\" (y/n)? " read answer
case "$answer" in
[nN]) echo "Changed your mind, huh?" exit $E_CHANGED_MIND
;;
echo "Deleting file \"$1\".";;
find . −inum $inum −exec rm {} \; echo "File "\"$1"\" deleted!"
exit 0
See Example 12−22 , Example 4−3 , and Example 10−8 for scripts using find. Its manpage provides more detail on this complex and powerful command.
12.2. Complex Commands |
140 |
Advanced Bash−Scripting Guide
xargs
A filter for feeding arguments to a command, and also a tool for assembling the commands themselves. It breaks a data stream into small enough chunks for filters and commands to process. Consider it as a powerful replacement for backquotes. In situations where backquotes fail with a too many arguments error, substituting xargs often works. Normally, xargs reads from stdin or from a pipe, but it can also be given the output of a file.
The default command for xargs is echo.
ls | xargs −p −l gzip gzips every file in current directory, one at a time, prompting before each operation.
An interesting xargs option is −n NN, which limits to NN the number of arguments passed.
ls | xargs −n 8 echo lists the files in the current directory in 8 columns.
Another useful option is −0, in combination with find −print0 or grep −lZ. This allows handling arguments containing whitespace or quotes.
find / −type f −print0 | xargs −0 grep −liwZ GUI | xargs −0 rm −f
grep −rliwZ GUI / | xargs −0 rm −f
Either of the above will remove any file containing "GUI". (Thanks, S.C.)
Example 12−4. Logfile using xargs to monitor system log
#!/bin/bash
#Generates a log file in current directory
#from the tail end of /var/log/messages.
#Note: /var/log/messages must be world readable
#if this script invoked by an ordinary user.
##root chmod 644 /var/log/messages
LINES=5
( date; uname −a ) >>logfile
# Time and machine name
echo −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− >>logfile tail −$LINES /var/log/messages | xargs | fmt −s >>logfile
echo >>logfile
12.2. Complex Commands |
141 |
Advanced Bash−Scripting Guide
echo >>logfile
exit 0
Example 12−5. copydir, copying files in current directory to another, using xargs
#!/bin/bash
#Copy (verbose) all files in current directory
#to directory specified on command line.
if [ −z "$1" ] # Exit if no argument given. then
echo "Usage: `basename $0` directory−to−copy−to" exit 65
fi
ls . | xargs −i −t cp ./{} $1
#This is the exact equivalent of
#cp * $1
#unless any of the filenames has "whitespace" characters.
exit 0
expr
All−purpose expression evaluator: Concatenates and evaluates the arguments according to the operation given (arguments must be separated by spaces). Operations may be arithmetic, comparison, string, or logical.
expr 3 + 5
returns 8
expr 5 % 3
returns 2
y=`expr $y + 1`
Increment a variable, with the same effect as let y=y+1 and y=$(($y+1)) This is an example of arithmetic expansion.
z=`expr substr $string $position $length`
Extract substring of $length characters, starting at $position.
Example 12−6. Using expr
#!/bin/bash
#Demonstrating some of the uses of 'expr'
#=======================================
echo
12.2. Complex Commands |
142 |
Advanced Bash−Scripting Guide
#Arithmetic Operators
#−−−−−−−−−− −−−−−−−−−
echo "Arithmetic Operators" echo
a=`expr 5 + 3` echo "5 + 3 = $a"
a=`expr $a + 1` echo
echo "a + 1 = $a"
echo "(incrementing a variable)"
a=`expr 5 % 3`
# modulo echo
echo "5 mod 3 = $a"
echo echo
#Logical Operators
#−−−−−−− −−−−−−−−−
#Returns 1 if true, 0 if false,
#+ opposite of normal Bash convention.
echo "Logical |
Operators" |
|
|
echo |
|
|
|
x=24 |
|
|
|
y=25 |
|
|
|
b=`expr $x = $y` |
# Test equality. |
||
echo "b = $b" |
|
# 0 |
( $x −ne $y ) |
echo |
|
|
|
a=3 |
|
|
|
b=`expr $a \> |
10` |
|
|
echo 'b=`expr |
$a \> 10`, therefore...' |
||
echo "If a > 10, b = 0 (false)" |
|||
echo "b = $b" |
|
# 0 |
( 3 ! −gt 10 ) |
echo |
|
|
|
b=`expr $a \< |
10` |
|
|
echo "If a < 10, b = 1 (true)" |
|
||
echo "b = $b" |
|
# 1 |
( 3 −lt 10 ) |
echo |
|
|
|
# Note escaping of operators. |
|
||
b=`expr $a \<= 3` |
|
|
|
echo "If a <= |
3, b = 1 (true)" |
|
|
echo "b = $b" |
|
# 1 |
( 3 −le 3 ) |
# There is also a "\>=" operator (greater than or equal to).
echo echo
#Comparison Operators
#−−−−−−−−−− −−−−−−−−−
12.2. Complex Commands |
143 |
Advanced Bash−Scripting Guide
echo "Comparison Operators" echo
a=zipper
echo "a is $a"
if [ `expr $a = snap` ]
# Force re−evaluation of variable 'a' then
echo "a is not zipper"
fi
echo echo
#String Operators
#−−−−−− −−−−−−−−−
echo "String Operators" echo
a=1234zipper43231
echo "The string being operated upon is \"$a\"."
#length: length of string b=`expr length $a`
echo "Length of \"$a\" is $b."
#index: position of first character in substring
#that matches a character in string b=`expr index $a 23`
echo "Numerical position of first \"2\" in \"$a\" is \"$b\"."
#substr: extract substring, starting position & length specified b=`expr substr $a 2 6`
echo "Substring of \"$a\", starting at position 2, and 6 chars long is \"$b\"."
#'match' operations similarly to 'grep'
#uses Regular Expressions
b=`expr match |
"$a" |
'[0−9]*'` |
|
echo Number of digits at the beginning of \"$a\" is $b. |
|||
b=`expr match |
"$a" |
'\([0−9]*\)'` |
# Note escaped parentheses. |
echo "The digits at the beginning of \"$a\" are \"$b\"."
echo
exit 0
The : operator can substitute for match. For example, b=`expr $a : [0−9]*` is the exact equivalent of b=`expr match $a [0−9]*` in the above listing.
#!/bin/bash
echo
echo "String operations using \"expr $string :\" construct" echo "−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−"
echo
a=1234zipper43231
12.2. Complex Commands |
144 |