Beginning Python (2005)

Other Features of the Language

Generating Lists for Loops

Python has a special feature that enables you to create lists: the range function:

list = range (10, 20)

print list

This code produces an obvious-looking result:

>>> print list

[10, 11, 12, 13, 14, 15, 16, 17, 18, 19]

By itself, this doesn’t seem profound, but it is essential for situations when you need to use a for loop that will continue for a specific number of iterations, and that isn’t based on an existing list; and this number may not be determined at the time when the program was written, but it becomes known only when the program is already running.

If range is only given a single number, it will count from zero to that number. The number can be positive or negative:

for number in range(10):

print “Number is now %d” % number

This produces the obvious output, which is what you want:

Number is now 0

Number is now 1

Number is now 2

Number is now 3

Number is now 4

Number is now 5

Number is now 6

Number is now 7

Number is now 8

Number is now 9

In addition, if you only want, for example, every other number, or every third number, you can use an even more optional third parameter, called the step, that describes what the interval will be between each number that range creates:

for number in range(5, 55, 4):

print “Number from 5 to 55, by fours: %d” % number

This results in the selective list of numbers that you specified:

Number from 5 to 55, by fours: 5

Number from 5 to 55, by fours: 9

Number from 5 to 55, by fours: 13

Number from 5 to 55, by fours: 17

Number from 5 to 55, by fours: 21

Number from 5 to 55, by fours: 25

131

TEAM LinG

Chapter 9

Number from 5 to 55, by fours: 29

Number from 5 to 55, by fours: 33

Number from 5 to 55, by fours: 37

Number from 5 to 55, by fours: 41

Number from 5 to 55, by fours: 45

Number from 5 to 55, by fours: 49

Number from 5 to 55, by fours: 53

In some situations, a program could be handling huge numbers of elements — perhaps hundreds of thousands, or millions. In this case, range does one thing wrong: It will create an array with every element that you’ve asked for — for example, from zero to the number of all the possible systems on the Internet. When this many things need to be examined, each element uses a bit of computer memory, which can eventually take up all of the memory on a system. To avoid any problems with this sort of really large list, a special built-in class called xrange creates fewer elements in memory, so it is perfect for really large sets of numbers. It behaves the same as range does, except instead of returning a list it returns an xrange object.

The following code will produce the same output as the range function, so the output is omitted:

# xrange provides a special case useful for large sets. This is unnecessary. for r in xrange(0, 10):

print r

Try It Out

Examining an xrange Object

Interestingly, note that xrange returns an object that behaves like a list. Note that this object has no public interfaces — just private methods that look like a subset of what most lists and tuples have:

>>>xr = xrange(0,10)

>>>dir(xr)

[‘__class__’, ‘__delattr__’, ‘__doc__’, ‘__getattribute__’, ‘__getitem__’, ‘__hash__’, ‘__init__’, ‘__iter__’, ‘__len__’, ‘__new__’, ‘__reduce__’, ‘__reduce_ex__’, ‘__repr__’, ‘__reversed__’, ‘__setattr__’, ‘__str__’]

Tying to call it directly doesn’t result in a list; it results in a representation of how it was called:

>>> xr

xrange(10)

You can, however, still access it by using the same dereferencing operation (the square brackets) that you can with lists, sequences, and dictionaries.

>>>xr[0]

0

>>>xr[1]

1

How It Works

Xrange produces an object that doesn’t have any public methods. The only methods it has are built-in methods that enable it to act as a very simple sequence. Internally, when you use the square brackets to

132

TEAM LinG

Other Features of the Language

access a list, tuple, or a dictionary, you are telling python to invoke the __getitem__ method of that list, tuple, or dictionary. An xrange object has this private method, so it can act as a sequence and be dereferenced this way.

When you call an xrange object, it doesn’t produce a list — instead, it tells you how it was created so you know what the parameters were, in case you wanted to know about the numbers it is generating.

The point is that even though it behaves like a sequence, it is different; and that’s kind of cool.

Special String Substitution Using

Dictionaries

One syntax you haven’t been shown yet is a special syntax for using dictionaries to populate string substitutions. This can come up when you want a configurable way to print out strings — such as a formatted report or something similar.

Try It Out

String Formatting with Dictionaries

When you are doing this, you want to take individual named elements from a known set of elements, such as what you have in a dictionary, and print them out in the order that you have specified, which can be defined outside of the program itself:

person = {“name”: “James”, “camera”: “nikon”, “handedness”: “lefty”, “baseball_team”: “angels”, “instrument”: “guitar”}

print “%(name)s, %(camera)s, %(baseball_team)s” % person

The output of this code looks like this:

>>> print “%(name)s, %(camera)s, %(baseball_team)s” % person James, nikon, angels

How It Works

Note that the information in the parentheses is the name of the key whose value will be substituted from the dictionary into the string. However, to use this properly, you still need to specify the type of the data being inserted after the closing parenthesis so that the string substitution knows what to do. Here, all the types were strings, but you could use the i for int, j for imaginary, l for long, and all the other format specifiers you’ve learned. To see different formats being used with this new format, try the following example. Notice that person should appear on the same line as the print statement — it’s not on the next line; it’s just the end of a long line:

person[“height”] = 1.6 person[“weight”] = 80

print “%(name)s, %(camera)s, %(baseball_team)s, %(height)2.2f, %(weight)2.2f” % person

133

TEAM LinG

Chapter 9

This gives you the following terse output:

>>> print “%(name)s, %(camera)s, %(baseball_team)s, %(height)2.2f, %(weight)2.2f” % person

James, nikon, angels, 1.60, 80.00

These examples work with almost the same syntax that you learned in the first three chapters.

Python 2.4 has added another form of string substitution within the String module, with a new syntax for a substitution grammar. This form has been created to enable you to give users — for example, of a program you’ve written — a format that may make more sense to them at first glance:

import string

person = {“name”: “James”, “camera”: “nikon”, “handedness”: “lefty”, “baseball_team”: “angels”, “instrument”: “guitar”}

person[“height”] = 1.6 person[“weight”] = 80

t = string.Template(“$name is $height m high and $weight kilos”) print t.substitute(person)

This produces output that’s no better or worse than the first way, except that you can’t control the format information anymore:

print t.substitute(person) James is 1.6 m high and 80 kilos

Think about using this feature when you are asking users to describe what information they want from a set of data. This can be used as an easily supported way for someone else to specify the data they want without saddling you with the need to rewrite your program. You just need to ask them to specify the template, and you can supply the string they’ve given you to the string.Template class to create a template object that will perform the desired substitution.

Featured Modules

Starting in Chapter 7, you’ve seen modules used to add functionality to Python. In Chapter 8, you learned how interaction with the operating system and its files is achieved through modules that provide interfaces to how the system works with the os module.

In this section, you’ll see examples of some other common modules that will help you to start building your own programs.

Getopt — Getting Options from the Command Line

On Unix systems, the most common way to specify the behavior of a program when it runs is to add parameters to the command line of a program. Even when a program is not run from the command line but is instead run using fork and exec (more on this later in this chapter), a command line

is constructed when it is invoked. This makes it a universal way of controlling the behavior of your programs.

134

TEAM LinG

Other Features of the Language

You may have seen, for instance, that many programs can be run so that they provide you with some basic information about how they should be run. Python enables you to do this with -h:

$ python –h

usage: python2.4 [option] ... [-c cmd | -m mod | file | -] [arg] ...

Options and arguments (and corresponding environment variables): -c cmd : program passed in as string (terminates option list) -d : debug output from parser (also PYTHONDEBUG=x)

-E : ignore environment variables (such as PYTHONPATH) [ etc. ]

In the past, different conventions were available on different Unix platforms to specify these options, but this has largely resulted in two forms of options being used by most projects: the short form, such as the help-message producing option to Python, and a long form, such as --help for help.

To accept these sorts of options makes sense. Ideally, you’d like to offer a short and a long form of commands that are common, and allow each one to optionally take a specification. So if you wanted to write a program that had a configuration file that the user could specify, you may want one option like -c short for experienced users, but provide a longer option too, like --config-file. In either case, you’d want them to be the same function in your program to save you time, but you’d like to give users the freedom to use these options however they want to use them.

The getopt module provides two functions to make this standard convention easy to use: getopt.getopt and getopt.gnu_getopt. They are both basically the same. The basic getopt only works until the first non-option is encountered — nothing else is checked.

For getopt to be useful, you have to know what options you want to be useful. Normally, it’s considered the least you can do for your users to write programs that provide them with information about how to run the program, such as how Python prints information with the -h option.

In addition, it’s often very useful to have a configuration file. Using these ideas as a starting point, you could start your new programs so that -h and --help both produce a minimal message about how your program is used, and using -c or --config-file=file would enable you to specify a configuration file that is different from the default configuration:

import sys import getopt

#Remember, the first thing in the sys.argv list is the name of the command

#You don’t need that.

cmdline_params = sys.argv[1:]

opts, args = getopt.getopt(cmdline_params, ‘hc:’, [‘help’, ‘config=’]) print opts, args

for option, parameter in opts:

if option == ‘-h’ or option == ‘--help’:

print “This program can be run with either -h or --help for this message,” print “or with -c or --config=<file> to specify a different configuration

file”

print

if option in (‘-c’, ‘--config’): # this means the same as the above print “Using configuration file %s” % parameter

135

TEAM LinG

Chapter 9

When long options are used and require a parameter (like --config in the preceding example), the equal sign must connect the option and the value of the parameter. However, when short options are used, one or more space or tab characters can separate the option from its corresponding value. This distinction is to duplicate the behavior of the options on older Unix machines that persist to the modern day. They persist because so many people expect that behavior. What can you do?

The preceding code snippet, if run in a program with the parameters -c test -h --config= secondtest, produces the following output:

[(‘-c’, ‘test’), (‘-h’, ‘’), (‘--config’, ‘secondtest’)] [] Using configuration file test

This program can be run with either -h or --help for this message,

or with -c or --config=<file> to specify a different configuration file

Using configuration file secondtest

Note how the second instance of the configuration file is accepted silently; and when it is reached, the same code that sets the config file is revisited so that the second instance is used.

The second list, the args data, is an empty list because all of the options provided to the program on the command line were valid options, or valid parameters to options. If you inserted other strings in the middle of your options, the normal getopt would behave differently. If the parameters used were instead -c test useless_information_here -h --config=secondtest, the output would say a lot less, and the args array would have a lot more in it.

[(‘-c’, ‘test’)] [‘useless_information_here’, ‘-h’, ‘--config=secondtest’] Using configuration file test

The gnu_getopt lets you mix and match on the command line so that nonoptions can appear anywhere in the midst of the options, with more options parsed afterward instead of stopping there:

opts, args = getopt.gnu_getopt(cmdline_params, ‘hc:’, [‘help’, ‘config=’]) print opts, args

for option, parameter in opts:

if option == ‘-h’ or option == ‘--help’:

print “This program can be run with either -h or --help for this message,” print “or with -c or --config=<file> to specify a different configuration

file”

print

if option in (‘-c’, ‘--config’): # this means the same as the above print “Using configuration file %s” % parameter

The important point to note is that if you use something that doesn’t meet the criteria for an option (by beginning with a – or a +, or following an option that takes a parameter), the two behave differently. Using the options -c test useless_information_here -h --config=secondtest, the gnu_getopt function provides the following output, with the odd duck being the only part of the command line left in the args array:

[(‘-c’, ‘test’), (‘-h’, ‘’), (‘--config’, ‘secondtest’)] [‘useless_information_here’]

136

TEAM LinG

Other Features of the Language

Using configuration file test

This program can be run with either -h or --help for this message,

or with -c or --config=<file> to specify a different configuration file

Using configuration file secondtest

Using More Than One Process

In Unix and Unix-like operating systems, the main way of performing certain kinds of subtasks is to create a new process running a new program. On Unix systems, this is done using a system call that is available in Python by using os.fork. This actually tells the computer to copy everything about the currently running program into a newly created program that is separate, but almost entirely identical. The only difference is that the return value for os.fork is zero in the newly created process (the child), and is the process ID (PID) of the newly created process in the original process (the parent). This can be difficult to understand, and the only way to really get it is to use it a few times and to read some other material on fork and exec that’s available on-line. Or talk to your nearest Unix guru.

Based on the one critical difference, a parent and child can perform different functions. The parent can wait for an event while the child processes, or vice versa. The code to do this is simple, and common, but it works only on Unix and Unix-like systems:

import os

pid = os.fork()

if pid == 0: # This is the child

print “this is the child” else:

print “the child is pid %d” % pid

One of the most common things to do after an os.fork call is to call os.execl immediately afterward to run another program. os.execl is an instruction to replace the running program with a new program, so the calling program goes away, and a new program appears in its place (in case you didn’t already know this, Unix systems use the fork and exec method to run all programs):

import os

pid = os.fork()

# fork and exec together print “second test”

if pid == 0: # This is the child print “this is the child”

print “I’m going to exec another program now” os.execl(‘/bin/cat’, ‘cat’, ‘/etc/motd’)

else:

print “the child is pid %d” % pid os.wait()

The os.wait function instructs Python that you want the parent to not do anything until the child process returns. It is very useful to know how this works because it works well only under Unix and Unixlike platforms such as Linux. Windows also has a mechanism for starting up new processes.

To make the common task of starting a new program easier, Python offers a single family of functions that combines os.fork and os.exec on Unix-like systems, and enables you to do something similar

137

TEAM LinG

Chapter 9

on Windows platforms. When you want to just start up a new program, you can use the os.spawn family of functions. They are a family because they are named similarly, but each one has slightly different behaviors.

On Unix-like systems, the os.spawn family contains spawnl, spawnle, spawnlp, spawnlpe, spawnv, spawnve, spawnvp, and spawnvpe. On Windows systems, the spawn family only contains spawnl, spawnle, spawnv, and spawnve.

In each case, the letters after the word spawn mean something specific. The v means that a list (a vector is what the v actually stands for) will be passed in as the parameters. This allows a command to be run with very different commands from one instance to the next without needing to alter the program at all. The l variations just requires a simple list of parameters.

The e occurrences require that a dictionary containing names and values that will be used as the environment for the newly created program will be passed in instead of using the current environment.

The p occurrence uses the value of the PATH key in the environment dictionary to find the program. The p variants are available only on Unix-like platforms. The least of what this means is that on Windows your programs must have a completely qualified path to be usable by the os.spawn calls, or you have to search the path yourself:

import os, sys

if sys.platform == ‘win32’:

print “Running on a windows platform” command = “C:\\winnt\\system32\\cmd.exe” params = []

if sys.platform == ‘linux2’:

print “Running on a Linux system, identified by %s” % sys.platform command = ‘/bin/uname’

params = [‘uname’, ‘-a’]

print “Running %s” % command os.spawnv(os.P_WAIT, command, params)

Of course, this example will only work on a limited range of systems. You can use the contents of sys.platform on your own computer and for something besides linux2 in case you are on another Unix system such as Solaris, Mac OS X, AIX, or others.

When you do this, you can either wait for the process to return (that is, until it finishes and exits) or you can tell Python that you’d prefer to allow the program to run on its own, and that you will confirm that it completed successfully later. This is done with the os.P_ family of values. Depending on which one you set, you will be given a different behavior when an os.spawn function returns.

If you need only the most basic invocation of a new command, sometimes the easiest way to do this is to use the os.system function. If you are running a program and just want to wait for it to finish, you can use this function very simply:

# Now system

if sys.platform == ‘win32’:

print “Running on a windows platform” command = “cmd.exe”

138

TEAM LinG

Other Features of the Language

if sys.platform == ‘linux2’: print “Running Linux” command = “uname -a”

os.system(command)

This can be much simpler because it uses the facilities that the operating system provides, and that users expect normally, to search for the program you want to run, and it defaults to waiting for the child process to finish.

Threads — Doing Many Things in the Same Process

Creating a new process using fork or spawn can sometimes be too much effort and not provide enough benefit. Specifically, regarding the too much effort, when a program grows to be large, fork has to copy everything in the program to the new program and the system must have enough resources to handle that. Another downside for fork is that sometimes when you need your program to do many things at the same time, some things may need to wait while others need to proceed. When this happens, you want to have all of the different components communicating their needs to other parts of the program.

Using multiple processes, this becomes very difficult. These processes share many things because the child was originally created using the data in the parent. However, they are separate entities — completely separate. Because of this, it can be very tricky to make two processes work together cooperatively.

So, to make some complex situations where subprocesses are not appropriate workable, the concept of threads is available.

Many cooperative threads of program execution are able to exist at the same time in the same program. Each one has potentially different objects, with different state, but they can all communicate, while also being able to run semi-independently of one another.

This means that in many situations, using threads is much more convenient than using a separate process. Note that the following example uses subclassing, which is covered in Chapter 10. To see how this works, try running it with a fairly large parameter, say two million (2000000):

import math

from threading import Thread import time

class SquareRootCalculator:

“””This class spawns a separate thread to calculate a bunch of square roots, and checks in it once a second until it finishes.”””

def __init__(self, target):

“””Turn on the calculator thread and, while waiting for it to finish, periodically monitor its progress.”””

self.results = []

counter = self.CalculatorThread(self, target) print “Turning on the calculator thread...” counter.start()

139

TEAM LinG

Chapter 9

while len(self.results) < target:

print “%d square roots calculated so far.” % len(self.results) time.sleep(1)

print “Calculated %s square root(s); the last one is sqrt(%d)=%f” % \ (target, len(self.results), self.results[-1])

class CalculatorThread(Thread):

“””A separate thread which actually does the calculations.”””

def __init__(self, controller, target):

“””Set up this thread, including making it a daemon thread so that the script can end without waiting for this thread to finish.”””

Thread.__init__(self) self.controller = controller self.target = target self.setDaemon(True)

def run(self):

“””Calculate square roots for all numbers between 1 and the target, inclusive.”””

for i in range(1, self.target+1): self.controller.results.append(math.sqrt(i))

if __name__ == ‘__main__’: import sys

limit = None

if len(sys.argv) > 1: limit = sys.argv[1] try:

limit = int(limit) except ValueError:

print “Usage: %s [number of square roots to calculate]” \ % sys.argv[0]

SquareRootCalculator(limit)

For many situations, such as network servers (see Chapter 16) or graphical user interfaces (see Chapter 13), threads make much more sense because they require less work from you as the programmer, and less resources from the system.

Note how separate threads can access each other’s names and data easily. This makes it very easy to keep track of what different threads are doing, an important convenience.

Subprocesses are really available only on Unix and Unix-like platforms on which they are well supported. A self-contained program using threads can be much more easily ported across different platforms.

Storing Passwords

You will frequently need to store passwords in an encrypted form. Most operating systems have their own way of doing this. On Unix, the standard encryption is a protocol called DES, though newer systems also allow a type of hashing called md5, and on some sha-1 is available. Windows systems usually keep passwords in an entirely different format in the registry.

140

TEAM LinG