Beginning Python (2005)

12

Testing

Like visits to the dentist, thorough testing of any program is something that you should be doing if you want to avoid the pain of having to trace a problem that you thought you’d taken care of. This lesson is one that normally takes a programmer many years to learn, and to be honest, you’re still going to be working on it for many years. However, the one thing that is of the utmost importance is that testing must be organized; and to be the most effective, you must start writing your programs knowing that it will be tested as you go along, and plan around having the time to write and confirm your test cases.

Fortunately, Python offers an excellent facility for organizing your testing called PyUnit. It is a Python port of the Java JUnit package, so if you’ve worked with JUnit, you’re already on firm ground when testing in Python — but if not, don’t worry. This chapter will show you the following:

The concept and use of assertions

The basic concepts of unit testing and test suites

A few simple example tests to show you how to organize a test suite

Thorough testing of the search utility from Chapter 11

The beauty of PyUnit is that you can set up testing early in the software development life cycle, and you can run it as often as needed while you’re working. By doing this, you can catch errors early on, before they’re painful to rework — let alone before anybody else sees them. You can also set up test cases before you write code, so that as you write, you can be sure that your results match what you expect! Define your test cases before you even start coding, and you’ll never find yourself fixing a bug only to discover that your changes have spiraled out of control and cost you days of work.

Asser tions

An assertion, in Python, is in practice similar to an assertion in day-to-day language. When you speak and you make an assertion, you have said something that isn’t necessarily proven but that

TEAM LinG

Chapter 12

you believe to be true. Of course, if you are trying to make a point, and the assertion you made is incorrect, then your entire argument falls apart.

In Python, an assertion is a similar concept. Assertions are statements that can be made within the code while you are developing it that you can use to test the validity of your code, but if the statement doesn’t turn out to be true, an AssertionError is raised, and the program will be stopped if the error isn’t caught (in general, they shouldn’t be caught, as AssertionErrors should be taken as a warning that you didn’t think something through correctly!)

Assertions enable you to think of your code in a series of testable cases. That way, you can make sure that while you develop, you can make tests along the lines of “this value is not None” or “this object is a String” or “this number is greater than zero.” All of these statements are useful while developing to catch errors in terms of how you think about the program.

Try It Out

Using Assert

Creating a set of simple cases, you can see how the assert language feature works:

# Demonstrate the use of assert() large = 1000

string = “This is a string” float = 1.0

broken_int = “This should have been an int”

assert large > 500

assert type(string) == type(“”) assert type(float) != type(1) assert type(broken_int) == type(4)

Try running the preceding with python -i.

How It Works

The output from this simple test case looks like this:

Traceback (most recent call last):

File “D:\Documents\ch12\try_assert.py”, line 13, in ?

assert type(broken_int) == type(4) AssertionError

You can see from this stack trace that this simply raises the error. assert is implemented very simply. If a special internal variable called __debug__ is True, assertions are checked; and if any assertion doesn’t succeed, an AssertionError is raised. Because assert is actually a combination of an if statement that, when there’s a problem, will raise an exception, you are allowed to specify a custom message, just as you would with raise, by adding a comma and the message that you’d want to see when you see the error in a try ... : and except ...: block. You should experiment by replacing the last assertion with this code and running it:

try:

assert type(broken_int) == type(4), “broken_int is broken”

except AssertionError, message:

print “Handle the error here. The message is: %s” % message

192

TEAM LinG

Testing

The variable __debug__, which activates assert, is special; it’s immutable after Python has started up, so in order to turn it off you need to specify the -O (a dash, followed by the capital letter O) parameter to Python. -O tells Python to optimize code, which among other things for Python means that it removes assert tests, because it knows that they’ll cause the program to slow down (not a lot, but optimization like this is concerned with getting every little bit of performance). -O is intended to be used when a program is deployed, so it removes assertions that are considered to be development-time features.

As you can see, assertions are useful. If you even think that you may have made a mistake and want to catch it later in your development cycle, you can put in an assertion to catch yourself, and move on and get other work done until that code is tested. When your code is tested, it can tell you what’s going wrong if an assertion fails instead of leaving you to wonder what happened. Moreover, when you deploy and use the -O flag, your assertion won’t slow down the program.

Assert lacks a couple of things by itself. First, assert doesn’t provide you with a structure in which to run your tests. You have to create a structure, and that means that until you learn what you want from tests, you’re liable to make tests that do more to get in your way than confirm that your code is correct.

Second, assertions just stop the program and they provide only an exception. It would be more useful to have a system that would give you summaries, so you can name your tests, add tests, remove tests, and compile many tests into a package that let you summarize whether your program tests out or not. These ideas and more make up the concepts of unit tests and test suites.

Test Cases and Test Suites

Unit testing revolves around the test case, which is the smallest building block of testable code for any circumstances that you’re testing. When you’re using PyUnit, a test case is a simple object with at least one test method that runs code; and when it’s done, it then compares the results of the test against various assertions that you’ve made about the results.

PyUnit is the name of the package as named by its authors, but the module you import is called the more generic-sounding name unittest.

Each test case is subclassed from the TestCase class, which is a good, memorable name for it. The simplest test cases you can write just override the runTest method of TestCase and enable you to define a basic test, but you can also define several different test methods within a single test case class, which can enable you to define things that are common to a number of tests, such as setup and cleanup procedures.

A series of test cases run together for a particular project is called a test suite. You can find some simple tools for organizing test suites, but they all share the concept of running a bunch of test cases together and recording what passed, what failed, and how, so you can know where you stand.

Because the simplest possible test suite consists of exactly one test case, and you’ve already had the simplest possible test case described to you, let’s write a quick testing example so you can see how all this fits together. In addition, just so you really don’t have anything to distract you, let’s test arithmetic, which has no external requirements on the system, the file system, or, really, anything.

193

TEAM LinG

Chapter 12

Try It Out

Testing Addition

1.Use your favorite editor to create a file named test1.py in a directory named ch12. Using your programming editor, edit your file to have the following code:

import unittest

class ArithTest (unittest.TestCase): def runTest (self):

“”” Test addition and succeed. “”” self.failUnless (1+1==2, ‘one plus one fails!’)

self.failIf (1+1 != 2, ‘one plus one fails again!’) self.failUnlessEqual (1+1, 2, ‘more trouble with one plus one!’)

def suite():

suite = unittest.TestSuite() suite.addTest (ArithTest()) return suite

if __name__ == ‘__main__’:

runner = unittest.TextTestRunner() test_suite = suite()

runner.run (test_suite)

2.Now run the code using python:

.

----------------------------------------------------------------------

Ran 1 tests in 0.000s

OK

How It Works

In step 1, after you’ve imported unittest (the module that contains the PyUnit framework), you define the class ArithTest, which is a subclass of the class from unittest, TestCase. ArithTest has only defined the runTest method, which performs the actual testing. Note how the runTest method has its docstring defined. It is at least as important to document your tests as it is to document your code. Lastly, a series of three assertions takes place in runTest.

TestCase classes beginning with fail, such as failUnless, failIf, and failUnlessEqual, come in additional varieties to simplify setting up the conditions for your tests. When you’re programming, you’ll likely find yourself resistant to writing tests (they can be very distracting; sometimes they are boring; and they are rarely something other people notice, which makes it harder to motivate yourself to write them). PyUnit tries to make things as easy as possible for you.

After the unit test is defined in ArithTest, you may like to define the suite itself in a callable function, as recommended by the PyUnit developer, Steve Purcell, in the modules documentation. This enables you to simply define what you’re doing (testing) and where (in the function you name). Therefore, after the definition of ArithTest, you have crated the suite function, which simply instantiates a vanilla, unmodified test suite. It adds your single unit test to it and returns it. Keep in mind that the suite function only invokes the TestCase class in order to make an object that can be returned. The actual test is performed by the returned TestCase object.

194

TEAM LinG

Testing

As you learned in Chapter 6, only when this is being run as the main program will Python invoke the TextTestRunner class to create the runner object. The runner object has a method called run that expects to have an object of the unittests.TestSuite class. The suite function creates one such object, so test_suite is assigned a reference to the TestSuite object. When that’s finished, the runner.run method is called, which uses the suite in test_suite to test the unit tests defined in test_suite.

The actual output in this case is dull, but in that good way you’ll learn to appreciate because it means everything has succeeded. The single period tells you that it has successfully run one unit test. If, instead of the period, you see an F, it means that a test has failed. In either case, PyUnit finishes off a run with a report. Note that arithmetic is run very, very fast.

Now, let’s see what failure looks like.

Try It Out

Testing Faulty Addition

1.Use your favorite text editor to add a second set of tests to test1.py. These will be based on the first example. Add the following to your file:

class ArithTestFail (unittest.TestCase): def runTest (self):

“”” Test addition and fail. “”” self.failUnless (1+1==2, ‘one plus one fails!’)

self.failIf (1+1 != 2, ‘one plus one fails again!’) self.failUnlessEqual (1+1, 2, ‘more trouble with one plus one!’) self.failIfEqual (1+1, 2, ‘expected failure here’)

self.failIfEqual (1+1, 2, ‘second failure’)

def suite_2():

suite = unittest.TestSuite() suite.addTest (ArithTest()) suite.addTest (ArithTestFail()) return suite

You also need to change the if statement that sets off the tests, and you need to make sure that it appears at the end of your file so that it can see both classes:

if __name__ == ‘__main__’:

runner = unittest.TextTestRunner()

test_suite = suite_2() runner.run (test_suite)

2.Now run the newly modified file (after you’ve saved it). You’ll get a very different result with the second set of tests. In fact, it’ll be very different from the prior test:

.F

======================================================================

FAIL: Test addition and fail.

----------------------------------------------------------------------

Traceback (most recent call last):

File “D:\Documents\ch12\test1.py”, line 27, in runTest self.failIfEqual (1+1, 2, ‘expected failure here’)

AssertionError: expected failure here

195

TEAM LinG

Chapter 12

----------------------------------------------------------------------

Ran 2 tests in 0.000s

FAILED (failures=1)

>>>

How It Works

Here, you’ve kept your successful test from the first example and added a second test that you know will fail. The result is that you now have a period from the first test, followed by an ‘F’ for ‘Failed’ from the second test, all in the first line of output from the test run.

After the tests are run, the results report is printed out so you can examine exactly what happened. The successful test still produces no output at all in the report, which makes sense: Imagine you have a hundred tests but only two fail — you would have to slog through a lot more output to find the failures than you do this way. It may seem like looking on the negative side of things, but you’ll get used to it.

Because there was a failed test, the stack trace from the failed test is displayed. In addition, a couple of different messages result from the runTest method. The first thing you should look at is the FAIL message. It actually uses the docstring from your runTest method and prints it at the top, so you can reference the test that failed. Therefore, the first lesson to take away from this is that you should document your tests in the docstring! Second, you’ll notice that the message you specified in the runTest for the specific test that failed is displayed along with the exception that PyUnit generated.

The report wraps up by listing the number of test cases actually run and a count of the failed test cases.

Test Fixtures

Well, this is all well and good, but real-world tests usually involve some work to set up your tests before they’re run (creating files, creating an appropriate directory structure, generally making sure everything is in shape, and other things that may need to be done to ensure that the right things are being tested). In addition, cleanup also often needs to be done at the end of your tests.

In PyUnit, the environment in which a test case runs is called the test fixture, and the base TestCase class defines two methods: setUp, which is called before a test is run, and tearDown, which is called after the test case has completed. These are present to deal with anything involved in creating or cleaning up the test fixture.

You should know that if setUp fails, tearDown isn’t called. However, tearDown is called even if the test case itself fails.

Remember that when you set up tests, the initial state of each test shouldn’t rely on a prior test having succeeded or failed. Each test case should create a pristine test fixture for itself. If you don’t ensure this, you’re going to get inconsistent test results that will only make your life more difficult.

To save time when you run similar tests repeatedly on an identically configured test fixture, subclass the TestCase class to define the setup and cleanup methods. This will give you a single class that you can

196

TEAM LinG

Testing

use as a starting point. Once you’ve done that, subclass your class to define each test case. You can alternatively define several test case methods within your unit case class, and then instantiate test case objects for each method. Both of these are demonstrated in the next example.

Try It Out

Working with Test Fixtures

1.Use your favorite text editor to add a new file test2.py. Make it look like the following example. Note that this example builds on the previous examples.

import unittest

class ArithTestSuper (unittest.TestCase): def setUp (self):

print “Setting up ArithTest cases”

def tearDown (self):

print “Cleaning up ArithTest cases”

class ArithTest (ArithTestSuper): def runTest (self):

“”” Test addition and succeed. “”” print “Running ArithTest”

self.failUnless (1+1==2, ‘one plus one fails!’) self.failIf (1+1 != 2, ‘one plus one fails again!’)

self.failUnlessEqual (1+1, 2, ‘more trouble with one plus one!’)

class ArithTestFail (ArithTestSuper): def runTest (self):

“”” Test addition and fail. “”” print “Running ArithTestFail”

self.failUnless (1+1==2, ‘one plus one fails!’) self.failIf (1+1 != 2, ‘one plus one fails again!’)

self.failUnlessEqual (1+1, 2, ‘more trouble with one plus one!’) self.failIfEqual (1+1, 2, ‘expected failure here’) self.failIfEqual (1+1, 2, ‘second failure’)

class ArithTest2 (unittest.TestCase): def setUp (self):

print “Setting up ArithTest2 cases” def tearDown (self):

print “Cleaning up ArithTest2 cases”

def runArithTest (self):

“”” Test addition and succeed, in one class. “”” print “Running ArithTest in ArithTest2” self.failUnless (1+1==2, ‘one plus one fails!’) self.failIf (1+1 != 2, ‘one plus one fails again!’)

self.failUnlessEqual (1+1, 2, ‘more trouble with one plus one!’)

def runArithTestFail (self):

“”” Test addition and fail, in one class. “”” print “Running ArithTestFail in ArithTest2” self.failUnless (1+1==2, ‘one plus one fails!’)

197

TEAM LinG

Chapter 12

self.failIf (1+1 != 2, ‘one plus one fails again!’) self.failUnlessEqual (1+1, 2, ‘more trouble with one plus one!’) self.failIfEqual (1+1, 2, ‘expected failure here’) self.failIfEqual (1+1, 2, ‘second failure’)

def suite():

suite = unittest.TestSuite()

#First style: suite.addTest (ArithTest())

suite.addTest (ArithTestFail())

#Second style:

suite.addTest (ArithTest2(“runArithTest”)) suite.addTest (ArithTest2(“runArithTestFail”))

return suite

if __name__ == ‘__main__’:

runner = unittest.TextTestRunner() test_suite = suite()

runner.run (test_suite)

2.Run the code:

Setting up ArithTest cases Running ArithTest

Cleaning up ArithTest cases

.Setting up ArithTest cases Running ArithTestFail FCleaning up ArithTest cases Setting up ArithTest2 cases Running ArithTest in ArithTest2 Cleaning up ArithTest2 cases

.Setting up ArithTest2 cases Running ArithTestFail in ArithTest2 FCleaning up ArithTest2 cases

======================================================================

FAIL: Test addition and fail.

----------------------------------------------------------------------

Traceback (most recent call last):

File “D:\Documents\ch12\test2.py”, line 25, in runTest self.failIfEqual (1+1, 2, ‘expected failure here’)

AssertionError: expected failure here

======================================================================

FAIL: Test addition and fail, in one class.

----------------------------------------------------------------------

Traceback (most recent call last):

File “D:\Documents\ch12\test2.py”, line 48, in runArithTestFail self.failIfEqual (1+1, 2, ‘expected failure here’)

AssertionError: expected failure here

198

TEAM LinG

Testing

----------------------------------------------------------------------

Ran 4 tests in 0.000s

FAILED (failures=2)

>>>

How It Works

Take a look at this code before moving along. The first thing to note about this is that you’re doing the same tests as before. One test is made to succeed and the other one is made to fail, but you’re doing two sets, each of which implements multiple unit test cases with a test fixture, but in two different styles.

Which style you use is completely up to you; it really depends on what you consider readable and maintainable.

The first set of classes in the code (ArithTestSuper, ArithTest, and ArithTestFail) are essentially the same tests as shown in the second set of examples in test1.py, but this time a class has been created called ArithTestSuper. ArithTestSuper implements a setUp and tearDown method. They don’t do much but they do demonstrate where you’d put in your own conditions. Each of the unit test classes are subclassed from your new ArithTestSuper class, so now they will perform the same setup of the test fixture. If you needed to make a change to the test fixture, you can now modify it in ArithTestSuper’s classes, and have it take effect in all of its subclasses.

The actual test cases, ArithTest and ArithTestFail, are the same as in the previous example, except that you’ve added print calls to them as well.

The final test case class, ArithTest2, does exactly the same thing as the prior three classes that you’ve already defined. The only difference is that it combines the test fixture methods with the test case methods, and it doesn’t override runTest. Instead ArithTest2 defines two test case methods: runArithTest and runArithTestFail. These are then invoked explicitly when you created test case instances during the test run, as you can see from the changed definition of suite.

Once this is actually run, you can see one change immediately: Because our setup, test, and cleanup functions all write to stdout, you can see the order in which everything is called. Note that the cleanup functions are indeed called even after a failed test. Finally, note that the tracebacks for the failed tests have been gathered up and displayed together at the end of the report.

Putting It All Together with

Extreme Programming

A good way to see how all of this fits together is to use a test suite during the development of an extended coding project. This strategy underlies the XP (Extreme Programming) methodology, which is a popular trend in programming: First, you plan the code; then you write the test cases as a framework; and only then do you write the actual code. Whenever you finish a coding task, you rerun the test suite to see how closely you approach the design goals as embodied in the test suite. (Of course, you are also debugging the test suite at the same time, and that’s fine!) This technique is a great way to find your programming errors early in the process, so that bugs in low-level code can be fixed and the code made stable before you even start on higher-level work, and it’s extremely easy to set up in Python using PyUnit, as you will see in the next example.

199

TEAM LinG

Chapter 12

This example includes a realistic use of text fixtures as well, creating a test directory with a few files in it and then cleaning up the test directory after the test case is finished. It also demonstrates the convention of naming all test case methods with test followed by the name, such as testMyFunction, to enable the unittest.main procedure to recognize and run them automatically.

Implementing a Search Utility in Python

The first step in this programming methodology, as with any, is to define your objectives — in this case, a general-purpose, reusable search function that you can use in your own work. Obviously, it would be a waste of time to anticipate all possible text-processing functionality in a single search utility program, but certain search tasks tend to recur a lot. Therefore, if you wanted to implement a general-purpose search utility, how would you go about it? The Unix find command is a good place to look for useful functionality — it enables you not only to iterate through the directory tree and perform actions on each file found but also to specify certain directories to skip, to specify rather complex logic combinations on the command line, and a number of other things, such as searching by file modification date and size.

On the other hand, the find command doesn’t include any searching on the content of files (the standard way to do this under Unix is to call grep from within find) and it has a lot of features involving the invocation of post-processing programs that we don’t really need for a general-purpose Python search utility.

What you might need when searching for files in Python could include the following:

Return values you can use easily in Python: A tuple including the full path, the filename, the extension, and the size of the file is a good start.

Specification of a regular expression for the filename to search for and a regular expression for the content (if no content search is specified, then the files shouldn’t be opened, to save overhead).

Optional specifications of additional search terms: The size of the file, its age, last modification, and so on are all useful.

A truly general search utility might include a function to be called with the parameters of the file, so that more advanced logic can be specified. The Unix find command enables very general logic combinations on the command line, but frankly, let’s face it — complex logic on the command line is hard to understand. This is the kind of thing that really works better in a real programming language like Python, so you could include an optional logic function for narrowing searches as well.

In general, it’s a good idea to approach this kind of task by focusing first on the core functionality, adding more capability after the initial code is already in good shape. That’s how the following example is structured — first you start with a basic search framework that encapsulates the functionality you covered in the examples for the os and re modules, and then you add more functionality once that first part is complete. This kind of incremental approach to software development can help keep you from getting bogged down in details before you have anything at all to work with, and the functionality of something like this general-purpose utility is complicated enough that it would be easy to lose the thread.

Because this is an illustration of the XP methodology as well, you’ll follow that methodology and first write the code to call the find utility, build that code into a test suite, and only then will you write the find utility. Here, of course, you’re cheating a little. Ordinarily, you would be changing the test suite as you go, but in this case, the test suite is already guaranteed to work with the final version of the tested code. Nonetheless, you can use this example for yourself.

200

TEAM LinG