Different kinds of tests	75

Listing 4.2 Timed test example

package com.clarkware.junitperf;

import junit.framework.Test; import junit.framework.TestSuite;

public class ExampleTimedTest {

public static final long toleranceInMillis = 100;

public static Test suite() {

long maxElapsedTimeInMillis = 1000 + toleranceInMillis;

Test testCase =

new ExampleTestCase("testOneSecondResponse"); Test timedTest =

new TimedTest(testCase, maxElapsedTimeInMillis);

TestSuite suite = new TestSuite(); suite.addTest(timedTest);

return suite;

}

public static void main(String args[]) {

junit.textui.TestRunner.run(suite());

}

}

Acceptance software testing

It’s important that an application perform well; but, in the end, the only question that matters is Does the application meet the customer’s needs? Acceptance tests are the final sphere of tests. These tests are usually conducted directly by the customer or someone acting as the customer’s proxy. Acceptance tests ensure that the application has met whatever goals the customer or stakeholder defined.

Acceptance tests are a superset of all other tests. Usually they start as functional and performance tests, but they may include subjective criteria like “ease of use” and “look and feel.” Sometimes, the acceptance suite may include a subset of the tests run by the developers, the difference being that this time the tests are run by the customer or a QA team.

For more about using acceptance tests with an agile software methodology, visit the Wiki site regarding Ward Cunningham’s fit framework (http://fit.c2.com/).

4.2.2The three flavors of unit tests

The focus of this book is automatic unit tests used by programmers. Unit testing, as we use the term, concentrates on testing the code from the inside (white box

76CHAPTER 4

Examining software tests

testing; see section 4.3.1). This activity is inextricably linked with coding and happens at the same time. Unit tests can ensure that your application is under test from the very beginning.

Of course, your application should undergo other forms of software testing, starting with unit tests and ending with acceptance tests. The previous section outlined the other types of software tests that should be applied to your application.

Most applications are divided into subsystems. As a developer, you want to ensure that each of your subsystems works correctly. As you write code, your first tests will probably be logic unit tests. As you write more tests and more code, you will begin to add integration and functional unit tests. At any one time, you will probably be working on a logic unit test, an integration unit test, or a functional unit test. Table 4.3 summarizes the different types of unit tests.

Table 4.3 Three flavors of unit tests: logic, integration, and functional

Test type	Description
Logic unit tests	Unit tests that focus on exercising the code logic. These tests are usually meant to
	exercise only a single method and no other. You can control the boundaries of a
	given method using mock objects or stubs (see chapters 6 and 7).
Integration unit	Unit tests that focus on testing the interaction between components in their real
tests	environment (or part of the real environment). For example, code that accesses a
	database has tests that effectively call the database, thus proving that the code-
	database interaction works (see chapter 11).
Functional unit	Unit tests that extend the boundaries of integration unit testing to confirm a
tests	stimulus-response. For example, imagine a web page that is protected and that
	you can access only after being logged in. If you are not logged in, accessing the
	page results in a redirect to the login page. A functional unit test verifies that
	behavior by sending an HTTP request to the page and verifying that the result is
	a 302 response code. It does not, however, verify that the full workflow leads to
	the login page. Workflow is the domain of pure, software functional testing (see
	section 4.2.1).

Figure 4.4 illustrates how these three flavors of unit tests interact. The sliders define the boundaries between the types of unit tests. The tests can be defined by the locations of the sliders. All three types of tests are needed to ensure your code works. If you use them, you can sleep well at night and come in the next day, well rested and eager to create more great code!

Strictly speaking, the functional unit tests are not pure unit tests, but neither are they pure functional tests. They are more dependent on an external environment than pure unit tests are, but they do not test a complete workflow, as

Determining how good tests are	77

Functional unit testing

Integration unit testing

System

Class

Logic unit testing

Figure 4.4 Unit testing within the application life cycle

expected by pure functional tests. We put functional unit tests in our scope because they are often useful as part of the battery of tests run in development.

An example is the StrutsTestCase (http://strutstestcase.sourceforge.net/) framework, which provides functional unit testing of the runtime Struts configuration. These tests tell a developer that the controller is invoking the appropriate software action and forwarding to the expected presentation page, but they do not confirm that the page is actually present and renders correctly.

4.3 Determining how good tests are

When you test a rivet, you can apply simple, objective standards. You can look at its dimensions, its weight, and whether it can withstand a certain amount of pressure. You can say that your tests cover just these aspects, and let the builder decide if the test coverage is sufficient.

But how do you express what aspects an application’s unit tests cover? “Everything that could possibly fail” is a fine standard, but it’s rather subjective. What kind of metrics can we apply?

78CHAPTER 4

Examining software tests

4.3.1Measuring test coverage

One metric of test coverage would be to count how many methods are being called by your tests. This doesn’t tell you whether the test is doing the right thing, but it does tell you whether you have a test in place.

Without unit tests, you can only write tests against the application’s public API methods. Because you don’t need to see inside the application to create the tests, these are called black box tests. Figure 4.5 diagrams what an application’s test coverage might look like using only black box tests.

Black box tests

Method

calls

Figure 4.5 Black box test coverage

Unit tests can be written with an intimate knowledge of how a method is implemented. If a conditional branch exists within the method, you can write two unit tests: one for each branch. Because you need to see into the method to create such a test, this is called white box testing. Figure 4.6 shows what an application’s test coverage might look like using white box unit tests.

Unit tests

Unit tests

Unit tests

Figure 4.6

Unit test coverage

With white box unit tests, it’s easier to achieve a higher test coverage metric, mainly because you have access to more methods and because you can control both the inputs to each method and the behavior of secondary objects called (using stubs or mock objects, as you’ll see in later chapters). White box unit tests can be written against both package-protected and public methods.