202 |
CHAPTER 7 |
The pillars of good tests |
expected value. This gives us a declarative way of creating a single test with different inputs.
The best thing about this is that, if one of the [RowTest] attributes fails, the other attributes are still executed by the test runner, so we see the full picture of pass/fail states in all tests.
Wrapping with try-catch
Some people think it’s a good idea to use a try-catch block for each assert to catch and write its exception to the console, and then continue to the next statement, bypassing the problematic nature of exceptions in tests. I think using parameterized tests is a far better way of achieving the same thing. Use parameterized tests instead of try-catch around multiple asserts.
Now that we know how to avoid multiple asserts acting as multiple tests, let’s look at multiple asserts being used to test multiple aspects of a single object.
7.2.6Avoiding testing multiple aspects of the same object
Let’s look at another example of a test with multiple asserts, but this time it’s not trying to act as multiple tests in one test, it’s trying to check multiple aspects of the same state. If even one aspect fails, we need to know about it. Listing 7.17 shows such a test.
Listing 7.17 Testing multiple aspects of the same object in one test
[Test] public void
Analyze_SimpleStringLine_UsesDefaulTabDelimiterToParseFields()
{
LogAnalyzer log = new LogAnalyzer(); AnalyzedOutput output =
log.Analyze("10:05\tOpen\tRoy");
Assert.AreEqual(1,output.LineCount);
Assert.AreEqual("10:05",output.GetLine(1)[0]);
Assert.AreEqual("Open",output.GetLine(1)[1]);
Assert.AreEqual("Roy",output.GetLine(1)[2]);
}
Writing maintainable tests |
203 |
This example is testing that the parse output from the LogAnalyzer worked by testing each field in the result object separately. They should all work, or the test should fail.
Making tests more maintainable
Listing 7.18 shows a way to refactor the test from listing 7.17 so that it’s easier to read and maintain.
Listing 7.18 Comparing objects instead of using multiple asserts
[Test] public void
Analyze_SimpleStringLine_UsesDefaulTabDelimiterToParseFields2()
{
LogAnalyzer log = new LogAnalyzer(); AnalyzedOutput expected = new AnalyzedOutput(); expected.AddLine("10:05", "Open", "Roy");
AnalyzedOutput output = log.Analyze("10:05\tOpen\tRoy");
Assert.AreEqual(expected,output);
}
Sets up
an expected object
Compares expected and actual objects
Instead of adding multiple asserts, we can create a full object to compare against, set all the properties that should be on that object, and compare the result and the expected object in one assert. The advantage of this approach is that it’s much easier to understand what we’re testing and to recognize that this is one logical block that should be passing, not many separate tests.
Note that, for this kind of testing, the objects being compared must override the Equals() method, or the comparison between the objects won’t work. Some people find this an unacceptable compromise. I use it from time to time, but am happy to go either way. Use your own discretion.
Overriding ToString()
Another approach you might try is to override the ToString() method of compared objects so that, if tests fail, you’ll get more meaningful
204 |
CHAPTER 7 |
The pillars of good tests |
error messages. For example, here’s the output of the test in listing 7.18 when it fails.
TestCase 'AOUT.CH7.LogAn.Tests.MultipleAsserts
.Analyze_SimpleStringLine_UsesDefaulTabDelimiterToParseFields2' failed:
Expected: <AOUT.CH789.LogAn.AnalyzedOutput> But was: <AOUT.CH789.LogAn.AnalyzedOutput>
C:\GlobalShare\InSync\Book\Code\ARtOfUniTesting
\LogAn.Tests\MultipleAsserts.cs(41,0): at AOUT.CH7.LogAn.Tests.MultipleAsserts
.Analyze_SimpleStringLine_UsesDefaulTabDelimiterToParseFields2()
Not very helpful, is it?
By implementing ToString() in both the AnalyzedOutput class and the LineInfo class (which are part of the object model being compared), we can get more readable output from the tests. Listing 7.19 shows the two implementations of the ToString() methods in the classes under test, followed by the resulting test output.
Listing 7.19 Implementing ToString() in compared classes for cleaner output
///Overriding ToString inside The AnalyzedOutput Object////////////// public override string ToString()
{
StringBuilder sb = new StringBuilder(); foreach (LineInfo line in lines)
{
sb.Append(line.ToString());
}
return sb.ToString();
}
///Overriding ToString inside each LineInfo Object////////////// public override string ToString()
{
StringBuilder sb = new StringBuilder();
for (int i = 0; i < this.fields.Length; i++)
{
sb.Append(this[i]);
sb.Append(",");
}
Writing maintainable tests |
205 |
return sb.ToString();
}
///TEST OUTPUT//////////////
------ Test started: Assembly: er.dll ------
TestCase 'AOUT.CH7.LogAn.Tests.MultipleAsserts
.Analyze_SimpleStringLine_UsesDefaulTabDelimiterToParseFields2' failed:
Expected: <10:05,Open,Roy,> But was: <>
C:\GlobalShare\InSync\Book\Code\ARtOfUniTesting
\LogAn.Tests\MultipleAsserts.cs(41,0): at AOUT.CH7.LogAn.Tests.MultipleAsserts
.Analyze_SimpleStringLine_UsesDefaulTabDelimiterToParseFields2()
Now the test output is much clearer, and we can understand that we got very different objects. Clearer output makes it easier to understand why the test fails and makes for easier maintenance.
Another way tests can become hard to maintain is when we make them too fragile by overspecification.
7.2.7Avoiding overspecification in tests
An overspecified test is one that contains assumptions about how a specific unit under test should implement its behavior, instead of only checking that the end behavior is correct.
Here are some ways unit tests are often overspecified:
A test specifies purely internal behavior for an object under test.
A test uses mocks when using stubs would be enough.
A test assumes specific order or exact string matches when it isn’t required.
TIP This topic is also discussed in xUnit Test Patterns by Gerard Meszaros.
Let’s look at some examples of overspecified tests.
Specifying purely internal behavior
Listing 7.20 shows a test against LogAnalyzer’s Initialize() method that tests internal state, and no outside functionality.
206CHAPTER 7 The pillars of good tests
Listing 7.20 An overspecified test that tests a purely internal behavior
[Test]
public void Initialize_WhenCalled_SetsDefaultDelimiterIsTabDelimiter()
{
LogAnalyzer log = new LogAnalyzer();
Assert.AreEqual(null,log.GetInternalDefaultDelimiter());
log.Initialize();
Assert.AreEqual('\t', log.GetInternalDefaultDelimiter());
}
This test is overspecified because it only tests the internal state of the LogAnalyzer object. Because this state is internal, it could change later on.
Unit tests should be testing the public contract and public functionality of an object. In this example, the tested code isn’t part of any public contract or interface.
Using mocks instead of stubs
Using mocks instead of stubs is a common mistake. Let’s look at an example.
Listing 7.21 shows a test that uses mocks to assert the interaction between LogAnalyzer and a provider it uses to read a text file. The test wrongly checks that LogAnalyzer calls the provider correctly to read the file’s text (an implementation detail that could change later and break our test). Instead, the test could use a stub to return the fake results from the text file, and assert against the public output of the LogAnalyzer’s method, which makes for a more robust, less brittle test.
Listing 7.21 shows the method we want to test, followed by an overspecified test for that code.
Listing 7.21 An overspecified test that uses mocks when stubs would do fine
public AnalyzeResults AnalyzeFile(string fileName)
{
int lineCount = logReader.GetLineCount(); string text = "";
Writing maintainable tests |
207 |
for (int i = 0; i < lineCount; i++)
{
text += logReader.GetText(fileName, i, i);
}
return new AnalyzeResults(text);
}
//////////////////////////the test///////////////// [Test]
public void AnalyzeFile_FileWith3Lines_CallsLogProvider3Times()
{
MockRepository mocks = new MockRepository(); ILogProvider mockLog = mocks.CreateMock<ILogProvider>(); LogAnalyzer log = new LogAnalyzer(mockLog); using(mocks.Record())
{
mockLog.GetLineCount();
LastCall.Return(3);
mockLog.GetText("someFile.txt", 1, 1); LastCall.Return("a");
mockLog.GetText("someFile.txt", 2, 2); LastCall.Return("b");
mockLog.GetText("someFile.txt", 3, 3); LastCall.Return("c");
}
AnalyzeResults results = log.AnalyzeFile("someFile.txt"); mocks.VerifyAll();
}
The test in listing 7.21 is overspecified because it tests the interaction between the interface of some LogReader (which reads text files) and the LogAnalzyer object. This means it’s testing the underlying reading algorithm inside the method under test, instead of testing for an expected result from the method under test. The test should let the method under test run its own internal algorithms, and test the results. By doing that, we make the test less brittle.
Listing 7.22 shows a modified test that only checks the outcome of the operation.
208CHAPTER 7 The pillars of good tests
Listing 7.22 Replacing mocks with stubs and checking outputs instead of interactions
[Test] public void
AnalyzeFile_With3Lines_CallsLog3TimesLessBrittle()
{
MockRepository mocks = new MockRepository(); ILogProvider stubLog = mocks.Stub<ILogProvider>(); using(mocks.Record())
{
SetupResult.For(stubLog.GetText("", 1, 1))
.IgnoreArguments() |
|
Stubs unknown |
|
||
.Repeat.Any() |
|
number of calls |
|
||
.Return("a"); |
|
|
SetupResult.For(stubLog.GetLineCount()).Return(3);
}
using(mocks.Playback())
{
LogAnalyzer log = new LogAnalyzer(stubLog); AnalyzeResults results = log.AnalyzeFile("someFile.txt");
Assert.That(results.Text,Is.EqualTo("aaa")); |
|
? Asserts on |
|
||
} |
|
end result |
}
The important thing about this test is that the end assert ? is against the end result, and it doesn’t care how many times the internal Text() method is called. We also use a stub that doesn’t care how many times it gets called, and it always returns the same result. This test is much less fragile, and it tests the right thing.
NOTE When you refactor internal state to be visible to an outside test, could it be considered a code smell (a sign that something might be wrong in the code’s design or logic)? It’s not a code smell when you’re refactoring to expose collaborators. It’s a code smell if you’re refactoring and there are no collaborators (so you don’t need to stub or mock anything).
I am using NUnit’s Assert.That syntax instead of Assert.AreEqual because the fluent nature of the new syntax is much cleaner and nicer to work with.