Code Coverage != Functional Coverage

To reach 100% testing coverage is a dream for many teams. The metric used is code coverage for tests, but is that enough? Unfortunately not. Code line coverage is not the same as functional coverage. And it is full functional coverage that really matters in the end.

Look at a simple method that formats a string.

public static string Format(int? value)
  const int defaultValue = 42;
    value = defaultValue;
  return "The value is " + defaultValue + ".";

There is a correct test for this method, that pass and gives 100% code coverage. Still there is a severe bug in there. Can you spot it? (If you don’t, just keep reading on, it will be obvious when another test is added later.)

Rerunning all Unit Tests with Different UI Culture

I was recently made aware that some unit tests for Kentor.AuthServices were failing on non-English computers. To handle that, I set up an Azure VM with Swedish installed and made a special unit test that would run all other tests with different UI cultures.

When I first understood that I had tests that were broken when run on non-English computers I of course felt that it should be fixed. The tests should not only run with other languages to enable developers from other countries. The tests should of course be possible to be run and used to find problems if someone reports errors on computers having a special language installed. There’s quite a few places in the code with string formatting and it can differ with different cultures, causing hard to find problems.

What I did was to write a special unit test that finds all other unit tests in the code and runs them with different UI culture. The unit tests are found using LINQ and Reflection (it’s an awesome combination that’s extremely powerful) and then they are run with reflection.

Improving Unit Testing with FluentAssertions

FluentAssertions is an alternative assertion library for unit tests, to use instead of the methods in Assert class that Microsoft provides. It has much better support for exceptions and some other stuff that improves readability and makes it easier to produce tests.

The coding of Kentor.AuthServices was a perfect opportunity for me to do some real TDD (Test Driven Development) again. I have long thought that the [ExpectedException] attribute that MsTest offers is not enough, so when Albin Sunnanbo suggested that I’d look at FluentAssertions I decided to try it.

Verifying Exceptions

FluentAssertions offers a ShouldThrow() extension method to the Action delegate type. It asserts that invoking a particular action will throw an exception.

// Code from
// Kentor.AuthServices.Tests/Saml2ResponseTests.cs
Action a = () => Saml2Response.Read(response).GetClaims();
.WithMessage("The Saml2Response must be validated first.");

Compared to the [ExpectedException] attribute this offers much better control.

Calling Non Public Setters

Properties with non public setters sometimes make sense – until a unit test requires an object to be set up in a specific state. I don’t want to make the setter public just because of the tests, so I wrote a small helper function that can call non public setters.

The helper returns the object itself, to allow a fluent syntax inside member initialization expressions and uses a lambda to select the member (I do like compile time checking whenever it’s possible).

This is actual code I wrote last week using the helper. It’s part of a set up of an object collection used for unit tests.

Orders = new List<order>
  new Order
    OrderId = 983427,
  }.CallNonPublicSetter(o => o.StatusId, OrderStatus.Confirmed),
  new Order
    OrderId = 18956,
  }.CallNonPublicSetter(o => o.StatusId, OrderStatus.Delivered)

The helper is implemented using reflection to access the property to bypass the protection level of the setter.

Test and Verification in Scrum

In a perfect Scrum world, the team tests everything themselves. I think that misses an important point – the developers have a code-centric view on the domain. Good testing requires a user- or business-centric view on the domain. I think that it is impossible to both have a deep understanding of the code and to be a good tester.

That doesn’t relieve the developers from tests – developers having any pride in what they do of course unit test all their code. To get high quality software unit tests (whether automated or not, I’ll leave that discussion outside this post) is important, but not alone sufficient. There have to be system level tests and user/acceptance tests too.

Software Development is a Job – Coding is a Passion

I'm Anders Abel, a systems architect and developer working for Kentor in Stockholm, Sweden.

profile for Anders Abel at Stack Overflow, Q&A for professional and enthusiast programmers

The complete code for all posts is available on GitHub.

Popular Posts



Powered by WordPress with the Passion for Coding theme.