When Code Coverage Betrayed Me

I’m a fan of code coverage as a way to ensure that there are covering tests. One area that I tend to rely heavily on Code Coverage for is to catch any tests that are no longer working correctly due to changes in the production code. That often works out well, but today I got betrayed by the code coverage engine.

The code that I worked on contained an if statement with a multi-step && expression.

void IsAllWrong(int importantValue, bool b)
  bool a = importantValue == GetAnswer();
  bool c = false;
  bool d = false;
  if (!a && !b && !c && !d)
    return true;
  return false;

Of course I had tests that made the evaluation fail both because of importantValue and b. So what happend later was that GetAnswer() was updated, without the test for when importantValue being updated. Of course (my bad) that test had set b to true, causing the evaluation to fail on b, causing true to be returned. So the test passed, but not due to the thing I wanted to test. In a complex application, this is bound to happen every now and then. But usually, the code coverage scores will reveal that there is an execution path not covered. But not this time! The trustworthy code coverage analysis betrayed me!

Code Coverage does Matter

Is it relevant to have a code coverage target? In a talk at NDC Oslo 2014 Uncle Bob said that the only reasonable goal is 100%. On the other hand Mark Seemann recently said on twitter and in a follow up blog post that “I thought it was common knowledge that nothing good comes from code coverage targets”. Those are two seemingly opposing views.

Before looking at the role of code coverage, I’d like to take a few steps back and look at the objectives of testing.

When working properly with test driven development, no production code is written unless there is a failing test driving the change. As soon as the test pass, no addition of production code is allowed until there is again a failing test. If that principle is followed, there will simply be no production code that is not covered by a test.

My objective is to achieve high code quality.

My way of achieving that is TDD.

An effect of that is that I do get full code coverage.

Code Coverage != Functional Coverage

To reach 100% testing coverage is a dream for many teams. The metric used is code coverage for tests, but is that enough? Unfortunately not. Code line coverage is not the same as functional coverage. And it is full functional coverage that really matters in the end.

Look at a simple method that formats a string.

public static string Format(int? value)
  const int defaultValue = 42;
    value = defaultValue;
  return "The value is " + defaultValue + ".";

There is a correct test for this method, that pass and gives 100% code coverage. Still there is a severe bug in there. Can you spot it? (If you don’t, just keep reading on, it will be obvious when another test is added later.)

NDC 2014 Highlights

Last week, I was in beautiful Oslo in Norway most of the week for NDC 2014. It was a great conference and I’d like to point out a few highlights.

For the first time, I was a speaker at a major conference. I’ve done quite a few internal talks before and a few externals too, but never at such a high profile event as NDC. Looking at the speaker list I’m really honoured to have been part of it. I think it was challenging and fun – but I also found myself much more nervous than I had anticipated.


There were so many great people at the conference (both speakers and participants), but there are some that I think stand out with exceptionally good talks.

  1. Troy Hunt’s How I hacked my way to Norway. Troy is entertaining and educating on the same time. A talk that is both fun to listen to and that actually gives some concrete advice on how to (not) do security. I’m a bit disappointed though that he never used the IKEA allen key when hacking sites ;-).
  2. Uncle Bob Martin’s Advanced TDD: The Transformation Priority Premise. I’m a huge fan of TDD, but Uncle Bob brings it to another level, talking about 10 second oscillations between writing test and production code. Despite the title, I think it’s worth listening too even if you have limited previous experience on TDD.
  3. Luke Wroblewski’s It’s a Write/Read (Mobile) Web. The keynote, which was an eye opener to me that mobile and touch devices are not only for passive consumption of material. I especially liked the count of the number of clicks required to book a hotel on the major sites (well over 100) compared to FOUR on the best one.
  4. Scott Meyer’s Effective Modern C++. An introduction to some of the “new” features of C++11. The talk makes sense to a C# developer too – C++ developers are often far ahead of us in being aware of the details of how the language works and what the pitfalls are. Although there are more pitfalls in C++, a lot of the things Scott talked about applies to C# as well.
Software Development is a Job – Coding is a Passion

I'm Anders Abel, a systems architect and developer working for Kentor in Stockholm, Sweden.

profile for Anders Abel at Stack Overflow, Q&A for professional and enthusiast programmers

Code for most posts is available on my GitHub account.

Popular Posts



Powered by WordPress with the Passion for Coding theme.