When Code Coverage Betrayed Me

I’m a fan of code coverage as a way to ensure that there are covering tests. One area that I tend to rely heavily on Code Coverage for is to catch any tests that are no longer working correctly due to changes in the production code. That often works out well, but today I got betrayed by the code coverage engine.

The code that I worked on contained an if statement with a multi-step && expression.

bool IsAllWrong(int importantValue, bool b)
{
  bool a = importantValue == GetAnswer();
  bool c = false;
  bool d = false;
 
  if (!a && !b && !c && !d)
  {
    return true;
  }
  return false;
}

Of course I had tests that made the evaluation fail both because of importantValue and b. So what happend later was that GetAnswer() was updated, without the test for when importantValue being updated. Of course (my bad) that test had set b to true, causing the evaluation to fail on b, causing true to be returned. So the test passed, but not due to the thing I wanted to test. In a complex application, this is bound to happen every now and then. But usually, the code coverage scores will reveal that there is an execution path not covered. But not this time! The trustworthy code coverage analysis betrayed me!

Code Coverage on GitHub PRs with Coveralls.IO

2016-06-05 18_01_53-KentorIT_authservices _ Coveralls - Test Coverage History & StatisticsWith Coveralls.IO it’s possible to get code coverage on all pull requests submitted. For Kentor.AuthServices I’ve set up AppVeyor builds that uses Coveralls.Net to upload coverage numbers to Coveralls.IO.

Why Coveralls.IO?

When I set up the code coverage I already had AppVeyor builds running. I wanted something that worked together with AppVeyor. I tried out a few options and quickly found out that I also wanted something that used the Visual Studio Code Coverage engine. The project had 100.00% coverage in Visual Studio when I started and when I tried another coverage engine it showed some lines as skipped. I wanted consistency with the Visual Studio Code Coverage, which coveralls could give me.

Software Development is a Job – Coding is a Passion

I'm Anders Abel, a systems architect and developer working for Kentor in Stockholm, Sweden.

profile for Anders Abel at Stack Overflow, Q&A for professional and enthusiast programmers

Code for most posts is available on my GitHub account.

Popular Posts

Archives

Series

Powered by WordPress with the Passion for Coding theme.