This is a guest post by Albin Sunnanbo sharing experiences on regression testing.
On several occasions I have worked with systems that processed lots of work items with a fairly complicated algorithm. When doing a larger rewrite of such an algorithm you want to regression test your algorithm. Of course you have a bunch of unit tests and/or integration tests that maps to your requirements, but unit tests tends to test systems from one angle and you need to complement with other test methods to test it from other angles too. We have used copy of production data to run a comprehensive regression test with just a few hours of work.
Our systems had the following workflow
- Users or imports produces some kind of work item in the system, i.e. orders.
- There is a completion phase of the work where the user commits each work item and make the result final, i.e. sends the order.
- Once each item is final the system processes the work item and produces an output that is saved in a database before it is exported to another system.
We have successfully used the following approach to regression testing for those kind of algorithms.
To reach 100% testing coverage is a dream for many teams. The metric used is code coverage for tests, but is that enough? Unfortunately not. Code line coverage is not the same as functional coverage. And it is full functional coverage that really matters in the end.
Look at a simple method that formats a string.
public static string Format(int? value)
const int defaultValue = 42;
value = defaultValue;
return "The value is " + defaultValue + ".";
There is a correct test for this method, that pass and gives 100% code coverage. Still there is a severe bug in there. Can you spot it? (If you don’t, just keep reading on, it will be obvious when another test is added later.)
When developing a system that sends mails, often the mails shouldn’t be sent for real when testing. Instead they should be made available for investigation. Fortunately, that functionality is built in with the .NET SmtpClient.
There is even no need to change the code. It’s just a matter of configuration. Add the following lines to the
web.config for web applications)
<smtp deliveryMethod="SpecifiedPickupDirectory" from="firstname.lastname@example.org">
<specifiedPickupDirectory pickupDirectoryLocation="c:\temp" />
<!-- The network host setting isn't used, but without it an exception
occurs when disposing of the SmtpClient.-->
The pickup directory setting is meant to be used with a local mail server that watches a directory for new mails. I have no mail server watching my
c:\temp directory. Instead, the mails are just dropped there as
.eml-files that can be opened using a mail program (e.g. outlook).
In a perfect Scrum world, the team tests everything themselves. I think that misses an important point – the developers have a code-centric view on the domain. Good testing requires a user- or business-centric view on the domain. I think that it is impossible to both have a deep understanding of the code and to be a good tester.
That doesn’t relieve the developers from tests – developers having any pride in what they do of course unit test all their code. To get high quality software unit tests (whether automated or not, I’ll leave that discussion outside this post) is important, but not alone sufficient. There have to be system level tests and user/acceptance tests too.