In a perfect Scrum world, the team tests everything themselves. I think that misses an important point – the developers have a code-centric view on the domain. Good testing requires a user- or business-centric view on the domain. I think that it is impossible to both have a deep understanding of the code and to be a good tester.
That doesn’t relieve the developers from tests – developers having any pride in what they do of course unit test all their code. To get high quality software unit tests (whether automated or not, I’ll leave that discussion outside this post) is important, but not alone sufficient. There have to be system level tests and user/acceptance tests too.
A simple V model for a scrum project could have three levels of tests.
- Unit tests performed by developers, against the sprint backlog item descriptions
- System tests performed by a tester in the team, against the product backlog item user stories
- Acceptance tests performed by the customer against the project goals.
Testers are Requirements Stakeholders
All of the tests above are performed against a specification. This means that not only the developers, but also the testers are stakeholders of the requirements in the project.
For the sprint backlog items and product backlog items it is quite simple, because those are produced anyways. For the acceptance tests it is harder, because in scrum there is no complete specification of the system produced. The developers don’t need it. For us, it is enough with product backlog items describing what to do in each sprint.
One of the fundamentals of agile software development is to embrace change. With scrum, that is handled through the product backlog which can contain both new features and changes to existing features. Consider for example a price calculation engine for a cab company. In each sprint, a new product backlog item which evolves the calculation is implemented.
- The fee is €2 per kilometer, plus a starting fee of €2 for each drive.
- On weekends and during nights the fees are 50% higher.
- There is no fixed starting fee, but a €2/minute fee when an ordered cab has to wait.
- There is a €1/minute fee during drive as well. The kilometer fee is €1.
For each of the four sprints, the requirements are clear. It is possible to do the system tests after each sprint against each of these product backlog items. When doing the acceptance tests on the complete system there is a problem: What requirements should the tests verify? Each of the product backlog items describes what to do in the sprint, but none of them holds the final, complete requirements.
A Complete Specification is Needed
To do the acceptance tests we have to know the final, complete, requirements. Even though no complete specification is produced for the developers, it is still needed for the tests. My experience is that the need for the complete specification grows over time.
When the deltas of the product backlog items are too many – or spans over a too long time frame – to be remembered, the complete specification has to be there. Especially when handing over a project to maintenance and getting the delivery accepted by the customer there has to be a specification of what the system does.
For an ordered cab, there is a €2/minute waiting fee until the cab starts driving. During driving the fee is €1/minute + €1/kilometer. On weekends and during nights the fees are 50% higher.
It doesn’t have to be more complicated than that, but the description has to be somewhere or there will inevitably be discussions of the order of the product backlog items and how they overrule each other.
When to do tests in Scrum?
The three levels of tests are performed at different times in a project. Starting at the bottom of the V model, unit testing should be part of the definition of done for sprint backlog items.
The system tests should be performed once the complete product backlog item is finished. I think that a good approach is to have the system tests performed in the sprint right after the functionality was implemented. The development and testing of a feature will be spread over several sprints.
- Sprint 1: The original PBI (product backlog item) is implemented and unit tested
- Sprint 2: The original PBI is system tested, and additional requirements are detailed.
- Sprint 3: An adjustment PBI based on the test results and the additional requirements is included in the sprint, implemented and unit tested
- Sprint 4: The adjustment BPI is system tested and the final specification is updated for the acceptance tests.
- At the end of the project the final result is tested against the final specification
Testing the features in the sprint right after they where implemented includes the testing in the iterative approach of scrum. With the testing done continuously during the project the overall risk of the project is reduced. The earlier the testing starts, the earlier any problems can be addressed.