Agile Testing Superpowers

In agile development teams, programmers and testers work together to create the next product increment. Unfortunately, Agile Testing is often implemented inconsistently. As a result, opportunities to increase effectiveness in agile projects are left untapped. Let's take a look at how testing changes in an agile context.

First a disclaimer: software development is complex and events can often only be understood in retrospect. Even when I describe the transformation to Agile Testing in a seemingly linear way here, this does not mean that it would always so predictably. Every context is different. From experience, however, I expect the following changes:

• risk-based approach,
• automated whitebox testing,
• focus of the whole team on testability.

The point of agile software development is its iterative incremental approach. Each increment developed is actually done and potentially shippable, meaning it can be shipped immediately at the product owner's discretion. This implies that all the necessary work for a release is completed. And that includes all necessary testing activities.

Agile testing therefore differs fundamentally from classical QA in that it does not happen after implementation, but at the same time. This means that all testing activities have to be completed within a sprint. In Kanban, testing activities are subject to a WIP limit, but the effect is similar. At limit, testing has to be finished before the next ticket can be implemented. This creates a form of backpressure: a team only takes on as much work as it can reliably deliver at the agreed upon standard of quality.

From a traditional tester's point of view, a new feature would of course have to be programmed before it could be tested. For the actual testing then remains only the short timespan at the end of a sprint. This takes the fundamental problem of testing — that there is never enough time for it — to an absurd extreme. How is that supposed to work out?

Taking Risks

A first shift toward an agile testing strategy is the realization that given limited resources, only a risk-based approach to testing makes sense. Risk-based testing is just as old as time constraints in testing themselves. In an agile context, however, the short iteration cycles make any pretense of comprehensive testing hopeless. Instead, test activities have to be prioritized by risk, which means by the probability of a defect occurring and the amount of the potential damage. This allows us to reduce the overall risk as best as possible in the time available.

Risk-based testing focuses on two areas: first, the changes made in the current increment, and second, the potential regressions caused by those changes. Since testers and programmers now work as a team, they are aware of the changes made and can test them in detail. Also, any regressions found are fixed immediately in the same sprint. This eliminates the ping-pong game of discovered bugs and fixes between development and QA that takes place over several sprints.

But it gets better. Due to this close collaboration between testers and programmers, they can use synergy effects to reduce time required for testing. If the same tests take less time to perform, then more risk can be covered in the same amount of time. The key to make testing fast is to get the programmers involved.

Welcome to the Whitebox

Programmers are specialists in automation. Their job is to write software after all. Software that is designed to support its users by automatically executing steps that would have otherwise been manual. Programmers also have a natural understanding of the structure of their software. This enables a structure-based approach to testing. Welcome to the whitebox!

Programmers are usually familiar with structure-based testing techniques such as unit or component testing. They use them to get feedback for the functionality of their solution during development and to quickly detect mistakes in their design. If functional testing is also repeated by testers after development, it is likely that tests are duplicated. This duplication is less common in agile development teams. Testers can convince themselves that a combination of multiple unit tests for corner cases and a single integration test already ensure functionality that they would otherwise test manually.

More importantly, developer tests are automated. Automated tests can cover in minutes what manual testing needs several days to cover. The difference is so immense that it is usually not even necessary to selectively run automated tests. They can be run in full and no later than during a CI build. High coverage with such automated tests provides excellent protection against regressions.

Additionally, every change or new development offers an opportunity to increase automated test coverage for the affected components. The unavoidable risk-based testing approach is complemented perfectly with techniques for automated whitebox testing. If the team consistently improves coverage of changed components, any actively developed parts of the system will eventually be covered with tests.

It now becomes very clear that testers in an agile team actually want to achieve a very high coverage with automated whitebox testing. This does not mean that testers do all the automation themselves. In fact, simply taking manual tests from QA and automating them will lead to very fragile tests that are expensive to maintain. It is important to get programmers involved in writing tests, so that they can add their knowledge of the software structure.

This also means that testers in an agile team will gradually learn more about this structure. How is the software broken down into components? What problems do these components solve? Also, is it immediately clear that the combination of these components represents a complete solution? Where do we need integration tests to make sure? Well-structured software usually allows tests to be distributed in the form of Mike Cohn's test automation pyramid.

Superpower Testability

As testers and programmers increasingly gel as a development team, they gain confidence in their test automation. They realize that in order to deliver quality, they all depend on a software structure that is simple to test. The whole team now has an incentive to make the software as testable as possible from the outset. They start to build quality in.

A testable design separates and isolates the functionality at risk (where subtle bugs are likely to cause real damage) from other aspects that are potentially more difficult to test. In enterprise application development, this means separating business and technology aspects, and is taken to the logical end in hexagonal architecture.

If there are legacy components that are not actively being developed and whose test coverage is insufficient, the team will strive to decouple from them as much as possible. This reduces the likelihood of regressions and thus the manual testing effort. This is again an improvement to the software structure brought about by feedback from testing in the same team.

To achieve high testability, it is important that the team has the ability to perform architectural changes as well. Time gained by an initial success in test automation should not be invested in new feature development alone. Software development teams require slack for redesign and refactoring. If the team is dependent on the expertise or decisions of an external software architect, it will be slowed down. Seniority and empowerment of the development team are the keys to success here.

The team's changing attitude to quality is also evident in refinement meetings. Their view of the requirements now increasingly integrates testability concerns. They will raise questions such as: Are the acceptance criteria testable? Is it clear how they will be tested? Is there an opportunity to adapt the software design to make it more testable?

Overall, an agile team that is mastering its testing strategy relies less and less on elaborate end-to-end testing or manual regression testing. More and more time remains for creative tasks. These consist at first of improving coverage with automated tests and improving testability. Even in an ideal world, manual exploratory testing activities will remain, although limited in time. They also represent a creative task and cannot be automated according to the current state of the art.

Provided that the team is not slowed down by external dependencies and develops no dysfunction, more time is gradually freed up for developing features at a pace that sustains quality. Testing is no longer perceived as a brake, but as a safety net that allows the team to move quickly and freely. The team becomes increasingly effective.

Testers as part of an agile development team play an important role in shaping the transformation to agile testing. Their tasks shift significantly — away from the execution of manual test plans after implementation, and towards their creative tasks such as identifying risks, coming up with and communicating an agile testing strategy, talking to stakeholders about testable acceptance criteria, and working closely with programmers in a highly productive team.

This post originally appeared in German on the Novatec company blog.