April 6, 2021

Leaky Scrum, Squishy Testing

When bugs are taking overhand, the sprint goal is frequently missed, team morale is down, and developers are working themselves into burnout, we may have a case of Leaky Scrum, Squishy Testing. Agile Testing helps. But how can you recognize this problem before it's too late?

In my view, this common failure mode of agile development is caused by not allowing the development team to take over enough testing activities. As a result, the team is not able to assume their responsibility for the quality of the product. When initially adopting agile methods, testing after development may still feel quite natural. However, under pressure to deliver, this situation can destabilize and ruin the development effort.

The symptoms of what I call a terminal state of Leaky Scrum, Squishy Testing are quickly summarized:

  • quality problems of increasing severity
  • inconsistent velocity makes planning difficult
  • low morale leading to burnout and increased turnover.

Fortunately, it takes some time for things to go this bad. But how can we recognize the situation in time and handle it?

Quality is the responsibility of the whole team

Last week, I described how in an agile context the responsibility for quality shifts to the development teams. It is crucial that all necessary testing activities take place within the development sprint so that the done increment meets the quality requirements of a release. Programmers and testers collaborate closely so that, upon completion, a user story or feature is also tested and covered by automated regression tests.

Pressuring a disempowered team leads to a quality death spiral

But what if the team is not aware of their role in maintaining quality or not fully enabled and empowered to meet it? This can quickly lead to a downward spiral, in which quality yields to time pressure. Let's look at this in more detail:

  1. Quality issues: Critical bugs are reported by QA, by operations or by the customer and have to be fixed.
  2. Pressure to deliver: This unexpected effort complicates release planning, it is increasingly unclear whether delivery deadlines can be met. The product owner or project management exert pressure on the team to get development done faster.
  3. Lack of Backpressure: The team, not being able to point to their responsibility to keep quality up, cannot resist. They yield to the pressure and accept more development work.
  4. Abandonment: The team reduces test activities and work on testability in order to have more time for implementation. Quality is moved out of focus and increasingly understood as someone else's problem.
  5. Positive feedback loop: Quality deteriorates and more critical bugs occur.

This gives rise to a vicious circle. Increasingly, quality issues cause everyone in the team to develop under more and more pressure. At the start of a sprint, it is unclear what can actually be delivered at the end. Developers are frustrated because they don't have the opportunity to make the necessary improvements. As management begins to doubt the development team, developers begin to doubt the agile development process itself. This leads to the kind of situation that Ron Jeffries refers to as "Dark Scrum".

Workarounds

When I recognize this situation in a client organization, it usually means that there is pain, but the organization is not collapsing. Instead, the situation is managed by changing the organization to stop the downward spiral described above. I will briefly outline three of these workarounds, or organizational "hacks" if you will. Perhaps some of them will sound familiar.

  • Testing Silo: Instead of collaborating with programmers during an iteration, testers work separately in a QA group and focus on detecting quality problems after a development iteration. The testing efforts of programmers and testers taken together keep quality stable, but often these efforts are redundant and wasteful. This is a rather common situation and in my view the result of an incomplete or failed adoption of agile testing.
  • Fake Kanban: The development process is split into implementation and testing on a Kanban board, raising the spectre of a mini-waterfall. In contrast to a proper Kanban, no WIP limits exist. Stories are accepted by the PO as done and demonstrated in stakeholder reviews even if they are not tested. The work of testers in the team is marginalized.
  • Hardening Sprints are entire sprints with the goal to raise the level of quality before release. Of course, this implies that the other sprints do not yield potentially shippable increments.

All these workarounds have in common is that they push testing out of the scope of the actual development iteration. The problem is that they disempower and limit the effectiveness of the development team. Delayed testing will naturally find bugs. Release planning has to account for time to fix them, and the development team has to deal with the increased stress of additional context switching. This costs time that can't be invested in developing new features or in improving testability and decreasing future testing efforts.

It also stunts the growth of an agile team and the surrounding organization. Imagine a DevOps team, deploying frequently to production, having hardening sprints.

The biggest risk with these workarounds, however, is that they can fail when pressure on the team increases. The development team already lacks the power to hold up under that pressure. Will testers in a QA silo be more empowered to block a release? Is a hardening sprint enough time to discover and fix the most egregious bugs? If not, the downward spiral I described above will start again, and quality deteriorate.

Increasing pressure

How can we tell that there is a problem? When I look at a software development organization, I pay attention to a handful of criteria derived from the chain of effects described above.

  1. Quality issues: Are a growing number of items in the sprint backlog bugs? Does the team invest more than about 10 to 20 percent of its development time in fixing defects?
  2. Pressure to deliver: Are PO or project management anxious about delivery deadlines? From their perspective, is there a perceived lack of time to improve quality?
  3. Lack of backpressure: Does the sprint backlog contain about one and a half to three times the amount of work than would be expected based on previous sprints? Or in the case of Kanban, are WIP limits ignored or nonexistent?
  4. Abandonment: Is there a separate QA after development? Does the Definition of Done not include all testing activities necessary for a story? Is it not being followed?
  5. Decompensation: Is the separate QA flooded with work and unable to keep up? Do features have to be finished in stabilization sprints? Are tickets piling up in the testing column of the Kanban board? Any signs, that testing after development is not sufficient?

If all five criteria are met, I assume that quality is going into a downward spiral. Unfortunately, the only thing to do here is to reduce the pressure over a longer period of time. The team has to be able to return to a workload that allows them to keep quality stable. Even better would be to enable them to take full responsibility for quality at the same time.

This brings us back to Leaky Scrum, Squishy Testing. A Scrum without clarified quality responsibility is leaky: The team cannot build up sufficient backpressure to guarantee the quality of their product. Software thus leaks unfinished and with defects to a QA stage after development, to the production environment, or to the customer.

Testing by its very nature is squishy: it is risk-based and risks do not manifest themselves immediately. Just because I don't run a particular test doesn't mean that a defect will occur. Merely the risk for later defects increases. That's why it's all too easy to cut corners regarding testing. When defects finally occur, it's not easy to restore the original level of quality.

Leaky Scrum, Squishy Testing is a problem because the development team cannot hold up under pressure to deliver and its effectiveness is limited. Development is more stressful and less satisfying for everyone involved. Potential to improve quality and sustainably increase development velocity remains untapped. It also becomes more difficult to evolve the organization towards a devops mode of delivery. Agile teams require slack to work on process improvements and achieve full adoption of Agile Testing.

Tags: Testing, Agile, Dysfunction