June 17, 2016

Trying out Mutation Testing

Yesterday, my colleague Stefan Ludwig and I had a look at PIT, a mutation testing tool for the JVM, and tried it on the latest release candidate of the shiny new WebTester 2. We posted about our experiment on the NovaTec blog. Have a look!

Mutation testing looks really promising to me, because it addresses an important question: how reliable is my test suite? Simple code coverage metrics, computed for example with JaCoCo, only provide a very indirect measure of test quality. Especially in projects that mandate a certain level of code coverage, the metric is practically useless for developers. Worse, sooner or later a stressed out team member will add meaningless tests to the suite, simply to exercise production code and raise coverage.

But if a mutation testing tool eliminates statements in your production code and checks if there is a test that detects the change, the true reliability of the test suite quickly becomes apparent.

Tags: Testing, Feedback, Metrics