Mutation testing, a.k.a. the power of mutants

Luckily, we have a Hulk! And by that, I mean mutation testing. In this article, I’ll tell you all about mutation testing in software development and how they can help you with unit tests verification.

How much do you trust your unit tests?

The main task of unit tests is to check results such as limit values, alignment or expected values returned by the methods. You certainly don’t want to neglect that so you take the time to write your units and always check if they work. The tests have always been green, so it seems reasonable to answer the aforementioned question with:

A LOT.

Unfortunately, it’s not always rainbows and unicorns. When your application is becoming bigger, the number of tests are also increasing. A hundred percent coverage of the code with tests doesn’t mean that the tests really “test” something. Believe it or not, but sometimes unit tests give “false positive” results.

What if the question was asked by your client? Would you still be sure that you trust your unit tests A LOT? Luckily, there’s a method allowing you to feel more confident with this type of software testing.

MUTANTS. Mutants everywhere.

It sounds like something straight outta “Fallout” video game but mutation testing is a real deal. Now, the burning questions are:

• What is mutation testing anyway?
• What are its benefits?
• What problems can mutation testing cause?
• How to perform that kind of tests?

Yep, that’s a lot, but don’t get discouraged – I’ll explain everything step-by-step. Let’s start with a simple definition.

Mutation testing verifies if the unit test works properly. They introduce random changes in the source code that should cause the test unit to fail.

Paraphrasing Allan Wats: Let’s suppose that every day you were able to test any unit you wanted to test. And naturally, as you began this testing adventure, you would fulfil all your wishes. You would have every kind of path in your code covered.

And after several days or weeks of total pleasure, each of you would say: “Well that was pretty great. But now let’s have a surprise, let’s have a test which isn’t under control, where something is gonna happen to it that I don’t know what it’s gonna be.”

So what are those mutants?

As I mentioned earlier, in software testing unit tests define what are the expected outputs returned by assertions or methods. Let’s assume a simple source code:

and simple unit test cases for it:

As it is said, the test verifies if method “hasStatus” returns true if status that is described matches the status that comes from ApiResponse class.

If we change the value ‘===’ to eg ‘>’ the test should stop working.

Mutations that can be used are, for example, substitution:

• + ► –
• * ► /
• >= ► ==
• true ► false.
• suppression of instruction, etc.

How can you carry out such tests without losing a lot of time to “swap” our source code? There is always a framework for that.

Currently, the most popular framework in the JS world is Stryker. The task of this framework is to create our mutants in as many numbers as possible. You have to keep in mind that this process can be resource-consuming. How does this framework work?

• It generates the mutants basing on the existing source code,
• Sets them against the wall,
• Shots to them with unit tests,
• Counts the bodies.

For single mutant, there are two options:

• survival (test is green),
• death (test is red).

Usually, you’d expect from your units to be green, but after making changes to your source code, you surely expect them to be red (at the end we have introduced changes that our tests should catch). Every mutant that survived is a potentially badly written test.

How does it look in the real world?

Initializing the tool in the project is easy, just follow the instructions on the Stryker Mutator website and then go through a short cli that will allow you to adjust the Stryker to your needs. While going through the configuration, it can be seen that the Stryker supports several tests and js frameworks. When the tool is configured, the only thing we need to do is run the command “stryker run”.

To present how the framework works we will build a single class that will provide three methods: isOldEnough, fbList and clubEntryPrice.

As I’ve said earlier, the Stryker will change the source code and then it will look up for unit tests which he will be able to kill mutants.

Let’s focus on the clubEntryPrice method. We will write a unit test for it:

So you have a basic unit test for one of the method conditions. Now you can give a job to Stryker. The result of the first mutation test is the following image:

As you can see, the framework found one file on which it made a code mutation. It generated 20 mutants, and then passed them through unit tests. After the first test, you have five mutants that survived.

So we start to patch that so no mutants survive. The first observation is the lack of testing edge values. The second is the lack of testing the negative path of our method (“noAccess”).

So let’s change the edge values in our tests, and add a test for the negative path:

After changes and re-launching the Stryker, We’ve got this:

One mutant that survived was still there. This is due to the fact that the woman’s path is not covered. So if we add the last “it”, we should get a result in which none of the mutants will survive.

And the result:

Looking at the tests that we’ve carried out on a single method, we can conclude that the tool allows you to:

• find the right values that you should put in the unit,
• find the right paths to test,
• cover all the possibilities in such a way that your tests become resistant to changes in the source code.

Unfortunately, like every tool, it has some downsides. Firstly, the complexity of the source code affects the time of performing mutation testing.

Note that for a simple method, tests are triggered up to 20 times and their execution time is 9 seconds. To make you aware of the scale – I wrote tests for the entire class and launched them. In 12 seconds Stryker generated as many as 40 mutants.

Secondly, you may have to spend a relatively long time to analyze the final results.

Touching land

As you can see in our test cases, mutation testing gives real results that can improve the quality of your unit tests. You have to realise that it is best to introduce mutation testing right at the beginning of the project.

If you want to implement the tool during the application’s development, the results returned by the tool could be massive, which will probably discourage you from viewing them and finding out what’s worth improving.