How to speed up automated tests? Been there, done that

In this article, I will teach you how to speed up automated software tests based on one of the projects in The Software House where our testing tool Kakunin was used.

Need for speed

My fellow Quality Assurance engineers, has the tests-take-far-too-long problem ever happened to you? I am 100% sure it has. Sooner or later we ALL face it. In the case of our TSH project (let’s call it Project Q – we like our clients and don’t want to reveal their details), there was no difference! Ain’t nobody got time for waiting for ages until a long suit is completed. Another speed-related issue was maintenance – which is really painful if you have to debug your tests for such a long time.

How to deal with it?

The simplest idea is to reduce the number of tests by keeping only those critical for the system. We did it and indeed it resolved the problem in the Project Q – unfortunately just for a while. We’re not interested in temporary solutions. What’s more, new features were on their way, so we had to find a better solution.

Stop. Idea time!

BUT WAIT, there’s more… issues to resolve. 😑 To be more precise, the external authorisation service that increases the execution time. In nearly every scenario a user has to be authorised, so imagine a situation where you have to add more than five seconds to each scenario. When you have 100 scenarios, it increases the test run by almost 10 minutes! Unacceptable!

Luckily, we found a solution through creating a mock service but in the end, we decided to just authenticate through UI.

Why did we decide to abandon the mock idea? The answer is simple – the service was a part of a big ecosystem and was under dynamic development. That’s why we had to make sure that the mock was always up-to-date so this approach was not the best at that moment.

In the next stage, the Continuous Integration process has been added. And guess what? Even though the time was reduced, tests run was exceeding 90 minutes! Again, couldn’t stress this enough – unacceptable!

After quick research, we decided that parallelization was the feature we looked for and we went for a built-in Protractor’s solution.

Everything was awesome – implementation was easy and quick, tests were executed in the specified number of browser instances… but not in the way we wanted. The list of features was split dynamically and, after a few scenarios were executed, it made browser restarting over and over.

It wasn’t doing any favours for Project Q, so we started to look for a better solution. Surprise – we didn’t find any library that did exactly what we wanted to. So the decision was made to implement parallelisation on our own terms and via our own framework.

Kakunin I choose you

Enough theory, let’s get practical. Kakunin is TSH’s original open-source tool that allows you to write E2E test scenarios. Kakunin parallelisation feature is available as a CLI command:

How does it work? Before the tests are executed, Kakunin assigns all test files to the list that later is split to smaller lists. The number of lists is controlled by <numberOfInstances> parameter passed in the command. What did we achieve by that? Browsers are no longer restarted after a few scenarios and indeed it is a huge boost for the speed.

BUT (there’s always a but… sadfrog.jpg)! It didn’t fulfil the last of our requirements – one user can be logged only once in the system, and trying to authenticate on the same user in another instance will cause deauthentication in the first. So the chunk functionality had to be controlled by us too.

The previous parallelisation command has been extended by “pattern”:

A number of patterns must be exact with a number of instances e.g.:

So to execute the following features into two instances…

…we have to use the command:

So thanks to that feature, we have full control over tests executed in each instance. All we had to do from this stage was changing the organisation of files containing tests.

The final battle

Great, finally we have total control of instances and the way we chunk tests. The only remaining problem was related to “fixture restart” mechanism which was used for cleaning data generated by automated tests. All of the instances operate on the same database, so reloading the database was sure to cause unexpected issues.

We decided (quite painlessly, since we’ve saved some time before) that some scenarios can be changed and time of executing them might be a bit increased. In tests that were changing the state of the application, by using Cucumber’s hooks we added a mechanism to revert it after a scenario. At the same time, this helped us increase the speed of tests.

Let’s take a look at the implementation now.

To control capabilities, we use “browserConfiguration” method where we pass two arguments.

The first argument is a config file created as a part Kakunin initialisation process and contains basic settings. The second argument comes from the CLI with set browser type, headless mode, number of instances and more. This method contains all the settings needed for a test run as shown below.

In the case of parallelisation, the number of all features is divided by the number of instances passed as a parameter in CLI. We use “lodash” to chunk the features…

…and then push it as a part of configured instances.

Right after all of the instances are ready, Kakunin returns the whole configuration…

…which is resolved by “getMultiCapabilities” method in protractor.conf.js file.

Ending this story

Aaand that’s it! Now you should know a bit more about how to speed up automated tests. The only thing left for me is to strongly recommend you checking out Kakunin framework – it is very intuitive, available on the open-source licence, so you can try it out for free!

If you want to check out other examples of Kakunin in action, you can read another article about how to introduce automated tests. More information can be found in the Kakunin documentation.

Don’t forget about using the search option! 😀