In today’s digital age, the demand for reliable measurement tools has increased. When it comes to measuring the effectiveness of advertising campaigns, behavioral data measurement products play a vital role. However, ensuring the accuracy of these products can be a challenging task. Given the multitude of devices and platforms available on the market, it is essential to ensure the smooth operation of your product across all of them. That’s where test automation comes in – it’s a way to test your product quickly and accurately across a range of platforms.

Automated Functional Testing vs Unit Tests

One can say that unit tests are enough to ensure the quality and detect issues during development. However, there are significant differences between unit tests and automated function testing. The following list describes just a few of them:

• Unit tests are running when the software is built. This is great because it provides quick feedback. But it also means that the environment where these tests are running is always the same and limited to the environment of the build server. Another related factor is that unit tests usually cannot communicate to real backends, so they are as good as mocks or emulation of backends they use.
• Unit tests work with internals of a software. This is great because they can target every function in isolation, but they never see the whole picture.Test automation works with a larger scope and tests the product as whole.
• Unit tests cannot help to detect problems related to changes in 3rd-party systems. This is especially important for the Ad Measurement domain, because the 3rd-party dependencies can change at any moment without notice. This leads to downtimes and losses. Test automation can run continuously and detect these kinds of problems.

Unit tests are great and every project should aim for 100% unit test coverage. At the same time, they are not a direct competitor of test automation systems. Both systems are important contributors towards product quality.

The goals of test automation

• Ensure problems are detected earlier in the development process, saving development costs.
• Iterate quickly between changes and testing.
• Make it safer to gradually rewrite, reducing the risk of introducing new bugs. For more information about gradual rewrite, see the Gradual Rewrite for Reducing Technical Debt blog post.
• Ensure the product meets the necessary quality standards before release, reducing the risk of releasing a feature with bugs.
• Detect changes in 3rd-party systems and react to these changes in a timely manner, ensuring the product remains stable and reliable.

Technologies

There are many technologies available for test automation. The choice depends on several factors, like what language your team is the most comfortable using, what language is already in use, and what language is the easiest to maintain. It is also important to evaluate the available options. For our test automation, we have chosen the stack described in the following sections:

• Script language
• Automation framework for Android
• Automation framework for browsers

Script language

We were considering Python, JavaScript, TypeScript, and Java. We used the following comparison between these languages:

Based on this comparison, we chose TypeScript, a programming language that offers static type-checking that helps to catch errors early and improve code quality. Our teams have already used this language for other projects, so the choice was natural.

Automation framework for Android

For Android test automation, we choose Appium – an open-source tool that is widely used for automating mobile app testing. It offers many features, including cross-platform support, robust testing capabilities, and a large community of developers. We run Android tests in an emulator inside a Docker container. We use a custom Docker image where the emulator is configured and required 3rd-party applications are installed. This makes it easy to start every test run from a predefined state by creating a new container from an image.

Automation framework for browsers

For the desktop browser testing, the choice was between Cypress and Playwright:

After evaluating all options, we decided to use Playwright – a newer browser automation tool that is gaining popularity due to its ease of use and cross-browser support. It offers a simple and intuitive API, which means less time spent writing complex code.

Test automation reports

Now we have some automatic tests that can run different scenarios and verify that the product behaves as expected. The next step is to decide how often to run these tests. In my experience, it is important to run several different flavors of test automation.

Pull request pipeline

Pull request pipeline is just like a unit test. It runs every time there is a change in the test automation or the product code. We want to verify those changes with test automation before they are merged. On the other hand we don’t want to hold merge requests for too long waiting for test automation to pass, so we select just a subset of the tests that provides good enough coverage and can be completed fast. In our case we try to keep the smoke test under 15 minutes.

Tests included into the smoke subset are marked with the @smoke tag.

Scheduled runs

Scheduled runs are critically important to verify that the integration with the 3rd party system works as expected. But even for stand-alone products, it is good to have up-to-date quality indicators. Frequent runs are good for addressing infrastructure instability and random failures: sometimes, tests may fail due to factors that we cannot control. Multiple runs show the severity of this kind of failure, so we can define a threshold under which failures are not critical and don’t generate alerts.

Scheduled pipeline runs all tests that are marked with the @stable tag.

Experimental pipeline

It is quite common when a new test scenario is introduced the test can be unstable or flaky at first. Sometimes it is difficult to stabilize it during initial development. Tests can pass in pull requests, but fail in 10% of runs, for example. To address this issue, we use a separate experimental pipeline. It runs a few times a day.

New tests are added to the experimental test set first, and in a week or so, when there is a statistics of runs, they may be marked as stable with the @stable tag.

Historical stats

The last point brings us to several important questions. How can we collect and monitor the test automation system from a historical point of view? How can we answer a question about the history of a particular test? To collect these kinds of stats, we use NewRelic: every test sends an event containing the result and other relevant statistical information. In NewRelic, we can monitor and query information about test history. We can also see how different product versions behaved with different tests. It is also possible to generate daily reports and alerts using historical information from NewRelic.

Who is the test automation for?

Test automation systems are powerful tools that can significantly improve the quality and efficiency of software development. However, it is important to understand that their true value can only be realized if all developers on the team know how to use them effectively.

One of the key benefits of automated testing is that it allows for faster feedback on code changes, which can help catch issues early in the development process. This means that developers can iterate and fix problems quickly, which can save time and reduce the likelihood of bugs making it to production.

However, if only a few test engineers or developers in the team know how to use the test automation system, then the rest of the team is missing out on these benefits. A big and complex test automation system requires continuous maintenance. If most of the team ignores it and expects that QA will create a ticket instead of going and checking test results directly, it may lead to continuously failed tests, and at the end, the system will not be trusted: “The test automation is red again, it’s always like that. Just ignore and release.”

How can we make sure that the knowledge is shared in the team? One approach is to have a rotation role in the team: each team member becomes responsible for monitoring and investigating test automation results for one week. It takes just a few minutes a day to check a daily or nightly report and tell everybody on the daily sync about the test status. The next week, another team member assumes this responsibility.

The more developers use the system, the more familiar they become with it, the better. At some point, it will be natural to write an automatic code, just like another step in the development process. If some information important to problem investigation is missing from the test automation, it can be spotted and added by developers: they know all aspects of the product and may have an idea of how to make test automation reports more useful.

Conclusion

Test automation system is a valuable tool for improving software development processes. However, to fully realize its benefits, all developers on the team must know how to use it effectively. By investigating problems and getting information from the system, developers can catch issues early, improve the quality of their code, and make the test automation even more useful.

Communicating with other team members and attending training sessions, sharing knowledge and best practices is the key to ensuring that the entire team is aligned and working towards the same goals.