How to make the most of test automation

Test automation is an important aspect of developing high-quality software. When properly implemented, it provides you with continuous insight into your code quality, every time and unattended, freeing up valuable time of testers and developers. When your software is thoroughly tested, it serves as insurance – helping to solve potential issues as quickly as possible.

Pick the test approach that best suits your project
There are many different ways in which you can apply test automation, says Erwin Bonsma, Software Architect at TOPIC. To effectively test your software, you need a collection of different types of tests. Typically your test suite is structured as a pyramid. The base is formed by a large collection of unit tests, which test units of your software in isolation and run very quickly. The top of the pyramid is formed by system tests that test the entire system to verify that system-level requirements are correctly implemented. Intermediate layers contain tests that test components in isolation or subsets of components. As you go up in the pyramid, the tests become more expensive to set-up and execute, and failures can be harder to analyze. You therefore want to test as much as you can in lower layers.

How to best test your software varies per project. You need to find the way that best meets the demand of that particular project, balancing the benefits that test automation brings versus the costs (the effort to develop and maintain the tests). Where to focus your test effort depends on the system design and associated risks.

The context in which the software is developed also matters. For a short-lived feasibility prototype you do not need the same amount of test coverage as for a longer-lived and more sophisticated end-user product. Test coverage requirements are particularly high for medical devices or for software in other domains with stringent safety regulations. In these cases, test automation is highly valuable. Extensive test coverage is also essential for software stacks that use continuous delivery, where you want to confidently be able to deliver updates to the production stack frequently.

Design for testability
Whatever the type of project, you need to consider testing from the outset. It starts with expressing the system requirements such that they are verifiable. Next you need to design your software with testability in mind. Build in decoupling points in your architecture. Ensure that there are clear interfaces between the layers in your system. This facilitates mocking parts of the system in order to efficiently test a subset of system components.

Erwin has good experiences using Protobuf (protocol buffers), a language- and platform-neutral extensible mechanism for serializing structured data. It can be used to specify the API between distributed components (for example between an IOT device and the cloud service it interfaces with) but also for defining communication between co-located processes. During development it facilitates incremental updates of the protocol, as it is an executable specification of the protocol that can be extended without breaking backwards compatibility. For testing it is also beneficial. As it is language agnostic, we were able to quickly develop Python-based simulators for low-level components that were eventually replaced by control software in C++ that interfaced with actual hardware.

A human readable specification for automated tests
To test formal requirements, it can be very helpful to express the tests in the “Given-When-Then” syntax. This is a human-readable test specification that lets customers without any software development experience review the tests and provide feedback. As an example, Erwin says, we used this for testing software that provides feedback to patients enrolled in a clinical trial. Our customers were very enthusiastic that they could read the tests, review them, and even contribute tests themselves. During development they provided us with many additional tests. This helped to ensure that the system performs as required, but also gave the customer confidence in the quality of the final system while development was still ongoing.

Semi-automated testing
It is not always possible to fully automate testing. An example is where the software controls a mechanical device and you want to verify the functioning of the entire system. However, even here you can apply and benefit from test automation techniques. For example, in a recent project we had to test a medical device which at start-up verified that cartridges were correctly loaded by scanning their QR codes. Scanning involved rotating the turret to adaptively position the cartridges in front of the camera. To test this, we again used test specified in Given-When-Then syntax, where certain steps were automated requests to the system (e.g. “the system scans the turret contents”), and others were requests to the tester (e.g. "the tester inserts cartridge X at position Y of the turret"). This minimizes the actions required by the tester, thereby speeding up test execution and reducing the chance of errors when carrying out the test.

Conclusion
By applying the discussed automated test approaches, the quality of software running on systems can improve drastically. To effectively test your software, you need to optimize your test suite to fit the demands of your project. To facilitate development of automated tests, make sure you design your software for testability. There is a wide selection of tools and techniques available for testing your software. Although it can be challenging to select the right approach, we highly encourage the use of test automation in software development processes, as it increases the quality of your product, reduces overall development effort and results in more predictable progress during development.

Pictured: Erwin Bonsma, Software Architect at TOPIC

Want to learn more about this topic? Feel free to contact us via contact@topic.nl.