12 characteristics of good, automated tests

There are several reasons for writing automated tests and writing them may seem relatively easy. However, writing “good” automated tests is significantly harder, and requires extensive experience and conscious training.  

In this post, I’ve gathered some (high-level) targets that need to be checked in order for the automated tests to become good. The targets are defined in twelve characteristics and when they are fulfilled the definition of “good” automated tests is achieved. 

Here are some high-level targets that might apply: 
 

• Tests should help us improve quality. 
• Tests should help us understand the system. 
• Tests should reduce (not increase) risk. 
• Tests should be easy to run.
• Tests should be easy to write and maintain.
• Tests should require minimal maintenance as the system evolves around them. 

Tests should help us improve quality 

1. Test as specification

If you use TDD – test-driven development or BDD – behavior-driven development (test-first development), the tests give you a way to capture what the system will do before you start building it. Thinking different scenarios through in order to turn them into tests helps us identify the areas where the requirements are ambiguous or contradictory. Such analysis clarifies the objective of the specification, which leads to a more accurate design, improving the quality of the software. 

2. Defect repellent

Automated tests find bugs, but that is not the main purpose of test automation. Automated testing prevents bugs from being introduced. Think of automated tests as rejection of defects, which prevents bugs from coming back into our software after we’ve made sure it’s bug-free. When we have good and complete regression tests, there will be no bugs because the tests will point out the defects before we even check in our code. 

3.Designation of defects

If the automated tests are fairly small (that is, we’re only testing a single behavior in each one), we’ll be able to pinpoint the error quickly, based on which test is failing. But to achieve that we need to write tests for all possible scenarios to cover every unit of the software. The tests must never contain ambiguities. Therefore, it is crucial that we keep the tests as small and trivial as possible (low complexity, consistent format, and testing a single behavior in each test). 

Tests should help us understand the system 

4. Tests as documentation

Automated tests can clearly illuminate how the code should work. They show what the result should be (it indicates the expected result of one or more statements). 

 If we want to know how the system does something, we can fire up the debugger, run the test, and step through the code to see how it works. The unit tests act as a form of documentation for the system. 

Tests should reduce (not increase) risk 

5. Tests as a safety net

Changing older code is risky because we often don’t know what we might break, and it’s also hard to know if we’ve broken something! We have to work very slowly and carefully, and do a lot of manual analysis before making any changes. 

 However, when working with code that has an automated test suite, we can work much faster. The tests will catch unexpected side effects of changes and let us know if we’ve broken something. In this way, the automatic tests act as a safety net that makes us dare to take chances.

6. No test risk 

We must be careful not to introduce new types of problems into the system as a result of automated testing. Keep test code separate from production code to avoid creating test-specific dependencies in the system (especially important for unit test code). All test-specific code and libraries must be linked in by the test and only in the test build and in the test environment. Test dependencies and test code must never be in the final code when it is built for production. 

Tests should be easy to run 

There are four specific characteristics that make automated tests easy to run. With these four characteristics you can just click a button (or better yet trigger automatically) to get the valuable feedback the tests provide: 

•  The tests must be fully automated so that they can be run without effort. 

• The tests must be self-evaluative so that they can detect and report errors without manual inspection. 
• The tests must be repeatable so that they can be run multiple times with the same results. 
• Each test should be independent so that it can run by itself.
  
  

7. Fully automated tests

A test that can be run without any manual intervention is a fully automated test. Fulfilling this characteristic is a prerequisite for fulfilling the others.

8.  Self-evaluation

A self-assessment test can encode everything that the test needs to verify that the expected result is correct. The test only notifies us when the outcome has not been approved; as a consequence, a clean test run requires zero manual effort. 

9. Repeatable tests

A repeatable test can be run over and over and still gives exactly the same results without any human intervention/analysis between runs. 

Tests should be easy to write and maintain 

When we change the behavior in a part of a system, we should expect a small number of tests to be affected by our modifications. One of the benefits of test automation is to make changes easily. We should therefore always strive to ensure that our tests do not do the opposite (make changes more difficult). Tests should require minimal maintenance as the system evolves around them. 

10. Simple tests

Keep the focus on the tests rather than how to actually code them. This means that the tests must be simple/trivial (easy to read, easy to write and easy to maintain). We should strive to verify one condition per test by creating a separate test method for each unique combination of conditions. Each test method should test the system through a single path in the system. 

11. Expressive tests

A library of helper methods that builds a domain-specific testing language allows the person writing the test code to express the concepts they want to test, without having to translate their thoughts into much more detailed code. 

12. Divide the problems

Keep the test code separate from the production code (keep the structure and logic from the production code but in a parallel structure). Each test should focus on a single problem to avoid complicated and unclear tests. 

Summary 

There is a difference between a test and a good test, but it is often difficult to know how to define a test that is “good”. In this checklist, I shared 12 characteristics for good test automation practices, which in turn results in easy-to-write tests and proper maintenance – both factors, which are very important for a system.