First Principle of Test Code

When we write code and tests, we are specifying the same behavior using two different mechanisms, one involving abstractions (the code) and the other using examples (the tests).—Martin Fowler, Domain Specific Languages, 3.6.3

There are many different approaches to writing automated tests for a program. Unit, integration, and system tests are the industry norm, but fuzzy, property, snapshot, and model-comparison tests also exist to blur the definition of test code. All of these various types of tests have one core principle in common, however, in that they all recreate some of the functionality of the production code in order to compare outcomes between the test code results and the production code results.

Unit, integration, system, and behavior tests, for instance, compare the production results to predefined example outcomes in order to find errors. These tests are referred to as functional tests. Functional tests usually only cover a small portion of the possible scenarios for production code, but these scenarios may be selected to produce the most meaningful results. These tests then sacrifice completeness for extreme simplicity.

Property tests, conversely, procedurally generate scenarios for tests, but then are only able to check properties of the production results, as they tend not to know the precise outcome. These tests then sacrifice precise assertion of outcomes for testing more scenarios.

Trade-offs

Completeness for simplicity isn’t the only trade off that test code can make. As long as it is unlikely that the test code produces the same wrong outcome as the production code, the test code can make any trade off that the developer sees fit, including many that would not suit the production code. One such example is to create a naive, non-performant alternative to a complex production algorithm, such that the test suite can invoke fuzzy comparison testing between the two implementations. This test suite could then have a complete reconstruction of the problem space with which to apply assertions, but with solutions that are less likely to be incorrect, and are very unlikely to be incorrect in the same way as the production solution.