Applying Spec-based generative testing to a transformation pipeline - tips and lessons learned

Generative tests are awesome but concrete cases of interest are easier to test with traditional example-based tests. Combine both kinds as appropriate. It is challenging to define the “properties” that generative tests check and they are necessarily somewhat general. They are good at checking many random inputs at a higher level. Compliment them with very concrete, example-based test(s) to verify the transformation in more detail and to test known, important corner cases that might not get generated. (Also, testing the same thing in two widely different ways gives us much more confidence that it really works.)

(We still generate random inputs in our example-based tests and then override the parts relevant to the test. That way there is less to hardcode, we test more, and there is less need to change the test as the shape of the input data evolves.)

For example apply-profile-compute-totals is responsible for summing up person’s usage into the 4 or 8 categories based on whether she has the basic or standard profile, applying the company’s limits on these categories and thus deriving how much of the costs should be paid by the user herself. The generative test checks the following properties:

The last property simplifies the input and output data (by adding them up and by not caring about individual categories while doing so) and then verifies that an important invariant holds (in this case, the amount of money coming in and out is the same). This is something we have used a lot as writing checks at a lower level would essentially require replicating the business logic in the test. But it doesn’t guarantee that the charges have been properly assigned to their respective categories and the correct relationship between charges, limits, and how much should be paid by the user vs. the company. Therefore we have also example-based tests for selected cases that check all the amounts (usage-over-limit, usage both including and excluding VAT, total-usage) in every category, as you can see below (you don’t need to understand anything beyond the fact that we check very concrete cases and individual data; BTW this is the most extensive example-based test in our codebase):

Ideally we would test the whole transformation pipeline. But it turned out to be more effective to test individual stages of the data transformation pipeline and only have a light “smoke test” for the whole pipeline.

We apply the Functional Core, Imperative Shell architecture and thus the top-level function takes in all the data necessary for the transformation, lot of it irrelevant for most stages. When we test an individual transformation stage, we only generate the subset of the data it needs, which will be faster and we will be able to run more tests in the same time span. The tests will also be more relevant to the function, we won’t be wasting cycles on varying data it doesn’t care about. Moreover it is much easier to tune the data to test the most important parts of the data space / avoid problematic ones.

For example many of the stages operate on a single user’s charges and it is thus much more efficient to focus on generating different sets of single-user charges than a number of different users, many with similar sets or charges. (Remember, the computer doesn’t know that it makes no difference to your program whether a particular set of charges belongs to the user “Sokrates” or “Buddha” so it might well try it with both.)

(Note: We have Spec-ed the most important arguments of the most important functions throughout the code and not just at the boundary of the system because we don’t care only about validating the data as it enters our system but also use it to document the shapes of the data flowing through for future maintainers / ourselves. Thus we can easily get test generation for these lower-level functions.)

Generative testing is dumb, it does not understand your data, it doesn’t know that variations in some parts of it are more interesting than others and that some variations are completely irrelevant for the test. For example we need user names but we don’t care what they are, we just pass them through so running the same test once with “John Doe” and once will “a#!” is useless. It also only matters to us whether “charge date” is before or after “invoice date” - how many days off it is makes no difference. The data space is huge (there is infinite number of “names” and dates) so we need to help the computer focus on the parts that are most interesting. Often we don’t care much about “variations in data” (though it is often beneficial to try different troublesome data, such as nil, ""`, very large numbers, floats that cannot be added exactly, etc), what we typically care about is`"variations in relationships." For example two charges belonging to the same user and category (so that we have to sum them up) or a charge with a type that is not among the known types.

You want to:

s/fdef allows you to specify the input and output specs but also the “properties” that should hold between the inputs and outputs via the :fn spec. We found it best to use named functions as the predicates:

because it makes both failures and the intent of the predicate easier to understand.

If a predicate function fails, the inputs are included in the error message as :args and :ret under (-> check-result first :clojure.spec.test.check/ret :shrunk :result-data :clojure.test.check.properties/error :data :clojure.spec.test.alpha/val). Alternatively, just use def inside the failing predicate function to capture the inputs.

What if a test fails? You likely don’t want to print the whole error that contains a lot of text and the whole input. You want to see the relevant parts of it - and you want to have the whole (shrunk) input available so that you can re-run and troubleshoot the function manually.

We do

Here are some useful subsets of the captured Spec error:

Here is our setup code and check enhanced with failure capturing and presentation:

Performance is a challenge for generative testing for a number of reasons:

Therefore you might want to limit the number of tests (which also limits the data generation time since the more tests, the larger / more complicated data is generated [well, not exactly, but…]). Perhaps you want to have quicker tests with fewer test cases while developing but run many more on the CI server. To set the number of tests:

Similarly, you can limit the size of the generated data, primarily the length of collections (BEWARE: This did nothing for me and I have to explore why):

Another reason for slowness is that generating test data that is valid may take time - and fail. The way this works is that there are base generators and ways to combine them and - most importantly here - ways to limit them. For example you want a string - but one that has 5 - 10 characters and contains only consonants. Test.check will generate a number of completely random strings and filter out those that do not satisfy these conditions (via its such-that). In this case it is likely to fail to generate any/many valid strings. Here is an example of such a failure:

You are better off providing a custom generator that only generates valid data (e.g. select randomly from the set of consonants → take randomly 5-10). However the problem - in my experience - is finding the spec that is causing the problem. Even if test-check fails to generate valid data, the error (as of now) does not contain any details useful in pinpointing the problematic spec (see CLJ-2097 and a workaround). And if it doesn’t fail, you just see that something is slow. Even a small, innocent-looking change to a spec may occasionally result in a noticeable slowdown. I wish I had a good troubleshooting recipe but I don’t. What worked for me to some extent is the classical divide-and-conquer approach: ask sg/sample to generate many samples (remember - the more different samples, the harder it has to work and the more likely is it to run into unsatisfiable such-that constraints) and, when I get a failure, repeat the process on the sub-specs of the failed spec (i.e. individual keys of an s/keys spec….) to see which one is causing it. I wish generators were linked to the specs and there was a way to enable time tracking that would give you a list of the slowest generators (per invocation / in total)….

st/check only checks the function you give to it. But you likely want to both discover if there is something wrong when invoking a helper function and possibly get an error that is closer to the source. Therefore enable spec checking for all functions invoked during the test: