How Clojure helped us recover from bad data
- How does Clojure help?
- Small, standalone functions with minimal dependencies
- Separation of calculations and actions
- Data as a first-class citizen
- How does Clojure REPL help?
On a calm autumn morning we got a desperate call from our customer service. Our biggest customer had just started a pilot of our “expense share” functionality - and was missing half of their data. And they absolutely needed them for sending salaries in a few days. We jumped into production, cross-checked the data against a few of the missing employees and quickly determined that they were missing from the incoming invoices. We were able to cut and glue pieces of production code to extract just the information we needed - counts of employees and invoices past, present, and expected for billing runs in the problematic period - and thus identify all affected customers. After a fix in the source system, we were able to repeat the billing runs without writing any data, just to verify it generated the correct results - and write the results later. Combining data from web services and the database, correlating them and massaging them just into the format needed, interactively, with immediate feedback - that was only possible thanks to the code being in Clojure and thanks to Clojure REPL. Without it, the troubleshooting and correction process would have been much more difficult and time-consuming. Let’s look in detail how exactly did it help.
(Published originally at the Telia Engineering blog.)
Primarily in three ways:
Small, standalone functions with minimal dependencies
Separation of calculations and actions (i.e. the Functional Core, Imperative Shell architecture)
Data as a first-class citizen
The code is essentially a pipeline of data retrieval, transformation, and storage, composed of small functions, each taking in just the bare minimum of what it really needs. It is thus easy to pick, chain, and add to just the functions you need. And even if a function does more than you want - for example it runs the input data through a 4-stage transformation pipeline while you only want the first two stages - you can easily make a copy with just that, it is only a few simple lines of code. Here is an example of a typical function:
(defn categorize-summarize-subscr-charges [feature-mappings subscr-charges] (->> subscr-charges (categorize+batch-filter-charges feature-mappings) (kd-rules/categorized-charges-check+filter) (charges->subscr-summary)))
As you can see it doesn’t do much - it just coordinates other (small) functions. The original, object-oriented version of this code was implemented as a graph of cooperating objects, each typically with a couple of dependencies of its own, with mostly non-reusable private methods, and an occasional data retrieval in the middle of its code. It was intended and only could be used as a monolith, you couldn’t pluck and re-combine its parts as we did here.
A key aspect of our design is the Functional Core, Imperative Shell architecture (described elsewhere) - we first assemble all the data (though some is supplied few at a time via
core.async channels due to memory limitations), then process it through side-effect-free “functional core,” where 90% of the code lives and which produces a description of the desired effects, then apply the effects. The first and last part happen in the thin “imperative shell” / outer layer of the code. All “actions” happen there while all side-effect-less calculations happen in the core.
We leveraged this in two ways. First, we were able to run the process to verify that all the source data has been corrected, yielding the expected counts of employees, without affecting anything, by running only the data retrieval and calculations. We were also able to store the desired “effects” from the run into files and apply them later after validation and a go from stakeholders without needing to rerun the whole resource-intensive job. Second, we were able to use only the most outer level of the data-extraction code to get all the raw data, move them to a more powerful machine, and continue the processing there. (That was before we added
core.async to solve the memory issues with large data sets.)
In general, having separated actions and calculations, we can safely run any function in the “core” (i.e. any calculation) and don’t need to worry about any side-effects. We can control which side-effects and when to perform.
Every data is in this form, no matter where it comes from - web services, database access code, ad hoc DB queries - so you can interactively get and combine data from all these sources to solve your problem.
You can also leverage generic data utilities - Clojure’s
print-table to show sequences of maps in a human-friendly form or a data diffing library to quickly find differences between two data graphs.
(I have written about this topic before, in Clojure vs Java: The benefit of Few Data Structures, Many Functions over Many Unique Classes.)
You can learn more about that in our post Introducing a Clojure REPL server into our predominantly Java application has been the greatest productivity boost ever.
We benefited a lot from small, standalone functions with minimal dependencies that we could pick and combine to find out just the answers we needed, from separation of calculations and actions so that we could perform them at separate machines and verify results without publishing them to our users, and data as a first-class citizen which made data transfer and comparison possible. The REPL put all the power of our development tools, programming language, and functions at our fingertips and enabled us to iterate quickly toward the answers we are looking for. I cannot imagine how much more time and effort we would have spent if we did not have Clojure and the REPL at our disposal. I only wish we also had the superpowers of Datomic instead of our RDBMS so that we could have looked at relevant data as of the time of the original billing run, which would make the comparison of the original and “fix” runs more reliable.