This has been a year of Clojure (and DevOps). Since the beginning of the year, I have worked to persuade my team to adopt Clojure instead of (or rather in addition to) to the current mixture of Java and Groovy, argued to rewrite a key batch job in Clojure (thank you, Monica, for providing the business case!), and did all I could to spread Clojure knowledge and skill in the team and surroundings. It has been a success - the job has been rewritten, we have already benefited from it, and we are finishing a new, Clojure-based micro-service. I have learned a lot about Java vs. Clojure, about the value of business-level tests and "living documentation", about leveraging Spec and property-based testing, about core.async (and error handling). On the DevOps front, I have grown to really dislike Terraform (wishing repeatedly to have a proper programming language instead of the frustrating Terraform DSL and the hacks it requires, not mentioning the nightmare of upgrading providers and Tf itself; 🤞for CDK/cdk-clj), have spent more time then I ever wanted with Kubernetes (and am working hard on simplifying our infrastructure and replacing K8s with AWS Fargate and thus spending less time on operations and more on development), and fell more and more in love with Clojure REPL, whether embdded in a Java app, in an actual Clojure code, or opening the door (securely!) to a "serverless" container on Fargate.

Error handling is something that core.async leaves to you (because it depends on your particular use case). People have written before about handling errors in the low-level go loops but there is little about the higher level constructs such as pipelines and transducers. I’d like to correct that. First, a little repetition.

You are running a Clojure microservice on AWS Fargate in a private VPC subnet and want to have a secure REPL access to it, only exposing the port inside the VPC. Moreover you would prefer to use SSM Session and SSM Port Forwarding instead of running and exposing SSH. Here is how to do it. (Beware: it is not as trivial as we would like.)

Clojure Zippers is a library for navigating and modifying tree data structures. While refactoring Cryogen, I needed an operation not supported out of the box (the removal of all nodes to the right of the current one) and thus had to learn a bit about the internal structure of zippers. I record it here for posterity.

A year ago I have moved my blog from Wordpress and its WYSIWYG editor to a static site written using React and GaphQL and generated by Gatsby, with entries in Markdown, hosted on Netlify. My motivation was control and ease of writing. Now I continue the trend by moving from Gatsby to Cryogen, a blog-focused static site generator in Clojure, and AsciiDoctor.

Sometimes the only way to get REPL into your production application is to tunnel it over HTTP. nREPL has a transport and Ring handler for that provided by Drawbridge. Heroku has a nice but too dated guide on using nREPL with Drawbridge. I would like to fill the missing bits here.

So you have just deployed your Docker container to AWS Fargate but it keeps on restarting with the event “service XYZ (..) is unhealthy ..” and you have no idea why. I have spent many bloody hours here and will gladly share my insights with you. The 3 key questions to ask are:

(+ a bonus question and more troubleshooting tips)

On a calm autumn morning we got a desperate call from our customer service. Our biggest customer had just started a pilot of our “expense share” functionality - and was missing half of their data. And they absolutely needed them for sending salaries in a few days. We jumped into production, cross-checked the data against a few of the missing employees and quickly determined that they were missing from the incoming invoices. We were able to cut and glue pieces of production code to extract just the information we needed - counts of employees and invoices past, present, and expected for billing runs in the problematic period - and thus identify all affected customers. After a fix in the source system, we were able to repeat the billing runs without writing any data, just to verify it generated the correct results - and write the results later. Combining data from web services and the database, correlating them and massaging them just into the format needed, interactively, with immediate feedback - that was only possible thanks to the code being in Clojure and thanks to Clojure REPL. Without it, the troubleshooting and correction process would have been much more difficult and time-consuming. Let’s look in detail how exactly did it help.

I want to split my application uberjar into a separate JAR with only the dependencies and a JAR with only the application code so that I can upload them as separate “layers” and thus leverage layer caching. While my code changes frequently and is tiny, the dependencies change rarely and are much bigger. If I can add them as a separate Docker layer or AWS Lambda layer then this can be cached on the server and reused when I upload a new version - saving time, bandwidth, and money.

When I was deciding what new language to learn, I could have picked the quite familiar Scala but chose instead Clojure - not despite of its lack of object-orientation, its immutable data structures, its too many parentheses on a single line - but because of it. (And because of Paul Graham’s Beating the Averages.) Why?!

The big struggle when entering a new code base is distinguishing the essential business logic from the incidental aspect of how it is implemented. Both are just code - but which parts must be there and the way they are and which can be changed? If you don’t know then you fear to change anything. And that is exactly what happened when we took over the application MB. What to do? How to save the code from becoming an incomprehensible mess of legacy code, and dying?! We found an answer:  clj-concordion. (Read: Functional Core, Imperative Shell and Specification by Example.) 

Once upon time, an API returned 0.5 GB JSON and our server crashed. We cried, we re-implemented it with streaming parser and a disk-backed Map and it worked again - but was slow. We measured and tweaked and finally got to a pretty good time and decent memory and CPU consumption. Come to learn about our journey, streaming JSON parsing, and MapDB!

Having the computer generate tests for you, trying tens of devious inputs you would never have thought of, is awesome. There is however far less experience with and knowledge of generative (a.k.a property-based) testing so I would like to share what we have learned and what worked for us when testing an important data transformation pipeline. We mostly leveraged our existing Clojure Spec data specifications to generate the tests, while regularly reaching down to clojure.test.check to create custom generators and for low-level control.

Common mistakes made by Clojure beginners and style recommendations.

That’s how I learned (again) to never use doubles for monetary amounts. I would never have thought of it myself (though I should have), hadn’t we used generative testing to produce random test data (popular troublemakers included) - data I wouldn’t have thought of, such as 0.01 + 0.05 that cannot be represented precisely with a double. Now that we switched safely over to BigDecimals and angry customers and law suits are off the table, you might wonder what is this generative testing about and how does it work.

Instead of hardcoding inputs and expected outputs, as in the traditional “example-based” testing, inputs are randomly generated and outputs are checked against rules (“properties”) that you define, such as “the output of (sort list) should have the same elements and length as list; also, each element is >= its predecessor”. And you can generate as many inputs as you want, for instance 100 is a popular choice.