Holy Dev Newsletter Aug 2023

The biggest news of August is of course Nathan Marz’s Red Planet Labs coming out of stealth mode and announcing their 100x productivity backend building platform, Rama. More on that below.

Most of my free time this month I have spent on rewriting my tiny ERP from Asami to Datomic (in particular, the embedded Datomic Local), which was surprisingly little work. I always wanted to learn and use Datomic, and since it is finally free and redistributable, I went for it. Two features of Asami were a bad match for my use case, namely its schemalessness and the fact that all attributes are multi-valued by default. Datomic has been created primarily as a transactional system for business applications, which aligns perfectly with my needs, and it has battle-tested code and a solid feature set. The only trap that caught me in the migration is that in Asami, enum values were just keywords, while in Datomic they are entities, and thus I had to add an extra join to get from id to value.

Adopting Datomic and its Entity Specs and transaction functions made it possible to simplify my code. I use Specs to ensure that all required attributes are provided, and a tx fn to make sure that you cannot delete anything that other entities refer to. You need to explicitly include both in transactions, so I have extended the fulcro-rad-datomic plugin with a datomic-common/append-to-raw-txn function (fn [env txt]) → env, which makes it possible to add anything to the transactions produced by the save/delete middlewares. I also leverage :db/retractEntity, while in Asami I had to implement this myself (though as a fellow developer Francis Avila observes, “Retract entity is not magic—it just queries for datoms and retracts them. You don’t have to use it, and in fact you often grow out of its simplistic assumptions. At shortcut we don’t use it at all because there are some attributes (with audit info) that we never want to retract”).

Aside of the migration, I have also finally implemented one of the two remaining "productionalization" features, namely shipping logs to a persistent external storage, as JSON. I took the easy path (though, once again, "easy" turned out to be multiple hours of work) and leveraged Fly.io’s support for observability via NATS. I deployed the fly-log-shipper app they created, which runs the Vector "tool for building observability pipelines" to hook into NATS to start recieving all logs, processes them by the transformers I defined (to filter some out, parse to JSON, throttle so I never will have too many), and finally ships them to my Cloudflare R2 bucket via its S3-compatibility API. The bucket automatically deletes files after a few weeks. I’d like to look into adding Loki to make the logs searchable and space-optimalized, if I ever get to it. I could perhaps combine fly-log-shipper with fly-log-local, to store Loki-processed logs locally, to a persistent volume, and only back them up to R2 once a day or so. (Currently, I write them up to every 1/2h, risking data loss during that period and higher by-operation costs.)

Finally, I want to share a lesson I have learned - locale-aware sorting of texts is really important to users, even for English, and it is easy to do. The default sort in both JS and Clojure produces A..Z .. a..z. If the text contains any accented characters, such the Czech "č", they will be at the very end, instead of where they belong. However, when you use a Collator for the text sort - even if it is a Collator for a different language - you will get much more appropriate results:

You can do this in JS, in Clojure ((sort (Collator/getInstance (Locale/forLanguageTag "en")) ["č" "d" "c"])), and in PostgreSQL (SELECT …​ ORDER BY name COLLATE "cs_CZ.utf8").

I have written about RPL’s efforts to make backend development much more productive before. Now they have finally delivered. And as a proof, they have implemented a Twitter "clone" that scales linearly to at least double Twitter transaction rate, in just 9 person-months. (While other similar projects would take many tens of person-years.) It is of course best to read their thorough announcement post, but I will try to summarize _my limited understanding_ of what is Rama, the platform that made this possible?

From one point of view, Rama is an immutable, distributed, partitioned database with integrated data processing, where this processing is co-located with data, and happens in a streaming or batched manner. You build distributed ETL pipelines, essentially by composing rather high-level data flows using the Rama API, to produce distributed, partitioned derived data - like materialized views, but with arbitrary structure and indexing (lists of IDs, maps of maps of stuff, …​). And you build similarly "query topologies" to run distributed queries that combine any of this data to return whatever the client needs. Rama is reactive, i.e. updates flow through the system automatically. Moreover, it provides a comprehensive solution for deploying, updating, scaling, and monitoring your code. It has high availability thanks to automatic data replication, and is very scalable, if you partition your data well, with respect to the use cases. I don’t event want to imagine how much complexity and how many diverse pieces of software would be needed to implement a robust, scalable, distributed application myself. With Rama, lot of the hard work is gone, and I can focus on the business logic.

I am still at the very beginning of digging into Rama, but I will keep you posted.