Holy Dev Newsletter Aug 2023

Welcome to the Holy Dev newsletter, which brings you gems I found on the web, updates from my blog, and a few scattered thoughts. You can get the next one into your mailbox if you subscribe.

What is happening

Rama, intro

The biggest news of August is of course Nathan Marz’s Red Planet Labs coming out of stealth mode and announcing their 100x productivity backend building platform, Rama. More on that below.

My ERP: from Asami to Datomic

Most of my free time this month I have spent on rewriting my tiny ERP from Asami to Datomic (in particular, the embedded Datomic Local), which was surprisingly little work. I always wanted to learn and use Datomic, and since it is finally free and redistributable, I went for it. Two features of Asami were a bad match for my use case, namely its schemalessness and the fact that all attributes are multi-valued by default. Datomic has been created primarily as a transactional system for business applications, which aligns perfectly with my needs, and it has battle-tested code and a solid feature set. The only trap that caught me in the migration is that in Asami, enum values were just keywords, while in Datomic they are entities, and thus I had to add an extra join to get from id to value.

Adopting Datomic and its Entity Specs and transaction functions made it possible to simplify my code. I use Specs to ensure that all required attributes are provided, and a tx fn to make sure that you cannot delete anything that other entities refer to. You need to explicitly include both in transactions, so I have extended the fulcro-rad-datomic plugin with a datomic-common/append-to-raw-txn function (fn [env txt]) → env, which makes it possible to add anything to the transactions produced by the save/delete middlewares. I also leverage :db/retractEntity, while in Asami I had to implement this myself (though as a fellow developer Francis Avila observes, “Retract entity is not magic—it just queries for datoms and retracts them. You don’t have to use it, and in fact you often grow out of its simplistic assumptions. At shortcut we don’t use it at all because there are some attributes (with audit info) that we never want to retract”).

Aside of the migration, I have also finally implemented one of the two remaining "productionalization" features, namely shipping logs to a persistent external storage, as JSON. I took the easy path (though, once again, "easy" turned out to be multiple hours of work) and leveraged Fly.io’s support for observability via NATS. I deployed the fly-log-shipper app they created, which runs the Vector "tool for building observability pipelines" to hook into NATS to start recieving all logs, processes them by the transformers I defined (to filter some out, parse to JSON, throttle so I never will have too many), and finally ships them to my Cloudflare R2 bucket via its S3-compatibility API. The bucket automatically deletes files after a few weeks. I’d like to look into adding Loki to make the logs searchable and space-optimalized, if I ever get to it. I could perhaps combine fly-log-shipper with fly-log-local, to store Loki-processed logs locally, to a persistent volume, and only back them up to R2 once a day or so. (Currently, I write them up to every 1/2h, risking data loss during that period and higher by-operation costs.)

Collators FTW!

Finally, I want to share a lesson I have learned - locale-aware sorting of texts is really important to users, even for English, and it is easy to do. The default sort in both JS and Clojure produces A..Z .. a..z. If the text contains any accented characters, such the Czech "č", they will be at the very end, instead of where they belong. However, when you use a Collator for the text sort - even if it is a Collator for a different language - you will get much more appropriate results:

// BAD:
// => [ "A", "Z", "a", "c", "d", "z", "č" ]
// GOOD:
["A","Z","a","č","c","d","z"].sort(new Intl.Collator('en').compare)
//=> [ "a", "A", "c", "č", "d", "z", "Z" ]

You can do this in JS, in Clojure ((sort (Collator/getInstance (Locale/forLanguageTag "en")) ["č" "d" "c"])), and in PostgreSQL (SELECT …​ ORDER BY name COLLATE "cs_CZ.utf8").

Rama, details

I have https://blog.jakubholy.net/2023/05-newsletter/#gems_from_the_world_wide_web[written about RPL’s efforts] to make backend development much more productive before. Now they have finally delivered. And as a proof, they have implemented a Twitter "clone" that scales linearly to at least double Twitter transaction rate, in just 9 person-months. (While other similar projects would take many tens of person-years.) It is of course best to read their thorough announcement post, but I will try to summarize _my limited understanding of what is Rama, the platform that made this possible?

From one point of view, Rama is an immutable, distributed, partitioned database with integrated data processing, where this processing is co-located with data, and happens in a streaming or batched manner. You build distributed ETL pipelines, essentially by composing rather high-level data flows using the Rama API, to produce distributed, partitioned derived data - like materialized views, but with arbitrary structure and indexing (lists of IDs, maps of maps of stuff, …​). And you build similarly "query topologies" to run distributed queries that combine any of this data to return whatever the client needs. Rama is reactive, i.e. updates flow through the system automatically. Moreover, it provides a comprehensive solution for deploying, updating, scaling, and monitoring your code. It has high availability thanks to automatic data replication, and is very scalable, if you partition your data well, with respect to the use cases. I don’t event want to imagine how much complexity and how many diverse pieces of software would be needed to implement a robust, scalable, distributed application myself. With Rama, lot of the hard work is gone, and I can focus on the business logic.

I am still at the very beginning of digging into Rama, but I will keep you posted.

Gems from the world wide web

There is little here today, because my Tumblr account has been temporarily disabled. And also because I was first in vacation mode and then in catch-up-after-summer mode, so there wasn’t much reading time.

I found The Repl’s episode Executable textbooks with Sam Ritchie about sicmutils - Computer Algebra, Physics and Differential Geometry in Clojure - quite interesting. Sicmutils is a tool "for [interactive] math and physics investigations" and a key part is symbolic computation - instead of working on numbers, it works on equations and symbols, with the ability to simplify the expressions, and to produce efficient code. In other words, it provides a high-level language to describe math and physics problems, and the compiler makes sure the result is executable and fast (and possibly faster than far more complex, manually written code). This seems to resonate with my First craft a language then build your software.

👓 Eraser – The whiteboard for engineering teams [SaaS, documentation]
A SaaS tool for drawing diagrams & diagrams as text, with a free draw canvas and more. Recommended by a colleague. Paid but with a reasonable free plan. You can also connet it to a GitHub repo to pull .md docs from there & edit diagrams in Eraser (they will be stored in a subdirectory, and replaced with PNG in the .md). Use it as a whiteboard, for documentation, wireframes, and much more.


Thank you for reading!

Tags: newsletter

Copyright © 2023 Jakub Holý
Powered by Cryogen
Theme by KingMob