Holy Dev Newsletter November 2024

👓 Rama is a testament to the power of Clojure [rama, clojure, experience]
A praise of Clojure, especially the fundamental ideas of immutability and data-oriented programming and its flexibility for defining abstractions (including macros and the ability to do in libraries that would require language support elsewhere): "Besides these being essential to maintaining simplicity in Rama’s implementation, Rama also embraces these principles in its approach to distributed programming and indexing." "Clojure’s principles are just sound ideas that really do make a huge impact on simplifying software development."
Rama contains a new programming language implementing a new programming paradigm (functions, or rather "fragments", form a reactive tree, and can emit 0, 1, or many times; built-in distributed programming). All this is written primarily in Clojure macros, and then Rama itself.
"These philosophies [data-oriented programming, immutability] have had a major impact on Rama’s development, helping a tremendous amount in managing complexity within Rama’s implementation."
If you are new to Rama, then quote is rather interesting:

👓 Migrating terabytes of data instantly (can your ALTER TABLE do this?) [rama, database]
Nathan points out that non-trivial schema migrations in most databases are at best complicated, at worst impossible, and introduces Rama's new, transparent migration support. You write the migration with full power of Rama, and it seems instantaneous (while, under the hood, Rama automatically migrates data on read, while it is running the migration on the persisted data in the background).
What is the problem with e.g. SQL migrations? You need to use SQL, possibly duplicating your business logic, you risk locking the whole table, you need to sync code deployment and make it work with both previous and new schema, while the migration is running.

👓 electric-sql/pglite: Lightweight WASM Postgres with real-time, reactive bindings. [wasm, database]
PGlite is a WASM Postgres build packaged into a TypeScript client library that enables you to run Postgres in the browser, Node.js, Bun and Deno, with no need to install any other dependencies. It is only 3mb gzipped and has support for many Postgres extensions...

👓 xadecimal/expose-api: A Clojure library to automatically generate public API namespaces by wrapping and exposing... [clojure, library]
Library for library builders - create a ns with your public API off other namespaces. Contrary to Potemkin, this is done at build time and thus static code analysis friendly (think Cursive, clj-kondo).

👓 anteoas/broch: A library for handling numbers with units. [clojure, library]
Brochs combines numbers with units into a quantity, and provide tools for working with these.

👓 Cloogle [clojure, tool, learning]
Semantic search across half a million Clojure functions. What can it do? A lot!
Cloogle can find specific implementations:

• https://cloogle.phronemophobic.com/doc-search.html?q=calculate+sha256+hash
• https://cloogle.phronemophobic.com/doc-search.html?q=create+and+return+a+temporary+file

• https://cloogle.phronemophobic.com/doc-search.html?q=error+handler+with+exponential+backoff
• https://cloogle.phronemophobic.com/doc-search.html?q=cancel+tasks+or+processes

👓 jeans11/demo-rama-electric: Demo of web application with Rama and Electric [learning, rama, clojure]
As it says on the tin: a demo of using Rama with a couple of modules (such as Session) for the backend together with Electric Clojure, handling the BE and the UI. The readme provides a pretty good overview. ❤️

👓 oakmac/standard-clojure-style-js: Standard Clojure Style in JavaScript [clojure, tool]
Clojurists Together sponsored a new, minimally configurable and highly portable formatter for Clojure, inspired by Tonsky's proposed simple formatting rules. Will be interesting to see where this goes... (Written initially in JS, so that it is easy to include in VS Code, make a CLI, put in a werb library, ... .)

👓 coder/wush simple & fast file transfer [tool, cli]

👓 igrishaev/deed: Fast, flexible, 0-deps (de)serialization library for Clojure [clojure, library] - Fast, flexible, 0-deps (de)serialization library for Clojure - igrishaev/deed
An interesting, if too fresh, alternative to Nippy for data serialization. Written in pure Java, claims to be 20-30% faster than Nippy, and offers a reportedly simpler API. Encryption and compression is not baked in as in Nippy, though wrapping with a GZIP*Stream is well supported. Serializes to an arbitrary stream, and knows how to serialize a number of types, including streams. Notice that Nippy by default serializes to a byte array, thus possibly doubling or tripling memory use. Deed also supports lazy reading and deserialization.

👓 Deploy web apps anywhere [tool, devops]
Interesting looking tool for when you want to run your services on your own hardware, but with the ease of use and features that cloud deploys offer, such as zero-downtime deploys and rolling restarts. From vanilla Linux to deploy in minutes, reportedly. Leverages Docker. By 37signals.

👓 Injee - The no configuration instant database for frontend developers [webdev, tool]
A non-persistent, REST & JSON "database" making it possible for frontend developers to create an "instant API," while waiting for a proper one to be created. POST some entities, get them back, search and sort... . Pretty neat idea! And written in Clojure...

👓 CEL  |  Common Expression Language [library]
I was looking for a library that would allow users to write expressions that transform data (primarily computing derived values), that would be simple and safe. CEL looks interesting - it is fast, extensible, allows subsetting, safe. It can be used also e.g. for encoding security/auth policies & checks, list filters in API calls, ... .

👓 On Seniority and Acceptance - Metosin [people, management]
Moving to more senior positions, the author has learned: 1. You see more problems than you can fix, 2. You can't expect [or make] a team to produce better quality than it knows how to. Actually, as you mature, you see more problems. Solution to #1 is prioritization (and saying no) and involving other people. Solution to #2 isn't quality control, but improving the team, giving the right mixture of support and freedom. And it requires that go of control. (Accept that you can't control the team's quality, and you can't solve all the problems.) Grow from intuitions into understanding / analysis. Make trade-offs. Multiple good points and food for thought here!

👓 How to max throughput when pulling data from a third party service [performance, api]
A good list of things to think of when trying to maximize the throughput of your service, depending on 3rd party services - batching requests and DB writes, rate limiting, retry with backoff, not starving other users, ... .

👓 Zag - Rapidly build UI components without sweating over the logic. - Zag [webdev, library, ui]
"A collection of framework-agnostic UI component patterns like accordion, menu, and dialog that can be used to build design systems for React, Vue, Solid.js and Svelte." But the really interesting part is that their logic is all powered by statecharts (like fine state machines, but more powerful). (In Fulcro, UI State Machines are used a lot for these purposes, and are likely to be eventually replaced by fulcrologic/statecharts.) A founding realization was that we need a better way to model component logic.

👓 Clojure's deadly sin [opinion, clojure, performance] - This article is about laziness in Clojure. It is intended to be a comprehensive and objective (however possible) critique of lazy sequences
"[..] intended to be a comprehensive and objective (however possible) critique of lazy sequences as a feature.
Pros: Avoiding unnecessary work (when you only (take 10 ...)), infinite sequences such as (range), processing data without having to load all of it into memory.
Cons: Aside of the performance overhead (in CPU and memory, due to creation of intermediary sequences), it is primarily the fact that lazy sequences don't play well with side effects. Namely, they will be triggered far from where they are in the code. F.ex. throwing exceptions (here, try-catch doesn't help when returning a lazy seq from it...), dynamic bindings (you are out of their scope when you come to actually executing the relevant code, releasing resources (e.g. if you try to return (with-open ... (line-seq ...))). Also, lazy seq chunking makes it hard to process a single element at a time. (Notice that some fns internally chunk the sequence they work on.)
To support all types of sequences including lazy, ISeq is not very optimal. From waling a 10k collection: "We see that it [loop with, effectively, first and rest] takes us 240 microseconds to merely iterate over that vector, and 400KB worth of objects gets allocated along the way. The second snippet uses doseq, which contains multiple chunking optimizations. Iteration with doseq is 6 times faster than with loop, producing 20 times less garbage on the heap. Finally, the reduce-based run! offers the same speed as doseq in this example while not allocating anything as it runs."
=> use mapv and friends, leverage transducers and possibyl eduction, remember doall...

👓 htmz - a low power tool for html [webdev, framework]
This is really cool and a very, very minimalist alternative to htmx. In its own words: "htmz is a minimalist HTML microframework for creating interactive and modular web user interfaces with the familiar simplicity of plain HTML. In a nutshell, htmz lets you swap page fragments on request using vanilla HTML.." 166B, no deps and JS bundles, plain HTML. Similar to and inspired by htmx, but much simpler. (Via Slipset.)
How does it work? You create a link such as to load the response HTML into the element with id mu-element. How? "htmz is an iframe named 'htmz'. You invoke htmz by loading a URL into the iframe via target=htmz. By using an iframe, we lean on the browser’s native capability to fetch the URL and parse the HTML. After loading the HTML resource, we take the resulting DOM via an onload handler." The latter, via document.querySelector(frame.contentWindow.location.hash || null)?.replaceWith(...frame.contentDocument.body.childNodes).

👓 Behavioral Programming in Clojure [clojure, research]
A short introduction into behavioral programming, with graphics and examples. As the original B.P. paper explains, B.P. is "A novel paradigm for programming reactive systems centered on naturally specified modular behavior." I.e. you can code each desired "behavior" independently of all others (= modularity, incremental specification of behavior), and they combine at runtime. A behavior is based around (sequences of) events and specifying what can, must, or may not happen following certain sequences of events. In particular, you can request a particular event to happen, wait on one, or block some. You can visualize a B.P program using live sequence charts. Reportedly, "Behavioral programming can be used to build very sophisticated behaviors from very simple building blocks. [...] Behavioral programs lend themselves to formal verification and to visualization. Behavioral programming has a sound theoretical foundation." Invented by David Harel, who also invented statecharts (not the Clojure lib of the same name :).)
IMHO it is nice and powerful to be able to specify behaviors independently (after all, it is how agents and human societies work) but it will surely be hard to understand the behavior of such system. Though visualization and formal methods may help.

👓 DORA | Capabilities: Trunk-based Development [software development, process, productivity]
Trunk-based development (merge to mainline at least daily) vs. feature branches (multiple days or longer). (Some) research reportedly "shows that teams achieve higher levels of software delivery and operational performance (delivery speed, stability, and availability) if they follow these practices [of trunk-based dev]." Ways to improve: develop in small batches, perform synchronous code review, implement comprehensive automated testing, have a fast build.

👓 Rama on Clojure’s terms, and the magic of continuation-passing style [rama]
At the heart of Rama is its dataflow language, a Clojure-based full-fledged programming language, which is based on continuation-passing style (CPS). This post explores how Rama works in comparison to equivalent Clojure code written in a CPS style. CPS through Rama greatly generalizes the basic concept of a function (into a fragment), which enables new ways of writing code in general, and is particularly liberating for writing parallel and asynchronous code. What makes Rama operations / fragments more general than functions is how they can emit multiple times, not emit at all, or emit asynchronously. The ability to emit asynchronously is what makes Rama so good for writing parallel and asynchronous code.
A good way to understand CPS and Rama's CPS better. More highlights:

• CPS and the ability to emit asynchronously unifies general purpose programming with distributed programming, by enabling parallel code to be expressed no differently than any other logic. Partitioners enable Rama code to precisely control not just what is executing, but where.
• Dataflow turns CPS into a full-fledged programming paradigm. Emitting zero times, multiple times, asynchronously, or to multiple output streams are major generalizations of functions that open up huge new avenues to explore in the craft of programming.

👓 My handy Clojure debugging tools [clojure, troubleshooting, tool]
A neat DIY way to capture local values, with corresponding expressions, with leveraging a user-level user.clj and profile and a custom reader tag.

👓 InstantDB: A Graph-Based Firebase [database, webdev, service]
An essay explaining the reasons for creating InstantDB, a "A Graph-Based Firebase" (database as a service with an API). InstantDB has relational queries, auth, and permissions and supports optimistic updates, "multiplayer" (user's changes are propagated to all others), and offline mode. These last three capabilities arguably make an app way better, and transform how you use it. The article discusses first the general architecture of such as webapp-and-DB system (a normalized in-browser store turned into a graph for each screen, and with mutations that apply both to the local store and the remote DB) and then dives into the integrated solution Instant delivers, with a "Local DB" that understands graph queries and handles normalization, caching, and syncing with the backend. Instant is based on a triple store (id, attribute, value), and has a datalog-like query language at its core, and a UI-friendly InstaQL on top of it.
Note: Firebase has reportedly two problems: 1) It is only a document store, without support for relations; 2) Its permission language is too weak and gets very complex very quickly.
Related: A Database in the Browser argues that the schleps we face as UI engineers are actually database problems in disguise (and InstantDB tries to be the solution for that).

Many think we need to resort to operational transforms to do stuff like this [resolving concurrent updates to data], but as figma showed, as long as we’re okay with having a single leader, and are fine with last-write-wins kind of semantics, we can drastically simplify this and just facts are enough. When time for even more serious resolution comes, you can open up the OT rabbit hole. -- Stepan Parunashvili in Database in the Browser, a Spec

👓 Database in the Browser, a Spec [webdev, database, opinion]
A predecessor to the InstantDB introduction post, an exploration of challenges in creating web applications and their backend for frontend, arguing that they would be solved by a database (with data authorization). Such as the fact that a lot of plumbing is necessary to get, cache, prepare data for views, over different channels (RPC/REST vs. websockets). It becomes even more complicated when you want to get updates of changes from the backend, undo/redo, ... . And on the backend you need to make all the endpoints...,

👓 Clojure 1.12 Field Guide [clojure]
A summary of what's new in Clojure 1.12 - clojure.repl.deps/add-lib(s), the new, more convenient clojure.java.process alternatve clojure.java.shell, array class syntax (^"[Ljava.lang.String;" -> String/1), Java methods as values (Class/.method, ../staticMethod, ../new) and type hints for them.

Choose the right programming language depends, among others, on the environment - do we expect changing requirements, is performance our top priority, or is it a high-stakes, high-risk environment?
Source: The talk How to transfer Clojure goodness to other languages by Elango Cheran and Timothy Pratley from Conj 2023

🎥 Electric Clojure is a brave, ground breaking experiment at radically simplifying the development of full-stack web applications....
Electric Clojure is a brave, ground breaking experiment at radically simplifying the development of full-stack web applications. With Electric, you don't need to manually transfer data to the client and deal with state management there. Instead, you write your full-stack app in cljc files, with e/client and e/server markers, and let the compiler figure out the split and network communication for you. I've been rather skeptical of the feasibility of this, but Dustin & Co. have put a lot of thought into it and it seems to actually work. (At least for rather interactive, backoffice and business apps without need for pixel perfection.)
You cannot just drop Electric into your code base and expect everything to work magically, there is some learning curve, even though the rules are sane and intuitive. You have to understand how Electric works and what piece of code runs where and when data crosses the network, and you need to learn Electric's superset of Clojure, with e/fn, e/for, an asynchronous let (since some parts of it may run on the server, others, concurrently, on the client), and get used to work with "differential reactive streams / signals" (= streams of diff info) instead of sequences of raw data. But compared to the complexity of React and manual state and network management, you are much better off anyway - and you get fine grained reactivity, where only changes are transferred and directly alter the relevant DOM (using the built-in dom library).
This talk introduces Electric v3 and improvements since v2, with fewer pitfalls and less need for macros. It is fascinating that all of this has been built by just four people. I am very much looking forward to actually trying this out and seeing for myself how well it works.

👓 Mike Acton’s Expectations of Professional Software Engineers [best practices, software development]
Mike has about 50 questions he expects all professional software engineers he works with to be able to answer. Many of them are widely applicable, and useful to think about. This mind-map provides a visual summary of them, useful as a quick reminder and review tool.

👓 phronmophobic/whisper.clj [clojure, library, media]
Clojure wrapper for whisper.cpp for recording audio and transcribing it to text.

👓 R2DBC [standard, database]
A non-blocking, reactive replacement for JDBC, based on t the Reactive Streams specification. There are drivers for all the popular RDBMS. V. 1.0 released in 4/2024. (Though the only pure-Java, non-Spring client has been archived 4 years ago... .)

👓 triple-threat/papers/A Spreadsheet Algebra for a Direct Data Manipulation Query Interface.pdf at master · dmwit/triple-threat [paper, data processing, programming languages]
A reference for constructing a data transformation DSL

Abstract— A spreadsheet-like “direct manipulation” interface is more intuitive for many non-technical database users compared to traditional alternatives, such as visual query builders. The construction of such a direct manipulation interface may appear straightforward, but there are some significant challenges. First, individual direct manipulation operations cannot be too complex, so expressive power has to be achieved through composing (long) sequences of small operations. Second, all intermediate results are visible to the user, so grouping and ordering are material after every small step. Third, users often find the need to modify previously specified queries. Since manipulations are specified one step at a time, there is no actual query expression to modify. Suitable means must be provided to address this need. Fourth, the order in which manipulations are performed by the user should not affect the results obtained, to avoid user confusion.
Shamelessly stolen from the Live Programming a Live Programming Environment: An Experience Report.

👓 This is very meta: using the moldable interactive coding tool Clerk to build a Clerk-based report builder inside Clerk... The UI... [research, clojure, ux]
This is very meta: using the moldable interactive coding tool Clerk to build a Clerk-based report builder inside Clerk... The UI is used to construct a data transformation pipeline by selecting one of offered, data-dependent choices at each new step, which reaches back to server-side and updates and re-renders the source code 🤯.
In any case, a good demonstration of how to develop and troubleshoot and compose custom Clerk viewers and its other capabilities. There is also a good lesson that finding the right DSL that allows building complex reports is very hard.
A lot of interesting references throughout and at the bottom of the paper.

Small Language Models

👓 graal/wasm at master · oracle/graal [wasm, tool]
GraalWasm is an open-source WebAssembly runtime compatible with the WebAssembly 1.0 specification. It runs WebAssembly programs in binary format and can be used to embed and leverage WebAssembly modules in Java applications. GraalWasm is in active development and implements a number of WebAssembly feature extensions. Feedback, bug reports, and contributions are welcome.

👓 PGlite [wasm, database]
Run a full Postgres database locally in WASM (with reactivity and live sync). PostgreSQL compiled to WASM and thus usable within the browser, with some popular extensions and extra reactivity/sync features, in 3MB compressed. Currently, only a single connection is allowed. In-memory or persistent via IndexedDB.

👓 bruno [tool, rest]
From ThoughtWorks Radar:

👓 Fast Open-Source OLAP DBMS - ClickHouse [devops, database, observability]
From ThoughtWorks radar 10/2024:

👓 Prossimo [rust, security]
A wonderful initiative to rewrite key building blocks of today's networks in memory-safe Rust - TLS, a reverse proxy, zlib for compression, ... .

👓 Spanking browser for performance: 100× speed improvement [webdev, performance]
Solving differential equations for efficient, physics-based animation in the browser. Now I wish I remembered more from my studies 😅. A custom rendering engine bypassing React, and doing direct DOM manipulation and drawing links into a 2D canvas, while having a layer for each node. We also learn that browsers may be dumb and waste 90% time trying to render off-screen layers. A lot of great engineering here!

🎥 This is wild. Fulcro purely on the server-side, with a custom render fn that either returns or prints hiccup - so that you can...
This is wild. Fulcro purely on the server-side, with a custom render fn that either returns or prints hiccup - so that you can play with all the concepts, focusing on them and data, instead of getting distracted by the UI. You could add a tap> and hook it into something like Portal to also display the UI... . Amazing! The repo is here: https://github.com/awkay/beginner-drafting