Holy Dev Newsletter March 2023
Welcome to the Holy Dev newsletter, which brings you gems I found on the web, updates from my blog, and a few scattered thoughts. You can get the next one into your mailbox if you subscribe.
Time is flying! This has been an intense period at work, which occupied most of my time, together with a little help I provided to two clients and coding the Fulcro ERP. When you maintain reporting capability for end users and have to upgrade the underlying query engine, which has extremely flexible and badly documented semantics and doesn’t care enough about backwards compatibility and stability, you are in for a treat. Hopefully, thanks to a few good ideas and lot of work, it is over with minimal impact on customers 😮💨. But it drives home how awesome the Clojure philosophy of stability and not breaking contracts is.
The ERP is hopefully drawing to a close (the final 20%? 😅). Right now I am refactoring from a monolithic order to an order with potentially multiple order lines, which is an interesting experience. I had to model this, move a bunch of attributes over, redesign my reports and forms, rewrite most queries, and implement cascading delete. Fortunately doable but not trivial. Some fallout from that is that fulcro-rad-asami has got docs improvements and the
::asami/no-batch? option to work around a Pathom 3 problem. (Troubleshooting it was quite an ordeal, and I am very thankful to Caleb MacDonald Black and Wilker for their help.)
I have published minimalist-fulcro-template-fly, which extends the existing minimalist-fulcro-template with support for building and deploying it to the Fly.io cloud, and with connectivity to PostgreSQL. I had picked this DB because I expected most developers to be familiar with relational databases, while something like Asami is more obscure, and less universally applicable.
Outside of Clojure, I have stretched my feable CSS skills to add dark mode to my blog, when I noticed how impractical it was to read it in the dark. And these last few days I have been playing with Clerk, to analyze REST communication around some issues at work.
In the wider world, Electric Clojure (see below) has been finally released in v1.0, and I am looking forward to trying it out. Perhaps I will try to rewrite the ERP with it (after it is done!), to see how it delivers on its promise of simplifying web development. Overall, I feel the pull of doing more work on the backend and less on the frontend, retreating from the world of SPAs, and looking for more simplicity. As you can see from some of the links posted below.
Lastly, I am going to attend Conj 🎉 and thus spent some time designing a Fulcro t-shirt to wear at the conference :-).
A tiny library by a colleague that enables you to capture the inputs and outputs of functions, e.g. in an atom - for example to record traces.
Ambient is a runtime for building high-performance multiplayer games and 3D applications, written in Rust, powered by WebGPU, and compatible with any language that compiles to/runs on WebAssembly. The goal is to make building multiplayer as straightforward as building singleplayer.
Ambient is built with data-oriented design in mind from top to bottom. All data is stored in, and interacted through, an entity component system backed by a centralized entity database on the server. This database is automatically replicated to every client, and each client has the ability to augment and extend entities with local state. The use of an ECS makes it easy to visualize the state of your application and provides excellent performance and scalability.
I am not into gaming but it sounds cool and worth studying.
I am looking for an alternative to a regular mouse. I have an inflammation in my right writst, possibly due to the mouse. I have tried a vertical mouse (#5 in the list) but find it impractical to have to lift my hand from the keyboard, and also not terribly precise or efficient, compared to a trackpad (#9). Perhaps #1, Roller Bar Mouse, sitting at the base of the keyboard, could be interesting. Or an ergonomic keyboard with a trackball (though their #1 doesn't look ergonomic to me at all; #5, Adesso WKB-3150UB, or #6 Perixx PERIBOARD-506, seem much more ergonomic to me).
Turn code in rich comments (in the form of `statement ;=>
A data viewer similar to Portal, but much simpler, with no dependencies, and only displaying the last tap-ed value as (nested, expandable) table. Uses a browser for the rendering. Leverages datafy.
"However, at a new startup, one of the core problems is to figure out if you are building something useful. A quick MVP can be invaluable in determining if you are on the right path or lost in the woods. All that time spent making readable, performant code might be wasted."
"Once you have that clarity, and you’ve shifted to “It’s time to scale” mode or “Perf actually is really important here” mode or even “We have a solid direction/roadmap and we don’t want too much tech debt” mode, Rust is a fantastic choice."
I haven't tried this but it sounds interesting. Use GPG keys for SSH (run gpg-agent instead of ssh-agent, config it to enable-ssh-support, even preconfigure what keys to ssh add automatically, tell the system to use it by pointing SSH_AUTH_SOCK to it).
Build a parser to parse a text into a data structure. The grammar is defined with function composition rather than the parser generator approach of using a DSL such as EBNF (as in Instaparse). Ex.: (def +space (p/+skip char/white?)).
See When to use a Parser Combinator? When to use a Parser Generator? => in general, parser combinators such as parsesso are for creating top-down (i.e. LL) parsers, with the ability to reuse common code (this lib). Parser Generators typically generate a finite state automaton for a bottom-up (LR) parser. Though nowadays there are also combinators for LR grammars and generators for LL ones (e.g. ANTLR). "Which one you should use, depends on how hard your grammar is, and how fast the parser needs to be." Especially if the grammar has lot of non-trivial ambiguities then it might be easier with the more flexible combinators approach.
Have a look at Chet Corcos's Introduction to Parsers article to understand more of the terms and background and learn more about parser combinators.
Robust support for the "reloaded" REPL dev workflow for deps-based projects. Integration for Stuart's Component (and possibly more in the future), so that the system is restarted if any component changes. Optional support for loading env variables. Can load dependent namespaces eagerly (default) or lazily. It makes it possible to declare a separate deps alias for each thing to start (the app, nrepl, file watcher, ...) and start any subset by combining the aliases on the command line.
I feel my current setup is working fine enough but perhaps I should give it a try...
A discussion of the newly released Electric Clojure by Hyperfiddle. What is Electric? "Electric Clojure, a reactive Clojure/Script dialect for web UI with compiler-managed client/server data sync." What does that mean? You write a single piece of code, e.g. a UI component function, and hint which parts need to run on the server vs. the client. "The Electric compiler performs deep graph analysis of your unified frontend/backend program to automatically determine the optimal network cut, and then compile it into separate client and server target programs that cooperate and anticipate each other's needs." Thus you program as if there was no hard client-server divide (no 2 separate files with REST calls in between) and Electric handles splitting it up and managing the communication between them on your behalf. It is groundbreaking and fascinating, with potential to significantly simplify web apps. Go read more about it.
Highlights from the discussion (many comments by the founder):
[..] we're seeing 10x LOC reduction (18k to 2k) in rebuilding Electric's sister project, Hyperfiddle (a spreadsheet like tool for robust UI development), as well as massive gains in performance.
NOTE: Til now focus was on correct program semantics, now started work on DX etc.
Our DOM module is only 300 LOC - it's bare metal DOM point writes + Electric (reactive language) + macros for JSX-y syntax. When the programming language itself is reactive, DOM rendering falls out for free.
Mechanically, Electric is comparable to Solid.js except the reactive engine (missionary) is general purpose, not coupled to DOM rendering, which is a special case of incremental view maintenance.
[..] over-abstracting is a primary risk and has been top of mind for us since project conception in ~2012. [..] Electric is an attempt to find exactly the right level of abstraction. The goal is to remove and flatten layers, not add them, thus decreasing abstraction weight in the end if we succeed. Maybe we fail, but first let me share some details about how we think about this:
- I've personally failed to build this project several times, Electric Clojure is something like the 7th attempt.
- strong composition model as a starting point, based on category theory generalization of "function" -> "async function" -> "reactive function" -> "stream function" -> "distributed function". [..] (This rigor is in response to the past failures.)
- Functional effect system (monad stuff) at the bottom, which provides strong semantics guarantees about glitch-free reactive propagation, process supervision (like Erlang) (transparent propagation of cancellation and failure), strong resource cleanup guarantees (DOM nodes can never be left hanging, event handlers can never fail to be detached and disposed). Already this results in tighter operational semantics than we have ever achieved with manual resource management (and, again, we tried, see past failures).
- Electric affords the programmer trapdoors to the underlying FRP/concurrency primitives. Electric is essentially a Clojure-to-FRP compiler, so if you code raw concurrency and effect management, that actually typechecks with what Electric generates, allowing seamless transition in and out of the abstraction.
- 3k LOC + 3k test LOC is the size of Electric today (includes a rewrite of the Clojure analyzer). Spring Framework is, let me go check, 59k just for spring-core/src/main/java, and there are like 20 other modules I excluded. Indeed it is not a fair comparison but certainly we have complexity budget to spare.
How LiveView in Phoenix, the unique Elixir web framework with first class realtime support, came to be. To me the most interesting part is that Phoenix, similarly to Electric Clojure and HTMX, moves state and logic to the backend (BE) and handles efficiently communicating changes to the (mostly?) static frontend (FE). A lot of webapp complexity IMO stems from synchronizing state on the FE and BE, so this is refreshing. Highlights:
FE and BE are connected by bidirectional Channels, which exploit Elixir/Erlang messages to talk with external clients. The FE opens a single WebSocket, which multiplexes between channels and processes. Because Elixir is preemptively scheduled, processes load-balance on both IO and CPU. You can block on one channel, transcode video on another, and the other channels stay snappy.
(An interesting feature is Phoenix Presence, a distributed group manager with metadata backed by CRDTs. Presence gets used in IOT apps to track devices, cars, and other things. If you don't have something like Presence, you probably don't think to build the kinds of features it enables.)
With React, any state change triggers a re-render, followed by an efficient patch of the browser DOM. LiveView is the same, except the server re-renders the template and holds the state. [LiveView] compute[s] a minimal diff to send to the client of only the dynamic data that has changed. [..] we send a payload better optimized than the best hand-written JSON API.
Experience report showing that Zig is "safer, faster, and easier to write" than corresponding unsafe Rust. The reason is that unsafe Rust is hard. A lot harder than C, this is because unsafe Rust has a lot of nuanced rules regarding its borrowing/ownership rules to make compiler optimizations, and you must not break those. Also, Zig had “more tools for working in a memory-unsafe environment, such as reporting memory leaks in tests”, making the overall process much better. Note that safe Rust is a completely different story.
A simple `idempotent(migration_name text,code text)` stored function that only `execute` the migration `code` if the migration hasn't been run yet.
Socket’s “safe npm” CLI tool transparently wraps the npm command and protects developers from malware, typosquats, install scripts, protestware, telemetry, and more—11 issues in all.
Translation, summarization, image recognition, image-to-text, and a whole lot more.
WebTransport is in my understanding essentially improved WebSockets, with support for newer protocols for better performance, and having multiplexing and choice between reliable streams or faster communication with the risk of reordering or loss of packets.
It works with HTTP/2 or 3, possibly on top of the TCP replacement QUICK. With that there is less overhead and it avoids the head-of-line blocking delays that WebSockets suffers from.
HTTP/3 over QUIC, it’s faster to establish connections, and congestion control feedback is available to help avoid problems
Really interesting, innovative language (in progress) with a unique focus: "Rune is a programming language developed to test ideas for improving security and efficiency." For example if you declare a type as secret, then computation on it will be done in constant time, preventing timing-based attacks. The other fascinating thing is that it is based on DataDraw, which is an ultra-fast in-memory, optionally persistent, database for high performance programs written in C. The database and DB access code is compiled and linked into your program, reportedly "making data manipulation even faster than if they were stored in native C data structures". It understands relationships, has cascading deletes, etc.
A fun and visually attractive insight into a big problem: burnout in tech. According to the report, about 40% tech workers are in high and 30% in medium risk of burnout. Being overworked and unable to recharge, they are exhausted an start feeling they achieve less than they should (self-inefficacy). As a defense, they become cynical, i.e. stop caring about the work and performance, and detach emotionally, becoming colder and harsher to their colleagues. Until they collapse or leave...
A nice snippet to add fn console.save to save a data into a file as json from the browser Console. (When it might be too large for the clipboard, which you can do with navigator.clipboard.writeText.)
Thank you for reading!