Holy Dev Newsletter Nov 2023

Welcome to the Holy Dev newsletter, which brings you gems I found on the web, updates from my blog, and a few scattered thoughts. You can get the next one into your mailbox if you subscribe.

What is happening

Exciting times! The gods of time have finally allowed me and Thomas Clark to get in sync and start reviving Wolframite, a bridge between Clojure and the awesome Wolfram Language and its built-in knowledge base. I am very excited about Wolfram because of the high level where it operates, its integration with scientific computing, and the integrated knowledge base. It truly makes everything "computable". With Wolframite, you can (soon) call to the free-for-dev Wolfram Engine or a full installation of Mathematica to do some work for you. If we are lucky then we will have Wolframite 1.0 by the end of the year. (Disclaimer: Wolframite is a refactoring of an older Clojuratica, and 99% of the work has been done by others. Me and Thomas just want to bring it to a conclusion and get it into your hands.)

Aside of Wolfram, I have also spent too much ๐Ÿ˜… time with improving my blog, namely updating cryogen-asciidoc to AsciidoctorJ v3, and consequently updating my builder to java > 8 (Netlify still only offers v8). I have also written a short note about The Four Heads of Complexity, based on a post by Kent Beck. (You should absolutely check out his newsletter Software Design. Click on the "No thanks >" link at the bottom of the page to get a preview and some free posts.)

Finally, I have spent many hours these past two weeks on implementing pagination. Youโ€™d think it is a solved problem, but it turns out to be pretty complicated, when the entries come from 6 different tables. I hope to have time to write some property-based tests for this beast, and work on simplification in future iterations.

Gems from the world wide web

๐Ÿ‘“ Photopea | Online Photo Editor [tool]
A free online image editor. I haven't tried it, but it may be handy. There is also reportedly vector editor Vectorpea...

๐Ÿ‘“ Google Rewriting Android's Binder In Rust With Promising Results - Phoronix [rust, experience]
Google is rewriting Binder, the inter-process communication module in the Android kernel, from C to Rust. The key motivation is the complexity of the existing code. Moving ensuring ownership from developers to the compiler makes it simpler and more secure. Another benefit is that error handling is checked by the compiler, and simpler.

๐Ÿ‘“ Sequoia-PGP [rust, security, tool, library]
Sequoia-PGP is a new OpenPGP implementation, focused on security and correctness. And while it comes with the sq command-line tool, it was primarily created as a library. You can read about SecureDrop's switch to it.

๐Ÿ‘“ ๐Ÿ”จ [git]: Write better commits with Gitmoji
Emojis in commit messages FTW!

๐Ÿ‘“ Should I Rust or should I go? [rust, opinion, criticism] - "Is Rust a great fit for this project?" I get this question quite frequently so I think it's time to write down my thoughts if i
A great and uncompromising analysis of the downsides of Rust (when compared to Go), and of when it still makes sense. Go is almost as easy to use as Python, and (reportedly) faster and more reliable than Java.
Some of the problems with Rust are how quickly it is changing (31 releases, while Go had 8 and Python 3), causing a lot of churn in the ecosystem. Moreover, it is becoming more and more complex. Additionally, it can still be consider just a "beta" in the sense that async support is not fully implemented yet (such as async traits). (Though async is really hard, and may well be the next Billion Dollar Mistake.) Moreover, the standard library is very anemic, and you have to pick from incompatible libraries & approaches.
Rust still makes sense for WebAssembly, replacing C/C++/Assembly, and when you need insane performance.

๐Ÿ‘“ oliyh/oxbow: A Server Sent Events (SSE) client for Clojurescript based on js/fetch [ClojureScript, library, networking] - A Server Sent Events (SSE) client for Clojurescript based on js/fetch - GitHub - oliyh/oxbow: A Server Sent Events (SSE) client for Clojures
Oxbow is a a cljs Server Sent Events client based on js/fetch, allowing you to use the full feature set of fetch and not be constrained by the old js/EventSource (which cannot e.g. send headers). js/fetch does not natively process streams, such as SSE. Oxbow handles this for you.

๐Ÿ‘“ igrishaev/pg: PostgreSQL driver in pure Clojure [postgresql, clojure, library, database]
PostgreSQL driver in pure Clojure, with an excellent support for (multi-dimensional) arrays (automatically converted from/to Clojure vectors) and json. Easy integration with HoneySQL.
It seems to be still very much in progres, and promises a more ergonomic interface than JDBC and first-class access to features of Pg not so well / at all supported by JDBC. F.ex. it offers "various ways to COPY data into or from the database. It's much more flexible than the official JDBC Postgres driver's standard CopyManager class." It also supports both the Simple and Extended Query / wire protocol, with their separate pros and cons.

๐Ÿ‘“ taoensso/tempel: Data security framework for Clojure [clojure, library, security]
Tempel is a lightweight encryption framework that wraps the JVM's native crypto facilities to provide a high-level Clojure API that is: idiomatic, simple, and easy-to-use even for non-experts. It incorporates best practices and reasonable defaults to help simplify many common data security needs such as log ins, encryption, signing, etc. Future-proof data formats with auto-updated algorithms and work factors over time

๐Ÿ‘“ fractl-io/fractl: Generative AI-powered Programming Language [research, programming languages]
Fractl is built on top of Clojure and exposes a dataflow-driven reactive programming model, with a functional core. (Existing Clojure/Java libraries are compatible with Fractl.) Its key aspects are ย higher level of abstraction and data-oriented syntaxย - which are intended to make it easier to create no/low code solution by using their SaaS build-code-by-connecting-boxes studio and being easier for AI code generation tools like Copilot. Fractl.io's pitch is "SaaS Apps 100x Faster". The SaaS product is there to earn them living, but the Fractal language itself and key tooling for it are open source.
Fractal's high-level abstractions:

  1. Graph-based Hierarchical Data Model - compose the high-level data model of an application as a hierarchical graph of business entities with relationships. Such entities and relationships are first-class constructs in Fractl.
  2. Zero-trust Programming - tightly control operations on business entities through declarative access-control (RBAC) encoded directly in the model itself.
  3. Declarative Dataflow - express business logic as purely-declarative patterns of data.
  4. Resolvers - use a simple, but powerful mechanism to interface with external systems. A resolver is associated with an entity and consists of a map of functions implementing CRUD + query
  5. Interceptors - extend the fractl runtime with custom capabilities.
  6. Entity-graph-Database Mapping - take advantage of an abstract persistence layer for fully-automated storage of entity instances. Essentially an ORM ๐Ÿ˜ฑ but hopefully better.
Check out this taste of Fractl code.
Quote: "Our early users tell us that theย  combination of graph data model and role+ownership-based (similar to row-level, but much more powerful) access control help with making their app complete and very secure."

Websockets and GraphQL are coming...

๐Ÿ‘“ zeitstein/brimm: Notion-inspired versatile knowledge management backed by a graph you own, full stack and open source. [tool, documentation, knowledge management]
A full-stack open-source versatile graph knowledge management webapp, inspired by Notion and infinite outliners like Worflowy/Roam/Logseq. Coming in Q1 2024.ย 
Built with XTDB, shadow-grove, SCI & more.
The primary motivation was to combine Notionโ€™s database and document editing experience with the structural flexibility of infinite outliners (apps like Workflowy or Logseq). The broader vision for Brimm is to provide you with a capable personal/team graph database with a default UI.
Brimm lets you model data as blocks of many shapes (text, tasks, bookmarks, personal info, events, books, music albums, contact info, etc.), categorise and organise it, make connections, and search across it. You can "mirror" blocks elsewhere, create live queries rendered e.g. in a table, .. .
The core data model is "graph of nodes [blocks] with properties" [such as a date, tags picker, ...]. You can mould your data and workflows using Clojure through SCI - derive new facts, write a script to import external facts, ... .

๐Ÿ‘“ Data-Driven Development is a Lie [clojure, design, criticism]
A sharp article with a good point: encoding complex logic in ad-hoc data DSL with a home-made, ad-hoc interpreter is a recipe for defects, inflexibility, and expensive maintenance. Using simple, rather dull functions is far .... simpler. Though I do not think the author covers all of what "data-driven" (or should we rather say data-oritented?) means. Look at the benefits of data-driven Reitit vs. macro-base Compojure.

๐Ÿ‘“ WebAssembly Garbage Collection (WasmGC) now enabled by default in Chrome - Chrome for Developers [wasm]
WebAssembly Garbage Collection is crucial for making higher-level languages practical with WASM, making it unnecessary to bundle their own, typically huge, GC. The standard is still under development, but obviously far along, as it is already enabled in both Chrome and Firefox. Kotlin/Wasm has been integrating it for some 1/2 year and can already run a lot of stuff.

๐Ÿ‘“ How to (and how not to) design REST APIs [design, networking, experience]
Some great advice here. A summary follows, but read the original post for its rich experience and reasoning behind the recommendations. I especially like and/or have been surprised by, but agree with, nr. 4, 6, 7, 8. 11 is wonderful.

  1. DO use plural nouns for collections - /products/
  2. DON'T add unnecessary path segments - e.g. Etsy's /shops/{shop_id}/listings/{listing_id} should have been /listings/{listing_id} b/c a listing_id is globally unique, and what if a listing switches shop?
  3. DON'T add .json or other extensions to the url, use established headers!
  4. DON'T return arrays as top level responses, always wrap them in an object - that way you can evolve it and add other keys in the future (e.g. for pagination)
  5. DON'T return map structures for collections, e.g. {key1: {key: "key1",..}, :key2: ...} - use an array of maps (see also #4)
  6. DO use strings for all identifiers, even if internally they are numbers - strings provide more flexibility, as things change over time (you can num -> str, not always the opposite)
  7. DO prefix your identifiers with a type marker - much more ergonomic, better for troubleshooting
  8. DON'T use 404 to indicate "not found" - non-intuitive (b/c that's what 404 is for!) but logical - b/c 404 could be returned by any intermediate proxy and thus could mean many different things, such as misconfigured client/proxy/load balancer hitting the wrong url. (The author suggests to use 410 gone.)
  9. BE consistent (Shopify has 6 subtly different schemas for an Address in their REST.)
  10. DO use a structured error format, if multiple layers of services are involved. Ex.: {message, type, types: ["Unauthorized", "Security",even a higher-level category for this error], cause: {}}
  11. DO provide idempotence mechanisms - ideally via an "idempotency key" or "client reference ID" - a client-provided value that you store temporarily or permanently and ensure its uniqness (by ignoring subsequent POSTs with the same one). Or let the client pick the id. On resubmit, you can either return 409 Conflict (which requires a smarter client that can handle it) or 201 with the original response (=> client can be dumb, but server becomes significantly more complex)
  12. DO use ISO8601 strings for timestamps, not numbers like milliseconds-since-epoch, because human readability matters. And all timestamps should be in UTC. Actually, use ISO8601 for all date/time-related values (durations, intervals, ...). But DON'T trust your language/platform defaults - verify them (e.g. to ensure it use UTC and not the local timezone).

๐Ÿ‘“ EuroRust 2023 [rust, learning]
Talks from EuroRust 2023. Especially Rust at Sentry: 7 Years Later and Lessons Learned from Teaching Rust seem interesting, though I haven't seen either.

๐Ÿ‘“ JUXT Blog: JUXTCast: Rama and its Clojure API โ€” with special guest Nathan Marz [research, rama, design]
A great way to learn about Rama, the programming platform for building backends with very diverse compute and storage needs end-to-end, and including deployment and monitoring. It is massively scalable and fault tolerant. Programmable datastore on steroids - mold it to fit your domain, rather than vice versa.
It all started by examining scalable backends from first principles: what backend does is answer questions => can be modelled as a function of all the data it has ever seen. To make it practical, introduce the concept of an index and thus make it a function of data + indices.
Normally, you use a set of tools to build a backend: perhaps a relational DB + ElasticSearch + perhaps other DBs for the storage data, and perhaps Storm + an API server + custom workers to compute the indices, and possibly something yet different for the querying part.
Our field is all about abstraction, automation, and reuse, so you'd expect that each backend consists primarily of what is unique to it - yet the opposite is true. Rama wants to change that.
Every app needs multiple types of indices, and it needs them fit its particular needs.
Rama's depot an PState storage is based on Fb's RocksDB.
Rama introduces data-flow programming paradigm. A key building block is a "fragment," which is a generalization of a function - it can emit 0, 1, or many times, possibly to multiple streams. (A fn emits exactly once, to a single stream, and is done after that.)


Thank you for reading, and Merry Christmas!

Tags: newsletter

Copyright © 2024 Jakub Holรฝ
Powered by Cryogen
Theme by KingMob