Holy Dev Newsletter May 2023
Welcome to the Holy Dev newsletter, which brings you gems I found on the web, updates from my blog, and a few scattered thoughts. You can get the next one into your mailbox if you subscribe.
What is happening
I am still finishing the tiny Fulcro ERP. This led to implementing support for "cascading delete" so that when a reference from an owner to an owned entity - e.g. from an Order to an OrderLine - is removed then the whole owned entity is deleted. I have also had some questions about subform validation, which prompted Tony to enable automatic validation of "edges" (so that an Order may require to have some OrderLines) and to also validate attributes of nested forms instead of just the top form. (This previously required a custom form validator.)
I am also helping Tony and Aram Zadikian with a new, exciting project to produce regularly Fulcro content of various kind, aimed at both beginners and more experienced users. We plan to produce some practical, cookbook-like articles, more design-oriented ones (like why things work the way they do, how to “think” Fulcro, etc.), and “success stories” (e.g. porting a non-trivial full-stack SPA to front-end-only in a day). You can see the initial, very rough draft of the cookbook with a first example here.
Finally, I have been to an excellent in-person meetup about Datomic here in Oslo. We learned about the benefits of Datomic our friends have experienced over a decade of using it, explored how to use it in code, and heard about its architecture and how to operate it (according to the DBA, he had never so few production problems with a database as with Datomic.)
Gems from the world wide web
👓 Microsoft is rewriting core Windows libraries in Rust • The Register [rust, trends]
Hurray! Though no surprise - memory safety bugs were at the heart of about 70 percent of the CVE-listed security vulnerabilities patched by the Windows maker in its own products since 2006. Now, Windows graphics device interface (Win32 GDI) is being ported to Rust and so far has 36 kLoC, and DWriteCore (text analysis, layout, and rendering) now consists of about 152 kLoC Rust and some C++. And last September, Microsoft Azure CTO declared that new software projects should use Rust rather than C/C++.
👓 Apple, Google and Microsoft team up on passwordless logins | TechCrunch [security]
Passkeys and logging in to any account as you already do with your phone will aoon become ubiquitous. Good by, passwords!
👓 Why fully distributed is by far the best way to run a software team [way of working, people]
Highlihhts: Most conversations are recorded and thus discoverable later, and searchable. More aync <_> more flexibility. No commute, cheaper, ... .
Check out the section "How distributed at RPL works" - including asking "silly" questions to get to know each other and, most importantly, pair-programming one or two times a day for 45 minutes a session. Standups. A lot of other interesting ideas there.
👓 Tour of our 250k line Clojure codebase [clojure, experience]
250k LoC Clojure, 50:50 code and test. Topics: custom programming language, type/schema checking, monorepo, polymorphic data with Specter, using (extended) Component, with-redefs for testing, deterministic simulation, ... .
Type/schema checking (with predicates enabling checks far beyond types): clearly document data x don't overdo it -> during the construction of types (via a custom defrecord+ macro) and at a few strategic spots throughout the codebase where lots of different types flow. Only occasionally annotate fn args / return val - "We find instead that being consistent about how we name things is good enough for understanding the code." They have 600 types (remember, they implement among others a compiler) and extra 500 assertions (checking higher-level properties rather than simple types).
Monorepo with 4 modules - leveraging AOT to speed up REPL startup in dependent modules.
Polymorphic data with Specter - a compiler is heavy on types and deals with nested and recursive data stuctures, and needs polymorphism - the perfect use case for Specter. Protocols are not flexible enough and composable enough, contrary to Sepcter's polymorphic defprotocolpath, which enables nested manipulation, i.e. a path such as [some-path polymorphic-subpaht sub-path...] for setval and similar. (A protocol can give you a polymorphic accessor function, but you can't leverage that in the middle of a Specter path.) "Protocol paths [..] greatly reduce the number of manipulation helper functions that would be required otherwise."
Organizing complex subsystems with Component: Component enables them to start/stop subsystems in a defined order, and to inject mocks in tests. They added support for doing part of the initialization / tear down on a separate thread, where necessary.
Using with-redefs for testing: to mock dependencies, inject failures, and, interestingly, most often to redefine log-like no-op fns placed at strategic places of the codebase to get insights into what is happening, f.ex. once called durable-state-checkpointed => check that the expected events happened at expected times/order.
Deterministic simulation: write 100% reproducible distributed systems tests by running our whole system on a single thread and randomizing the order in which entities execute events (details). It heavily utilizes Component's support for dependency overrides (primarily of the executor-service-factory component used by anything that needs threads, and the system clock component) and redefs (e.g. of promise).
From the Conclusion: "[Clojure] enabled us to build powerful abstractions not possible in other languages, remove all ceremony whatsoever, and utilize powerful testing techniques." Also noteworthy: "Plus we’ve had multiple members on our team start with no Clojure or functional programming experience and they were able to get up to speed quickly."
RPL's new language and tool
RPL is creating a developer "tool reduces the cost of building large-scale end-to-end applications by multiple orders of magnitude," and its foundation is a new general-purpose language build withing Clojure with ± 60 macros (though they produce bytecode, not Clojure), with "semantics [..] substantially different than Clojure".
It has first-class continuations, which "makes it extremely good at async, parallel, and reactive programming. All of these are foundational to the large-scale distributed infrastructure we’re building."
RPL codebase consists of this new language and its compiler, abstractions for parallel programming, their implementation as a distributed cluster, a webui (=> see what's running on the cluster, the current status of operations like scaling, and telemetry showing what’s going on in users' applications), and an enhanced fork of Specter. I.e. the key parts seem to be a language with first-class support for async, reactive, and parallel programming, and a built-in cluster(ing).
The RPL Tool isn't "just for the initial construction of an application, but also encapsulates deployment, monitoring, and maintenance. It implements the first truly cohesive model for building software applications – a set of abstractions that can build any application at any scale with greatly reduced engineering cost." Why? Because "Building scalable software applications today is extremely expensive, requiring the combination of dozens of different tools. The engineering involves countless arcane tasks that are far removed from actual application logic. The narrow focus of each individual tool, and the lack of a cohesive model of building end-to-end applications, causes the engineering cost to be orders of magnitude greater than it could be." (Both from RPL's About page.)
👓 Neon — Serverless, Fault-Tolerant, Branchable Postgres [rust, database, serverless, SaaS]
An open-source, postgres-compatible database written in Rust (at least the storage part, the compute is perhaps just Postgres as-is) and aiming at serverless applications. A fully managed multi-cloud SaaS offering. It looks really interesting: separated compute and storage for better scalability, can scale down to 0, cold storage offloads to S3/similar for cost efficiency, cold start reportedly in under 3s, create an instant "branch" (copy) of a DB (via copy-on-write) - "This eliminates expensive full-data backup and restore". Autoscaling.
Functional-navigational programming in Clojure(Script) with Specter - thoughts from the red planet - thoughts from the red planet
👓 Functional-navigational programming in Clojure(Script) with Specter - thoughts from the red planet - thoughts from the red... [clojure, library, productivity]
An introduction of the Specter library and it's core value proposition: separate the details of navigating into nested (possibly recursive) data structures for retrieval and modification from your business logic - while keeping it extremely performant. This is thanks to the concept of paths, a composable series of navigational steps. You can thus write a generic business logic function working on any data structure, just taking the input/target paths as params. With this, you can replace many bespoke data retrieval and update utilities with a handful of simple, composable paths. "This is the crux of functional-navigational programming: better a handful of generic navigators than a lot of specific operations."
👓 Google debuts API to check security status of dependencies [security, tool] - Is this what the kids mean by owning the libs?
The deps.dev API and web site enables you to understand the complete, transitive dependency graph of a project, including security vulnerabilities, licenses, recent releases, and more. It is intended to combat software supply chain attacks.
The article also links to Google's Assured Open Source Software (Assured OSS) service for Java and Python, which mirrors repositories of more than 1,000 popular software packages that get scanned for vulnerabilities and get signed to prevent any tampering.
👓 Egeria Project [enterprise, tool, integration]
A Linux Foundation project for open metadata and governance for enterprises - automatically capturing, managing and exchanging metadata between tools and platforms. I.e. it is a solution designed to set a standard for leveraging metadata in line of business applications. The project site is somewhat overwhelming; the announcement post might give you a better undersanding if its value proposition.
Egeria consists of multiple elements:
Open standards for what the metadata about essentially anything an enteprprise cares should like like - Locations, Endpoints, Hosts, Storage, Software Servers, Applications nad Processes, Software Service, Netowrsk and Gateways, Cloud Platforms and Services; Actors, People, Teams, Feedback; Deployed APIs, Software Components, Files and Folders, Graph Stores, Databases (as well as metadata for Tables etc); Glossary and Dictionary to describe terms; Governance Controls, Licenses, Agreements; and much more.
A number of pre-built connectors to various sources of information and for auto-discovery (e.g. to extract metadata about databases via JDBC, about Kafka topics, about OpenAPI - enabled services).
A set of services for collecting and serving this metadata.
All this looks huge and complex (quite enterprisy, huh?), but you may find there useful / inspiring bits and pieces, especially if you work with any kind of the metadata they standardize.
👓 DevTools Tips [tool, webdev, learning]
A ton of bite-sized tips for getting most out of browser Dev Tools, marked with which browsers support each one.
👓 Next-Generation Event Processing Platform | Memphis.dev [tool, eventing, software architecture]
And a quote from a user: "All that time you spend scaling, managing, and maintaining Kafka is better spent building your actual product. Memphis frees you to do just that."
This is still under heavy development (now v 1.0) but can be an interesting tool when building event-centric architectures, reportedly much easier than Kafka (see comparison). Use it as a queue between components, to ingest high throughput data streams, and much more. I like the focus on developer experience, simplicity, and observability. I have been afraid of Kafka bacause it is reportedly difficult to operate, but this looks better. Though I am not sure what their business model is, perhaps living off their cloud offering and paid support?
Some features: Its broker is distributed and persistent. Schemaverse - evolution, validation, and enforcement for data contracts. (Protobuf, Avro, JSON, and GraphQL.) Coming in 1.5: Inline enrichment and analysis of ingested data using serverless functions. Multitenancy.
Run on Kubernetes for clustering. SDKs in Python, Go, Node.js, Java, REST, and more. Communication via TCP and HTTP. TLS, WebSockets, gRPC, MQTT, and AMQP are coming "soon".
Read more on its architecture. (A "station" is similar to Kafka topics or Rabbit queues.) It seems Memphis is written in Go and based on NATS Jetstream.
👓 SHACL - Wikipedia [data, standard, validation]
SHACL (Shapes Constraint Language) is a powerful and flexible data validation language for RDF graphs (possibly expressed as JSON-LD). It can specify constraints on individual properties and nodes, using simle or complex paths, even using SPARQL. There are many built-in predicates, and you can even use JS or SPARQL for more complex ones. See an example in the shacl-playground.
I have first learned about it from the presentation of the Clojure DB Fluree, which I suppose uses it under the hood, even though the docs do not mention it. (It also uses RDF, JSON-LD, SPARQL.)
KingMob/TrueGrit: A data-driven, functionally-oriented, idiomatic Clojure library for circuit breakers, bulkheads, retries, rate limiters, timeouts, etc.
👓 KingMob/TrueGrit: A data-driven, functionally-oriented, idiomatic Clojure library for circuit breakers, bulkheads, retries, rate... [clojure, library, devops]
Based on Resilience4j, freeing you from needing to deal with it's classes.
AbhinavOmprakash/snitch: Snitch injects inline defs in your functions and multimethods. This enables a repl-based, editor-agnostic, clojure and clojurescript debugging workflow. It is inline-defs on steroids.
👓 AbhinavOmprakash/snitch: Snitch injects inline defs in your functions and multimethods. This enables a repl-based,... [clojure, library, productivity, troubleshooting]
Just add to your dev/user.clj
(require '[snitch.core :refer [defn* defmethod* *fn *let]])
which, will intern these macros inside clojure.core & cljs.core, so you don't have to import them in every namespace.
Using any of them instead of the original will capture any (possibly destructured) bindings in them and their bodies as top-level defs. And *[de]fn captures the urn value as fn-name> and fn-name< renders a call that un-destructures the captured arguments so that our can call the fn.
--
Thank you for reading!