Holy Dev Newsletter February 2022

👓 Test Desiderata [testing, best practice]
Desirable properties of tests by Kent Beck - Isolated, Composable, Fast, Inspiring, Writable, Readable, Behavioral (= if code behavior changes then the test fails), Structure-insensitive (= if code structure changes the test still passes), Automated, Specific (= easy to find the cause), Deterministic, Predictive. You make trade-offs between these for different kinds of tests (e.g. unit x acceptance).

👓 MicroProfile LRA [standard, architecture, distributed systems]
Long Running Action: "The proposal introduces annotations and APIs for services to coordinate long running activities whilst still maintaining loose coupling and doing so in such a way as to guarantee a globally consistent outcome without the need to take locks on data."
An alternative to distributed transactions - a compensation based approach in which participants make changes visible but register a compensatory action which is performed if something goes wrong. (Via a central coordinator, I assume.)
Handles failure conditions, maintains state for those flows that span long periods of time and ensures that remedial activities are called correctly. Goals

• support long running actions
• no strong coupling between services
• allow actions to finish early
• allow some parts of a computation to be cancelled without affecting other parts of the computation (nested activities)
• guarantee execution of compensating actions if a business activity is cancelled

👓 Explore.org [psychology]
Science tells us that watching nature is deeply soothing for humans. Here you can select from a number of life feeds of animals, oceans, waterfalls, "zenscapes" and more. Silent, with a calm music, or the sounds of the nature. Perfect during a long meeting :-) Some of my favorites are the Cayman Reef (music), Lisbon Falls - Blyde River (water sounds), Jellyfish tank (music), Tembe Elephant Park (nature sounds).
They have also a bunch of documentary films mostly about nature.
(This link is out of character for this technical feed - but we are humans and managing one's energy and well-being is crucial for productivity.)

🎥 A mindblowing and engaging talk about performance measurement and optimization, primarily in the context of Linux binaries. Part...
A mindblowing and engaging talk about performance measurement and optimization, primarily in the context of Linux binaries.
Part 1: The "random" memory layout of your program can have 10s of % impact on the performance (as it changes locality and such). Any code change can influence it in unpredictable ways. So the 10% speedup result of your optimization effort may actually be a random consequence of changed memory layout (on your test HW). To eliminate the effect of memory layout, you want to randomize it. Do it many times during the performance test (e.g. every 1/2s) to ensure normal distribution of the execution times (thanks to the Central Limit Theorem), which enables you to use powerful statistical tools to decide whether the results are statistically significant or not. The author's & Co.' tool Stabilizer does such randomization. (On the test app, gcc's -O2 proved significantly faster than -O1 while -O3 provided minimal speedup that might be random.)
Part 2: Profiles are not very good for concurrent, multi-threaded programs. In a concurrent program we want to know what is the effect of speeding up a particular process on the whole program? We cannot speed it up - but can slow down everything else. The Linux "causual profiler" Coz does that. (Note: Sometimes speeding something up can slow everything down - e.g. due to increased congestion.) Coz allowed them to detect simple but hard to find optimization improvements, leading to the speedup of 25% in sqlite with just +3, -3 lines of code and of 10% in memcached for +2,-6 LoC.
More amazing software and materials at the group's site https://plasma-umass.org.

👓 Sigma.js [library, javascript, data visualization]
A mature library (since at least 2015!) aimed at use cases where you have giant graphs containing thousands of nodes and edges that you need to render quickly.

👓 github/hotkey: Trigger an action on an element with a keyboard shortcut. [library, javascript, ux]
Use Hotkey, set the data-hotkey attribute on your elements, and you get keyboard shortcuts. Reportedly used by GitHub. Via JavaScript Weekly.

👓 Welcome to IMS | Introduction to Modern Statistics [book, learning, statistics, data science]
Open and free textbook introducing the student to statistics - exploratory data analysis, regression modeling, hypothesis testing etc.

👓 HarvardX Data Science Professional Certificate [learning, data science]
Self-paced, commercial MOOC program in data science, with R. The program covers concepts such as probability, inference, regression, and machine learning and helps you develop an essential skill set that includes R programming, data wrangling (to process and convert raw data into formats needed for analysis) with dplyr, data visualization with ggplot2. Motivating case studies.
The online, free book Introduction to Data Science - Data Analysis and Prediction Algorithms with R by Rafael A. Irizarry is based on the program.

👓 The 9 indispensable features to learn for the new Rust programmer [rust, learning]
"less than 20% of the features will bring you more than 80% of the results". Those 20% include: Enums, async-await, Traits, Smart pointers (Arc & Co., so you don't need to fiidle with lifetimes), Collections, Iterators, Combinators (map, filter etc.), Streams (± iterators for async), no_std (for mebedded dev).

👓 Animate.css | A cross-browser library of CSS animations. [css, library]
Easily add animations to html or SVG. Already at v4, leveraging CSS variables. Worth checking out!

👓 hikerpig/pintora: An extensible text-to-diagrams library that works in both browser and node.js [library, javascript, diagram]
Pintora is an extensible javascript text-to-diagrams library that works in both browser and Node.js. So far it only supports Sequence, Entity Relationship, Component, and Activity Diagram and Mind Map but you can add your own with some TypeScript.
Heavily inspired by Mermaid.js and PlantUML.

👓 magnars/optimus: A Ring middleware for frontend performance optimization. [clojure, library, webdev, performance]
A Ring middleware for optimization (minification, concatenation, etc.) of static resources such as JS and CSS. There is even an addon for images. It can serve the assets live in dev and optimized in prod. It can also encode the file version in the URL for eternal cacheability (and has functions to produce these URLs, to use in your html templates/..).

👓 phronmophobic/snowball: View the sizes of your dependencies. [clojure, tool]
View the sizes of your dependencies in a tree map - for a deps-based project or any maven library.

👓 Condition Systems in an Exceptional Language - Chris Houser [clojure]
A discussion of approaches to error handling (throw an exception, return a special value, condition systems), based on Peter Seibel's chapter 19. Beyond Exception Handling: Conditions and Restarts but adapted to Clojure. Condition systems such as Common Lisp's are the most powerful and more widely useful. "The condition system is more flexible than exception systems because instead of providing a two-part division between the code that signals an error and the code that handles, it the condition system splits the responsibilities into three parts - signaling a condition, handling it, and restarting." I.e. the low-level place where the error occurs can offer a number of recovery strategies - restarts - such as "skip the faulty item" or "replace it with a default value" and something up the stack can decide which of the strategies to use (and supply it with configuration, such as the default value to use). The talk also demonstrates how to implement it yourself, using Clojure's dynamic variables (or rather, in this case, functions). A modern Clojure(Script) library implementing conditions and restarts is farolero. (You might also listen to the defn podcast with its author, Joshua Suskalo.)

👓 FDA Approves Video Game Based on UCSF Brain Research as ADHD Therapy for Kids | UC San Francisco [society, future] - The FDA has approved the first video game therapeutic as a treatment for attention deficit hyperactivity disorder (ADHD) in children, based
The future is here - the first ever video game has been approved as a treatment for the brain (in particular, ADHD). There is a huge potential in "experiential medicine", ie designing experiences that get our brains and bodies to improve, leveraging neuroplasticity and bio-feedback through various sensors. Though it has to be based on serious science and not wishful thinking and marketing, as is sometimes the case:-)
You can learn more about "digital medicine" and improving the brain through closed-loop (i.e. quick feedback) video games in the 2018 essay by Adam Gazzaley titled The Cognition Crisis: The Perils and Promise of Technology and the Brain. Quote: "Neuroscape also is working on virtual reality, multi-sensory immersion, and motion capture. It is building games that target cognition. They incorporate efforts in art, music, story, and multiple award cycles."

🎥 Valentin Waeselynck demonstrates interactive development in Clojure, exploring a 3rd party API and building a backend for a...
Valentin Waeselynck demonstrates interactive development in Clojure, exploring a 3rd party API and building a backend for a search site one function at a time.

👓 A Data Pipeline is a Materialized View [theory, data processing]
Good thoughts about the connection between data pipelines (that take data inputs and produce new data) and materialized views (that show a particular view of the source data, and are persisted). A pipeline is essentially a tree/DAG of views where the leaves are the input, primary data and the root is the output we want. For performance we don't want to recompute every time the data is needed so we want to cache some parts of the graph - hence the materialized view. We associate this term with relational databases but it applies here just as well. As do concerns such as update triggers and update granularity. Nowadays there are interesting projects for building such materialized views such as dbt - "Pipelines as Batch-Updated SQL Queries" and Materialize - "Pipelines as Live-Updated Materialized Views" (of streaming sources such as Kafka; "very efficient incremental updates against very flexible and expressive queries").

👓 wotbrew/relic: Functional relational programming for Clojure(Script). [clojure, library, clojurescript, database] - Functional relational programming for Clojure(Script). - GitHub - wotbrew/relic: Functional relational programming for Clojure(Script).
An in-memory database - a Clojure map - with indexed SQL-style but data-based query (that can call Clojure functions), materialized views with incremental maintenance, powerful data integrity constraints, reactive (=> recompute derive data, update UI upon change) and efficient at that. The db is a map of "entity name" -> vector of entities and Relic makes sure it is indexed and fast, joins work, materialized views are always up to date. Might we worth exploring e.g. as the client-side data store if you work with complex data, need reactivity, and performant derived views.

👓 rorokimdim/pp-grid: A clojure library to construct formatted text [clojure, library, cli]
Print grids, tables (with nested tables), charts in ASCII. Not sure where I would need this (output from CLI tools? REPL-making ASCII "graphics"?) but if I ever do, it will be a life-saver.

👓 AsyncAPI Initiative for event-driven APIs [standard, api]
The AsyncAPI specification is an industry standard for defining asynchronous - i.e. message-driven - APIs, for example over WebSockets, HTTP, Kafka. It is the async twin of OpenAPI (formerly Swagger) and describes the available Operations, Messages, subscribable Channels, etc. Plus tools to generate docs, client libraries and server code

👓 Dare to ask for more #rust2024 [rust]
An inspiring look at what the 2024 Rust edition could be, from one (I assume) of the core contributors. Essentially more expresiveness and ergonomics in Rust, async, unsafe Rust etc. Fewer rough edges. Even awesomer tooling.

👓 A Rust match made in hell [rust, deep dive]
A very far reaching exploration of a deadlock bug caused by match's "borrow-through" (extending the lifetime of the thing being matched over the whole match) in interaction with mutexes and async (namely Arc>). We learn about borrow-through, why standard mutexes cannot be held across .await (contrary to Tokio's), fixing mutex moving into a block by switching to a reference, and a lot more. In essence, the problem is that this match mutex.lock().foo() { true => { mutex.lock().bar(); } false => {}} deadlocks.

👓 Beautiful images of nature that you are free to use (contribution not required). A great resource for my talks! [blogging]
Beautiful images of nature that you are free to use (contribution not required). A great resource for my talks!

Why you cannot apply a process to software development (via The GROWS Method® Institute)

Wrt the Cynefin framework's 4 domains of complexity: Use of a process is perfect in the Clear domain. It might be a useful part of the Complicated domain. It's generally not useful at all in the Complex or Chaos domains. And within those first two domains, where a process is helpful, you probably shouldn't be developing custom software in the first place.
We develop software from scratch because we're trying to doing something that hasn't been done before. If it has been done before, then you're probably better off buying it or renting it as a service.

👓 GUN - data sync protocol and Web3 P2P graph database [web3, database]
GUN is a small, easy, and fast protocol for syncing data. Because GUN is small, it can be added to your app to improve a single feature. But because GUN is easy, it can also replace hundreds or thousands of lines of code, networking calls, storage boilerplate, pub/sub routing, cache invalidation, and more. The power of GUN is that it abstracts this complexity into a unified simple API that is very different than traditional systems, with the aim of making it easy to build cool apps quickly.
The best way to think about GUN is synchronizing video game state: In 2 lines of code we can gun.get('thing').put({x: 0.01, y: 0.02, z: 0.03}) to send updates and gun.get('thing').on(data => object.moveTo(data.x, data.y, data.z)) to read it on another machine. Here .put({x: 0.09}) merges with the existing data.
GUN is a graph database that can store SQL-like tables, JSON-like documents, files and livestreaming video, plus relational and hypergraph data!

👓 Pencil - OSS GUI prototyping tool [design, tool, ui, prototyping]
An open-source GUI prototyping tool that's available for ALL platforms. Includes "stensils" for desktop, mobile etc apps. Elements in a design can link to other pages to represent interaction flow.

👓 A Critique of React Hooks | Hacker News [javascript, react, criticism]
An interesting discussion of pros (eg. simplified library APIs) and cons (eg. magic, not pure FP) of React Hooks.
All in all, there seems to be both pros and cons: "I  converted a medium-sized React codebase from classes to hooks. In most cases it simplified the components and eliminated boilerplate. But it also introduced more than a few very tricky bugs and serious performance regressions that were not trivial to fix."

👓 ulid/spec: The canonical spec for ulid, a lexically sortable, uuid-compatible universally unique identifier [specification]
Universally Unique Lexicographically Sortable Identifier as an alternative to UUID. UUID can be suboptimal for many use-cases because, among other, it isn't the most character efficient way of encoding 128 bits of randomness and its randomness can cause fragmentation in many data structures (such as DB indices). ULID is lexicographically sortable (much better for indices). Canonically encoded as a 26 character string, as opposed to the 36 character UUID. And more!

👓 yetanalytics/pathetic: Utility Library for handling JSONPath and navigating JSON structures [clojure, library, clojurescript]
Parse JSONPath to data, extract a map of paths to values from a json using a JSONPath, turn a wildcard / partial path into a list of actual paths,...

👓 Reliably Send an HTTP Request as a User Leaves a Page | CSS-Tricks - CSS-Tricks [webdev, analytics]
Outstanding requests that have not finished in time (eg on slow 3G) are cancelled when navigating away from / closing a page. If you want to inform eg your analytics reliably, you have a few options. You can use the send beacon api:
document.getElementById('link').addEventListener('click', (e) => { const blob = new Blob([JSON.stringify({ some: "data" })], { type: 'application/json; charset=UTF-8' }); navigator.sendBeacon('/log', blob)); });
Or, if that isn't good enough, You might go with fetch() + keepalive if: You need to easily pass custom headers with the request / You want to make a GET request to a service, rather than a POST / You’re supporting older browsers (like IE) and already have a fetch polyfill being loaded.
It’s also worth mentioning that a growing number of browsers support the ping attribute. When attached to links, it’ll fire off a small POST request.