Hands on Rama, day 1: Setup, idempotent create & update

I struggled a little with basic setup, before I understood that I needed to run lein with both the profiles dev and provided, the latter to pull in Rama itself while running outside of a cluster. This time I already remembered to avoid pulling in Clojure itself, since Rama has it built in (customized, to handle the dataflow "language"). I also had to add a dependency on nrepl for Cursive’s REPL to work. (Which would be unnecessary in Calva, I believe.)

Side note: Getting support for clj-kondo is currently somewhat manual process - you need to copy com.rpl into your .clj-kondo/com.rpl and add it to :config-path in your config.edn. I expect that Rama post v0.11.0 will have it baked in and then it will suffice to run With Rama v0.11.4+ you just need to run clj-kondo …​ --copy-configs in your project to get clj-kondo config for Rama in place.

After that, everything worked fine, with only minor limitations in Cursive (inability to show docstrings for some macros, and not understanding that Rama includes Clojure at v.1.11 and thus with random-uuid.) There are also few more known limitations.

I have read the initial announcement blog post in detail, and most of the Introducing Rama’s Clojure API: build end-to-end scalable backends in 100x less code, and small parts of the documentation. I typically prefer to study more but I really wanted to get my hands dirty. I paid for that rush :-).

Syntax: I immediately run into my lack of working knowledge of Rama syntax. A cheat sheet would be extremely useful, both for Rama in general and the dataflow language in particular. Fortunately, the API docs are quite decent, though they could be even better. I kind of lacked understanding the bigger picture, for example the general structure of dataflow. I had to read a few examples to figure out that the way to write processed data to a PState is via local-transform>.

Dataflow vs. Clojure: Another struggle was understanding what Clojure functions and special forms can be used inside dataflows and where I have to use Rama’s specific ones. Now I first check whether com.rpl.rama has an alternative (as it does for if, assert, or, and others). It seems that especially special forms are problematic. When I used or instead of or>, I got a mysterious "Don’t know how to create ISeq from: clojure.lang.Symbol" somewhere in com.rpl.ramaspecter.navs$fn__15709 during the Rama to Clojure compilation / transpilation, which wasn’t very helpful.

Even if I am able to put together working code, I have no idea whether it is any good or not. Fortunately, Nathan is very helpful in the #rama channel in Clojurians Slack.

The ProfileModule and BankTransferModule were especially useful for my case - the former deals with creating and updating entities, the latter with "transactional" updates. I wouldn’t have thought of a separate depot for create and update (i.e. edit) events. I have opted for simplicity over efficiency and not used rama-helpers' ModuleUniqueIdPState for more space-efficient IDS, contrary to those examples. I was unsure about how to describe my data - do I want to use something strict, such as Thrift? Is there any benefit to or need for using Records? I don’t know so I used records, as the examples do. I wouldn’t even know how to integrate Thrift, or how to handle our custom fields with it.

Data structure: I was also unsure how to model the data, in this case Components, especially with respect to the user-defined custom fields. In RDBMS we store them as a single JSON column, while in code we inline them into the Component as top-level properties. I assumed I wanted to sub-index them, since I often need to search or filter based on their values, but I lack a complete understanding of what sub-indexing means and its consequences. I ended up with the schema {UUID (map-schema Keyword Object {:subindex? true})}, i.e. component-id (a uuid) → component as a map from a keyword to an arbitrary object. I could also have used fixed-keys-schema for the Ardoq-defined Component properties, with custom fields nested under a sub-indexed property, similarly as in the current DB. I am not sure what are the tradeoffs and which one to pick.

Returning data: Currently, a Create operation in Ardoq returns the complete entity. I wanted to keep that, and also to make it idempotent so that re-submitting the same request (with the same uuid) would simply return the original entity. That was quite simple with a local-select> to check for existence and the recently added ack-return> for returning the entity from the dataflow. But it broke when the entity already existed, because the select returned a …​durable.RocksDBWrapper, which isn’t serializable. Nathan kindly explained that it is because a sub-indexed thing could potentially be much larger than memory, and suggested turning it into a simple map with Specter’s view (which I had no idea existed) and old good (into {} …​). Many thanks!

I have also tried to return the full PState, for troubleshooting, but that wasn’t possible either - you always need to provide a path with the key navigator, so that Rama can get the correct partition. I’ve spent some time fighting it over this 😅.

CAS: My last struggle and success was implementing "compare-and-set" semantics for updates, i.e. only set a component property to the new value if it still has the expected value. I have quickly discovered assert! and throw! and used them to try to prevent the update from happening. That did not work, because Rama just kept retrying the operation. Thus I learned not to use exceptions for flow control and instead used an if> that either updates the data with local-transform> or returns an error message with ack-return>. Nathan confirmed that this was a reasonable way to do it. Though I am not finished yet because currently this would prevent updates to properties that have changed in the meantime but still allowed those that did not. That may be desirable in some cases, but I would rather prefer transactional semantics of "all or nothing". I believe it will be straightforward to change to that.