My Highlights From EuroClojure 2013
EuroClojure 2013 was a nice, small conference. The talks were mostly interesting and often useful and it was wonderful to meet in reality people I only knew from the virtual life or from stories. You can get an impression what it was like from the #euroclojure tweets.
Below are some noteworthy things from the talks and chats.
The Clojure-driven drone project Sormilla at GitHub is an example of using Stuart Sierra's Reload pattern to be able to realod code cleanly w/o restarting REPL, see the interface Service. (I cannot find this part of the code at GitHub and it is 2m old, perhaps the changes with this aren't there yet?) Using clojure/tools.namespace's reload[-all]
Ruby programmers are perhaps more open-minded towards Clojure than experienced Java devs?
Local and contextual quality: Christopher Alexander, the father of design patterns, examines one special "quality without a name" that embodies that a design is good. It is hard to describe and is special by depending very much on the local context and cannot be described and prescribed generally, in the absolute platonic way. How does this quality apply to software?
Performance and reliability oppose each other when going from the least performant, brute force, through caching to mutability => when in doubt, use brute force (i.e. old good Clojure without perf. optimization). Caching is often good enough.
Uses the library Criterium for benchmarking.
Overview (via Richard West):
Some of the code smells from OOP apply to Clojure as well, perhaps with some modifications (too large class => too large namespace) but Clojure has also some of its own, such as:
See also Tero Parviainen's extensive summary of the Narcissistic Design talk.
Narcissistic Design recorded at NoFluffJustStuff.com.
The 10 steps to job security are:
Small x big: It is not only important that you fix bugs or solve problems, you should also ask whether you are actually solving the big problems / defects. Some approaches help with small bugs / design problems but that is not enough.
Complexity: Work around problems in a layer above rather than fixing them where they are. Ex.: fixing shortcomings of Java in Clojure (f.ex. Java allows multiple instances of false) - duing this guarantees increased complexity.
Xively started originally as a Rails app, storing both metadata and data from the customer's devices in PostgreSQL. This didn't work well since most devices sent data in the same time (full hour, ...) and Rails/PG did not handle the spikes well.
They have therefore replaced PG with Cassandra nad considered Node.js (but were discouraged by testimonials from large apps and callback hell) and Go (too new at the time) until they settled for Clojure b/c it was coold and had first-class quality driver for Cassandra (which alone got the performance 10* up).
They liked Leiningen (even for a Java app) and simple multithreading with pmap. Internally the app uses RabbitMQ.
Currently replacing REST in devices-server communication with better Message Queue Telemetry Transport (MQTT) and
Wins
Thanks to that, Jon & Co. were able to create a Clojure app proof of concept that was able to render the non-trivial front page of Daily Mail in a week, without coupling to the old system. During the following 6 months, they were able to replace the publishing part of the software that has grown for 14 years.
Technologies:
Far more detailed notes by Tero Parviainen.
As you might know, Datomic uses a single atom to hold its state. I have been wondering how to manage my state as a huge map in one atom (and atoms referenced by an atom seemed strange ...). Now Christophe has clearly thought about the same and came with a nice solution.
In the first iteration, megaref was just a combination of update-in and swap! but later got better by trying to avoid update conflicts on unrelated branches of the state tree/map (e.g. [:a :b] and [:a :c] as oppossed to the conflicting [:a :b] and [:a :b :c]) and being smart about what to protect against concurrenct access. In the backround, it can use atoms, STM, or (coming) agents to handle concurrent access to the map.
Unless your app is CPU bound, it is worth giving a try for holding its state.
Experiences with integration platoforms/tools from Daily Mail:
Below are some noteworthy things from the talks and chats.
Other stuff
- Rich Hickey’s Greatest Hits - links to the most popular talks (currently 6) such as Simple Made Easy
- Rich Hiceky's demo simulation of Ants in Clojure - high concurrency, no locking
Slides and code from the talks
Go to EuroClojure 2013 at Lanyrd for links to slides and source code used in the talks.Talks
Here are some things I have found worth remembering from the programme.Side notes
The MOOC Introduction to Complexity has been mentioned and recommended.The Clojure-driven drone project Sormilla at GitHub is an example of using Stuart Sierra's Reload pattern to be able to realod code cleanly w/o restarting REPL, see the interface Service. (I cannot find this part of the code at GitHub and it is 2m old, perhaps the changes with this aren't there yet?) Using clojure/tools.namespace's reload[-all]
Ruby programmers are perhaps more open-minded towards Clojure than experienced Java devs?
Zach Tellman's keynote: States and Nomads: Handling Software Complexity
Abstractions are essentially simplifications that try to hide complexity by either treating multiple moving parts as one or by ignoring some relationships between the moving parts. Therefore all abstractions are imperfect and there are always situations where the underlying complexity leaks through. Every abstraction (library, framework) has thus a boundary beyond which it isn't useful anymore due to the underlying complexity leaking through. A civilized discussion should focus on discovering the boundary rather than claiming that something is rubbish because you tried to use it outside of its boundary of usefulness or claiming that it is a silver bullet.Local and contextual quality: Christopher Alexander, the father of design patterns, examines one special "quality without a name" that embodies that a design is good. It is hard to describe and is special by depending very much on the local context and cannot be described and prescribed generally, in the absolute platonic way. How does this quality apply to software?
There is a central quality which is the root criterion of life and spirit in a man, a town, a building, a wilderness. This quality is objective and precise, but it cannot be named.Zach's reading list containing f.ex. Gabriel, Richard P. Patterns of Software is interesting
- Ch. Alexander: The Timeless Way of Building, Volume 8, page ?19?
Liberator – free your data with RFC 2616
Liberator is a REST library for Clojure with great troubleshooting capabilities, having a log of past requests and for each of them the decision steps it went through, complete with a graph.James Reeves: Functional 3D Game Design
A 3D game has ~ 16 ms/frame which is not too short for using clojure.Performance and reliability oppose each other when going from the least performant, brute force, through caching to mutability => when in doubt, use brute force (i.e. old good Clojure without perf. optimization). Caching is often good enough.
Uses the library Criterium for benchmarking.
Jen Smith: Common Clojure Smells
Slides (pdf).Overview (via Richard West):
Some of the code smells from OOP apply to Clojure as well, perhaps with some modifications (too large class => too large namespace) but Clojure has also some of its own, such as:
- Magic Keys - if many functions need to know about keys within (= structure of) a map => harder to maintain => try to decouple
- Locally scoped atoms
- Indirection by partiality - many derived functions using (partial .. )
- Macromania - overuse of macros
- Excessive use of lazy evaluation - a troubleshooting issue since an error may appear much later when the lazy seq is evaluated, far from where it originates => consider evaluating seqs at borders of responsibility (modules, ...)
- Many discrete steps with a final one that executes the seq (a.k.a. long pipelines; map + filter + reduce + ...) - an error in the middle will be hard to troubleshoot => break down
- Parens proliferation (a.k.a. over-nesting, doing too many things at once) => break into multiple functions, use ->
Stuart Halloway's keynote: Narcissistic Design
A thought-provoking presentation about "tips" for increasing complexity including a masked criticism of (bad) unit testing (many tests with different setup, ...). It is worth listening to and thinking about.See also Tero Parviainen's extensive summary of the Narcissistic Design talk.
Narcissistic Design recorded at NoFluffJustStuff.com.
The 10 steps to job security are:
- Use OO, and don't forget those setter methods!
- Prefer APIs over data.
- Start with DSLs.
- Always connect (and never enqueue).
- Create abstractions for information (i.e. wrap all data in specific objects/classes)
- Use static typing across subsystem boundaries.
- Put language semantics on the wire. (Instead of using a language-agnostic data language like json, edn, or hessian.)
- Write lots of unit tests.
- Leverage context.
- Update information in place.
- Design an information model
- Design an API
- Design DSL
Small x big: It is not only important that you fix bugs or solve problems, you should also ask whether you are actually solving the big problems / defects. Some approaches help with small bugs / design problems but that is not enough.
Complexity: Work around problems in a layer above rather than fixing them where they are. Ex.: fixing shortcomings of Java in Clojure (f.ex. Java allows multiple instances of false) - duing this guarantees increased complexity.
Paul Bellamy & Martin Trojer: Using Clojure to Serve The Internet of Things
https://github.com/martintrojer/euroclojure-demo - including code for Arduion, clojure webapp collecting tweets, etc.Xively started originally as a Rails app, storing both metadata and data from the customer's devices in PostgreSQL. This didn't work well since most devices sent data in the same time (full hour, ...) and Rails/PG did not handle the spikes well.
They have therefore replaced PG with Cassandra nad considered Node.js (but were discouraged by testimonials from large apps and callback hell) and Go (too new at the time) until they settled for Clojure b/c it was coold and had first-class quality driver for Cassandra (which alone got the performance 10* up).
They liked Leiningen (even for a Java app) and simple multithreading with pmap. Internally the app uses RabbitMQ.
Currently replacing REST in devices-server communication with better Message Queue Telemetry Transport (MQTT) and
Wins
- Powerful abstractions - immutable data structures, Lisp power, ....
- Much more declarative than Rails
- Language aesthetics - elegant composability of core functions, ...
- Much better performance than Rails (but then, R. wasn't created for performance)
- JVM startup time - partially countd by using Stuart Sierra's Reload workflow to minimize REPL restarts
- Hard to read / use stack traces (especially from anonymous functions in lazy collections)
- It is easy to get confusing version clashes in Leiningen when an app and its dependency depend on different versions of the same lib - whatever lands first on classpath is loaded (some lein. plugins may help?)
- Tooling is most mature around Emacs
- No debugger => println-based troubleshooting (ritz isn't there yet) (my tip: see night-vision.goggles/introspect-ns!)
Clifton Cunningham (CEO) & Jon Pither (dev): Newspaper Massacre
Lot of inspiring stuff here. Daily Mail had 14 years old legacy up with intertwined CMS, publishing platform, and Oracle DB, over 300k LoC of Java/JSP/JS that everybody feared to touch. They did two great things: (1) The CEO decided to foster teams that take end-to-end ownership, have minimal make-your-own process, do not fear change. (2) They modified the old system to publish events via messaging (initially Mule) whenever anything important happened (new article, ...), these were stored together with the data as JSON in ElasticSearch.Thanks to that, Jon & Co. were able to create a Clojure app proof of concept that was able to render the non-trivial front page of Daily Mail in a week, without coupling to the old system. During the following 6 months, they were able to replace the publishing part of the software that has grown for 14 years.
Technologies:
- Moustache for templating - it is cross-language, can also render in the browser for more dynamism, and is fron-end dev friendly
- Zookeeper and its Clojure wrapper Avout for shared configuration (=> every service has atom(s) that get updated when the config is changed, can use Clojure's add-watch on the atoms to be able to react to the changes
- Riemann with Logstash and Graphite for monitoring exceptions and other events of interest
- Redis as a golden hammer :) for different things
- Memcached for pure data caching (which is perhaps not an ideal situation)
- Apache Camel (that replaced Mule) for integration and messaging (though not really happy with it)
- Use Datomic instead of ElasticSearch - it has a define schema (x amorphic JSON), it is possible to get snapshot of data at any time for troubleshooting, ...
- core.contracts and perhaps core.typed or Prismatic's Schema to make the data more defined
Ryan Greenhall: Templating In Clojure
Noteworthy:- Hiccup not suitable for fron-end devs not comfortable with Clojure
- Enlive breaks easily when relying on selectors and the structure of a page changes (though, reportedly, the proper use is predictor-based rather then selector-based)
- clj-jade - Java+clojure implementation of Jade templating, which is, if I understand it correctly, a combination of HAML (conscise syntax for HTML) and minimalistic templating
- Personally, I would also give a try to Mustache for Clojure (if you can live with HTML's verbosity)
Christophe Grand: Megarefs
GitHub: https://github.com/cgrand/megarefFar more detailed notes by Tero Parviainen.
As you might know, Datomic uses a single atom to hold its state. I have been wondering how to manage my state as a huge map in one atom (and atoms referenced by an atom seemed strange ...). Now Christophe has clearly thought about the same and came with a nice solution.
In the first iteration, megaref was just a combination of update-in and swap! but later got better by trying to avoid update conflicts on unrelated branches of the state tree/map (e.g. [:a :b] and [:a :c] as oppossed to the conflicting [:a :b] and [:a :b :c]) and being smart about what to protect against concurrenct access. In the backround, it can use atoms, STM, or (coming) agents to handle concurrent access to the map.
Unless your app is CPU bound, it is worth giving a try for holding its state.
Frazer Irving: Enterprise integration with Clojure
About the Enterprise Integration Patterns book: written by smart people but in a boring style and very prescriptive (i.e. "this is how to do it" instead of a discussion)Experiences with integration platoforms/tools from Daily Mail:
- Initially used Mule - but nobody understood or wanted to understand its tons of Spring XML files
- Currently using Apache Camel and its Holy Grail Clojure DSL (with ActiveMQ underneath) but not perfectly happy with it
- Camel implements Enterprise Integration Patterns from the book; the disadvantage is that it constraints thinking to these patterns only
- Hard things are hard (inst. of "possible")
- Positive: mature, plenty of adapters, ...
- Tried following clojure tools
- Lamina (a tool for "describing and analyzing streams of data") - not primarily for enteprise integration (EI); implements channels but currenlty supports only HTTP and websockets (not JMS, ...); good: visualization of what happened to a message, i.e. what & why it went though
- EEP (Embedded Event Processing) - young and experimental, not designed for EI