Most interesting links of August '13
Sorry folks, this month it will be very brief. I have many more great stuff in the queue but haven't managed to write it down yet. Next month will be heavy :-)
Recommended Readings
- Interested in native vs. webapp? Check out Why mobile web apps are slow (mobile browser much slower, not much real improvements, weak CPUs,...; seems to be really high-quality, plenty of data) and Sencha's 5 Myths About Mobile Web Performance (Mobile web performance is mostly driven by JavaScript performance on the CPU, CPU-Bound JavaScript has only become faster because of HW improvements, Mobile browsers are already fully optimized, Future hardware improvements are unlikely to help, JavaScript garbage collection is a performance killer).
- Why Software Projects are Terrible and How Not To Fix Them - many teams are not ready to embrace new/better software practices, primarly for two reasons: 1) most of them are nonintuitive (f.ex. adding more people will slow dev down) and need to be sold through a high hierarchy of managament - but people/managers/organizations don't really care, it takes years for good/bad practices to have an impact, which is not relevant "now." 2) Businss objectives change too quickly and SW is blamed for not delivering. Based on evaluating many failed projects. Conclusion: Choose carefully people/organizations your work with. Avoid blame-driven ones. Quote on middle managers: "He has to put more developers on the project, call a meeting and yell at people, and other arbitrary bad ideas. Not because he thinks those will solve the problem. In fact, managers often do this in spite of the fact that they know it’s bad. Because that’s what will convince upper management that they’re doing their best." "In the vast majority of failed projects I’ve been called to looked at, the managers have not read one book on software engineering."
- Big Data: Kafka for uSwitch's Event Pipeline - a better alternative to log files - use LinkedIn's Kafka for messaging, have MR jobs to import latest messages into Hadoop/HDFS. The advantage of Kafka is that it persists the messages for a period of time so it is easy to batch-import and even re-import them. The uSwitch's talk Users As Data explains the downsides of log files. LinkedIn's Camus is a tool for importing messages from Kafka to HDFS.
- Realtime Analytics with Storm and Hadoop (at Twitter; presentation deck) - pre-aggregate some data into a read-only, random read DB such as ElephantDB, Voldemort, or Manhattan. For newer data use Storm and aggregated data in a read-write, big-data DB such as HBase, Riak, or Cassandra. For stuff that cannot be pre-aggregated you might use Storm's Distributed RPC.
- The Unified Logging Infrastructure for Data Analytics at Twitter - a paper from late 2012 that presents "Twitter’s production logging infrastructure and its evolution from application-specific logging to a uni- fied “client events” log format, where messages are captured in common, well-formatted, flexible Thrift messages" - with the benefit of "s streamlined log collection and data analysis".
- Development and Deployment at Facebook (Kent Beck et. al., 8/2013, 13p paper) - "More than one billion users log in to Facebook at least once a month to connect and share content with each other. Among other activities, these users upload over 2.5 billion content items every day. In this article we describe the development and deployment of the software that supports all this activity, focusing on the site's primary codebase for the Web front-end."
Talks
- One of the most valuable talks I've seen, in just 18 min: The Progress Principle - about the disengagement crisis and motivation at work by Teresa Amabile at TEDx Atlanta (via @thovden). Disengagement from work is increasing, at all age and salary levels, and leads to unhappy people, low productivity, huge financial losses. Based on analysing diaries of 12k participants, the single most important engaging and motivating factor is making progress in a meaningful work (including small wins). A culture of management by fear and punishment for failure creates disengagement and can crush even an innovative, profitable, praised company in a few years. Everybody, though especially the management, creates the culture through their everyday, small actions. If everybody focuses on catalysing progress and supporting their fellow humans through good and bad times, engagement and success will follow. Remove progress inhibitors, nourish the human spirit (acknowledge what we humans value, encourage people). Yet of the managers asked, very few knew of the significance of making progress (or, I can assume, of supporting people and making them happy(er) and the impact of our inner work life (perceptions, emotions, etc.) on our productivity and creativity). The study included two seemingly similar, successfull companies, one with great engagement, another with a new management that managed to destroy the engagement and thus eventually the company. Actions to take: catalyse progress, celebrate wins, encourage and support your colleagues.
Clojure Corner
- Wonderful Clojure Cheatsheet 1.5 with tooltips showing the doc and summary of information available at clojuredocs.org (other Clj versions), by Andy Fingerhut
- Chas Emerick's Clojure type selection flowchart to help you decide whether to use a map, a record, reify, proxy, gen-class, or deftype. (Reify and proxy don't produce a class but just an instance of an anonymous class; proxy can extend a base class, reify cannot. gen-class produces a class visible from Java and can extend Java classes. ...)
Tools/Libs
- Docker.io - pack, ship and run any application (and its dependencies) as a lightweight container, i.e. essentially "a VM without the overhead of a VM," using linux containers (chroot on steroids with resource limits via control groups) see reports of some uses such as Java app deployment, desktop virtualization, automatic app deployment in GitHub commit. Docker also supports evolving the containers over time, i.e. deploying new version, by pushing just diffs so it's low-overhead. You can build a container (include files, SW, forward ports, ...) using a Dockerfile. See dotScale 2013 - Solomon Hykes - Why we built Docker for an intro (20 min).
- Packer.io - tool for building pre-configured VM images for different platforms (EC2, VirtualBox, ...), remotely similar to Netflix's Aminator. See Immutable Servers With Packer and Puppet for an example use case.
- Ubuntu-build Vagrant boxes at cloud-images.ubuntu.com/vagrant/
- SlimerJS - PhantomJS-compatible headless browser engine based on Firefox/Gecko (well, it is not fully headless yet but that is planned; the main focus now is full compatibility with PhantomJS' API) (Both work with CasperJS for navigational steps/testing.)
- localtunnel - instantly show locally running webapp/server to the rest of the world (gem install localtunnel, localtunnel <port to share>, => share the url returned, e.g. http://xyz.localtunnel.com) - I haven't tried it but it looks simple and very convenient
- Logstash + Kibana (via @mortenberg): take control of your logs - while Logstash can collect (from multiple servers/services), parse (over 100 built-in patterns), store, index, search your logs, Kibana is a web interface to seach them, view them in realtime (based on a query) etc. See this Logstash slides (9/2012) and an overview of Kibana's powers. PS: Logstash can also compute metrics and send them to graphite etc. It is typically used with ElasticSearch.
- ncdu is an interactive, command-like disk usage browser that shows a list of directories sorted by size shown in human-friendly units, you can navigate with arrows and enter and i to show the current dir/file info, d to delete it, q to quit; check out this article about ncdu with screenshots and ncdu man page. Install via Apt etc., run f.ex. with ncdu -x / .
- vagrant-cachier - Vagrant plugin for caching apt/yum/.. packages locally, thus speeding up destroy+up