Holy on Dev

Archive for October, 2012

Most interesting links of October ’12

Posted by Jakub Holý on October 31, 2012

Recommended Readings

  • David Veksler: Some lesser-known truths about programming – things newcomers into the field of IT don’t know and don’t expect, true and an interesting read. Not backed by good data but anyway. F.ex.: “[..] a programmer spends about 10-20% of his time writing code [..] much of the other 90% thinking, researching, and experimenting”. “A good programmer is ten times more productive than an average programmer. A great programmer is 20-100 times more productive than the average [..]” “Bad programmers write code which lacks conceptual integrity, non-redundancy, hierarchy, and patterns, and so is very difficult to refactor.” “Continuous change leads to software rot, which erodes the conceptual integrity of the original design.” “A 2004 study found that most software projects (51%) will fail in a critical aspect, and 15% will fail totally.”
  • Brett L. Schuchert: Modern Mocking Tools and Black Magic – An example of power corrupting – interesting for two reasons: a good analysis of a poorly written piece of code and discussion of the code injection black magic (JMockIt) vs. actually breaking dependencies to enable tests.  The author presents a typical example of low-quality method (mixing multiple concerns, mixing different levels of abstractions, untestable due to a hardcoded use of an external call) and discusses ways to improve it and to make it testable. Recommended to read.
  • It’s Not About the Unit Tests – Learning from iOS Developers: iOS developers don’t do much testing yet they manage to produce high quality. How is that possible? The key isn’t testing itself, but caring for the code. (Of course, iOS is little special: small apps, no legacy, a powerful platform that does lot for the apps, very visual apps.) “It’s not about the practices. It’s about the spirit and intent behind them, and how they are applied.” (M. Fowler had a similar observation about a team that used mock-based testing exclusively and thus lacked integration tests yet all worked. [I’ve lost the link to the post and would be grateful for it])
  • Java Code Quality Tools – Overview – brief descriptions of 44 quality-related tools including some interesting tools and Eclipse plugins I didn’t know or knew but forgot. F.ex. analysis of dependencies with JBoss Tattletale or JarAnalyzer, Clirr to check libraries for source and binary backwards compatibility, JDiff generates JavaDoc-based report of removed/added/changed in an API. Spoon – read and check or transform Java code. Java PathFinder (NASA) – special JVM capable of checking all execution path to discover concurrency defects etc.


  • DirB, Directory Bookmarks for Bash (home) – moving efficiently among favourite directories (s <name> to create a bookmark for pwd, g <bookmark | relative/abs dir path> to enter a dir (=> works both for bookmarks and as a replacement for cd); also support for relative path bookmarks & more; sl lists bookmakrs in the last used order) (You might also want to check out Autojump, described in Dec 11; bashmarks is another similar project. Another similar project is rupa’s z and j2 and the fish clone z-fish)

Clojure Corner

  •  Jon Pither: Clojure at a Bank – Moving from Java –  the justification (productivity, dynamism, FP a better match for the domain) and process behind moving from Java to Clojure with a monolithic 1M LOC Spring/Hibernate app. (Random quotes: “I had used some dynamical languages before and it was quite obvious that we were essentially forcing lots of schema and type definition on to a problem domain that just didn’t want or need it.” “[..] it [dependency injection] just looks redundant in retrospect now that I’m working 95% with FP code.”) There is also a EuroClojure talk about their experiences one year later (35 min).
  • Prismatic’s “Graph” at Strange Loop – an interesting desing problem, its solution, and a resulting OSS library. The problem: How to break a large function into independently usable small ones that might depend on each other without ever needing to recompute a value once the function producing is called. The solution: Graph – “Graph is a simpledeclarative abstraction to express compositional structure.” (Enabling explicit declaration of data dependencies and pluging in different implementations.)
  • The Oblong: Blog about 2/3 D game programming in Clojure, starting from scratch (w/o an engine); interesting experiences
  • Ironclad: Steam Legions – Clojure game development battle report (the game on Github)
  • Building the Wishlisted.org webapp in Clojure – experiences from learning Clojure for real by building a webapp in Noir
  • Clojure vs. Scala smackdown (“Just kidding with the title of this post :-)”) – a short post with interesting discussion. Dmitri Sotnikov’s opinion resonates with me: “I found that for me Clojure wins on simplicity and consistency. While it looks more alien initially, once you learn the basics, you just reuse the same patterns everywhere.” Some more comments: “One major concern was maintainability, since it’s fairly easy to write very dense code. This turned out to not be a problem in practice. Because Clojure code is written as a tree, refactoring it is very easy.” REPL seems to be a big win (applies to Scala too). Scala’s type system might get tedious and learning its quirks takes time but there is lot of potential and both have they strong sides.
  • Code Fatigue – discussion of the advantages of learning, using, and combining the (many) standard Clojure functions instead of a “basic solution” using recursion etc. The argument is in favor of higher-level code with less complexity in the form of branching, recursion, nested expressions etc. and thus less mental fatigue.

Favorite Quotes

A classic test only cares about the final state – not how that state was derived. Mockist tests are thus more coupled to the implementation [emphasis mine] of a method. Changing the nature of calls to collaborators usually cause a mockist test to break.

– Martin Fowler in his classical Mocks Aren’t Stubs

I’m afraid of code. When I see a big pile of code, I get scared ;-). Some classes and method make me cry. I had troubles explaining why I prefer short pieces of code keeping the same level of abstraction, cohesive and loosely coupled. The following quote captures the essence – improved communication.

One way to improve communication is to reduce the need for it and the same can be said for code. […] Since we tend to read code more than write it, anything we can do to reduce the need to read code is time well invested in the life of a project.

– Brett L. Schuchert in Modern Mocking Tools and Black Magic – An example of power corrupting justifying extraction of code into another class or method

Posted in General, Languages, Testing, Tools, Top links of month | Tagged: , , , , , , | Comments Off on Most interesting links of October ’12

Do You Know Why You Are Testing?! (On The Principles Underlying TDD)

Posted by Jakub Holý on October 27, 2012

Kent Beck in his recent post Functional TDD: A Clash of Cultures summarizes well the key principles and benefits that underlie test-driven development. I think it is really worthwhile becoming aware of and thinking over these foundation stones of TDD (and testing in general). Knowing them enables you to apply TDD in the most effective way with respect to a particular context to gain the maximum of these benefits. People that do not really understand the value of their tests and TDD tend to write hard to maintain tests of limited (and occasionally even negative) value. Are you one of them?

Read the rest of this entry »

Posted in Testing | Tagged: , , | Comments Off on Do You Know Why You Are Testing?! (On The Principles Underlying TDD)

Tip: Import Leiningen Project to IntelliJ (With Dependencies)

Posted by Jakub Holý on October 27, 2012

To import a Leiningen-based project into IntelliJ with the Clojure plugin, the best way seems to be:

  1. Run lein pom to generate a Maven pom.xml from project.clj
  2. Import the project as a Maven project (File – New Project… – Import project from external model – Maven – browse to the directory – …)

Versions: Leiningen 2.0.0-preveiw10, IntelliJ 11. You don’t need the Leiningen plugin for IntelliJ (it seems to be useful only for running the leiningen tasks, which you can well do from the command line).

If you change the dependencies, regenerate the pom.

(Notice that Leiningen 1 did download all the dependencies into ./lib/ when executing lein deps but that doesn’t work anymore in Leiningen 2.)

Posted in Tools | Tagged: , , | Comments Off on Tip: Import Leiningen Project to IntelliJ (With Dependencies)

Tool Tip: Byob – Screen With Text UI

Posted by Jakub Holý on October 17, 2012

Screen (man) is very useful for running terminal sessions on remote computers that enable the user to disconnect and re-connect. Byobu (man), formerly also called screen-profiles, is a wrapper script for screen that adds status lines with useful info to screen and provides text UI for configuring it (byobu-config).

Screen allows you to have multiple “windows” (terminal sessions) opened at the same time and jump between them (C-a n). You can also display multiple windows if you split your screen.

Few screen/byobu tips (C-a means Contol-a):

  • Configuration
    • Increase scroll history via “screen -h <num lines>” or “defscrollback <num lines>” in ~/.screenrc
  • Detach from screen: C-a d
  • Re-attach to screen: screen -rd
  • Split vertically: C-a |
    • Jump to the new split (or back): C-a tab
    • Create new window in the split (=> shell): C-a c
    • Remove all splits but the current one: C-a Q
  • Next window: C-a n
  • Pass “C-a” to the terminal running under screen: “C-a a”
  • Help “C-a ?”
  • Byobu: disable the Fn keys and only use the traditional screen commands: run byobu-config and select to use only them
    • To access Bybou menu (config) type, instead of F9, C-a @

Posted in Tools | Tagged: | Comments Off on Tool Tip: Byob – Screen With Text UI

Puppet Troubleshooting: Compiling Catalog, Locating a Cached Catalog

Posted by Jakub Holý on October 17, 2012

Few troubleshooting tips for Puppet.

Where to Find the Cached Catalog On Client

Puppet Agent caches its compiled (text/Ruby) catalog under its $vardir, for example Puppet 0.25.x stores it into /var/lib/puppet/client_yaml/catalog/<hostname>.yaml. It might be useful when troubleshooting.

Compiling Catalog Manually

You can compile the catalog for a particular node manually on the Puppet Master. Ex.:

puppetmasterd --compile mynode.example.com > mynode_catalog

The compilation uses locally cached facts, typically in /var/lib/puppet/yaml/facts/<hostname>.yaml .

You can try the catalog manually on the node (though retrieving files via puppet://puppet/… will fail). You will first need delete the leading lines with messages/warnings from the compilation process, if any (such as “[warning] …”):

puppet --noop --apply mynode_catalog

Posted in Tools | Tagged: , | Comments Off on Puppet Troubleshooting: Compiling Catalog, Locating a Cached Catalog

My Scala vs. Clojure Impression In Pictures

Posted by Jakub Holý on October 13, 2012

(By kristobalite)


(By agiamba)


Clojure has a zen-like quality to it. There is extreme focus on simplicity, on defining few elementary orthogonal concepts that can be combined in powerful ways. For example it took 3 years for Clojure to get named parameters – but the result, destructuring, is something much richer and more applicable, that fits the language perfectly and has become a core part of idiomatic Clojure.

Scala feels as if trying to empower the programmer in any possible way, throwing in shortcuts, syntactic sugar, and plenty of methods so that anything can be done with the minimal amount of code. It is overwhelming.

Disclaimer: I have only a few weeks of experience with Scala and are in no position to judge it. This is just an impression I have gained so far and nothing more. I’m sure it is a great language.

Posted in General, Languages | Tagged: , , , | 1 Comment »

Note: Loading Tab-Separated Data In Cascalog

Posted by Jakub Holý on October 9, 2012

To load all fields from a tab-separated text file in Cascalog we need to use the generic hfs-tap and specify the “scheme” (notice that loading all fields and expecting tab as the separator is the default behavior of TextDelimited):


With a custom separator and fields:

   (cascading.scheme.hadoop.TextDelimited. (cascalog.workflow/fields ["?f1" "?f2"]) "\t") ; or cascading.tuple.Fields/ALL inst. of (fields ...)

Hadoop doesn’t manage to load data files from nested sub-directories (for example from a Hive partitioned table). To load them, you need to use a “glob pattern” to turn the standard Hfs tap into a GlobHfs tap. This is how we would match all the subdirectories (Hadoop will then handle loading the files in them):

   :source-pattern "epoch_week=*/")


Posted in General, Languages, Tools | Tagged: , , , | Comments Off on Note: Loading Tab-Separated Data In Cascalog