Most interesting links of April '13

Recommended Readings

The top top article

How To Survive a Ground-Up Rewrite Without Losing Your Sanity (recommended by Kent Beck) - sometimes you need to actually rewrite an important part of a system; here we learn about two such rewrites, one which went well and one that failed badly - and what are the important differences.

The pain of a rewrite: "it's [a major rewrite] going to take insanely longer than you expect" - because: "there's this endless series of weird crap encoded in the data in surprising ways" and it takes days to convert them, "It's brutally hard to reduce scope" (you cannot drop features, edge cases), "There turn out to be these other system that use 'your' data".

To succeed you need: 1) Determine clear business-visible wins to justify the effort that will be much higher than expected and to know when to give up / what to postpone; 2) Do it extremely incrementally (<->  Succession) - break it into a series of small, safe steps, each generating a business value and learning of its own thus enabling early and frequent economical tradeoffs (stop, shift priorities, ...) - ex.: rewrite a single reports, migrate its data, switch customers to it, go on to the next one - complete slice of functionality => a more realistic estimate soon => reprioritisation; incrementalism requires you to be able to write data both to the old and new system, which is hard but always pays off: "Here's what I'm going to say: always insert that dual-write layer. Always. It's a minor, generally somewhat fixed cost that buys you an incredible amount of insurance." 3) "Abandoning the Project Should Always Be on the Table" (<- known biz value, better estimate based on early feedback).

Some Specific Tactics: Shrink Ray FTW (a graph of how much has been already replaced => motivation), Engineer The Living Hell Out Of Your Migration Scripts (tests, robustness, error handling, restartability), If Your Data Doesn't Look Weird, You're Not Looking Hard Enough.

Methodology, agile, lean

• M. Fowler: The New Methodology - a good description of the rise of Agile, the motivation for it, the various Agile methodologies (XP, Lean, Scrum etc.) and what is required to be able to apply an agile approach. Main points: Agile is adaptive (vs. predictive) and relies heavily on people and their judgement and skills (vs. treating them as same, replacable units) - which also leads to the need of leadership instead of (command&control) management. Discusses unpredictability of requirements and scope, foolishness of separating design and implementation, difficulty of measurement of SW development, continuous improvement etc. Quotes: "However letting go of predictability doesn't mean you have to revert to uncontrollable chaos. Instead you need a process that can give you control over an unpredictability. That's what adaptivity is all about."
• The Toyota concept of 'respect for people' - many state that they respect their workers but fail to really understand what it means; it is not about freedom of act, it is about a mutual respect, leveraging the strengths of each other: worker's experience and insight and manager's broader overview, as demonstrated by the problem-solving dialog and challenges (problem - root cause - solution - measure of success, the manager challenging the worker's answers). Also a nice example how the evaluation of individual performance leads to a much worse system and high turnover compared to a whole-oriented company.
• Fixed Bid Agile Without Cognitive Dissonance - a refreshing take on fixed-scope projects and Agile; yes, they are bad but sometimes the client has no other choice so what best we can make out of it? The core advice: Agree "a pragmatic change management protocol (along with a contingency built into the pricing)" (push for lower initial requirements granularity, customer involvement, flexibility of functionality) => "you can gain significant agile benefits for clients who wouldn't otherwise accept them".
• Agile Atlas: Scrum - a good description of Scrum and its values, roles, artifacts, and activities

Learning, psychology, estimates

• How Developers Stop Learning: Rise of the Expert Beginner - sometimes you meet people with experience-indicating titles that are actually little competent, perhaps leading incompetent IT departments. Why? They, unchallenged by competent peers or broader IT community, came to believe that they are "experts" while actually being only little more advanced beginners, better than their beginner colleagues but still lacking any understanding of the big picture and the knowledge of what they do not know, trapped in the "unconscious incompetence" stage. The post explains this in a more detail and is followed up an explanation how it can lead to the rise of a mediocre SW group in "How Software Groups Rot: Legacy of the Expert Beginner".
• Coding, Fast and Slow: Developers and the Psychology of Overconfidence (via @peterskeide) - why are we so bad at estimating (inherent complexity of SW vs. our overconfidence) and why it cannot be fixed. We can learn to somehow estimate tasks of few hours length (less complex, plenty of practice opportunities). The question is: "how you can your dev team generate a ton of value, even though you can not make meaningful long-term estimates?"
• Cognitive Overhead, Or Why Your Product Isn’t As Simple As You Think (via @JiriJerabek) - to make apps more accessible to users, we try to make them simple - but "simple" might be different from what you expect. The important thing is not less steps, less features, less elements, but lower cognitive overhead, i.e. “how many logical connections or jumps your brain has to make in order to understand or contextualize the thing you’re looking at.” Good examples of unexpectadly high / pleasantly low cognitive overhead, some tips, even suprising ones such as make people do more (to be more involved in the process - e.g. bump their phones), slow down your product.

Other

• Economies of Scala - a case for using Scala over Java, supported by data: many capable developers want to use it but there are few opportunities for them - and getting developers is one of the main challenges.
• A canonical Repository test - a nice standard way to test a "DAO"; highlights: use of  FEST assert 2 for clean and nice checks, no unimportant details in the test (f.ex. details of the test data hidden in randomPerson() and randomOder(Person)).
• How To Think Like An Engineer - some nice ideas such as: "Build A Simple First Version: With People, Not Code" - "Technology is not always the best solution, because technology is not always the simplest solution.", i.e. don't automate everything from the start (examples from Netflix, Amazon); "Rather than trying to do everything at once, break down the functions of your company into smaller goals." - and focus at one at a time
• Economies Of Scale As A Service (do not mix up with Scala! :-))- an interesting description of the trend away from ownership to the rental of important resources (servers, manufacturing capabilities, personal cars, ...) and the resulting changes in the society, business, and industry
• Troy Hunt: Our password hashing has no clothes (or the much shorter though biased How To Safely Store A Password) - MD5 and SHA are not safe enough due to brute-force attack enabled by GPUs, irrespective key size; it's crucial to use hashing algorithms designed for passwords (and thus sufficiently slow) - f.ex. bcrypt, or PBKDF2 or the newer scrypt.
• Everything about Java 8 - a well-made summary of what should come in Java 8, based on the current state, discussing the finer points: static and default (non-static, overridable) methods on interfaces, lambdas (do I need to mentione that?!) and method references (String::valueOf, Object::toString, myVar::toString, ArrayList::new); good discussion of the various use cases and limitations of lambdas (capturing x non-c., ..); java.util.stream for functional operations on value streams (filter, map, reduce etc.); java.time inspired by Joda, more concurrency utilities (e.g. CompletableFuture for chaining futures); String.join (finally!), Optional ~ Scala's Option & more; yummy!
• How To Keep Your Best Programmers - what motivates capable programmers to stay/leave? The author lists some common reasons and concludes that, ultimately, all are linked to the desire for autonomy, mastery, or purpose. However he goes further and proposes that, to keep talented devs, you must offer them an appealing narrative (regarding their actions and a result, related to autonomy/mastery/purpose) and reaffirm/update it frequently; ex.: “With the work that we’re giving you over the next few months, you’re going to become the foremost NoSQL expert in our organization.” "At any point, both you and the developers on your team should know their narratives." - so that they will be "constant points of job satisfaction and purpose."

Clojure Corner

• Clojure Data Analysis Cookbook review - "The book provides a collection of recipes for accomplishing common tasks associated with analyzing different types of data sets. It starts out by showing how to read data from a variety of sources such as JSON, CSV, and JDBC. [..] how to sanitize the collected data and sample large data sets. [..] a number of different strategies for processing it." How to present them with ClojureScript and  NVD3 (D3.js components). "Some of the highlights include using the Clojure STM, parallel processing of the data, including useful tricks for partitioning, using reducers, and distributed processing with Hadoop and Casalog."

Favorite Quotes

once again, trying to do it *and* do it right was too much all at once, resulting in little progress and little learning.

- Kent Beck's tweet 2013-04-16

A true agile development process can be recognized by its continual evolution:

A project that begins using an adaptive process won't have the same process a year later. Over time, the team will find what works for them, and alter the process to fit.

- Martin Fowler in The New Methodology

Tags: learning clojure methodology human data