Real Options—a Mindset - an intro into the Real Options approach, which has been quite a hot topic and a transformational way of thinking for a number of inspiring people (Dan North, Liz Keogh etc.). "Real Options help us to better make decisions and commitments with three simple principles: Options have value. Options expire. Never commit early unless you know why." We can "pay" to keep our options open longer, i.e. to avoid commiting prematurely.
Demystifying the CHAOS report's claim of ~ 1/2 features being unused: the Standish Group's CHAOS report has been often quoted for its "finding" that a large percentage of features in applications is never/rarely used. However this claim seems to have never been confirmed, their "research" is reportedly not very scientific and not publicly available for scrutiny. Critique by Laurent Bossavit (2013), Jorge Aranda's Standish, the CHAOS report, and science. Thx to @smalltalk80 for pointing this out! However there is one research, Online Experimentation at Microsoft, that supports the claim, in a different context but the same problem applies to features: "Evaluating well-designed and executed experiments that were designed to improve a key metric, only about one-third were successful at improving the key metric!"
Why Yammer believes the traditional engineering organizational structure is dead - small teams, small projects (2-10 people, 2-10 weeks), no separation into front/middle tier/backend team (=> communication, design obstacle); have instead people specializing in these areas and construct feature teams from them based on the actual needs; engineers, not managers do eng. decisions; all aligned via focusing on the same 3 key metrics. Small projects => constant sense of urgency (and excitement): Often very long projects cause engineers to lose track of the end goal. Think of it in terms of hiking: start fresh & excited, get tired and losing track of the goal, excited again at the end => cut out the middle part, keep them in the exciting state where they can measure progress and see it visually; it’s the only way to maintain urgency and morale. Focus: people alwasy work only at one (short) project at a time (there are special bug-fixing teams for maintenance tasks with people rotating in&out).
Agile development is more culture than process - Why thinking of agile as culture and not just process explains resistance and difficulty in teaching and learning the approach - and should be taught so => 1. Underscore agile values that motivate practice; 2. Identify organization values that compete with agile values, conflict of values; 3. Be sensitive to culture shock.
Mark Zuckerberg's Letter to Investors: 'The Hacker Way' (quite long, you might want to read only "The Hacker Way" part at the end) - about Facebook's "unique culture and management approach" - "Hackers believe that something can always be better, and that nothing is ever complete." "Hackers try to build the best services over the long term by quickly releasing and learning from smaller iterations rather than trying to get everything right all at once." "Instead of debating for days [..], hackers would rather just prototype something and see what works." "Hacker culture is also extremely open and meritocratic." "Many of our most successful products came out of hackathons, [..]." <=> five core values: Focus on Impact (focus on solving the most important problems, be good at finding the biggest problems to work on); Move Fast ("[..] if you never break anything, you’re probably not moving fast enough."); Be Bold ("Building great things means taking risks."); Be Open (=> effort to make as much info as possible visible to all); Build Social Value ("[..] Facebook exists to make the world more open and connected, and not just to build a company. ")
Dave Nicolett: I know how to tie my shoes - on the difficulty of convincing people to try unfamiliar software development techniques - "People change the way they operate when they are experiencing some sort of inconvenience or negative feedback. As long as things are going along reasonably well, people don’t go out of their way to change the way they work." (with few exceptions) You can learn to tie your shoes in a split second, but why to invest the effort? You'd need to set aside assumptions, suppress habits, practice. You can argument there are many inconveniences (bugs, criticism for slow delivery, ...) but "Unfortunately, that’s all pretty normal, and most people in the software field are accustomed to it. They don’t see it as a problem that calls for them to change their practices. Most of them probably have a hard time visualizing a different reality." => Maybe that’s the reason there’s been no satisfactory answer to the question of how to convince people to adopt different practices. We shouldn’t be trying to convince people to do anything. We should be helping people solve their problems and achieve their goals. If they are satisfied with the outcomes they achieve using their current methods, then there is no problem to solve.
Kent Beck: Pace of Progress = Pace of Feedback - '"The pace of my progress is completely constrained by the pace of my feedback". If I want to go faster, it's hard to achieve by going faster. I can almost always optimize my feedback loop, though.' "The second lesson from this episode is that it's not just the duration of the feedback loop that matters, it's also the quality. All week I was working in tiny little iterations. Without producing useful information, though, those iterations could be as small or as large as I liked, I was still just going to spin my wheels." => "The next time I seem to be going slow, I'm going to look at my whole feedback loop--duration, quality and my ability to respond to the information."
What Google Has Learned About How to Hire People - interview results have no relation to actual performance on the job: "We looked at tens of thousands of interviews, and everyone who had done the interviews and what they scored the candidate, and how that person ultimately performed in their job. We found zero relationship. It’s a complete random mess." "Instead, what works well are structured behavioral interviews, where you have a consistent rubric for how you assess people, [..]" 'Behavioral interviewing also works — where you’re not giving someone a hypothetical, but you’re starting with a question like, “Give me an example of a time when you solved an analytically difficult problem.”' Link to an interesting book, Hiring Geeks That Fit.
Cool tech stuff
The Elixir language - Clojure + Ruby + Erlang - a functional meta-programming aware language built on top of the Erlang VM; a dynamic language with flexible syntax with macros support that leverages Erlang's abilities to build concurrent, distributed, fault-tolerant applications with hot code upgrades. First-class support for pattern matching, polymorphism via protocols, etc. (via @bodil)
Random Testing seems to be gaining popularity and looks very interesting; at NDC Oslo, John Hughes has presented how QuickCheck, which generates random sequences of API calls, has been successfully used to find bugs in the Riak DB and a file system that a human would never think of, and Stuart Halloway has presented simulation testing with Simulant, which runs predefined actions according to a probabilistic model (e.g. 100 traders, each having 1h mean time between trades and mean traded amount 100, the test runs for 4 simulated hours). Something worth exploring!
The Secret To 10 Million Concurrent Connections - The Kernel Is The Problem, Not The Solution: To have really fast SW, you need to implement your own core services (FS, net driver (packet handling), thread scheduling, ..) tuned for your app. You need to be aware of the clock-time cost of cache misses, memory access etc.. Custom solutions are times faster than what the general OS kernel can offer. => "data plane oriented system" Core areas and solutions for them: packet scalability, multi-core scalability (locks are expensive), memory scalability.
M. Fowler: EmbeddedDocument - a pattern for working with JSON flowing in/out of our services (REST <-> JSON-friendly DB) without unnecessary conversions but with good encapsulation; naive approach: json -> object graph -> (processing) -> json; "In many of these situtiations a better way to proceed is to keep the data in a JSONish form, but still wrap it with objects to coordinate manipulation." - use a lib to parse the JSON into a generic structure (e.g. a structure of lists, and maps/dicts) and store in a field of an object defining methods that encapsulate it - f.ex. for an Order we could have a method returning the customer and another computing the cost, accessing the underlying generic structure. The user of the wrapper object doesn't need to know/care about the underlying structure. "The sweet spot for an embedded document is when you're providing the document in the same form that you get it from the data store, but still want to do some manipulation of that data. [..] The order object needs only a constructor and a method to return its JSON representaiton. On the other hand as you do more work on the data - more server side logic, transforming into different representations - then it's worth considering whether it's easier to turn the data into an object graph."
Wired.com, Nassim Taleb: Beware the Big Errors of 'Big Data' - in big data, noise has much stronger effect and in a large enough dataset we will always find spurious (i.e. false) relationships => beware! "Well, if I generate (by simulation) a set of 200 variables — completely random and totally unrelated to each other — with about 1,000 data points for each, then it would be near impossible not to find in it a certain number of “significant” correlations of sorts. But these correlations would be entirely spurious."
A Taste of Salt: Like Puppet, Except It Doesn’t Suck - a deescription of Salt and the tools around by an enthusiastic user with deep experience with Puppet. Highlights: Light-weight communication over ZeroMQ, very active community, simplicity, configuration is YAML, Salt-cloud can spin instances in EC2/Openstack/..., Salt-virt does the same for virtual machines (KVM/Xen/...), Salt-vagrant, Salt-monitor (work in progess) can ask all the server for their stats. "Having stood up a number of different configuration management systems across a wide variety of environments, I’ve yet to find a solution that’s as rapid to deploy, simple to scale, or as well architected as Salt."
Robin Ward: AngularJS vs Ember - a nice overview of the different approaches of the two; the author is strongly pro-Ember, claiming that AngularJS is much closer to low-end libraris like Backbone/Knockout and that you will often need the additional features of Ember. The comments provide the right countrweight to the biased post and form thus a good whole together.
After Your Job Is Gone - an interesting essay on the future, which, according to the author, we can already see happening, when technology will take away most of our work and we will not need to work all day. Not very optimistic, though (the author predicts few reach and many poor people).
Clojure Cup 2013, Sept 28-29 - create something cool with Clojure/ClojureScript within 48h and perhaps win a price! #fun
Stuart Sierra's My Clojure Workflow, Reloaded (6/2013) - mainly about reloading changes into REPL, working around things that are not reloaded/left over => restart the app from scratch after significant changes => the app as a transient object => no global state, careful management of resources, :dev profile with :source-paths to a dir with user.clj (autoloaded by repl, pre-loading useful stuff) and dev util deps
In Clojure-based Machine Learning: "Our backend is 99.4% coded in Clojure, and 66% of the team [of 3] had never programmed seriously in any Lisp, let alone Haskell or Prolog (heck, not even I (the remaining 33%) had actually tried anything non-mainstream for real in a big project!) Maybe some Ruby, and lots and lots of Java and C and C++. But they accepted the challenge after reading around and learning the basics, and 3 months later you couldn’t take Clojure from their prying hands."
J. Pither: TDD and Clojure - "If you were to create a shopping list of things you really want for your development experience then what would you put at the top?" => 1. rapid feedback on changes, 2. REPL (place to explore and to play with your code <=> TDD), 3. FP and Immutability ("FP and dynamic languages lead to a lot less code. There’s less ceremony, less modeling. Because you’re managing less code you do less large scale refactorings." => TDD needed less), 4. Regression Tests ("It’s my current opinion that what you get left out of TDD once you have amazingly fast feedback and a REPL is regression testing.")
John Hughes and Stuart Halloway had very interesting talks about random testing at NDC Oslo, a topic I have been ignorant of but want to explore more now. Contrary to the typical example-based unit tests where the developer specifies inputs, interactions, and specific validations, random testing generates random input data and/or sequences of interactions and the verification is based on more general checks. Random testing can check many more cases than a developer would ever write and cases that a human would never think of. It can thus discover defects impossible to find by the traditional testing, as has been demonstrated f.ex. on Riak.
Random testing typically starts by creating (likely a very simplified) model of the system under test. The model is then used to generate the random data inputs and/or sequences of actions (method calls). Then the tests are executed and their input and output data captured. Finally the results are validated, either against predefined "system properties," i.e. invariants that should always hold true, or manually by the developer.
Highlights from Dan North's excellent, inspiring, and insightful talk Patterns of Effective Delivery at RootConf 2011. North has a unique take on what agile development is, going beyond the established (and rather limitied and rigid) views. I really recommend this talk to learn more about effective teams, about North's "shocking," beyond-agile experience, and for great ideas on improving your team.
The talk challenges the absolutism of some widely accepted principles of "right" software development such as TDD, naming, the evilness of copy&paste. However the challenge is in a positive way: it makes us think in which contexts these principles really help (in many) and when it might be more effective to (temporarily) postpone them. The result is a much more balanced view and better undestanding of their value. A lot of it is inspired by the theory (and practice) of Real Options.
What are Patterns of Effective Delivery?
Patterns - strategies that work in a particular context - and not in another (too often we forget the context and to consider the context where a strategy doesn't work / is contra-productive); beware: a part of the context is the experience of the developer; for unexperienced devs it might be better to just stick to a process and applying TDD etc. all the time than trying to guess when they are appropriate and when not without having the experience to decide it right
Effective - optimize for something: volume of SW produced? time to market? learning/discovery? certanity? user experience?
Delivery - get stuff that is useful out of the door; software is not important, the utility it provides is; know why you write the SW to be able to get better at it
Programmer Competency Matrix - a nice overview of a programmer needs to know in different areas (computer science: data structures etc., SW engineering: SCM etc., programming: defensive coding, IDE etc.) + what experience and knowledge they need, split into 4 levels. Where are you?
The 12 most disruptive technologies of the next 10 years according to McKinsey - 1. mobile internet, 2. automation of knowledge work, 3. internt of things, 4. cloud, 5. advanced robotics, 6. [near-]autonomous vehicles, 7. next-generation genomics, 8. energy storage, 9. 3D printing, 10. advanced materials, ...; the full 178 page report.
Discussion: World's Biggest 'Agile' Software Project Close To Failure - what is/isn't agile, agile vs. waterfall etc; a nice collection of all the possible opinions and misunderstandings. Some of my favorites: waste: '[..]An "agile" project cannot fail and cost Billions because it must always deliver runnable software with a maximum of a few weeks delay[..]' (runnable = delivering value), separation: '[..] my experience has been that separating the designer/architect role from the developer role is fraught with pitfalls. The people writing the code should be the ones designing it, [..]', stability: '[..] On the successful projects that I've worked with in Agile, there's strong stakeholders, good architecture keeping the vision in place and project management that keeps things well orchestrated. Without those in the mix, it'll fail just like all software projects. [..]', waterfall success, people issue: 'The problem here isn't waterfall/agile. The problem here isn't .Net/Linux. The problem here is the parties involved. [politicians and IT dinosaurs]' (learn what's needed from drunk, bitching employees; ignore official nonsense requirements),: simplicity '[..] It would probably be a lot easier if they started by making a simpler tool - instead of trying to calculate everybody's entitlements everywhere [..]', agile suitability: '[..] You cannot use Agile to build a 100-mile canal, as the whole thing would be useless even if you completed 99 miles. [..]'.
Some people seem to believe that agile means no architecture and no/too little planning. Some believe that agile = hack now, fix later.
Clojure in the Enterprise? - about the differences between the Clojure and the enterprise java (with heavy frameworks such as JPA, JSF, mutable state) ways and difficulties of introducing Clojure due to old-fashioned thinking, limited skills, etc. "Take away objects, mutable state, variables, loops... a lot of Java developers are immediately all at sea and have no idea how to solve even basic problems. They'll try to bend Clojure to their OOP way of thinking and they'll most likely fail."
World Singles' experience with Clojure: "We love Clojure. It's made development a lot more fun. We're able to solve harder problems, make changes faster, leverage multi-core concurrency more effectively, and we have a much smaller code base to maintain."
Replace Temp with Query in Clojure - in this post we can follow an interesting refactoring from a deeply nested if & let code to a much flatter one (featuring cond to test multiple conditions (instead of guard conditions used in imperative languages) delay to be ably to bind an expression to a local variable without evaluating it yet)
Reconstructing Clojure Macros With Speclj - a good idea to learn macros by trying to create one's own implementation of existing Clojure macros, driven by tests that define the expected behavior of the macro (using macroexpand, macroexpand-1, macroexpand-all)
This is an experiment and I hope to hear from the community if this is something that can really help people or not. I would also love to get contributions from other developers, to cover a broader range of opinions and examples. Comments and contributions are welcomed and appreciated!
This book is about a great person, about change, about one of the largest bureaucracies and dysfunctional organizations, about projects gone astray, about warfare and its latest evolution. Many of the challenges and ideas that we encounter in the book are not limited to the military domain but apply also to business and IT. It is worth reading whether you are a military person, somebody trying to push through a change, a business person, or interested in thinking and organizations.