jakub holý

building the right thing, building it right, fast


Testing in General

Naming Conventions

Based on Working Effectively with Legacy Code, ch.18:

  • Fake* (ex.: FakeEmailSender) - mocked dependencies
  • Testing* (ex.: TestingOrderController) - test subclasses of production classes (the Sublcass and Override Method pattern)
  • *IT, *Test - integration / unit test classes

Test Doubles: Fake x mock x stub differences explained by Fowler:

  • Dummy objects - passed around but never actually used
  • Fake - a working implementation with shortcuts making it unsuitable for production (e.g. in-memory DB)
  • Stubs - provide configured answers to calls made during test and usually nothing outside of these; may also record information about the calls
  • Mocks - "objects pre-programmed with expectations which form a specification of the calls they are expected to receive" => behavior verification (x state verif.)

TDD & Value of Testing

Do I always need to write tests first? No, it is not suitable for example when exploring or in general when - for same rare reasons - the benefit is less than the cost (-> Economics of Coding).

Do I need 100% coverage? Hell no, what is the value of testing getters/setters? (versus the cost?) See Brian Marick's How to Misuse Code Coverage (also reflected in Martin Fowler's post on Test Coverage).

Kent Beck answering a question about how much testing to do (highlighted by me):

I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence (I suspect this level of confidence is high compared to industry standards, but that could just be hubris). If I don’t typically make a kind of mistake (like setting the wrong variables in a constructor), I don’t test for it. I do tend to make sense of test errors, so I’m extra careful when I have logic with complicated conditionals. When coding on a team, I modify my strategy to carefully test code that we, collectively, tend to get wrong.

Different people will have different testing strategies based on this philosophy, but that seems reasonable to me given the immature state of understanding of how tests can best fit into the inner loop of coding. Ten or twenty years from now we’ll likely have a more universal theory of which tests to write, which tests not to write, and how to tell the difference. In the meantime, experimentation seems in order.

Kent Beck has posted a good summary of the benefits of TDD in "Functional TDD: A Clash of Cultures" – double checking of the logic (by the implementation and the, preferably quite different, test), solution decomposition (focus on part of the problem, once solve be sure it stays solved),  automatic checking of correctness, outside in design (API first, implementation after that). Plus the pleasant experience of the continuous cycle of tension (failing test) – relief (green).

Update: See Do You Know Why You Are Testing?! (On The Principles Underlying TDD)


  • It’s Not About the Unit Tests - Learning from iOS Developers: iOS developers don't do much testing yet they manage to produce high quality. How is that possible? The key isn't testing itself, but caring for the code. (Of course, iOS is little special: small apps, no legacy, a powerful platform that does lot for the apps, very visual apps.)

Unit Testing Toolbox (for Java)

A collection of test-related links to libraries, tools, utilities, concepts etc.

Notes about Java testing

  • Access to tested methods - non-private methods & test in the same directory or private methods accessed via java.lang.reflect and setAccessible http://www.onjava.com/pub/a/onjava/2003/11/12/reflection.html?page=last&x-showcontent=text.
  • JUni 4.4: assertThat and fluent API via Hamcrest (use junit-dep and full Hamcrest to get much more matchers than the few integrated into JUnit)

    • JUnit 4: @Test, @Before, @After, @RunWith(value=Parameterized.class) and @Parameters public static Collection<Object[]> data() { ... }; and public void testXY(Object param...);
  • Great support for testing in Groovy

    • Gmock - very easy to use and very readable code, e.g. mock only selected methods and static methods... . Used inside a GmockTasteCase or GroovyTestCase
    • GroovyTestCase - uses just the standard Java assert instead of the various assertEquals etc. Ex.: assert bluegroupsService.groupExist("noSuchGroup") => java.lang.AssertionError: Expression: bluegroupsService.groupExist(noSuchGroup)
    • ObjectGraphBuilder

Libraries for unit testing

  • Unitils (Java 5) -

    an open source library aimed at making unit testing easy and maintainable. Unitils builds further on existing libraries like dbunit and integrates with JUnit and TestNG. Unitils provides general asserion utilities (e.g. via reflection), support for database testing, support for testing with mock objects and offers integration with [or mocking out] SpringHibernate and the Java Persistence API (JPA). It has been designed to offer these services to unit tests in a very configurable and loosely coupled way. As a result, services can be added and extended very easily. EasyMock integration

    (I'd personally prefer Mockito). See for example ReflectionAssert.assertReflectionEquals, which recursively compares JavaBeans without using their equals, i.e. comparing only primitive values - extremely useful! (Groovy has something similar.)

  • Mocking

    • PowerMock (Java 5) - mj. umí mock pro statické metody; rozšiřuje EasyMock/Mockito použitím custom classloaderu a bytekódové manipulace tkž může mockovat statické, privátní a final metody aj. Podporuje EasyMock a Mockito.
    • JBoss jsf-mock: can be used to create mock objects for all the main JSF entities. Also the library is fully compatible with JSF 2.0
  • Concurrent code testing (some of these are old but it doesn't mean they're worthless!)

    • MultithreadedTC (2007) is a wonderful framework for testing concurrent applications; it enables you to test a particular interleaving of blocking method calls from different threads (once all threads block, it increases its counter and notifies any thread waiting for the counter).
    • junit-toolbox' MultithreadingTester (run concurrently, several times), ParallelRunner (run @Test/@Theory methods in parallel)
    • tempus-fugit's ConcurrentTestRunner (2011)- each @Test runs in its own thread; its ConcurrentRule with @Concurrent(count = #threads) enables you to run a particular test in several threads at once and can be combined with RepeatingRule + @Repeating(repetition = #) to run in multiple times in each thread. It has also other goodies worth checking out.
  • Integration

    • JBoss Arquillian - in-container (integration) testing of JEE
    • Dumbster - mock SMTP server for unit testing (start in @Before, get sent messages, stop)
  • Specific environments

    • LDAP: embedded LDAP for unit testing w/ apacheDS: apacheds-for-unit-tests (v1.0) (or for v1.5, which adds dynamic schema and config.)
    • J(2)EE - Mockrunner “stubs out J2EE by magic”; it has stubs that provide out of the box functionality for JMS, JDBC, Servlet and many other J2EE APIs. It should allow unit testing of code that depends on J2EE APIs outside a container.
  • Various

    • MagicTest: Allow to check multiple exceptional states in 1 method with nearly no overhead (exceptions are catched and stored and you can then say that they're the expected result); an intro article
    • parallel-junit - A small library that lets you run JUnit tests in parallel.
    • NetBeans' Memory Leak Unit Test library (assertGC performs some dirty tricks to make sure that GC is indeed run and, if the weak reference under test hasn't been GC-ed, prints details of who references the object)
    • JBoss Byteman - tool for fault injection into the tested code via AOP and orchestrating multipe threads to exercise possible concurrency issues
  • JUnit addons and extensions

    • TwiP (JUnit 4, Java 5) - parameters for individual methods and more - more flexible than @ParameterizedRunner
  • Mutation testing

    • Jester (and Maven plugin Gester) - Kent Beck: “Why just think your tests are good when you can know for sure? Sometimes Jester tells me my tests are airtight, but sometimes the changes it finds come as a bolt out of the blue. Highly recommended.” An intro article.
  • QuickCheck - The goal of QuickCheck is to replace scenario-based testing with specification-based testing. A QuickCheck-based test tries to cover the laws of a domain whereas classical scenario-based testing can only test the validity for heuristically (manually) picked values. Basically QuickCheck is about generators of data.

Test execution

  • Test Load Balancer - partitions your tests for executing them in parallel in multiple threads or machines while doing also some other interesting stuff like re-ordering them to have the last failed ones first. It makes sure no tests is executed more times and none is omitted. A benefit is that it is standalone tool that can be integrated into private builds, CI etc. It's written in Java but supports also non-JVM languages. Currently (10/2011) it supports for example. JUnit, Cucumber, Ant, Java, Ruby. Support for Maven is in progress. What it does in detail: "Balancer gets a list of all the tests that need to be executed from the build script (after the build script has been invoked, and before tests start running). It then prunes that list to make a subset using the historical test information obtained from the Server. This smaller subset is passed-on to the test framework to execute. Balancer continues to listens to events published by the test framework as these tests execute, to record result and time taken by each test. This data is then posted across to the TLB server and acts as seed data for balancing/ordering future builds."

Testing-Related Links

  • The Way of Testivus - ancient eastern testing wisdom :-) - really worthy brochure for all pragmatic testers
  • Blog Cost of Testing (10/09)  - a two-week one-man experiment in coding with testing with interesting results such as that only 10% of time was spent writing tests even though their extent is 40-50% of the whole codebase (mainly because tests are usually quite simple, no complicated logic etc.)
  • Alberto Savoia: Beautiful Tests (a chapter from Beautiful Code) - a good and inspiring example of the though process behind creating a set of tests. Included: a smoke test, gradual addition of boundary tests, randomized testing, testing of performance characteristics. Conclusion: Writing test code can be every bit creative and challenging and tests document the code’s intended behavior and boundaries; help you improve the code; help you gain confidence in the functionality and performance.
  • Getting Started with Tdd in Java using Eclipse [Screencast] - TDD in practice with application of a number of best practices and interesting ideas regarding testing (such as not having for 1 production class just 1 test but more, one for each specific test setup). It has 50m but is really worth the time
  • xUnitPatterns - one of the elementary sources that anybody interested in testing should read through. Not only it explains all the basic concepts (mocks, stubs, fakes,...) but also many pitfalls to avoid (various test smells such as fragile tests due to Data Sensitivity, Behavior Sensitivity, Overspecified Software [due to mocks] etc.), various strategies (such as for fixture setup), and general testing principles.
  • Infinitest is an interesting tool that brings Continuous Testing to your IDE (similar to Kent Beck's JUnitMax) & an experience report
  • When software testing failes -ARIANE 5 Flight 501 FailureReport by the Inquiry Board
  • Blog: extremely readable test of a web app and even more readable webapp test - really inspiring
  • F. Martinig's 10 + 1 Favorite Unit Testing Articles, 2009
  • Upcoming: Javascript Testing Framework Comparisons

Testing services etc.

Code quality metrics

Bad quality => harder testing, that's why it is here.

  • Testability-explorer by Misko Havery, the Google testing guru - an open-source tool, which analyzes Java bytecode and computes how difficult it will be to write unit tests for the code
  • ckjm— Chidamber and Kemerer Java Metrics - WMC: Weighted methods per class, DIT: Depth of Inheritance Tree, NOC: Number of Children, CBO: Coupling between object classes, RFC: Response for a Class, LCOM: Lack of cohesion in methods. Bonus - Ca: Afferent couplings,NPM: Number of public methods
  • Simian (Similarity Analyser; free for non-commercial use)


Test code samples & discussion


What can be improved with test method name @Test public void empty_credit_card_number_should_return_false ?

  • I believe it should rather be called ...should_be_invalid because we want tests to "tell a story", to be more business logic-oriented ("card invalid") than implementation-oriented ("return false"); with my name it's OK to change the validate(..) method to throw an exception instead of returning false while the original name would need to be changed

Decide in the latest responsible moment

What could be improved with the following 1st test method code in the TDD approach? ... assertFalse(CreditCardValidator.validate("")); ...

  • I believe it's completely unnecessary and harmful to assume and enforce that there 1) exists the class CreditCardValidator 2) and that it has a statically-caled method; what if we later decide that we actually need an instance method instead (which is quite likely)?
  • I think that a better solution is to begin by adding the method validate(..) to the test class itself and let this method decide how to perform the validation -- thus we can start implementation right there before migrating to a suitable target class -- it's no-brainer to switch from static call to instance call because there is just 1 place to change -- this approach allows us to postpone the decision how the vlaidation is called til the latest possible moment => one of the Lean thinking gold principles -- when things become more stable, it's a simple refactoring to inline this helper method