Testing
- Testing in General
- Unit Testing Toolbox (for Java)
- Testing-Related Links
- Test code samples & discussion
Testing in General
Naming Conventions
Based on Working Effectively with Legacy Code, ch.18:
Fake* (ex.: FakeEmailSender) - mocked dependencies
Testing* (ex.: TestingOrderController) - test subclasses of production classes (the Sublcass and Override Method pattern)
*IT, *Test - integration / unit test classes
Test Doubles: Fake x mock x stub differences explained by Fowler:
Dummy objects - passed around but never actually used
Fake - a working implementation with shortcuts making it unsuitable for production (e.g. in-memory DB)
Stubs - provide configured answers to calls made during test and usually nothing outside of these; may also record information about the calls
Mocks - “objects pre-programmed with expectations which form a specification of the calls they are expected to receive” ⇒ behavior verification (x state verif.)
TDD & Value of Testing
Do I always need to write tests first? No, it is not suitable for example when exploring or in general when - for same rare reasons - the benefit is less than the cost (→ Economics of Coding).
Do I need 100% coverage? Hell no, what is the value of testing getters/setters? (versus the cost?) See Brian Marick’s How to Misuse Code Coverage (also reflected in Martin Fowler’s post on Test Coverage).
Kent Beck answering a question about how much testing to do (highlighted by me):
I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence (I suspect this level of confidence is high compared to industry standards, but that could just be hubris). If I don’t typically make a kind of mistake (like setting the wrong variables in a constructor), I don’t test for it. I do tend to make sense of test errors, so I’m extra careful when I have logic with complicated conditionals. When coding on a team, I modify my strategy to carefully test code that we, collectively, tend to get wrong.
Different people will have different testing strategies based on this philosophy, but that seems reasonable to me given the immature state of understanding of how tests can best fit into the inner loop of coding. Ten or twenty years from now we’ll likely have a more universal theory of which tests to write, which tests not to write, and how to tell the difference. In the meantime, experimentation seems in order.
Kent Beck has posted a good summary of the benefits of TDD in “Functional TDD: A Clash of Cultures” – double checking of the logic (by the implementation and the, preferably quite different, test), solution decomposition (focus on part of the problem, once solve be sure it stays solved), automatic checking of correctness, outside in design (API first, implementation after that). Plus the pleasant experience of the continuous cycle of tension (failing test) – relief (green).
References
It’s Not About the Unit Tests - Learning from iOS Developers: iOS developers don’t do much testing yet they manage to produce high quality. How is that possible? The key isn’t testing itself, but caring for the code. (Of course, iOS is little special: small apps, no legacy, a powerful platform that does lot for the apps, very visual apps.)
Unit Testing Toolbox (for Java)
A collection of test-related links to libraries, tools, utilities, concepts etc.
Notes about Java testing
Access to tested methods - non-private methods & test in the same directory or private methods accessed via java.lang.reflect and setAccessible http://www.onjava.com/pub/a/onjava/2003/11/12/reflection.html?page=last&x-showcontent=text.
JUni 4.4: assertThat and fluent API via Hamcrest (use junit-dep and full Hamcrest to get much more matchers than the few integrated into JUnit)
JUnit 4: @Test, @Before, @After, @RunWith(value=Parameterized.class) and @Parameters public static Collection<Object[]> data() \{ … }; and public void testXY(Object param…);
Great support for testing in Groovy
Gmock - very easy to use and very readable code, e.g. mock only selected methods and static methods… . Used inside a GmockTasteCase or GroovyTestCase
GroovyTestCase - uses just the standard Java assert instead of the various assertEquals etc. Ex.: assert bluegroupsService.groupExist(“noSuchGroup”) ⇒ java.lang.AssertionError: Expression: bluegroupsService.groupExist(noSuchGroup_)_
Libraries for unit testing
Unitils (Java 5) -
an open source library aimed at making unit testing easy and maintainable. Unitils builds further on existing libraries like dbunit and integrates with JUnit and TestNG. Unitils provides general asserion utilities (e.g. via reflection), support for database testing, support for testing with mock objects and offers integration with [or mocking out] Spring, Hibernate and the Java Persistence API (JPA). It has been designed to offer these services to unit tests in a very configurable and loosely coupled way. As a result, services can be added and extended very easily. EasyMock integration
(I’d personally prefer Mockito). See for example ReflectionAssert.assertReflectionEquals, which recursively compares JavaBeans without using their equals, i.e. comparing only primitive values - extremely useful! (Groovy has something similar.)
Mocking
PowerMock (Java 5) - mj. umí mock pro statické metody; rozšiřuje EasyMock/Mockito použitím custom classloaderu a bytekódové manipulace tkž může mockovat statické, privátní a final metody aj. Podporuje EasyMock a Mockito.
JBoss jsf-mock: can be used to create mock objects for all the main JSF entities. Also the library is fully compatible with JSF 2.0
Concurrent code testing (some of these are old but it doesn’t mean they’re worthless!)
MultithreadedTC (2007) is a wonderful framework for testing concurrent applications; it enables you to test a particular interleaving of blocking method calls from different threads (once all threads block, it increases its counter and notifies any thread waiting for the counter).
junit-toolbox’ MultithreadingTester (run concurrently, several times), ParallelRunner (run @Test/@Theory methods in parallel)
tempus-fugit’s ConcurrentTestRunner (2011)- each @Test runs in its own thread; its ConcurrentRule with @Concurrent(count = #threads) enables you to run a particular test in several threads at once and can be combined with RepeatingRule + @Repeating(repetition = #) to run in multiple times in each thread. It has also other goodies worth checking out.
Integration
JBoss Arquillian - in-container (integration) testing of JEE
Dumbster - mock SMTP server for unit testing (start in @Before, get sent messages, stop)
Specific environments
LDAP: embedded LDAP for unit testing w/ apacheDS: apacheds-for-unit-tests (v1.0) (or for v1.5, which adds dynamic schema and config.)
J(2)EE - Mockrunner “stubs out J2EE by magic”; it has stubs that provide out of the box functionality for JMS, JDBC, Servlet and many other J2EE APIs. It should allow unit testing of code that depends on J2EE APIs outside a container.
Various
MagicTest: Allow to check multiple exceptional states in 1 method with nearly no overhead (exceptions are catched and stored and you can then say that they’re the expected result); an intro article
parallel-junit - A small library that lets you run JUnit tests in parallel.
NetBeans’ Memory Leak Unit Test library (assertGC performs some dirty tricks to make sure that GC is indeed run and, if the weak reference under test hasn’t been GC-ed, prints details of who references the object)
JBoss Byteman - tool for fault injection into the tested code via AOP and orchestrating multipe threads to exercise possible concurrency issues
JUnit addons and extensions
TwiP (JUnit 4, Java 5) - parameters for individual methods and more - more flexible than @ParameterizedRunner
Mutation testing
Jester (and Maven plugin Gester) - Kent Beck: "`Why just think your tests are good when you can know for sure? Sometimes Jester tells me my tests are airtight, but sometimes the changes it finds come as a bolt out of the blue. Highly recommended.`" An intro article.
QuickCheck - The goal of QuickCheck is to replace scenario-based testing with specification-based testing. A QuickCheck-based test tries to cover the laws of a domain whereas classical scenario-based testing can only test the validity for heuristically (manually) picked values. Basically QuickCheck is about generators of data.
Test execution
Test Load Balancer - partitions your tests for executing them in parallel in multiple threads or machines while doing also some other interesting stuff like re-ordering them to have the last failed ones first. It makes sure no tests is executed more times and none is omitted. A benefit is that it is standalone tool that can be integrated into private builds, CI etc. It’s written in Java but supports also non-JVM languages. Currently (10/2011) it supports for example. JUnit, Cucumber, Ant, Java, Ruby. Support for Maven is in progress. What it does in detail: “Balancer gets a list of all the tests that need to be executed from the build script (after the build script has been invoked, and before tests start running). It then prunes that list to make a subset using the historical test information obtained from the Server. This smaller subset is passed-on to the test framework to execute. Balancer continues to listens to events published by the test framework as these tests execute, to record result and time taken by each test. This data is then posted across to the TLB server and acts as seed data for balancing/ordering future builds.”
Testing-Related Links
The Way of Testivus - ancient eastern testing wisdom :-) - really worthy brochure for all pragmatic testers
Blog Cost of Testing (10/09) - a two-week one-man experiment in coding with testing with interesting results such as that only 10% of time was spent writing tests even though their extent is 40-50% of the whole codebase (mainly because tests are usually quite simple, no complicated logic etc.)
Alberto Savoia: Beautiful Tests (a chapter from Beautiful Code) - a good and inspiring example of the though process behind creating a set of tests. Included: a smoke test, gradual addition of boundary tests, randomized testing, testing of performance characteristics. Conclusion: Writing test code can be every bit creative and challenging and tests document the code’s intended behavior and boundaries; help you improve the code; help you gain confidence in the functionality and performance.
Getting Started with Tdd in Java using Eclipse [Screencast] - TDD in practice with application of a number of best practices and interesting ideas regarding testing (such as not having for 1 production class just 1 test but more, one for each specific test setup). It has 50m but is really worth the time
xUnitPatterns - one of the elementary sources that anybody interested in testing should read through. Not only it explains all the basic concepts (mocks, stubs, fakes,…) but also many pitfalls to avoid (various test smells such as fragile tests due to Data Sensitivity, Behavior Sensitivity, Overspecified Software [due to mocks] etc.), various strategies (such as for fixture setup), and general testing principles.
Infinitest is an interesting tool that brings Continuous Testing to your IDE (similar to Kent Beck’s JUnitMax) & an experience report
When software testing failes -ARIANE 5 Flight 501 FailureReport by the Inquiry Board
Blog: extremely readable test of a web app and even more readable webapp test - really inspiring
F. Martinig’s 10 + 1 Favorite Unit Testing Articles, 2009
Testing services etc.
VM wth IE 6-9 - ievms (make take hours to install)
http://www.browserstack.com/ – testing with proper machine through the browser, support for IE, mobile, and more
Code quality metrics
Bad quality ⇒ harder testing, that’s why it is here.
Testability-explorer by Misko Havery, the Google testing guru - an open-source tool, which analyzes Java bytecode and computes how difficult it will be to write unit tests for the code
ckjm— Chidamber and Kemerer Java Metrics - WMC: Weighted methods per class, DIT: Depth of Inheritance Tree, NOC: Number of Children, CBO: Coupling between object classes, RFC: Response for a Class, LCOM: Lack of cohesion in methods. Bonus - Ca: Afferent couplings,NPM: Number of public methods
Simian (Similarity Analyser; free for non-commercial use)
Books
Martin Fowler: XUnit Test Patterns - Refactoring Test Code, 2008 - patterns and anti-patterns for unit tests (I haven’t read it yet)
Lasse Koskela: Effective Unit Testing: A guide for Java Developers (2013) - guide for writting better tests for those who already know how to test. A major part is a catalog of test smells in the form description - example - how to fix - summary, grouped by the topics of readability, maintainability, trustworthiness. There are also chapters on mocking, testign with Groovy, testable design, speading up test execution etc. Read this review.
Test code samples & discussion
Naming
What can be improved with test method name @Test public void empty_credit_card_number_should_return_false ?
I believe it should rather be called …should_be_invalid because we want tests to “tell a story”, to be more business logic-oriented (“card invalid”) than implementation-oriented (“return false”); with my name it’s OK to change the validate(..) method to throw an exception instead of returning false while the original name would need to be changed
Decide in the latest responsible moment
What could be improved with the following 1st test method code in the TDD approach? … assertFalse(CreditCardValidator.validate("")); …
I believe it’s completely unnecessary and harmful to assume and enforce that there 1) exists the class CreditCardValidator 2) and that it has a statically-caled method; what if we later decide that we actually need an instance method instead (which is quite likely)?
I think that a better solution is to begin by adding the method validate(..) to the test class itself and let this method decide how to perform the validation — thus we can start implementation right there before migrating to a suitable target class — it’s no-brainer to switch from static call to instance call because there is just 1 place to change — this approach allows us to postpone the decision how the vlaidation is called til the latest possible moment ⇒ one of the Lean thinking gold principles — when things become more stable, it’s a simple refactoring to inline this helper method
