(Unit) Testing Swiss Knife: All the Tools You Wanted to Know
I love testing. And I like productivity. There are many tools and libraries that make writing tests easier, more convenient, more fun. I would like to introduce here those that I found the most useful during the years, from selected advanced features of JUnit to assertion libraries, powerful behavior/fault injection, testing of database-related code, and finally to boosting your testing productivity hundredfold with Groovy.
This post accompanies my JavaZone 2012 lightning talk and goes more in depth and introduces additional tools and tips.
You can use @Rule with a public instance field for rules that should act before/after each test method and @ClassRule with a public static field for rules that should act before/after the whole test case.
You can use the existing rule implementations to simplify your tests. Some of the most useful rules are (click on the name for JavaDoc with examples):
Before introducing @Rule, every DbUnit Express test required the declaration of a db tester and it initialization in setUp:
Introducing EmbeddedDbTesterRule simplified that to one annotated line:
(The EmbeddedDbTesterRule is more complicated than usual because I wanted it to extend the original class, EmbeddedDbTester, and thus couldn't extend ExternalResource.)
See also DRY: Use JUnit @Rule Instead of Repeating Setup/@Before in Each Test.
The Parametrized runner provides a different way by creating a new instance of the test class for each data set with the data passed to its constructor and available to all test methods, the data itself provided by a static method:
The downside is that it requires quite lot of typing and thus it pays off only sometimes. A limitation is that you cannot apply a set of values to a single test method, which stems from the design of JUnit which expects you to create a different TestCase for each individual fixture, i.e. test data/context (which isn't a bad idea but sometimes it is just too costly to do).
Runner is the most powerful and flexible part of the JUnit architecture, it can decide how the test class is instantiated and initialized, what methods are executed and how and in what order etc. You can therefore use it for some neat tricks.
For example Johannes Brodwall of Steria wrote a custom runner that switches between using a fast in-memory database and the true stand-alone database based on whether the tests run on a developer's machine or on the continuous integration server.
You'd typically extend the default runner BlockJUnit4ClassRunner (source), as e.g. Mockito and Spring do.
Example: Ignore DB tests if a particular system property is set:
(Assume is also often used together with Theories to define what values are legal for a particular theory.)
Jens Schauder wrote an example test where each test method (theory) is executed for all possible 2-element combinations of three string values. Jacob Childress uses theories as an alternative to Parametrized by using only a single parameter (thus there are no combinations) and a @DataPoints method. Adam Hepner has written brief instructions for two parameter method and assume at StackOverflow.
Contrary to Parametrized, the Theories runner doesn't supply fixed sets of parameters to the test methods but generates all combinations of the available values and runs the tests for each of them, making it easier to test many more cases.
Notice that you can exclude some combinations/values by using assume (discussed above).
The main feature of DbUnit [Express] is the ability to ensure a clean, defined state of the database before each test by removing and inserting data based on (optionally test class-specific) data set files (XML, csv, Excel, ...). In addition to that it provides access to the database via a Connection or DataSource and utilities for checking the data. The goal of DbUnit Express is to help you to get your first complete DB test ready in 10 minutes in a form that can be easily distributed to other developers (in-memory DB automatically initialized from a provided .ddl file).
Example:
Read more information about set up, usage, and benefits on the DbUnit Express documentation page.
Related:
Personally I prefer FEST-Assert because it has a nice fluent API that is intuitive to use and easy to discover via autocompletion in IDE. Hamcrest is older and suffers from issues with generics - earlier or later you run into a case where you have to add same strange casts or convert a generic list to List and then cast it to List<SomethingElse> to get Hamcrest working. It even sometimes happens that the code works with Maven but fails with Eclipse or vice versa.
Hamcrest & generics hell by J.F. Smart and at StackOverflow (e.g. explicit cast to collection element type).
Learn some great tips and tricks for Fest-Assert such as extracting a particular property from beans in a collection or retaining only objects matching a condition.
Why to use Groovy?
Alternatives: ScalaTest if you know Scala - it also integrates very well with JUnit and production code in Java
There are two primary uses of Byteman:
Read more at Cool Tools: Fault Injection into Unit Tests with JBoss Byteman – Easier Testing of Error Handling and get the complete ExampleByteman project from GitHub.
Beware that with great power comes also great responsibility. You should only use Byteman as the last resort since such tests are harder to understand and more brittle.
Alternative: JMockit (might be easier to use since you only write code as Java code and not as text as in the case of Byteman)
Update: Brett L. Schuchert argues very well in Modern Mocking Tools and Black Magic - An example of power corrupting why using such black magic (JMockit in his case) should be avoided as only treating the symtomps in favor of actually fixing the code.
JBoss Arquillian is the best tool for this job - it let you define what classes and resources to include in a test and runs it on an embedded server (Glassfish, JBoss, ...). If you use Java EE then you absolutely should have a look at Arquillian.
If you have to run your tests manually then you will tend to run them less often, after larger changes. With Infinitest you learn about a failure that you introduced soon after having done so, making it much easier to understand what's wrong and to fix it.
Infinitest can be easily configured to skip some tests (typically slow/integration tests) and set JVM arguments for the tests and supports both JUnit and TestNG. It isn't completely perfect but it does a great job anyway. Highly recommended.
PS: Feedback is welcomed.
This post accompanies my JavaZone 2012 lightning talk and goes more in depth and introduces additional tools and tips.
Table of Content
- Advanced JUnit 4: Useful to Know
- Testing DAOs with DbUnit Express
- Readable Tests with Matchers
- Attaining Eternal Bliss with Groovy
- Fault Injection with JBoss Byteman
- Other Stuff to Know
- The Code
Advanced JUnit 4: Useful to Know
Removing Duplication And Simplyfing Tests With @Rule
Rules have been introduced to JUnit around version 4.8. Simply said, rules can perform actions before and after each test or the whole test case, similarly to your @Before[Class] and @After[Class] methods but even before/after them. To use a rule you would assign it a public field of your test class. Example:
public class ExampleTest {
@Rule public ExpectedException thrown = ExpectedException.none(); // *must* be public
@Test
public void throwsNullPointerExceptionWithMessage() {
thrown.expect(NullPointerException.class); // same as @Test(expected=...)
thrown.expectMessage("cool failure"); // check message substring
throw new NullPointerException("My cool failure message");
}
}
You can use @Rule with a public instance field for rules that should act before/after each test method and @ClassRule with a public static field for rules that should act before/after the whole test case.
You can use the existing rule implementations to simplify your tests. Some of the most useful rules are (click on the name for JavaDoc with examples):
ExpectedException: check that an exception has been thrown and that the message is as expected- See my post JUnit Tip: Verifying that an Exception with a Particular Message was Thrown for more details
TemporaryFolder: create fresh files/folders before test and delete them after test
Before introducing @Rule, every DbUnit Express test required the declaration of a db tester and it initialization in setUp:
public class SimpleNonExtendingEmbeddedDbJUnit4Test {
private final EmbeddedDbTester testDb = new EmbeddedDbTester();
@Before
public void setUp() throws Exception {
testDb.onSetup();
}
// ...
Introducing EmbeddedDbTesterRule simplified that to one annotated line:
public class EmbeddedDbTesterRuleTest {
@Rule
public final EmbeddedDbTesterRule embeddedDb = new EmbeddedDbTesterRule();
// ...
(The EmbeddedDbTesterRule is more complicated than usual because I wanted it to extend the original class, EmbeddedDbTester, and thus couldn't extend ExternalResource.)
See also DRY: Use JUnit @Rule Instead of Repeating Setup/@Before in Each Test.
@RunWith(Parametrized): Re-Run Tests With Different Data
Occasionally we need to execute the same test method(s) repeatedly for different data. The typical solution is to copy, paste and adjust (brrr, a certain path to hell) or to use iteration:
@Test public void acceptAllValidEmails() {
String emails = {"word@example.com", "under_score@a.b.cz", "d.ot@here.org"};
for (String email: emails) {
assertTrue("Should accept " + email, this.filter.accept(email));
}
}
The Parametrized runner provides a different way by creating a new instance of the test class for each data set with the data passed to its constructor and available to all test methods, the data itself provided by a static method:
@RunWith(value = Parameterized.class)
public class FilterTest {
private String email;
private Filter filter = ...;
public FilterTest(String email) { this.email = email; } // parametrized constructor
@Parameters // the 1st element of each array will be assigned to the 1st constructor parameter etc.
public static Collection<Object[]> data() {
return Arrays.asList(new String[][] { {"word@example.com"}, {"under_score@a.b.cz"}, {"d.ot@here.org"}});
}
@Test public void acceptAllValidEmails() {
assertTrue("Should accept " + this.email, this.filter.accept(this.email));
}
}
The downside is that it requires quite lot of typing and thus it pays off only sometimes. A limitation is that you cannot apply a set of values to a single test method, which stems from the design of JUnit which expects you to create a different TestCase for each individual fixture, i.e. test data/context (which isn't a bad idea but sometimes it is just too costly to do).
Alternatives:
- Use the TwiP runner (available separately) that adds support for parameters to test methods and supplies values from a provided set or randomly from the whole space
- Use the JUnitParamsRunner (available separately) that permits more flexible specification of data and to do it on per-method basis
- Use TestNG :-)
- Use JUnit Theories, see below
The Almighty Custom Runners
You have certainly used an alternative JUnit runner (activated by the @RunWith(<runner class>) annotation), such as Parametrized, MockitoJUnitRunner, or SpringJUnit4ClassRunner. But have you ever considered writing your own?Runner is the most powerful and flexible part of the JUnit architecture, it can decide how the test class is instantiated and initialized, what methods are executed and how and in what order etc. You can therefore use it for some neat tricks.
For example Johannes Brodwall of Steria wrote a custom runner that switches between using a fast in-memory database and the true stand-alone database based on whether the tests run on a developer's machine or on the continuous integration server.
You'd typically extend the default runner BlockJUnit4ClassRunner (source), as e.g. Mockito and Spring do.
Conditionally Ignore Tests With Assume
It's useful to be able to @Ignore tests based on a condition, for example to disable database integration tests in environments where the database isn't available. You can use theorg.junit.Assume class for that - if you call any of its assume* methods in a test method and it evaluates to false than the test will be treated is if it were marked with @Ignore. You can ignore all tests by doing this in a @Before method.Example: Ignore DB tests if a particular system property is set:
@Before public void ignoreTestsIfDbTestsDisabled() {
assumeThat(System.getProperties().containsKey("tests.db.disable"), is(false));
}
(Assume is also often used together with Theories to define what values are legal for a particular theory.)
Theories (Experimental!)
Theories are parametrized test methods that are executed for each possible combination of values of their parameters. The values are defined as fields annotated with @DataPoint or provided by a method annotated with @DataPoints. You use @RunWith(Theories.class) and annotate each test method with @Theory instead of @Test.Jens Schauder wrote an example test where each test method (theory) is executed for all possible 2-element combinations of three string values. Jacob Childress uses theories as an alternative to Parametrized by using only a single parameter (thus there are no combinations) and a @DataPoints method. Adam Hepner has written brief instructions for two parameter method and assume at StackOverflow.
Contrary to Parametrized, the Theories runner doesn't supply fixed sets of parameters to the test methods but generates all combinations of the available values and runs the tests for each of them, making it easier to test many more cases.
Notice that you can exclude some combinations/values by using assume (discussed above).
Testing DAOs with DbUnit Express
DbUnit Express is a thin wrapper around DbUnit (unit testing of code interacting with a database) intended to make starting with DB testing as quick and as easy as possible by introducing convention over configuration, by using an embedded Derby database out of the box (though you can change that), and by providing various utilities for initializing, using, and checking the content of the test database.The main feature of DbUnit [Express] is the ability to ensure a clean, defined state of the database before each test by removing and inserting data based on (optionally test class-specific) data set files (XML, csv, Excel, ...). In addition to that it provides access to the database via a Connection or DataSource and utilities for checking the data. The goal of DbUnit Express is to help you to get your first complete DB test ready in 10 minutes in a form that can be easily distributed to other developers (in-memory DB automatically initialized from a provided .ddl file).
Example:
public class ExampleJUnit4WithRuleTest {
/**
* Initialize the test and instruct it to use a custom data set file instead of the default dbunit-test_data_set.xml.
* The set up of the test DB will be executed automaticaly thanks to the magic of @Rule.
*/
@Rule
public EmbeddedDbTesterRule testDb = new EmbeddedDbTesterRule("EmbeddedDbTesterRuleTest-data.xml");
@Test
public void should_contain_data_supplied_by_dbunit() throws Exception {
// 1. TODO: Invoke the database-using class that you want to test, passing to it the test database
// via testDb.getDataSource() or testDb.getSqlConnection()
// ex.: new MyUserDao(testDb.getDataSource()).save(new User("Jakub", "Holy"));
// 2. Verify the results ...
// Here we use a checker to check the content of the my_test_table loaded from the EmbeddedDbTesterRuleTest-data.xml
testDb.createCheckerForSelect("select some_text from my_test_schema.my_test_table")
.withErrorMessage("No data found => onSetup wasn't executed as expected")
.assertRowCount(1)
.assertNext("EmbeddedDbTesterRuleTest data"); // the 1st row
}
}
Read more information about set up, usage, and benefits on the DbUnit Express documentation page.
Related:
- Spring's Embedded database support - automatically create and load an embedded DB (Derby, H2, HSQL).
Readable Tests with Matchers
Tests are much better if they clearly express what they test. However JUnit itself doesn't make it possible (perhaps by design?) to express more complex conditions such as checking the content of a list or particular atributes of entities within a list. You can make your tests much more conscise and easier to understand by using matcher libraries such as FEST-Assert (currently at version 2 milestone 7) or Hamcrest. As a bonus, you will get much more descriptive and clear error messages.
// JUnit
assertNotNull(list);
assertEquals(6, list.size());
assertTrue(list.contains(sam));
assertTrue(list.contains(frodo));
// FEST-Assert
assertThat(list)
.hasSize(6)
.contains(frodo, sam);
// Hamcrest
assertThat(list.size(), is(6));
assertThat(list, hasItems(frodo, sam));
Personally I prefer FEST-Assert because it has a nice fluent API that is intuitive to use and easy to discover via autocompletion in IDE. Hamcrest is older and suffers from issues with generics - earlier or later you run into a case where you have to add same strange casts or convert a generic list to List and then cast it to List<SomethingElse> to get Hamcrest working. It even sometimes happens that the code works with Maven but fails with Eclipse or vice versa.
Hamcrest & generics hell by J.F. Smart and at StackOverflow (e.g. explicit cast to collection element type).
Learn some great tips and tricks for Fest-Assert such as extracting a particular property from beans in a collection or retaining only objects matching a condition.
Attaining Eternal Bliss with Groovy
The happiest moment in my testing life was when I realized that I can use one language - typically Java - for the production code and another, much more productive one for tests - Groovy.Why to use Groovy?
- It's syntax is 99.9% of Java + 1000% more => you can copy & paste to Groovy (and back), you can learn the productive features of Groovy gradually
- Powerful, concise, productive
- It has literals for lists, maps, regular expressions etc.
- Groovy adds many extremely useful and poweful methods to JDK classes that skyrocket your productivity:
- Collections: filter, map, ...
- File: getText(), ...
- A single powerful assert ("assert <expressions>") providing very clear failure messages about all parts of the expression involved. With Groovy you generally don't need to learn and use a matcher library
- Closures now & here :-)
- Multi-line strings and strings with variable replacement ("text $replacedWithVariableValue text")
- Useful libraries such as Spock
Alternatives: ScalaTest if you know Scala - it also integrates very well with JUnit and production code in Java
Fault Injection with JBoss Byteman
Byteman is pure magic. It can change behavior of any method on the call stack executed during a test. And you don't need to have access to the object owning the method. The behavior is injected into the target method before a test and removed when it finishes (contrary to most other AOP tools that change code at load time and leave it so).There are two primary uses of Byteman:
- Testing of failure handling by "injecting" failures, usually in the form of throwing an exception such as SocketTimeoutException or FileNotFoundException, into methods somewhere down the call stack
- Testing of legacy code - Byteman enables you to shortcut/replace dependencies that would otherwise prevent you from testing a class (e.g. a web service client instantiated internally in the class)
@RunWith(BMUnitRunner.class)
public class MyMainTest {
@Test(expected = IllegalStateException.class)
@BMRule(name="throw IllegalStateException from the helper class",
targetClass = "MainsHelper",
targetMethod = "sayHello",
action = "throw new java.lang.IllegalStateException(\"Exception injected by Byteman\")")
public void testSayHello() {
new MyMain().sayHello();
fail("sayHello should have failed due to Byteman injecting an exception into sayHello");
}
}
Read more at Cool Tools: Fault Injection into Unit Tests with JBoss Byteman – Easier Testing of Error Handling and get the complete ExampleByteman project from GitHub.
Beware that with great power comes also great responsibility. You should only use Byteman as the last resort since such tests are harder to understand and more brittle.
Alternative: JMockit (might be easier to use since you only write code as Java code and not as text as in the case of Byteman)
Update: Brett L. Schuchert argues very well in Modern Mocking Tools and Black Magic - An example of power corrupting why using such black magic (JMockit in his case) should be avoided as only treating the symtomps in favor of actually fixing the code.
Other Stuff to Know
Java EE Integration Testing With JBoss Arquillian
Java EE in the version 5/6 incarnations has become a lean yet powerful platform worth considering. If you go this way and delegate cross-cutting concerns to the container (in the form of interceptors, dependency injection, eventing etc.) then you will also need to verify that these container-managed pieces are assembled correctly and work as expected.JBoss Arquillian is the best tool for this job - it let you define what classes and resources to include in a test and runs it on an embedded server (Glassfish, JBoss, ...). If you use Java EE then you absolutely should have a look at Arquillian.
Continuous Integration In Your IDE With Infinitest
Infinitest (User Guide) is my absolutely favorite Eclipse/IntelliJ plugin. It provides you with immediate feedback whenever you change your production or test code by running the tests affected by the change and reporting their failures directly in the source code as markers.If you have to run your tests manually then you will tend to run them less often, after larger changes. With Infinitest you learn about a failure that you introduced soon after having done so, making it much easier to understand what's wrong and to fix it.
Infinitest can be easily configured to skip some tests (typically slow/integration tests) and set JVM arguments for the tests and supports both JUnit and TestNG. It isn't completely perfect but it does a great job anyway. Highly recommended.
And Even More...
See my wiki page on Testing for even more tools and libraries.The Code
The code is available in my UnitTestingSwissKnife GitHub repository.PS: Feedback is welcomed.