Separating Integration and Unit Tests with Maven, Sonar, Failsafe, and JaCoCo

Goal: Execute the slow integration tests separately from unit tests and show as much information about them as possible in Sonar.

The first part - executing IT and UT separately - is achieved by using the maven-failsafe-plugin and by naming the integration tests *IT (so that the unit test running surefire-maven-plugin will ignore them while failsafe will execute them in the integration-test phase and collect results in the verify phase).

The second part - showing information about integration tests in Sonar - is little more tricky. Metrics of integration tests will not be included in the Test coverage + Unit tests success widget. You can add Integration test coverage (IT coverage) widget if you enable JaCoCo but there is no alternative for the test success metrics. But don't despair, read on!

Important notice: The integration of Sonar, JaCoCo and Failsafe evolves quite quickly so this information may easily get outdated with the next releases of Sonar

Versions: Sonar 2.12, Maven 3.0.3

Prerequisity: How Sonar Works

It's important to understand how Sonar works and how it is integrated into the build process. Briefly:

• Sonar is run after the project is built (and thus various artifacts are already generated): either as a post- action in Jenkins or after executing mvn install manually
• Sonar comes bundled with various integrated quality analysis plugins such as PMD, Checkstyle, Findbugs (depending on the quality profile chosen). You do not need to add them manually to your POM (but you can, if you need to configure them). However if you need something it doesn't do yet such as collecting coverage for integration tests, you have to do it manually.
• Sonar may either reuse existing reports (checkstyle, ...) or generate its own.
• F.ex. if you choose JaCoCo as the default code coverage provider, Sonar will automatically rerun unit tests with JaCoCo enabled to collect the coverage metrics. You can see the file target/sonar/sonar-pom.xml that it generates to see what it does.

Fooling Sonar to Display IT Test Success

Executing unit tests via Surefire and integration tests via Failsafe results in not beeing able to see when integration tests fail. (Because this is shown by the Unit test success widget, which doesn't support Failsafe.) However it's possible to fool Sonar to show test success for both unit and integration tests together by instructing Failsafe to store its test reports to the same directory as Surefire instead of the default failsafe-reports, and that's what we do:

<!-- pom.xml, the build/plugins section -->
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <configuration>
        <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
    </configuration>
</plugin>

How to display IT Code Coverage in Sonar

Sonar 2.12 currently isn't able to compute test coverage for integration tests automatically and thus we have to do it manually. Summary:

1. Add Failsafe to your Maven configuration as shown above.
2. Add JaCoCo to Maven and combine it with Failsafe to produce the coverage report when integration tests are run
3. Instruct Sonar to execute the verify phase and pass the path to the integration test report to it

1. Add Failsafe to your Maven configuration as shown above

See the pom.xml fragment under "Fooling Sonar to Display IT Test Success" above.

2. Add JaCoCo to Maven and combine it with Failsafe to produce the coverage report when integration tests are run

As Sonar 2.12 doesn't automatically reconfigure Failsafe to collect code coverage, we have to instruct Failsafe manually to load the JaCoCo java agent that will collect and store the coverage report (to target/jacoco.exec by default):

<!-- pom.xml fragment: -->
<build>
 ...
     <!-- Run integration tests (*IT) -->
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-failsafe-plugin</artifactId>
        <configuration>
            <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
            <argLine>${jacoco.agent.argLine}</argLine>
        </configuration>
    </plugin>
    <!--
        Compute integration test coverage for Sonar
        BEWARE: Sonar doesn't run the verify phase, it has to be forced by setting -Dsonar.phase=verify
    -->
    <plugin>
        <groupId>org.jacoco</groupId>
        <artifactId>jacoco-maven-plugin</artifactId>
        <configuration>
            <propertyName>jacoco.agent.argLine</propertyName> <!-- default: argLine -->
            <includes>
                <include>com/comoyo/**</include>
            </includes>
            <destFile>${project.build.directory}/jacoco-integration.exec</destFile> <!-- agent -->
            <dataFile>${project.build.directory}/jacoco-integration.exec</dataFile> <!-- report -->
        </configuration>
        <executions>
            <execution>
                <id>agent</id>
                <goals><goal>prepare-agent</goal></goals>
            </execution>
        </executions>
    </plugin>
</build>

The key lines are 10 (argLine) and 28 (prepare-agent before verify - the default phase, if unspecified, is init).

3. Instruct Sonar to execute the verify phase and pass the path to the integration test report to it

Important: If you use Sonar prior to 2.12 then you must install the Sonar JaCoCo plugin manually.

Whether running Sonar from Jenkins or locally we have to make sure that the verify phase is run and pass to Sonar the path to the generated JaCoCo integration test coverage report. This is best done in Sonar in the JaCoCo plugin configuration (on the project level?) but you could also do it manually by passing it to maven via "-Dsonar.jacoco.itReportPath=target/jacoco-integration.exec" or by setting the property in pom.xml under project/properties.

Jenkins Sonar plugin does not require any special configuration if path to the report is set inside Sonar's JaCoCo plugin.

Command line (the recommended way; the first line runs among others the phase verify and thus generates the report):

payment$ mvn clean install -Dtest=false -DfailIfNoTests=false
payment$ mvn sonar:sonar

We could actually invoke Maven with a single line, though that isn't recommended because it would run tests multiple times:

payment$ mvn -Dsonar.phase=verify sonar:sonar

Some logs of interest from running mvn sonar:sonar:

...
JaCoCo agent (version 0.5.3.201107060350) extracted: /var/folders/k0/2842tm752zv1dh4q77_gmgdr0000gn/T/jacocoagent2548420105762793132.jar
JVM options: -javaagent:/var/folders/k0/2842tm752zv1dh4q77_gmgdr0000gn/T/jacocoagent2548420105762793132.jar=destfile=target/jacoco.exec,excludes=*_javassist_*
Initializer FindbugsMavenInitializer...
Initializer FindbugsMavenInitializer done: 4 ms
Execute maven plugin maven-surefire-plugin...
Execute org.apache.maven.plugins:maven-surefire-plugin:2.8.1:test...
...
Execute maven plugin maven-surefire-plugin done: 9856 ms
Initializer JacocoMavenInitializer...
..
Sensor SquidSensor done: 2207 ms
Sensor JaCoCoSensor...
Analysing /myproject/target/jacoco.exec
Sensor JaCoCoSensor done: 559 ms
Sensor JaCoCoItSensor...
Analysing /myproject/target/jacoco-integration.exec
Sensor JaCoCoItSensor done: 89 ms
Sensor SurefireSensor...

• Notice that I've JaCoCo set as my code coverage provider in Sonar and Sonar does use its own copy of it  (the line JaCoCo agent .. extracted), which it uses in the test phase
• Notice that Sonar runs surefire:test automatically (with instrumentation) to collect code coverage
• Notice that JaCoCo processed both *.exec files (the first generated by Sonar for unit tests, the other generated by Maven in verify prior to calling Sonar)

Tip: Compute the total code coverage of unit + integration tests

Unit and integration test coverage are computed separately; to see the total code coverage we would need to merge the two (notice we can't just sum them as both kinds of tests can cover some of the same lines). It would be possible by using JaCoCo both to compute unit test coverage (supported out of the box) and integration test coverage into different files and using its Ant task to merge the two coverage files, passing the resulting file as the IT test coverage file to Sonar (for we cannot get a 3rd widget to display this summed coverage). However I haven't tried it.

Note on Testing JaCoCo, Failsafe and Maven Integration

If Sonar doesn't show the IT coverage widget though it is on the dashboard (click on Configure widgets while logged in) or shows 0% though it should be higher, you may check the data that JaCoCo is generating by producing HTML report from them. There is both an Ant task for that, which didn't work for me, and Maven goal. This is how you instruct JaCoCo to generate the report when "mvn site" is run - notice the lines 13, 14 (remember that you must run "mvn verify" first to generate the binary coverage report):

<!-- pom.xml build/plugins fragment -->
<plugin>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <executions>
        <execution>
            <id>agent</id>
            <phase>pre-integration-test</phase>
            <goals><goal>prepare-agent</goal></goals>
        </execution>
        <execution>
            <id>report</id>
            <phase>site</phase>
            <goals><goal>report</goal></goals>
        </execution>
    </executions>
</plugin>

References

JaCoCo Maven plugin has minimal documentation but you can get useful info via:

mvn help:describe -Dplugin=org.jacoco:jacoco-maven-plugin -Ddetail

Sonar: Measure Code Coverage by Integration Tests with Sonar (9/2010) - doesn't use the JaCoCo Maven plugin and thus has to configure the argLine manually

The Complete POM

See the complete POM at GitHub.

Tags: testing java tool