Java Code Coverage Tools Evaluation

Over the past week, I’ve been evaluating various test Code Coverage tools for our Java application.  We have a large Java project with about 1500 production classes and 550 test classes.  My goal was to find a tool that integrates well into our Ant test target, automatically producing html reports at the application, package, class, and line level without requiring code changes.  The tool can’t be too slow.  Good documentation and examples, historical reports, and an IntelliJ IDEA plugin were nice-to-have but not necessary.  Cost is a consideration.  I looked at ten free tools and one commercial tool during the evaluation.  I found that some projects are dead, some have some work to do, and some have created excellent tools that should work for any team.

Introduction

JavaRanch provides a nice overview of code coverage, explaining the goals of code coverage and demonstrating a few tools.  I’ll rely on that and Google to provide my introduction to the subject.  I will say that most successful tools work by instrumenting the bytecode to track which code is accessed.  When a line of bytecode is executed, the instrumentation records the event, and those records are later processed into the reports.  The instrumented bytecode is usually created in a separate directory which needs to be placed in the classpath for the JUnit task.

The best listing of free Java code coverage tools is at java-source.net. I considered each of the tools on that page and Clover, a commercial product.  Here are my evaluations, from least suitable to most suitable.

Dead projects

InsECT is a dead project.  Its Sourceforge activity is 0%, having not released anything since version 0.91 in 2003.  I didn’t evaluate it.

JVDMI Code Coverage Analyser is also a dead project.  Its Sourceforge activity is 26%, having not released anything since version 0.2 in 2002.  I didn’t evaluate it.

Interesting approach, but not for us

Jester does not instrument the byte code to present a report, either.  Instead it finds code that hasn’t been tested and literally changes the code.  It changed branching conditions, hard-coded numbers, and String literals to see which tests break.  While this is an interesting idea that did really well for Robert Martin and Robert Koss’s bowling scoring example, it is not what we are looking for.

Hansel does not instrument the byte code to present a report.  Instead, it requires the developer to insert a CoverageDecorator in the suite() method of the test.  This decorator will cause the test to fail if any of the code in the target class is not executed.  While this approach would be very effective, it would not work for us.  Since we have over 500 tests now, this change would require a lot of work to change the tests, and the sudden failure of a few thousand test cases would make our current tests irrelevant.  This tool might be effective for a new project, but not for us.

Closer, but didn’t work for us

NoUnit is a promising project that could work well for some projects, but it didn’t quite meet our needs.  The biggest problem is that it doesn’t come with an Ant task.  While I did see on forums that people have written their own Ant tasks, I want that functionality to be built into the tool.  Also, NoUnit reports to the method level, while most other tools go a step further to check lines within the methods.

Quilt is popular, but I can’t figure out why.  Sourceforge activity is high, yet the latest version is from October 2003.  Perhaps the project will improve significantly soon, but I was not impressed.  The documentation was produced in Maven, so it looked good, but there was very little meat to it.  In fact, I couldn’t find a single screenshot of the final report.  More troublesome, I couldn’t get the tutorial to work with our application, and the example that came with the binary distribution did not work.  There are enough good alternatives not to consider this tool.

jcoverage/gpl is the free version of a product that also has two different commercial versions.  I was very frustrated with the site because it also didn’t show a screen shot or the cost of the non-free versions.  The tutorial that came with the gpl version (the only one I tried) was minimal, but I thought it had enough to show me how to use the tool.  Unfortunately, Ant couldn’t find all of their tasks despite my following their directions, so I never really got the tool to work.  Since there are better free and commercial application with much better documentation, I didn’t press the issue.

JBlanket is an academic project that seems to have a similar approach to many other tools here.  The screen shots in the documentation look great.  I really wanted it to work, but there was a failure somewhere between JBlanket and JDOM.  I had to move on.

Quilt, JBlanket, and jcoverage didn’t work for our application and the examples given; perhaps I would have solved the problems with more stubbornness, but the poor documentation did not encourage me to think that they would be better than Emma, which I had already evaluated.

GroboUtils would be a great product for a smaller product, but it needs much better performance before we could use it.  Instrumenting our bytecode took almost twenty minutes on my machine, and the documentation implies that the instrumentation should happen on every compilation.  Also, I got an OutOfMemory error during report generation.  On the other hand, GroboUtils has nice functionality and examples, and I’m sure it would be a great solution for a smaller project.

The winners

Emma is a great free product.  It started out as an internal tool, and it has been on Sourceforge since May 2004.  Its instrumentation and report generation are very fast, taking less than two minutes each where GroboUtils took over ten minutes for each.  The reports are very nice, although they don’t use pretty green and red bars like some of the other tools use.  There are HTML, XML, and text reports, and the HTML report allows drill-down to the line.  Emma uses an interesting approach to defining coverage that often results in lines being only partially covered.  For example, lines with ternary operators with only one branch executed will show as partially executed, which is very nice.  Emma is the only free tool that worked on our application and satisfied all my requirements.  The examples and documentation were thorough, although I ran into a couple errors in the basic Ant tutorial.  Emma does not provide automated historical reports or an IntelliJ plugin.

Clover is the best tool of the bunch.  At its price ($2500 for the first five developers, $100 for each after that), it should be.  Its instrumentation and report generation are very fast, similar to Emma.  The HTML reports are even nicer than Emma’s, showing nice green and red bars to demonstrate coverage percentage in addition to the number.  The HTML reports also allow drill-down to the line level, and they are designed with both frame and no-frame versions that will be familiar to anyone who knows Javadoc.  There are also XML and PDF reports.  Clover also provides an executive report and historical reports to show a high-level overview of coverage, and how the coverage has changed over time.  Also, Clover integrates completely with IntelliJ and other IDEs.

The biggest problem I had with Clover may have been an artifact of evaluation; I couldn’t evaluate both the plugin and the Ant version at the same time because Clover instruments the actual bytecode instead of a copy, and the two installations of Clover fought over which had control.  I hope that this problem goes away with the full, purchased version of Clover.  Either that, or I hope that I will find a fix to the problem in a forum somewhere.

Conclusion

Emma and Clover were the best tools.  Emma is fast, free, integrates with Ant well, and produces nice reports.  Clover is fast, integrates with Ant and IntelliJ well, produces even nicer reports, and also has historical reports.  However, it costs $2,500 to get the full set of features.  Both have good documentation.

I do not know which we will start using; that will depend on getting budgetary approval for Clover.  However, Emma is good enough that we’ll be very happy with it as a free alternative.

Appendix

Tool License sf.net Activity (Sep 14, 2004) Advantages Disadvantages
Clover Various
Pricing
N/A Great
reports, IDE integration, history reports, nice Ant
integration
Because it instruments the main
bytecode, it takes over; cost
Emma Common Public License v1.0 98.0256% Very nice reports, fast Have to create our own historical reports
GroboUtils MIT
License
82.2061% Good
examples and documentation – good for small application
Instrumentation and report generation are too slow for our large application
NoUnit GPL 57.1157% Nice report No Ant task; measures method-level coverage exercised by JUnit tests only, not statement and branch coverage
Jester Open License 56.5148% Tests fragility of tests It
doesn’t really test coverage
Hansel BSD License 44.0677% Makes the tests themselves fail Relies on changing all the suite() methods, not on instrumenting the byte code
Quilt Apache
License
and Artistic License
82.3901% None Weak
tutorial and documentation; couldn’t
get it working
jcoverage/gpl GPL N/A None Couldn’t
get it working
JBlanket GPL N/A None Couldn’t
get it working
JVDMI Code Coverage Analyser LGPL 26.703% None Inactive
InsECT GPL 0% None Inactive
Advertisements

About Lance Finney

Father of two boys, Java developer, Ethical Humanist, and world traveler (when I can sneak it in). Contributor to Grounded Parents.
This entry was posted in Programming. Bookmark the permalink.

23 Responses to Java Code Coverage Tools Evaluation

  1. Nadeem says:

    I have been using the maven JCoverage report and it has been working great. It’s as simple as adding <reports>
    <report>maven-jcoverage-plugin</report> </reports> to project.xml.

  2. Lance Finney says:

    Nadeem,

    Is that the gpl version of JCoverage or one of the commercial versions? I evaluated the gpl version, and the documentation I saw didn’t mention that task. I also checked the jar file and didn’t see it there.

  3. Lance Finney says:

    I just found the JCoverage plugin page off the Maven site. I’m sorry I didn’t evaluate that, but we’re not using Maven, so it wouldn’t work for us. Thanks for the comment, though.

  4. Nils-H says:

    Great article!

  5. idcmp says:

    Agreed, thanks for taking the time to review the options out there and make the information available!

  6. Vlad says:

    Lance, thanks for a good writeup on EMMA. You’ve mentioned a couple of errors in the basic ANT tutorial — if you can give me details on those, I will certainly correct them.

  7. Ilja Preuß says:

    You can introduce Hansel class by class – when you have to touch the tests for a class, instrument it with Hansel and make it run again. We are doing this at work with a product of ~500,000 LOC. It has been quite helpfull until know.

  8. Prashant says:

    Nice article and good comparison.

  9. Jeff Conrad says:

    It’s too bad you could not get JCoverage working. I think it was easy to set up. I used the provided example to modify my JUnit target. Maybe check with http://www.javaranch.com/newsletter/200401/Journal200401.jsp#a5. My only problem was that I was instrumented the unit test classes and ended up with 100% coverage right off the bat, but with some tweaking, I got it to instrument my original source and the numbers dropped. Oh well.

  10. Param says:

    Thanks for your valuable research. We were hard-pressed for time & this page helped our decision making!

  11. mano says:

    I would like to know if there is a tool that could be used for J2ME-MIDP2.0 ? I tried Emma to work but Emma depends on Class Loader which J2ME-MIDP2.0 does not have.

  12. Arndt says:

    InsECT doesn’t look dead to me. The last webpage update was 2003 and the current release is 0.9.
    Wonder where you looked at?
    Nice report anyway.

  13. Lance Finney says:

    You’re right, Arndt, there was a release in 2003, and they are at 0.9, not at 0.1. I don’t know if I saw an invalid page back then, or if I made a typo, or if I got InsectJ confused with something else. It is possible, though, that the sourceforge activity might have been 0% at that the point I wrote this. It’s currently 18%.

    However, since InsECT hasn’t realeased in about 16 months and has such a low activity, I would still consider it inactive.

    I have updated the evaluation for the 0.91 release in September 2003.

  14. Jimmy Jarrett says:

    This blog is a bit out of date, you should take a look at the new GroboUtils. In the new version, it’s instrumentation is literally 10x faster (estimate on my part), and the out of memory error has been fixed.

  15. How did you get emma to run your junit task? I see there is an task but I can’t see how to get it to run junit, without reimplementing what the junit task does.

    If this question is too hard to answer, please just post your build.xml that tested your unit tests with emma.

    Good article.

  16. Lance Finney says:

    Here are the relevant snippets from our build.xml:

    A necessary taskdef:

    <taskdef resource=’emma_ant.properties’ classpathref=’build.classpath’/>

    We modified our test target. This part goes at the top to decorate the bytecode:

    <property name=”coverage.dir” value=”${basedir}/coverage”/>
    <mkdir dir=”${coverage.dir}”/>
    <emma enabled=”${emma.enabled}”>
    <instr instrpathref=”classes.path”
    destdir=”${out.instr.dir}”
    metadatafile=”${coverage.dir}/metadata.emma”
    merge=”true”>
    <filter excludes=”*Test*,*Mock*”/>
    </instr>
    </emma>

    We added these within the junit call:

    <jvmarg value=”-Demma.coverage.out.file=${coverage.dir}/coverage.emma”/>
    <jvmarg value=”-Demma.coverage.out.merge=true”/>

    We added this at the end of the test target for cleanup:

    <emma enabled=”${emma.enabled}”>
    <report sourcepath=”${basedir}/src”>
    <fileset dir=”${coverage.dir}”>
    <include name=”*.emma”/>
    </fileset>
    <txt outfile=”${coverage.dir}/coverage.txt”/>
    <html outfile=”${coverage.dir}/coverage.html”/>
    </report>
    </emma>
    <delete dir=”${out.instr.dir}”/>

    And here’s another target that is used to turn emma on for testing:

    <target name=”emma” description=”turns on EMMA instrumentation/reporting”>
    <property name=”emma.enabled” value=”true”/>
    <property name=”out.instr.dir” value=”${basedir}/outinstr”/>
    <mkdir dir=”${out.instr.dir}”/>
    </target>

  17. Nicolas says:

    Does someone has tested Rational Test RealTime ?
    It seems to be a very good tool to test coverage, runtime analysis and many things else.
    Any opinion ?

  18. anjan bachu says:

    hi there,

    I’ve heard about an open-source tool called Cobertura(https://sourceforge.net/projects/cobertura/, http://cobertura.sourceforge.net/) which is in the same space as emma. Currently, it’s at v.1.4.

    BR,
    ~A

  19. Lance Finney says:

    anjan,

    I’ve seen good things about Cobertura. Unfortunately, I’m no longer at the same job I was at when I made this comparison, so I could not legitimately compare Cobertura to the other tools under the same conditions. I hope it’s a good tool, too.

  20. ish says:

    has anyone found something that works for J2ME? I have evaluated the following tools without any success: Emma, Clover, GlassJar Toolkit. I have an idea of how to actually implement one based on source code instrumentation but don’t have the time right now to do it. If anyone knows of anything else that might work please email me.

  21. Prasad Bhuvanam says:

    Really Very Nice Analysis and this helped me a lot in decision making(which tool to choose).
    Thanks a lot to you all.

  22. Ajay KR says:

    I have used Cobertura (http://cobertura.sourceforge.net/index.html) and pretty happy with that. Good thing is its getting improved after every release and so looks promising. As of now its at version 1.8.
    -Ajay KR

  23. Michael says:

    There are many plugins based on EMMA for all the main IDEs. For example, the http://codecoverage.netbeans.org for Netbeans or EclEmma (and others) for Eclipse. We also did some research in this area and the conclusion was that EMMA is the best choice

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s