automatic jUnit test generators - review

This could be scary one. 

When I heard term “automatic jUnit test generator” I was thinking: “no one should go that path”, ”units test should be written by the developers”, “do you want to replace devs?”, “test should be written before code!”.

But I gave it closer look, and actually “closer think” – did I really expect that there is some AI module which will go through code and automatically write tests – if so why there is no AI which automatically write code in first place … [maybe soon ;-)]

Before you send thousands of hate comments, let’s give it quick look what is definition and what should you expect by automatic jUnit test generator:

• it's everywhere - right click on class in eclipse or intelliJ and you’ll have it - that's point first and should be breakthrough in your thinking about generators ;-)
• if you want to write jUnit for class A that is dependent on: log, dbDao, jmsHelper, etc... - there are few different schools, but for simplicity let’s say that you'll write mocks for each of them
• then if you'll have:
• class B,C dependent on jmsHelper and log
• class D,E dependent on dbDao and log
• etc ...
• and there is team of people working on same code

Result => we will see developer per class / copy-paste pattern.

What if instead standard eclipse jUnit plugin you’ll use tool that where you'll create "mappers/patterns" for each class that will produce that mocks automatically?

Then you'll run "generator" and it will create same structure as eclipse/intelliJ "right click", but with all mocks already copied - no need to create boilerplate code by each developer separately?

And … there will be no real test code there - plain output jUnit just like in "right click".

If I calculate it properly, I spend like 50% of testing time - writing boilerplate code [setting up all mocks, adding standard "patterns", etc...].

To speed up it I usually finish with copy-paste pattern which from other hand I'm extreme oppositionist.

Using generators I'll not to do it again - yes, of course I need to do it once at beginning, maybe some maintenance, but then it will be reused, and whole solution will be consistent.

So, now we should be on same page.

At first I was skeptic if something like that could exist, but ..

Generators:

I've take 4 generators for review:

• AgitarOne
• JTest - Parasoft
• Randoop
• CodePro AnalytiX - google

there are probably more - but to speed up process of speeding up process I  also have some time constrains  ;-)

Test scenario:

1.     Choose few most “typical” classes for that project – together easy and more complicated.

2.     Using different tools generate test cases.

3.     Compare generated jUnits from usability perspective (what was generated, how many methods, what amount of code was there,..)  also from percentage of code covered.

Output:

• AgitarOne - quite nice, but is not generating plain jUnits, there is strong dependency on Agitar version of jUnits and if you don’t have license it will not work - so I’ll not take it into consideration [commercial]
• Randoop - created random "stress" test cases, quite different from others - delivers almost full tests, finds typical corner case scenarios (NPE, OutOfIndex, ..) [opensource]
• jTest Parasoft - created boilerplate test code using plain jUnit (which is significant plus compared to Agitar) - it works just like standard jUnit eclipse/intelliJ plugin which means one test for one method, no code analysis[commercial] 
• CodePro AnalytiX - created boilerplate test code using plain jUnit and EasyMock, with static code analysis - created number of test per method depends on "if-cases" which was quite good especially in the case when we go through existing stuff - definitely that will speed up writing, there are other cool tools in the pack, supporting: code coverage, dependency analysis, metrics computation, audit, etc... which also is pretty cool as developer don't need to wait till Sonar give you stats after commits - you can have it on-demand [freeware]

Recommendation / Winner:

CodePro AnalytiX

Discussion:

CodePro is good call - is generating lots of boilerplate code for a developer and save lots of time. Still complex code cannot be completely automated, and developer need to fill gaps - but it is showing you that gaps, suggesting what should be tested.

In different scenario (and I'm thinking also about code reviews) I will give a try to randoop - as is generating whole test, developer  don't need to touch the code (almost ;-) ), and is testing specific corner cases - which you can miss otherwise.

Note:

Of course there are some promises that "generators will exercise" code, which at end will look like (i.e.: method that is adding two ints):

public void testAdd_1() throws Exception

{

     int x=1;

     int y=2;

     int result = Tested.add(x, y);

    // add test code here

}

Still developer need to go there and:

- add asserts,

- change name of the method to something meaningful,

- add comments,

- [change whatever is specific to your team process/code quality]

I my opinion CodePro is really good replacement for standard eclipse jUnit generator - in worst case (day first) you'll have exactly same output, in best case - you'll need to add only asserts!