SVN hooks to automatically check code with PMD (or checkstyle)

I just finished PoC for creating SVN hook with PMD (easily replaced with PMD or  git instead svn).

Functionality:

Every time developer commit java source code to repository (svn), pre-commit hook will call PMD http://pmd.sourceforge.net/. If there will be violations - error with detailed description will be generated, and developer will be stopped from committing bad code. 

Solution is based on standard svn example for validate-files. I did PoC on windows machine - so I have to spend some additional time to fight with .bat->.py integration, but mac/linux should me much more easier /straight forward solution.
Prerequisites are: svn + java + python + pmd.

Installation steps:
1) Create your SVN repository:

svnadmin create c:\svnrepo

2) Copy into hooks directory: (c:\svnrepo\hooks)
2.a) pre-commit.py

#!/usr/bin/env python 
"""Subversion pre-commit hook script that runs PMD static code analysis.

Functionality:
Runs PMD checks on java source code.
Commit will be rejected if PMD rules are voilated.
If there are more than 40 files committed at once - commit will be rejected.
Don't kill SVN server - commit in smaller chunks. 
To avoid PMD checks - put NOPMD into SVN log.
The script expects a pmd-check.conf file placed in the conf dir under
the repo the commit is for."""
 
import sys
import os
import subprocess
import fnmatch
import tempfile
 
# Deal with the rename of ConfigParser to configparser in Python3
try:
    # Python >= 3.0
    import configparser
except ImportError:
    # Python < 3.0
    import ConfigParser as configparser
 
class Commands:
    """Class to handle logic of running commands"""
    def __init__(self, config):
        self.config = config
 
    def svnlook_changed(self, repo, txn):
        """Provide list of files changed in txn of repo"""
        svnlook = self.config.get('DEFAULT', 'svnlook')
        cmd = "%s changed -t %s %s" % (svnlook, txn, repo)
        # sys.stderr.write("Command:: %s\n" % cmd)
        p = subprocess.Popen(cmd, shell=True,
                             stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 
        lines = (line.strip() for line in p.stdout)
        # Only if the contents of the file changed (by addition or update)
        # directories always end in / in the svnlook changed output
        changed = [line[4:] for line in lines if line[-1] != "/"
            and line[0] in ("A","U") ]

        # wait on the command to finish so we can get the
        # returncode/stderr output
        data = p.communicate()
        if p.returncode != 0:
            sys.stderr.write(data[1].decode())
            sys.exit(2)
 
        return changed
 
    def svnlook_getlog(self, repo, txn):
        """ Gets content of svn log"""
        svnlook = self.config.get('DEFAULT', 'svnlook')
 
        cmd = "%s log -t %s %s" % (svnlook, txn, repo)
 
        p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, 
                             stderr=subprocess.PIPE)
        data = p.communicate()
 
        return (p.returncode, data[0].decode())
 
   
    def svnlook_getfile(self, repo, txn, fn, tmp):
        """ Gets content of svn file"""
        svnlook = self.config.get('DEFAULT', 'svnlook')
 
        cmd = "%s cat -t %s %s %s > %s" % (svnlook, txn, repo, fn, tmp)
 
        p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, 
                             stderr=subprocess.PIPE)
        data = p.communicate()
 
        return (p.returncode, data[1].decode())
 
    def pmd_command(self, repo, txn, fn, tmp):
        """ Run the PMD scan over created temporary java file"""
        pmd = self.config.get('DEFAULT', 'pmd')
        pmd_rules = self.config.get('DEFAULT', 'pmd_rules')
 
        cmd = "%s -f text -R %s -d %s" % (pmd, pmd_rules, tmp)
 
        p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, 
                             stderr=subprocess.PIPE)
        data = p.communicate()
 
        # pmd is not working on error codes ..
        return (p.returncode, data[0].decode())
 
 
def main(repo, txn):
    exitcode = 0
    config = configparser.SafeConfigParser()
    config.read(os.path.join(repo, 'conf', 'pmd-check.conf'))
    commands = Commands(config)
 
    # check if someone put magic string to not process code with PMD
    (returncode, log) = commands.svnlook_getlog(repo, txn)
    if returncode != 0:
        sys.stderr.write(
            "\nError retrieving log from svn " \
            "(exit code %d):\n" % (returncode))
        sys.exit(returncode);
       
    if "NOPMD" in log:
        sys.stderr.write("No PMD check - mail should be sent instead.")
        sys.exit(0)
       
    # get list of changed files during this commit
    changed = commands.svnlook_changed(repo, txn)
 
    # this happens when you adding new project to repo
    if len(changed) == 0:
        sys.stderr.write("No files changed in SVN!!!\n")
        sys.exit(0)
 
    # we don't want to kill svn server or wait hours for commit
    if len(fnmatch.filter(changed, "*.java")) >= 40:
                sys.stderr.write(
            "Too many files to process, try commiting " \
            " less than 40 java files per session \n" \
            " Or put 'NOPMD' in comment, if you need " \
            " to work with bigger chunks!\n")
        sys.exit(1)
 
    # create temporary file
    f = tempfile.NamedTemporaryFile(suffix='.java',prefix='x',delete=False)
    f.close()
 
    # only java files
    for fn in fnmatch.filter(changed, "*.java"):
        (returncode, err_mesg) = commands.svnlook_getfile(
            repo, txn, fn, f.name)
        if returncode != 0:
            sys.stderr.write(
                "\nError retrieving file '%s' from svn " \
                "(exit code %d):\n" % (fn, returncode))
            sys.stderr.write(err_mesg)
           
        (returncode, err_mesg) = commands.pmd_command(
            repo, txn, fn, f.name)
        if returncode != 0:
            sys.stderr.write(
                "\nError validating file '%s'" \
                "(exit code %d):\n" % (fn, returncode))
            sys.stderr.write(err_mesg)
            exitcode = 1
        if len(err_mesg) != 0:
            sys.stderr.write(
                "\nPMD violations in file '%s' \n" % fn)
            sys.stderr.write(err_mesg)
            exitcode = 1
           
    os.remove(f.name)
    return exitcode
 
if __name__ == "__main__":
    if len(sys.argv) != 3:
        sys.stderr.write("invalid args\n")
        sys.exit(1)
 
    try:
        sys.exit(main(sys.argv[1], sys.argv[2]))
    except configparser.Error as e:
       sys.stderr.write("Error with the pmd-check.conf: %s\n" % e)
        sys.exit(1)

2.b) pre-commit.bat [ if this is windows ]

c:/python27/python pre_commit.py %1 %2
exit %ERRORLEVEL%;

you can change paths to python/repository


3) Copy pmd-check.conf into conf directory (c:\svnrepo\conf)

[DEFAULT]
svnlook = c:\\opt\\svn\\svnlook.exe
pmd = c:\\opt\\pmd\\bin\\pmd.bat
pmd_rules = java-basic,java-braces,java-clone,java-codesize,java-comments,java-design,java-empty,java-finalizers,java-imports,java-j2ee,java-javabeans,java-junit,java-logging-java,java-naming,java-optimizations,java-strictexception,java-strings,java-typeresolution,java-unnecessary,java-unusedcode
 
#not in scope
#java-controversial, java-coupling, java-android, java-javabeans, java-logging-jakarta-commons, java-sunsecure
#java-migrating, java-migrating_to_13, java-migrating_to_14, java-migrating_to_15, java-migrating_to_junit14

And this is file that you should change depending on your svn/pmd installation paths and what PMD rules you want to check.


Final tip: To avoid PMD check on top of standard PMD suppressers - you can just put "NOPMD" into comment.

Have fun!
/Jaro

automatic jUnit test generators - review

This could be scary one. 

When I heard term “automatic jUnit test generator” I was thinking: “no one should go that path”, ”units test should be written by the developers”, “do you want to replace devs?”, “test should be written before code!”.

But I gave it closer look, and actually “closer think” – did I really expect that there is some AI module which will go through code and automatically write tests – if so why there is no AI which automatically write code in first place … [maybe soon ;-)]

Before you send thousands of hate comments, let’s give it quick look what is definition and what should you expect by automatic jUnit test generator:

• it's everywhere - right click on class in eclipse or intelliJ and you’ll have it - that's point first and should be breakthrough in your thinking about generators ;-)
• if you want to write jUnit for class A that is dependent on: log, dbDao, jmsHelper, etc... - there are few different schools, but for simplicity let’s say that you'll write mocks for each of them
• then if you'll have:
• class B,C dependent on jmsHelper and log
• class D,E dependent on dbDao and log
• etc ...
• and there is team of people working on same code

Result => we will see developer per class / copy-paste pattern.

What if instead standard eclipse jUnit plugin you’ll use tool that where you'll create "mappers/patterns" for each class that will produce that mocks automatically?

Then you'll run "generator" and it will create same structure as eclipse/intelliJ "right click", but with all mocks already copied - no need to create boilerplate code by each developer separately?

And … there will be no real test code there - plain output jUnit just like in "right click".

If I calculate it properly, I spend like 50% of testing time - writing boilerplate code [setting up all mocks, adding standard "patterns", etc...].

To speed up it I usually finish with copy-paste pattern which from other hand I'm extreme oppositionist.

Using generators I'll not to do it again - yes, of course I need to do it once at beginning, maybe some maintenance, but then it will be reused, and whole solution will be consistent.

So, now we should be on same page.

At first I was skeptic if something like that could exist, but ..

Generators:

I've take 4 generators for review:

• AgitarOne
• JTest - Parasoft
• Randoop
• CodePro AnalytiX - google

there are probably more - but to speed up process of speeding up process I  also have some time constrains  ;-)

Test scenario:

1.     Choose few most “typical” classes for that project – together easy and more complicated.

2.     Using different tools generate test cases.

3.     Compare generated jUnits from usability perspective (what was generated, how many methods, what amount of code was there,..)  also from percentage of code covered.

Output:

• AgitarOne - quite nice, but is not generating plain jUnits, there is strong dependency on Agitar version of jUnits and if you don’t have license it will not work - so I’ll not take it into consideration [commercial]
• Randoop - created random "stress" test cases, quite different from others - delivers almost full tests, finds typical corner case scenarios (NPE, OutOfIndex, ..) [opensource]
• jTest Parasoft - created boilerplate test code using plain jUnit (which is significant plus compared to Agitar) - it works just like standard jUnit eclipse/intelliJ plugin which means one test for one method, no code analysis[commercial] 
• CodePro AnalytiX - created boilerplate test code using plain jUnit and EasyMock, with static code analysis - created number of test per method depends on "if-cases" which was quite good especially in the case when we go through existing stuff - definitely that will speed up writing, there are other cool tools in the pack, supporting: code coverage, dependency analysis, metrics computation, audit, etc... which also is pretty cool as developer don't need to wait till Sonar give you stats after commits - you can have it on-demand [freeware]

Recommendation / Winner:

CodePro AnalytiX

Discussion:

CodePro is good call - is generating lots of boilerplate code for a developer and save lots of time. Still complex code cannot be completely automated, and developer need to fill gaps - but it is showing you that gaps, suggesting what should be tested.

In different scenario (and I'm thinking also about code reviews) I will give a try to randoop - as is generating whole test, developer  don't need to touch the code (almost ;-) ), and is testing specific corner cases - which you can miss otherwise.

Note:

Of course there are some promises that "generators will exercise" code, which at end will look like (i.e.: method that is adding two ints):

public void testAdd_1() throws Exception

{

     int x=1;

     int y=2;

     int result = Tested.add(x, y);

    // add test code here

}

Still developer need to go there and:

- add asserts,

- change name of the method to something meaningful,

- add comments,

- [change whatever is specific to your team process/code quality]

I my opinion CodePro is really good replacement for standard eclipse jUnit generator - in worst case (day first) you'll have exactly same output, in best case - you'll need to add only asserts!