Progressing from prototype: incremental Groovy/Java joint compiler

Everything I blogged about previously was based on a prototype. A bit too much hacking here and there but I wanted to see if the hard problems could be solved rather than waste time on simple problems that were 'just a matter of coding' (hah). Since then I've taken a radical approach to improving the codebase... I've thrown everything away and started again, and the code is much better because of that.

But of all the things that worried me about the prototype the worst thing was probably having no tests. I'd get grails to build then realise I'd broken something that allowed groovyc to build. I couldn't go on without tests!

Testing
I spent a day looking at the JDT compiler and builder tests and created a variant of the harness that allowed me to write mixed groovy/java tests. The resultant tests are beautiful. Here is a basic compiler test:

public void testStandaloneGroovyFile() {
  this.runConformTest(new String[] {
    "p/X.groovy",
    "package p;\n" + 
    "public class X {\n" + 
    "  public static void main(String[] argv) {\n"+
    "    print \"success\"\n" + 
    "  }\n"+
    "}\n",
  },"success");  
}

It is so simple I don't even have to explain to you what it is doing... and the harness is flexible enough to support checking for expected errors, compiling multiple files. And best of all everything is one place, right there in the test method - each test isn't a test method plus a bit of xml config plus some source files from that directory over there (that you worked on with the command line then forgot to refresh in eclipse so the contents didn't get uploaded to the repo on sync - sigh).

From that humble test beginning, here is a test that is now working:

public void testDefaultValueMethods1() {
  this.runConformTest(new String[] {
    "p/C.java",
    "package p;\n" +
    "public class C {\n"+
    "  public static void main(String[] argv) {\n"+
    "    G o = new G();\n"+
    "    o.m(\"abc\",3);\n"+
    "    o.m(\"abc\");\n"+
    "  }\n"+
    "}\n",    

    "p/G.groovy",
    "package p;\n"+
    "public class G {\n" + 
    "  public void m(String s,Integer i=3) { print s }\n"+
    "}\n",
  },
  "abcabc");
}

That shows calling methods with default parameters from .java files.

On top of those compiler tests, I can also create builder tests that verify the usage of this modified compiler as it would be exercised when incrementally building an eclipse project. Here is a test that checks incremental compilation for groovy/java:

public void testIncrementalCompilationTheBasics() throws JavaModelException {
    IPath projectPath = env.addProject("Project");
    env.addExternalJars(projectPath, Util.getJavaClassLibs());
    env.addGroovyJars(projectPath);
    fullBuild(projectPath);
    
    // remove old package fragment root so that names don't collide
    env.removePackageFragmentRoot(projectPath, "");
    
    IPath root = env.addPackageFragmentRoot(projectPath, "src");
    env.setOutputFolder(projectPath, "bin");

    env.addClass(root, "pkg", "Hello",
      "package pkg;\n"+
      "public class Hello {\n"+
      "   public static void main(String[] args) {\n"+
      "      System.out.println(new GHello().run());\n"+
      "   }\n"+
      "}\n"
      );

    env.addGroovyClass(root, "pkg", "GHello",
      "package pkg;\n"+
      "public class GHello {\n"+
      "   public int run() { return 12; }\n"+
      "}\n"
      );
      
    incrementalBuild(projectPath);
    expectingCompiledClassesV("pkg.Hello","pkg.GHello");
    expectingNoProblems();
    executeClass(projectPath, "pkg.Hello", "12", "");

    // whitespace change to groovy file
    env.addGroovyClass(root, "pkg", "GHello",
        "package pkg;\n"+
        "public class GHello {\n"+
        "  \n"+ // new blank line
        "   public int run() { return 12; }\n"+
        "}\n"
        );
    incrementalBuild(projectPath);
    expectingCompiledClassesV("pkg.GHello");
    expectingNoProblems();

    // structural change to groovy file 
    // - did the java file record its dependency correctly?
    env.addGroovyClass(root, "pkg", "GHello",
        "package pkg;\n"+
        "public class GHello {\n"+
        // return type now String
        "   public String run() { return \"abc\"; }\n"+ 
        "}\n"
        );
    incrementalBuild(projectPath);
    expectingCompiledClassesV("pkg.GHello","pkg.Hello");
    expectingNoProblems();
}

As you can see, it is pretty straightforward:

- Build a pair of files (one groovy, one java)
- make a whitespace change to the groovy file and incremental build (this should only rebuild the groovy file).
- make a structural change to the groovy file and incremental build (should rebuild both groovy and the java file dependent on it)

I can even run the code at any time to check it is OK (I can *even* disassemble it and check the internal bytecode structure - love those JDT guys!)

With the ability to test, productivity has massively improved. The existence of the tests enabling the confidence to make more serious changes and refactorings as the code is developed.

Architecture
The architecture of the prototype was ... well perhaps 'architecture' isn't really the right word here. All jars that might be needed were dumped in a directory, then glued together with 'duct tape coding' - just join the necessary classes together and if that fails, apply more tape/code. The driving goals around the architecture of the new version are:

- do not damage JDT as a Java compiler (do not modify it to support a new language to the detriment of how fast and great it is at compiling Java). Changes are needed to JDT to plug groovy in, but they should be absolutely minimal.
- preserve modularity. Consume jdt.core as a plugin/bundle, consume the groovy compiler and dependencies as a plugin/bundle and create the glue code in a new plugin/bundle.
- ... do NOT damage JDT as a Java compiler!
- Design for future upgrades. Make the changes to JDT and the dependencies on groovy as simple as possible so that moving to a new JDT (eg. that in Eclipse 3.5) or a new groovy is simple.

And so far this has been achieved. The JDT has been modified to attempt to load a 'LanguageSupport' implementation dynamically. If this fails then it behaves as it always did - a pure Java compiler. If this succeeds then the language support hooks in where necessary. The LanguageSupport implementation is in a separate plugin and wires JDT to the existing groovy compiler plugin. The particularly great feature is that if you run an environment without the groovy language support plugin, then jdt just behaves as normal - it doesn't have a hard dependency on the groovy plugin which would cause it to fail if the groovy language support was missing.

Latest version
So what is the state of things now if I started again? My 50 tests are confirming that the basics already behave *but* generics is causing a headache (when doesn't it...) as I'm having some trouble turning the JDT representation of a type into what groovy wants to see, with respect to type variables, etc.

Where/When can I get this thing!!!
Within two months from now there should be something downloadable. We are looking to integrate this compiler into the existing Groovy eclipse plugin.