Groovy eclipse: from alpha towards milestone

After the initial splash of releasing the alpha, I don't want anyone to think we aren't working hard to improve things and get to that first milestone. There are already numerous fixes and enhancements and I'd like to thank everyone who has given it a spin.

In this post I'll pull together a flavour of what has changed since that first alpha - and perhaps when you see the kinds of issues that have been reported, it may give you the confidence to give it a try!

Back level groovyc
The most serious issue that arose was simply that we were based on an old development driver of groovy 1.7. It was a couple of months out of date, and the jars built at that time were incomplete. This is now remedied and the compiler has been rebased on the recently released official groovy 1.7 beta 1.

Generics checking of groovy code
Some of you have been playing with this feature more than I anticipated. It quickly became apparent that there was a need to support @SuppressWarnings so that users could hide warnings if they didn't want to mess around trying to correctly specify generic constraints. It now works:


(*cough* Just pretend you didn't see the final 's' in SuppressWarnings that isn't highlighted correctly...)

Coping with bad code
When any parser is integrated into an IDE for some filetype, and that IDE supports features like early error indication (as you type), that parser needs to be really robust. In Eclipse full compiles kick off for the code in the editor any time the user stops typing for even a moment - and not just when they Ctrl+S to save. This means the parser and compiler are often being asked to process incomplete/unfinished code. We are actively fixing issues where incomplete code crashes the groovy compiler.

JUnit
Based on a requirement from the Spock test framework we now have support for a monospace font in the JUnit failure trace view. This is surprisingly useful (even when not using Spock) and I now run with it as my default mode for JUnit. This shot shows the aligned output in the failure window (click to zoom):



The option to set the font is under Window>Preferences>Groovy

Error positions
The Eclipse Java support is great at positioning the 'squigglies' at just the right location to indicate where the problem is in your code. As any alpha users will have noticed, things aren't so great for groovy code. We are working to fix these situations.

Here is the 'before' shot taken on the first alpha (don't laugh):



We always report the errors, we were just having a few issues with properly locating them in the editor:
- the error that Gravy doesn't implement Comparable is recorded against the annotation
- the warning about Stack is not quite underlining the whole word
- the error about violated generic bounds is one character to the left of where it should be
(admittedly this was taken on Windows which I think was worse than mac/linux)

And now with recent fixes:



(there's that problem with the 's' in SuppressWarnings again...)

AST Transforms
The alpha had very basic support for AST Transforms. By specifying the jar containing the transform in the groovy.properties file in the root of the project, it would be picked up and any transforms within available during the project build. However, already we have seen cases where basic transforms (for example those triggered by @Entity) require much more than a single jar. For now we are still making AST transform support 'opt-in' but if you specify this particular entry in the groovy.properties file:

org.eclipse.jdt.core.compiler.groovy.groovyClassLoaderPath=%projclasspath%

Then the classloader responsible for loading AST transforms and their dependencies will run with the same classpath as the project. This will be the default behaviour before release (I think) but for now it is a helpful way for us to work out if transforms are involved in particular bugs that we get in.

Boring compiler internal bugs
I'm afraid I don't have any fancy screenshots here, and these are the bugs that can take a lot of effort to fix. We've had some great jira issues raised by people really mixing it up with groovy/java generic types extended/implemented via other groovy/java types. The basics are behaving pretty well and really it is only when users stray out into generics, covariance, etc that the compiler starts to misbehave. However, we can only fix bugs we know about so if you are having a problem, please let us know ! I am aware that some grails users are having issues with their projects but I can't seem to recreate it locally - if you have a failing grails project and can share it with us, please do!


Finally, we are also running Continuous Integration now, so we can get those fixes out faster.This means you should regularly update and if you haven't installed at all yet, what are you waiting for?


Eclipse update site for Eclipse 3.5:
http://dist.codehaus.org/groovy/distributions/greclipse/snapshot/e3.5

Eclipse update site for Eclipse 3.4.2:
http://dist.codehaus.org/groovy/distributions/greclipse/snapshot/e3.4

(The 3.4 stream is a bit behind the 3.5 stream, but will receive periodic updates)

Raise any issues at the JIRA:
http://jira.codehaus.org/browse/GRECLIPSE

Progressing from prototype: incremental Groovy/Java joint compiler

Everything I blogged about previously was based on a prototype. A bit too much hacking here and there but I wanted to see if the hard problems could be solved rather than waste time on simple problems that were 'just a matter of coding' (hah). Since then I've taken a radical approach to improving the codebase... I've thrown everything away and started again, and the code is much better because of that.

But of all the things that worried me about the prototype the worst thing was probably having no tests. I'd get grails to build then realise I'd broken something that allowed groovyc to build. I couldn't go on without tests!

Testing
I spent a day looking at the JDT compiler and builder tests and created a variant of the harness that allowed me to write mixed groovy/java tests. The resultant tests are beautiful. Here is a basic compiler test:

public void testStandaloneGroovyFile() {
  this.runConformTest(new String[] {
    "p/X.groovy",
    "package p;\n" + 
    "public class X {\n" + 
    "  public static void main(String[] argv) {\n"+
    "    print \"success\"\n" + 
    "  }\n"+
    "}\n",
  },"success");  
}

It is so simple I don't even have to explain to you what it is doing... and the harness is flexible enough to support checking for expected errors, compiling multiple files. And best of all everything is one place, right there in the test method - each test isn't a test method plus a bit of xml config plus some source files from that directory over there (that you worked on with the command line then forgot to refresh in eclipse so the contents didn't get uploaded to the repo on sync - sigh).

From that humble test beginning, here is a test that is now working:

public void testDefaultValueMethods1() {
  this.runConformTest(new String[] {
    "p/C.java",
    "package p;\n" +
    "public class C {\n"+
    "  public static void main(String[] argv) {\n"+
    "    G o = new G();\n"+
    "    o.m(\"abc\",3);\n"+
    "    o.m(\"abc\");\n"+
    "  }\n"+
    "}\n",    

    "p/G.groovy",
    "package p;\n"+
    "public class G {\n" + 
    "  public void m(String s,Integer i=3) { print s }\n"+
    "}\n",
  },
  "abcabc");
}

That shows calling methods with default parameters from .java files.

On top of those compiler tests, I can also create builder tests that verify the usage of this modified compiler as it would be exercised when incrementally building an eclipse project. Here is a test that checks incremental compilation for groovy/java:

public void testIncrementalCompilationTheBasics() throws JavaModelException {
    IPath projectPath = env.addProject("Project");
    env.addExternalJars(projectPath, Util.getJavaClassLibs());
    env.addGroovyJars(projectPath);
    fullBuild(projectPath);
    
    // remove old package fragment root so that names don't collide
    env.removePackageFragmentRoot(projectPath, "");
    
    IPath root = env.addPackageFragmentRoot(projectPath, "src");
    env.setOutputFolder(projectPath, "bin");

    env.addClass(root, "pkg", "Hello",
      "package pkg;\n"+
      "public class Hello {\n"+
      "   public static void main(String[] args) {\n"+
      "      System.out.println(new GHello().run());\n"+
      "   }\n"+
      "}\n"
      );

    env.addGroovyClass(root, "pkg", "GHello",
      "package pkg;\n"+
      "public class GHello {\n"+
      "   public int run() { return 12; }\n"+
      "}\n"
      );
      
    incrementalBuild(projectPath);
    expectingCompiledClassesV("pkg.Hello","pkg.GHello");
    expectingNoProblems();
    executeClass(projectPath, "pkg.Hello", "12", "");

    // whitespace change to groovy file
    env.addGroovyClass(root, "pkg", "GHello",
        "package pkg;\n"+
        "public class GHello {\n"+
        "  \n"+ // new blank line
        "   public int run() { return 12; }\n"+
        "}\n"
        );
    incrementalBuild(projectPath);
    expectingCompiledClassesV("pkg.GHello");
    expectingNoProblems();

    // structural change to groovy file 
    // - did the java file record its dependency correctly?
    env.addGroovyClass(root, "pkg", "GHello",
        "package pkg;\n"+
        "public class GHello {\n"+
        // return type now String
        "   public String run() { return \"abc\"; }\n"+ 
        "}\n"
        );
    incrementalBuild(projectPath);
    expectingCompiledClassesV("pkg.GHello","pkg.Hello");
    expectingNoProblems();
}

As you can see, it is pretty straightforward:

- Build a pair of files (one groovy, one java)
- make a whitespace change to the groovy file and incremental build (this should only rebuild the groovy file).
- make a structural change to the groovy file and incremental build (should rebuild both groovy and the java file dependent on it)

I can even run the code at any time to check it is OK (I can *even* disassemble it and check the internal bytecode structure - love those JDT guys!)

With the ability to test, productivity has massively improved. The existence of the tests enabling the confidence to make more serious changes and refactorings as the code is developed.

Architecture
The architecture of the prototype was ... well perhaps 'architecture' isn't really the right word here. All jars that might be needed were dumped in a directory, then glued together with 'duct tape coding' - just join the necessary classes together and if that fails, apply more tape/code. The driving goals around the architecture of the new version are:

- do not damage JDT as a Java compiler (do not modify it to support a new language to the detriment of how fast and great it is at compiling Java). Changes are needed to JDT to plug groovy in, but they should be absolutely minimal.
- preserve modularity. Consume jdt.core as a plugin/bundle, consume the groovy compiler and dependencies as a plugin/bundle and create the glue code in a new plugin/bundle.
- ... do NOT damage JDT as a Java compiler!
- Design for future upgrades. Make the changes to JDT and the dependencies on groovy as simple as possible so that moving to a new JDT (eg. that in Eclipse 3.5) or a new groovy is simple.

And so far this has been achieved. The JDT has been modified to attempt to load a 'LanguageSupport' implementation dynamically. If this fails then it behaves as it always did - a pure Java compiler. If this succeeds then the language support hooks in where necessary. The LanguageSupport implementation is in a separate plugin and wires JDT to the existing groovy compiler plugin. The particularly great feature is that if you run an environment without the groovy language support plugin, then jdt just behaves as normal - it doesn't have a hard dependency on the groovy plugin which would cause it to fail if the groovy language support was missing.

Latest version
So what is the state of things now if I started again? My 50 tests are confirming that the basics already behave *but* generics is causing a headache (when doesn't it...) as I'm having some trouble turning the JDT representation of a type into what groovy wants to see, with respect to type variables, etc.

Where/When can I get this thing!!!
Within two months from now there should be something downloadable. We are looking to integrate this compiler into the existing Groovy eclipse plugin.

Grails built with Eclipse JDT Incremental Joint groovy/java compiler

Just a quick entry on this. Each time I tackle compiling a new mixed java/groovy codebase, a new set of issues appear. No surprises really, just features I knew were not implemented yet that the particular codebase exploits. But here is the good news (this is Eclipse JDT compiler debug output):

>FULL BUILD STATS for: grails
>   compiled 139801 lines in 63692ms:2194.9lines/s
>   parse: 7437 ms (11.6%), resolve: 1150 ms (1.8%),
>   analyze: 50634 ms (79.4%), generate: 2836 ms (4.4%)
Recording new state : State for grails (#0 @ Tue May 26 11:47:22 PDT 2009)

Remember, those 'lines' counts are just including the java code right now. And don't read much into the time taken as I have more debug code than real code right now :)

Anyway, there were a couple of interesting cases in getting grails to compile that are worth talking about, just to give a feel for what is possible with this incremental compiler:

Properties
In Grails, the GROOVY type Mapping defines:

  boolean tablePerHierarchy = true

This is then referenced from a JAVA type (GrailsDomainBinder) through

  mapping.isTablePerHierarchy()

The modified JDT compiler handles this no problem.


Default parameter values
The GROOVY type GrailsPluginUtils defines this method with a default value for the closure parameter:

static synchronized Resource[] getArtefactResources(
 String basedir,
 Closure resourceResolver = DEFAULT_RESOURCE_RESOLVER) {

The JAVA type GrailsAwareGroovyTestSuite then accesses it through:

GrailsPluginUtils.getArtefactResources(grailsSettings.getBaseDir().getPath());

(supplying just the first parameter on the call)

The modified JDT compiler handles this no problem.


Generics
The GROOVY type NaturalId looks like this:

public class NaturalId {
   List<String> propertyNames = []
   boolean mutable = false
}

The JAVA type GrailsDomainBinder then attempts to access the propertyNames through a getter and treat the elements as Strings.

for (String propertyName : naturalId.getPropertyNames()) {

The modified JDT compiler handles this no problem.

But, of course, the best thing is that incremental compilation works. Let me break that property 'propertyNames' - change the type to just 'List' and save:

Starting build of grails @ Tue May 26 12:11:56 PDT 2009
Found source delta for: grails
Clearing last state : State for grails (#3 @ Tue May 26 11:58:31 PDT 2009)
INCREMENTAL build
Compile this changed source file src/.../NaturalId.groovy
About to compile src/.../NaturalId.groovy
Type has structural changes ...\NaturalId
 will look for dependents of NaturalId in org/codehaus/groovy/grails/orm/hibernate/cfg
Writing changed class file NaturalId.class
Found match in org.codehaus.groovy.grails.orm.hibernate.cfg to NaturalId
 adding affected source file src/.../GrailsDomainBinder.java
... edited out a list of other affected types ...
About to compile src/.../GrailsDomainBinder.java
Recording new state : State for grails (#4 @ Tue May 26 12:11:56 PDT 2009)
Finished build of grails @ Tue May 26 12:12:04 PDT 2009

it recognized that GrailsDomainBinder need a rebuild and now recorded an error against the attempted iteration:

Type mismatch: cannot convert from element type Object to String 

Fixing the problem (converting propertyNames back to type List<String>) and resaving gives a similar incremental build log and the problem vanishes.

Anyway, that is probably the last codebase I'll experiment upon - the size of the task ahead is apparent, so the focus now is making what there is robust enough to distribute to other users.

Modifying JDT to Joint compile Groovy #2

Just a follow up post on progress with the modified compiler.

With GANT working, I wanted to try it on another codebase. I initially chose the groovy eclipse plugin - and it was quite straightforward to fix a few things in the modified compiler in order to get that building. But I had a hard time running the tests. So I decided to try to go for a bit more of a challenge and build the groovy compiler itself (as an example of a mixed codebase) from:

https://svn.codehaus.org/groovy/trunk/groovy/groovy-core
(version 16084)

This really pushed the modified compiler and revealed quite a few gotchas that it hadn't previously been handling properly. Types like GroovyShellTestCase (groovy type) which extends GroovyTestCase (java type) and also has a GroovyShell (java type) delegate defined caused no end of headaches. The problem I had was that compilation would succeed but the tests would just explode. The curse of dynamic programming: everything goes wrong at runtime.

I eventually figured out how to run some groovy compiler tests inside eclipse. I wasn't sure which of the suites to try and launch so just selected the "src/test/(defaultpackage)" then "Right Click > Run as Junit". That comes to about 5400 - but I think includes some duplicates. It is certainly enough to be getting on with! However, debugging failures was horrible and I was reduced to comparing my .class file with that from a groovy distribution. Observing what methods had or hadn't been created, or why the method bodies differed and working out how the modified compiler had made such awful mistakes.

But today I'm back on track, it came down to a few things
- ensuring the representation of groovy types which the JDT then sees is absolutely correct.
- realising that the verifier visitor runs twice in joint compilation, and the implications that first run has on downstream code transformations (like DelegateAstTransformation)
- realising that the build process also needed me to run the DGMConverter app to create some classes needed later (DOH, this had me stuck for a while)

Today I'm down to:



So not 100% passing, but I'm not sure all the tests were designed to launch in eclipse in the way I am launching them (and in fact some of the failing cases pass when run standalone)

When it is behaving, the incremental builds are great. Here I made a structural change to ComplexCommandSupport.groovy and saved it - see the JDT doing the necessary impact analysis on what else needs building:

Starting build of groovyc @ Wed May 13 14:43:20 PDT 2009
Found source delta for: groovyc
Clearing last state : State for groovyc (#2 @ Wed May 13 14:43:10 PDT 2009)
INCREMENTAL build
Compile this changed source file src/main/org/codehaus/groovy/tools/shell/ComplexCommandSupport.groovy
About to compile src/main/org/codehaus/groovy/tools/shell/ComplexCommandSupport.groovy
Writing changed class file ComplexCommandSupport$_closure1.class
Type has structural changes org\codehaus\groovy\tools\shell\ComplexCommandSupport
  will look for dependents of ComplexCommandSupport in org/codehaus/groovy/tools/shell
Writing changed class file ComplexCommandSupport.class
Writing changed class file ComplexCommandSupport$_closure1_closure3.class
Writing changed class file ComplexCommandSupport$_createCompletors_closure2.class
Found match in org.codehaus.groovy.tools.shell to ComplexCommandSupport
  adding affected source file src/main/org/codehaus/groovy/tools/shell/commands/ShowCommand.groovy
Found match in org.codehaus.groovy.tools.shell to ComplexCommandSupport
  adding affected source file src/main/org/codehaus/groovy/tools/shell/commands/HistoryCommand.groovy
Found match in org.codehaus.groovy.tools.shell to ComplexCommandSupport
  adding affected source file src/main/org/codehaus/groovy/tools/shell/commands/ShadowCommand.groovy
Found match in org.codehaus.groovy.tools.shell to ComplexCommandSupport
  adding affected source file src/main/org/codehaus/groovy/tools/shell/commands/PurgeCommand.groovy
Found match in org.codehaus.groovy.tools.shell to ComplexCommandSupport
Found match in org.codehaus.groovy.tools.shell to ComplexCommandSupport
  adding affected source file src/main/org/codehaus/groovy/tools/shell/commands/RecordCommand.groovy
About to compile src/main/org/codehaus/groovy/tools/shell/commands/ShowCommand.groovy
About to compile src/main/org/codehaus/groovy/tools/shell/commands/HistoryCommand.groovy
About to compile src/main/org/codehaus/groovy/tools/shell/commands/ShadowCommand.groovy
About to compile src/main/org/codehaus/groovy/tools/shell/commands/PurgeCommand.groovy
About to compile src/main/org/codehaus/groovy/tools/shell/commands/RecordCommand.groovy
Skipped over unchanged class file ShowCommand$_closure1_closure5.class
Skipped over unchanged class file ShowCommand$_closure1.class
Skipped over unchanged class file ShowCommand$_closure2.class
Skipped over unchanged class file ShowCommand$_closure3_closure7.class
Skipped over unchanged class file ShowCommand$_closure4.class
Skipped over unchanged class file ShowCommand$_closure2_closure6.class
Skipped over unchanged class file ShowCommand$_closure4_closure8.class
Writing changed class file ShowCommand.class
Skipped over unchanged class file ShowCommand$_closure3.class
Skipped over unchanged class file HistoryCommand$_createCompletors_closure5_closure7.class
Writing changed class file HistoryCommand.class
Skipped over unchanged class file HistoryCommand$_closure1_closure6.class
Skipped over unchanged class file HistoryCommand$_closure1.class
Skipped over unchanged class file HistoryCommand$_closure2.class
Skipped over unchanged class file HistoryCommand$_createCompletors_closure5.class
Skipped over unchanged class file HistoryCommand$_closure4.class
Skipped over unchanged class file HistoryCommand$_closure3.class
Skipped over unchanged class file ShadowCommand$_closure2.class
Skipped over unchanged class file ShadowCommand$_closure4.class
Writing changed class file ShadowCommand.class
Skipped over unchanged class file ShadowCommand$_closure3.class
Skipped over unchanged class file ShadowCommand$_closure1.class
Skipped over unchanged class file PurgeCommand$_closure4.class
Skipped over unchanged class file PurgeCommand$_closure2.class
Skipped over unchanged class file PurgeCommand$_closure1.class
Writing changed class file PurgeCommand.class
Skipped over unchanged class file PurgeCommand$_closure3.class
Skipped over unchanged class file RecordCommand$_recordError_closure5.class
Skipped over unchanged class file RecordCommand$_closure1.class
Skipped over unchanged class file RecordCommand$_closure2.class
Writing changed class file RecordCommand.class
Skipped over unchanged class file RecordCommand$_closure4.class
Skipped over unchanged class file RecordCommand$_closure3.class
Recording new state : State for groovyc (#3 @ Wed May 13 14:43:20 PDT 2009)
Finished build of groovyc @ Wed May 13 14:43:21 PDT 2009

Modifying Eclipse JDT to joint compile groovy

This is a little project I've been working on for a week or so. Right up front I'll say I'm a groovy newbie (grewbie?), but I do know a little bit about compilers.

After getting a little intro to groovy joint compilation (and groovy compiler structure) from Jochen Theodorou, I wondered how hard it would be to go about things slightly differently. Let the Eclipse JDT compiler take control of building a mixed code base but whenever it needed to deal with groovy - call the groovy compiler. If JDT could take control I could see all sorts of things 'just working' in Eclipse, problems upon which we are still expending large amounts of efforts to try and solve for AspectJ. If I could just plug groovy type reference resolution and classfile generation into JDT correctly then references between java and groovy artifacts would work and JDT would ensure incremental compilation worked, even across restarts of eclipse (this latter problem still hasn't been solved for AspectJ).

Effectively I just saw the problem as needing to bring together two phased compilers. The groovy one has many phases whilst the JDT has fewer and they are more of an internal notion than something clearly visible in the source code.

The main problem to address is handling references: java references into groovy files, and groovy references into java files. Clearly we want both and today the groovy joint compiler handles it via stub generation prior to calling javac which can then see the stubs.

For my 'mashup' I considered JDT as 3 phases:
1) diet parsing - where the basic structure of the types are determined, no method internals are processed and no references are chased down
2) complete type bindings - wire up the type references, work out which Foo they mean? com.a.Foo or com.b.Foo
3) code generation - process the method bodies and generate the code (there are really multiple phases here but we don't need to know about anything more granular for now)

These phases exist in groovy too, with other names - I didn't really know which phases they were so had a guess :)

The general algorithm is then:

1) JDT compiler receives a bunch of .groovy and .java files
2) diet parse all the .java files to learn their structure
3) drive the .groovy files through the first few phases (stopping before reference resolution)
4) do some wiring: so JDT can see the result of the groovy parse, and Groovy can see the result of the JDT parse
5) complete type bindings for java - resolving references to other JDT types and to groovy types
6) drive the rest of the groovy phases for each groovy file - resolving references to other groovy types and to JDT types
7) pass the .class files from groovy code gen back to JDT (so it can do incremental analysis)
8) generate the .class files for the .java files

And it worked. It is similar to the joint compilation strategy already employed by groovy but the communication between the compilers is much tighter. Initially a trivial little HelloWorld was all that worked:

-- HelloWorld.java --
public class HelloWorld {
public void run() {
 System.out.println("Hello World");
}

public static void main(String[] argv) {
    System.out.println("HelloWorld.main()");
    new pkg.GroovyClass().m(); // reference to groovy
}
}
--

-- GroovyClass.groovy --
package pkg

public class GroovyClass {
public void m() {
      System.out.println("GroovyClass.m() running")
}
public static void main(String []argv) {
    System.out.println("GroovyClass.main()");
    new HelloWorld().run(); // reference to java
}
}
--

But motivated by that I picked up GANT (Groovy based ANT) from http://gant.codehaus.org as a larger mixed project. This afternoon this successfully compiled:

>FULL BUILD STATS for: Gant_Trunk_160
> compiled 6501 lines in 1201ms:5412.9lines/s
> parse: 424 ms (35.3%), resolve: 56 ms (4.6%), analyze: 5 ms (0.4%), generate: 699 ms (58.2%)
Recording new state : State for Gant_Trunk_160 (#0 @ Mon May 04 17:33:50 PDT 2009)
Finished build of Gant_Trunk_160 @ Mon May 04 17:33:52 PDT 2009

Those stats are possibly not telling the entire story, they may be excluding some numbers for groovy, but it built! And the tests ran (all in eclipse):


But perhaps the most pleasing feature that worked was incremental compilation. I can change any file (groovy or java) and save it, triggering an incremental build. Here I modified an exception class (GantException.groovy) with a structural change (added a field), and the output reports:

INCREMENTAL build
Compile this changed source file src/main/groovy/gant/GantException.groovy
About to compile src/main/groovy/gant/GantException.groovy
Type has structural changes gant\GantException
will look for dependents of GantException in gant
Found match in gant to GantException
   adding affected source file src/main/groovy/gant/MissingTargetException.groovy
Found match in gant to GantException
   adding affected source file src/main/groovy/gant/TargetMissingPropertyException.groovy
Found match in gant to GantException
   adding affected source file src/main/groovy/gant/TargetExecutionException.groovy
About to compile src/main/groovy/gant/MissingTargetException.groovy
About to compile src/main/groovy/gant/TargetMissingPropertyException.groovy
About to compile src/main/groovy/gant/TargetExecutionException.groovy
Recording new state : State for Gant_Trunk_160 (#5 @ Mon May 04 17:41:11 PDT 2009)
Finished build of Gant_Trunk_160 @ Mon May 04 17:41:11 PDT 2009
GroovyIntegration: creating type declaration for gant.GantException

You can see that after saving that structural change, the JDT decided to rebuild a few other files that it knew depended upon it - but total incremental build time was less than one second. If .java files had depended upon the change, they would also have been recompiled.

Now - this is not ready for prime time yet, but proves the idea I had that integration was possible and that a good deal of JDT infrastructure (for incremental builds) would work. All these changes are being made to Eclipse JDT and the use of that compiler in Eclipse but, of course, the Eclipse compiler can be used in batch mode. So what we also have here is a java compiler that will happily allow groovy to be included in the mix and it can be run on the command line ... or from an Ant build script ... perhaps one generated by PDE that is being used to build an OSGi bundle that mixes java and groovy.

Look out for more updates soon...

AspectJ load-time weaving and memory usage

To skip the background and just learn about the new weaver flags to control memory usage - just scroll to the end of this article.

Weaving is a complicated process and to reduce the complexity within the weaver it likes to take control of everything and know what it is dealing with. It builds a large data structure called a World. This World knows all about types - it knows intimate details (the bytecode) of the types being woven, and it knows the basic structure of types from the classpath that are being referenced during weaving (but are not themselves to be woven).

Knowing all this causes Worlds to get quite large, and they don't really give up the space they consume. This is mainly by design, a legacy design from when weaving was more something to play with than something to seriously use and rely on in your production quality application.

This article is about how to solve the Worlds problems (Badum-tish!).

The key data structure in the World is a typemap and the design is that it holds two categories of entity:

1) a representation of a type that would be hard to recover later
2) a representation of a type that would be easy to recover later

What falls into these two categories? From an AspectJ 1.6.4 point of view:
- aspects and anything that gets woven is category (1)
- types just referenced from woven types are category (2)

Category (2) can be basically treated like a cache - where if the information is in memory that is great, but if it isn't we can recover it quickly. Category (1) is the data we can't recovery quickly/easily and so represents a growing fixed part of the World - as more types are woven it gets larger and the space is never released.

For AspectJ 1.6.5 we are looking at what to do here. There are a few issues that with a little work could contribute a lot of benefit to the memory footprint.

Firstly, it is not true that just because something gets woven that it should be considered category (1). Sometimes weaving affects the type structure (intertype declarations), sometimes it does not (basic advice weaving). If the structure of the type is not affected then we don't need to remember it in detail and once it is woven we can forget about it. If we need to recover information about it later we can do so from the classpath or from the VM that loaded it (via reflection). The fact that it was woven will no longer matter because we will only be reliant on the structure, which we know did not change due to weaving.

Secondly, there are links between the two categories. Using either weak or soft references for category (2) entities is great but if there is a hard reference from a category (1) object to a category (2) object then it will never be garbage collected. Being smarter about type references will enable category (1) objects to have a 'softer grip' on the category (2) entities, allowing them to be GC'd.

Some initial weaver options are live today in AspectJ 1.6.5 dev builds. These options enable the weaver to consider types only affected by advice as category (2) - disposable and recoverable.

New weaver options.

The new options are related to what we shall call 'type demotion'. Once enabled, then after the weaver weaves a type it will look at whether it can be demoted from the fixed set of types in the World to the garbage collectable set of types. If it can be demoted then it will be eligible for GC later.

As an example, I used yourkit to attach to a weaver that had just (load time) woven 1400 classes with simple before advice. With type demotion switched off the size of the weaver was 14Meg and fixed, with no amount of forced GC's shrinking it. With type demotion switched ON, the size of the weaver was 2Meg.

The options can be specified in the aop.xml file:

<weaver options="-Xset:typeDemotion=true,typeDemotionDebug=true"/>

The former option 'typeDemotion=true' switches it on. The latter option will produce a few diagnostics about demotion when it occurs. If enough people try it out and it behaves then it could become default in AspectJ1.6.5 (I am expecting some issues that need solving).

I haven't fully thought through how this option will affect source compilation/weaving so use it in that scenario at your own risk (but I'd be interested in feedback!). I suspect:

TypeDemotion + IncrementalCompilation = headache

The work is being done under bug https://bugs.eclipse.org/bugs/show_bug.cgi?id=227484 in Eclipse bugzilla.

AspectJ 1.6.4 released

I am trying to keep on track with a release every 3 months. It worked well throughout 2008 and here is the first proper release of 2009. The download is here:

http://eclipse.org/aspectj/downloads.php

and the README giving a high level overview of the changes is here:

http://www.eclipse.org/aspectj/doc/released/README-164.html

I always try and have themes for the releases, but these things inevitably change during development. Although I didn't originally intend to look at incremental compilation for 1.6.4, the difference between the JDT and AJDT experience is really hurting adoption. And thanks to some great feedback from our users (collecting diagnostics and discussing configuration scenarios) we have made huge improvements in AspectJ 1.6.4 and the associated AJDT 1.6.5 release. I tried to include a graph in the readme that showed the change in build times but the build times were so short with 1.6.4 that they hardly showed up!

Over the next few weeks I'll write some posts describing the new features in more detail.