AspectJ and Eclipse Programming

Groovy-Eclipse and Eclipse Java7 (3.7.1)

2011-09-23T12:55:00.000-07:00

Just a quick post. Today the Eclipse team released Eclipse 3.7.1 - this is the first release to integrate their Java7 compilation support. Groovy-Eclipse integrates groovy into eclipse by patching jdt.core. I was concerned our patch would need a lot of rework given that supporting Java7 is probably the biggest set of changes in the Eclipse Compiler for a while. However, it is all done and todays development builds of Groovy-Eclipse for Eclipse 3.7 will work with 3.7.1 and allow you to mix Java7 source code and Groovy code. Here is the update site from which you can grab a dev build:

http://dist.codehaus.org/groovy/distributions/greclipse/snapshot/e3.7/

If you install that dev build into a regular Eclipse 3.7.0, it will work - it will 'upgrade' you to the Eclipse Java7 compiler in the process but you'll be missing the UI to exploit that without upgrading other parts of eclipse.

STS 2.8.0.M2 is only based on Eclipse 3.7.0 but the final version of 2.8.0 will be based on Eclipse 3.7.1.

Groovy: How fast is your AST transform?

2011-02-22T12:09:00.000-08:00

I spend a lot of time trying to get Ast transforms to behave properly in Groovy-Eclipse. But something that has annoyed me for the longest time is that I couldn't actually tell what transforms were running, or how long they were taking. So, whilst fixing the latest problem with transforms I augmented the Groovy Event Console to show what is going on with respect to transforms.

To see the console just choose 'Groovy Event Console' from the Console view pulldown:

Lots of events are recorded in there (code assist timings, code select timings, ast transform stuff). Currently we don't have a filter to ignore those you aren't interested in, that will be added shortly.

Here are the interesting ast transform events that get recorded:


Ast Transforms : 12:46:18 Classpath for GroovyClassLoader (used to discover transforms): 
 ----8<--- snipped long string which you can check to see if your transform would be found ----8<----

Ast Transforms : 12:45:59 Local transform com.foo.MyTransform on com.example.Code:com.example.Code = 1ms

Ast Transforms : 12:46:21 Global transform org.spockframework.compiler.SpockTransform on /Spocky/src/WorkflowTest.groovy = 48ms

Ast Transforms : 12:46:31 Global transform org.spockframework.compiler.SpockTransform on /Spocky/src/WorkflowTest.groovy = 3ms

Notice the differing timings for the Spock transform, 48ms then 3ms. The former is due to classloading as the transform is loaded up ready for use.

What about global transforms? Well they look at all files, so what I've done for now is not report global transforms unless they take >0ms to run. If they take more than 0ms I'm assuming they are doing something and you might want to know about. I may need to revise that number up from 0.


Ast Transforms : 12:55:04 Global transform 
  org.codehaus.groovy.grails.compiler.injection.GlobalEntityASTTransformation 
  on /XXYXY/grails-app/domain/com/foo/Bar.groovy = 83ms

Ast Transforms : 12:55:04 Global transform 
  org.codehaus.groovy.grails.compiler.injection.GlobalPluginAwareEntityASTTransformation on 
  /XXYXY/grails-app/conf/spring/resources.groovy = 89ms

These timers are in the latest snapshot of groovy eclipse.

AJDT memory usage reduction

2010-07-15T13:19:00.001-07:00

I've had some prototype code ticking along in the background for a while now that should enable AJDT to use less memory. I've just released these changes into AJDT dev builds - a pair of options must be set for your projects to turn them on (as they are experimental). If I get some positive feedback that they behave, I'll switch them to on by default. I thought I'd blog a bit about what I'd done (scroll down if you just want to know the options!)

AJDT maintains per-project AspectJ compiler instances when running in Eclipse. Each compiler instance maintains a lot of state - mostly to enable things like:
- the markers in the UI
- code assist when ITDs are around
- incremental compilation

If you have looked at a heap profile of an AspectJ workspace, you would have seen two large objects for each project. A world, and a model.

The 'world' is the per project type system, it can lookup types by name and stores everything about them (their structure, their code, etc). It is based around the compiled form of the type.

The 'model' is more about storing the source structure of the type. It basically stops at the member level, not going as deep as the code within methods. Alongside this source structure the model stores a relationship map which records two way relationships between nodes in the source structure. For example it records 'SomeAspect.beforeAdvice advises SomeType.method'.

Together these two objects help AspectJ eat all your memory.

There are two new options, one influences how the model behaves, one influences how the world behaves.


-Xset:minimalModel=true

This option changes the model behaviour. With it switched on we discard any pieces of the model that are probably not useful. From my description of the model above, you can imagine that the interesting parts of the model are those pieces that are relationship targets. If a type is compiled and it doesn't define relationships and is not a target for them, we don't need to keep it around as no UI functionality will be needing to use it. Both the markers in the editor and ITD code completion don't care about these types. When you switch on minimalModel, after a type has been through the weaver we will discard the model representation of it if the weaver did nothing to it. We know if that piece of the model will ever be needed that the source file defining it will be recompiled, which will recreate it.


-Xset:typeDemotion=true

This option influences the behaviour of the world. It has existed for a while but didn't really do anything in the IDE, since the initial use for it was in the load time weaving world where memory is even more precious. Now this option will influence how AspectJ behaves under AJDT. Similarly to what the model option does, this option makes the world throw things away. The problem has always been how to be sure we aren't throwing away something we might need again later. What AspectJ now does is keep track of whether a woven type has been written out to the output folder. If it has we know we can discard our representation and retrieve it later from disk. If the type hasn't been written out yet, we hold onto our representation.

The first option recovers a decent chunk of memory, the second option recovers a larger chunk. To see the difference I recommend you activate the eclipse heap usage monitor (Window>Preferences>General 'show heap status') which will then give you the heap usage in the bottom right of your eclipse. Do a few builds, wait for the memory to settle down (or attach jconsole to it and force some GCs). Then activate the two options and do a similar measurement. How much might you save? 100s of megabytes for a large workspace.

To set the options, open the AspectJ Compiler preferences page for the project, then 'Enable project specific settings', open 'Other' and in the 'Non-standard compiler options' section set '-Xset:minimalModel=true,typeDemotion=true'. Or you can set them at the workspace level in the main compiler preferences.

Feedback welcome!

AspectJ 1.6.9 released

2010-07-05T13:18:00.001-07:00

I just released AspectJ 1.6.9. The full readme is accessible here: http://eclipse.org/aspectj/doc/released/README-169.html

Some highlights:
- reduction in class files sizes for compiled code
- overweaving as an alternative to reweaving (as used in the GoogleIO/Roo keynote!)
- aop.xml scoping for controlling set of types affected by aspects
- message inserts for declare warning/error
- type category type patterns
- preserved name/visibility for woven intertype field declarations
- preliminary support for innertype intertypes.

plus a load of bug fixes.

The download is available from http://eclipse.org/aspectj and is already included in the recent AJDT 2.1.0 release for Eclipse 3.5/3.6.

Enjoy!

AspectJ: size is important

2010-05-21T11:12:00.000-07:00

I've been doing some work to reduce the size of the class files produced by AspectJ. Anyone who has javap'd a compiled aspect or woven type will know there is a lot of 'stuff' in there. Most of it is for two reasons:
- to enable binary weaving. We preserve a lot about the original aspect declarations so that we can do the right thing during binary weaving and produce nice weaveinfo/warning/error messages.
- to enable reweaving. The reweaving mechanism was discussed in a previous post I made. The data stored in the classfile basically enables us to recover the original unwoven class file when a secondary weave is required.

In my recent changes I've focussed on the binary weaving case, and reviewed the serialized format we are using for data that we store in the class file attributes. Here is a small scale example:

aspect X {
  public int Foo.getMeAnInt() { return 42; }
  public String Foo.string;
  declare parents: Foo implements java.io.Serializable;
}

public class Foo {
}

Compiled with 1.6.9m2:

X.class 5048bytes
Foo.class 1290bytes

Compiled with 1.6.9dev:

X.class 3615bytes
Foo.class 812bytes

What about the javap output?

For 1.6.9m2 a typical attribute that represents an intertype declaration would look like this:

org.aspectj.weaver.TypeMunger: length = 0x118
 02 01 00 05 4C 46 6F 6F 3B 00 00 00 01 00 0A 67
 65 74 4D 65 41 6E 49 6E 74 00 03 28 29 49 00 00
 00 00 00 1D 00 00 00 28 00 00 00 00 00 01 00 00
 00 00 00 01 49 00 00 00 00 FFFFFFAC FFFFFFED 00 05 73 72 00
 11 6A 61 76 61 2E 6C 61 6E 67 2E 42 6F 6F 6C 65
 61 6E FFFFFFCD 20 72 FFFFFF80 FFFFFFD5 FFFFFF9C FFFFFFFA FFFFFFEE 02 00 01 5A 00 05
 76 61 6C 75 65 78 70 01 73 72 00 0C 6A 61 76 61
 2E 69 6F 2E 46 69 6C 65 04 2D FFFFFFA4 45 0E 0D FFFFFFE4 FFFFFFFF
 03 00 01 4C 00 04 70 61 74 68 74 00 12 4C 6A 61
 76 61 2F 6C 61 6E 67 2F 53 74 72 69 6E 67 3B 78
 70 74 00 14 4E 3A 5C 61 73 70 65 63 74 6A 31 36
 39 6D 32 5C 58 2E 61 6A 77 02 00 5C 78 73 72 00
 11 6A 61 76 61 2E 6C 61 6E 67 2E 49 6E 74 65 67
 65 72 12 FFFFFFE2 FFFFFFA0 FFFFFFA4 FFFFFFF7 FFFFFF81 FFFFFF87 38 02 00 01 49 00 05
 76 61 6C 75 65 78 72 00 10 6A 61 76 61 2E 6C 61
 6E 67 2E 4E 75 6D 62 65 72 FFFFFF86 FFFFFFAC FFFFFF95 1D 0B FFFFFF94 FFFFFFE0
 FFFFFF8B 02 00 00 78 70 00 00 00 02 73 71 00 7E 00 06
 00 00 00 1D 00 00 00 00

Now it looks like this:

org.aspectj.weaver.TypeMunger: length = 0x29
 02 01 01 00 1C 00 00 00 01 00 1D 00 1E 00 00 00
 00 00 1D 00 00 00 28 00 00 00 00 1F 00 02 00 4B
 00 00 00 02 00 00 00 1D 00

The improvements are achieved in a couple of ways:

1) moving string data from inside the attribute into the constant pool. Usually the strings in question are actually already in the constant pool, so the size of the constant pool isn't even increasing when I do this. This means we go from storing some long string (maybe a fully qualified type name) inside an attribute to just storing a short index into the pool. This is something we talked about doing a long time ago but never found the time.

2) better representation of the data. Collapsing multiple booleans into bitflags in a single byte. Changing from ints to shorts/bytes where suitable. Doing some data encoding: one attribute was a sequence of ints (one per line in the file) - this was changed to a encoded sequence where we only stored the change from the previous int value. Encoding in this way enabled me to switch to using shorts instead of ints. Saving 2bytes * <number of lines in the file>.

All these little changes add up.

That little example above was just a small case - imagine a much larger scenario. A good example is the Spring Roo petclinic sample. Just download Spring Roo, start the shell and run the clinic script which will build the petclinic Roo application. This uses many many intertype declarations and declares. What difference do the changes above make to it?

With 1.6.9m2. If I compile petclinic src and tests I get 62 class files. Total space consumed is 1,033,516bytes. Two interesting classes are the Pet class and the Pet Entity aspect. These are 27k and 45k respectively.

With dev builds. Same build, same 62 class files. Total space consumed is 632,026bytes. Pet class is down to 15k, Pet entity aspect down to 26k.

That is about a 40% reduction in space consumed.

So why does it matter? Disk is cheap, zip files will sort out some of that wasted space. But I don't think that gives me an excuse to be lazy - more optimal class files will be loaded just that little bit faster by the weaver, will consume a little bit less memory when load time weaving, and a little less bandwidth went shipped around the network - there are no downsides.

Perhaps the trickiest thing to do was maintain compatibility. 1.6.9 builds will now start producing these smaller class files but I still wanted to allowed the weaver to load older classes (built with 1.6.8 and earlier). As ever, AspectJ is not expecting you to recompile your entire application when we put out a new version.

HOWEVER, although AspectJ will understand code built with 1.6.8 and earlier, IF you have already been using 1.6.9 dev builds (like m1 or m2), you will need to do a recompile of your code.

I've decided to blog about this now because it is going live in AJDT imminently. If there are failures they will manifest as problems processing the attributes. My tests are pretty good here but there may be some gremlines lurking. If you find one, let me know!

SpringSource Tool Suite and Grails 1.3 - getting it working

2010-05-19T14:54:00.000-07:00

UPDATED 20-May: with STS 2.3.2 information, scroll to the end...

We have been aware of an issue with STS and Grails 1.3. Here is how to get things working.

I tried this on the STS 2.3.3M1 released this morning at GoogleIO - which you can get from here: http://www.springsource.com/products/springsource-google-download. It should also work on 2.3.2.

Once you have the latest Groovy and Grails support installed from the extensions page on the dashboard, you need to make a further update to the Grails support. You do this by opening Help > Install New Software... - in the dialog that appears, add a new update site with this URL: http://dist.springsource.com/milestone/TOOLS/update/e3.5/ . Once that refreshes you will see quite a few updates - but you want the grails one. See this image for where to find it, and then select it:

Also deselect 'Contact all update sites' at the bottom (just in case). And click Next. Follow the dialogs to complete installation. After restart you should find your projects working with Grails 1.3.

Let me know if that does/doesnt work for you!

STS 2.3.2 update: 20-May

A few users have told me about issues on 2.3.2 installing that update. Nikolaj Joergensen commented on my blog about how to get it to work. Before using the instructions above, add the update site: http://update.atlassian.com/atlassian-eclipse-plugin/e3.4

Now you have that, follow the instructions above, but ensure the 'Contact all update sites during install to find required software' - that way dependencies are resolved and it should install OK on 2.3.2.

AspectJ 1.6.9.M2 is out (and it is in a maven repo)

2010-05-13T17:04:00.000-07:00

1.6.9.M2 is now out - the README has more information. It is only a month since M1 but there are some important bug fixes that have gone in and a couple of new features that I wanted to start getting feedback on.

Also, following on from a previous post about dev builds of AspectJ being in a maven repo, you can now access milestone builds there too.

The repo is added with this magic:

<repository>
    <id>maven.springframework.org</id>
    <name>SpringSource milestones</name>
    <url>http://maven.springframework.org/milestone</url>
</repository>

and the version to depend upon is: 1.6.9.M2

AspectJ Overweaving

2010-05-13T08:14:00.000-07:00

For many years AspectJ has a mechanism that allows classes to woven repeatedly with different aspects. It is called 'reweaving'.

Reweaving is implemented by storing a diff inside a class file that can be applied to the woven class to recover the original (unwoven) form. Going back to the original and weaving new aspects alongside originally applied aspects 'altogether' enables preservation of semantics - which can be important where, for example, different aspects are interacting at the same join point.

Reweaving works well, but does have one serious limitation. Due to the use of a diff and the need to revert to the original class, it doesn't work if another kind of class transformation is done between the AspectJ weaves.

For example: I build my regular application using a number of aspects. It is built to be reweavable and diffs are included (this is the default behaviour). Now I run it through some other tool that modifies the class file in some way. Finally I deploy my application and there is some load-time weaving at startup to further modify the application.

The modification that occurred between the two weaves will mean the 'diff' is now invalid as it was computed based on the difference between the original unwoven form and the first woven form. So we cannot 'undo' the weaving. We do have the option to switch from diff mode and instead preserve the entire unwoven class file in the woven class file instead - but whatever we do along that route, if we go back to the original we will lose the transformations performed by the other tool.

These days, with the increase in popularity of AspectJ and the rise in use of other tools that modify bytecode, this situation is coming up more and more.

Enter overweaving...

Although the basics of this have been in since 1.6.7 it is getting some real attention now in 1.6.9. Overweaving does not attempt to go back to the original - it simply applies the new aspects over the top of what was there before. It does mean semantics are a little affected - but there are realistic situations where that isn't critical. Suppose the secondary weave at loadtime is applying monitoring aspects - it doesn't matter what aspects have previously done, as long as the monitoring can still be meaningfully applied.

Where overweaving really gets tricky is that the new aspects will see joinpoints that never existed in the original user code, these joinpoints were created by the AspectJ weaving process during the first weave. If the pointcuts for the second weave are vague in what they want to capture, they would be picking out these phantom join points. What do I mean? Here is an example:

X.aj

aspect X {
  before(): cflow(execution(* main(..))) && execution(* *(..)) {
    System.out.println("X:"+thisJoinPointStaticPart);
  }
}

C.java

package com.andy;
public class C {
  public String name = "andy";

  public static void main(String []argv) {
    new C().run();
  }

  public void run() {
    System.out.println("hello "+name);
  }
}

After compiling it, let's take a look inside the C.main method:

public static void main(java.lang.String[]);
  Code:
   Stack=2, Locals=3, Args_size=1
   0: getstatic #75; //Field X.ajc$cflowCounter$0:Lorg/aspectj/runtime/internal/CFlowCounter;
   3: invokevirtual #84; //Method org/aspectj/runtime/internal/CFlowCounter.inc:()V
   6: getstatic #75; //Field X.ajc$cflowCounter$0:Lorg/aspectj/runtime/internal/CFlowCounter;
   9: invokevirtual #81; //Method org/aspectj/runtime/internal/CFlowCounter.isValid:()Z
   12: ifeq 24
   15: invokestatic #65; //Method X.aspectOf:()LX;
   18: getstatic #67; //Field ajc$tjp_0:Lorg/aspectj/lang/JoinPoint$StaticPart;
   21: invokevirtual #71; //Method X.ajc$before$X$1$83d8b2a6:(Lorg/aspectj/lang/JoinPoint$StaticPart;)V
   24: new #1; //class com/andy/C
   27: dup
   28: invokespecial #23; //Method "":()V
   31: invokevirtual #24; //Method run:()V
   34: goto 46
   37: astore_2
   38: getstatic #75; //Field X.ajc$cflowCounter$0:Lorg/aspectj/runtime/internal/CFlowCounter;
   41: invokevirtual #87; //Method org/aspectj/runtime/internal/CFlowCounter.dec:()V
   44: aload_2
   45: athrow
   46: getstatic #75; //Field X.ajc$cflowCounter$0:Lorg/aspectj/runtime/internal/CFlowCounter;
   49: invokevirtual #87; //Method org/aspectj/runtime/internal/CFlowCounter.dec:()V
   52: return

In the middle of it you can see the actual original code:

   24: new #1; //class com/andy/C
   27: dup
   28: invokespecial #23; //Method "":()V
   31: invokevirtual #24; //Method run:()V

Everything else is generated guff. I chose cflow in my example as that does quite a bit of codegen. The cflow pointcut is used to verify whether some join point is in the control flow of another. It is implemented by maintaining a counter that is incremented as the cflow() referenced join point is entered and decremented as it is exited. That is the calls to 'CFlowCounter.inc()' and 'CFlowCounter.dec()' above. To check if a join point is in the control flow a call is inserted to 'CFlowCounter.isValid()'. And finally advice is called, see the call to 'X.ajc$before$X$1$83d8b2a6'.

In a reweaving setup that generated code doesn't matter, as we jump back to the original compiled code prior to weaving and it doesn't contain any of that stuff. In an overweaving setup we have to do a bit of unpicking and avoid these rogue join points. For many aspects they don't matter, because pointcuts will be picking specific methods. But once you get into wildcard territory, for example 'call(* *(..))', we have to be careful that this doesn't match those CFlowCounter and other methods.

So the implementation of overweaving is really an exercise in avoiding rogue join points. AspectJ 1.6.9m2 will include some big steps forward in this area, and more improvements will follow. If it is eventually as robust and reliable as reweaving, it may become the default behaviour. I haven't done any performance comparison yet. Although you might think reweaving sounds very expensive, overweaving won't be 'free' due to need to process more bytecode and avoid rogue join points. When it is more complete, I will measure and publish some numbers.

Oh, one more thing - to try it out yourself use the option -Xset:overWeaving=true. Specify that on the command line or in your weaver options section. It vaguely works in 1.6.9m1 but will be much improved in the imminent 1.6.9m2.

1.6.9M1 released

2010-04-17T21:40:00.000-07:00

Just a quick note to say it is out, and the readme is here:

http://eclipse.org/aspectj/doc/released/README-169.html

Unlike the last few releases that were 'simply' speeding things up whilst reducing memory consumption, this one has new features!

AspectJ 1.6.9M1 - the message 'unable to continue'

2010-04-05T11:52:00.001-07:00

Edit: 1.6.9m1 is now out, readme with new feature info is here

AspectJ 1.6.9 milestone 1 is going to be out shortly. With 1.6.9 I'm trying a longer release cycle that will include a couple of milestones. The release schedule of once-per-quarter was starting to feel a bit too frequent.

This article is about something that changes in 1.6.9 milestone 1 that hasn't changed in a while and I find whenever it changes the same question arises a number of times. For 1.6.9 I am having to change the bytecode version number. In order to explain what is going on and help users solve the problem, I created this blog article that should appear when they google the error :) (the release notes will explain it, but who reads them...)

When an AspectJ program is compiled, the AspectJ constructs are captured as attributes in the resultant class file. The format of these attributes is versioned - this enables a later version of AspectJ to process attributes created by an older version of AspectJ, since by knowing the version it understands the format. As language constructs are enhanced, the version is increased, usually indicating that the attribute contains more information than it used to.

Compatibility is (right now) supported in one direction. Newer AspectJs will *always* understand the output of older AspectJ versions, there is never a need to recompile your old code. However, the inverse is not true. When the version number for an attribute is increased, an old AspectJ will not understand what is created by the newer AspectJ.

What does this mean? It means once you start using a version of AspectJ with the new attribute version format, you must upgrade 'downstream' AspectJs as well. So if you upgrade your eclipse to use an AJDT that includes one of these AspectJ builds, you must upgrade any weaver you might be using later for load-time weaving otherwise it won't be able to understand what AJDT builds.

How does it manifest? It used to manifest as a crash when this happened - the older AspectJ would just be unable to cope with the extra data in the attribute. But for a while now we have been producing a message indicating what has happened. Here it is:


java.lang.RuntimeException: Problem processing attributes in N:\aspectj169-dev\A.class
Caused by: org.aspectj.weaver.BCException: 
Unable to continue, this version of AspectJ supports classes built with weaver version 6.0 but the class A is version 7.0

This means the weaver being used supports version 6 attributes but the class it encountered contains version 7 attributes. The solution to this is simply to upgrade the AspectJ you are using to process the classes. There is no real need to understand the version numbers as the cause and solution are always the same (although it is useful info for me when performing a precise diagnosis) - the versions are mismatched, upgrade to the latest. However, in the interests of sharing the history of this, here are the versions:


WEAVER_VERSION_MAJOR_AJ121 = 1

WEAVER_VERSION_MAJOR_AJ150M1 = 2
WEAVER_VERSION_MAJOR_AJ150M4 = 3

WEAVER_VERSION_MAJOR_AJ160M1 = 4
WEAVER_VERSION_MAJOR_AJ160M2 = 5

// added annotation value binding
WEAVER_VERSION_MAJOR_AJ161 = 6

// new style field ITDs
WEAVER_VERSION_AJ169 = 7

So you will see to support the new style weaving of field ITDs (where names and visibility are preserved rather than always creating public members with mangled names), the version number had to be increased.

Groovy-Eclipse: groovy hotswap support

2010-03-24T12:36:00.000-07:00

I tweeted the other day that this was progressing well. The latest dev builds of Groovy-Eclipse now contain all the pieces. In this article I'll talk about how to try it out. The update site for the latest dev builds is:


http://dist.codehaus.org/groovy/distributions/greclipse/snapshot/e3.5/

Not perfect

The hot code replacement is working, however we need to improve the UI to make the experience much smoother. There are two things that are quite annoying and will be fixed in the future:

1) The inclusion of the agent on the debug launch configuration is a manual process, so you'll have to find it and add the line to the launch config.

2) The stack unwinding can be a little annoying. It may not take you to the start of the method you just changed, it may unwind a bit further into the groovy infrastructure and generated code - so you will need to take care to step back to where you were (use a breakpoint and just run to it).

Given that those are the current pitfalls, you can't say I didn't warn you and you can decide if you want to proceed and try it out :)

Setting up the agent

Debugging groovy is run in just the same way as debugging java, except right now there is one tweak to make to your debug launch configuration.

The reset agent is required to clear the call site caches that groovy code creates when it runs. They exist to make latter calls faster but when you change the destination of the call and hot-code-replace, you need the cache to be cleared otherwise it will continue to call the original destination. The agent is in two places right now, find it at whichever is easier for you:

1) With the hot-code-replace enabled recent groovy-eclipse dev builds, the agent is inside the lib folder of the org.codehaus.groovy plugin. After I just installed, that is here for me:


N:\sts231\plugins\org.codehaus.groovy_1.7.0.20100324-1200-e35\lib\groovyReset.jar

So you will need to know where the plugins folder for your installation of eclipse resides and find out the timestamp for the most recent version of that plugin that you have installed.

2) The agent is also here: groovyReset.jar if you want to just grab it and save it to your hard drive in an easier to remember location.

As the agent clears these caches, you really only want to use it when debugging (not when running the app properly) - so might want to create a separate 'debug' launch configuration that has it specified. Either create a new one or duplicate the 'Run' one you use and tag it for debugging. How do you specify the agent setting? In the launch config window, go to the properties page, arguments tab, then under VM arguments, add:


-javaagent:<path_to_the_agent>/groovyReset.jar

Stepping

As I mentioned in the drawbacks, you may find after you hot code replace something that the stack for the thread you were debugging has unwound further than you'd like. Here is an example. My debugger was stopped at in the Enuming.run() method. Here is the stack at the breakpoint:


Enuming.run() line: 8 
Enuming$run.call(Object, Object[]) line: not available 
CallSiteArray.defaultCall(CallSite, Object, Object[]) line: 40 
AbstractCallSite.call(Object, Object[]) line: 117 
AbstractCallSite.call(Object) line: 121 
Enuming.main(String...) line: 14

I then make a change to run() and save it, the debugger rewinds back to:


AbstractCallSite.call(Object) line: 121 
Enuming.main(String...) line: 14

And I am some way away from the run() method. This will be fixed, but is an annoyance right now. To get back into the method in question (run() in my case), I recommend putting a breakpoint at the start of run() and clicking Resume to continue and stop back in that method.

So that's it - this is very preliminary support, I'd appreciate any feedback on it as I haven't really tested it in anger on anything other than little toy projects. (I did try a few mixed java/groovy projects).

Groovy-Eclipse: won an Eclipse Community Award

2010-03-23T08:31:00.000-07:00

I had a look back through my blog and it was 1st May last year when I wrote about tinkering with JDT to see if I could get it to integrate with the groovy compiler ( Modifying eclipse JDT to joint compile groovy).

This week it was EclipseCon 2010 and the new version of Groovy-Eclipse based on that original architecture won an Eclipse Community Award ("Best Open Source Developer Tool"). Of course it wasn't all about that compiler integration - it is a vital underpinning but a lot of work had to be done on top of that to add the customary UI features and behaviours that users expect (code assist, refactoring, search, etc) - and Andrew Eisenberg has done a great job in that area - making my clunky low level compiler consumable!

We are a small team but by sticking to our original well scoped design vision of just enhancing the common every day tasks of developers (edit/save/compile/test) - we have achieved something special. But there is always more to do and we are about to embark on Eclipse 3.6 support - whilst at the same time continuing to develop the STS Grails support that builds upon Groovy-Eclipse.

Thanks to everyone for the support and feedback since our first alpha release.

AspectJ dev builds in a maven repo

2010-03-22T12:44:00.000-07:00

I wanted to make it easier for users to pick up dev builds of AspectJ. These development builds aren't suitable candidates for putting into maven central but with so many users consuming AspectJ from maven I wanted to make it as painless as possible for them to be picked up. So, we now have a new repo where I will publish dev builds periodically.

The maven compatible repo is maven.springframework.org/snapshot/org/aspectj - and if you browse to it you will see it currently contains 1.6.9 dev builds under the name 1.6.9.BUILD-SNAPSHOT.

The repo is added with this magic:


<repository>
    <id>maven.springframework.org</id>
    <name>SpringSource snapshots</name>
    <url>http://maven.springframework.org/snapshot</url>
</repository>

and then the version to depend upon is:

1.6.9.BUILD-SNAPSHOT

Let me know if you try it out !

Running the Groovy-Eclipse joint compiler on the command line or from Ant

2010-02-18T11:11:00.000-08:00

Groovy-Eclipse v2.0.0 closely integrates the groovy compiler and eclipse compiler in order to compile mixed language projects. Unlike the traditional approach used by groovyc, groovy-eclipse does this in a stub-less way.

Now, the Eclipse compiler has always been usable outside of Eclipse either directly on the command line or via Ant. Some developers choose to use it in this mode because then their back end build systems are using exactly the same compiler as their IDE (since there are situations where different java compilers produce different bytecode for the same input - meaning your build system could produce different code to that produced/tested in the IDE!).

In this article I'm going to describe how to run the groovy-eclipse compiler in batch mode. It is a little messy right now, hopefully some of the fiddling about with classpaths can be reduced over time.

The entry point to the compiler is exactly the same for groovy-eclipse as it is for normal eclipse compiler batch usage (namely org.eclipse.jdt.internal.compiler.batch.Main), all the complexity is around pulling together the right set of jars to satisfy all dependencies.

All the code/jars I'm going to discuss here are available either as part of the Eclipse 3.5.1 distribution or as part of the Groovy Eclipse v2.0.0 download.

First things we need are from the Groovy Eclipse distribution:

- the patched eclipse compiler. A patched form of the standard JDT compiler that allows other compilers to fully participate in compilation. Nothing groovy specific in this patched compiler
- the groovy compiler. A standard Groovy 1.7.0 distribution.
- the groovy patches. A small set of patches on the groovy compiler that fix a few issues and allow it to be plugged into the patched eclipse compiler.
- the integration logic to plug the modified groovy compiler into the modified JDT compiler.

Going forward we'd like to reduce the patches by contributing the groovy changes back to groovy and eclipse changes back to eclipse, but these things take time.

The patched eclipse compiler is this plugin jar:

org.eclipse.jdt.core_3.5.1.xx-20100115-0900-e35-RELEASE.jar

The patches for the groovy compiler (and a copy of groovy 1.7.0) are contained in this plugin which is not currently packaged as a self contained jar:

org.codehaus.groovy_1.7.0.xx-20100115-0900-e35-RELEASE

within that plugin folder you will find:
groovy-eclipse.jar (the patches for the standard groovy distribution)
lib/* - the groovy 1.7.0 distribution and its dependencies (asm/antlr/etc)

There is then a plugin that connects the patched (now extensible) jdt compiler to the groovy compiler:

org.eclipse.jdt.groovy.core_1.0.0.xx-20100115-0900-e35-RELEASE.jar

Now we need to satisfy the dependencies that the eclipse java compiler has, all these can be found in a standard Eclipse 3.5.1 distribution:


org.eclipse.core.runtime_3.5.0.v20090525.jar
org.eclipse.osgi_3.5.1.R35x_v20090827.jar
org.eclipse.core.resources_3.5.1.R35x_v20090826-0451.jar
org.eclipse.core.filesystem_1.2.0.v20090507.jar
org.eclipse.equinox.common_3.5.1.R35x_v20090807-1100.jar
org.eclipse.core.jobs_3.4.100.v20090429-1800.jar
org.eclipse.text_3.5.0.v20090513-2000.jar

To satisfy groovyc, in addition to:

org.codehaus.groovy_1.7.0.xx-20100115-0900-e35-RELEASE/lib/groovy-1.7.0.jar

we also pull in

org.codehaus.groovy_1.7.0.xx-20100115-0900-e35-RELEASE/lib/asm-3.2.jar
org.codehaus.groovy_1.7.0.xx-20100115-0900-e35-RELEASE/lib/antlr-2.7.7.jar

Phew. Creating a CLASSPATH with all that stuff on it is a bit hellish:

org.eclipse.jdt.core_3.5.1.xx-20100115-0900-e35-RELEASE.jar;org.codehaus.groovy_1.7.0.xx-20100115-0900-e35-RELEASE/groovy-eclipse.jar;org.codehaus.groovy_1.7.0.xx-20100115-0900-e35-RELEASE/lib/groovy-1.7.0.jar;org.codehaus.groovy_1.7.0.xx-20100115-0900-e35-RELEASE/lib/asm-3.2.jar;org.codehaus.groovy_1.7.0.xx-20100115-0900-e35-RELEASE/lib/antlr-2.7.7.jar;org.eclipse.jdt.groovy.core_1.0.0.xx-20100115-0900-e35-RELEASE.jar;org.eclipse.core.runtime_3.5.0.v20090525.jar;org.eclipse.osgi_3.5.1.R35x_v20090827.jar;org.eclipse.core.resources_3.5.1.R35x_v20090826-0451.jar;org.eclipse.core.filesystem_1.2.0.v20090507.jar;org.eclipse.equinox.common_3.5.1.R35x_v20090807-1100.jar;org.eclipse.core.jobs_3.4.100.v20090429-1800.jar;org.eclipse.text_3.5.0.v20090513-2000.jar

But means we can do this:


====B.groovy===
print 'hello world'

====A.java===
public class A {
  public static void main(String[]argv) {
    new B().run();
  }
}

java org.eclipse.jdt.internal.compiler.batch.Main A.groovy B.java

java B
'hello world'

Yey!

For usage from Ant we need two more little things:
- a compiler adapter that will tell javac to use the JDT compiler
- a variant of the javac task that will pick up .groovy files in addition to .java files

The compiler adapter is readily available inside the eclipse jdt compiler plugin (in the same way as it is available from the unpatched jdt compiler plugin):

jar -xvf org.eclipse.jdt.core_3.5.1.xx-20100115-0900-e35-RELEASE.jar jdtCompilerAdapter.jar

The minimal task is available inside another part of groovy eclipse:
org.codehaus.groovy.eclipse.ant_2.0.0.xx-20100115-0900-e35-RELEASE\groovyAntEclipse.jar

Here is the Ant snippet that pulls it altogether. It defines a variant of the javac task called 'gjavac' that will build mixed codebases.


<?xml version="1.0" encoding="UTF-8"?>
<project name="My Project" default="build" basedir=".">

<target name="init">
  <property name="base" value="n:/somewhere"/>
  <path id="gjavac.classpath">
     <pathelement path="${base}/groovyAntEclipse.jar"/>
     <pathelement path="${base}/jdtCompilerAdapter.jar"/>
     <pathelement path="${base}/org.eclipse.jdt.core_3.5.1.xx-20100115-0900-e35-RELEASE.jar"/>
     <pathelement path="${base}/org.codehaus.groovy_1.7.0.xx-20100115-0900-e35-RELEASE/groovy-eclipse.jar"/>
          <pathelement path="${base}/org.codehaus.groovy_1.7.0.xx-20100115-0900-e35-RELEASE/lib/groovy-1.7.0.jar"/>
     <pathelement path="${base}/org.codehaus.groovy_1.7.0.xx-20100115-0900-e35-RELEASE/lib/asm-3.2.jar"/>
     <pathelement path="${base}/org.codehaus.groovy_1.7.0.xx-20100115-0900-e35-RELEASE/lib/antlr-2.7.7.jar"/>
     <pathelement path="${base}/org.eclipse.jdt.groovy.core_1.0.0.xx-20100115-0900-e35-RELEASE.jar"/>
     <pathelement path="${base}/org.eclipse.core.runtime_3.5.0.v20090525.jar"/>
     <pathelement path="${base}/org.eclipse.osgi_3.5.1.R35x_v20090827.jar"/>
     <pathelement path="${base}/org.eclipse.core.resources_3.5.1.R35x_v20090826-0451.jar"/>
     <pathelement path="${base}/org.eclipse.core.filesystem_1.2.0.v20090507.jar"/>
     <pathelement path="${base}/org.eclipse.equinox.common_3.5.1.R35x_v20090807-1100.jar"/>
     <pathelement path="${base}/org.eclipse.core.jobs_3.4.100.v20090429-1800.jar"/>
     <pathelement path="${base}/org.eclipse.text_3.5.0.v20090513-2000.jar"/>
  </path>
  <property name="build.compiler" value="org.eclipse.jdt.core.JDTCompilerAdapter"/>
  <taskdef name="gjavac" classname="org.codehaus.groovy.eclipse.ant.GroovyJDTCompileTask">
     <classpath refid="gjavac.classpath" />
  </taskdef>
</target>

<target name="build" depends="init" >
  <gjavac destDir="bin">
    <src path="src/"/>
  </gjavac>
</target>

</project>

As a special reward for anyone who read this far, let me say that I've packaged up all the jars from above in one super-jar that you can grab:
http://dist.springsource.org/milestone/GRECLIPSE/e3.5/GroovyEclipseBatch.zip

Armed with that download and the Ant snippet above, you should be good to go. Feedback welcome!

Groovy and AspectJ: load-time weaving groovy scripts

2010-01-26T14:18:00.000-08:00

Norman Elton raised an interesting question on the AspectJ mailing list:

"The AspectJ documentation seems to indicate that you can restrict weaving to a particular ClassLoader, instead of specifying a javaagent at runtime. I'm using LTW to weave into Groovy scripts, which are loaded from a GroovyClassLoader."

I thought, whoa - that'd be cool. Norman tried what the documentation possibly suggested would work:


WeavingURLClassLoader wucl = 
  new WeavingURLClassLoader(this.getClass().getClassLoader());
GroovyClassLoader loader = new GroovyClassLoader(wucl);
Class groovy_class = loader.parseClass(
  new File("bundle/GroovyPoint.groovy"));

but that seemed to do nothing. Basically that is because the classloaders are entirely separate, the WeavingURLClassLoader is intended to weave code it loads, and it doesn't load the script, the GroovyClassLoader loads the script. So it started me thinking, how can I achieve what Norman wanted to. There now follows some sample code. It may not be the most elegant way to get Groovy and AspectJ interoperating but it does work!

First, we need a groovy script 'Foo.script':


void printMessage(String message) {
  print message
}

printMessage('hello world')

And let's also create a Tracer aspect 'Tracer.aj':


public aspect Tracer {
  before(): execution(* *(..)) && !within(Tracer) {
    System.out.println("Entering "+thisJoinPoint);
  }
}

The plan is to loadtime weaving the tracer aspect into the groovy script when it is dynamically loaded through a GroovyClassLoader. First we compile the aspect into a jar:


ajc Tracer.aj -outxml -outjar aspects.jar

This builds the .class files for the aspect and also the META-INF/aop.xml file which will configure loadtime weaving later on.

Now the code that joins the two things together. Here is the driver program and the script loader routine:


public class Driver {
  public static void main(String[] args) throws Exception { 
    Script script = new Driver().loadScript("Foo");
    script.run();
  }

  public Script loadScript(String scriptname) throws Exception {
    WeavingGroovyClassLoader magicLoader = 
      new WeavingGroovyClassLoader(this.getClass().getClassLoader());  
    Class scriptClass = magicLoader.parseClass(
      new File("code/"+scriptname+".groovy"));
    return scriptClass.newInstance();
  }
}

Obviously all the magic happens in the WeavingGroovyClassLoader. The idea here is that WeavingGroovyClassLoader is an extension of the GroovyClassLoader that will call out to the AspectJ weaver just before a class is defined.

Here is the WeavingGroovyClassLoader:


class WeavingGroovyClassLoader extends GroovyClassLoader {

  ClassLoaderWeavingAdaptor adaptor = null;
  InnerLoader loader;
  
  public WeavingGroovyClassLoader(ClassLoader classLoader) {
    super(classLoader);
    // Copy of the code that is normally in createCollector - so that we
    // have the inner loader for AspectJ to register classes against
    loader = (InnerLoader) AccessController.doPrivileged(new PrivilegedAction() {
      public Object run() {
        return new InnerLoader(WeavingGroovyClassLoader.this);
      }
    });
    adaptor = new ClassLoaderWeavingAdaptor();
    adaptor.initialize(loader, null); // pass the InnerLoader
  }
 
  // mostly a copy of the variant in GroovyClassLoader 
  protected ClassCollector createCollector(CompilationUnit unit, SourceUnit su) {
    return new WeavingClassCollector(loader, unit, su);
  }
  
  class WeavingClassCollector extends ClassCollector {
    protected WeavingClassCollector(InnerLoader arg0, CompilationUnit arg1,
      SourceUnit arg2) {
      super(arg0, arg1, arg2);
    }

    protected Class createClass(byte[] code, ClassNode classNode) {
      try {
        code = adaptor.weaveClass(classNode.getName(), code);
      } catch (IOException e) {
        System.err.println("Weaving failed");
        e.printStackTrace();
      }
      return super.createClass(code,classNode);
    }
  }
}

On creation it builds an AspectJ ClassLoaderWeavingAdaptor instance. This instance will configure itself based on the classpath of the loader - this means it will be looking for META-INF/aop.xml files on the classpath.

Normally the createCollector() in the GroovyClassLoader is left to create an InnerLoader and that is used to define the built scripts. However (as discovered by Norman in some testing of the above), the InnerLoader must also be used by AspectJ so that it can record any generated closures (for example to support around advice). So that is why the InnerLoader creation is now in the ctor - so that it can be passed to the adaptor. This works on simple cases but there may be some situations it doesn't handle (I haven't tried script reloading, for example).

The class collection mechanism is intercepted GroovyClassLoader so that we can invoke the AspectJ weaver to see if the class needs a further change, before finally defining it to the VM.

To run all of this we have to set it up as follows:
- the script (in our case Foo.script) should be placed in the code subdirectory.
- the aspects.jar we built earlier should be on the classpath
- that's it!


java Driver

Entering execution(CallSite[] Foo.$getCallSiteArray())
Entering execution(CallSiteArray Foo.$createCallSiteArray())
Entering execution(Class Foo.$get$$class$Foo())
Entering execution(Class Foo.class$(String))
Entering execution(Object Foo.run())
Entering execution(CallSite[] Foo.$getCallSiteArray())
Entering execution(void Foo.printMessage(String))
Entering execution(CallSite[] Foo.$getCallSiteArray())
hello world

The 'Entering' lines coming from the aspect whilst the 'hello world' comes from the script. Obviously infrastructure is also being advised in the groovy case but the tracer aspect is very basic. If I want to see a little debug I can create another META-INF/aop.xml and put it onto the classpath:


<aspectj>
  <weaver options="-verbose"/>
</aspectj>

This will merge with the one in aspects.jar when the adaptor is configured and turn on debug:


java Driver

[AppClassLoader@a39137] info AspectJ Weaver Version 1.6.8 built on Friday Jan 8, 2010 at 21:53:37 GMT
[AppClassLoader@a39137] info register classloader sun.misc.Launcher$AppClassLoader@a39137
[AppClassLoader@a39137] info using configuration /N:/workspaces/groovy35/BleedingEdge/bin/META-INF/aop.xml
[AppClassLoader@a39137] info using configuration file:/N:/workspaces/groovy35/BleedingEdge/aspects.jar!/META-INF/aop-ajc.xml
[AppClassLoader@a39137] info register aspect Tracer
Entering execution(CallSite[] Foo.$getCallSiteArray())
Entering execution(CallSiteArray Foo.$createCallSiteArray())
Entering execution(Class Foo.$get$$class$Foo())
Entering execution(Class Foo.class$(String))
Entering execution(Object Foo.run())
Entering execution(CallSite[] Foo.$getCallSiteArray())
Entering execution(void Foo.printMessage(String))
Entering execution(CallSite[] Foo.$getCallSiteArray())
hello world

I could even turn on weaving information messages in aop.xml (with -showWeaveInfo) if I wished. There are no doubt some nice use cases around exploiting AspectJ declare warning and error where the script is checked after it is loaded and compiled.

Groovy eclipse version 2 is out

2010-01-15T16:42:00.001-08:00

Version 2.0.0 is now available for download !

The New and Noteworthy for 2.0.0 contains more information, including the update site location.

It has been a long road from the initial hacking I did on an incremental joint compiler last May and I'll write up more of a retrospective shortly. But for now, enjoy the release!

Groovy eclipse and javadoc hovers

2009-12-16T13:49:00.001-08:00

In order to support task tags in eclipse (see a previous article), I had to modify the groovy parser to preserve comments. This enabled me to look through them post-parse and recognize tags. The natural next step was then to look at attaching the comments correctly to the eclipse data structures in order to get javadoc hovers to work. By javadoc hovers, I mean the ability to hover your mouse over a type or other reference and have any javadoc related to it be shown in a small popup box that remains in place whilst you hover.

Here is an example of it absolutely *not* working, this is groovy eclipse M2:

Notice hovering over printSomething() shows just the method signature, with no documentation. Now, *here* is an example of the latest dev build:

Notice all the text is there and it is formatted according to any markup included in the comment.

It is not flawless by any means, but it is basically hanging together. Please feedback through the GRECLIPSE Jira or the mailing list if you encounter any issues or missing functionality. Currently the approach taken to determine which comment to attach to which element is a little 'crude' because the parser is not yet recording comments against specific elements during a parse, it is just collecting together all comments in a big list. Once the parser is doing the right thing, the mechanism for determining which comment to attach can be improved.

Here's another shot of it working whilst in the GPars code (SingleRunActor is a groovy type):

AspectJ 1.6.7 and faster load time weaving

2009-12-09T10:07:00.000-08:00

AspectJ 1.6.7 has been all about performance. The release is due at the end of the month (December 09) and I've already written a few articles about how much it has improved. One part I've recently been looking at is loadtime weaving and what can be done to speed that up too.

It turns out quite a lot can be done... Recent dev builds include all the latest greatest changes and I really hope everyone will notice it running faster and in less memory. What has changed under the covers?

Primarily the processing of the aop.xml files has been improved. Previously we were only optimizing for one kind of include/exclude pattern. If you used 'com.foo..*' (ie. the ..* notation) then we would be able to do a simple 'classname.startsWith("com.foo.")' when looking at whether to weave classes being passed to the weaver. Any other kind of pattern would in fact use AspectJ pattern matching. This seems the logical thing to do, but it can be an expensive waste of processing power, particularly if you eventually end up excluding the type. Suppose your exclude was:

<exclude within="*Bar"/>

you wanted to exclude all types ending with 'Bar'. We didn't recognize that pattern and resorted to AspectJ pattern matching - this means all types ending in Bar would be loaded into the weaver and then rejected. They *would* eventually be removed from the weaver but for a while they'd sit there eating your memory and loading them was not free - and all this when the pattern is clearly matchable purely by processing the name of the type. You may think that is an unlikely kind of pattern but I have seen the use of "*Exception" to say no exception types should be woven...

I put out a call on the aspectj-users mailing list for common patterns but didn't get much of a response - so I looked to other sources to determine patterns that should be optimized for. By optimized I mean what can we do to include or exclude a type without having to ask the weaver to process the bytes for the class. AspectJ 1.6.7 will now optimize for these patterns:

- an exact name 'com.foo.Bar' - yes, previously it didn't optimize for exact names!
- a trailing suffix '*Bar'
- types not in the default package containing a string. This is a big one and optimizes for patterns like '*..*CGLIB*' which is the way we recommend exclusion of CGLIB generated types
- any type '*'

It turns out just these few, plus some minor enhancements to how patterns are processed, can make a big difference. Here is a comparison showing 10 iterations of starting tomcat loadtime weaving a small web app with spring-aspects and another few monitoring aspects too. In the top picture is startup time in milliseconds. In the bottom picture is heap usage in bytes.

The memory usage graph looks insane - what on earth were we doing before?

Some of that is the reduction in types loaded by the weaver because more pattern matching to exclude types is done up front, but it is *mainly* showing the impact of another change I've made, switching from softrefs to weakrefs. I find VMs treat softrefs in a rather unpredictable/unreliable way and users tend to think if they force one GC and take a memory snapshot, they will see true memory usage. Then I get heapdumps emailed to me asking why I'm using all the memory. Softrefs are rather more stubborn than going after a single GC but unfortunately the rules can vary across VMs concerning when they are collected (at least this is my experience). Rather than spend my time constantly defending the use of softrefs, I've switched to weakrefs which are GC'd far sooner. It should mean less heap dumps in my inbox :) Of course, it does mean processing power may be wasted later repopulating the caches if we need something later that was actually weakly referenced, but that seems a price worth paying to reduce concern about memory consumption.

Thanks to Tim Schraepen and Ramnivas Laddad for working with me to determine common include/exclude patterns that we should optimize for.

Using eclipse to edit your groovy scripts

2009-12-07T12:45:00.000-08:00

I've seen this question come up a couple of times lately and thought I'd write something up about how to do it in the least invasive way.

This article is aimed at users who have a few groovy scripts in some directory and want to use the sophisticated eclipse editor for them. Although eclipse does like to have a 'workspace' and a 'project' containing anything being worked on, those don't have to cause havoc, and you aren't forced to import/move your scripts around to keep eclipse happy.

I have a folder n:\groovyscripts, inside is one script:

PrintHelloWorld.groovy

There is nothing else in the folder. I want to use eclipse to edit that script, and I do *not* want to move it.

After downloading either Eclipse or the SpringSource Tool Suite - and then installing the groovy eclipse plugin, we are ready to get going.

The Workspace

Eclipse needs a workspace. Just run eclipse and let it default if you don't want to think too hard - it will likely want to create a folder in your user account area on your machine. The scripts are not going to be moved here, so it doesn't really matter where it is.

The Project

The key step is when defining the project. During creation the project wizard must be told to use the existing directory. To create a new project, click the 'File' option on the toolbar, 'New' then 'Groovy Project'. Give it a name (doesn't *really* matter what it is), but most importantly in the 'Contents' section of the 'New Project Wizard', select 'Create project from existing source', choose your script folder. Here is mine filled in:

Now click Finish to create the project.

Edit!

That's it, we are ready to edit. In my case my package explorer now looks like this:

(In the filters setting for package explorer, I have it set to show '.' prefixed files, so that we can see everything in this screenshot). The scripts haven't been moved, all that has happened is that my scripts folder has been promoted to being a fully fledged eclipse project. What does that mean? It means if I look in my scripts folder I now see two new '.' prefixed files:

The .project file defines our project and how it should be compiled. Take a look inside, you'll see it has the name we specified 'MyScripts' (in my case) and the groovynature and javanature - 'natures' indicate to eclipse how to treat particular projects. Alongside the .project file is a .classpath file, this does exactly what you think it does :) - defining all the classpath entries for our project.

We can now just work with the scripts in eclipse and get all those nice editor features. Here I've typed in 'part' and pressed Ctrl+Space to activate code assist:

Groovy version and libraries.

Groovy eclipse currently adds Groovy 1.7rc1 to the classpath for any new projects you create - and that includes our new scripts project. If you open the 'Groovy Libraries' entry in the new project we have created, you will see the reference to Groovy 1.7. This means all code assist and references are going to be resolved against that version of Groovy. Manipulating the classpath to change this, or even to add new library dependencies, should all be done through the eclipse menus. To open the edit page for the classpath, right click the project, select 'Properties' then in the window that appears, select 'Java Build Path' and you will get a window like this:

You can then add new jars, that either already exist in the project ('Add JARs') or exist externally to the project somewhere else on your disk ('Add External JARs').

Script compilation.

Something you may have observed in the screenshot above, although I didn't call it out, is that the .class file for our script was in the scripts folder. This wasn't the case before I imported it, but at the moment the project was defined, eclipse started compiling the scripts. The compiled form is there ready for you to execute immediately if you wish to run them as java applications rather than through the 'groovy' script launcher.

So there you have it. You can commit the .project/.classpath into your repo alongside your scripts if you want to preserve the classpath settings you are using or allow others to easily pick up the fact that your scripts folder is now an eclipse project. I hope I've shown that using eclipse for something like this isn't like the bad old days where an IDE (mentioning no names here...) forced you to submit your code to it and then it would hide it from you in some grand repository on your disk - eclipse is making it easy to work with the setup you already have.

Groovy Eclipse: task tags

2009-12-01T08:36:00.000-08:00

Not exactly talking about changing the world in this post, but something recently added to groovy eclipse for both Groovy 1.6 and 1.7 is task tag support. What is a task tag? Just those markers you leave in the code (then fail to address by the time you ship the product):

// TODO optimize this
// MUSTFIX cannot ship with this problem

I find these incredibly useful for Java and now groovy eclipse supports them. All the combinations too (some of which I didn't know about until implementing it):

What works?
- the same tags as used in your Java code are respected. If you add new ones or define them to be case insensitive, that is all respected by groovy
- they appear the same in the Tasks view and in the editor with appropriate gutter markers
- navigation from Tasks view to editor works fine
- importance defined for the task (high/normal/low) is respected
- adjacent tags separated by a single space get the same text

What else can you see there?
- the underlining of somethin is indicating it cannot be resolved statically (which may or may not be a problem in your code, but is just a hint in case you expected it to be...)

Supporting task tags wasn't trivial because the standard groovy parser discards comments during the parse - I've modified it to preserve them for later use. Not only does this enable things like task tag support, it will also allow javadocs to appear when hovering over constructs in the editor - but that isn't quite done yet...

AspectJ: How much faster is AspectJ 1.6.7?

2009-11-10T14:35:00.001-08:00

In the previous post I talked about the new timing infrastructure that is now in place for determining which pointcuts are proving expensive to match. Since getting that going I've been actively working on some systems that are load-time weaving aspects and attempting to optimize the matching process. The system I was working on used many of what I could call 'typical' pointcuts and so as well as performing some general optimizations that will affect any kind of pointcut, I've put in some specific optimizations for these typical/common pointcuts.

What is a common pointcut?

I see this all the time: 'execution(* Foo.*(..))'. An execution pointcut with an exact typename used for the declaring type in execution (Foo in this example). By exact, I mean no wildcards are used. AspectJ 1.6.7 supports fast match now if the type name is an exact name ( fast match is discussed in the previous article). The exact name may or may not be fully qualified, but the key thing is if there are no wildcards. What difference does that make? Here is a simple aspect:

aspect SimpleAspect {
before(): execution(* CharSequence+.*e*(..)) {}
}

Let's weave that into rt.jar (which contains 16,000 or so classes). The aspect above will actually hit 380 join points in that jar. The command I'm running:

ajc -timers -inpath rt.jar SimpleAspect.aj -outjar wovenrt.jar

AspectJ1.6.6: 21728ms
AspectJ1.6.7: 11954ms

Almost 10seconds faster due to optimized matching!

Related to that, I see a lot of users exploiting the ability to recognize join points by their annotations and in execution I see this all the time: 'execution(* (@Bar *).*(..))' - execution of any method in any type annotated by @Bar. For the case of an exact annotation and wildcarded type name (just '*'), fast match will now kick in. What difference does it make? Similar kind of improvement:

AspectJ1.6.6: 21944ms
AspectJ1.6.7: 11417ms

Of course a pattern like that doesn't actually match anything in rt.jar but that means the numbers here are showing purely the improvement in match speed, because no weaving occurs.

WATCH OUT: This close variant does not get fast matched: 'execution(* (@Bar *..*).*(..))' - this variation is choosing types not in the default package that happen to be annotated. It could be optimized for, but so far I've found all users are happy to switch to '(@Bar *)' so have not invested time in optimizing this pattern.

Optimizations have also been made for all pointcuts containing signatures with these characteristics, because I often see them occur in the wild:
- using an ellipsis for method arguments - '* *(..)'
- using a single star for method arguments - '* *(*)'
- specifying no arguments - '* *()'
- specifying void or !void as the return type - 'void *(..)' '!void *(..)'
- a declaring type of simply one wildcard - '* *(..)'

How much faster? For these improvements I created a micro benchmark that attempted to match these patterns a million times against a method.

AspectJ1.6.6:
Signature pattern [* *(..))] took 1375ms for 1,000,000
Signature pattern [* *(*))] took 949ms for 1,000,000
Signature pattern [* *())] took 908ms for 1,000,000
Signature pattern [!void *(..))] took 1279ms for 1,000,000
Signature pattern [void *(..))] took 1120ms for 1,000,000
Signature pattern [* *a*b*()] took 570ms for 1,000,000

AspectJ1.6.7:
Signature pattern [* *(..))] took 357ms for 1,000,000
Signature pattern [* *(*))] took 293ms for 1,000,000
Signature pattern [* *())] took 285ms for 1,000,000
Signature pattern [!void *(..))] took 364ms for 1,000,000
Signature pattern [void *(..))] took 362ms for 1,000,000
Signature pattern [* *a*b*()] took 309ms for 1,000,000

Fast match is also now supported for the initialization pointcut too - so that will see similar benefits to execution.

Beyond these improvements, other general optimizations include:

1) avoiding type resolution unless absolutely necessary. This involves doing more examination of patterns up front to ensure the types really need to be loaded.

2) avoiding repeated type resolution. Rather than resolving a type, then passing the signature to a downstream method where it needs to be re-resolved, AspectJ will now pass the resolved entity to save that extra resolve call.

3) faster binary weaving for jars. Changed the algorithm for processing jar entries, it now does far less work with collections.

4) Altered type walking mechanism to be iterator based. Previously it calculated the entire hierarchy before looking through it and often found the type it was looking for early. Switching to iterators AspectJ now only explores as far as necessary and if the required type is found early, it walks (and resolves!) no further.

5) Reduced collection usage in the join point creation code. Join point creation code runs *a lot* so even tiny improvements here manifest as big savings.

6) Moved to an iterator approach when searching for methods - rather than determining them all then searching through. AspectJ now only searches as far as necessary.

How to show these improvements? I've collected full tracing yourkit profiles and compared them. This is for a load-time weaving system involving tomcat plus some aspects. Startup time (when not profiled) of the server with these improvements has gone from 9seconds with AspectJ 1.6.6 to 6 seconds with AspectJ 1.6.7.

Demonstrating improvements: type resolution

Type resolution calls all funnel through World.resolve(). Here is the comparison run for 1.6.6 (old) versus 1.6.7 (new):

441,922 less calls to resolve. Due to a combination of avoiding repeated resolution and just more pointcut analysis so types are not chased down unnecessarily. To reinforce that AspectJ is really chasing down fewer types, here is a look at the calls to lookupClass() which is how .class files are loaded by the weaver:

3,068 less calls to locate and load class files.

Demonstrating improvements: fast match

Obviously because fast match is now implemented for more pointcuts, the time spent in fast match has increased:

Up from 2seconds to 12seconds. But that 10seconds is more than offset by the reduction in time spent in the second stage of matching:

That shows a reduction of almost 60seconds in second stage matching due to the improvements in fast matching. (Remember all these numbers are particularly large because full profiling was done).

Conclusion

So there you have it, what has changed in AspectJ 1.6.7 and why it is faster. I will practically guarantee that any user of the weaver will see an improvement - but how much improvement will depend upon whether your pointcuts fit the common patterns outlined above. With the new timer infrastructure I'm hoping it will be much easier to diagnose slow weaving in future and over the next few releases I'm sure more enhancements will be put in place for further common pointcut patterns.

AspectJ 1.6.7 dev builds already include the enhancements mentioned above. The latest AJDT dev builds for Eclipse 3.5 also include this AspectJ, however the ability to switch on timers in AJDT is not possible yet.

AspectJ: profiling pointcut matching

2009-11-09T09:28:00.000-08:00

This is the first of two articles about AspectJ 1.6.7 and pointcut matching. In this first one I'm going to talk about the new support for producing timing information showing where the time is spent whilst matching. In the second article I'll show how I've used that information to make AspectJ 1.6.7 the fastest release yet.

I'm not looking at compile time here and I'm not looking at the actual time taken to modify the bytecode. I'm purely focusing on the time taken to match pointcuts against bytecode. The pointcut syntax is very powerful and very flexible - the individual components can be combined into all manor of complex pointcuts - but how do you know if you are suffering poor match speeds due to a choice on wildcarding or combination of pointcut components? Until 1.6.7 that wasn't really possible. Even if you profile the weaver with something like yourkit, you don't see a breakdown of the cost of each individual pointcut element.

For a while now we have had the -timers option, but it hasn't produced detailed output:

> ajc -timers Foo.java Bar.aj
Compiler took 560ms

That is just the time taken for the whole process (compile/match/weave/output). With 1.6.7, if you additionally turn on verbose, you get two new kinds of pointcut timing information:

ajc -timers -verbose Simple.aj -inpath rt.jar
Pointcut fast matching cost (total=23ms for 16450 fast match calls):
...snip...
Pointcut matching cost (total=4282ms for 150000 joinpoints):
...snip...

Fast matching, what's that? AspectJ takes a two pass approach to matching. First fast match runs, where all the pointcuts are given a chance to say if they are interested in weaving some type. Fast match decisions must be made very quickly and are made purely at the type level - pointcuts must not say NO unless absolutely sure but saying YES when in fact detailed analysis proves the pointcut won't match is not a fast match error, it is just sub-optimal. Until 1.6.7 there was limited support for fast match and most pointcuts just said 'yes, I am interested' and deferred all their checks until the second stage. A good example of fast match is the within() pointcut - this can very easily quickly say yes/no as to whether it is going to weave a type.

After fast matching some type X we know the list of pointcuts that may weave X and call them all to do their detailed matching and work out if they actually do apply.

The pointcut that gave those numbers above was actually 'execution(* CharSequence+.*e*(..))'. There is no fast match support in 1.6.6 for that and so we see very little time spent in fast matching (23ms), but quite a while spent in detailed matching (4282ms). Of course purely seeing the time spent matching is interesting, but really you want to know which pointcuts contributed to that time.

Here is a more complete aspect that I'm going to weave into rt.jar (from a 1.6 JDK):

aspect SimpleAspect {
before(): execution(* CharSequence+.*e*(..)) {}
before(): execution(* *.*f*(..)) {}
before(): execution(* *t*.*(*)) {}
before(): staticinitialization(*y*) {}
before(): within(*p*) {}
}

And here is the fast match timer output:

Pointcut fast matching cost (total=283ms for 82250 fast match calls):

Time:40ms (types:#16450) fast matching against

(staticinitialization(*y*.()) && persingleton(SimpleAspect))

Time:41ms (types:#16450) fast matching against

(execution(* *t*.*(*)) && persingleton(SimpleAspect))

Time:34ms (types:#16450) fast matching against

(execution(* *f*(..)) && persingleton(SimpleAspect))

Time:35ms (types:#16450) fast matching against

(execution(* java.lang.CharSequence+.*e*(..)) && persingleton(SimpleAspect))

Time:132ms (types:#16450) fast matching against

(within(*p*) && persingleton(SimpleAspect))

As you can see, there is a complete breakdown of the time spent in fastMatch for each pointcut of interest. However, as you can see, very little time is spent fast matching - in fact the only significant time is spent in within() fast matching and that is what we'd expect since within() properly implements fast match.

Here are the timing numbers for the second stage of matching:


Pointcut matching cost (total=6532ms for 675000 joinpoint match calls):Time:482ms (jps:#168585) matching against 
    (staticinitialization(*y*.()) && persingleton(SimpleAspect))
Time:3970ms (jps:#168585) matching against 
    (execution(* *t*.*(*)) && persingleton(SimpleAspect))
Time:538ms (jps:#168584) matching against 
    (execution(* *f*(..)) && persingleton(SimpleAspect))
Time:1536ms (jps:#168584) matching against 
    (execution(* java.lang.CharSequence+.*e*(..)) && persingleton(SimpleAspect))
Time:4ms (jps:#662) matching against 
    (within(*p*) && persingleton(SimpleAspect))

See the difference that up front fast match made to the latter within() matching? It dismissed 1000s of types and so they were not processed by the within() in the secondary stage. But the real goal here is to gain some insight into the matching process and here we see explicitly that the execution(* *t*.*(*)) pointcut is taking forever to match - and that is a cue to:

- think about whether the pointcut captures what you really want it to
- perhaps ask on the mailing list about whether there is a better way to say the same thing
- raise a bug because it can't possibly suck that much :)

Notice how the second stage matching information also tells you how many joinpoints that pointcut was matched against - this can be a hint that perhaps it is worth trying to add some scoping to their pointcut definition. Add a within() to those pointcuts matching 100,000s of join points, if possible. Why will that make a difference? AspectJ processes pointcuts and rewrites them into an optimal component ordering - putting the components cheapest to evaluate first, like within(). If those cheap matches evaluate to false, the more expensive pieces later in the pointcut won't even be called to match.

Finally I should say that this timing information isn't just dumped out at the end of a build. I wanted it to be useful in environments like load time weaving - and in that scenario you never really know you are 'finished' as some other type may yet be loaded. So, currently this information is produced periodically - fast match information is produced every 250 fast match calls. Full match information is produced every time 25000 join points are processed. Both these numbers can be configured.

ajc .... -timers -verbose -Xset:timersPerJoinpoint=100,timersPerFastMatchCall=20

(produce fast match info every 20 calls to fast match and produce match info every 100 joinpoint match calls). Sometimes it is necessary to use numbers lower than the default in cases where the amount of weaving you are doing is never enough to trigger the default values.

In addition to the command line, this should work from Ant, with the timers="true" option (I've coded it in, just not written the tests). It definetly works from loadtime weaving, just specify "-timers -verbose" and optionally the Xset values in your aop.xml weaver options="" section.

The timers are not yet available in AJDT - that will come.

The latest dev builds of AspectJ 1.6.7 include this timer support and also some preliminary optimizations. Give it a go. Any feedback is welcomed.

I've been experimenting with this on some large systems and been able to identify bottlenecks in matching and identify common pointcut usages that occur so frequently they ought to be specifically optimized for (for example: execution(* (@Foo *).*(..)) - any method in a type annotated with @Foo). In the next post I'll show you what I've been able to do with the timing information and discuss why AspectJ 1.6.7 is so much faster... Just to wet your appetite for that article:

aspect SimpleAspect {

before(): execution(* CharSequence+.*e*(..)) {}

}

AspectJ1.6.6:

ajc -timers SimpleAspect.aj -inpath rt.jar -outjar woven.jar

Compiler took 19260ms

AspectJ1.6.7 dev builds:

ajc -timers SimpleAspect.aj -inpath rt.jar -outjar woven.jar

Compiler took 12531ms

Groovy eclipse: from alpha towards milestone

2009-08-06T09:50:00.000-07:00

After the initial splash of releasing the alpha, I don't want anyone to think we aren't working hard to improve things and get to that first milestone. There are already numerous fixes and enhancements and I'd like to thank everyone who has given it a spin.

In this post I'll pull together a flavour of what has changed since that first alpha - and perhaps when you see the kinds of issues that have been reported, it may give you the confidence to give it a try!

Back level groovyc
The most serious issue that arose was simply that we were based on an old development driver of groovy 1.7. It was a couple of months out of date, and the jars built at that time were incomplete. This is now remedied and the compiler has been rebased on the recently released official groovy 1.7 beta 1.

Generics checking of groovy code
Some of you have been playing with this feature more than I anticipated. It quickly became apparent that there was a need to support @SuppressWarnings so that users could hide warnings if they didn't want to mess around trying to correctly specify generic constraints. It now works:

(*cough* Just pretend you didn't see the final 's' in SuppressWarnings that isn't highlighted correctly...)

Coping with bad code
When any parser is integrated into an IDE for some filetype, and that IDE supports features like early error indication (as you type), that parser needs to be really robust. In Eclipse full compiles kick off for the code in the editor any time the user stops typing for even a moment - and not just when they Ctrl+S to save. This means the parser and compiler are often being asked to process incomplete/unfinished code. We are actively fixing issues where incomplete code crashes the groovy compiler.

JUnit
Based on a requirement from the Spock test framework we now have support for a monospace font in the JUnit failure trace view. This is surprisingly useful (even when not using Spock) and I now run with it as my default mode for JUnit. This shot shows the aligned output in the failure window (click to zoom):

The option to set the font is under Window>Preferences>Groovy

Error positions
The Eclipse Java support is great at positioning the 'squigglies' at just the right location to indicate where the problem is in your code. As any alpha users will have noticed, things aren't so great for groovy code. We are working to fix these situations.

Here is the 'before' shot taken on the first alpha (don't laugh):

We always report the errors, we were just having a few issues with properly locating them in the editor:
- the error that Gravy doesn't implement Comparable is recorded against the annotation
- the warning about Stack is not quite underlining the whole word
- the error about violated generic bounds is one character to the left of where it should be
(admittedly this was taken on Windows which I think was worse than mac/linux)

And now with recent fixes:

(there's that problem with the 's' in SuppressWarnings again...)

AST Transforms
The alpha had very basic support for AST Transforms. By specifying the jar containing the transform in the groovy.properties file in the root of the project, it would be picked up and any transforms within available during the project build. However, already we have seen cases where basic transforms (for example those triggered by @Entity) require much more than a single jar. For now we are still making AST transform support 'opt-in' but if you specify this particular entry in the groovy.properties file:

org.eclipse.jdt.core.compiler.groovy.groovyClassLoaderPath=%projclasspath%

Then the classloader responsible for loading AST transforms and their dependencies will run with the same classpath as the project. This will be the default behaviour before release (I think) but for now it is a helpful way for us to work out if transforms are involved in particular bugs that we get in.

Boring compiler internal bugs
I'm afraid I don't have any fancy screenshots here, and these are the bugs that can take a lot of effort to fix. We've had some great jira issues raised by people really mixing it up with groovy/java generic types extended/implemented via other groovy/java types. The basics are behaving pretty well and really it is only when users stray out into generics, covariance, etc that the compiler starts to misbehave. However, we can only fix bugs we know about so if you are having a problem, please let us know ! I am aware that some grails users are having issues with their projects but I can't seem to recreate it locally - if you have a failing grails project and can share it with us, please do!

Finally, we are also running Continuous Integration now, so we can get those fixes out faster.This means you should regularly update and if you haven't installed at all yet, what are you waiting for?

Eclipse update site for Eclipse 3.5:
http://dist.codehaus.org/groovy/distributions/greclipse/snapshot/e3.5

Eclipse update site for Eclipse 3.4.2:
http://dist.codehaus.org/groovy/distributions/greclipse/snapshot/e3.4

(The 3.4 stream is a bit behind the 3.5 stream, but will receive periodic updates)

Raise any issues at the JIRA:
http://jira.codehaus.org/browse/GRECLIPSE

Progressing from prototype: incremental Groovy/Java joint compiler

2009-06-03T09:58:00.001-07:00

Everything I blogged about previously was based on a prototype. A bit too much hacking here and there but I wanted to see if the hard problems could be solved rather than waste time on simple problems that were 'just a matter of coding' (hah). Since then I've taken a radical approach to improving the codebase... I've thrown everything away and started again, and the code is much better because of that.

But of all the things that worried me about the prototype the worst thing was probably having no tests. I'd get grails to build then realise I'd broken something that allowed groovyc to build. I couldn't go on without tests!

Testing
I spent a day looking at the JDT compiler and builder tests and created a variant of the harness that allowed me to write mixed groovy/java tests. The resultant tests are beautiful. Here is a basic compiler test:

public void testStandaloneGroovyFile() {
  this.runConformTest(new String[] {
    "p/X.groovy",
    "package p;\n" + 
    "public class X {\n" + 
    "  public static void main(String[] argv) {\n"+
    "    print \"success\"\n" + 
    "  }\n"+
    "}\n",
  },"success");  
}

It is so simple I don't even have to explain to you what it is doing... and the harness is flexible enough to support checking for expected errors, compiling multiple files. And best of all everything is one place, right there in the test method - each test isn't a test method plus a bit of xml config plus some source files from that directory over there (that you worked on with the command line then forgot to refresh in eclipse so the contents didn't get uploaded to the repo on sync - sigh).

From that humble test beginning, here is a test that is now working:

public void testDefaultValueMethods1() {
  this.runConformTest(new String[] {
    "p/C.java",
    "package p;\n" +
    "public class C {\n"+
    "  public static void main(String[] argv) {\n"+
    "    G o = new G();\n"+
    "    o.m(\"abc\",3);\n"+
    "    o.m(\"abc\");\n"+
    "  }\n"+
    "}\n",    

    "p/G.groovy",
    "package p;\n"+
    "public class G {\n" + 
    "  public void m(String s,Integer i=3) { print s }\n"+
    "}\n",
  },
  "abcabc");
}

That shows calling methods with default parameters from .java files.

On top of those compiler tests, I can also create builder tests that verify the usage of this modified compiler as it would be exercised when incrementally building an eclipse project. Here is a test that checks incremental compilation for groovy/java:

public void testIncrementalCompilationTheBasics() throws JavaModelException {
    IPath projectPath = env.addProject("Project");
    env.addExternalJars(projectPath, Util.getJavaClassLibs());
    env.addGroovyJars(projectPath);
    fullBuild(projectPath);
    
    // remove old package fragment root so that names don't collide
    env.removePackageFragmentRoot(projectPath, "");
    
    IPath root = env.addPackageFragmentRoot(projectPath, "src");
    env.setOutputFolder(projectPath, "bin");

    env.addClass(root, "pkg", "Hello",
      "package pkg;\n"+
      "public class Hello {\n"+
      "   public static void main(String[] args) {\n"+
      "      System.out.println(new GHello().run());\n"+
      "   }\n"+
      "}\n"
      );

    env.addGroovyClass(root, "pkg", "GHello",
      "package pkg;\n"+
      "public class GHello {\n"+
      "   public int run() { return 12; }\n"+
      "}\n"
      );
      
    incrementalBuild(projectPath);
    expectingCompiledClassesV("pkg.Hello","pkg.GHello");
    expectingNoProblems();
    executeClass(projectPath, "pkg.Hello", "12", "");

    // whitespace change to groovy file
    env.addGroovyClass(root, "pkg", "GHello",
        "package pkg;\n"+
        "public class GHello {\n"+
        "  \n"+ // new blank line
        "   public int run() { return 12; }\n"+
        "}\n"
        );
    incrementalBuild(projectPath);
    expectingCompiledClassesV("pkg.GHello");
    expectingNoProblems();

    // structural change to groovy file 
    // - did the java file record its dependency correctly?
    env.addGroovyClass(root, "pkg", "GHello",
        "package pkg;\n"+
        "public class GHello {\n"+
        // return type now String
        "   public String run() { return \"abc\"; }\n"+ 
        "}\n"
        );
    incrementalBuild(projectPath);
    expectingCompiledClassesV("pkg.GHello","pkg.Hello");
    expectingNoProblems();
}

As you can see, it is pretty straightforward:

- Build a pair of files (one groovy, one java)
- make a whitespace change to the groovy file and incremental build (this should only rebuild the groovy file).
- make a structural change to the groovy file and incremental build (should rebuild both groovy and the java file dependent on it)

I can even run the code at any time to check it is OK (I can *even* disassemble it and check the internal bytecode structure - love those JDT guys!)

With the ability to test, productivity has massively improved. The existence of the tests enabling the confidence to make more serious changes and refactorings as the code is developed.

Architecture
The architecture of the prototype was ... well perhaps 'architecture' isn't really the right word here. All jars that might be needed were dumped in a directory, then glued together with 'duct tape coding' - just join the necessary classes together and if that fails, apply more tape/code. The driving goals around the architecture of the new version are:

- do not damage JDT as a Java compiler (do not modify it to support a new language to the detriment of how fast and great it is at compiling Java). Changes are needed to JDT to plug groovy in, but they should be absolutely minimal.
- preserve modularity. Consume jdt.core as a plugin/bundle, consume the groovy compiler and dependencies as a plugin/bundle and create the glue code in a new plugin/bundle.
- ... do NOT damage JDT as a Java compiler!
- Design for future upgrades. Make the changes to JDT and the dependencies on groovy as simple as possible so that moving to a new JDT (eg. that in Eclipse 3.5) or a new groovy is simple.

And so far this has been achieved. The JDT has been modified to attempt to load a 'LanguageSupport' implementation dynamically. If this fails then it behaves as it always did - a pure Java compiler. If this succeeds then the language support hooks in where necessary. The LanguageSupport implementation is in a separate plugin and wires JDT to the existing groovy compiler plugin. The particularly great feature is that if you run an environment without the groovy language support plugin, then jdt just behaves as normal - it doesn't have a hard dependency on the groovy plugin which would cause it to fail if the groovy language support was missing.

Latest version
So what is the state of things now if I started again? My 50 tests are confirming that the basics already behave *but* generics is causing a headache (when doesn't it...) as I'm having some trouble turning the JDT representation of a type into what groovy wants to see, with respect to type variables, etc.

Where/When can I get this thing!!!
Within two months from now there should be something downloadable. We are looking to integrate this compiler into the existing Groovy eclipse plugin.

Grails built with Eclipse JDT Incremental Joint groovy/java compiler

2009-05-26T11:48:00.000-07:00

Just a quick entry on this. Each time I tackle compiling a new mixed java/groovy codebase, a new set of issues appear. No surprises really, just features I knew were not implemented yet that the particular codebase exploits. But here is the good news (this is Eclipse JDT compiler debug output):


>FULL BUILD STATS for: grails
>   compiled 139801 lines in 63692ms:2194.9lines/s
>   parse: 7437 ms (11.6%), resolve: 1150 ms (1.8%),
>   analyze: 50634 ms (79.4%), generate: 2836 ms (4.4%)
Recording new state : State for grails (#0 @ Tue May 26 11:47:22 PDT 2009)

Remember, those 'lines' counts are just including the java code right now. And don't read much into the time taken as I have more debug code than real code right now :)

Anyway, there were a couple of interesting cases in getting grails to compile that are worth talking about, just to give a feel for what is possible with this incremental compiler:

Properties
In Grails, the GROOVY type Mapping defines:

  boolean tablePerHierarchy = true

This is then referenced from a JAVA type (GrailsDomainBinder) through

  mapping.isTablePerHierarchy()

The modified JDT compiler handles this no problem.

Default parameter values
The GROOVY type GrailsPluginUtils defines this method with a default value for the closure parameter:

static synchronized Resource[] getArtefactResources(
 String basedir,
 Closure resourceResolver = DEFAULT_RESOURCE_RESOLVER) {

The JAVA type GrailsAwareGroovyTestSuite then accesses it through:

GrailsPluginUtils.getArtefactResources(grailsSettings.getBaseDir().getPath());

(supplying just the first parameter on the call)

The modified JDT compiler handles this no problem.

Generics
The GROOVY type NaturalId looks like this:

public class NaturalId {
   List<String> propertyNames = []
   boolean mutable = false
}

The JAVA type GrailsDomainBinder then attempts to access the propertyNames through a getter and treat the elements as Strings.

for (String propertyName : naturalId.getPropertyNames()) {

The modified JDT compiler handles this no problem.

But, of course, the best thing is that incremental compilation works. Let me break that property 'propertyNames' - change the type to just 'List' and save:

Starting build of grails @ Tue May 26 12:11:56 PDT 2009
Found source delta for: grails
Clearing last state : State for grails (#3 @ Tue May 26 11:58:31 PDT 2009)
INCREMENTAL build
Compile this changed source file src/.../NaturalId.groovy
About to compile src/.../NaturalId.groovy
Type has structural changes ...\NaturalId
 will look for dependents of NaturalId in org/codehaus/groovy/grails/orm/hibernate/cfg
Writing changed class file NaturalId.class
Found match in org.codehaus.groovy.grails.orm.hibernate.cfg to NaturalId
 adding affected source file src/.../GrailsDomainBinder.java
... edited out a list of other affected types ...
About to compile src/.../GrailsDomainBinder.java
Recording new state : State for grails (#4 @ Tue May 26 12:11:56 PDT 2009)
Finished build of grails @ Tue May 26 12:12:04 PDT 2009

it recognized that GrailsDomainBinder need a rebuild and now recorded an error against the attempted iteration:


Type mismatch: cannot convert from element type Object to String

Fixing the problem (converting propertyNames back to type List<String>) and resaving gives a similar incremental build log and the problem vanishes.

Anyway, that is probably the last codebase I'll experiment upon - the size of the task ahead is apparent, so the focus now is making what there is robust enough to distribute to other users.

Modifying JDT to Joint compile Groovy #2

2009-05-07T10:29:00.000-07:00

Just a follow up post on progress with the modified compiler.

With GANT working, I wanted to try it on another codebase. I initially chose the groovy eclipse plugin - and it was quite straightforward to fix a few things in the modified compiler in order to get that building. But I had a hard time running the tests. So I decided to try to go for a bit more of a challenge and build the groovy compiler itself (as an example of a mixed codebase) from:

https://svn.codehaus.org/groovy/trunk/groovy/groovy-core
(version 16084)

This really pushed the modified compiler and revealed quite a few gotchas that it hadn't previously been handling properly. Types like GroovyShellTestCase (groovy type) which extends GroovyTestCase (java type) and also has a GroovyShell (java type) delegate defined caused no end of headaches. The problem I had was that compilation would succeed but the tests would just explode. The curse of dynamic programming: everything goes wrong at runtime.

I eventually figured out how to run some groovy compiler tests inside eclipse. I wasn't sure which of the suites to try and launch so just selected the "src/test/(defaultpackage)" then "Right Click > Run as Junit". That comes to about 5400 - but I think includes some duplicates. It is certainly enough to be getting on with! However, debugging failures was horrible and I was reduced to comparing my .class file with that from a groovy distribution. Observing what methods had or hadn't been created, or why the method bodies differed and working out how the modified compiler had made such awful mistakes.

But today I'm back on track, it came down to a few things
- ensuring the representation of groovy types which the JDT then sees is absolutely correct.
- realising that the verifier visitor runs twice in joint compilation, and the implications that first run has on downstream code transformations (like DelegateAstTransformation)
- realising that the build process also needed me to run the DGMConverter app to create some classes needed later (DOH, this had me stuck for a while)

Today I'm down to:

So not 100% passing, but I'm not sure all the tests were designed to launch in eclipse in the way I am launching them (and in fact some of the failing cases pass when run standalone)

When it is behaving, the incremental builds are great. Here I made a structural change to ComplexCommandSupport.groovy and saved it - see the JDT doing the necessary impact analysis on what else needs building:


Starting build of groovyc @ Wed May 13 14:43:20 PDT 2009
Found source delta for: groovyc
Clearing last state : State for groovyc (#2 @ Wed May 13 14:43:10 PDT 2009)
INCREMENTAL build
Compile this changed source file src/main/org/codehaus/groovy/tools/shell/ComplexCommandSupport.groovy
About to compile src/main/org/codehaus/groovy/tools/shell/ComplexCommandSupport.groovy
Writing changed class file ComplexCommandSupport$_closure1.class
Type has structural changes org\codehaus\groovy\tools\shell\ComplexCommandSupport
  will look for dependents of ComplexCommandSupport in org/codehaus/groovy/tools/shell
Writing changed class file ComplexCommandSupport.class
Writing changed class file ComplexCommandSupport$_closure1_closure3.class
Writing changed class file ComplexCommandSupport$_createCompletors_closure2.class
Found match in org.codehaus.groovy.tools.shell to ComplexCommandSupport
  adding affected source file src/main/org/codehaus/groovy/tools/shell/commands/ShowCommand.groovy
Found match in org.codehaus.groovy.tools.shell to ComplexCommandSupport
  adding affected source file src/main/org/codehaus/groovy/tools/shell/commands/HistoryCommand.groovy
Found match in org.codehaus.groovy.tools.shell to ComplexCommandSupport
  adding affected source file src/main/org/codehaus/groovy/tools/shell/commands/ShadowCommand.groovy
Found match in org.codehaus.groovy.tools.shell to ComplexCommandSupport
  adding affected source file src/main/org/codehaus/groovy/tools/shell/commands/PurgeCommand.groovy
Found match in org.codehaus.groovy.tools.shell to ComplexCommandSupport
Found match in org.codehaus.groovy.tools.shell to ComplexCommandSupport
  adding affected source file src/main/org/codehaus/groovy/tools/shell/commands/RecordCommand.groovy
About to compile src/main/org/codehaus/groovy/tools/shell/commands/ShowCommand.groovy
About to compile src/main/org/codehaus/groovy/tools/shell/commands/HistoryCommand.groovy
About to compile src/main/org/codehaus/groovy/tools/shell/commands/ShadowCommand.groovy
About to compile src/main/org/codehaus/groovy/tools/shell/commands/PurgeCommand.groovy
About to compile src/main/org/codehaus/groovy/tools/shell/commands/RecordCommand.groovy
Skipped over unchanged class file ShowCommand$_closure1_closure5.class
Skipped over unchanged class file ShowCommand$_closure1.class
Skipped over unchanged class file ShowCommand$_closure2.class
Skipped over unchanged class file ShowCommand$_closure3_closure7.class
Skipped over unchanged class file ShowCommand$_closure4.class
Skipped over unchanged class file ShowCommand$_closure2_closure6.class
Skipped over unchanged class file ShowCommand$_closure4_closure8.class
Writing changed class file ShowCommand.class
Skipped over unchanged class file ShowCommand$_closure3.class
Skipped over unchanged class file HistoryCommand$_createCompletors_closure5_closure7.class
Writing changed class file HistoryCommand.class
Skipped over unchanged class file HistoryCommand$_closure1_closure6.class
Skipped over unchanged class file HistoryCommand$_closure1.class
Skipped over unchanged class file HistoryCommand$_closure2.class
Skipped over unchanged class file HistoryCommand$_createCompletors_closure5.class
Skipped over unchanged class file HistoryCommand$_closure4.class
Skipped over unchanged class file HistoryCommand$_closure3.class
Skipped over unchanged class file ShadowCommand$_closure2.class
Skipped over unchanged class file ShadowCommand$_closure4.class
Writing changed class file ShadowCommand.class
Skipped over unchanged class file ShadowCommand$_closure3.class
Skipped over unchanged class file ShadowCommand$_closure1.class
Skipped over unchanged class file PurgeCommand$_closure4.class
Skipped over unchanged class file PurgeCommand$_closure2.class
Skipped over unchanged class file PurgeCommand$_closure1.class
Writing changed class file PurgeCommand.class
Skipped over unchanged class file PurgeCommand$_closure3.class
Skipped over unchanged class file RecordCommand$_recordError_closure5.class
Skipped over unchanged class file RecordCommand$_closure1.class
Skipped over unchanged class file RecordCommand$_closure2.class
Writing changed class file RecordCommand.class
Skipped over unchanged class file RecordCommand$_closure4.class
Skipped over unchanged class file RecordCommand$_closure3.class
Recording new state : State for groovyc (#3 @ Wed May 13 14:43:20 PDT 2009)
Finished build of groovyc @ Wed May 13 14:43:21 PDT 2009

Modifying Eclipse JDT to joint compile groovy

2009-05-01T13:30:00.000-07:00

This is a little project I've been working on for a week or so. Right up front I'll say I'm a groovy newbie (grewbie?), but I do know a little bit about compilers.

After getting a little intro to groovy joint compilation (and groovy compiler structure) from Jochen Theodorou, I wondered how hard it would be to go about things slightly differently. Let the Eclipse JDT compiler take control of building a mixed code base but whenever it needed to deal with groovy - call the groovy compiler. If JDT could take control I could see all sorts of things 'just working' in Eclipse, problems upon which we are still expending large amounts of efforts to try and solve for AspectJ. If I could just plug groovy type reference resolution and classfile generation into JDT correctly then references between java and groovy artifacts would work and JDT would ensure incremental compilation worked, even across restarts of eclipse (this latter problem still hasn't been solved for AspectJ).

Effectively I just saw the problem as needing to bring together two phased compilers. The groovy one has many phases whilst the JDT has fewer and they are more of an internal notion than something clearly visible in the source code.

The main problem to address is handling references: java references into groovy files, and groovy references into java files. Clearly we want both and today the groovy joint compiler handles it via stub generation prior to calling javac which can then see the stubs.

For my 'mashup' I considered JDT as 3 phases:
1) diet parsing - where the basic structure of the types are determined, no method internals are processed and no references are chased down
2) complete type bindings - wire up the type references, work out which Foo they mean? com.a.Foo or com.b.Foo
3) code generation - process the method bodies and generate the code (there are really multiple phases here but we don't need to know about anything more granular for now)

These phases exist in groovy too, with other names - I didn't really know which phases they were so had a guess :)

The general algorithm is then:

1) JDT compiler receives a bunch of .groovy and .java files
2) diet parse all the .java files to learn their structure
3) drive the .groovy files through the first few phases (stopping before reference resolution)
4) do some wiring: so JDT can see the result of the groovy parse, and Groovy can see the result of the JDT parse
5) complete type bindings for java - resolving references to other JDT types and to groovy types
6) drive the rest of the groovy phases for each groovy file - resolving references to other groovy types and to JDT types
7) pass the .class files from groovy code gen back to JDT (so it can do incremental analysis)
8) generate the .class files for the .java files

And it worked. It is similar to the joint compilation strategy already employed by groovy but the communication between the compilers is much tighter. Initially a trivial little HelloWorld was all that worked:


-- HelloWorld.java --
public class HelloWorld {
public void run() {
 System.out.println("Hello World");
}

public static void main(String[] argv) {
    System.out.println("HelloWorld.main()");
    new pkg.GroovyClass().m(); // reference to groovy
}
}
--

-- GroovyClass.groovy --
package pkg

public class GroovyClass {
public void m() {
      System.out.println("GroovyClass.m() running")
}
public static void main(String []argv) {
    System.out.println("GroovyClass.main()");
    new HelloWorld().run(); // reference to java
}
}
--

But motivated by that I picked up GANT (Groovy based ANT) from http://gant.codehaus.org as a larger mixed project. This afternoon this successfully compiled:


>FULL BUILD STATS for: Gant_Trunk_160
> compiled 6501 lines in 1201ms:5412.9lines/s
> parse: 424 ms (35.3%), resolve: 56 ms (4.6%), analyze: 5 ms (0.4%), generate: 699 ms (58.2%)
Recording new state : State for Gant_Trunk_160 (#0 @ Mon May 04 17:33:50 PDT 2009)
Finished build of Gant_Trunk_160 @ Mon May 04 17:33:52 PDT 2009

Those stats are possibly not telling the entire story, they may be excluding some numbers for groovy, but it built! And the tests ran (all in eclipse):

But perhaps the most pleasing feature that worked was incremental compilation. I can change any file (groovy or java) and save it, triggering an incremental build. Here I modified an exception class (GantException.groovy) with a structural change (added a field), and the output reports:


INCREMENTAL build
Compile this changed source file src/main/groovy/gant/GantException.groovy
About to compile src/main/groovy/gant/GantException.groovy
Type has structural changes gant\GantException
will look for dependents of GantException in gant
Found match in gant to GantException
   adding affected source file src/main/groovy/gant/MissingTargetException.groovy
Found match in gant to GantException
   adding affected source file src/main/groovy/gant/TargetMissingPropertyException.groovy
Found match in gant to GantException
   adding affected source file src/main/groovy/gant/TargetExecutionException.groovy
About to compile src/main/groovy/gant/MissingTargetException.groovy
About to compile src/main/groovy/gant/TargetMissingPropertyException.groovy
About to compile src/main/groovy/gant/TargetExecutionException.groovy
Recording new state : State for Gant_Trunk_160 (#5 @ Mon May 04 17:41:11 PDT 2009)
Finished build of Gant_Trunk_160 @ Mon May 04 17:41:11 PDT 2009
GroovyIntegration: creating type declaration for gant.GantException

You can see that after saving that structural change, the JDT decided to rebuild a few other files that it knew depended upon it - but total incremental build time was less than one second. If .java files had depended upon the change, they would also have been recompiled.

Now - this is not ready for prime time yet, but proves the idea I had that integration was possible and that a good deal of JDT infrastructure (for incremental builds) would work. All these changes are being made to Eclipse JDT and the use of that compiler in Eclipse but, of course, the Eclipse compiler can be used in batch mode. So what we also have here is a java compiler that will happily allow groovy to be included in the mix and it can be run on the command line ... or from an Ant build script ... perhaps one generated by PDE that is being used to build an OSGi bundle that mixes java and groovy.

Look out for more updates soon...

AspectJ load-time weaving and memory usage

2009-04-09T09:59:00.000-07:00

To skip the background and just learn about the new weaver flags to control memory usage - just scroll to the end of this article.

Weaving is a complicated process and to reduce the complexity within the weaver it likes to take control of everything and know what it is dealing with. It builds a large data structure called a World. This World knows all about types - it knows intimate details (the bytecode) of the types being woven, and it knows the basic structure of types from the classpath that are being referenced during weaving (but are not themselves to be woven).

Knowing all this causes Worlds to get quite large, and they don't really give up the space they consume. This is mainly by design, a legacy design from when weaving was more something to play with than something to seriously use and rely on in your production quality application.

This article is about how to solve the Worlds problems (Badum-tish!).

The key data structure in the World is a typemap and the design is that it holds two categories of entity:

1) a representation of a type that would be hard to recover later
2) a representation of a type that would be easy to recover later

What falls into these two categories? From an AspectJ 1.6.4 point of view:
- aspects and anything that gets woven is category (1)
- types just referenced from woven types are category (2)

Category (2) can be basically treated like a cache - where if the information is in memory that is great, but if it isn't we can recover it quickly. Category (1) is the data we can't recovery quickly/easily and so represents a growing fixed part of the World - as more types are woven it gets larger and the space is never released.

For AspectJ 1.6.5 we are looking at what to do here. There are a few issues that with a little work could contribute a lot of benefit to the memory footprint.

Firstly, it is not true that just because something gets woven that it should be considered category (1). Sometimes weaving affects the type structure (intertype declarations), sometimes it does not (basic advice weaving). If the structure of the type is not affected then we don't need to remember it in detail and once it is woven we can forget about it. If we need to recover information about it later we can do so from the classpath or from the VM that loaded it (via reflection). The fact that it was woven will no longer matter because we will only be reliant on the structure, which we know did not change due to weaving.

Secondly, there are links between the two categories. Using either weak or soft references for category (2) entities is great but if there is a hard reference from a category (1) object to a category (2) object then it will never be garbage collected. Being smarter about type references will enable category (1) objects to have a 'softer grip' on the category (2) entities, allowing them to be GC'd.

Some initial weaver options are live today in AspectJ 1.6.5 dev builds. These options enable the weaver to consider types only affected by advice as category (2) - disposable and recoverable.

New weaver options.

The new options are related to what we shall call 'type demotion'. Once enabled, then after the weaver weaves a type it will look at whether it can be demoted from the fixed set of types in the World to the garbage collectable set of types. If it can be demoted then it will be eligible for GC later.

As an example, I used yourkit to attach to a weaver that had just (load time) woven 1400 classes with simple before advice. With type demotion switched off the size of the weaver was 14Meg and fixed, with no amount of forced GC's shrinking it. With type demotion switched ON, the size of the weaver was 2Meg.

The options can be specified in the aop.xml file:


<weaver options="-Xset:typeDemotion=true,typeDemotionDebug=true"/>

The former option 'typeDemotion=true' switches it on. The latter option will produce a few diagnostics about demotion when it occurs. If enough people try it out and it behaves then it could become default in AspectJ1.6.5 (I am expecting some issues that need solving).

I haven't fully thought through how this option will affect source compilation/weaving so use it in that scenario at your own risk (but I'd be interested in feedback!). I suspect:


TypeDemotion + IncrementalCompilation = headache

The work is being done under bug https://bugs.eclipse.org/bugs/show_bug.cgi?id=227484 in Eclipse bugzilla.

AspectJ 1.6.4 released

2009-04-02T09:45:00.000-07:00

I am trying to keep on track with a release every 3 months. It worked well throughout 2008 and here is the first proper release of 2009. The download is here:

http://eclipse.org/aspectj/downloads.php

and the README giving a high level overview of the changes is here:

http://www.eclipse.org/aspectj/doc/released/README-164.html

I always try and have themes for the releases, but these things inevitably change during development. Although I didn't originally intend to look at incremental compilation for 1.6.4, the difference between the JDT and AJDT experience is really hurting adoption. And thanks to some great feedback from our users (collecting diagnostics and discussing configuration scenarios) we have made huge improvements in AspectJ 1.6.4 and the associated AJDT 1.6.5 release. I tried to include a graph in the readme that showed the change in build times but the build times were so short with 1.6.4 that they hardly showed up!

Over the next few weeks I'll write some posts describing the new features in more detail.

Compiler variation: javac and jdt

2009-03-03T08:40:00.000-08:00

In the post about adopting ajc, I mentioned that care should be taken if choosing to run different compilers on the front and back end of your development process - since there can be some variation in the actual built .class files. The classes should function the same, of course, but those in the final project jar may not be exactly what you tested in your IDE. Something like AspectJ that exposes context based on the bytecode form (rather than the source form) can reveal these differences. These things get reported to me as bugs against AspectJ, when it is just that a compiler implementation choice has been revealed, and everything is working as designed.

I'm going to show you a scenario that demonstrates the kinds of difference I see - this came to my attention under bug ( https://bugs.eclipse.org/bugs/show_bug.cgi?id=255661 ).

public class C {
  public void foo() {
  }
  static class InnerType {
    private InnerType() {}
  }
}

Slightly peculiar program in that the ctor for the inner type is private. Compiling it will give the same results for JDT and javac:


javap -private C$InnerType
class C$InnerType extends java.lang.Object{
    private C$InnerType();
}

Now to make things interesting, I will invoke the constructor from the containing outer type:


public class C {
  public void foo() {
    new InnerType();
  }
  static class InnerType {
    private InnerType() {}
  }
}

Now if these were two different classes, that would be an error because the constructor is private, but as it is an inner type it is allowed. But how does the compiler support/implement it? This is where there is variation:

javac C.java
javap -private C$InnerType

class C$InnerType extends java.lang.Object{
    private C$InnerType();
    C$InnerType(C$1);
}



'jdt' C.java
javap -private C$InnerType

class C$InnerType extends java.lang.Object{
    private C$InnerType();
    C$InnerType(C$InnerType);
}

In either case a new constructor is added, which is visible to the outer class. This constructor is synthetic and takes a parameter simply to distinguish it from the existing constructor. The synthetic ctor simply calls the existing one. The choice of what type the parameter is what varies across the compiler implementations. Javac chooses to conjure up a new class (so there is a new C$1.class file created) whilst JDT chooses to use the type of the inner type. What happens at the call site?

JAVAC:

  4:   aconst_null
  5:   invokespecial   #3; //Method C$InnerType."":(LC$1;)V

JDT:

  3:   aconst_null
  4:   invokespecial   #18; //Method C$InnerType."":(LC$InnerType;)V

Both just pass null in. It doesn't matter what that value is, the parameter is only there to avoid a clash with the existing private ctor.

Effectively JDT has saved creation of a new class file by using the class for the inner type. But it could get unstuck in the unusual case:


public class C {
  public void foo() {
    new InnerType();
  }
  static class InnerType {
    private InnerType() {}
    private InnerType(InnerType t) {}
  }
}

Oh no! What happens now... the JDT compiler just adds another parameter:


class C$InnerType extends java.lang.Object{
    private C$InnerType();
    private C$InnerType(C$InnerType);
    C$InnerType(C$InnerType, C$InnerType);
}

That is where the javac solution helps - since it conjures up a class from nowhere, there can't be a clash for the synthetic it ctor it creates. However, I'd say I prefer the JDT compilers implementation, it avoids an entire extra class being created and this latter case which can cause some long constructors is very unusual. What about the call site for this final case? Just as expected:


   3:   aconst_null
   4:   aconst_null
   5:   invokespecial   #18; //Method C$InnerType."":(LC$InnerType;LC$InnerType;)V

I hope that has given you a flavour of the kind of variation I tend to see on a day-to-day basis. Either class will function exactly the same, but any tool or activity performed post compilation on the bytecode might behave differently in each case.

Load time weaving: the basics

2009-02-26T08:28:00.001-08:00

Today I'm going to show a basic AspectJ load-time weaving setup and which configuration options can be used to debug a scenario that just doesn't seem to be working as expected.

Here is a trivial application:

public class Simple {

  public static void main(String[]argv) {
    countFast(1000);
    countSlow(1000);
  }

  public static void countSlow(int value) {
    count(value,5);
  }

  public static void countFast(int value) {
    count(value,0);
  }

  private static void count(int value, int delay) {
    for (int i=0;i<value;i++) {
      try {Thread.sleep(delay);} catch (Exception e) {}
    }
  }
}

Awesome eh? Rather contrived, but I've introduced a delay down one of the execution paths so I can do some LTW profiling and pinpoint the problematic code.

Pure java so it can be compiled and run normally:
javac Simple.java
java Simple

Now to build a suitable profiling aspect.

aspect WhereDoesTheTimeGo {

  pointcut methodsOfInterest(): execution(* *(..)) &&
                              !within(WhereDoesTheTimeGo);

  private int nesting = 0;

  Object around(): methodsOfInterest() {
    nesting++;
    long stime=System.currentTimeMillis();
    Object o = proceed();
    long etime=System.currentTimeMillis();
    nesting--;
    StringBuilder info = new StringBuilder();
    for (int i=0;i<nesting;i++) {
      info.append("  ");
    }
    info.append(thisJoinPoint+" took "+(etime-stime)+"ms");
    System.out.println(info.toString());
    return o;
  }
}

This aspect could be compiled and woven at source time, but it is more likely something to only be used occasionally when the code is run. To include at source time:


ajc WhereDoesTheTimeGo.java Simple.java
java Simple

    execution(void Simple.count(int, int)) took 2ms
  execution(void Simple.countFast(int)) took 4ms
    execution(void Simple.count(int, int)) took 5049ms
  execution(void Simple.countSlow(int)) took 5049ms
execution(void Simple.main(String[])) took 5054ms

But I want to show load-time weaving. Now load time weaving needs me to do two things:

- launch the VM with the AspectJ agent registered
- provide an xml file to configure the weaver

The AspectJ agent is a simple JVMTI agent that can get involved in the class loading process and weave any types before they are defined in the VM. In either source compilation or binary weaving, the AspectJ compiler knows the set of aspects involved, either by being given the source for them or the jars containing them. In a load time weaving setup they could be anywhere on the classpath and so to save a very costly scan of all the classpath contents, it relies on an xml config file to name the aspects it should use. There are a lot of potential options that can be specified in the xml file, but a minimal one suitable for my needs here is:


<aspectj>
 <aspects>
   <aspect name="WhereDoesTheTimeGo"/>
 </aspects>
</aspectj>

It just defines one aspect called WhereDoesTheTimeGo (the weaver will expect to find the class for this type on the classpath). In fact for a simple case like this, the compiler can produce suitable xml just by specifying a flag at compile time. So now I'll use that flag and build the aspect into a reusable aspect library jar.


ajc WhereDoesTheTimeGo.java -outxml -outjar timing.jar

The timing.jar will now contain an aop-ajc.xml (in a META-INF directory) and the class file for the compiled aspect.

jar -tvf timing.jar
  55 Wed Feb 25 22:06:30 GMT 2009 META-INF/MANIFEST.MF
5149 Wed Feb 25 22:06:32 GMT 2009 WhereDoesTheTimeGo.class
  85 Wed Feb 25 22:06:32 GMT 2009 META-INF/aop-ajc.xml

The one generated by -outxml is called META-INF/aop-ajc.xml (being in the META-INF folder is important!). That name is chosen for the generated one so it does not clash with (or overwrite) any aop.xml I might be working on separately that contains more advanced options. The agent is going to merge together the contents of all the xml config files it finds and use the combination to configure the weaver.

It is ready to go, I just need to launch the VM with the AspectJ agent, see the -javaagent option here:


java -javaagent:<pathToAspectj>/lib/aspectjweaver.jar
     -classpath "code;timing.jar;<pathToAspectj>/lib/aspectjrt.jar" Simple

   execution(void Simple.count(int, int)) took 1ms
 execution(void Simple.countFast(int)) took 6ms
   execution(void Simple.count(int, int)) took 5114ms
 execution(void Simple.countSlow(int)) took 5114ms
execution(void Simple.main(String[])) took 5121ms

The Simple class is woven as the VM loads it. The classpath requires: my Simple class (now in the code directory), the aspect library (containing the aspect .class and the xml) and finally the aspectj runtime jar. Running without either the agent specified or the aspect library on the classpath will execute Simple program normally.

Now that's a nice simple scenario, but what about when things aren't going so well... What on earth should I do if just don't see the aspect weaving anything? How do I know what is going on at load time?

To explore what the system is doing, I need to expand my aop-ajc.xml file. Firstly I just want to know the basics about what AspectJ is up to. The xml file can have a weaver section where various options can be configured - many of which match the options that can be specified when calling the compiler on the command line. AspectJ will merge together any suitable aop xml files it discovers on the classpath in order to define the behaviour of a weaver, so I can either modify my existing aop-ajc.xml inside the jar or create another one just for trying out some options.

For now I will modify the existing one - so I extract it from the jar, update it and pack it back into the jar. Here is the new one:


<aspectj>
 <aspects>
   <aspect name="WhereDoesTheTimeGo"/>
 </aspects>
 <weaver options="-verbose"/>
</aspectj>

I've turned on verbose mode for the weaver. Now on running it I will see:


[AppClassLoader@9fbe93] info AspectJ Weaver Version DEVELOPMENT 
                        built on Wednesday Feb 25, 2009 at 21:17:03 GMT
[AppClassLoader@9fbe93] info register classloader sun.misc.Launcher$AppClassLoader@9fbe93
[AppClassLoader@9fbe93] info using configuration
                        file:/C:/blog/timing.jar!/META-INF/aop-ajc.xml
[AppClassLoader@9fbe93] info register aspect WhereDoesTheTimeGo
[AppClassLoader@9fbe93] info processing reweavable type WhereDoesTheTimeGo: 
                        WhereDoesTheTimeGo.java

   execution(void Simple.count(int, int)) took 0ms
 execution(void Simple.countFast(int)) took 3ms
   execution(void Simple.count(int, int)) took 5003ms
  execution(void Simple.countSlow(int)) took 5003ms
execution(void Simple.main(String[])) took 5007ms

What does it all mean? Firstly all weaver messages are prefixed with the classloader the weaver instance is attached to - in a multi classloader setup this can really help me understand the mixed up verbose output. The first message gives the version number of the weaver being used then it tells me the xml configuration file that will be used for this weaver (file:/C:/blog/timing.jar!/META-INF/aop-ajc.xml) and then the messages tell me which aspects have been defined based on that configuration.

Running with the verbose option can at least tell me if the weaver is being created and that the agent setup is correct. But it still will not tell me if any weaving is occurring. For that I would use the additional option '-showWeaveInfo' exactly the same as I would on the command line.


<aspectj>
 <aspects>
   <aspect name="WhereDoesTheTimeGo"/>
 </aspects>
 <weaver options="-verbose -showWeaveInfo"/>
</aspectj>


[AppClassLoader@9fbe93] info AspectJ Weaver Version DEVELOPMENT 
                        built on Wednesday Feb 25, 2009 at 21:17:03 GMT
[AppClassLoader@9fbe93] info register classloader sun.misc.Launcher$AppClassLoader@9fbe93
[AppClassLoader@9fbe93] info using configuration 
                        file:/C:/blog/timing.jar!/META-INF/aop-ajc.xml
[AppClassLoader@9fbe93] info register aspect WhereDoesTheTimeGo
[AppClassLoader@9fbe93] weaveinfo Join point 
                        'method-execution(void Simple.main(java.lang.String[]))'
                        in Type 'Simple' (Simple.java:3) advised by around advice
                        from 'WhereDoesTheTimeGo' (WhereDoesTheTimeGo.java:7)
[AppClassLoader@9fbe93] weaveinfo Join point 
                        'method-execution(void Simple.countSlow(int))' 
                        in Type 'Simple' (Simple.java:8) advised by around advice
                        from 'WhereDoesTheTimeGo' (WhereDoesTheTimeGo.java:7)
[AppClassLoader@9fbe93] weaveinfo Join point 
                        'method-execution(void Simple.countFast(int))' 
                        in Type 'Simple' (Simple.java:12) advised by around advice
                        from 'WhereDoesTheTimeGo' (WhereDoesTheTimeGo.java:7)
[AppClassLoader@9fbe93] weaveinfo Join point 
                        'method-execution(void Simple.count(int, int))' 
                        in Type 'Simple' (Simple.java:16) advised by around advice 
                        from 'WhereDoesTheTimeGo' (WhereDoesTheTimeGo.java:7)
[AppClassLoader@9fbe93] info processing reweavable type WhereDoesTheTimeGo: 
                        WhereDoesTheTimeGo.java
    execution(void Simple.count(int, int)) took 1ms
  execution(void Simple.countFast(int)) took 3ms
    execution(void Simple.count(int, int)) took 5005ms
  execution(void Simple.countSlow(int)) took 5005ms
execution(void Simple.main(String[])) took 5016ms

See the new messages prefixed 'weaveinfo' - just as I would expect to see on the command line if weaving via ajc.

The weaver section of the xml configuration can also define which types should be included in or excluded from weaving. Although pointcuts can limit where aspects apply, it is sometimes useful to write these specifications in the xml configuration. As a simple example I can exclude my Simple class from being woven:

<aspectj>
 <aspects>
   <aspect name="WhereDoesTheTimeGo"/>
 </aspects>
 <weaver options="-verbose -showWeaveInfo">
   <exclude within="Simple">
 </weaver>
</aspectj>

Running with that configuration, I can see no weaving messages. Although it might have been nice for the weaver to tell me it excluded my class... and that brings us to the '-debug' option. If it looked like my code was not getting woven even though the right aspects were being defined - I would use the debug option:


<aspectj>
 <aspects>
   <aspect name="WhereDoesTheTimeGo"/>
 </aspects>
 <weaver options="-debug">
   <exclude within="Simple">
 </weaver>
</aspectj>

Now when I run it (notice I've removed -verbose and -showWeaveInfo to limit the output):


[AppClassLoader@9fbe93] debug not weaving 'Simple'

This message indicates the load time weaving infrastructure chose not to weave Simple - and that is due to the include/exclude constraints specified. Finally I will remove my exclude section from the xml and re-run:


[AppClassLoader@9fbe93] debug weaving 'Simple'
[AppClassLoader@9fbe93] debug generating class 'Simple$AjcClosure1'
[AppClassLoader@9fbe93] debug generating class 'Simple$AjcClosure3'
[AppClassLoader@9fbe93] debug generating class 'Simple$AjcClosure5'
[AppClassLoader@9fbe93] debug generating class 'Simple$AjcClosure7'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.runtime.reflect.Factory'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.lang.reflect.SourceLocation'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.runtime.reflect.MethodSignatureImpl'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.lang.reflect.MethodSignature'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.runtime.reflect.CodeSignatureImpl'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.runtime.reflect.MemberSignatureImpl'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.runtime.reflect.SignatureImpl'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.runtime.reflect.ConstructorSignatureImpl'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.lang.reflect.ConstructorSignature'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.runtime.reflect.UnlockSignatureImpl'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.lang.reflect.UnlockSignature'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.runtime.reflect.LockSignatureImpl'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.lang.reflect.LockSignature'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.runtime.reflect.AdviceSignatureImpl'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.lang.reflect.AdviceSignature'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.runtime.reflect.InitializerSignatureImpl'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.lang.reflect.InitializerSignature'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.runtime.reflect.CatchClauseSignatureImpl'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.lang.reflect.CatchClauseSignature'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.runtime.reflect.FieldSignatureImpl'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.lang.reflect.FieldSignature'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.lang.JoinPoint'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.runtime.reflect.SignatureImpl$Cache'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.runtime.reflect.JoinPointImpl$StaticPartImpl'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.runtime.reflect.SourceLocationImpl'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.runtime.reflect.JoinPointImpl'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.lang.ProceedingJoinPoint'
[AppClassLoader@9fbe93] debug weaving 'WhereDoesTheTimeGo'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.lang.NoAspectBoundException'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.runtime.internal.Conversions'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.runtime.reflect.StringMaker'
[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.runtime.reflect.SignatureImpl$CacheImpl'
    execution(void Simple.count(int, int)) took 1ms
  execution(void Simple.countFast(int)) took 5ms
    execution(void Simple.count(int, int)) took 5001ms
  execution(void Simple.countSlow(int)) took 5002ms
execution(void Simple.main(String[])) took 5008ms

Due to the weaver now actually doing something, a few more types got loaded during the application run. The debug output is telling me important information:

[AppClassLoader@9fbe93] debug cannot weave 'org.aspectj.lang.ProceedingJoinPoint'
'cannot weave' - indicates the type is in a package for which weaving is forbidden. By default org.aspectj, java. and javax. are all forbidden (so I don't need to exclude them myself). The latter two packages can be woven by the use of extra options in the weaver section.

[AppClassLoader@9fbe93] debug weaving 'Simple'
- the weaver got a chance to process the specified type. IT DOES NOT MEAN THE WEAVER ACTUALLY CHANGED IT (I ought to change that message really, to indicate this). To also see if the type was modified during weaving, I would use -showWeaveInfo in addition to -debug

Armed with those 3 options I can investigate any basic wierdness I see when trying to get started with load time weaving. Even more advanced options can involve turning on weaver trace or dumping the bytecode (before and after it is woven) - but those tend to be used when an AspectJ compiler developer is asking for diagnostics.

More info?
- load time weaving configuration documentation: http://www.eclipse.org/aspectj/doc/released/devguide/ltw-configuration.html
- enabling weaving of java. and javax. packages:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=149261#c11

Joined Planet Eclipse

2009-02-25T09:52:00.000-08:00

To promote the Eclipse AspectJ project and share tips on best practice with AspectJ, I've joined Planet Eclipse (Hello guys!). On this blog you will find mainly tips and tricks for AspectJ but also general articles on software development that relate to what I'm currently working on. I also come across all manner of strangeness with Java as I sometimes have to go deep into the corners when an AspectJ bug is raised, and I'll share some of those findings here.

Right this minute, I am working towards releasing AspectJ 1.6.4, and working with Andrew E on the related AJDT 1.6.4 (see his AJDT musings at http://contraptionsforprogramming.blogspot.com/ ).

Andy Clement.

AspectJ: fixing 'reverse cascade errors'

2009-02-17T10:03:00.000-08:00

I do hate to see invalid cascading errors in an editor. This is where there is an error in the source code I happen to be editing and the parser recovery can't quite ever recover from it so I see further errors later on the editor that aren't really problems - they vanish as soon as I address the first error.

A problem AspectJ had for a while is the worst kind of cascade error, these cascade UP the editor rather than down. If I have an error then I start to see problems reported earlier in the source file than where the real problem is. What do I have to fix? I have to scroll through all of them and attempt to work out which one is the real error. Truly horrible.

AspectJ 1.6.4 fixes this problem.

Here is an example of what I mean. A simple piece of code:


public class C {
      public static void main(String[] args) {
            String s = new String();
            s.toCharArray();
      }

Clearly the problem is the missing ending curly bracket to close the class declaration.

Compile that with AspectJ 1.6.3:


>ajc C.java
C.java:4 [error] Syntax error on tokens, valid member declaration expected instead
s.toCharArray();

C.java:5 [error] Syntax error, insert "}" to complete ClassBody
  }

Er, what?? What is that first error? Here there are just two problems but in a large file there can be hundreds of occurrences of this issue.

Now if I compile with a recent AspectJ 1.6.4 dev build:


>ajc C.java
C.java:5 [error] Syntax error, insert "}" to complete ClassBody
  }

AspectJ now only reports the proper error and no incorrect cascades. Although I am only showing command line usage here, the problem is also addressed in AJDT by the same fix. What was the problem? Some rules in the parser grammar only exist to enable recovery and will only attempt to be used to match input tokens once a single error is found. There was a bad recovery rule in there that matched valid code as erroneous code once the parser got into a recovery state.

I think it is a sign of compiler maturity when I am spending my time fixing how the compiler copes with broken code - it is an indication that it is handling valid code just fine.

AspectJ: optimized annotation value binding

2009-02-17T09:24:00.000-08:00

Just a quick tip today. Since AspectJ 1.5.0 it has been possible to bind annotation values for use in advice, here is an entire program, called Demo.java:


import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;

enum Colour {
 RED, GREEN, BLUE;
}

@Retention(RetentionPolicy.RUNTIME)
@interface ColouredAnnotation {
 Colour value();
}

public class Demo {
 public static void main(String[] args) {
  // warmup a bit
  runOne(); runOne(); runOne(); runOne();
  runOne(); runOne(); runOne(); runOne();
  long stime = System.nanoTime();
  runOne();
  System.out.println("Took " + (System.nanoTime() - stime)+"ns");
 }

 public static void runOne() {
  for (int i = 0; i < 10000; i++) {
   colouredMethod();
  }
 }
 
 @ColouredAnnotation(Colour.RED)
 public static void colouredMethod() { }

}

It simply runs an annotated method 10000 times and prints out the time taken. Now here is some advice that binds the annotation so the colour value of the annotation can be accessed in the advice:


aspect ColourMonitor {

   // Binding the whole annotation for access in the advice:
   pointcut colouredMethods(ColouredAnnotation anno):
     execution(@ColouredAnnotation * colouredMethod(..)) && 
     @annotation(anno);

   before(ColouredAnnotation anno): colouredMethods(anno) {
     Colour colour = anno.value();
     String colourName = colour.name();
   }
}

After applying that aspect to the Demo class and executing it, typical times reported on my machine are:


Took 2096884381ns
Took 2062844928ns
Took 2097097187ns

About 2 seconds per run - there is a high price to pay for the flexibility of having the entire annotation in the advice. In AspectJ 1.6.1 a new syntax was added for just this kind of usage of annotations. Here is the same aspect but the value binding is done directly in the pointcut and advice signature:


aspect ColourMonitor {

   // Bind just the part of the annotation of interest:
   pointcut colouredMethods(Colour colour):
    execution(@ColouredAnnotation * colouredMethod(..)) &&
    @annotation(ColouredAnnotation(colour));

   before(Colour colour): colouredMethods(colour) {
     String colourName = colour.name();
   }
}

Is it quicker? Er, just a bit:


Took 319245ns
Took 313797ns
Took 320990ns

That is about 0.0003 seconds.

There are currently restrictions around this syntax usage. It is not possible at all join points and it is not possible for all kinds of annotation value (above I showed how it is used with enum values). Adding support for other values will be straightforward and I plan to address them as they come up in real use cases, but it cannot necessarily be supported at all join points.

But if you have a scenario that fits, it can offer a nice performance boost.

AspectJ: Choosing ajc

2009-02-16T09:40:00.000-08:00

I often read on the AspectJ mailing list that users are starting to learn about AspectJ via load-time weaving. Often it seems to be due to concerns about changing the compiler they are using earlier in their process (during development time or in product build). Although learning via load-time weaving is perfectly valid, there are currently some drawbacks if developing outside of Eclipse AJDT tools:

- there are no helpful gutter markers indicating where pointcuts match
- there is no immediate feedback about errors made with the pointcut syntax
- diagnosing problems is tricky

I plan to write a separate post about load-time weaving (and tackle the problem diagnosis area) but in this post I want to address any issues around the ajc compiler. I'll do this in an FAQ style and any questions asked in post responses, I'll fold into the overall post at a later date.

Q) Why would I risk switching to a new compiler technology?

The AspectJ compiler, ajc, is not a new compiler technology. It is a copy of the Eclipse compiler with extensions to support the AspectJ language. I have seen (and worked on) many projects where the development tools are eclipse and then after the code is committed to the source code repo, the backend build process uses standard javac to create the product distribution. That is an example of using very different compiler technologies during a project lifecycle. Yes, the Java language is well specified, but the implementation specifics vary by compiler. They can sometimes create different bytecode based on the same source, and the quality/consistency of warnings and errors can vary. In this kind of scenario using ajc instead of javac on the backend is likely to give a more consistent experience because it will be the same compiler technology as used in the IDE.

Q) What kind of extensions are made to the Eclipse compiler?

Two examples of the typical kind of extension:
- extended grammar to allow the new keywords and constructs: 'aspect', 'before', 'after'
- extended internally to allow for features like intertype declarations. For example: the compiler is about to produce an error because a method does not exist, but before it does a check is made to see if it would be satisfied via an intertype declaration.

Q) Do these extensions slow the compiler down?

When writing pure Java, some of these extensions are still exercised but find they have nothing to do. For example: Consider the case above when the check is made to see whether an ITD satisfies what looks like a missing method. If there are no ITDs then that extension terminates immediately. If the extensions did slow it down to a significant degree I would consider that a bug.

Q) Do these extensions 'damage' pure java compilation?

Over the last several years there have been on average less than one bug per year where a compiler extension has damaged the compilers ability to compile pure Java. These are fixed extremely quickly. More frequently it tends to be a bit of Java that also fails on the equivalent Eclipse compiler level and so the fix is to upgrade AspectJ to the latest Eclipse compiler and pickup their fix.

Q) What version of the Eclipse compiler is it based on?

A simple 'ajc -version' will always reveal the current internal compiler level:

AspectJ Compiler 1.6.3 (1.6.3 - Built: Thu Feb 12, 2009 at 20:28:24 GMT) - Eclipse Compiler 0.785_R33x, 3.3

So that is the Eclipse 3.3 compiler. In JDT terms, build 785 of the compiler.

Q) Why does AspectJ not track the very latest Eclipse compiler versions?

Moving to a new compiler and porting the extensions across is a little bit complicated. Usually it is only done for a very good reason. AspectJ moved to Eclipse 3.3 to pick up the Java6 support within it, before that AspectJ was based on Eclipse 3.1. Yes, Eclipse 3.4 is out but there are no open AspectJ bugs that would be fixed by moving to a new version of the Eclipse compiler. Also there is a big benefit that the AspectJ compiler can be plugged into multiple levels of Eclipse (for example, AspectJ1.6.X will happily run in Eclipse 3.3, 3.4 or 3.5) and each time an upgrade is made there is a risk the compiler dependencies will have changed to such a degree that it will no longer run in an older Eclipse. There is finite resource on the team so managing multiple branches is not really feasible.

Q) How are the performance and memory profiles for this modified Eclipse compiler?

The cost of the extensions when not using aspects should be negligible, otherwise I would consider it a bug. I haven't done any direct performance comparisons on large projects - I'd be interested if anyone has and wants to share the results with me? Throughout the 1.6 AspectJ releases a key focus has been performance (in terms of time and memory usage) and as can be seen in the AspectJ 1.6.1 readme - some significant improvements have been made.

Q) If my project breaks due to ajc, how soon can I get a fix?

The AspectJ team is small, and that has meant all the processes have had to be fully automated. A continuous integration approach is used and ALL tests are automated. The project has just passed 4000 tests, and of course it is based on the JDT compiler which itself has 10s of thousands of tests run against it. No build is published to the Eclipse AspectJ download site unless it passes every test. Each build adds new regression tests and so every build is ideally better than the one before. There is no drop in quality as a rush is made to integrate new features. And a released build of AspectJ is just a development build where the build tag is changed from DEVELOPMENT to 1.6.X. All this means fixes can be released within 24hours - sometimes quicker when a helpful user contributes a failing testcase when raising the bug.

Q) Doesn't that need to constantly maintain working dev builds limit the kind of work that can be done in the codebase? Is their room for innovation?

During the 1.6 releases alone, the weaver part of the compiler has undergone at least two large overhauls. During 1.6.1 development we discarded 20% of the weaver during a refactoring exercise - compare the sizes of aspectjweaver in 1.6.0 and 1.6.1 : AspectJ in the Maven repo . The fact that HEAD is always working and that there are so many tests gives us incredible confidence to make large scale refactorings without damaging the quality. During 1.6.3 development the weaving and matching were split apart so that the matcher would be easier to use standalone.

Q) Where can I ask more questions?

Feel free to post here or on the AspectJ mailing list which is accessible through:
http://www.eclipse.org/aspectj/userlists.php.

I hope that has filled in a few blanks for any users wondering about picking up ajc and trying it out.

AspectJ: advising methods based on parameter annotations

2009-02-08T17:31:00.000-08:00

Since version 1.5.0, AspectJ has been able to advise annotated types, methods and fields. Buried in the readme for 1.6.0 is a note that support has now been added for matching on parameter annotations. In this short post I'll describe what that means, and how to very quickly build an aspect that can help you benefit from this feature.

Here's a bit of Java showing the use of parameter annotations. The printString() method has the @NonNull annotation attached to the first parameter:

public class TestClass {
    public static void printString(@NonNull Message message) {
        System.out.println(message.getMessage());
    }

    public static void main(String[] argv) {
        printString(new Message("Hi!"));
        printString(null);
    }
}

class Message {
    private String text;

    Message(String text) {
        this.text = text;
    }

    public String getMessage() {
        return this.text;
    }
}

// Marker Annotation
@Retention(RetentionPolicy.RUNTIME)
@interface NonNull { }

The @NonNull annotation is intended to declare a simple constraint that the method never expects to get a null passed in as the value for that parameter. Running that simple code causes (unsurprisingly) an NPE:

> java TestClass
Exception in thread "main" java.lang.NullPointerException
 at TestClass.printString(TestClass.java:3)

Nothing is enforcing the constraint. I want to enforce it using an aspect.

Since AspectJ 1.5.0, the pattern for matching execution of a method involving annotated types has been something like this:


execution(public void someMethod(@SomeAnnotation SomeType))

This would match "execution of a public method called someMethod which takes only one argument of type SomeType, and that type SomeType has the annotation SomeAnnotation". Effectively this requires that SomeType was declared like this:


@SomeAnnotation
class SomeType {
 ...
}

For the NonNull case there is no annotation on the parameter type (Message), it is solely on the parameter. In AspectJ 1.6.0 the syntax for the execution pointcut has been extended to allow a grouping to be specified for the annotation - to indicate whether it is expected to be an annotation associated directly with the type of the parameter or with the parameter itself.

Here is the new syntax variant:


execution(public void someMethod([parameterannotations] ([typeannotations] SomeType)))

Extra parentheses in the method argument section of the pointcut determine whether a specified annotation is to be found on the type itself or the parameter. Now we can write a NonNull matching pointcut:


execution(* *(@NonNull (*)))

"execution of any method where the first parameter has the @NonNull annotation."

Contrast that with this variant (that has no parentheses):


execution(* *(@NonNull *))

"execution of any method where the type of the first parameter has the @NonNull annotation."

Or this variant:


execution(* *(@NonNull (@SomeAnnotation *)))

"execution of any method where the type of the first parameter has the annotation @SomeAnnotation, and the parameter itself has the @NonNull annotation."

Multiple annotations can be specified at either of those positions.

By extending the syntax in this way for execution() pointcuts, AspectJ avoids the need to introduce yet another top level pointcut construct for this case. It can seem like a further step into parentheses hell, it is not that difficult to understand and the pointcut signature continues to look like the method declaration when supported in this way directly in the execution pointcut syntax.

Now I can wrap the pointcut up in a Null detection aspect


public aspect NullParameterChecker {
    // Note that args() is used to bind the first parameter
    // so it can be checked in the advice
    pointcut methodExpectingNonNullFirstArg(Object o):
     execution(* *(@NonNull (*),..)) && args(o,..);

    before(Object o): methodExpectingNonNullFirstArg(o) {
        if (o==null) {
            throw new IllegalArgumentException(
              "First argument is null at "+thisJoinPoint.getSignature());
        }
    }
}

In both the execution() and args() pointcut the ",.." syntax is used to indicate I do not mind if the method has more parameters, I am just interested in the first one.

The simple before() advice here will run at the start of the method and for a null argument value it will produce an IllegalArgumentException with a message of our choice. Further down the stack trace will indicate which offending caller passed null.

Extending the aspect to check multiple parameters currently requires the definition of multiple pointcuts and advice:


pointcut methodExpectingNonNullFirstArg():
execution(* *(@NonNull (*),..));

pointcut methodExpectingNonNullSecondArg():
execution(* *(*,@NonNull (*),..));

pointcut methodExpectingNonNullThirdArg():
execution(* *(*,*, @NonNull (*),..));

pointcut methodExpectingNonNullFourthArg():
execution(* *(*,*,*,@NonNull (*),..));

In each case there, the '*' is used to stand in for parameters that are not of interest. There is an open AspectJ enhancement request to provide a neater syntax here, but for now I need to write the above code. Clearly it will start to get a bit 'verbose' if there are numerous parameters, but I would say if I start having methods with that 6, 7, or more, parameters - the codebase has other problems ;)

It is also possible to match on annotation parameters to constructors in a similar way (here, using 'new' instead of a method name or method name wildcard):


pointcut firstParameterNonNull():
execution(new(@NonNull (*),..));

And if I were uncomfortable with the AspectJ code style, everything shown above can be written in an annotation style pure Java aspect, using pointcuts like this:


@Pointcut("execution(* *(@NonNull (*),..)) && args(o)")
public void methodExpectingNonNullFirstArg(Object o) {}

Building the code

There are many options for linking the aspect into an existing Java code - here I'm just going to show two possibilities: source compilation and binary weaving. I'll try and cover some of the other options in a later post. I could have used AJDT (the AspectJ Eclipse support: http://eclipse.org/ajdt), but I'm a command line kind of guy. So I grabbed the latest dev build of AspectJ from http://eclipse.org/aspectj/downloads.php and installed it. Once installed and setup, I can run 'ajc' from the command line:


ajc -1.5 TestClass.java Message.java NonNull.java NullParameterChecker.aj

If I wanted to know where the aspect was affecting my code, I could add the -showWeaveInfo option:

ajc -1.5 -showWeaveInfo TestClass.java Message.java NonNull.java NullParameterChecker.aj

Join point 'method-execution(void TestClass.printString(Message))' in Type 'TestClass' 
  (TestClass.java:5) advised by before advice from 'NullParameterChecker'

If all source was in a single src folder, I can:

ajc -1.5 -sourceroots src -d bin

Once built, I can run it with any old JVM (1.5 or later). The compiler produces standard Java bytecode, but that bytecode will have a dependency on the aspectj runtime - a small jar that must be on the classpath when executing an AspectJ application. It provides implementations of AspectJ support functions, like the 'thisJoinPoint' construct that can be seen used in the before advice.

java -classpath "bin;c:\aspectj163\lib\aspectjrt.jar" TestClass

Exception in thread "main" java.lang.IllegalArgumentException:
  First parameter is null at void TestClass.printString(Message)

AspectJ actually modifies the code using bytecode transformation (rather than source transformation). This means it does not require source code in order to apply an aspect to a codebase. Suppose my build process already creates a nicely package application.jar file and I would rather not change the compiler from javac to ajc on the front end of that process. In that case I can use binary weaving. On the command line it would look like this:

ajc -1.5 -inpath application.jar NullParameterChecker.aj -outjar applicationWoven.jar
The jar applicationWoven.jar will contain all the classes from application.jar (modified where appropriate by the aspect) and the compiled aspect.

It is even possible to build the aspect up front into a reusable aspect library. To build the nullCheckerLibrary.jar, I can:

ajc -1.5 NullParameterChecker.aj -classpath "application.jar;c:\aspectj163\lib\aspectjrt.jar"
    -outjar nullCheckerLibrary.jar

Notes about this compiler invocation:
- application.jar is on the classpath when compiling the aspect so that NonNull can be resolved (it is referenced from the pointcut)
- there will be a warning message about the advice in the aspect not matching. This is fine, I am just building it into a library now, I'm not trying to weave it into a codebase.

The library can be used as input on an ajc call to link it into some codebase (using the aspectpath option). Or I could use Ant if I were adding the weaving of this aspect as a final step in my build process:

<project name="simple-project">

  <taskdef
    resource="org/aspectj/tools/ant/taskdefs/aspectjTaskdefs.properties">
    <classpath>
      <pathelement location="c:/aspectj162/lib/aspectjtools.jar"/>
    </classpath>
  </taskdef>

  <target name="weave" >
    <iajc inpath="application.jar" aspectpath="nullCheckerLibrary.jar"
      outjar="applicationWoven.jar" showWeaveInfo="true">
    </iajc>
  </target>
</project>

Now I can run just that step:


> ant weave

Buildfile: build.xml

weave:
  [iajc] weaveinfo Join point 'method-execution(void TestClass.printString(Message))'
  in Type 'TestClass' (TestClass.java:5) advised by before advice from 'NullParameterChecker'

BUILD SUCCESSFUL
Total time: 1 second

Running the application is the same as when it was entirely built from source, except now I specify the woven jar and the aspect library on the classpath:


java -classpath "applicationWoven.jar;nullCheckerLibrary.jar;c:\aspectj163\lib\aspectjrt.jar"
     TestClass

Exception in thread "main" java.lang.IllegalArgumentException:
  First parameter is null at void TestClass.printString(Message)

The aspect can even be linked to the codebase at load-time if I wanted to selectively launch the application with or without the checking aspect. However, load time weaving will be the subject of a later post.

I hope that has shown a little of the power of the latest AspectJ syntax, and the flexibility in the options available for introducing an aspect to an existing codebase. More soon...