Eclipse Build Systems in Perspective

It is easy to get confused over the "build system options" available when developing with Eclipse - there is JDT and ANT, PDE (interactive), and PDE/Build (headless) - to start with the "classics". Then there is Athena, an elaboration on the also classic 'releng' system used to build Eclipse itself, then the newer Buckminster. The two latest additions to the family of build related projects at Eclipse are Eclipse b3 (2009), and Tycho (proposal 2010).

With this blog post I want to put eclipse build technology in perspective.

So, what does all these technologies do?

The classics

JDT - builds java source, interactively, under control of the project and workspace preferences.

PDE - builds OSGi components and the Eclipse specific extensions (features, products, etc.) under the control of PDE preferences. PDE consists of both the interactive parts (incremental builds, export commands, dialogs etc.), and the headless logic that performs the actual work. This logic is also made available as ANT tasks.

PDE/Build - generates ANT scripts that are then executed in headless fashion to build. The generated ANT scripts makes use of the same ANT tasks used by the interactive PDE.

If we stop there for a moment - this was the level of support for building provided by Eclipse a few years back. Although PDE does a very good job of building things interactively, it is fair to say that the headless PDE/Build has been the source of much pain and frustration. To complete the picture; at this time there was also the "releng" system in use at Eclipse, which only ran on the Eclipse servers.

Improving on PDE

While PDE/Build has more or less remained unchanged there has been many different approaches to how to provide good support for headless builds. Some took the scripting route - improving on releng base builder (Athena), some wrote their own scripts, other started modeling PDE like this project in PDE incubator, and some wrote better/easier to use script generators like pluginbuilder.

Eclipse Buckminster

When the Eclipse Buckminster project was introduced in 2005 it focused on additional things — when using generated scripts and technologies external to the IDE there are often issues caused by differences in how things are built i.e. it works just fine in the IDE but breaks in the build. An important goal for Buckminster was (and still is) to provide exactly the same build interactively as on the servers. This means that Buckminster has a tight integration with the builders running in a workspace made available in an efficient packaging for headless execution. Another important design decision was to use the existing information (i.e. the meta data used by PDE in the IDE) without generation of scripts and without roundtrip engineering, instead using advice/decoration to modify discovered metadata when the original information is not enough.

In addition to the important philosophical difference, Buckminster also provides unique support for materialization of a workspace, automatic provisioning of a target platform, running JUnit tests, EclEmma code coverage, Hudson integration, and much more.

Note that when Buckminster builds PDE related material (Buckminster can build other things as well), it calls on the same PDE logic that is used when building with interactive PDE.

Tycho

Tycho is a set of Maven plugins that provide building of OSGi and Eclipse related components (features, plugins, RCP products, etc.) and is an alternative to PDE suitable for those that have a Maven centric setup. Tycho does not use the original PDE logic.

Eclipse b3

The b3 project is about making it easy to work with build systems — discovering and modernizing or integrating existing build systems should be just as easy as building meta data rich components interactively or in continuos integration fashion. Eclipse b3 starts at the very other end of the spectrum than "which compiler to use" or which meta data dialect is used to describe components.

Eclipse b3 does this by providing EMF based models for build (i.e. components, their relationships, versions, types, etc.), expressions (i.e. the processing in form of tasks, builders, compilers, etc.), and p2 (with support for aggregation, re-categorization, mirroring, maven meta data publishing, and more), as well as a concrete syntax in form of a DSL implemented with Xtext that provides a rich text editing environment, and an evaluator that makes it possible to run b3 scripts.

As an example - there is nothing in Eclipse b3 that restrict it to using the original PDE logic to build the PDE related artifacts, it can just as well make use of Tycho's alternate way of building the same things.

The very first way Eclipse b3 will be building things is to use Buckminster as the execution engine. A small, and easy to understand b3 script will drive the entire build — combining the ease of use in the b3 DSL with the proven stable builds provided by Buckminster.

Eclipse b3 - a success at Eclipsecon

As you may have seen, the Eclipse b3 project is about creating a new generation of Eclipse technology to simplify software build and assembly. There has been lots of activity since the project was created, and there was much interest in b3 at Eclipsecon - so, here is a status update.

In case you did not know; a seminar on b3 was held at Eclipse Summit Europe 09 where the initial ideas were presented and discussed. A lot of new ideas about how people want to work with builds were generated, and there was lots of positive feedback on the original ideas. As always, some darlings were also completely killed in the process (the message that XPath queries are anything but easy to understand was received).

The feedback told us that these things are important:

• Ease of use
• Based on Modeling
• Debuggable
• Flexible / Extensible

Armed with all that input the time between ESE and Eclipsecon 2010 was devoted to developing a version of b3 that demonstrates the ideas, with a focus on Ease of Use, while not sacrificing flexibility or capability of dealing with real world complexities when building. At Eclipsecon we reached the first milestone of b3 consisting of:

• A build ecore model
• A process/expression ecore model
• A p2 ecore model with aggregation and rewrite support
• A concrete syntax implemented with XText (i.e. a feature rich eclipse editor and much more).
• An evaluator (i.e. making it possible to run the build scripts).
• Documentation of the concrete syntax
• A website with links to all b3 related information (documentation, for developers, etc).

The b3 interest at Eclipsecon 2010 was huge - the room was packed, not everyone could get in, and of those that did, close to 80% liked the presentation (i.e. voted +1). To the two individuals who voted -1; I am hoping you put some comments on your votes so I know what you did not like (it simply has to be 'lack of chairs' :)).

For the next milestone (around Helios release) we are adding concrete things to b3:

• Use a b3 script to drive a buckminster build
• Publish b3 build units to a p2 repository - i.e. author installable units

As always - love to hear your questions and comments. I have already received quite a few questions regarding the relationship between b3 and other eclipse related build technologies (buckminster, athena, PDE build, and now the Tycho proposal), and that will be the topic of my next blog post about b3.

"Chester the test-data molester" comes to town

Introduction

"Chester the test-data molester" is a http test server that delivers various error scenarios consistently.

In the p2 project we have the need to test various communication error scenarios, such as when a web server reports illegal last modified dates, reports the wrong file size, endlessly redirects, hits internal server errors, when connection is made via a "hotel style" payment service etc. This was a real pain to set up in an ad-hoc manner, so I decided to create a small equinox based http testserver that is now ready for use.

The testserver has already been invaluable in finding trasport related issues. I thought it was worth writing this short introduction as it may help others test and fix error reporting issues in RCP apps with custom p2 user interface, for those that need to be able to replicate p2 problems where for security/practical reasons it is not possible to access the real repositories, as well as for those that simply need a http server that can create various error scenarios consistently.

How to get it

The testserver resides in the p2 CVS repository - org.eclipse.equinox.p2.testserver. To use it you need to check it out, as well as the org.eclipse.equinox.http bundle. (If you use the p2 team project sets in the p2 releng project you will get everything you need). There is a launch configuration in the testserver project that starts the testserver on "localhost:8080". (You can change the port in the launch configuration if you want).

Basic Services

The testserver starts some basic testing services on the following paths:

• /timeout[/anything] - will wait 10 minutes and then produce no response
• /status/nnn[/anything] - returns html content with the http response status code set to nnn, e.g. /status/500 for an internal server error
• /redirect/nnn[/location] - redirects nnn times and then redirects to location (a path on testserver). If no location given, a html page with a message is generated as the final redirect. Examples:
  • /redirect/3/status/500 - redirects 3 times and then generates a 500 - internal error.
  • /redirect/3 - redirects 3 times and produces a message
  • /redirect/30 - redirects 30 times, and will trigger "too many redirects error" in most configurations
• /never[/anything] - has basic authentication turned on, but will not accept any username/password as valid.

Content Delivery

The testserver also has content available - there is an index.html in the project, as well as some p2 repository test data. The testserver has also "mounted" the eclipse updates 3.4 repository on different paths with different types of wrappers/"molestors" in place.
This set of paths goes to the testserver bundle's web content - i.e. index.html and some p2 test data:

• /public/... - normal access
• /private/... - same as public but requires login with "Aladdin" and password "open sesame"
• /truncated/... - truncates files by delivering less content than stated in length e.g. /truncated/index.html
• /molested/... - returns garbage instead of real content in the later part of the file e.g. /molested/index.html
• /decelerate/... - delivers content chopped up in small packets with delay, e.c. /decelerate/index.html (interesting to watch in firefox which delivers content as it comes in).

This set of paths has mounted http://download.eclipse.org/eclipse/updates/3.4 with various "molestors":

• /proxy/private/... - requires login with "Aladdin" and password "open sesame"
• /proxy/public/... - unmolested access (useful in redirects)
• /proxy/decelerate/... - chops up content and delays delivery
• /proxy/decelerate2/.... - chops up content and delays the last 20% of the delivery
• /proxy/truncated/... - truncates all files
• /proxy/molested/... - generates gibberish for later part of all files
• /proxy/modified/... - delivers various errors in "last modified" (see below)
  
  • .../zero/... - all times are returned as 0
  • .../old/... - all times are very old
  • .../now/... - all times are the same as the request time
  • .../future/... - all times are in the future (which is illegal in HTTP)
  • .../bad/... - the time is not a date at all - the client should throw an error
• /proxy/length/... - delivers various content length errors (see below)
  • .../zero/... - length is reported as 0 (but all content written to stream)
  • .../less/... - less than the correct size is reported (all content written)
  • .../more/... - double the correct size is reported (but only available content is written)

Get in touch

I had fun writing this - I mean, how often do you get to write classes called "Molestor" :). I hope you find the testserver useful, and that if you would like it to perform other forms of content maiming and mutilation that you contribute by submitting patches to the "p2" project marking issues with the text [testserver].

Progress Monitor Patterns

Are you working with Progress Monitors, and wish you did not have too?

I recently got to investigate issues in p2 why the progress bar did not move while it was very clear that things where happening as subtasks changed the text. Looking into what was going on made me find a new Progress Monitor anti pattern.

The problem is how to handle a case where you need to do something like this:

for (int i = 0; i < candidates.length; i++)       if (loadCandidate1(i))         break;

Where a call to loadCandidate is potentially a long running task. The first loaded candidate means we are done.

Antipattern

Here is an implementation of the above example using the antipattern - i.e. "don't do this:

public void antiPattern(IProgressMonitor monitor) {     SubMonitor sub = SubMonitor.convert(monitor, candidates.length);     for (int i = 0; i < candidates.length; i++) {       if (loadCandidate(i, sub.newChild(1)))         break;     }   }   public boolean loadCandidate(int index, IProgressMonitor monitor) {     SubMonitor sub = SubMonitor.convert(monitor, 1000);     try {       for (int i = 0; i < 1000; i++)         sub.worked(1);       // ...       return true;     } finally {       monitor.done();     }     return false;   }

Well, what is wrong with this, you may ask… Well, the length of the progress bar will be divided into as many slots as there are candidates, and if the first candidate succeeds and uses its 1000 ticks, and the remaining candidates are never considered, we will end up reporting the 1000 ticks on a fraction of the overall progress bar. This means that for 10 candidates, you will see the progress-bar slowly go to about 10% of the overall length, to suddenly jump to 100%.

Good pattern

Here is the good pattern that makes use of the full progress bar:

public void goodPattern(IProgressMonitor monitor) {     SubMonitor sub = SubMonitor.convert(monitor, 1000);     for (int i = 0; i < candidates.length; i++) {       sub.setWorkRemaining(1000);       if (goodLoadCandidate(i, sub.newChild(1000)))         break;     }   }   public boolean goodLoadCandidate(int index, IProgressMonitor monitor) {     SubMonitor sub = SubMonitor.convert(monitor, 1000);     sub.beginTask(null, 1000);     try {       // … code that loads       for (int i = 0; i < 1000; i++)         sub.worked(1);       return true;     } finally {       // ignore errors     }     return false;   }

Notice how “setWorkRemaining” is called each time in the loop. This reallocates the remaining ticks, but there is no need to compute how many that actually remains, the child allocation of 1000 ticks will always give the child 1000 ticks to report, even if there were not enough ticks left in the parent. All the scaling is performed by the SubMonitor, so you don’t have to worry about it.

Now, let’s say that the routine you are calling do need to perform a bit of work even if it does not need all of its ticks. Don’t worry, that will work too as long as it is a small portion of the allocation. If you risk consuming a larger part, you may be better off doing a true partitioning of the progress-bar as shown in the next section.

Good pattern - regular loop

Here is the good pattern for a regular loop where each iteration does consume ticks

public void goodLoopPattern(IProgressMonitor monitor) {     SubMonitor sub = SubMonitor.convert(monitor, candidates.length*100);     for (int i = 0; i < candidates.length; i++) {       if (goodLoopCandidate(i, sub.newChild(100)))         break;     }   }   public boolean goodLoopCandidate(int index, IProgressMonitor monitor) {     SubMonitor sub = SubMonitor.convert(monitor, 1000);     sub.beginTask(null, 1000);     try {       // … code that loads       for (int i = 0; i < 1000; i++)         sub.worked(1);       return true;     } finally {       // ignore errors     }     return false;   }

Here I simply use SubMonitor’s newChild, this works without calling “done” because the next call to newChild (or “done” for that matter) will consume the ticks allocated for the child.

Rules of Thumb

1. Use SubMonitor
2. Always begin by converting the monitor you get to a SubMonitor
3. Use newChild(n) to create a sub monitor to pass to methods that take an IProgressMonitor as argument.
4. Never call “done” - but document that the caller must do so unless they used a SubMonitor

Presentation in Stockholm Jan 27 - p2 provisioning your world

Everyone in Stockholm with an interest in Eclipse and provisioning with p2 are welcome to the Scandic Anglais Hotel at Humlegården 23, Stockholm, Sweden on January 27 - 6 pm.

A p2 presentation and demo will be presented to the Swedish Eclipse user group. If you are interested, you should register (it is free) here.

How to include RSS feeds into your components using OPML

It is really easy to include RSS feeds and links in your components - all you have to do is to place a buckminster.opml file in the root of your component (or tweak the properties if you want to call it something else, and place it elsewhere).

The OPML file itself has a very simple XML syntax. You start of with the usual xml declaration and declaring that you are using OPML 2.0.

<?xml version="1.0" encoding="ISO-8859-1"?>
<opml version="2.0">

This is followed by a head declaration where some information about the content of the OPML file is kept.

<head>
<title>Component information for org.demo.exampleComponent</title>
<dateCreated>Mon, 14 Apr 2008 14:18:51 GMT</dateCreated>
<ownerName>Your name, or name of project</ownerName>
<ownerEmail>contact.us@somewhere.com</ownerEmail>
</head>

And after the head comes the body part that consists of a sequence of outline elements - they can be nested if you want to use "subfolders".

<body>
<!-- outline elements --!>
</body>

A regular link (non feed) is expressed in an outline element like this:

<outline text="Cloudsmith" description="Cloudsmith's site" url="http://www.cloudsmith.com" language="unknown" title="Cloudsmith" type="link" />

And a feed is just as simple:

<outline text="My Feed" description="This is my feed" htmlUrl="http://www.somewhere.org" language="unknown" title="My Example Feed" type="rss" version="RSS2" xmlUrl="http://feeds.somewhere.org/examplefeed"/>

In both of the above examples - the "text" attribute is the label typically used in the RSS reader's bookmarks. Some OPML viewers use the title instead, and the reader may or may not show the description. Regular links (type="link") should use the "url" attribute for the link, and a feed ("type="rss") should use the "xmlUrl" attribute for the feed, but also add a link to a human readable web page - this is done by using the "htmlUrl" attribute, and it is often a link to the page where it is possible to subscribe to the feed, or read its content online.

In the feed example above, the feed type is "rss" - and the version "RSS2". A feed should always have the type set to "rss" (including atom feeds, but for atom feeds, the OPML spec is vague - I use "atom" as the version and it works just fine). These tags are mainly for a processor of the OPML itself, a feed reader will look at the actual feed to determine its type anyway. You can read more about the OPML 2.0 standard here.

And then finally, a subfolder is very simple to create:

<outline text="A Subfolder">
<outline .... />
<outline .... />
</outline>

Even if the creation of a component's OPML (at least for now) is done via manual editing of XML, I hope the examples above show that it is really quite easy.

Feed your Buckminster

Did you know that Buckminster can deliver RSS feeds and links that appear in the Eclipse UI?

We wanted to make it possible to share content that relates to a component, such as a links to wiki, bugzilla, and documentation, and to RSS feeds like a version feed, open and closed issues in bugzilla, examples feed, why not a "checkin feed", or why not a feed with the lastest available plugins...

We selected the OPML 2.0 XML definition for Outline Processing - essentially describing a structure of links and feeds - or bookmarks if you like. And we decided to include this into the Buckminster model. This means that a component type can auto generate meta data from components that have extra "community" information and make this available in the form of OPML. Buckminster recognizes that a component can have an OPML file embedded in the component (by default it is called buckminster.opml and is placed in the root of the component, but it is possible to have it in some other directory and give it a different name by setting properties).

Buckminster includes two new views that makes use of the new OPML feature, a Component Explorer that shows all components known to Buckminster, and a Component Outline that follows the current selection showing the related component information.

Here is a screenshot of the Component Outline, which also shows Buckminster's integration with the RSS Owl Reader.

If you want to try out the RSS OWL integration - there is a preview available at our update site http://dev.cloudsmith.com/updates3_4 - it is based on RSS OWL 2.0 M7 but with some patches applied.

And here is a screenshot of RSS OWL showing the example feed:

Clicking on the links in RSS OWL will start buckminster materialization, or open the cquery editor!