Migrating b3 from Xtext 0.8 to 1.0 nightly > M6

Until there is migration documentation, my experiences of migrating the Eclipse b3 project from Xtext 0.8 (~M4) to 1.0 nightly (> M6) may be of value to others. I did this by first migrating to M6, and then using the nightly - this so I had a state to roll back to in case the nightly would fail me completely.

Migrating to 1.0 M6 version

Merge of o.e.xtext.ui.common and o.e.xtext.ui.core into o.e.xtext.ui

Almost everything that was either in ui.common, or ui.core is now in just ui - all that is needed is to change the imports, and update any dependencies to the two merged bundles with the new bundle. (See below for some additional changes).

Label Provider

The label provider changed more than just being moved to ui.

import org.eclipse.xtext.ui.common.DefaultLabelProvider;
changed to

import org.eclipse.xtext.ui.label.DefaultEObjectLabelProvider;

and my label provider is now derived from this class instead.

Proposal Provider
References to
import org.eclipse.b3.ui.AbstractBeeLangProposalProvider;
changed to

import org.eclipse.b3.ui.contentassist.AbstractBeeLangProposalProvider;

New Structure
After running the mwe workflow, I got 4 new packages in my dsl project with the suffix ".ui". These packages contained the corresponding classes found in the existing packages without the ".ui" suffix. I moved/merged my code over to the new packages and deleted the old packages.

UI Module change
The UIModule for my DSL had to change to the following signature and constructor:

public class BeeLangUiModule extends org.eclipse.b3.ui.AbstractBeeLangUiModule {

public BeeLangUiModule(AbstractUIPlugin plugin) {

super(plugin);

}

The new structure has the UIModule in a new package (with ."ui" suffix), and the Activator (also in a new package "...ui.internal") uses the UIModule in this new package.

Mwe workflow change

I had to replace the JavaScopingFragment with ImportURIScopingFragment:

fragment class="org.eclipse.xtext.generator.scoping.JavaScopingFragment"

fragment class="org.eclipse.xtext.generator.scoping.ImportURIScopingFragment"

since the JavaScopingFragment no longer exist. Don't know if the ImportURIScopingFragment is what I want, but I had to pick one.

Converting Java Strings

Strings.convertFromJavaString now has an extra boolean argument useUnicode which should be set to true to process \\uXXXX escapes. (I did set it to true). I use this method in some terminal converters.

Changes in plugin.xml

A manual merge of all changes in plugin.xml_gen to my plugin.xml (basically changes related to use of "ui" in package names) was required.

Manifest change

Manifest file needed update as the activator is in a different package:

Bundle-Activator: org.eclipse.b3.ui.internal.BeeLangActivator

(using "ui" in the package name)

Migrating to latest nightly

The 42 Easter Egg

The method

protected void configureImportantInformation(IEditStrategyAcceptor acceptor)

has been dropped from DefaultAutoEditStrategy. It was only there to block an easteregg (typing 42 displays a funny comment about 'the meaning of life' - but the easter egg and method seems to both be gone in the nightly).

Serialization

I have not done much with b3 serialization yet, so required changes were small. I needed to add a single method:

public class BeeLangGrammarSerialization implements ITransientValueService

needs an implementation of the method

public boolean isCheckElementsIndividually(EObject owner, EStructuralFeature feature)

I added one that returns false, which hopefully is the same as the default.

Guice from Orbit

com.google.guice (from itemis) changed to com.google.inject from orbit (dependencies changed, and my launch configurations needed to be changed).

Open Issues

I had attached commands to the popup menu that should appear (and they did in 0.8 M4) over the editor's outline. But this stopped working. I am waiting on some wisdom from the Xtext gurus on this...
This was a temporary issue with plugin.xml changes not taking effect. After a restart and clean build it now works just like before.

Syntax highlight has changed, and I am trying to figure out how it works now... It stopped working because I forgot to move things over to the new UIModule (as described above).

Summary

All in all, the migration was quite painless. Knowing that changes were to take place in several of the services, I only have minimal implementations (the default, or just a few lines to fix something glaring) in many places (in wait for the 1.0 release and new documentation). If you have a lot of code and using everything "to the hilt" in 0.7.2, you may want to wait for the official release and the documentation.

I will update this article as I find more things that needs to be changed, or if I changed something in error.

Buckminster 3.6 New & Noteworthy

The Helios release of Buckminster has the following new and noteworthy features available in 3.6M7

• Support for Git - uses, updates, or clones git repository as needed
• Headless JUnit and EclEmma (code coverage) support
• Comprehensive documentation available - introduction, examples, and reference. Download PDF, 250 pages, includes descriptions of the new features described here.
• Graphical dependency visualizer - resolutions can be viewed and navigated/filtered with a Zest based viewer
• Much improved target platform support - using new features in PDE to automatically manage/materialize target platform config
• Provisioning and management of API baseline
• New EMF based editors for MSPEC and RMAP - much easier to use than editing XML
• Reader type for Project Set (.psf) files - makes it easy to integrate or migrate projects that are using .psf files to describe where the source is
• p2 repository size reduction to 1/3 using improved pack200 support
• OmniVersion support - the support for non OSGi version has been changed to use the p2 OmniVersion implementation for increased flexibility - backwards compatible with Buckminster version-type, version scheme used in earlier versions.
• Qualifier generator using Build Identifier - use a property to control the content of a version qualifier
• LDAP style filters on RMAP providers, CQUERY advisors, and MSPEC nodes - makes it possible to parameterize more things, reduces the need for multiple slightly different copies of these files.
• Smart version range generation for feature 'includes' - heuristics result in natural choices
• Support for category.xml files - the new PDE mechanism for categorizing result in p2 repository is supported
• Headless 'install JRE' support
• Better defaults often renders the MSEPC unnecessary - automatic materialization to Target Platform for binaries often removes the need to use a MSPEC.
• Using new p2 API, p2 'pure' reader, and using separate p2 agent - reduces risk of contamination of the running instance's p2 data.

Eclipse Build Systems in Perspective

It is easy to get confused over the "build system options" available when developing with Eclipse - there is JDT and ANT, PDE (interactive), and PDE/Build (headless) - to start with the "classics". Then there is Athena, an elaboration on the also classic 'releng' system used to build Eclipse itself, then the newer Buckminster. The two latest additions to the family of build related projects at Eclipse are Eclipse b3 (2009), and Tycho (proposal 2010).

With this blog post I want to put eclipse build technology in perspective.

So, what does all these technologies do?

The classics

JDT - builds java source, interactively, under control of the project and workspace preferences.

PDE - builds OSGi components and the Eclipse specific extensions (features, products, etc.) under the control of PDE preferences. PDE consists of both the interactive parts (incremental builds, export commands, dialogs etc.), and the headless logic that performs the actual work. This logic is also made available as ANT tasks.

PDE/Build - generates ANT scripts that are then executed in headless fashion to build. The generated ANT scripts makes use of the same ANT tasks used by the interactive PDE.

If we stop there for a moment - this was the level of support for building provided by Eclipse a few years back. Although PDE does a very good job of building things interactively, it is fair to say that the headless PDE/Build has been the source of much pain and frustration. To complete the picture; at this time there was also the "releng" system in use at Eclipse, which only ran on the Eclipse servers.

Improving on PDE

While PDE/Build has more or less remained unchanged there has been many different approaches to how to provide good support for headless builds. Some took the scripting route - improving on releng base builder (Athena), some wrote their own scripts, other started modeling PDE like this project in PDE incubator, and some wrote better/easier to use script generators like pluginbuilder.

Eclipse Buckminster

When the Eclipse Buckminster project was introduced in 2005 it focused on additional things — when using generated scripts and technologies external to the IDE there are often issues caused by differences in how things are built i.e. it works just fine in the IDE but breaks in the build. An important goal for Buckminster was (and still is) to provide exactly the same build interactively as on the servers. This means that Buckminster has a tight integration with the builders running in a workspace made available in an efficient packaging for headless execution. Another important design decision was to use the existing information (i.e. the meta data used by PDE in the IDE) without generation of scripts and without roundtrip engineering, instead using advice/decoration to modify discovered metadata when the original information is not enough.

In addition to the important philosophical difference, Buckminster also provides unique support for materialization of a workspace, automatic provisioning of a target platform, running JUnit tests, EclEmma code coverage, Hudson integration, and much more.

Note that when Buckminster builds PDE related material (Buckminster can build other things as well), it calls on the same PDE logic that is used when building with interactive PDE.

Tycho

Tycho is a set of Maven plugins that provide building of OSGi and Eclipse related components (features, plugins, RCP products, etc.) and is an alternative to PDE suitable for those that have a Maven centric setup. Tycho does not use the original PDE logic.

Eclipse b3

The b3 project is about making it easy to work with build systems — discovering and modernizing or integrating existing build systems should be just as easy as building meta data rich components interactively or in continuos integration fashion. Eclipse b3 starts at the very other end of the spectrum than "which compiler to use" or which meta data dialect is used to describe components.

Eclipse b3 does this by providing EMF based models for build (i.e. components, their relationships, versions, types, etc.), expressions (i.e. the processing in form of tasks, builders, compilers, etc.), and p2 (with support for aggregation, re-categorization, mirroring, maven meta data publishing, and more), as well as a concrete syntax in form of a DSL implemented with Xtext that provides a rich text editing environment, and an evaluator that makes it possible to run b3 scripts.

As an example - there is nothing in Eclipse b3 that restrict it to using the original PDE logic to build the PDE related artifacts, it can just as well make use of Tycho's alternate way of building the same things.

The very first way Eclipse b3 will be building things is to use Buckminster as the execution engine. A small, and easy to understand b3 script will drive the entire build — combining the ease of use in the b3 DSL with the proven stable builds provided by Buckminster.

Eclipse b3 - a success at Eclipsecon

As you may have seen, the Eclipse b3 project is about creating a new generation of Eclipse technology to simplify software build and assembly. There has been lots of activity since the project was created, and there was much interest in b3 at Eclipsecon - so, here is a status update.

In case you did not know; a seminar on b3 was held at Eclipse Summit Europe 09 where the initial ideas were presented and discussed. A lot of new ideas about how people want to work with builds were generated, and there was lots of positive feedback on the original ideas. As always, some darlings were also completely killed in the process (the message that XPath queries are anything but easy to understand was received).

The feedback told us that these things are important:

• Ease of use
• Based on Modeling
• Debuggable
• Flexible / Extensible

Armed with all that input the time between ESE and Eclipsecon 2010 was devoted to developing a version of b3 that demonstrates the ideas, with a focus on Ease of Use, while not sacrificing flexibility or capability of dealing with real world complexities when building. At Eclipsecon we reached the first milestone of b3 consisting of:

• A build ecore model
• A process/expression ecore model
• A p2 ecore model with aggregation and rewrite support
• A concrete syntax implemented with XText (i.e. a feature rich eclipse editor and much more).
• An evaluator (i.e. making it possible to run the build scripts).
• Documentation of the concrete syntax
• A website with links to all b3 related information (documentation, for developers, etc).

The b3 interest at Eclipsecon 2010 was huge - the room was packed, not everyone could get in, and of those that did, close to 80% liked the presentation (i.e. voted +1). To the two individuals who voted -1; I am hoping you put some comments on your votes so I know what you did not like (it simply has to be 'lack of chairs' :)).

For the next milestone (around Helios release) we are adding concrete things to b3:

• Use a b3 script to drive a buckminster build
• Publish b3 build units to a p2 repository - i.e. author installable units

As always - love to hear your questions and comments. I have already received quite a few questions regarding the relationship between b3 and other eclipse related build technologies (buckminster, athena, PDE build, and now the Tycho proposal), and that will be the topic of my next blog post about b3.

"Chester the test-data molester" comes to town

Introduction

"Chester the test-data molester" is a http test server that delivers various error scenarios consistently.

In the p2 project we have the need to test various communication error scenarios, such as when a web server reports illegal last modified dates, reports the wrong file size, endlessly redirects, hits internal server errors, when connection is made via a "hotel style" payment service etc. This was a real pain to set up in an ad-hoc manner, so I decided to create a small equinox based http testserver that is now ready for use.

The testserver has already been invaluable in finding trasport related issues. I thought it was worth writing this short introduction as it may help others test and fix error reporting issues in RCP apps with custom p2 user interface, for those that need to be able to replicate p2 problems where for security/practical reasons it is not possible to access the real repositories, as well as for those that simply need a http server that can create various error scenarios consistently.

How to get it

The testserver resides in the p2 CVS repository - org.eclipse.equinox.p2.testserver. To use it you need to check it out, as well as the org.eclipse.equinox.http bundle. (If you use the p2 team project sets in the p2 releng project you will get everything you need). There is a launch configuration in the testserver project that starts the testserver on "localhost:8080". (You can change the port in the launch configuration if you want).

Basic Services

The testserver starts some basic testing services on the following paths:

• /timeout[/anything] - will wait 10 minutes and then produce no response
• /status/nnn[/anything] - returns html content with the http response status code set to nnn, e.g. /status/500 for an internal server error
• /redirect/nnn[/location] - redirects nnn times and then redirects to location (a path on testserver). If no location given, a html page with a message is generated as the final redirect. Examples:
  • /redirect/3/status/500 - redirects 3 times and then generates a 500 - internal error.
  • /redirect/3 - redirects 3 times and produces a message
  • /redirect/30 - redirects 30 times, and will trigger "too many redirects error" in most configurations
• /never[/anything] - has basic authentication turned on, but will not accept any username/password as valid.

Content Delivery

The testserver also has content available - there is an index.html in the project, as well as some p2 repository test data. The testserver has also "mounted" the eclipse updates 3.4 repository on different paths with different types of wrappers/"molestors" in place.
This set of paths goes to the testserver bundle's web content - i.e. index.html and some p2 test data:

• /public/... - normal access
• /private/... - same as public but requires login with "Aladdin" and password "open sesame"
• /truncated/... - truncates files by delivering less content than stated in length e.g. /truncated/index.html
• /molested/... - returns garbage instead of real content in the later part of the file e.g. /molested/index.html
• /decelerate/... - delivers content chopped up in small packets with delay, e.c. /decelerate/index.html (interesting to watch in firefox which delivers content as it comes in).

This set of paths has mounted http://download.eclipse.org/eclipse/updates/3.4 with various "molestors":

• /proxy/private/... - requires login with "Aladdin" and password "open sesame"
• /proxy/public/... - unmolested access (useful in redirects)
• /proxy/decelerate/... - chops up content and delays delivery
• /proxy/decelerate2/.... - chops up content and delays the last 20% of the delivery
• /proxy/truncated/... - truncates all files
• /proxy/molested/... - generates gibberish for later part of all files
• /proxy/modified/... - delivers various errors in "last modified" (see below)
  
  • .../zero/... - all times are returned as 0
  • .../old/... - all times are very old
  • .../now/... - all times are the same as the request time
  • .../future/... - all times are in the future (which is illegal in HTTP)
  • .../bad/... - the time is not a date at all - the client should throw an error
• /proxy/length/... - delivers various content length errors (see below)
  • .../zero/... - length is reported as 0 (but all content written to stream)
  • .../less/... - less than the correct size is reported (all content written)
  • .../more/... - double the correct size is reported (but only available content is written)

Get in touch

I had fun writing this - I mean, how often do you get to write classes called "Molestor" :). I hope you find the testserver useful, and that if you would like it to perform other forms of content maiming and mutilation that you contribute by submitting patches to the "p2" project marking issues with the text [testserver].

Progress Monitor Patterns

Are you working with Progress Monitors, and wish you did not have too?

I recently got to investigate issues in p2 why the progress bar did not move while it was very clear that things where happening as subtasks changed the text. Looking into what was going on made me find a new Progress Monitor anti pattern.

The problem is how to handle a case where you need to do something like this:

for (int i = 0; i < candidates.length; i++)       if (loadCandidate1(i))         break;

Where a call to loadCandidate is potentially a long running task. The first loaded candidate means we are done.

Antipattern

Here is an implementation of the above example using the antipattern - i.e. "don't do this:

public void antiPattern(IProgressMonitor monitor) {     SubMonitor sub = SubMonitor.convert(monitor, candidates.length);     for (int i = 0; i < candidates.length; i++) {       if (loadCandidate(i, sub.newChild(1)))         break;     }   }   public boolean loadCandidate(int index, IProgressMonitor monitor) {     SubMonitor sub = SubMonitor.convert(monitor, 1000);     try {       for (int i = 0; i < 1000; i++)         sub.worked(1);       // ...       return true;     } finally {       monitor.done();     }     return false;   }

Well, what is wrong with this, you may ask… Well, the length of the progress bar will be divided into as many slots as there are candidates, and if the first candidate succeeds and uses its 1000 ticks, and the remaining candidates are never considered, we will end up reporting the 1000 ticks on a fraction of the overall progress bar. This means that for 10 candidates, you will see the progress-bar slowly go to about 10% of the overall length, to suddenly jump to 100%.

Good pattern

Here is the good pattern that makes use of the full progress bar:

public void goodPattern(IProgressMonitor monitor) {     SubMonitor sub = SubMonitor.convert(monitor, 1000);     for (int i = 0; i < candidates.length; i++) {       sub.setWorkRemaining(1000);       if (goodLoadCandidate(i, sub.newChild(1000)))         break;     }   }   public boolean goodLoadCandidate(int index, IProgressMonitor monitor) {     SubMonitor sub = SubMonitor.convert(monitor, 1000);     sub.beginTask(null, 1000);     try {       // … code that loads       for (int i = 0; i < 1000; i++)         sub.worked(1);       return true;     } finally {       // ignore errors     }     return false;   }

Notice how “setWorkRemaining” is called each time in the loop. This reallocates the remaining ticks, but there is no need to compute how many that actually remains, the child allocation of 1000 ticks will always give the child 1000 ticks to report, even if there were not enough ticks left in the parent. All the scaling is performed by the SubMonitor, so you don’t have to worry about it.

Now, let’s say that the routine you are calling do need to perform a bit of work even if it does not need all of its ticks. Don’t worry, that will work too as long as it is a small portion of the allocation. If you risk consuming a larger part, you may be better off doing a true partitioning of the progress-bar as shown in the next section.

Good pattern - regular loop

Here is the good pattern for a regular loop where each iteration does consume ticks

public void goodLoopPattern(IProgressMonitor monitor) {     SubMonitor sub = SubMonitor.convert(monitor, candidates.length*100);     for (int i = 0; i < candidates.length; i++) {       if (goodLoopCandidate(i, sub.newChild(100)))         break;     }   }   public boolean goodLoopCandidate(int index, IProgressMonitor monitor) {     SubMonitor sub = SubMonitor.convert(monitor, 1000);     sub.beginTask(null, 1000);     try {       // … code that loads       for (int i = 0; i < 1000; i++)         sub.worked(1);       return true;     } finally {       // ignore errors     }     return false;   }

Here I simply use SubMonitor’s newChild, this works without calling “done” because the next call to newChild (or “done” for that matter) will consume the ticks allocated for the child.

Rules of Thumb

1. Use SubMonitor
2. Always begin by converting the monitor you get to a SubMonitor
3. Use newChild(n) to create a sub monitor to pass to methods that take an IProgressMonitor as argument.
4. Never call “done” - but document that the caller must do so unless they used a SubMonitor

Presentation in Stockholm Jan 27 - p2 provisioning your world

Everyone in Stockholm with an interest in Eclipse and provisioning with p2 are welcome to the Scandic Anglais Hotel at Humlegården 23, Stockholm, Sweden on January 27 - 6 pm.

A p2 presentation and demo will be presented to the Swedish Eclipse user group. If you are interested, you should register (it is free) here.