The b3 aggregator

The Eclipse b3 Aggregator is based on and part of the Eclipse b3 project. Eclipse b3 provides a versatile and adaptable framework supporting build, assembly and deployment processes. It supports a rich set of use cases. One of those - the aggregation of repositories - is the focus of the b3 Aggregator tool.

The Eclipse b3 Aggregator combines repositories from various sources into a new aggregated p2 repository. It can also be configured to produce a hybrid p2/Maven2 repository. There are many situations where using aggregated repositories is a good solution, here are some examples:

1. Projects want to provide convenient access to their products - Installation instructions requiring the user to visit several repos for a complete install are not uncommon. An aggregated repo for all those locations provides a convenient one-stop-shop strategy. The aggregation can perform mirroring of all consumed p2 repos or selectively provide indirection via a composite repo.
   
2. Organizations or teams want control over internally used components - It may be necessary to have gated access to relevant/"blessed" p2 repos where an organizational "healthcheck" has been performed prior to internal distribution. Furthermore, internally used aggregated repos can provide a common basis for all organizational users (i.e. for both IDE distribution as well as for content used when building internal applications).
3. Increase repository availability - by aggregating and mirroring what is used from multiple update sites into internally controlled servers.
4. Distributed Development Support - an overall product repository is produced by aggregating contributions from multiple teams.
5. Owners of a p2 repo for a given project may not be in position to host all required or recommended components due to licensing issues - Buckminster's SVN support can serve as an example here, as it requires components available in the main Eclipse p2 repo as well as third-party components. Hence users have to visit several repos for a complete install.
   

The b3 Aggregator is focused on supporting these specific requirements, and it plays an important role in the full scope of the b3 project. The Aggregator is however used in scenarios outside of the traditional "build domain" and this has been reflected in the user interface which does not delve into the details of "building" and should therefore be easy to use by non build experts.

Functional Overview

The b3 Aggregator performs aggregation and validation of repositories. The input to the aggregator engine (that tells it what to do) is a b3aggr EMF model. Such a model is most conveniently created by using the b3 Aggregator editor. This editor provides both editing and interactive execution of aggregation commands. The editor is based on a standard EMF "tree and properties view" style editor where nodes are added and removed to form a tree, and the details of nodes are edited in a separate properties view. Once a b3aggr model has been created it is possible to use the command line / headless aggregator to perform aggregation (and other related commands). (Note that since the b3aggr is "just and EMF model", it can be produced via EMF APIs, transformation tools, etc. and thus support advanced use cases).

The model mainly consists of Contributions; specifications of what to include from different repositories, and Validation Repositories; repositories that are used when validating, but that are not included in the produced aggregation (i.e. they are not copied). The model also contains specification of various processing rules (exclusions, transformation of names, etc.), and specification of Contacts; individuals/mailing-lists to inform when processing fails.

Here are some of the important features supported by the b3 Aggregator in Eclipse 3.6M7:

• p2 and maven2 support — the aggregator can aggregate from and to both p2 and maven2 repositories.
• Maven2 name mapping support — names in the p2 domain are automatically mapped to maven2 names using built in rules. Custom rules are also supported.
• Mirroring — artifacts from repositories are mirrored/downloaded/copied to a single location
• Selective mirroring — an aggregation can produce an aggregation consisting of a mix of references to repositories and mirrored repositories.
• Cherry picking — it is possible to pick individual items when the entire content of a repository is not wanted. Detailed picking is supported as well as picking transitive closures like a product, or a category to get everything it contains/requires.
• Pruning — it is possible to specify mirroring based on version ranges. This can be used to reduce the size of the produced result when historical versions are not needed in the aggregated result.
• Categorization — categorization of installable units is important to the consumers of the aggregated repository. Categories are often choosen by repository publishers in a fashion that makes sense when looking at a particular repository in isolation, but when they are combined with others it can be very difficult for the user to understand what they relate to. An important task for the constructor of an aggregation is to be able to organize the aggregated material in an easily consumable fashion. The b3 aggregator has support for category prefixing, category renaming, addition of custom categories, as well as adding and removing features in categories.
• Validation — the b3 aggregator validates the aggregated result to ensure that everything in the repository is installable.
• Blame Email — when issues are found during validation the aggregator supports sending emails describing the issue. This is very useful when aggregating the result of many different projects. Advanced features include specifying contacts for parts of the aggregation which is useful in large multi layer project structures where issues may related to the combination of a group of projects rather than one individual project - someone responsible for the aggregation itself should be informed about these cross-project issues. The aggregator supports detailed control over email generation including handling of mock emails when testing aggregation scripts.

Documentation

The b3 aggregator documentation is available here on the Eclipse Wiki.

Buckminster 3.6 New & Noteworthy

The Helios release of Buckminster has the following new and noteworthy features available in 3.6M7

• Support for Git - uses, updates, or clones git repository as needed
• Headless JUnit and EclEmma (code coverage) support
• Comprehensive documentation available - introduction, examples, and reference. Download PDF, 250 pages, includes descriptions of the new features described here.
• Graphical dependency visualizer - resolutions can be viewed and navigated/filtered with a Zest based viewer
• Much improved target platform support - using new features in PDE to automatically manage/materialize target platform config
• Provisioning and management of API baseline
• New EMF based editors for MSPEC and RMAP - much easier to use than editing XML
• Reader type for Project Set (.psf) files - makes it easy to integrate or migrate projects that are using .psf files to describe where the source is
• p2 repository size reduction to 1/3 using improved pack200 support
• OmniVersion support - the support for non OSGi version has been changed to use the p2 OmniVersion implementation for increased flexibility - backwards compatible with Buckminster version-type, version scheme used in earlier versions.
• Qualifier generator using Build Identifier - use a property to control the content of a version qualifier
• LDAP style filters on RMAP providers, CQUERY advisors, and MSPEC nodes - makes it possible to parameterize more things, reduces the need for multiple slightly different copies of these files.
• Smart version range generation for feature 'includes' - heuristics result in natural choices
• Support for category.xml files - the new PDE mechanism for categorizing result in p2 repository is supported
• Headless 'install JRE' support
• Better defaults often renders the MSEPC unnecessary - automatic materialization to Target Platform for binaries often removes the need to use a MSPEC.
• Using new p2 API, p2 'pure' reader, and using separate p2 agent - reduces risk of contamination of the running instance's p2 data.

Eclipse Build Systems in Perspective

It is easy to get confused over the "build system options" available when developing with Eclipse - there is JDT and ANT, PDE (interactive), and PDE/Build (headless) - to start with the "classics". Then there is Athena, an elaboration on the also classic 'releng' system used to build Eclipse itself, then the newer Buckminster. The two latest additions to the family of build related projects at Eclipse are Eclipse b3 (2009), and Tycho (proposal 2010).

With this blog post I want to put eclipse build technology in perspective.

So, what does all these technologies do?

The classics

JDT - builds java source, interactively, under control of the project and workspace preferences.

PDE - builds OSGi components and the Eclipse specific extensions (features, products, etc.) under the control of PDE preferences. PDE consists of both the interactive parts (incremental builds, export commands, dialogs etc.), and the headless logic that performs the actual work. This logic is also made available as ANT tasks.

PDE/Build - generates ANT scripts that are then executed in headless fashion to build. The generated ANT scripts makes use of the same ANT tasks used by the interactive PDE.

If we stop there for a moment - this was the level of support for building provided by Eclipse a few years back. Although PDE does a very good job of building things interactively, it is fair to say that the headless PDE/Build has been the source of much pain and frustration. To complete the picture; at this time there was also the "releng" system in use at Eclipse, which only ran on the Eclipse servers.

Improving on PDE

While PDE/Build has more or less remained unchanged there has been many different approaches to how to provide good support for headless builds. Some took the scripting route - improving on releng base builder (Athena), some wrote their own scripts, other started modeling PDE like this project in PDE incubator, and some wrote better/easier to use script generators like pluginbuilder.

Eclipse Buckminster

When the Eclipse Buckminster project was introduced in 2005 it focused on additional things — when using generated scripts and technologies external to the IDE there are often issues caused by differences in how things are built i.e. it works just fine in the IDE but breaks in the build. An important goal for Buckminster was (and still is) to provide exactly the same build interactively as on the servers. This means that Buckminster has a tight integration with the builders running in a workspace made available in an efficient packaging for headless execution. Another important design decision was to use the existing information (i.e. the meta data used by PDE in the IDE) without generation of scripts and without roundtrip engineering, instead using advice/decoration to modify discovered metadata when the original information is not enough.

In addition to the important philosophical difference, Buckminster also provides unique support for materialization of a workspace, automatic provisioning of a target platform, running JUnit tests, EclEmma code coverage, Hudson integration, and much more.

Note that when Buckminster builds PDE related material (Buckminster can build other things as well), it calls on the same PDE logic that is used when building with interactive PDE.

Tycho

Tycho is a set of Maven plugins that provide building of OSGi and Eclipse related components (features, plugins, RCP products, etc.) and is an alternative to PDE suitable for those that have a Maven centric setup. Tycho does not use the original PDE logic.

Eclipse b3

The b3 project is about making it easy to work with build systems — discovering and modernizing or integrating existing build systems should be just as easy as building meta data rich components interactively or in continuos integration fashion. Eclipse b3 starts at the very other end of the spectrum than "which compiler to use" or which meta data dialect is used to describe components.

Eclipse b3 does this by providing EMF based models for build (i.e. components, their relationships, versions, types, etc.), expressions (i.e. the processing in form of tasks, builders, compilers, etc.), and p2 (with support for aggregation, re-categorization, mirroring, maven meta data publishing, and more), as well as a concrete syntax in form of a DSL implemented with Xtext that provides a rich text editing environment, and an evaluator that makes it possible to run b3 scripts.

As an example - there is nothing in Eclipse b3 that restrict it to using the original PDE logic to build the PDE related artifacts, it can just as well make use of Tycho's alternate way of building the same things.

The very first way Eclipse b3 will be building things is to use Buckminster as the execution engine. A small, and easy to understand b3 script will drive the entire build — combining the ease of use in the b3 DSL with the proven stable builds provided by Buckminster.