XRobots at EclipseCon Europe 2014

EclipseCon Europe 2014 was special to me: Miro, Holger and me had been preparing a robot game for our booth, and I was very curious whether it would work as expected and more than that whether people would like it. It turned out to be a blast: Some attendees even spent the after hours to work on their programs offline, and returned to our booth the next day to prove it. And even for those who didn't choose to compete, it was still a great show to watch.

The game works as follows: You write a script to control a robot in a wrestling fight. You win the game when your robot turns the opponent over or pushes it outside of the ring. Within the script, you can define how your robot should react on specific conditions, e.g. when it is close to the opponent. The robots can move, rotate, drive curves and lift up their scoop. You can read the own and the opponent’s position, view direction and speed.

The DSL is of course implemented with Xtext. You only need a web browser to join the game. We have enhanced Holger’s Xtext in the Web code to provide a web-editor with the usual Xtext goodies such as live validation, content assist, syntax highlighting, hovers etc. It worked really well and we plan to spend more time on Xtext in the web soon. As we have embedded the expression library Xbase, you can use all kind of expression and control structures.

To run your script you have enter a token which is shown on the game display at our booth. As soon as both robots are assigned, the scripts are executed by an interpreter on our game server, which makes two Lego Mindstorms robots engage in a fight.

The involved tools include Lejos, Xtend, Xtext with Xbase, Eclipse, Orion, Jetty, JavaFX and more. To locate the robots precisely enough we had to go for a video solution based on OpenCV. The Making-Of has been rather exciting, so we decided to file a talk on this for EclipseCon NA 2015. If you plan to attend and want to hear the story, vote for it!

The robots’ next gig will be at Devoxx in Antwerp next week. Of course we’ll bring them to  EclipseCon 2015 in San Francisco, too.

Apart from the robots, EclipseCon Europe 2014 was again gorgeous. The three sessions I gave went fine, I met a lot of really nice people and for the first time I had the time to watch the amazing Circus Eclipse. Thanks to the organizers and to the community to make EclipseCon such a great event.

PS: Our talk The Making of XRobots has been selected as one of three early birds for EclipseCon NA 2015! See you there!

Xtext, Xtend, JavaFX and Robots

It is autumn again and EclipseCon Europe 2014 is just around the corner. To me as an Eclipse committer from Germany ECE always feels a bit like meeting the family. In fact, I have been attending for seven years in a row now. I even remember the times when it was called Eclipse Summit Europe. Oh man, I am getting old…

As every year, my colleagues and me are contributing the program: I have counted 14 sessions by itemis and I am going to give three myself. Of course there is a lot about Xtext and Xtend, but also on other topics like Java performance or home automation. And itemis is sponsoring again.

For me it starts Tuesday morning: Sven and me are going to give a tutorial on Functional Programming With Xtend. Xtend — the JVM language developed at Eclipse — offers a great set of features to exploit the benefits of functional programming without the verbosity of Java. This session not only gives an introduction on functional programming in general but it is also a good opportunity to look beyond Duke’s nose. Bring your laptop, you are going to get your hands dirty!

No EclipseCon without me on graphical editing: In Diagrams, Xtext and UX I am going demonstrate how you can add diagrams to your textual Xtext languages without adding usability pains. I am going to use the JavaFX based framework FXDiagram which offers superior UI metaphors for the end users in combination with more freedom for the developer. Watch it in action on Tuesday afternoon.

On Wednesday, Holger and me are going to show you what’s New & Noteworthy in Xtext. We have recently finalized the Xbase API for DSLs with expressions, concurrency in the IDE has been reworked, we have parallelized the builder and much more. We plan to give a sneak preview of the IntelliJ support as well.

But the ultimate reason why you must attend EclipseCon Europe 2014 is going to be at our booth. Miro, Holger and me have created a new game for you using all the technologies above (and more). So be prepared to encounter these two guys.

'X' Is For Abstraction

We at itemis have created a bunch of projects with an ‘X‘ during the last years: Xtext, Xtend and Xbase. In this blog post I am trying to elaborate the different use cases for each of them.
 

It all starts with Java...

...which is a nice language understood by many programmers worldwide. A giant ecosystem and a vivid community as well as great IDEs are keys to its success.

But Java has a couple of crucial problems: Its syntax is inflexible and very verbose, making fluent APIs the closest you can get to domain-specific languages. Design patterns and structural conventions as JavaBeans have to be manually implemented over and over again with lot of error-prone boiler plate code. Frameworks require a big deal of XML pointing to Java classes, often ignored by the static analysis at compile time. IDEs can help us here, but they are usually limited to scaffolding or a single way of using a framework. The diagnosis is: Java is lacking the right means of abstraction.

All technologies presented in the following enable you to define your own set of abstractions on top of Java. The solutions differ in syntactic flexibility, turnaround times, required user skills and multi-platform coverage.

External DSLs (Xtext)

Xtext is a framework for external, textual, domain-specific languages. Xtext makes it easy to build an entire IDE for a completely new language, including your own code generators or interpreters to execute it. The Xtext homepage lists a lot of example projects.

When should you use Xtext? First of all, when you need a maximum of syntactical freedom. As Xtext DSLs are external they are defined by their own grammar, thus not constrained by the syntax of a host language.

Second, use plain Xtext if your language has no correspondence to Java or if you want to avoid dependencies on JDT at all cost. Languages that target different platforms at the same time are most likely based on a plain Xtext-based language, too. The technical skills needed to work with a Xtext-based language IDE is relatively low. I personally do not believe in "programming business men". But Xtext has been successfully used for non-technical, purely domain-oriented languages.
As the generated IDE is already very powerful by default and can be customized easily, some people even use Xtext to build the IDE front end for an existing language with an existing compiler.

If you change the grammar, you have to regenerate the language infrastructure and redeploy the language's plug-ins to the user's machines. The latter can be a challenge in some scenarios.

Mostly structural languages with simple expressions only will go smoothly. A few complicated expressions can pragmatically be integrated using the common integration techniques (generation gap pattern, black-box literals, protected regions, etc.). But all of these raise the complexity, break the abstraction, and are hard to maintain. If expressions are the rule and not the exception, consider

Java DSLs (Xtext with Xbase)

Under the codename Xbase, we have developed a rich expression language, that you can embed in your Xtext-based DSLs. It reuses Java's typesystem, can express everything that Java expressions can do and adds even more features like type inference, operator overloading, lambda expressions etc.

The default way of using Xbase is to define a transformation from your language to a structural Java model and embed the expressions in the appropriate Java contexts. A generic code generator builds Java code from this model, so no need to build your own. As we generate plain Java code, these Xbase languages integrate seamlessly with hand-written Java code and work for platforms that are sensitive to bytecode manipulation, like Android.

Java DSLs are the way to go for languages that compile to Java and need rich behavior or expressions. These are probably much more than you'd probably expect. Have a look at our seven languages for some examples. There is also an interpreter for Xbase allowing to skip the compilation step on the language user's side.

We have only implemented an execution environment for Java, so it's up to you if you need another. Restricting the available expressions is easy and could help here significantly. You can also add new expressions to the expression language, but the syntactic space is already quite crowded.

Non-technical users are usually ruled out here. You don't want to teach them managers about closures, do you? You still have complete syntactical freedom for your language. But you still have to regenerate the plug-ins and redeploy them on grammar changes as before.

Internal DSLs (Xtend)

Just because Java doesn't offer the means of abstraction you need doesn't mean you have to define a new language yourself. We have created Xtend, a general purpose language (GPL) based on Java but adding a lot of fancy stuff like type inference, operator overloading, lambda expressions, etc. You guessed it: Xtend is based on the Xbase expression library, but also defines the structural parts (types, fields, methods, annotations etc.), adds extension methods, multi-dispatch methods, template expressions, and offers an even better IDE.

This is already a mighty toolset and can bring you pretty close to internal DSLs in other languages. Look at our examples from EclipseCon 2012 (especially the SWT, Distances and HTML Builder examples).

One additional leap ahead are Active Annotations (new in 2.4, beta). These annotations allow you to participate in the Xtend to Java compilation. They are particularly useful for implementing patterns,  but can be surprisingly powerful in other scenarios, too. Have a look at a few cool examples: Observer pattern, Vistors, JSON classes, a simple REST API, micro benchmarking...

Having everything you need in the language itself gives you short turnarounds as no more regeneration and redeployment is needed. The toolstack stays nicely low. But you're restricted to generating Java and use the syntactical elements of the Xtend language. I would never recommend Xtend to non-programmers, but to everyone fighting with Java.

Seven Languages By Seven Geeks

One of the most powerful new features of Xtext is the capability to build languages for the JVM. In my opinion, these languages are a real breakthrough in domain-specific language design:

• These DSLs directly interoperate with Java. They compile to Java code and allow to refer to Java artifacts.
• They not only cover structural aspects but also behavior. DSLs become really powerful with a rich expression language offering today what Java has been missing for years, like type inference, lambda expressions, extension methods etc.
• Complicated integration patterns like Generation Gap or Protected Regions become obsolete. Everything can be expressed on the same level of abstraction
• They offer top-notch Eclipse-based tooling and integrate into the Java IDE. Switch between the DSL and the Java code during debugging, refactor across multiple languages, navigate to source etc.

Yet we have the impression that the concept and the benefits of JVM DSLs have not been well understood and adopted. This is why we decided it was time for another example and documentation offensive. The result is the 7 Languages project: Seven example JVM languages highlighting the benefits of this approach in various use cases of Java development:

1. A scripting language. Write code without any boilerplate.
2. A build language similar to Gradle but statically typed.
3. JavaBeans facades for mongoDB objects.
4. A DSL to define Guice modules.
5. A language to route HTTP requests with conditions, variables and dependency injection.
6. A template language
7. A LOGO-like turtle drawing environment.

Clone or fork the code from github and enjoy the documentation on theXtext website.

Multitouch Gestures in the Generic Graph View

In my previous post, I've already described the discovery mechanisms of the Generic Graphical View project: Starting from one node you can step by step reveal other connected nodes in the same model. The model itself is fixed - it is still a view! - but the user can decide herself which subset of elements should appear in the diagram. That behavor allows to create diagram views focussed on a specific semantic aspect of the model.

The challenge is to find a decent UI for nodes wich have several hidden connections. How should a user select any subset of these?

Since the Helios release SWT supports four multitouch gestures: PAN, ZOOM, SWIPE and ROTATE. On my Mac's touchpad these gestures are actually easier to perform than a conventional drag operation because there is no extra mouse button. So I decided to give it a try and have a look how gestures could be used in the graph view. Here is a screencast showing the results:



SWT's multitouch API is fairly simple (example snippet). The first step was to add multitouch events to GEF, such that the can be handled in the same way as mouse or keyboard events. I had to extend a couple of GEF classes. If you're interested in the details, have a look at the classes in the package org.eclipse.xtext.graphview.behavior.gestures and the GraphViewRootEditPart.

These changes allow to use a ZOOM gesture on a ScalableFreeformRootEditPart.
Note that the PAN gesture is already supported by GEF because the FigureCanvas uses hardware scrolling. Unfortunately, these event is always consumed, such that it will not make it through to other GEF elements.

The ZOOM gesture can also be used to drill down into elements. If the mapping defines an open reference to another diagram, you can drill down by selecting it and performing the ZOOM in gesture. If you‘re inside a drill down diagram, you can ZOOM out into the container diagram.

Any Tool can now handle multitouch gestures by implementing the IViewerGestureHandler interface. For a DragTracker this makes no sense, as it is exited as soon as the mouse key is released and you cannot start a drag and a multitouch gesture simultaneously.

The RevealGestureTool works as follows: Click on the + button to reveal the hidden connected elements, Then use ROTATE or SWIPE to rotate elements, ZOOM to change the distance and right-click to select and pin individual elements. Another click will commit the input.

Discovery Diagrams for the Generic Graphical View

If you've read my previous posts or seen my presentation at EclipseCon Europe 2011, you'll already know the Generic Graphical View framework: Using two Xtext-based DSLs - one for mapping semantic to graphical elements and the other for styling the figures - you can easily create nice looking diagrams for any Java based model.


In its first version the contents of the diagram was entirely defined in the mapping DSL. It was only possible to restrict the visible elements to a subset of the model by adding filtering expressions in the mapping. For bigger models it was hard to focus on a subset that is not structural defined but based on semantics, e.g. all classes that deal with addresses in a CRM application.

Many graphical frameworks handle the selection of the visible elements in a diagram by means of dialogs. The user is then forced to switch between keyboard and mouse which is annoying, and these dialogs are often complicated and unintuitive. The goal for the Generic Graph View project was to build a UI for choosing the diagram contents that uses the mouse or trackpad only and avoid dialogs at all cost. The following screencast demonstrates the basic functionality.



As a first step, actions were introduced that allow to hide existing graphical elements - nodes, labels or connections. If you hover over an element, you'll get an popup button allowing to remove it from the view. So you can start with a big diagram and strip it down to what you want to see. Once an element inside a node has been removed, you can restore the node completely using another popup button. You can define elements as initially hidden using a hidden flag in the mapping definition.

In addition to the subtractive approach, you can step by step extend the diagram contents from an initial node. If you're hovering over a node that has hidden connection, several + buttons will popup. When one of these buttons is triggered, the hidden elements and connections will appear in transparent arranged in a circle around the source element. In the same drag operation you can change the distance and the angle of that circle. This behavior will get really cool with multitouch gestures, which I will cover in a separate post.

Join My Pie Fight Session at EclipseCon Europe 2011

Competition is hard at EclipseCon: Everybody has to make sure to get the most participants into his talk. As Sven already announced a quiz for the same timeslot as mine I've had to figure out something even more exciting.

So here's my idea for my session:

I'll begin the session ranting on every Eclipse project that's somehow related to graphics. I'll shamelessly reveal all of their weaknesses. Everybody involved in one of these projects will be so offended that they start to get personal, first against me then against each other. When emotions boil up, I expect people to self-organize in teams, like The Knights of GMF Runtime or The Zest'o'Maniacs. My session is at 5pm so I expect a good supply of cake as in the years before. I am starting the fight by throwing the first cake, e.g. on the GMF Tooling Ultras. They try to answer but unluckily hit a member of the Spray Front, who's already in the midst of a quarrel with a Graphiti Warrior. Thinking it was a GEF Cyclope who threw the cake... A real pie fight breaks loose.

Hehehe (demonic laughter)

Well... thinking again... the cake at EclipseCon is usually quite yummy. It would be a pity to waste it in such a way. Maybe I should come up with something more peaceful.

OK, I see, I have to make that differently: I am going to point to some general issues with graphical editing. I'll weigh the cost against the usefullness and show, why an interpreted graphical view is so much simpler to implement and to use than a graphical editor. Second part will be a live demo of a prototype for such a framework.

Maybe I should bring a helmet, just in case someone did not notice my change of plan.

Extending Xbase

The new expression language library Xbase allows to integrate expressions in your own Xtext 2.0 DSLs. Xbase offers a grammar, a large runtime library and a compiler as well as an interpreter to execute these expressions. Thus, Xbase opens the field for a much wider class of DSLs and annihilates prior advantages of internal DSLs over external DSLs based on Xtext.

In a series of internal workshops on Xbase, I've often encountered one question: Can we extend the expressions of Xbase? In this blogpost, I am going to show you how to do that. It is based on the domain model example shipped with Xtext 2.0, which you can instantiate by choosing File > New > Example... > Xtext > Xtext Domain-Model Example.

The Grammar

By default, Xbase only knows integer literals. In this example we are adding decimal literals.

If you want to add a new concept to Xbase expressions syntactically, you have to hook into the right rule of Xbase. As the name suggests a literal can be added in the Xbase rule XLiteral. So in the Dominamodel.xtext you have to add

import "http://www.eclipse.org/xtext/xbase/Xbase"
...
XLiteral returns XExpression:
  XClosure |
  XBooleanLiteral |
  XIntLiteral |
  XNullLiteral |
  XStringLiteral |
  XTypeLiteral |
  DecimalLiteral; // our new alternative

DecimalLiteral:
  intPart=INT '.' decimalPart=INT;

As the EPackage of the domainmodel language is generated this will introduce a new EClass DecimalLiteral as soon as you regenerate the language infrastructure.

Type Provider

For the sake of simplicity, our decimal literal will be mapped to java.lang.Double. We have to adapt the type provider accordingly

public DomainmodelTypeProvider {
  @Inject
  private TypeReferences typeReferences;  
  ...
  protected JvmTypeReference _type(DecimalLiteral literal, 
                                   boolean rawType) {
    return typeReferences.getTypeForName(Double.class, 
                                         literal);
  }
}

Compiler

The compiler has to understand what to do with decimal literals. We have to add a method that appends the concrete syntax of the Java double literal for a DecimalLiteral

public class DomainmodelCompiler {
  ...  
  protected void _toJavaExpression(DecimalLiteral expr,
                                   IAppendable b) {
    b.append(expr.getIntPart() + "." 
      + expr.getDecimalPart());
  }

  protected void _toJavaStatement(DecimalLiteral expr, 
                                  IAppendable b, 
                                  boolean isReferenced) {
    generateComment(expr, b, isReferenced);
  }
}

We can now use decimal literals in your expressions and they are correctly converted to java.lang.Doubles. The following domain model should now compile:

entity DecimalLiteralTest {
  op approxPi() : double {
    3.1415926535
  }
}  

Note that autoboxing/unboxing works in Xbase the same as in Java. Still we do not have any operations for decimal literals. The next section shows how to add them.

Operators

Operators in Xbase are implemented as Java methods following a naming convention. For example, the operator + will be mapped to a call to a method named operator_plus(). There are several places where Xbase tries to find this method

1. in the current type as operator_plus(OperandType1, OperandType2)


2. in the class of the first operand, as OperandType1.operator_plus(OperandType2)


3. in the extension scope as operator_plus(OperandType1, OperandType2)

The first alternative is ruled out, as we want to provide global operations. The second does not apply as we cannot change the final class java.lang.Double, so we're eventually dealing with alternative 3.

Let's write a new extension library class holding the operations as static methods. These methods need to be available at runtime, too, as the generated Java code will call them. That's why it is a good idea to put such extension classes into a separate plug-in org.eclipse.xtext.example.domainmodel.lib.

package org.eclipse.xtext.example.domainmodel.lib;
public class DecimalExtensions {
  public static Double operator_plus(Double x, 
                                     Double y) {
    return x + y;
  }
  public static Double operator_minus(Double x, 
                                      Double y) {
    return x - y;
  }
  public static Double operator_multiply(Double x, 
                                         Double y) {
    return x * y;
  }
  public static Double operator_divide(Double x,
                                       Double y) {
    return x / y;
  }
}

Globally available extension classes and literal classes have to be registered in the
StaticMethodsFeatureForTypeProvider. We override it

package org.eclipse.xtext.example.domainmodel.scoping;
...
public class DomainmodelStaticMethodsProvider 
    extends StaticMethodsFeatureForTypeProvider {
  @Override
  protected Iterable getVisibleTypesContainingStaticMethods
    (JvmTypeReference reference) {
    Iterable resultFromSuper = 
      super.getVisibleTypesContainingStaticMethods
        (reference);
    if (reference != null && reference.getType() != null
        && "java.lang.Double"
          .equals(reference.getType().getIdentifier())) {
      return Iterables.concat(Collections.singletonList
("org.eclipse.xtext.example.domainmodel.lib.DecimalExtensions"),
          resultFromSuper);
    }
    return resultFromSuper;
  } 
}

and bind our customized implementation

public class DomainmodelRuntimeModule 
  extends AbstractDomainmodelRuntimeModule {
...
  public Class 
      bindStaticMethodsFeatureForTypeProvider() {
    return DomainmodelStaticMethodsProvider.class;
  }
}

If you add the plug-in org.eclipse.xtext.example.domainmodel.lib to the classpath of the project in the runtime workspace, the following domain model should compile:

entity DecimalLiteralTest2 {
  op circleArea(double radius) : double {
    3.1415926535 * radius * radius
  }
}  

Using Xbase to Define a Generic Graphical View

While textual representations such as code are perfectly suited to process the details of an object model, a graphical view can be very helpful to display the relationships between objects.

Most graphics frameworks in the Eclipse ecosystem provide graphical editors. Usually, the semantic model is mapped to its graphical representation using a hard wired transformation, be it in code (GEF, Graphiti) or in models and a code generator (GMF, the upcoming Spray project). There are two issues here: First, as opposed to a graphical view, a graphical editor requires the mapping to be bidirectional. That constrains the transformation a lot, e.g. requires having the same structure on the graphical as on the semantic side. Second, hard wiring the mapping introduces extra compilation/generation/deployment turnarounds.

I've created a prototype of a graphical view that is configured using two textual DSLs: A mapping DSL that declaratively describes the transformation from semantic objects to graphical elements, and a styling DSL to modify the graphical representation. The graphical view interprets the mapping and the stylesheet on a given input model, giving immediate feedback on changes of these live models.

Both DSLs are implemented in Xtext 2.0 and make heavy use of the new Xbase expression language library. As Xbase uses Java's typesystem, we can refer to any POJO as an input model. Xbase's rich semantics allow comfortably navigating the models references, and its interpreter can be customized easily to fit our needs. The graphics are implemented using plain GEF and Zest layouts.

The following screencast gives you a short demo. (Sorry for the poor video quality, it gets better if you got to full screen mode)

EclipseCon 2011 Slides

EclipseCon 2011 was once again a cool conference, even though I really needed a week of holiday to recover from the bad weather, the jet-lag and the general lack of sleep :-) This is why it took me some time to upload the slides of our presentations. But better late than never, here they are.

First, Sebastian and me presented What's Cooking in Xtext 2.0. The room was almost full, and when I asked a vast majority of the audience had already had contact with Xtext. To me that shows that Xtext has really become mainstream, a really good feeling. Here are the slides:

What's Cooking in Xtext 2.0

On Thursday, Sven, Sebastian and me gave the tutorial on Pragmatic DSL Design with Xtext. There were so many participants, that we had to switch rooms with Stephan Herrmann's tutorial on Object Teams. Thanks for that, Stephan! And here are our slides:

Pragmatic DSL Design with Xtext, Xbase and Xtend 2

Did I mention EclipseCon?

I just realized that I haven't blogged on this year's EclipseCon yet. Time to do that. You might have guessed it:


As a committer to several Eclipse projects, I am really looking forward to meeting "the family". It's always been big fun and I guess it will be the same this year. But the main reason for going there is of course:


Together with Sebastian I will give a talk on What's cooking in Xtext 2.0 on Tuesday at 2pm. We're eager to show you all the new features of Xtext we've been working on during the last months, especially the expression language library Xbase and the code generation langauge Xtend.

You should not miss our tutorial
Pragmatic DSL Design with Xtext, Xbase and Xtend 2 on Thursday at 10:30am. Sven will join us to give you a walk through tutorial on a modern way of creating DSLs. We just had another rehearsal on this one and I promise you it will be a real burner.


All of that wouldn't have been possible without the support from the company we're working for: itemis. Into the bargain we're bronze sponsors for this years conference.
It is great to work for a company that really lives open source.