Monday, May 20, 2013

Another Xtend and JavaFX Story

For my spare time JavaFX project, I was looking for a way to export a scene to SVG. Googling a bit I stumbled over a blog entry by Gerrit Grunwald who implemented an JavaFX shape to SVG path converter class for the JFXtras project. It looked good so I decided to give it a try.

 

Java to Xtend

The class is of course written in Java. Even though there is no technical reason - Java and Xtend classes can coexist in the same project without any problems - I wanted to convert it to Xtend as well. I am a lazy guy. Luckily Krzysztof Rzymkowski had recently posted on the Xtend group that he had started to implement a Java to Xtend converter. It even has a web interface, and except for one slight issue with a for-loop it worked like a charm. Great work, Krzysztof!

Of course the resulting code is pretty Java-like, so I wanted to improve on it using more of the cool Xtend features. I found quite a few spots to do so, reducing the amount of code significantly and enhancing readability a lot. The remainder of this post is about this ongoing love of JavaFX and Xtend. The complete source code will likely be made open-source soon.

 

Dispatch Methods

The class starts with a method that only contains an instanceof-cascade to delegate to the conversion method for the specific subclass of Shape the parameter has:
public static String shapeToSvgString(final Shape SHAPE) {
   final StringBuilder fxPath = new StringBuilder();
   if (Line.class.equals(SHAPE.getClass())) {
      fxPath.append(convertLine((Line) SHAPE));
   } else if (Arc.class.equals(SHAPE.getClass())) { 
      fxPath.append(convertArc((Arc) SHAPE)); 
   } else if (QuadCurve.class.equals(SHAPE.getClass())) {
      fxPath.append(convertQuadCurve((QuadCurve) SHAPE));
   } ... 
In Xtend I can use dispatch methods to realize this: They must have the same method and number of parameters, but different parameter types. The Xtend compiler then generates the dispatcher method with the instanceof-cascade automatically. So renaming the delegate methods and marking them as dispatch made the dispatcher method obsolete.
def dispatch String toSvgString(Line line) ...
def dispatch String toSvgString(Arc arc) ...
def dispatch String toSvgString(QuadCurve quadCurve) ...
Tip: If you want to delegate from one dispatch case to another, the original methods are available with an underscore preceeding their name.

Templates

An SVG path is kind of a cryptic string. In the Java code it is assembled using a StringBuilder, e.g.
final StringBuilder fxPath = new StringBuilder();
fxPath.append("M ").append(CENTER_X).append(" ")
   .append(CENTER_Y - RADIUS).append(" ");
fxPath.append("C ")
   .append(CENTER_X + CONTROL_DISTANCE)
   .append(" ").append(CENTER_Y - RADIUS).append(" ")
   .append(CENTER_X + RADIUS).append(" ")
   .append(CENTER_Y - CONTROL_DISTANCE)
   .append(" ")
   .append(CENTER_X + RADIUS).append(" ")
   .append(CENTER_Y).append(" ");
...
That's the best you can do with Java. In Xtend we have template expressions - multiline strings which can be interrupted with values from expressions. Even IF-conditions and FOR-loops are supported. With carefully chosen regular expressions for find/replace and some manual fine-tuning the above becomes nicely readable
'''M «centerX» «centerY - radius»
   C «centerX + controlDistance» «centerY - radius» 
   «centerX + radius» «centerY - controlDistance» 
   «centerX + radius» «centerY»
...

Switch Expression

I found another finer-grained instanceof-cascade in the convertPath method:
final StringBuilder fxPath = new StringBuilder();
for (PathElement element : PATH.getElements()) {
   if (MoveTo.class.equals(element.getClass())) {
      fxPath.append("M ")
         .append(((MoveTo) element).getX()).append(" ")
         .append(((MoveTo) element).getY()).append(" ");
   } else if (LineTo.class.equals(element.getClass())) {
      fxPath.append("L ")
         .append(((LineTo) element).getX()).append(" ")
         .append(((LineTo) element).getY()).append(" ");
   } else if (CubicCurveTo.class.equals(element.getClass())) { 
      fxPath.append("C ")
...
As the bodies of the if-statements are so simple, I decided to use Xtend's switch instead. It allows to use type guards for the cases and automatically cast the switch variable to that type inside the case's body:
val it = new StringBuilder
for (element : path.elements) {
   switch element {
      MoveTo: append('''M «element.x» «element.y» ''')
      LineTo: append('''L «element.x» «element.y» ''')
      CubicCurveTo: append('''C «element.controlX1»...
I could further use the operator => instead of the Builder class and the operator <=> replacing Double.compare.

Extension Import (Client Side)

From a client side, the shapeToSvgString method is a utility method for Shapes. In Xtend, you can import such methods using a static extension import. That makes them callable in extension syntax, as if the method was defined in the class of the first parameter. To further improve readability, I renamed the methods to toSvgString such that I can now write
import static extension ...ShapeConverter.*
... 
new Rectangle.toSvgString

Wednesday, April 3, 2013

'X' Is For Abstraction

We at itemis have created a bunch of projects with an ‘X‘ during the last years: Xtext, Xtend and Xbase. In this blog post I am trying to elaborate the different use cases for each of them.
 

It all starts with Java...

...which is a nice language understood by many programmers worldwide. A giant ecosystem and a vivid community as well as great IDEs are keys to its success.

But Java has a couple of crucial problems: Its syntax is inflexible and very verbose, making fluent APIs the closest you can get to domain-specific languages. Design patterns and structural conventions as JavaBeans have to be manually implemented over and over again with lot of error-prone boiler plate code. Frameworks require a big deal of XML pointing to Java classes, often ignored by the static analysis at compile time. IDEs can help us here, but they are usually limited to scaffolding or a single way of using a framework. The diagnosis is: Java is lacking the right means of abstraction.

All technologies presented in the following enable you to define your own set of abstractions on top of Java. The solutions differ in syntactic flexibility, turnaround times, required user skills and multi-platform coverage.

External DSLs (Xtext)

Xtext is a framework for external, textual, domain-specific languages. Xtext makes it easy to build an entire IDE for a completely new language, including your own code generators or interpreters to execute it. The Xtext homepage lists a lot of example projects.

When should you use Xtext? First of all, when you need a maximum of syntactical freedom. As Xtext DSLs are external they are defined by their own grammar, thus not constrained by the syntax of a host language.

Second, use plain Xtext if your language has no correspondence to Java or if you want to avoid dependencies on JDT at all cost. Languages that target different platforms at the same time are most likely based on a plain Xtext-based language, too. The technical skills needed to work with a Xtext-based language IDE is relatively low. I personally do not believe in "programming business men". But Xtext has been successfully used for non-technical, purely domain-oriented languages.
As the generated IDE is already very powerful by default and can be customized easily, some people even use Xtext to build the IDE front end for an existing language with an existing compiler.

If you change the grammar, you have to regenerate the language infrastructure and redeploy the language's plug-ins to the user's machines. The latter can be a challenge in some scenarios.

Mostly structural languages with simple expressions only will go smoothly. A few complicated expressions can pragmatically be integrated using the common integration techniques (generation gap pattern, black-box literals, protected regions, etc.). But all of these raise the complexity, break the abstraction, and are hard to maintain. If expressions are the rule and not the exception, consider

Java DSLs (Xtext with Xbase)

Under the codename Xbase, we have developed a rich expression language, that you can embed in your Xtext-based DSLs. It reuses Java's typesystem, can express everything that Java expressions can do and adds even more features like type inference, operator overloading, lambda expressions etc.

The default way of using Xbase is to define a transformation from your language to a structural Java model and embed the expressions in the appropriate Java contexts. A generic code generator builds Java code from this model, so no need to build your own. As we generate plain Java code, these Xbase languages integrate seamlessly with hand-written Java code and work for platforms that are sensitive to bytecode manipulation, like Android.

Java DSLs are the way to go for languages that compile to Java and need rich behavior or expressions. These are probably much more than you'd probably expect. Have a look at our seven languages for some examples. There is also an interpreter for Xbase allowing to skip the compilation step on the language user's side.

We have only implemented an execution environment for Java, so it's up to you if you need another. Restricting the available expressions is easy and could help here significantly. You can also add new expressions to the expression language, but the syntactic space is already quite crowded.

Non-technical users are usually ruled out here. You don't want to teach them managers about closures, do you? You still have complete syntactical freedom for your language. But you still have to regenerate the plug-ins and redeploy them on grammar changes as before.

Internal DSLs (Xtend)

Just because Java doesn't offer the means of abstraction you need doesn't mean you have to define a new language yourself. We have created Xtend, a general purpose language (GPL) based on Java but adding a lot of fancy stuff like type inference, operator overloading, lambda expressions, etc. You guessed it: Xtend is based on the Xbase expression library, but also defines the structural parts (types, fields, methods, annotations etc.), adds extension methods, multi-dispatch methods, template expressions, and offers an even better IDE.

This is already a mighty toolset and can bring you pretty close to internal DSLs in other languages. Look at our examples from EclipseCon 2012 (especially the SWT, Distances and HTML Builder examples).

One additional leap ahead are Active Annotations (new in 2.4, beta). These annotations allow you to participate in the Xtend to Java compilation. They are particularly useful for implementing patterns,  but can be surprisingly powerful in other scenarios, too. Have a look at a few cool examples: Observer pattern, Vistors, JSON classes, a simple REST API, micro benchmarking...

Having everything you need in the language itself gives you short turnarounds as no more regeneration and redeployment is needed. The toolstack stays nicely low. But you're restricted to generating Java and use the syntactical elements of the Xtend language. I would never recommend Xtend to non-programmers, but to everyone fighting with Java.

Sunday, January 6, 2013

Accumulating JavaFX Transforms With Xtend

I was a bit vague in my last post on JavaFX's transformation API, so I decided to write a separate post on this topic.

In JavaFX, each node in the scenegraph can be translated, rotated, scaled or sheared relative to its parent. In mathematical terms, each node maintains a transformation describing its own local coordinate system. This transformation can be defined with a matrix represented as a Transform object in JavaFX.

The JavaFX API offers some convenience methods to manipulate this transformation. E.g., you can scale a node in x-direction and translate it by 42 units in y-direction by calling
  node.setScaleX(2);
  node.setTranslateY(42)
Unfortunately, these convenience methods do not accumulate as one would expect. Instead of multiplying the matrices for subsequent transformations, the only set specific entries in the matrix. As a result, translating an object before rotating yields the same result as applying these transformations in reverse order. This is mathematically wrong and does also not match the users expectations.

In my diagram editor application, I want to scroll (translate), zoom (scale) and rotate the canvas using mouse gestures. From the user's point of view it's imperative that each transformation builds on the previous state. The convenience methods don't match this usecase.

The correct solution to this problem is to multiply the transformation matrices. Unfortunately, JavaFX lacks any kind of calculation API for Transform and its subclasses.

Once again, this is where Xtend comes to our aid. An extension method in Xtend can be used to define functions for existing (closed) types, which syntactically look like being methods of the type on the caller's side. Affine is a subtype of Transform that allows its matrix entries to change. So I wrote extension methods to translate, rotate, scale and shear an existing Affine by multiplying the respective transformation matrix, e.g.
class TransformExtensions {
  def static scale(Affine it, double x, double y) {
    // left multiply a scale matrix to it
    // highly optimized as there are many zeros in the scale matrix
    mxx = x * mxx 
    xy = x * mxy 
    mxz = x * mxz
    tx = x * tx // take existing translation into account 

    myx = y * myx
    myy = y * myy
    myz = y * myz
    ty = y * ty 
 }
}
Importing these as extensions
import static extension ...TransformExtensions.*
now allows to accumulate the transformations, e.g.
val diagram = scene.root
val diagramTransform = new Affine
diagram.transforms.clear
diagram.transforms += diagramTransform

val EventHandler scrollHandler = [
  diagramTransform.translate(deltaX, deltaY)
] 
scene.onScrollStarted = scrollHandler 
scene.onScroll = scrollHandler
scene.onScrollFinished = scrollHandler
  
val EventHandler rotateHandler = [
  diagramTransform.rotate(angle, sceneX, sceneY)
] 
scene.onRotationStarted = rotateHandler
scene.onRotate = rotateHandler
scene.onRotationFinished = rotateHandler
...
The resulting behavior looks like the following screenshot. Note that the mouse position is the pivot for rotations and zoom. 

video 

The same mechanism can be used to properly place labels along connections:

PS: I went for mutable transformation matrices, but the same mechnism will hold for immutable matrices. The extension methods then return a new Affine instead of modifying the receiver.

Tuesday, January 1, 2013

JavaFX Loves Xtend

Inspired by a talk by Gerrit Grunwald and the work of Tom Schindl I started to dig a bit deeper into JavaFX. Being one of the committers of the Xtend language, it was a matter of honour to use Xtend in my experiments. The result: JavaFX and Xtend seem to be made for each other.

If you know me a bit, you won't be surprised that I tried implementing a simple graph editor first. I will blog on that particular application in a separate post. Even though JavaFX is not primarily a graphical editing framework, it offers a lot of the required features, e.g.
  • State of the art rendering and CSS styling
  • A life scene-graph with affine transformations in all nodes
  • An easy way to do data-binding and event propagation
  • Easy to use APIs for effects and animations
  • Built-in support for tablets and touch-aware devices
The developers of JavaFX did a pretty good job in creating a clear Java API, but sometimes Java's rigid syntax was a bit in their way. This is where Xtend comes into play. The resulting code is astonishingly short and easy to read.

Lambda Expressions For Event Listeners

Each JavaFX Node has a couple of convenience methods to react on on UI events, such as mouse clicks, key strokes or multi-touch gestures. A mouse click can for example be handled as
Node node = new Rectangle();
node.setOnMouseClicked(new EventHandler<MouseEvent>() {
   @Override
   public void handle(MouseEvent event) {
      System.out.println(event.getButton() + " button clicked");  
  }
});
Xtend's lambda expressions (aka closures) are automatically coerced to interfaces with a single method. Using the implicit parameter it and the shortcut syntax for property access the above becomes
val node = new Rectangle
node.onMouseClicked = [
   println(button + " button clicked")
]

The With Operator => Obsoletes Builders

JavaFX has a couple of objects that need quite a few parameters to be customized. Instead of implementing a new constructor with a huge number of parameters for each use case, the JavaFX developers generated fluent builder APIs, e.g.
Rectangle rectangle = RectangleBuilder.create()
  .width(80)
  .height(30)
  .fill(Color.BLUE)
  .stroke(Color.RED)
  .strokeWidth(1.2)
  .arcWidth(12)
  .arcHeight(12)
  .build();
In Xtend, we have the with operator '=>' that binds the preceding argument to the parameter of a lambda expression and executes the latter. So we can achieve the same as above with an even shorter syntax and without the need for an accompanying builder class:
val rectangle = new Rectangle => [
   width = 80
   height = 30
   fill = Color::BLUE
   stroke = Color::RED
   strokeWidth = 1.2
   arcWidth = 12
   arcHeight = 12
]
The with operator also facilitates the creation of object trees, e.g. subtrees of JavaFX's scenegraph.

Extension Methods For High-Level Property Binding

In JavaFX, properties can be derived from other properties by means of another fluent API for the calculation. The result is the best you can achieve with pure Java, e.g. given two DoubleProperties a and b, you can bind a mean value property which will be automatically updated when a or be change:
mean.bind(a.add(b).divide(2));
If these calculations get more difficult and you do a lot of them it will get quite unreadable. This is the right moment to think about overloaded operator extensions for DoubleProperties. With these, the last line can become as simple as
mean << a + b / 2
Extension methods and operator overloading can also be used to add the missing APIs for geometry calculation, e.g. to apply a Transform to a Point3D or to accumulate Transforms.

Monday, November 19, 2012

Xtext tip: How do I get the Guice Injector of my language?

This post is about one of the most frequent mistakes people are making when starting to use Xtext beyond the basic functionality. I stumble across this in our newsgroup about three times a week.

The problem usually starts by the necessity to load a file in your language in the UI, lets say in a custom action. You already know that in Xtext everything is wired up with dependency injection (DI), and that you need a DI configured XtextResourceSet to load a model. So you start looking for an Injector to create your XtextResourceSet and find that one is returned by the generated [DSL]StandaloneSetup.createInjectorAndDoEMFRegistration(). This seems to work fine first, but in the followoing you'll get a lot of strange errors. Things that worked before are suddenly broken.

A variant of the error – more popular among former oAW users – is to run a generator MWE2 workflow containing such a [DSL]StandaloneSetup from the UI.

Never ever use the [DSL]StandaloneSetup for your language within Eclipse!


Why not? Xtext relies on EMF and EMF uses on a bunch of global registries to work. E.g. the EPackage.Registry for the implementation classes for EPackages or the Resource.Factory.Registry for parsers for model files. These are global singletons, and they are usually populated by means of Eclipse extension points, e.g. org.eclipse.emf.ecore.generated_package or org.eclipse.emf.ecore.extension_parser. If you run a standalone Java application – the term standalone means without equinox here – there is no concept of extension points. That means you have to register the DSL to EMF's registries yourself. This is what the [DSL]StandaloneSetup is meant for. If you're using it within Eclipse, you'll overwrite the contributions from the extension points with new instances from the standalone setup. This will crash your application. 

It gets even worse: Xtext has a couple of singletons, too, but they are not global but scoped within the (singleton) Injector of your language. If you just create a new Injector, you'll end up with two instances for all of these singletons. This will likely hit you badly.

Finally, some Xtext components have different implementations for Eclipse and for standalone mode. E.g., Xtext's Java reflection layer (aka JVM model) uses JDT's Java model in Eclipse and Java Reflection in standalone mode. The Injector from the standalone setup will register different implementation classes, thus also breaking functionality.

So how do I use dependency injection in Eclipse properly then?


The Equinox aware Injector you're looking for can be obtained from the Activator of your language (in the UI plug-in). But it would be bad advise to just tell you to go there. You should rather use DI properly. If you think about where to get the Injector you're most often already on the wrong track.

A central aspect of DI is not to care about object creation at all, but just declare dependencies (via @Inject in Guice). Every dependency that is injected will also be created by the Injector so its own dependencies will be created on-demand and injected recursively. As a result, in a well-designed application using DI, there is only one entry class that is explicitly created or populated by the Injector and the rest is automatically DI aware as it is injected somewhere.

IDE components in Eclipse are usually registered by contributing to an extension point with an executable extension, usually an attribute with name class, for example

   <extension point="org.eclipse.ui.editorActions">
      
      <editorContribution targetID="..." id="...">

         <action
 class="[some class extending IActionDelegate]"
  
   ... 

Xtext generates a so called executable extension factory for you that will instantiate the given class using the Injector, so use

  ... class="[DSL]ExecutableExtensionFactory:[some class name]"

and the class will be generated with all its dependencies injected. This is also covered in the documentation.

If there is no such extension point, and your class is not yet instantiated using DI, you might have to access the Injector via the Activator and use it to inject all dependencies, i.e.

   [DSL]Activator.getInstance().getInjector("").injectMembers(this);

Note that you should only have to do this for the entry class, from which all other components are injected. If you have to use the Injector, make sure it is only once.

How do I use DI in tests?


There is a whole section on using DI in tests in the documentation. I am not going to repeat this here. The base configuration requires using JUnit4 and is automatically generated for your language has been created with a recent version of Xtext. If not, make sure to add

   pathTestProject = "[DSL].tests" 

in the section on paths in your MWE2 workflow and

   // generates junit test support classes into Generator#pathTestProject
   fragment = junit.Junit4Fragment {}
 


to the fragments and regenerate. This will create the preconfigured test plug-in. Use the InjectorProvider for standalone tests and the UiInjectorProvider for plug-in tests. Once again, your code should not show any references to an Injector.

And the Injector of another language?


Sometimes, you want to switch to a service implementation from another language, e.g. when your DSL allows cross references to another. Usually, you are in the DI context of one language and have an URI to a resource or element of the other language at hand. In this case use the IResourceServiceProvider.Registry:

   @Inject IResourceServiceProvider.Registry reg;

and use that to get the component, e.g. to get the default ILabelProvider

   ILabelProvider otherLangLabelProvider = 
      reg.getResourceServiceProvider(otherLangURI)
           .get(ILabelProvider.class);

Thursday, July 12, 2012

Seven Languages By Seven Geeks

One of the most powerful new features of Xtext is the capability to build languages for the JVM. In my opinion, these languages are a real breakthrough in domain-specific language design:
  • These DSLs directly interoperate with Java. They compile to Java code and allow to refer to Java artifacts.
  • They not only cover structural aspects but also behavior. DSLs become really powerful with a rich expression language offering today what Java has been missing for years, like type inference, lambda expressions, extension methods etc.
  • Complicated integration patterns like Generation Gap or Protected Regions become obsolete. Everything can be expressed on the same level of abstraction
  • They offer top-notch Eclipse-based tooling and integrate into the Java IDE. Switch between the DSL and the Java code during debugging, refactor across multiple languages, navigate to source etc.
Yet we have the impression that the concept and the benefits of JVM DSLs have not been well understood and adopted. This is why we decided it was time for another example and documentation offensive. The result is the 7 Languages project: Seven example JVM languages highlighting the benefits of this approach in various use cases of Java development:
  1. A scripting language. Write code without any boilerplate.
  2. A build language similar to Gradle but statically typed.
  3. JavaBeans facades for mongoDB objects.
  4. A DSL to define Guice modules.
  5. A language to route HTTP requests with conditions, variables and dependency injection.
  6. A template language
  7. A LOGO-like turtle drawing environment.
Clone or fork the code from github and enjoy the documentation on theXtext website.

Wednesday, March 28, 2012

EclipseCon 2012 - I have had my share

I am at EclipseCon 2012, this year at Reston (VA). I have just finished my parts and can now relax and take it easy for the remainder of the conference.

Yesterday I gave a a tutorial on "DSLs for Xtext Developers" together with Sebastian. Around 50 participants learnded how to wirte their own DSLs that run on the JVM and integrate tightly with Java. Looking at the feedback so far, this was a real success, even thought it was a whole lot of stuff. Have a look at the slides:





Today I talked about "A Fresh Look at graphical Editing". In comparison to the talk at EclipseCon Europe, I changed the focus a bit and spent more time to make clear, why most graphical model editors really suck. This time I seem to have made my point, at least noone has asked me so far to convert the Generic Graph View framework into a model editor. The new features, like multitouch gestures and discovery diagrams went fine, too. Here are the slides