FXDiagram goes IDEA

You may have heard already that the Xtext team is currently porting Xtext and Xtend to IntelliJ IDEA. To get acquainted with the new APIs I decided to spend a few hours of my spare time to port FXDiagram as well. Here is a screencast of the first shot with a class diagram for Java code:



As the core of FXDiagram is independent of Eclipse, this was easier than expected. IDEA is a Swing (!) application, and the JFXPanel allows to embed JavaFX controls within Swing. The hardest part was actually to use Java as Xtend is not yet available on IDEA ;-)  FXDiagram's configurable model access made it easy to integrate with IDEA's PSI model and its transactions.

I must admit that the overall experience is a bit snappier in IDEA than in Eclipse. I assume this is because the Oracle team has spent more effort on the JFXPanel than on the FXCanvas. Unfortunately, multi-touch gestures don't work yet, but I had to port them to the the FXCanvas myself as well.

Graphical Views for Xtext ctd.

(Read my previous post for a rationale on graphical views)

No matter with what graphics technology you choose to implement a diagram for your Xtext-based language, there are a few things you should know in order to connect the diagram view/editor to the rest of the infrastructure.

Let us first have a look how to implement a context menu action for the Xtext editor to show the element at the current cursor position in the diagram. In Eclipse, you have to implement a handler for such an action. Xtext offers a bunch of classes making your life easier. EditorUtils detects the Xtext editor for the handler’s action and EObjectAtOffsetHelper finds the semantic model element at a given text offset.

As the document – and thereby the model parsed from it – is subject to editing, you have to make sure nobody is changing it while you traverse it to derive your diagram. You can execute code in a read transaction on the XtextDocument by wrapping it in an IUnitOfWork that you pass to the XtextDocument#readOnly method.

The following snipped shows an example handler written in Xtend.

Note that the @Inject annotation in the example requires you to use your language's executable extension factory when you register the handler class it in the plugin.xml of your language's UI plug-in. See the Xtext documentation for details.

To navigate from a diagram element back to it’s element definition in the text, you need to store some back reference. The model element itself is not suitable, as its lifecycle is bound to the one of the editor, and the parser is even free to expunge model elements whenever the document is re-parsed. As Xtext is based on EMF, each model element has a unique URI. You should always use these URIs to store references to model elements beyond the boundaries of a read/write transaction. An element’s URI is provided by EMF's EcoreUtil#getURI method.

Having the URI, the navigation to the respective model element’s definition is easiest to implement using Xtext’s IURIEditorOpener. If your diagram view supports selection listeners, the respective Xtend code could look like this:

Graphical Views for Xtext

Xtext provides you with a powerful IDE and a rich featured text editor for your domain-specific language with little effort. But sometimes, a picture says more than a thousand words: You want to have some additional graphical representation of your models, a set of diagrams.

Diagrams are superior to code when it comes to high-level views. But while programmers can easily cope with files that contain several hundred lines of code, the same amount of information usually blows a diagram and destroys all its suggestiveness. Diagrams with just a few nodes and edges showing on a certain aspect of the models only are best fit for human readers. Such aspects are often spread across various model files. So in order to add the best value for the language users on top of an Xtext infrastructure, we need to allow them to create multiple diagrams, each highlighting just a subset of model information, picked from multiple model files. In other words, diagrams that have a completely different structure than their associated textual models.

Traditional graphical editing frameworks focus on editing the underling model through the diagram.
But synchronizing the model changes from textual and diagram editors is very hard if their content's structures differ. In most cases, the integration will lead to major usability quirks like unexpected editor behavior, forced save operations, blocked editors or even data loss.

So rather than working around the hard challenges of integrating graphical and textual model editing, we can leave model modification to Xtext. For the graphical stuff we can concentrate on diagram editing and leave the underlying model read-only. This way we can spend our energy on the things that really matter to the user, like easy and useful ways to populate diagrams or best visual appearance.

In Eclipse, one could build such graphical views using Zest or GEF. If model and diagram do not differ too much in structure, a small code generator targeting GraphViz is a very simple solution with high quality output.

The general integration with Xtext is covered in a follow up post.

The following screencast shows another solution for an Xtext-based domain model language. The graphical editor is using FXDiagram, a diagram framework based on JavaFX. FXDiagram offers a very smooth user experience, excellent rendering, support for touch-pad gestures, animated undo/redo, diagram persistence, export to scalable vector graphics and much more. If you are interested in learning more, feel free to contact me.

Eclipse Diagram Editors From A User's Perspective

I believe that the Eclipse's diagram editing frameworks have a lot to catch up when it comes to usability. For illustration I've created a screencast on how a typical Eclipse graphical editor appears to a user. I have used the Ecore Tools editor as an example, but most issues presented are caused by defaults of GMF or GEF which are pretty hard to change. So don't be surprised if you re-encounter these or similar issues in Graphiti, Sirius etc. The screencast doesn't even cover the rendering errors described in my previous blog post.



So what has gone wrong?

In my opinion, diagram editor behavior and editing metaphors haven't really changed since the first commits on the GEF project in 2002. Advances in UI design and usability have had almost no priority on Eclipse's graphical frameworks since then. They were built from a pure developer's perspective. So a lot of effort has been spent on:

• Facilitating the development of new editors. This usually causes a restriction to very specific use cases. 
• Establishing weird processes to implement new diagram editors. 
• Repeatedly putting new abstraction layers on top of the existing ones. Hide the seemingly ugly or incomprehensible concepts of a base framework. The problem is to find better abstractions that don't leak. In the end you often have to learn both (or more) frameworks in order to use them. 
• Building on top of existing frameworks with serious limitations, like integer coordinates, missing alpha channel, no hardware acceleration etc. 
• Supporting very large diagrams. Yes, there are very big models. But diagrams with hundreds of nodes and connections only make good wallpapers.
• Creating new compatibility APIs, e.g. to replace the rendering engine. These will limit the available functionality to the common minimum set of features of all backends. 
• Changing a model graphically. Note the difference between editing a diagram – e.g. to make it look nicer – and changing the underlying model. Did you ever wonder why GEF Zest is the only Eclipse diagram framework with its own diagram layout algorithms and basic support for multi-touch gestures? 

Computers don't need diagrams. Diagrams are for humans. Their strength is to explain a part of a model in a very suggestive way. Taking the human user into focus changes the requirements for a graphical framework drastically. Issues like the following must have top priority:

• Appealing visual design, 
• Intuitive editing metaphors, 
• Snappy behavior, 
• Support for modern input devices, and 
• The possibility for individual customization. 

How do we get there? Attend my talk Eclipse Diagram Editors - An Endangered Species at EclipseCon Europe for some propositions and an extensive demo.

Obi-Wan at the Edge

There are two hard problems in computer science: cache validation, variable substitution, and off-by-one errors.      (unknown source)

The uncounted latter is often referred to as OBOE or, with a geek sense of humor, "Obi-Wan error".

Obi-Wan In Eclipse Diagram Editors

In an older post, I already mentioned one major weakness of GEF (3.x): All coordinates are integers. One problem with that is that it can easily result in off-by-one rendering bugs. That unfortunately holds for many frameworks built on top of GEF. Have a close look at the gradient fill, anchor points (where connections are touching the nodes), clipping bounds and the arrow head in the following screenshots from the standard examples:

Integer vs. double coordinates is comparable to raster vs. vector images. While in integer coordinates a point usually means a pixel, in double coordinates a point has no area and a line can be infinitesimally thin. In a world of double precision it is obvious that we need some rendering. We are forced to think about things like line width, stroke type, line caps, line joins, miters, etc. There are excellent explanatory pictures on these in the JavaDocs of JavaFX's Shape class.

"Smooth" Curves

You have to take round-off errors into account when approximating smooth curves (splines) with integer polylines. Otherwise your allegedly smooth curve will go zigzag. This might be the reason why GEF does not have any curves. GMF does:

 

Arrow Heads

I'd expect an edges arrow head them to be aligned with the tangent of the edge at the anchor point. For polylines this is the line connecting the anchor with the last bendpoint. Same for Bézier curves, but apparently not for splines like these:

  

It is also a good idea to prohibit bendpoints too close to the anchor: The inclination of the curve may become too steep:

The last screenshot also shows that the line cap style plays a role at the tip of the arrow head.

 

Use the Force

GEF 4 will have double coordinates and Bézier curves. But at the time of writing this blogpost, there hasn't been an official release yet. It is not clear either whether and when it will be adopted by downstream projects.

I came across all this when trying to calculate edges and arrow heads for my sparetime JavaFX diagram editor project. Luckily, JavaFX uses double coordinates only. JavaFX also includes quadratic and cubic Bézier curves. These curves are really smooth and drawn fast, though I must admit I'd  have expected B-Splines at least.

I am going to demonstrate this at EclipseCon Europe 2013. Stay tuned for more.

Accumulating JavaFX Transforms With Xtend

I was a bit vague in my last post on JavaFX's transformation API, so I decided to write a separate post on this topic.

In JavaFX, each node in the scenegraph can be translated, rotated, scaled or sheared relative to its parent. In mathematical terms, each node maintains a transformation describing its own local coordinate system. This transformation can be defined with a matrix represented as a Transform object in JavaFX.

The JavaFX API offers some convenience methods to manipulate this transformation. E.g., you can scale a node in x-direction and translate it by 42 units in y-direction by calling

  node.setScaleX(2);
  node.setTranslateY(42)

Unfortunately, these convenience methods do not accumulate as one would expect. Instead of multiplying the matrices for subsequent transformations, the only set specific entries in the matrix. As a result, translating an object before rotating yields the same result as applying these transformations in reverse order. This is mathematically wrong and does also not match the users expectations.

In my diagram editor application, I want to scroll (translate), zoom (scale) and rotate the canvas using mouse gestures. From the user's point of view it's imperative that each transformation builds on the previous state. The convenience methods don't match this usecase.

The correct solution to this problem is to multiply the transformation matrices. Unfortunately, JavaFX lacks any kind of calculation API for Transform and its subclasses.

Once again, this is where Xtend comes to our aid. An extension method in Xtend can be used to define functions for existing (closed) types, which syntactically look like being methods of the type on the caller's side. Affine is a subtype of Transform that allows its matrix entries to change. So I wrote extension methods to translate, rotate, scale and shear an existing Affine by multiplying the respective transformation matrix, e.g.

class TransformExtensions {
  def static scale(Affine it, double x, double y) {
    // left multiply a scale matrix to it
    // highly optimized as there are many zeros in the scale matrix
    mxx = x * mxx 
    xy = x * mxy 
    mxz = x * mxz
    tx = x * tx // take existing translation into account 

    myx = y * myx
    myy = y * myy
    myz = y * myz
    ty = y * ty 

 }
}

Importing these as extensions

import static extension ...TransformExtensions.*

now allows to accumulate the transformations, e.g.

val diagram = scene.root
val diagramTransform = new Affine
diagram.transforms.clear
diagram.transforms += diagramTransform

val EventHandler scrollHandler = [
  diagramTransform.translate(deltaX, deltaY)

] 
scene.onScrollStarted = scrollHandler 
scene.onScroll = scrollHandler
scene.onScrollFinished = scrollHandler
  
val EventHandler rotateHandler = [
  diagramTransform.rotate(angle, sceneX, sceneY)
] 
scene.onRotationStarted = rotateHandler
scene.onRotate = rotateHandler
scene.onRotationFinished = rotateHandler
...

The resulting behavior looks like the following screenshot. Note that the mouse position is the pivot for rotations and zoom. 

 

The same mechanism can be used to properly place labels along connections:

PS: I went for mutable transformation matrices, but the same mechnism will hold for immutable matrices. The extension methods then return a new Affine instead of modifying the receiver.

Impressions on GEF

For the Generic Graphical View project I've been digging deep into the Graphical Editing Framework (GEF) lately. GEF is the base technology below almost any graphical framework in the Eclipse environment. I wanted to keep the tool stack as low as possible and have the maximum control on everything so I decided to use GEF directly. Here are my impressions.

I really like GEF's base architecture: The separation between Draw2D and GEF, the concepts of Tools, Requests, EditParts and EditPolicies. It helps you to find the component to change in order to implement a new feature quickly. The pictures in the GEF online help really help
understanding this.

Pecularities

In some areas GEF appears to be kind of rusty. A whole bunch of relevant commercial products has been relying on it for years. Long term backward API compatibility has taken its toll. Some things definitely need a serious overhaul and often a better naming. I am very happy to hear that a 3.0 version is planned. Here is my wishlist for improvements:

• Make coordinates double by default. Integer coordinates often result in rendering glitches due to rounding errors and are not well suited for scaling.
• Add alpha channel (transparency) support.
• Revise scaling of fonts: Sometimes a label's bounds don't match its text's extend.
  
• Rename methods of the update mechanism. I am often confused about the semantics validate(), invalidate(), invalidateTree() or revalidate().
• Rectangular bounds are not well suited for round things like ellipses. Similar to that, clipping child figures only makes sense for Viewports but nothing else. The need of an invisible container figure for side affixed children is really an ugly workaround.
  
• Diagram layout does not take connection labels into account.
• Add support for curve connections.
• The feedback figures - the surrogates when a figure is moved or resized - seem to come from a time where hardware graphics acceleration was pure luxury. If you have nested edit parts you might have to reimplement the nested figure construction twice for nicer feedback.
  
• Using a bit less inheritance and more composition would be fine.
  

Mutable Geometric Primitives

As many other graphics frameworks, GEF code tries to create as little new objects as possible. Geometrical primitives such as points, dimensions and rectangles are therefore mutable and reused often. There are even things as a static Rectangle.SINGLETON for temporary calculation. This pretty much reminds me of good old C++. I am not sure whether such optimization is still necessary. It results in pretty unreadable code:

Point p = figure.getLocation(); 
// p is the location in local coordinates
p.getParent().transformToParent(p);
// now p is in parent coordinates
Point q = p.add(p).scale(2):
// this will also modify p

You see: As the semantics of the variable changes, it is no surprise you cannot find a more expressive name than p. In addition, you have to be very careful if you are dealing with a reference or a copy. Modifying a Translatable (once again bad name, as Dimension also implements this interface) directly can also circumvent an event mechanism. This is really error prone. I'd strongly advise to introduce immutable primitives.

Coordinate Transformations

The way transformation between local, global and parent coordinates are handled is somehow unsuggestive. I've used various 2D and 3D scenegraph-based frameworks but I never got as confused with coordinates as in GEF. For example, if you have a ScalableLayeredPane showing some scaled content, its translateToAbsolute(Point) method will not take the scaling into account while translateToParent(Point) will.

Some Hints

Starting with GEF from zero is hard. You will likely spend a serious amount of time debugging why your diagram doesn't show anything at all. Better copy some basic setup from one of the GEF examples. Here are my favorite pitfalls:

• missing minimum size on figure - figure does not show up at all in some layouts
• wrong coordinate system - figures don't appear where they should, somtimes in nirvana
  
• wrong layout manager, e.g. a plain XYLayout on a FreeformFigure - results in endless layout loop
• reusing a geometry element instead of making a copy - completely unpredictable editor behavior
  
• the default ChangeBoundsRequest does not take local coordinate systems into account

For better diagram layout algorithms the KIELER project offers a good choice.

Multitouch Gestures in the Generic Graph View

In my previous post, I've already described the discovery mechanisms of the Generic Graphical View project: Starting from one node you can step by step reveal other connected nodes in the same model. The model itself is fixed - it is still a view! - but the user can decide herself which subset of elements should appear in the diagram. That behavor allows to create diagram views focussed on a specific semantic aspect of the model.

The challenge is to find a decent UI for nodes wich have several hidden connections. How should a user select any subset of these?

Since the Helios release SWT supports four multitouch gestures: PAN, ZOOM, SWIPE and ROTATE. On my Mac's touchpad these gestures are actually easier to perform than a conventional drag operation because there is no extra mouse button. So I decided to give it a try and have a look how gestures could be used in the graph view. Here is a screencast showing the results:



SWT's multitouch API is fairly simple (example snippet). The first step was to add multitouch events to GEF, such that the can be handled in the same way as mouse or keyboard events. I had to extend a couple of GEF classes. If you're interested in the details, have a look at the classes in the package org.eclipse.xtext.graphview.behavior.gestures and the GraphViewRootEditPart.

These changes allow to use a ZOOM gesture on a ScalableFreeformRootEditPart.
Note that the PAN gesture is already supported by GEF because the FigureCanvas uses hardware scrolling. Unfortunately, these event is always consumed, such that it will not make it through to other GEF elements.

The ZOOM gesture can also be used to drill down into elements. If the mapping defines an open reference to another diagram, you can drill down by selecting it and performing the ZOOM in gesture. If you‘re inside a drill down diagram, you can ZOOM out into the container diagram.

Any Tool can now handle multitouch gestures by implementing the IViewerGestureHandler interface. For a DragTracker this makes no sense, as it is exited as soon as the mouse key is released and you cannot start a drag and a multitouch gesture simultaneously.

The RevealGestureTool works as follows: Click on the + button to reveal the hidden connected elements, Then use ROTATE or SWIPE to rotate elements, ZOOM to change the distance and right-click to select and pin individual elements. Another click will commit the input.

Discovery Diagrams for the Generic Graphical View

If you've read my previous posts or seen my presentation at EclipseCon Europe 2011, you'll already know the Generic Graphical View framework: Using two Xtext-based DSLs - one for mapping semantic to graphical elements and the other for styling the figures - you can easily create nice looking diagrams for any Java based model.


In its first version the contents of the diagram was entirely defined in the mapping DSL. It was only possible to restrict the visible elements to a subset of the model by adding filtering expressions in the mapping. For bigger models it was hard to focus on a subset that is not structural defined but based on semantics, e.g. all classes that deal with addresses in a CRM application.

Many graphical frameworks handle the selection of the visible elements in a diagram by means of dialogs. The user is then forced to switch between keyboard and mouse which is annoying, and these dialogs are often complicated and unintuitive. The goal for the Generic Graph View project was to build a UI for choosing the diagram contents that uses the mouse or trackpad only and avoid dialogs at all cost. The following screencast demonstrates the basic functionality.



As a first step, actions were introduced that allow to hide existing graphical elements - nodes, labels or connections. If you hover over an element, you'll get an popup button allowing to remove it from the view. So you can start with a big diagram and strip it down to what you want to see. Once an element inside a node has been removed, you can restore the node completely using another popup button. You can define elements as initially hidden using a hidden flag in the mapping definition.

In addition to the subtractive approach, you can step by step extend the diagram contents from an initial node. If you're hovering over a node that has hidden connection, several + buttons will popup. When one of these buttons is triggered, the hidden elements and connections will appear in transparent arranged in a circle around the source element. In the same drag operation you can change the distance and the angle of that circle. This behavior will get really cool with multitouch gestures, which I will cover in a separate post.

Eclipse Con Europe 2011

The slides of my talk A Fresh Look at Graphical Editing at Eclipse Con Europe 2011 are online now. If you're interested in the code it's on github, but remember it's a prototype - no tests so far, no support. Unfortunately, there was no cake being served at 5pm, so the pie fight had to be cancelled.

As I have been asked this a couple of times at the booth: I am convined it makes no sense turning the presented view framework into a graphical editing framework. I am not entirely opposed to graphical editors but keep in mind that

1. You're going to need much more than graphical editing frameworks provide (projection, navigation, browsing, ...) I'd estimate a factor of 10-20 in the effort of building a graphical tool as opposed to a textual tool.
   
2. The bi-directional mapping from the semantic model to the graphical one is source of a lot of problems. Not all of them can be hidden from the users.
   
3. The structure of the semantic model must match the diagram model. This is not even true for the simplest models like UML2 class diagrams. You should be in a position to change the semantic metamodel to have a perfect structural match.
   

Would have been interested to discuss the last point with Bran Selic, who claimed in his keynote that UML2 is cool but the tools suck...

Eclipse Con Europe 2011 was an excellent event as always. I really enjoyed meeting all the nice people from the community. Thanks a lot for the organization and especially for the delicious food.

But remember: After the conference is before the conference. The CfP for Eclipse Con 2012 in Reston, Virgina, is open. Hope to see all of you there.

Join My Pie Fight Session at EclipseCon Europe 2011

Competition is hard at EclipseCon: Everybody has to make sure to get the most participants into his talk. As Sven already announced a quiz for the same timeslot as mine I've had to figure out something even more exciting.

So here's my idea for my session:

I'll begin the session ranting on every Eclipse project that's somehow related to graphics. I'll shamelessly reveal all of their weaknesses. Everybody involved in one of these projects will be so offended that they start to get personal, first against me then against each other. When emotions boil up, I expect people to self-organize in teams, like The Knights of GMF Runtime or The Zest'o'Maniacs. My session is at 5pm so I expect a good supply of cake as in the years before. I am starting the fight by throwing the first cake, e.g. on the GMF Tooling Ultras. They try to answer but unluckily hit a member of the Spray Front, who's already in the midst of a quarrel with a Graphiti Warrior. Thinking it was a GEF Cyclope who threw the cake... A real pie fight breaks loose.

Hehehe (demonic laughter)

Well... thinking again... the cake at EclipseCon is usually quite yummy. It would be a pity to waste it in such a way. Maybe I should come up with something more peaceful.

OK, I see, I have to make that differently: I am going to point to some general issues with graphical editing. I'll weigh the cost against the usefullness and show, why an interpreted graphical view is so much simpler to implement and to use than a graphical editor. Second part will be a live demo of a prototype for such a framework.

Maybe I should bring a helmet, just in case someone did not notice my change of plan.

Graphical Syntax View for Xtext

I demoed it so often that I almost forgot blogging about it: In Xtext 2.0 you can now visualize the syntax of your DSLs grammar in a live railroad diagram view. The view supports navigation between the editor and the view and you can export your diagrams to a file. We even have a special representation for unordered groups. Watch this screencast to see it in action:



The syntax graph is implemented in plain Draw2d using a custom layout algorithm. It is technically not related to the generic graphical view I presented in my previous blogpost.

Using Xbase to Define a Generic Graphical View

While textual representations such as code are perfectly suited to process the details of an object model, a graphical view can be very helpful to display the relationships between objects.

Most graphics frameworks in the Eclipse ecosystem provide graphical editors. Usually, the semantic model is mapped to its graphical representation using a hard wired transformation, be it in code (GEF, Graphiti) or in models and a code generator (GMF, the upcoming Spray project). There are two issues here: First, as opposed to a graphical view, a graphical editor requires the mapping to be bidirectional. That constrains the transformation a lot, e.g. requires having the same structure on the graphical as on the semantic side. Second, hard wiring the mapping introduces extra compilation/generation/deployment turnarounds.

I've created a prototype of a graphical view that is configured using two textual DSLs: A mapping DSL that declaratively describes the transformation from semantic objects to graphical elements, and a styling DSL to modify the graphical representation. The graphical view interprets the mapping and the stylesheet on a given input model, giving immediate feedback on changes of these live models.

Both DSLs are implemented in Xtext 2.0 and make heavy use of the new Xbase expression language library. As Xbase uses Java's typesystem, we can refer to any POJO as an input model. Xbase's rich semantics allow comfortably navigating the models references, and its interpreter can be customized easily to fit our needs. The graphics are implemented using plain GEF and Zest layouts.

The following screencast gives you a short demo. (Sorry for the poor video quality, it gets better if you got to full screen mode)