It all starts with Java...
...which is a nice language understood by many programmers worldwide. A giant ecosystem and a vivid community as well as great IDEs are keys to its success.
But Java has a couple of crucial problems: Its syntax is inflexible and very verbose, making fluent APIs the closest you can get to domain-specific languages. Design patterns and structural conventions as JavaBeans have to be manually implemented over and over again with lot of error-prone boiler plate code. Frameworks require a big deal of XML pointing to Java classes, often ignored by the static analysis at compile time. IDEs can help us here, but they are usually limited to scaffolding or a single way of using a framework. The diagnosis is: Java is lacking the right means of abstraction.All technologies presented in the following enable you to define your own set of abstractions on top of Java. The solutions differ in syntactic flexibility, turnaround times, required user skills and multi-platform coverage.
External DSLs (Xtext)
Xtext is a framework for external, textual, domain-specific languages. Xtext makes it easy to build an entire IDE for a completely new language, including your own code generators or interpreters to execute it. The Xtext homepage lists a lot of example projects.When should you use Xtext? First of all, when you need a maximum of syntactical freedom. As Xtext DSLs are external they are defined by their own grammar, thus not constrained by the syntax of a host language.
Second, use plain Xtext if your language has no correspondence to Java or if you want to avoid dependencies on JDT at all cost. Languages that target different platforms at the same time are most likely based on a plain Xtext-based language, too. The technical skills needed to work with a Xtext-based language IDE is relatively low. I personally do not believe in "programming business men". But Xtext has been successfully used for non-technical, purely domain-oriented languages.
As the generated IDE is already very powerful by default and can be customized easily, some people even use Xtext to build the IDE front end for an existing language with an existing compiler.
If you change the grammar, you have to regenerate the language infrastructure and redeploy the language's plug-ins to the user's machines. The latter can be a challenge in some scenarios.
Mostly structural languages with simple expressions only will go smoothly. A few complicated expressions can pragmatically be integrated using the common integration techniques (generation gap pattern, black-box literals, protected regions, etc.). But all of these raise the complexity, break the abstraction, and are hard to maintain. If expressions are the rule and not the exception, consider
Java DSLs (Xtext with Xbase)
Under the codename Xbase, we have developed a rich expression language, that you can embed in your Xtext-based DSLs. It reuses Java's typesystem, can express everything that Java expressions can do and adds even more features like type inference, operator overloading, lambda expressions etc.The default way of using Xbase is to define a transformation from your language to a structural Java model and embed the expressions in the appropriate Java contexts. A generic code generator builds Java code from this model, so no need to build your own. As we generate plain Java code, these Xbase languages integrate seamlessly with hand-written Java code and work for platforms that are sensitive to bytecode manipulation, like Android.
Java DSLs are the way to go for languages that compile to Java and need rich behavior or expressions. These are probably much more than you'd probably expect. Have a look at our seven languages for some examples. There is also an interpreter for Xbase allowing to skip the compilation step on the language user's side.
We have only implemented an execution environment for Java, so it's up to you if you need another. Restricting the available expressions is easy and could help here significantly. You can also add new expressions to the expression language, but the syntactic space is already quite crowded.
Non-technical users are usually ruled out here. You don't want to teach them managers about closures, do you? You still have complete syntactical freedom for your language. But you still have to regenerate the plug-ins and redeploy them on grammar changes as before.
Internal DSLs (Xtend)
Just because Java doesn't offer the means of abstraction you need doesn't mean you have to define a new language yourself. We have created Xtend, a general purpose language (GPL) based on Java but adding a lot of fancy stuff like type inference, operator overloading, lambda expressions, etc. You guessed it: Xtend is based on the Xbase expression library, but also defines the structural parts (types, fields, methods, annotations etc.), adds extension methods, multi-dispatch methods, template expressions, and offers an even better IDE.This is already a mighty toolset and can bring you pretty close to internal DSLs in other languages. Look at our examples from EclipseCon 2012 (especially the SWT, Distances and HTML Builder examples).
One additional leap ahead are Active Annotations (new in 2.4, beta). These annotations allow you to participate in the Xtend to Java compilation. They are particularly useful for implementing patterns, but can be surprisingly powerful in other scenarios, too. Have a look at a few cool examples: Observer pattern, Vistors, JSON classes, a simple REST API, micro benchmarking...
Having everything you need in the language itself gives you short turnarounds as no more regeneration and redeployment is needed. The toolstack stays nicely low. But you're restricted to generating Java and use the syntactical elements of the Xtend language. I would never recommend Xtend to non-programmers, but to everyone fighting with Java.
