DSL developement through language embedding and extension

Size: px

Start display at page:

Download "DSL developement through language embedding and extension"

Ashlee Eaton
6 years ago
Views:

1 DSL developement through language embedding and extension February 5, 2010 Contents 1 Introduction How language embedding and extension relate Classifying extension techniques Paraphrase Orthophrase Metaphrase Concrete languages Assimilation with MetaBorg A (small) preprocessor language Augmenting Haskell Generalization of modular design 10 5 Conclusion 11

2 1 Introduction Developing Domain Specific Languages (DSLs) often is a process in which compiler specialists and domain experts create the language envisioned by the latter applying restrictions and rules known only to the former. During the developement, many things have to be taken into account by the compiler specialists for which well-known and tested solutions exist - for example typing and scope rules. Modern compiler generators ease the task of applying those techniques to new languages. The idea of language embedding and extension is to even remove those basic tasks from the language developement process by using an existing language with established typing and scoping rules (as well as further functionality) and augment that now so-called host language to express domain specific ideas by means such as modifying the syntax of the host language or integrating already present tools and language constructs. The goal of this paper is to give the reader an overview about the general ideas behind DSL design through language embedding and extension, present some abstract concepts as well as their concrete counterparts and briefly discuss some of their advantages and disadvantages. Embedding techniques such as PHP in HTML are not to be discussed as they don t integrate both languages into a DSL. The main source of information about general concepts are a quite old but classic paper by Standish [Sta75] and a more recent paper by Zingaro [Zin07] that discusses the situation of embedding and extension today. Modern concepts and concrete implementations are taken from [Hud98], [Slo02] and [BGV05]. The paper is structured as follows: In section 2, I will show a classification of embedding and extension techniques which I will use to classify the concrete techniques and languages presented as examples in section 3. Section 4 presents a special concept of generalization proposed by Hudak that takes extension to the extreme. Section 5 concludes the paper. 1.1 How language embedding and extension relate Language embedding and extension are two sides of the same medal. As mentioned above, the aim of both techniques is the augmentation of an established programming language (the host language) such that a domain expert can use the resulting language to express domain specific ideas with it. The idea of language embedding takes the host language and embeds within it a guest language that can express domain specific ideas. Language extension takes the host language and augments it with new syntax and semantics such that the domain specific ideas can be expressed in the new language. In both cases the domain expert is able to express domain specific ideas in the resulting language using the basic structure of the host language augmented by ideas of his domain. The only difference is the view on the augmentation process. Where language embedding requires a pre-existing guest language expressing domain specific ideas, language extension integrates those ideas directly into the host language. An example for a (com- 2

3 pletely fictional) language embedding is shown in listing 1. It uses Java Annotations to declare the annotation language and backticks to include external code fragments. p u b l i c c l a s s MyEmbeddingClass ( " bash " ) p u b l i c v oid embeddedmethod ( S t r i n g x ) { r e t u r n echo $x sed e " s / foo / b a r / g " ; Listing 1: Fictional language embedding Listing 2 shows the same code fragment but in a (again fictional) language called Java- Bash. It tries to capture the essence of bash-programming (namely the concept of pipes and filters) and applies it to standard Java. Both listings are only given to show the main differences and similarities between language embedding and extension. They are in no way meant to be examples for feasible or even working languages. p u b l i c c l a s s MyExtensionClass { p u b l i c v oid extendedmethod ( S t r i n g x ) { r e t u r n s t r i n g T o P i p e x s u b s t i t u t e " foo " " b a r " ; Listing 2: Fictional language extension JavaBash 2 Classifying extension techniques Lexical analysis paraphrase orthophrase Syntactic analysis metaphrase Semantic analysis Figure 1: Extension techniques and compiler phases Nowadays, language embedding and extension comes in various forms but it was in 1975 already, that Thomas A. Standish coined the terms of paraphrase, orthophrase 3

4 and metaphrase [Sta75] that are still in use today [Zin07]. Figure 1 shows how those techniques relate to compiler phases. An in-depth explanation is found below. 2.1 Paraphrase Paraphrase means to define something new by showing how to exchange it for something whose meaning is known [Sta75] and thus adding new features by relying on features already present in the language [Zin07]. Typical cases of paraphrase include user-defined macros and preprocessors. The new language defined by the macro- or preprocessor-engine transforms valid programs in its own syntax into valid programs or program fragments of the target language. Paraphrasive techniques are in most cases applied before or directly after the lexical analysis of the source program. The scanner and parser themselves can be left unmodified. Listing 3 shows paraphrase as implemented in C with preprocessor statements. Before the code is fed into the actual compiler, the preprocessor-statements are evaluated. Every occurence of (the letters) PI is then replaced by # i n c l u d e < s t d i o. h> # d e f i n e PI i n t main ( void ) { p r i n t f ( " Pi i s %f \ n ", PI ) ; return 0 ; Listing 3: Paraphrase - The C preprocessor Advantages and disadvantages An advantage of paraphrasive techniques is that they require relatively little knowledge about the internals of the compilation process. The actual compiler is to a great extent left untouched and the preprocessing or macro extension is handled by an external tool. A big problem when using paraphrase is the fact that the transformation step often has no knowledge about the structure and semantics of the target program in which the generated code will be inserted. This may result in unwanted behaviour or binding of variables if the transformation introduces names that are already bound in the scope the fragment is to be inserted into. Listing 4 shows a program in which the textual replacement leads to the wrong result because 5*PI_PLUS_ONE is replaced by 5* and the multplication has a higher precedence than the addition. There are more such problems with textual replacement and the C preprocessor community has come up with various tips and suggestions for avoiding such problems. More information about the C preprocessor and common problems with its usage can be found in [EBN02]. The terms hygienic and unhygienic expansion are also used in this context. 4

5 Extension systems that are hygienic have some mechanism for avoidance of problems as mentioned above whereas unhygienic techniques don t bother with those problems and leave that to the user. # i n c l u d e < s t d i o. h> # d e f i n e PI_PLUS_ONE i n t main ( void ) { p r i n t f ( " 5 piplusone = %f \ n ",5 PI_PLUS_ONE ) ; return 0 ; Listing 4: Paraphrase gone wrong 2.2 Orthophrase Orthophrase means adding orthogonal features to a language where an orthogonal feature is a feature which lies outside the space of features expressible by paraphrase. Its defining expression cannot be composed in the language [Sta75]. It thus means adding features not expressible in terms of available language primitives [Zin07]. The feature has to be added via manipulation of the language definition itself. In most cases, orthophrasive techniques require a modification of all parts of the compiler toolchain, as no syntax or semantics can be reused. The term orthophrase is mainly used for low-level extensions of languages such as adding IO-features to languages which don t support direct access to the filesystem. (e.g. to the formula language of modern spreadsheet applications) Advantages and disadvantages Orthophrase seems to be the most powerful approach to language extension as it enables the user to specify completely new syntax and semantics - preferrably for his domain. The big disadvantage is, that orthophrasive techniques require heavy modifications of most of the compiler phases and the corresponding tools. As orthophrasive techniques mostly aren t easily and quickly applicable, they are irrelevant for easy DSL design. 2.3 Metaphrase Metaphrase means altering the interpretation rules of a language so that it processes old expressions in new ways [Sta75]. In most cases, these techniques alter the context-free grammar of a host language by adding or changing productions such that domain specific concepts are expressible in the resulting language. Most metaphrasive techniques try to find a language that meets most of the users needs regarding general functionality 5

6 and add only domain specific parts to the expressible ideas. The host language should be selected such that no orthogonal features have to be added - otherwise orthophrase would have to be used. The concrete languages discussed in this paper use mainly metaphrasive techniques Advantages and disadvantages The main advantage of metaphrasive techniques is, that most of the host language can be left untouched and can be reused in the new language. A disadvantage is, that the use of those techniques requires insight into the inner working of the compiler. 3 Concrete languages In this section, I will introduce some concrete extension and embedding techniques for DSL design, explain their main concepts regarding the above classification and explain their concrete advantages and disadvantages. 3.1 Assimilation with MetaBorg Bravenboer et al. propose the MetaBorg method as a general way of providing domainspecific notation for domain abstractions [BGV05]. They integrate two languages completely, so that the resulting language uses notations of both parent languages. Their idea is the approach of assimilation which means that a guest language is not only embedded in a host language but that inside the embedded code fragments it is possible to use notations of the host language thus allowing a host language (collective) to incorporate and assimilate external domains (cultures) [BGV05]. MetaBorg can be regarded as a two-way metaphrasive system as two existing language are integrated very tightly by modifying and combining the grammars of both languages. The technique works on the syntax level by specifying the exact symbols and productions that have to be modified for the assimilation. An example given in [BGV05] combines Java and regular expression syntax to enable Perl-like use of regular expressions. Java has support for regular expressions but saves them in Strings which under some circumstances require heavy backslashescaping which makes more complicated expressions quite unreadable. The new language is called JavaRegex and allows programs like the one below (example taken from [BGV05]) in which a regular expression for valid IP-addresses is defined and bound to the identifier ipline. Later, this identifier is used with a String-variable input and the new ~?-Operator to check if input matches the regular expression. regex ipline = [/ (([0-1]?\d{1,2\.) (2[0-4]\d\.) (25[0-5]\.)){3 (([0-1]?\d{1,2) (2[0-4]\d) (25[0-5])) 6

7 /] ; if( input ~? ipline ) System.out.println("Input is an ip-number."); else System.out.println("Input is NOT an ip-number."); The tight integration is achieved by first renaming all non-terminals of both langauges (e.g. by prefixing them with letters such that all Java-symbols begin with the letter J and all Regex-symbols begin with the letter R) and then specifying the concrete productions that are to be added and/or modified. The language Stratego/XT is used to specify those transformations. To check the validity of JavaRegex-code, the compiler needs the syntax definitions of valid Java code, the syntax definition of valid regular expressions and information about which productions in both definitions have to be modified to allow for the integration. The actual compilation process of JavaRegex works much like a preprocessor by converting the regular expressions to strings and the ~?-Operator to Java API calls. The advantage of MetaBorg is that it - unlike the C preprocessor - works on syntax level. Error messages can now be displayed in context of the code the user sees and not the generated Java code. 3.2 A (small) preprocessor language The C preprocessor is a paraphrasive system that works on the source text even before it is fed into the syntactic analysis phase. It augments the C language and its derivatives by the possibility to define macros and include other files. The source code itself can now contain terms defined to be replaced and as such can be viewed as source code of an extended language. The C preprocessor is unhygienic and many attempts have been made by C programmers to avoid binding collisions or even unwanted behaviour as mentioned in section The following code shows a small DSL which allows to define attributes for C++ classes in a simple manner. The user of the DSL only has to write attribute(name,type,format) in his class body where name is a valid identifier, type is a valid type and format is a valid character representing the formatting of the attribute in a printf-statement. A small paraphrasive DSL # d e f i n e g e t t e r ( name, t y p e ) t y p e g e t ##name ( ) \ { r e t u r n t h i s >name ; # d e f i n e s e t t e r ( name, t y p e ) void s e t ##name ( t y p e newval ) \ { t h i s >name = newval ; # d e f i n e p r i n t e r ( name, type, f o r m a t ) void p r i n t ##name ( ) \ { p r i n t f ( " ( " # t y p e " ) " # name"=%"# f o r m a t " \ n ", name ) ; # d e f i n e a t t r i b u t e ( name, type, f o r m a t ) p r i v a t e : t y p e name ; \ p u b l i c : \ 7

8 g e t t e r ( name, t y p e ) ; \ s e t t e r ( name, t y p e ) ; \ p r i n t e r ( name, type, f o r m a t ) ; c l a s s r e c t a n g l e { a t t r i b u t e ( width, i n t, d ) a t t r i b u t e ( h e i g h t, i n t, d ) ; 3.3 Augmenting Haskell Hudak [Hud98] proposes a technique in which he uses the functional programming language Haskell as host language and augments it via its own notation. The technique is used by Sloane [Slo02] to re-implement a given language called Odin while trying to keep the new language as close to the original language as possible. For us as software engineers, this technique seems to only define a special API - but if one looks at the resulting notation, it is clearly visible that a domain expert seems to get a language from his own domain. In Haskell it is possible to specify functions and datatypes. This built-in feature is used to define the syntax for a new, domain specific language. The code written in this new language is still perfectly valid Haskell code (given the Haskell functions and datatypes defined by the language designer) but can be understood and written by a domain expert as if it was in a language designed only for his domain Points and Regions Hudak presents a small example language that enables the domain user to work with points and regions as well as intersections and disjunctions of regions. The main datatypes used for calculations in that domain are Point, which represents a single point in the plane (with given x- and y-coordinates) and Region, which represents a region in said plane. A Region is modeled by a function Point -> Bool which returns true iff the given Point is in the given Region. For the specification of circles, another datatype Radius is used, which is simply another name for a real number. The language comes from a naval background and as such can only specify circles around a fixed midpoint (the ship). The functions used on the given datatypes are shown below. Their signatures are in Curry-form. 8

9 Functions in augmented Haskell Function Definition Description inregion Point -> Region -> Bool Checks if a given point lies in a given Region circle Radius -> Region Creates a circular Region with given Radius (/\) Region -> Region -> Region Calculates the Regions that results by intersecting to Regions (\/) Region -> Region -> Region Calculates the Regions that results by combining to Regions The user can now use domain notation such as p r s to check if point p is in the intersection of regions r and s. In the new language, this query looks as follows: p inregion r /\ s. It is evaluated as shown below. It is very nice to see how the domain specific notation (using points and regions) comes closer to the software engineer s view on the language (using functions) with every transformation step Pictures and Animations Evaluation of a simple expression p inregion r /\ s = (r /\ s) p r p && s p Another example by Hudak relies on functional concepts even more. He defines a datatype Picture which represents a picture. Functions to create and manipulate those pictures are the following (defined in [Hud98]): -- Atomic objects: circle -- a unit circle square -- a unit square import "p.gif" -- an imported bit-map -- Composite objects: scale v p -- scale picture p by vector v color c p -- color picture p with color c trans v p -- translate picture p by vector v p1 over p2 -- overlay p1 on p2 p1 above p2 -- place p1 above p2 p1 beside p2 -- place p1 beside p2 Hudak captures the notions of the animation domain by defining the datatype Animation as Time -> Picture - an animation is a function from time to pictures. As not only 9

10 animations have that property, he specifies a more general Behavior for things that can change over time. An animation is now only a special case of a behavior. type Behavior a = Time -> a type Animation = Behavior Picture Hudak then proceeds to lift the operators used for basic picture manipulation to the behavior/animation level by specifying (b1 overb b2) t = b1 t over b2 t (b1 aboveb b2) t = b1 t above b2 t (b1 besideb b2) t = b1 t beside b2 t (scaleb v b) t = scale (v t) (b t) (colorb c b) t = color (c t) (b t) (transb v b) t = trans (v t) (b t) The new operators are now applicable to animations instead of pictures but reuse the underlying operators on pictures to bring their functionality to the animation level. They capture the animation domain by adding a layer that represents behavior (change over time) to the picture domain. It is now possible to add even more layers of concepts by lifting the operators to those levels. In this modular concept it might be possible to describe 3D-animations by first extending the picture domain to a 3D-body domain and then by behavior. This approach leads to the generalization mentioned in section 4 4 Generalization of modular design The idea of (modular) language extension as mentioned in section can be taken to the extremes. Such an approach called Modular Monadic Interpreter is proposed by Hudak. In regular interpreted languages, a term is mapped to an answer. The monadic interpreter maps terms first to computations (called monads, which can easily be expressed in Haskell using the do{-notation) and then to answers. As an example, a regular interpreter would map the term 3*4+1 to the answer 13, whereas a monadic interpreter maps it first to arith(3*4+1) and then to 13 where arith is a function that can calculate results for arithmetic expressions. If the user now wishes to add functionality such as variables or function definitions, he has to specify a function to be used instead of arith. He can do so by transforming the given arith function with a higer-order-function and thus preserving its functionality in his new function. The arith-module can be reused in certain parts of the variable-enabled language, but new concepts need to be added. A new var-function has to be designed, which is a higher-order-function that transforms another function (the monad for arithmetic expressions) into a monad for language that can handle arithmetic expressions using variables. The term 3*a+1 now maps to var(arith)(3*a+1) and then to the result. The next 10

11 layer, function definitions, would map to the result via func(var(arith))(3*a+1). Layer after layer, such transformation steps can be applied to create a more powerful language and its interpreter in every iteration. 5 Conclusion There are many approaches to language embedding and extension which all share the idea of reuse. In most cases, the structure or even the compiler of a host language is reused so that already present solutions for which there exists no need to be adjusted to the problem domain can be used as a part of the new language. Metaphrasic techniques are to be mentioned as having the best tradeoff between ease of implementation and gainable expressiveness of the resulting language. Here, I presented and discussed only some of them. References [BGV05] Martin Bravenboer, Rene De Groot, and Eelco Visser. Metaborg in action: Examples of domain-specific language embedding and assimilation using stratego/xt. In Participants Proceedings of the Summer School on Generative and Transformational Techniques in Software Engineering (GTTSE 05). Springer Verlag, [EBN02] Michael D. Ernst, Greg J. Badros, and David Notkin. An empirical analysis of c preprocessor use. IEEE Transactions on Software Engineering, 28: , [Hud98] [Slo02] [Sta75] [Zin07] P. Hudak. Modular domain specific languages and tools. In ICSR 98: Proceedings of the 5th International Conference on Software Reuse, page 134, Washington, DC, USA, IEEE Computer Society. A. Sloane. Post-design domain-specific language embedding: A case study in the software engineering domain. In HICSS 02: Proceedings of the 35th Annual Hawaii International Conference on System Sciences (HICSS 02)- Volume 9, page 281, Washington, DC, USA, IEEE Computer Society. Thomas A. Standish. Extensibility in programming language design. SIG- PLAN Not., 10(7):18 21, Daniel Zingaro. Modern extensible languages. Technical Report 47, McMaster University SQRL,

The role of semantic analysis in a compiler

Semantic Analysis Outline The role of semantic analysis in a compiler A laundry list of tasks Scope Static vs. Dynamic scoping Implementation: symbol tables Types Static analyses that detect type errors