Interim Report: Build Your Efficient Programming Language Quickly

Transcription

1 Interim Report: Build Your Efficient Programming Language Quickly Final Year Project Department of Computer Science University of Hong Kong Xiang Ji Supervisor: Prof. Bruno c.d.s. Oliveira January 24,

2 Contents 1 Introduction 4 2 Methodology Pretty Printer Correctness Aesthetic Qualities Work Accomplished Bug Fixes for the Pretty Printing Processor Bug Fix: Type Parameter Mismatch Bug Fix: Spaces Bug Fix: Annotation Processing Error Pretty Formatting Precise Representation of Language Primitives Testing for the Pretty Printer Generator Problems Encountered Finding a Pretty Printing Library Modularity Work Remaining & Plans Pretty Printer Case Study on Ruby Emacs/Vim Plugins Conclusion 17 List of Figures 1 Unformatted Code Formatted Code AnnotatedGrammar PPProcessor PPExpAlg Original Language Parameters JppLib Documentation

3 8 Prettifying Effect Manual override Wadler s Printer Emacs Plugin Vim Plugin

4 Abstract The programming language group at HKU (HKUPLG) aims to develop a language prototyping framework on top of Oracle s new research Graal Java Virtual Machine and its Truffle interpreter. In this project, which is a part of the aforementioned ongoing effort at HKU- PLG, various tools for the framework are developed, including a pretty printer generator and syntax highlighting plugins based on Emacs/Vim text editors for the new languages being prototyped. Currently, the pretty printer generator is nearly complete, having been tested on several sample languages; the syntax highlighting plugins are expected to be developed in the next iteration. Together with case studies using Python and Ruby language, the project aims to demonstrate that it is possible to build efficient programming languages quickly on a large scale, using Graal VM and an advanced design pattern called Object Algebra. Potential future works include other tools such as logger/debugger, and a IDE-like development environment for the framework. 1 Introduction Each year, there are hundreds of new programming languages created for various purposes. However, the creation process is usually complicated and involved, with each component of the language rewritten from scratch, instead of leveraging resources for existing languages. Also, if an interpreter is used for the language instead of a compiler, efficiency can become a serious issue. With the above concerns of extensibility and efficiency in sight, the programming language group at HKU (HKUPLG) aims to develop a framework on top of Oracles new research Graal Java Virtual Machine and its Truffle interpreter project, in order to allow developers build efficient programming language implementations quickly. Extensibility: The framework will provide various ready-to-use language components. In order to build a new programming language, a programmer just needs to pick the desired features for the language, assemble them together just like assembling Lego blocks, add some additional custom features if he/she wishes, and then quickly get a prototype implementation, thus eliminating the need of writing everything from scratch. Such modularity will be provided by Object Algebra 4

5 [1], a design pattern researched and proposed by the supervisor of this project, Bruno c.d.s. Oliveira. Efficiency: With the support of high-efficiency Graal virtual machine and Truffle framework, the interpreted languages will perform with high efficiency close to that of a compiled language.[2] In this project, which is a part of the larger project going on at HKUPLG, various tools for the framework are developed, including: A pretty printer which helps users in the process of prototyping and debugging. Emacs & Vim plugins which provide supports such as syntax highlighting for the new languages designed by users Also, a case study on Ruby language will be carried out in the next phase of the project. The remainder of this report first presents a detailed discussion on the methodology employed during the development of this project. Then, work accomplished and work remaining are described, which detail various functionalities of the pretty printer that we ve implemented so far and the tests we ve performed. In addition, problems encountered and future planning are provided. Finally, the report closes with a summary and information on potential future research directions. 2 Methodology The development of framework tools encompasses a variety of software. The pretty printer and related tools are being developed in Java in cohesion with the framework s existing code; it will also leverage existing Java libraries. The Emacs/Vim plugins will be written in Emacs Lisp / Vimscript respectively and will be relatively more independent. 2.1 Pretty Printer A pretty printer is a tool that formats and prints out application source code from the Abstract Syntax Tree (AST). It will help in producing a clear 5

6 representation of the code a programmer is writing. After pretty printing, the originally unintelligible code in Figure 1 is transformed into well-formatted code in Figure 2. Also, it will help the debugging process. Figure 1: Unformatted Code Figure 2: Formatted Code Since we are providing a language prototyping framework, the developer might create a wide variety of languages to his/her liking. Therefore, a single pretty printer would not suffice for all the languages. The framework shall instead automatically generate an appropriate printer for each new language that the developer creates. The way we achieve it is through Java annotations. Figure 3 demonstrates the developer-written Java interface which specifies the language syntax. After the developer has finished writing, he/she will prepend the to the interface. 6

7 Figure 3: AnnotatedGrammar Then, our pretty printer generator will collect all available interfaces annotated as shown in Figure 4. An analysis will be performed on each of the interfaces, and an appropriate printer will be generated based on the syntax definitions collected during the analysis. Figure 4: PPProcessor An example can be seen in Figure 5, where the appropriate printing method for the add operation in the language Expression Algebra is generated. All the code shown in this figure is automatically produced by our pretty printer generator, without extra modification. 7

8 Figure 5: PPExpAlg Before the project started, there was already a prototype of the annotation processor present in the framework, developed by other members of HKUPLG. However, it had two drawbacks upon which improvements are being made: 1. It was still largely primitive and only works for the simplest type of language. When the target language got more complicated, it produced some error and was unable to generate the printer. 2. The printing format was still crude, which means it was not really a "pretty" printer: The outputted strings were mostly an amalgamation of text and were not really pleasant on the eyes Correctness As the prototype printer was more a concept model than a real, complete piece of software, it contains multiple bugs, which will be discussed in details below. They have been fixed during the development process. 8

9 2.1.2 Aesthetic Qualities A "pretty" printer should not only print the abstract syntax tree correctly, but also do so with style. A good pretty printer should be able to handle line breaks and indentations appropriately. For this purpose, a layout algorithm is needed. There have been several algorithms around, and a Java pretty printing library, JppLib[3], which implements the algorithm of Oppen[4], has been selected as the library to provide such a layout algorithm. The library is integrated into our annotation processor in order to generate printers that are able to print prettily. 3 Work Accomplished 3.1 Bug Fixes for the Pretty Printing Processor Currently, the bugs within the prototype processor are mostly fixed. Those bugs include: Bug Fix: Type Parameter Mismatch Before, the generated printer would always only have one type parameter in its interface signature regardless of the number of type parameters in the algebra interface of the target language. For example, for parameters <E, P> (See figure 6), the generated printer would still produce just <interface _PPMumAlg extends MumAlg<IPrint> instead of <interface _PPMumAlg extends MumAlg<IPrint, IPrint>. This prevented the pretty printer from working on more complicated languages. Now this bug is fixed and the correct type parameters are produced. Figure 6: Original Language Parameters 9

10 3.1.2 Bug Fix: Spaces The generated printer would not produce appropriate spaces. For example, the abstract syntax tree alg.ifnode(alg.booleannode(true), alg.stringnode("it s true"), alg.stringnode("it s false")) should be printed as ( if true "It s true" "It s false"), however the actual output was (iftrue"it s true""it s false"), which means everything was jumbled together. Now the correct output is printed Bug Fix: Annotation Processing Error Whenever there is an annotation in the form = ( SYMBOL form@ * ) "), that is to say, a symbol followed by a list of arguments, the annotation processor reported an index out of range error and failed to generate any code for the printer. Now this error is fixed and the printers can be generated properly. 3.2 Pretty Formatting As mentioned previously, for the pretty part of the printers generated, we resorted to a library JppLib by Martin Geise, which provides a full set of documentation(see Figure 7) and sample usage. Designed with a focus on object-oriented programming style, JppLib uses one single object to record the information accumulated while traversing the document. In order to adapt to such a feature, we had to utilize an object algebra interface IPrint similar to that used in Factory design pattern in our code, so that the printer can properly print the parent node of an abstract syntax tree first, before proceeding to its children nodes. Without this pattern, the printer will always print children nodes first, which will result in erroneous outputs such as (true it is True it is False if) instead of (if true it is True it is False ). After successfully incorporating the library, the pretty printer is now able to correctly insert line breaks and whitespaces into the printed output. As seen in Figure 8a, the original output is all piled together in one line, while the prettified output, as seen in Figure 8b, has linebreaks and whitespaces which make the code much easier for the programmer to read. Notice that in this example, the parameter LINE_WIDTH is set as 20. Varying this parameter will lead to different line-breaking behaviors in accordance with the programmer s 10

11 Figure 7: JppLib Documentation preference. 3.3 Precise Representation of Language Primitives The form of language primitives, such as boolean, can vary greatly from one language to another. For example, in Mumbler language it s #t and #f while in Java language it s true and false. In our framework, a token file is supplied for each language for the parser to correctly identify and convert language primitives from source code into AST representation. Unfortunately, the reverse is not easy, i.e. converting the language primitive from AST back into their original source code form. This resulted in certain inaccuracies in our printer. For example, for Mumbler language, the output (if true resulta resultb) should really be (if #t "resulta" "resultb"). The current solution is to let users manually supply the correct way of printing out language primitives (See figure 9). Some language workbenches (tools for domain-specific language creation) seem to be capable of incorporating language tokens for reserved keywords defined by the developer. As a part of potential future work, we might be able to look into those language workbenches and come up with our an automatic solution which would spare the users of the manual labor. 11

12 (a) Unformatted Output Figure 8: Prettifying Effect (b) Formatted Output 3.4 Testing for the Pretty Printer Generator The pretty printer generator has been tested on two sample languages: ExpAlg which is a simple language involving only arithmetic operations, and Mumbler [5] which is a more sophisticated language resembling Lisp in its design. In the next step it will be combined with the other part of our project carried out by the other group members, namely a case study attempting to implement Python using our framework. It will be tested on our implementation of Python once the implementation is complete. 4 Problems Encountered There were several main difficulties which were dealt with in the process. Setting up the Graal/Truffle framework was actually quite complicated. Because it is still an experimental project under active development by Oracle, each new version brings breaking changes. Therefore, we had to fix our version of Graal to a specific version instead of the newest one. Also, we had to fix the version of Java Runtime Environment installed on our machine to version 1.8.0_31 instead of version 1.8.0_45 since the newest JRE is incompatible with Graal. After fixing the versions the program runs normally. 12

13 Figure 9: Manual override 4.1 Finding a Pretty Printing Library Finding a suitable pretty printing library took some time. Originally we intended to translate a library written in Haskell by Wadler[6] (see Figure 10a) to Java. However, doing so proved difficult since Java and Haskell are two distinctly different languages in various aspects, and such differences are also embodied in the design patterns of their programs. A prototypical implementation in Java was attempted, as shown in Figure 10b, however it could not satisfactorily provide all the functionalities needed. Therefore, we resorted to an open-source library JppLib written in Java according to Oppen s algorithm, which is actually effectively equivalent to Wadler s algorithm. 4.2 Modularity One important feature of object algebra is modularity. For this purpose, multiple algebraic interfaces should be able to be combined together: Listing 1: Combining Interfaces s t a t i c <E> E make1 ( ExpAlg<E> a l g ) { return a l g. add ( a l g. l i t ( 3 ), a l g. mul ( a l g. l i t ( 4 ), a l g. l i t ( 5 ) ) ) ; } 13

14 (a) Haskell Source Figure 10: Wadler s Printer (b) Java Translation Attempt s t a t i c <E> E make2 ( ExpAlg<E> a l g ) {... } PPExpAlg p1 = new PPExpAlg ( ) ; I P r i n t pp1 = make1 ( p1 ) ; PPExpAlg p2 = new PPExpAlg ( ) ; I P r i n t pp2 = make2 ( p2 ) ; PPExpAlg p3 = new PPExpAlg ( ) ; // Here we try to combine pp1 and pp2 t o g e t h e r. I P r i n t pp3 = p3. add ( pp1, pp2 ) ; However, in the previous implementation, each interface contained its own layouter instance, which cannot communicate with each other. The solution (provided by the supervisor) is to only instantiate a printer when the function print() of the interface is called, instead of when the interface is instantiated. In this way, this printer can traverse the structures of all children interfaces each time, instead of having to rely on other printers for information. p u b l i c i n t e r f a c e I P r i n t { d e f a u l t StringBackend p r i n t ( ) { StringBackend back = new StringBackend (DEFAULT_LINE_WIDTH) ; Layouter<NoExceptions> pp = new Layouter<NoExceptions >(back, DEFAULT_INDENTATION) ; p r i n t L o c a l ( pp ) ; 14

15 } return back ; } void p r i n t L o c a l ( Layouter<NoExceptions> pp ) ; 5 Work Remaining & Plans The remaining work will be focused on the case study of Ruby programming language and the development of Emacs/Vim plugins. 5.1 Pretty Printer The eventual goal for the pretty printer is to let it serve as a component of the entire framework. An important usage of it will be printing out debugging information in a legible and user-friendly way. Also it should be useful for code formatting as well as transforming abstract syntax tree of the target language into source code. 5.2 Case Study on Ruby Since the idea of our framework is to share common components when creating different language implementations, we would like to perform a case study on another language, which would reuse considerable work we have done during the case study of Python. We decided to use Ruby programming language as our case study target, because: Ruby and Python share a lot of similarities. They are both dynamic, interpreted languages widely used for scripting and rapid software developments. Similar to the zippy project with Python, there s also a Truffle-based Ruby implementation called RubyTruffle, which would serve as a reasonable comparison target against our implementation using Object Algebra for program benchmarks. Specifically, we ll first break down the algebraic interfaces for Python syntax definitions into modules, and reuse the appropriate components for Ruby 15

16 syntax definitions. For example, the syntaxes of arithmetic computations for Python and Ruby are largely similar. Therefore, we might be able to extend the algebraic interface for Python arithmetic computations, instead of having to rewrite a definition particular to Ruby. We ll also seek to modularize the interpreter implementation for Python, and see how we will be able to reuse it in Ruby interpreter. 5.3 Emacs/Vim Plugins In order to write codes efficiently in the newly created language, it is better for the developers to have something like an IDE(Integrated Development Environment). However, developing an IDE from scratch is a huge task which would be out of the scope of a final year project. An alternative would be to write plugins for Emacs/Vim (two widely used extensible text editors) which will generate appropriate syntax highlighting and corresponding editing facilities respectively for the target languages being developed by the users, according to their algebraic interface annotations. Depending on the remaining time available for the project, plugins for Emacs and Vim will be developed. The plugin development for Emacs will be done in Emacs Lisp (See Figure 11), while the plugin for Vim will be written in Vimscript (See Figure 12). Ideally they will be tested against various sample languages. Figure 11: Emacs Plugin 16

17 Figure 12: Vim Plugin 6 Conclusion Together with the case study using Python carried out by the other members of this group, we hope this project could demonstrate the viability of rapid prototyping of programming languages using Oracle Graal/Truffle combined with the Object Algebra design pattern. With the completion and improvement of the various components of our framework, we hope that the framework could be ready for use of real-world programming languages in the near future. Potential areas for future work include the development of loggers and debuggers for the framework, the creation of more language components for developers to use, and the development of full-blown IDE support for languages created. 17

18 References [1] B. C. d. S. Oliveira and W. R. Cook, Extensibility for the Masses: Practical Extensibility with Object Algebras, in Proceedings of the 26th European Conference on Object-Oriented Programming, ser. ECOOP 12, Berlin, Heidelberg: Springer-Verlag, 2012, pp. 2 27, isbn: doi: / {\_}2. [Online]. Available: [2] Oracle, Graal Project, [Online]. Available: net/projects/graal. [3] M. Giese, The Java Pretty Printer Library, [Online]. Available: [4] D. Oppen, Pretty printing, Stanford University, Tech. Rep., 1979, p. 26. [Online]. Available: au/%7b~%7dpmoulder/line-breaking/oppen-pretty-printing.pdf. [5] C. Esquivias, Mumbler, [Online]. Available: https : / / github. com/cesquivias/mumbler. [6] P. Wadler, A prettier printer, Tech. Rep. 1980, 1998, pp [Online]. Available: doi= %7b%5c&%7damp;rep=rep1%7b%5c&%7damp;type= pdf. 18