Automatic OO Parser Generation using Visitors for Ada 2005

Transcription

1 Automatic OO Parser Generation using Visitors for Ada 2005 Martin C. Carlisle US Air Force Academy 2354 Fairchild Dr, Ste 6G101, USAFA, CO ABSTRACT We describe AdaGOOP 2005 (the Ada Generator of Object- Oriented Parsers). AdaGOOP 2005 takes a specification of tokens and an LALR(1) grammar and creates a lexer, a parser that automatically creates a parse tree, and a traversal of the parse tree using the visitor pattern. AdaGOOP generates output that is similar to that of the Java tool SableCC. It takes advantage of the new interface feature available in Ada 2005 to create a visitor. Categories and Subject Descriptors D.3.4 [Programming Languages]: Translator writing systems and compiler generation, D.1.5 [Programming Techniques] Object-oriented programming General Terms Languages, Theory Keywords AdaGOOP, automatic parser generation, visitor pattern, Ada INTRODUCTION A large number of tools have been developed to automate lexer and parser generation [1, 2, 3, 4, 5, 6, 7, 8, etc]. Probably the most well known of these are Lex and Yacc [4,5] and their GNU cousins Flex and Bison [3,7]. Although these tools were developed for C programmers, Ada versions (Aflex and Ayacc) were developed at the University of California-Irvine [1]. Using these tools, the programmer provides a set of regular expressions and an LALR(1) grammar, and obtains a completely functional lexer and parser. AdaGOOP 2005 is an Ada 2005 automatic parser generator built on top of the SCATC versions of aflex and ayacc [2]. As SableCC does, AdaGOOP automatically generates code to build a parse tree, and a traversal of the parse tree using the visitor pattern. This tool is distributed in the public Copyright 2006 Association for Computing Machinery. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the U.S. Government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only. SIGAda 06, November 12 16, 2006, Albuquerque, New Mexico, USA. Copyright 2006 ACM /06/ $5.00. domain as a service of the United States Air Force; but it is unsupported and without any warranty. It is currently available at: ftp://ftp.usafa.af.mil/pub/dfcs/carlisle/usafa/adagoop. 2. USING THE TOOL AdaGOOP is distributed as Ada 2005 source code. To use the tool, simply download and compile the source (adagoop.adb contains the main procedure). AdaGOOP outputs will be run through SCAFLEX and SCAYACC. Source code for these tools is distributed in subfolders of the same names (scaflex.adb and scayacc.adb are the main procedures). AdaGOOP is executed from a command prompt as follows: adagoop input_file prefix (e.g. adagoop ada05.g ada05) Prefix should be a simple name for the project. This name will be used several places in the output files. The input file to AdaGOOP consists of three parts: token_macros, tokens, and the grammar. Token macros are regular expressions that form parts of tokens. For example, token_macros DIGIT [0-9] INTEGER ({DIGIT}+) defines two macros, DIGIT and INTEGER, that can be used to create other tokens. Tokens are formed in the same fashion; however, their regular expressions can only contain token macros, not other tokens. The regular expression operators supported are given in Table 1. Comments should be defined with the token ignore, e.g. ignore "--"[^\n]* Both tokens and token macros should be specified 1 per line. Comments may be interspersed with --. This is why the ignore token above must have the -- in quotation marks. The final section, grammar gives an LALR(1) grammar. It should with a %start, indicating which non-terminal symbol is the starting symbol of the grammar, e.g. grammar %start A A : A token1 B token2 ; B : token2; 3

2 Each grammar rule consists of a non-terminal name, followed by a colon, followed by 1 or more sequences of terminals and nonterminals, separated by. Note that a null production is specified by simply having one of these sequences be empty. The rule is terminated with a semicolon. White space is irrelevant. A complete sample file is given in Appendix A. [0-9A-FZ] [^xyz] \n New line Any 1 of 0-9 A-F,Z Any 1 except x,y,z. Any 1 character \.,\( Period, Right paren X* 0 or more X s X+ 1 or more X s X? 0 or 1 X X Y Either X or Y X* Literally X* (ignore operators) {MACRO} Replace with macro defn () Precedence Table 1. Regular expression operators of AdaGOOP 3. ADAGOOP OUTPUT Throughout this section, prefix refers to the command-line argument passed to AdaGOOP. AdaGOOP generates prefix.l, an input file for scaflex. This file contains the specifications of tokens. The output file prefix.y is an input file for scayacc. It contains the grammar, plus code to build an object-oriented parse tree. Once AdaGOOP has been run, you can create the lexer and parser by running (for our example): scaflex prefix.l gnatchop -w prefix.a gnatchop -w prefix_io.a gnatchop -w prefix_dfa.a scayacc prefix.y gnatchop -w prefix.a gnatchop -w prefix_goto.a sed -e 1d prefix_tokens.a > prefix_modified_tokens.a gnatchop -w prefix_modified_tokens.a gnatchop -w prefix_shift_reduce.a Note gnatchop is needed only if your compilation environment (such as GNAT) requires package specifications and bodies to appear in separate files. sed is used only to delete the first line of prefix_tokens.a. We needed to add with context clauses to this file, but they appeared after the package declaration. To resolve this, we added the context clauses, another package declaration, and within a batch script used sed to delete the first package declaration. For the object-oriented parse tree, AdaGOOP creates an abstract object, Parseable, that will serve as the root of our hierarchy (it has no fields): type Parseable is abstract tagged null record; We also create a single object type, Parseable_Token Pto handle all nonterminal symbols. type Parseable_Token is new Parseable with record Line : Natural; Column : Natural; Token_String : String_Ptr; Token_Type : Token; -- enumeration type of all tokens Both Parseable and Parseable_Token are found in the package Prefix_Model. A production A => B C, where B and C are nonterminals, corresponds to a node in the parse tree containing pointers to the subtrees derived through B and C. We therefore create the following class declaration (which will appear in the package specification a_model.ads): type is new Parseable with record B_part : access B_nonterminal Class; C_part : access C_nonterminal Class; Since we have a single class for all terminal symbols, if we have A => B terminal1 C, we simply add a pointer to a Parseable_Token as the second field of the record. The parse tree then contains all of the information to completely recreate the original source text, with the exception of comment tokens (which are ignored). To address the issue of multiple productions for a single nonterminal, we insert an abstract class for the non-terminal, and have concrete classes corresponding to each possible production. For example, if we have A => B terminal1 C D, then AdGOOP outputs the following record declarations in the package A_Model: type is abstract new Parseable with null record; type _Ptr is access all Class; 4

3 type 1 is new B_part : access B_nonterminal Class Terminal1_Part : Parseable_Token_Ptr; C_part : access C_nonterminal Class; type 2 is new D_part : access D_nonterminal Class; The tool also will automatically number the fields in the case where a non-terminal or terminal appears more than once on the right hand side of a production. So, if we had A => B terminal1 B instead, the record would look like: type 1 is new B_part1 : access B_nonterminal Class Terminal1_Part : Parseable_Token_Ptr; B_part2 : access B_nonterminal Class; AdaGOOP also generates code for traversing the parse tree using the Visitor pattern [9]. Each class in the parse tree hierarchy has an acceptor method, which is used to dispatch to the appropriate visitor method: procedure Acceptor(This : access Parseable; I : access prefix_visitor_interface. Visit_prefix_Interface'Class) is abstract; This automatically generated interface specifies a visitor method that has no additional parameters and does not return a value. It is output to the file prefix_visitor_interface.ads. As can be seen from the following sample body (automatically generated), Acceptor merely calls the appropriate visitor method found in the interface: procedure Acceptor( This : access 2; I : access Visit_prefix_Interface'Class) is I.Visit_2(This); end Acceptor; Visit_prefix_Interface contains one visitor method for each leaf of the hierarchy, plus supports both pre-order and postorder depth first traversals by providing before and after methods that can be overloaded: limited with A_Model; limited with prefix_model; package prefix_visitor_interface is type Visit_prefix_Interface is interface; procedure Visit_Parseable_Token( T : access prefix_model. Parseable_Token procedure Before_1( procedure Visit_1( 'Class) is abstract; procedure After_1( procedure Before_2( procedure Visit_2( 'Class) is abstract; procedure After_2( end prefix_visitor_interface; A depth first traversal class, DFS, is generated into the package prefix_dfs. DFS implements the visitor interface, and its visit methods perform a depth first traversal of the parse tree. For example, following is the visitor method corresponding to the grammar rule A A PROCEDURE B: procedure Visit_1( A_Model.1'Class) is I_Classwide : access DFS'Class := I; I_Classwide.Before_1( N); N.A_part.Acceptor(I); I_Classwide.Visit_Parseable_Token( N.PROCEDURE_part); N.B_part.Acceptor(I); I_Classwide.After_1( N); end Visit_1; 5

4 This visitor method calls the before method (for pre-order traversals), then calls Acceptor for each of the children to dispatch to the appropriate visitor, and finally calls the after method (for post-order traversals). Tokens don t require dispatching, so the visit method can be called directly. It is interesting to note that Ada, unlike Java, does not perform dynamic dispatching by default. Therefore, it is necessary to convert the parameter I to a classwide type before calling Acceptor on each child so that dispatching occurs. To create your own visitor, you simply create a child class for DFS. In this child class, you override the before and after methods for those parse tree classes that you wish to process: limited with A_Model; with Prefix_DFS; package DFS_Example is type DFS is new Prefix_DFS.DFS with null record; overriding procedure After_1( A_Model.1'Class); overriding procedure Before_2( A_Model.2'Class); end DFS_Example; Creating a main program for a tool generated with AdaGOOP is simple. You need simply call the run procedure in the prefix_parser package with a filename, then use get_parse_tree to get the root of resulting parse tree. The depth first traversal is started by calling the method Acceptor on the root of the tree. with Ada.Command_Line; use Ada.Command_Line; with ada.text_io; use ada.text_io; with prefix_parser; with prefix_model; with DFS_Print; with a_model; procedure tool is type DFS_Access is access all DFS_Print.DFS'Class; Parse_Tree : prefix_model.parseable_ptr; Visitor : DFS_Access := new DFS_Print.DFS; if argument_count /= 1 then put_line("usage: tool filename"); return; end if; prefix_parser.run(argument(1)); parse_tree := prefix_parser.get_parse_tree; parse_tree.acceptor(visitor); end tool; 4. CONCLUSIONS AND FUTURE WORK AdaGOOP allows a programmer to quickly create a parser that generates an object-oriented parse tree. As a result, it is quite useful for generating simple language translators. Fagin [10] used the previous version of AdaGOOP to generate a translator from a subset of Ada 95 to NQC [11]. This allows developers to use Ada to program the Lego Mindstorms robots. Pedersen and Constantinides [12] created an Aspect Ada language using AdaGOOP. Future projects using AdaGOOP should be even easier with the new enhancements. One possibility for future work is to figure out how to include comments in the parse tree (since they do not appear in the grammar they are not currently included). This would allow AdaGOOP to be used to create a code reformatter. We would welcome additional contributions to this project (either grammars or improvements to the tool). 5. REFERENCES [1] Arcadia Project. Aflex and Ayacc. ~arcadia/aflex-ayacc/aflex-ayacc.html [2] Conn, Richard The Source Code Analysis Tool Construction Project. Proceedings of Tri-Ada 97, See also scatcdsk.htm [3] Free Software Foundation Bison-GNU Project-Free Software Foundation. bison.html [4] Johnson, S.C Yacc yet another compiler compiler. C.S. Technical Report #32. Murray Hill, NJ: Bell Telephone Laboratories. [5] Lesk, M.E., and Schmidt, E Lex a lexical analyzer generator. In Unix Programmer s Manual 2. Murray Hill, NJ: AT&T Bell Laboratories. [6] Mauney, Jon, and Fischer, Charles N An improvement to immediate error detection in Strong LL(1) parsers. Information Processing Letters 12(5): [7] Paxson, Vern Flex users manual. Ithaca, NY: Cornell University. [8] SableCC Home Page. [9] Martin, Robert C The Visitor Family of Design Patterns. In The Principles, Patterns and Practices of Agile Software Development. Prentice Hall. ISBN: [10] Fagin, B Using Ada-based robotics to teach computer science. In Proceedings of the 5th Annual SIGCSE/SIGCUE Conference on Innovation and Technology in Computer Science Education (Helsinki, Finland, July 11-13, 2000), [11] NQC Home Page. 6

5 [12] Pedersen, K. H. and Constantinides, C AspectAda: aspect oriented programming for Ada 95. Proceedings of SIGAda 2005 (Atlanta, GA, USA, November 13-17, 2005), A. SAMPLE ADAGOOP GRAMMAR token_macros DIGIT [0-9] INTEGER ({DIGIT}*) EXPONENT ([ee](\+? -){INTEGER}) DECIMAL_LITERAL {INTEGER}\.{DIGIT}{INTEGER}{EXPONENT}? tokens -- Reserved Words OPEN [oo][pp][ee][nn] PROCEDURE [pp][rr][oo][cc][ee][dd][uu][rr][ee] WRITE [ww][rr][ii][tt][ee] --Other NUMBER ({INTEGER} {DECIMAL_LITERAL}) grammar -- testing %start A A : A PROCEDURE B B; B : B OPEN C C; C : WRITE NUMBER; 7