Parser Generators. Aurochs and ANTLR and Yaccs, Oh My. Tuesday, December 1, 2009 Reading: Appel 3.3. Department of Computer Science Wellesley College

Transcription

1 Parser Generators Aurochs and ANTLR and Yaccs, Oh My Tuesday, December 1, 2009 Reading: Appel 3.3 CS235 Languages and Automata Department of Computer Science Wellesley College Compiler Structure Source Program (character stream) Lexer (a.k.a. Scanner, Tokenizer) Tokens Front End (CS235) Global Analysis Information (Symbol and Attribute Tables) Parser Middle Stages (CS251/ CS301) Abstract Syntax Tree (AST) Type Checker Intermediate Representation Semantic Analysis Intermediate Representation Optimizer Intermediate Representation (used by all phases of the compiler) Back End (CS301) Code Generator Machine code or byte code Parser Generators

2 Motivation We now know how to construct LL(k) and LR(k) parsers by hand. But constructing parser tables from grammars and implementing parsers from parser tables is tedious and error prone. Isn t there a better way? Idea: Hey, aren t computers good at automating these sorts of things? Parser Generators 34-3 Parser Generators A parser generator is a program whose inputs are: 1. a grammar that specifies a tree structure for linear sequences of tokens (typically created by a separate lexer program). 2. a specification of semantic actions to perform when building the parse tree. Its output is an automatically constructed program that parses a token sequence into a tree, performing the specified semantic actions along the way In the remainder of this lecture, we will explore and compare a two parser generators: Yacc and Aurochs. Parser Generators

3 ml-yacc: An LALR(1) Parser Generator Slip.yacc.yacc file (token datatype, parsing rules, precedence & associativity info) Slip.lex.lex file (token rules & token-handling code) ml-yacc ml-lex Slip.lex.sml Slip.yacc.sig token datatype Slip.yacc.desc LR parsing table info for debugging shift-reduce and reduce-reduce conflicts Slip.yacc.sml parsing program program (as character stream) A Slip program ant scanner program (SML) token stream Slip tokens ant parser program (SML) abstract syntax tree Slip AST Parser Generators 34-5 Format of a.yacc File Header section with SML code %% Specification of terminals & variables and various declarations (including precedence and associativity) %% Grammar productions with semantic actions Parser Generators

4 IntexpUnambiguousAST.yacc (Part 1) (* no SML code in this case *) %% (* separates initial section from middle section of.yacc file *) Name used to prefix various names created by ml-yacc %name Intexp %pos int Declares the type of positions for terminals. Each terminal has a left and right position. Here, position is just an int (a character position in a string or file) %term INT of int ADD SUB MUL DIV EXPT LPAREN (* "(" *) RPAREN (* ")" *) EOF (* End of file*) (* Middle section continued on next slide *) Specifies the terminals of the language. ml-yacc automatically constructs a Tokens module based on this specification. Tokens specified without "of" will have a constructor of two args: (1) its left position and (2) its right position. Tokens specified with "of" will have a constructor of three args: (1) the component datum (whose type follows "of"); (2) its left position; and (3) its the right position. %nonterm Start of AST.exp Specifies the non-terminals of the language and the kind of value Exp of AST.exp that the parser will generate for them (an AST in this case). Term of AST.exp The "start" non-terminal should not appear on the right-hand Factor of AST.exp sides of any productions. Unit of AST.exp %start Start Start symbol of the grammar Parser Generators 34-7 IntexpUnambiguousAST.yacc (Part 2) (* Middle section continued from previous slide *) %keyword %eop EOF %noshift EOF %nodefault Lists the keywords of the language. The error handler of the ml-yacc generated parser treats keywords specially. In this example, there aren t any keywords. Indicates tokens that the input Tells the parser never to shift the specfied tokens Suppresses generation of default reduction %verbose Generates a description of the LALR parser table in filename.yacc.desc. This is helpful for debugging shift/reduce and reduce/reduce conflicts %value INT(0) Specifies default values for value-bearing terminals. Terminals with default values may be created by an ml-yacc-generated parser as part of error-correction. Note; in this middle section, you can also specify associativity and precedence of operators. See IntexpPrecedence.yacc (later slide) for an example. (* Grammar productions and semantic actions on next slide *) Parser Generators

5 AST module for integer expressions structure AST = struct datatype exp = Int of int BinApp of binop * exp * exp and binop = Add Sub Mul Div Expt (* Various AST functions omitted from this slide *) Parser Generators 34-9 IntexpUnambiguousAST.yacc (Part 3) %% (* separates middle section from final section *) (* Grammar productions and associated semantic actions (in parens) go here *) Start: Exp (Exp) Within parens, nonterminals stand for the value generated by the parser for the nonterminal, as specified in the %nonterm declaration (an AST expression in this case). Exp : Term (Term) (* The following rules specify left associativity *) Exp ADD Term (AST.BinApp(AST.Add,Exp,Term)) Exp SUB Term (AST.BinApp(AST.Sub,Exp,Term)) Term : Factor (Factor) (* The following rules specify left associativity *) Term MUL Factor (AST.BinApp(AST.Mul,Term,Factor)) Term DIV Factor (AST.BinApp(AST.Div,Term,Factor)) Factor : Unit (Unit) (* The following rule specifies right associativity *) Unit EXPT Factor (AST.BinApp(AST.Expt,Unit,Factor)) Unit : INT (AST.Int(INT)) LPAREN Exp RPAREN (Exp) Within parens, terminals stand for the component datum specified after of in the %term declaration. E.g., the datum associated with INT is an integer. Parser Generators

6 Interfacing with the Lexer (*contents of Intexp.lex *) open Tokens type pos = int type lexresult= (svalue,pos) token fun eof () = Tokens.EOF(0,0) 0) fun integer(str,lexpos) = case Int.fromString(str) of NONE => raise Fail("Shouldn't happen: sequence of digits not recognized as integer -- " ^ str) SOME(n) => INT(n,lexPos,lexPos) (* For scanner initialization (not needed here) *) fun init() = () %% %header (functor IntexpLexFun(structure Tokens: Intexp_TOKENS)); alpha=[a-za-z]; digit=[0-9]; whitespace=[\ \t\n]; symbol=[+*^()\[]; any= [^]; %% {digit}+ => (integer(yytext,yypos)); "+" => (ADD(yypos,yypos)); (* more lexer rules go here *) Parser Generators Parser interface code, Part 1 structure Intexp = struct (* Module for interfacing with parser and lexer *) (* Most of the following is "boilerplate" code that needs to be slightly edited on a per parser basis. *) structure IntexpLrVals = IntexpLrValsFun(structure Token = LrParser.Token) structure IntexpLex = IntexpLexFun(structure Tokens = IntexpLrVals.Tokens); structure IntexpParser = Join(structure LrParser = LrParser structure ParserData = IntexpLrVals.ParserData structure Lex = IntexpLex) val invoke = fn lexstream => (* The invoke function invokes the parser given a lexer *) let val print_error = fn (str,pos,_) => TextIO.output(TextIO.stdOut, "***Intexp Parser Error at character position " ^ (Int.toString pos) ^ "***\n" ^ str^ "\n") in IntexpParser.parse(0,lexstream,print_error,()) fun newlexer fcn = (* newlexer creates a lexer from a given input-reading function *) let val lexer = IntexpParser.makeLexer fcn val _ = IntexpLex.UserDeclarations.init() in lexer (* continued on next slide *) ml-yacc

7 Parser interface code, Part 2 (* continued from previous slide *) fun stringtolexer str = (* creates a lexer from a string *) let val done = ref false in newlexer (fn n => if (!done) then " else (done := true; str)) fun filetolexer filename = (* creates a lexer from a file *) let val instream = TextIO.openIn(filename) in newlexer (fn n => TextIO.inputAll(inStream)) fun lexertoparser (lexer) = (* creates a parser from a lexer *) let val dummyeof = IntexpLrVals.Tokens.EOF(0,0) val (result,lexer) = invoke lexer val (nexttoken,lexer) = IntexpParser.Stream.get lexer in if IntexpParser.sameToken(nextToken,dummyEOF) then result else (TextIO.output(TextIO.stdOut, "*** INTEXP PARSER WARNING -- unconsumed input ***\n"); result) val parsestring = lexertoparser o stringtolexer (* parses a string *) val parsefile = lexertoparser o filetolexer (* parses a file *) (* struct *) Parser Generators Putting it all together: the load file (* This is the contents of load-intexp-unambiguous-ast.sml *) CM.make("$/basis.cm"); (* loads SML basis library *) CM.make("$/ml-yacc-lib.cm"); (* loads SML YACC library *) use "AST.sml"; (* datatype for integer expression abstract syntax trees *) use "IntexpUnambiguousAST.yacc.sig"; (* defines Intexp_TOKENS and other datatypes *) use "IntexpUnambiguousAST.yacc.sml"; (* defines shift-reduce parser *) use "Intexp.lex.sml"; (* load lexer *after* parser since it uses tokens defined by parser *) use "Intexp.sml"; (* load top-level parser interface *) Control.Print.printLength := 1000; (* set printing parameters so that *) Control.Print.printDepth := 1000; (* we ll see all details of parse trees *) Control.Print.stringDepth := 1000; (* and strings *) open Intexp; (* open the parsing module so that we can use parsestring and parsefile without qualification. *) Parser Generators

8 Taking it for a spin [fturbak@cardinal intexp-parsers] ml-lex Intexp.lex Number of states = 14 Number of distinct rows = 3 Approx. memory size of trans. tabl e = 387 bytes [fturbak@cardinal intexp-parsers] ml-yacc IntexpUnambiguousAST.yacc [fturbak@cardinal intexp-parsers] Linux shell - use "load-intexp-unambiguous-ast.sml"; (* lots of printed output omitted here *) - parsestring "1+2*3^4^5*6+7"; val it = BinApp SML (Add, BinApp (Add,Int 1, BinApp (Mul, BinApp (Mul,Int 2,BinApp (Expt,Int 3,BinApp (Expt,Int 4,Int 5))), Int 6)),Int 7) : IntexpParser.result Parser Generators intepreter IntexpUnambiguousAST.yacc.desc state 0: state 1: Start :. Exp INT shift 6 LPAREN shift 5 Start goto 19 Exp goto 4 Term goto 3 Factor goto 2 Unit goto 1. error Factor : Unit. (reduce by rule 7) Factor : Unit. EXPT Factor ADD reduce by rule 7 SUB reduce by rule 7 MUL reduce by rule 7 DIV reduce by rule 7 EXPT shift 7 RPAREN reduce by rule 7 EOF reduce by rule 7 lots of states omitted state 18: state 19: Unit : LPAREN Exp RPAREN. (reduce by rule 10) ADD reduce by rule 10 SUB reduce by rule 10 MUL reduce by rule 10 DIV reduce by rule 10 EXPT reduce by rule 10 RPAREN reduce by rule 10 EOF reduce by rule 10. error EOF accept. error 72 of 104 action table entries left after compaction 21 goto table entries. error Parser Generators

9 IntexpUnambiguousCalc.yacc We can calculate the result of an integer expression rather than generating an AST. (* An integer exponentiation function *) fun expt (base,0) = 1 expt(base,power) = base*(expt(base,power-1)) %% (* Only changed parts shown *) %nonterm Start of int Exp of int Term of int Factor of int Unit of int ) %% Start: Exp (Exp) Exp : Term (Term) (* The following rules specify left associativity *) Exp ADD Term (Exp + Term) Exp SUB Term (Exp - Term) Term : Factor (Factor) (* The following rules specify left associativity *) Term MUL Factor (Term * Factor) Term DIV Factor (Term div Factor) (* div is integer division *) Factor : Unit (Unit) (* The following rule specifies right associativity *) Unit EXPT Factor (expt(unit,factor)) Unit : INT (INT) LPAREN Exp RPAREN (Exp) Parser Generators Testing IntexpUnambiguousCalc [fturbak@cardinal intexp-parsers] ml-yacc IntexpUnambiguousCalc.yacc [fturbak@cardinal intexp-parsers] Linux shell - use "load-intexp-unambiguous-calc.sml"; (* lots of printed output omitted here *) - parsestring "1+2*3"; val it = 7 : IntexpParser.result result SML intepreter - parsestring "1-2-3"; val it = ~4 : IntexpParser.result Parser Generators

10 IntexpAmbiguousAST.yacc We can use an ambiguous grammar. (In this case, it leads to shift/reduce conflicts.) %% (* Only changed parts shown *) %nonterm Start of AST.exp Exp of AST.exp ) %% Start: Exp (Exp) Exp : INT (AST.Int(INT)) Exp ADD Exp (AST.BinApp(AST.Add,Exp1,Exp2)) Exp SUB Exp (AST.BinApp(AST.Sub,Exp1,Exp2)) Exp MUL Exp (AST.BinApp(AST.Mul,Exp1,Exp2)) Exp DIV Exp (AST.BinApp(AST.Div,Exp1,Exp2)) Exp EXPT Exp (AST.BinApp(AST.Expt,Exp1,Exp2)) Parser Generators IntexpAmbiguousAST.yacc.desc 25 shift/reduce conflicts error: state 8: shift/reduce conflict (shift EXPT, reduce by rule 6) error: state 8: shift/reduce conflict (shift DIV, reduce by rule 6) error: state 8: shift/reduce conflict (shift MUL, reduce by rule 6) error: state 8: shift/reduce conflict (shift SUB, reduce by rule 6) error: state 8: shift/reduce conflict (shift ADD, reduce by rule 6) error: state 9: shift/reduce conflict (shift EXPT, reduce by rule 5) error: state 9: shift/reduce conflict (shift DIV, reduce by rule 5) error: state 9: shift/reduce conflict (shift MUL, reduce by rule 5) error: state 9: shift/reduce conflict (shift SUB, reduce by rule 5) error: state 9: shift/reduce conflict (shift ADD, reduce by rule 5) error: state 10: shift/reduce conflict (shift EXPT, reduce by rule 4) error: state 10: shift/reduce conflict (shift DIV, reduce by rule 4) error: state 10: shift/reduce conflict (shift MUL, reduce by rule 4) error: state 10: shift/reduce conflict (shift SUB, reduce by rule 4) error: state 10: shift/reduce conflict (shift ADD, reduce by rule 4) error: state 11: shift/reduce conflict (shift EXPT, reduce by rule 3) error: state 11: shift/reduce conflict (shift DIV, reduce by rule 3) error: state 11: shift/reduce conflict (shift MUL, reduce by rule 3) error: state 11: shift/reduce conflict (shift SUB, reduce by rule 3) error: state 11: shift/reduce conflict (shift ADD, reduce by rule 3) error: state 12: shift/reduce conflict (shift EXPT, reduce by rule 2) error: state 12: shift/reduce conflict (shift DIV, reduce by rule 2) error: state 12: shift/reduce conflict (shift MUL, reduce by rule 2) error: state 12: shift/reduce conflict (shift SUB, reduce by rule 2) error: state 12: shift/reduce conflict (shift ADD, reduce by rule 2) state descriptions omitted Parser Generators

11 IntexpPrecedenceAST.yacc We can resolve conflicts with precedence/associativity declarations (can also handle dangling else problem this way). %% (* Only changed parts shown *) %nonterm Start of AST.exp Exp of AST.exp ) (* Specify associativity and precedence from low to high *) %left ADD SUB %left MUL DIV %right EXPT %% Start: Exp (Exp) Exp : INT (AST.Int(INT)) Exp ADD Exp (AST.BinApp(AST.Add,Exp1,Exp2)) Exp SUB Exp (AST.BinApp(AST.Sub,Exp1,Exp2)) Exp MUL Exp (AST.BinApp(AST.Mul,Exp1,Exp2)) Exp DIV Exp (AST.BinApp(AST.Div,Exp1,Exp2)) Exp EXPT Exp (AST.BinApp(AST.Expt,Exp1,Exp2)) Parser Generators Stepping Back Whew! That s a lot of work! Is it all worth it? There s a tremous amount of initial setup and glue code to get Lex and Yacc working together. But once that s done, we can focus on specifying and debugging high-level declarations of the token structure (in Lex) and grammar (in Yacc) rather than on SML programming. In Yacc, generally sp most of the time debugging shift/reduce and reduce/reduce conflicts must be very familiar with the LALR(1) table descriptions in the.yacc.desc file. Parser Generators

12 Other Parser Generators There are a plethora of other LALR parser generators for many languages, including: GNU Bison (Yacc for C, C++) jb (Bison for Java) CUP (Yacc for Java) There are also LL(k) parser generators, particularly ANTLR, which generates parsers and other tree-manipulation programs for Java, C, C++, C#, Python, Ruby, and JavaScript. There is also an sml-antlr. There are a relatively new (starting in 2002) collection of so- called packrat parsers for Parsing Expression Grammars (PEGs) for a wide range of target languages. We will study one of these: Aurochs. Parser Generators Parsing Expression Grammars (PEGs) Key problem in building top-down (recursive-descent) parsers: deciding which production to use on tokens. Three strategies: Predictive parser: determine which production to use based on k tokens of lookahead. Often mangle grammars to make this work. Backtracking parser: when have several choices, try them in order. If one fails try the next. Very general and high-level, but can take exponential time to parse input. PEG Parser: have a prioritized choice operator in which unconditionally use first successful match. Can directly express common disambiguation rules like longest-match match, followed-by by, and not-followed-by. Can express lexing rules and parsing rules in same grammar! This eliminates glue code between lexer and parser. Can implement PEGs via a packrat parser that memoizes parsing results and guarantees parsing in linear time (but possibly exponential space) Parser Generators

13 Aurochs: a PEG Parser We will explore PEG parsing using the Aurochs parser generator, developed by Berke Durak and available from The input to Aurochs is (1) a file to be parsed and (2) a.peg file specifying PEG declarations for lexing and parsing rules. The output of Aurochs is an XML parse tree representation. (It is also possible to generate parse trees in particular languages, but for simplicity we ll stick with the XML trees.) Parser Generators Aurochs Example [lynux@localhost slipmm-parsers]$ aurochs -parse simple.slip SlipmmParens.peg Aurochs Parsing file simple.slip using Nog interpreter - Grammar loaded from file SlipmmParens.peg - RESULT <Root> <StmList> <Assign var="x"> <BinApp binop="+"> <Int value="3"/> <Int value="4"/> </BinApp> </Assign> <Print> <BinApp binop="*"> <BinApp binop="-"> <Id name="x"/> <Int value="1"/> </BinApp> <BinApp binop="+"> <Id name="x"/> <Int value="2"/> </BinApp> </BinApp> </Print> </StmList> </Root> - No targets Contents of the file simple.slip begin x := (3+4); print ((x-1)*(x+2)); Parser Generators

14 Aurochs Grammar Expressions Expression 'c' "hello" BOF, EOF sigma Meaning The single ASCII character 'c' The string hello Beginning and of file tokens Any character epsilon The empty string [a-za-z0-9] grep-like character sets [^\n\t ] grep-like complements of character sets e1 e2 Concatenation: parse e1, then e2 e* Repetition: parses as many occurrences of e as possible, including zero. e+ Nonzero repetition: parses at least one occurrences of e. e? Option: parses zero or one occurrence of e. &e Conjunction: Parses e, but without consuming it. ~e Negation: Attempts to parse e. If parsing e fails, then ~e succeeds without consuming any input. If parsing e succeeds, then ~e fails (also without consuming input). e1 e2 (e) Ordered disjunction (prioritized choice): Attempt to parse e1 at the current position. If this succeeds, it will never try to parse e2 at the current position. If it fails, it will never try to parse e1 again at that position, and will try to parse e2. If e2 fails, the whole disjunction fails. Parentheses used for explicit grouping name Parse according to e from production name ::= e Parser Generators Some Simple Expressions Expression Meaning -? [1-9][0-9]* Any nonempty sequence of digits not beginning with 0, preceded by an optional minus sign &[aeiou] [a-z]+ Any nonempty sequence of lowercase letters beginning with a vowel. ~0 [0-9]+ Any nonempty sequence of digits not beginning with 0 ~( begin let ) &[a-z] [a-za-z0-9]+ exp ::= (exp + exp) id = exp id num Alphanumeric names beginning with lowercase letters, excluding the keywords begin,, and let. Assume id matches identifiers and num matches numbers. Then this can parse (x=3 + x) and (x=y=3 + (1 + z=y)) Due to prioritized choice, this will never parse x=3. exp ::= (exp + exp) Due to prioritized choice this will never parse x=3 id id = exp num if exp then stm else stm if exp then stm Solves dangling else problem by specifying that else goes with the closest if. Parser Generators

15 Numerical Expressions in Aurochs (* Tokens *) float ::= '-'? decimal+ ('.' decimal+)? ('e' ('-' '+')? decimal+)?; integer ::= '-'? decimal+; decimal ::= [0-9]; sp ::= (' ' '\n' '\r' '\t')*; (* any number of whitespaces *) ADD ::= sp ' + ' sp; SUB ::= sp '-' sp; MUL ::= sp '*' sp; DIV ::= sp '/' sp; LPAREN ::= sp '(' sp; RPAREN ::= sp ')' sp; (* Expressions *) start ::= exp sp EOF; exp ::= term ADD exp (* right associative *) term SUB exp (* right associative *) term; term ::= factor MUL term (* right associative *) factor DIV term (* right associative *) SUB factor (* unary negation *) factor; factor ::= number LPAREN exp RPAREN; number ::= float integer; Parser Generators XML Parse Trees for Numexps in Aurochs (* Tokens the same as before, so omitted *) (* Expressions *) start ::= exp sp EOF; exp ::= <add> term ADD exp </add> (* right associative *) _{term SUB exp} (* right associative *) term; term ::= <mul> factor MUL term </mul> (* right associative *) <div> factor DIV term </div> (* right associative *) <neg> SUB factor </neg> factor; factor ::= number LPAREN exp RPAREN; number ::= <number> value:(float integer) </number>; Parser Generators

16 Testing Numexp Parser calculator]$ aurochs -quiet -parse exp.txt numexp.peg <Root> <mul> _{<number value="1"/> _{<number value="2"/> <number value="3"/> Contents of the file} exp.txt (1-2-3)*-( e-9)} <neg> <add> <neg> <number value="4"/> </neg> <add> <number value="5.6"/> <number value="7.8e-9"/> </add> </add> </neg> </mul> </Root> Parser Generators Transforming Left Recursion (* Expressions *) start ::= exp sp EOF; exp ::= term <addsub> (op:(add SUB) ( exp)* </addsub>; term ::= <neg> SUB factor </neg> factor <muldiv> (op:(mul DIV) term)* </muldiv>; factor ::= number LPAREN exp RPAREN; number ::= <number> value:(float integer) </number>; Parser Generators

17 Left Recursion Transformation Exampl calculator]$ aurochs -quiet -parse sub.txt numexp-leftrec.peg <Root> <number value="1"/> <muldiv/> <addsub op="-"> <number value="2"/> <muldiv/> <addsub op="-"> Contents of the file sub.txt <number value="3"/> ( ) <muldiv/> <addsub op="-"> <number value="4"/> <muldiv/> <addsub/> </addsub> </addsub> </addsub> <muldiv/> <addsub/> </Root> Parser Generators Moving Spaces from Tokens to Expressions (* Tokens *) float ::= '-'? decimal+ ('.' decimal+)? ('e' ('-' '+')? decimal+)?; integer ::= '-'? decimal+; decimal ::= [0-9]; sp ::= (' ' '\n' '\r' '\t')*; (* any number of whitespaces *) ADD ::= ' + '; SUB ::= '-'; MUL ::= '*'; DIV ::= '/'; /; LPAREN ::= '('; RPAREN ::= ')'; (* Expressions *) start ::= sp exp sp EOF; exp ::= <add> sp term sp ADD sp exp sp </add> (* right associative *) _{sp term sp SUB sp exp sp} (* right associative *) term; term ::= <mul> sp factor sp MUL sp term sp </mul> (* right associative *) <div> sp factor sp DIV sp term sp </div> (* right associative *) <neg> sp SUB sp factor sp </neg> sp factor sp; factor ::= sp number sp sp LPAREN sp exp sp RPAREN sp; number ::= <number> sp value:(float integer) sp </number>; Parser Generators

18 Modifiers: Surrounding, Outfixing, and Fris The following modifiers take an expression x as parameter and transform each production a ::= b as follows: apping x do... done transforms them into a ::= b x preping x do... done transforms them into a ::= x b surrounding x do... done transforms them into a ::= x b x The following modifiers transform each concatenation a b c: outfixing x do... done transforms them into x a x b x c x suffixing x do... done transforms them into a x b x c x prefixing x do... done transforms them into x a x b x c Note that these modifiers also act within implicit concatenations produced by the operators '*' and '+'. These rules found at Parser Generators A Shorthand for all those Expression Spaces start ::= sp exp sp EOF; outfixing sp do surrounding sp do exp ::= <add> term ADD exp </add> (* right associative *) _{term SUB exp} (* right associative *) term; term ::= <mul> factor MUL term </mul> (* right associative *) <div> factor DIV term </div> (* right associative *) <neg> SUB factor </neg> factor; factor ::= number LPAREN exp RPAREN; number ::= <number> value:(float integer) </number>; done; done; Parser Generators

19 Aurochs Productions for Slip-- Tokens whitespace ::= [ \n\r\t] BOF; (* BOF = beginning of file token *) (* sp is at least one ignorable unit (space or comment) *) sp ::= whitespace*; PRINT ::= "print"; BEGIN ::= "begin"; END ::= ""; keyword ::= <keyword> value: (PRINT BEGIN END) </keyword>; ADD ::= '+'; SUB ::= '-'; MUL ::= '*'; DIV ::= '/'; op ::= <op> value: (ADD SUB MUL DIV) </op>; LPAREN ::= "("; RPAREN ::= ")"; SEMI ::= ";"; GETS ::= ":="; punct ::= <punct> value: (LPAREN RPAREN SEMI GETS) </punct> ; alpha ::= [a-za-z]; alphanumund ::= [a-za-z0-9_]; digit ::= [0-9]; reserved ::= "print" "begin" ""; ident ::= <ident> value: (~(reserved sp) alpha alphanumund*) </ident>; integer ::= <integer> value: digit+ </integer>; token ::= sp (keyword punct op ident integer) sp; tokenlist ::= <tokenlist> token* </tokenlist>; start ::= tokenlist EOF; Parser Generators Slip-- Tokenizing example [lynux@localhost slipmm-parsers]$ aurochs -quiet -parse simple.slip SlipmmTokens.peg <Root> <tokenlist> <keyword value="begin"/> <ident value="x"/> <punct value=":="/> <punct value="("/> <integer value="3"/> <op value="+"/> <integer value="4"/> <punct value=")"/> <punct value=";"/> <keyword value="print"/> <punct value="("/> <punct value="("/> <ident value="x"/> <op value="-"/> <integer value="1"/> <punct value=")"/> <op value="*"/> <punct value="("/> <ident value="x"/> <op value="+"/> <integer value="2"/> <punct value=")"/> <punct value=")"/> <punct value=";"/> <keyword value=""/> </tokenlist> </Root> Contents of the file simple.slip begin x := (3+4); print ((x-1)*(x+2)); Parser Generators

20 Slip-- Tokens with Positions slipmm-parsers]$ aurochs -quiet -parse simple.slip SlipmmTokensPositions.peg <Root> <tokenlist> <token start="0" stop="5"> (* Only Change to.peg file. *) <keyword value="begin"/> </token> <token start="9" stop="10"> <ident value="x"/> </token> <token start="11" stop="13"> <punct value=":="/> </token> <token start="14" stop="15"> <punct value="("/> </token> <token start="15" stop="16"> <integer value="3"/> </token> <token start="16" stop="17"> <op value="+"/> </token> many more tokens. token ::= <token> sp start:position (keyword punct op ident integer) stop:position sp </token>; Contents t of the file simple.slip begin x := (3+4); print ((x-1)*(x+2)); Parser Generators Line Comments How to add line-terminated comments introduced by #? begin sum := 5+3; # Set sum to 8 print(sum); # Display sum (* Only changes in.peg file *) whitespace ::= [ \n\r\t] BOF; (* BOF = beginning of file token *) linecomment ::= '#' [^\n]* '\n'; (* Comment goes from '#' to of line *) (* sp is at least one ignorable unit (space or comment) *) sp ::= (whitespace linecomment )*; Parser Generators

21 Nonnestable Block Comments begin sum := 5+3; # Set sum to 8 { Comment out several lines: x := sum * 2; z := x * x; } print(sum); # Display sum (* Only changes in.peg file *) whitespace ::= [ \n\r\t] BOF; (* BOF = beginning of file token *) linecomment ::= '#' [^\n]* '\n'; (* Comment goes from '#' to of line *) blockcomment ::= '{' ((~ '}') sigma)* '}'; (* Nonnestable block comment *) (* sp is at least one ignorable unit (space or comment) *) sp ::= (whitespace linecomment blockcomment)*; Parser Generators Nestable Block Comments begin sum := 5+3; # Set sum to 8 { Comment out several lines: x := sum * 2; { Illustrate nested block comments: y = sum - 3:} z := x * x; } print(sum); # Display sum (* Only changes in.peg file *) whitespace ::= [ \n\r\t] BOF; (* BOF = beginning of file token *) linecomment ::= '#' [^\n]* '\n'; (* Comment goes from '#' to of line *) blockcomment ::= '{' (blockcomment ((~ '}') sigma))* '}'; (* Properly nesting block comment *) (* sp is at least one ignorable unit (space or comment) *) sp ::= (whitespace linecomment blockcomment)*; Parser Generators

22 Slip -- Programs (* Assume tokens unchanged from before *) start ::= program; outfixing sp do apping sp do program ::= stm EOF; stm ::= <Assign> var:ident GETS exp </Assign> <Print> PRINT exp </Print> <StmList> BEGIN stmlist END </StmList>; stmlist ::= (stm SEMI)*; exp ::= <Id> name:ident </Id> <Int> value:integer </Int> <BinApp> LPAREN exp binop:op exp RPAREN </BinApp>; done; done; Parser Generators Aurouchs Errors -Fatal error: exception Aurochs.Parse_error(1033) This means that the.peg file has an error at the specified character number (1033 in this case). Note: start position is 0-indexed; compared to 1-indexed emacs position in M-x goto-char. -PARSE ERROR IN FILE nesting-block-comment-test.slip AT CHARACTER 12 This means that the parser encountered an error in parsing the input file at the specified ed character number. Parser Generators