Languages and Compilers (SProg og Oversættere) Lecture 3 recap Bent Thomsen Department of Computer Science Aalborg University

Transcription

1 Languages and Compilers (SProg og Oversættere) Lecture 3 recap Bent Thomsen Department of Computer Science Aalborg University With acknowledgement to Norm Hutchinson whose slides this lecture is based on. 1

2 Syntax Analysis: Parser Dataflow chart Source Program Stream of Characters Scanner Error Reports Stream of Tokens Parser Error Reports Abstract Syntax Tree 2

3 Top-Down vs. Bottom-Up parsing LL-Analyse (Top-Down) Left-to-Right Left Derivative LR-Analyse (Bottom-Up) Left-to-Right Right Derivative Reduction Derivation Look-Ahead Look-Ahead 3

4 Recursive Descent Parsing Recursive descent parsing is a straightforward top-down parsing algorithm. We will now look at how to develop a recursive descent parser from an EBNF specification. Idea: the parse tree structure corresponds to the call graph structure of parsing procedures that call each other recursively. 4

5 Recursive Descent Parsing Sentence ::= Subject Verb Object. Subject ::= I a Noun the Noun Object ::= me me a Noun the Noun Noun ::= cat mat rat Verb ::= like is is see sees Define a procedure parsen for each non-terminal N private void parsesentence() ; private void parsesubject(); private void parseobject(); private void parsenoun(); private void parseverb(); 5

6 Recursive Descent Parsing: Auxiliary Methods public class MicroEnglishParser private TerminalSymbol currentterminal private void accept(terminalsymbol expected) if if (currentterminal matches expected) currentterminal = next input terminal ; else report a syntax error

7 Recursive Descent Parsing: Parsing Methods Sentence ::= Subject Verb Object. private void parsesentence() parsesubject(); parseverb(); parseobject(); accept(. ); 7

8 Recursive Descent Parsing: Parsing Methods Subject ::= I a Noun the Noun private void parsesubject() if if (currentterminal matches I ) accept( I ); else if if (currentterminal matches a ) accept( a ); parsenoun(); else if if (currentterminal matches the ) accept( the ); parsenoun(); else report a syntax error 8

9 Recursive Descent Parsing: Parsing Methods Noun ::= cat mat rat private void parsenoun() if if (currentterminal matches cat ) accept( cat ); else if if (currentterminal matches mat ) accept( mat ); else if if (currentterminal matches rat ) accept( rat ); else report a syntax error 9

10 Top-down parsing The parse tree is constructed starting at the top (root). Sentence Subject Verb Object. Noun Noun The cat sees a rat. 10

11 A little bit of useful theory We will now look at a few useful bits of theory. These will be necessary later when we implement parsers. Grammar transformations A grammar can be transformed in a number of ways without changing the meaning (i.e. the set of strings that it defines) The definition and computation of starter sets 11

12 Left factorization 1) Grammar Transformations X Y X Z X ( Y Z ) X Y= ε Z Example: single-command ::= V-name := := Expression if if Expression then single-command if if Expression then single-command else single-command single-command ::= V-name := := Expression if if Expression then single-command ( ε else single-command) 12

13 1) Grammar Transformations (ctd) Elimination of Left Recursion N ::= X N Y N ::= X Y* Example: Identifier ::= Letter Identifier Letter Identifier Digit Identifier ::= Letter Identifier (Letter Digit) Identifier ::= Letter (Letter Digit)* 13

14 1) Grammar Transformations (ctd) Substitution of non-terminal symbols N ::= X M ::= α N β Example: N ::= X M ::= α X β single-command ::= for contrvar := := Expression to-or-dt Expression do do single-command to-or-dt ::= to to downto single-command ::= for contrvar := := Expression (to downto) Expression do do single-command 14

15 Developing RD Parser for Mini Triangle Before we begin: The following non-terminals are recognized by the scanner They will be returned as tokens by the scanner Identifier := := Letter (Letter Digit)* Integer-Literal ::= Digit Digit* Operator ::= + - * / < > = Comment ::=! Graphic* eol Assume scanner produces instances of: public class Token byte kind; String spelling; final static byte IDENTIFIER = 0, 0, INTLITERAL = 1; 1;... 15

16 (1+2) Express grammar in EBNF and factorize... Program ::= single-command Command ::= single-command Command Left ; single-command recursion elimination needed single-command Left factorization needed ::= V-name := := Expression Identifier ( Expression ) if if Expression then single-command else single-command while Expression do do single-command let Declaration in in single-command begin Command end V-name ::= Identifier... 16

17 (1+2)Express grammar in EBNF and factorize... After factorization etc. we get: Program ::= single-command Command ::= single-command (;single-command)* single-command ::= Identifier ( := := Expression ( Expression ) ) if if Expression then single-command else single-command while Expression do do single-command let Declaration in in single-command begin Command end V-name ::= Identifier... 17

18 Developing RD Parser for Mini Triangle Expression Left recursion elimination needed ::= primary-expression Expression Operator primary-expression primary-expression ::= Integer-Literal V-name Operator primary-expression ( Expression ) Declaration Left recursion elimination needed ::= single-declaration Declaration ; single-declaration single-declaration ::= const Identifier ~ Expression var Identifier : Type-denoter Type-denoter ::= Identifier 18

19 (1+2) Express grammar in EBNF and factorize... After factorization and recursion elimination : Expression ::= primary-expression ( Operator primary-expression )* )* primary-expression ::= Integer-Literal Identifier Operator primary-expression ( Expression ) Declaration ::= single-declaration (;single-declaration)* single-declaration ::= const Identifier ~ Expression var Identifier : Type-denoter Type-denoter ::= Identifier 19

20 (3) Create a parser class with... public class Parser private Token currenttoken; private void accept(byte expectedkind) if if (currenttoken.kind == == expectedkind) currenttoken = scanner.scan(); else report syntax error private void acceptit() currenttoken = scanner.scan(); public void parse() acceptit(); //Get the the first token parseprogram(); if if (currenttoken.kind!=!= Token.EOT) report syntax error

21 (4) Implement private parsing methods: Program ::= single-command private void parseprogram() parsesinglecommand(); 21

22 (4) Implement private parsing methods: single-command ::= Identifier ( := := Expression ( Expression ) ) if if Expression then single-command else single-command... more alternatives... private void parsesinglecommand() switch (currenttoken.kind) case Token.IDENTIFIER : case Token.IF : more cases default: report a syntax error 22

23 (4) Implement private parsing methods: single-command ::= Identifier ( := := Expression ( Expression ) ) if if Expression then single-command else single-command while Expression do do single-command let Declaration in in single-command begin Command end From the above we can straightforwardly derive the entire implementation of parsesinglecommand (much as we did in the microenglish example) 23

24 Algorithm to convert EBNF into a RD parser The conversion of an EBNF specification into a Java implementation for a recursive descent parser is so mechanical that it can easily be automated! => JavaCC Java Compiler Compiler We can describe the algorithm by a set of mechanical rewrite rules N ::= ::= X private void parsen() parse X 24

25 Algorithm to convert EBNF into a RD parser parse tt accept(t); parse N parsen(); where ttis is a terminal where N is is a non-terminal parse ε // // a dummy statement parse XY XY parse X parse Y 25

26 Algorithm to convert EBNF into a RD parser parse X* X* while (currenttoken.kind is is in in starters[x]) parse X parse X Y X Y switch (currenttoken.kind) cases in in starters[x]: parse X break; cases in in starters[y]: parse Y break; default: report syntax error 26

27 Example: Generation of parsecommand Command ::= ::= single-command ((;; single-command )* )* private void void void parsecommand() parsesinglecommand(); single-command ((;; single-command )* )* parse while ( ( ; ; single-command (currenttoken.kind==token.semicolon) )* )* parse acceptit(); ; ; single-command parse parsesinglecommand(); single-command 27

28 Example: Generation of parsesingledeclaration single-declaration ::= ::= const Identifier ~ Type-denoter var varidentifier :: Expression private void parsesingledeclaration() switch (currenttoken.kind) private case void void parsesingledeclaration() Token.CONST: parse switch const (currenttoken.kind) acceptit(); Identifier ~ Type-denoter case var var Token.CONST: parseidentifier(); :: Expression parse acceptit(); acceptit(token.is); const Identifier ~ Type-denoter case parseidentifier(); Token.VAR: parsetypedenoter(); parse acceptit(token.is); parse case ~ var varidentifier :: Expression Token.VAR: default: parsetypedenoter(); acceptit(); Type-denoter report syntax error case Token.VAR: parseidentifier(); parse acceptit(token.colon); var var Identifier : : Expression default: parseexpression(); report syntax error default: report syntax error 28

29 LL(1) Grammars The presented algorithm to convert EBNF into a parser does not work for all possible grammars. It only works for so called LL(1) grammars. What grammars are LL(1)? Basically, an LL(1) grammar is a grammar which can be parsed with a top-down parser with a lookahead (in the input stream of tokens) of one token. How can we recognize that a grammar is (or is not) LL(1)? There is a formal definition which we will skip for now We can deduce the necessary conditions from the parser generation algorithm. 29

30 parse X* X* LL(1) Grammars while (currenttoken.kind is is in in starters[x]) parse X parse X Y X Y switch (currenttoken.kind) cases in instarters[x]: parse X break; cases in instarters[y]: parse Y break; default: report syntax error Condition: starters[x] must be be disjoint from the the set set of of tokens that can immediately follow X * Condition: starters[x] and starters[y] must be be disjoint sets. 30

31 2) Starter Sets Informal Definition: The starter set of a RE X is the set of terminal symbols that can occur as the start of any string generated by X Example : starters[ (+ - ε)(0 1 9)* ] = +,-, 0,1,,9 Formal Definition: starters[ε] = starters[t] =t (where t is a terminal symbol) starters[x Y] = starters[x] starters[y] (if X generates ε) starters[x Y] = starters[x] (if not X generates ε) starters[x Y] = starters[x] starters[y] starters[x*] = starters[x] 31

32 2) Starter Sets (ctd) Informal Definition: The starter set of RE can be generalized to extended BNF Formal Definition: starters[n] = starters[x] (for production rules N ::= X) Example : starters[expression] = starters[primaryexp (Operator PrimaryExp)*] = starters[primaryexp] = starters[identifiers] starters[(expression)] = starters[a b c... z] ( = a, b, c,, z, ( 32

33 LL(1) grammars and left factorisation The original Mini Triangle grammar is not LL(1): For example: single-command ::= V-name := := Expression Identifier ( Expression )... V-name ::= Identifier Starters[V-name := Expression] = Starters[V-name] = Starters[Identifier] Starters[Identifier ( Expression )] = Starters[Identifier] NOT DISJOINT! 33

34 LL(1) grammars: left factorisation What happens when we generate a RD parser from a non LL(1) grammar? single-command ::= V-name := := Expression Identifier ( Expression )... private void void parsesinglecommand() switch (currenttoken.kind) case Token.IDENTIFIER: parse V-name := := Expression case Token.IDENTIFIER: parse Identifier ( Expression )...other cases... default: report syntax error wrong: overlapping cases 34

35 LL(1) grammars: left factorisation single-command ::= V-name := := Expression Identifier ( Expression )... Left factorisation (and substitution of V-name) single-command ::= Identifier ( := := Expression ( Expression ) )... 35

36 LL1 Grammars: left recursion elimination Command ::= single-command Command ; single-command What happens if we don t perform left-recursion elimination? public void voidparsecommand() switch (currenttoken.kind) case in in starters[single-command] parsesinglecommand(); case in in starters[command] parsecommand(); accept(token.semicolon); parsesinglecommand(); default: report syntax error wrong: overlapping cases 36

37 LL1 Grammars: left recursion elimination Command ::= single-command Command ; single-command Left recursion elimination Command ::= single-command (; (; single-command)* 37

38 Systematic Development of RD Parser (1) Express grammar in EBNF (2) Grammar Transformations: Left factorization and Left recursion elimination (3) Create a parser class with private variable currenttoken methods to call the scanner: accept and acceptit (4) Implement private parsing methods: add private parsen method for each non terminal N public parse method that gets the first token form the scanner calls parses (S is the start symbol of the grammar) 38

39 Algorithm to convert EBNF into a RD parser The conversion of an EBNF specification into a Java implementation for a recursive descent parser is straightforward We can describe the algorithm by a set of mechanical rewrite rules N ::= ::= X private void void parsen() parse X 39

40 Algorithm to convert EBNF into a RD parser parse tt accept(t); parse N parsen(); where ttis is a terminal where N is is a non-terminal parse ε // // a dummy statement parse XY XY parse X parse Y 40

41 Algorithm to convert EBNF into a RD parser parse X* X* while (currenttoken.kind is is in in starters[x]) parse X parse X Y X Y switch (currenttoken.kind) cases in instarters[x]: parse X break; cases in instarters[y]: parse Y break; default: report syntax error 41

42 LL(1) Grammars The presented algorithm to convert EBNF into a parser does not work for all possible grammars. It only works for so called LL(1) grammars. What grammars are LL(1)? Basically, an LL1 grammar is a grammar which can be parsed with a top-down parser with a lookahead (in the input stream of tokens) of one token. How can we recognize that a grammar is (or is not) LL1? We can deduce the necessary conditions from the parser generation algorithm. There is a formal definition we can use. 42

43 parse X* X* LL 1 Grammars while (currenttoken.kind is is in in starters[x]) parse X parse X Y X Y switch (currenttoken.kind) cases in instarters[x]: parse X break; cases in instarters[y]: parse Y break; default: report syntax error Condition: starters[x] must be be disjoint from the the set set of of tokens that can immediately follow X * Condition: starters[x] and starters[y] must be be disjoint sets. 43

44 Formal definition of LL(1) A grammar G is LL(1) iff for each set of productions M ::= X 1 X 2 X n : 1. starters[x 1 ], starters[x 2 ],, starters[x n ] are all pairwise disjoint 2. If X i =>* ε then starters[x j ] follow[x]=ø, for 1 j n.i j If G is ε-free then 1 is sufficient 44

45 What does X i =>* ε mean? Derivation It means a derivation from X i leading to the empty production What is a derivation? A grammar has a derivation: αaβ => αγβ iff A γ P (Sometimes A ::= γ ) =>* is the transitive closure of => Example: G = (E, a,+,*,(,), P, E) where P = E E+E, E E*E, E a, E (E) E => E+E => E+E*E => a+e*e => a+e*a => a+a*a E =>* a+a*a 45

46 Follow Sets Follow(A) is the set of prefixes of strings of terminals that can follow any derivation of A in G $ follow(s) (sometimes <eof> follow(s)) if(b αaβ) P, then first(β) follow(b) follow(a) The definition of follow usually results in recursive set definitions. In order to solve them, you need to do several iterations on the equations. 46

47 A few provable facts about LL(1) grammars No left-recursive grammar is LL(1) No ambiguous grammar is LL(1) Some languages have no LL(1) grammar A ε-free grammar, where each alternative X j for N ::= X j begins with a distinct terminal, is a simple LL(1) grammar 47

48 Converting EBNF into RD parsers The conversion of an EBNF specification into a Java implementation for a recursive descent parser is so mechanical that it can easily be automated! => JavaCC Java Compiler Compiler 48

49 JavaCC JavaCC is a parser generator JavaCC can be thought of as Lex and Yacc for implementing parsers in Java JavaCC is based on LL(k) grammars JavaCC transforms an EBNF grammar into an LL(k) parser The lookahead can be change by writing LOOKAHEAD( ) The JavaCC can have action code written in Java embedded in the grammar JavaCC has a companion called JJTree which can be used to generate an abstract syntax tree 49

50 JavaCC and JJTree JavaCC is a parser generator Inputs a set of token definitions, grammar and actions Outputs a Java program which performs lexical analysis Finding tokens Parses the tokens according to the grammar Executes actions JJTree is a preprocessor for JavaCC Inputs a grammar file Inserts tree building actions Outputs JavaCC grammar file From this you can add code to traverse the tree to do static analysis, code generation or interpretation. 50

51 JavaCC and JJTree 51

52 JavaCC input format One file with extension.jj containing Header Token specifications Grammar Example: TOKEN: <INTEGER_LITERAL: ([ 1-9 ]([ 0-9 ])* 0 )> void StatementListReturn() : (Statement())* return Expression() ; 52

53 JavaCC token specifications use regular expressions Characters and strings must be quoted ;, int, while Character lists [ ] is shorthand for [ a - z ] matches a b c z [ a, e, i, o,u ] matches any vowel [ a - z, A - Z ] matches any letter Repetition shorthand with * and + [ a - z, A - Z ]* matches zero or more letters [ a - z, A - Z ]+ matches one or more letters Shorthand with? provides for optionals: ( + - )?[ 0-9 ]+ matches signed and unsigned integers Tokens can be named TOKEN : <IDENTIFIER:<LETTER>(<LETTER> <DIGIT>)*> TOKEN : <LETTER: [ a - z, A - Z ] > <DIGIT:[ 0-9 ]> Now <IDENTIFIER> can be used in defining syntax 53

54 A bigger example options LOOKAHEAD=2; PARSER_BEGIN(Arithmetic) public class Arithmetic PARSER_END(Arithmetic) SKIP : " " "\r" "\t" TOKEN: < NUMBER: (<DIGIT>)+ ( "." (<DIGIT>)+ )? > < DIGIT: ["0"-"9"] > double expr(): term() ( "+" expr() "-" expr() )* double term(): unary() ( "*" term() "/" term() )* double unary(): "-" element() element() double element(): <NUMBER> "(" expr() ")" 54

55 Generating a parser with JavaCC javacc filename.jj generates a parser with specified name Lots of.java files javac *.java Compile all the.java files Note the parser doesn t do anything on its own. You have to either Add actions to grammar by hand Use JJTree to generate actions for building AST Use JBT to generate AST and visitors 55

56 Adding Actions by hand options LOOKAHEAD=2; PARSER_BEGIN(Calculator) public class Calculator public static void main(string args[]) throws ParseException Calculator parser = new Calculator(System.in); while (true) parser.parseoneline(); PARSER_END(Calculator) SKIP : " " "\r" "\t" TOKEN: < NUMBER: (<DIGIT>)+ ( "." (<DIGIT>)+ )? > < DIGIT: ["0"-"9"] > < EOL: "\n" > void parseoneline(): double a; a=expr() <EOL> System.out.println(a); <EOL> <EOF> System.exit(-1); 56

57 Adding Actions by hand (ctd.) double expr(): double a; double b; a=term() ( "+" b=expr() a += b; "-" b=expr() a -= b; )* return a; double term(): double a; double b; a=unary() ( "*" b=term() a *= b; "/" b=term() a /= b; )* return a; double unary(): double a; "-" a=element() return -a; a=element() return a; double element(): Token t; double a; t=<number> return Double.parseDouble(t.toString()); "(" a=expr() ")" return a; 57

58 Using JJTree JJTree is a preprocessor for JavaCC JTree transforms a bare JavaCC grammar into a grammar with embedded Java code for building an AST Classes Node and SimpleNode are generated Can also generate classes for each type of node All AST nodes implement interface Node Useful methods provided include: Public void jjtgetnumchildren()- returns the number of children Public void jjtgetchild(int i) - returns the i th child The state is in a parser field called jjtree The root is at Node rootnode() You can display the tree with ((SimpleNode)parser.jjtree.rootNode()).dump( ); JJTree supports the building of abstract syntax trees which can be traversed using the visitor design pattern 58

59 JBT JBT Java Tree Builder is an alternative to JJTree It takes a plain JavaCC grammar file as input and automatically generates the following: A set of syntax tree classes based on the productions in the grammar, utilizing the Visitor design pattern. Two interfaces: Visitor and ObjectVisitor. Two depth-first visitors: DepthFirstVisitor and ObjectDepthFirst, whose default methods simply visit the children of the current node. A JavaCC grammar with the proper annotations to build the syntax tree during parsing. New visitors, which subclass DepthFirstVisitor or ObjectDepthFirst, can then override the default methods and perform various operations on and manipulate the generated syntax tree. 59

60 The Visitor Pattern For object-oriented programming the visitor pattern enables the definition of a new operator on an object structure without changing the classes of the objects When using visitor pattern The set of classes must be fixed in advance Each class must have an accept method Each accept method takes a visitor as argument The purpose of the accept method is to invoke the visitor which can handle the current object. A visitor contains a visit method for each class (overloading) A method for class C takes an argument of type C The advantage of Visitors: New methods without recompilation! 60

61 Summary: Parser Generator Tools JavaCC is a Parser Generator Tool It can be thought of as Lex and Yacc for Java There are several other Parser Generator tools for Java We shall look at some of them later Beware! To use Parser Generator Tools efficiently you need to understand what they do and what the code they produce does Note, there can be bugs in the tools and/or in the code they generate! 61

62 So what can I do with this in my project? Language Design What type of language are you designing? C like, Java Like, Scripting language, Functional.. Does it fit with existing paradigms/generations? Which language design criteria to emphasise? Readability, writability, orthogonality, What is the syntax? Informal description CFG in EBNF (Human readable, i.e. may be ambiguous) LL(1) or (LALR(1)) grammar Grammar transformations left-factorization, left-recursion elimination, substitution Compiler Implementation Recursive Decent Parser (Tools generated Parser) 62