EDA180: Compiler Construc6on Top- down parsing Görel Hedin Revised: 2013-01- 30a
Compiler phases and program representa6ons source code Lexical analysis (scanning) Intermediate code genera6on tokens intermediate code Syntac6c analysis (parsing) Op6miza6on AST APributed AST intermediate code Seman6c analysis Analysis Machine code genera6on Synthesis machine code 2
A closer look at the parser text Scanner tokens Defined by: regular expressions Parser Pure parsing concrete parse tree (implicit) This lecture context- free grammar AST building AST abstract grammar 3
Different parsing algorithms Unambiguous LR LL Ambiguous This lecture All context- free grammars LL: Left-to-right scan Leftmost derivation Builds tree top-down Simple to understand LR: Left-to-right scan Rightmost derivation Builds tree bottom-up More powerful 4
CompoundStmt IfStmt LL and LR parsers, main idea Id Assign Id Assign Id Id if ID then ID = ID ; ID... if ID then ID = ID ; ID... LL(1): decides to build Assign after seeing the first token of its subtree. The tree is built top down. LR(1): decides to build Assign after seeing the first token following its subtree. The tree is built bottom up. The token is called lookahead. LL(k) and LR(k) use k lookahead tokens. 5
Recursive- descent parsing A way of programming an LL(1) parser by recursive method calls Assume an EBNF grammar with exactly one produc6on rule for each nonterminal symbol. For each nonterminal, a method is constructed. A nonterminal method matches tokens and calls other nonterminal methods, according to the grammar. If the lookahead token does not match, an error is reported. A -> B C D B -> a C b D C ->... D ->... 6
Example Java implementa6on: overview statement -> assignment compoundstmt assignment-> ID ASSIGN expr SEMICOLON compoundstmt -> LBRACE statement* RBRACE expr ->... class Parser { private int token; "// current lookahead token void accept(int t) {...} "// accept t and read in next token void error(string str) {...}"// generate error message void statement() {...} void assignment () {...} void compoundstmt () {...}... } 7
Example: recursive descent methods statement -> assignment compoundstmt assignment-> ID ASSIGN expr SEMICOLON compoundstmt -> LBRACE statement* RBRACE class Parser { void statement() { switch(token) { case ID: assignment(); break; case LBRACE: compoundstmt(); break; default: error("expecting statement, found: " + token); } } void assignment() { accept(id); accept(assign); expr(); accept(semicolon); } void compoundstmt() { accept(lbrace); while (token!=rbrace) { statement(); } accept(rbrace); }... } 8
Example: Parser skeleton details statement -> assignment compoundstmt assignment-> ID ASSIGN expr SEMICOLON compoundstmt -> LBRACE statement* RBRACE expr ->... class Parser { final static int ID=1, WHILE=2, DO=3, ASSIGN=4,...; private int token; "// current lookahead token void accept(int t) { "// accept t and read in next token if (token==t) { token = nexttoken(); } else { error("expected " + t + ", but found " + token); } } void error(string str) {...}"// generate error message private int nexttoken() {...} // read next token from scanner void statement()...... } 9
Are these grammars LL(1)? expr -> name params name Common prefix expr -> expr "+" term term term -> ID Led recursion What would happen in a recursive- descent parser? Could they be LL(2)? LL(k)? 10
Dealing with common prefix of limited length: Local lookahead LL(2) grammar: statement -> assignment compoundstmt callstmt assignment-> ID ASSIGN expr SEMICOLON compoundstmt -> LBRACE statement* RBRACE callstmt -> ID LPAR expr RPAR SEMICOLON void statement()... 11
Dealing with common prefix of limited length: Local lookahead LL(2) grammar: statement -> assignment compoundstmt callstmt assignment-> ID ASSIGN expr SEMICOLON compoundstmt -> LBRACE statement* RBRACE callstmt -> ID LPAR expr RPAR SEMICOLON void statement() { switch(token) { case ID: if (lookahead(2) == ASSIGN) { assignment(); } else { callstmt(); } break; case LBRACE: compoundstmt(); break; default: error("expecting statement, found: " + token); } } 12
Common prefix? If two produc6ons can derive a sentence star6ng in the same way, they share a common prefix. A -> a B A -> a C B -> b C -> c A -> B a A -> B b B -> c d A -> a B B -> a C B -> b C C -> c A has two rules that can derive the prefix a The grammar is LL(2) A has two rules that can derive the prefix c d The grammar is LL(3) No problem. The two rules that start the same cannot be derived from the same nonterminal. The grammar is LL(1) Which nonterminals have common prefix produc6ons? How long is the common prefix? Is the grammar LL(1), LL2(),...? 13
Common prefix? A -> B A -> C A -> D B -> b a C -> b d D -> e A has two rules that can derive the prefix b The grammar is LL(2) A -> B a A -> B b B -> B c B -> d A has two rules that can derive the prefix d c* So, the prefix can become arbitrarily long. The grammar is not LL(k), no matter what k we use. We need to rewrite the grammar, or use another parsing method. Which nonterminals have common prefix produc6ons? How long is the common prefix? Is the grammar LL(1), LL2(),...? 14
Elimina6ng the common prefix Rewrite to an equivalent grammar without the common prefix Exp -> Name Params Exp -> Name With common prefix - not LL(1) 15
Elimina6ng the common prefix Rewrite to an equivalent grammar without the common prefix Exp -> Name Params Exp -> Name With common prefix - not LL(1) Exp -> Name OptParams OptParams -> Params OptParams -> ε Without common prefix - LL(1) Eliminating a common prefix this way is called "left factoring". 16
Elimina6ng the common prefix Rewrite to an equivalent grammar without the common prefix A -> B A -> C B -> b a B -> e D B -> f C -> b d D -> B C Indirect common prefix 17
Elimina6ng the common prefix Rewrite to an equivalent grammar without the common prefix A -> B A -> C B -> b a B -> e D B -> f C -> b d D -> B C Indirect common prefix First, make the common prefix directly visible: Substitute all B right-hand sides into the A -> B rule We can't remove the B rules since B is used in other places. Similarly for the A -> C rule A -> b a A -> e D A -> f B -> b a B -> e D B -> f A -> b d C -> b d D -> B C Direct common prefix Then, eliminate the direct common prefix, as previously. 18
Dealing with led recursion in LL parsers Method 1: Rewrite to an equivalent grammar without led recursion (A bit cumbersome) Left-recursive grammar not LL(k) E -> E "+" T E -> T T-> ID Rewrite to right-recursion! But there is now a common prefix! Still not LL(k). E -> T "+" E E -> T T-> ID Eliminate the common prefix. The grammar is now LL(1) E -> T E' E' -> "+" E E' -> ε T-> ID A left-recursive AST can be built during the right-recursive parse. 19
Dealing with led recursion in LL parsers Method 2: Rewrite to EBNF (Easy!) Left-recursive grammar not LL(k) E -> E "+" T E -> T T-> ID Rewrite to EBNF! E -> T ( "+" T )* T-> ID A left-recursive AST can be built during the iteration. 20
JavaCC: An LL- based parser generator CFG (in Java-like spec langauge) JavaCC Parser (in Java code) 21
JavaCC specifica6on CFG: statement -> assignment compoundstmt assignment-> ID ASSIGN expr SEMICOLON compoundstmt -> LBRACE statement* RBRACE JavaCC: void statement() : {} { assignment() compoundstmt() } void assignment() : {} { id() <ASSIGN> expr() <SEMICOLON> } void compoundstmt() : {} { <LBRACE> (statement())* <RBRACE> } void id() : {} { <ID> } Place where Java code can be added. You can also add Java code inside the rules. (For semantic actions, e.g. build the AST) Good idea to add a nonterminal id for ID tokens. This way you can avoid code duplication in the semantic actions. 22
Using local lookahead in JavaCC Local lookahead can be used to discriminate between an assignment and a procedure call: statement -> assignment callstmt whilestmt assignment -> ID ASSIGN expr SEMICOLON callstmt -> ID LPAR expr RPAR SEMICOLON JavaCC: void statement() : {} { LOOKAHEAD(2) assignment() callstmt() whilestmt() }... A lookahead of 2 tokens will be used before selecting assignment. If that fails, ordinary single-token lookahead will be used in the following alternatives. 23
Using EBNF in JavaCC Straight forward! expr -> term (PLUS term)* term -> factor (TIMES factor)* factor -> ID INT LPAR expr RPAR JavaCC: void expr() : {} { term() (<PLUS> term())* } void term() : {} { factor() (<TIMES> factor())* } void factor() : {} { id() intexpr() <LPAR> expr () <RPAR> } 24
Algorithm for construc6ng an LL(1) parser Fairly simple. The non-trivial part: how to select the correct production p for X, based on the lookahead token. X p1: X ->... p2: X ->... Which tokens can occur in the FIRST position?... t 1... t n t n+1... FIRST FOLLOW Can one of the productions derive the empty string? I.e., is it "NULLABLE"? If so, which tokens can occur in the FOLLOW position? 25
Steps in construc6ng an LL(1) parser 1. Write the grammar on canonical form 2. Analyze the grammar to construct a table. The table shows what production to select, given the current lookahead token. 3. Conflicts in the table? The grammar is not LL(1). 4. No conflicts? Straight forward implementation using table-driven parser or recursive descent. t 1 t 2 t 3 t 4 X 1 p1 p2 X 2 p3 p3 p4 26
Example: Construct the LL(1) table for this grammar: p1: statement -> assignment p2: statement -> compoundstmt p3: assignment -> ID "=" expr ";" p4: compoundstmt -> "{" statements "}" p5: statements -> statement statements p6: statements -> ε statement assignment compoundstmt statements ID "=" ";" "{" "}" For each production p: X -> γ, we are interested in: FIRST(γ) the tokens that occur first in a sentence derived from γ. NULLABLE(γ) is it possible to derive ε from γ? And if so: FOLLOW(X) the tokens that can occur immediately after an X-sentence. 27
Example: Construct the LL(1) table for this grammar: p1: statement -> assignment p2: statement -> compoundstmt p3: assignment -> ID "=" expr ";" p4: compoundstmt -> "{" statements "}" p5: statements -> statement statements p6: statements -> ε ID "=" ";" "{" "}" statement p1 p2 assignment p3 compoundstmt p4 statements p5 p5 p6 To construct the table, look at each production p: X -> γ. Compute the token set FIRST(γ). Add p to each corresponding entry for X. Then, check if γ is NULLABLE. If so, compute the token set FOLLOW(X), and add p to each corresponding entry for X. 28
Example: Dealing with End of File: p1: vardecl -> type ID optinit p2: type -> "integer" p3: type -> "boolean" "=" expr ";" p4: optinit -> "=" INT p5: optinit -> ε ID integer boolean "=" ";" INT vardecl type optinit 29
Example: Dealing with End of File: p0: S -> vardecl EOF p1: vardecl -> type ID optinit p2: type -> "integer" p3: type -> "boolean" "=" expr ";" p4: optinit -> "=" INT p5: optinit -> ε ID integer boolean "=" ";" INT EOF S p0 p0 vardecl p1 p1 type p2 p3 optinit p4 p5 30
Example: Ambiguous grammar: p1: E -> E "+" E p2: E -> ID p3: E -> INT E "+" ID INT 31
Example: Ambiguous grammar: p1: E -> E "+" E p2: E -> ID p3: E -> INT "+" ID INT E p1, p2 p1, p3 Collision in a table entry! The grammar is not LL(1) An ambiguous grammar is not even LL(k) adding more lookahead does not help. 32
Example: Unambiguous, but led- recursive grammar: p1: E -> E "*" F p2: E -> F p3: F -> ID p4: F -> INT E F "*" ID INT 33
Example: Unambiguous, but led- recursive grammar: p1: E -> E "*" F p2: E -> F p3: F -> ID p4: F -> INT "*" ID INT E p1,p2 p1,p2 F p3 p4 Collision in a table entry! The grammar is not LL(1) A grammar with left-recursion is not even LL(k) adding more lookahead does not help. 34
Example: Grammar with common prefix: p1: E -> F "*" E p2: E -> F p3: F -> ID p4: F -> INT p5: F -> "(" E ")" E F "*" ID INT "(" ")" 35
Example: Grammar with common prefix: p1: E -> F "*" E p2: E -> F p3: F -> ID p4: F -> INT p5: F -> "(" E ")" "*" ID INT "(" ")" E p1,p2 p1,p2 p1,p2 F p3 p4 p5 Collision in a table entry! The grammar is not LL(1) A grammar with common prefix is not LL(1). Some grammars with common prefix are LL(k), for some k, but not this one. 36
Example: Another grammar with common prefix: p1: Stmt -> ID "(" IdList ")" p2: Stmt -> ID "=" Exp Stmt... ID "(" ")" "=" 37
Example: Another grammar with common prefix: p1: Stmt -> ID "(" IdList ")" p2: Stmt -> ID "=" Exp Stmt... ID "(" ")" "=" p1, p2 Collision in a table entry! The grammar is not LL(1) A grammar with common prefix is not LL(1) But this grammar is LL(2) 38
We could create an LL(2) table! (global lookahead 2) p1: Stmt -> ID "(" IdList ")" p2: Stmt -> ID "=" Exp ID ID ID "(" ID "=" ID ")" "(" ID "(" "("... Stmt p1 p2... No conflicts! The grammar is LL(2)! But k > 1 gives very large tables inefficient! 39
A beper alterna6ve: Local lookahead! p1: Stmt -> ID "(" IdList ")" p2: Stmt -> ID "=" Exp Stmt "(" "="... p1 ID "(" ")" "=" p2 No collisions! A slightly more complex table structure for the local lookahead. Will be efficient. JavaCC can generate both LL(k) and local lookahead parsers. Using k > 1 is not recommended. Too slow parsing. Use local lookahead if needed. 40
Summary: construc6ng an LL(1) parser 1. Write the grammar on canonical form 2. Analyze the grammar using FIRST, NULLABLE, and FOLLOW. 3. Use the analysis to construct a table. The table shows what production to select, given the current lookahead token. 4. Conflicts in the table? The grammar is not LL(1). 5. No conflicts? Straight forward implementation using table-driven parser or recursive descent. 41
Summary ques6ons Construct a CFG for a simple part of a programming language. Construct a recursive descent parser for a simple language. Give typical examples of ambigui6es in CFGs What is the difference between LL(1) and LL(k)? What is a "common prefix", and how can it be eliminated? What is meant by "led factoring"? What is "led recursion" and how can it be eliminated? In what way can an LL syntax tree differ from the desired AST? Construct an EBNF grammar for conven6onal arithme6c expressions that respect standard precedence and associa6vity. What is NULLABLE(X), FIRST(X), and FOLLOW(X)? Construct an LL(1) table for a grammar. What does it mean if there is a collision in an LL(1) table? What is the difference between local lookahead and global lookahead? Why can it be useful to add an end- of- file rule to some grammars? How can we decide if a grammar is LL(1) or not? 42
Readings F4: Predic6ve parsing. Recursive descent. LL grammars and parsing. Led recursion and factoriza6on. Appel, chapter 3.2 43