CC 4181 Compiler Construction Parsing 1 Outline Top-down v.s. Bottom-up Top-down parsing Recursive-descent parsing LL1) parsing LL1) parsing algorithm First and follow sets Constructing LL1) parsing table Error recovery Bottom-up parsing hift-reduce parsers LR0) parsing LR0) items Finite automata of items LR0) parsing algorithm LR0) grammar LR1) parsing LR1) parsing algorithm LR1) grammar Parsing conflict Parsing 2 Introduction Parsing is a process that constructs a syntactic structure i.e. parse tree) from the stream of tokens. We already learned how to describe the syntactic structure of a language using context-free) grammar. o, a parser only needs to do this? tream of tokens Context-free grammar Parser Parse tree Parsing 3 1
Top Down Parsing Bottom Up Parsing A parse tree is created from root to leaves The traversal of parse trees is a preorder traversal Tracing leftmost derivation Two types: Backtracking parser Predictive parser Backtracking: Try different structures and backtrack if it does not matched the input A parse tree is created from leaves to root The traversal of parse trees is a reversal of postorder traversal Tracing rightmost derivation More powerful than topdown parsing Predictive: Guess the structure of the parse tree from the next input Parsing 4 Parse Trees and Derivations E E E + id E * E id id Top-down parsing E E + E id E * E id id Bottom-up parsing E E + E id + E id + E * E id + id * E id + id * id E E + E E + E * E E + E * id E + id * id id + id * id Parsing 5 TOP DOWN PARING Parsing 6 2
Top-down Parsing What does a parser need to decide? Which production rule is to be used at each point of time? How to guess? What is the guess based on? What is the next token? Reserved word if, open parentheses, etc. What is the structure to be built? If statement, expression, etc. Parsing 7 Top-down Parsing Why is it difficult? Cannot decide until later Next token: if tructure to be built: t t Matchedt Unmatchedt Unmatchedt if E) t if E) Matchedt else Unmatchedt Matchedt if E) Matchedt else Matchedt... Production with empty string Next token: id tructure to be built: par par parlist parlist exp, parlist exp Parsing 8 Recursive-Descent Write one procedure for each set of productions with the same nonterminal in the LH Each procedure recognizes a structure described by a nonterminal. A procedure calls other procedures if it needs to recognize other structures. A procedure calls match procedure if it needs to recognize a terminal. Parsing 9 3
Recursive-Descent: Example E E O F F O + - F E ) id E ::= F {O F} O ::= + - F ::= E ) id procedure E { E; O; F; } procedure F { switch token { case : match ); E; match ) ); case id: matchid); default: error; } } For this grammar: We cannot decide which rule to use for E, and If we choose E E O F, it leads to infinitely recursive loops. Rewrite the grammar into EBNF procedure E { F; while token=+ or token=-) { O; F; } } Parsing 10 Match procedure procedure matchexptok) { if token==exptok) then gettoken else error } The token is not consumed until gettoken is executed. Parsing 11 Problems in Recursive-Descent Difficult to convert grammars into EBNF Cannot decide which production to use at each point Cannot decide when to use -production A Parsing 12 4
LL1) LL1) Parsing Read input from L) left to right imulate L) leftmost derivation 1 lookahead symbol Use stack to simulate leftmost derivation Part of sentential form produced in the leftmost derivation is stored in the stack. Top of stack is the leftmost nonterminal symbol in the fragment of sentential form. Parsing 13 Concept of LL1) Parsing imulate leftmost derivation of the input. Keep part of sentential form in the stack. If the symbol on the top of stack is a terminal, try to match it with the next input token and pop it out of stack. If the symbol on the top of stack is a nonterminal X, replace it with Y if we have a production rule X Y. Which production will be chosen, if there are both X Y and X Z? Parsing 14 Example of LL1) Parsing E TX FNX E)NX TX)NX FNX)NX nnx)nx nx)nx natx)nx n+tx)nx n+fnx)nx n+e)nx)nx n+tx)nx)nx n+fnx)nx)nx n+nnx)nx)nx n+nx)nx)nx n+n)nx)nx n+n)x)nx n+n))nx n+n))mfnx n+n))*fnx n+n))*nnx n+n))*nx n+n))*n F n T N n + n ) ) * n $ E X n A + F ) E T X T N X A T X Finished E X M * A + - T F N ) F n N M F N T N M * F E ) n E X $ Parsing 15 5
LL1) Parsing Algorithm Push the start symbol into the stack WHILE stack is not empty $ is not on top of stack) and the stream of tokens is not empty the next input token is not $) WITCH Top of stack, next token) CAE terminal a, a): Pop stack; Get next token CAE nonterminal A, terminal a): IF the parsing table entry M[A, a] is not empty THEN Get A X 1 X 2... X n from the parsing table entry M[A, a] Pop stack; Push X n... X 2 X 1 into stack in that order ELE Error CAE $,$): Accept OTHER: Error Parsing 16 LL1) Parsing Table If the nonterminal N is on the top of stack and the next token is t, which production rule to use? Choose a rule N X such that X * ty or X * and * WNtY t N X Y Q t N X t Y Parsing 17 First et Let X be or be in V or T. FirstX ) is the set of the first terminal in any sentential form derived from X. If X is a terminal or, then FirstX ) ={X }. If X is a nonterminal and X X 1 X 2... X n is a rule, then FirstX 1 ) -{} is a subset of FirstX) FirstX i )-{} is a subset of FirstX) if for all j<i FirstX j ) contains {} is in FirstX) if for all j n FirstX j )contains Parsing 18 6
Examples of First et exp exp addop term term st ifst ifst other if exp ) st elsepart addop + - elsepart else st term term mulop factor exp 0 1 factor mulop * Firstexp) = {0,1} factor exp) num Firstelsepart) = {else, } Firstaddop) = {+, -} Firstmulop) = {*} Firstfactor) = {, num} Firstterm) = {, num} Firstexp) = {, num} Firstifst) Firstst) = {if} = {if, other} Parsing 19 Algorithm for finding FirstA) For all terminals a, Firsta) = {a} For all nonterminals A, FirstA) := {} While there are changes to any FirstA) For each rule A X 1 X 2... X n For each X i in {X 1, X 2,, X n } If for all j<i FirstX j ) contains, Then add FirstX i )-{} to FirstA) If is in FirstX 1 ), FirstX 2 ),..., and FirstX n ) Then add to FirstA) If A is a terminal or, then FirstA) = {A}. If A is a nonterminal, then for each rule A X 1 X 2... X n, FirstA) contains FirstX 1 ) - {}. If also for some i<n, FirstX 1 ), FirstX 2 ),..., and FirstX i ) contain, then FirstA) contains FirstX i+1 )-{}. If FirstX 1 ), FirstX 2 ),..., and FirstX n ) contain, then FirstA) also contains. Parsing 20 Finding First et: An Example exp term exp exp addop term exp addop + - term factor term term mulop factor term mulop * factor exp ) num exp exp addop term term mulop factor First + - num * num Parsing 21 7
Follow et Let $ denote the end of input tokens If A is the start symbol, then $ is in FollowA). If there is a rule B X A Y, then FirstY) - {} is in FollowA). If there is production B X A Y and is in FirstY), then FollowA) contains FollowB). Parsing 22 Algorithm for Finding FollowA) Follow) = {$} FOR each A in V-{} FollowA)={} WHILE change is made to some Follow sets FOR each production A X 1 X 2... X n, FOR each nonterminal X i Add FirstX i+1 X i+2...x n )-{} into FollowX i ). NOTE: If i=n, X i+1 X i+2...x n = ) IF is in FirstX i+1 X i+2...x n ) THEN Add FollowA) to FollowX i ) If A is the start symbol, then $ is in FollowA). If there is a rule A Y X Z, then FirstZ) - {} is in FollowX). If there is production B X A Y and is in FirstY), then FollowA) contains FollowB). Parsing 23 Finding Follow et: An Example exp term exp exp addop term exp addop + - term factor term term mulop factor term mulop * factor exp ) num exp exp addop term term mulop factor First num + - + - num * * num Follow $ ) $ ) + - $ ) Parsing 24 8
Constructing LL1) Parsing Tables FOR each nonterminal A and a production A X FOR each token a in FirstX) A X is in MA, a) IF is in FirstX) THEN FOR each element a in FollowA) Add A X to MA, a) Parsing 25 Example: Constructing LL1) Parsing Table First Follow exp {, num} {$,)} exp {+,-, } {$,)} addop {+,-} {,num} term {,num} {+,-,),$} term {*, } {+,-,),$} mulop {*} {,num} factor {, num} {*,+,-,),$} 1 exp term exp 2 exp addop term exp 3 exp 4 addop + 5 addop - 6 term factor term 7 term mulop factor term 8 term 9 mulop * 10 factor exp ) 11 factor num exp exp addop term term mulop factor ) + - * n $ 1 1 3 2 2 3 4 5 6 6 8 8 8 7 8 9 10 11 Parsing 26 LL1) Grammar A grammar is an LL1) grammar if its LL1) parsing table has at most one production in each table entry. Parsing 27 9
LL1) Parsing Table for non-ll1) Grammar 1 exp exp addop term 2 exp term 3 term term mulop factor 4 term factor 5 factor exp ) 6 factor num 7 addop + 8 addop - 9 mulop * Firstexp) = {, num } Firstterm) = {, num } Firstfactor) = {, num } Firstaddop) = { +, - } Firstmulop) = { * } ) + - * num $ exp 1,2 1,2 term 3,4 3,4 factor 5 6 addop 7 8 mulop 9 Parsing 28 Causes of Non-LL1) Grammar What causes grammar being non-ll1)? Left-recursion Left factor Parsing 29 Immediate left recursion A A X Y Left Recursion A=Y X* A A X 1 A X 2 A X n Y 1 Y 2... Y m A={Y 1, Y 2,, Y m } {X 1, X 2,, X n }* General left recursion A => X =>* A Y Can be removed very easily A Y A, A X A A Y 1 A Y 2 A... Y m A, A X 1 A X 2 A X n A Can be removed when there is no empty-string production and no cycle in the grammar Parsing 30 10
Removal of Immediate Left Recursion exp exp + term exp - term term term term * factor factor factor exp ) num Remove left recursion exp term exp exp = term term)* exp + term exp - term exp term factor term term = factor * factor)* term * factor term factor exp ) num Parsing 31 Bad News! General Left Recursion Can only be removed when there is no emptystring production and no cycle in the grammar. Good News!!!! Never seen in grammars of any programming languages Parsing 32 Left Factoring Left factor causes non-ll1) Given A X Y X Z. Both A X Y and A X Z can be chosen when A is on top of stack and a token in FirstX) is the next token. A X Y X Z can be left-factored as A X A and A Y Z Parsing 33 11
Example of Left Factor ift if exp ) st else st if exp ) st can be left-factored as ift if exp ) st elsepart elsepart else st seq st ; seq st can be left-factored as seq st seq seq ; seq Parsing 34 BOTTOM UP PARING Parsing 35 Bottom-up Parsing Use explicit stack to perform a parse imulate rightmost derivation R) from left L) to right, thus called LR parsing More powerful than top-down parsing Left recursion does not cause problem Two actions hift: take next input token into the stack Reduce: replace a string B on top of stack by a nonterminal A, given a production A B Parsing 36 12
Example of hift-reduce Parsing Grammar ) Parsing actions tack Input Action $ ) ) $ shift $ ) ) $ shift $ ) ) $ reduce $ ) ) $ shift $ ) ) $ reduce $ ) ) $ reduce ) $ ) $ shift $ ) $ reduce $ ) $ reduce ) $ $ accept Reverse of rightmost derivation from left to right 1 ) ) 2 ) ) 3 ) ) 4 ) ) 5 ) ) 6 ) ) 7 ) 8 ) 9 ) 10 Parsing 37 Example of hift-reduce Parsing Grammar ) Parsing actions tack Input Action $ ) ) $ shift $ ) ) $ shift $ ) ) $ reduce $ ) ) $ shift $ ) ) $ reduce $ ) ) $ reduce ) $ ) $ shift $ ) $ reduce $ ) $ reduce ) $ $ Viable prefix accept 1 ) ) 2 ) ) 3 ) ) 4 ) ) 5 ) ) 6 ) ) 7 ) 8 ) 9 ) 10 handle Parsing 38 Terminology Right sentential form sentential form in a rightmost derivation Viable prefix sequence of symbols on the parsing stack Handle right sentential form + position where reduction can be performed + production used for reduction LR0) item production with distinguished position in its RH Right sentential form ) ) ) Viable prefix ), ),, ), ),,, Handle ). with ). with ). ) with ) LR0) item ). ).. ). ). ) Parsing 39 13
hift-reduce parsers There are two possible actions: shift and reduce Parsing is completed when the input stream is empty and the stack contains only the start symbol The grammar must be augmented a new start symbol is added a production is added To make sure that parsing is finished when is on top of stack because never appears on the RH of any production. Parsing 40 LR0) parsing Keep track of what is left to be done in the parsing process by using finite automata of items An item A w. B y means: A w B y might be used for the reduction in the future, at the time, we know we already construct w in the parsing process, if B is constructed next, we get the new item A w B. Y Parsing 41 LR0) items LR0) item production with a distinguished position in the RH Initial Item Item with the distinguished position on the leftmost of the production Complete Item Item with the distinguished position on the rightmost of the production Closure Item of x Item x together with items which can be reached from x via -transition Kernel Item Original item, not including closure items Parsing 42 14
Finite automata of items Grammar: ) Items:...).).) ). ).....)..) )..) ) ). Parsing 43 DFA of LR0) Items...)..).) ) )...)..).)...) )..). ) ). ). Parsing 44 LR0) parsing algorithm Item in state token Action A-> x.by where B is terminal B shift B and push state s containing A -> xb.y A-> x.by where B is terminal not B error A -> x. - reduce with A -> x i.e. pop x, backup to the state s on top of stack) and push A with new state ds,a) ->. none accept ->. any error Parsing 45 15
LR0) Parsing Table A.A A A.A) A.a 0 a a A.A) A.A) A A.a 3 A A. 1 A a. 2 A A.) 4 ) A A). 5 tate Action Rule a ) A 0 shift 3 2 1 1 reduce A -> A 2 reduce A -> a 3 shift 3 2 4 4 shift 5 5 reduce A -> A) Parsing 46 Example of LR0) Parsing tate Action Rule a ) A 0 shift 3 2 1 1 reduce A -> A 2 reduce A -> a 3 shift 3 2 4 4 shift 5 tack Input 5 reduce Action A -> A) $0 a ) ) $ shift $03 a ) ) $ shift $033 a ) ) $ shift $033a2 ) ) $ reduce $033A4 ) ) $ shift $033A4)5 ) $ reduce $03A4 ) $ shift $03A4)5 $ reduce $0A1 $ accept Parsing 47 Non-LR0)Grammar Conflict hift-reduce conflict A state contains a complete item A x. and a shift item A x.by Reduce-reduce conflict A state contains more than one complete items. A grammar is a LR0) grammar if there is no conflict in the grammar...). 0.).). 2. 1.) 3 )..). 4 ). 5 ) Parsing 48 16
LR1) parsing imple LR with 1 lookahead symbol Examine the next token before deciding to shift or reduce If the next token is the token expected in an item, then it can be shifted into the stack. If a complete item A x. is constructed and the next token is in FollowA), then reduction can be done using A x. Otherwise, error occurs. Can avoid conflict Parsing 49 LR1) parsing algorithm Item in state token Action A-> x.by B is terminal) B shift B and push state s containing A -> xb.y A-> x.by B is terminal) not B error A -> x. in reduce with A -> x i.e. pop x, FollowA) backup to the state s on top of stack) and push A with new state ds,a) A -> x. not in error FollowA) ->. none accept ->. any error Parsing 50 Conflict hift-reduce conflict LR1) grammar A state contains a shift item A x.wy such that W is a terminal and a complete item B z. such that W is in FollowB). Reduce-reduce conflict A state contains more than one complete item with some common Follow set. A grammar is an LR1) grammar if there is no conflict in the grammar. Parsing 51 17
LR1) Parsing Table A A) a A.A A A A. 1 A.A) A.a a 0 A.A) A.A) A.a 3 a A A a. 2 A A.) 4 ) A A). 5 tate a ) $ A 0 3 2 1 1 AC 2 R2 3 3 2 4 4 5 5 R1 Parsing 52 LR1) Grammar not LR0)..). 0.).). 2. 1.) 3 )..). 4 ) ) tate ) $ 0 2 R2 R2 1 1 AC 2 2 R2 R2 3 3 4 4 2 R2 R2 5 5 R1 R1 ). 5 Parsing 53 Disambiguating Rules for Parsing Conflict hift-reduce conflict Prefer shift over reduce In case of nested if statements, preferring shift over reduce implies most closely nested rule for dangling else Reduce-reduce conflict Error in design Parsing 54 18
Dangling Else. 0.I.other I.if I.if else other.other 3 other I if. 5 I if. else I if if other other. 1 I. 2 I else I if. 4 I if. else.i.other I.if I.if else I if I if else. 6.I.other I.if I.if else I.if else 7 state if else other $ I 0 4 3 1 2 1 ACC 2 R1 R1 3 R2 R2 4 4 3 5 2 5 6 R3 6 4 3 7 2 7 R4 R4 Parsing 55 LALR1) parsing Goal: reduce number of states in LR1) parser. ome states LR1) automaton have the same core items and differ only in the possible lookahead. tates I 3 and I 3 ', I 5 and I 5 ', I 7 and I 7 ', I 8 and I 8 ' We shrink our parser by merging such states. LR : 10 states, LR1): 14 states, LALR1) : 10 states 56 Conflicts in LALR1) parsing Most conflicts that existed in LR1) parser can be eliminated with LALR1) Can LALR1) parsers introduce conflicts that did not exist in the LR1) parser? Unfortunately YE. BUT, only reduce/reduce conflicts. YACC generates LALR1) parser 57 19