Lexical and Syntax Analysis. Bottom-Up Parsing

Size: px

Start display at page:

Download "Lexical and Syntax Analysis. Bottom-Up Parsing"

Malcolm Fletcher
5 years ago
Views:

1 Lexical and Syntax Analysis Bottom-Up Parsing

2 Parsing There are two ways to construct derivation of a grammar. Top-Down: begin with start symbol; repeatedly replace an instance of a production s LHS with its RHS. Bottom-Up: begin with terminal symbols; repeatedly replace an instance of a production s RHS with its LHS. (See animation.)

3 Derivation in reverse Bottom-up parsers construct a derivation in reverse. Example: ( x + x ) ( v + x ) ( e + x ) ( e + v ) ( e + e ) e (Here we have a rightmost derivation in reverse.)

4 Shift-Reduce Parsing A non-deterministic procedure for bottom-up parsing involving: a stack holding grammar symbols; an input buffer holding the rest of the string to be parsed. Symbol $ is used to mark the bottom of the stack and the end of the input buffer.

5 Shift-Reduce Parsing There are four primary operations: 1. Shift. Consume the next input symbol and push it onto the stack. 2. Reduce. If the RHS of a production appears on top of the stack, then replace it with the LHS. 3. Accept. Announce success when the stack contains only the start symbol and the input is empty. 4. Error. Fail when the input is empty and no reduction can be applied.

6 Exercise 1 Consider the following grammar G. e e + t t t t * f f f ( e ) x Show steps taken by a shift-reduce parser to successfully parse the following input. x * x

7 Shift-reduce parsing is complete If an input string conforms to a grammar, then there exists at least one series of shift-reduce steps leading to acceptance. (See Aho et al. for a proof.)

8 Problem But how do we decide when to shift and when to reduce? Stack Input Action $ x * x $ Shift $ x * x $ Reduce by f x $ f * x $ Reduce by t f $ t * x $ Shift or Reduce? Both shift and reduce (by e t) are possible, but reduce leads to failure.

9 Possible solutions How do we decide when to shift and when to reduce? Solution 1: guess, and backtrack upon failure. (Inefficient!) Solution 2: LR parsing. L for left-to-right scan of the input, and R for constructing a rightmost derivation in reverse.

10 PART 1: LR(0) PARSING (When to shift and when to reduce) LR(0) Items Reducible Items LR(0) States LR(0) Automata Constructing an LR(0) Automaton LR(0) Parsing

11 LR(0) Items An item of a grammar H is a production of H with a dot at some point in the body. For example, the production e e + t has four possible items: e e + t e e + t e e + t e e + t

12 LR(0) Items Note that the production n ε has just one possible item: n

13 Intuition An item indicates how much of a production we have seen at some point during parsing. For example, the item e e + t indicates that we have just seen a substring derivable from e and that next we hope to see one derivable from + t.

14 Reducible items An item is reducible if it has a dot at the rightmost position. For example: t t * f Intuition: we have reached a point where we are ready to reduce t * f to t. Exception: an item defining the start symbol is not reducible.

15 LR(0) States A state of an LR parser is a set of items. For example, here is a state S 2 : S 2 e e + v v x v y In state S 2 we could be in the process of parsing one of three possible productions.

16 LR(0) Automaton An LR(0) automaton has states (sets of LR(0) items); and transitions labelled by grammar symbols. For example, here is a fragment of an LR(0) automaton: x S 2 e e + v v x v y S 5 v x v y S 6 v y S 3 e e + v

17 Constructing an LR(0) automaton An LR(0) automaton is defined by two main functions: GOTO(L, X): the LR(0) state reached by taking an X transition in state L. STATES(G): the set of all LR(0) states of grammar G. To define these we need an auxiliary function called CLOSURE.

18 The CLOSURE function If L is a set of items then CLOSURE(L) is the set of items such that: every item in L is in CLOSURE(L); if item n α m β is in CLOSURE(L) and m γ is a production then m γ is also in CLOSURE(L).

19 Exercise 2 Consider the following grammar P. s e e e + v v v x y Compute the CLOSURE of the set of items { s e }.

20 The GOTO function If L is a set of items and X is a grammar symbol and n α X β L then GOTO(L, X) is the CLOSURE of the set of all items n α X β

21 Exercise 3 Consider the following LR(0) state. S 0 s e e e + v e v v x v y Give the states that can be reached from S 0 on each of the following grammar symbols: e, v, x and y.

22 Exercise 4 Consider the following LR(0) state. S 1 s e e e + v Give the states that can be reached from S 1 on symbol +.

23 Constructing the LR(0) states Aim: to compute C, the set of all LR(0) states for some grammar G. Pre-processing step: if s is the start symbol of G, introduce a new start symbol s' and a new production s' s. (Motivation: success is detected when s is reduced to s'.)

24 The STATES function STATES(G) { } C := { CLOSURE({ s' s }) } repeat for each L C for each grammar symbol X C := C { GOTO(L, X) } until C is unchanging return C

25 Constructing the LR(0) automaton States: obtained by the STATES function. Transitions: for each state L and grammar symbol X compute GOTO(X, L). Final state: if s' is the start symbol and s' s is an item in state L then add a transition labelled $ from L to the accepting state.

26 Exercise 5 Construct an LR(0) automaton for the following grammar. s e e e + v v v x y (A new start symbol s has already been introduced.)

27 LR(0) Parsing An LR(0) parser has: an LR(0) automaton (the GOTO function in particular); a pointer next to the next character in the input string; a stack of states S, initially containing the start state of the LR(0) automaton; a current state, always at the top of the stack.

28 LR(0) Parsing while top(s) accept if n α is a reducible item in top(s) then pop α elements push( GOTO(top(S), n ) ) else if GOTO(top(S), *next) = L then next++ push(l) else raise syntax error Shift! Reduce!

29 Exercise 6 (page 1) Consider the LR(0) automaton: start v S 0 s e e e + v e v v x v y S 1 s e e e + v $ e accept + x y S 4 e v S 5 v x x S 2 e e + v v x v y v y S 6 v y S 3 e e + v

30 Exercise 6 (page 2) Show the steps taken by an LR(0) parser on the following input string. x + x For each step, show the stack, the input buffer, and the action taken.

31 Shift-Reduce Conflicts If at some state L there is a reducible item and a transition on a terminal symbol then: there exists a shift-reduce conflict; and the grammar is not an LR(0) grammar. Grammar P contains no shiftreduce conflicts.

32 Reduce-Reduce Conflicts If at some state L there is a more than one reducible item then: there exists a reduce-reduce conflict; and the grammar is not an LR(0) grammar. Grammar P contains no reducereduce conflicts.

33 PART 2: LR(1) PARSING (When to shift and when to reduce) LR(1) Items Reducible Items SLR(1) Parsing LR(1) Parsing LALR(1) Parsing

34 Motivation LR(0) automata say when to shift and when to reduce. But conflicts are very common: many grammars are not LR(0) grammars. If we enrich the structure of LR(0) items, we can decrease conflicts significantly.

35 LR(1) Items Idea: extend LR(0) items to have the form n α β { σ } where { σ } is a set of terminal symbols called the lookahead set.

36 Reducible items An item is reducible if it has a dot at the rightmost position and the next input symbol is in the lookahead set. Example: t t * f { *, +, ) } Here we have reached a point where we can reduce t * f to t provided that the next input symbol is *, + or ).

37 LR(1) parsing LR(1) parsing is like LR(0) parsing except that: LR(1) states contain LR(1) items rather than LR(0) items. An item is only reducible if the next input symbol is in the lookahead set.

38 Aside: an equivalence If an LR(1) state contains several items differing only in their lookahead sets, e.g. t t * f { *, + } t t * f { +, ) } then these are equivalent to the single item formed by unioning the lookahead sets, e.g. t t * f { *, +, ) }

39 LR(1) parsing There are three main forms of LR(1) parsing, each differing in how the lookahead sets of items are defined. LR(1) LALR(1) SLR(1)

40 SLR(1) PARSING SLR(1) stands for simple LR(1)

41 SLR(1) Automata An SLR(1) automaton is an LR(0) automaton with every item n α β replaced by n α β { σ } where {σ } is the follow set of n.

42 Exercise 7 Compute the follow set of each non-terminal in the following grammar. s e e e + v v v x y

43 Answer follow( s ) = { $ } follow( e ) = { +, $ } follow( v ) = { +, $ }

44 Example of an SLR(1) automaton start v S 4 e v {+, $} S 0 s e {$} e e + v {+, $} e v {+, $} v x {+, $} v y {+, $} e S 1 s e {$} e e + v {+, $} $ accept + x y S 5 v x {+, $} S 6 v y {+, $} x S y 2 e e + v {+, $} v x {+, $} v y {+, $} v S 3 e e + v {+, $}

45 Power of SLR(1) SLR(1) Grammars LR(0) Grammars An SLR(1) parser observes fewer reducible items than an LR(0) parser. Therefore, fewer reduce-reduce and shift-reduce conflicts.

46 LR(1) PARSING More powerful than SLR(1)

47 Idea of LR(1) In SLR(1), the lookahead set of an item is computed in the context of any possible derivation. Idea: have different lookahead sets for item n α β for each occurrence site of n in the grammar.

48 Example Consider the following grammar. s c c c c c d The follow sets are: follow( s ) = { $ } follow( c ) = { c, d, $ }

49 Example (continued) In SLR(1) parsing, we have items such as c d { c, d, $ } But depending on whether we are parsing the first or second c in an s, we want items such as c d { c, d } or c d { $ }

50 Construction an LR(1) automaton To construct an LR(1) automaton, we make minor modifications to the CLOSURE and GOTO functions.

51 The CLOSURE function every item in L is in CLOSURE(L); if item n α m β {σ} is in CLOSURE(L) and m γ is a production then m γ {σ'} is in CLOSURE(L) where σ' contains every terminal in first(β) and also, if ε first(β), every terminal in σ.

52 Exercise 8 Consider the following grammar Q. s' s s c c c c c d Compute the CLOSURE of the set of items { s' s {$} }.

53 The GOTO function If L is a set of items and X is a grammar symbol and n α X β {σ} L then GOTO(L, X) is the CLOSURE of the set of all items n α X β {σ}

54 Exercise 9 Consider the following LR(1) state. S 0 s' s {$} s c c {$} c c c {c, d} c d {c, d} Give the states that can be reached from S 0 on each of the following grammar symbols: s, c, c and d.

55 Exercise 10 Construct an LR(1) automaton for the following grammar. s' s s c c c c c d (A new start symbol s' has already been introduced.)

56 LALR(1) PARSING Merging LR(1) States

57 Problem LR(1) automata can contain very many states, requiring large amounts of memory to store.

58 LALR(1) Parsing Idea: first construct an LR(1) automaton and then merge states where possible. Two LR(1) states are merged if, by ignoring the lookahead sets, they contain the same items. The merged state contains the union of all the items in the original two states.

59 Example Ignoring lookahead sets, states S 4 and S 7 have the same items. S 4 c d {c, d} S 7 c d {$} After merging: S 47 c d {c, d} c d {$} Or more simply: S 47 c d {$, c, d}

60 Exercise 11 Construct an LALR(1) automaton for the following grammar. s' s s c c c c c d (A new start symbol s' has already been introduced.)

61 Facts about LALR(1) LALR(1) and LR(0) automata contain the same number of states. For the C language, the LALR(1) automaton contains hundreds of states whereas the LR(1) automaton contains thousands.

62 More facts about LALR(1) If the LR(1) automaton for a grammar contains no shiftreduce conflicts, then neither does the LALR(1) automaton. But, an LALR(1) automaton may contain reduce-reduce conflicts not present in an LR(1) automaton.

63 Expressive power of LR parsing techniques LR(1) LALR(1) SLR(1) LR(0)

64 Concluding remarks LR parsers can handle a larger class of grammars than LL parsers. But LR parsers are often too complex to construct by hand. So the main use of LR parsing is in automatic parser generator tools such as Bison.

65 Acknowledgements

MIT Parse Table Construction. Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology

MIT Parse Table Construction. Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology MIT 6.035 Parse Table Construction Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology Parse Tables (Review) ACTION Goto State ( ) $ X s0 shift to s2 error error goto s1