Introduction to parsers

Size: px
Start display at page:

Download "Introduction to parsers"

Transcription

1 Syntax Analysis Introduction to parsers Context-free grammars Push-down automata Top-down parsing LL grammars and parsers Bottom-up parsing LR grammars and parsers Bison/Yacc - parser generators Error Handling: Detection & Recovery 1

2 Introduction to parsers source code Lexical Analyzer token next token Parser syntax tree Semantic Analyzer Symbol Table 2

3 Context Free Grammar CFG & Terminology Rewrite vs. Reduce Derivation Language and CFL Equivalence & CNF Parsing vs. Derivation lm/rm derivation & parse tree Ambiguity & resolution Expressive power Derivation is the reverse of Parsing. If we know how sentences are derived, we may find a parsing method in the reversed direction. 3

4 CFG: An Example Terminals: id, +, -, *, /, (, ) Nonterminals: expr, op Productions: expr expr op expr expr ( expr ) expr - expr expr id op + - * / The start symbol: expr 4

5 Notational Conventions in CFG a,b,c, [+-0-9], id: symbols in A,B,C,,S, expr,stmt: symbols in N U,V,W,,X,Y,Z: grammar symbols in( +N) denotes strings in( +N) * u,v, w, denotes strings in * A is an abbreviation of Alternatives: at RHS A A A 5

6 Context-Free Grammars A set of terminals: basic symbols from which sentences are formed A set of nonterminals: syntactic variables denoting sets of strings A set of productions: rules specifying how the terminals and nonterminals can be combined to form sentences The start symbol: a distinguished nonterminal denoting the language 7

7 CFG: Components Specification for Structures & Constituency CFG: formal specification of structure (parse trees) G = {, N, P, S} : terminal symbols N: non-terminal symbols P: production rules S: start symbol 8

8 CFG: Components : terminal symbols the input symbols of the language programming language: tokens (reserved words, variables, operators, ) natural languages: words or parts of speech pre-terminal: parts of speech (when words are regarded as terminals) N: non-terminal symbols groups of terminals and/or other non-terminals S: start symbol: the largest constituent of a parse tree 9

9 CFG: Components P: production (re-writing) rules form: A β (A: non-terminal, β: string of terminals and non-terminals) meaning: A re-writes to ( consists of, derived into )β, or β reduced to A start with S-productions (S β) 10

10 Derivations A derivation step is an application of a production as a rewriting rule E -E A sequence of derivation steps E -E -(E) -(id ) is called a derivation of - ( id ) from E The symbol * denotes derives in zero or more steps ; the symbol + denotes derives in one or more steps 11

11 CFG: Accepted Languages Context-Free Language Language accepted by a CFG L(G) = { S + (strings of terminals that can be derived from start symbol)} Proof of acceptance: by induction On the number of derivation steps On the length of input string 12

12 Context-Free Languages A context-free language L(G) is the language defined by a context-free grammar G A string of terminals is in L(G) if and only if S +, is called a sentence of G If S *, where may contain nonterminals, then we call a sentential form of G E -E -(E) -(id ) G 1 is equivalent to G 2 if L(G 1 ) = L(G 2 ) 13

13 CFG: Equivalence Chomsky Normal Form (CNF) (Chomsky, 1963): ε-free, and Every production rule is in either of the following form: A A1 A2 A a (A1, A2: non-terminal, a: terminal) i.e., two non-terminals or one terminal at the RHS Properties: Generate binary parse tree Good simplification for some algorithms e.g., grammar training with the inside-outside algorithm (Baker 1979) Good tool for theoretical proving e.g., time complexity 14

14 CFG: Equivalence Every CFG can be converted into a weakly equivalent CNF equivalence: L(G1) = L(G2) strong equivalent: assign the same phrase structure to each sentence (except for renaming non-terminals) weak equivalent: do not assign the same phrase structure to each sentence e.g., A B C D == {A B X, X CD} 15

15 CFG: An Example Terminals: id, +, -, *, /, (, ) Nonterminals: expr, op Productions: expr expr op expr [R1] expr ( expr ) [R2] expr - expr [R3] expr id [R4] op + - * / The start symbol: expr 16

16 Left- & Right-most Derivations Each derivation step needs to choose a nonterminal to rewrite an alternative to apply A leftmost derivation always chooses the leftmost nonterminal to rewrite E lm - E lm - ( E ) lm - ( E + E ) lm - ( id + E ) lm - ( id + id ) A rightmost (canonical) derivation always chooses the rightmost nonterminal to rewrite E rm - E rm - ( E ) rm - ( E + E ) rm - (E + id ) rm - ( id + id ) 17

17 Left- & Right-most Derivations Representation of leftmost/rightmost derivations: Use the sequence of productions (or production numbers) to represent a derivation sequence. Example: E rm - E rm - ( E ) rm - ( E + E ) rm - (E + id ) rm - ( id + id ) => [3], [2], [1], [4], [4] (~ R3, R2, R1, R4, R4) Advantage: A compact representation for parse tree (data compression) Each parse tree has a unique leftmost/rightmost derivation 18

18 Parse Trees A parse tree is a graphical representation for a derivation that filters out the order of choosing nonterminals for rewriting NP NP PP NP girl in the park 19

19 Context Free Grammar (CFG): Specification for Structures & Constituency Parse Tree: graphical representation of structure Root node (S): a sentencial level structure Internal nodes: constituents of the sentence Arcs: relationship between parent nodes and their children (constituents) Terminal nodes: surface forms of the input symbols (e.g., words) Bracketed notation: Alternative representation e.g., [I saw [the [girl [in [the park]]]]] 20

20 Parse Tree: I saw the girl in the park S NP VP 1st parse NP PP NP pron v det n p det n I saw the girl in the park 21

21 Parse Tree: I saw the girl in the park S NP VP 2nd parse NP NP PP NP pron v det n p det n I saw the girl in the park 22

22 LM & RM: An Example E - E E lm - E lm - ( E ) lm - ( E + E ) lm - ( id + E ) lm - ( id + id ) ( E ) E + E id id E rm - E rm - ( E ) rm - ( E + E ) rm - ( E + id ) rm - ( id + id ) 23

23 Parse Trees & Derivations Many derivations may correspond to the same parse tree, but every parse tree has associated with it a unique leftmost and a unique rightmost derivation 24

24 Ambiguous Grammar A grammar is ambiguous if it produces more than one parse tree for some sentence more than one leftmost/rightmost derivation E E + E id + E id + E * E id + id * E id + id * id E E * E E + E * E id + E * E id + id * E id + id * id 25

25 Ambiguous Grammar E E E + E E * E id E * E E + E id id id id id 26

26 Resolving Ambiguity Use disambiguating rules to throw away undesirable parse trees Rewrite grammars by incorporating disambiguating rules into grammars 27

27 An Example The dangling-else grammar stmt if expr then stmt if expr then stmt else stmt other Two parse trees for if E 1 then if E 2 then S 1 else S 2 28

28 An Example S if E then S else S if E then S S if E then S Preferred parse: closest then if E then S else S 29

29 Disambiguating Rules Rule: match each else with the closest previous unmatched then Remove undesired state transitions in the pushdown automaton shift/reduce conflict on else 1 st parse: reduce 2 nd parse: shift 30

30 Grammar Rewriting stmt m_stmt unm_stmt ; with only paired then-else m_stmt if expr then m_stmt else m_stmt other unm_stmt if expr then stmt if expr then m_stmt else unm_stmt 31

31 RE vs. CFG Every language described by a RE can also be described by a CFG Example: (a b)*abb A0 a A0 b A0 a A1 A1 ba2 A2 ba3 A3 (1) Right branching (2) Starts with a terminal symbol 32

32 a( b) A0 A0 RE vs. CFG Regular Grammar: Right branching Starts with a a( b) A0 terminal symbol a A1 b A2 (a b)* abb b A3 33

33 RE vs. CFG A0 a A0 b A0 a A1 A1 ba2 A2 ba3 A3 A0 a RE: (a b) * abb A2 start a b b b A1 A3 34

34 RE vs. CFG a DFA for (a b) * abb A2 A0 start A0 b A0 a A1 A1 aa1 b A2 A2 aa1 b A3 A3 aa1 b A0 b a b b 0 a a b a A1 A3 35

35 CFG: Expressive Power (cont.) Writing a CFG for a FSA (RE) define a non-terminal Ni for a state with state number i start symbol S = N0 (assuming that state 0 is the initial state) for each transition δ(i,a)=j (from state i to stet j on input alphabet a), add a new production Ni a Nj to P (a== ε Ni Nj) for each final state i, add a new production Ni εto P 36

36 CFG: Expressive Power CFG vs. Regular Expression (R.E.) Every R.E. can be recognized by a FSA Every FSA can be represented by a CFG with production rules of the form: A a B ε Therefore, L(RE) L(CFG) 38

37 CFG: Expressive Power (cont.) Chomsky Hierarchy: R.E.: Regular set (recognized by FSAs) CFG: Context-free (Pushdown automata) CSG: Context-sensitive (Linear bounded automata) Unrestricted: Recursively enumerable (Tuning Machine) 39

38 Push-Down Automata Input Stack Finite Automata Output 40

39 RE vs. CFG Why use REs for lexical syntax? do not need a notation as powerful as CFGs are more concise and easier to understand than CFGs More efficient lexical analyzers can be constructed from REs than from CFGs Provide a way for modularizing the front end into two manageable-sized components 41

40 CFG vs. Finite-State Machine Inappropriateness of FSA Constituents: only terminals Recursion: do not allow A => B => A RTN (Recursive Transition Network) FSA with augmentation of recursion arc: terminal or non-terminal if arc is non-terminal: call to a sub-transition network & return upon traversal 42

41 Nonregular Constructs REs can denote only a fixed number of repetitions or an unspecified number of repetitions of one given construct E.g. a*b* A nonregular construct: L = {a n b n n 1} 43

42 Non-Context-Free Constructs CFGs can denote only a fixed number of repetitions or an unspecified number of repetitions of one or two (paired) given constructs E.g. a n b n Some non-context-free constructs: L 1 = {wcw w is in (a b)*} declaration/use of identifiers L 2 = {a n b m c n d m n 1 and m 1} #formal arguments/#actual arguments L 3 = {a n b n c n n 0} e.g., b: Backspace, c: under score 44

43 Context-Free Constructs FA (RE) cannot keep counts CFGs can keep count of two items but not three Similar context-free constructs: L 1 = {wcw R w is in (a b)*, R: reverse order} L 2 = {a n b m c m d n n 1 and m 1} L 2 = {a n b n c m d m n 1 and m 1} L 3 = {a n b n n 1} 45

44 CFG Parsers 46

45 Types of CFG Parsers Universal: can parse any CFG grammar CYK, Earley CYK: Exhaustively matching sub-ranges of input tokens against grammar rules, from smaller ranges to larger ranges Earley: Exhaustively enumerating possible expectations from left-to-right, according to current input token and grammar Non-universal: e.g., recursive descent parser Universal (to all grammars) is NOT always efficient 47

46 Types of CFG Parsers Practical Parsers: [ what is a good parser? ] Simple: simple program structure Left-to-right (or right-to-left) scan middle-out or island driven is often not preferred Top-down or Bottom up matching Efficient: efficient for good/bad inputs Parse normal syntax quickly Detect errors immediately on next token Deterministic: No alternative choices during parsing given next token Small lookahead buffer (also contribute to efficiency) 48

47 Types of CFG Parsers Top Down: Matching from start symbol down to terminal tokens Bottom Up: Matching input tokens with reducible rules from terminal up to start symbol 49

48 Efficient CFG Parsers Top Down: LL Parsers Matching from start symbol down to terminal tokens, left-to-right, according to a leftmost derivation sequence Bottom Up: LR Parsers Matching input tokens with reducible rules, left-to-right, from terminal up to start symbol, in a reverse order of rightmost derivation sequence 50

49 Efficient CFG Parsers Efficient & Deterministic Parsing only possible for some subclasses of grammars with special parsing algorithms Top Down: Parsing LL Grammars with LL Parsers Bottom Up: Parsing LR Grammars with LR Parsers LR grammar is a larger class of grammars than LL 51

50 Parsing Table Construction for Parsing Table: Efficient Parsers A pre-computed table (according to the grammar), indicating the appropriate action(s) to take in any predefined state when some input token(s) is/are under examination Lookahead symbol(s): the input symbol(s) under examination for determining next action(s) id + * State-0 State-1 State-2 action-1 action-2 Good parsers do not change their codes when the grammar is revised. Table driven. action-3 action-4 num action-5 52

51 Parsing Table Construction for Efficient Parsers Parsing Table Construction: Decide a pre-defined number of lookaheads to use for predicting next state Define and enumerate all the unique states for the parsing method Decide the actions to take in all states with all possible lookahead(s) 53

52 Parsing Table Construction for Efficient Parsers X-Parser: you can invent any parser and call it the X-Parser But its parsing algorithm may not handle all grammars deterministically, thus efficiently. X-Grammar: Any grammar whose parsing table for the X- parsing method/x-parser has no conflicting actions in all states Non-X Grammar: has more than one action to take under any state 54

53 Parsing Table Construction for Efficient Parsers k: The number of lookahead symbols used by a parser to determine the next action A larger number of lookahead symbols tends to make it less possible to have conflicting actions But may result in a much larger table that grows exponentially with the number of lookaheads Does not guarantee unambiguous for some grammars (inherently ambiguous) X(k) Parser: X Parser that uses k lookahead symbols to determine the next action X(k) Grammar: any grammar deterministically parsable with X(k) Parser 55

54 Types of Grammars Capable of Efficient Parsing LL(k) Grammars Grammars that can be deterministically parsed using an LL(k) parsing algorithm e.g., LL(1) grammar LR(k) Grammars Grammars that can be deterministically parsed using an LR(k) parsing algorithm e.g., SLR(1) grammar, LR(1) grammar, LALR(1) grammar 56

55 Top-Down CFG Parsers Recursive Descent Parser vs. Non-Recursive LL(1) Parser 57

56 Top-Down Parsing Construct a parse tree from the root to the leaves using leftmost derivation S c A B A a b a B d input: cad S S S S c A B c A B c A B c A B a b a a d 58

57 Predictive Parsing A top-down parsing without backtracking there is only one alternative production to choose at each derivation step stmt if expr then stmt else stmt while expr do stmt begin stmt_list end 59

58 LL(k) Parsing The first L stands for scanning the input from left to right The second L stands for producing a leftmost derivation The k stands for the number of input symbols for lookahead used to choose alternative productions at each derivation step 60

59 LL(1) Parsing Use one input symbol of lookahead Same as Recursive-descent parsing But, Nonrecursive predictive parsing 61

60 Recursive Descent Parsing (more) The parser consists of a set of (possibly recursive) procedures Each procedure is associated with a nonterminal of the grammar The sequence of procedures called in processing the input implicitly defines a parse tree for the input 62

61 An Example type simple id array [ simple ] of type simple integer char num dotdot num 63

62 An Example array [ num dotdot num ] of integer type array [ simple ] of type num dotdot num simple integer 64

63 An Example procedure type; begin if lookahead is in { integer, char, num } then simple else if lookahead = id then match(id) else if lookahead = array then begin match(array); match('['); simple; match(']'); match(of); type end else error end; 65

64 An Example procedure match(t : token); begin if lookahead = t then lookahead := nexttoken else error end; 66

65 An Example procedure simple; begin if lookahead = integer then match(integer) else if lookahead = char then match(char) else if lookahead = num then begin match(num); match(dotdot); match(num) end else error end; 67

66 LL(k) Constraint: Left Recursion A grammar is left recursive if it has a nonterminal A such that A + A A A A R R R A A A A * A R R R R 68

67 Direct/Immediate Left Recursion A A 1 A 2... A m n is equivalent to A A i j (i=1,m ; j=1,n) A 1 A' 2 A'... n A' A' 1 A' 2 A'... m A' ( n ) ( m )* 69

68 An Example E E + T T T T * F F F ( E ) id E T E' E' + T E' T F T' T' * F T' F ( E ) id 70

69 Indirect Left Recursion G0: S A a b A A c S d Problem: Indirect Left-Recursion: S A a S d a Scan rules top-down Do not start with symbols defined earlier (=> substitute them if any) Resolve direct recursion Solution-Step1: Indirect to Direct Left-Recursion: A A c A a d b d Solution-Step2: Direct Left-Recursion to Right-Recursion: S A a b A b d A' A' 71 A' ca' a d A'

70 Indirect Left Recursion Input. Grammar G with no cycles or -production. Output. An equivalent grammar with no left recursion. 1. Arrange the nonterminals in some order A 1, A 2,..., A n 2. for i := 1 to n do begin // Step1: Substitute 1 st -symbols of Ai for j := 1 to i - 1 do begin // which are previous Aj s replace each production of the form A i A j ( j < i ) by the production A i k where A j k are all the current A j -productions; end eliminate direct left recursion among A i -productions // Step2 end 72

71 Left Factoring Two alternatives of a nonterminal A have a nontrivial common prefix if, and A 1 2 A A' A'

72 An Example S i E t S i E t S e S a E b S i E t S S' a S' es E b 74

73 Top-Down Parsing: as Stack Matching Construct a parse tree from the root to the leaves using leftmost derivation S c A B A a b a B d input: cad S S S S c A B c A B c A B c A B a b a a d 76

74 Nonrecursive Predictive Parsing General State a b c x y z Input Stack X Non- Recursive: Stack + Driver Program (instead of Recursive procedures) Parsing program (parser/driver) Parsing table Output M[X,a]= {X -> Y 1 Y 2 Y k } Predictive: precomputed parsing actions 77

75 Nonrecursive Predictive Parsing Expand Non-terminal a b c x y z Input Stack Y 1 Non- Recursive: Stack + Driver Program (instead of Recursive procedures) Y 2 Y k Parsing program (parser/driver) Parsing table Output M[X,a]= {X -> Y 1 Y 2 Y k } Predictive: precomputed parsing actions 78

76 Nonrecursive Predictive Parsing Match Terminal a b c x y z Input Stack Y 1 =a Non- Recursive: Stack + Driver Program (instead of Recursive procedures) Y 2 Y k Parsing program (parser/driver) Parsing table Output M[X,a]= {X -> Y 1 Y 2 Y k } Predictive: precomputed parsing actions 79

77 Nonrecursive Predictive Parsing - Error Recovery a b c x y z Input Stack Y 1 =a Non- Recursive: Stack + Driver Program (instead of Recursive procedures) Y 2 Y k =c Parsing program (parser/driver) Parsing table Output M[X,a]= {X -> Y 1 Y 2 Y k } Predictive: precomputed parsing actions 80

78 Nonrecursive Predictive Parsing - Error Recovery a b c x y z Input Stack Y 1 =a Non- Recursive: Stack + Driver Program (instead of Recursive procedures) Y 2 Y k =c Parsing program (parser/driver) Parsing table Output M[X,a]= {X -> Y 1 Y 2 Y k } Predictive: precomputed parsing actions 81

79 Stack Operations Match when the top stack symbol is a terminal and it matches the input symbol, pop the top stack symbol and advance the input pointer Expand when the top stack symbol is a nonterminal, replace this symbol by the right hand side of one of its productions Leftmost RHS symbol at Top-of-Stack 83

80 An Example type simple id array [ simple ] of type simple integer char num dotdot num 84

81 An Example Action Stack Input E type array [ num dotdot num ] of integer M type of ] simple [ array array [ num dotdot num ] of integer M type of ] simple [ [ num dotdot num ] of integer E type of ] simple num dotdot num ] of integer M type of ] num dotdot num num dotdot num ] of integer M type of ] num dotdot dotdot num ] of integer M type of ] num num ] of integer M type of ] ] of integer M type of of integer E type integer E simple integer M integer integer 85

82 Parsing program push $S onto the stack, where S is the start symbol set ip to point to the first symbol of w$; // try to match S$ with w$ repeat let X be the top stack symbol and a the symbol pointed to by ip; if X is a terminal or $ then if X = a then pop X from the stack and advance ip else error // or error_recovery() else // X is a nonterminal if M[X, a] = X Y 1 Y 2... Y k then pop X from and push Y k... Y 2 Y 1 onto the stack else error // or error_recovery() until X = $ 86

83 Parser Driven by a Parsing Table: Non-recursive Descent a b c d X X Y1 Y2 Yk X Z1 Z2 Zm Y1 Y1 1 Y1 2 Z1 Z1 1 Z1 2 X() { // WITHOUT ε-production: X ε if (LA= a ) then Y1(); Y2(); Yk(); else if (LA= b ) Z1(); Z2(); ; Zm(); a in FirstSet( Y1 Y2 Yk ) else ERROR(); // no X ε b in FirstSet( Z1 Z2 Zm ) // else RETURN; if X exists } // Recursive decent procedure for matching X 87

84 Parser Driven by a Parsing Table: Non-recursive Descent a b c d X X Y1 Y2 Yk X Z1 Z2 Zm X Y1 Y1 1 Y1 2 Z1 Z1 1 Z1 2 a in FirstSet( Y1 Y2 Yk ) b in FirstSet( Z1 Z2 Zm ) d in FollowSet(X) (S => * X d ) X() { // WITH ε-production: X ε if (LA= a ) then else if (LA= b ) Y1(); Y2(); Yk(); Z1(); Z2(); ; Zm(); // else ERROR(); // no X ε else if (LA=??) RETURN; // if X exists } // Recursive decent procedure for matching X 88

85 First Sets The first set of a string is the set of terminals that begin the strings derived from. If *, then is also in the first set of. Used simply to flag whether can be null for computing First Set Not for matching any real input when parsing FIRST( ) = {a * a }+{, if * } FIRST( ) includes { }: means that * 89

86 Compute First Sets If X is terminal, then FIRST(X) is {X} If X is nonterminal and X is a production, then add to FIRST(X) If X is nonterminal and X Y 1 Y 2... Y k is a production, then add a to FIRST(X) if for some i, a is in FIRST(Y i ) and is in all of FIRST(Y 1 ),..., FIRST(Y i-1 ). If is in FIRST(Y j ) for all j, then add to FIRST(X) 90

87 Follow Sets What to do with matching null: A? TD Recursive Descent Parsing: assumes success LL: more predictive => Follow Set of A The follow set of a nonterminal A is the set of terminals that can appear immediately to the right of A in some sentential form, namely, S * A a a is in the follow set of A. 91

88 Compute Follow Sets Initialization: Place $ in FOLLOW(S), where S is the start symbol and $ is the input right end marker. If there is a production A B, then everything in FIRST( ) except for is placed in FOLLOW(B) is not considered a visible input to follow any symbol If there is a production A B or A B where FIRST( ) contains (i.e., * ), then everything in FOLLOW(A) is in FOLLOW(B) S * A a implies S * B a YES: every symbol that can follow A will also follow B NO!: every symbol that can follow B will also follow A 92

89 An Example E T E' E' +TE' T F T' T' *FT' F ( E ) id FIRST(E) = FIRST(T) = FIRST(F) = { (, id } FIRST(E') = { +, } FIRST(T') = { *, } FOLLOW(E) = FOLLOW(E') = { ), $ } FOLLOW(T) = FOLLOW(T') = { +, ), $ } FOLLOW(F) = { +, *, ), $ } 93

90 Constructing Parsing Table Input. Grammar G. Output. Parsing Table M. Method. 1. For each production A of the grammar, do steps 2 and For each terminal a in FIRST( ), add A to M[A, a]. 3. If is in FIRST( ) [A * ], add A to M[A, b] for each terminal b [including $ ] in FOLLOW(A). - If is in FIRST( ) and $ is in FOLLOW(A), add A to M[A, $]. 4. Make each undefined entry of M be error. 94

91 LL(1) Parsing Table Construction A B C a in First( ) A b in Follow(A) c not in First( ) or Follow(A) A ( * ) error When to apply A? including A A() { // WITH/WITHOUT ε-productions: A ( * ) if (LA= a in First(Y1 Y2 Yk)) then Y1(); Y2(); Yk(); else if (LA= b in Follow(A) & εin First(Z1 Z2... )) Z1(); Z2(); ; Zm(); // Nullable else ERROR(); } // Recursive version of LL(1) parser 95

92 An Example id + * ( ) $ E E TE' E TE' E' E' +TE' E' E' T T FT' T FT' T' T' T' *FT' T' T' F F id F (E) 96

93 An Example Stack Input Output $E id + id * id$ $E'T id + id * id$ E TE' $E'T'F id + id * id$ T FT' $E'T'id id + id * id$ F id $E'T' + id * id$ $E' + id * id$ T' $E'T+ + id * id$ E' +TE' $E'T id * id$ $E'T'F id * id$ T FT' $E'T'id id * id$ F id $E'T' * id$ $E'T'F* * id$ T' *FT' $E'T'F id$ $E'T'id id$ F id $E'T' $ $E' $ T' $ $ E' 97

94 LL(1) Grammars A grammar is an LL(1) grammar if its predictive parsing table has no multiplydefined entries 98

95 A Counter Example S ietss' a S' es E b e FOLLOW(S ) a b e i t $ S S a S ietss' S' S' S' S' es E E b Disambiguation: matching closest then e FIRST(e S) 99

96 LL(1) Grammars or Not?? A grammar G is LL(1) iff whenever A are two distinct productions of G, the following conditions hold: For no terminal a do both and derive strings beginning with a. or M[A, first( )&first( )] entries will have conflicting actions At most one of and can derive the empty string or M[A, follow(a)] entries have conflicting actions If *, then does not derive any string beginning with a terminal in FOLLOW(A). or M[A, first( )&follow(a)] entries have conflicting actions 100

97 Non-LL(1) Grammar: Ambiguous According to LL(1) Parsing Table Construction a in First( ) & First( ) b in Follow(A) a in First( ) & Follow(A) A A A ( * ) A (/ * ) (but * a ) A A ( * ) A ( * ) B C When will A & A appear in the same table cell?? 101

98 LL(1) Grammars or Not?? If G is left-recursive or ambiguous, then M will have at least one multiply-defined entry => non-ll(1) E.g., X Xa b => FIRST(X) = {b} (and, of course, FIRST(b) = {b}) => M[X,b] includes both {X Xa} and {X b} Ambiguous G and G with left-recursive productions can not be LL(1). No LL(1) grammar can be ambiguous 102

99 Error Recovery for LL Parsers 103

100 Syntactic Errors Empty entries in a parsing table: Syntactic error is encountered when the lookahead symbol corresponding to this entry is in input buffer Error Recovery information can be encoded in such entries to take appropriate actions upon error Error Detection: (1) Stacktop = x && x!= input (a) (2) Stacktop = A && M[A, a] = empty (error) 104

101 Error Recovery Strategies Panic mode: skip tokens until a token in a set of synchronizing tokens appears INS(eration) type of errors sync at delimiters, keywords,, that have clear functions Phrase Level Recovery local INS(eration), DEL(eation), SUB(stitution) types of errors Error Production define error patterns in grammar Global Correction [Grammar Correction] minimum distance correction 105

102 Error Recovery Panic Mode Panic mode: skip tokens until a token in a set of synchronizing tokens appears Commonly used Synchronizing tokens: SUB(A,ip): use FOLLOW(A) as sync set for A (pop A) use the FIRST set of a higher construct as sync set for a lower construct INS(ip): use FIRST(A) as sync set for A *ip= : use the production deriving as the default DEL(ip): If a terminal on stack cannot be matched, pop the terminal 106

103 Error Recovery Panic Mode Action Stack Input SUB(A,ip) INS(ip) DEL(ip) X A *ip Follow(A) A A *ip First(A) A x *ip x X X A A x *ip Follow(A) *ip First(A) x *ip 107

104 Error Recovery Actions Using Follow & First Sets to Sync Expanding non-terminal A: M[A,a] = error (blank): Skip a in input = delete all such a (until sync with sync symbol, b) /* panic */ M[A,b] = sync (at FOLLOW(A)) Pop A from stack = b is a sync symbol following A M[A,b] = A (sync at FIRST(A)) Expand A (same as normal parsing action) Matching terminal x : (*sp= x )!= a Pop(x) from stack = missing input token x 108

105 FOLLOW(X) An is Example used to Expand productions or Sync (on errors) FOLLOW(E)=FOLLOW(E )={),$} id + * ( ) $ E E TE' E TE' sync sync E' E' +TE' E' E' T T FT' sync T FT' sync sync T' T' T' *FT' T' T' F F id sync sync F (E) sync sync FOLLOW(F)={+,*,),$} FIRST(X) is used to Expand non- productions or Sync (on errors) 109

106 An Example Stack Input Output $E ) id * + id$ error, skip ) $E id * + id$ id is in FIRST(E) $E'T id * + id$ E TE' $E'T'F id * + id$ T FT' $E'T'id id * + id$ F id $E'T' * + id$ $E'T'F* * + id$ T' *FT' $E'T'F + id$ error, M[F,+]=synch / FOLLOW(F) $E'T' + id$ F popped $E' + id$ T' $E'T+ + id$ E' +TE' $E'T id$ $E'T'F id$ T FT' $E'T'id id$ F id $E'T' $ $E' $ T' $ $ E' 110

107 Parse Tree - Error Recovered ) id * + id => id * F + id E ) T E F T + T E id * F T F T ε ε id ε 111

Parsing. Roadmap. > Context-free grammars > Derivations and precedence > Top-down parsing > Left-recursion > Look-ahead > Table-driven parsing

Parsing. Roadmap. > Context-free grammars > Derivations and precedence > Top-down parsing > Left-recursion > Look-ahead > Table-driven parsing Roadmap > Context-free grammars > Derivations and precedence > Top-down parsing > Left-recursion > Look-ahead > Table-driven parsing The role of the parser > performs context-free syntax analysis > guides

More information

Syntax Analysis. Martin Sulzmann. Martin Sulzmann Syntax Analysis 1 / 38

Syntax Analysis. Martin Sulzmann. Martin Sulzmann Syntax Analysis 1 / 38 Syntax Analysis Martin Sulzmann Martin Sulzmann Syntax Analysis 1 / 38 Syntax Analysis Objective Recognize individual tokens as sentences of a language (beyond regular languages). Example 1 (OK) Program

More information

Concepts Introduced in Chapter 4

Concepts Introduced in Chapter 4 Concepts Introduced in Chapter 4 Grammars Context-Free Grammars Derivations and Parse Trees Ambiguity, Precedence, and Associativity Top Down Parsing Recursive Descent, LL Bottom Up Parsing SLR, LR, LALR

More information

8 Parsing. Parsing. Top Down Parsing Methods. Parsing complexity. Top down vs. bottom up parsing. Top down vs. bottom up parsing

8 Parsing. Parsing. Top Down Parsing Methods. Parsing complexity. Top down vs. bottom up parsing. Top down vs. bottom up parsing 8 Parsing Parsing A grammar describes syntactically legal strings in a language A recogniser simply accepts or rejects strings A generator produces strings A parser constructs a parse tree for a string

More information

3. Parsing. Oscar Nierstrasz

3. Parsing. Oscar Nierstrasz 3. Parsing Oscar Nierstrasz Thanks to Jens Palsberg and Tony Hosking for their kind permission to reuse and adapt the CS132 and CS502 lecture notes. http://www.cs.ucla.edu/~palsberg/ http://www.cs.purdue.edu/homes/hosking/

More information

Monday, September 13, Parsers

Monday, September 13, Parsers Parsers Agenda Terminology LL(1) Parsers Overview of LR Parsing Terminology Grammar G = (Vt, Vn, S, P) Vt is the set of terminals Vn is the set of non-terminals S is the start symbol P is the set of productions

More information

CA Compiler Construction

CA Compiler Construction CA4003 - Compiler Construction David Sinclair A top-down parser starts with the root of the parse tree, labelled with the goal symbol of the grammar, and repeats the following steps until the fringe of

More information

3. Syntax Analysis. Andrea Polini. Formal Languages and Compilers Master in Computer Science University of Camerino

3. Syntax Analysis. Andrea Polini. Formal Languages and Compilers Master in Computer Science University of Camerino 3. Syntax Analysis Andrea Polini Formal Languages and Compilers Master in Computer Science University of Camerino (Formal Languages and Compilers) 3. Syntax Analysis CS@UNICAM 1 / 54 Syntax Analysis: the

More information

Top down vs. bottom up parsing

Top down vs. bottom up parsing Parsing A grammar describes the strings that are syntactically legal A recogniser simply accepts or rejects strings A generator produces sentences in the language described by the grammar A parser constructs

More information

Compilerconstructie. najaar Rudy van Vliet kamer 140 Snellius, tel rvvliet(at)liacs(dot)nl. college 3, vrijdag 22 september 2017

Compilerconstructie. najaar Rudy van Vliet kamer 140 Snellius, tel rvvliet(at)liacs(dot)nl. college 3, vrijdag 22 september 2017 Compilerconstructie najaar 2017 http://www.liacs.leidenuniv.nl/~vlietrvan1/coco/ Rudy van Vliet kamer 140 Snellius, tel. 071-527 2876 rvvliet(at)liacs(dot)nl college 3, vrijdag 22 september 2017 + werkcollege

More information

Table-Driven Parsing

Table-Driven Parsing Table-Driven Parsing It is possible to build a non-recursive predictive parser by maintaining a stack explicitly, rather than implicitly via recursive calls [1] The non-recursive parser looks up the production

More information

Sometimes an ambiguous grammar can be rewritten to eliminate the ambiguity.

Sometimes an ambiguous grammar can be rewritten to eliminate the ambiguity. Eliminating Ambiguity Sometimes an ambiguous grammar can be rewritten to eliminate the ambiguity. Example: consider the following grammar stat if expr then stat if expr then stat else stat other One can

More information

PART 3 - SYNTAX ANALYSIS. F. Wotawa TU Graz) Compiler Construction Summer term / 309

PART 3 - SYNTAX ANALYSIS. F. Wotawa TU Graz) Compiler Construction Summer term / 309 PART 3 - SYNTAX ANALYSIS F. Wotawa (IST @ TU Graz) Compiler Construction Summer term 2016 64 / 309 Goals Definition of the syntax of a programming language using context free grammars Methods for parsing

More information

Syntactic Analysis. Top-Down Parsing

Syntactic Analysis. Top-Down Parsing Syntactic Analysis Top-Down Parsing Copyright 2017, Pedro C. Diniz, all rights reserved. Students enrolled in Compilers class at University of Southern California (USC) have explicit permission to make

More information

Wednesday, August 31, Parsers

Wednesday, August 31, Parsers Parsers How do we combine tokens? Combine tokens ( words in a language) to form programs ( sentences in a language) Not all combinations of tokens are correct programs (not all sentences are grammatically

More information

Syntax Analysis Part I

Syntax Analysis Part I Syntax Analysis Part I Chapter 4: Context-Free Grammars Slides adapted from : Robert van Engelen, Florida State University Position of a Parser in the Compiler Model Source Program Lexical Analyzer Token,

More information

Parsers. Xiaokang Qiu Purdue University. August 31, 2018 ECE 468

Parsers. Xiaokang Qiu Purdue University. August 31, 2018 ECE 468 Parsers Xiaokang Qiu Purdue University ECE 468 August 31, 2018 What is a parser A parser has two jobs: 1) Determine whether a string (program) is valid (think: grammatically correct) 2) Determine the structure

More information

WWW.STUDENTSFOCUS.COM UNIT -3 SYNTAX ANALYSIS 3.1 ROLE OF THE PARSER Parser obtains a string of tokens from the lexical analyzer and verifies that it can be generated by the language for the source program.

More information

Part 3. Syntax analysis. Syntax analysis 96

Part 3. Syntax analysis. Syntax analysis 96 Part 3 Syntax analysis Syntax analysis 96 Outline 1. Introduction 2. Context-free grammar 3. Top-down parsing 4. Bottom-up parsing 5. Conclusion and some practical considerations Syntax analysis 97 Structure

More information

Parsing Wrapup. Roadmap (Where are we?) Last lecture Shift-reduce parser LR(1) parsing. This lecture LR(1) parsing

Parsing Wrapup. Roadmap (Where are we?) Last lecture Shift-reduce parser LR(1) parsing. This lecture LR(1) parsing Parsing Wrapup Roadmap (Where are we?) Last lecture Shift-reduce parser LR(1) parsing LR(1) items Computing closure Computing goto LR(1) canonical collection This lecture LR(1) parsing Building ACTION

More information

LR Parsing Techniques

LR Parsing Techniques LR Parsing Techniques Introduction Bottom-Up Parsing LR Parsing as Handle Pruning Shift-Reduce Parser LR(k) Parsing Model Parsing Table Construction: SLR, LR, LALR 1 Bottom-UP Parsing A bottom-up parser

More information

CS 314 Principles of Programming Languages

CS 314 Principles of Programming Languages CS 314 Principles of Programming Languages Lecture 5: Syntax Analysis (Parsing) Zheng (Eddy) Zhang Rutgers University January 31, 2018 Class Information Homework 1 is being graded now. The sample solution

More information

COP4020 Programming Languages. Syntax Prof. Robert van Engelen

COP4020 Programming Languages. Syntax Prof. Robert van Engelen COP4020 Programming Languages Syntax Prof. Robert van Engelen Overview n Tokens and regular expressions n Syntax and context-free grammars n Grammar derivations n More about parse trees n Top-down and

More information

SYNTAX ANALYSIS 1. Define parser. Hierarchical analysis is one in which the tokens are grouped hierarchically into nested collections with collective meaning. Also termed as Parsing. 2. Mention the basic

More information

Wednesday, September 9, 15. Parsers

Wednesday, September 9, 15. Parsers Parsers What is a parser A parser has two jobs: 1) Determine whether a string (program) is valid (think: grammatically correct) 2) Determine the structure of a program (think: diagramming a sentence) Agenda

More information

Parsers. What is a parser. Languages. Agenda. Terminology. Languages. A parser has two jobs:

Parsers. What is a parser. Languages. Agenda. Terminology. Languages. A parser has two jobs: What is a parser Parsers A parser has two jobs: 1) Determine whether a string (program) is valid (think: grammatically correct) 2) Determine the structure of a program (think: diagramming a sentence) Agenda

More information

COP4020 Programming Languages. Syntax Prof. Robert van Engelen

COP4020 Programming Languages. Syntax Prof. Robert van Engelen COP4020 Programming Languages Syntax Prof. Robert van Engelen Overview Tokens and regular expressions Syntax and context-free grammars Grammar derivations More about parse trees Top-down and bottom-up

More information

CS2210: Compiler Construction Syntax Analysis Syntax Analysis

CS2210: Compiler Construction Syntax Analysis Syntax Analysis Comparison with Lexical Analysis The second phase of compilation Phase Input Output Lexer string of characters string of tokens Parser string of tokens Parse tree/ast What Parse Tree? CS2210: Compiler

More information

Types of parsing. CMSC 430 Lecture 4, Page 1

Types of parsing. CMSC 430 Lecture 4, Page 1 Types of parsing Top-down parsers start at the root of derivation tree and fill in picks a production and tries to match the input may require backtracking some grammars are backtrack-free (predictive)

More information

CS308 Compiler Principles Syntax Analyzer Li Jiang

CS308 Compiler Principles Syntax Analyzer Li Jiang CS308 Syntax Analyzer Li Jiang Department of Computer Science and Engineering Shanghai Jiao Tong University Syntax Analyzer Syntax Analyzer creates the syntactic structure of the given source program.

More information

Context-free grammars

Context-free grammars Context-free grammars Section 4.2 Formal way of specifying rules about the structure/syntax of a program terminals - tokens non-terminals - represent higher-level structures of a program start symbol,

More information

LL(k) Parsing. Predictive Parsers. LL(k) Parser Structure. Sample Parse Table. LL(1) Parsing Algorithm. Push RHS in Reverse Order 10/17/2012

LL(k) Parsing. Predictive Parsers. LL(k) Parser Structure. Sample Parse Table. LL(1) Parsing Algorithm. Push RHS in Reverse Order 10/17/2012 Predictive Parsers LL(k) Parsing Can we avoid backtracking? es, if for a given input symbol and given nonterminal, we can choose the alternative appropriately. his is possible if the first terminal of

More information

Parsing. Handle, viable prefix, items, closures, goto s LR(k): SLR(1), LR(1), LALR(1)

Parsing. Handle, viable prefix, items, closures, goto s LR(k): SLR(1), LR(1), LALR(1) TD parsing - LL(1) Parsing First and Follow sets Parse table construction BU Parsing Handle, viable prefix, items, closures, goto s LR(k): SLR(1), LR(1), LALR(1) Problems with SLR Aho, Sethi, Ullman, Compilers

More information

Compiler Construction: Parsing

Compiler Construction: Parsing Compiler Construction: Parsing Mandar Mitra Indian Statistical Institute M. Mitra (ISI) Parsing 1 / 33 Context-free grammars. Reference: Section 4.2 Formal way of specifying rules about the structure/syntax

More information

Section A. A grammar that produces more than one parse tree for some sentences is said to be ambiguous.

Section A. A grammar that produces more than one parse tree for some sentences is said to be ambiguous. Section A 1. What do you meant by parser and its types? A parser for grammar G is a program that takes as input a string w and produces as output either a parse tree for w, if w is a sentence of G, or

More information

Compiler Construction 2016/2017 Syntax Analysis

Compiler Construction 2016/2017 Syntax Analysis Compiler Construction 2016/2017 Syntax Analysis Peter Thiemann November 2, 2016 Outline 1 Syntax Analysis Recursive top-down parsing Nonrecursive top-down parsing Bottom-up parsing Syntax Analysis tokens

More information

Bottom-up parsing. Bottom-Up Parsing. Recall. Goal: For a grammar G, withstartsymbols, any string α such that S α is called a sentential form

Bottom-up parsing. Bottom-Up Parsing. Recall. Goal: For a grammar G, withstartsymbols, any string α such that S α is called a sentential form Bottom-up parsing Bottom-up parsing Recall Goal: For a grammar G, withstartsymbols, any string α such that S α is called a sentential form If α V t,thenα is called a sentence in L(G) Otherwise it is just

More information

Note that for recursive descent to work, if A ::= B1 B2 is a grammar rule we need First k (B1) disjoint from First k (B2).

Note that for recursive descent to work, if A ::= B1 B2 is a grammar rule we need First k (B1) disjoint from First k (B2). LL(k) Grammars We need a bunch of terminology. For any terminal string a we write First k (a) is the prefix of a of length k (or all of a if its length is less than k) For any string g of terminal and

More information

Compilers. Yannis Smaragdakis, U. Athens (original slides by Sam

Compilers. Yannis Smaragdakis, U. Athens (original slides by Sam Compilers Parsing Yannis Smaragdakis, U. Athens (original slides by Sam Guyer@Tufts) Next step text chars Lexical analyzer tokens Parser IR Errors Parsing: Organize tokens into sentences Do tokens conform

More information

Top-Down Parsing and Intro to Bottom-Up Parsing. Lecture 7

Top-Down Parsing and Intro to Bottom-Up Parsing. Lecture 7 Top-Down Parsing and Intro to Bottom-Up Parsing Lecture 7 1 Predictive Parsers Like recursive-descent but parser can predict which production to use Predictive parsers are never wrong Always able to guess

More information

EDAN65: Compilers, Lecture 06 A LR parsing. Görel Hedin Revised:

EDAN65: Compilers, Lecture 06 A LR parsing. Görel Hedin Revised: EDAN65: Compilers, Lecture 06 A LR parsing Görel Hedin Revised: 2017-09-11 This lecture Regular expressions Context-free grammar Attribute grammar Lexical analyzer (scanner) Syntactic analyzer (parser)

More information

VIVA QUESTIONS WITH ANSWERS

VIVA QUESTIONS WITH ANSWERS VIVA QUESTIONS WITH ANSWERS 1. What is a compiler? A compiler is a program that reads a program written in one language the source language and translates it into an equivalent program in another language-the

More information

Top-Down Parsing and Intro to Bottom-Up Parsing. Lecture 7

Top-Down Parsing and Intro to Bottom-Up Parsing. Lecture 7 Top-Down Parsing and Intro to Bottom-Up Parsing Lecture 7 1 Predictive Parsers Like recursive-descent but parser can predict which production to use Predictive parsers are never wrong Always able to guess

More information

A left-sentential form is a sentential form that occurs in the leftmost derivation of some sentence.

A left-sentential form is a sentential form that occurs in the leftmost derivation of some sentence. Bottom-up parsing Recall For a grammar G, with start symbol S, any string α such that S α is a sentential form If α V t, then α is a sentence in L(G) A left-sentential form is a sentential form that occurs

More information

Chapter 4. Lexical and Syntax Analysis. Topics. Compilation. Language Implementation. Issues in Lexical and Syntax Analysis.

Chapter 4. Lexical and Syntax Analysis. Topics. Compilation. Language Implementation. Issues in Lexical and Syntax Analysis. Topics Chapter 4 Lexical and Syntax Analysis Introduction Lexical Analysis Syntax Analysis Recursive -Descent Parsing Bottom-Up parsing 2 Language Implementation Compilation There are three possible approaches

More information

Acknowledgements. The slides for this lecture are a modified versions of the offering by Prof. Sanjeev K Aggarwal

Acknowledgements. The slides for this lecture are a modified versions of the offering by Prof. Sanjeev K Aggarwal Acknowledgements The slides for this lecture are a modified versions of the offering by Prof. Sanjeev K Aggarwal Syntax Analysis Check syntax and construct abstract syntax tree if == = ; b 0 a b Error

More information

CS1622. Today. A Recursive Descent Parser. Preliminaries. Lecture 9 Parsing (4)

CS1622. Today. A Recursive Descent Parser. Preliminaries. Lecture 9 Parsing (4) CS1622 Lecture 9 Parsing (4) CS 1622 Lecture 9 1 Today Example of a recursive descent parser Predictive & LL(1) parsers Building parse tables CS 1622 Lecture 9 2 A Recursive Descent Parser. Preliminaries

More information

CS 321 Programming Languages and Compilers. VI. Parsing

CS 321 Programming Languages and Compilers. VI. Parsing CS 321 Programming Languages and Compilers VI. Parsing Parsing Calculate grammatical structure of program, like diagramming sentences, where: Tokens = words Programs = sentences For further information,

More information

MIT Parse Table Construction. Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology

MIT Parse Table Construction. Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology MIT 6.035 Parse Table Construction Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology Parse Tables (Review) ACTION Goto State ( ) $ X s0 shift to s2 error error goto s1

More information

Chapter 3. Parsing #1

Chapter 3. Parsing #1 Chapter 3 Parsing #1 Parser source file get next character scanner get token parser AST token A parser recognizes sequences of tokens according to some grammar and generates Abstract Syntax Trees (ASTs)

More information

CSC 4181 Compiler Construction. Parsing. Outline. Introduction

CSC 4181 Compiler Construction. Parsing. Outline. Introduction CC 4181 Compiler Construction Parsing 1 Outline Top-down v.s. Bottom-up Top-down parsing Recursive-descent parsing LL1) parsing LL1) parsing algorithm First and follow sets Constructing LL1) parsing table

More information

UNIT III & IV. Bottom up parsing

UNIT III & IV. Bottom up parsing UNIT III & IV Bottom up parsing 5.0 Introduction Given a grammar and a sentence belonging to that grammar, if we have to show that the given sentence belongs to the given grammar, there are two methods.

More information

CS415 Compilers. Syntax Analysis. These slides are based on slides copyrighted by Keith Cooper, Ken Kennedy & Linda Torczon at Rice University

CS415 Compilers. Syntax Analysis. These slides are based on slides copyrighted by Keith Cooper, Ken Kennedy & Linda Torczon at Rice University CS415 Compilers Syntax Analysis These slides are based on slides copyrighted by Keith Cooper, Ken Kennedy & Linda Torczon at Rice University Limits of Regular Languages Advantages of Regular Expressions

More information

Compiler Design 1. Top-Down Parsing. Goutam Biswas. Lect 5

Compiler Design 1. Top-Down Parsing. Goutam Biswas. Lect 5 Compiler Design 1 Top-Down Parsing Compiler Design 2 Non-terminal as a Function In a top-down parser a non-terminal may be viewed as a generator of a substring of the input. We may view a non-terminal

More information

Compilers: CS31003 Computer Sc & Engg: IIT Kharagpur 1. Top-Down Parsing. Lect 5. Goutam Biswas

Compilers: CS31003 Computer Sc & Engg: IIT Kharagpur 1. Top-Down Parsing. Lect 5. Goutam Biswas Compilers: CS31003 Computer Sc & Engg: IIT Kharagpur 1 Top-Down Parsing Compilers: CS31003 Computer Sc & Engg: IIT Kharagpur 2 Non-terminal as a Function In a top-down parser a non-terminal may be viewed

More information

Syntax Analysis/Parsing. Context-free grammars (CFG s) Context-free grammars vs. Regular Expressions. BNF description of PL/0 syntax

Syntax Analysis/Parsing. Context-free grammars (CFG s) Context-free grammars vs. Regular Expressions. BNF description of PL/0 syntax Susan Eggers 1 CSE 401 Syntax Analysis/Parsing Context-free grammars (CFG s) Purpose: determine if tokens have the right form for the language (right syntactic structure) stream of tokens abstract syntax

More information

CS502: Compilers & Programming Systems

CS502: Compilers & Programming Systems CS502: Compilers & Programming Systems Top-down Parsing Zhiyuan Li Department of Computer Science Purdue University, USA There exist two well-known schemes to construct deterministic top-down parsers:

More information

The analysis part breaks up the source program into constituent pieces and creates an intermediate representation of the source program.

The analysis part breaks up the source program into constituent pieces and creates an intermediate representation of the source program. COMPILER DESIGN 1. What is a compiler? A compiler is a program that reads a program written in one language the source language and translates it into an equivalent program in another language-the target

More information

Syntax Analysis Check syntax and construct abstract syntax tree

Syntax Analysis Check syntax and construct abstract syntax tree Syntax Analysis Check syntax and construct abstract syntax tree if == = ; b 0 a b Error reporting and recovery Model using context free grammars Recognize using Push down automata/table Driven Parsers

More information

Abstract Syntax Trees & Top-Down Parsing

Abstract Syntax Trees & Top-Down Parsing Review of Parsing Abstract Syntax Trees & Top-Down Parsing Given a language L(G), a parser consumes a sequence of tokens s and produces a parse tree Issues: How do we recognize that s L(G)? A parse tree

More information

4 (c) parsing. Parsing. Top down vs. bo5om up parsing

4 (c) parsing. Parsing. Top down vs. bo5om up parsing 4 (c) parsing Parsing A grammar describes syntac2cally legal strings in a language A recogniser simply accepts or rejects strings A generator produces strings A parser constructs a parse tree for a string

More information

Syn S t yn a t x a Ana x lysi y s si 1

Syn S t yn a t x a Ana x lysi y s si 1 Syntax Analysis 1 Position of a Parser in the Compiler Model Source Program Lexical Analyzer Token, tokenval Get next token Parser and rest of front-end Intermediate representation Lexical error Syntax

More information

CSE P 501 Compilers. LR Parsing Hal Perkins Spring UW CSE P 501 Spring 2018 D-1

CSE P 501 Compilers. LR Parsing Hal Perkins Spring UW CSE P 501 Spring 2018 D-1 CSE P 501 Compilers LR Parsing Hal Perkins Spring 2018 UW CSE P 501 Spring 2018 D-1 Agenda LR Parsing Table-driven Parsers Parser States Shift-Reduce and Reduce-Reduce conflicts UW CSE P 501 Spring 2018

More information

Derivations of a CFG. MACM 300 Formal Languages and Automata. Context-free Grammars. Derivations and parse trees

Derivations of a CFG. MACM 300 Formal Languages and Automata. Context-free Grammars. Derivations and parse trees Derivations of a CFG MACM 300 Formal Languages and Automata Anoop Sarkar http://www.cs.sfu.ca/~anoop strings grow on trees strings grow on Noun strings grow Object strings Verb Object Noun Verb Object

More information

Syntax Analysis. Amitabha Sanyal. (www.cse.iitb.ac.in/ as) Department of Computer Science and Engineering, Indian Institute of Technology, Bombay

Syntax Analysis. Amitabha Sanyal. (www.cse.iitb.ac.in/ as) Department of Computer Science and Engineering, Indian Institute of Technology, Bombay Syntax Analysis (www.cse.iitb.ac.in/ as) Department of Computer Science and Engineering, Indian Institute of Technology, Bombay September 2007 College of Engineering, Pune Syntax Analysis: 2/124 Syntax

More information

Context-free grammars (CFG s)

Context-free grammars (CFG s) Syntax Analysis/Parsing Purpose: determine if tokens have the right form for the language (right syntactic structure) stream of tokens abstract syntax tree (AST) AST: captures hierarchical structure of

More information

Compilers. Predictive Parsing. Alex Aiken

Compilers. Predictive Parsing. Alex Aiken Compilers Like recursive-descent but parser can predict which production to use By looking at the next fewtokens No backtracking Predictive parsers accept LL(k) grammars L means left-to-right scan of input

More information

Parsing. Note by Baris Aktemur: Our slides are adapted from Cooper and Torczon s slides that they prepared for COMP 412 at Rice.

Parsing. Note by Baris Aktemur: Our slides are adapted from Cooper and Torczon s slides that they prepared for COMP 412 at Rice. Parsing Note by Baris Aktemur: Our slides are adapted from Cooper and Torczon s slides that they prepared for COMP 412 at Rice. Copyright 2010, Keith D. Cooper & Linda Torczon, all rights reserved. Students

More information

CSE302: Compiler Design

CSE302: Compiler Design CSE302: Compiler Design Instructor: Dr. Liang Cheng Department of Computer Science and Engineering P.C. Rossin College of Engineering & Applied Science Lehigh University February 20, 2007 Outline Recap

More information

Part 5 Program Analysis Principles and Techniques

Part 5 Program Analysis Principles and Techniques 1 Part 5 Program Analysis Principles and Techniques Front end 2 source code scanner tokens parser il errors Responsibilities: Recognize legal programs Report errors Produce il Preliminary storage map Shape

More information

shift-reduce parsing

shift-reduce parsing Parsing #2 Bottom-up Parsing Rightmost derivations; use of rules from right to left Uses a stack to push symbols the concatenation of the stack symbols with the rest of the input forms a valid bottom-up

More information

Building a Parser III. CS164 3:30-5:00 TT 10 Evans. Prof. Bodik CS 164 Lecture 6 1

Building a Parser III. CS164 3:30-5:00 TT 10 Evans. Prof. Bodik CS 164 Lecture 6 1 Building a Parser III CS164 3:30-5:00 TT 10 Evans 1 Overview Finish recursive descent parser when it breaks down and how to fix it eliminating left recursion reordering productions Predictive parsers (aka

More information

Lexical and Syntax Analysis. Top-Down Parsing

Lexical and Syntax Analysis. Top-Down Parsing Lexical and Syntax Analysis Top-Down Parsing Easy for humans to write and understand String of characters Lexemes identified String of tokens Easy for programs to transform Data structure Syntax A syntax

More information

Lecture 7: Deterministic Bottom-Up Parsing

Lecture 7: Deterministic Bottom-Up Parsing Lecture 7: Deterministic Bottom-Up Parsing (From slides by G. Necula & R. Bodik) Last modified: Tue Sep 20 12:50:42 2011 CS164: Lecture #7 1 Avoiding nondeterministic choice: LR We ve been looking at general

More information

Syntax Analysis. Prof. James L. Frankel Harvard University. Version of 6:43 PM 6-Feb-2018 Copyright 2018, 2015 James L. Frankel. All rights reserved.

Syntax Analysis. Prof. James L. Frankel Harvard University. Version of 6:43 PM 6-Feb-2018 Copyright 2018, 2015 James L. Frankel. All rights reserved. Syntax Analysis Prof. James L. Frankel Harvard University Version of 6:43 PM 6-Feb-2018 Copyright 2018, 2015 James L. Frankel. All rights reserved. Context-Free Grammar (CFG) terminals non-terminals start

More information

Bottom-Up Parsing. Parser Generation. LR Parsing. Constructing LR Parser

Bottom-Up Parsing. Parser Generation. LR Parsing. Constructing LR Parser Parser Generation Main Problem: given a grammar G, how to build a top-down parser or a bottom-up parser for it? parser : a program that, given a sentence, reconstructs a derivation for that sentence ----

More information

CSE 3302 Programming Languages Lecture 2: Syntax

CSE 3302 Programming Languages Lecture 2: Syntax CSE 3302 Programming Languages Lecture 2: Syntax (based on slides by Chengkai Li) Leonidas Fegaras University of Texas at Arlington CSE 3302 L2 Spring 2011 1 How do we define a PL? Specifying a PL: Syntax:

More information

Abstract Syntax Trees & Top-Down Parsing

Abstract Syntax Trees & Top-Down Parsing Abstract Syntax Trees & Top-Down Parsing Review of Parsing Given a language L(G), a parser consumes a sequence of tokens s and produces a parse tree Issues: How do we recognize that s L(G)? A parse tree

More information

Abstract Syntax Trees & Top-Down Parsing

Abstract Syntax Trees & Top-Down Parsing Review of Parsing Abstract Syntax Trees & Top-Down Parsing Given a language L(G), a parser consumes a sequence of tokens s and produces a parse tree Issues: How do we recognize that s L(G)? A parse tree

More information

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING KATHMANDU UNIVERSITY SCHOOL OF ENGINEERING DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING REPORT ON NON-RECURSIVE PREDICTIVE PARSER Fourth Year First Semester Compiler Design Project Final Report submitted

More information

Ambiguity, Precedence, Associativity & Top-Down Parsing. Lecture 9-10

Ambiguity, Precedence, Associativity & Top-Down Parsing. Lecture 9-10 Ambiguity, Precedence, Associativity & Top-Down Parsing Lecture 9-10 (From slides by G. Necula & R. Bodik) 9/18/06 Prof. Hilfinger CS164 Lecture 9 1 Administrivia Please let me know if there are continued

More information

EDAN65: Compilers, Lecture 04 Grammar transformations: Eliminating ambiguities, adapting to LL parsing. Görel Hedin Revised:

EDAN65: Compilers, Lecture 04 Grammar transformations: Eliminating ambiguities, adapting to LL parsing. Görel Hedin Revised: EDAN65: Compilers, Lecture 04 Grammar transformations: Eliminating ambiguities, adapting to LL parsing Görel Hedin Revised: 2017-09-04 This lecture Regular expressions Context-free grammar Attribute grammar

More information

ΕΠΛ323 - Θεωρία και Πρακτική Μεταγλωττιστών

ΕΠΛ323 - Θεωρία και Πρακτική Μεταγλωττιστών ΕΠΛ323 - Θεωρία και Πρακτική Μεταγλωττιστών Lecture 5b Syntax Analysis Elias Athanasopoulos eliasathan@cs.ucy.ac.cy Regular Expressions vs Context-Free Grammars Grammar for the regular expression (a b)*abb

More information

Syntax Analysis (Parsing)

Syntax Analysis (Parsing) An overview of parsing Functions & Responsibilities Context Free Grammars Concepts & Terminology Writing and Designing Grammars Syntax Analysis (Parsing) Resolving Grammar Problems / Difficulties Top-Down

More information

Bottom up parsing. The sentential forms happen to be a right most derivation in the reverse order. S a A B e a A d e. a A d e a A B e S.

Bottom up parsing. The sentential forms happen to be a right most derivation in the reverse order. S a A B e a A d e. a A d e a A B e S. Bottom up parsing Construct a parse tree for an input string beginning at leaves and going towards root OR Reduce a string w of input to start symbol of grammar Consider a grammar S aabe A Abc b B d And

More information

CSE P 501 Compilers. Parsing & Context-Free Grammars Hal Perkins Winter /15/ Hal Perkins & UW CSE C-1

CSE P 501 Compilers. Parsing & Context-Free Grammars Hal Perkins Winter /15/ Hal Perkins & UW CSE C-1 CSE P 501 Compilers Parsing & Context-Free Grammars Hal Perkins Winter 2008 1/15/2008 2002-08 Hal Perkins & UW CSE C-1 Agenda for Today Parsing overview Context free grammars Ambiguous grammars Reading:

More information

Introduction to Syntax Analysis

Introduction to Syntax Analysis Compiler Design 1 Introduction to Syntax Analysis Compiler Design 2 Syntax Analysis The syntactic or the structural correctness of a program is checked during the syntax analysis phase of compilation.

More information

Lecture 8: Deterministic Bottom-Up Parsing

Lecture 8: Deterministic Bottom-Up Parsing Lecture 8: Deterministic Bottom-Up Parsing (From slides by G. Necula & R. Bodik) Last modified: Fri Feb 12 13:02:57 2010 CS164: Lecture #8 1 Avoiding nondeterministic choice: LR We ve been looking at general

More information

Parser Generation. Bottom-Up Parsing. Constructing LR Parser. LR Parsing. Construct parse tree bottom-up --- from leaves to the root

Parser Generation. Bottom-Up Parsing. Constructing LR Parser. LR Parsing. Construct parse tree bottom-up --- from leaves to the root Parser Generation Main Problem: given a grammar G, how to build a top-down parser or a bottom-up parser for it? parser : a program that, given a sentence, reconstructs a derivation for that sentence ----

More information

Principles of Programming Languages COMP251: Syntax and Grammars

Principles of Programming Languages COMP251: Syntax and Grammars Principles of Programming Languages COMP251: Syntax and Grammars Prof. Dekai Wu Department of Computer Science and Engineering The Hong Kong University of Science and Technology Hong Kong, China Fall 2006

More information

Chapter 4. Lexical and Syntax Analysis

Chapter 4. Lexical and Syntax Analysis Chapter 4 Lexical and Syntax Analysis Chapter 4 Topics Introduction Lexical Analysis The Parsing Problem Recursive-Descent Parsing Bottom-Up Parsing Copyright 2012 Addison-Wesley. All rights reserved.

More information

Introduction to Syntax Analysis. The Second Phase of Front-End

Introduction to Syntax Analysis. The Second Phase of Front-End Compiler Design IIIT Kalyani, WB 1 Introduction to Syntax Analysis The Second Phase of Front-End Compiler Design IIIT Kalyani, WB 2 Syntax Analysis The syntactic or the structural correctness of a program

More information

CMSC 330: Organization of Programming Languages

CMSC 330: Organization of Programming Languages CMSC 330: Organization of Programming Languages Context Free Grammars and Parsing 1 Recall: Architecture of Compilers, Interpreters Source Parser Static Analyzer Intermediate Representation Front End Back

More information

Syntax Analysis. COMP 524: Programming Language Concepts Björn B. Brandenburg. The University of North Carolina at Chapel Hill

Syntax Analysis. COMP 524: Programming Language Concepts Björn B. Brandenburg. The University of North Carolina at Chapel Hill Syntax Analysis Björn B. Brandenburg The University of North Carolina at Chapel Hill Based on slides and notes by S. Olivier, A. Block, N. Fisher, F. Hernandez-Campos, and D. Stotts. The Big Picture Character

More information

The procedure attempts to "match" the right hand side of some production for a nonterminal.

The procedure attempts to match the right hand side of some production for a nonterminal. Parsing A parser is an algorithm that determines whether a given input string is in a language and, as a side-effect, usually produces a parse tree for the input. There is a procedure for generating a

More information

Formal Languages and Compilers Lecture VII Part 3: Syntactic A

Formal Languages and Compilers Lecture VII Part 3: Syntactic A Formal Languages and Compilers Lecture VII Part 3: Syntactic Analysis Free University of Bozen-Bolzano Faculty of Computer Science POS Building, Room: 2.03 artale@inf.unibz.it http://www.inf.unibz.it/

More information

Syntax Analysis. The Big Picture. The Big Picture. COMP 524: Programming Languages Srinivas Krishnan January 25, 2011

Syntax Analysis. The Big Picture. The Big Picture. COMP 524: Programming Languages Srinivas Krishnan January 25, 2011 Syntax Analysis COMP 524: Programming Languages Srinivas Krishnan January 25, 2011 Based in part on slides and notes by Bjoern Brandenburg, S. Olivier and A. Block. 1 The Big Picture Character Stream Token

More information

Introduction to Parsing. Lecture 8

Introduction to Parsing. Lecture 8 Introduction to Parsing Lecture 8 Adapted from slides by G. Necula Outline Limitations of regular languages Parser overview Context-free grammars (CFG s) Derivations Languages and Automata Formal languages

More information

Question Bank. 10CS63:Compiler Design

Question Bank. 10CS63:Compiler Design Question Bank 10CS63:Compiler Design 1.Determine whether the following regular expressions define the same language? (ab)* and a*b* 2.List the properties of an operator grammar 3. Is macro processing a

More information

Parsing III. CS434 Lecture 8 Spring 2005 Department of Computer Science University of Alabama Joel Jones

Parsing III. CS434 Lecture 8 Spring 2005 Department of Computer Science University of Alabama Joel Jones Parsing III (Top-down parsing: recursive descent & LL(1) ) (Bottom-up parsing) CS434 Lecture 8 Spring 2005 Department of Computer Science University of Alabama Joel Jones Copyright 2003, Keith D. Cooper,

More information