Introduction to Parsing. Lecture 5. Professor Alex Aiken Lecture #5 (Modified by Professor Vijay Ganesh)

Transcription

1 Introduction to Parsing Lecture 5 (Modified by Professor Vijay Ganesh) 1

2 Outline Regular languages revisited Parser overview Context-free grammars (CFG s) Derivations Ambiguity 2

3 Languages and Automata Formal languages are very important in CS specially in programming languages Regular languages The weakest formal languages wely used Many applications We will also study context-free languages, tree languages 3

4 Beyond Regular Languages Many languages are not regular Strings of balanced parentheses are not regular: {() i i i 0} 4

5 What Can Regular Languages xpress? Languages requiring counting modulo a fixed integer Intuition: A finite automaton that runs long enough must repeat states Finite automaton can t remember # of times it has visited a particular state 5

6 The Functionality of the Parser Input: sequence of tokens from lexer Output: parse tree of the program (But some parsers never produce a parse tree...) 6

7 xample Cool if x = y then 1 else 2 fi Parser input IF ID = ID THN INT LS INT FI Parser output IF-THN-LS = INT INT ID ID 7

8 Comparison with Lexical Analysis Phase Input Output Lexer Parser String of characters String of tokens String of tokens Parse tree 8

9 The Role of the Parser Not all strings of tokens are programs parser must distinguish between val and inval strings of tokens We need A language for describing val strings of tokens A method for distinguishing val from inval strings of tokens 9

10 Context-Free Grammars Programming language constructs have recursive structure An XPR is if XPR then XPR else XPR fi while XPR loop XPR pool Context-free grammars are a natural notation for this recursive structure 10

11 CFGs (Cont.) A CFG consists of A set of terminals T A set of non-terminals N A start symbol S (a non-terminal) A set of productions X YY L Y 1 2 n where { } X N and Y T N ε i 11

12 Notational Conventions In these lecture notes Non-terminals are written upper-case Terminals are written lower-case The start symbol is the left-hand se of the first production 12

13 xamples of CFGs A fragment of Cool: XPR if XPR then XPR else XPR fi while XPR loop XPR pool 13

14 xamples of CFGs (cont.) Simple arithmetic expressions: + ( ) 14

15 The Language of a CFG Read productions as rules: Means X YL Y 1 n X can be replaced by 1 YL Y n 15

16 Key Idea 1. Begin with a string consisting of the start symbol S 2. Replace any non-terminal X in the string by a the right-hand se of some production X YL Y 1 n 3. Repeat (2) until there are no non-terminals in the string 16

17 The Language of a CFG (Cont.) More formally, write X L X L X X L X YL Y X L X 1 i n 1 i 1 1 m i+ 1 n if there is a production X YL Y i 1 m 17

18 The Language of a CFG (Cont.) Write if X L X YL Y 1 n 1 m in 0 or more steps X L X L L YL Y 1 n 1 m 18

19 The Language of a CFG Let G be a context-free grammar with start symbol S. Then the language of G is: { and every is a terminal } a K a Sa K a a 1 n 1 n i 19

20 Terminals Terminals are so-called because there are no rules for replacing them Once generated, terminals are permanent Terminals ought to be tokens of the language 20

21 xamples L(G) is the language of CFG G Strings of balanced parentheses {() i i i 0} Two grammars: S S ( S) ε OR S ( S) ε 21

22 Cool xample A fragment of COOL: XPR if XPR then XPR else XPR fi while XPR loop XPR pool 22

23 Cool xample (Cont.) Some elements of the language if then else fi while loop pool if while loop pool then else if if then else fi then else fi 23

24 Arithmetic xample Simple arithmetic expressions: + () Some elements of the language: + () () () 24

25 Notes The ea of a CFG is a big step. But: Membership in a language is yes or no ; also need parse tree of the input Must handle errors gracefully Need an implementation of CFG s (e.g., bison) 25

26 More Notes Form of the grammar is important Many grammars generate the same language Tools are sensitive to the grammar Note: Tools for regular languages (e.g., flex) are sensitive to the form of the regular expression, but this is rarely a problem in practice 26

27 Derivations and Parse Trees A derivation is a sequence of productions S L L L A derivation can be drawn as a tree Start symbol is the tree s root X YL Y n For a production add children 1 to node X Y1 L Y n 27

28 Derivation xample Grammar + () String + 28

29 Derivation xample (Cont.) *

30 Derivation in Detail (1) 30

31 Derivation in Detail (2)

32 Derivation in Detail (3) * 32

33 Derivation in Detail (4) * + 33

34 Derivation in Detail (5) * + 34

35 Derivation in Detail (6) *

36 Notes on Derivations A parse tree has Terminals at the leaves Non-terminals at the interior nodes An in-order traversal of the leaves is the original input The parse tree shows the association of operations, the input string does not 36

37 Left-most and Right-most Derivations The example is a leftmost derivation At each step, replace the left-most non-terminal There is an equivalent notion of a right-most derivation

38 Right-most Derivation in Detail (1) 38

39 Right-most Derivation in Detail (2)

40 Right-most Derivation in Detail (3)

41 Right-most Derivation in Detail (4) * + 41

42 Right-most Derivation in Detail (5) * + 42

43 Right-most Derivation in Detail (6) *

44 Derivations and Parse Trees Note that right-most and left-most derivations have the same parse tree The difference is the order in which branches are added 44

45 Summary of Derivations We are not just interested in whether s e L(G) We need a parse tree for s A derivation defines a parse tree But one parse tree may have many derivations Left-most and right-most derivations are important in parser implementation 45

46 Ambiguity Grammar + () String + 46

47 Ambiguity (Cont.) This string has two parse trees + * * + 47

48 Ambiguity (Cont.) A grammar is ambiguous if it has more than one parse tree for some string quivalently, there is more than one right-most or left-most derivation for some string Ambiguity is BAD Leaves meaning of some programs ill-defined 48

49 Dealing with Ambiguity There are several ways to handle ambiguity Most direct method is to rewrite grammar unambiguously + ' ' ʹ () ʹ () ' nforces precedence of * over + 49

50 Ambiguity in Arithmetic xpressions Recall the grammar + * ( ) int The string int * int + int has two parse trees: + * * int int + int int int int 50

51 Ambiguity: The Dangling lse Conser the grammar if then if then else OTHR This grammar is also ambiguous 51

52 The Dangling lse: xample The expression if 1 then if 2 then 3 else 4 has two parse trees if if 1 if 4 1 if Typically we want the second form 52

53 The Dangling lse: A Fix else matches the closest unmatched then We can describe this in the grammar MIF /* all then are matched */ UIF /* some then is unmatched */ MIF if then MIF else MIF OTHR UIF if then if then MIF else UIF Describes the same set of strings 53

54 The Dangling lse: xample Revisited The expression if 1 then if 2 then 3 else 4 if if 1 if 1 if A val parse tree (for a UIF) Not val because the then expression is not a MIF 54

55 Ambiguity No general techniques for handling ambiguity Impossible to convert automatically an ambiguous grammar to an unambiguous one Used with care, ambiguity can simplify the grammar Sometimes allows more natural definitions We need disambiguation mechanisms 55

56 Precedence and Associativity Declarations Instead of rewriting the grammar Use the more natural (ambiguous) grammar Along with disambiguating declarations Most tools allow precedence and associativity declarations to disambiguate grammars xamples 56

57 Associativity Declarations Conser the grammar + int Ambiguous: two parse trees of int + int + int int int + int int int int Left associativity declaration: %left + 57

58 Precedence Declarations Conser the grammar + * int And the string int + int * int * + + int int * int int int Precedence declarations: %left + %left * int 58