Context-free grammars (CFGs)

Transcription

1 Context-free grammars (CFGs)

2 Roadmap Last time RegExp == DFA Jlex: a tool for generating (Java code for) a lexer/scanner Mainly a collection of regexp, action pairs This time CFGs, the underlying abstraction for parsers Next week Java CUP: a tool for generating (Java code for) a parser Mainly a collection of CFG-rule, action pairs regexp : JLex :: CFG : Java CUP

3 RegExps Are Great! Perfect for tokenizing a language However, they have some limitations Can only define a limited family of languages Cannot use a RegExp to specify all the programming constructs we need No notion of structure Let s explore both of these issues

4 Limitations of RegExps Cannot handle matching E.g., language of balanced parentheses L () = { ( n ) n where n > 0} No DFA exists for this language Intuition: A given FSM only has a fixed, finite amount of memory For an FSM, memory = the states With a fixed, finite amount of memory, how could an FSM remember how many ( characters it has seen?

5 Theorem: No RegExp/DFA can describe the language L () Proof by contradiction: Suppose that there exists a DFA A for L () and A has N states A has to accept the string ( N ) N with some path q 0 q 1 q N q 2N+1 By the pigeonhole principle some state has to repeat: q i = q j for some i<j<n Therefore the run q 0 q 1 q i q j+1 q N q 2N+1 is also accepting A accepts the string ( N-(j-i) ) N L (), which is a contradiction!

6 Limitations of RegExps: No Structure Our Enhanced-RegExp scanner can emit a stream of tokens: X = Y + Z ID ASSIGN ID PLUS ID but this doesn t really enforce any order of operations

7 The Chomsky Hierarchy LANGUAGE CLASS: power efficiency Recursively enumerable Turing machine Context-Sensitive Context-Free Happy medium? Regular FSM Noam Chomsky

8 Context Free Grammars (CFGs) A set of (recursive) rewriting rules to generate patterns of strings Can envision a parse tree that keeps structure

9 CFG: Intuition S ( S ) A rule that says that you can rewrite S to be an S surrounded by a single set of parenthesis Before applying rule S After applying rule S ( S )

10 Context Free Grammars (CFGs) A CFG is a 4-tuple (N,Σ,P,S) N is a set of non-terminals, e.g., A, B, S, Σ is the set of terminals P is a set of production rules S N is the initial non-terminal symbol ( start symbol )

11 Context Free Grammars (CFGs) A CFG is a 4-tuple (N,Σ,P,S) N is a set of non-terminals, e.g., A, B, S Σ is the set of terminals P is a set of production rules S (in N) is the initial non-terminal symbol Placeholder / interior nodes in the parse tree Tokens from scanner Rules for deriving strings If not otherwise specified, use the non-terminal that appears on the LHS of the first production as the start

12 Production Syntax LHS RHS Single nonterminal symbol Expression: Sequence of terminals and nonterminals Examples: S à ( S ) S à ε

13 Production Shorthand Nonterm expression Nonterm ε equivalently: Nonterm expression ε equivalently: Nonterm expression ε S à ( S ) S à ε S à ( S ) ε S à ( S ) ε

14 Derivations To derive a string: Start by setting Current Sequence to the start symbol Repeatedly, Find a Nonterminal X in the Current Sequence Find a production of the form X α Apply the production: create a new current sequence in which α replaces X Stop when there are no more non-terminals This process derives a string of terminal symbols

15 Derivation Syntax We ll use the symbol for derives We ll use the symbol & for derives in one or more steps (also written as & ) We ll use the symbol for derives in zero or more steps (also written as )

16 An Example Grammar

17 Terminals begin end semicolon assign id plus An Example Grammar

18 For readability, bold and lowercase Terminals begin end semicolon assign id plus An Example Grammar

19 For readability, bold and lowercase Terminals begin Program end boundary semicolon assign id plus An Example Grammar

20 An Example Grammar For readability, bold and lowercase Terminals begin Program end boundary semicolon Represents ; assign Separates statements id plus

21 An Example Grammar For readability, bold and lowercase Terminals begin Program end boundary semicolon Represents ; assign Separates statements id plus Represents = in an assignment statement

22 An Example Grammar For readability, bold and lowercase Terminals begin Program end boundary semicolon Represents ; assign Separates statements id plus Represents = in an assignment statement Identifier / variable name

23 An Example Grammar For readability, bold and lowercase Terminals begin Program end boundary semicolon Represents ; assign Separates statements id plus Represents = in an assignment statement Identifier / variable name Represents + operator in an expression

24 For readability, bold and lowercase Terminals begin end semicolon assign id plus An Example Grammar Nonterminals Prog Stmts Stmt Expr

25 For readability, bold and lowercase Terminals begin end semicolon assign id plus An Example Grammar For readability, Italics and UpperCamelCase Nonterminals Prog Stmts Stmt Expr

26 For readability, bold and lowercase Terminals begin end semicolon assign id plus An Example Grammar For readability, Italics and UpperCamelCase Nonterminals Prog Root of the parse tree Stmts Stmt Expr

27 For readability, bold and lowercase Terminals begin end semicolon assign id plus An Example Grammar For readability, Italics and UpperCamelCase Nonterminals Prog Root of the parse tree Stmts List of statements Stmt Expr

28 For readability, bold and lowercase Terminals begin end semicolon assign id plus An Example Grammar For readability, Italics and UpperCamelCase Nonterminals Prog Root of the parse tree Stmts List of statements Stmt A single statement Expr

29 For readability, bold and lowercase Terminals begin end semicolon assign id plus An Example Grammar For readability, Italics and UpperCamelCase Nonterminals Prog Root of the parse tree Stmts List of statements Stmt A single statement Expr A mathematical expression

30 An Example Grammar For readability, bold and lowercase Terminals begin end semicolon assign id plus For readability, Italics and UpperCamelCase Nonterminals Prog Stmts Stmt Expr Defines the syntax of legal programs Productions Prog begin Stmts end Stmts Stmts semicolon Stmt Stmt Stmt id assign Expr Expr id Expr plus id

31 An Example Grammar For readability, bold and lowercase Terminals begin Program end boundary semicolon Represents ; assign Separates statements id plus Represents = statement Identifier / variable name Represents + expression For readability, Italics and UpperCamelCase Nonterminals Prog Root of the parse tree Stmts List of statements Stmt A single statement Expr An expression Defines the syntax of legal programs Productions Prog begin Stmts end Stmts Stmts semicolon Stmt Stmt Stmt id assign Expr Expr id Expr plus id

32 Productions 1. Prog begin Stmts end 2. Stmts Stmts semicolon Stmt 3. Stmt 4. Stmt id assign Expr 5. Expr id 6. Expr plus id

33 Productions 1. Prog begin Stmts end 2. Stmts Stmts semicolon Stmt 3. Stmt 4. Stmt id assign Expr 5. Expr id 6. Expr plus id Derivation Sequence

34 Productions 1. Prog begin Stmts end Parse Tree 2. Stmts Stmts semicolon Stmt 3. Stmt 4. Stmt id assign Expr 5. Expr id 6. Expr plus id Derivation Sequence

35 Productions 1. Prog begin Stmts end Parse Tree 2. Stmts Stmts semicolon Stmt 3. Stmt 4. Stmt id assign Expr 5. Expr id 6. Expr plus id Derivation Sequence Key terminal Nonterminal Rule used

36 Productions 1. Prog begin Stmts end 2. Stmts Stmts semicolon Stmt 3. Stmt 4. Stmt id assign Expr 5. Expr id 6. Expr plus id Parse Tree Prog Derivation Sequence Prog Key terminal Nonterminal Rule used

37 Productions 1. Prog begin Stmts end 2. Stmts Stmts semicolon Stmt 3. Stmt 4. Stmt id assign Expr 5. Expr id 6. Expr plus id Parse Tree Prog Derivation Sequence Prog begin Stmts end 1 Key terminal Nonterminal Rule used

38 Productions 1. Prog begin Stmts end 2. Stmts Stmts semicolon Stmt 3. Stmt begin 4. Stmt id assign Expr 5. Expr id 6. Expr plus id Parse Tree Prog Stmts end Derivation Sequence Prog begin Stmts end 1 Key terminal Nonterminal Rule used

39 Productions 1. Prog begin Stmts end 2. Stmts Stmts semicolon Stmt 3. Stmt begin Prog Stmts end 4. Stmt id assign Expr 5. Expr id Stmts semicolon Stmt 6. Expr plus id Parse Tree Derivation Sequence Prog begin Stmts end 1 begin Stmts semicolon Stmt end 2 Key terminal Nonterminal Rule used

40 Productions 1. Prog begin Stmts end 2. Stmts Stmts semicolon Stmt 3. Stmt begin Prog Stmts end 4. Stmt id assign Expr 5. Expr id Stmts semicolon Stmt 6. Expr plus id Stmt Parse Tree Derivation Sequence Prog begin Stmts end 1 begin Stmts semicolon Stmt end begin Stmt semicolon Stmt end 2 3 Key terminal Nonterminal Rule used

41 Productions 1. Prog begin Stmts end 2. Stmts Stmts semicolon Stmt 3. Stmt begin Prog Stmts end 4. Stmt id assign Expr 5. Expr id Stmts semicolon Stmt 6. Expr plus id Stmt Parse Tree Derivation Sequence id Prog begin Stmts end 1 2 begin Stmts semicolon Stmt end 3 begin Stmt semicolon Stmt end begin id assign Expr semicolon Stmt end assign Expr 4 Key terminal Nonterminal Rule used

42 Productions 1. Prog begin Stmts end 2. Stmts Stmts semicolon Stmt 3. Stmt begin Prog Stmts end 4. Stmt id assign Expr 5. Expr id Stmts Parse Tree semicolon Stmt 6. Expr plus id Stmt id assign Expr Derivation Sequence id assign Prog begin Stmts end 1 2 begin Stmts semicolon Stmt end 3 begin Stmt semicolon Stmt end 4 begin id assign Expr semicolon Stmt end begin id assign Expr semicolon id assign Expr end Expr 4 Key terminal Nonterminal Rule used

43 Productions 1. Prog begin Stmts end 2. Stmts Stmts semicolon Stmt 3. Stmt begin Prog Stmts end 4. Stmt id assign Expr 5. Expr id Stmts Parse Tree semicolon Stmt 6. Expr plus id Stmt id assign Expr Derivation Sequence id assign Prog begin Stmts end 1 2 begin Stmts semicolon Stmt end 3 begin Stmt semicolon Stmt end 4 begin id assign Expr semicolon Stmt end begin id assign Expr semicolon id assign Expr end begin id assign id semicolon id assign Expr end Expr id 4 5 Key terminal Nonterminal Rule used

44 Productions 1. Prog begin Stmts end 2. Stmts Stmts semicolon Stmt 3. Stmt begin Prog Stmts end 4. Stmt id assign Expr 5. Expr id Stmts Parse Tree semicolon Stmt 6. Expr plus id Stmt id assign Expr Derivation Sequence id assign Expr Prog begin Stmts end 1 2 begin Stmts semicolon Stmt end id 3 begin Stmt semicolon Stmt end 4 4 begin id assign Expr semicolon Stmt end 5 begin id assign Expr semicolon id assign Expr end begin id assign id semicolon id assign Expr end begin id assign id semicolon id assign Expr plus id end 6 Expr plus id Key terminal Nonterminal Rule used

45 Productions 1. Prog begin Stmts end 2. Stmts Stmts semicolon Stmt 3. Stmt begin Prog Stmts end 4. Stmt id assign Expr 5. Expr id Stmts Parse Tree semicolon Stmt 6. Expr plus id Stmt id assign Expr Derivation Sequence id assign Expr Prog begin Stmts end 1 2 begin Stmts semicolon Stmt end id 3 begin Stmt semicolon Stmt end 4 4 begin id assign Expr semicolon Stmt end 5 begin id assign Expr semicolon id assign Expr end begin id assign id semicolon id assign Expr end begin id assign id semicolon id assign Expr plus id end begin id assign id semicolon id assign id plus id end 5 6 Expr plus id id Key terminal Nonterminal Rule used

46 A five minute introduction MAKEFILE

47 Makefiles: Motivation Typing the series of commands to generate our code can be tedious Multiple steps that depend on each other Somewhat complicated commands May not need to rebuild everything Makefiles solve these issues Record a series of commands in a script-like DSL Specify dependency rules and Make generates the results

48 Makefiles: Basic Structure <target>: <dependency list> (tab) <command to satisfy target> Example Example.class: Example.java IO.class javac Example.java IO.class: IO.java javac IO.java

49 Makefiles: Basic Structure <target>: <dependency list> (tab) <command to satisfy target> Example Example.class: Example.java IO.class javac Example.java IO.class: IO.java javac IO.java

50 Makefiles: Basic Structure <target>: <dependency list> (tab) <command to satisfy target> Example Example.class: Example.java IO.class javac Example.java IO.class: IO.java javac IO.java Example.class depends on example.java and IO.class

51 Makefiles: Basic Structure <target>: <dependency list> (tab) <command to satisfy target> Example Example.class: Example.java IO.class javac Example.java IO.class: IO.java javac IO.java Example.class depends on example.java and IO.class Example.class is generated by javac Example.java

52 Makefiles: Dependencies Example.class Example.java IO.class Example Example.class: Example.java IO.class javac Example.java IO.class: IO.java javac IO.java IO.java

53 Makefiles: Dependencies Internal Dependency graph Example.class Example.java IO.class Example Example.class: Example.java IO.class javac Example.java IO.class: IO.java javac IO.java IO.java

54 Makefiles: Dependencies Internal Dependency graph Example.class Example.java Example Example.class: Example.java IO.class javac Example.java IO.class: IO.java javac IO.java IO.class IO.java A file is rebuilt if one of it s dependencies changes

55 Makefiles: Variables You can thread common configuration values through your makefile

56 Makefiles: Variables You can thread common configuration values through your makefile Example JC = /s/std/bin/javac JFLAGS = -g

57 Makefiles: Variables You can thread common configuration values through your makefile Example JC = /s/std/bin/javac JFLAGS = -g Build for debug

58 Makefiles: Variables You can thread common configuration values through your makefile Example JC = /s/std/bin/javac JFLAGS = -g Build for debug Example.class: Example.java IO.class $(JC) $(JFLAGS) Example.java IO.class: IO.java $(JC) $(JFLAGS) IO.java

59 Makefiles: Phony Targets You can run commands via make Write a target with no dependencies (called phony) Will cause it to execute the command every time Example clean: rm f *.class test: java cp. Test.class

60 Makefiles: Phony Targets You can run commands via make Write a target with no dependencies (called phony) Will cause it to execute the command every time Example clean: rm f *.class test: java cp. Test.class

61 Makefiles: Phony Targets You can run commands via make Write a target with no dependencies (called phony) Will cause it to execute the command every time Example clean: rm f *.class test: java cp. Test.class

62 Recap We ve defined context-free grammars More powerful than regular expressions Learned a bit about makefiles Next time: we ll look at grammars in more detail