Compilers and Language Processing Tools

Transcription

1 Compilers and Language Processing Tools Summer Term 2011 Prof. Dr. Arnd Poetzsch-Heffter Software Technology Group TU Kaiserslautern c Prof. Dr. Arnd Poetzsch-Heffter 1

2 Parser Generators c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 2

3 Preparations $ mkdir clp11 $ cd clp11 c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 3

4 ANTLR ANTLR Parser generator for LL(k) grammars (LL(*)) Integrates Scanner generator and AST generator c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 4

5 ANTLR ANTLR Parser generator for LL(k) grammars (LL(*)) Integrates Scanner generator and AST generator $ java -jar /home/j_schaef/public/antlrworks jar c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 5

6 Simple Arithmetic Expressions Simple arithmetic expressions: c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 6

7 Simple Arithmetic Expressions Simple arithmetic expressions: Enter the following grammar into ANTLR: grammar test; expr : expr + expr NUM; NUM : ( )*; c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 7

8 Simple Arithmetic Expressions Simple arithmetic expressions: Enter the following grammar into ANTLR: grammar test; expr : expr + expr NUM; NUM : ( )*; ANTLR Error: Rule expr is left-recursive c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 8

9 Remove Left-Recursion c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 9

10 Remove Left-Recursion grammar test; expr : NUM exprp NUM; exprp : + expr; NUM : ( )*; c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 10

11 Java CUP Java CUP Parser generator for LALR grammars c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 11

12 Simple Expressions expr ::= expr + expr number number ::= 0 [1-9][0-9]+ c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 12

13 Steps c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 13

14 Steps 1. Identify and name the terminal and non-terminal symbols c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 14

15 Steps 1. Identify and name the terminal and non-terminal symbols 2. Define a grammar c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 15

16 Steps 1. Identify and name the terminal and non-terminal symbols 2. Define a grammar 3. Define an unambiguous grammar c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 16

17 First Try Parser.cup: terminal PLUS, NUMBER; non terminal expr; expr ::= expr PLUS expr NUMBER ; c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 17

18 First Try Parser.cup: terminal PLUS, NUMBER; non terminal expr; expr ::= expr PLUS expr NUMBER ; $ java -jar java-cup.jar Parser.cup c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 18

19 Java CUP Output Warning : *** Shift/Reduce conflict found in state #5 between expr ::= expr PLUS expr (*) and expr ::= expr (*) PLUS expr under symbol PLUS Resolved in favor of shifting. Error : *** More conflicts encountered than expected -- parser generation aborted c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 19

20 Debugging the Generated Parser Use the -dump_states option of JavaCUP. $ java -jar java-cup.jar -dump_states Parser.cup c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 20

21 ===== Viable Prefix Recognizer ===== START lalr_state [0]: { [expr ::= (*) NUMBER, {EOF PLUS }] [$START ::= (*) expr EOF, {EOF }] [expr ::= (*) expr PLUS expr, {EOF PLUS }] } transition on expr to state [2] transition on NUMBER to state [1] lalr_state [1]: { [expr ::= NUMBER (*), {EOF PLUS }] } lalr_state [2]: { [$START ::= expr (*) EOF, {EOF }] [expr ::= expr (*) PLUS expr, {EOF PLUS }] } transition on EOF to state [4] transition on PLUS to state [3] lalr_state [3]: { [expr ::= (*) NUMBER, {EOF PLUS }] [expr ::= expr PLUS (*) expr, {EOF PLUS }] [expr ::= (*) expr PLUS expr, {EOF PLUS }] } transition on expr to state [5] transition on NUMBER to state [1] lalr_state [4]: { [$START ::= expr EOF (*), {EOF }] } lalr_state [5]: { [expr ::= expr PLUS expr (*), {EOF PLUS }] [expr ::= expr (*) PLUS expr, {EOF PLUS }] } transition on PLUS to state [3] c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 21

22 LALR(1) Item Automata s4 start expr EOF. EOF s0 expr. NUMBER start. expr EOF expr. expr PLUS expr EOF, PLUS EOF EOF, PLUS expr EOF s2 start expr. EOF expr expr. PLUS expr EOF EOF, PLUS s1 expr NUMBER. NUMBER EOF, PLUS NUMBER PLUS s3 expr. NUMBER expr expr PLUS. expr expr. expr PLUS expr EOF, PLUS EOF, PLUS EOF, PLUS s5 PLUS expr expr expr PLUS expr. expr expr. PLUS expr EOF, PLUS EOF, PLUS c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 22

23 Unambiguous Grammar expr ::= expr PLUS expr NUMBER ; Unambiguous Grammar? c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 23

24 Unambiguous Grammar expr ::= expr PLUS expr NUMBER ; Unambiguous Grammar? Grammar Γ 3 from lecture (p.71): expr ::= expr PLUS num num ; num ::= NUMBER; c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 24

25 JavaCUP States ===== Viable Prefix Recognizer ===== START lalr_state [0]: { [expr ::= (*) num, {EOF PLUS }] [expr ::= (*) expr PLUS num, {EOF PLUS }] [num ::= (*) NUMBER, {EOF PLUS }] [$START ::= (*) expr EOF, {EOF }] } transition on expr to state [3] transition on NUMBER to state [2] transition on num to state [1] lalr_state [1]: { [expr ::= num (*), {EOF PLUS }] } lalr_state [2]: { [num ::= NUMBER (*), {EOF PLUS }] } lalr_state [3]: { [expr ::= expr (*) PLUS num, {EOF PLUS }] [$START ::= expr (*) EOF, {EOF }] } transition on EOF to state [5] transition on PLUS to state [4] lalr_state [4]: { [expr ::= expr PLUS (*) num, {EOF PLUS }] [num ::= (*) NUMBER, {EOF PLUS }] } transition on NUMBER to state [2] transition on num to state [6] lalr_state [5]: { [$START ::= expr EOF (*), {EOF }] } lalr_state [6]: { [expr ::= expr PLUS num (*), {EOF PLUS }] } c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 25

26 LALR(1) Item Automata s1 expr num. EOF, PLUS s5 start expr EOF. EOF num s0 expr. num expr. expr PLUS expr num. NUMBER start. expr EOF EOF, PLUS EOF, PLUS EOF, PLUS EOF expr EOF s3 expr expr. PLUS expr start expr. EOF EOF, PLUS EOF s2 num NUMBER. NUMBER EOF, PLUS NUMBER PLUS s4 expr expr PLUS. expr num. NUMBER EOF, PLUS EOF, PLUS num s6 expr expr PLUS num. EOF, PLUS c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 26

27 Action and Reduce Tables -dump_tables -dump_grammar ACTION_TABLE From state #0 [term 3:SHIFT(to state 2)] From state #1 [term 0:REDUCE(with prod 2)] [term 2:REDUCE(with prod 2)] From state #2 [term 0:REDUCE(with prod 3)] [term 2:REDUCE(with prod 3)] From state #3 [term 0:SHIFT(to state 5)] [term 2:SHIFT(to state 4)] From state #4 [term 3:SHIFT(to state 2)] From state #5 [term 0:REDUCE(with prod 0)] From state #6 [term 0:REDUCE(with prod 1)] [term 2:REDUCE(with prod 1)] REDUCE_TABLE From state #0 [non term 0->state 3] [non term 1->state 1] From state #1 From state #2 From state #3 From state #4 [non term 1->state 6] From state #5 From state #6 Terminals: [0]EOF [1]error [2]PLUS [3]NUMBER Non terminals: [0]expr [1]num Productions: [0] $START ::= expr EOF [1] expr ::= expr PLUS num [2] expr ::= num [3] num ::= NUMBER c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 27

28 Example => (Scanner) => NUMBER PLUS NUMBER PLUS NUMBER EOF = NPNPNE State Stack Input Rest Action #0 #0 NPNPNE shift and goto #2 #2 #0 N#2 PNPNE reduce and goto #1 #1 #0 num#1 PNPNE reduce and goto #3 #3 #0 expr#3 PNPNE shift and goto #4 #4 #0 expr#3 P#4 NPNE shift and goto #2 #2 #0 expr#3 P#4 N#2 PNE reduce and goto #6 #6 #0 expr#3 P#4 num#6 PNE reduce and goto #3 #3 #0 expr#3 PNE shift and goto #4 #4 #0 expr#3 P#4 NE shift and goto #2 #2 #0 expr#3 P#4 N#2 E reduce and goto #6 #6 #0 expr#3 P#4 num#6 E reduce and goto #3 #3 #0 expr#3 E shift and goto #5 #5 #0 expr#3 E#5 reduce and goto #3 #3 #0 start#3 c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 28

29 Associativities = (4 + 5 ) + 6 c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 29

30 Associativities = (4 + 5 ) + 6 JavaCUP: Associativities on terminal symbols: left, right, nonassoc c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 30

31 Associativities = (4 + 5 ) + 6 JavaCUP: Associativities on terminal symbols: left, right, nonassoc Example: precedence left PLUS; expr ::= expr PLUS expr NUMBER ; c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 31

32 Precedences * 3 = 5 + (6 * 3) c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 32

33 Precedences * 3 = 5 + (6 * 3) JavaCUP: Precedence on terminal symbols: same line = equal precedences, farther down = higher precedence c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 33

34 Precedences * 3 = 5 + (6 * 3) JavaCUP: Precedence on terminal symbols: same line = equal precedences, farther down = higher precedence Example: precedence left PLUS, MINUS; precedence left MULT, DIV; // PLUS same as MINUS // higher than PLUS and MINUS expr ::= expr infixop expr NUMBER ; infixop ::= PLUS MINUS MULT DIV; c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 34

35 Rule Precedences Rules have the precedence of their last terminal symbol c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 35

36 Rule Precedences Rules have the precedence of their last terminal symbol ? c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 36

37 Rule Precedences Rules have the precedence of their last terminal symbol ? expr ::= expr MINUS expr MINUS expr NUMBER ; c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 37

38 Rule Precedences Rules have the precedence of their last terminal symbol ? expr ::= expr MINUS expr MINUS expr NUMBER ; Solution: Introduce "dummy" terminal symbol UMINUS: precedence left MINUS; precedence left UMINUS; expr ::= expr MINUS expr MINUS expr %prec UMINUS NUMBER ; Note: UMINUS is never returned by the scanner! c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 38

39 Semantic versus Syntax Arithmetic and Boolean expressions For example: a := 5 b := 4+a c := 3 = 3 d := 2+1 = 3 & c c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 39

40 Semantic versus Syntax Arithmetic and Boolean expressions For example: a := 5 b := 4+a c := 3 = 3 d := 2+1 = 3 & c Grammar: terminal ID, EQUAL, PLUS, ASSIGN, AND; non terminal ae, be, stm; precedence left AND; precedence left PLUS; stm ::= ID ASSIGN ae ID ASSIGN be; be ::= ae EQUAL ae be AND be ID; ae ::= ae PLUS ae ID; c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 40

41 Handle Semantics Later terminal ID, EQUAL, PLUS, ASSIGN, AND; non terminal e, stm; precedence left AND; precedence left PLUS; precedence left EQUAL; stm ::= ID ASSIGN e; e ::= ID e EQUAL e e AND e e PLUS e; c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 41

42 Error Recovery Problem Typically JavaCUP stops when it encounters an error Only the first error is shown Solution Local error recovery c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 42

43 Example expr ::= expr PLUS expr ID; exprlist ::= expr SEMI expr; We want to be able to skip erroneous expressions in a list. c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 43

44 Example expr ::= expr PLUS expr ID; exprlist ::= expr SEMI expr; We want to be able to skip erroneous expressions in a list. Introduce error production with special non-terminal error symbol. exprlist ::= error SEMI expr; Important: the error symbol should always be followed by a terminal synchronization symbol, SEMI in this example. c Prof. Dr. Arnd Poetzsch-Heffter Practical Exercise 44