Compila(on Lecture 2: Lexical Analysis Syntax Analysis (1): CFLs, CFGs, PDAs. Noam Rinetzky

Transcription

1 Compila(on Lecture 2: Lexical Analysis Syntax Analysis (1): CFLs, CFGs, PDAs Noam Rinetzky 1

2 2

3 3

4 What is a Compiler? source language target language Source text txt Executale code exe Compiler 4

5 Compiler vs. Interpreter 5

6 Toy compiler 6

7 The Real Anatomy of a Compiler 7

8 The Real Anatomy of a Compiler Source text txt Process text input characters Lexical Syntax tokens Analysis Analysis Annotated AST AST Sem. Analysis Intermediate code generation IR Intermediate code optimization IR Code generation Symolic Instructions Target code optimization SI Machine code generation MI Write executale output Executale code exe 8

9 Lexical Analysis Lexical analyzers are also known as scanners 9

10 Lexical Analysis: from Text to Tokens txt x = * 4*a*c Token Stream <ID, x > <EQ> <ID, > <MULT> <ID, > <MINUS> <INT,4> <MULT> <ID, a > <MULT> <ID, c > Lexical Analysis Syntax Analysis Sem. Analysis Inter. Rep. Code Gen. 10

11 What does Lexical Analysis do? Scan the input Par((ons the text into stream of tokens Numers Iden(fiers Keywords Punctua(on word in the source language meaningful to the syntac(cal analysis Tokens usually represented as (kind, value) Defined using regular expressions * 11

12 What does Lexical Analysis do? Language: fully parenthesized expressions Context free language Regular languages Expr Num LP Expr Op Expr RP Num Dig Dig Num Dig LP ( RP ) Op + * ( ( ) * 19 ) 12

13 What does Lexical Analysis do? Language: fully parenthesized expressions Context free language Regular languages Expr Num LP Expr Op Expr RP Num Dig Dig Num Dig LP ( RP ) Op + * ( ( ) * 19 ) LP LP Num Op Num RP Op Num RP 13

14 From scanning to parsing program text ((23 + 7) * x) Lexical Analyzer token stream ( LP ( LP 23 Num + OP 7 Num ) RP * OP? Id ) RP Grammar: Expr... Id Id a... z syntax error Parser Op(*) valid Astract Syntax Tree Op(+) Id(?) Num(23) Num(7) 14

15 Some asic terminology Lexeme (aka symol) - a series of lecers separated from the rest of the program according to a conven(on (space, semi- column, comma, etc.) Pacern - a rule specifying a set of strings. Example: an iden(fier is a string that starts with a lecer and con(nues with lecers and digits (Usually) a regular expression Token - a pair of (pacern, acriutes) 15

16 Example void match0(char *s) /* find a zero */ { if (!strncmp(s, 0.0, 3)) return 0. ; } VOID ID(match0) LPAREN CHAR DEREF ID(s) RPAREN LBRACE IF LPAREN NOT ID(strncmp) LPAREN ID(s) COMMA STRING(0.0) COMMA NUM(3) RPAREN RPAREN RETURN REAL(0.0) SEMI RBRACE EOF 16

17 Regular languages Formal languages Σ = finite set of lecers Word = sequence of lecer Language = set of words Regular languages defined equivalently y Regular expressions Finite- state automata 17

18 Example: Integers w/o Leading Zeros Digit = Digit0 = 0 Digit Pos = Digit Digit0* Integer = 0 Pos - Pos 18

19 Challenge: Amiguity If = if Id = Lecer (Lecer Digit)* if is a valid iden(fiers what should it e? iffy is also a valid iden(fier Solu(on Longest matching token Break (es using order of defini(ons Keywords should appear efore iden(fiers 19

20 Building a Scanner Take I Input: String Output: Sequence of tokens 20

21 Building a Scanner Take I Token nexttoken() { char c ; loop: c = getchar(); switch (c){ case ` `: goto loop ; case `;`: return SemiColumn; case `+`: c = getchar() ; switch (c) { case `+': return PlusPlus ; case '= return PlusEqual; default: ungetc(c); return Plus; }; case `<`: case `w`: } 21

22 There must e a ecer way! 22

23 A ecer way AutomaGcally generate a scanner Define tokens using regular expressions Use finite- state automata for detec(on 23

24 Reg- exp vs. automata Regular expressions are declara(ve Good for humans Not executale Automata are opera(ve Define an algorithm for deciding whether a given word is in a regular language Not a natural nota(on for humans 24

25 Overview Define tokens using regular expressions Construct a nondeterminis(c finite- state automaton (NFA) from regular expression Determinize the NFA into a determinis(c finite- state automaton (DFA) DFA can e directly used to iden(fy tokens 25

26 Automata theory: a ird s- eye view 26

27 Determinis(c Automata (DFA) M = (Σ, Q, δ, q 0, F) Σ - alphaet Q finite set of state q 0 Q ini(al state F Q final states δ : Q Σ à Q - transi(on func(on For a word w, M reach some state x M accepts w if x F 27

28 DFA in pictures An automaton is defined y states and transi(ons transition a, accepting state a c start start state,c a,,c a,,c 28

29 Accep(ng Words Words are read leq- to- right a c a c start Missing transi(on = non- acceptance Stuck state 29

30 Accep(ng Words Words are read leq- to- right a c a c start 30

31 Accep(ng Words Words are read leq- to- right a c a c start 31

32 Accep(ng Words Words are read leq- to- right a c a c start 32

33 Rejec(ng Words Words are read leq- to- right c a c start 33

34 Rejec(ng Words Missing transi(on means non- acceptance c a c start 34

35 Non- determinis(c Automata (NFA) M = (Σ, Q, δ, q 0, F) Σ - alphaet Q finite set of state q 0 Q ini(al state F Q final states δ : Q (Σ {ε}) 2 Q - transi(on func(on DFA: δ : Q Σ à Q For a word w, M can reach a numer of states X M accepts w if X M {} Possile: X = {} Possile ε- transi(ons 35

36 NFA Allow mul(ple transi(ons from given state laeled y same lecer a c start a c 36

37 Accep(ng words a c a c start a c 37

38 Accep(ng words Maintain set of states a c a c start a c 38

39 Accep(ng words a c a c start a c 39

40 Accep(ng words Accept word if reached an accep(ng state a c a c start a c 40

41 NFA+Є automata Є transi(ons can fire without reading the input start a c Є 41

42 NFA+Є run example a c start a c Є 42

43 NFA+Є run example Now Є transi(on can non- determinis(cally take place a c start a c Є 43

44 NFA+Є run example a c start a c Є 44

45 NFA+Є run example a c start a c Є 45

46 NFA+Є run example Є transi(ons can fire without reading the input a c start a c Є 46

47 Word accepted NFA+Є run example a c start a c Є 47

48 From regular expressions to NFA Step 1: assign expression names and otain pure regular expressions R 1 R m Step 2: construct an NFA M i for each regular expression R i Step 3: comine all M i into a single NFA Amiguity resolu8on: prefer longest accep8ng word 48

49 From reg. exp. to automata Theorem: there is an algorithm to uild an NFA +Є automaton for any regular expression Proof: y induc8on on the structure of the regular expression start 49

50 Basic constructs R = ε R = a a R = φ 50

51 Composi(on R = R1 R2 ε M1 ε ε M2 ε R = R1R2 ε M1 ε M2 ε 51

52 Repe((on R = R1* ε ε M1 ε ε 52

53 53

54 Naïve approach Try each automaton separately Given a word w: Try M 1 (w) Try M 2 (w) Try M n (w) Requires rese{ng aqer every acempt 54

55 Actually, we comine automata comines a a a*+ aa 0 ε ε ε ε 1 a 2 3 a a a a a*+ a a aa 55

56 Corresponding DFA a*+ 8 7 a 6 8 a a*+ 0# ε# ε# ε# ε# a# 1# 2# 3# a# 4# # 7# 8# 9# a# a# 10# a# # # a*+# # 5# 11# # a# a# 6# # 12# 13# aa# 75# a a a aa a a*+ 56

57 Scanning with DFA Run un(l stuck Rememer last accepgng state Go ack to accep(ng state Return token 57

58 Amiguity resolu(on a# 1# 2# a# ε# ε# 3# a# 4# # 5# # 6# a# 0# ε# a# # 7# 8# # a*+# ε# 9# a# 10# # 11# a# 12# # 13# aa# 75# Longest word Tie- reaker ased on order of rules when words have same length 58

59 Examples a# 1# 2# a# a*+ 8 7 a 6 8 a a*+ 0# ε# ε# ε# ε# 3# a# 4# # 7# 8# 9# a# a# 10# # # a*+# # 5# 11# # a# a# 6# # 12# 13# aa# 75# a a a aa a a*+ aaa: gets stuck aqer aa in state 12, acks up to state (5 8 11) pacern is a*+, token is a Tokens: <a*+, a> <a,a><a,a> 59

60 Examples a# 1# 2# a# a*+ 8 7 a 6 8 a a*+ 0# ε# ε# ε# ε# 3# a# 4# # 7# 8# 9# a# a# 10# # # a*+# # 5# 11# # a# a# 6# # 12# 13# aa# 75# a a a aa a a*+ aa: stops aqer second in (6 8), token is a ecause it comes first in spec Tokens: <a, a> <a,a> 60

61 Summary of Construc(on Descrie tokens as regular expressions Decide acriutes (values) to save for each token Regular expressions turned into a DFA Also, records which acriutes (values) to keep Lexical analyzer simulates the run of an automata with the given transi(on tale on any input string 61

62 A Few Remarks Turning an NFA to a DFA is expensive, ut Exponen(al in the worst case In prac(ce, works fine The construc(on is done once per- language At Compiler construc(on (me Not at compila(on (me 62

63 Implementa(on 63

64 if xy, i, zs98 3,32, , comm\n \n, \t, Implementa(on y Example if { return IF; } [a- z][a- z0-9]* { return ID; } [0-9]+ { return NUM; } [0-9]. [0-9]+ [0-9]*. [0-9]+ { return REAL; } (\- \- [a- z]*\n) ( \n \t) { ; }. { error(); } ID 2 3 IF ID error REAL 0 1 NUM REAL 12 w.s. error error w.s. 64

65 ID 2 3 IF ID error REAL 0 1 NUM REAL 12 w.s. error error w.s. int edges[][256]= { /*, 0, 1, 2, 3,..., -, e, f, g, h, i, j,... */ /* state 0 */ {0,, 0, 0,, 0, 0, 0, 0, 0,..., 0, 0, 0, 0, 0, 0}, /* state 1 */ {13,, 7, 7, 7, 7,, 9, 4, 4, 4, 4, 2, 4,, 13, 13}, /* state 2 */ {0,, 4, 4, 4, 4,..., 0, 4, 3, 4, 4, 4, 4,, 0, 0}, /* state 3 */ {0,, 4, 4, 4, 4,, 0, 4, 4, 4, 4, 4, 4,, 0, 0}, /* state 4 */ {0,, 4, 4, 4, 4,, 0, 4, 4, 4, 4, 4, 4,, 0, 0}, /* state 5 */ {0,, 6, 6, 6, 6,, 0, 0, 0, 0, 0, 0, 0,, 0, 0}, /* state 6 */ {0,, 6, 6, 6, 6,, 0, 0, 0, 0, 0, 0, 0,..., 0, 0}, /* state 7 */ /* state */... /* state 13 */ {0,, 0, 0, 0, 0,, 0, 0, 0, 0, 0, 0, 0,, 0, 0} }; 65

66 Pseudo Code for Scanner char* input = ; Token nexttoken() { lastfinal = 0; currentstate = 1 ; inputpositionatlastfinal = input; currentposition = input; while (not(isdead(currentstate))) { nextstate = edges[currentstate][*currentposition]; if (isfinal(nextstate)) { lastfinal = nextstate ; inputpositionatlastfinal = currentposition; } currentstate = nextstate; advance currentposition; } input = inputpositionatlastfinal + 1; return action[lastfinal]; } 66

67 Example ID 2 3 IF ID error REAL 0 1 NUM REAL 12 w.s. error error w.s. Input: if - - not- a- com 2 lanks 67

68 ID 2 3 IF ID error REAL final state input 0 1 NUM REAL 0 1 if - - not- a- com 12 w.s. error error w.s. 2 2 if - - not- a- com 3 3 if - - not- a- com return IF 3 0 if - - not- a- com 68

69 final state input not-a-com not-a-com not-a-com found whitespace not-a-com 69

70 ID 2 3 IF ID error REAL final state input not- a- com 12 w.s. error NUM REAL error w.s not- a- com not- a- com not- a- com error not- a- com not- a- com 70

71 ID 2 3 IF ID error REAL final state input 0 1 NUM REAL not- a- com 12 w.s. error error w.s not- a- com not- a- com error not- a- com not- a- com 71

72 Concluding remarks Efficient scanner Minimiza(on Error handling Automa(c crea(on of lexical analyzers 72

73 Efficient Scanners Efficient state representa(on Input uffering Using switch and gotos instead of tales 73

74 Minimiza(on Create a non- determinis(c automaton (NDFA) from every regular expression Merge all the automata using epsilon moves (like the construc(on) Construct a determinis(c finite automaton (DFA) State priority Minimize the automaton separate accep(ng states y token kinds 74

75 Example if { return IF; } [a- z][a- z0-9]* { return ID; } [0-9]+ { return NUM; } IF ID error NUM Modern compiler implementa(on in ML, Andrew Appel, (c)1998, Figures 2.7,2.8 75

76 Example if { return IF; } [a- z][a- z0-9]* { return ID; } [0-9]+ { return NUM; } IF ID ID IF ID ID error NUM NUM NUM error Modern compiler implementa(on in ML, Andrew Appel, (c)1998, Figures 2.7,2.8 76

77 Example ID IF if { return IF; } [a- z][a- z0-9]* ID { return ID; } ID IF [0-9]+ { return NUM; } ID ID ID IF error IF NUM ID NUM NUM NUM NUM error error NUM error Modern compiler implementa(on in ML, Andrew Appel, (c)1998, Figures 2.7,2.8 77

78 Example ID IF if { return IF; } [a- z][a- z0-9]* { return ID; } ID [0-9]+ { return NUM; } ID IF ID NUM NUM error NUM error Modern compiler implementa(on in ML, Andrew Appel, (c)1998, Figures 2.7,2.8 78

79 Error Handling Many errors cannot e iden(fied at this stage Example: fi (a==f(x)). Should fi e if? Or is it a rou(ne name? We will discover this later in the analysis At this point, we just create an iden(fier token Some(mes the lexeme does not match any pacern Easiest: eliminate lecers un(l the eginning of a legi(mate lexeme Alterna(ves: eliminate/add/replace one lecer, replace order of two adjacent lecers, etc. Goal: allow the compila(on to con(nue Prolem: errors that spread all over 79

80 Automa(cally generated scanners Use of Program- Genera(ng Tools Specifica(on è Part of compiler Compiler- Compiler regular expressions JFlex input program scanner Stream of tokens 80

81 Use of Program- Genera(ng Tools Input: regular expressions and ac(ons Ac(on = Java code Output: a scanner program that Produces a stream of tokens Invoke ac(ons when pacern is matched regular expressions JFlex input program scanner Stream of tokens 81

82 Line Coun(ng Example Create a program that counts the numer of lines in a given input text file 82

83 Crea(ng a Scanner using Flex int num_lines = 0; %% \n ++num_lines;. ; %% main() { yylex(); printf( "# of lines = %d\n", num_lines); } 83

84 Crea(ng a Scanner using Flex int num_lines = 0; %% \n ++num_lines;. ; %% main() { yylex(); printf( "# of lines = %d\n", num_lines); } ini(al \n ^\n newline other 84

85 %% %% JFLex Spec File User code: Copied directly to Java file JFlex direc(ves: macros, state names Lexical analysis rules: Op(onal state, regular expression, ac(on How to reak input to tokens Ac(on when token matched Possile source of javac errors down the road DIGIT= [0-9] LETTER= [a- za- Z] YYINITIAL {LETTER} ({LETTER} {DIGIT})* 85

86 Crea(ng a Scanner using JFlex import java_cup.runtime.*; %% %cup %{ private int linecounter = 0; %} %eofval{ System.out.println("line numer=" + linecounter); return new Symol(sym.EOF); %eofval} NEWLINE=\n %% {NEWLINE} { linecounter++; } [^{NEWLINE}] { } 86

87 Catching errors What if input doesn t match any token defini(on? Trick: Add a catch- all rule that matches any character and reports an error Add aqer all other rules 87

88 A JFlex specifica(on of C Scanner import java_cup.runtime.*; %% %cup %{ private int linecounter = 0; %} Letter= [a- za- Z_] Digit= [0-9] %% \t { } \n { linecounter++; } ; { return new Symol(sym.SemiColumn);} ++ { return new Symol(sym.PlusPlus); } += { return new Symol(sym.PlusEq); } + { return new Symol(sym.Plus); } while { return new Symol(sym.While); } {Letter}({Letter} {Digit})* { return new Symol(sym.Id, yytext() ); } <= { return new Symol(sym.LessOrEqual); } < { return new Symol(sym.LessThan); } 88

89 Missing Crea(ng a lexical analysis y hand Tale compression Symol Tales Nested Comments Handling Macros 89

90 Lexical Analysis: What Input: program text (file) Output: sequence of tokens 90

91 Lexical Analysis: How Define tokens using regular expressions Construct a nondeterminis(c finite- state automaton (NFA) from regular expression Determinize the NFA into a determinis(c finite- state automaton (DFA) DFA can e directly used to iden(fy tokens 91

92 Lexical Analysis: Why Read input file Iden(fy language keywords and standard iden(fiers Handle include files and macros Count line numers Remove whitespaces Report illegal symols [Produce symol tale] 92

93 Syntax Analysis (1) Context Free Languages Context Free Grammars Pushdown Automata 93

94 The Real Anatomy of a Compiler Source text txt Process text input characters Lexical Syntax tokens Analysis Analysis Annotated AST AST Sem. Analysis Intermediate code generation IR Intermediate code optimization IR Code generation Symolic Instructions Target code optimization SI Machine code generation MI Write executale output Executale code exe 94

95 Frontend: Scanning & Parsing program text ((23 + 7) * x) Lexical Analyzer token stream ( LP ( LP 23 Num + OP 7 Num ) RP * OP x Id ) RP Grammar: E... Id Id a... z syntax error Parser Op(*) valid Astract Syntax Tree Op(+) Id() Num(23) Num(7) 95

96 From scanning to parsing program text ((23 + 7) * x) Lexical Analyzer token stream ( LP ( LP 23 Num + OP 7 Num ) RP * OP x Id ) RP Grammar: E... Id Id a... z syntax error Parser Op(*) valid Astract Syntax Tree Op(+) Id() Num(23) Num(7) 96

97 Parsing Construct a structured representa(on of the input Challenges How do you descrie the programming language? How do you check validity of an input? Is a sequence of tokens a valid program in the language? How do you construct the structured representa(on? Where do you report an error? 97

98 Some founda(ons 98

99 Context free languages (CFLs) L 01 = { 0 n 1 n n > 0 } L polyndrom = {pp p Σ*, p =reverse(p)} L polyndrom# = {p#p p Σ*, p =reverse(p), # Σ} 99

100 Context free grammars (CFG) V non terminals (syntac(c variales) T terminals (tokens) P deriva(on rules Each rule of the form V t (T 4 V)* S start symol G = (V,T,P,S) 100

101 What can CFGs do? Recognize CFLs S t 0T1 T t 0T1 ℇ 101

102 Recognizing CFLs Context Free Grammars (CFG) ~ language- defining power Nondeterminis(c push down automata (PDA) 102

103 Pushdown Automata (PDA) Nondeterminis(c PDAs define all CFLs Determinis(c PDAs model parsers. Most programming languages have a determinis(c PDA Efficient implementa(on 103

104 Intui(on: PDA An ε- NFA with the addi(onal power to manipulate one stack Top X Y IF $ stack control (ε- NFA) 104

105 Intui(on: PDA Think of an ε- NFA with the addi(onal power that it can manipulate a stack PDA moves are determined y: The current state (of its ε- NFA ) The current input symol (or ε) The current symol on top of its stack 105

106 Intui(on: PDA Current input if (oops) then stat:= lah else aort Top X Y IF $ stack control (ε- NFA) 106

107 Intui(on: PDA Moves: Change state Replace the top symol y 0 n symols 0 symols = pop ( reduce ) 0 < symols = sequence of pushes ( shiq ) Nondeterminis(c choice of next move 107

108 PDA Formalism PDA = (Q, Σ, Γ, δ, q 0, $, F): Q: finite set of states Σ: Input symols alphaet Γ: stack symols alphaet δ: transi(on func(on q 0 : start state $: start symol F: set of final states 108

109 The Transi(on Func(on δ(q, a, X) = { (p 1, σ 1 ),,(p n, σ n )} Input: triplet A state q Q An input symol a Σ or ε A stack symol X Γ Output: set of 0 k ac(ons of the form (p, σ) A state p Q σ a sequence X 1 X n Γ* of stack symols 109

110 Ac(ons of the PDA Say (p, σ) δ(q, a, X) If the PDA is in state q and X is the top symol and a is at the front of the input Then it can Change the state to p. Remove a from the front of the input (ut a may e ε). Replace X on the top of the stack y σ. 110

111 Example: Determinis(c PDA Design a PDA to accept {0 n 1 n n > 1}. The states: q = We have not seen 1 so far start state p = we have seen at least one 1 and no 0s since f = final state; accept. 111

112 Example: Stack Symols $ = start symol. Also marks the ocom of the stack, Indicates when we have counted the same numer of 1 s as 0 s. X = counter used to count the numer of 0s we saw 112

113 Example: Transi(ons δ(q, 0, $) = {(q, X$)}. δ(q, 0, X) = {(q, XX)}. These two rules cause one X to e pushed onto the stack for each 0 read from the input. δ(q, 1, X) = {(p, ε)}. When we see a 1, go to state p and pop one X. δ(p, 1, X) = {(p, ε)}. Pop one X per 1. δ(p, ε, $) = {(f, $)}. Accept at ocom. 113

114 Ac(ons of the Example PDA q $ 114

115 Ac(ons of the Example PDA q X $ 115

116 Ac(ons of the Example PDA q X X $ 116

117 Ac(ons of the Example PDA q X X X $ 117

118 Ac(ons of the Example PDA p X X $ 118

119 Ac(ons of the Example PDA p X $ 119

120 Ac(ons of the Example PDA p $ 120

121 Ac(ons of the Example PDA f $ 121

122 Example: Non Determinis(c PDA A PDA that accepts palindromes L {pp Σ* p =reverse(p)} 122

123 123