Compilation

Size: px

Start display at page:

Download "Compilation"

Mervyn Preston
5 years ago
Views:

1 Compiltion Lecture 2: Lexicl Anlysis Nom Rinetzky 1

2 2

3 Lexicl Anlysis Modern Compiler Design: Chpter 2.1 3

4 Conceptul Structure of Compiler Compiler Source text txt Frontend Semntic Representtion Bckend Executle code exe Lexicl Anlysis Syntx Anlysis Prsing Semntic Anlysis Intermedite Representtion (IR) Code Genertion 4

5 Conceptul Structure of Compiler Compiler Source text txt Frontend Semntic Representtion Bckend Executle code exe Lexicl Anlysis Syntx Anlysis Prsing Semntic Anlysis Intermedite Representtion (IR) Code Genertion words sentences 5

6 Wht does Lexicl Anlysis do? Lnguge: fully prenthesized expressions Expr Num LP Expr Op Expr RP Num Dig Dig Num Dig LP ( RP ) Op + * ( ( ) * 19 ) 6

7 Wht does Lexicl Anlysis do? Lnguge: fully prenthesized expressions Context free lnguge Regulr lnguges Expr Num LP Expr Op Expr RP Num Dig Dig Num Dig LP ( RP ) Op + * ( ( ) * 19 ) 7

8 Wht does Lexicl Anlysis do? Lnguge: fully prenthesized expressions Context free lnguge Regulr lnguges Expr Num LP Expr Op Expr RP Num Dig Dig Num Dig LP ( RP ) Op + * ( ( ) * 19 ) 8

9 Wht does Lexicl Anlysis do? Lnguge: fully prenthesized expressions Context free lnguge Regulr lnguges Expr Num LP Expr Op Expr RP Num Dig Dig Num Dig LP ( RP ) Op + * ( ( ) * 19 ) 9

10 Wht does Lexicl Anlysis do? Lnguge: fully prenthesized expressions Context free lnguge Regulr lnguges Expr Num LP Expr Op Expr RP Num Dig Dig Num Dig LP ( RP ) Op + * ( ( ) * 19 ) LP LP Num Op Num RP Op Num RP 10

11 Wht does Lexicl Anlysis do? Lnguge: fully prenthesized expressions Context free lnguge Regulr lnguges Expr Num LP Expr Op Expr RP Num Dig Dig Num Dig LP ( RP ) Op + * Vlue Kind ( ( ) * 19 ) LP LP Num Op Num RP Op Num RP 11

12 Wht does Lexicl Anlysis do? Context free lnguge Regulr lnguges Vlue Lnguge: fully prenthesized expressions Expr Num LP Expr Op Expr RP Num Dig Dig Num Dig LP ( RP ) Op + * Token Token ( ( ) * 19 ) Kind LP LP Num Op Num RP Op Num RP 12

13 Wht does Lexicl Anlysis do? Prtitions the input into strem of tokens Numers Identifiers Keywords Punctution word in the source lnguge meningful to the syntcticl nlysis Usully represented s (kind, vlue) pirs (Num, 23) (Op, * ) 13

14 From scnning to prsing progrm text ((23 + 7) * x) Lexicl Anlyzer token strem ( LP ( LP 23 Num + OP 7 Num ) RP * OP? Id ) RP Grmmr: Expr... Id Id... z syntx error Prser Op(*) vlid Astrct Syntx Tree Op(+) Id(?) Num(23) Num(7) 14

15 Why Lexicl Anlysis? Well, not strictly necessry, ut Regulr lnguges Í Context-Free lnguges Simplifies the syntx nlysis (prsing) And lnguge definition Modulrity Reusility Efficiency 15

16 Lecture gols Understnd role & plce of lexicl nlysis Lexicl nlysis theory Using progrm generting tools 16

17 Lecture Outline ürole & plce of lexicl nlysis Wht is token? Regulr lnguges Lexicl nlysis Error hndling Automtic cretion of lexicl nlyzers 17

18 Wht is token? (Intuitively) A word in the source lnguge Anything tht should pper in the input to syntx nlysis Identifiers Vlues Lnguge keywords Usully, represented s pir of (kind, vlue) 18

19 Exmple Tokens Type Exmples ID foo, n_14, lst NUM 73, 00, 517, 082 REAL 66.1,.5, 5.5e-10 IF if COMMA, NOTEQ!= LPAREN ( RPAREN ) 19

20 Exmple Non Tokens Type Exmples comment /* ignored */ preprocessor directive #include <foo.h> #define NUMS 5.6 mcro NUMS whitespce \t, \n, \, 20

21 Some sic terminology Lexeme (k symol) - series of letters seprted from the rest of the progrm ccording to convention (spce, semi-column, comm, etc.) Pttern - rule specifying set of strings. Exmple: n identifier is string tht strts with letter nd continues with letters nd digits (Usully) regulr expression Token - pir of (pttern, ttriutes) 21

22 Exmple void mtch0(chr *s) /* find zero */ { } if (!strncmp(s, 0.0, 3)) return 0.0 ; VOID ID(mtch0) LPAREN CHAR DEREF ID(s) RPAREN LBRACE IF LPAREN NOT ID(strncmp) LPAREN ID(s) COMMA STRING(0.0) COMMA NUM(3) RPAREN RPAREN RETURN REAL(0.0) SEMI RBRACE EOF 22

23 Exmple Non Tokens Type Exmples comment /* ignored */ preprocessor directive #include <foo.h> #define NUMS 5.6 mcro NUMS whitespce \t, \n, \, Lexemes tht re recognized ut get consumed rther thn trnsmitted to prser if i/*comment*/f 23

24 Lecture Outline ürole & plce of lexicl nlysis üwht is token? Regulr lnguges Lexicl nlysis Error hndling Automtic cretion of lexicl nlyzers 24

25 How cn we define tokens? Keywords esy! if, then, else, for, while, Identifiers? Numericl Vlues? Strings? Chrcterize unounded sets of vlues using ounded description? 25

26 Regulr lnguges Forml lnguges Σ = finite set of letters Word = sequence of letter Lnguge = set of words Regulr lnguges defined equivlently y Regulr expressions Finite-stte utomt 26

Common formt for reg-exps Bsic Ptterns Mtching x The chrcter x. Any chrcter, usully except new line [xyz] Any of the chrcters x,y,z ^x Any chrcter except x Repetition Opertors R?

27 Common formt for reg-exps Bsic Ptterns Mtching x The chrcter x. Any chrcter, usully except new line [xyz] Any of the chrcters x,y,z ^x Any chrcter except x Repetition Opertors R? An R or nothing (=optionlly n R) R* Zero or more occurrences of R R+ One or more occurrences of R Composition Opertors R1R2 An R1 followed y R2 R1 R2 Either n R1 or R2 Grouping (R) R itself 27

28 Exmples * cd? = ( )* = ( )* = 28

29 Escpe chrcters Wht is the expression for one or more + symols? (+)+ won t work (\+)+ will ckslsh \ efore n opertor turns it to stndrd chrcter \*, \?, \+, \(\+\*, (\(\+\*)+, ckslsh doule quotes surrounds text (+*, (+* + 29

30 Shorthnds Use nmes for expressions letter = z A B Z letter_ = letter _ digit = id = letter_ (letter_ digit)* Use hyphen to denote rnge letter = -z A-Z digit =

31 Exmples if = if then = then relop = < > <= >= = <> digit = 0-9 digits = digit+ 31

32 Exmple A numer is numer = ( ) + ( e \. ( ) + ) ( e E ( ) + ) Using shorthnds it cn e written s numer = digits (e \.digits (e E (e + -) digits ) ) 32

33 Exercise 1 - Question Lnguge of rtionl numers in deciml representtion (no leding, ending zeros) Not 007 Not

34 Exercise 1 - Answer Lnguge of rtionl numers in deciml representtion (no leding, ending zeros) Digit = Digit0 = 0 Digit Num = Digit Digit0* Frc = Digit0* Digit Pos = Num \.Frc 0\.Frc Num\.Frc PosOrNeg = (Є -)Pos R = 0 PosOrNeg 34

35 Exercise 2 - Question Equl numer of opening nd closing prenthesis: [ n ] n = [], [[]], [[[]]], 35

36 Exercise 2 - Answer Equl numer of opening nd closing prenthesis: [ n ] n = [], [[]], [[[]]], Not regulr Context-free Grmmr: S ::= [] [S] 36

37 Chllenge: Amiguity If = if Id = Letter (Letter Digit)* if is vlid identifiers wht should it e? iffy is lso vlid identifier Solution Longest mtching token Brek ties using order of definitions Keywords should pper efore identifiers 37

38 Creting lexicl nlyzer Given list of token definitions (pttern nme, regex), write progrm such tht Input: String to e nlyzed Output: List of tokens How do we uild n nlyzer? 38

39 Building Scnner Tke I Input: String Output: Sequence of tokens 39

40 Building Scnner Tke I Token nexttoken() { chr c ; loop: c = getchr(); switch (c){ cse ` `: goto loop ; cse `;`: return SemiColumn; cse `+`: c = getchr() ; switch (c) { cse `+': return PlusPlus ; cse '= return PlusEqul; defult: ungetc(c); return Plus; }; cse `<`: cse `w`: } 40

41 There must e etter wy! 41

42 A etter wy Automticlly generte scnner Define tokens using regulr expressions Use finite-stte utomt for detection 42

43 Reg-exp vs. utomt Regulr expressions re declrtive Good for humns Not executle Automt re opertive Define n lgorithm for deciding whether given word is in regulr lnguge Not nturl nottion for humns 43

44 Overview Define tokens using regulr expressions Construct nondeterministic finite-stte utomton (NFA) from regulr expression Determinize the NFA into deterministic finite-stte utomton (DFA) DFA cn e directly used to identify tokens 44

45 Automt theory: ird s-eye view 45

46 Deterministic Automt (DFA) M = (S, Q, d, q 0, F) S - lphet Q finite set of stte q 0 Î Q initil stte F Í Q finl sttes δ : Q S à Q - trnsition function For word w, M rech some stte x M ccepts w if x Î F 46

47 DFA in pictures An utomton is defined y sttes nd trnsitions trnsition, c ccepting stte strt,c,,c,,c strt stte 47

48 Accepting Words Words re red left-to-right c c strt Missing trnsition = non-cceptnce Stuck stte 48

49 Accepting Words Words re red left-to-right c c strt 49

50 Accepting Words Words re red left-to-right c c strt 50

51 Accepting Words Words re red left-to-right c c strt 51

52 Rejecting Words Words re red left-to-right c c strt 52

53 Rejecting Words Missing trnsition mens non-cceptnce c c strt 53

54 Non-deterministic Automt (NFA) M = (S, Q, d, q 0, F) S - lphet Q finite set of stte q 0 Î Q initil stte F Í Q finl sttes δ : Q (S È {e}) 2 Q - trnsition function DFA: δ : Q S à Q For word w, M cn rech numer of sttes X M ccepts w if X M {} Possile: X = {} Possile e-trnsitions 54

55 NFA Allow multiple trnsitions from given stte leled y sme letter c strt c 55

56 Accepting words c c strt c 56

57 Accepting words Mintin set of sttes c c strt c 57

58 Accepting words c c strt c 58

59 Accepting words Accept word if reched n ccepting stte c c strt c 59

60 NFA+Є utomt Є trnsitions cn fire without reding the input strt c Є 60

61 NFA+Є run exmple c strt c Є 61

62 NFA+Є run exmple Now Є trnsition cn non-deterministiclly tke plce c strt c Є 62

63 NFA+Є run exmple c strt c Є 63

64 NFA+Є run exmple c strt c Є 64

65 NFA+Є run exmple Є trnsitions cn fire without reding the input c strt c Є 65

66 Word ccepted NFA+Є run exmple c strt c Є 66

67 From regulr expressions to NFA Step 1: ssign expression nmes nd otin pure regulr expressions R 1 R m Step 2: construct n NFA M i for ech regulr expression R i Step 3: comine ll M i into single NFA Amiguity resolution: prefer longest ccepting word 67

68 From reg. exp. to utomt Theorem: there is n lgorithm to uild n NFA+Є utomton for ny regulr expression Proof: y induction on the structure of the regulr expression strt 68

69 Bsic constructs R = e R = R = f 69

70 Composition R = R1 R2 e M1 e e M2 e R = R1R2 e M1 e M2 e 70

71 Repetition R = R1* e e M1 e e 71

72 72

73 Nïve pproch Try ech utomton seprtely Given word w: Try M 1 (w) Try M 2 (w) Try M n (w) Requires resetting fter every ttempt 73

74 Actully, we comine utomt comines *+ e e e e *

75 Corresponding DFA * *+ 0# ε# ε# ε# ε# # 1# 2# 3# # 4# # 7# 8# 9# # # 10# # # # *+# # 5# 11# # # # 6# # 12# 13# # 75# *+ 75

76 Scnning with DFA Run until stuck Rememer lst ccepting stte Go ck to ccepting stte Return token 76

77 Amiguity resolution # 1# 2# # ε# ε# 3# # 4# # 5# # 6# # 0# ε# # # 7# 8# # *+# ε# 9# # 10# # 11# # 12# # 13# # 75# Longest word Tie-reker sed on order of rules when words hve sme length 77

78 Exmples # 1# 2# # * *+ 0# ε# ε# ε# ε# 3# # 4# # 7# 8# 9# # # 10# # # *+# # 5# 11# # # # 6# # 12# 13# # 75# *+ : gets stuck fter in stte 12, cks up to stte (5 8 11) pttern is *+, token is 78 Tokens: <*+, > <,><,>

79 Exmples # 1# 2# # * *+ 0# ε# ε# ε# ε# 3# # 4# # 7# 8# 9# # # 10# # # *+# # 5# 11# # # # 6# # 12# 13# # 75# *+ : stops fter second in (6 8), token is ecuse it comes first in spec Tokens: <, > <,> 79

80 Summry of Construction Descrie tokens s regulr expressions Decide ttriutes (vlues) to sve for ech token Regulr expressions turned into DFA Also, records which ttriutes (vlues) to keep Lexicl nlyzer simultes the run of n utomt with the given trnsition tle on ny input string 80

81 A Few Remrks Turning n NFA to DFA is expensive, ut Exponentil in the worst cse In prctice, works fine The construction is done once per-lnguge At Compiler construction time Not t compiltion time 81

82 Implementtion 82

83 Implementtion y Exmple if xy, i, zs98 3,32, , comm\n \n, \t, if { return IF; } [-z][-z0-9]* { return ID; } [0-9]+ { return NUM; } [0-9]. [0-9]+ [0-9]*. [0-9]+ { return REAL; } (\-\-[-z]*\n) ( \n \t) { ; }. { error(); } ID 2 3 IF ID error REAL 0 1 NUM REAL 12 w.s. error error w.s. 83

84 ID 2 3 IF ID error REAL 0 1 NUM REAL 12 w.s. error error w.s. int edges[][256]= { /*, 0, 1, 2, 3,..., -, e, f, g, h, i, j,... */ /* stte 0 */ {0,, 0, 0,, 0, 0, 0, 0, 0,..., 0, 0, 0, 0, 0, 0}, /* stte 1 */ {13,, 7, 7, 7, 7,, 9, 4, 4, 4, 4, 2, 4,, 13, 13}, /* stte 2 */ {0,, 4, 4, 4, 4,..., 0, 4, 3, 4, 4, 4, 4,, 0, 0}, /* stte 3 */ {0,, 4, 4, 4, 4,, 0, 4, 4, 4, 4, 4, 4,, 0, 0}, /* stte 4 */ {0,, 4, 4, 4, 4,, 0, 4, 4, 4, 4, 4, 4,, 0, 0}, /* stte 5 */ {0,, 6, 6, 6, 6,, 0, 0, 0, 0, 0, 0, 0,, 0, 0}, /* stte 6 */ {0,, 6, 6, 6, 6,, 0, 0, 0, 0, 0, 0, 0,..., 0, 0}, /* stte 7 */ /* stte */... /* stte 13 */ {0,, 0, 0, 0, 0,, 0, 0, 0, 0, 0, 0, 0,, 0, 0} }; 84

85 Pseudo Code for Scnner chr* input = ; Token nexttoken() { lstfinl = 0; currentstte = 1 ; inputpositionatlstfinl = input; currentposition = input; while (not(isded(currentstte))) { nextstte = edges[currentstte][*currentposition]; if (isfinl(nextstte)) { lstfinl = nextstte ; inputpositionatlstfinl = currentposition; } currentstte = nextstte; dvnce currentposition; } input = inputpositionatlstfinl + 1; return ction[lstfinl]; } 85

86 Exmple ID 2 3 IF ID error REAL 0 1 NUM REAL 12 w.s. error error w.s. Input: if --not--com 2 lnks 86

87 ID 2 3 IF ID error REAL finl stte input 0 1 NUM REAL 0 1 if --not--com 12 w.s. error error w.s. 2 2 if --not--com 3 3 if --not--com return IF 3 0 if --not--com 87

88 finl stte input not--com not--com not--com found whitespce not--com 88

89 ID 2 3 IF ID error REAL finl stte input not--com 12 w.s. error NUM REAL error w.s not--com not--com not--com error not--com not--com 89

90 ID 2 3 IF ID error REAL finl stte input 0 1 NUM REAL 0 1 -not--com 12 w.s. error error w.s not--com 9 0 -not--com error 9 0 -not--com 9 0 -not--com 90

91 Concluding remrks Efficient scnner Minimiztion Error hndling Automtic cretion of lexicl nlyzers 91

92 Efficient Scnners Efficient stte representtion Input uffering Using switch nd gotos insted of tles 92

93 Minimiztion Crete non-deterministic utomton (NDFA) from every regulr expression Merge ll the utomt using epsilon moves (like the construction) Construct deterministic finite utomton (DFA) Stte priority Minimize the utomton seprte ccepting sttes y token kinds 93

96 Exmple IF if { return IF; } [-z][-z0-9]* ID { return ID; } ID IF [0-9]+ { return NUM; } ID ID ID ID IF error IF NUM ID NUM NUM NUM NUM error error NUM error Modern compiler implementtion in ML, Andrew Appel, (c)1998, Figures 2.7,2.8 96

98 Error Hndling Mny errors cnnot e identified t this stge Exmple: fi (==f(x)). Should fi e if? Or is it routine nme? We will discover this lter in the nlysis At this point, we just crete n identifier token Sometimes the lexeme does not mtch ny pttern Esiest: eliminte letters until the eginning of legitimte lexeme Alterntives: eliminte/dd/replce one letter, replce order of two djcent letters, etc. Gol: llow the compiltion to continue Prolem: errors tht spred ll over 98

99 Automticlly generted scnners Use of Progrm-Generting Tools Specifiction è Prt of compiler Compiler-Compiler regulr expressions JFlex input progrm scnner Strem of tokens 99

100 Use of Progrm-Generting Tools Input: regulr expressions nd ctions Action = Jv code Output: scnner progrm tht Produces strem of tokens Invoke ctions when pttern is mtched regulr expressions JFlex input progrm scnner Strem of tokens 100

101 Line Counting Exmple Crete progrm tht counts the numer of lines in given input text file 101

102 Creting Scnner using Flex int num_lines = 0; %% \n ++num_lines;. ; %% min() { yylex(); printf( "# of lines = %d\n", num_lines); } 102

103 Creting Scnner using Flex int num_lines = 0; %% \n ++num_lines;. ; %% min() { yylex(); printf( "# of lines = %d\n", num_lines); } initil \n ^\n newline other 103

104 %% %% JFLex Spec File User code: Copied directly to Jv file JFlex directives: mcros, stte nmes Lexicl nlysis rules: Optionl stte, regulr expression, ction How to rek input to tokens Action when token mtched Possile source of jvc errors down the rod DIGIT= [0-9] LETTER= [-za-z] YYINITIAL {LETTER} ({LETTER} {DIGIT})* 104

105 Creting Scnner using JFlex import jv_cup.runtime.*; %% %cup %{ privte int linecounter = 0; %} %eofvl{ System.out.println("line numer=" + linecounter); return new Symol(sym.EOF); %eofvl} NEWLINE=\n %% {NEWLINE} { linecounter++; } [^{NEWLINE}] { } 105

106 Ctching errors Wht if input doesn t mtch ny token definition? Trick: Add ctch-ll rule tht mtches ny chrcter nd reports n error Add fter ll other rules 106

107 A JFlex specifiction of C Scnner import jv_cup.runtime.*; %% %cup %{ privte int linecounter = 0; %} Letter= [-za-z_] Digit= [0-9] %% \t { } \n { linecounter++; } ; { return new Symol(sym.SemiColumn);} ++ { return new Symol(sym.PlusPlus); } += { return new Symol(sym.PlusEq); } + { return new Symol(sym.Plus); } while { return new Symol(sym.While); } {Letter}({Letter} {Digit})* { return new Symol(sym.Id, yytext() ); } <= { return new Symol(sym.LessOrEqul); } < { return new Symol(sym.LessThn); } 107

108 Missing Creting lexicl nlysis y hnd Tle compression Symol Tles Nested Comments Hndling Mcros 108

109 Lexicl Anlysis: Wht Input: progrm text (file) Output: sequence of tokens 109

110 Lexicl Anlysis: How Define tokens using regulr expressions Construct nondeterministic finite-stte utomton (NFA) from regulr expression Determinize the NFA into deterministic finite-stte utomton (DFA) DFA cn e directly used to identify tokens 110

111 Lexicl Anlysis: Why Red input file Identify lnguge keywords nd stndrd identifiers Hndle include files nd mcros Count line numers Remove whitespces Report illegl symols [Produce symol tle] 111

112 The Rel Antomy of Compiler Source text txt Process text input chrcters Lexicl Syntx tokens Anlysis Anlysis Annotted AST AST Sem. Anlysis Intermedite code genertion IR Intermedite code optimiztion IR Code genertion Symolic Instructions Trget code optimiztion SI Mchine code genertion MI Write executle output Executle code exe 112

CS321 Languages and Compiler Design I. Winter 2012 Lecture 5

CS321 Languages and Compiler Design I. Winter 2012 Lecture 5 CS321 Lnguges nd Compiler Design I Winter 2012 Lecture 5 1 FINITE AUTOMATA A non-deterministic finite utomton (NFA) consists of: An input lphet Σ, e.g. Σ =,. A set of sttes S, e.g. S = {1, 3, 5, 7, 11,