Fall Compiler Principles Lecture 1: Lexical Analysis. Roman Manevich Ben-Gurion University

Transcription

1 Fll Compiler Principles Lecture 1: Lexicl Anlysis Romn Mnevich Ben-Gurion University

2 Agend Understnd role of lexicl nlysis in compiler Lexicl nlysis theory Implementing professionl scnner vi scnner genertor 2

3 Jvscript exmple Cn you some identify sic units in this code? vr curroption = 0; // Choose content to disply in lower pne. function choose ( id ) { vr menu = ["out-me", "pulictions", "teching", "softwre", "ctivities"]; } for (i = 0; i < menu.length; i++) { curroption = menu[i]; vr elt = document.getelementbyid(curroption); if (curroption == id && elt.style.disply == "none") { elt.style.disply = "lock"; } else { elt.style.disply = "none"; } } 3

4 Jvscript exmple Cn you some identify sic units in this code? keyword????? vr curroption = 0; // Choose content to disply in lower pne. function choose ( id ) { vr menu = ["out-me", "pulictions", "teching", "softwre", "ctivities"]; } for (i = 0; i < menu.length; i++) { curroption = menu[i]; vr elt = document.getelementbyid(curroption); if (curroption == id && elt.style.disply == "none") { elt.style.disply = "lock"; } else { elt.style.disply = "none"; } }?? 4

5 Jvscript exmple Cn you some identify sic units in this code? keyword whitespce identifier opertor numeric literl punctution vr curroption = 0; // Choose content to disply in lower pne. function choose ( id ) { vr menu = ["out-me", "pulictions", "teching", "softwre", "ctivities"]; } comment string literl for (i = 0; i < menu.length; i++) { curroption = menu[i]; vr elt = document.getelementbyid(curroption); if (curroption == id && elt.style.disply == "none") { elt.style.disply = "lock"; } else { elt.style.disply = "none"; } } 5

6 Role of lexicl nlysis First prt of compiler front-end High-level Lnguge Lexicl Anlysis Syntx Anlysis Prsing AST Symol Tle etc. Inter. Rep. (IR) Code Genertion Executle Code (scheme) Convert strem of chrcters into strem of tokens Split text into most sic meningful strings Simplify input for syntx nlysis 6

7 From scnning to prsing progrm text 59 + (1257 * xposition) Lexicl Anlyzer Lexicl error vlid token strem num + ( num * id ) Grmmr: E id E num E E + E E E * E E ( E ) syntx error + Prser vlid Astrct Syntx Tree num * num x 7

8 Scnner output vr curroption = 0; // Choose content to disply in lower pne. function choose ( id ) { vr menu = ["out-me", "pulictions, "teching", "softwre", "ctivities"]; } for (i = 0; i < menu.length; i++) { curroption = menu[i]; vr elt = document.getelementbyid(curroption); if (curroption == id && elt.style.disply == "none") { elt.style.disply = "lock"; } else { elt.style.disply = "none"; } } Strem of Tokens LINE: ID(vlue) 1: VAR 1: ID(currOption) 1: EQ 1: INT_LITERAL(0) 1: SEMI 3: FUNCTION 3: ID(choose) 3: LP 3: ID(id) 3: EP 3: LCB... 8

9 Tokens 9

10 Wht is token? Lexeme sustring of originl text constituting n identifile unit Identifiers, Vlues, reserved words, Record type storing: Kind Vlue (when pplicle) Strt-position/end-position Any informtion tht is useful for the prser Different for different lnguges 10

11 Exmple tokens Type Identifier Exmples x, y, z, foo, r NUM 42 FLOATNUM STRING so long, nd thnks for ll the fish LPAREN ( RPAREN ) IF if 11

12 C++ exmple 1 Splitting text into tokens cn e tricky How should the code elow e split? vector<vector<int>> myvector >> opertor >, > or two tokens? 12

13 C++ exmple 2 Splitting text into tokens cn e tricky How should the code elow e split? vector<vector<int> > myvector >, > two tokens 13

14 Seprting tokens Type Exmples Comments /* ignore code */ // ignore until end of line White spces \t \n Lexemes re recognized ut get consumed rther thn trnsmitted to prser if i f i/*comment*/f 14

15 Preprocessor directives in C Type Include directives Exmples #include<foo.h> Mcros #define THE_ANSWER 42 15

16 First step of designing scnner Define ech type of lexeme Reserved words: vr, if, for, while Opertors: < = ++ Identifiers: myfunction Literls: 123 hello How cn we define lexemes of unounded length? 16

17 First step of designing scnner Define ech type of lexeme Reserved words: vr, if, for, while Opertors: < = ++ Identifiers: myfunction Literls: 123 hello How cn we define lexemes of unounded length? Regulr expressions 17

18 Agend Understnd role of lexicl nlysis in compiler Convert text to strem of tokens Lexicl nlysis theory Implementing professionl scnner vi scnner genertor 18

19 Regulr expressions 19

20 Regulr lnguges refresher Forml lnguges Alphet = finite set of letters Word = sequence of letter Lnguge = set of words Regulr lnguges defined equivlently y Regulr expressions Finite-stte utomt 20

21 Regulr expressions Empty string: Є Letter: Conctention: R 1 R 2 Union: R 1 R 2 Kleene-str: R* Shorthnd: R + stnds for R R* scope: (R) Exmple: (0* 1*) (1* 0*) Wht is this lnguge? 21

22 Exercise 1 - Question Lnguge of Jv identifiers Identifiers strt with either n underscore _ or letter Continue with either underscore, letter, or digit 22

23 Exercise 1 - Answer Lnguge of Jv identifiers Identifiers strt with either n underscore _ or letter Continue with either underscore, letter, or digit (_ z A Z)(_ z A Z 0 9)* 23

24 Exercise 1 Better nswer Lnguge of Jv identifiers Identifiers strt with either n underscore _ or letter Continue with either underscore, letter, or digit (_ z A Z)(_ z A Z 0 9)* Using shorthnd mcros First = _ z A Z Next = First 0 9 R = First Next* 24

25 Exercise 2 - Question Lnguge of rtionl numers in deciml representtion (no leding, ending zeros) Positive exmples: Negtive exmples:

26 Exercise 2 - Answer Lnguge of rtionl numers in deciml representtion (no leding, ending zeros) Digit = Digit0 = 0 Digit Num = Digit Digit0* Frc = Digit0* Digit Pos = Num.Frc 0.Frc Num.Frc PosOrNeg = (Є -)Pos R = 0 PosOrNeg 26

27 Exercise 3 - Question Equl numer of opening nd closing prenthesis: [ n ] n = [], [[]], [[[]]], 27

28 Exercise 3 - Answer Equl numer of opening nd closing prenthesis: [ n ] n = [], [[]], [[[]]], Not regulr Context-free Grmmr: S ::= [] [S] 28

29 Finite utomt 29

30 Finite utomt An utomton is defined y sttes nd trnsitions trnsition ccepting stte c strt strt stte 30

31 Automton running exmple Words re red left-to-right c c strt 31

32 Automton running exmple Words re red left-to-right c c strt 32

33 Automton running exmple Words re red left-to-right c c strt 33

34 Automton running exmple Words re red left-to-right c word ccepted c strt 34

35 Word outside of lnguge c c strt 35

36 Word outside of lnguge Missing trnsition mens non-cceptnce c c strt 36

37 Word outside of lnguge c strt 37

38 Word outside of lnguge c strt 38

39 Word outside of lnguge Finl stte is not n ccepting stte c strt 39

40 Exercise - Question Wht is the lnguge defined y the utomton elow? c strt 40

41 Exercise - Answer Wht is the lnguge defined y the utomton elow? * c Generlly: ll pths leding to ccepting sttes c strt 41

42 A little out me Joined Ben-Gurion University two yers go Reserch interests Advnced compiltion nd synthesis techniques Lnguge-supported prllelism Sttic nlysis nd verifiction 42

43 I m here for Teching you theory nd prctice of populr compiler lgorithms Hopefully mke you think out solving prolems y exmples from the compilers world Answering questions out mteril Contcting me e-mil: romnm@cs.gu.c.il Office hours: see course we-pge Announcements Forums (per ssignment) 43

44 Tenttive syllus Front End Intermedite Representtion Optimiztions Code Genertion Scnning Lowering Locl Optimiztions Register Alloction Top-down Prsing (LL) Dtflow Anlysis Instruction Selection Bottom-up Prsing (LR) Loop Optimiztions Attriute Grmmrs mid-term exm 44

45 Nondeterministic Finite utomt 45

46 Non-deterministic utomt Allow multiple trnsitions from given stte leled y sme letter c strt c 46

47 NFA run exmple c c strt c 47

48 Mintin set of sttes NFA run exmple c c strt c 48

49 NFA run exmple c c strt c 49

50 NFA run exmple Accept word if ny of the sttes in the set is ccepting c c strt c 50

51 NFA+Є utomt Є trnsitions cn fire without reding the input strt c Є 51

52 NFA+Є run exmple c strt c Є 52

53 NFA+Є run exmple Now Є trnsition cn non-deterministiclly tke plce c strt c Є 53

54 NFA+Є run exmple c strt c Є 54

55 NFA+Є run exmple c strt c Є 55

56 NFA+Є run exmple c strt c Є 56

57 Word ccepted NFA+Є run exmple c strt c Є 57

58 Reg-exp vs. utomt Regulr expressions re declrtive Offer compct wy to define regulr lnguge y humns Don t offer direct wy to check whether given word is in the lnguge Automt re opertive Define n lgorithm for deciding whether given word is in regulr lnguge Not nturl nottion for humns 58

59 From Regulr expressions to NFA 59

60 From reg. exp. to utomt Theorem: there is n lgorithm to uild n NFA+Є utomton for ny regulr expression Proof: y induction on the structure of the regulr expression For ech su-expression R we uild n utomton with exctly one strt stte nd one ccepting stte Strt stte hs no incoming trnsitions Accepting stte hs no outgoing trnsitions 60

61 From reg. exp. to utomt Theorem: there is n lgorithm to uild n NFA+Є utomton for ny regulr expression Proof: y induction on the structure of the regulr expression strt 61

62 Bse cses R = strt R = strt 62

63 Construction for R 1 R 2 R 1 strt R 2 63

64 Construction for R 1 R 2 R 1 R 2 strt 64

65 Construction for R* R strt 65

66 NFA determiniztion 66

67 From NFA+Є to DFA Construction requires O(n) sttes for reg-exp of length n Running n NFA+Є with n sttes on string of length m tkes O(m n 2 ) time Cn we reduce the n 2 fctor? 67

68 From NFA+Є to DFA Construction requires O(n) sttes for reg-exp of length n Running n NFA+Є with n sttes on string of length m tkes O(m n 2 ) time Cn we reduce the n 2 fctor? Theorem: for ny NFA+Є utomton there exists n equivlent deterministic utomton Proof: determiniztion vi suset construction Numer of sttes in the worst-cse O(2 n ) Running time O(m) 68

69 Suset construction For n NFA+Є with sttes M={s 1,,s k } Construct DFA with one stte per set of sttes of the corresponding NFA M ={ [], [s 1 ], [s 1,s 2 ], [s 2,s 3 ], [s 1,s 2,s 3 ], } Simulte trnsitions etween individul sttes for every letter NFA+Є s s 1 2 s s 4 7 DFA [s 1,s 4 ] [s 2,s 7 ] 69

70 Suset construction For n NFA+Є with sttes M={s 1,,s k } Construct DFA with one stte per set of sttes of the corresponding NFA M ={ [], [s 1 ], [s 1,s 2 ], [s 2,s 3 ], [s 1,s 2,s 3 ], } Extend mcro sttes y sttes rechle vi Є trnsitions NFA+Є DFA s Є s 1 4 [s 1,s 2 ] [s 1,s 2,s 4 ] 70

71 Recp We know how to define ny single type of lexeme We know how to convert ny regulr expression into recognizing utomton But is this enough for scnning? 71

72 Designing scnner 72

73 Scnning chllenges Regulr expressions llow us to recognize whether given text is sequence of legl lexemes Define the lnguge of ll sequences of lexemes Automt theory provides n lgorithm for checking memership of words But we re interested in splitting the text not just deciding on memership 1. How do we split the text into lexemes? 2. How do we hndle miguities lexemes mtching more thn one token? 73

74 Seprting lexemes ID = ( z) ( z)* ONE = 1 Input: 1 How do we return ID(), ONE? 74

75 Seprting lexemes ID = ( z) ( z)* ONE = 1 Input: 1 How do we return ID(), ONE? -z ID -z strt 1 ONE 75

76 Mximl munch lgorithm ID = ( z) ( z)* ONE = 1 Input: 1 How do we return ID(), ONE? Solution: find longest mtching lexeme Keep reding text until utomton fils rememer text position of lst ccepting stte Return token corresponding to lst ccepting stte Reset go ck to strt stte nd continue reding input from rememered text position 76

77 Hndling miguities ID = ( z) ( z)* IF = if Input: if Mtches oth tokens Wht should the scnner output? -z ID -z strt i NFA f IF 77

78 Hndling miguities ID = ( z) ( z)* IF = if Input: if Mtches oth tokens Wht should the scnner output? Solution: rek tie using order of definitions Output: ID(if) -z ID -z strt i NFA f IF 78

79 Hndling miguities ID = ( z) ( z)* IF = if Input: if Mtches oth tokens Wht should the scnner output? Solution: rek tie using order of definitions Output: ID(if) strt -z\i i ID ID -z\f f -z -z ID < IF DFA 79

80 Hndling miguities IF = if ID = ( z) ( z)* Input: if Mtches oth tokens Wht should the scnner output? Solution: rek tie using order of definitions Output: IF strt -z\i i ID ID -z\f f Conclusion: list keyword token definitions efore identifier definition -z -z IF < ID DFA 80

81 Filtering illegl comintions Which tokens should the scnner return for 123foo? 81

82 Filtering illegl comintions Which tokens should the scnner return for 123foo? We sometimes wnt to rule out certin token conctentions prior to prsing How cn we do tht with wht we ve seen so fr? 82

83 Filtering illegl comintions Which tokens should the scnner return for 123foo? We sometimes wnt to rule out certin token conctentions prior to prsing How cn we do tht with wht we ve seen so fr? Define error lexemes 83

84 Putting the lgorithm pieces together 84

85 Scnner construction recp List of regulr expressions (one per lexeme) NFA+Є DFA minimiztion Code implementing mximl munch with tie reking policy 85

86 Running the scnner Mintin vriles for ltpos = position from which we scn for next token lpos = Lst position of ccepting stte stte = Current stte While not reching end of text do Red until getting stuck If not end-of-file return token = input[ltpos, lpos] ltpos = lpos + 1 stte = q1 (initil) lpos = N/A 86

87 Scnning exercise You re given the following lexeme: RAT: (1-9)(0-9)*. (0-9)*(1-9) 0. (0-9)*(1-9) Construct the corresponding scnner utomton Run it on the inputs

88 Scnner run on Scn from Lst ccept pos. curr. position curr. letter curr. stte token 0 N/A 0 1 q1 0 N/A 1. q2 0 N/A 2 2 q q q N/A 4. q1 ERROR q2. 0 q4 strt q1 0. q6 q3 1-9 q

89 Scnner run on Scn from Lst ccept pos. curr. position curr. letter curr. stte token 0 N/A 0 1 q1 0 N/A 1. q2 0 N/A 2 2 q q q q N/A 4 0 q1 4 N/A 5. q6 4 N/A 6 2 q3 4 6 EOF EOF q q2. 0 q4 strt q1 0. q6 q3 1-9 q

90 Agend Understnd role of lexicl nlysis in compiler Convert text to strem of tokens Lexicl nlysis theory Theory of regulr lnguges + mximl munch + precedence Implementing professionl scnner vi scnner genertor 90

91 Implementing scnner 91

92 Implementing modern scnners Mnul construction of utomt + determiniztion + mximl munch + tie reking Very tedious Error-prone Non-incrementl Fortuntely there re tools tht utomticlly generte roust code from specifiction for most lnguges C: Lex, Flex Jv: JLex, JFlex 92

93 Using JFlex Define tokens (nd sttes) Run JFlex to generte Jv implementtion Usully MyScnner.nextToken() will e clled in loop y prser MyScnner.lex Strem of chrcters Lexicl Specifiction JFlex MyScnner.jv Tokens 93

94 Common formt for reg-exps Bsic Ptterns Mtching x The chrcter x. Any chrcter, usully except new line [xyz] Any of the chrcters x,y,z Repetition Opertors R? An R or nothing (=optionlly n R) R* Zero or more occurrences of R R+ One or more occurrences of R Composition Opertors R1R2 An R1 followed y R2 R1 R2 Either n R1 or R2 Grouping (R) R itself 94

95 Escpe chrcters Wht is the expression for one or more + symols? (+)+ won t work (\+)+ will ckslsh \ efore n opertor turns it to stndrd chrcter \*, \?, \+, Newline: \n or \r\n depending on OS T: \t 95

96 Shorthnds Use nmes for expressions letter = z A B Z letter_ = letter _ digit = id = letter_ (letter_ digit)* Use hyphen to denote rnge letter = -z A-Z digit =

97 Ctching errors Wht if input doesn t mtch ny token definition? Wnt to grcefully signl n error Trick: dd ctch-ll rule tht mtches ny chrcter nd reports n error Add fter ll other rules 97

98 Next lecture: prsing