Fall Compiler Principles Lecture 1: Lexical Analysis. Roman Manevich Ben-Gurion University of the Negev

Transcription

1 Fll Compiler Principles Lecture 1: Lexicl Anlysis Romn Mnevich Ben-Gurion University of the Negev

2 Agend Understnd role of lexicl nlysis in compiler Regulr lnguges reminder Lexicl nlysis lgorithms Scnner genertion 2

3 Jvscript exmple Cn you some identify sic units in this code? vr curroption = 0; // Choose content to disply in lower pne. function choose ( id ) { vr menu = ["out-me", "pulictions", "teching", "softwre", "ctivities"]; } for (i = 0; i < menu.length; i++) { curroption = menu[i]; vr elt = document.getelementbyid(curroption); if (curroption == id && elt.style.disply == "none") { elt.style.disply = "lock"; } else { elt.style.disply = "none"; } } 3

4 Jvscript exmple Cn you some identify sic units in this code? keyword????? vr curroption = 0; // Choose content to disply in lower pne. function choose ( id ) { vr menu = ["out-me", "pulictions", "teching", "softwre", "ctivities"]; } for (i = 0; i < menu.length; i++) { curroption = menu[i]; vr elt = document.getelementbyid(curroption); if (curroption == id && elt.style.disply == "none") { elt.style.disply = "lock"; } else { elt.style.disply = "none"; } }?? 4

5 Jvscript exmple Cn you some identify sic units in this code? keyword whitespce identifier opertor numeric literl punctution vr curroption = 0; // Choose content to disply in lower pne. function choose ( id ) { vr menu = ["out-me", "pulictions", "teching", "softwre", "ctivities"]; } comment string literl for (i = 0; i < menu.length; i++) { curroption = menu[i]; vr elt = document.getelementbyid(curroption); if (curroption == id && elt.style.disply == "none") { elt.style.disply = "lock"; } else { elt.style.disply = "none"; } } 5

6 Role of lexicl nlysis First prt of compiler front-end High-level Lnguge Lexicl Anlysis Syntx Anlysis Prsing AST Symol Tle etc. Inter. Rep. (IR) Code Genertion Executle Code (scheme) Convert strem of chrcters into strem of tokens Split text into most sic meningful strings Simplify input for syntx nlysis 6

7 From scnning to prsing progrm text 59 + (1257 * xposition) Lexicl Anlyzer Lexicl error vlid token strem num + ( num * id ) Grmmr: E id E num E E + E E E * E E ( E ) syntx error + Prser vlid Astrct Syntx Tree num * num x 7

8 Scnner output where is the white spce? where is the white spce? vr curroption = 0; // Choose content to disply in lower pne. function choose ( id ) { vr menu = ["out-me", "pulictions, "teching", "softwre", "ctivities"]; } for (i = 0; i < menu.length; i++) { curroption = menu[i]; vr elt = document.getelementbyid(curroption); if (curroption == id && elt.style.disply == "none") { elt.style.disply = "lock"; } else { elt.style.disply = "none"; } } Strem of Tokens LINE: ID(vlue) 1: VAR 1: ID(currOption) 1: EQ 1: INT_LITERAL(0) 1: SEMI 3: FUNCTION 3: ID(choose) 3: LP 3: ID(id) 3: EP 3: LCB... 8

9 Tokens 9

10 Wht is token? Lexeme sustring of originl text constituting n identifile unit Identifiers, vlues, reserved words, Record type storing: Kind Vlue (when pplicle) Strt-position/end-position Any informtion tht is useful for the prser Different for different lnguges 10

11 Exmple tokens Type Identifier Exmples x, y, z, foo, r NUM 42 FLOATNUM STRING so long, nd thnks for ll the fish LPAREN ( RPAREN ) IF if 11

12 C++ exmple 1 Splitting text into tokens cn e tricky How should the code elow e split? vector<vector<int>> myvector >> opertor >, > or two tokens? 12

13 C++ exmple 2 Splitting text into tokens cn e tricky How should the code elow e split? vector<vector<int> > myvector >, > two tokens 13

14 Seprting tokens Type Exmples Comments /* ignore code */ // ignore until end of line White spces \t \n Lexemes tht re recognized ut get consumed rther thn trnsmitted to prser if i f i/*comment*/f 14

15 Preprocessor directives in C Type Include directives Exmples #include<foo.h> Mcros #define THE_ANSWER 42 15

16 First step of designing scnner Define ech type of lexeme Reserved words: vr, if, for, while Opertors: < = ++ Identifiers: myfunction Literls: 123 hello How cn we define lexemes of unounded length? 16

17 First step of designing scnner Define ech type of lexeme Reserved words: vr, if, for, while Opertors: < = ++ Identifiers: myfunction Literls: 123 hello How cn we define lexemes of unounded length? Regulr expressions 17

18 Agend Understnd role of lexicl nlysis in compiler Convert text to strem of tokens Regulr lnguges reminder Lexicl nlysis lgorithms Scnner genertion 18

19 Regulr lnguges reminder 19

20 Bsic definitions nd fcts Forml lnguges Alphet = finite set of letters Word = sequence of letter Lnguge = set of words Regulr lnguges defined equivlently y Regulr expressions Finite-stte utomt 20

21 Regulr expressions Empty string: Є Letter: 1,, k Alphet Conctention: R 1 R 2 Union: R 1 R 2 Kleene-str: R* Shorthnd: R + stnds for R R* scope: (R) Exmple: (0* 1*) (1* 0*) Wht is this lnguge? 21

22 Exercise 1 - Question Lnguge of Jv identifiers Identifiers strt with either n underscore _ or letter Continue with either underscore, letter, or digit 22

23 Exercise 1 - Answer Lnguge of Jv identifiers Identifiers strt with either n underscore _ or letter Continue with either underscore, letter, or digit (_ z A Z)(_ z A Z 0 9)* 23

24 Exercise 1 Better nswer Lnguge of Jv identifiers Identifiers strt with either n underscore _ or letter Continue with either underscore, letter, or digit (_ z A Z)(_ z A Z 0 9)* Using shorthnd mcros First = _ z A Z Next = First 0 9 R = First Next* 24

25 Exercise 2 - Question Lnguge of rtionl numers in deciml representtion (no leding, ending zeros) Positive exmples: Negtive exmples:

26 Exercise 2 - Answer Lnguge of rtionl numers in deciml representtion (no leding, ending zeros) Digit = Digit0 = 0 Digit Num = Digit Digit0* Frc = Digit0* Digit Pos = Num.Frc 0.Frc Num.Frc PosOrNeg = (Є -)Pos R = 0 PosOrNeg 26

27 Exercise 3 - Question Equl numer of opening nd closing prenthesis: [ n ] n = [], [[]], [[[]]], 27

28 Exercise 3 - Answer Equl numer of opening nd closing prenthesis: [ n ] n = [], [[]], [[[]]], Not regulr Context-free Grmmr: S ::= [] [S] 28

29 Finite utomt 29

30 Finite utomt: known results Types of finite utomt: Deterministic (DFA) Non-deterministic (NFA) Non-deterministic + epsilon trnsitions Theorem: trnsltion of regulr expressions to NFA+epsilon (liner time) Theorem: trnsltion of NFA+epsilon to DFA Worst-cse exponentil time Theorem [Myhill-Nerode]: DFA cn e minimized 30

31 Finite utomt An utomton M = Q,,, q 0, F is defined y sttes nd trnsitions trnsition ccepting stte c strt strt stte 31

32 Exercise - Question Wht is the lnguge defined y the utomton elow? c strt 32

33 Exercise - Answer Wht is the lnguge defined y the utomton elow * c? Generlly: ll pths leding to ccepting sttes c strt 33

34 Non-deterministic utomt Allow multiple trnsitions from given stte leled y sme letter c strt c 34

35 NFA+Є utomt Є trnsitions cn fire without reding the input strt c Є 35

36 A little out me Joined Ben-Gurion University in 2012 Reserch interests Inductive progrmming nd synthesis Sttic nlysis nd verifiction Lnguge-supported prllelism 36

37 I m here for Teching you theory nd prctice of populr compiler lgorithms Hopefully mke you think out solving prolems y exmples from the compilers world Answering questions out mteril Contcting me e-mil: romnm@cs.gu.c.il Office hours: see course we-pge Announcements Forums (per ssignment) 37

38 Tenttive syllus Front End Intermedite Representtion Optimiztions Code Genertion Scnning Opertionl Semntics Dtflow Anlysis Register Alloction Top-down Prsing (LL) Lowering Loop Optimiztions Energy Optimiztion Bottom-up Prsing (LR) Instruction Selection mid-term exm 38

39 Reg-exp vs. utomt Regulr expressions re declrtive Offer compct wy to define regulr lnguge y humns Don t offer direct wy to check whether given word is in the lnguge Automt re opertive Define n lgorithm for deciding whether given word is in regulr lnguge Not nturl nottion for humns A high-level lnguge A mchine lnguge 39

40 From Regulr expressions to utomt 40

41 From reg. exp. to NFA+Є utomt Theorem: there is n lgorithm to uild n NFA+Є utomton for ny regulr expression Proof: y induction on the structure of the regulr expression strt 41

42 Inductive constructions R = strt R 1 R 2 R 1 R = strt strt R 2 R * strt R R 1 R 2 strt R 1 R 2 42

43 Running time of NFA+Є Construction requires O(k) sttes for reg-exp of length k Running n NFA+Є with k sttes on string of length n tkes O(n k 2 ) time Cn we reduce the k 2 fctor? Ech stte in configurtion of O(k) sttes my hve O(k) outgoing edges, so processing n input letter my tke O(k 2 ) time 43

44 From NFA+Є to DFA Construction requires O(k) sttes for reg-exp of length k Running n NFA+Є with k sttes on string of length n tkes O(n k 2 ) time Cn we reduce the k 2 fctor? Theorem: for ny NFA+Є utomton there exists n equivlent deterministic utomton Proof: determiniztion vi suset construction Numer of sttes in the worst-cse O(2 k ) Running time O(n) 44

45 Recp We know how to define ny single type of lexeme We know how to convert ny regulr expression into recognizing utomton But how do we use this for scnning? 45

46 The forml scnning prolem 46

47 Wht is scnner Lexicl Specifiction: List of regulr expressions (one per lexeme) vr curroption = 0; // Choose content function choose ( id ) {... R 1 R k Scnner Strem of Tokens LINE: ID(vlue) 1: VAR 1: ID(currOption) 1: EQ 1: INT_LITERAL(0) 1: SEMI... 47

48 Scnning prolem Input: Lexicl specifiction: R 1,, R k (regulr expressions, one per lexeme) input: string of n chrcters Output: sequence of tokens R 1 (lex 1 ) R n (lex n ) such tht The lexemes prtition the input lex 1 lex n = input R 1 R n mtch the lexeme type from the specifiction 48

49 Exmple 1: prtitioning ID = ( z) ( z)* ONE = 1 Input: 1 Wht should the output e? 1. ID() ID() ID() ONE 2. ID() ID() ONE 3. ID() ID() ONE 4. ID() ONE First mtch semntics Mximl munch semntics 49

50 Mximl munch semntics ID = ( z) ( z)* ONE = 1 Input: 1 How do we return ID() ONE? Solution: find longest mtching lexeme Automton my enter nd leve ccepting stte mny times efore longest mtch is found Intuition: some tokens, such s identifiers re prefix-closed 50

51 Exmple 2: hndling miguities ID = ( z) ( z)* IF = if Input: if Mtches oth tokens Wht should the scnner output e? -z strt q0 -z\i i ID -z\f -z DFA ID f ID, IF 51

52 Solution: precedence semntics ID = ( z) ( z)* IF = if Input: if Mtches oth tokens Wht should the scnner output e? Brek tie using order of definitions Output: ID(if) strt q0 -z\i i ID ID -z\f f -z -z ID, IF DFA 52

53 Solution: precedence semntics IF = if ID = ( z) ( z)* Input: if Mtches oth tokens Wht should the scnner output e? Brek tie using order of definitions Output: IF strt q0 -z\i i ID ID -z\f f Conclusion: list keyword token definitions efore identifier definition -z -z IF, ID DFA 53

54 Putting together n lgorithm 54

55 Overll lgorithm structure List of regulr expressions (one per lexeme) R 1 R k High-level intermedite representtion minimiztion Medium-level intermedite representtion Crucil: Assign semntics NFA+Є R 1 R k DFA for R 1 R k How do we implement mximl munch? Scnner implementtion (efficient dt structures) 55

56 A First mtch lgorithm 56

57 Suggestions? First mtch lgorithm Wht is the complexity? 57

58 A Mximl munch lgorithm 58

59 Mximl munch scnning lgorithm Input: input: string of n chrcters M: DFA for union of tokens Output: positions of in input tht re the finl chrcters of ech token Dt: Stck of stte, index of sttes nd their positions encountered since lst ccepting stte i: index of next chrcter in input q: current stte or Bottom (no stte) 59

60 Mximl munch pseudo-code Reset DFA to look for next token Used to indicte n error sitution (no token is found) 60

61 Mximl munch run exmple Assume R 1 = input = q0 q1 R 2 = * q3 q2 q i stck q0 1 B,1 Output = 61

62 Mximl munch run exmple Assume R 1 = input = q0 q1 R 2 = * q3 q2 q i stck q0 1 B,1 q1 2 B,1 q0,1 Output = 62

63 Mximl munch run exmple Assume R 1 = input = q0 q1 R 2 = * q3 q2 q i stck q0 1 B,1 q1 2 B,1 q0,1 q3 3 q1,2 Output = 63

64 Mximl munch run exmple Assume R 1 = input = q0 q1 R 2 = * q3 q2 q i stck q0 1 B,1 q1 2 B,1 q0,1 q3 3 q1,2 q3 4 q1,2 q3,3 Output = 64

65 Mximl munch run exmple Assume R 1 = input = q0 q1 R 2 = * q3 q2 q i stck q0 1 B,1 q1 2 B,1 q0,1 q3 3 q1,2 q3 4 q1,2 q3,3 q3 3 q1,2 Output = 65

66 Mximl munch run exmple Assume R 1 = input = q0 q1 R 2 = * q3 q2 q i stck q0 1 B,1 q1 2 B,1 q0,1 q3 3 q1,2 q3 4 q1,2 q3,3 q3 3 q1,2 q1 2 Output = 66

67 Mximl munch run exmple Assume R 1 = input = q0 q1 R 2 = * q3 q2 q i stck q0 1 B,1 q1 2 B,1 q0,1 q3 3 q1,2 q3 4 q1,2 q3,3 q3 3 q1,2 q1 2 Output = 1 67

68 Mximl munch run exmple Assume R 1 = input = q0 q1 R 2 = * q3 q2 q i stck q0 2 B,2 Output = 1 68

69 Mximl munch run exmple Assume R 1 = input = q0 q1 R 2 = * q3 q2 q i stck q0 2 B,2 q1 3 B,2 q0,2 Output = 1 69

70 Mximl munch run exmple Assume R 1 = input = q0 q1 R 2 = * q3 q2 q i stck q0 2 B,2 q1 3 B,2 q3 4 q1,3 Output = 1 70

71 Mximl munch run exmple Assume R 1 = input = q0 q1 R 2 = * q3 q2 q i stck q0 2 B,2 q1 3 B,2 q3 4 q1,3 Output = 1 71

72 Mximl munch run exmple Assume R 1 = input = q0 q1 R 2 = * q3 q2 q i stck q0 2 B,2 q1 3 B,2 q3 4 q1,3 q1 3 Output = 1 72

73 Mximl munch run exmple Assume R 1 = input = q0 q1 R 2 = * q3 q2 q i stck q0 2 B,2 q1 3 B,2 q3 4 q1,3 q1 3 Output =

74 Mximl munch run exmple Assume R 1 = input = q0 q1 R 2 = * q3 q2 q i stck q0 3 B,3 Output =

75 Mximl munch run exmple Assume R 1 = input = q0 q1 R 2 = * q3 q2 q i stck q0 3 B,3 q1 4 B,3 q0,3 Output =

76 Mximl munch run exmple Assume R 1 = input = q0 q1 R 2 = * q3 q2 q i stck q0 3 B,3 q1 4 B,3 q0,3 Output =

77 Mximl munch run exmple Assume R 1 = input = q0 q1 R 2 = * q3 q2 q i stck q0 3 B,3 q1 4 B,3 q0,3 Output =

78 Mximl munch run exmple Assume R 1 = input = q0 q1 R 2 = * q3 q2 q i stck q0 3 B,3 q1 4 B,3 q0,3 Output =

79 Complexity of mximl munch Wht is the complexity of tokenizing text of n chrcters y mtching longest tokens? 79

80 Complexity of mximl munch Wht is the complexity of tokenizing text of n chrcters y mtching longest tokens? Assume the following token clsses R 1 = R 2 = * For input= n it is O(n 2 ) Cn we improve the worst-cse complexity? q q q q n n 80

81 Improved scnning lgorithm Ide: use work done on leftover stck to improve future decisions Rememer for ech index which sttes hve filed cnnot e extended to token Mximl-Munch Tokeniztion in Liner Time Tom Reps [TOPLAS 1998] 81

82 Improved lgorithm pseudo-code Wht is the running time? How mny times cn this test fil for given index? 82

83 Agend Understnd role of lexicl nlysis in compiler Convert text to strem of tokens Regulr lnguges reminder Lexicl nlysis lgorithms Precedence + First mtch Precedence + Mximl munch Scnner genertion 83

84 Implementing scnner 84

85 Implementing modern scnners Mnul construction of utomt + determiniztion + mximl munch + tie reking Very tedious Error-prone Non-incrementl Fortuntely there re tools tht utomticlly generte roust code from specifiction for most lnguges C: Lex, Flex Jv: JLex, JFlex 85

86 Using JFlex Define tokens (nd sttes) Run JFlex to generte Jv implementtion Usully MyScnner.nextToken() will e clled in loop y prser MyScnner.lex Strem of chrcters Lexicl Specifiction JFlex MyScnner.jv Tokens 86

87 Filtering illegl comintions Which tokens should the scnner return for 123foo? 87

88 Filtering illegl comintions Which tokens should the scnner return for 123foo? We sometimes wnt to rule out certin token conctentions prior to prsing How cn we do tht with wht we ve seen so fr? 88

89 Filtering illegl comintions Which tokens should the scnner return for 123foo? We sometimes wnt to rule out certin token conctentions prior to prsing How cn we do tht with wht we ve seen so fr? Define error lexemes 89

90 Ctching errors Wht if input doesn t mtch ny token definition? Wnt to grcefully signl n error Trick: dd ctch-ll rule tht mtches ny chrcter nd reports n error Add fter ll other rules 90

91 Next lecture: prsing