Topic 2: Lexing and Flexing

Transcription

1 Topic 2: Lexing nd Flexing COS 320 Compiling Techniques Princeton University Spring 2016 Lennrt Beringer 1

2 2 The Compiler

3 Lexicl Anlysis Gol: rek strem of ASCII chrcters (source/input) into sequence of tokens Token: sequence of chrcters treted s unit (cf. word) Ech token hs token type (cf. clssifiction ver - noun - punctution symol): IDENTIFIER REAL SEMI foo, x, quicksort, 6.7, 3.9E-33, -4.9 ; NUM IF THEN 1, 50, -100 if then LPAREN ( RPAREN ) EQ = PLUS + Mny tokens hve ssocited semntic informtion: NUM(1), NUM(50), IDENTIFIER(foo), IDENTIFIER(x), ut typiclly not SEMI(;), LPAREN(() Definition of tokens (mostly) prt of lnguge definition White spce nd comments often discrded. Pros/Cons? 3

4 Lexicl Anlysis Gol: rek strem of ASCII chrcters (source/input) into sequence of tokens Token: sequence of chrcters treted s unit (cf. word) Ech token hs token type (cf. clssifiction ver - noun - punctution symol): IDENTIFIER REAL SEMI foo, x, quicksort, 6.7, 3.9E-33, -4.9 ; NUM IF THEN 1, 50, -100 if then LPAREN ( RPAREN ) EQ = PLUS + Mny tokens hve ssocited semntic informtion: NUM(1), NUM(50), IDENTIFIER(foo), IDENTIFIER(x), ut typiclly not SEMI(;), LPAREN(() Definition of tokens (mostly) prt of lnguge definition White spce nd comments often discrded. Pros/Cons? 4

5 Lexicl Anlysis Gol: rek strem of ASCII chrcters (source/input) into sequence of tokens Token: sequence of chrcters treted s unit (cf. word) Ech token hs token type (cf. clssifiction ver - noun - punctution symol): IDENTIFIER REAL SEMI foo, x, quicksort, 6.7, 3.9E-33, -4.9 ; NUM IF THEN 1, 50, -100 if then LPAREN ( RPAREN ) EQ = PLUS + Mny tokens hve ssocited semntic informtion: NUM(1), NUM(50), IDENTIFIER(foo), IDENTIFIER(x), ut typiclly not SEMI(;), LPAREN(() Definition of tokens (mostly) prt of lnguge definition White spce nd comments often discrded. Pros/Cons? 5

6 Lexicl Anlysis Gol: rek strem of ASCII chrcters (source/input) into sequence of tokens Token: sequence of chrcters treted s unit (cf. word) Ech token hs token type (cf. clssifiction ver - noun - punctution symol): IDENTIFIER REAL SEMI foo, x, quicksort, 6.7, 3.9E-33, -4.9 ; NUM IF THEN 1, 50, -100 if then LPAREN ( RPAREN ) EQ = PLUS + Mny tokens hve ssocited semntic informtion: NUM(1), NUM(50), IDENTIFIER(foo), IDENTIFIER(x), ut typiclly not SEMI(;), LPAREN(() Definition of tokens (mostly) prt of lnguge definition White spce nd comments often discrded. Pros/Cons? 6

7 Lexicl Anlysis Gol: rek strem of ASCII chrcters (source/input) into sequence of tokens Token: sequence of chrcters treted s unit (cf. word) Ech token hs token type (cf. clssifiction ver - noun - punctution symol): IDENTIFIER REAL SEMI foo, x, quicksort, 6.7, 3.9E-33, -4.9 ; NUM IF THEN 1, 50, -100 if then LPAREN ( RPAREN ) EQ = PLUS + Mny tokens hve ssocited semntic informtion: NUM(1), NUM(50), IDENTIFIER(foo), IDENTIFIER(x), ut typiclly not SEMI(;), LPAREN(() Definition of tokens (mostly) prt of lnguge definition White spce nd comments often discrded. Pros/Cons? 7

8 8 Lexicl Anlysis Exmple

9 Lexicl Anlysis Exmple IDENTIFIER(x) EQ LPAREN IDENTIFIER(y) PLUS NUM(4.0) RPAREN SEMI 9

10 Implementing Lexicl Anlyzer (Lexer) Option 1: write it from scrtch: Source: text file Lexer for lnguge L Strem of tokens 10

11 Implementing Lexicl Anlyzer (Lexer) Option 1: write it from scrtch: Source: text file Lexer for lnguge L Strem of tokens Option 2: et your own dog food! (use lexicl nlyzer genertor): Source: text file Token strem wrt L1 Lexer for L1 Lexer for L2 Token strem wrt L2 Input: lexing rules for L1 Lexer genertor Input: lexing rules for L2 11

12 Implementing Lexicl Anlyzer (Lexer) Option 1: write it from scrtch Source: text file Lexer for lnguge L Strem of tokens Option 2: et your own dog food! (use lexicl nlyzer genertor) Source: text file Token strem wrt L1 Lexer for L1 Lexer for L2 Token strem wrt L2 Input: lexing rules for L1 Lexer genertor Input: lexing rules for L2 12 Q: how do we descrie the tokens for L1, L2,? A: using nother lnguge, of course! Yeh, ut how do we descrie the tokens of tht lnguge???

13 Theory to the rescue: regulr expressions Some definitions An lphet is (finite) collection of symols. Exmples: ASCII, {0, 1}, {A,..Z,,.. Z}, {0,..9} 13

14 Theory to the rescue: regulr expressions Some definitions An lphet is (finite) collection of symols. Exmples: ASCII, {0, 1}, {A,..Z,,.. Z}, {0,..9} A string/word (over lphet A) is finite sequence of symols from A. 14

15 Theory to the rescue: regulr expressions Some definitions An lphet is (finite) collection of symols. Exmples: ASCII, {0, 1}, {A,..Z,,.. Z}, {0,..9} A string/word (over lphet A) is finite sequence of symols from A. A lnguge (over A) is (finite or infinite) set of strings over A. 15

16 Theory to the rescue: regulr expressions Some definitions An lphet is (finite) collection of symols. Exmples: ASCII, {0, 1}, {A,..Z,,.. Z}, {0,..9} A string/word (over lphet A) is finite sequence of symols from A. A lnguge (over A) is (finite or infinite) set of strings over A. Exmples: the ML lnguge: set of ll strings representing correct ML progrms (infinite) the lnguge of ML keywords: set of ll strings tht re ML keywords (finite) the lnguge of ML tokens: set of ll strings tht mp to ML tokens (infinite) 16

17 Theory to the rescue: regulr expressions Some definitions An lphet is (finite) collection of symols. Exmples: ASCII, {0, 1}, {A,..Z,,.. Z}, {0,..9} A string/word (over lphet A) is finite sequence of symols from A. A lnguge (over A) is (finite or infinite) set of strings over A. Exmples: the ML lnguge: set of ll strings representing correct ML progrms (infinite) the lnguge of ML keywords: set of ll strings tht re ML keywords (finite) the lnguge of ML tokens: set of ll strings tht mp to ML tokens (infinite) Q: How to descrie lnguges? A(for lexing): regulr expressions! REs re finite descriptions/representtions of (certin) finite or infinite lnguges, including the lnguge of (progrmming) lnguge s tokens (eg the lnguge of ML tokens) the lnguge descriing the lnguge of (progrmming) lnguge s tokens, the lnguge descriing 17

18 Constructing regulr expressions Bse cses Inductive cses: given RE s M nd N, 18

19 Constructing regulr expressions Bse cses the RE ε (epsilon): the (finite) lnguge contining only the empty string. for ech symol from A, the RE denotes the (finite) lnguge contining only the string. Inductive cses: given RE s M nd N, 19

20 Constructing regulr expressions Bse cses the RE ε (epsilon): the (finite) lnguge contining only the empty string. for ech symol from A, the RE denotes the (finite) lnguge contining only the string. Inductive cses: given RE s M nd N, the RE M N (lterntion, union) descries the lnguge contining the strings in M or N. Exmple: denotes the two-element lnguge {, } 20

21 Constructing regulr expressions Bse cses the RE ε (epsilon): the (finite) lnguge contining only the empty string. for ech symol from A, the RE denotes the (finite) lnguge contining only the string. Inductive cses: given RE s M nd N, the RE M N (lterntion, union) descries the lnguge contining the strings in M or N. Exmple: denotes the two-element lnguge {, } The RE MN (conctention) denotes the strings tht cn e written s the conctention mn where m in from M nd n is from N. Exmple: ( )( c) denotes the lnguge {, c,, c} 21

22 Constructing regulr expressions Bse cses the RE ε (epsilon): the (finite) lnguge contining only the empty string. for ech symol from A, the RE denotes the (finite) lnguge contining only the string. Inductive cses: given RE s M nd N, the RE M N (lterntion, union) descries the lnguge contining the strings in M or N. Exmple: denotes the two-element lnguge {, } The RE MN (conctention) denotes the strings tht cn e written s the conctention mn where m in from M nd n is from N. Exmple: ( )( c) denotes the lnguge {, c,, c} The RE M* (Kleene closure/str) denotes the (infinitely mny) strings otined y conctenting finitely mny elements from M. Exmple: ( )* denotes the lnguge {ε,,,,,,,,,,, } 22

23 Regulr Expression Exmples For lphet Σ = {,}: Strings with n even numer of s: RE = Strings tht with n odd numer of s: RE =

24 Regulr Expression Exmples For lphet Σ = {,}: Strings with n even numer of s: RE = Solutions not unique! * ( * * )* Strings tht with n odd numer of s: RE =

25 Regulr Expression Exmples For lphet Σ = {,}: Strings with n even numer of s: RE = Strings tht with n odd numer of s: RE = Solutions not unique! * ( * * )* * * ( * *)*

26 Regulr Expression Exmples For lphet Σ = {,}: Strings with n even numer of s: RE = Strings tht with n odd numer of s: RE = * ( * * )* * * ( * *)* Strings with n even numer of s OR n odd numer of s:, RE = Strings tht cn e split into string with n even numer of s, followed y string with n odd numer of s:

27 Regulr Expression Exmples For lphet Σ = {,}: Strings with n even numer of s: RE = Strings tht with n odd numer of s: RE = * ( * * )* * * ( * *)* Strings with n even numer of s OR n odd numer of s:, RE = RE RE Strings tht cn e split into string with n even numer of s, followed y string with n odd numer of s:

28 Regulr Expression Exmples For lphet Σ = {,}: Strings with n even numer of s: RE = Strings tht with n odd numer of s: RE = * ( * * )* * * ( * *)* Strings with n even numer of s OR n odd numer of s:, RE = RE RE Strings tht cn e split into string with n even numer of s, followed y string with n odd numer of s: RE, = RE RE

29 Regulr Expression Exmples For lphet Σ = {,}: Strings with n even numer of s: RE = Strings with n odd numer of s: RE = * ( * * )* * * ( * *)* Strings with n even numer of s OR n odd numer of s:, RE = RE RE Strings tht cn e split into string with n even numer of s, followed y string with n odd numer of s: RE, = RE RE Optionl Homework: Strings with n even numer of s nd n odd numer of s.

30 Implementing RE s: finite utomt Source: text file Token strem wrt L1 Automton Lexer for L1 Automton Lexer for L2 Token strem wrt L2 RE Input: lexing rules for L1 Lexer genertor RE Input: lexing rules for L2 Finite utomt (k finite stte mchines, FSM s): computtionl model of mchines with finite memory Components of n utomton over lphet A: 30 finite set S of nodes ( sttes ) set of directed edges ( trnsitions ) s t, ech linking two sttes nd leled with symol from A designted strt stte s0 from S, indicted y rrow from nowhere nonempty set of finl ( ccepting ) sttes (indicted y doule circle)

31 Finite Automt recognize lnguges Definition: the lnguge recognized y n FA is the set of (finite) strings it ccepts. Q: how does the finite utomton D c ccept string A? A: follow the trnsitions: 1. strt in the strt stte s0 nd inspect the first symol, 1 2. when in stte s nd inspecting symol, trverse one edge leled to get to the next stte. Look t the next symol. 3. After reding in ll n symols: if the current stte s is finl one, ccept. Otherwise, reject. 4. whenever there s no edge whose lel mtches the next symol: reject. 31

32 Clsses of Finite Automt Deterministic finite utomton (DFA) ll edges leving node re uniquely leled. Nondeterministic finite utomton (NFA) two (or more) edges leving node my e identiclly uniquely leled. Any choice tht leds to cceptnce is fine. edges my lso e leled with ε. So cn jump to the next stte without consuming n input symol. 32

33 Clsses of Finite Automt Deterministic finite utomton (DFA) ll edges leving node re uniquely leled. Nondeterministic finite utomton (NFA) two (or more) edges leving node my e identiclly uniquely leled. Any choice tht leds to cceptnce is fine. edges my lso e leled with ε. So cn jump to the next stte without consuming n input symol. Strtegy for otining DFA tht recognizes exctly the lnguge descried y n RE: 1. convert RE into n NFA tht recognizes the lnguge 2. trnsform the NFA into n equivlent DFA Rememer Tuesdy s quiz? 33

34 NFA Exmples Over lphet {, }: Strings with n even numer of s: Strings with n odd numer of s:

35 NFA Exmples Over lphet {, }: Strings with n even numer of s: D : Strings with n odd numer of s:

36 NFA Exmples (dhoc) Over lphet {, }: Strings with n even numer of s: D : Strings with n odd numer of s: D : Cn we systemticlly generte NFA s from RE s?

37 37 RE to NFA Rules

38 38 RE to NFA Rules

39 39 RE to NFA Rules

40 40 RE to NFA Rules

41 RE to NFA Conversion: exmples for nd conct Strings with n even numer of s OR n odd numer of s: RE RE D D ε ε Strings tht cn e split into string with n even numer of s, followed y string with n odd numer of s: RE RE D D 41

42 RE to NFA Conversion: exmples for nd conct Strings with n even numer of s OR n odd numer of s: RE RE D D ε ε Strings tht cn e split into string with n even numer of s followed y string with n odd numer of s: RE RE D D 42

43 NFA to DFA Conversion c ε 5 c ε 6 Ide: comine identiclly leled NFA trnsitions DFA sttes represent sets of equivlent NFA sttes 43

44 NFA to DFA conversion Auxiliry definitions: edge(s, ) = { t s t } set of NFA sttes rechle from NFA stte s y n step closure(s) = S U ( U edge(s, ε)) s є S set of NFA sttes rechle from ny s є S y n ε step Min clcultion: DFA-edge(D,) = closure( U edge(s,)) s є D set of NFA sttes rechle from D y mking one step nd (then) ny numer of ε steps 44

45 NFA to DFA Exmple c ε 5 c ε 6 45 closure(s) S U ( U edge(s, ε)) s є S

46 NFA to DFA Exmple c ε 5 c ε 6 Step 1: closure sets 1 : {1} 2:{2} 3:{3,5} 4:{4,6} 5:{5} 6:{6} 46 closure(s) S U ( U edge(s, ε)) s є S edge(s, ) { t s t }

47 NFA to DFA Exmple c ε 5 c ε 6 Step 1: closure sets 1 {1} 2 {2} 3 {3,5} 4 {4,6} 5 {5} 6 {6} Step 2: edge sets c 1 2, , closure(s) edge(s, ) DFA-edge(D,) 47 S U ( U edge(s, ε)) s є S { t s t } closure( U edge(s,)) s є D

48 NFA to DFA Exmple 1 1 {1} 2 {2} 3 {3,5} 4 {4,6} 5 {5} 6 {6} c ε 5 c ε 6 Step 3: DFA-sets c 1 2, , D c {1} Cl(2) + Cl(3) = {2,3,5} {} {} {2,3,5} {} Cl(2)+Cl(4) = {2,4,6} {2,4,6} {} Cl(4) = {4,6} {} {4,6} {} {} {} Cl(4) + Cl(6) = {4,6} closure(s) edge(s, ) DFA-edge(D,) 48 S U ( U edge(s, ε)) s є S { t s t } closure( U edge(s,)) s є D

49 NFA to DFA Exmple c ε 5 c ε 6 Step 4: Trnsition mtrix A B C D D c {1} Cl(2) + Cl(3) = {2,3,5} {} {} {2,3,5} {} Cl(2)+Cl(4) = {2,4,6} {2,4,6} {} Cl(4) = {4,6} {} {4,6} {} {} {} A B C c Cl(4) +Cl(6) = {4,6} D 49

50 NFA to DFA Exmple c ε 5 c ε 6 Step 5: Initil stte: closure of initil NFA stte A B C D D c {1} Cl(2) + Cl(3) = {2,3,5} {} {} {2,3,5} {} Cl(2)+Cl(4) = {2,4,6} {2,4,6} {} Cl(4) = {4,6} {} {4,6} {} {} {} A B C c Cl(4) + Cl(6) = {4,6} D 50

51 NFA to DFA Exmple c ε 5 c ε 6 Step 6: Finl stte(s): DFA sttes contining finl NFA stte 51 A B C D D c {1} Cl(2) + Cl(3) = {2,3,5} {} {} {2,3,5} {} Cl(2)+Cl(4) = {2,4,6} {2,4,6} {} Cl(4) = {4,6} {} {4,6} {} {} {} A Algorithm in pseudo-code: Appel, pge 27 B C c Cl(4) + Cl(6) = {4,6} D

52 The Longest Token Lexer should identify the longest mtching token: ifz8 should e lexed s IDENTIFIER, not s two tokens IF, IDENTIFIER Hence, the implementtion sves the most recently encountered ccepting stte of the DFA (nd the corresponding strem position) nd updtes this info when pssing through nother ccepting stte Uses the order of rules s tie-reker in cse severl tokens (of equl length) mtch 52

53 Other Useful Techniques (see exercise 2.7) (generlized NFA s: trnsitions my e leled with RE s) 53

54 Summry Motivted use of lexer genertors for prtitioning input strem into tokens Three formlisms for descriing nd implementing lexers: Regulr expressions NFA s DFA s Conversions RE -> NFA -> DFA Next lecture: prcticlities of lexing (ML-LEX) 54

55 55 The Compiler

56 56 Prcticlities of lexing: ML Lex, Lex, Flex,

57 57 ML Lex: lexer genertor for ML (similr tools for C: lex, flex)

58 Lexicl Specifiction Specifiction of lexer hs three prts: User Declrtions %% ML-LEX Definitions %% Rules User declrtions: definitions of vlues to e used in the ction frgments of rules Two vlues must e defined in the section: type lexresult: type of the vlue returned y the rule ctions fun eof(): function to e clled y the generted lexer when end of input strem is reched (eg cll prser, print done ) 58

59 Lexicl Specifiction Specifiction of lexer hs three prts: User Declrtions %% ML-LEX Definitions %% Rules ML-LEX Definitions: definitions of regulr expressions revitions: DIGITS=[0..9]+; LETTER = [-za-z]; definitions of strt sttes to permit multiple lexers to run together: %s STATE1 STATE2 STATE3; Exmple: entering comment mode, e.g. for supporting nested comments 59

60 Lexicl Specifiction Specifiction of lexer hs three prts: optionl, sttes must e defined in ML-LEX section User Declrtions %% ML-LEX Definitions %% Rules ML expression (eg construct token nd return it to reg.expr the invoking function) Rules: formt: <strt-stte-list> pttern => (ction_code); Intuitive reding: if you re in stte mode, lex strings mtching the pttern s descried y the ction. 60

61 61 Rule Ptterns

62 62 Rule Actions

63 63 Strt Sttes

64 64 Rule Mtching nd Strt Sttes

65 65 Rule Dismigution

66 66 Exmple

67 67 Exmple in Action