CMPT 379 Compilers. Lexical Analysis

Transcription

1 CMPT 379 Compilers Anoop Srkr 9//7 Lexicl Anlysis Also clled scnning, tke input progrm string nd convert into tokens Exmple: T_DOUBLE ( doule ) T_IDENT ( f ) T_OP ( = ) doule f = sqrt(); T_IDENT ( sqrt ) T_LPAREN ( ( ) T_OP ( ) T_INTCONSTANT ( ) T_RPAREN ( ) ) T_SEP ( ; ) 9//7

2 Token Attriutes Some tokens hve ttriutes T_IDENT sqrt T_INTCONSTANT Other tokens do not T_WHILE Token=T_IDENT, Lexeme= sqrt, Pttern Source code loction for error reports 9//7 3 Lexicl errors Wht if user omits the spce in doulef? No lexicl error, single token T_IDENT( doulef ) is produced insted of sequence T_DOUBLE, T_IDENT( f )! Typiclly few lexicl error types E.g., illegl chrs, opened string constnts or comments tht re not closed 9//7 4

3 Adhoc Scnners 9//7 5 Implementing Lexers: Loop nd switch scnners Ad hoc scnners Big nested switch/cse sttements Lots of getc()/ungetc() clls Buffering; Sentinels for pushcks; strems Cn e errorprone, use only if Your lnguge s lexicl structure is very simple The tools do not provide wht you need for your token definitions Chnging or dding keyword is prolemtic Hve look t n ctul implementtion of n dhoc scnner 9//7 6 3

4 Implementing Lexers: Loop nd switch scnners Another prolem: how to show tht the implementtion ctully cptures ll tokens specified y the lnguge definition? How cn we show correctness Key ide: seprte the definition of tokens from the implementtion Prolem: we need to reson out ptterns nd how they cn e used to define tokens (recognize strings). 9//7 7 Specifiction of Ptterns using Regulr Expressions 9//7 8 4

5 Forml Lnguges: Recp Symols:,, c Alphet : finite set of symols Σ = {, } String: sequence of symols Empty string: Set of ll strings: Σ* Define: Σ = Σ {} (Forml) Lnguge: set of strings { n n : n > } cf. The Lirry of Bel, Jorge Luis Borges 9//7 9 Regulr Lnguges The set of regulr lnguges: ech element is regulr lnguge Ech regulr lnguge is n exmple of (forml) lnguge, i.e. set of strings e.g. { m n : m, n re +ve integers } 9//7 5

6 Regulr Lnguges Defining the set of ll regulr lnguges: The empty set nd {} for ll in Σ re regulr lnguges If L nd L nd L re regulr lnguges, then: (union) (Kleene closure) re lso regulr lnguges There re no other regulr lnguges (conctention) 9//7 Forml Grmmrs A forml grmmr is concise description of forml lnguge A forml grmmr uses specilized syntx For exmple, regulr expression is concise description of regulr lnguge ( )* : is the set of ll strings over the lphet {, } which end in We will use regulr expressions (regexps) in order to define tokens in our compiler, e.g. lexemes for string token cn e defined s \" Σ* \" 9//7 6

7 Regulr Expressions: Definition Every symol of Σ { } is regulr expression E.g. if Σ = {,} then, re regexps If r nd r re regulr expressions, then the core opertors to comine two regexps re Conctention: r r, e.g. or Alterntion: r r, e.g. Repetition: r *, e.g. * or * No other core opertors re defined But other opertors cn e defined using the sic opertors (s in lex regulr expressions) e.g. + = * 9//7 3 Lex regulr expressions Expression Mtches Exmple Using core opertors c nonopertor chrcter c \c chrcter c literlly \* "s" string s literlly "**". ny chrcter ut newline.* ^ eginning of line ^c used for mtching $ end of line c$ used for mtching [s] ny one of chrcters in string s [c] ( c) [^s] ny one chrcter not in string s [^] ( c) where S = {,,c} r* zero or more strings mtching r * r+ one or more strings mtching r + * r? zero or one r? ( ) r{m,n} etween m nd n occurences of r {,3} ( ) r r n r followed y n r r r n r or n r (r) sme s r ( ) r /r r when followed y n r c/3 used for mtching 9//7 4 7

8 Regulr Expressions: Definition Note tht opertors pply recursively nd these pplictions cn e miguous E.g. is c equl to ( )c or (() )c? Avoid such cses of miguity provide explicit rguments for ech regexp opertor For convenience, for exmples on this pge, let us use the symol to denote the opertor for conctention Remove miguity with n explicit regexp tree ( )c is written s ( ( ( ))c) or in postfix: c (() )c is written s ( ( ( ))c) or in postfix: c 9//7 5 Regulr Expressions: Definition Remove miguity with n explicit regexp tree ( )c is written s ( ( ( ))c) or in postfix: c c (() )c is written s ( ( ( ))c) or in postfix: c Does the order of conctention mtter? c 9//7 6 8

9 Regulr Expressions: Exmples Alphet {, } All strings tht represent inry numers divisile y 4 (ut ccept ) (( )*) All strings tht do not contin s sustring ** 9//7 7 Equivlence of Regexps (R S) T == R (S T) == R S T (RS)T == R(ST) (R S) == (S R) R*R* == (R*)* == R* == RR* R** == R* (R S)T = RT ST R(S T) == RS RT (R S)* == (R*S*)* == (R*S)*R* == (R* S*)* RR* == R*R (RS)*R == R(SR)* R = R R = R 9//7 8 9

10 Equivlence of Regexps ()* ()* ()()* ()* ()()* ()()* ()()* ()* (RS)*R == R(SR)* RS == (RS) R* == RR* R == R R R* == RR* 9//7 9 Regulr Expressions To descrie ll lexemes tht form token s pttern ( )+ Need decision procedure: to which token does given sequence of chrcters elong (if ny)? Finite Stte Automt Cn e deterministic (DFA) or nondeterministic (NFA) 9//7

11 Implementing Regulr Expressions with Finitestte Automt 9//7 Deterministic Finite Stte Automt: DFA A set of sttes S One strt stte q, zero or more finl sttes F An lphet of input symols A trnsition function: δ: S x Σ S Exmple: δ(, ) = 9//7

12 DFA: Exmple Wht regulr expression does this utomton ccept? A B C A: strt stte C: finl stte Answer: ( )* 9//7 3 DFA simultion A B C Input string: Strt stte: A. δ(a,) = B DFA simultion tkes t most n steps for input of length n to return ccept or reject. δ(b,) = C 3. δ(c,) = A 4. δ(a,) = B 5. δ(b,) = C no more input nd C is finl stte: ccept 9//7 4

13 FA: Pscl Exmple letter letter digit < > 8 9 { digit digit } } 3 = > = B A C * 4 +, 5 : = 7 6 Any ut } =. ; ( ) D F H E G 9//7 5 Building Lexicl Anlyzer Token Pttern Pttern Regulr Expression Regulr Expression NFA NFA DFA DFAs or NFAs for ll the tokens Lexicl Anlyzer Two sic rules to del with multiple mtching: greedy mtch + regexp ordering Note tht greedy mens longest leftmost mtch 9//7 6 3

14 %{ #include <stdio.h> #define NUMBER 56 #define IDENTIFIER 57 %} /* regexp definitions */ num [9]+ %% {num} { return NUMBER; } [zaz9]+ { return IDENTIFIER; } %% Lexicl Anlysis using Lex int min () { int token; while ((token = yylex())) { switch (token) { cse NUMBER: printf("number: %s, LENGTH:%d\n", yytext, yyleng); rek; cse IDENTIFIER: printf("identifier: %s, LENGTH:%d\n", yytext, yyleng); rek; defult: printf("error: %s not recognized\n", yytext); } } } 9//7 7 NFAs NFA: like DFA, except A trnsition cn led to more thn one stte, tht is, δ: S x Σ S One stte is chosen nondeterministiclly Trnsitions cn e leled with, mening sttes cn e reched without reding ny input, tht is, δ: S x Σ { } S 9//7 8 4

15 Thompson s construction Converts regexps to NFA Build NFA recursively from regexp tree Build NFA with lefttoright prse of postfix string using stck n n5 n n4 n7 c n3 n6 Input = c red, push n = nf() red, push n = nf() red, push n3 = nf() red, n3=pop(); n=pop(); push n4 = nf(or, n, n3) red, n4 = pop(); n = pop(); push n5 = nf(ct, n, n4) red c, push n6 = nf(c) red, n6 = pop(); n5 = pop(); push n7 = nf(ct, n5, n6) 9//7 9 Thompson s construction Converts regexps to NFA Six simple rules Empty lnguge Symols Empty String Alterntion (r or r ) Conctention (r followed y r ) Repetition (r *) Used y Ken Thompson for ptternsed serch in text editor QED (968) 9//7 3 5

16 Thompson Rule For the empty lnguge φ (optionlly include sinkhole stte) Σ Σ 9//7 3 Thompson Rule For ech symol x of the lphet, there is NFA tht ccepts it (include sinkhole stte) x Σ\x Σ Σ 9//7 3 6

17 Thompson Rule There is n NFA tht ccepts only Σ Σ 9//7 33 Thompson Rule 3 Given two NFAs for r, r, there is NFA tht ccepts r r r r 9//7 34 7

18 Thompson Rule 3 Given two NFAs for r, r, there is NFA tht ccepts r r r r 9//7 35 Thompson Rule 4 Given two NFAs for r, r, there is NFA tht ccepts r r r r 9//7 36 8

19 Thompson Rule 4 Given two NFAs for r, r, there is NFA tht ccepts r r r r 9//7 37 Thompson Rule 5 Given NFA for r, there is n NFA tht ccepts r * r 9//7 38 9

20 Thompson Rule 5 Given NFA for r, there is n NFA tht ccepts r * r 9//7 39 Exmple Set of ll inry strings tht re divisile y four (include in this set) Defined y the regexp: (( )*) Apply Thompson s Rules to crete n NFA 9//7 4

21 Bsic Blocks nd (this version does not report errors: no sinkholes) 9//7 4 9//7 4

22 ( )* 9//7 43 ( )* 9//7 44

23 (( )*) 9//7 45 Simulting NFAs Similr to DFA simultion But hve to del with trnsitions nd multiple trnsitions on the sme input Insted of one stte, we hve to consider sets of sttes Simulting NFAs is prolem tht is closely linked to converting given NFA to DFA 9//7 46 3

24 NFA to DFA Conversion Suset construction Ide: susets of set of ll NFA sttes re equivlent nd ecome one DFA stte Algorithm simultes movement through NFA Key prolem: how to tret trnsitions? 9//7 47 Closure Strt stte: q closure(s): S is set of sttes 9//7 48 4

25 Closure (T: set of sttes) push ll sttes in T onto stck initilize closure(t) to T while stck is not empty do egin pop t off stck for ech stte u with u move(t, ) do if u closure(t) do egin dd u to closure(t) push u onto stck end end 9//7 49 NFA Simultion After computing the closure move, we get set of sttes On some input extend ll these sttes to get new set of sttes 9//7 5 5

26 NFA Simultion Strt stte: q Input: c,, c k 9//7 5 Conversion from NFA to DFA Conversion method closely follows the NFA simultion lgorithm Insted of simulting, we cn collect those NFA sttes tht ehve identiclly on the sme input Group this set of sttes to form one stte in the DFA 9//7 5 6

27 Exmple: suset construction //7 53 closure(q ) //7 54 7

28 move(closure(q ), ) //7 55 closure(move(closure(q ), )) //7 56 8

29 move(closure(q ), ) //7 57 closure(move(closure(q ), )) //7 58 9

30 Suset Construction dd closure(q ) to Dsttes unmrked while unmrked T Dsttes do egin mrk T; for ech symol c do egin U := closure(move(t, c)); if U Dsttes then dd U to Dsttes unmrked Dtrns[d, c] := U; end end 9//7 59 Suset Construction sttes[] = closure({q }) p = j = while j p do egin for ech symol c do egin e = DFAedge(sttes[j], c) if e = sttes[i] for some i p then Dtrns[j, c] = i else p = p+ sttes[p] = e Dtrns[j, c] = p j = j + end end 9//7 6 3

31 DFA (prtil) [,, 3, 4, 6, 9, ] [3, 4, 5, 6, 8, 9,, 3, 4] [3, 4, 6, 7, 8, 9] 9//7 6 [,, 3, 4, 6, 9, ] DFA for (( )*) [3, 4, 5, 6, 8, 9,, 3, 4] [3, 4, 6, 7, 8, 9] [3, 4, 5, 6, 8, 9, ] [3, 4, 5, 6, 8, 9,,, 4] 9//7 6 3

32 Minimiztion of DFAs [,, 3, 4, 6, 9, ] [3, 4, 6, 7, 8, 9] [3, 4, 5, 6, 8, 9, ] [3, 4, 5, 6, 8, 9,,, 4] 9//7 63 Minimiztion of DFAs [3, 4, 5, 6, 8, 9,,, 4] [3, 4, 6, 7, 8, 9] [3, 4, 5, 6, 8, 9, ] 9//7 64 3

33 Minimiztion of DFAs Algorithm for minimizing the numer of sttes in DFA Step : prtition sttes into groups: ccepting nd nonccepting A D B 9//7 65 E C Minimiztion of DFAs Step : in ech group, find sugroup of sttes hving property P P: The sttes hve trnsitions on ech symol (in the lphet) to the sme group A, : lue A, : yellow E, : lue E, : yellow D, : yellow D, : yellow A D B 9//7 66 E C B, : lue B, : yellow C, : lue C, : yellow 33

34 Minimiztion of DFAs Step 3: if sugroup does not oey P split up the group into seprte group Go ck to step. If no further sugroups emerge then continue to step 4 A, : lue A, : green E, : lue E, : green D, : yellow D, : green A D B 9//7 67 E C B, : lue B, : green C, : lue C, : green Minimiztion of DFAs Step 4: ech group ecomes stte in the minimized DFA Trnsitions to individul sttes re mpped to single stte representing the group of sttes A D B E C 9//

35 NFA to DFA Suset construction converts NFA to DFA Complexity: For FSAs, we mesure complexity in terms of initil cost (creting the utomton) nd per string cost Let r e the length of the regexp nd n e the length of the input string NFA, Initil cost: O(r); Per string: O(rn) DFA, Initil cost: O(r s); Per string: O(n) DFA, common cse, s = r, ut worst cse s = r 9//7 69 NFA to DFA A regexp of size r cn ecome r stte DFA, n exponentil increse in complexity Try the suset construction on NFA uilt for the regexp A*A n where A is the regexp ( ) Note tht the NFA for regexp of size r will hve r sttes Minimiztion cn reduce the numer of sttes But minimiztion requires determiniztion 9//7 7 35

36 NFA to DFA 9//7 7 NFA to DFA 9//7 7 36

37 NFA to DFA 5 = 3 sttes 9//7 73 NFA vs. DFA in the wild Engine Type DFA Trditionl NFA POSIX NFA Hyrid NFA/DFA Progrms wk (most versions), egrep (most versions), flex, lex, MySQL, Procmil GNU Emcs, Jv, grep (most versions), less, more,.net lnguges, PCRE lirry, Perl, PHP (pcre routines), Python, Ruy, sed (most versions), vi mwk, MKS utilities, GNU Emcs (when requested) GNU wk, GNU grep/egrep, Tcl 9//

38 Extensions to Regulr Expressions Most modern regexp implementtions provide extensions: mtching groups; \ refers to the string mtched y the first grouping (), \ to the second mtch, etc., e.g. ([z]+)\ which mtches where \= mtch nd replce opertions, e.g. s/([z]+)/\\/g which chnges into where \= These extensions re no longer regulr. In fct, extended regexp mtching is NPhrd Extended regulr expressions (including POSIX nd Perl) re clled REGEX to distinguish from regexp (which re regulr) In order to cpture these difficult cses, the lgorithms used even for simple regexp mtching run in time exponentil in the length of the input 9//7 75 Converting Regulr Expressions directly into DFAs This lgorithm ws first used y Al Aho in egrep, nd used in wk, lex, flex 9//

39 Regexp to DFA: (() ())*# {} () {,3} (,4) * {,3} (,4) node {,3,5} (5) # 5 {3} (4) {5} (5) firstpos = {} lstpos = () {} () {} () {3} (3) 3 {4} (4) 4 9//7 77 Regexp to DFA: followpos followpos(p) tells us which positions cn follow position p There re two rules tht use the firstpos {} nd lstpos () informtion {k,l} * (i,j) (i,j) c {k,l} c followpos(i)+=k,l followpos(j)+=k,l followpos(i)+=k,l followpos(j)+=k,l 9//

40 {} Regexp to DFA: (() ())*# fp()+=,3 fp(4)+=,3 () fp()+= {} () {,3} (,4) * {3} (3) {,3,5} (5) 3 fp(3)+=4 9//7 79 # 5 {4} (4) {5} (5) 4 fp()+=5 fp(4)+=5 root={,3,5} fp()= fp(3)=4 fp()=,3,5 fp(4)=,3,5 Regexp to DFA: (() ())*# root={,3,5} fp()= fp(3)=4 fp()=,3,5 fp(4)=,3,5 : : 3: 4: 5:# # 3 4 9//7 8 # # 5 4

41 Regexp to DFA: (() ())*# root={,3,5} fp()= fp(3)=4 fp()=,3,5 fp(4)=,3,5 : : 3: 4: 5:# A {,3,5} A A: fp(), {}, B, A: fp(3), {4}, C, B C # A: fp(5),# {},# E,# B: fp(), {,3,5}, A, C: fp(4), {,3,5}, A, 9//7 8 E A B C Converting n NFA into Regulr Expression 9//7 8 4

42 NFA to RegExp A B D C Wht is the regulr expression for this NFA? 9//7 83 NFA to RegExp A B D C A = B B = D C D = B C = D 9//7 84 4

43 NFA to RegExp Three steps in the lgorithm (pply in ny order):. Sustitution: for B = X pick every A = B T nd replce to get A = X T. Fctoring: (R S) (R T) = R (S T) nd (R T) (S T) = (R S) T 3. Arden's Rule: For ny set of strings S nd T, the eqution X = (S X) T hs X = (S*) T s solution. 9//7 85 NFA to RegExp A = B B = D C D = B C = D Sustitute: A = B B = D D D = B Fctor: A = B B = ( ) D D = B Sustitute: A = ( ) D D = ( ) D 9//

44 NFA to RegExp A = ( ) D D = ( ) D Fctor: A = ( ) D D = ( ) D Arden: A = ( ) D D = ( )* Remove epsilon: A = ( ) D D = ( )* Sustitute: A = ( ) ( )* Simplify: A = ( )+ 9//7 87 NFA to Regexp using GNFAs i r r4 rip j r3 Generlized NFA: trnsition function tkes stte nd regexp nd returns set of sttes i ( (r)(r)*(r3) ) (r4) j r Algorithm:. Add new strt & ccept stte. For ech stte s: rip stte s creting GNFA, consider ech stte i nd j djcent to s 3. Return regexp from strt to ccept stte 9//

45 NFA to Regexp using GNFAs s s,, ( )* s *( )* 9//7 89 NFA to Regexp using GNFAs 3 Rip sttes,, 3 in tht order, nd we get: (( )* )(( )( )* )*(( )( )* ) ( )* 9//7 9 45

46 Implementing Lexicl Anlyzer 9//7 9 Lexicl Anlyzer using NFAs For ech token convert its regexp into DFA or NFA Crete new strt stte nd crete trnsition on to the strt stte of the utomton for ech token For input i, i,, i n run NFA simultion which returns some finl sttes (ech finl stte indictes token) If no finl stte is reched then rise n error Pick the finl stte (token) tht hs the longest mtch in the input, e.g. prefer DFA #8 over ll others ecuse it red the input until i 3 nd none of the other DFAs reched i 3 If two DFAs rech the sme input chrcter then pick the one tht is listed first in the ordered list 9//7 9 46

47 Lexicl Anlysis using NFAs TOKEN_A = TOKEN_B = 7 8 TOKEN_C = *+ 9//7 93 Lexicl Anlysis using NFAs TOKEN_A = Input: ,, 3, 7 8 TOKEN_B = TOKEN_C = *+ TOKEN_C mtches,3, 4, NONE 7 9//7 94 TOKEN_A mtches, 47

48 Lexicl Anlysis using NFAs TOKEN_A = Input: ,, 3, 7 8 TOKEN_B = TOKEN_C = *+ 3 4 Output:, $ TOKEN_C [,3] 4, NONE TOKEN_A [3,4] 7 TOKEN_A mtches 3,4 9//7 95 Lexicl Anlyzer using DFAs Ech token is defined using regexp r i Merge ll regexps into one ig regexp R = (r r r n ) Convert R to n NFA, then DFA, then minimize rememer orig NFA finl sttes with ech DFA stte 9//

49 Lexicl Anlyzer using DFAs The DFA recognizer hs to find the longest leftmost mtch for token continue mtching nd report the lst finl stte reched once DFA simultion cnnot continue e.g. longest mtch: <print> nd not <pr>, <int> e.g. leftmost mtch: for input string the regexp + will mtch nd not If two ptterns mtch the sme token, pick the one tht ws listed erlier in R e.g. prefer finl stte (in the originl NFA) of r over r 3 9//7 97 Lookhed opertor Implementing r /r : mtch r when followed y r e.g. *+/*c ccepts string c ut not d The lexicl nlyzer mtches r r up to position q in the input But rememers the position p in the input where r mtched ut not r Reset to strt stte nd strt from position p 9//

50 Efficient dtstructures for DFAs 9//7 99 Implementing DFAs D rry storing the trnsition tle Adjcency list, more spce efficient ut slower Merge two ides: rry structures used for sprse tles like DFA trnsition tles se & next rrys: Trjn nd Yo, 979 Drgon ook (defult+se & next+check) 9//7 5

51 5 9//7 Implementing DFAs d c c d c 9//7 Implementing DFAs d c se next check nextstte(s, x) : L := se[s] + x return next[l] if check[l] eq s

52 se 3 Implementing DFAs c d defult 9// next check nextstte(s, x) : L := se[s] + x return next[l] if check[l] eq s else return nextstte(defult[s], x) Summry Token Pttern Pttern Regulr Expression Regulr Expression NFA Thompson s Rules NFA DFA Suset construction DFA miniml DFA Minimiztion Lexicl Anlyzer (multiple ptterns) 9//7 4 5