Example: Source Code. Lexical Analysis. The Lexical Structure. Tokens. What do we really care here? A Sample Toy Program:

Transcription

1 Lexicl Anlysis Red source progrm nd produce list of tokens ( liner nlysis) source progrm The lexicl structure is specified using regulr expressions Other secondry tsks: (1) get rid of white spces (e.g., \t,\n,\sp) nd comments () line numering lexicl nlyzer token get next token prser A Smple Toy Progrm: Exmple: Source Code (* define vlid mutully recursive procedures *) let function do_nothing1(: int, : string)= do_nothing(+1) function do_nothing(d: int) = do_nothing1(d, str ) in end do_nothing1(0, str ) Wht do we relly cre here? Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 1 of 0 Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge of 0 The Lexicl Structure Output fter the Lexicl Anlysis token + ssocited vlue LET 51 FUNCTION 56 ID(do_nothing1) 65 LPAREN 76 ID() 77 COLON 78 ID(int) 80 COMMA 8 ID() 85 COLON 86 ID(string) 88 RPAREN 9 EQ 95 ID(do_nothing) 99 LPAREN 110 ID() 111 PLUS 11 INT(1) 11 RPAREN 11 FUNCTION 117 ID(do_nothing) 16 LPAREN 17 ID(d) 18 COLON 19 ID(int) 11 RPAREN 1 EQ 16 ID(do_nothing1) 150 LPAREN 161 ID(d) 16 COMMA 16 STRING(str) 165 RPAREN 170 IN 17 ID(do_nothing1) 177 LPAREN 188 INT(0) 189 COMMA 190 STRING(str) 19 RPAREN 198 END 00 EOF 0 Tokens Tokens re the tomic unit of lnguge, nd re usully specific strings or instnces of clsses of strings. Tokens Smple Vlues Informl Description LET let keyword LET END end keyword END PLUS + LPAREN ( COLON : STRING str RPAREN ) INT 9, 8 integer constnts ID do_nothing1,, int, string EQ = EOF letter followed y letters, digits, nd underscores end of file Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge of 0 Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge of 0

2 Lexicl Anlysis, How? First, write down the lexicl specifiction (how ech token is defined?) using regulr expression to specify the lexicl structure: identifier = letter (letter digit underscore) * letter =... z A... Z digit = Second, sed on the ove lexicl specifiction, uild the lexicl nlyzer (to recognize tokens) y hnd, Regulr Expression Spec ==> NFA ==> DFA ==>Trnsition Tle ==> Lexicl Anlyzer Or just y using lex --- the lexicl nlyzer genertor Regulr Expression Spec (in lex formt) ==> feed to lex ==> Lexicl Anlyzer Regulr Expressions regulr expressions re concise, linguistic chrcteriztion of regulr lnguges (regulr sets) identifier = letter (letter digit underscore) * or ech regulr expression define regulr lnguge --- set of strings over some lphet, such s ASCII chrcters; ech memer of this set is clled sentence, or word we use regulr expressions to define ech ctegory of tokens 0 or more For exmple, the ove identifier specifies set of strings tht re sequence of letters, digits, nd underscores, strting with letter. Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 5 of 0 Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 6 of 0 Regulr Expressions nd Regulr Lnguges Given n lphet the regulr expressions over nd their corresponding regulr lnguges re ) denotes,the empty string, denotes the lnguge { }. ) for ech in, denotes { } --- lnguge with one string. c) if R denotes L R nd S denotes L S then R S denotes the lnguge L R L S, i.e, { x x L R or x L S }. d) if R denotes L R nd S denotes L S then RS denotes the lnguge L R L S, tht is, { xy x L R nd y L S }. e) if R denotes L R then R * denotes the lnguge L R * where L * is the union of ll L i (i=0,...,nd L i is just {x 1 x...x i x 1 L,..., x i L}. Exmple Regulr Expression Explntion * 0 or more s + 1 or more s ( ) * ll strings of s nd s (including ) ( ) * ll strings of s nd s of even length [-za-z] shorthnd for... z A... Z [0-9] shorthnd for ([0-9]) * 0 numers tht strt nd end with 0 ( ) * ( e)?? ll strings tht contin foo s sustring the following is not regulr expression: n n (n > 0) f) if R denotes L R then (R) denotes the sme lnguge L R. Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 7 of 0 Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 8 of 0

3 Lexicl Specifiction Using regulr expressions to specify tokens keyword = egin end if then else identifier = letter (letter digit underscore) * integer = digit + relop = < <= = <> > >= letter =... z A B... Z digit = Amiguity : is egin keyword or n identifier? Next step: to construct token recognizer for lnguges given y regulr expressions --- y using finite utomt! given string x, the token recognizer sys yes if x is sentence of the specified lnguge nd sys no otherwise Trnsition Digrms Flowchrt with sttes nd edges; ech edge is lelled with chrcters; certin suset of sttes re mrked s s Trnsition from stte to stte proceeds long edges ccording to the next input chrcter letter digit underscore strt letter delimiter 0 1 Every string tht ends up t is ccepted If get stuck, there is no trnsition for given chrcter, it is n error Trnsition digrms cn e esily trnslted to progrms using cse sttements (in C). Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 9 of 0 Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 10 of 0 Trnsition Digrms (cont d) The token recognizer (for identifiers) sed on trnsition digrms: stte0: stte1: c = getchr(); if (islph(c)) goto stte1; error();... c = getchr(); if (islph(c) isdigit(c) isunderscore(c)) goto stte1; if (c ==,... c == ) ) goto stte; error();... stte: ungetc(c,stdin); /* retrct current chr */ return(id,... the current identifier...); Next: 1. finite utomt re generlized trnsition digrms!. how to uild finite utomt from regulr expressions? Finite Automt Finite Automt re similr to trnsition digrms; they hve sttes nd lelled edges; there re one unique strt stte nd one or more thn one finl sttes Nondeterministic Finite Automt (NFA) : ) cn lel edges (these edges re clled -trnsitions) ) some chrcter cn lel or more edges out of the sme stte Deterministic Finite Automt (DFA) : ) no edges re lelled with ) ech chrcter cn lel t most one edge out of the sme stte NFA nd DFA ccepts string x if there exists pth from the strt stte to leled with chrcters in x NFA: multiple pths DFA: one unique pth Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 11 of 0 Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 1 of 0

4 Exmple: NFA Exmple: DFA strt stte strt 0 1 An NFA ccepts ( ) * strt stte strt 0 1 A DFA ccepts ( ) * There re mny possile moves --- to ccept string, we only need one sequence of moves tht led to. There is only one possile sequence of moves --- either led to nd ccept or the input string is rejected input string: One sucessful sequence: Another unsuccessful sequence: input string: The sucessful sequence: Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 1 of 0 Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 1 of 0 Trnsition Tle Finite Automt cn lso e represented using trnsition tles NFA with -trnsitions 1. NFA cn hve -trnsitions --- edges lelled with For NFA, ech entry is set of sttes: For DFA, ech entry is unique stte: 1 STATE 0 {0,1} {0} 1 - {} - {} - - STATE ccepts the regulr lnguge denoted y ( * * ) Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 15 of 0 Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 16 of 0

5 Regulr Expressions -> NFA How to construct NFA (with -trnsitions) from regulr expression? Algorithm : pply the following construction rules, use unique nmes for ll the sttes. (inportnt invrint: lwys one!) 1. Bsic Construction i f. Inductive Construction R 1 R RE -> NFA (cont d) i initil stte for N 1 nd N N 1 N for N 1 nd N f N 1 : NFA for R 1 N : NFA for R the new nd unique i f R 1 R merge : of N 1 nd initil stte of N initil stte for N 1 N 1 N for N Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 17 of 0 Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 18 of 0 RE -> NFA (cont d). Inductive Construction (cont d) Exmple : RE -> NFA Converting the regulr expression : ( ) * (in )===> R 1 * initil stte for N 1 for N 1 N 1 : NFA for R 1 (in )===> 5 i N 1 f ====> Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 19 of 0 Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 0 of 0

6 Exmple : RE -> NFA (cont d) Converting the regulr expression : ( ) * Exmple : RE -> NFA (cont d) Converting the regulr expression : ( ) * ( ) * ====> ( ) * ====> X ====> (severl steps re omitted) Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 1 of 0 Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge of 0 NFA -> DFA NFA re non-deterministic; need DFA in order to write deterministic prorgm! There exists n lgorithm ( suset construction ) to convert ny NFA to DFA tht ccepts the sme lnguge Sttes in DFA re sets of sttes from NFA; DFA simultes in prllel ll possile moves of NFA on given input. Definition: for ech stte s in NFA, -CLOSURE(s) = { s } { t s cn rech t vi -trnsitions } Definition: for ech set of sttes S in NFA, -CLOSURE(S) = i -CLOSURE(s) for ll s i in S NFA -> DFA (cont d) ech DFA-stte is set of NFA-sttes suppose the strt stte of the NFA is s, then the strt stte for its DFA is - CLOSURE(s) ; the s of the DFA re those tht include NFA-finl-stte Algorithm : converting n NFA N into DFA D ---- Dsttes = {e-closure(s 0 ),s 0 is N s strt stte} Dsttes re initilly unmrked while there is n unmrked D-stte X do { mrk X for ech in S do { T = {sttes reched from ny s i in X vi } Y = e-closure(t) if Y not in Dsttes then dd Y to Dsttes unmrked dd trnsition from X to Y, lelled with } } Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge of 0 Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge of 0

7 Exmple : NFA -> DFA converting NFA for ( )* to DFA The strt stte A = -CLOSURE(0) = {0, 1,,, 7}; Dsttes={A} 1st itertion: A is unmrked; mrk A now; -trnsitions: T = {, 8} new stte B= -CLOSURE() -CLOSURE(8) = {, 6, 1,,, 7} 8) = {1,,,, 6, 7, 8} dd trnsition from A to B lelled with -trnsitions: T = {5} new stte C = -CLOSURE(5) = {1,,, 5, 6, 7} dd trnsition from A to C lelled with Dsttes = {A, B, C} nd itertion: B, C re unmrked; we pick B nd mrk B first; B = {1,,,, 6, 7, 8} B s -trnsitions: T = {, 8}; T s -CLOSURE is B itself. dd trnsition from B to B lelled with Exmple : NFA -> DFA (cont d) B s -trnsitions: T = {5, 9}; new stte D = -CLOSURE({5, 9}) = {1,,, 5, 6, 7, 9} dd trnsition from B to D lelled with Dsttes = {A, B, C, D} then we pick C, nd mrk C C s -trnsitions: T = {, 8}; its -CLOSURE is B. dd trnsition from C to B lelled with C s -trnsitions: T = {5}; its -CLOSURE is C itself. dd trnsition from C to C lelled with next we pick D, nd mrk D D s -trnsitions: T = {, 8}; its -CLOSURE is B. dd trnsition from D to B lelled with D s -trnsitions: T = {5, 10}; new stte E = -CLOSURE({5, 10}) = {1,,, 5, 6, 7, 10} Dsttes = {A, B, C, D, E}; E is since it hs 10; next we pick E, nd mrk E Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 5 of 0 Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 6 of 0 Exmple : NFA -> DFA (cont d) E s -trnsitions: T = {, 8}; its -CLOSURE is B. dd trnsition from E to B lelled with E s -trnsitions: T = {5}; its -CLOSURE is C itself. dd trnsition from E to C lelled with Other Algorithms How to minimize DFA? (see Drgon Book.9, pp11) ll sttes in Dsttes re mrked, the DFA is constructed! A B D C E How to convert RE to DFA directly? (see Drgon Book.9, pp15) How to prove two Regulr Expressions re equivlent? (see Drgon Book pp150, Exercise.) Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 7 of 0 Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 8 of 0

8 Lex Lex is progrm genertor it tkes lexicl specifiction s input, nd produces lexicl processor written in C. DIGITS [0-9]... Lex Specifiction lex definition Lex Specifiction foo.l Lex lex.yy.c expression ction integer printf( INT ); trnsltion rules lex.yy.c input text C Compiler.out.out sequence of tokens chr getc() {... } user s C functions (optionl) Implementtion of Lex: Lex Spec -> NFA -> DFA -> Trnsition Tles + Actions -> yylex() expression is regulr expression ; ction is piece of C progrm; for detils, red the Lesk&Schmidt pper Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 9 of 0 Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 0 of 0 ML-Lex ML-Lex is like Lex it tkes lexicl specifiction s input, nd produces lexicl processor written in Stndrd ML. Lex Specifiction foo.lex foo.lex.sml input text ML-Lex ML Compiler Mlex Implementtion of ML-Lex is similr to implementtion of Lex foo.lex.sml module Mlex sequence of tokens ML-Lex Specifiction type pos = int vl linenum =... vl lexresult = %s COMMENT STRING; SPACE=[ \t\n\01]; DIGITS=[0-9];... expression => (ction); integer => (print( INT ));... => (...linenum...); user s ML declrtions ml-lex definitions trnsltion rules cn cll the ove ML declrtions expression is regulr expression ; ction is piece of ML progrm; when the input mtches the expression, the ction is executed, the text mtched is plced in the vrile yytext. Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 1 of 0 Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge of 0

9 Wht does ML-Lex generte? foo.lex ML-Lex foo.lex.sml smple foo.lex.sml: everything in prt 1 of foo.lex structure Mlex = struct structure UserDeclrtions = struct... end... fun mkelexer yyinput =... end To use the generted lexicl processor: vl lexer = Mlex.mkeLexer(fn _ => input (openin toy )); vl nexttoken = lexer() ech cll returns one token! input filenme ML-Lex Definitions Things you cn write inside the ml-lex definitions section (nd prt): %s COMMENT STRING define new strt sttes %reject REJECT() to reject mtch %count count the line numer %structure {identifier} the resulting structure nme (the defult is Mlex) (hint: you proly don t need use %reject, %count,or %structure for ssignment.) Definition of nmed regulr expressions : identifier = regulr expression SPACE=[ \t\n\01] IDCHAR=[_-zA-Z0-9] Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge of 0 Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge of 0 ML-Lex Trnsltion Rules Ech trnsltion rule (rd prt) re in the form <strt-stte-list> regulr expression => (ction); Vlid ML-Lex regulr expressions: (see ML-Lex-mnul pp -6) chrcter stnds for itself except for the reserved chrs:? * + ( ) ^ $ / ;. = < > [ { \ to use these chrs, use ckslsh! for exmple, \\\ represents the string \ using squre rckets to enclose set of chrcters ( \ - ^ re reserved) [c] chr, or, or c [^c] ll chrs except,, c [-z] ll chrs from to z [\n\t\] new line, t, or ckspce [-c] chr - or or or c ML-Lex Trnsltion Rules (cont d) Vlid ML-Lex regulr expressions: (cont d) escpe sequences: (cn e used inside or outside squre rckets) \ ckspce \n newline \t t \ddd ny scii chr (ddd is digit deciml). ny chr except newline (equivlent to [^\n]) x mtch string x exctly even if it contins reserved chrs x? n optionl x x* 0 or more x s x+ 1 or more x s x y x or y ^x if t the eginning, mtch t the eginning of line only {x} sustitute definition x (defined in the lex definition section) (x) sme s regulr expression x x{n} repeting x for n times x{m-n} repeting x from m to n times Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 5 of 0 Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 6 of 0

10 ML-Lex Trnsltion Rules (cont d) wht re vlid ctions? Actions re siclly ML code (with the following extensions) All ctions in lex file must return vlues of the sme type Use yytext to refer to the current string [-z]+ => (print yytext); [0-9]{} => (print (Chr.ord(su(yytext,0)))); Cn refer to nything defined in the ML-Declrtion section (1st prt) YYBEGIN strt-stte enter into nother strt stte lex() nd continue() to reinvoking the lexing function yypos --- refer to the current position Amiguity wht if more thn one trnsltion rules mtches? 1 A. longest mtch is preferred B. mong rules which mtched the sme numer of chrcters, the rule given first is preferred while => (Tokens.WHILE(...)); [-za-z][-za-z0-9_]* => (Tokens.ID(yytext,...)); < => (Tokens.LESS(...)); <= => (Tokens.LE(yypos,...)); input while mtches rule 1 ccording B ove input <= mtches rule ccording A ove Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 7 of 0 Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 8 of 0 Strt Sttes (or Strt Conditions) strt sttes permit multiple lexicl nlyzers to run together. ech trnsltion rule cn e prefixed with <strt-stte> the lexer is initilly in predefined strt ste clled INITIAL define new strt sttes (in ml-lex-definitions): %s COMMENT STRING to switch to nother strt sttes (in ction): YYBEGIN COMMENT exmple: multi-line comments in C %s COMMENT <INITIAL> /* => (YYBEGIN COMMENT; continue()); <COMMENT> */ => (YYBEGIN INITIAL; continue()); <COMMENT>. \n => (continue()); <INITIAL>... Implementtion of Lex construct NFA for sum of Lex trnsltion rules (regexp/ction); convert NFA to DFA, then minimize the DFA to recognize the input, simulte DFA to termintion; find the lst DFA stte tht includes NFA, execute ssocited ction (this pickes longest mtch). If the lst DFA stte hs >1 NFA s, pick one for rule tht ppers first how to represent DFA, the trnsition tle: D rry indexed y stte nd input-chrcter too ig! ech stte hs linked list of (chr, next-stte) pirs too slow! hyrid scheme is the est Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 9 of 0 Copyright Zhong Sho, Yle University Lexicl Anlysis : Pge 0 of 0