Recognition of Tokens

Transcription

1 42 Recognton o Tokens The queston s how to recognze the tokens? Exmple: ssume the ollowng grmmr rgment to generte specc lnguge: stmt expr expr then stmt expr then stmt else stmt term relop term term term d num where the termnls, then, else, relop, d, nd num generte sets o strngs gven y the ollowng regulr dentons: then then else else relop < <= = <> > >= d letter(letter dgt)* num dgts optonl-rcton optonl-exponent Where letter nd dgts re s dened prevously. For ths lnguge rgment the lexcl nlyzer wll recognze the keywords, then, else, s well s the lexemes denoted y relop, d, nd num. To smply mtters, we ssume keywords re reserved; tht s, they cnnot e used s denters. The num represents the unsgned nteger nd rel numers o Pscl. In ddton, we ssume lexemes re seprted y whte spce, consstng o nonnull sequences o lnks, ts, nd newlnes. The lexcl nlyzer wll strp out whte spce. It wll do so y comprng strng gnst the regulr denton ws, elow. delm lnk t newlne ws delm + I mtch or ws s ound, the lexcl nlyzer does not return token to the prser.

2 42 Trnston Dgrms (TD) As n ntermedte step n the constructon o lexcl nlyzer, we rst produce lowchrt, clled Trnston dgrm. Trnston dgrms depct the ctons tht tke plce when lexcl nlyzer s clled y the prser to get the next token. The TD uses to keep trck o normton out chrcters tht re seen s the orwrd ponter scns the nput. t dose tht y movng rom poston to poston n the dgrm s chrcters re red. Components o Trnston Dgrm 1. One stte s leled the Strt Stte; strt t s the ntl stte o the trnston dgrm where control resdes when we egn to recognze token. 2. Postons n trnston dgrm re drwn s crcles nd re clled sttes. 3. The sttes re connected y Arrows, clled edges. Lels on edges re ndctng the nput chrcters. 4. The Acceptng sttes n whch the tokens hs een ound. 5. Retrct one chrcter use * to ndcte sttes on whch ths nput retrcton.

3 42 Exmple: A Trnston Dgrm or the token relton opertors "relop" s shown n Fgure elow: strt < 0 1 = 2 Return (relop,le) > 3 Return (relop,ne) Other 4 * Return (relop,lt) = > 5 6 Return (relop,eq) = 7 Return (relop,ge) Other 8 * Return (relop,gt) Trnston Dgrm or relton opertors Exmple: A Trnston Dgrm or the denters nd keywords: Letter or dgt strt 9 letter other * Trnston Dgrm or denters nd keywords

4 42 Exmple: A Trnston Dgrm or Unsgned Numers n Pscl: dgt dgt dgt strt 12 dgt dgt 15 E 16 +or- 17 dgt 18 other * 19 E dgt dgt dgt strt 20 dgt dgt 23 other 24 dgt strt 25 dgt 26 other 27 * num dgt + (. dgt + )(E(+ - )dgt + ) Trnston Dgrm or unsgned numers n Pscl Tretment o Whte Spce (WS): delm strt 28 delm 29 other * 30 Trnston Dgrm or Whte Spce Nothng s returned when the cceptng stte s reched; we merely go ck to the strt stte o the rst trnston dgrm to look or nother pttern.

5 42 Fnte Automt (FA) It generlzed trnston dgrm TD, constructed to comple regulr expresson RE nto recognzer. Recognzer or Lnguge: s progrm tht tkes strng X s n nput nd nswers "Yes" X s sentence o the lnguge nd "No" otherwse. FA Nondetermnstc Fnte Automt (NFA) Determnstc Fnte Automt (DFA) Note: Both NFA nd DFA re cple o recognzng wht regulr expresson cn denote. Nondetermnstc Fnte Automt (NFA) NFA: mens tht more thn one trnston out o stte my e possle on sme nput symol. 1 2 Also trnston on nput ( -Trnston) s possle

6 42 A nondetermnstc nte utomton NFA s mthemtcl model conssts o 1) A set o sttes S; 2) A set o nput symol,, clled the nput symols lphet. 3) A set o trnston to move the symol to the sets o sttes. 4) A stte S0 clled the ntl or the strt stte. 5) A set o sttes F clled the cceptng or nl stte. Exmple: The NFA tht recognzes the lnguge ( ) * s shown elow: strt Exmple: The NFA tht recognzes the lnguge * * s shown elow:

7 03 Determnstc Fnte Automt (DFA) A determnstc nte utomton (DFA, or short) s specl cse o non-determnstc nte utomton (NFA) n whch 1. No stte hs n -trnston,.e., trnston on nput, nd 2. For ech stte S nd nput symol, there s t most one edge leled levng S. A determnstc nte utomton DFA hs t most one trnston rom ech stte on ny nput. Exmple: The ollowng gure shows DFA tht recognzes the lnguge ( )*. strt The Trnston Tle s: Stte

8 03 Converson o n NFA nto DFA It s hrd or computer progrm to smulte n NFA ecuse the trnston uncton s multvlued. The lgorthm tht clled the suset constructon wll convert n NFA or ny lnguge nto DFA tht recognzes the sme lnguges. Algorthm: (Suset constructon): constructng DFA rom NFA. Input: NFA N. Output: DFA D cceptng the sme lnguge. Method: ths lgorthm constructs trnston tle Dtrn or D. Ech DFA stte s set o NFA sttes nd we construct Dtrn so tht D wll smulte "n prllel" ll possle moves N cn mke on gven nput strng. It use the opertons n elow to keep trck o sets o NFA sttes (s represents n NFA stte nd T set o NFA sttes). Opertons -closure(s) -closure(t) Move(T, ) Descrpton Set o NFA sttes rechle rom NFA stte s on - trnstons lone. Set o NFA sttes rechle rom some NFA stte s n T on -trnstons lone. Set o NFA sttes to whch there s trnston on nput symol rom some NFA stte s n T. 1) -closure (s0) s the strt stte o D. 2) A stte o D s cceptng t contns t lest one cceptng stte n N.

9 04 Algorthm: (Suset constructon): Intlly, -closure(s0) s the only stte n Dsttes nd t s unmrked; whle there s n unmrked stte T n Dsttes do egn mrk T; For ech nput symol do egn U: = ( -closure (move (T, )) ; U s not n Dsttes then dd U s n unmrked stte to Dsttes; Dtrn [T, ]: = U End End We construct Dsttes, the set o sttes o D, nd Dtrn, the trnston tle or D, n the ollowng mnner. Ech stte o D corresponds to set o NFA sttes tht N could e n ter redng some sequence o nput symols ncludng ll possle -trnstons eore or ter symols re red. Algorthm: Computton o -closure Push ll sttes n T onto slck; Intlze -closure (T) to T; Whle slck s not empty do egn Pop t, the top clement, o o stck; For ech stte u wth n edge rom t to u leled do I u s not n -closure (T) do egn Add u to -closure (T); Push u onto stck End End A smple lgorthm to compute -closure (T) uses stck to hold sttes whose edges hve not een checked or -leled trnstons.

10 00 Exmple: The gure elow shows NFA N cceptng the lnguge ( )* Sol: pply the Algorthm o Suset constructon s ollow: 1) Fnd the strt stte o the equvlent DFA s -closure (0), whch s consst o strt stte o NFA nd the ll sttes rechle rom stte 0 v pth n whch every edge s leled. A= {0, 1, 2, 4, 7} 2) Compute move (A, ), the set o sttes o NFA hvng trnstons on rom memers o A. Among the sttes 0, 1, 2, 4 nd 7, only 2 nd 7 hve such trnstons, to 3 nd 8, so move (A, )={3, 8} Compute the -closure (move (A, )) = -closure ({3, 8}), -closure ({3, 8}) = {1, 2, 3, 4, 6, 7, 8} Let us cll ths set B. strt A B

11 02 3) Compute move (A, ), the set o sttes o NFA hvng trnstons on rom memers o A. Among the sttes 0, 1, 2, 4 nd 7, only 4 hve such trnstons, to 5 so move (A, )={5} Compute the -closure (move (A, )) = -closure ({5}), -closure ({5}) = {1, 2, 4, 5, 6, 7} Let us cll ths set C. So the DFA hs trnston on rom A to C. strt A C 4) We pply the steps 2 nd 3 on the B nd C, ths process contnues or every new stte o the DFA untl ll sets tht re sttes o the DFA re mrked. The ve derent sets o sttes we ctully construct re: A = {0, 1, 2, 4, 7} B = {1, 2, 3, 4, 6, 7, 8} C = {1, 2, 4, 5, 6, 7} D = {1, 2, 4, 5, 6, 7, 9} E = {1, 2, 4, 5, 6, 7, 10} Stte A s the strt stte, nd stte E s the only cceptng stte. The complete trnston tle Dtrn s shown n elow: STATE INPUT SYMBOL A B C B C B B D C D B E E B C Trnston tle Dtrn or DFA

12 02 Also, trnston grph or the resultng DFA s shown n elow. It should e noted tht the DFA lso ccepts ( )*. C strt A B D E From Regulr Expresson to n NFA Now gve n lgorthm to construct n NFA rom regulr expresson. The lgorthm s syntx-drected n tht t uses the syntctc structure o the regulr expresson to gude the constructon process. Algorthm: (Thompson's constructon): Input: regulr expresson R over lphet. Output: NFA N cceptng L(R). 1- For, construct the NFA strt Here s strt stte nd cceptng stte. Clerly ths NFA recognzes { }.

13 02 2- For n, construct the NFA strt Agn s strt stte nd cceptng stte. Ths mchne recognzes {}. 3- For the regulr expresson construct the ollowng composte NFA N( ). 4- For the regulr expresson construct the ollowng composte NFA N(). strt 5- For the regulr expresson * construct the ollowng composte NFA N(*). strt

14 02 Exmple: let us use lgorthm Thompson's constructon to construct the ollowng regulr expressons: 1) RE = ()* 2) RE= ( )* 3) RE= ( )*

15 02 4) RE= * ( ) strt Lexcl Errors Wht user omts the spce n For? No lexcl error, sngle token IDENT ( For ) s produced nsted o sequence For, IDENT ( ). Typclly ew lexcl error types 1) the llegl chrs, or exmple: Wrt@ln (x); 2) untermnted comments, or exmple: {Mn progrm 3) Ill-ormed constnts How s Scnner Progrmmed? 1) Descre tokens wth regulr expressons. 2) Drw trnston dgrms. 3) Code the dgrm s tle/progrm.