Table-driven look-ahead lexical analysis

Transcription

1 Tle-riven look-he lexil nlysis WUU YANG Computer n Informtion Siene Deprtment Ntionl Chio-Tung University, HsinChu, Tiwn, R.O.C. Astrt. Moern progrmming lnguges use regulr expressions to efine vli tokens. Tritionl lexil nlyzers se on minimum eterministi finite utomt for regulr expressions nnot hnle the look-he prolem. The snner writer nees to expliitly ientify the look-he sttes n oe the uffering n re-snning opertions y hn. We ientify the lss of finite look-he finite utomt, whih is generl enough to inlue ll finite utomt of prtil lexil nlyzers. Finite look-he finite utomt re then trnsforme into suffix finite utomt. A new lexil nlyzer mkes use of the suffix finite utomt to ientify tokens. The new lexil nlyzer solves the look-he prolem in tle-riven pproh n it n etet lexil errors t n erlier time thn tritionl lexil nlyzers. The extr ost of the new lexil nlyzers is the lrger stte trnsition tle n three itionl 1-imensionl tles. Keywors: ompilers, finite utomt, finite-lookhe finite utomt, inrementl lexil nlysis, lnguge proessors, lexil nlysis, regulr expressions, suffix finite utomt 1. Introution Moern progrmming lnguges use regulr expressions to efine vli tokens. A snner is uilt from the finite utomton (FA) tht orrespons to the set of regulr expressions. In etermining the en of token, snner usully nees to exmine n extr symol from the input. This is lle 1-symol lookhe. In some ses, the snner nees to look he multiple symols. For instne, to sn in Psl n A, the snner nees to look he two symols (i.e., the two ".") fter the 10 []. A similr exmple in Fortrn is "10.EQ.IX". There re two iffiulties when the snner nees to look he symols. First, the previewe symols re usully hnle y uffering mehnism. The size of the uffer must e no less thn the mximum numer of look-he symols require y the snner. It is not strightforwr to etermine the mximum numer of look-he symols from the regulr-expression speifition of tokens. Seon, the previewe symols must e uffere n re-snne when the next token is requeste. For snner genertors tht generte trnsition tles, rther thn oe, the uffering n re-snning opertions must e expliitly ientifie n hn-oe y the snner writer. Though some snner genertors my utomtilly generte oe to solve the look-he prolem, these genertors work only in n ho wy. Aoring to Pxson, the look-he prolem (lle ktrking in flex) is "messy n often my e n enormous mount of work for omplite snner". The look-he prolem origintes from the longest-mth onvention, whih sttes tht the snner shoul fin, strting from the urrent lotion of the input, longest string stisfying token efinition. This onvention n e trnslte into the stte-trnsition ehvior of the finite utomt of the regulr expressions. A tritionl snner mkes stte trnsitions oring to finite utomton. When the This work ws supporte in prt y Ntionl Siene Counil, Tiwn, R.O.C. uner grnts NSC E T n NSC S CL. Copyright 1994 y Wuu Yng. All rights reserve. An extene version of this pper hs een epte y At Informti.

2 snner fils to trnsit to next stte, it etermines the en of the urrent token n puts the previewe symols into uffer. The finite utomton tht unerlies the tritionl snner is usully eterministi n minimum. The reson for hoosing the minimum eterministi FA is to reue the size of the trnsition tle of the snner. However, the minimum eterministi FA oes not help the snner to eie the en of urrent token n to proess the previewe symols. By using suffix finite utomton, we n uil new snners tht solve the look-he prolem in tle-riven pproh. The snner writer is ompletely free from onerns of the look-he prolem. The suffix utomt re lso relte to prllel lexil nlysis. For the ske of revity, prllel lexil nlysis is omitte from the pper.. The ompute-lookhe lgorithm Given set of regulr expressions efining tokens for progrmming lnguge, n lgorithm is presente in this setion tht etermines the mximum numer of look-he symols require y the snner. We ssume tht the snner employs the longest-mth onvention to resolve onflits. The input is string of symols. There is finite voulry of istint symols. We use α, β, n γ to enote strings of symols. The length of string α, enote y α, is the numer of symols in it. The nottion α β mens α is proper prefix of β. A yle in irete grph is pth tht strts n ens t the sme stte (or noe). In DFA, there is n initil stte n one or more epting sttes. The initil stte is pointe to y n rrow whose til is lele strt; the epting sttes re inite y oule irles. An exmple FA is shown in Figure. An epting-to-epting pth is pth from n epting stte to n epting stte tht oes not pss through ny epting sttes, exept the first n the lst epting sttes on the pth. The first n the lst epting sttes on n epting-to-epting pth nee not e istint. Definition. A finite-lookhe finite utomton (FFA) is eterministi finite utomton tht looks he t most finite numer of input symols when etermining the en of token. Consier the exmple of in Psl or A gin. A lose look revels tht the reson why the snner nees to look he two symols fter the 10 is euse oth the integer 10 n eiml numer suh s 10. re vli tokens ut 10. is not. This oservtion les to the following theorem. Theorem 1. The mximum numer of lookhe symols require y the snner = mx ( γ α ), where α n γ re vli tokens, α γ, n for ll β suh tht α β γ, β is not vli token. The mximum numer of look-he symols is the length of the longest epting-to-epting pths. This length is ompute y moifie ynmi progrmming metho. Exmple. Consier the two regulr expressions: { { } * } * n { } * { { } * } *. A eterministi finite utomton orresponing to the two regulr expressions is shown in Figure (). The sttes re numere 1 through. Stte 4 is the initil stte; sttes 4 n re epting sttes. In this exmple, 4 is the length of the epting-to-epting pth 1. This mens the snner nees to look he t most 4 symols in orer to etet the en of token. Figure () is expline in the next setion. The following theorem emonstrtes tht the mximum numer of look-he symols is etermine from the regulr expressions, rther thn the prtiulr DFA hosen for the regulr expressions. Theorem. Let M 1 n M e equivlent DFAs. The mximum numers of look-he symols require y M 1 n M re the sme. Corollry. Let M 1 n M e equivlent DFAs. M 1 is n FFA if n only M is n FFA.

3 Algorithm: Compute-Lookhe /* Given set of regulr expressions, this lgorithm etermines the mximum numer */ /* of lookhe symols require y the snner. */ Trnsform the set of regulr expressions into eterministi finite utomton. Numer the non-epting sttes 1,,..., k ; numer the epting sttes k +1, k +,..., n. Delete ll sttes tht re not rehle from n epting stte n ll epting sttes. Chek whether there is ny yle in the resulting grph. if the resulting grph ontins yles then return( ) /* the FA is not n FFA */ Put ll the sttes n eges elete in the ove step k into the grph. for := 1 to n o for := 1 to n o if = then A 0 [, ] := 0 else if there n ege then A 0 [, ] := 1 else A 0 [, ] := for i := 1 to k o for := 1 to n o for := 1 to n o A i [, ] := mx ( A i 1 [, ], A i 1 [, i ] + A i 1 [i, ] ) The mximum numer of lookhe symols require y the snner is the mximum of A k [, ], where = k +1,..., n n = k +1,..., n. Figure 1. The Compute-Lookhe Algorithm strt 4 1 () DFA 6 strt 4 1 () suffix FA () steps in the ynmi progrmming A A A 1 4 A Figure. An exmple to illustrte the Compute-Lookhe lgorithm. It is our opinion tht ll rel-worl snners shoul use only finite-lookhe FAs sine it is iffiult to hnle potentilly unoune uffer. Furthermore, n FA tht my look he n infinite numer of symols nees O (n ) time on input of length n. By ontrst, finite-lookhe FA nees only O (n ) time on ny input of length n.

4 4. Suffix finite utomt When the DFAs orresponing to the regulr expressions re FFAs, tle-riven look-he snners n e uilt for the regulr expressions. A pth on DFA n e ivie into segments y the epting sttes on the pth. The lst segment on the pth is lle the suffix. Definition. A suffix of non-epting stte s is pth from n epting stte to s tht oes not go through ny epting sttes. If stte is not rehle from ny epting sttes, its suffix is the empty pth. The suffix of n epting stte is lwys the empty pth. Definition. A suffix FA (SFA) is eterministi finite utomton in whih, for eh stte s, ll the suffixes of stte s rry the sme lels. In suffix FA, eh stte s hs unique suffix lel. This suffix lel represents the look-he symols when the finite utomton hlts t stte s. Note tht ll SFAs re lso FFAs n only FFAs hve equivlent SFAs. To trnsform n FFA into n equivlent SFA, we split ll the non-epting sttes tht hve two or more ifferent suffixes. For instne, onsier stte 1 in Figure (). Stte 1 hs two ifferent suffixes, tht is, the two pths 1 n 4 1, whih re lele n, respetively. After splitting stte 1, we get n equivlent suffix FA shown in Figure (). Theorem 4. All n only FFAs hve equivlent SFAs. 4. Lexil tles The suffix FA is the sis for the new snner. In ition to the tritionl stte trnsition tle, three new tles re generte: ontinution tle, n tion tle, n n output tle. Rememer tht the suffix lels of stte s represent the previewe symols from input when the snner hlts t stte s. When the next token is requeste, this suffix lel shoul e snne gin. Sine in n SFA, eh stte hs unique suffix lel, it is possile to pre-ompute the pth whih the SFA psses through when the suffix lel of stte is fe into the SFA. We first efine funtion f inl tht tkes string s rgument n returns pir, of whih the first is sequene of tokens n the seon is string. Roughly speking, f inl (α) returns the output of the SFA n the remining suffix of α when α is use s input. Given string α, if the SFA n sn α ompletely, then f inl (α) = (nil, α). On the other hn, if the SFA nnot sn α ompletely n some non-null prefix of α n move the SFA to n epting stte, let β e the longest prefix of α tht moves the SFA from the initil stte to n epting stte f. We my write α s βα. Then f inl (α) = (ont (Tf, f irst (f inl (α ))), seon (f inl (α ))), where Tf is the token ssoite with stte f, f irst n seon return the first n seon elements of pir, respetively, n ont ontente token to sequene. If the SFA nnot sn α ompletely n no prefix of α n move the SFA to n epting stte, f inl (α) = (nil, error ). There is one entry for eh stte of the SFA in the ontinution, tion, n output tles. These entries re etermine from the suffix lel of the stte. Let α e the suffix lel of stte s. Output [s ] is f irst (f inl (α)). If seon (f inl (α)) is not error, let p e the pth whih the SFA psses through when seon (f inl (α)) is use s input. Continution [s ] is the lst stte on the pth p. Ation [s ] is the token ssoite with the lst epting stte on the pth p, exept the very first initil stte (note tht the initil stte might well e n epting stte). If there is no epting stte on the pth p, exept the very first initil stte, tion [s ] is NULL.

5 Suppose seon (f inl (α)) is n error. Then ontinution [s ] is error, whih mens tht the SFA eventully etets the lexil error when the suffix lel α of stte s is use s input. Ation [s ] is NULL in this se. Exmple. Figure 4() shows n FFA. An equivlent SFA is shown in Figure 4(). We will explin the WSFA of Figure 4() lter. The three tles re shown in Figure 4().. The new snner river Figure is the snner river. It mkes use of the stte trnsition tle for hnging the sttes of the SFA. A vrile token is use to reor the token ssoite with the lst epting stte tht the SFA psses through uring the trnsitions. Let s e the stte when the SFA nnot mke further trnsition. The token in the vrile token is reognize n printe. The suffix lel of stte s represents the previewe symols. When the suffix lel is fe into the SFA, we know tht the tokens in output [s ] will e reognize. Hene, they re printe one y one. Finlly, the SFA will reh the stte speifie y ontinution [s ]. Ation [s ] is the token ssoite with the lst epting stte when the suffix lel of s is fe into the SFA, fter ll the tokens in output [s ] re reognize. () FFA strt () SFA strt () WSFA strt () CON, ACT, n OUT tles sttes CON error 6 7 ACT NULL NULL T6 T7 NULL NULL NULL T NULL T6 T7 OUT nil nil nil nil nil nil nil nil T nil nil Figure 4. An FFA, its equivlent SFA n WSFA, the ontinution, tion, n output tles of the SFA.

6 6 Algorithm: New-Snner-Driver /* Given the stte trnsition tle ST, the ontinution tle CON, the tion tle*/ /* ACT, n the output tle OUT of n SFA, the snner river groups symols*/ /* from input into tokens. We ssume tht there is no trnsition from ny stte*/ /* when the next input symol is the en-of-file symol. */ urrent_stte := the initil stte of the SFA token := NULL next_symol := next symol from input repet if ST [urrent_stte, next_symol ] no-trnsition then egin urrent_stte := ST [urrent_stte, next_symol ] if urrent_stte is n epting stte then token := the token ssoite with the stte urrent_stte next_symol := next symol from input en else /* no trnsition is possile t this point. */ if token = NULL then lexil_error () else egin print token print the tokens in OUT [urrent_stte ] token := ACT [urrent_stte ] if CON [urrent_stte ] is error then lexil_error () else urrent_stte := CON [urrent_stte ] en until next_symol = en-of-file n urrent_stte = the initil stte of the SFA Figure. The new snner-river. In the suffix FA, ll suffixes of stte rry the sme lel. This requirement is too strong for the purpose of lexil nlysis. For instne, stte in the FFA of Figure 4() is split into three sttes, 8, n 10 ue to ifferent suffixes. However, sttes n 10 my e omine into single stte sine their entries in the ontinution, tion, n output tles re the sme, respetively. Similrly, sttes 4 n 11 my e omine. The resulting FA is shown in Figure 4(). Bse on the ove oservtion, we introue the notion of wek suffix FA. 6. Conlusion n relte work The look-he prolem is exmine in etil in this pper. It is foun tht tritionl snners se on minimum eterministi finite utomt solve the look-he prolem in n ho wy. We propose new tle-riven snner tht solves the look-he prolem utomtilly. There re mny pulishe reports on lexil nlysis. They inlue lex, sngen, flex, TOOLS, Alex, LexAGen, Nwroki s work, GLA, Rex, Mksn, ALADIN, n Glxy. (Referenes to these works re ville from the uthor.) Our pproh to the look-he prolem is similr in spirit to the string pttern mthing lgorithm [] n the string mthing lgorithm [1]. REFERENCES 1. Aho, A.V. n Corsik, M.J., Effiient string mthing: An i to iliogrphi serh, Comm. ACM 18(6) pp. -40 (June 197).. Fisher, C.N. n LeBln, R.J. Jr., Crfting Compiler with C, Benjmin/Cummings, Reing, MA (1991).. Knuth, D.E., Morris, Jr., J.H., n Prtt, V.R., Fst pttern mthing in strings, SIAM J. on Computing 6()(1977).