Scanner Termination. Multi Character Lookahead. to its physical end. Most parsers require an end of file token. Lex and Jlex automatically create an

Transcription

1 Scnner Termintion A scnner reds input chrcters nd prtitions them into tokens. Wht hppens when the end of the input file is reched? It my be useful to crete n Eof pseudo-chrcter when this occurs. In Jv, for exmple, InputStrem.red(), which reds single byte, returns -1 when end of file is reched. A constnt, EOF, defined s -1 cn be treted s n extended ASCII chrcter. This chrcter then llows the definition of n Eof token tht cn be pssed bck to the prser. An Eof token is useful becuse it llows the prser to verify tht the logicl end of progrm corresponds to its physicl end. Most prsers require n end of file token. Lex nd Jlex utomticlly crete n Eof token when the scnner they build tries to scn n EOF chrcter (or tries to scn when eof() is true) Multi Chrcter Lookhed We my llow finite utomt to look beyond the next input chrcter. This feture is necessry to implement scnner for FORTRAN. In FORTRAN, the sttement O 10 J = 1,100 specifies loop, with index J rnging from 1 to 100. The sttement O 10 J = is n ssignment to the vrible O10J. (Blnks re not significnt except in strings.) A FORTRAN scnner decides whether the O is the lst chrcter of O token only fter reding s fr s the comm (or period). A milder form of extended lookhed problem occurs in Pscl nd Ad. The token is rel literl, wheres is three different tokens. We need two-chrcter lookhed fter the 10 prefix to decide whether we re to return 10 (n integer literl) or ( rel literl)

2 Suppose we use the following FA.... Given we scn three chrcters nd stop in nonccepting stte. Whenever we stop reding in nonccepting stte, we bck up long ccepted chrcters until n ccepting stte is found. Chrcters we bck up over re rescnned to form lter tokens. If no ccepting stte is reched during bckup, we hve lexicl error. Performnce Considertions Becuse scnners do so much chrcter-level processing, they cn be rel performnce bottleneck in production compilers. Speed is not concern in our project, but let s see why scnning speed cn be concern in production compilers. Let s ssume we wnt to compile t rte of 1000 lines/sec. (so tht most progrms compile in just few seconds). Assuming 30 chrcters/line (on verge), we need to scn 30,000 chr/sec On 30 SPECmrk mchine (30 million instructions/sec.), we hve 1000 instructions per chrcter to spend on ll compiling steps. If we llow 25% of compiling to be scnning ( compiler hs lot more to do thn just scn!), tht s just 250 instructions per chrcter. A key to efficient scnning is to group chrcter-level opertions whenever possible. It is better to do one opertion on n chrcters rther thn n opertions on single chrcters. In our exmples we ve red input one chrcter s time. A subroutine cll cn cost hundreds or thousnds of instructions to execute fr too much to spend on single chrcter. We prefer routines tht do block reds, putting n entire block of chrcters directly into buffer. Specilized scnner genertors cn produce prticulrly fst scnners. The GLA scnner genertor clims tht the scnners it produces run s fst s: while(c!= Eof) { c = getchr(); }

3 Lexicl Error Recovery A chrcter sequence tht cn t be scnned into ny vlid token is lexicl error. Lexicl errors re uncommon, but they still must be hndled by scnner. We won t stop compiltion becuse of so minor n error. Approches to lexicl error hndling include: elete the chrcters red so fr nd restrt scnning t the next unred chrcter. elete the first chrcter red by the scnner nd resume scnning t the chrcter following it. Both of these pproches re resonble. The first is esy to do. We just reset the scnner nd begin scnning new. The second is bit hrder but lso is bit sfer (less is immeditely deleted). It cn be implemented using scnner bckup. Usully, lexicl error is cused by the ppernce of some illegl chrcter, mostly t the beginning of token. (Why t the beginning?) In these cse, the two pproches re equivlent The effects of lexicl error recovery might well crete lter syntx error, hndled by the prser. Consider...for$tnight... The $ termintes scnning of for. Since no vlid token begins with $, it is deleted. Then tnight is scnned s n identifier. In effect we get...for tnight... which will cuse syntx error. Such flse errors re unvoidble, though syntctic error-repir my help. Error Tokens Certin lexicl errors require specil cre. In prticulr, runwy strings nd runwy comments ought to receive specil error messges. In Jv strings my not cross line boundries, so runwy string is detected when n end of line is red within the string body. Ordinry recovery rules re inpproprite for this error. In prticulr, deleting the first chrcter (the double quote chrcter) nd restrting scnning is bd decision. It will lmost certinly led to cscde of flse errors s the string text is inppropritely scnned s ordinry input

4 One wy to hndle runwy strings is to define n error token. An error token is not vlid token; it is never returned to the prser. Rther, it is pttern for n error condition tht needs specil hndling. We cn define n error token tht represents string terminted by n end of line rther thn double quote chrcter. For vlid string, in which internl double quotes nd bck slshes re escped (nd no other escped chrcters re llowed), we cn use " ( Not( " Eol \ ) \" \\ )* " For runwy string we use " ( Not( " Eol \ ) \" \\ )* Eol (Eol is the end of line chrcter.) When runwy string token is recognized, specil error messge should be issued. Further, the string my be repired into correct string by returning n ordinry string token with the closing Eol replced by double quote. This repir my or my not be correct. If the closing double quote is truly missing, the repir will be good; if it is present on succeeding line, cscde of inpproprite lexicl nd syntctic errors will follow. Still, we hve told the progrmmer exctly wht is wrong, nd tht is our primry gol In lnguges like C, C++, Jv nd CSX, which llow multiline comments, improperly terminted (runwy) comments present similr problem. A runwy comment is not detected until the scnner finds close comment symbol (possibly belonging to some other comment) or until the end of file is reched. Clerly specil, detiled error messge is required. Let s look t Pscl-style comments tht begin with { nd end with }. Comments tht begin nd end with pir of chrcters, like /* nd */ in Jv, C nd C++, re bit trickier. Correct Pscl comments re defined quite simply: { Not( } )* } To hndle comments terminted by Eof, this error token cn be used: { Not( } )* Eof We wnt to hndle comments unexpectedly closed by close comment belonging to nother comment: {... missing close comment... { norml comment }... We will issue wrning (this form of comment is lexiclly legl). Any comment contining n open comment symbol in its body is most probbly missing } error

5 We split our legl comment definition into two token definitions. The definition tht ccepts n open comment in its body cuses wrning messge ("Possible unclosed comment") to be printed. We now use: { Not( } )* } nd { (Not( { } )* { Not( { } )* )+ } The first definition mtches correct comments tht do not contin n open comment in their body. The second definition mtches correct, but suspect, comments tht contin t lest one open comment in their body. Single line comments, found in Jv, CSX nd C++, re terminted by Eol. They cn fll prey to more subtle error wht if the lst line hs no Eol t its end? The solution? Another error token for single line comments: // Not(Eol) * This rule will only be used for comments tht don t end with n Eol, since scnners lwys mtch the longest rule possible Regulr Expressions nd Finite Automt Regulr expressions re fully equivlent to finite utomt. The min job of scnner genertor like JLex is to trnsform regulr expression definition into n equivlent finite utomton. First it trnsforms regulr expression into nondeterministic finite utomton (NFA). Unlike n ordinry deterministic finite utomton, n NFA need not mke unique (deterministic) choice of successor stte to visit. For exmple, s shown below, n NFA is llowed to hve stte tht hs two trnsitions (rrows) coming out of it, lbeled by the sme symbol. An NFA my lso hve trnsitions lbeled with λ. λ Trnsitions re normlly lbeled with individul chrcters in Σ, nd lthough λ is string (the string with no chrcters in it), it is definitely not chrcter. In the bove exmple, when the utomton is in the stte t the left nd the next input chrcter is, it my choose to use the

6 trnsition lbeled or first follow the λ trnsition (you cn lwys find λ wherever you look for it) nd then follow n trnsition. FAs tht contin no λ trnsitions nd tht lwys hve unique successor sttes for ny symbol re deterministic. 165