Lexical analysis, scanners. Construction of a scanner

Transcription

1 Lexicl nlysis scnners (NB. Pges 4-5 re for those who need to refresh their knowledge of DFAs nd NFAs. These re not presented during the lectures) Construction of scnner Tools: stte utomt nd trnsition digrms. Regulr expressions enle the utomtic construction of scnners. Function. Red the input strem (sequence of chrcters) group the chrcters into primitives (tokens). Returns token s <type vlue>.. Throw out certin sequences of chrcters (s comments etc.). 3. Build the symol tle string tle constnt tle etc. 4. Generte error messges. 5. Convert for exmple string integer. Tokens re descried using regulr expressions. Scnner genertor (e.g. Lex): In: Regulr expression. Out: Scnner. Environment: Scnner Give me the next token Find nd return the next token Prser Lecture Lexicl nlysis Pge 4 Lecture Lexicl nlysis Pge 4 Finite stte utomt nd digrms (Finite utomton) Assume: regulr expression RU = + + = L(RU) = { n m n m } Recognizer A progrm which tkes string x nd nswers yesno depending on whether x is included in the lnguge. The first step in constructing recognizer for the lnguge L(RU) is to drw stte digrm (trnsition digrm). Method of opertion:. Strt in the strting node.. Repet until there is no more input: ) Red input. ) Follow suitle edge. 3. When there is no more input: Check whether we re in finl stte. In this cse ccept the string. There is n error in the input if there is no suitle edge to follow. Tke this mesure: Add one or severl error nodes. Exmple of input: strt 9 3 Step Current stte Input ε error stte ccepting stte stte digrm (DFA) for n m Then ccept ecuse there is no more input nd stte 3 is n ccepting stte. Lecture Lexicl nlysis Pge 43 Lecture Lexicl nlysis Pge 44

2 Stte digrms re represented y trnsition tles: Exmple: NFA for ( ) * Trnsition tle Stte Accept Found Next stte no ε 9 no 9 no yes no? 9 strt stte digrm for ( ) * (Suitle for computer representtion). The previous grph is DFA (Deterministic Finite Automton). It is deterministic ecuse t ech step there is exctly one stte to go to nd there is no trnsition mrked ε. A regulr expression denotes regulr set nd corresponds to n NFA (Nondeterministic Finite Automton). stte Accept {} {} no {} no yes Trnsition tle for ( ) * It requires more clcultions to simulte n NFA with computer progrm e.g. for input. Lecture Lexicl nlysis Pge 45 Lecture Lexicl nlysis Pge 46 Theorem Any NFA cn e trnsformed to corresponding DFA. When generting scnner utomticlly the following is done: regulr expression NFA. NFA DFA. DFA miniml DFA. DFA corresponding progrm code or tle. DFA for ( ) * Exmple: Design DFA which cn recognize wht cn follow INTEGER in FORTRAN declrtion: INTEGER A INTEGER X Y I(3) INTEGER B(3 J K) L Alphet Σ = { v c ( ) } (v = vrile c = integer constnt) singlev = v v ( (v c)( (v c)) * ) singlev denotes single vrile or rry. strt r.e. = singlev ( singlev) * stte = strt = vrile is found = vrile is found then ( 3 = vrile is found then ( then vrile which sttes dimension 4 = vrile is found then ( then integer constnt which sttes dimension 5 = ) fter 3 4 mrks end of dimension 6 = fter 3 4 mrks dimension seprtor 7 = fter 5 mrks vrile seprtor E = error stte Lecture Lexicl nlysis Pge 47 Lecture Lexicl nlysis Pge 48

3 3 ) v v strt v ( 6 5 c c v 4 ) 7 E which gives us: Optimized trnsition tle stte v c ( ) Accept E E E E no E E E yes 3 3 E E E no 3 E E E 4 no 4 E E E E yes E E E E E E no NB. Not miniml DFA. Cn e optimized y merging sttes. Study the trnsition tle: This corresponds to miniml DFA elow: Trnsition tle stte v c ( ) Accept E E E E no E E E 7 yes 3 4 E E E no 3 E E E 5 6 no 4 E E E 5 6 no 5 E E E E 7 yes E E E no 7 E E E E no E E E E E E no strt v ( E v c 3 ) 4 Merge ll similr sttes (similr rows). Note tht you cnnot merge two sttes if only one of them is n ccepting stte. ( = 7 = 6 3 = 4) Re-numer the sttes Lecture Lexicl nlysis Pge 49 Lecture Lexicl nlysis Pge 5 How is scnner progrmmed?. Descrie tokens with regulr expressions.. Drw trnsition digrms. 3. Code the digrm s tleprogrm. Exmple. Write scnner for the following tokens. Severl ctegories of tokens: keyword = BEGIN END id = letter (letter ) * integer = + op = + - * = := Simplifiction: Assume tht there is chrcter fter ech token. This simplifiction cn esily e removed! The scnner represents tokens s tuples: Tuple type < Typecode vlue > undef. < > id < tle-pointer > integer < vlue > BEGIN < 3 > END < 4 > + < 5 > - < 6 > * < 7 > < 8 > < 9 > < > = < > := < > Prcticl trick for -chrcter tokens: Initilize vector chtyp A B + - * = Lecture Lexicl nlysis Pge 5 Lecture Lexicl nlysis Pge 5

4 Drw the trnsition digrm: Comments: strt letter letter tuple( convert()) tuple( lookup()) or tuple(3 ) or tuple(4 ) tuple(8 ) convert() converts text to integers. lookup() returns index to symol tle. BEGIN END delt with y putting them in the symol tle from the eginning. When they re found return 3 or 4 insted of. ch lwys contins the next chrcter prv_ch lwys contins the next to the lst chrcter. Automtic trnsition to stte fter ech recognized token (even fter stte 4). 7 8 tuple(9 ) : 9 = tuple( ) +-* = 3 tuple( chtyp(prv_ch) ) not E 4 tuple( ) Lecture Lexicl nlysis Pge 53 Lecture Lexicl nlysis Pge 54 Coding. Trnspose the digrm to trnsition tle perform simple interprettion of the tle. Different vrints of coding the tle ) Interpreting the tle Exmple: Trnsition- digrm nd -tle for integers: strt tuple( convert()) E not stte Blnk Other Accept E flse E flse true E E E flse stte Blnk Other Accept E flse E flse true E E E flse Red in or initite tle tle[sttech]; ch := getchr; stte := ; (* strte stte *) while not eof do egin oldstte := stte; stte := tle[stte ch]; ccumulte(ch); if stte = error_stte then error_hndling(oldstte ch ) else if ccept[stte] then perform_ccept(stte); (* return tuple nd stte := *) ch := getchr; end; Lecture Lexicl nlysis Pge 55 Lecture Lexicl nlysis Pge 56

5 ) GOTO-representtion of tle stte Blnk Other Accept E E E E E stte: ch := getchr; if ch = then goto stte; if ch = then goto stte; goto sttee; (* in cses *) stte: c) using CASE sttement cse stte of : cse ch of.. 9 :stte := ; : stte := ; s: stte := E; end; :. Direct coding of digrms (not vi tle) Glol vriles: toktyp = current symol clss vlue = vlue ch = current chrcter eofile = end-of-file flg chtyp = vector for -chrcter tokens symt = symol tle Routines: getchr; skip_s; ccumulte(ch); lookup(id); letter(ch); (ch); Initite: eofile := flse; initite chtyp ccording to the previous description; initite the symol tle with reserved words; strt E not tuple( convert()) Lecture Lexicl nlysis Pge 57 Lecture Lexicl nlysis Pge 58 Incomplete suggested progrm for the scnner ch := getchr; vl := ; while not eofile do egin skip_s; cse ch of A.. Z : ch := getchr; while letter(ch) or (ch) do ch := getchr; if ch = then lookup(ccumulted string) else error();.. 9 : vl := ord(ch) - ord( ); ch := getchr; while (ch) do egin vl:=vl*+ord(ch)-ord( ); ch := getchr; end; if ch = then toktyp := else error(); s: toktyp := chtyp[ch]; if toktyp = then error(); end (* cse *); end (* while *); Simplifiction removed (i.e. not necessrily concluding chrcter): strt letter : +-* = letter not tuple( convert()) tuple( lookup()) or tuple(3 ) or tuple(4 ) tuple(8 ) tuple(9 ) = 6 E 8 tuple( ) tuple(chtyp(prv_ch) ) tuple( ) Lecture Lexicl nlysis Pge 59 Lecture Lexicl nlysis Pge 6

6 Prolem: Lookhed is sometimes needed to determine symol type. Exmple: in FORTRAN DO I =.5 is n ssignment ut DO I = 5 is for-sttement. It is. or which determines whether the scnner returns DOI or DO Exmple: in Pscl Lecture Lexicl nlysis Pge 6