In the last lecture, we discussed how valid tokens may be specified by regular expressions.

Transcription

1 LECTURE 5 Scnning

2 SYNTAX ANALYSIS We know from our previous lectures tht the process of verifying the syntx of the progrm is performed in two stges: Scnning: Identifying nd verifying tokens in progrm. Prsing: Identifying nd verifying the ptterns of tokens in progrm.

3 LEXICAL ANALYSIS In the lst lecture, we discussed how vlid tokens my e specified y regulr expressions. In this lecture, we will discuss how we cn go out identifying tokens tht re specified y regulr expressions.

4 RECOGNIZING TOKENS A recognizer for lnguge is progrm tht tkes string x s input nd nswers yes if x is sentence of the lnguge nd no otherwise. In the context of lexicl nlysis, given string nd regulr expression, recognizer of the lnguge specified y the regulr expression nswers yes if the string is in the lnguge. How cn we recognize regulr expression (int)? Wht out (int for)? We could, for exmple, write n d hoc scnner tht contined simple conditions to test, the ility to peek hed t the next token, nd loops for numerous chrcters of the sme type.

5 RECOGNIZING TOKENS For exmple, to recognize (int for): if cur_chr == i : pek t next chrcter if it is n : pek t next chrcter if it is t : return int if cur_chr == f : pek t next chrcter if it is o : pek t next chrcter if it is r : return for else error

6 FINITE AUTOMATA A set of regulr expressions cn e compiled into recognizer utomticlly y constructing finite utomton using scnner genertor tools (lex, for exmple). The dvntge of using n utomticlly generted scnner over n d-hoc implementtion is ese-of-modifiction. If there re ever ny chnges to my token definitions, I only need to updte my regulr expressions nd regenerte my scnner. Where performnce is concern, hnd-written scnners or optimized scnners my e necessry ut for development scnner genertors re most useful.

7 FINITE AUTOMATA A finite utomton is simple idelized mchine tht is used to recognize ptterns within some input. A finite utomton will ccept or reject n input depending on whether the pttern defined y the finite utomton occurs in the input. The elements of finite utomton, given set of input chrcters, re A finite set of sttes (or nodes). A specilly-denoted strt stte. A set of finl (ccepting) sttes. A set of leled trnsitions (or rcs) from one stte to nother.

8 FINITE AUTOMATA Here s n exmple finite utomton tht ccepts ny sequence of 1 s nd 0 s which hs n odd numer of 1 s s f Strting stte 1 Finl stte Trnsitions leled with input

9 FINITE AUTOMATA We ccept string only if its chrcters provide pth from the strting stte to some finl stte s f Strting stte 1 Finl stte Trnsitions leled with input

10 FINITE AUTOMATA Finite utomt come in two flvors. Deterministic Never ny miguity. For ny given stte nd ny given input, only one possile trnsition. Non-deterministic There my e more thn one trnsition from ny given stte for ny given chrcter. There my e epsilon trnsitions trnsitions leled y the empty string. There is no ovious lgorithm for converting regulr expressions to DFAs.

11 FINITE AUTOMATA Typiclly scnner genertors crete DFAs from regulr expressions in the following wy: Crete NFA equivlent to regulr expression. Construct DFA equivlent to NFA. Minimize the numer of sttes in the DFA.

12 CONSTRUCTING NFAS Let s sy we hve some regulr expression tht specifies the tokens we llow in our progrmming lnguge. How do we turn this into n NFA? Well, there re couple of sic uilding locks we cn use. Let s sy our regulr expression is simply, mening we only ccept one instnce of the chrcter s token. s f Similrly, we could hve two stte NFA tht ccepts the empty string.

13 CONSTRUCTING NFAS Conctention: s f Alterntion: s f

14 CONSTRUCTING NFAS Kleene Closure: * s f

15 CONSTRUCTING NFAS There re three importnt properties of these sic NFAs tht we should tke note of: There re no trnsitions ck into the initil stte. There is single finl stte. There re no trnsitions out of the finl stte. Becuse of these invrint properties, we cn comine smller NFAs to crete lrger NFAs. When trnslting regulr expression into n NFA, strt with smll components of the regulr expression nd use them to crete lrger constructions.

16 CONSTRUCTING NFAS Let s put this ll together. Let s sy I hve the following regulr expression: ( ( ) c )* which descries the set {, c, c, cc, cc, cc, cc, ccc.} Let s strt with ( ). We lredy know the corresponding NFA is: s f

17 CONSTRUCTING NFAS Let s put this ll together. Let s sy I hve the following regulr expression: ( ( ) c )* which descries the set {, c, c, cc, cc, cc, cc, ccc.} Conctenting ( ) with c gives us: s c f

18 CONSTRUCTING NFAS Let s put this ll together. Let s sy I hve the following regulr expression: ( ( ) c )* which descries the set {, c, c, cc, cc, cc, cc, ccc.} Adding in the Kleene closure gives us the NFA for ( ( ) c )*: s c f

19 FROM NFAS TO DFAS If we cn esily convert our regulr expressions into n NFA, then why do we need to further convert it to DFA? Well, NFAs dmit numer of trnsitions from single stte for the sme input. For exmple, in the lst NFA, we cn move to stte tht dmits or stte tht dmits on n empty string. To implement n NFA, we d need to explore ll possile trnsitions to find the right pth. Insted, we implement DFA which, for given input, moves to stte tht represents the set of sttes we could rrive t in n equivlent NFA.

20 FROM NFAS TO DFAS Let s look t our NFA gin. Notice we ve dded lels to identify individul sttes c

21 FROM NFAS TO DFAS Before consuming ny input, we could e in the stte 1. But we could lso e in the sttes 2, 3, 5 nd 9 vi epsilon trnsitions. A [1,2,3,5,9] Lets mke strting stte clled A which is the set of ll possile strting sttes.

22 FROM NFAS TO DFAS If we receive n input, from A we could trnsition to 4 or 7 (vi epsilon trnsitions). A [1,2,3,5,9] B [4,7]

23 FROM NFAS TO DFAS If we receive n input, from A we could trnsition to 6 or 7 (vi epsilon trnsitions). A [1,2,3,5,9] B [4,7] C [6,7]

24 FROM NFAS TO DFAS If we receive n input c, we cnnot go nywhere from A, ut oth B nd C cn go to 2,3,5,8,9. A [1,2,3,5,9] B [4,7] c C [6,7] c D [2,3,5,8,9]

25 FROM NFAS TO DFAS From D, we cn go to B on n input of or go to C on n input of. A [1,2,3,5,9] B [4,7] c C [6,7] c D [2,3,5,8,9]

26 FROM NFAS TO DFAS Finlly, we ve grphed ll possile scenrios. Let s circle the finl sttes. A [1,2,3,5,9] B [4,7] c C [6,7] c D [2,3,5,8,9]

27 FROM NFAS TO DFAS Here s our finl DFA. For reference, our regulr expression ws ( ( ) c )* which descries the set {, c, c, cc, cc, cc, cc, ccc.}. A B c C c D

28 FROM NFAS TO DFAS It is possile tht the resulting DFA cn e minimized further. Strt y comining ll non-finl sttes together nd ll finl sttes together. A B c C c D

29 FROM NFAS TO DFAS Now comine ll of the non-finl sttes together. B c C c A D

30 FROM NFAS TO DFAS Now comine ll of the non-finl sttes together. B C c A D

31 FROM NFAS TO DFAS Note tht AD is our strting stte s well s n ccepting stte. Note tht, in some cses, we my hve to pull prt some sttes to void miguity, ut here there is none. B C c A D

32 FROM NFAS TO DFAS Here s nother simple exmple of DFA tht cn e minimized: digit digit. A B C digit

33 FROM NFAS TO DFAS This is simplistic exmple, ut we cn minimize this DFA in the following wy: digit digit A. B,C Be creful tht your minimiztion doesn t introduce miguity! Split sttes s needed until none remin.

34 FROM DFAS TO SCANNERS Now tht we hve our DFA, we cn ctully crete scnner. In the next lecture, we ll tlk out creting scnner tht models the structure of the DFA we creted.