Dr. D.M. Akbar Hussain

Transcription

1 Dr. D.M. Akr Hussin Lexicl Anlysis. Bsic Ide: Red the source code nd generte tokens, it is similr wht humns will do to red in; just tking on the input nd reking it down in pieces. Ech token is sequence of chrcters representing some informtion. Possile Types of Token: Keywords: if, while, Specil Symols: +, *, <, >.. Identifiers: User defined strings of chrcters.. Scnning is specil cse of pttern mtching: Method used re Regulr Expressions nd Finite Automt. 1 Lexicl Anlysis. It is simple ut time consuming process: Must e efficient Responsiility is to recognize tokens nothing else: int foo 1; foo int 1 ; Wht is the out come? Is this its jo? (Syntx nlysis.) Lexicl nlysis my lso perform one or more of the following: Deleting comments. Inserting line numers. Evluting constnts. (this is controversil, whether it should e done here or not) Compiler Construction F6S/Chpter 1

2 Dr. D.M. Akr Hussin Scnning Genertor #include.. #include min() { chr in; in getch ( ); if ( islph (in) ) in getch ( ); else error (); while ( islph (in) isdigit (in) ) in getch ( ); } For n identifier Regulr Expressions (RE) RE represent pttern of strings composed of chrcters. It is very convenient method to represent identifiers nd constnt. A lnguge is set of strings. String is finite sequence of symols. Symols themselves re tken from finite lphet. RE r is defined y set of strings with which it mtches: L(r) Which mens r contin chrcters (could e lphet + specil symols e.g. met-chrcters). Some Bsic RE: L(x) {x} Expression mtches x. L() {} Empty string with no chrcters. L(Ø) { } Ø mtches no string (empty set). 4 Compiler Construction F6S/Chpter

3 Dr. D.M. Akr Hussin RE Opertions Choice (lterntion) Conctention Repetition Choice: Given two RE M nd N, it is written with the lterntion opertor (verticl r) s M N. A string is in the lnguge M N if it is in the lnguge M or in the lnguge N. It is union, e.g., for lnguge it contins strings nd. L( ) L() U L() {,} Conctention: Given two RE M nd N, conctention opertor mkes new RE M.N. Which mens string is in the lnguge M.N if it is the conctention of ny two strings nd such tht is in lnguge M nd is in the lnguge N. e.g.: ( ). defines the lnguge contining nd. e.g.: S1 {, }, S {, } S1.S {,,, } 5 RE Opertions Repetition: Given RE M its repetition is M* (lso clled kleene closure). A string is in the lnguge M* if it is conctention of zero or more strings ll of which re in M. e.g.: * mtches,,,,,. * {}UUUU.. U * n 0 Bsiclly it is infinite set of union. Rememer ech element is finite conctention of strings. n e.g. ( )*,,,,,,,, 0 e.g. (0 1)*.0 0, 00,10,010, (Binry numers) 6 Compiler Construction F6S/Chpter

4 Dr. D.M. Akr Hussin RE Opertions *?? ( )* ()* ( )*,,,,,,, ()*,,,,,. * Hs the highest precedence then conctention nd lterntion Some short cuts (revitions): c mens (.) c ( ) mens ( ) [cd] men ( c d) [ - g] mens [cdefg] 7 RE Opertions Extension Why it is required: Exmple: * repetition zero or more, this could e prolem for nturl numers, we need to hve one instnce in which it gurntees tht we hve mtch, disllowing the empty string. Binry numer exmple: (0 1)* Although, we cn solve it y: (0 1) (0 1)*, 0, 00, 11,.. But this is simple sitution nd it is esy, ut things my not e tht simple in lnguge. So the solution is M + (M.M*). 8 Compiler Construction F6S/Chpter 4

5 Dr. D.M. Akr Hussin Extension RE Opertions Another Sitution Demnds: ny chrcter Common requirement is to hve mtch of ny chrcter in the lphet. Which mens we hve to literlly write ech chrcter with lterntion. The extension is to use met chrcter. period, which gives the option not to put every chrcter lterntive. Exmple:.*.* lest one. True for ll strings contining t 9 Rnge of Chrcters Usully we do like this for rnge of chrcters: c d.. z So the extension is [ - z] [0-9] Any Chrcter not in given set: It is not possile to get to sitution, where we hve RE for ll chrcters except one chrcter. Tilde or Crt cn e used. For exmple: ~ chrcter which is not ~( c) chrcter which is neither, or c. In Lex: ^, ^( c) Optionl Occurrence: M? (M ) 10 Compiler Construction F6S/Chpter 5

6 Dr. D.M. Akr Hussin RE Nottions An ordinry chrcter stnds for itself The empty string M N Alterntion choosing from M or N M.N / MN Conctention, n M followed y N M* Repetition zero or more times M + Repetition one or more times M? Optionl, Zero or one occurrence of M [-za-z] Chrcter set lterntion. Period, ny single chrcter except new line ~M /^M A chrcter which is not M (Tilde or Crt).+* Quottion, string in quotes stnds for itself literlly 11 Finite Automt RE re very convenient for specifying token, ut some formlism is required to implement it. Finite utomt is mthemticl wy of descriing mchines. In prticulr for the process of recognizing input tokens FA is the proper formlism. FA re the est construct to implement scnners. Forml Definition: FA is finite utomton with finite set of sttes. Edges led from one stte to other sttes. Ech edge is leled with symol. One is unique strt stte nd some re finl sttes. 1 Compiler Construction F6S/Chpter 6

7 Dr. D.M. Akr Hussin Finite Automt strt ccept letter [ za Z] digit [0 9] letter letter 1 Ole, Ole004 digit O 1 l e O 1 l e Exmple Finite Automt for Rel digit digit digit Compiler Construction F6S/Chpter 7

8 Dr. D.M. Akr Hussin Deterministic Finite Automt (DFA) In DFA no two edges leving from sme stte re leled with the sme symol. It follows: Strting from the strt stte utomton follows exctly one pth (edge) to get to the next stte. The edge hs to e leled with the input chrcter. After mking n trnsitions for n long chrcter string, if the utomton is in the finl stte. It ccepts it otherwise rejects. A - Z Fox exmple for n ID: A - Z Deterministic Finite Automt (DFA) Lst exmple hs no such provision for exmple if there is chrcter other thn lphet. A - Z A - Z 1 ID other other 0-9 error 16 Compiler Construction F6S/Chpter 8

9 Dr. D.M. Akr Hussin Exmples of DFA Signed nturl numer (+ -)?digit Signed numer (+ -)?digit.?digit Exmples of DFA * * 1 ( )* Compiler Construction F6S/Chpter 9

10 Dr. D.M. Akr Hussin DFA Conventions Error trnsitions re not explicitly shown. Input symols tht result in the sme trnsition re grouped together (this set cn even e given nme). Still not displyed: stopping conditions nd ctions. Principle of Longest Su-string (or Mximl Munch): o The DFA mtches the longest possile input string efore stopping. 19 Non Deterministic Finite Automt (NFA) DFA ctully does not represent every thing, it provide us n outline of its opertions. Mthemticlly, DFA must hve trnsition for every stte nd chrcter. Wht ction is to e tken when n error occurs. Wht ction is to e tken when n ccepting condition reched. 0 Compiler Construction F6S/Chpter 10

11 Dr. D.M. Akr Hussin Non Deterministic Finite Automt (NFA) 6 Bsiclly NFA is n utomton which hs choice of edges, leled with the sme symols to follow out of stte. Or it cn hve specil edge leled with tht cn e followed without eting ny symol from the input (without consulting the input) NFA Exmple Compiler Construction F6S/Chpter 11

12 Dr. D.M. Akr Hussin How to NFA Suppose we hve three tokens :, <, : : : < < How to NFA : : < < Suppose we hve tokens <, <>, < cn we do tht < < < > < <> < 4 Compiler Construction F6S/Chpter 1

13 Dr. D.M. Akr Hussin How to NFA Solution: < < > <> others < 5 How to NFA : : < < 6 Compiler Construction F6S/Chpter 1

14 Dr. D.M. Akr Hussin NFA [+ * *] The definition of NFA is similr ut we need to expnd the lphet set to include. 1 which cn e ccepted y ny of the Following sequences: NFA c RE ( c)* c c Compiler Construction F6S/Chpter 14

15 Dr. D.M. Akr Hussin RE to DFAs RE nd DFAs re equivlent. However RE re preferred over DFAs ecuse of their compctness in token description. Typiclly, scnner genertion strts with RE nd through DFAs finlly end with scnner progrm. Generlly, this trnsltion construct n intermedite construction using NFA which finlly construct the DFA. There is possiility of direct construction from RE to DFA ut generlly not preferred ecuse of complexity. RE NFA DFA PRO 9 Thomson Construction It construct mchine y using the trnsition to glue together the mchines for ech piece of RE, which corresponds to the whole expression. 0 Compiler Construction F6S/Chpter 15

16 Dr. D.M. Akr Hussin Thomson Construction Conctention: RE MN so Thomson construction follows: M M N Alterntion/Choice: RE M N so Thomson construction follows: M N 1 Thomson Construction Repetition: RE M* so Thomson construction follows: M Thomson construction is not unique, there re other possiilities e.g., for MN: M N However, it is only possile if the ccepting stte hs no trnsitions to other sttes. Compiler Construction F6S/Chpter 16

17 Dr. D.M. Akr Hussin Suset Construction (NFA to DFA) Oviously, first thing to remove is trnsitions nd second the multiple trnsitions from stte on single chrcter input. 1 For exmple: RE M* 4 closure of single stte S is set of sttes rechle y series of zero or more trnsitions nd denoted y S. 1 {1,, 4}, {}, {,, 4}, 4 {4} {1,, 4} {,, 4} Suset Construction Exmple RE * 1 {, 6, 1}, {}, {, 4}, 4 {4}, 5 {8, 5}, 6 {6} 7 {8, 7}, 8 {8} {, 6, 1} {, 4, 7, 8} {5, 8} 4 Compiler Construction F6S/Chpter 17

18 Dr. D.M. Akr Hussin Another Exmple (First NFA) RE ( )* Suset Construction 0 {0,1,,4,7} 1 {1,,4} {} {,6,1,,4,7} 4 {4} 5 {5,6,1,,4,7} 6 {6,1,,4,7} 7 {7} 8 {8} 9 {9} 10 {10} {0,1,,4,7} {1,,4,5,6,7} {1,,,4,6,7,8} {1,,4,5,6,7,9} {1,,4,5,6,7,10} 6 Compiler Construction F6S/Chpter 18

19 Dr. D.M. Akr Hussin Minimiztion of Sttes in DFA For *: We hve seen tht derivtion from RE to DFA result in more sttes (complex). Building scnner with more sttes is not going to e pprecited. But good news is tht FA theory sttes tht for given DFA there is n equivlent DFA contining minimum sttes. This minimum stte DFA is unique. 7 Minimiztion of Sttes in DFA Divide the given set of sttes into: Accepting sttes Not ccepting sttes ( )* Actul DFA 1 Minimized Version {1} {, } 8 Compiler Construction F6S/Chpter 19

20 Dr. D.M. Akr Hussin Exmple DFA for C comments other / * * / other other * / * * / other? 9 TINY DFA For Numers nd Identifiers digit INNUM white spce digit letter letter INID [other] [other] START DONE + - * / < ( ) ; 40 Compiler Construction F6S/Chpter 0

21 Dr. D.M. Akr Hussin TINY DFA For Scnner digit INNUM START white spce digit letter : letter INID INASSIGN [other] [other] DONE [other] { } other INCOMMENT other 41 Compiler Construction F6S/Chpter 1