CSE 401 Compilers. Agenda. Lecture 4: Implemen:ng Scanners Michael Ringenburg Winter 2013

Transcription

1 CSE 401 Compilers Leture 4: Implemen:ng Snners Mihel Ringenurg Winter 013 Winter 013 UW CSE 401 (Mihel Ringenurg) Agend Lst week we overed regulr expressions nd finite utomt. Tody, we ll finish our finl exmple (NFA to DFA onversion) nd then tlk out how snners re implemented. Wednesdy, we ll egin our disussion of prsing. Winter 013 UW CSE 401 (Mihel Ringenurg) 1

2 Announements Prt 1 of the projet (the snner) will e relesed tomorrow morning. If you or your prtner hven t emiled the ourse stff to let us know your tem, do so TODAY. If you hven t een le to find prtner, emil me nd I ll pir you up with someone else who hsn t. You n lso hek the disussion ord there hve een few posts y people looking for prtners. We urrently hve n even numer of students (54), so everyone should e le to hve prtner. Winter 013 UW CSE 401 (Mihel Ringenurg) 3 Convert NFA to DFA: Exmple Step 1: Find losure of strt stte: {1,,5} Winter 013 UW CSE 401 (Mihel Ringenurg) 4

3 Convert NFA to DFA: Exmple {1,,5} Step : Mke new DFA stte orresponding to this losure. Mrk it s unvisited (yellow in this digrm). Winter 013 UW CSE 401 (Mihel Ringenurg) 5 Convert NFA to DFA: Exmple {1,,5}??? Loop: As long s there re unvisited DFA nodes, pik one. Consider trnsi:ons from its orresponding NFA sttes for every symol in the lphet. Winter 013 UW CSE 401 (Mihel Ringenurg) 6 3

4 Convert NFA to DFA: Exmple {3} {1,,5}?? Only trnsi:on on from 1,, or 5 is to 3. Winter 013 UW CSE 401 (Mihel Ringenurg) 7 Convert NFA to DFA: Exmple {1,,5}? {3}? losure of {3} is just 3 (no trnsi:ons), so {1,,5} trnsi:ons to {3} on. This DFA stte does not exist yet, so mke it nd mrk it unvisited. Winter 013 UW CSE 401 (Mihel Ringenurg) 8 4

5 Convert NFA to DFA: Exmple {3} {1,,5}? No trnsi:ons from 1,, or 5 on symol. Winter 013 UW CSE 401 (Mihel Ringenurg) 9 Convert NFA to DFA: Exmple {1,,5} {3} {6} Only trnsi:on from 1,, or 5 on is to 6. Winter 013 UW CSE 401 (Mihel Ringenurg) 10 5

6 Convert NFA to DFA: Exmple {1,,5} {3} {6,7} Epsilon losure of {6} is {6,7}, so {1,,5} trnsi:ons to {6,7} on. This doesn t exist, so rete nd mrk unvisited. Winter 013 UW CSE 401 (Mihel Ringenurg) 11 Convert NFA to DFA: Exmple {1,,5} {3} {6,7} Done with {1,,5}. Mrk s visited (lk in our digrm). Winter 013 UW CSE 401 (Mihel Ringenurg) 1 6

7 Convert NFA to DFA: Exmple {3} {4,7} {1,,5} {6,7} Repet for nother unvisited node ({3}). Cretes {4,7}. Winter 013 UW CSE 401 (Mihel Ringenurg) 13 Convert NFA to DFA: Exmple {3} {4,7} {1,,5} {6,7} Repet for unvisited node ({4,7}). No trnsi:ons. Winter 013 UW CSE 401 (Mihel Ringenurg) 14 7

8 Convert NFA to DFA: Exmple {3} {4,7} {1,,5} {6,7} Repet for unvisited node {6,7}. No trnsi:ons. Winter 013 UW CSE 401 (Mihel Ringenurg) 15 Convert NFA to DFA: Exmple {3} {4,7} {1,,5} {6,7} No more unvisited nodes. Mrk s finl ll sttes whih inlude n NFA finl stte in their set. Winter 013 UW CSE 401 (Mihel Ringenurg) 16 8

9 Building Snner We ve seen the theory (RE to NFA to DFA), ut how is this onverted to pr:e? A snner needs to tke n input strem nd onvert it to tokens. Following the longest mth priniple i.e., uild the longest legl token str:ng t the urrent input posi:on. Then repet. Generl ide: Crete n RE for every token type. E.g., n RE for +, nd RE for integers, et. Build DFA for the union of the REs Modify DFA implement:on to reognize the longest mthing sustring (rther thn only ep:ng the whole string). This is some:mes free/unneessry for ertin DFAs Repetedly invoke (typilly y the prser to otin next token). Winter 013 UW CSE 401 (Mihel Ringenurg) 17 Snning DFA How does this modified DFA work? Must not just ept, ut ept nd tell us whih RE generted the string (i.e., whih token we found). Iden:fy the token y the finl stte we end in. Wht if our DFA finl stte orresponds to mul:ple REs from the originl? This n hppen if text mthes mul'ple tokens. E.g., for my mth the for keyword RE nd the iden:fier RE. Compiler writer must define priority order (e.g., keywords > IDs). Must lso find longest mth my get this for free If needed, run DFA un:l no more trnsi:ons. If not in finl stte, ktrk to lst seen finl stte. Not lwys neessry. Winter 013 UW CSE 401 (Mihel Ringenurg) 18 9

10 Punng it together A snner is DFA tht finds the next token eh :me it is lled (nd dvnes the input pointer to the token s end). Every finl stte of DFA emits (returns) token. For exmple: == eomes <equl> (not <ssign> <ssign>) ( eomes <leqpren> privte eomes <privte> Compiler writer (you) hoose the token nmes Also, there my e ddi:onl dt ssoited with tokens \r\n might ount lines; ll tokens might inlude line #; integer literls inlude vlue; et. Winter 013 UW CSE 401 (Mihel Ringenurg) 19 DFA => Code, y Hnd Op:on 1: One proedure per DFA stte Reds in hrter, nd uses swith sttement to determine the next stte to ll Finl sttes return token. Op:ons : Single proedure for DFA, swith sed on first hrter We ll see n exmple of this in few slides. Pros Firly strighvorwrd to write. If wriwen well, n e fster thn generted snners (pr:ulrly op:on ). Cn hndle ny weird lnguge orner ses tht don t mp perfetly to the RE/NFA/DFA model. Redle ode (mostly). Cons A lot of tedious work thus, error prone. Winter 013 UW CSE 401 (Mihel Ringenurg) 0 10

11 DFA => ode, utom: Op:on 1: use tool to generte tle driven snner Rows: sttes of DFA Columns: input hrters Entries: :on Go to next stte Aept token, go to strt stte Error Pros Convenient just feed it the token regulr expressions Extly mthes speifi:on you give it, if tool orret Cons Mgi Some:mes lnguge onstruts don t mp perfetly to FA model Not effiient Winter 013 UW CSE 401 (Mihel Ringenurg) 1 DFA => ode, utom: Op:on : use tool to generte diret- oded snner Trnsi:ons emedded in the ode, using ondi:onl sttements, loops, possily goto Pros Convenient just feed it the REs Extly mthes speifi:on you give it, if tool orret More effiient thn tle driven snners Cons Mgi Code is unredle Genertes lots of ode (ut n e firly fst) Winter 013 UW CSE 401 (Mihel Ringenurg) 11

12 The Rel World In ommeril senngs (nd most g front ends) hnd wriwen snners used more oqen thn not. Espeilly for lrger lnguges, e.g., C++/Jv. Cn purhse, e.g., EDG C/C++ front end (used y Cry, Intel, others). Why? Fstest Cn hndle lnguge orner ses C++ espeilly d. Redle/deuggle ode. Winter 013 UW CSE 401 (Mihel Ringenurg) 3 Exmple: A hnd- wriwen DFA nd snner To demonstrte, we ll show hnd- wriwen DFA for some typil progrmming lnguge onstruts Then use to onstrut hnd- wriwen snner Senng: Snner is lled whenever the prser needs new token Snner stores urrent posi:on in input From there, use DFA to reognize the longest possile input sequene tht mkes up token nd return tht token; sve updted posi:on for next :me Dislimer: Exmple for illustr:on only you ll use tools for the ourse projet. Credit: Hl Perkins wrote this DFA nd ode. Winter 013 UW CSE 401 (Mihel Ringenurg) 4 1

13 Snner DFA Exmple (1) 0 whitespe or omments Advne to end of whitespe nd/or omment efore we egin snning the next token. Comments n use DFA or simple proedure. end of input ( ) ; Aept EOF Aept LPAREN Aept RPAREN Aept SCOLON No other tokens strt with ny of these hrters. Thus no trnsi:ons out of these finl sttes. If we do this everywhere, no need for ktrking. Effiient Winter 013 UW CSE 401 (Mihel Ringenurg) 5 Snner DFA Exmple () 5 = 6 Aept NEQ < 8 [other ] = 7 9 Aept NOT, don t dvne input pointer Aept LEQ One gin, no need for trnsi:ons out of finl sttes. [other ] 10 Aept LESS, don t dvne input pointer Winter 013 UW CSE 401 (Mihel Ringenurg) 6 13

14 Snner DFA Exmple (3) [0-9] 11 [0-9] [other ] 1 Aept INT, don t dvne Only integers in this lnguge. Winter 013 UW CSE 401 (Mihel Ringenurg) 7 Snner DFA Exmple (4) [-za-z] 13 [-za-z0-9_] [other ] Aept ID or keyword, don t dvne Strtegies for hndling iden:fiers vs keywords Hnd- wriwen snner: look up iden:fiers in tle of keywords (good ppli:on of perfet hshing i.e., given knowledge of keys hed of :me, n ensure no ollisions.) Mhine- generted snner: generte DFA with pproprite trnsi:ons to reognize keywords (> priority thn IDs). 14 Winter 013 UW CSE 401 (Mihel Ringenurg) 8 14

15 Bktrking As we sw, ktrking is not neessry in our DFA. More effiient In mny ses, token syntx n e hosen (nd DFA onstruted refully) suh tht ktrking is rre (or n e voided en:rely). Esier to ensure this hppens in hnd- wriwen snner Prt of why well- wri3en hnd- wriwen snners re the most effiient. Winter 013 UW CSE 401 (Mihel Ringenurg) 9 Implemen:ng Snner y Hnd Token Represent:on A token is simple, tgged struture (Compilers wriwen in C/C++ oqen use tgged union style struture) puli lss Token { puli int kind; // token s lexil lss puli int intvl;// integer vlue if lss = INT puli String id;// tul identifier if lss = ID // lexil lsses puli stti finl int EOF = 0; // end of file // token puli stti finl int ID = 1; // identifier, // not keyword puli stti finl int INT = ; // integer puli stti finl int LPAREN = 4; // ( puli stti finl int SCOLN = 5; // ; puli stti finl int WHILEK = 6; // ) // et. et. et. Winter 013 UW CSE 401 (Mihel Ringenurg) 30 15

16 Simple Snner Exmple // glol stte nd methods // next unproessed input hrter stti hr nexth; // dvne to next input hr void geth() { } // skip whitespe nd omments void skipwhitespe() { } // input is letter, digit, or _ oolen isidchr(hr ); Winter 013 UW CSE 401 (Mihel Ringenurg) 31 Snner gettoken() method // Clled y prser to retrieve the next input token puli Token gettoken() { Token result; skipwhitespe(); if (/*no more input*/) { result = new Token(Token.EOF); return result; } swith(nexth) { se '(': result = new Token(Token.LPAREN); geth(); return result; se )': result = new Token(Token.RPAREN); geth(); return result; se ;': result = new Token(Token.SCOLON); geth(); return result; // Repet for other single hrter tokens Winter 013 UW CSE 401 (Mihel Ringenurg) 3 16

17 gettoken() () se '': // or = geth(); if (nexth == '=') { result = new Token(Token.NEQ); geth(); return result; } else { result = new Token(Token.NOT); return result; } se '<': // < or <= geth(); if (nexth == '=') { result = new Token(Token.LEQ); geth(); return result; } else { result = new Token(Token.LESS); return result; } Winter 013 UW CSE 401 (Mihel Ringenurg) 33 gettoken() (3) se '0': se '1': se '': se '3': se '4': se '5': se '6': se '7': se '8': se '9': // integer onstnt String num = nexth; geth(); while (Chrter.isdigit(nexth)) { num = num + nexth; geth(); } result = new Token(Token.INT, Integer(num).intVlue()); return result; Winter 013 UW CSE 401 (Mihel Ringenurg) 34 17

18 gettoken() (4) se '': se 'z': se 'A': se 'Z': // id or keyword string s = nexth; geth(); while (isidchr(nexth)) // letter, digit, _ { s = s + nexth; geth(); } if (keywordtle.iskeyword(s)) { result = new Token(keywordTle.getKind(s)); } else { result = new Token(Token.ID, s); } return result; Winter 013 UW CSE 401 (Mihel Ringenurg) 35 MiniJv Snner Gener:on We ll use the JFlex tool to utom:lly rete snner from speifi:on file. We ll use the CUP tool to utom:lly rete prser from speifi:on file. Token lss is shred y jflex nd CUP. Lexil lsses (token kinds) re listed in CUP s input file nd it genertes the token lss defini:on. So you ll need to modify oth speifi:on files for the snner por:on of your projet Prser mods will e smll. Winter 013 UW CSE 401 (Mihel Ringenurg) 36 18

19 JFlex Speifi:on Exmple Open sr/snner/minijv.jflex in your projet strter ode. You ll see tht few tokens hve lredy een done for you, to demonstrte how it works, e.g.: "+" { return symol(sym.plus); } {letter} ({letter} {digit} _)* { return symol(sym.identifier, yytext()); } Formt is Token RE, followed y ode to exeute. Cn define helper revi:ons, e.g.: letter = [-za-z] digit = [0-9] Winter 013 UW CSE 401 (Mihel Ringenurg) 37 Speifying the tokens Tokens re speified in the CUP file sr/prser/minijv.up /* Terminls (tokens returned y the snner) */ /* reserved words: */ terminl DISPLAY; /* opertors: */ terminl PLUS, BECOMES; /* delimiters: */ terminl LPAREN, RPAREN, SEMICOLON; /* tokens with vlues: */ terminl String IDENTIFIER; Winter 013 UW CSE 401 (Mihel Ringenurg) 38 19

20 JFlex Demo Your projet strter ode hs few tokens defined lredy. We ll dd mul:pli:on. Winter 013 UW CSE 401 (Mihel Ringenurg) 39 Coming Awr:ons Str:ng next leture: prsing Will do LR prsing first we need this for the projet, then LL (reursive- desent) prsing, whih you should lso know. My tke the rest of Jnury it s ig topi Se:ons more detils out using JFlex for your projet. The full detils n e found in the JFlex nd CUP doument:on. Winter 013 UW CSE 401 (Mihel Ringenurg) 40 0