CS321 Languages and Compiler Design I. Winter 2012 Lecture 5

Transcription

1 CS321 Lnguges nd Compiler Design I Winter 2012 Lecture 5 1

2 FINITE AUTOMATA A non-deterministic finite utomton (NFA) consists of: An input lphet Σ, e.g. Σ =,. A set of sttes S, e.g. S = {1, 3, 5, 7, 11, 97}. A designted strt stte, e.g. stte 1. A designted set of finl sttes, e.g. {5, 97}. A set of trnsitions from sttes to sttes, lelled y elements of Σ or ǫ, e.g PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 2

3 NFA S ACCEPT STRINGS Cn lso write s trnsition tle. Input Stte ǫ 0 {0,1} {0} {2} {3} 2 - {3} An NFA ccepts the string x if there is pth from strt to finl stte leled y the the chrcters of x, possily including some ǫ s. Exmple: NFA ove ccepts the strings,, nd, mong mny others. PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 3

4 NFA S AND REGULAR LANGUAGES An NFA ccepts the lnguge L if it ccepts exctly the strings in L. Exmple: NFA ove ccepts the lnguge defined y the R.E. ( ) ( ǫ ). Fct: For every regulr lnguge L, there exists n NFA tht ccepts L. PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 4

5 NFA S FROM R.E. S Cn give n lgorithm for constructing n NFA from n R.E., such tht the NFA ccepts the lnguge defined y the R.E. Algorithm is recursive, nd is sed on the recursively defined structure of R.E. s. Mkes hevy use of ǫ -trnsitions. Bse Constructions in Σ Inductive Constructions uild new mchines y connecting existing mchines using ǫ -trnsitions to existing initil sttes nd from existing finl sttes. Note tht ech constructed mchine hs exctly one initil stte nd one finl stte. PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 5

6 INDUCTIVE CONSTRUCTIONS R R 1 2 M 1 M 2 R 01 R 1 2 M 1 M 2 R * M PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 6

7 Exmple: ( )* (c) NFA CONSTRUCTION EXAMPLE ( )* c c PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 7

8 Exmple (continued) c c ( )* (c) c PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 8

9 Exmple (continued) Cn simplify NFA s y removing useless empty-string trnsitions: ( )* (c) c Or even simpler: c PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 9

10 NFA S FOR LEXICAL PATTERN R.E.S ID letter (letter digit)* Non-deterministic letter shorthnd letter digit or, simpler: letter letter digit Still non-deterministic (when to stop??) IF i f RELATION < = > = = Non-deterministic Resolve non-determinism y ccepting longest possile string! PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 10

11 Lexicl nlyzer must find mong set of ptterns. est mtch Try: NFA for pttern #1 Then try: NFA for pttern #2... Finlly, try: NFA for pttern #n Must reset input string fter ech unsuccessful mtch ttempt. Alwys choose pttern tht llows longest input string to mtch. Must specify which pttern should win if two or more mtch the sme length of input. PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 11

12 Alterntively, comine ll the NFA s into one gint NFA, with distinguished finl sttes: NFA for pttern #1 NFA for pttern # NFA for pttern #n Now cn hve non-determinism etween ptterns, s well s within single pttern, e.g: i letter f letter Found keyword IF Found n identifier digit PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 12

13 IMPLEMENTING NFA S Behvior of n NFA on given input string is miguous. So NFA s don t led to deterministic computer progrm. Cn convert to deterministic finite utomton (DFA). (Also clled finite stte mchine. ) Like NFA, ut hs no ǫ -trnsitions nd no symol lels more thn one trnsition from ny given node. Esy to simulte on computer. There is n lgorithm ( suset construction ) tht cn convert ny NFA to DFA tht ccepts the sme lnguge. Alterntive pproch: Simulte NFA directly y pretending to follow ll possile pths t once. To hndle longest mtch requirement, must keep trck of lst finl stte entered, nd cktrck to tht stte ( unreding chrcters) if get stuck. PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 13

14 DFA AND BACKTRACKING EXAMPLE Given the following set of ptterns: + We wnt to uild mchine to find the longest mtch; in cse of ties, fvor the pttern listed first. Here s the NFA: * PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 14

15 * CORRESPONDING DFA * *+ Consider input : Mchine stops in stte ( ). Pttern is. Lexeme is. PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 15

16 * CORRESPONDING DFA * *+ Conider input : Mchine stops in stte (8) Pttern is +. Lexeme is. PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 16

17 * CORRESPONDING DFA * *+ Consider input : Mchine stops fter second in stte (6 8). Pttern is ecuse it comes first in spec. Lexeme is ; finl will e red gin next time. PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 17

18 * CORRESPONDING DFA * *+ Consider input : Mchine gets stuck fter in stte (12). Bcks up to stte (5 8 11), unreding Pttern is + Lexeme is ; finl will e red gin next time. PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 18

19 JFLEX JFlex is lexicl nlyzer genertor Jv version of originl AT&T lex tool for C; mny similr tools exist. Detils of use my vry. ccepts specifiction of lexicl nlyzer. produces Jv progrm tht implements specifiction. PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 19

20 JFlex source JFlex (foo.lex) run y: jv JFlex.Min foo.lex Lexicl nlyzer clss Yylex method yylex() in Jv (Yylex.jv) Jv Compiler prser or driver source files Chrcter input strem Lexicl Anlyzer Executle Tokens PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 20

21 JFLEX RULE SPECIFICATIONS The min input to JFlex is sequence of rules, ech consisting of Pttern regulr expression (using ASCII s lphet) Action frgment of Jv code When prefix of input mtches pttern, the generted nlyzer executes the corresponding ction. Actions cn mke use of uilt-in vriles nd methods yytext() returns lexeme s String yyline contins current line numer (must use %line option). PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 21

22 JFLEX RULES EXAMPLE %% %% integer {println("found keyword INTEGER");} [0-9]+ {println("found numer");} [A-Z][A-Z]* {println("found ident " + yytext());} [ \t\n] { /* ignore white spce */ } As usul, if more thn one pttern mtches, the longest mtch is preferred; ties re roken in fvor of rule tht ppers first. PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 22

23 JFLEX PATTERNS Ptterns include literl text nd met-level opertors. Pttern Mtches x chrcter x "x" chrcter x even if it s n opertor \x ditto [xy] x or y [x-y] chrcters etween x nd y inclusive [^s] ny chrcter not in set s. ny chrcter ut \n p? n optionl p p* zero or more p s p+ one or more p s p q p or q () grouping {d} sustitute definition for d PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 23

24 JFLEX ACTIONS Actions cn e ny vlid Jv sttement lock. Ordinrily ech ction termintes with sttement return t; which cuses yylex() to return with the token vlue t. Otherwise, yylex() throws wy the lexeme nd continues serching for nother pttern. This is suitle for hndling white spce. The simplest possile ction is just the empty lock {}. yylex() rises n exception if no pttern mtches. So it is good ide to include ctch-ll pttern s the lst rule, e.g.:. { System.err.println("Unexpected chrcter"); } PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 24

25 JFLEX DIRECTIVES The complete form of JFlex specifiction is: user code %% JFlex directives %% rules Directives include control instructions, such s %line, which sys the generted code should keep trck of line numers. Directives cn lso include mcro definitions, which revite regulr expressions for lter use in ptterns, e.g., %% LETTERS=[-zA-Z_] DIGITS=[0-9] %% {LETTERS}({LETTERS} {DIGITS})* {return new Token(ID);} PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 25

26 JFLEX USER CODE User code is just copied directly to the top of the generted.jv file; it cn contin functions nd glols to e invoked from the ctions. Such code cn lso e included in the directives section if enclosed etween %{ nd %}; in this cse, it is copied into the inside of the generted Yylex clss. PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 26

27 JFLEX STATES JFlex permits multiple sets of rules to coexist in the sme specifiction. Ech set of rules is ssocited with stte. Rules prefixed with <nme> re recognized (only) when yylex() is in the stte nme. When yylex() strts running it is in the stte with the predefined nme YYINITIAL. You declre new stte nmes in %stte line in the definitions section of the spec. You put yylex() into stte nme y including the method cll yyegin(nme); in n ction. PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 27

28 EXAMPLE USING JFLEX STATES Exmple: multi-line comments in Jv. %% %stte COMMENT %% <YYINITIAL>"/*" { yyegin(comment); } <COMMENT>"*/" { yyegin(yyinitial); } <COMMENT>. "\n" { /* ignore comments */ } <YYINITIAL>... ordinry rules follow <YYINITIAL>... PSU CS321 W 12 LECTURE 5 c ANDREW TOLMACH 28