Lexical Analysis and Lexical Analyzer Generators

Transcription

1 1 Lexicl Anlysis nd Lexicl Anlyzer Genertors Chpter 3 COP5621 Compiler Construction Copyright Roert vn Engelen, Florid Stte University,

2 2 The Reson Why Lexicl Anlysis is Seprte Phse Simplifies the design of the compiler LL(1) or LR(1) prsing with 1 token lookhed would not e possile (multiple chrcters/tokens to mtch) Provides efficient implementtion Systemtic techniques to implement lexicl nlyzers y hnd or utomticlly from specifictions Strem uffering methods to scn input Improves portility Non-stndrd symols nd lternte chrcter encodings cn e normlized (e.g. trigrphs)

3 3 Interction of the Lexicl Anlyzer with the Prser Source Progrm Lexicl Anlyzer Token, tokenvl Get next token Prser error error Symol Tle

4 4 Attriutes of Tokens y := *x Lexicl nlyzer <id, y > <ssign, > <num, 31> <+, > <num, 28> <*, > <id, x > token tokenvl (token ttriute) Prser

5 5 Tokens, Ptterns, nd Lexemes A token is clssifiction of lexicl units For exmple: id nd num Lexemes re the specific chrcter strings tht mke up token For exmple: c nd 123 Ptterns re rules descriing the set of lexemes elonging to token For exmple: letter followed y letters nd digits nd non-empty sequence of digits

6 6 Specifiction of Ptterns for Tokens: Definitions An lphet Σ is finite set of symols (chrcters) A string s is finite sequence of symols from Σ s denotes the length of string s denotes the empty string, thus = 0 A lnguge is specific set of strings over some fixed lphet Σ

7 7 Specifiction of Ptterns for Tokens: String Opertions The conctention of two strings x nd y is denoted y xy The exponenttion of string s is defined y s 0 = s i = s i-1 s for i > 0 note tht s = s = s

8 8 Specifiction of Ptterns for Tokens: Lnguge Opertions Union L M = {s s L or s M} Conctention LM = {xy x L nd y M} Exponentition L 0 = {}; L i = L i-1 L Kleene closure L * = i=0,, L i Positive closure L + = i=1,, L i

9 9 Specifiction of Ptterns for Tokens: Regulr Expressions Bsis symols: is regulr expression denoting lnguge {} Σ is regulr expression denoting {} If r nd s re regulr expressions denoting lnguges L(r) nd M(s) respectively, then r s is regulr expression denoting L(r) M(s) rs is regulr expression denoting L(r)M(s) r * is regulr expression denoting L(r) * (r) is regulr expression denoting L(r) A lnguge defined y regulr expression is clled regulr set

10 10 Specifiction of Ptterns for Tokens: Regulr Definitions Regulr definitions introduce nming convention: d 1 r 1 d 2 r 2 d n r n where ech r i is regulr expression over Σ {d 1, d 2,, d i-1 } Any d j in r i cn e textully sustituted in r i to otin n equivlent set of definitions

11 11 Specifiction of Ptterns for Tokens: Regulr Definitions Exmple: letter A B Z z digit id letter ( letter digit ) * Regulr definitions re not recursive: digits digit digits digit wrong!

12 12 Specifiction of Ptterns for Tokens: Nottionl Shorthnd The following shorthnds re often used: r + = rr * r? = r [-z] = c z Exmples: digit [0-9] num digit + (. digit + )? ( E (+ -)? digit + )?

13 13 Grmmr Regulr Definitions nd stmt if expr then stmt if expr then stmt else stmt expr term relop term term term id num Grmmrs Regulr definitions if if then then else else relop < <= <> > >= = id letter ( letter digit ) * num digit + (. digit + )? ( E (+ -)? digit + )?

14 14 Coding Regulr Definitions in Trnsition Digrms relop < <= <> > >= = strt 0 < = > id letter ( letter digit ) * = 2 > 3 other 4 * return(relop, LE) return(relop, NE) return(relop, LT) return(relop, EQ) = 7 return(relop, GE) other 8*return(relop, GT) letter or digit strt 9 letter 10 other 11*return(gettoken(), instll_id())

15 Coding Regulr Definitions in 15 Trnsition Digrms: Code token nexttoken() { while (1) { switch (stte) { cse 0: c = nextchr(); if (c==lnk c==t c==newline) { stte = 0; lexeme_eginning++; } else if (c== < ) stte = 1; else if (c== = ) stte = 5; else if (c== > ) stte = 6; else stte = fil(); rek; cse 1: cse 9: c = nextchr(); if (isletter(c)) stte = 10; else stte = fil(); rek; cse 10: c = nextchr(); if (isletter(c)) stte = 10; else if (isdigit(c)) stte = 10; else stte = 11; rek; Decides the next strt stte to check int fil() { forwrd = token_eginning; swith (strt) { cse 0: strt = 9; rek; cse 9: strt = 12; rek; cse 12: strt = 20; rek; cse 20: strt = 25; rek; cse 25: recover(); rek; defult: /* error */ } return strt; }

16 16 The Lex nd Flex Scnner Genertors Lex nd its newer cousin flex re scnner genertors Systemticlly trnslte regulr definitions into C source code for efficient scnning Generted code is esy to integrte in C pplictions

17 17 Creting Lexicl Anlyzer with Lex nd Flex lex source progrm lex.l lex.yy.c lex or flex compiler C compiler lex.yy.c.out input strem.out sequence of tokens

18 18 Lex Specifiction A lex specifiction consists of three prts: regulr definitions, C declrtions in %{ %} %% trnsltion rules %% user-defined uxiliry procedures The trnsltion rules re of the form: p 1 { ction 1 } p 2 { ction 2 } p n { ction n }

19 19 Regulr Expressions in Lex x mtch the chrcter x \. mtch the chrcter. string mtch contents of string of chrcters. mtch ny chrcter except newline ^ mtch eginning of line $ mtch the end of line [xyz] mtch one chrcter x, y, or z (use \ to escpe -) [^xyz]mtch ny chrcter except x, y, nd z [-z] mtch one of to z r* closure (mtch zero or more occurrences) r+ positive closure (mtch one or more occurrences) r? optionl (mtch zero or one occurrence) r 1 r 2 mtch r 1 then r 2 (conctention) r 1 r 2 mtch r 1 or r 2 (union) ( r ) grouping r 1 \r 2 mtch r 1 when followed y r 2 {d} mtch the regulr expression defined y d

20 20 Exmple Lex Specifiction 1 Trnsltion rules %{ #include <stdio.h> %} %% [0-9]+ { printf( %s\n, yytext); }. \n { } %% min() { yylex(); } Contins the mtching lexeme Invokes the lexicl nlyzer lex spec.l gcc lex.yy.c -ll./.out < spec.l

21 21 Exmple Lex Specifiction 2 Trnsltion rules %{ #include <stdio.h> int ch = 0, wd = 0, nl = 0; %} delim [ \t]+ %% \n { ch++; wd++; nl++; } ^{delim} { ch+=yyleng; } {delim} { ch+=yyleng; wd++; }. { ch++; } %% min() { yylex(); printf("%8d%8d%8d\n", nl, wd, ch); } Regulr definition

22 22 Exmple Lex Specifiction 3 Trnsltion rules %{ #include <stdio.h> %} digit [0-9] letter [A-Z-z] Regulr definitions id {letter}({letter} {digit})* %% {digit}+ { printf( numer: %s\n, yytext); } {id} { printf( ident: %s\n, yytext); }. { printf( other: %s\n, yytext); } %% min() { yylex(); }

23 Exmple Lex Specifiction 4 23 %{ /* definitions of mnifest constnts */ #define LT (256) %} delim [ \t\n] ws {delim}+ letter [A-Z-z] digit [0-9] id {letter}({letter} {digit})* numer {digit}+(\.{digit}+)?(e[+\-]?{digit}+)? %% {ws} { } if {return IF;} then {return THEN;} else {return ELSE;} {id} {yylvl = instll_id(); return ID;} {numer} {yylvl = instll_num(); return NUMBER;} < {yylvl = LT; return RELOP;} <= {yylvl = LE; return RELOP;} = {yylvl = EQ; return RELOP;} <> {yylvl = NE; return RELOP;} > {yylvl = GT; return RELOP;} >= {yylvl = GE; return RELOP;} %% int instll_id() Token ttriute Return token to prser Instll yytext s identifier in symol tle

24 24 Design of Lexicl Anlyzer Genertor Trnslte regulr expressions to NFA Trnslte NFA to n efficient DFA Optionl regulr expressions NFA DFA Simulte NFA to recognize tokens Simulte DFA to recognize tokens

25 25 Nondeterministic Finite Automt An NFA is 5-tuple (S, Σ, δ, s 0, F) where S is finite set of sttes Σ is finite set of symols, the lphet δ is mpping from S Σ to set of sttes s 0 S is the strt stte F S is the set of ccepting (or finl) sttes

26 26 Trnsition Grph An NFA cn e digrmmticlly represented y leled directed grph clled trnsition grph strt S = {0,1,2,3} Σ = {,} s 0 = 0 F = {3}

27 27 Trnsition Tle The mpping δ of n NFA cn e represented in trnsition tle δ(0,) = {0,1} δ(0,) = {0} δ(1,) = {2} δ(2,) = {3} Stte Input Input 0 {0, 1} {0} 1 2 {2} {3}

28 28 The Lnguge Defined y n NFA An NFA ccepts n input string x if nd only if there is some pth with edges leled with symols from x in sequence from the strt stte to some ccepting stte in the trnsition grph A stte trnsition from one stte to nother on the pth is clled move The lnguge defined y n NFA is the set of input strings it ccepts, such s ( )* for the exmple NFA

29 29 Design of Lexicl Anlyzer Genertor: RE to NFA to DFA Lex specifiction with regulr expressions NFA p 1 { ction 1 } p 2 { ction 2 } p n { ction n } strt s 0 N(p 1 ) N(p 2 ) N(p n ) ction 1 ction 2 ction n Suset construction DFA

30 30 From Regulr Expression to NFA (Thompson s Construction) strt i f strt i f r 1 r 2 r 1 r 2 strt i N(r 1 ) N(r 2 ) strt i N(r 1 ) N(r 2 ) f r* strt i N(r) f f

31 31 Comining the NFAs of Set of Regulr Expressions strt 1 2 { ction 1 } { ction 2 } *+ { ction 3 } strt strt strt

32 32 Simulting the Comined NFA Exmple 1 strt ction 1 ction 3 6 ction none 7 8 ction 3 Must find the longest mtch: Continue until no further moves re possile When lst stte is ccepting: execute ction

33 33 Simulting the Comined NFA Exmple 2 strt ction 1 ction 3 6 ction none ction 2 ction 3 When two or more ccepting sttes re reched, the first ction given in the Lex specifiction is executed

34 34 Deterministic Finite Automt A deterministic finite utomton is specil cse of n NFA No stte hs n -trnsition For ech stte s nd input symol there is t most one edge leled leving s Ech entry in the trnsition tle is single stte At most one pth exists to ccept string Simultion lgorithm is simple

35 35 Exmple DFA A DFA tht ccepts ( )* strt

36 36 Conversion of n NFA into DFA The suset construction lgorithm converts n NFA into DFA using: -closure(s) = {s} {t s t} -closure(t) = s T -closure(s) move(t,) = {t s t nd s T} The lgorithm produces: Dsttes is the set of sttes of the new DFA consisting of sets of sttes of the NFA Dtrn is the trnsition tle of the new DFA

37 37 -closure nd move Exmples strt closure({0}) = {0,1,3,7} move({0,1,3,7},) = {2,4,7} -closure({2,4,7}) = {2,4,7} move({2,4,7},) = {7} -closure({7}) = {7} move({7},) = {8} -closure({8}) = {8} move({8},) = 0 2 none Also used to simulte NFAs

38 38 Simulting n NFA using -closure nd move S := -closure({s 0 }) S prev := := nextchr() while S do S prev := S S := -closure(move(s,)) := nextchr() end do if S prev F then execute ction in S prev return yes else return no

39 39 The Suset Construction Algorithm Initilly, -closure(s 0 ) is the only stte in Dsttes nd it is unmrked while there is n unmrked stte T in Dsttes do mrk T for ech input symol Σ do U := -closure(move(t,)) if U is not in Dsttes then dd U s n unmrked stte to Dsttes end if Dtrn[T,] := U end do end do

40 Suset Construction Exmple 1 40 strt strt A C B D E Dsttes A = {0,1,2,4,7} B = {1,2,3,4,6,7,8} C = {1,2,4,5,6,7} D = {1,2,4,5,6,7,9} E = {1,2,4,5,6,7,10}

41 Suset Construction Exmple 2 41 strt strt 1 3 A C D B E 3 F 2 3 Dsttes A = {0,1,3,7} B = {2,4,7} C = {8} D = {7} E = {5,8} F = {6,8}

42 42 Minimizing the Numer of Sttes of DFA C strt A B D E strt A B D E

43 43 From Regulr Expression to DFA Directly The importnt sttes of n NFA re those without n -trnsition, tht is if move({s},) for some then s is n importnt stte The suset construction lgorithm uses only the importnt sttes when it determines -closure(move(t,))

44 44 From Regulr Expression to DFA Directly (Algorithm) Augment the regulr expression r with specil end symol # to mke ccepting sttes importnt: the new expression is r# Construct syntx tree for r# Trverse the tree to construct functions nullle, firstpos, lstpos, nd followpos

45 45 From Regulr Expression to DFA Directly: Syntx Tree of ( )*# conctention # 6 closure 4 5 lterntion * position numer (for lefs )

46 46 From Regulr Expression to DFA Directly: Annotting the Tree nullle(n): the sutree t node n genertes lnguges including the empty string firstpos(n): set of positions tht cn mtch the first symol of string generted y the sutree t node n lstpos(n): the set of positions tht cn mtch the lst symol of string generted e the sutree t node n followpos(i): the set of positions tht cn follow position i in the tree

47 From Regulr Expression to DFA 47 Directly: Annotting the Tree Node n nullle(n) firstpos(n) lstpos(n) Lef true Lef i flse {i} {i} / \ c 1 c 2 c 1 / \ c 2 nullle(c 1 ) or nullle(c 2 ) nullle(c 1 ) nd nullle(c 2 ) firstpos(c 1 ) firstpos(c 2 ) if nullle(c 1 ) then firstpos(c 1 ) firstpos(c 2 ) else firstpos(c 1 ) lstpos(c 1 ) lstpos(c 2 ) if nullle(c 2 ) then lstpos(c 1 ) lstpos(c 2 ) else lstpos(c 2 ) * true firstpos(c 1 ) lstpos(c 1 ) c 1

48 48 From Regulr Expression to DFA Directly: Syntx Tree of ( )*# {1, 2, 3} {6} {1, 2, 3} {5} {6}# {6} 6 nullle {1, 2, 3} {1, 2} {1, 2} * {1, 2, 3} {3} {4} {3} {3} 3 {4} {4} 4 {5} {5} 5 firstpos lstpos {1, 2} {1} {1} 1 {1, 2} {2} {2} 2

49 49 From Regulr Expression to DFA Directly: followpos for ech node n in the tree do if n is ct-node with left child c 1 nd right child c 2 then for ech i in lstpos(c 1 ) do followpos(i) := followpos(i) firstpos(c 2 ) end do else if n is str-node for ech i in lstpos(n) do followpos(i) := followpos(i) firstpos(n) end do end if end do

50 From Regulr Expression to DFA 50 Directly: Algorithm s 0 := firstpos(root) where root is the root of the syntx tree Dsttes := {s 0 } nd is unmrked while there is n unmrked stte T in Dsttes do mrk T for ech input symol Σ do let U e the set of positions tht re in followpos(p) for some position p in T, such tht the symol t position p is if U is not empty nd not in Dsttes then dd U s n unmrked stte to Dsttes end if Dtrn[T,] := U end do end do

51 51 From Regulr Expression to DFA Directly: Exmple Node followpos 1 {1, 2, 3} 2 {1, 2, 3} 3 {4} 4 {5} 5 {6} strt 1,2, 1,2,3 3,4 1,2, 3,5 1,2, 3,6

52 52 Time-Spce Trdeoffs Automton Spce (worst cse) Time (worst cse) NFA O( r ) O( r x ) DFA O(2 r ) O( x )