Compiler Construction D7011E

Transcription

1 Compiler Construction D7011E Lecture 3: Lexer genertors Viktor Leijon Slides lrgely y John Nordlnder with mteril generously provided y Mrk P. Jones. 1

2 Recp: Hndwritten Lexers: Don t require sophisticted progrmming. Often requires cre to void non-determinism, or potentilly expensive cktrcking. Cn e fine tuned for performnce nd for the lnguge concerned. But it might lso e something we would wnt to utomte 2

3 Cn Mchine do Better? It cn e hrd to write (correct) lexer y hnd But tht s not surprising: finite stte mchines re low level n ssemly lnguge of lexicl nlysis Cn we uild lexicl nlyzer genertor tht will tke cre of ll the dirty detils, nd let the humns work t higher level? If so, wht would its input look like? 3

4 Forml Lnguges: Pick n lphet A: set of symols. n n For lexicl nlysis, symol = chrcter. For prsing, symol = token. The set of ll finite strings of symols tken from A is written A*. A lnguge (over A) is suset of A* Two issues: n How do we descrie certin lnguge over A (suset of A*)? n How do we check whether string elongs to the lnguge? 4

5 Regulr Expressions: A syntx for descriing regulr lnguges. A widely used nottion for descriing ptterns in text strings: emcs, vi, grep, wk, perl, ut lso good for descriing the lexicl structure of progrmming lnguge 5

6 Regulr Expressions: Expression r 1 r 2 r 1 r 2 Mening Sequencing: text mtching r 1 followed y text mtching r 2. Alterntives: text mtching r 1 or text mtching r 2. r* Repetition: text mtching r zero or more times. 'c' Constnt: mtches text c literlly 'c 1 '..'c 2 ' Rnge: mtches single chrcter in the rnge etween c1 nd c2, inclusively. 6

7 Regulr Expressions: Expression r + Mening Repetition: text mtching r one ore more times. r + = rr* ( r ) Grouping: text mtching r. r? Optionl: Optionl text mtching r.. Wildcrd: Mtches ny single chrcter. ~r Negtion: Mtches ny single chrcter not mtched y r. R Rule: Mtches the sme text mtched y the regulr expression nmed R. 7

8 Relevnt Exmples: Expression ('0'..'9') + 'if' ('A'..'Z' ''..'z')('a'..'z' ''..'z' '0'..'9' '_')* (' ' '\t' '\n')+ '//'.* '\n' INT FRAC? EXP? frgment INT : ('0'..'9')+ FRAC : '.' INT EXP : ('E' 'e') ('+' '-')? INT Possile Use Integer literls The keyword if Identifier in C++ Whitespce C++ comment Floting point literl 8

9 Automted Lexer Genertors 9

10 The lex Fmily: lex is clssicl Unix tool for generting C progrms to implement lexicl nlyzers. It cn lso e used s quick wy of generting simple text processing utilities. lex dtes from the mid-seventies nd hs spwned fmily of clones: flex, ML lex, JLex, JFlex, Alex, etc Lex nd its successors re ll sed on ides from the theory of forml lnguges nd utomt 10

11 Vritions on the Theme: There re quite few lex-like tools ech ctering to prticulr n n progrmming lnguges operting system/environments They vry in minor detils of syntx etc., ut the lex heritge is usully pretty cler. Most of them hve lot of fetures; red the mnuls! In this course we will use ANTLR ( which is powerful tool for generting oth lexers nd prsers from common syntctic specifiction. 11

12 The Ins nd Outs of ANTLR: The input to ANTLR is grmmr file which my contin either lexer specifiction, or prser specifiction, or oth. If ANTLR is given lexer grmmr in file MyLex.g, its output is file MyLex.jv defining clss MyLex tht extends lirry clss Lexer. The importnt method of clss Lexer is Token nexttoken(); 12

13 The Ins nd Outs of ANTLR: The importnt methods of interfce Token re int gettype(); String gettext(); The importnt ttriutes of clss MyLex re integer constnts denoting token types, with one constnt defined for ech lexer rule in MyLex.g. To seprte lexer rules from prser rules, the former nmes lwys strt with Cpitl letter 13

14 ANTLR lexer grmmr formt: In file MyLex.g: lexer grmmr MyLex; optionl declrtions RULE 1 : regexp 1 { optionl Jv ctions } ; RULE n : regexp n { optionl Jv ctions } ; 14

15 Simple ANTLR lexicl rules: Lexer rules must egin with n uppercse letter: ID : (''..'z')+ ; Actions re optionl, the most useful ction prevents the mtched token from eing returned: WS : (' ' '\n') { skip(); } ; 15

16 Simple ANTLR lexicl rules: Tokens re normlly consumed y the code tht clls lexer.nexttoken() However, rule ctions my lso produce nother strem of output Current lexeme text is ville vi gettext(). Exmple: rule for copying input to System.out without ny chnge: TEXT :. { System.out.print(getText());} ; 16

17 Simple ANTLR lexicl rules: The mtched lexeme text my lso e mnipulted vi method settext(). A glol serch nd replce utility: CR : ('\r\n' '\n') { settext("\n"); }; THING : 'compiler' { settext("thing"); }; TEXT :. ; 17

18 Simple ANTLR lexicl rules: Suexpressions in rules my e referenced using the $NNN nottion. SPEC : ID '#' { settext($id.text); } ; ID : ''..'z'+ ; SEP : ('#' ' ' '\n' '\r')+ ; 18

19 Simple ANTLR lexicl rules: Alterntively, explicit nming of suexpressions my e used. SPEC : i=id '#' ID? { settext($i.text); } ; ID : ''..'z'+ ; SEP : ('#' ' ' '\n' '\r')+ ; 19

20 Simple ANTLR lexicl rules: Output for given pttern my depend on { oolen flg = flse; } BEGIN : 'BEGIN' { flg = true; } ; END : 'END' { flg = flse; } ; ID : (''..'z')+ { if (!flg) skip(); } ; WS : ' ' '\n' ; 20

21 Simple ANTLR lexicl rules: The mximl munch rule pplies DOUBLE : ' ' { settext("="); } ; SINGLE : ' ' { settext("-"); } ; TEXT :. ; 21

22 Simple ANTLR lexicl rules: If two rules mtch the sme lexeme, the first one wins: IF : 'if' { settext("if"); }; ID : (''..'z')+ WS : ' ' '\n' 22

23 Towrds n ANTLR lexer: A "complete" lexer grmmr file. lexer grmmr MyLex; IF : 'if' ; ELSE : 'else' ; IDENT LPAR : '(' ; RPAR : ')' ; : ('A'..'Z' ''..'z') ('A'..'Z' ''..'z' '0'..'9')* ; WS : (' ' '\n' '\t')+ { skip(); } ; COMMENT : '//'.* '\n' { skip(); } ; 23

24 Towrds n ANTLR lexer: Clling the lexer repetedly: ANTLRInputStrem input = new ANTLRInputStrem(System.in); MyLex lexer = new MyLex(input); Token t; do { t = lexer.nexttoken(); } while (t.gettype()!= MyLex.EOF); System.out.println("Success"); Filures throw n exception, defult hndling is utomticlly in plce! 24

25 The Theory Behind Regulr Expression Recognition 25

26 How lex & friends work: Lex, et el., tckle the genertion of lexers vi two intermedite stges: regulr expression 26

27 How lex & friends work: Lex, et el., tckle the genertion of lexers vi two intermedite stges: regulr expression A non-deterministic finite utomton (NFA) 26

28 How lex & friends work: Lex, et el., tckle the genertion of lexers vi two intermedite stges: regulr expression A non-deterministic finite utomton (NFA) A deterministic finite utomton (DFA) 26

29 How lex & friends work: Lex, et el., tckle the genertion of lexers vi two intermedite stges: regulr expression A non-deterministic finite utomton (NFA) A deterministic finite utomton (DFA) C/Jv/ code 26

30 Finite Automt Astrct devices for recognizing (checking) whether string elongs to some regulr lnguge For given string, finite utomton either ccepts it (the string elongs to the corresponding lnguge), or rejects it Intuitive wy of opertion: sed on the current stte nd the next input chrcter, either reject the chrcter, or ccept it nd go to the next stte 27

31 Finite Automt: We will use the following symols to descrie utomt: n stte numer n n stte numer n, ccepting A trnsition on symol A trnsition without ny input 28

32 (Non) Deterministic? A mchine is non-deterministic (n NFA) if there is stte with: n n more thn one trnsition on the sme symol; or with n trnsition. Otherwise, the mchine is deterministic ( DFA). 29

33 Exmple: Given regulr expression ( )*, we cn uild corresponding NFA: An equivlent DFA:

34 Building NFAs: We will descrie the construction of NFAs s process tht tkes: n A regulr expression r; nd n A pointer s to nother mchine. The result is mchine: NFA(r,s) r s which recognizes r, nd then goes to s. 31

35 Building the Min Recognizer: For n input file with ptterns r 1,,r n nd corresponding ctions 1,, n, we construct the following NFA: r 1 r n Do ction 1 if we ccept here Do ction n if we ccept here 32

36 Simple Cses: NFA(, s) = s NFA(c, ) = s c s NFA(r 1 r 2, ) s = NFA(r 1, NFA(r 2, s )) = r 1 r 2 s 33

37 Alterntives nd Repetition: NFA(r 1 r 2, ) = s r 1 r 2 s s NFA(r*, s ) = s r 34

38 Following the Algorithm: Let s use the lgorithm to uild n NFA for the regulr expression ( )*: 35

39 Following the Algorithm: Let s use the lgorithm to uild n NFA for the regulr expression ( )*: NFA(( )*, s) ( )* s 35

40 Following the Algorithm: NFA(( )*, NFA(, s)) ( )* NFA(, s) s 36

41 Following the Algorithm: NFA(( )*, NFA(, s)) NFA(,NFA(, s)) ( )* NFA(,s) s 37

42 Following the Algorithm: NFA(( )*, NFA(, s)) NFA(,s 1 ) s 1 ( )* 38

43 Following the Algorithm: NFA(( )*, s 2 ) s 2 ( )* 39

44 Following the Algorithm: NFA(( )*, s 2 ) NFA(, m) s 2 m 40

45 Following the Algorithm: NFA(( )*, s 2 ) NFA(,m) s 2 m NFA(,m) 41

46 Following the Algorithm: NFA(( )*, s 2 ) s 2 m NFA(,m) 42

47 Following the Algorithm: NFA(( )*, s 2 ) s 2 m 43

48 Following the Algorithm: NFA(( )*, s 2 ) s 2 m The right structure, ut more complex thn we might hve hoped we ll fix this soon 43

49 From NFAs to DFAs: Consider the simple NFA for ( )*: How do we know which rnch to tke in stte 0 when we see n? n We might end up in stte 0 or in stte 1. n Let s mke this lterntive explicit! 44

50 Illustrting NFA to DFA: Key ide: lel every stte in the DFA with set of NFA sttes: {0} {0,1} {0,2} 45

51 Mking DFA for the Exmple: Strt stte is {0} From {0}, we cn rech {0,1,2,3,4} without ny input (this is the -closure of {0}). {0,1,2,3,4} goes to {0,5}, hence {0,1,2,3,4,5} on. {0,1,2,3,4} goes to {0}, hence {0,1,2,3,4} on. 46

52 -closure: If S is set of NFA sttes, the -closure of S is the set of ll sttes tht cn e reched from S y n -trnsition. If s S, then s -closure(s). If s -closure(s), nd there is n -trnsition from s to t, then t -closure(s). 47

53 Mking DFA for the Exmple: {0,1,2,3,4,5} goes to {0,5}, hence {0,1,2,3,4,5} on. {0,1,2,3,4,5} goes to {0,6}, hence {0,1,2,3,4,6} on. 48

54 Mking DFA for the Exmple: {0,1,2,3,4,6} goes to {0,5}, hence {0,1,2,3,4,5} on. {0,1,2,3,4,6} goes to {0}, hence {0,1,2,3,4,6} on. {0,1,2,3,4,6} is n ccept stte in the DFA ecuse 6 is n ccept in the NFA. 49

55 The Finl DFA: 0,1,2, 3,4 0,1,2,3, 4,5 0,1,2,3, 4,6 50

56 More Formlly: If the strt stte of the NFA is leled n, then the strt stte of the DFA is leled -closure({n}). For ech chrcter A nd for ech stte X, there is single trnsition from X on which goes to the stte: Y = -closure({ y x X nd x goes to y on in the NFA }) A stte X in the DFA is n ccept stte if ny x X is n ccept stte in the NFA. We cn delete/ignore ny trnsitions to the empty stte, {}, ecuse trnsition to {} will never led to n ccept stte. 51

57 Stte Explosion? In theory, given n NFA with n distinct sttes, the corresponding DFA might hve s mny s 2 n distinct sttes! In prctice, we need only consider those sttes tht re rechle from the initil strt stte; this is usully much lower thn 2 n. 52

58 DFA Minimiztion: Is this DFA the only DFA tht will recognize ( )*? Clerly not: But every regulr set is recognized y minimum stte DFA tht is unique up to stte nmes. DFA minimiztion, however, is eyond the scope of this clss 53

59 From DFA to Implementtion: The lex pproch use generl, tle driven lgorithm: for (;;) { stte = tle[stte][getchr()]; if (stte==accept) rek; if (stte == X) throw (new ScnError()); } /* user code */ return TOKEN23; Stte c d e ACCEPT X X 54

60 From DFA to Implementtion: The flex pproch generte custom executle code: stte34 : switch(getchr()) { cse : goto stte37; cse : goto stte14; cse c : goto ccept; defult : goto stuck; } ccept: /* user code */ { return TOKEN23; } 55

61 From DFA to Implementtion: The ANTLR pproch tle-driven s lex, ut DFAs re customized for ech decision point (in oth the lexer nd the prser) int lt = dfn.predict(input); switch (lt) { cse 1 : mtchident(); rek; cse 2 : mtchint(); rek; cse 3 : mtchfloat(); rek; cse 4 : mtchcomment(); continue; } return stte.token; More on this technique in reltion to prsing! 56

62 Hndwritten or Mchine Generted? A mildly controversil topic! Issues include: n How efficient is the generted lexer? n How esily does it interfce to other code? n How nturl is the input? If the lnguge you re compiling hs some wkwrd fetures, the lexers produced y tool might need some mssging to do the right thing. n How good re the error messges? 57

63 Summry: Regulr expressions provide high-level lnguge for descriing lexemes. lex is useful tool for text processing nd lso for writing lexers lex works y mpping regulr expressions to NFAs, nd then mpping them to DFAs. n Automtes the tsk of removing non-determinism. We use ANTLR in the course project, ut fter tht, the choice is yours! 58

64 L Project (step 1): Implement lexer for MiniJv, using ANTLR. The clss Min shll contin min method tht clls the lexer repetedly when given -l s rgument., nd print Lexicl nlysis succeeded/filed s pproprite. The lexer shll hndle the MiniJv test suite correctly. There is n utomtic testsuite on the student mchines: /fs/ltu.se/students/l/e/leijon/testsuite/testsuite.pl The MiniJv syntx is descried in Appel, Appendix 1, ut we dd one exception: No /* nested /* comments */ necessry! */ See the wepge for detils, dedline on the 24th. 59

65 Reding instructions You should now hve finished Chpter 3. n Next we move on to prsing. Strt with chpter , Grmmrs & Top- Down prsing. 60