J. Xue. Tutorials. Tutorials to start in week 3 (i.e., next week) Tutorial questions are already available on-line

Size: px

Start display at page:

Download "J. Xue. Tutorials. Tutorials to start in week 3 (i.e., next week) Tutorial questions are already available on-line"

Georgia Fields
5 years ago
Views:

1 Tutorials Tutorials to start in week 3 (i.e., next week) Tutorial questions are already available on-line COMP3131/9102 Page 65 March 4, 2018

2 Assignment 1: Scanner +5 = two tokens: + and 5 the scanner understands how tokens are formed but not anything else COMP3131/9102 Page 66 March 4, 2018

3 COMP3131/9102: Programming Languages and Compilers Jingling Xue School of Computer Science and Engineering The University of New South Wales Sydney, NSW 2052, Australia Jingling Xue COMP3131/9102 Page 67 March 4, 2018

4 NFA REs The Big Picture DFA minimal-state DFA DFA The two conversions in dashed arrows are not covered: REs DFA (pages , Red Dragon/ 3.7, Purple Dragon) DFA REs: Chapter 3, J. Hopcroft, R. Motwani and J. Ullman, Introduction to Automata Theory, Languages, and Computation, Addison-Wesley, 2nd Edition, See www-db.stanford.edu/~ullman/ullman-books.html. DFA minimal-state DFA (pages , Red Dragon/ 3.9.6, Purple Dragon) Tools: COMP3131/9102 Page 68 March 4, 2018

5 Week 2: Regular Expressions, DFA and NFA 1. Definitions of REs, DFA and NFA 2. REs = NFA (Thompson s construction, Algorithm 3.3, Red Dragon/Algorithm 3.23, Purple Dragon) 3. NFA = DFA (subset construction, Algorithm 3.2, Red Dragon/Algorithm 3.20, Purple Dragon) 4. DFA = minimal-state DFA (state minimisation, Algorithm 3.6, Red Dragon/Algorithm 3.39, Purple Dragon) 5. Scanner generators How to use them (straightforward) How to write them (the most techniques introduced today) COMP3131/9102 Page 69 March 4, 2018

6 Applications of Regular Expressions Anywhere when patterns of text need to be specified Specifying restriction enzymes Google analytics Unix system, database and networking administration: grep, fgrep, egrep, sed, awk HTML documents: Javascript and VBScript Perl: J. Friedl, Mastering Regular Expressions, O reilly, 1997 Token Specs for scanner generators (lex, Jflex, etc.) COMP3131/9102 Page 70 March 4, 2018

7 Applications of Finite Automata (i.e., Finite State Machines) Hardware design (minimising states = minimising cost) Language theory Computational complexity Scanner generators (lex and Jflex) Automata tools: downloads/ 39c c-49a3-ac9c-40d807010c07/ COMP3131/9102 Page 71 March 4, 2018

8 Alphabet, Strings and Languages Alphabet denoted Σ: any finite set of symbols The binary alphabet {0,1} (for machine languages) The ASCII alphabet (for high-level languages) String: a finite sequence of symbols drawn from Σ: Length s of a string s: the number of symbols in s ǫ: the empty string ( ǫ = 0) Language: any set of strings overσ; its two special cases: : the empty set {ǫ} COMP3131/9102 Page 72 March 4, 2018

9 Examples of Languages Σ = {0,1} a string is an instruction The set of M68K instructions The set of Pentium instructions The set of MIPS instructions Σ = the ASCII set a string is a program the set of Haskell programs the set of C programs the set of VC programs COMP3131/9102 Page 73 March 4, 2018

10 Terms for Parts of a String (Figure 3.7 of Text) TERM prefix ofs suffix ofs substring ofs proper prefix suffix, substring ofs DEFINITION a string obtained by removing 0 or more trailing symbols ofs a string obtained by removing 0 or more leading symbols ofs a string obtained by deleting a prefix and a suffix from s Any nonempty string x that is, respectively, a prefix, suffix or substring ofssuch that s x COMP3131/9102 Page 74 March 4, 2018

11 String Concatenation If x and y are strings, xy is the string formed by appending y to x Examples: x y xy key word keyword java script javascript ǫ is the identity: ǫx = xǫ = x COMP3131/9102 Page 75 March 4, 2018

12 Operations on Languages (Figure 3.8 of Text) OPERATION DEFINITION union: L M L M = {s s L or s M} concatenation: LM LM = {st s L and t M} Kleene Closure: L L = i=0 Li = L 0 L LL LLL... wherel 0 = {ǫ} (0 or more concatenations of L) Positive Closure: L + L + = i=1 Li = L LL LLL... (1 or more concatenations of L) COMP3131/9102 Page 76 March 4, 2018

13 Examples: Operations on Languages L = {a,...,z,a,...,z, } D = {0,...,9} EXAMPLE LANGUAGE (THE SET OF ) L D L 3 LD L L(L D) D + COMP3131/9102 Page 77 March 4, 2018

14 Examples: Operations on Languages L = {a,...,z,a,...,z, } D = {0,...,9} EXAMPLE LANGUAGE L D letters and digits L 3 all 3-letter strings LD strings consisting of a letter followed by a digit L all strings of letters, including the empty string ǫ L(L D) all strings of letters and digits beginning with a letter D + all strings of one or more digits COMP3131/9102 Page 78 March 4, 2018

15 Regular Expressions (REs) Over Alphabet Σ Inductive Base: 1. ǫ is a RE, denoting the RL {ǫ} 2. a Σ is a RE, denoting the RL {a} Inductive Step: Suppose r and s are REs, denoting the RLsL(r) and L(s). Then (next slide): 1. (r) (s) is a RE, denoting the RL L(r) L(s) 2. (r)(s) is a RE, denoting the RL L(r)L(s) 3. (r) is a RE, denoting the RL L(r) 4. (r) is a RE, denoting the RL L(r) REs define regular languages (RL) or regular sets COMP3131/9102 Page 79 March 4, 2018

16 Precedence and Associativity of Regular Operators Precedence: has the highest precedence Concatenation has the second highest precedence has the lowest precedence Associativity: all are left-associative Example: (a) ((b) (c)) a b c Unnecessary parentheses can be avoided! COMP3131/9102 Page 80 March 4, 2018

17 An Example (Following the Definition of REs) Alphabet: Σ = {0,1} RE: 0(0 1) Question: What is the language defined by the RE? Answer: L(0(0 1) ) = L(0)L((0 1) ) = {0}L(0 1) = {0}(L(0) L(1)) = {0}({0} {1}) = {0}{0,1} = {0}{ǫ,0,1,00,01,10,11,...} = {0,00,01,000,001,010,011,...} The RE describes the set of strings of 0 s and 1 s beginning with a 0. COMP3131/9102 Page 81 March 4, 2018

18 More Example Regular Expressions: Σ = {0, 1} RE 1 {1} 0 1 {0,1} LANGUAGE 1 {ǫ,1,11,111,...} 1 1 {1,11,111,...} the set containing 0 and all strings consisting of zero or more 0 s followed by a 1. COMP3131/9102 Page 82 March 4, 2018

19 Notational Shorthands One or more instances + : r + = rr denotes the language (L(r)) + has the same precedence and associativity as Zero or one instance?: r? = r ǫ denotes the language L(r) {ǫ} written as(r)? to indicate grouping (e.g., (12)?) Character classes: [A Za z ][A Za z0 9 ] COMP3131/9102 Page 83 March 4, 2018

20 Regular Expressions for VC (or C) TOKEN Identifiers Integers digit + Reals RE letter(letter digit) A bit long but can be obtained from the following page by substitutions In the VC spec, letter includes In Java, letters and digits may be drawn from the entire Unicode character set. Examples of identifiers are: abc αβγ COMP3131/9102 Page 84 March 4, 2018

21 Regular Grammars for Integers and Reals in VC Integers: Reals: digit: intliteral: digit + digit: fraction:.digit + exponent: (E e)(+ -)?digit + floatliteral: digit fraction exponent? digit +. digit +.?exponent Regular grammars are a special case of CFGs (Week 3). COMP3131/9102 Page 85 March 4, 2018

22 Finite Automata (or Finite State Machines) A finite automaton consists of a 5-tuple: (Σ,S,T,F,I) where Σ is an alphabet S is a finite set of states T is a state transition function: T : S Σ S F is a finite set of final or accepting states I is the start state: I S. COMP3131/9102 Page 86 March 4, 2018

23 Representation and Acceptance Transition graph: state A transition A a B start state final state start A S Acceptance: A FA accepts an input string x iff there is some path in the transition graph from the start state to some accepting state such that the edge labels spell out x. COMP3131/9102 Page 87 March 4, 2018

24 What Language does this FA accept? start S 1 A COMP3131/9102 Page 88 March 4, 2018

25 Example 1 The language: strings of 0 and 1 with an odd number of 1 (ǫ not included) start 0 S 1 1 A 0 S: even number of 1 s seen A: odd number of 1 s seen Σ {0,1} S {S,A} T T(S,0) = S,T(S,1) = A,T(A,0) = A, T(A,1) = S F {A} I S is recognised because S 0 S 1 A 0 A 1 S 1 A COMP3131/9102 Page 89 March 4, 2018

26 Implicit Error State By definition, T is a function from S Σ tos, but S A start 1 If T(s,a) is undefined at the state s on inputa, then T(s,a) = error S A error start 1 The error state and transitions to it aren t drawn (by convention) COMP3131/9102 Page 90 March 4, 2018

27 Deterministic FA (DFA) and Nondeterministic FA (NFA) A FA is a DFA if no state has an ǫ-transition, i.e., an transition on input ǫ, and for each state s and input symbol a, there is at most one edge labeled a leaving s A FA is an NFA if it is not a DFA: Nondeterministic: can make several parallel transitions on a given input Acceptance: the existence of some path as per Slide 87 COMP3131/9102 Page 91 March 4, 2018

28 DFA or NFA? What are the Languages Recognised? a a a a a start a a a a ǫ ǫ a start a a COMP3131/9102 Page 92 March 4, 2018

29 Two Examples NFA 1: NFA 2: a a a a a start a a a a ǫ ǫ a start a a The same language: the set of all strings of a s such that the length of each of these strings is a multiple of 2 or 3 (ǫ included) COMP3131/9102 Page 93 March 4, 2018

30 Real-Life DFAs The ghosts in Pac-Man have four behaviors: 1. Randomly wander the maze 2. Chase Pac-Man, when he is within line of sight 3. Flee Pac-Man, after Pac-Man has consumed a power pellet 4. Return to the central base to regenerate COMP3131/9102 Page 94 March 4, 2018

31 Real-Life DFAs The behavior of a vending machine: accepts dollars and 25 cents, and charges $1.25 per coke. COMP3131/9102 Page 95 March 4, 2018

32 What About this Non-Real-Life NFA? ǫ COMP3131/9102 Page 96 March 4, 2018

33 Week 2: Regular Expressions, DFA and NFA 1. Definitions of REs, DFA and NFA 2. REs = NFA (Thompson s construction, Algorithm 3.3, Red Dragon/Algorithm 3.23, Purple Dragon) 3. NFA = DFA (subset construction, Algorithm 3.2, Red Dragon/Algorithm 3.20, Purple Dragon) 4. DFA = minimal-state DFA (state minimisation, Algorithm 3.6, Red Dragon/Algorithm 3.39, Purple Dragon) 5. Scanner generators How to use them (straightforward) How to write them (the most techniques introduced today) COMP3131/9102 Page 97 March 4, 2018

34 Thompson s Construction of NFA from REs Syntax-driven Inductive: The cases in the construction of the NFA follow the cases in the definition of REs Important: if a symbol a occurs several times in a RE r, a separate NFA is constructed for each occurrence Thompson s method is one of many available COMP3131/9102 Page 98 March 4, 2018

35 Thompson s Construction Inductive Base: 1. For ǫ, construct the NFA start S ǫ A 2. For a Σ, construct the NFA start S a A Inductive step: suppose N(r) and N(s) are NFAs for REs r and s. Then COMP3131/9102 Page 99 March 4, 2018

36 Thompson s Construction (Cont d) REr s : start S ǫ ǫ N(r) N(s) ǫ ǫ A RE rs : start S N(r) N(s) A ǫ RE r : start S ǫ N(r) ǫ A ǫ RE (r) : N((r)) is the same as N(r) COMP3131/9102 Page 100 March 4, 2018

37 Example: RE = NFA Converting(0 10 1) 10 to an NFA COMP3131/9102 Page 101 March 4, 2018

38 Example: RE = NFA Regular expression: (0 10 1) 10 NFA: ǫ ǫ start ǫ ǫ 0 ǫ ǫ ǫ ǫ ǫ ǫ 9 10 ǫ 15 ǫ 1 ǫ 0 ǫ ǫ ǫ COMP3131/9102 Page 102 March 4, 2018

39 Week 2: Regular Expressions, DFA and NFA 1. Definitions of REs, DFA and NFA 2. REs = NFA (Thompson s construction, Algorithm 3.3, Red Dragon/Algorithm 3.23, Purple Dragon) 3. NFA = DFA (subset construction, Algorithm 3.2, Red Dragon/Algorithm 3.20, Purple Dragon) 4. DFA = minimal-state DFA (state minimisation, Algorithm 3.6, Red Dragon/Algorithm 3.39, Purple Dragon) 5. Scanner generators How to use them (straightforward) How to write them (the most techniques introduced today) COMP3131/9102 Page 103 March 4, 2018

40 Example: DFA (Cont d) start 0,1,2,5,10 1 6,7,9,11,12, ,2,3,4,5,10 1,2,4,5,10, ,8,9, 12,13,14 0 The algorithm used is known as the subset construction, because a DFA state corresponds to a subset of NFA states There are at most 2 n DFA states, where n is the total number of the NFA states COMP3131/9102 Page 104 March 4, 2018

41 Subset Construction: The Operations Used OPERATION ǫ-closure(s) ǫ-closure(t ) move(t, a) DESCRIPTION Set of NFA states readable from NFA stateson ǫ-transitions Set of NFA states readable from some state s int on ǫ-transitions Set of NFA states to which there is a transition on input a from some statesin T s: a NFA state T : a set of NFA states COMP3131/9102 Page 105 March 4, 2018

42 Subset Construction: The Algorithm Lets 0 be the start state of the NFA; DFAstates contains the only unmarked state ǫ-closure(s 0 ); while there is an unmarked statet in DFAstatesdo begin mark T for each input symbol ado begin U := ǫ-closure(move(t, a)); if U is not in DFAstatesthen Add U as an unmarked state in DFAstates; DFATrans[T,a] := U; end; end; COMP3131/9102 Page 106 March 4, 2018

43 Subset Construction: The Definition of the DFA Let(Σ,S,T,F,s 0 ) be the original NFA. The DFA is: The alphabet: Σ The states: all states in DFAstates The start state: ǫ-closure(s 0 ) The accepting states: all states in DFAstates containing at least one accepting state inf of the NFA The transitions: DFATrans COMP3131/9102 Page 107 March 4, 2018

44 Week 2: Regular Expressions, DFA and NFA 1. Definitions of REs, DFA and NFA 2. REs = NFA (Thompson s construction, Algorithm 3.3, Red Dragon/Algorithm 3.23, Purple Dragon) 3. NFA = DFA (subset construction, Algorithm 3.2, Red Dragon/Algorithm 3.20, Purple Dragon) 4. DFA = minimal-state DFA (state minimisation, Algorithm 3.6, Red Dragon/Algorithm 3.39, Purple Dragon) 5. Scanner generators How to use them (straightforward) How to write them (the most techniques introduced today) COMP3131/9102 Page 108 March 4, 2018

45 An Algorithm to Mimimise DFA Statements Initially, let Π be the partition with the two groups: (1) one is the set of all final states (2) the other is the set of all non-final states Let Π new = Π for (each group G in Π new ) { partition G into subgroups such that two states s and t are in the same subgroup iff for all input symbols a, states s and t have transitions on a to states in the same group of Π new replace G in Π new by the set of subgroups formed } Begins with the most optimistic assumption Also used in global value numbering (COMP4133) COMP3131/9102 Page 109 March 4, 2018

46 Example (Cont d): States Re-Labeled start A 0 D 1 0 B E 0 C 0 0 A,D,E start state minimisation 1 B,C 0 1 Theoretical Result: every regular language can be recognised by a minimal-state DFA that is unique up to state names COMP3131/9102 Page 110 March 4, 2018

47 Various Conversions for an Example (0 10 1) 10 NFA in Slide 102 DFA in Slide 110 DFA minimal-state DFA in Slide 110 However, the conversions in dashed arrows are not covered. COMP3131/9102 Page 111 March 4, 2018

48 Week 2: Regular Expressions, DFA and NFA 1. Definitions of REs, DFA and NFA 2. REs = NFA (Thompson s construction, Algorithm 3.3, Red Dragon/Algorithm 3.23, Purple Dragon) 3. NFA = DFA (subset construction, Algorithm 3.2, Red Dragon/Algorithm 3.20, Purple Dragon) 4. DFA = minimal-state DFA (state minimisation, Algorithm 3.6, Red Dragon/Algorithm 3.39, Purple Dragon) 5. Scanner generators How to use them (straightforward) How to write them (the most techniques introduced today) COMP3131/9102 Page 112 March 4, 2018

49 Scanner Generators Scanners generated in C lex (UNIX) flex GNU s fast lex (UNIX) mks lex (MS-DOS and OS/2) Scanners generated in Java Jflex JavaCC (SUN Microsystems) COMP3131/9102 Page 113 March 4, 2018

50 The Scanner Spec in Jflex user code -- copied verbatim to the scanner file %% Jflex directives %% regular expression rules COMP3131/9102 Page 114 March 4, 2018

51 How a Scanner Generator Works Token Spec using REs NFA DFA minimal-state DFA table-driven code (Jflex) simulating a DFA on an input or hard-wired code (Assignment 1 or 3.4 of either Dragon Book) COMP3131/9102 Page 115 March 4, 2018

52 An Example: Spec some user code %% LETTER=[A-Za-z_] DIGIT=[0-9] %% "if" { return new Token(Token.IF, "if", pos); } "<" { return new Token(Token.LT, "<", pos); } "<=" { return new Token(Token.LE, "<=", pos); } {LETTER}({LETTER} {DIGIT})* { return new Token(Token.ID, "itsspelling", pos); } Two rules: The first pattern used when more than one are matched if as a keyword not as an id The longest prefix of the input is always matched <= as one token COMP3131/9102 Page 116 March 4, 2018

53 An Example: NFA N(if) start S ǫ ǫ ǫ ǫ N(<) N(<=) N(id) A DFA can also be used for each pattern. COMP3131/9102 Page 117 March 4, 2018

54 An Example: NFA i f start 0 ǫ ǫ ǫ ǫ 4 < 5 6 < 7 = 8 LETTER 9 10 LETTER, DIGIT COMP3131/9102 Page 118 March 4, 2018

55 DIGIT LETTER but f An Example: DFA f 2, 10 3, 10 LETTER, DIGIT start i < = 5, 7 8 0,1,4,6,9 LETTER but i 10 LETTER, DIGIT Already a minimal-state DFA! COMP3131/9102 Page 119 March 4, 2018

56 A DFA Represented as a Transition Table State Character < = i f LETTER but i LETTER but f DIGIT (0,1,4,6,9) (5,7) (2,10) (10) (2,10) (3,10) (10) (10) (5,7) (8) (8) (10) (10) (10) (10) (10) (10) (3,10) (10) (10) (10) (10) (10) Letter ={i} letter but i Character classes reduce the table size The blank entries are errors The tables are usually sparse (pages of text for compression techniques) COMP3131/9102 Page 120 March 4, 2018

57 The Scanner Driver for Simulating a DFA state = initial_state while (TRUE) { next_state = T[state][current_char]; if (next_state == ERROR) // cannot move any further break; state = next_state; if (current_char == EOF) // input exhausted break; current_char = getchar(); // fetch the next char } Backtrack to the most recent accepting state if (such a state exists) /* return the corresponding token reset current_char to the first after the token */ else lexical_error(state); There should be a column in the transition table for EOF Need to backtrack COMP3131/9102 Page 121 March 4, 2018

58 The Output of Running Jflex on a Sample Scanner Spec Scanner.l: the spec for the scanner generator Jflex jflex Scanner.l Constructing NFA : 267 states in NFA Converting NFA to DFA : 139 states before minimization, 106 states in minimized DFA Old file Scanner.java saved as Scanner.java~ Writing code to Scanner.java Scanner.l.java: the scanner generated javac Scanner.l.java java Scanner < test.vc COMP3131/9102 Page 122 March 4, 2018

59 Limitations of Regular Expressions (or FAs) Cannot count Cannot recognise palindromes (e.g., racecar & rotator) The language of the balanced parentheses is not a regular language {( n ) n n 1} cannot build a FA to recognise the language for any n (can trivially build a FA for n=3, for example) but can be specified by a CFG (Week 3): P (P) () COMP3131/9102 Page 123 March 4, 2018

60 Chomsky s Hierarchy Depending on the form of production α β four types of grammars (and accordingly, languages) are distinguished: GRAMMAR KNOWN AS DEFINITION LANGUAGE MACHINE Type 0 unrestricted grammar α ǫ Type 0 Turing machine Type 1 Type 2 context-sensitive grammar CSGs context-free grammar CFGs α β Type 1 linear bounded automaton A α Type 2 stack automaton Type 3 Regular grammars A w Bw Type 3 finite state automaton COMP3131/9102 Page 124 March 4, 2018

61 Reading Sections of either Dragon Book Week 3 tutorial questions (available on-line) Week 3: Context-Free Grammars COMP3131/9102 Page 125 March 4, 2018

COMP-421 Compiler Design. Presented by Dr Ioanna Dionysiou

COMP-421 Compiler Design. Presented by Dr Ioanna Dionysiou COMP-421 Compiler Design Presented by Dr Ioanna Dionysiou Administrative! [ALSU03] Chapter 3 - Lexical Analysis Sections 3.1-3.4, 3.6-3.7! Reading for next time [ALSU03] Chapter 3 Copyright (c) 2010 Ioanna