Lecture 3: Lexical Analysis

Size: px

Start display at page:

Download "Lecture 3: Lexical Analysis"

Rose Terry
5 years ago
Views:

1 Lecture 3: Lexical Analysis COMP 524 Programming Language Concepts tephen Olivier January 2, 29 Based on notes by A. Block, N. Fisher, F. Hernandez-Campos, J. Prins and D. totts

Goal of Lecture Character tream Token tream Parse

analysis) Parser (syntax analysis) emantic

optimization (optional) ymbol Table Machine

2 Goal of Lecture Character tream Token tream Parse Tree Abstract syntax tree canner (lexical analysis) Parser (syntax analysis) emantic analysis & intermediate code gen. This includes regular expressions. Modified intermediate form Machine-independent optimization (optional) ymbol Table Machine language Modified target language Target code generation. Machine-specific optimization (optional)

3 canning The main task of scanning is to identify tokens.

4 Pseudo-Code canner (Fig 2.5) We skip any initial white spaces we read the next character if it is a ( we look at the next character if that is a * we have a comment; we skip forward through the terminating *) otherwise we return a ( and reuse the look-ahead If it is one of the one-character tokens ([],;=+- etc.) we return that token...

5 Pseudo-Code canner (Fig 2.5) We skip any initial white spaces we read the next character if it is a ( we look at the next character if that is We a could * we have just turn a comment; this into real code and we skip use forward that as through the scanner, the terminating and that would *) be otherwise we return a ) and reuse the look-ahead If it is one of the fine one-character for small programs... tokens ([],;=+- etc.) we return that token...

6 Pseudo-Code canner (Fig 2.5) We skip any initial white spaces we read the next character if it is a ( we look at the next character if that... is However, a * have for a larger comment; programs that must we skip be forward correct through a more the formal terminating approach *) is more otherwise we return a ) and reuse the look-ahead If it is one of the one-character appropriate. tokens ([],;=+- etc.) we return that token...

7 Regular expressions digit non_zero_digit natural_number non_zero_digit digit* non_neg_number ( natural_number) ( (. digit* non_zero_digit) )

8 Regular expressions digit non_zero_digit natural_number non_zero_digit digit* denotes assignment non_neg_number ( natural_number) ( (. digit* non_zero_digit) )

9 Regular expressions digit non_zero_digit natural_number non_zero_digit digit* denotes or non_neg_number ( natural_number) ( (. digit* non_zero_digit) )

10 Regular expressions digit non_zero_digit natural_number non_zero_digit digit* Thus, digit equal or or 2 or... non_neg_number ( natural_number) ( (. digit* non_zero_digit) )

11 Regular expressions * denotes zero or more of this type. digit non_zero_digit natural_number non_zero_digit digit* non_neg_number ( natural_number) ( (. digit* non_zero_digit) )

12 Regular expressions Two REs next to each other denotes concatenation. digit non_zero_digit natural_number non_zero_digit digit* non_neg_number ( natural_number) ( (. digit* non_zero_digit) )

13 Regular expressions o natural number equals at least one non-zero digit followed by digit zero or 2 more digits non_zero_digit natural_number non_zero_digit digit* non_neg_number ( natural_number) ( (. digit* non_zero_digit) )

7 8 9 natural_number non_zero_digit digit*

14 Regular expressions means empty. digit non_zero_digit natural_number non_zero_digit digit* non_neg_number ( natural_number) ( (. digit* non_zero_digit) )

15 Regular expressions o, what does this mean? digit non_zero_digit natural_number non_zero_digit digit* non_neg_number ( natural_number) ( (. digit* non_zero_digit) )

16 Regular expressions It means or a natural number followed by nothing or by. and zero or more digit digits concluded by a non_zero number non_zero_digit natural_number non_zero_digit digit* non_neg_number ( natural_number) ( (. digit* non_zero_digit) )

17 Regular Expression Rules A RE consist of: A character (e.g.,,,...) The empty string (i.e., ) Two REs next to each other (e.g., non_negative_digit digit ) to denote concatenation. Two REs separated by next to each other (e.g., non_negative_digit digit ) to denote one RE or the other. An RE followed by * (called the Kleene star) to denote zero or more iterations of the RE. Parentheses (in order to avoid ambiguity).

18 Regular Expression Rules A RE consist of: A character (e.g.,,,...) The empty string (i.e., ) A RE is NEVER defined in terms of itself! Thus, REs cannot define recursive statements. Two REs next to each other (e.g., non_negative_digit digit ) to denote concatenation. Two REs separated by next to each other (e.g., non_negative_digit digit ) to denote one RE or the other. An RE followed by * (called the Kleene star) to denote zero or more iterations of the RE. Parentheses (in order to avoid ambiguity).

19 Regular Expression Rules A RE consist of: A character (e.g.,,,...) The empty string (i.e., ) The set of tokens that can be recognized by Two REs next to each other (e.g., non_negative_digit digit ) to regular expressions is called a regular set. denote concatenation. Two REs separated by next to each other (e.g., non_negative_digit digit ) to denote one RE or the other. An RE followed by * (called the Kleene star) to denote zero or more iterations of the RE. Parentheses (in order to avoid ambiguity).

20 Deterministic finite automaton (DFA) Every regular set can be defined by using deterministic finite automaton (DFA). DFAs are turing machines that have a finite number of states and deterministically move between states.

21 Deterministic finite automaton (DFA) Every regular set can be defined by using deterministic finite automaton (DFA). tart tate DFAs are turning machines that have a finite number of states and deterministically move between states.

22 Deterministic finite automaton (DFA) Every regular set can be defined by using deterministic finite automaton (DFA). Intermediate tate DFAs are turning machines that have a finite number of states and deterministically move between states.

23 Deterministic finite automaton (DFA) Every regular set can be defined by using deterministic finite automaton (DFA). End tate (double circle) DFAs are turning machines that have a finite number of states and deterministically move between states.

24 Deterministic finite automaton (DFA) Every regular set can be defined by using deterministic finite automaton This (DFA). stands for: **( )* DFAs are turning machines that have a finite number of states and deterministically move between states.

25 Constructing DFAs A DFA can be constructed from a RE via two steps.. Construct a nondeterministic finite automaton (NFA) from the RE. 2. Construct a DFA from the NFA. 3. Minimize the DFA

26 What is an NFA? An NFA is similar to a DFA, except that state transitions are nondeterministic. This nondeterminism is encapsulated via the epsilon transition (written as ).

27 What is an NFA? An NFA is similar to a DFA, except that state transitions are nondeterministic. This stands for: This nondeterminism is encapsulated via the epsilon transition (written as ).

28 What is an NFA? The transitions imply that either An NFA transition is similar can to be a DFA, taken except with any that (or state transitions are nondeterministic. no) input. This nondeterminism is encapsulated via the epsilon transition (written as ).

29 The four RE rules and NFA Rule --Base case: a a

30 The four RE rules and NFA Rule 2--Concatenation: AB plus A B AB

31 tands for ome NFA called A The four RE rules and NFA Rule 2--Concatenation: AB plus A B AB

32 The four RE rules and NFA Rule 3--Alternation: A B or A B A B A B

33 The four RE rules and NFA Rule 3--Alternation: A B Notice the epsilon transitions. or A B A B A B

34 The four RE rules and NFA Rule 4--Kleene Closure: A* empty or repeated A A A*

35 The four RE rules and NFA Rule 4--Kleene Closure: A* Notice the epsilon transitions. empty or repeated A A A*

36 ome examples: a * A B AB* F (GH*) Z* Y*X A B A

37 Constructing DFAs A DFA can be constructed from a RE via two steps.. Construct a nondeterministic finite automaton (NFA) from the RE. 2. Construct a DFA from the NFA. 3. Minimize the DFA

38 Constructing DFAs A DFA can be constructed from a RE via two steps.. Construct a nondeterministic finite automaton (NFA) from the RE. FOR_KEYWORD for IDENTIFIER Alph ( Alph Dig )* REAL Dig Dig*. Dig* INT Dig Dig*

39 Constructing DFAs A DFA can be constructed from a RE via two steps.. Construct a nondeterministic finite automaton (NFA) from the RE. 2. Construct a DFA from the NFA. 3. Minimize the DFA

40 Constructing a DFA from an NFA. Construct the DFA by collapsing the states of an NFA. Three steps.identify set of states that can be reached from the start state via epsilon-transitions and make this one state. 2.For a given DFA state (which is a set of NFA states) consider each possible input and combine the resulting NFA states into one DFA state. 3.Repeat tep 2 until all states have been added.

41 An example ** tart,a,b,f A C E C,G,E,F B D B,D,F F G G,E,F NFA DFA

42 All the states that we can reach via are in this state An example ** tart,a,b,f A C E C,G,E,F B F D G G,E,F B,D,F NFA DFA

43 All the states that we can reach via or from ABF ** An example tart,a,b,f A C E C,G,E,F B F D G G,E,F B,D,F NFA DFA

44 All the states that we can reach via or from ABF ** An example tart,a,b,f A C E C,G,E,F B F D G G,E,F B,D,F NFA DFA

45 All the states that we can reach via or from B,D,F or C,G,F ** An example tart,a,b,f A C E C,G,E,F B F D G G,E,F B,D,F NFA DFA

46 An example ** elf Loop. tart,a,b,f A C E C,G,E,F B D B,D,F F G G,E,F NFA DFA

47 An example ** tart,a,b,f A C E C,G,E,F B D B,D,F F G G,E,F NFA DFA

48 Minimize via partitioning First, partition states into final and non-final econd, determine the effect of the state transition based on what partition the transition goes to. Third, Create new partition for those states that have different transitions. Fourth, repeat.

49 An example tart C,G,E,F,A,B,F X- X-2 tate ABF X-2 X- BDF X-2 X- B,D,F CGEF X-2 N/A G,E,F GEF X-2 N/A DFA

50 An example tart X- X-2 X- X-2 tate ABF X-2 X- BDF X-2 X- CGEF X-2 N/A GEF X-2 N/A

51 canner Code Can create canner from the DFA one of two ways: Nested case statements (Handwritten) Tables (easy to generate from code, hard to write by hand)

53 Two complications--nested Case Keywords It is possible to maintain a DFA for keywords, but the number of states would be even larger! o, they are handled as exceptions to the rule. Dot-Dot Pascal uses.. to denote a range of numbers; however, to determine the meaning of the.. we need to look ahead after reading the first. to determine if. denotes the end of a token or a beginning of a new token. 3.4 one token three tokens

54 This code specifies a twodimensional transition table, which tells whether to move, return token, or announce error

55 A second table tells when we might have hit the end of a token (for backing up)

56 Pragmas Pragmas are comments that provide direction for the compiler. For example, Variable x is used a lot, keep it in memory if possible. These are often handled by the parser since this makes the grammar much simpler.

Lexical Analysis. COMP 524, Spring 2014 Bryan Ward

Lexical Analysis. COMP 524, Spring 2014 Bryan Ward Lexical Analysis COMP 524, Spring 2014 Bryan Ward Based in part on slides and notes by J. Erickson, S. Krishnan, B. Brandenburg, S. Olivier, A. Block and others The Big Picture Character Stream Scanner