CS308 Compiler Principles Lexical Analyzer Li Jiang


 Nicholas McLaughlin
 2 years ago
 Views:
Transcription
1 CS308 Lexical Analyzer Li Jiang Department of Computer Science and Engineering Shanghai Jiao Tong University
2 Content: Outline Basic concepts: pattern, lexeme, and token. Operations on languages, and regular expression Recognition of tokens Finite automata, including NFA and DFA Conversion from regular expression to NFA and DFA Optimization of lexical analyzer 2
3 Lexical Analyzer Lexical Analyzer reads the source program character by character to produce tokens. strips out comments and whitespaces returns a token when the parser asks for correlates error messages with the source program 3
4 Token A token is a pair of a token name and an optional attribute value. Token name specifies the pattern of the token Attribute stores the lexeme of the token Tokens Keyword: begin, if, else, Identifier: string of letters or digits, starting with a letter Integer: a nonempty string of digits Punctuation symbol:,, ;, (, ), Regular expressions are widely used to specify patterns of the tokens. 4
5 Attributes of Token Information for subsequent compiler phases about the particular lexeme Token name influences parsing decision attribute value influences translation of tokens after the parse Attributes of identifier Lexeme, type, location Stored in symbol table Tricky problem DO 5 I = 1.25 VS. DO 5 I = 1,25 5
6 Token Example 6
7 Content: Outline Basic concepts: pattern, lexeme, and token. Operations on languages, and regular expression Recognition of tokens Finite automata, including NFA and DFA Conversion from regular expression to NFA and DFA Optimization of lexical analyzer 7
8 Input Buffering Why a compiler needs buffers? Buffer Pairs: alternately reload Two pointers lexemebegin forward Sentinels: a mark for buffer end If length of lexeme + look ahead distance > buffer size 8
9 Lookahead with Sentinels 9
10 Terminology of Languages Alphabet: a finite set of symbols ASCII Unicode String: a finite sequence of symbols on an alphabet is the empty string s is the length of string s Concatenation: xy represents x followed by y Exponentiation: s n = s s s.. s ( n times) s 0 = Language: a set of strings over some fixed alphabet the empty set is a language The set of wellformed C programs is a language 10
11 Operations on Languages Union: L 1 L 2 = { s s L 1 or s L 2 } Concatenation: L 1 L 2 = { s 1 s 2 s 1 L 1 L 2 } and s 2 (Kleene) Closure: Positive Closure: L * i0 1 L i i L i L 11
12 Example L 1 = {a,b,c,d} L 2 = {1,2} L 1 L 2 = {a,b,c,d,1,2} L 1 L 2 = {a1,a2,b1,b2,c1,c2,d1,d2} L 1 * = L 1+ = all strings using letters a,b,c,d including the empty string all strings using letters a,b,c,d without the empty string 12
13 Regular Expressions Regular expression is a representation of a language that can be built from the operators applied to the symbols of some alphabet. A regular expression is built up of smaller regular expressions (using defining rules). Each regular expression r denotes a language L(r). A language denoted by a regular expression is called as a regular set. 13
14 Regular Expressions (Rules) Regular expressions over alphabet Reg. Expr a (r 1 ) (r 2 ) L(r 1 ) L(r 2 ) (r 1 ) (r 2 ) L(r 1 ) L(r 2 ) (r) * (L(r)) * (r) L(r) Language it denotes L() = {} L(a) = {a} Extension (r) + = (r)(r) * (L(r)) + Positive closure (r)? = (r) L(r) {} zero or one instance [a 1 a n ] L(a 1 a 2 a n ) character class 14
15 Regular Expressions (cont.) We may remove parentheses by using precedence rules: * highest concatenation second highest lowest ab* c (a(b) * ) (c) Example: = {0,1} 0 1 => {0,1} (0 1)(0 1) => {00,01,10,11} 0 * => {,0,00,000,0000,...} (0 1) * => all strings with 0 and 1, including the empty string 15
16 Lex regular expression 16
17 Regular Definitions We can give names to regular expressions, and use these names as symbols to define other regular expressions. 17 A regular definition is a sequence of the definitions of the form: d 1 r 1 where d i is a innovative symbol and d 2 r 2 r i is a regular expression over symbols in {d 1,d 2,...,d i1 } d n r n alphabet previously defined symbols
18 Regular Definitions Example Example: Identifiers in Pascal letter A B... Z a b... z digit id letter (letter digit ) * If we try to write the regular expression representing identifiers without using regular definitions, that regular expression will be complex. (A... Z a... z) ( (A... Z a... z) (0... 9) ) * Q: unsigned numbers (integer or floating point) 18
19 Quiz * 1. All strings of lowercase letters that contain the five vowels in order. 2. All strings of lowercase letters in which the letters are in ascending lexicographic order. 3. Comments, consisting of a string surrounded by /* and */, without an intervening */, unless it is inside doublequotes ( ). [HOMEWORK] 19
20 Content: Outline Basic concepts: pattern, lexeme, and token. Operations on languages, and regular expression Recognition of tokens Finite automata, including NFA and DFA Conversion from regular expression to NFA and DFA Optimization of lexical analyzer 21
21 Recognition of token Express the pattern Grammar 22 Find a prefix that is a lexeme matching the pattern Regular Definitions
22 Transition Diagram * State: represents a condition that could occur during scanning start/initial state: accepting/final state: lexeme found intermediate state: Edge: directs from one state to another, labeled with one or a set of symbols 23
23 Transition Diagram for relop Among the lexemes that match the pattern for relop, what can we only be looking at? Transition Diagram for ``relop < > < = >= = <> 24
24 TransitionDiagramBased Lexical Analyzer Switch statement or multi way branch Holds the number of the current state Determines the next state by reading and examining the next input character Find the edge Take action 25 Implementation of relop transition diagram
25 Transition Diagram for Others * What about the Transition Diagram of letter/digit? A transition diagram for id's A transition diagram for unsigned numbers 26
26 Content: Outline Basic concepts: pattern, lexeme, and token. Operations on languages, and regular expression Recognition of tokens Finite automata, including NFA and DFA Conversion from regular expression to NFA and DFA Optimization of lexical analyzer 29
27 Finite Automata A finite automaton is a recognizer that takes a string, and answers yes if the string matches a pattern of a specified language, and no otherwise. * Two kinds: Nondeterministic finite automaton (NFA) no restriction on the labels of their edges Deterministic finite automaton (DFA) exactly one edge with a distinguished symbol goes out of each state Both NFA and DFA have the same capability We may use NFA or DFA as lexical analyzer 30
28 Nondeterministic Finite Automaton (NFA) A NFA consists of: S: a set of states Σ: a set of input symbols (alphabet) A transition function: maps statesymbol pairs to sets of states s 0 : a start (initial) state F: a set of accepting states (final states) NFA can be represented by a transition graph Accepts a string x, if and only if there is a path from the starting state to one of accepting states such that edge labels along this path spell out x. Remarks The same symbol can label edges from one state to several different states An edge may be labeled by ε, the empty string 31
29 NFA Example (1) The language recognized by this NFA is (a b) * a b 32
30 NFA Example (2) NFA accepting aa* bb* 33
31 Implementing an NFA S closure({s 0 }) c nextchar() while (c!= eof) { begin S closure(move(s,c)) { set all of states can be accessible from s 0 by transitions } { set of all states can be accessible from a state in S by a transition on c} c nextchar end if (SF!= ) then { if S contains an accepting state } return yes else return no backtrack may be needed to identify the longest match. Subset Construction 34
32 Excise 3 For NFA in the following figure, indicate all the paths labeled aabb. Does the NFA accept aabb? Give the transition table.  (0) a> (1) a> (2) b> (2) b> ((3)) (0) a> (1) a> (2) b> (2) b> (2)  (0) a> (0) a> (0) b> (0) b> (0) (0) a> (0) a> (1) b> (1) b> (1)  (0) a> (1) a> (1) b> (1) b> (1) (0) a> (1) a> (2) b> (2) ε> (0) b> (0)  (0) a> (1) a> (2) ε> (0) b> (0) b> (0) 35
33 Deterministic Finite Automaton (DFA) A Deterministic Finite Automaton (DFA) is a special form of a NFA. No state has ε transition For each symbol a and state s, there is at most one a labeled edge leaving s. start The language recognized by this DFA is?(a b) * a b 36
34 Practice * Draw the transition diagram for recognizing the following regular expression a(a b)*a a b b a a a Nondeterministic a a b Deterministic 37
35 Implementing a DFA s s 0 { start from the initial state } c nextchar { get the next character from the input string } while (c!= eof) do { do until the end of the string } begin s move(s,c) { transition function } c nextchar end if (s in F) then { if s is an accepting state } return yes else return no 38
36 NFA vs. DFA Compactibility Readability Speed NFA Good Good Slow DFA Bad Bad Fast DFAs are widely used to build lexical analyzers. Maintaining a set of state is more complex than keeping track a single state. 39 NFA DFA The language recognized (a b) * a b
37 Pop Quiz 1) What are the languages presented by the two FAs? (a) Fixed pattern Solution: 01 strings with length 4, except 0110 Closure a a a a (b) Solution: a(aaaaa)* a 40
38 Content: Outline Basic concepts: pattern, lexeme, and token. Operations on languages, and regular expression Recognition of tokens Finite automata, including NFA and DFA Conversion from regular expression to NFA and DFA Optimization of lexical analyzer 42
39 Regular Expression NFA McNaughtonYamadaThompson (MYT) construction Simple and systematic (recursive up the parse tree for the regular expression) Construction starts from the simplest parts (alphabet symbols). For a complex regular expression, subexpressions are combined to create its NFA. Guarantees the resulting NFA will have exactly one final state, and one start state. 43
40 MYT Construction Basic rules: for subexpressions with no operators For expression start i f For a symbol a in the alphabet start i a f 44
41 MYT Construction Cont d Inductive rules: for constructing larger NFAs from the NFAs of subexpressions (Let N(r 1 ) and N(r 2 ) denote NFAs for regular expressions r 1 and r 2, respectively) For regular expression r 1 r 2 start i N(r 1 ) f N(r 2 ) 45
42 MYT Construction Cont d For regular expression r 1 r 2 start i N(r 1 ) N(r 2 ) f For regular expression r * start i N(r) f 46
43 Example: (a b) * a a: b: a b (a b): a b (a b) * : a b (a b) * a: a b a 47 47
44 Properties of the Constructed NFA 1. N(r) has at most twice as many states as there are operators and operands in r. This bound follows from the fact that each step of the algorithm creates at most two new states. 2. N(r) has one start state and one accepting state. The accepting state has no outgoing transitions, and the start state has no incoming transitions. 3. Each state of N(r) other than the accepting state has either one outgoing transition on a symbol in {} or two outgoing transitions, both on. 48
45 Conversion of an NFA to a DFA Approach: Subset Construction each state of the constructed DFA corresponds to a set / combination of NFA states Details 1 Create transition table Dtran for the DFA 2 Insert closure(s 0 ) to Dstates as initial state 3 Pick a not visited state T in Dstates 4 For each symbol a, Create state closure(move(t, a)), and add it to Dstates and Dtran 5 Repeat step (3) and (4) until all states in Dstates are visited 49
46 The Subset Construction Simulate in parallel all possible moves NFA can make on the input a 50
47 NFA to DFA Example NFA for (a b) * abb A = closure({0}) = {0,1,2,4,7} A into DS as an unmarked state mark A closure(move(a,a)) = closure({3,8}) = {1,2,3,4,6,7,8} = B B into DS closure(move(a,b)) = closure({5}) = {1,2,4,5,6,7} = C C into DS transfunc[a,a] B transfunc[a,b] C mark B closure(move(b,a)) = closure({3,8}) = {1,2,3,4,6,7,8} = B closure(move(b,b)) = closure({5,9}) = {1,2,4,5,6,7,9} = D transfunc[b,a] B transfunc[b,b] D mark C closure(move(c,a)) = closure({3,8}) = {1,2,3,4,6,7,8} = B closure(move(c,b)) = closure({5}) = {1,2,4,5,6,7} = C transfunc[c,a] B transfunc[c,b] C 51
48 NFA to DFA Example NFA for (a b) * abb Transition table for DFA Equivalent DFA 4 52
49 Quiz 1 Suppose we have two tokens: (1) the keyword if, and (2) identifiers, which are strings of letters other than if. Show: 1. The NFA for these tokens, and 2. The DFA for these tokens NFA DFA 55
50 Regular Expression DFA First, augment the given regular expression by concatenating a special symbol # r r# augmented regular expression Second, create a syntax tree for the augmented regular expression. All leaves are alphabet symbols (plus # and the empty string) All inner nodes are operators Third, number each alphabet symbol (plus #) (position numbers) 56
51 Regular Expression DFA Cont d (a b) * a (a b) * a# augmented regular expression a 1 * b 2 a 3 # a b a 3 4 # F Syntax tree of (a b) * a# each symbol is at a leaf each symbol is numbered (positions) inner nodes are operators 57
52 followpos Then we define the function followpos for the positions (positions assigned to leaves). followpos(i)  the set of positions which can follow the position i in the strings generated by the augmented regular expression. Example: ( a b) * a # followpos(1) = {1,2,3} followpos(2) = {1,2,3} followpos(3) = {4} followpos(4) = {} followpos() is just defined for leaves, not defined for inner nodes. 58
53 firstpos, lastpos, nullable To compute followpos, we need three more functions defined for the nodes (not just for leaves) of the syntax tree. firstpos(n)  the set of the positions of the first symbols of strings generated by the subexpression rooted by n. lastpos(n)  the set of the positions of the last symbols of strings generated by the subexpression rooted by n. nullable(n)  true if the empty string is a member of strings generated by the subexpression rooted by n; false otherwise 59
54 Usage of the Functions (a b) * a (a b) * a# augmented regular expression m * n a 3 # 4 nullable(n) = false nullable(m) = true firstpos(n) = {1, 2, 3} a 1 b 2 lastpos(n) = {3} Syntax tree of (a b) * a# 60
55 Computing nullable, firstpos, lastpos n nullable(n) firstpos(n) lastpos(n) leaf labeled true leaf labeled with position i false {i} {i} nullable(c 1 ) or c 1 c 2 nullable(c 2 ) nullable(c 1 ) c 1 c 2 and nullable(c 2 ) firstpos(c 1 ) firstpos(c 2 ) if (nullable(c 1 )) firstpos(c 1 )firstpos(c 2 ) else firstpos(c 1 ) lastpos(c 1 ) lastpos(c 2 ) if (nullable(c 2 )) lastpos(c 1 )lastpos(c 2 ) else lastpos(c 2 ) * true firstpos(c 1 ) lastpos(c 1 ) c 1 Straightforward recursion on the height of the tree 61
56 Thinking Extend the above table to include two more operations (a)? (b) + n nullable(n) firstpos(n) lastpos(n)? c 1 + TRUE firstpos(c 1 ) lastpos(c 1 ) c 1 Nullable(c 1 ) firstpos(c 1 ) lastpos(c 1 ) 62
57 How to evaluate followpos Tworules define the function followpos: 1. If n is concatenationnode with left child c 1 and right child c 2, and i is a position in lastpos(c 1 ), then all positions in firstpos(c 2 ) are in followpos(i). 2. If n is a starnode, and i is a position in lastpos(n), then all positions in firstpos(n) are in followpos(i). If firstpos and lastpos have been computed for each node, followpos of each position can be computed by making one depthfirst traversal of the syntax tree. 63
58 Example  ( a b) * a # {1,2,3} {4} {1,2,3} {3} {4}# 4 {1,2}* {1,2}{3} a{3} 3 {1,2} {1,2} {1} a {1} {2} b {2} 1 2 {4} red firstpos blue lastpos Then we can calculate followpos followpos(1) = {1,2,3} followpos(2) = {1,2,3} followpos(3) = {4} followpos(4) = {} After we calculate follow positions, we are ready to create DFA for the regular expression. 64
59 Algorithm (RE DFA) 1. Create the syntax tree of (r) # 2. Calculate nullable, firstpos, lastpos, followpos 3. Put firstpos(root) into the states of DFA as an unmarked state. 4. while (there is an unmarked state S in the states of DFA) do mark S for each input symbol a do let s 1,...,s n are positions in S and symbols in those positions are a S followpos(s 1 )... followpos(s n ) Dtran[S,a] S if (S is not in the states of DFA) put S into the states of DFA as an unmarked state. the start state of DFA is firstpos(root) the accepting states of DFA are all states containing the position of # 65
60 Example  ( a b) * a # followpos(1)={1,2,3} followpos(3)={4} followpos(2)={1,2,3} followpos(4)={} S 1 =firstpos(root)={1,2,3} mark S 1 a: followpos(1) followpos(3)={1,2,3,4}=s 2 Dtran[S 1,a]=S 2 b: followpos(2)={1,2,3}=s 1 Dtran[S 1,b]=S 1 mark S 2 a: followpos(1) followpos(3)={1,2,3,4}=s 2 Dtran[S 2,a]=S 2 b: followpos(2)={1,2,3}=s 1 Dtran[S 2,b]=S 1 start state: S 1 accepting states: {S 2 } b S 1 a S 2 a 66 b
61 Example  ( a ) b c * # followpos(1)={2} Let s continue followpos(2)={3,4} followpos(3)={3,4} followpos(4)={} S 1 =firstpos(root)={1,2} mark S 1 a: followpos(1)={2}=s 2 Dtran[S 1,a]=S 2 b: followpos(2)={3,4}=s 3 Dtran[S 1,b]=S 3 mark S 2 b: followpos(2)={3,4}=s 3 Dtran[S 2,b]=S 3 mark S 3 c: followpos(3)={3,4}=s 3 Dtran[S 3,c]=S 3 S 1 a b S 2 b S 3 c start state: S 1 accepting states: {S 3 } 67
62 Minimizing Number of DFA States For any regular language, there is always a unique minimum state DFA, which can be constructed from any DFA of the language. Algorithm: Partition the set of states into two groups: G 1 : set of accepting states G 2 : set of nonaccepting states For each new group G partition G into subgroups such that states s 1 and s 2 are in the same group iff for all input symbols a, states s 1 and s 2 have transitions to states in the same group. Start state of the minimized DFA is the group containing the start state of the original DFA. Accepting states of the minimized DFA are the groups containing the accepting states of the original DFA. 68
63 Minimizing DFA Example (1) 1 a b a 2 b 3 a G 1 = {2} G 2 = {1,3} G 2 cannot be partitioned because Dtran[1,a]=2 Dtran[3,a]=2 Dtran[1,b]=3 Dtran[3,b]=3 b So, the minimized DFA (with minimum states) is b a 1 a b 2 69
64 Minimizing DFA Example (2) a a 2 a 1 b 4 b a 3 b b a Minimized DFA 1 b Groups: {1,2,3} {4} {1,2} {3} no more partitioning b 2 a b a b 1>2 1>3 2>2 2>3 3>4 3>3 70 a 3 70
65 Architecture of A Lexical Analyzer 71 71
66 An NFA for Lex program Create an NFA for each regular expression Combine all the NFAs into one Introduce a new start state Connect it with ε transitions to the start states of the NFAs 72
67 Pattern Matching with NFA 1 The lexical analyzer reads in input and calculates the set of states it is in at each symbol. 2 Eventually, it reach a point with no next state. 3 It looks backwards in the sequence of sets of states, until it finds a set including one or more accepting states. 4 It picks the one associated with the earliest pattern in the list from the Lex program. 5 It performs the associated action of the pattern. 73
68 Pattern Matching with NFA  Example Input: aaba 74 Report pattern: a*b +
69 Pattern Matching with DFA 1 Convert the NFA for all the patterns into an equivalent DFA. For each DFA state with more than one accepting NFA states, choose the pattern, who is defined earliest, the output of the DFA state. 2 Simulate the DFA until there is no next state. 3 Trace back to the nearest accepting DFA state, and perform the associated action. Input: abba Report pattern abb 75
70 Summary How lexical analyzers work Convert REs to NFA Convert NFA to DFA Minimize DFA Use the minimized DFA to recognize tokens in the input Use priorities, longest matching rule 76
71 Homework Check the web page!!! 77
Module 6 Lexical Phase  RE to DFA
Module 6 Lexical Phase  RE to DFA The objective of this module is to construct a minimized DFA from a regular expression. A NFA is typically easier to construct but string matching with a NFA is slower.
More information[Lexical Analysis] Bikash Balami
1 [Lexical Analysis] Compiler Design and Construction (CSc 352) Compiled By Central Department of Computer Science and Information Technology (CDCSIT) Tribhuvan University, Kirtipur Kathmandu, Nepal 2
More informationZhizheng Zhang. Southeast University
Zhizheng Zhang Southeast University 2016/10/5 Lexical Analysis 1 1. The Role of Lexical Analyzer 2016/10/5 Lexical Analysis 2 2016/10/5 Lexical Analysis 3 Example. position = initial + rate * 60 2016/10/5
More informationPrinciples of Compiler Design Presented by, R.Venkadeshan,M.TechIT, Lecturer /CSE Dept, Chettinad College of Engineering &Technology
Principles of Compiler Design Presented by, R.Venkadeshan,M.TechIT, Lecturer /CSE Dept, Chettinad College of Engineering &Technology 6/30/2010 Principles of Compiler Design R.Venkadeshan 1 Preliminaries
More informationDixita Kagathara Page 1
2014 Sem  VII Lexical Analysis 1) Role of lexical analysis and its issues. The lexical analyzer is the first phase of compiler. Its main task is to read the input characters and produce as output a sequence
More informationUNIT II LEXICAL ANALYSIS
UNIT II LEXICAL ANALYSIS 2 Marks 1. What are the issues in lexical analysis? Simpler design Compiler efficiency is improved Compiler portability is enhanced. 2. Define patterns/lexeme/tokens? This set
More informationCOMP421 Compiler Design. Presented by Dr Ioanna Dionysiou
COMP421 Compiler Design Presented by Dr Ioanna Dionysiou Administrative! [ALSU03] Chapter 3  Lexical Analysis Sections 3.13.4, 3.63.7! Reading for next time [ALSU03] Chapter 3 Copyright (c) 2010 Ioanna
More informationLexical Analysis/Scanning
Compiler Design 1 Lexical Analysis/Scanning Compiler Design 2 Input and Output The input is a stream of characters (ASCII codes) of the source program. The output is a stream of tokens or symbols corresponding
More informationFinite automata. We have looked at using Lex to build a scanner on the basis of regular expressions.
Finite automata We have looked at using Lex to build a scanner on the basis of regular expressions. Now we begin to consider the results from automata theory that make Lex possible. Recall: An alphabet
More informationLexical Analysis. Implementation: Finite Automata
Lexical Analysis Implementation: Finite Automata Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Nondeterministic Finite Automata (NFAs)
More informationPRINCIPLES OF COMPILER DESIGN UNIT II LEXICAL ANALYSIS 2.1 Lexical Analysis  The Role of the Lexical Analyzer
PRINCIPLES OF COMPILER DESIGN UNIT II LEXICAL ANALYSIS 2.1 Lexical Analysis  The Role of the Lexical Analyzer As the first phase of a compiler, the main task of the lexical analyzer is to read the input
More informationFormal Languages and Compilers Lecture VI: Lexical Analysis
Formal Languages and Compilers Lecture VI: Lexical Analysis Free University of BozenBolzano Faculty of Computer Science POS Building, Room: 2.03 artale@inf.unibz.it http://www.inf.unibz.it/ artale/ Formal
More informationLexical Analysis. Dragon Book Chapter 3 Formal Languages Regular Expressions Finite Automata Theory Lexical Analysis using Automata
Lexical Analysis Dragon Book Chapter 3 Formal Languages Regular Expressions Finite Automata Theory Lexical Analysis using Automata Phase Ordering of FrontEnds Lexical analysis (lexer) Break input string
More informationLexical Analysis. Prof. James L. Frankel Harvard University
Lexical Analysis Prof. James L. Frankel Harvard University Version of 5:37 PM 30Jan2018 Copyright 2018, 2016, 2015 James L. Frankel. All rights reserved. Regular Expression Notation We will develop a
More informationFront End: Lexical Analysis. The Structure of a Compiler
Front End: Lexical Analysis The Structure of a Compiler Constructing a Lexical Analyser By hand: Identify lexemes in input and return tokens Automatically: LexicalAnalyser generator We will learn about
More informationLexical Analysis. Chapter 2
Lexical Analysis Chapter 2 1 Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexers Regular expressions Examples
More informationImplementation of Lexical Analysis
Implementation of Lexical Analysis Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Nondeterministic Finite Automata (NFAs) Implementation
More informationLexical Analysis. Lecture 24
Lexical Analysis Lecture 24 Notes by G. Necula, with additions by P. Hilfinger Prof. Hilfinger CS 164 Lecture 2 1 Administrivia Moving to 60 Evans on Wednesday HW1 available Pyth manual available on line.
More informationImplementation of Lexical Analysis
Implementation of Lexical Analysis Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Nondeterministic Finite Automata (NFAs) Implementation
More informationCSc 453 Lexical Analysis (Scanning)
CSc 453 Lexical Analysis (Scanning) Saumya Debray The University of Arizona Tucson Overview source program lexical analyzer (scanner) tokens syntax analyzer (parser) symbol table manager Main task: to
More informationConcepts Introduced in Chapter 3. Lexical Analysis. Lexical Analysis Terms. Attributes for Tokens
Concepts Introduced in Chapter 3 Lexical Analysis Regular Expressions (REs) Nondeterministic Finite Automata (NFA) Converting an RE to an NFA Deterministic Finite Automatic (DFA) Lexical Analysis Why separate
More informationImplementation of Lexical Analysis. Lecture 4
Implementation of Lexical Analysis Lecture 4 1 Tips on Building Large Systems KISS (Keep It Simple, Stupid!) Don t optimize prematurely Design systems that can be tested It is easier to modify a working
More informationImplementation of Lexical Analysis
Implementation of Lexical Analysis Lecture 4 (Modified by Professor Vijay Ganesh) Tips on Building Large Systems KISS (Keep It Simple, Stupid!) Don t optimize prematurely Design systems that can be tested
More informationLexical Analysis. Sukree Sinthupinyo July Chulalongkorn University
Sukree Sinthupinyo 1 1 Department of Computer Engineering Chulalongkorn University 14 July 2012 Outline Introduction 1 Introduction 2 3 4 Transition Diagrams Learning Objectives Understand definition of
More informationCS6660 COMPILER DESIGN L T P C
COMPILER DESIGN CS6660 COMPILER DESIGN L T P C 3 0 0 3 UNIT I INTRODUCTION TO COMPILERS 5 TranslatorsCompilation and InterpretationLanguage processors The Phases of CompilerErrors Encountered in Different
More informationLexical Analyzer Scanner
Lexical Analyzer Scanner ASU Textbook Chapter 3.1, 3.3, 3.4, 3.6, 3.7, 3.5 Tsansheng Hsu tshsu@iis.sinica.edu.tw http://www.iis.sinica.edu.tw/~tshsu 1 Main tasks Read the input characters and produce
More informationFormal Languages and Compilers Lecture IV: Regular Languages and Finite. Finite Automata
Formal Languages and Compilers Lecture IV: Regular Languages and Finite Automata Free University of BozenBolzano Faculty of Computer Science POS Building, Room: 2.03 artale@inf.unibz.it http://www.inf.unibz.it/
More informationCS415 Compilers. Lexical Analysis
CS415 Compilers Lexical Analysis These slides are based on slides copyrighted by Keith Cooper, Ken Kennedy & Linda Torczon at Rice University Lecture 7 1 Announcements First project and second homework
More informationIntroduction to Lexical Analysis
Introduction to Lexical Analysis Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexical analyzers (lexers) Regular
More informationFigure 2.1: Role of Lexical Analyzer
Chapter 2 Lexical Analysis Lexical analysis or scanning is the process which reads the stream of characters making up the source program from lefttoright and groups them into tokens. The lexical analyzer
More informationLexical Analysis. Lecture 34
Lexical Analysis Lecture 34 Notes by G. Necula, with additions by P. Hilfinger Prof. Hilfinger CS 164 Lecture 34 1 Administrivia I suggest you start looking at Python (see link on class home page). Please
More information2010: Compilers REVIEW: REGULAR EXPRESSIONS HOW TO USE REGULAR EXPRESSIONS
2010: Compilers Lexical Analysis: Finite State Automata Dr. Licia Capra UCL/CS REVIEW: REGULAR EXPRESSIONS a Character in A Empty string R S Alternation (either R or S) RS Concatenation (R followed by
More informationLexical Analyzer Scanner
Lexical Analyzer Scanner ASU Textbook Chapter 3.1, 3.3, 3.4, 3.6, 3.7, 3.5 Tsansheng Hsu tshsu@iis.sinica.edu.tw http://www.iis.sinica.edu.tw/~tshsu 1 Main tasks Read the input characters and produce
More informationCSE302: Compiler Design
CSE302: Compiler Design Instructor: Dr. Liang Cheng Department of Computer Science and Engineering P.C. Rossin College of Engineering & Applied Science Lehigh University February 13, 2007 Outline Recap
More informationECS 120 Lesson 7 Regular Expressions, Pt. 1
ECS 120 Lesson 7 Regular Expressions, Pt. 1 Oliver Kreylos Friday, April 13th, 2001 1 Outline Thus far, we have been discussing one way to specify a (regular) language: Giving a machine that reads a word
More informationCompiler course. Chapter 3 Lexical Analysis
Compiler course Chapter 3 Lexical Analysis 1 A. A. Pourhaji Kazem, Spring 2009 Outline Role of lexical analyzer Specification of tokens Recognition of tokens Lexical analyzer generator Finite automata
More informationChapter 3 Lexical Analysis
Chapter 3 Lexical Analysis Outline Role of lexical analyzer Specification of tokens Recognition of tokens Lexical analyzer generator Finite automata Design of lexical analyzer generator The role of lexical
More informationLexical Analysis. Chapter 1, Section Chapter 3, Section 3.1, 3.3, 3.4, 3.5 JFlex Manual
Lexical Analysis Chapter 1, Section 1.2.1 Chapter 3, Section 3.1, 3.3, 3.4, 3.5 JFlex Manual Inside the Compiler: Front End Lexical analyzer (aka scanner) Converts ASCII or Unicode to a stream of tokens
More informationCOMPILER DESIGN UNIT I LEXICAL ANALYSIS. Translator: It is a program that translates one language to another Language.
UNIT I LEXICAL ANALYSIS Translator: It is a program that translates one language to another Language. Source Code Translator Target Code 1. INTRODUCTION TO LANGUAGE PROCESSING The Language Processing System
More informationUNIT 2 LEXICAL ANALYSIS
OVER VIEW OF LEXICAL ANALYSIS UNIT 2 LEXICAL ANALYSIS o To identify the tokens we need some method of describing the possible tokens that can appear in the input stream. For this purpose we introduce
More informationRegular Expressions. Agenda for Today. Grammar for a Tiny Language. Programming Language Specifications
Agenda for Today Regular Expressions CSE 413, Autumn 2005 Programming Languages Basic concepts of formal grammars Regular expressions Lexical specification of programming languages Using finite automata
More informationCSEP 501 Compilers. Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter /8/ Hal Perkins & UW CSE B1
CSEP 501 Compilers Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter 2008 1/8/2008 200208 Hal Perkins & UW CSE B1 Agenda Basic concepts of formal grammars (review) Regular expressions
More informationCS Lecture 2. The Front End. Lecture 2 Lexical Analysis
CS 1622 Lecture 2 Lexical Analysis CS 1622 Lecture 2 1 Lecture 2 Review of last lecture and finish up overview The first compiler phase: lexical analysis Reading: Chapter 2 in text (by 1/18) CS 1622 Lecture
More informationDavid Griol Barres Computer Science Department Carlos III University of Madrid Leganés (Spain)
David Griol Barres dgriol@inf.uc3m.es Computer Science Department Carlos III University of Madrid Leganés (Spain) OUTLINE Introduction: Definitions The role of the Lexical Analyzer Scanner Implementation
More informationConcepts. Lexical scanning Regular expressions DFAs and FSAs Lex. Lexical analysis in perspective
Concepts Lexical scanning Regular expressions DFAs and FSAs Lex CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 1 CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 2 Lexical analysis
More informationLexical Analysis. Lexical analysis is the first phase of compilation: The file is converted from ASCII to tokens. It must be fast!
Lexical Analysis Lexical analysis is the first phase of compilation: The file is converted from ASCII to tokens. It must be fast! Compiler Passes Analysis of input program (frontend) character stream
More informationChapter 4. Lexical analysis. Concepts. Lexical scanning Regular expressions DFAs and FSAs Lex. Lexical analysis in perspective
Chapter 4 Lexical analysis Lexical scanning Regular expressions DFAs and FSAs Lex Concepts CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 1 CMSC 331, Some material 1998 by Addison Wesley
More informationChapter Seven: Regular Expressions
Chapter Seven: Regular Expressions Regular Expressions We have seen that DFAs and NFAs have equal definitional power. It turns out that regular expressions also have exactly that same definitional power:
More informationLast lecture CMSC330. This lecture. Finite Automata: States. Finite Automata. Implementing Regular Expressions. Languages. Regular expressions
Last lecture CMSC330 Finite Automata Languages Sets of strings Operations on languages Regular expressions Constants Operators Precedence 1 2 Finite automata States Transitions Examples Types This lecture
More informationComputer Science Department Carlos III University of Madrid Leganés (Spain) David Griol Barres
Computer Science Department Carlos III University of Madrid Leganés (Spain) David Griol Barres dgriol@inf.uc3m.es Introduction: Definitions Lexical analysis or scanning: To read from lefttoright a source
More informationLexical Analysis. Introduction
Lexical Analysis Introduction Copyright 2015, Pedro C. Diniz, all rights reserved. Students enrolled in the Compilers class at the University of Southern California have explicit permission to make copies
More informationLexical Analysis  1. A. Overview A.a) Role of Lexical Analyzer
CMPSC 470 Lecture 02 Topics: Regular Expression Transition Diagram Lexical Analyzer Implementation A. Overview A.a) Role of Lexical Analyzer Lexical Analysis  1 Lexical analyzer does: read input character
More informationFormal Languages and Grammars. Chapter 2: Sections 2.1 and 2.2
Formal Languages and Grammars Chapter 2: Sections 2.1 and 2.2 Formal Languages Basis for the design and implementation of programming languages Alphabet: finite set Σ of symbols String: finite sequence
More informationAdministrivia. Lexical Analysis. Lecture 24. Outline. The Structure of a Compiler. Informal sketch of lexical analysis. Issues in lexical analysis
dministrivia Lexical nalysis Lecture 24 Notes by G. Necula, with additions by P. Hilfinger Moving to 6 Evans on Wednesday HW available Pyth manual available on line. Please log into your account and electronically
More informationImplementation of Lexical Analysis
Outline Implementation of Lexical nalysis Specifying lexical structure using regular expressions Finite automata Deterministic Finite utomata (DFs) Nondeterministic Finite utomata (NFs) Implementation
More informationCS 403 Compiler Construction Lecture 3 Lexical Analysis [Based on Chapter 1, 2, 3 of Aho2]
CS 403 Compiler Construction Lecture 3 Lexical Analysis [Based on Chapter 1, 2, 3 of Aho2] 1 What is Lexical Analysis? First step of a compiler. Reads/scans/identify the characters in the program and groups
More informationCS 314 Principles of Programming Languages
CS 314 Principles of Programming Languages Lecture 2: Syntax Analysis Zheng (Eddy) Zhang Rutgers University January 22, 2018 Announcement First recitation starts this Wednesday Homework 1 will be release
More informationCompiler Construction
Compiler Construction Thomas Noll Software Modeling and Verification Group RWTH Aachen University https://moves.rwthaachen.de/teaching/ss16/cc/ Conceptual Structure of a Compiler Source code x1 := y2
More informationCS412/413. Introduction to Compilers Tim Teitelbaum. Lecture 2: Lexical Analysis 23 Jan 08
CS412/413 Introduction to Compilers Tim Teitelbaum Lecture 2: Lexical Analysis 23 Jan 08 Outline Review compiler structure What is lexical analysis? Writing a lexer Specifying tokens: regular expressions
More informationCMSC 330: Organization of Programming Languages
CMSC 330: Organization of Programming Languages Context Free Grammars and Parsing 1 Recall: Architecture of Compilers, Interpreters Source Parser Static Analyzer Intermediate Representation Front End Back
More informationWeek 2: Syntax Specification, Grammars
CS320 Principles of Programming Languages Week 2: Syntax Specification, Grammars Jingke Li Portland State University Fall 2017 PSU CS320 Fall 17 Week 2: Syntax Specification, Grammars 1/ 62 Words and Sentences
More informationInterpreter. Scanner. Parser. Tree Walker. read. request token. send token. send AST I/O. Console
Scanning 1 read Interpreter Scanner request token Parser send token Console I/O send AST Tree Walker 2 Scanner This process is known as: Scanning, lexing (lexical analysis), and tokenizing This is the
More informationCS 314 Principles of Programming Languages. Lecture 3
CS 314 Principles of Programming Languages Lecture 3 Zheng Zhang Department of Computer Science Rutgers University Wednesday 14 th September, 2016 Zheng Zhang 1 CS@Rutgers University Class Information
More informationLexical Analysis  2
Lexical Analysis  2 More regular expressions Finite Automata NFAs and DFAs Scanners JLex  a scanner generator 1 Regular Expressions in JLex Symbol  Meaning. Matches a single character (not newline)
More informationDr. D.M. Akbar Hussain
1 2 Compiler Construction F6S Lecture  2 1 3 4 Compiler Construction F6S Lecture  2 2 5 #include.. #include main() { char in; in = getch ( ); if ( isalpha (in) ) in = getch ( ); else error (); while
More informationOutline. 1 Scanning Tokens. 2 Regular Expresssions. 3 Finite State Automata
Outline 1 2 Regular Expresssions Lexical Analysis 3 Finite State Automata 4 Nondeterministic (NFA) Versus Deterministic Finite State Automata (DFA) 5 Regular Expresssions to NFA 6 NFA to DFA 7 8 JavaCC:
More informationStructure of Programming Languages Lecture 3
Structure of Programming Languages Lecture 3 CSCI 6636 4536 Spring 2017 CSCI 6636 4536 Lecture 3... 1/25 Spring 2017 1 / 25 Outline 1 Finite Languages Deterministic Finite State Machines Lexical Analysis
More informationLexical Analysis. Lecture 3. January 10, 2018
Lexical Analysis Lecture 3 January 10, 2018 Announcements PA1c due tonight at 11:50pm! Don t forget about PA1, the Cool implementation! Use Monday s lecture, the video guides and Cool examples if you re
More informationCSE 413 Programming Languages & Implementation. Hal Perkins Autumn 2012 Grammars, Scanners & Regular Expressions
CSE 413 Programming Languages & Implementation Hal Perkins Autumn 2012 Grammars, Scanners & Regular Expressions 1 Agenda Overview of language recognizers Basic concepts of formal grammars Scanner Theory
More informationLexical Analysis 1 / 52
Lexical Analysis 1 / 52 Outline 1 Scanning Tokens 2 Regular Expresssions 3 Finite State Automata 4 Nondeterministic (NFA) Versus Deterministic Finite State Automata (DFA) 5 Regular Expresssions to NFA
More informationImplementation of Lexical Analysis
Written ssignments W assigned today Implementation of Lexical nalysis Lecture 4 Due in one week :59pm Electronic handin Prof. iken CS 43 Lecture 4 Prof. iken CS 43 Lecture 4 2 Tips on uilding Large Systems
More informationCS143 Handout 20 Summer 2011 July 15 th, 2011 CS143 Practice Midterm and Solution
CS143 Handout 20 Summer 2011 July 15 th, 2011 CS143 Practice Midterm and Solution Exam Facts Format Wednesday, July 20 th from 11:00 a.m. 1:00 p.m. in Gates B01 The exam is designed to take roughly 90
More information1. INTRODUCTION TO LANGUAGE PROCESSING The Language Processing System can be represented as shown figure below.
UNIT I Translator: It is a program that translates one language to another Language. Examples of translator are compiler, assembler, interpreter, linker, loader and preprocessor. Source Code Translator
More informationAnnouncements! P1 part 1 due next Tuesday P1 part 2 due next Friday
Announcements! P1 part 1 due next Tuesday P1 part 2 due next Friday 1 Finitestate machines CS 536 Last time! A compiler is a recognizer of language S (Source) a translator from S to T (Target) a program
More informationTHE COMPILATION PROCESS EXAMPLE OF TOKENS AND ATTRIBUTES
THE COMPILATION PROCESS Character stream CS 403: Scanning and Parsing Stefan D. Bruda Fall 207 Token stream Parse tree Abstract syntax tree Modified intermediate form Target language Modified target language
More informationCOP4020 Programming Languages. Syntax Prof. Robert van Engelen
COP4020 Programming Languages Syntax Prof. Robert van Engelen Overview n Tokens and regular expressions n Syntax and contextfree grammars n Grammar derivations n More about parse trees n Topdown and
More informationCS 403: Scanning and Parsing
CS 403: Scanning and Parsing Stefan D. Bruda Fall 2017 THE COMPILATION PROCESS Character stream Scanner (lexical analysis) Token stream Parser (syntax analysis) Parse tree Semantic analysis Abstract syntax
More informationCSE 413 Programming Languages & Implementation. Hal Perkins Winter 2019 Grammars, Scanners & Regular Expressions
CSE 413 Programming Languages & Implementation Hal Perkins Winter 2019 Grammars, Scanners & Regular Expressions 1 Agenda Overview of language recognizers Basic concepts of formal grammars Scanner Theory
More informationCS321 Languages and Compiler Design I. Winter 2012 Lecture 4
CS321 Languages and Compiler Design I Winter 2012 Lecture 4 1 LEXICAL ANALYSIS Convert source file characters into token stream. Remove contentfree characters (comments, whitespace,...) Detect lexical
More informationImplementation of Lexical Analysis
Written ssignments W assigned today Implementation of Lexical nalysis Lecture 4 Due in one week y 5pm Turn in In class In box outside 4 Gates Electronically Prof. iken CS 43 Lecture 4 Prof. iken CS 43
More informationCSE 105 THEORY OF COMPUTATION
CSE 105 THEORY OF COMPUTATION Spring 2017 http://cseweb.ucsd.edu/classes/sp17/cse105ab/ Today's learning goals Sipser Ch 1.2, 1.3 Design NFA recognizing a given language Convert an NFA (with or without
More informationRegular Languages and Regular Expressions
Regular Languages and Regular Expressions According to our definition, a language is regular if there exists a finite state automaton that accepts it. Therefore every regular language can be described
More informationArchitecture of Compilers, Interpreters. CMSC 330: Organization of Programming Languages. Front End Scanner and Parser. Implementing the Front End
Architecture of Compilers, Interpreters : Organization of Programming Languages ource Analyzer Optimizer Code Generator Context Free Grammars Intermediate Representation Front End Back End Compiler / Interpreter
More informationWhere We Are. CMSC 330: Organization of Programming Languages. This Lecture. Programming Languages. Motivation for Grammars
CMSC 330: Organization of Programming Languages Context Free Grammars Where We Are Programming languages Ruby OCaml Implementing programming languages Scanner Uses regular expressions Finite automata Parser
More informationG Compiler Construction Lecture 4: Lexical Analysis. Mohamed Zahran (aka Z)
G22.2130001 Compiler Construction Lecture 4: Lexical Analysis Mohamed Zahran (aka Z) mzahran@cs.nyu.edu Role of the Lexical Analyzer Remove comments and white spaces (aka scanning) Macros expansion Read
More informationCMSC 330: Organization of Programming Languages. Architecture of Compilers, Interpreters
: Organization of Programming Languages Context Free Grammars 1 Architecture of Compilers, Interpreters Source Scanner Parser Static Analyzer Intermediate Representation Front End Back End Compiler / Interpreter
More informationLexical Analysis (ASU Ch 3, Fig 3.1)
Lexical Analysis (ASU Ch 3, Fig 3.1) Implementation by hand automatically ((F)Lex) Lex generates a finite automaton recogniser uses regular expressions Tasks remove white space (ws) display source program
More informationRegular Languages. MACM 300 Formal Languages and Automata. Formal Languages: Recap. Regular Languages
Regular Languages MACM 3 Formal Languages and Automata Anoop Sarkar http://www.cs.sfu.ca/~anoop The set of regular languages: each element is a regular language Each regular language is an example of a
More informationA simple syntaxdirected
Syntaxdirected is a grammaroriented compiling technique Programming languages: Syntax: what its programs look like? Semantic: what its programs mean? 1 A simple syntaxdirected Lexical Syntax Character
More informationCOP4020 Programming Languages. Syntax Prof. Robert van Engelen
COP4020 Programming Languages Syntax Prof. Robert van Engelen Overview Tokens and regular expressions Syntax and contextfree grammars Grammar derivations More about parse trees Topdown and bottomup
More information2. Lexical Analysis! Prof. O. Nierstrasz!
2. Lexical Analysis! Prof. O. Nierstrasz! Thanks to Jens Palsberg and Tony Hosking for their kind permission to reuse and adapt the CS132 and CS502 lecture notes.! http://www.cs.ucla.edu/~palsberg/! http://www.cs.purdue.edu/homes/hosking/!
More informationCompiler Construction
Compiler Construction Lecture 2: Lexical Analysis I (Introduction) Thomas Noll Lehrstuhl für Informatik 2 (Software Modeling and Verification) noll@cs.rwthaachen.de http://moves.rwthaachen.de/teaching/ss14/cc14/
More information1. Lexical Analysis Phase
1. Lexical Analysis Phase The purpose of the lexical analyzer is to read the source program, one character at time, and to translate it into a sequence of primitive units called tokens. Keywords, identifiers,
More informationLexical Analysis. Finite Automata
#1 Lexical Analysis Finite Automata Cool Demo? (Part 1 of 2) #2 Cunning Plan Informal Sketch of Lexical Analysis LA identifies tokens from input string lexer : (char list) (token list) Issues in Lexical
More informationLanguages, Automata, Regular Expressions & Scanners. Winter /8/ Hal Perkins & UW CSE B1
CSE 401 Compilers Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter 2010 1/8/2010 200210 Hal Perkins & UW CSE B1 Agenda Quick review of basic concepts of formal grammars Regular
More informationRoll No. :... Invigilator's Signature :. CS/B.Tech(CSE)/SEM7/CS701/ LANGUAGE PROCESSOR. Time Allotted : 3 Hours Full Marks : 70
Name : Roll No. :... Invigilator's Signature :. CS/B.Tech(CSE)/SEM7/CS701/201112 2011 LANGUAGE PROCESSOR Time Allotted : 3 Hours Full Marks : 70 The figures in the margin indicate full marks. Candidates
More informationLexical Analysis. Note by Baris Aktemur: Our slides are adapted from Cooper and Torczon s slides that they prepared for COMP 412 at Rice.
Lexical Analysis Note by Baris Aktemur: Our slides are adapted from Cooper and Torczon s slides that they prepared for COMP 412 at Rice. Copyright 2010, Keith D. Cooper & Linda Torczon, all rights reserved.
More informationQuestion Bank. 10CS63:Compiler Design
Question Bank 10CS63:Compiler Design 1.Determine whether the following regular expressions define the same language? (ab)* and a*b* 2.List the properties of an operator grammar 3. Is macro processing a
More informationCS 432 Fall Mike Lam, Professor. Finite Automata Conversions and Lexing
CS 432 Fall 2017 Mike Lam, Professor Finite Automata Conversions and Lexing Finite Automata Key result: all of the following have the same expressive power (i.e., they all describe regular languages):
More informationCSCIGA Compiler Construction Lecture 4: Lexical Analysis I. Hubertus Franke
CSCIGA.2130001 Compiler Construction Lecture 4: Lexical Analysis I Hubertus Franke frankeh@cs.nyu.edu Role of the Lexical Analyzer Remove comments and white spaces (aka scanning) Macros expansion Read
More information