Lexical Analysis/Scanning

Size: px
Start display at page:

Download "Lexical Analysis/Scanning"

Transcription

1 Compiler Design 1 Lexical Analysis/Scanning

2 Compiler Design 2 Input and Output The input is a stream of characters (ASCII codes) of the source program. The output is a stream of tokens or symbols corresponding to different syntactic categories. The output also contains attributes of tokens. Examples of tokens are different keywords, identifiers, constants, operators, delimiters etc.

3 Compiler Design 3 Note The scanner removes the comments, white spaces, evaluates the constants, keeps track of the line numbers etc. This stage performs the main I/O and reduces the complexity of the syntax analyzer. The syntax analyzer invokes the scanner whenever it requires a token.

4 Compiler Design 4 Token A token is an identifier (name/code) corresponding to a syntactic category of the language grammar. In other word it is the terminal alphabet of the grammar. Often we use an integer code for this.

5 Compiler Design 5 Pattern A pattern is a description (formal or informal) of the set of objects corresponding to a terminal (token) symbol of the grammar. Examples are the set of identifier in C language, set of integer constants etc.

6 Compiler Design 6 Lexeme and Attribute A lexeme is an actual string of characters that matches with a pattern and generates a token. An attribute of a token is a value that the scanner extracts from the corresponding lexeme and supplies to the syntax analyzer.

7 Compiler Design 7 Specification of Token The set of strings corresponding to a token (terminals) of a programming language is often a regular set and is specified by a regular expression.

8 Compiler Design 8 Scanner from the Specification The collection of tokens of a programming language can be specified by a set of regular expressions. A scanner or lexical analyzer for the language uses a DFA (recognizer of regular languages) in its core. Different final states of the DFA identifies different tokens. Synthesis of this DFA from the set of regular expressions can be automated.

9 Compiler Design 9 Regular Expression 1. ε, and all a Σ are regular expressions. 2. If r and s are regular expressions, then so are (r s), (rs), (r ) and (r). Nothing else is a regular expression. We can reduce the use of parenthesis by introducing precedence and associativity rules. Binary operators are left associative and the precedence rule is > concat >.

10 Compiler Design 10 IEEE POSIX Regular Expression An enlarged set of operators (defined) for the regular expressions are introduced in different softwares e.g. awk, grep, lex etc. a. \x is the character itself (a few exceptions are \n, \t, \r etc.).. is any character other than \n. [xyz] is x y z. a Consult the manual pages of lex/flex and Wikipedia for the details of IEEE POSIX standard of regular expressions.

11 Compiler Design 11 IEEE POSIX Regular Expression [abg-pt-y] any character a, b,g,, p, T,, Y. [^G-Q] not any one of G, H,, P, Q. r+ one or more r s. r? one or zero r s. r{2,} two or more r s etc.

12 Compiler Design 12 Language of a Regular Expression The language of a regular expression is defined in a usual way on the inductive structure of the definition. L(ε) = {ε}, L( ) =, L(a) = {a} for all a Σ, L(r s) = L(r) L(s), L(rs) = L(r)L(s), L(r ) = L(r), L(r?) = L(r) {ε} etc.

13 Compiler Design 13 C Identifier The regular expression for the C identifier is [ a-za-z][ a-za-z0-9]* The first character is an underscore or an English alphabet. From the second character on a decimal digit can also be used.

14 Compiler Design 14 Regular Definition We can give name to a regular expression for the convenience of use. A name name of a regular expression can be used regular expressions following the name.

15 Compiler Design 15 Examples of a Regular Definition sign: + - ε digit: [0-9] digits: {digit}* frac: \.{digits} ε frace: \.{digit}{digits} expo: ((E e){sign}{digit}{digit}?) ε num: {sign}(({digit}+ {frac} {expo}) ({frace} {expo}))

16 Compiler Design 16 RE to NFA: Thompson s Construction We can mechanically construct a non-deterministic finite automaton (NFA) with only one initial and only one final state from a given regular expression. The total number of states of the NFA is linear in the number of symbols of the regular expression a a The construction is on the inductive structure of the definition of the regular expression.

17 Compiler Design 17 s φ s ε f s a f a Σ

18 Compiler Design 18 (r s) and (rs) ε s1 N(r) f1 ε s f ε s2 N(s) f2 ε s ε s1 N(s) f1 ε s2 N(r) f2 ε f

19 Compiler Design 19 Kleene Closure: s ε s ε s1 N(s) f1 ε f ε

20 Compiler Design 20 Properties of Thompson s Construction Q 2length(r), where Q is the number of states of the NFA and length(r) is the number of alphabet and operator symbols in r. Only one initial and one final state. No incoming edge to the initial state and no outgoing edge from the final state.

21 Compiler Design 21 Properties of Thompson s Construction At most one incoming and one outgoing transition on a symbol of the alphabet. At most two incoming and two outgoing ǫ transitions.

22 Compiler Design 22 a+(ab) - An Example a 0 1 ε ε 8 9 ε ε ε a ε ε ε b ε

23 Compiler Design 23 Context-Free Grammar of RE The set of regular expression can be specified by a context-free grammar. E ε σ, σ Σ E.E E +E E (E) We have put a. for concatenation to make it an operator grammar and have replaced by + for clarity a. a This ambiguous grammar can be used with proper precedence and associativity rules.

24 Compiler Design 24 Syntax Directed Thompson s Construction Rules of Thompson s construction can be associated with the non-terminals of the production rules of the grammar. This is known as an attribute grammar. We assume the following data structures.

25 Compiler Design 25 Syntax Directed Thompson s Construction Global state counter S initialized to 0, and the state transition table: T[][]. With every occurrence of the non-terminal E we associate two attributes E.ini and E.fin to store the initial and the final states of the NFA, corresponding to the regular expression generated by this occurrence of E.

26 Compiler Design 26 Some of the Rules: Basis E ε: {T[S][ε] = S+1; E.ini = S; S = S+1; E.fin = S; S = S+1;} E a: {T[S][a] = S+1; E.ini = S; S = S+1; E.fin = S; S = S+1;} The second rule depends on the symbol of the alphabet

27 Compiler Design 27 Concatenation Rules E E1.E2: {E.ini = S; S = S+1; E.fin = S+1; S = S+1; T[E.ini][ε]=E1.ini; T[E1.fin][ε]=E2.ini; T[E2.fin][ε]=E.fin;} Similarly other rules can be derived.

28 Compiler Design 28 The Final NFA The states of the final NFA are {0,1,,S 1}. The initial state is in E.ini and the final state is in E.fin. The state transitions are in T[][].

29 Compiler Design 29 a+(a.b) - An Example E.ini=10; E.fin=11; T[10][]=E1.ini; T[10][]=E2.ini; E T[E1.fin][]=11; T[E2.fin][]=11 E1.ini=0 E1.fin=1 T[0][a]=1 E1 + E2 T[E3.fin][]=E3.ini E3.ini=E4.ini a E3.fin=E4.fin E3 E2.ini=8; E2.fin=9;T[8][]=9; T[8][]=E3.ini; T[E3.fin][]=9; * Q={0,1,..., 11} E5 E5.ini=2 E5.fin=3 T[2][a]=3 a ( ) E4. E6 b E4.ini=6 E4.fin=7 T[6][]=E5.ini (2); T[E5.fin(3)][]=E6.ini (4); T[E6.fin(5)][]=7 E6.ini=4 E6.fin=5 T[4][b]=5

30 Compiler Design 30 NFA to DFA Let the constructed ε-nfa be (N,Σ,δ n,n 0,n F ). By taking ε-closure of states and doing the subset construction we can get an equivalent DFA (Q,Σ,δ d,q 0,Q F ).

31 Compiler Design 31 Algorithm Q = L = ε-closure({q 0 }) while(l ) q = removeelm(l) for all σ Σ t = ε-closure(δ n (q,σ)) T[q][σ] = t if t Q Q = Q {t} L = L {t}

32 Compiler Design 32 ε-closure(t) for all q T push(st,q) εt = T while(isempty(st) == false) t = pop(st) for all u δ(t,ε) if u εt εt = εt {u} push(st, u)

33 Compiler Design 33 Note The time complexity of the ε-closure algorithm for each state is O( M ) = O( N + δ ).

34 Compiler Design 34 Final State of the DFA The set of final states of the equivalent DFA is Q F = {q Q : n F q}. It is to be noted that different final states will recognize different tokens. It is also possible that one final state identifies more than one tokens.

35 Compiler Design 35 Time Complexity of Subset Construction The size of Q is O(2 N ) and so the time complexity is also O(2 N ), where N is the set of states of the NFA. But this is one time construction.

36 Compiler Design 36 a+(ab) - NFA to DFA The state transition table of the DFA is Initial Final State State a b A : {0,2,6,7,8,9} {1,3,4,9} B : {1,3,4,9} {2,5,7,9} C : {2,5,7,9} {3,4} D : {3,4} {2,5,7,9}

37 Compiler Design 37 a+(ab) - NFA to DFA A b a D a φ a b a a B b b b C

38 Compiler Design 38 Note It may be of advantage to drop the transitions to for designing a scanner. This makes the DFA incompletely specified. Absence of a transition from a final state may identify a token.

39 Compiler Design 39 DFA State Minimization The constructed DFA may have set of equivalent states a and can be minimized. It is to be noted that the time complexity of a scanner of a DFA with a larger number of states is not different from the scanner of a DFA having a smaller number of states. Their code sizes are different and that may give rise to some difference in their speeds. a Let M = (Q,Σ,δ,s,F) be a DFA. Two states p,q Q are said to be equivalent if there is no x Σ so that δ(p,x) δ(q,x).

40 Compiler Design 40 DFA State Minimization We start with two non-equivalent partitions of Q: F and Q\F. If p,q belongs to the same initial partition P but there is some σ Σ so that δ(p,σ) P 1 and δ(q,σ) P 2, where P 1 and P 2 are two distinct partitions, then p, q cannot remain in the same partition i.e. they are not equivalent.

41 Compiler Design 41 DFA to Scanner Given a regular expression r we can construct the recognizer of L(r). For every token class or syntactic category of a language we have a regular expression. Let {r 1,r 2,,r k } be the total collection of all regular expressions of a language. The regular expression r = r 1 r 2 r k represents objects of all syntactic categories.

42 Compiler Design 42 DFA to Scanner Give the NFAs of r 1,r 2,,r k we construct the NFA for r = r 1 r 2 r k by introducing a new start state and adding ε-transitions from this state to the initial states of the component NFAs. But we keep different final states as they are to identify different token classes.

43 Compiler Design 43 Final Composite NFA sr1 N(r1) fr1 ε s ε sr2 N(r2) fr2 ε srk N(rk) frk

44 Compiler Design 44 DFA to Scanner The DFA corresponding to r can be constructed from the composite NFA. It can be implemented as a C program that will be used as a scanner of the language. But the following points are to be noted.

45 Compiler Design 45 Note A program is not a single word but a stream of words and the notion of acceptance of a scanner should be different from a simple DFA. The following questions are of importance: when does the scanner report an acceptance? what does it do if the word (lexeme) matches with more than one regular expressions?

46 Compiler Design 46 Example Consider the following subset of C language operators: = * *= < << <= <<=

47 Compiler Design 47 State Transition Diagram

48 Compiler Design = * = < other other $ 4 $ 5 < = = other 8 $ 9 other 6 $ 7

49 Compiler Design 49 Note At the final state 1 we know that we have ++. But we cannot decide whether it is pre or post increment operator. Though scanner can take that decision, but it is better to delay it for the parser.

50 Compiler Design 50 Note At the final state 3 we know that we have +. But we do not know whether it is binary or unary. Again that decision is defered. Moreover the last consumed symbol is not part of the lexeme. It is a look-ahead symbol. We mark such a final state with the number of look-ahead symbols to un-read before going back to the start state. Here we have done that by one $.

51 Compiler Design 51 Note There are situations where there may be more than one look-ahead. Fortran: DO 10 I = 1, 10 and DO 10 I = 1.10 The first one is a do-loop and the second one is an assignment DO10I=1.10. PL/I: IF ELSE THEN THEN = ELSE; ELSE ELSE = THEN IF THEN are not reserved as keyword.

52 Compiler Design 52 Maximum Word Length Matching The scanner will go on reading input as long as there is a transition. Let there be no transitions for the current state q on the input σ (the machine is incompletely specified). The state q may or may not be final.

53 Compiler Design 53 q is Final If the final state q corresponds to only one regular expression r i, the scanner returns the corresponding token a. But if it matches with more than one regular expressions then it is necessary to resolve the conflict. This is often done by specifying priority of expressions e.g. keyword over an identifier. a It is necessary to identify the final state with the regular expression r i.

54 Compiler Design 54 q is not Final It is possible that while consuming symbols the scanner has crossed one or more final states. The decision may be to report the last final state. But then it is necessary to keep track of the final states and the position of the input.

55 Compiler Design 55 Components of a Scanner 1. The transition table of the DFA or NFA. 2. Set of actions corresponding to a final states. 3. Other essential functions.

56 Compiler Design 56 Maximum Prefix on NFA 1. Read input and keep track of the sequence of the set of states. Stop when no more transition is possible (maximum prefix). 2. Trace back the last set of states with a final state. 3. Push back the look-ahead symbols in the buffer and emit appropriate token along with attribute value(s).

57 Compiler Design 57 Note It is possible that the last set of states has more than one final states corresponding to different patterns. Take action corresponding to a pattern with highest priority a. a A pattern specified earlier may have higher priority.

58 Compiler Design 58 From DFA to Code Three possible implementations of DFA - table driven, direct coded, hand coded.

59 Compiler Design 59 Table Driven Scanner There is a driver code and a set of tables. The driver code essentially has three parts: Initialization, Main scanner loop, Roll-back loop, Token or error return.

60 Compiler Design 60 Initialization currect state <-- start state lexeme <-- Nil push(stack,$)

61 Compiler Design 61 Main Scanner Loop while currect state not = error state lexeme <-- lexme + (c = getchar()) if current state is an accept state clear(stack) push(stack, current state) sym <-- translate[c] next state <-- delta(current state, sym)

62 Compiler Design 62 Roll Back Loop while not a final state or stack is not empty state <-- pop(stack) unget() last symbol of lexeme

63 Compiler Design 63 Token or Error if final state return token[state] else Error

64 Compiler Design 64 Tables translate[] converts a character to a DFA symbol (reduces the size of the alphabet). delta[] is the state transition table. token[] have token values corresponding to final states.

65 Compiler Design 65 Note At times roll-back may be costly - consider the language ab (ab) c and the input ababababab$. There will be roll-back of = 20 characters.

66 Compiler Design 66 Direct Coded Scanner Each state is implemented as a fragment of code. It eliminates memory reference for transition table access.

67 Compiler Design 67 Code Corresponding to a State Code is labelled by the state name. Read a character and append it to lexeme. Update the roll-back stack. Go to next appropriate state - a valid transition, roll-back and token return state etc.

68 Compiler Design 68 Reading Characters: Input Buffer A scanner or lexical analyzer reads the input character by character. The process will be very inefficient if it sends request to the OS for every character read.

69 Compiler Design 69 Input Buffer OS reads a block of data, supplies the requesting process the required amount, and stored the remaining portion in a buffer called buffer cache. In subsequent calls, the actual IO does not take place as long as the data is available in the buffer. Requesting OS for single character is also costly due to context-switching overhead. So the scanner uses its own buffer.

70 Compiler Design 70 Input Buffer A buffer at its end may contain an initial portion of a lexeme. It creates problem in refilling the buffer. So a 2-buffer scheme is used. The buffers are filled alternatively. A sentinel-character is placed at the end-of-buffer to avoid two comparisons - character and end-of-buffer. We may run out of buffer space for a long

71 Compiler Design 71 character string or a comment.

72 Compiler Design 72 Direct DFA Construction from a Regular Expression A deterministic finite automaton can be constructed directly from the given regular expression.

73 Compiler Design 73 Important States: a Definition All initial states of an NFA are important. Any other state p of an NFA is called important if p has an out-transition on some a Σ. The ε-closure of the important states of an NFA are used to calculate the next state of the equivalent DFA.

74 Compiler Design 74 Important States Important states are introduced in the NFA by the start states and each symbol positions of the regular expression. In our example, a+(ab) the important states are 8,0,2,4.

75 Compiler Design 75 a+(ab) - An Example a 0 1 ε ε 8 9 ε ε ε a ε ε ε b ε

76 Compiler Design 76 End Marker and Final State We introduce a special end marker # Σ to the regular expression, r (r)#. This makes the final state(s) of the original NFA important. It also helps to detect the final state(s) (a state that has transition on #).

77 Compiler Design 77 Syntax Tree of a Regular Expression The regular expression is represented by a syntax tree where each leaf node corresponds to a symbol of the alphabet, a Σ, or ε. Each internal nodes corresponds to an operator symbol.

78 Compiler Design 78 Syntax Tree of a+(ab) # + # a * a b

79 Compiler Design 79 Labelling the Leaf Nodes We associate a positive integer p with each leaf node of a Σ (not of ε). The positive integer p is called the position of the symbol of the leaf node. Following are a few definitions where n is a node and p is a position.

80 Compiler Design 80 Definitions nullable(n): A node n is nullable if the language of its subexpression contains ǫ. firstpos(n): It is the set of positions in the subtree of n, from where the first symbol of any string of the language corresponding to the subexpression of n may come.

81 Compiler Design 81 DFA directly from Regular Expression lastpos(n): it is similar to the firstpos(n) except that these are the positions of the last symbols. followpos(p): It is the set positions in the syntax tree from where a symbol may come after the symbol of the position p in a string of L((r)#).

82 Compiler Design 82 Computation of nullable(n) n is a leaf node with label ε: true. leaf node with label a Σ: false. internal node of the form n 1 +n 2 : nullable(n 1 ) nullable(n 2 ). internal node of the form n 1 n 2 : nullable(n 1 ) nullable(n 2 ). internal node of the form n 1 : true.

83 Compiler Design 83 Computation of firstpos(n) n is a leaf node with label ε:. leaf node with label a Σ: {a}. internal node of the form n 1 +n 2 : firstpos(n 1 ) firstpos(n 2 ). internal node of the form n 1 n 2 : if nullable(n 1 ), then firstpos(n 1 ) firstpos(n 2 ), else firstpos(n 1 ). internal node of the form n 1: firstpos(n 1 ).

84 Compiler Design 84 Computation of lastpos(n) n is a leaf node with label ε:. leaf node with label a Σ: {a}. internal node of the form n 1 +n 2 : lastpos(n 1 ) lastpos(n 2 ). internal node of the form n 1 n 2 : if nullable(n 2 ), then lastpos(n 1 ) lastpos(n 2 ), else lastpos(n 2 ). internal node of the form n 1: lastpos(n 2 ).

85 Compiler Design 85 Example In our example there are two nullable nodes, the + and the nodes. We decorate the syntax tree with ifirstpos() and lastpos() data.

86 Compiler Design 86 ({1,2,4},{4}) ({1,2},{1,3}) ({1},{1}) a 1 + * # ({4},{4}) 4 ({2},{3}) ({2},{3}) ({2},{2}) a b 2 3 ({3},{3})

87 Compiler Design 87 Computation of followpos(p) Given a regular expression r, a symbol of a particular position can be followed by a symbol of another position in a string of L(r) in two different ways. If n is a concatenation node n 1 n 2 of the syntax tree, then for each position p in lastpos(n 1 ), each position q of firstpos(n 2 ) is in followpos(p).

88 Compiler Design 88 Computation of followpos(p) If n is a Kleene-star node of the syntax tree, then for each position p in lastpos(n), each position q of firstpos(n) is in followpos(p).

89 Compiler Design 89 Example In our example, from the concatenation node we get that 3 followpos(2), 4 followpos(1) and followpos(3). from the Kleene-star node we get 2 followpos(3).

90 Compiler Design 90 Example The following table summaries followpos() of different positions. Position p followpos(p) 1 {4} 2 {3} 3 {2, 4} 4

91 Compiler Design 91 Directed Graph of followpos() Each position p is represented by a node. There is a directed edge from a position p to a position q, if q followpos(p).

92 Compiler Design 92 Directed Graph of the Example

93 Compiler Design 93 Directed Graph to NFA This directed graph is actually an NFA without ε-transition. All positions in the firstpos(root) are initial states. A transition from p q is labelled by the symbol of position p. The node corresponding to the position of # is the accepting state.

94 Compiler Design 94 Directed Graph to NFA: the Example a b b a

95 Compiler Design 95 DFA from Regular Expression - Direct Construction Input: A regular expression r over Σ Output: A DFA M = (Q,Σ,s,F,δ). Algorithm: 1. Construct a syntax tree T corresponding to the augmented regular expression (r)#, where # Σ.

96 Compiler Design 96 DFA from Regular Expression - Directly 2. Compute nullable, firstpos, lastpos and followpos of the syntax tree T. 3. The construction of M is as follows: The set of states Q of M are the subsets of the positions of T. The start state s = firstpos(root(t)). The final states are all the subsets containing the position of #.

97 Compiler Design 97 Construction of δ tag[firstpos(root(t))] 0 Q firstpos(root(t)) while (α Q and tag[α] = 0) do tag[α] 1 a Σ do positions p α of a Σ, collect followpos(p) in a set β if (β Q) tag[β] 0 Q Q {β} δ(α,a) β.

98 Compiler Design 98 DFA of the Example The state transition table: Initial Final State State a b A : {1,2,4} {3,4} B : {3,4} {2,4} C : {2,4} {3} D : {3} {2,4} Start state: A{1,2,4}, Final states:{a{1, 2, 4}, B{3, 4}, C{2, 4}}.

99 Compiler Design 99 DFA State Transition Diagram 1,2,4 A a 3,4 2,4 3 B b C a D b

Module 6 Lexical Phase - RE to DFA

Module 6 Lexical Phase - RE to DFA Module 6 Lexical Phase - RE to DFA The objective of this module is to construct a minimized DFA from a regular expression. A NFA is typically easier to construct but string matching with a NFA is slower.

More information

CS308 Compiler Principles Lexical Analyzer Li Jiang

CS308 Compiler Principles Lexical Analyzer Li Jiang CS308 Lexical Analyzer Li Jiang Department of Computer Science and Engineering Shanghai Jiao Tong University Content: Outline Basic concepts: pattern, lexeme, and token. Operations on languages, and regular

More information

UNIT II LEXICAL ANALYSIS

UNIT II LEXICAL ANALYSIS UNIT II LEXICAL ANALYSIS 2 Marks 1. What are the issues in lexical analysis? Simpler design Compiler efficiency is improved Compiler portability is enhanced. 2. Define patterns/lexeme/tokens? This set

More information

[Lexical Analysis] Bikash Balami

[Lexical Analysis] Bikash Balami 1 [Lexical Analysis] Compiler Design and Construction (CSc 352) Compiled By Central Department of Computer Science and Information Technology (CDCSIT) Tribhuvan University, Kirtipur Kathmandu, Nepal 2

More information

COMP-421 Compiler Design. Presented by Dr Ioanna Dionysiou

COMP-421 Compiler Design. Presented by Dr Ioanna Dionysiou COMP-421 Compiler Design Presented by Dr Ioanna Dionysiou Administrative! [ALSU03] Chapter 3 - Lexical Analysis Sections 3.1-3.4, 3.6-3.7! Reading for next time [ALSU03] Chapter 3 Copyright (c) 2010 Ioanna

More information

Dixita Kagathara Page 1

Dixita Kagathara Page 1 2014 Sem - VII Lexical Analysis 1) Role of lexical analysis and its issues. The lexical analyzer is the first phase of compiler. Its main task is to read the input characters and produce as output a sequence

More information

Lexical Analysis. Dragon Book Chapter 3 Formal Languages Regular Expressions Finite Automata Theory Lexical Analysis using Automata

Lexical Analysis. Dragon Book Chapter 3 Formal Languages Regular Expressions Finite Automata Theory Lexical Analysis using Automata Lexical Analysis Dragon Book Chapter 3 Formal Languages Regular Expressions Finite Automata Theory Lexical Analysis using Automata Phase Ordering of Front-Ends Lexical analysis (lexer) Break input string

More information

Formal Languages and Compilers Lecture IV: Regular Languages and Finite. Finite Automata

Formal Languages and Compilers Lecture IV: Regular Languages and Finite. Finite Automata Formal Languages and Compilers Lecture IV: Regular Languages and Finite Automata Free University of Bozen-Bolzano Faculty of Computer Science POS Building, Room: 2.03 artale@inf.unibz.it http://www.inf.unibz.it/

More information

Zhizheng Zhang. Southeast University

Zhizheng Zhang. Southeast University Zhizheng Zhang Southeast University 2016/10/5 Lexical Analysis 1 1. The Role of Lexical Analyzer 2016/10/5 Lexical Analysis 2 2016/10/5 Lexical Analysis 3 Example. position = initial + rate * 60 2016/10/5

More information

COMPILER DESIGN UNIT I LEXICAL ANALYSIS. Translator: It is a program that translates one language to another Language.

COMPILER DESIGN UNIT I LEXICAL ANALYSIS. Translator: It is a program that translates one language to another Language. UNIT I LEXICAL ANALYSIS Translator: It is a program that translates one language to another Language. Source Code Translator Target Code 1. INTRODUCTION TO LANGUAGE PROCESSING The Language Processing System

More information

Figure 2.1: Role of Lexical Analyzer

Figure 2.1: Role of Lexical Analyzer Chapter 2 Lexical Analysis Lexical analysis or scanning is the process which reads the stream of characters making up the source program from left-to-right and groups them into tokens. The lexical analyzer

More information

Lexical Analysis. Prof. James L. Frankel Harvard University

Lexical Analysis. Prof. James L. Frankel Harvard University Lexical Analysis Prof. James L. Frankel Harvard University Version of 5:37 PM 30-Jan-2018 Copyright 2018, 2016, 2015 James L. Frankel. All rights reserved. Regular Expression Notation We will develop a

More information

Formal Languages and Compilers Lecture VI: Lexical Analysis

Formal Languages and Compilers Lecture VI: Lexical Analysis Formal Languages and Compilers Lecture VI: Lexical Analysis Free University of Bozen-Bolzano Faculty of Computer Science POS Building, Room: 2.03 artale@inf.unibz.it http://www.inf.unibz.it/ artale/ Formal

More information

Compiler Construction

Compiler Construction Compiler Construction Lecture 2: Lexical Analysis I (Introduction) Thomas Noll Lehrstuhl für Informatik 2 (Software Modeling and Verification) noll@cs.rwth-aachen.de http://moves.rwth-aachen.de/teaching/ss-14/cc14/

More information

Compiler Construction

Compiler Construction Compiler Construction Thomas Noll Software Modeling and Verification Group RWTH Aachen University https://moves.rwth-aachen.de/teaching/ss-16/cc/ Conceptual Structure of a Compiler Source code x1 := y2

More information

1. INTRODUCTION TO LANGUAGE PROCESSING The Language Processing System can be represented as shown figure below.

1. INTRODUCTION TO LANGUAGE PROCESSING The Language Processing System can be represented as shown figure below. UNIT I Translator: It is a program that translates one language to another Language. Examples of translator are compiler, assembler, interpreter, linker, loader and preprocessor. Source Code Translator

More information

Lexical Analysis. Lexical analysis is the first phase of compilation: The file is converted from ASCII to tokens. It must be fast!

Lexical Analysis. Lexical analysis is the first phase of compilation: The file is converted from ASCII to tokens. It must be fast! Lexical Analysis Lexical analysis is the first phase of compilation: The file is converted from ASCII to tokens. It must be fast! Compiler Passes Analysis of input program (front-end) character stream

More information

CSc 453 Lexical Analysis (Scanning)

CSc 453 Lexical Analysis (Scanning) CSc 453 Lexical Analysis (Scanning) Saumya Debray The University of Arizona Tucson Overview source program lexical analyzer (scanner) tokens syntax analyzer (parser) symbol table manager Main task: to

More information

Front End: Lexical Analysis. The Structure of a Compiler

Front End: Lexical Analysis. The Structure of a Compiler Front End: Lexical Analysis The Structure of a Compiler Constructing a Lexical Analyser By hand: Identify lexemes in input and return tokens Automatically: Lexical-Analyser generator We will learn about

More information

UNIT -2 LEXICAL ANALYSIS

UNIT -2 LEXICAL ANALYSIS OVER VIEW OF LEXICAL ANALYSIS UNIT -2 LEXICAL ANALYSIS o To identify the tokens we need some method of describing the possible tokens that can appear in the input stream. For this purpose we introduce

More information

2010: Compilers REVIEW: REGULAR EXPRESSIONS HOW TO USE REGULAR EXPRESSIONS

2010: Compilers REVIEW: REGULAR EXPRESSIONS HOW TO USE REGULAR EXPRESSIONS 2010: Compilers Lexical Analysis: Finite State Automata Dr. Licia Capra UCL/CS REVIEW: REGULAR EXPRESSIONS a Character in A Empty string R S Alternation (either R or S) RS Concatenation (R followed by

More information

Implementation of Lexical Analysis

Implementation of Lexical Analysis Implementation of Lexical Analysis Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Non-deterministic Finite Automata (NFAs) Implementation

More information

Lexical Analysis 1 / 52

Lexical Analysis 1 / 52 Lexical Analysis 1 / 52 Outline 1 Scanning Tokens 2 Regular Expresssions 3 Finite State Automata 4 Non-deterministic (NFA) Versus Deterministic Finite State Automata (DFA) 5 Regular Expresssions to NFA

More information

Compiler Design 1. Bottom-UP Parsing. Goutam Biswas. Lect 6

Compiler Design 1. Bottom-UP Parsing. Goutam Biswas. Lect 6 Compiler Design 1 Bottom-UP Parsing Compiler Design 2 The Process The parse tree is built starting from the leaf nodes labeled by the terminals (tokens). The parser tries to discover appropriate reductions,

More information

2. Lexical Analysis! Prof. O. Nierstrasz!

2. Lexical Analysis! Prof. O. Nierstrasz! 2. Lexical Analysis! Prof. O. Nierstrasz! Thanks to Jens Palsberg and Tony Hosking for their kind permission to reuse and adapt the CS132 and CS502 lecture notes.! http://www.cs.ucla.edu/~palsberg/! http://www.cs.purdue.edu/homes/hosking/!

More information

CS412/413. Introduction to Compilers Tim Teitelbaum. Lecture 2: Lexical Analysis 23 Jan 08

CS412/413. Introduction to Compilers Tim Teitelbaum. Lecture 2: Lexical Analysis 23 Jan 08 CS412/413 Introduction to Compilers Tim Teitelbaum Lecture 2: Lexical Analysis 23 Jan 08 Outline Review compiler structure What is lexical analysis? Writing a lexer Specifying tokens: regular expressions

More information

ECS 120 Lesson 7 Regular Expressions, Pt. 1

ECS 120 Lesson 7 Regular Expressions, Pt. 1 ECS 120 Lesson 7 Regular Expressions, Pt. 1 Oliver Kreylos Friday, April 13th, 2001 1 Outline Thus far, we have been discussing one way to specify a (regular) language: Giving a machine that reads a word

More information

Regular Expressions. Agenda for Today. Grammar for a Tiny Language. Programming Language Specifications

Regular Expressions. Agenda for Today. Grammar for a Tiny Language. Programming Language Specifications Agenda for Today Regular Expressions CSE 413, Autumn 2005 Programming Languages Basic concepts of formal grammars Regular expressions Lexical specification of programming languages Using finite automata

More information

Outline. 1 Scanning Tokens. 2 Regular Expresssions. 3 Finite State Automata

Outline. 1 Scanning Tokens. 2 Regular Expresssions. 3 Finite State Automata Outline 1 2 Regular Expresssions Lexical Analysis 3 Finite State Automata 4 Non-deterministic (NFA) Versus Deterministic Finite State Automata (DFA) 5 Regular Expresssions to NFA 6 NFA to DFA 7 8 JavaCC:

More information

Introduction to Lexical Analysis

Introduction to Lexical Analysis Introduction to Lexical Analysis Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexical analyzers (lexers) Regular

More information

Implementation of Lexical Analysis

Implementation of Lexical Analysis Implementation of Lexical Analysis Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Non-deterministic Finite Automata (NFAs) Implementation

More information

Lexical Analysis. Chapter 2

Lexical Analysis. Chapter 2 Lexical Analysis Chapter 2 1 Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexers Regular expressions Examples

More information

Lexical Analysis. Introduction

Lexical Analysis. Introduction Lexical Analysis Introduction Copyright 2015, Pedro C. Diniz, all rights reserved. Students enrolled in the Compilers class at the University of Southern California have explicit permission to make copies

More information

CSEP 501 Compilers. Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter /8/ Hal Perkins & UW CSE B-1

CSEP 501 Compilers. Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter /8/ Hal Perkins & UW CSE B-1 CSEP 501 Compilers Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter 2008 1/8/2008 2002-08 Hal Perkins & UW CSE B-1 Agenda Basic concepts of formal grammars (review) Regular expressions

More information

Languages and Compilers

Languages and Compilers Principles of Software Engineering and Operational Systems Languages and Compilers SDAGE: Level I 2012-13 3. Formal Languages, Grammars and Automata Dr Valery Adzhiev vadzhiev@bournemouth.ac.uk Office:

More information

CSE 413 Programming Languages & Implementation. Hal Perkins Autumn 2012 Grammars, Scanners & Regular Expressions

CSE 413 Programming Languages & Implementation. Hal Perkins Autumn 2012 Grammars, Scanners & Regular Expressions CSE 413 Programming Languages & Implementation Hal Perkins Autumn 2012 Grammars, Scanners & Regular Expressions 1 Agenda Overview of language recognizers Basic concepts of formal grammars Scanner Theory

More information

David Griol Barres Computer Science Department Carlos III University of Madrid Leganés (Spain)

David Griol Barres Computer Science Department Carlos III University of Madrid Leganés (Spain) David Griol Barres dgriol@inf.uc3m.es Computer Science Department Carlos III University of Madrid Leganés (Spain) OUTLINE Introduction: Definitions The role of the Lexical Analyzer Scanner Implementation

More information

About the Tutorial. Audience. Prerequisites. Copyright & Disclaimer. Compiler Design

About the Tutorial. Audience. Prerequisites. Copyright & Disclaimer. Compiler Design i About the Tutorial A compiler translates the codes written in one language to some other language without changing the meaning of the program. It is also expected that a compiler should make the target

More information

Finite automata. We have looked at using Lex to build a scanner on the basis of regular expressions.

Finite automata. We have looked at using Lex to build a scanner on the basis of regular expressions. Finite automata We have looked at using Lex to build a scanner on the basis of regular expressions. Now we begin to consider the results from automata theory that make Lex possible. Recall: An alphabet

More information

Computer Science Department Carlos III University of Madrid Leganés (Spain) David Griol Barres

Computer Science Department Carlos III University of Madrid Leganés (Spain) David Griol Barres Computer Science Department Carlos III University of Madrid Leganés (Spain) David Griol Barres dgriol@inf.uc3m.es Introduction: Definitions Lexical analysis or scanning: To read from left-to-right a source

More information

Lexical Analysis. Implementation: Finite Automata

Lexical Analysis. Implementation: Finite Automata Lexical Analysis Implementation: Finite Automata Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Non-deterministic Finite Automata (NFAs)

More information

CS 314 Principles of Programming Languages. Lecture 3

CS 314 Principles of Programming Languages. Lecture 3 CS 314 Principles of Programming Languages Lecture 3 Zheng Zhang Department of Computer Science Rutgers University Wednesday 14 th September, 2016 Zheng Zhang 1 CS@Rutgers University Class Information

More information

Compiler phases. Non-tokens

Compiler phases. Non-tokens Compiler phases Compiler Construction Scanning Lexical Analysis source code scanner tokens regular expressions lexical analysis Lennart Andersson parser context free grammar Revision 2011 01 21 parse tree

More information

Formal Languages and Grammars. Chapter 2: Sections 2.1 and 2.2

Formal Languages and Grammars. Chapter 2: Sections 2.1 and 2.2 Formal Languages and Grammars Chapter 2: Sections 2.1 and 2.2 Formal Languages Basis for the design and implementation of programming languages Alphabet: finite set Σ of symbols String: finite sequence

More information

Lexical Analysis. Lecture 2-4

Lexical Analysis. Lecture 2-4 Lexical Analysis Lecture 2-4 Notes by G. Necula, with additions by P. Hilfinger Prof. Hilfinger CS 164 Lecture 2 1 Administrivia Moving to 60 Evans on Wednesday HW1 available Pyth manual available on line.

More information

Lexical Analysis. COMP 524, Spring 2014 Bryan Ward

Lexical Analysis. COMP 524, Spring 2014 Bryan Ward Lexical Analysis COMP 524, Spring 2014 Bryan Ward Based in part on slides and notes by J. Erickson, S. Krishnan, B. Brandenburg, S. Olivier, A. Block and others The Big Picture Character Stream Scanner

More information

Lexical Analyzer Scanner

Lexical Analyzer Scanner Lexical Analyzer Scanner ASU Textbook Chapter 3.1, 3.3, 3.4, 3.6, 3.7, 3.5 Tsan-sheng Hsu tshsu@iis.sinica.edu.tw http://www.iis.sinica.edu.tw/~tshsu 1 Main tasks Read the input characters and produce

More information

Lexical Analysis. Lecture 3. January 10, 2018

Lexical Analysis. Lecture 3. January 10, 2018 Lexical Analysis Lecture 3 January 10, 2018 Announcements PA1c due tonight at 11:50pm! Don t forget about PA1, the Cool implementation! Use Monday s lecture, the video guides and Cool examples if you re

More information

Part 5 Program Analysis Principles and Techniques

Part 5 Program Analysis Principles and Techniques 1 Part 5 Program Analysis Principles and Techniques Front end 2 source code scanner tokens parser il errors Responsibilities: Recognize legal programs Report errors Produce il Preliminary storage map Shape

More information

Chapter 4. Lexical analysis. Concepts. Lexical scanning Regular expressions DFAs and FSAs Lex. Lexical analysis in perspective

Chapter 4. Lexical analysis. Concepts. Lexical scanning Regular expressions DFAs and FSAs Lex. Lexical analysis in perspective Chapter 4 Lexical analysis Lexical scanning Regular expressions DFAs and FSAs Lex Concepts CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 1 CMSC 331, Some material 1998 by Addison Wesley

More information

CS Lecture 2. The Front End. Lecture 2 Lexical Analysis

CS Lecture 2. The Front End. Lecture 2 Lexical Analysis CS 1622 Lecture 2 Lexical Analysis CS 1622 Lecture 2 1 Lecture 2 Review of last lecture and finish up overview The first compiler phase: lexical analysis Reading: Chapter 2 in text (by 1/18) CS 1622 Lecture

More information

A Simple Syntax-Directed Translator

A Simple Syntax-Directed Translator Chapter 2 A Simple Syntax-Directed Translator 1-1 Introduction The analysis phase of a compiler breaks up a source program into constituent pieces and produces an internal representation for it, called

More information

Lexical Analyzer Scanner

Lexical Analyzer Scanner Lexical Analyzer Scanner ASU Textbook Chapter 3.1, 3.3, 3.4, 3.6, 3.7, 3.5 Tsan-sheng Hsu tshsu@iis.sinica.edu.tw http://www.iis.sinica.edu.tw/~tshsu 1 Main tasks Read the input characters and produce

More information

Syntactic Analysis. CS345H: Programming Languages. Lecture 3: Lexical Analysis. Outline. Lexical Analysis. What is a Token? Tokens

Syntactic Analysis. CS345H: Programming Languages. Lecture 3: Lexical Analysis. Outline. Lexical Analysis. What is a Token? Tokens Syntactic Analysis CS45H: Programming Languages Lecture : Lexical Analysis Thomas Dillig Main Question: How to give structure to strings Analogy: Understanding an English sentence First, we separate a

More information

CSE302: Compiler Design

CSE302: Compiler Design CSE302: Compiler Design Instructor: Dr. Liang Cheng Department of Computer Science and Engineering P.C. Rossin College of Engineering & Applied Science Lehigh University February 01, 2007 Outline Recap

More information

CSE 130 Programming Language Principles & Paradigms Lecture # 5. Chapter 4 Lexical and Syntax Analysis

CSE 130 Programming Language Principles & Paradigms Lecture # 5. Chapter 4 Lexical and Syntax Analysis Chapter 4 Lexical and Syntax Analysis Introduction - Language implementation systems must analyze source code, regardless of the specific implementation approach - Nearly all syntax analysis is based on

More information

CSE302: Compiler Design

CSE302: Compiler Design CSE302: Compiler Design Instructor: Dr. Liang Cheng Department of Computer Science and Engineering P.C. Rossin College of Engineering & Applied Science Lehigh University February 13, 2007 Outline Recap

More information

Languages, Automata, Regular Expressions & Scanners. Winter /8/ Hal Perkins & UW CSE B-1

Languages, Automata, Regular Expressions & Scanners. Winter /8/ Hal Perkins & UW CSE B-1 CSE 401 Compilers Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter 2010 1/8/2010 2002-10 Hal Perkins & UW CSE B-1 Agenda Quick review of basic concepts of formal grammars Regular

More information

PRINCIPLES OF COMPILER DESIGN UNIT II LEXICAL ANALYSIS 2.1 Lexical Analysis - The Role of the Lexical Analyzer

PRINCIPLES OF COMPILER DESIGN UNIT II LEXICAL ANALYSIS 2.1 Lexical Analysis - The Role of the Lexical Analyzer PRINCIPLES OF COMPILER DESIGN UNIT II LEXICAL ANALYSIS 2.1 Lexical Analysis - The Role of the Lexical Analyzer As the first phase of a compiler, the main task of the lexical analyzer is to read the input

More information

The Language for Specifying Lexical Analyzer

The Language for Specifying Lexical Analyzer The Language for Specifying Lexical Analyzer We shall now study how to build a lexical analyzer from a specification of tokens in the form of a list of regular expressions The discussion centers around

More information

CS6660 COMPILER DESIGN L T P C

CS6660 COMPILER DESIGN L T P C COMPILER DESIGN CS6660 COMPILER DESIGN L T P C 3 0 0 3 UNIT I INTRODUCTION TO COMPILERS 5 Translators-Compilation and Interpretation-Language processors -The Phases of CompilerErrors Encountered in Different

More information

Concepts. Lexical scanning Regular expressions DFAs and FSAs Lex. Lexical analysis in perspective

Concepts. Lexical scanning Regular expressions DFAs and FSAs Lex. Lexical analysis in perspective Concepts Lexical scanning Regular expressions DFAs and FSAs Lex CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 1 CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 2 Lexical analysis

More information

Principles of Compiler Design Presented by, R.Venkadeshan,M.Tech-IT, Lecturer /CSE Dept, Chettinad College of Engineering &Technology

Principles of Compiler Design Presented by, R.Venkadeshan,M.Tech-IT, Lecturer /CSE Dept, Chettinad College of Engineering &Technology Principles of Compiler Design Presented by, R.Venkadeshan,M.Tech-IT, Lecturer /CSE Dept, Chettinad College of Engineering &Technology 6/30/2010 Principles of Compiler Design R.Venkadeshan 1 Preliminaries

More information

Lexical Analysis. Chapter 1, Section Chapter 3, Section 3.1, 3.3, 3.4, 3.5 JFlex Manual

Lexical Analysis. Chapter 1, Section Chapter 3, Section 3.1, 3.3, 3.4, 3.5 JFlex Manual Lexical Analysis Chapter 1, Section 1.2.1 Chapter 3, Section 3.1, 3.3, 3.4, 3.5 JFlex Manual Inside the Compiler: Front End Lexical analyzer (aka scanner) Converts ASCII or Unicode to a stream of tokens

More information

MIT Specifying Languages with Regular Expressions and Context-Free Grammars

MIT Specifying Languages with Regular Expressions and Context-Free Grammars MIT 6.035 Specifying Languages with Regular essions and Context-Free Grammars Martin Rinard Laboratory for Computer Science Massachusetts Institute of Technology Language Definition Problem How to precisely

More information

Non-deterministic Finite Automata (NFA)

Non-deterministic Finite Automata (NFA) Non-deterministic Finite Automata (NFA) CAN have transitions on the same input to different states Can include a ε or λ transition (i.e. move to new state without reading input) Often easier to design

More information

Nondeterministic Finite Automata (NFA): Nondeterministic Finite Automata (NFA) states of an automaton of this kind may or may not have a transition for each symbol in the alphabet, or can even have multiple

More information

Lexical Analysis. Lecture 3-4

Lexical Analysis. Lecture 3-4 Lexical Analysis Lecture 3-4 Notes by G. Necula, with additions by P. Hilfinger Prof. Hilfinger CS 164 Lecture 3-4 1 Administrivia I suggest you start looking at Python (see link on class home page). Please

More information

Lexical Analysis. Finite Automata

Lexical Analysis. Finite Automata #1 Lexical Analysis Finite Automata Cool Demo? (Part 1 of 2) #2 Cunning Plan Informal Sketch of Lexical Analysis LA identifies tokens from input string lexer : (char list) (token list) Issues in Lexical

More information

Concepts Introduced in Chapter 3. Lexical Analysis. Lexical Analysis Terms. Attributes for Tokens

Concepts Introduced in Chapter 3. Lexical Analysis. Lexical Analysis Terms. Attributes for Tokens Concepts Introduced in Chapter 3 Lexical Analysis Regular Expressions (REs) Nondeterministic Finite Automata (NFA) Converting an RE to an NFA Deterministic Finite Automatic (DFA) Lexical Analysis Why separate

More information

Lexical Analysis (ASU Ch 3, Fig 3.1)

Lexical Analysis (ASU Ch 3, Fig 3.1) Lexical Analysis (ASU Ch 3, Fig 3.1) Implementation by hand automatically ((F)Lex) Lex generates a finite automaton recogniser uses regular expressions Tasks remove white space (ws) display source program

More information

DVA337 HT17 - LECTURE 4. Languages and regular expressions

DVA337 HT17 - LECTURE 4. Languages and regular expressions DVA337 HT17 - LECTURE 4 Languages and regular expressions 1 SO FAR 2 TODAY Formal definition of languages in terms of strings Operations on strings and languages Definition of regular expressions Meaning

More information

CS 315 Programming Languages Syntax. Parser. (Alternatively hand-built) (Alternatively hand-built)

CS 315 Programming Languages Syntax. Parser. (Alternatively hand-built) (Alternatively hand-built) Programming languages must be precise Remember instructions This is unlike natural languages CS 315 Programming Languages Syntax Precision is required for syntax think of this as the format of the language

More information

1. Lexical Analysis Phase

1. Lexical Analysis Phase 1. Lexical Analysis Phase The purpose of the lexical analyzer is to read the source program, one character at time, and to translate it into a sequence of primitive units called tokens. Keywords, identifiers,

More information

Lexical Analysis. Finite Automata

Lexical Analysis. Finite Automata #1 Lexical Analysis Finite Automata Cool Demo? (Part 1 of 2) #2 Cunning Plan Informal Sketch of Lexical Analysis LA identifies tokens from input string lexer : (char list) (token list) Issues in Lexical

More information

Chapter 4. Lexical and Syntax Analysis. Topics. Compilation. Language Implementation. Issues in Lexical and Syntax Analysis.

Chapter 4. Lexical and Syntax Analysis. Topics. Compilation. Language Implementation. Issues in Lexical and Syntax Analysis. Topics Chapter 4 Lexical and Syntax Analysis Introduction Lexical Analysis Syntax Analysis Recursive -Descent Parsing Bottom-Up parsing 2 Language Implementation Compilation There are three possible approaches

More information

Lecture 4: Syntax Specification

Lecture 4: Syntax Specification The University of North Carolina at Chapel Hill Spring 2002 Lecture 4: Syntax Specification Jan 16 1 Phases of Compilation 2 1 Syntax Analysis Syntax: Webster s definition: 1 a : the way in which linguistic

More information

CSE 413 Programming Languages & Implementation. Hal Perkins Winter 2019 Grammars, Scanners & Regular Expressions

CSE 413 Programming Languages & Implementation. Hal Perkins Winter 2019 Grammars, Scanners & Regular Expressions CSE 413 Programming Languages & Implementation Hal Perkins Winter 2019 Grammars, Scanners & Regular Expressions 1 Agenda Overview of language recognizers Basic concepts of formal grammars Scanner Theory

More information

Compiler Construction LECTURE # 3

Compiler Construction LECTURE # 3 Compiler Construction LECTURE # 3 The Course Course Code: CS-4141 Course Title: Compiler Construction Instructor: JAWAD AHMAD Email Address: jawadahmad@uoslahore.edu.pk Web Address: http://csandituoslahore.weebly.com/cc.html

More information

The Front End. The purpose of the front end is to deal with the input language. Perform a membership test: code source language?

The Front End. The purpose of the front end is to deal with the input language. Perform a membership test: code source language? The Front End Source code Front End IR Back End Machine code Errors The purpose of the front end is to deal with the input language Perform a membership test: code source language? Is the program well-formed

More information

4. Lexical and Syntax Analysis

4. Lexical and Syntax Analysis 4. Lexical and Syntax Analysis 4.1 Introduction Language implementation systems must analyze source code, regardless of the specific implementation approach Nearly all syntax analysis is based on a formal

More information

Compiler course. Chapter 3 Lexical Analysis

Compiler course. Chapter 3 Lexical Analysis Compiler course Chapter 3 Lexical Analysis 1 A. A. Pourhaji Kazem, Spring 2009 Outline Role of lexical analyzer Specification of tokens Recognition of tokens Lexical analyzer generator Finite automata

More information

3. Syntax Analysis. Andrea Polini. Formal Languages and Compilers Master in Computer Science University of Camerino

3. Syntax Analysis. Andrea Polini. Formal Languages and Compilers Master in Computer Science University of Camerino 3. Syntax Analysis Andrea Polini Formal Languages and Compilers Master in Computer Science University of Camerino (Formal Languages and Compilers) 3. Syntax Analysis CS@UNICAM 1 / 54 Syntax Analysis: the

More information

CSCI312 Principles of Programming Languages!

CSCI312 Principles of Programming Languages! CSCI312 Principles of Programming Languages!! Chapter 3 Regular Expression and Lexer Xu Liu Recap! Copyright 2006 The McGraw-Hill Companies, Inc. Clite: Lexical Syntax! Input: a stream of characters from

More information

MIT Specifying Languages with Regular Expressions and Context-Free Grammars. Martin Rinard Massachusetts Institute of Technology

MIT Specifying Languages with Regular Expressions and Context-Free Grammars. Martin Rinard Massachusetts Institute of Technology MIT 6.035 Specifying Languages with Regular essions and Context-Free Grammars Martin Rinard Massachusetts Institute of Technology Language Definition Problem How to precisely define language Layered structure

More information

Question Bank. 10CS63:Compiler Design

Question Bank. 10CS63:Compiler Design Question Bank 10CS63:Compiler Design 1.Determine whether the following regular expressions define the same language? (ab)* and a*b* 2.List the properties of an operator grammar 3. Is macro processing a

More information

CS 432 Fall Mike Lam, Professor. Finite Automata Conversions and Lexing

CS 432 Fall Mike Lam, Professor. Finite Automata Conversions and Lexing CS 432 Fall 2017 Mike Lam, Professor Finite Automata Conversions and Lexing Finite Automata Key result: all of the following have the same expressive power (i.e., they all describe regular languages):

More information

CSE 401/M501 Compilers

CSE 401/M501 Compilers CSE 401/M501 Compilers Languages, Automata, Regular Expressions & Scanners Hal Perkins Spring 2018 UW CSE 401/M501 Spring 2018 B-1 Administrivia No sections this week Read: textbook ch. 1 and sec. 2.1-2.4

More information

Chapter 4. Lexical and Syntax Analysis

Chapter 4. Lexical and Syntax Analysis Chapter 4 Lexical and Syntax Analysis Chapter 4 Topics Introduction Lexical Analysis The Parsing Problem Recursive-Descent Parsing Bottom-Up Parsing Copyright 2012 Addison-Wesley. All rights reserved.

More information

Regular Languages. MACM 300 Formal Languages and Automata. Formal Languages: Recap. Regular Languages

Regular Languages. MACM 300 Formal Languages and Automata. Formal Languages: Recap. Regular Languages Regular Languages MACM 3 Formal Languages and Automata Anoop Sarkar http://www.cs.sfu.ca/~anoop The set of regular languages: each element is a regular language Each regular language is an example of a

More information

Introduction to Lexical Analysis

Introduction to Lexical Analysis Introduction to Lexical Analysis Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexers Regular expressions Examples

More information

UNIT III & IV. Bottom up parsing

UNIT III & IV. Bottom up parsing UNIT III & IV Bottom up parsing 5.0 Introduction Given a grammar and a sentence belonging to that grammar, if we have to show that the given sentence belongs to the given grammar, there are two methods.

More information

Lexical Analysis. Sukree Sinthupinyo July Chulalongkorn University

Lexical Analysis. Sukree Sinthupinyo July Chulalongkorn University Sukree Sinthupinyo 1 1 Department of Computer Engineering Chulalongkorn University 14 July 2012 Outline Introduction 1 Introduction 2 3 4 Transition Diagrams Learning Objectives Understand definition of

More information

Chapter 3 Lexical Analysis

Chapter 3 Lexical Analysis Chapter 3 Lexical Analysis Outline Role of lexical analyzer Specification of tokens Recognition of tokens Lexical analyzer generator Finite automata Design of lexical analyzer generator The role of lexical

More information

Chapter Seven: Regular Expressions

Chapter Seven: Regular Expressions Chapter Seven: Regular Expressions Regular Expressions We have seen that DFAs and NFAs have equal definitional power. It turns out that regular expressions also have exactly that same definitional power:

More information

J. Xue. Tutorials. Tutorials to start in week 3 (i.e., next week) Tutorial questions are already available on-line

J. Xue. Tutorials. Tutorials to start in week 3 (i.e., next week) Tutorial questions are already available on-line Tutorials Tutorials to start in week 3 (i.e., next week) Tutorial questions are already available on-line COMP3131/9102 Page 65 March 4, 2018 Assignment 1: Scanner +5 = two tokens: + and 5 the scanner

More information

Implementation of Lexical Analysis

Implementation of Lexical Analysis Implementation of Lexical Analysis Lecture 4 (Modified by Professor Vijay Ganesh) Tips on Building Large Systems KISS (Keep It Simple, Stupid!) Don t optimize prematurely Design systems that can be tested

More information

Bottom-Up Parsing. Lecture 11-12

Bottom-Up Parsing. Lecture 11-12 Bottom-Up Parsing Lecture 11-12 (From slides by G. Necula & R. Bodik) 9/22/06 Prof. Hilfinger CS164 Lecture 11 1 Bottom-Up Parsing Bottom-up parsing is more general than topdown parsing And just as efficient

More information

Decision, Computation and Language

Decision, Computation and Language Decision, Computation and Language Regular Expressions Dr. Muhammad S Khan (mskhan@liv.ac.uk) Ashton Building, Room G22 http://www.csc.liv.ac.uk/~khan/comp218 Regular expressions M S Khan (Univ. of Liverpool)

More information

10/4/18. Lexical and Syntactic Analysis. Lexical and Syntax Analysis. Tokenizing Source. Scanner. Reasons to Separate Lexical and Syntactic Analysis

10/4/18. Lexical and Syntactic Analysis. Lexical and Syntax Analysis. Tokenizing Source. Scanner. Reasons to Separate Lexical and Syntactic Analysis Lexical and Syntactic Analysis Lexical and Syntax Analysis In Text: Chapter 4 Two steps to discover the syntactic structure of a program Lexical analysis (Scanner): to read the input characters and output

More information