UNIT II LEXICAL ANALYSIS

Size: px

Start display at page:

Download "UNIT II LEXICAL ANALYSIS"

Eustacia Harris
6 years ago
Views:

1 UNIT II LEXICAL ANALYSIS 2 Marks 1. What are the issues in lexical analysis? Simpler design Compiler efficiency is improved Compiler portability is enhanced. 2. Define patterns/lexeme/tokens? This set of strings described by a rule called pattern associated with the token. A lexeme is a sequence of characters in the source program that is matched by the pattern for a token. Token is a sequence of character that can be treated as a single logical entity. 3. Give the algebraic properties of regular expression? AXIOM DESCRIPTION i) r/s = s/r / is commutative ii) r/(s/t)=(r/s)/t / is associative iii) (rs)t=r(st) concatenation is associative iv) r(s/t)=rs/rt concatenation distributes over / v) r**=r* * is idempotent 4. What are the systems referred to data flow engine? i) Compiler-compilers ii) Compiler-generators iii) Translator writing systems. 5. Give the parts of a string? Prefix of s, suffix of s, substring of s, proper prefix, proper suffix, proper substring and subsequence of s. 6. What are the operations on language? Union Concatenation Kleene closure or star closure and Star closure. 7. Give the error recovery actions in lexical errors? i) Deleting an extraneous character ii) Inserting a missing character iii) Replacing an incorrect character by a correct character. 8. What are the implementations of lexical analyzer? a) Use a lexical analyzer generator, such as Lex compiler, to produce the

2 the lexical analyzer from a regular expression based specification b) Write the lexical analyzer in a conventional systems-programming language using the I/O facilities of that language to read the input. c) Write the lexical analyzer in assembly language and explicitly manage reading of input. 9. Define regular expression? It is built up out of simpler regular expression using a set of defining rules. Each regular expression r denotes a language L(r). The defining rules specify how L(r) is formed by combining in various ways the languages denoted by the sub expressions of r. 10. Give the precedence of regular expression operator? i) The unary operator * has the highest precedence and is left associative. ii) Concatenation has the second highest precedence and is left associative. iii) / has the lowest precedence and is left associative. 11. Define the length of a string? It is the number of occurrences of symbols in string, s denoted by s. Example: s=abc, s = Give the rules in regular expression? 1) is a regular expression that denotes { }, that is the set containing the empty string. 2) If a is a symbol in _, then a is a regular expression that denoted {a}, i.e., the set containing the string a. 3) Suppose r and s are regular expression denoting the languages L( r) and L(s). 13. Define regular set? A language denoted by a regular expression is said to be a regular set. 14. Give the types of notational shorthand s? Zero or more instance One or more instance Character classes 15. Write short notes on buffer pair. Concerns with efficiency issues Used with a lookahead on the input

3 It is a specialized buffering technique used to reduce the overhead required to process an input character. Buffer is divided into two N-character halves. Use two pointers. Used at times when the lexical analyzer needs to look ahead several characters beyond the lexeme for a pattern before a match is announced. 16. What are the error recovery actions in a lexical analyzer? Deleting an extraneous character, inserting a missing character, replacing an incorrect character by a correct character, transposing two adjacent characters. 17. What is the input to the lexical analyzer generator? What is its output? 18. What is the function of a scanner? The scanner scans the source program and separates the tokens. 19. Write a short note on LEX. A LEX source program is a specification of lexical analyzer consisting of set of regular expressions together with an action for each regular expression. The action is a piece of code, which is to be executed whenever a token specified by the corresponding regular expression is recognized. The output of a LEX is a lexical analyzer program constructed from the LEX source specification. 20. What are the drawbacks of using buffer pairs? i. This buffering scheme works quite well most of the time but with it amount of lookahead is limited. ii. Limited lookahead makes it impossible to recognize tokens in situations where the distance, forward pointer must travel is more than the length of buffer. 21. In a string of length n, how many of the following are there? a. Prefixes b. Suffixes c. Proper prefixes a) n+1 prefixes

4 Eg: ban Prefixes = {ε, ban,ba,b} b) n+1 suffixes Eg: bana Suffixes = { ε, bana,ana,na,a} c) n-1 proper prefixes Eg: banan Proper prefixes = {bana,ban,ba,b} 22. Describe the languages denoted by the following regular expression: 0(0/1)*0 The set pf all strings of 0 s and 1 s starting and ending with Write regular expressions for the following language. The set of strings over {a,b,c} that contain exactly one b. (a/c)*b(a/c)* 24. Write the regular definition for the following language. Pascal identifier. Letter [a-z A-Z] Digit [0-9] id Letter(Letter Digit}* White Space delimiter blank tab newline ws delim Write regular expression for the following. a) if b) relational operator c) Keyword if i.f relational operator < <= > >= = <> Keyword IF ELSE BEGIN END 16 Marks 1. Identify the lexemes that make up the tokens in the following C program. int max(i,j) int i,j; /* return maximum of integers i and j */ {return i>j:?i:j; }

5 Solution: LEXEME TOKEN ATTRIBUTE VALUES int int - max id Ptr to table entry ( ( - i id Ptr to table entry ) ) - j id Ptr to table entry ) ) - { { - return return - > rel-op GT?? - : : - ; ; - } } - 2. Identify the lexemes that make up the tokens in the following FORTRAN program. E = M * C **2 Solution: LEXEME TOKEN ATTRIBUTE VALUES E id Ptr to table entry = assign-op - M id Ptr to table entry * mul - C id Ptr to table entry ** exp-op - 2 num 2 3. Construct the NFA from the (a b)*a(a b) using Thompson.s construction algorithm. The algorithm is syntax directed in that it uses the syntactic structure of the regular expression to guide the construction process. First, parse the regular expression r into its constituent sub expressions. Then using various rules, construct NFA.s for each of the basic symbols in r.

7 4. Explain about Input buffering technique. Determining the next lexeme requires reading the input beyond the end of the lexeme. Buffer Pairs: Concerns with efficiency issues Used with a look ahead on the input It is a specialized buffering technique used to reduce the overhead required to process an input character. Buffer is divided into two N-character halves. Use two pointers. Used at times when the lexical analyzer needs to look ahead several characters beyond the lexeme for a pattern before a match is announced. One pointer called forward pointer, points to first character of the next lexeme found. The string of characters between two forms the lexeme.

8 Increment procedure for forward pointer: (2) If forward at end of first half then reload second half forward+=1 else if forward at end of second half reload the first half move forward to beginning of first half else forward+=1 Sentinels: It is the special character which cannot be a part of source program. It is used to reduce the two tests into one. e.g. eof Increment procedure for forward pointer using sentinels: forward+=1 if forward =eof then If forward at end of first half then reload second half forward+=1 else if forward at end of second half reload the first half move forward to beginning of first half else terminate lexical analysis 5. Explain specification of tokens. Regular expressions are the notations for specifying the patterns. Each pattern matches a set of strings Strings and languages: An alphabet is a finite set of symbols. A string over an alphabet is a finite sequence of symbols from the alphabet. Terms for parts of a string: Prefix, Suffix, Substring, Proper prefix and proper suffix Language: It is a set of strings over some fixed alphabet. Operations on languages: Concatenation Union Kleene closure Positive closure Regular expressions: ε is a regular expression that denotes { ε } if a is a symbol in, then a is a regular expression that denotes {a} Suppose r and s are regular expressions denoting the languages L(r) and L(s). Then, (r) (s) is a regular expression denoting L(r) U L(s) (r) (s) is a regular expression denoting L(r) L(s)

9 (r)* is a regular expression denoting L(r)* (r) is a regular expression denoting L(r) A language denoted by a regular expression is said to be a regular set. Unary operator * has the highest precedence and is left associative Concatenation has the second highest precedence and is left associative has lowest precedence and is left associative Regular definitions: It is a sequence of definitions of the form d1->r1, d2->r2. dn->rn Where each di is a distinct name and each ri is a regular expression over the symbols in U {d1, d2,.. di-1} 6. Give the minimized DFA for the following expression (a b)*abb. Syntax tree for (a b)*abb#: Calculation of firstpos, lastpos and nullable for nodes in syntax tree:

10 Calculation of followpos: Node followpos 1 {1, 2, 3} 2 {1, 2, 3} 3 {4} 4 {5} 5 {6} Now, the start state of DFA is firstpos of the root So, A= {1, 2, 3} Consider the input symbol ëaí: Position 1 and 3 are for ëaí in A So, let B = followpos(1) U followpos(3) = {1, 2, 3} U {4} = {1, 2, 3, 4} DTrans[A, a] = B Consider the input symbol ëbí: Position 2 is for ëbí in A So, let B = followpos(2) = {1, 2, 3} = A DTrans[A, b] = A Now continue with B, Consider the input symbol ëaí: Position 1 and 3 are for ëaí in A So, followpos(1) U followpos(3) = {1, 2, 3} U {4} = {1, 2, 3, 4} = B DTrans[B, a] = B Consider the input symbol ëbí: Position 2 and 4 are for ëbí in B So, followpos(2) U followpos(4) = {1, 2, 3, 4, 5} = C DTrans[B, b] = C Now continue with C, Consider the input symbol ëaí: Position 1 and 3 are for ëaí in A So, followpos(1) U followpos(3)

11 = {1, 2, 3} U {4} = {1, 2, 3, 4} = B DTrans[C, a] = B Consider the input symbol ëbí: Position 2 and 5 are for ëbí in C So, followpos(2) U followpos(5) = {1, 2, 3, 6} = D DTrans[C, b] = D Now continue with D, Consider the input symbol ëaí: Position 1 and 3 are for ëaí in D So, followpos(1) U followpos(3) = {1, 2, 3} U {4} = {1, 2, 3, 4} = B DTrans[D, a] = B Consider the input symbol ëbí: Position 2 is for ëbí in D So, followpos(2) = {1, 2, 3} = A DTrans[D, b] = A The position associated with the end marker #, 6 is in D. So, D is the final state. DFA 7. Draw the transition diagram for unsigned numbers

12 8. For the regular expression, (a / b)*abb. Draw the NFA. Obtain DFA from NFA. Minimize DFA using new construction. Hints: NFA ε-closure computation DFA diagram Minimization steps Minimized DFA 9. Construct a NFA using Thompsonís construction algorithm for the regular expression (a b)*abb(a b)* and convert it into DFA.

13 The start state of DFA is Â-closure (0) Â-closure (0) = {0, 1, 2, 4, 7} = A The input symbol alphabet is {a, b} Â-closure (move(a, a)) = Â-closure (move({0,1, 2, 4, 7}, a)) = Â-closure (3, 8) = {1, 2, 3, 4, 6, 7, 8} = B DTrans [A, a] = B Â-closure (move(a, b)) = Â-closure (move({0,1, 2, 4, 7}, b)) = Â-closure (5) = {1, 2, 4, 5, 6, 7} = C DTrans [A, b] = C Â-closure (move(b, a)) = Â-closure (move({1, 2, 3, 4, 6, 7, 8}, a)) = Â-closure (3, 8) = {1, 2, 3, 4, 6, 7, 8} = B DTrans [B, a] = B Â-closure (move(b, b)) = Â-closure (move({1, 2, 3, 4, 6, 7, 8}, b)) = Â-closure (5, 9) = {1, 2, 4, 5, 6, 7, 9} = C DTrans [B, b] = D Â-closure (move(c, a)) = Â-closure (move({1, 2, 4, 5, 6, 7, 8}, a)) = Â-closure (3, 8) = B DTrans [C, a] = B Â-closure (move(c, b)) = Â-closure (move({1, 2, 4, 5, 6, 7, 8}, b)) = Â-closure (5) = C DTrans [C, b] = C Â-closure (move(d, a)) = Â-closure (move({1, 2, 4, 5, 6, 7, 9}, a)) = Â-closure (3, 8) = B DTrans [D, a] = B Â-closure (move(d, b)) = Â-closure (move({1, 2, 4, 5, 6, 7, 9}, b)) = Â-closure (5, 10) = {1, 2, 4, 5, 6, 7, 10, 11, 12, 14, 17} = E DTrans [D, b] = E Â-closure (move(e, a)) = Â-closure (move({1, 2, 4, 5, 6, 7, 10, 11, 12, 14, 17}, a)) = Â- closure (3, 8, 13) = {1, 2, 3, 4, 6, 7, 8, 11, 12, 13, 14, 16, 17} = F DTrans [E, a] = F Â-closure (move(e, b)) = Â-closure (move({1, 2, 4, 5, 6, 7, 10, 11, 12, 14, 17}, b)) = Â- closure (5, 15) = {1, 2, 4, 5, 6, 7, 11, 12, 14, 15, 16, 17} = G

14 DTrans [E, b] = G Â-closure (move(f, a)) =Â-closure (3, 8, 13) = F DTrans [F, a] = F Â-closure (move(f, b)) =Â-closure (5, 9, 15) ={1, 2, 4, 5, 6, 7, 9, 11, 12, 14, 15, 16, 17}= H DTrans [F, b] = H Â-closure (move(g, a)) =Â-closure (3, 8, 13) = F DTrans [G, a] = F Â-closure (move(g, b)) =Â-closure (5, 15) = G DTrans [G, b] = G Â-closure (move(h, a)) =Â-closure (3, 8, 13) = F DTrans [H, a] = F Â-closure (move(h, b)) =Â-closure (5, 10, 15) ={1, 2, 4, 5, 6, 7, 10, 11, 12, 14, 15, 16, 17} = I DTrans [H, b] = I Â-closure (move(i, a)) =Â-closure (3, 8, 13) = F DTrans [I, a] = F Â-closure (move(i, b)) =Â-closure (5, 15) = G DTrans [I, b] = G

COMP-421 Compiler Design. Presented by Dr Ioanna Dionysiou

COMP-421 Compiler Design. Presented by Dr Ioanna Dionysiou COMP-421 Compiler Design Presented by Dr Ioanna Dionysiou Administrative! [ALSU03] Chapter 3 - Lexical Analysis Sections 3.1-3.4, 3.6-3.7! Reading for next time [ALSU03] Chapter 3 Copyright (c) 2010 Ioanna