Engineering. discovery. Discovery Engineering REVIEWS COMPUTER ENGINEERING. Lexical analysis - a brief study

Size: px

Start display at page:

Download "Engineering. discovery. Discovery Engineering REVIEWS COMPUTER ENGINEERING. Lexical analysis - a brief study"

Cornelius Welch
5 years ago
Views:

REVIEWS COMPUTER ENGINEERING Discovery Engineering, Volume 2, Number 7, October 2013 ISSN 2320 6675 EISSN 2320 6853 discovery Engineering Lexical analysis - a brief study Harish R, Shahbaz Ali Khan,

1 REVIEWS COMPUTER ENGINEERING Discovery Engineering, Volume 2, Number 7, October 2013 ISSN EISSN discovery Engineering Lexical analysis - a brief study Harish R, Shahbaz Ali Khan, Shokat Ali, Rajat, Vaibhav Jain, Nitish Raj CSE department, Dronacharya College of engineering, Gurgaon, Haryana-06, India Received 13 August; accepted 21 September; published online 01 October; printed 16 October 2013 ABSTRACT The intention of this paper is to provide an overview on the subject of compiler design. The overview includes previous and existing concepts, current technologies. This paper also covers definition, design, overview, and advantages of compiler and its different parts. Through this paper we are creating awareness among the people about this rising field of compiler design. This paper also offers a comprehensive number of references for each concept in Lexical Analysis. Keywords: lexemes, scanner, lexer, parser. To Cite This Article Harish R, Shahbaz Ali Khan, Shokat Ali, Rajat, Vaibhav Jain, Nitish Raj. Lexical analysis - a brief study. Discovery Engineering, 2013, 2(7), INTRODUCTION The first phase of compilation Also known as lexer, scanner Takes a stream of characters and Returns tokens (words) Each token has a type and an optional value Called by the parser each time a new token is needed Typical tokens of programming languages Reserved words: class, int, char, bool, Identifiers: abc, def, mmm, mine, Constant numbers: 123, , 1.2E3 Operators and separators: (, ), <, <=, +, -, 1.2. Goal recognize token classes, report error if a string does not match any class 2. LEXICAL ANALYSIS Lexical analysis or scanning is the process where the stream of characters making up the source program is read from left-to-right and grouped into tokens. Tokens are sequences of characters with a collective meaning. There are usually only a small number of tokens for a programming language: constants (integer, double, char, string, etc.), operators (arithmetic, relational, logical), punctuation, and reserved words. The lexical analyzer takes a source program as input, and produces a stream of tokens as output. The lexical analyzer might recognize particular instances of tokens such as: 3 or 255 for an integer constant token "Fred" or "Wilma" for a string constant token numtickets or queue for a variable token Such specific instances are called lexemes. A lexeme is the actual character sequence forming a token, the token is the general class that a lexeme belongs to. Some tokens have exactly one lexeme (e.g., the > character); for others, there are many lexemes (e.g., integer constants). The scanner is tasked with determining that the input stream can be divided into valid symbols in the source language, but has no smarts about which token should come where. Few errors can be detected at the lexical level alone because the scanner has a localized view of the source program without any context. The scanner can report about characters that are not valid tokens (e.g., an illegal or unrecognized symbol) and a few other malformed entities (illegal characters within a string constant, unterminated comments, etc.) It does not look for or detect garbled sequences, tokens out of place, undeclared identifiers, misspelled keywords, mismatched types and the like. For example, the following input will not generate any errors in the lexical analysis phase, because the scanner has no concept of the appropriate arrangement Page30

Compilation: The action or process of producing something of tokens for declaration. The syntax analyzer will catch this error later in the next phase.

In the above sequence, it returns b, [, 2, and] as four separate tokens, having no idea they collectively form an array access.

compilation (although often these are handled by some sort of preprocessor which filters the input before the compiler runs). 2.1.

2 Compilation: The action or process of producing something of tokens for declaration. The syntax analyzer will catch this error later in the next phase. int a double } switch b[2] =; Furthermore, the scanner has no idea how tokens are grouped. In the above sequence, it returns b, [, 2, and] as four separate tokens, having no idea they collectively form an array access. The lexical analyzer can be a convenient place to carry out some other chores like stripping out comments and white space between tokens and perhaps even some features like macros and conditional compilation (although often these are handled by some sort of preprocessor which filters the input before the compiler runs) Task A token is a string of one or more characters that is significant as a group. The process of forming tokens from an input stream of characters is called tokenization. Tokens are identified based on the specific rules of the lexer. Some methods used to identify tokens include: regular expressions, specific sequences of characters known as a flag, specific separating characters called delimiters, and explicit definition by a dictionary. Special characters, including punctuation characters, are commonly used by lexers to identify tokens because of their natural use in written and programming languages. Tokens are often categorized by character content or by context within the data stream. Categories are defined by the rules of the lexer. Categories often involve grammar elements of the language used in the data stream. Programming languages often categorize tokens as identifiers, operators, grouping symbols, or by data type. Written languages commonly categorize tokens as nouns, verbs, adjectives, or punctuation. Categories are used for Comparison: It is an act of assessment or evaluation of things side by side in order to see to what extent they are similar or different. It is used to bring out similarities or diffrences between two things of same type mostly to discover essential features or meaning either scientifically or otherwise. Content: The amount of things contained in something. Things written or spoken in a book, an article, a programme, a speech, etc. Also known as tokenizer or scanner. In Spanish, called analizadormorfologico Purpose: translation of the source code into a sequence of symbols. The symbols identified by the morphological analyzer will be considered terminal symbols in the grammar used by the syntactic analyzer Other Tasks Identification of lexical errors, e.g., starting an identifier with a digit where the language does not allow this: 2abc Deletion of white-space: Usually, the function of white-space is only to separate tokens. Exceptions: languages where whitespace indicates code block, e.g.,python: if 1 == 2: print 1 print 2 Deletion of comments: not relevant to execution of program. 3. TOKENS post-processing of the tokens either by the parser or by other functions in the program. A lexical analyzer generally does nothing with combinations of tokens, a task left for a Page31

$parser. For example, a typical lexical analyzer recognizes parentheses as tokens, but does nothing to ensure that each "(" is matched with a ")"). delimiter (i.e. matching the string " " or regular expression /\s{1}/).$

3 parser. For example, a typical lexical analyzer recognizes parentheses as tokens, but does nothing to ensure that each "(" is matched with a ")"). delimiter (i.e. matching the string " " or regular expression /\s{1}/). The tokens could be represented in XML, Demarcating: Set the boundaries or limits of Consider this expression in the C programming language: sum = 3 + 2; Tokenized and represented by the following table: 4. LEXEMES AND PATTERNS Lexeme: A specific instance of a token. Used to differentiate tokens. For instance, both position and initial belong to the identifier class, however each a different lexeme. Lexical analyzer may return a token type to the Parser, but must also keep track of attributes that distinguish one lexeme from another. Examples of attributes: Identifiers: string Numbers: value Attributes are used during semantic checking and code generation. They are not needed during parsing. A lexeme, however, is only a string of characters known to be of a certain kind (e.g., a string literal, a sequence of letters). In order to construct a token, the lexical analyzer needs a second stage, the evaluator, which goes over the characters of the lexeme to produce a value. The lexeme's type combined with its value is what properly constitutes a token, which can be given to a parser. (Some tokens such as parentheses do not really have values, and so the evaluator function for these can return nothing. The evaluators for integers, identifiers, and strings can be considerably more complex. Sometimes evaluators can suppress a lexeme entirely, concealing it from the parser, which is useful for whitespace and comments.) Patterns: Rule describing how tokens are specified in a program. Needed because a language can contain infinite possible strings. They all cannot be enumerated. Formal mechanisms used to represent these patterns. Formalism helps in describing precisely 1. Which strings belong to the language, 2. Which do not? Also, form basis for developing tools that can automatically determine if a string belongs to a language. 5. SCANNERS The first stage, the scanner, is usually based on a finitestate machine (FSM). It has encoded within it information on the possible sequences of characters that can be contained within any of the tokens it handles (individual instances of these character sequences are known as lexemes). For instance, an integer token may contain any sequence of numerical digit characters. In many cases, the first nonwhitespace character can be used to deduce the kind of token that follows and subsequent input characters are then processed one at a time until reaching a character that is not in the set of characters acceptable for that token (this is known as the maximal munch rule, or longest match rule). In some languages, the lexeme creation rules are more complicated and may involve backtracking over previously read characters. For example, in C, a single 'L' character is not enough to distinguish between an identifier that begins with 'L' and a wide-character string literals. 6. TOKENIZTION Tokenization is the process of demarcating and possibly classifying sections of a string of input characters. The resulting tokens are then passed on to some other form of processing. The process can be considered a sub-task of parsing input. Take, for example, The quick brown fox jumps over the lazy dog The string isn't implicitly segmented on spaces, as an English speaker would do. The raw input, the 43 characters, must be explicitly split into the 9 tokens with a given space s-expression 7. LEXICAL GENERATOR Lexical analysis can often be performed in a single pass if reading is done a character at a time. Single-pass lexers can be generated by tools such as flex. The lex/flex family of generators uses a table-driven approach which is much less efficient than the directly coded approach. With the latter approach the generator produces an engine that directly jumps to follow-up states via goto statements. Tools like re2c and Quex have proven (e.g. RE2C - A More Versatile Scanner Generator (1994) to produce engines that are between two to three times faster than flex produced engines. It is in general difficult to hand-write analyzers that perform better than engines generated by these latter tools. The simple utility of using a scanner generator should not be discounted, especially in the developmental phase, when a language specification might change daily. The ability to express lexical constructs as regular expressions facilitates the description of a lexical analyzer. Some tools offer the specification of pre- and post-conditions which are hard to program by hand. In that case, using a scanner generator may save a lot of development time, at least where extreme performance optimality is not a concern. 8. LEXICAL ANALYZER GENERATOR ANTLR - Can generate lexical analyzers and parsers. Flex - Alternative variant of the classic "lex" (C/C++). JFlex - A rewrite of JLex. Ragel - A state machine and lexical scanner generator with output support for C, C++, C#, Objective-C, D, Java, Go and Ruby source code. The following lexical analysers can handle Unicode: JavaCC - JavaCC generates lexical analyzers written in Java. JLex - A lexical analyzer generator for Java. Quex (or "Queχ") - A fast universal lexical analyzer generator for C and C++ 9. ADVANTAGES AND ROLES OF LEXICAL ANALYZER Primary role: Scan a source program (a string) and break it up into small, meaningful units, called tokens. Example: position := initial + rate * 60; Page32

4 Transform into meaningful units: identifiers, constants, operators, and punctuation. Other roles: o Removal of comments o Case conversion o Removal of white spaces o Interpretation of compiler directives or pragmas: For instance, in Turbo Pascal {$R+ means range checking is enabled. o Communication with symbol table: Store information regarding an identifier in the symbol table. Not advisable in cases where scopes can be nested. o Preparation of output listing: Keep track of source program, line numbers, and correspondences between error messages and line numbers. Why separate LA from parser? o Simpler design of both LA and parser o More efficient compiler o More portable compiler Role of lexical analyzer in the compiler Lexical and syntax analysis of program Syntactical: Conforming to the rules of syntax Lexical analysis 10. LEXICAL ERRORS Scanner may come across certain errors: invalid character, invalid token etc. Usually detected by reaching a state that is not final and there are no transitions for the current input symbol. Cannot uncover syntactical, semantic or logical errors: view of the lexical analyzer is localized. What to do when lexical errors occur? o Delete the characters read so far and restart scanning at the next unread character. o Delete the first character read by the scanner and resume scanning at the character following it. o Local transformations: replace a char by another, transpose adjacent chars etc. Note that error recovery at this stage may create errors in the parsing stage: for instance, replace beg#in by beg in which will cause error during the parsing phase. Other approach could be for the scanner to provide a certain warning token to the parser. Parser can use this information to do syntactic error-repair. Error recovery of a common problem: runaway comments and strings. Possible solutions: Introduce error token that represent a runaway string or comment. Once the runaway error token is recognized, a special error message may be issued. 11. CERTAIN DIAGRAMS FOR LEXICAL ANALYSIS 12. SUMMARY 1. In computer science, lexical analysis is the process of converting a sequence of characters into a sequence of tokens. 2. A program or function that performs lexical analysis is called a lexical analyzer, lexer, or scanner. A lexer often exists as a single function which is called by a parsers or another function, or can be combined with the parser in scanner less parsing. Page33

5 DISCLOSURE STATEMENT There is no financial support for this research work from the funding agency. ACKNOWLEDGMENTS We thank our guide for his timely help, giving outstanding ideas and encouragement to finish this research work successfully. REFERENCE 1. Compiling with C# and Java, Pat Terry, 2005, ISBN X 2. Algorithms + Data Structures = Programs, Niklaus Wirth, 1975, ISBN Compiler Construction, Niklaus Wirth, 1996, ISBN Sebesta, R. W. (2006). Concepts of programming languages (Seventh edition) pp Boston: Pearson/Addison-Wesley. RELATED RESOURCES 1. Aho, R. Sethi, J. Ullman, Compilers: Principles, Techniques, and Tools. Reading, MA: Addison-Wesley, J. Backus, The History of FORTRAN I, II, III., SIGPLAN Notices, Vol. 13, no. 8, August, 1978, pp J.P. Bennett, Introduction to Compiling Techniques. Berkshire, England: McGraw-Hill, D. Cohen, Introduction to Computer Theory, New York: Wiley, Fischer, R. LeBlanc, Crafting a Compiler. Menlo Park, CA: Benjamin/Cummings, J. Hopcroft, J. Ullman, Introduction to Automata Theory, Languages, and Computation, Reading MA: Addison- Wesley, S. Kleene, "Representation of Events in Nerve Nets and Finite Automata," in C. Shannon and J. McCarthy (eds), Automata Studies, Princeton, NJ: Princeton University Press, McGettrick, The Definition of Programming Languages. Cambridge: Cambridge University Press, T. Sudkamp, Languages and Machines: An Introduction to the Theory of Computer Science, Reading, MA: Addison- Wesley, R.L.Wexelblat, History of Programming Languages. London: Academic Press, Page34

Introduction to Lexical Analysis

Introduction to Lexical Analysis Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexical analyzers (lexers) Regular