CSC 306 LEXICAL ANALYSIS PROJECT SPRING 2011

Transcription

1 CSC 306 LEXICAL ANALYSIS PROJECT SPRING 2011 PROF. GODFREY C. MUGANDA DEPARTMENT OF COMPUTER SCIENCE In this assignment, we take the first step in writing a compiler for the Imp 2011 programming language, and construct a lexical analyzer Imp. The lexical analyzer will take an Imp source file as input, and output a stream of tokens making up the the Imp program in the file. In addition, the lexical analyzer strips out comments, making them invisible to subsequent phases of the compiler. Comments start with // and end at the end of the line. Identifier tokens consist of a sequence of alphanumeric characters and underscores. Identifiers may not start with a digit and are case sensitive. Number tokens are sequences of digits. Strings tokens consist of a sequence of characters delimited by double quotes. Each token is formed by the longest sequence of characters that obey the rules for the formation of that token. Here is a table of Imp reserved words: and begin boolean break call end else false function halt if input integer is loop not null newline or output procedure return then true var The rest of the tokens are the punctuation symbols, : ( ) ; the relational operators < <= > >= == # and the arithmetic operators + - * / % The job of the lexical analyzer being to break source code into a sequence of tokens, the output of your program will consist of lexeme - token pairs (the token is really an internal representation, while the lexeme is the string that actually appears in the source program.) For example, when the source file 1

2 2 PROF. GODFREY C. MUGANDA DEPARTMENT OF COMPUTER SCIENCE procedure main( ) begin output("hello World"); end main; is fed to the lexical analyzer as input, the output will be procedure : procedure t_id : : : ) begin : begin output : : ( t_string : Hello : : ; end : end t_id : : ; Notice that the printing format of a token is the internal name of the token prefixed with character. The exceptions to this rule are the reserved keywords, number tokens, and string tokens. The printing formats of these is the internal name of the token prefixed with a t_. This will make more sense as you continue to read. Following is an outline of the design for the lexical analyzer. Further information and hints to be given in class. 1. The Lexer.h file The lexical analyzer will consist of a single class called Lexer. C++ allows you to put the declaration (specification) of a class in a header file (Lexer.h), and put the definitions of the class methods in a separate implementation file (Lexer.cpp). Typically, header file contain type declarations: class declarations and /or enumeration types. A header file should never contain variable or function definitions. #ifndef LEXER_H #define LEXER_H #include <iostream> #include <fstream> #include <sstream> #include <cstdlib> #include <string> #include <vector> #include <map> using namespace std; enum TokenType t_and, t_begin, t_boolean, t_break,

3 CSC 306 LEXICAL ANALYSIS PROJECT SPRING t_call, t_end, t_else, t_elsif, t_false, t_function, t_halt, t_if, i_input, i_integer, t_is, t_loop, t_not, t_null, t_newline, t_or, t_output, t_procedure, t_return, t_then, t_true, t_var, t_comma, t_colon, t_lparen, t_rparen, t_semi, t_lt, t_le, t_gt, t_ge, t_eq, t_ne, t_plus, t_minus, t_mult, t_div, t_mod, t_assign, t_error, t_id, t_number, t_string, t_eof, ; class Lexer public: Lexer(istream& source, ostream &listing); TokenType gettoken(); string getlexeme()return lexeme; string tokentostring(tokentype t) return tokentostringvector[t]; private: char getchar(); // source and listing files istream& source; ostream& listing; string lexeme; // vector maps TokenType to string vector<string> tokentostringvector; // maps identifier strings identifier tokens map<string, TokenType> lexemetokenmap; ; #endif /* LEXER_H */ Note that the class declaration is enclosed inside the preprocessor directives #ifndef LEXER_H #define LEXER_H... #endif These tells the preprocessor to only include this header file if the simple LEXER_H is not already defined. In this case the symbol is defined by the second line and the file is included. Here is a description of the public members of the Lexer class: Lexer(istream& source, ostream &listing);

4 4 PROF. GODFREY C. MUGANDA DEPARTMENT OF COMPUTER SCIENCE This is the constructor. It takes two streams as parameters. The source contains the source program. The listing stream will hold a listing of the source program, interspersed with compiler messages pointing out the the location of syntax errors. TokenType gettoken(); This function returns the next token from the source file each time it is called. As a side effect it sets the lexeme variable. It skips comments and whitespace. string getlexeme()return lexeme; This function returns the lexeme that was stored by the last call to gettoken(). string tokentostring(tokentype t) return tokentostringvector[t]; This function returns the string associated with a particular token. The private members of the class are as follows: char getchar(); This function will be called to return the next character from the source file. This function will also be in charge of copying lines from the source stream to the listing stream. This method will be discussed in class. istream& source; ostream& listing; These variables keep track of the source and listing streams. They must be reference variables to work correctly, and must be initialized by the Lexer constructor in a very particular way. More on this later. string lexeme; This variable will have its value set by the gettoken() method during each call. vector<string> tokentostringvector; This vector associates TokenType values with string objects. For example, the token t id is associated with the id. map<string, TokenType> lexemetokenmap; This map is the opposite of the tokentostringvector. It associates strings with the appropriate token. For example var is associated with t var. 2. The Lexer.cpp file This file consists of 3 functions, two of which you will need to write. #include "Lexer.h" #include <cctype> Lexer::Lexer(istream &source1, ostream& listing1)

5 CSC 306 LEXICAL ANALYSIS PROJECT SPRING :source(source1), listing(listing1) TokenType Lexer::getToken() char Lexer::getChar() static string currentline = ""; static istringstream lineinput(""); char ch = lineinput.get(); while (ch == EOF) //lineinput needs to be replenished from the source lineinput.clear(); // need to clear flags on lineinput stream getline(source, currentline); // restore new line discarded by getline currentline += \n ; if (!source) return EOF; lineinput.str(currentline); listing << currentline; ch = lineinput.get(); return ch; The constructor 3. The Constructor Lexer::Lexer(istream &source1, ostream& listing1) :source(source1), listing(listing1) must initialize all the member variables of the class so the lexical analyzer will work correctly. As previously mentioned, the source and listing variables must be reference variables, and must be initialized in a constructor initialization list that precedes the body of the constructor function. This is already done in the code fragment above. The body of the constructor needs to initialize the two variables // vector maps TokenType to string

6 6 PROF. GODFREY C. MUGANDA DEPARTMENT OF COMPUTER SCIENCE vector<string> tokentostringvector; // maps identifier strings identifier tokens map<string, TokenType> lexemetokenmap; Hints on how to do this will be given in lecture. 4. The gettoken function This function is the heart of the lexical analyzer. Basically, it just needs to make sure that it returns the next token from the source file whenever it is called. How to do this will be outlined in class. Basically gettoken() will make calls to getchar() each time it needs a new character. It will skip white space between tokens, and will always look to assemble the longest stream of characters that can form a token. There are two basic rules. The first rule is that the first character of a token always tells what kind of token it is. This rule works pretty well, but you have to finesse things a little from time to time, because a few characters can begin more than one token. The second rule is this. Before you call gettoken(), make sure the current character in the source stream is either the first character of the next token to be returned, or is white space preceding such a character. This is the precondition for calling gettoken(). There is also a postcondition: Just before returning from gettoken(), make sure that the current character is the character just after the token that is to be returned. The next page has a main.cpp file you can use to test your code. Notice the clever use istringstream to create an inmemory source stream.

7 CSC 306 LEXICAL ANALYSIS PROJECT SPRING Test Harness Use the following file to test your lexical analyzer. /* * File: main.cpp * Author: gcm * * Created on April 7, 2012, 8:06 PM */ #include <cstdlib> #include "Lexer.h" #include <sstream> #include <fstream> #include <iostream> using namespace std; int main(int argc, char** argv) istringstream in("x x and begin boolean break call end else false\nbegin " "function halt if input integer is loop not null\n " "newline or output procedure return then true var\n" "123,:()< <= > >= > = == # + - * / %!<\n" " \"This is a string token\" mom is nice\n " "so is dad //but little sis is naughty x\n " "y\n" "x \n" " bye " ); Lexer lex(in, cerr); TokenType token = lex.gettoken(); while (token!= t_eof) cerr << lex.tokentostring(token) << ":" << lex.getlexeme() << endl; token = lex.gettoken(); cerr << endl << lex.tokentostring(token); cerr << endl; return 0;