Assignment 1: Lexer and tokenout utility

Transcription

1 Assignment 1: Lexer and tokenout utility Worth 10% of the Project grade Due: Tuesday, September 10th at 11:59 pm (just before Midnight) Overview Over the semester, you will be building pieces of a compiler for a simple C like language called C-like, (also known as clike, especially from the command line). C-like will not implement all of C, but much of C to gain an understanding of what is needed to put together a full implementation. This assignment contains most of the first part of the five piece project which makes up the semester long compiler project. To make this assignment at least nominally useful, we will be using the lexer (which you will be implementing in this assignment) as a tool to massage the input to remove certain kinds of input. As a reminder: each assignment in this class depends on the previous assignment. In other words, you will be reusing much of this code in your next assignment, so get it right NOW or you will be making more work for your future self. This Assigment This is the beginnings of a lexical processor for clike. Knowing how to write just a lexer can be useful for tranformations. In this portion, we implement the lexer for C-like and use it to do something moderately useful. We will implementing something called tokenout using the C-like lexer and the flex Unix tool. You are required to use flex and C for this assignment. If you don't use C and flex, you will get a 0.: Recall: When using Makefiles, all dependencies are checked using timestamps on files. EXAMPLE: If a timestamp on a file ex1.c is newer than file ex1.o, and file ex1.o depends on file ex1.c, ex1.o must be rebuilt to pick up the potential changes in 'ex1.c'). The problem, especially with C, is that this can cause major rebuilds even if all that changed was a comment. For example, if only a comment or whitespace in a.h file changes, there is no reason to rebuild everything that depends on that.h file. (Why do we have to recompile anything that include a.h file if it changes? Discuss on piazza.com) We can use a lexical stream of tokens to check whether or not the file has changed or JUST the comments/whitespace has changed. By outputting just the tokens of a file and no whitespace or comments, we can compare (say) an original file from a source code repository to the changed file and see if we need to rebuild. (This example may sound contrived, but is a technique that has been used in workplaces to try to mitigate rebuilding of an extremely large codebase where code was installed and running.) What does tokenout do? It will output a stream of tokens, discarding whitespace and comments. We will (more or less) use the definitions of the tokens and reserved words that come from "C: A Reference Manual" (5th Edition) by Harbison and Steele, Chapter 2: "Lexical Elements". When in doubt, do what C99 would do. When in doubt on what's a token, defer to the Harbison Steele book. If that is unclear, defer to the testcases (see below). If they are unclear, ask us on piazza. In general, though, the testcases will be the last authority on what is correct. Definitions These definitions all come from Harbison and Steele. We strongly recommend you pick up a copy of the book for the class and for a reference in real life. We will use the following definition of whitespace (derived from Section 2.1: "Character Set"):

2 space ' ' horizontal tab (or tab) '\t' vertical tab '\v', formfeed '\f' newline '\n' and return '\r' (optional elements) For comments, we will use both C-style and C++-style. See Section 2.2: "Comments": /* start comment - C-style */ end comment // starts a comment, ends at end of line Recall that comments in C DO NOT nest (i.e., you can't have a comment inside of another comment). Do not worry about implementing #if comments. The definition of tokens is given on page 29, Section 2.3: "Tokens". This gives a good definition of what a token is in C. The operators and separators are given on page 21 in Section 2.4: "Operators and Separators". Don't worry about the "alternate token spellings" (i.e., completely ignore them). Identifiers are described in Section 2.5, starting on page 21. The reserved words (keywords) of the language are given on page 23 of the C: A Reference Manual (5th Edition), Section 2.6 "Keywords". Use ONLY the keywords in the table (Table 2-4) and DO NOT worry about asm, fortran or others listed just below the table. Disregard Section 2.6.1: "Predefined Identifiers". Constants are described in Section 2.7. There are 4 types of contants: integer, float, character, and string constants. When in doubt, refer to the Harbison-Steele book for questions of ambiguity and prefer C99. DO NOT worry about trigraphs, token respellings, checking constant limits (at least for now), wide strings, additional C++ reserved words, or implementing the preprocessor (#, #define, etc). The basic rules for tokenout: If whitespace is encountered, it is ignored (normally C replaces it with a space) If an identifier is encountered, it is replaced on output by id:name-of-identifier If a keyword is encountered, it is replaced on output by kw:name-of-keyword If a string constant is encountered, it is replaced on output by str:"the-string-constant" If an int constant is encountered, it is replaced on output by int:int-const If a floating point constant is encountered, it is replaced on output by float:float-const If a char constant is is encountered, it is replaced on output by char:'char'

3 Any other token comes out as-as A space is inserted between every token EXAMPLES: Example 1: Consider the following hello.c file: // Example 1 # include "stdio.h" int main() { printf("hello there!\n"); exit(1); } The tokenout utility would output: # id:include str:"stdio.h" kw:int id:main ( ) { id:printf ( str:"hello there!\n" ) ; id:exit ( int:1 ) ; } Note that the comment was completely discarded, and a single space separates each token. Surprisingly, printf is not a keyword in the language: it is a function call into the standard library, so it just a plain identifier. The strings and ints also annotate what is there. Example 2: Consider the following simple input file: 1 +++a The tokenout utility would output: int: id:a Why? C follows the "maximal munch rule" and grabs the biggest token it can at the start of the +++, which is 2 plusses. The following 3 preprocessing directives should all have the same output: # include "stdio.h" #include "stdio.h" #include"stdio.h" should all output: # id:include str:"stdio.h" In other words, # is a token, include is a token, and the constant "stdio.h" is a token. (In a real C implementation, strictly speaking, #include wouldn't be seen by the tokenizer; the macro preprocessor would have inserted the contents of the file "stdio.h" instead. For the purposes of this compiler, we will simply see include as an identifier for this first assignment.)

4 Errors There's a few cases where you will output an error to standard error. If a /* comment is unfinished (i.e., no matching */ before the end of the file), you should output an error to stderr with the line number. For an input of /*/ The output would be empty, and the stderr would report something like: 2, Unterminated Comment Where the 2 is the line number that the end of file would be on. If you input a token that is unrecognized by the lexer, then you should report on stderr. See the test cases for more details on stderr. Similar, if a string is unterminated. Handling the unterminated string and char constant is usually left to the default rules (when a token doesn't match). Building We will building and running your code on lectura.cs.arizona.edu. You will turn in at least 2 files: a Makefile and a Flex file named 'tokenout.l'. The Makefile builds an executable named tokenout on lectura. Like most UNIX tools, tokenout reads from standard input and writes to standard output.: % make tokenout % tokenout < input.txt > output.txt # reads standard input and writes to standard output (We strongly suggest you support 'make clean', 'make test' in your Makefile as well to make your life easier). The file tokenout.l will be built using the UNIX command tool flex and create lex.yy.c which will be compiled and folded into the final executable. You will probably want your main routine in your tokenout.l. Grading We will giving you a bunch of sample inputs and expected outputs where we will compare (using the UNIX diff utility) the expected output with your output. If you match exactly, you will pass the test case, if you don't, you will fail. There are approximately 50 test cases on the class web page under: We reserve the right to add more test cases. Note that they check both the standard output and standard error. In the Testing dir is a README which describes how to run the testcases. You will also be graded on being valgrind clean: In other words, your program should have any memory leaks OR memory errors that valgrind finds. If time permits, we may may have trophies for 'cleanest code' and 'fastest code'. Hints: Do one class of tokens at a time (keywords, then integers, etc) at a time and test them fully before moving on to the next token type. That way you can keep your testing for all the other characters you have processed.

5 Start simple (the input doesn't HAVE to be a C program, just a string of tokens) and build from there. Real C programs make good final tests.