CS 316 (Kong): TinyJ Assignment 1. {<methoddecl>} } [ [<printargument>] )

Transcription

1 CS 316 (Kong): TinyJ Assignment 1 To be submitted no later than: Thursday, November 29. [Note: I expect euclid to be up until midnight that evening, but there is no guarantee that it will be: If euclid unexpectedly goes down after 6 p.m., the deadline will not be extended. If you try to submit after 6 p.m. that evening and find that euclid is down, you may have to make a late submission!] This assignment counts 1.5% towards your grade if the grade is computed using rule A. The TinyJ language is an extremely small subset of Java. Every valid TinyJ program is a valid Java program, and has the same semantics whether it is regarded as a TinyJ or a Java program. The syntax of TinyJ is given by the following EBNF rules: A Java program is a TinyJ program if and only if it can be generated by these EBNF rules, except that TinyJ doesn t support non-decimal (i.e., hexadecimal, octal, or binary) integer literals, method name overloading, program arguments, printing of Boolean values, return; statements within the main() method, and ints that are ³ ,147,418,112. Reserved words of TinyJ are shown in boldface. Some names used by Java library packages, classes, and their methods (e.g., java, Scanner, and nextint) are reserved words of TinyJ, as is main. Otherwise, IDENTIFIER here means any Java identifier consisting of ASCII characters. indicates a nonterminal whose method you must complete in Parser.java--other methods have already been written for you! <program> [<importstmt>] <importstmt> import java. util. Scanner ; <maindecl> <methoddecl> <parameterdecllist> <methoddecl> <maindecl> <datafielddecl> <vardecl> <singlevardecl> <parameterdecl> <compoundstmt> <statement> <assignmentorinvoc> <argumentlist> <ifstmt> <whilestmt> <outputstmt> class IDENTIFIER '' <datafielddecl> '' static <vardecl> int <singlevardecl>, <singlevardecl> ; Scanner IDENTIFIER new Scanner '(' IDENTIFIER '[' ']' [ <expr3> System. in ')' ; ] public static void main '(' String IDENTIFIER '[' ']' ')' <compoundstmt> static ( void int '[' ']') IDENTIFIER '(' <parameterdecllist> ')' <compoundstmt> [<parameterdecl>, <parameterdecl> ] '[' ']' '' ; return [<expr3>] ; <vardecl> <assignmentorinvoc> <compoundstmt> <ifstmt> <whilestmt> <outputstmt> IDENTIFIER ( '['<expr3>']' <expr3> ; <argumentlist> '('[<expr3>,<expr3>]')' if '(' <expr7> ')' <statement> [else <statement>] int '' IDENTIFIER <statement> while '(' <expr7> ')' System. out. ( ; ) <statement> print '(' <printargument> ')' ; println '(' [<printargument>] ')' ; ) <printargument> <expr7> <expr6> <expr5> <expr4> <expr3> <expr2> CHARSTRING <expr3> ' ' <expr6> & <expr5> <expr4> (!) <expr4> <expr3> [(> < > <) <expr3> ] <expr2> (+ ) <expr2> (* / %) '(' <expr7> ')' (+!) UNSIGNEDINT null new int '[' <expr3> ']' '[' ']' IDENTIFIER (. nextint '(' ')' [<argumentlist>]'[' <expr3> <expr6> <expr5> ']' This is the first of three TinyJ assignments. After completing all three assignments you will have a program that can compile any TinyJ program into a simple virtual machine code, and then execute the virtual machine code it has generated. (Execution should produce the same run-time behavior as you would get if you compiled the same TinyJ program using javac into a.class file and then executed that.class file using a Java VM.) There will be exam questions relating to the TinyJ assignments, which may count 25 50% towards your grade. Page 1 of 5 )

2 TinyJ Assignment 1 will not deal with compilation of TinyJ programs, nor with execution of virtual machine code, but only with syntax analysis of TinyJ programs. The goal of TinyJ Assignment 1 is to complete a program that will: (a) determine whether its input file is a <program> according to the above EBNF rules, and (b) output a parse tree of the input file, if the input file is a <program>. Regarding (a), note that a <program> is a syntactically valid TinyJ program, but may contain errors like undeclared variable or array index out of range. A sideways representation of ordered trees, described below, will be used for (b). A Sideways Representation of an Ordered Rooted Tree T If T has just one node, then Otherwise, representation of T the unique node of T representation of T the root of T representation of the 1st subtree of the root of T representation of the 2nd subtree of the root of T... representation of the last subtree of the root of T In this sideways representation, sibling nodes always have the same indentation, but each non-root node is further indented than its parent; the indentation of a node is proportional to the depth of that node in the tree. Here are the ordinary and the sideways representations of a tree: <expr4> <expr3> / \ <expr2> + <expr2> / / \ * UNSIGNEDINT IDENTIFIER UNSIGNEDINT <expr4> <expr3> <expr2> UNSIGNEDINT + <expr2> IDENTIFIER * UNSIGNEDINT Indentation level depth of node in the tree Indentation levels of consecutive lines are equal or differ by just 1: If previous line is a nonterminal, then the current line's indentation level is higher by 1. If previous line is a token/terminal, then the current line's indentation level is the same. If previous line is, the current line's indentation level is lower by 1. How to Install the TinyJ Assignment 1 Files on euclid, venus, and Your PC Do 1 5, and optionally 6 11, before our class on Monday, November 19. (See Seven Files You May Need to Refer to on p. 3.) Remember that Unix/Linux file and command names are case-sensitive when following the instructions below! 1. Login to euclid and enter: /users/kong300/316/tj1setup [The 1 in TJ1setup is the digit 1, not the letter l.] 2. Wait for the line TJ1setup done to appear on the screen, and then enter the following command on euclid: java -cp TJ1solclasses:. TJ1asn.TJ CS316ex12.java 12.sol Note the period after the colon in this command. This command executes my solution to this assignment with CS316ex12.java as the input file and 12.sol as the output file. A listing of CS316ex12.java should be displayed on the screen, and 12.sol should contain a sideways representation of the program s parse tree afterwards. There should not be any error message. To view the tree, you can use less 12.sol or just open 12.sol in an editor. 3. Logout from euclid and login to venus. 4. Enter the following on venus: /home/faculty/ykong/tj1setup [Again, the 1 in TJ1setup is the digit 1, not the letter l.] 5. Repeat step 2 above on venus. The following 6 steps are needed only if you are interested in doing TinyJ assignments on your PC rather than euclid or venus. (Regardless of where you do your work, you must submit on euclid.) These instructions assume you are using Oracle s JDK. Page 2 of 5

3 A Correct TinyJ Input File and the Corresponding Parse Tree That is Written to the Output File by a Solution to TinyJ Assignment 1 Input File (a TinyJ program): import java.util.scanner; class Simple2 static Scanner input new Scanner(System.in); public static void main(string args[]) int x input.nextint(); x x % 3; System.out.println(x + 2); Output File (the above program's parse tree, in sideways representation): <program> <importstmt> Reserved Word: import Reserved Word: java. Reserved Word: util. Reserved Word: Scanner ; Reserved Word: class IDENTIFIER: Simple2 <datafielddecl> Reserved Word: static <vardecl> Reserved Word: Scanner IDENTIFIER: input Reserved Word: new Reserved Word: Scanner ( Reserved Word: System. Reserved Word: in ) ; <maindecl> Reserved Word: public Reserved Word: static Reserved Word: void Reserved Word: main ( Reserved Word: String IDENTIFIER: args [ ] ) <compoundstmt> <statement> <vardecl> Reserved Word: int <singlevardecl> IDENTIFIER: x <expr3> <expr2> IDENTIFIER: input. Reserved Word: nextint ( ) ; <statement> <assignmentorinvoc> IDENTIFIER: x <expr3> <expr2> IDENTIFIER: x % UNSIGNED INTEGER LITERAL: 3 ; <statement> <outputstmt> Reserved Word: System. Reserved Word: out. Reserved Word: println ( <printargument> <expr3> <expr2> IDENTIFIER: x + <expr2> UNSIGNED INTEGER LITERAL: 2 ) ;

4 A TinyJ Input File with a Syntax Error, and the Output That is Generated by a Solution to TinyJ Assignment 1 Input File (the error is that public should be preceded by a semicolon): import java.util.scanner; class Simple2 static Scanner input new Scanner(System.in) public static void main(string args[]) int x input.nextint(); x x % 3; System.out.println(x + 2); Wrong token: Symbols.SEMICOLON expected, not Symbols.PUBLIC Output on the Screen (shows the tokens that are read before the error is detected): 1: 2: 3: 4: 5: 6: 7: import java.util.scanner; class Simple2 static Scanner input new Scanner(System.in) public ERROR! Something's wrong--maybe the following token is missing: ; input.currentchar ' ' LexicalAnalyzer.currentToken Reserved Word: public Output File (an incomplete parse tree whose leaves are the tokens that appear in the input file before the syntax error): <program> <importstmt> Reserved Word: import Reserved Word: java. Reserved Word: util. Reserved Word: Scanner ; Reserved Word: class IDENTIFIER: Simple2 <datafielddecl> Reserved Word: static <vardecl> Reserved Word: Scanner IDENTIFIER: input Reserved Word: new Reserved Word: Scanner ( Reserved Word: System. Reserved Word: in ) Last correct token: Symbols.RPAREN

5 Symbols.java.txt /23/2015 package TJlexer; import java.util.enumset; public enum Symbols // TinyJ reserved words INT("Reserved Word: int", "int"), VOID("Reserved Word: void", "void"), STATIC("Reserved Word: static", "static"), IF("Reserved Word: if", "if"), WHILE("Reserved Word: while", "while"), ELSE("Reserved Word: else", "else"), NEW("Reserved Word: new", "new"), OUT("Reserved Word: out", "out"), PRINT("Reserved Word: print", "print"), SYSTEM("Reserved Word: System", "System"), PRINTLN("Reserved Word: println", "println"), RETURN("Reserved Word: return", "return"), IN("Reserved Word: in", "in"), NULL("Reserved Word: null", "null"), NEXTINT("Reserved Word: nextint", "nextint"), MAIN("Reserved Word: main", "main"), JAVA("Reserved Word: java", "java"), UTIL("Reserved Word: util", "util"), CLASS("Reserved Word: class", "class"), STRING("Reserved Word: String", "String"), PUBLIC("Reserved Word: public", "public"), IMPORT("Reserved Word: import", "import"), SCANNER("Reserved Word: Scanner", "Scanner"), // End of TinyJ reserved words. (See the definition // Other TinyJ tokens that have just one instance LBRACE(""), RBRACE(""), COMMA(","), SEMICOLON(";"), BECOMES(""), LPAREN("("), RPAREN(")"), LBRACKET("["), RBRACKET("]"), DOT("."), OR(" "), AND("&"), NOT("!"), EQ(""), For each Symbols.X enum constant object, the field X.symbolRepresentationForOutputFile (see line 100) is set to the string passed as 1st argument of the constructor call for Symbols.X---see line 106. A call of TJ.output.printSymbol(Symbols.X) or a call of TJ.output.printSymbol(Symbols.X, null) will print the string X.symbolRepresentationForOutputFile to the output file as a node of the sideways parse tree (with the correct indentation). of the reservedwords EnumSet below.)

6 Symbols.java.txt NE("!"), GT(">"), LT("<"), GE(">"), LE("<"), TIMES("*"), DIV("/"), MOD("%"), PLUS("+"), MINUS("-"), // TinyJ tokens that have more than one instance IDENT("IDENTIFIER"), UNSIGNEDINT("UNSIGNED INTEGER LITERAL"), CHARSTRING("CHARACTER STRING LITERAL"), // Fictitious tokens ENDOFINPUT("EOF"), BADTOKEN("?????????? BAD TOKEN"), EMPTY("<empty>"), NONE(""), // Nonterminals NTprogram("<program>"), NTimport("<importStmt>"), NTdataFieldDecl("<dataFieldDecl>"), NTvarDecl("<varDecl>"), NTsingleVarDecl("<singleVarDecl>"), NTmainDecl("<mainDecl>"), NTmethodDecl("<methodDecl>"), NTparameterDeclList("<parameterDeclList>"), NTparameterDecl("<parameterDecl>"), NTcompoundStmt("<compoundStmt>"), NTstatement("<statement>"), NTassignmentOrInvoc("<assignmentOrInvoc>"), NTargumentList("<argumentList>"), NTifStmt("<ifStmt>"), NTwhileStmt("<whileStmt>"), NToutputStmt("<outputStmt>"), NTprintArgument("<printArgument>"), NTexpr7("<expr7>"), NTexpr6("<expr6>"), NTexpr5("<expr5>"), NTexpr4("<expr4>"), NTexpr3("<expr3>"), NTexpr2("<expr2>"), 11/23/2015

7 Symbols.java.txt NTexpr1(""); static final EnumSet<Symbols> reservedwords EnumSet.range(INT, SCANNER); public final String symbolrepresentationforoutputfile; final String reservedwordspelling; Symbols(String symbolrepresentationforoutputfile, String reservedwordspelling) this.symbolrepresentationforoutputfile symbolrepresentationforoutputfile; this.reservedwordspelling reservedwordspelling; Symbols(String symbolrepresentationforoutputfile) this(symbolrepresentationforoutputfile, null); 11/23/2015

8 A Correct TinyJ Input File and the Corresponding Parse Tree That is Written to the Output File by a Solution to TinyJ Assignment 1 <singlevardecl> IDENTIFIER: x import java.util.scanner; <expr3> <expr2> class Simple2 IDENTIFIER: input static Scanner input new Scanner(System.in);. Reserved Word: nextint public static void main(string args[]) ( ) int x input.nextint(); x x % 3; System.out.println(x + 2); ; Each line of the output file / parse tree is printed by 1 call of TJ.output.printSymbol(...) or by 1 call of TJ.output.decTreeDepth(). Output File (the above <statement> <assignmentorinvoc> program's parse tree, in IDENTIFIER: x sideways representation): <expr3> printed by calls of TJ.output.printSymbol(Symbols.IDENT, "x") printed by TJ.output.printSymbol(Symbols.NTprogram) <expr2> treedepth 0<program> <importstmt> printed by TJ.output.printSymbol(Symbols.NTimport) treedepth 1 IDENTIFIER: x treedepth 2 Reserved Word: import Reserved Word: java printed by TJ.output.printSymbol(Symbols.UNSIGNEDINT, 3). printed by TJ.output.printSymbol(Symbols.IMPORT, null) % Reserved Word: util UNSIGNED INTEGER LITERAL: 3. Reserved Word: Scanner ; printed by TJ.output.printSymbol(Symbols.SEMICOLON, null) printed by TJ.output.decTreeDepth() ; treedepth 1 Reserved Word: class IDENTIFIER: Simple2 printed by TJ.output.printSymbol(Symbols.LBRACE, null) <statement> <datafielddecl> <outputstmt> treedepth 2 Reserved Word: static printed by TJ.output.printSymbol(Symbols.NTvarDecl) Reserved Word: System <vardecl> treedepth 3. Reserved Word: Scanner Reserved Word: out IDENTIFIER: input printed by TJ.output.printSymbol(Symbols.NEW, null). Reserved Word: println Reserved Word: new ( Reserved Word: Scanner <printargument> ( <expr3> Reserved Word: System <expr2>. Reserved Word: in IDENTIFIER: x ) ; printed by TJ.output.decTreeDepth() treedepth 2 printed by TJ.output.decTreeDepth() + <expr2> treedepth 1 <maindecl> treedepth 2 Reserved Word: public UNSIGNED INTEGER LITERAL: 2 Reserved Word: static Reserved Word: void Reserved Word: main ( Reserved Word: String ) IDENTIFIER: args ; [ ] ) <compoundstmt> treedepth 3 <statement> treedepth 4 <vardecl> Reserved Word: int treedepth 5 Input File (a TinyJ program):

9 6. In a cmd.exe (command prompt) window* on your PC, enter the following: *You can open a cmd.exe window on your Windows PC as follows: md c:\316java 1. Type Win-r (i.e., hold down the Windows key and type r) to open the Run dialog box. 2. Type cmd into the Open: textbox and press E. 7. Enter javac -version in the cmd.exe window. If you get an error message or if the version number that is printed is older than 1.8.0, then download and install the latest version of the Java JDK e.g., the Java SE (LTS) JDK from and set the PATH variable as explained at The correct value of PATH for the Java SE JDK is typically: c:\program files\java\jdk \bin If you have difficulty with step 1 of these instructions for setting the PATH, try the following instead of that step: 1. Type Win-r to open the Run dialog box. 2. Type control sysdm.cpl into the Open: textbox and press E. 8. Make c:\316java your working directory by entering the following in the cmd.exe window: cd /d c:\316java 9. Using an scp or sftp client, download TJ1asn.jar from your home directory on venus or euclid into the c:\316java folder on your PC. For example, if the PuTTY programs have been installed in the c:\program Files (x86)\putty folder on a PC running a 64-bit version of Windows, or in the c:\program Files\PuTTY folder on a PC running a 32-bit version of Windows (which can be done, e.g., by downloading the file putty-0.70-installer.msi from and running that installer), and assuming c:\316java is your working directory in the cmd.exe window (see step 8), when you are connected to the Internet you can download TJ1asn.jar by entering the appropriate one of the following in the cmd.exe window: "c:\program files (x86)\putty\pscp" xxxxx316@euclid.cs.qc.cuny.edu:tj1asn.jar. "c:\program files\putty\pscp" xxxxx316@euclid.cs.qc.cuny.edu:tj1asn.jar. Here xxxxx316 means your euclid username. Note the double quotes in "c:\program... pscp" and the space followed by a period at the end of the line! 10. Enter the following two commands in the cmd.exe window: jar xvf TJ1asn.jar javac -cp. TJ1asn\TJ.java 11. Enter the following command in the cmd.exe window: java -cp TJ1solclasses;. TJ1asn.TJ CS316ex12.java 12.sol The comments on step 2 also apply here, except that a semicolon rather than a colon precedes the period. A version of less for Windows PCs is available at: Seven Files You May Be Expected to Refer to in Class Starting on Monday, November 19 Either bring a laptop / tablet to class so you can quickly bring any of these seven files up on your screen, or bring printouts of these seven files to class: From your TJ1asn directory on euclid: OutputFileHandler.java.txt Parser.java.txt SourceFileErrorException.java.txt TJ.java.txt From your TJlexer directory on euclid (the l in TJlexer is the letter l, not the digit 1): LexicalAnalyzer.java.txt SourceHandler.java.txt Symbols.java.txt These are the source files of the program, with line numbers added. The actual source files (without line numbers) are in the same directories and have the same names, but their extension is.java. If you have done steps 6 11 above, you can find the same files in C:\316java\TJ1asn and C:\316java\TJlexer on your PC, and these files can be opened using, e.g., any of the editors recommended in the last paragraph of p. 4 of the Lisp Assignment 2 document. Otherwise, you can the files to yourself e.g., you can send TJ1asn/TJ.java.txt to yourself by entering the following on euclid: pine -attach TJ1asn/TJ.java.txt your- -address [After pine starts up, enter Fx y to send the file.] How to Execute My Solution to This Assignment Steps 1 and 4 put 16 files named CS316exk.java (k 0 15) into your home directories on euclid and venus. These are all valid TinyJ source files. If you did step 10, it will have put copies of the same 16 files on your PC. You should be able to execute my solution to this assignment either on euclid or on venus by entering the following command: java -cp TJ1solclasses:. TJ1asn.TJ TinyJ-source-file-name output-file-name args[1] [Your current working directory has to be your home directory for this args[0] to work.] This should also work in a cmd.exe window on your PC if you have done steps 6 11, except that you need a semicolon instead of a colon after TJ1solclasses on a PC: java -cp TJ1solclasses;. TJ1asn.TJ TinyJ-source-file-name output-file-name [Your working directory has to be C:\316java for this to work (see step 8).] EXAMPLES: java java -cp -cp TJ1solclasses:. TJ1asn.TJ TJ1solclasses;. TJ1asn.TJ Page 3 of 5 CS316ex12.java CS316ex12.java 12.sol 12.sol (on euclid or venus) (on a PC)

10 How to Do TinyJ Assignment 1 The file TJ1asn/Parser.java is incomplete. It was produced by taking a complete version of that file and replacing parts of the code with comments of the follwing two forms: /*???????? */ or (in two places) /*???????? default: throw... */ To complete this assignment, replace every such comment in TJ1asn/Parser.java with appropriate code, and recompile the file. On venus or euclid, you can use any text editor to edit the file. If you are working on your PC, do not use Notepad as your editor; I suggest you use one of the editors listed in the last paragraph on p. 4 of the Lisp Assignment 2 document. (For the second type of comment, the appropriate code should include the default: throw... statement.) Do not put Parser.java or Parser.class into any directory other than TJ1asn..java and.class files. Do not change or move other To recompile TJ1asn/Parser.java after editing it, enter the following command: javac -cp. TJ1asn/Parser.java IMPORTANT: If you are doing this on venus or euclid, your current working directory has to be your home directory. If you are doing this on your PC (in a cmd.exe window), your working directory has to be c:\316java (see installation step 8); otherwise javac will not be able to find other classes that are used in Parser.java! As stated on p. 3 of the first-day handout, keep backups of your edited versions of Parser.java on venus and elsewhere. How to Test Your Solution Test your completed version of Parser.java by recompiling it, and then executing the TJ1asn.TJ program with each of the 16 files CS316exk.java (k 0 15) as the TinyJ source file and k.out as the output file. You can do this by entering the following two commands: args[1] javac -cp. TJ1asn/Parser.java args[0] java -cp. TJ1asn.TJ CS316exk.java k.out If you are doing this on venus or euclid, your current working directory has to be your home directory. If you are doing this on your PC (in a cmd.exe window), your working directory has to be c:\316java (see installation step 8). If your program is correct then in each case the output file k.out should be identical to the output file k.sol that is produced by running my solution with the same source file as follows: java -cp TJ1solclasses:. TJ1asn.TJ CS316exk.java k.sol [on euclid or venus] java -cp TJ1solclasses;. TJ1asn.TJ CS316exk.java k.sol [on a PC] On euclid and venus, use diff -c to compare the output files produced by your and my solutions. (This outputs a report of the differences, if any, between the two files.) On a PC, use fc /n to compare files. For example, the commands diff -c k.sol k.out > k.dif [on venus or euclid] and fc /n k.sol k.out > k.dif [on a PC] output to k.dif the differences between k.sol and k.out. (If your solution is correct, there should be no differences.) How to Submit a Solution to This Assignment This assignment is to be submitted no later than Thursday, November 29. It counts 1.5% towards your grade if the grade is computed using Rule A. [Note: If euclid unexpectedly goes down after 6 p.m. on the due date, the deadline will not be extended. Try to submit no later than noon that day, and on an earlier day if possible.] To submit: 1. Add a comment at the beginning of your completed version of Parser.java that gives your name and the names of the students you worked with (if any). As usual, you may work with up to two other students. 2. Leave your final version of Parser.java on euclid in your TJ1asn directory, so it replaces the original version of Parser.java, before midnight on the due date. When two or three students work together, each of the students must leave the completed file in his/her directory. If you are working on venus or your PC, you can transfer Parser.java to your TJ1asn directory on euclid by following the instructions on the next page. 3. Be sure to test your submission on euclid. Note that if your modified version of Parser.java cannot even be compiled without error on euclid, then you will receive no credit at all for your submission! IMPORTANT: Do NOT open your submitted file Parser.java in an editor on euclid after the due date, unless you are resubmitting a corrected version of your solution as a late submission. Also do not execute mv, chmod, or touch with your submitted file as an argument after the due date. (However, it is OK to view a submitted file using the less file viewer after the due date.) Remember that, as stated on page 3 of the first-day handout, you are required to keep a backup copy of your submitted file on venus, and another copy elsewhere. Page 4 of 5

11 3 examples of???????? comments in Parser.java that you need to replace with appropriate 11/17/2007 code. Parser.java.txt private static void outputstmt() throws SourceFileErrorException TJ.output.printSymbol(NToutputStmt); accept(system); accept(dot); accept(out); accept(dot); switch (getcurrenttoken()) /*???????? default: throw new SourceFileErrorException("print() or println() expected, not " + getcurrenttoken().symbolrepresentationforoutputfile); */ private static void printargument() throws SourceFileErrorException TJ.output.printSymbol(NTprintArgument); /*???????? */ private static void expr7() throws SourceFileErrorException TJ.output.printSymbol(NTexpr7); /*???????? */ 7

12 TJ.java.txt /3/2007 package TJ1asn; import import import import TJlexer.SourceHandler; TJlexer.LexicalAnalyzer; TJlexer.Symbols; java.io.printwriter; public final class TJ public static SourceHandler input; public static OutputFileHandler output; the "TJ.output" in TJ.output.printSymbol(...) etc. public static final int DATA_MEMORY_SIZE 20000; public static final int data[] new int [DATA_MEMORY_SIZE]; /* space for string literals */ public static void main(string args[]) final String inputfilename args.length 0? null : args[0]; final String outputfilename args.length < 1? null : args[1]; try output new OutputFileHandler(outputFileName); input new SourceHandler(inputFileName); LexicalAnalyzer.setIO(input, output); LexicalAnalyzer.setStringTable(data); LexicalAnalyzer.nextToken(); Parser.program(); The most important line: This tries to read a <program> from the input file and output the parse tree of that <program>. if (LexicalAnalyzer.getCurrentToken()! Symbols.ENDOFINPUT) throw new SourceFileErrorException("Token encountered after end of program"); catch (SourceFileErrorException theerror) System.out.println("\n\n\nERROR! " + theerror.errormessage); if (input! null) Error messages are if (input.getcurrentchar()! SourceHandler.eofDesignator) System.out.println("input.currentChar '" + (char) input.getcurrentchar() + '\''); printed to the screen by else this exception handler. System.out.println("input.currentChar EOF"); System.out.print("LexicalAnalyzer.currentToken "); TJ.output.outputSymbol(LexicalAnalyzer.getCurrentToken(), LexicalAnalyzer.getTokenValue(), new PrintWriter(System.out, true)); System.out.println(); finally if (output! null) 1

13 A TinyJ Input File with a Syntax Error, and the Output That is Generated by a Solution to TinyJ Assignment 1 Input File (the error is that public should be preceded by a semicolon): import java.util.scanner; class Simple2 static Scanner input new Scanner(System.in) public static void main(string args[]) int x input.nextint(); x x % 3; System.out.println(x + 2); Output on the Screen (shows the tokens that are read before the error is detected): 1: 2: 3: 4: 5: 6: 7: import java.util.scanner; class Simple2 static Scanner input new Scanner(System.in) public ERROR! Something's wrong--maybe the following token is missing: ; input.currentchar ' ' LexicalAnalyzer.currentToken Reserved Word: public Output File (an incomplete parse tree whose leaves are the tokens that appear in the input file before the syntax error): <program> <importstmt> Reserved Word: import Reserved Word: java. Reserved Word: util. Reserved Word: Scanner ; Reserved Word: class IDENTIFIER: Simple2 <datafielddecl> Reserved Word: static <vardecl> Reserved Word: Scanner IDENTIFIER: input Reserved Word: new Reserved Word: Scanner ( Reserved Word: System. Reserved Word: in )

14 Parser.java.txt /17/2007 package TJ1asn; import import import import static TJlexer.LexicalAnalyzer.getCurrentToken; static TJlexer.LexicalAnalyzer.nextToken; static TJlexer.Symbols.*; TJlexer.Symbols; // ************************************ Recursive Descent Parser ************************************** public final class Parser private static void accept (Symbols expectedtoken) throws SourceFileErrorException if (getcurrenttoken() expectedtoken) nexttoken(); else throw new SourceFileErrorException("Something's wrong--maybe the following token is missing: " + expectedtoken.symbolrepresentationforoutputfile); static void program () throws SourceFileErrorException TJ.output.printSymbol(NTprogram); if (getcurrenttoken() IMPORT) importstmt(); accept(class); accept(ident); accept(lbrace); while (getcurrenttoken() STATIC) datafielddecl(); maindecl(); while (getcurrenttoken() STATIC) methoddecl(); accept(rbrace); private static void importstmt() throws SourceFileErrorException 1

15 OutputFileHandler.java.txt /3/2007 package TJ1asn; import java.io.*; import java.util.scanner; import TJlexer.Symbols; public class OutputFileHandler protected PrintWriter outfilewriter; public final PrintWriter getoutfilewriter() return outfilewriter; protected int treedepth 0; public final int gettreedepth() return treedepth; public final void inctreedepth() treedepth++; public final void dectreedepth() for (int i 0; i < treedepth; i++) outfilewriter.print(" "); outfilewriter.println(""); treedepth--; public final void printsymbol(symbols nodename) printsymbol(nodename, null); called by Parser.java's parsing methods to print nonterminals public final void printsymbol(symbols nodename, Object nodevalue) if (nodename! Symbols.NONE) for (int i 0; i < treedepth; i++) prints treedepth spaces outfilewriter.print(" "); outputsymbol(nodename, nodevalue, outfilewriter); 1 called by nexttoken() to print currenttoken

16 OutputFileHandler.java.txt /3/2007 public void outputsymbol(symbols nodename, Object nodevalue, PrintWriter out) out.print(nodename.symbolrepresentationforoutputfile); if (nodevalue null) out.println(); else out.println(": " + nodevalue); protected final void openoutputfile (String filename) throws SourceFileErrorException System.out.print("Enter name for output file: "); if (filename! null) System.out.print(filename+"\n\n"); else System.out.flush(); filename (new Scanner(System.in)).nextLine(); System.out.println(); try outfilewriter new PrintWriter(new FileWriter(filename)); catch (IOException e) throw new SourceFileErrorException("Failed to open output file"); protected OutputFileHandler(String filename) throws SourceFileErrorException openoutputfile(filename); 2

17 LexicalAnalyzer.java.txt /23/2015 package TJlexer; import static TJlexer.Symbols.*; allows Symbols.SEMICOLON to be written as just SEMICOLON, etc. import TJ1asn.OutputFileHandler; import TJ1asn.SourceFileErrorException; public final class LexicalAnalyzer private static SourceHandler input; private static OutputFileHandler output; private static int stringtable[]; public static void setio(sourcehandler sourcehandler, OutputFileHandler outputfilehandler) input sourcehandler; output outputfilehandler; public static void setstringtable(int[] tbl) stringtable tbl; private static Symbols currenttoken NONE; currenttoken contains the Symbols.X constant which represents the TinyJ token that this program "is now looking at". public static Symbols getcurrenttoken() currenttokenneedstobeinspected false; return currenttoken; getcurrenttoken() is the public getter method for currenttoken. private static boolean currenttokenneedstobeinspected; private static int currentvalue; // numerical value of UNSIGNEDINT token public static int getcurrentvalue() return currentvalue; private static String currentspelling; public static String getcurrentspelling() return currentspelling; private static int startofstring; private static int endofstring -1; // spelling of IDENT

18 LexicalAnalyzer.java.txt /23/2015 public static int getstartofstring() return startofstring; public static int getendofstring() return endofstring; public static void setendofstring(int addr) // called in ParserAndTranslator's program() method endofstring addr; private static Object tokenvalue; // passed to output.printsymbol() at start of nexttoken(); // contains information used to output the currenttoken public static Object gettokenvalue() return tokenvalue; public static void nexttoken() throws SourceFileErrorException output.printsymbol(currenttoken, tokenvalue); if (currenttokenneedstobeinspected) throw new SourceFileErrorException("Internal error in parser: Token discarded without being inspected"); else A successful call of nexttoken() does the following: currenttokenneedstobeinspected true; StringBuilder currenttokenstring new StringBuilder(10); while (input.getcurrentchar() ' ') input.nextchar(); tokenvalue null; if (Character.isLetter((char) input.getcurrentchar()) input.getcurrentchar() '_' input.getcurrentchar() '$') /* identifier or reserved word */ 1. Prints currenttoken to the output file. 2. Skips white space, then reads the characters of the next token from the input file and sets currenttoken to the Symbols.X constant which represents that token. 3. Sets tokenvalue appropriately if that token is an identifier, a string literal, or an unsigned int literal; otherwise tokenvalue will be null (from line 81). do currenttokenstring.append((char) input.getcurrentchar()); input.nextchar(); while (Character.isLetterOrDigit((char) input.getcurrentchar()) input.getcurrentchar() '_' input.getcurrentchar() '$');

19 LexicalAnalyzer.java.txt currentspelling currenttokenstring.tostring(); for (Symbols resword : Symbols.reservedWords) if (currentspelling.equals(resword.reservedwordspelling)) currenttoken resword; return; currenttoken IDENT; tokenvalue currentspelling; return; /* identifier or reserved word */ else switch (input.getcurrentchar()) case '0': case '1': case '2': case '3': case '4': case '5': case '6': case '7': case '8': case '9': /* unsigned integer */ currenttoken UNSIGNEDINT; do currenttokenstring.append((char) input.getcurrentchar()); input.nextchar(); while (input.getcurrentchar() > '0' && input.getcurrentchar() < '9'); currentvalue Integer.parseInt(currentTokenString.toString()); tokenvalue currentvalue; return; case '"': currenttoken CHARSTRING; startofstring endofstring + 1; int linenum input.getsourcefilereader().getlinenumber(); input.nextchar(); int c; while ((c input.getcurrentchar())! '"') if (c SourceHandler.eofDesignator) throw new SourceFileErrorException("End of file occurred within a string."); else if (c '\\') input.nextchar(); switch (input.getcurrentchar()) case 'n': c '\n'; break; case '\\': c '\\'; break; case '"': c '\"'; break; 11/23/2015

20 LexicalAnalyzer.java.txt default: c input.getcurrentchar(); break; currenttokenstring.append((char) c); stringtable[++endofstring] c; input.nextchar(); if (input.getsourcefilereader().getlinenumber()! linenum) throw new SourceFileErrorException("Newline not allowed within a string."); tokenvalue '"' + currenttokenstring.tostring() + '"'; input.nextchar(); return; case '': input.nextchar(); if (input.getcurrentchar() '') currenttoken EQ; input.nextchar(); else currenttoken BECOMES; return; case '!': input.nextchar(); if (input.getcurrentchar() '') currenttoken NE; input.nextchar(); else currenttoken NOT; return; case '<': input.nextchar(); if (input.getcurrentchar() '') currenttoken LE; input.nextchar(); else currenttoken LT; return; 11/23/2015

21 LexicalAnalyzer.java.txt /23/2015 case '>': input.nextchar(); if (input.getcurrentchar() '') currenttoken GE; input.nextchar(); else currenttoken GT; return; case '+': input.nextchar(); if (input.getcurrentchar() '+') currenttoken BADTOKEN; tokenvalue "\"++\""; throw new SourceFileErrorException("Unrecognized token: " + tokenvalue); else currenttoken PLUS; return; case '-': input.nextchar(); if (input.getcurrentchar() '-') currenttoken BADTOKEN; tokenvalue "\"--\""; throw new SourceFileErrorException("Unrecognized token: " + tokenvalue); else currenttoken MINUS; return; case case case case case case case case case case case case case case '&': ' ': '': '': ',': ';': '(': ')': '[': ']': '.': '*': '/': '%': currenttoken currenttoken currenttoken currenttoken currenttoken currenttoken currenttoken currenttoken currenttoken currenttoken currenttoken currenttoken currenttoken currenttoken AND; input.nextchar(); return; OR; input.nextchar(); return; LBRACE; input.nextchar(); return; RBRACE; input.nextchar(); return; COMMA; input.nextchar(); return; SEMICOLON; input.nextchar(); return; LPAREN; input.nextchar(); return; RPAREN; input.nextchar(); return; LBRACKET; input.nextchar(); return; RBRACKET; input.nextchar(); return; DOT; input.nextchar(); return; TIMES; input.nextchar(); return; DIV; input.nextchar(); return; MOD; input.nextchar(); return;

22 LexicalAnalyzer.java.txt case SourceHandler.eofDesignator: currenttoken ENDOFINPUT; return; default: currenttoken BADTOKEN; tokenvalue "'" + (char) input.getcurrentchar() + "'"; throw new SourceFileErrorException("Unrecognized token: " + tokenvalue); /* nexttoken */ 11/23/2015

23 This is the file you have to complete. Parser.java.txt /17/2007 package TJ1asn; import import import import static TJlexer.LexicalAnalyzer.getCurrentToken; allows LexicalAnalyzer.getCurrentToken() to be written as just getcurrenttoken(). allows LexicalAnalyzer.nextToken() to be written as just nexttoken(). static TJlexer.LexicalAnalyzer.nextToken; static TJlexer.Symbols.*; allows Symbols.SEMICOLON to be written as just SEMICOLON, etc. TJlexer.Symbols; // ************************************ Recursive Descent Parser ************************************** public final class Parser private static void accept (Symbols expectedtoken) throws SourceFileErrorException if (getcurrenttoken() expectedtoken) nexttoken(); else throw new SourceFileErrorException("Something's wrong--maybe the following token is missing: " + expectedtoken.symbolrepresentationforoutputfile); A call of accept(x) does the following: static void program () throws SourceFileErrorException TJ.output.printSymbol(NTprogram); if (getcurrenttoken() IMPORT) importstmt(); accept(class); accept(ident); accept(lbrace); 1. It checks that currenttoken Symbols.X. 2. Assuming currenttoken Symbols.X it calls nexttoken(), which: (a) Prints token Symbols.X to the output file. (b) Reads in the next token from the input file and sets currenttoken (& tokenvalue) accordingly. But if currenttoken! Symbols.X when accept(x) is called, an exception is thrown and the program will terminate after printing an error message on the screen! while (getcurrenttoken() STATIC) datafielddecl(); maindecl(); Calling accept(x) has the same effect as calling nexttoken() unless currenttoken! Symbols.X, in which case the program terminates with an error message. while (getcurrenttoken() STATIC) methoddecl(); accept(rbrace); private static void importstmt() throws SourceFileErrorException 1

24 RECURSIVE DESCENT PARSING For each nonterminal <n> of the EBNF specification, Parser.java has a 11/17/2007 corresponding static parsing method n(). When n() is called, it expects currenttoken to be a possible first token* TJ.output.printSymbol(NTimport); of an instance of <n>. If this is so, then the call of n() will (if possible): accept(import); 1. Read in the tokens in the rest of an instance of <n>. accept(java); 2. Output (with an indentation of TJ.output.treeDepth) a sideways parse accept(dot); accept(util); tree, with root <n>, which generates the instance of <n> that is read. Parser.java.txt accept(dot); accept(scanner); accept(semicolon); On return from a successful call of n(), currenttoken will be the first token after the instance of <n> that it read. *If <n> can generate an empty string, as in the case <n> <parameterdecllist>, then when n() is called currenttoken might also be the first token after an empty sequence that is derived from <n>. private static void datafielddecl() throws SourceFileErrorException TJ.output.printSymbol(NTdataFieldDecl); accept(static); vardecl(); private static void vardecl() throws SourceFileErrorException TJ.output.printSymbol(NTvarDecl); if (getcurrenttoken() INT) nexttoken(); singlevardecl(); while (getcurrenttoken() COMMA) nexttoken(); singlevardecl(); accept(semicolon); else if (getcurrenttoken() SCANNER) nexttoken(); if (getcurrenttoken() IDENT) nexttoken(); 2

25 Parser.java.txt /17/2007 else throw new SourceFileErrorException("Scanner name expected"); accept(becomes); accept(new); accept(scanner); accept(lparen); accept(system); accept(dot); accept(in); accept(rparen); accept(semicolon); else throw new SourceFileErrorException("\"int\" or \"Scanner\" expected"); private static void singlevardecl() throws SourceFileErrorException TJ.output.printSymbol(NTsingleVarDecl); /*???????? */ private static void maindecl() throws SourceFileErrorException TJ.output.printSymbol(NTmainDecl); accept(public); accept(static); accept(void); accept(main); accept(lparen); accept(string); accept(ident); accept(lbracket); accept(rbracket); accept(rparen); 3

26 Parser.java.txt /17/2007 compoundstmt(); private static void methoddecl() throws SourceFileErrorException TJ.output.printSymbol(NTmethodDecl); /*???????? */ private static void parameterdecllist() throws SourceFileErrorException TJ.output.printSymbol(NTparameterDeclList); if (getcurrenttoken() INT) parameterdecl(); while (getcurrenttoken() COMMA) nexttoken(); parameterdecl(); else TJ.output.printSymbol(EMPTY); private static void parameterdecl() throws SourceFileErrorException TJ.output.printSymbol(NTparameterDecl); accept(int); accept(ident); while (getcurrenttoken() LBRACKET) nexttoken(); accept(rbracket); 4

27 Parser.java.txt /17/2007 private static void compoundstmt() throws SourceFileErrorException TJ.output.printSymbol(NTcompoundStmt); /*???????? */ private static void statement() throws SourceFileErrorException TJ.output.printSymbol(NTstatement); switch (getcurrenttoken()) case SEMICOLON: nexttoken(); break; case RETURN: nexttoken(); if (getcurrenttoken()! SEMICOLON) expr3(); accept(semicolon); break; case INT: case SCANNER: vardecl(); break; case IDENT: assignmentorinvoc(); break; case LBRACE: compoundstmt(); break; case IF: ifstmt(); break; case WHILE: whilestmt(); break; case SYSTEM: outputstmt(); break; default: throw new SourceFileErrorException("Expected first token of a <statement>, not " + getcurrenttoken().symbolrepresentationforoutputfile); private static void assignmentorinvoc() throws SourceFileErrorException TJ.output.printSymbol(NTassignmentOrInvoc); /*???????? */ 5

28 Parser.java.txt /17/2007 private static void argumentlist() throws SourceFileErrorException TJ.output.printSymbol(NTargumentList); /*???????? */ private static void ifstmt() throws SourceFileErrorException TJ.output.printSymbol(NTifStmt); accept(if); accept(lparen); expr7(); accept(rparen); statement(); if (getcurrenttoken() ELSE) nexttoken(); statement(); private static void whilestmt() throws SourceFileErrorException TJ.output.printSymbol(NTwhileStmt); /*???????? */ 6

29 Parser.java.txt /17/2007 private static void outputstmt() throws SourceFileErrorException TJ.output.printSymbol(NToutputStmt); accept(system); accept(dot); accept(out); accept(dot); switch (getcurrenttoken()) /*???????? default: throw new SourceFileErrorException("print() or println() expected, not " + getcurrenttoken().symbolrepresentationforoutputfile); */ private static void printargument() throws SourceFileErrorException TJ.output.printSymbol(NTprintArgument); /*???????? */ private static void expr7() throws SourceFileErrorException TJ.output.printSymbol(NTexpr7); /*???????? */ 7

30 Parser.java.txt /17/2007 private static void expr6() throws SourceFileErrorException TJ.output.printSymbol(NTexpr6); /*???????? */ private static void expr5() throws SourceFileErrorException TJ.output.printSymbol(NTexpr5); /*???????? */ private static void expr4() throws SourceFileErrorException TJ.output.printSymbol(NTexpr4); /*???????? */ private static void expr3() throws SourceFileErrorException TJ.output.printSymbol(NTexpr3); /*???????? */ private static void expr2() throws SourceFileErrorException TJ.output.printSymbol(NTexpr2); 8

31 Parser.java.txt /17/2007 expr1(); while ( getcurrenttoken() TIMES getcurrenttoken() DIV getcurrenttoken() MOD) nexttoken(); expr1(); private static void expr1() throws SourceFileErrorException TJ.output.printSymbol(NTexpr1); switch (getcurrenttoken()) /*???????? default: throw new SourceFileErrorException("Malformed expression"); */ 9

32 How to Transfer TJ1asn/Parser.java from venus or a PC to euclid s TJ1asn Directory The following instructions assume that xxxxx316 is your username on euclid. If you are working on venus, and your current working directory is your home directory, enter the following command to transfer TJ1asn/Parser.java to your TJ1asn directory on euclid: scp TJ1asn/Parser.java xxxxx316@euclid.cs.qc.cuny.edu:tj1asn You will be asked to enter your euclid password. If you are working on a PC that is running a 64-bit version of Windows and the PuTTY programs have been installed into the c:\program Files (x86)\putty folder, or you are working on a PC that is running a 32-bit version of Windows and the PuTTY programs* have been installed into the c:\program Files\PuTTY folder, then you can transfer TJ1asn/Parser.java into your TJ1asn directory on euclid by entering the appropriate one of the following commands in a cmd.exe window: 64-bit Windows: "c:\program files (x86)\putty\pscp" TJ1asn/Parser.java xxxxx316@euclid.cs.qc.cuny.edu:tj1asn 32-bit Windows: "c:\program files\putty\pscp" TJ1asn/Parser.java xxxxx316@euclid.cs.qc.cuny.edu:tj1asn The double quotes and the backslashes in "c:\program... pscp" are needed! You will be asked to enter your euclid password. IMPORTANT: After you have transferred TJ1asn/Parser.java to your TJ1asn directory on euclid, you should test your code on euclid see the How to Test Your Solution instructions on the previous page. (It is not enough to have tested your code on venus or your PC, because testing on a machine other than euclid does not test the file you actually submitted!) *You can get the PuTTY programs by downloading the installer file putty-0.70-installer.msi from and then running this installer on your PC. Page 5 of 5

33 A Mistake to Avoid When Doing TinyJ Assignment 1 A common mistake in writing recursive descent parsing code is to write getcurrenttoken() X or accept(x) [which performs a getcurrenttoken() X test] using a Symbols constant X that represents a nonterminal. This is wrong, as getcurrenttoken() returns a Symbols constant that represents a token. Here are two examples of this kind of mistake. 1. When writing the method argumentlist(), which should be based on the EBNF rule <argumentlist> '('[<expr3>,<expr3>]')' it would be wrong to write: accept(lparen); if (getcurrenttoken() NTexpr3) /* INCORRECT! */ expr3();... // a while loop that deals with,<expr3> accept(rparen); Here it would be correct to write code of the following form: accept(lparen); if (getcurrenttoken()! RPAREN) /* CORRECT */ expr3();... // a while loop that deals with,<expr3> accept(rparen); 2. When writing the method expr1(), one case you need to deal with relates to the following part of the EBNF rule that defines : IDENTIFIER (. nextint '(' ')' [<argumentlist>]'[' <expr3> ']' ) Here it would be wrong to write something like: case IDENT: nexttoken(); if (getcurrenttoken()! DOT) if (getcurrenttoken() NTargumentList /* INCORRECT! */ ) argumentlist();... // a while loop that deals with '[' <expr3> ']' else... // code to deal with. nextint '(' ')' break; Instead, you can write something like: case IDENT: nexttoken(); if (getcurrenttoken()! DOT) if (getcurrenttoken() LPAREN /* CORRECT */ ) argumentlist();... // a while loop that deals with '[' <expr3> ']' else... // code to deal with. nextint '(' ')' break; The use of LPAREN in the above code is correct because the first token of any instance of <argumentlist> must be a left parenthesis, as we see from the EBNF rule <argumentlist> '('[<expr3>,<expr3>]')'