Improving Exception Messages with ExceptionDoctor

Transcription

1 Improving Exception Messages with ExceptionDoctor Michael Woods 1 and Stephen H. Edwards 1 1 Department of Computer Science, Virginia Tech, Blacksburg VA US Abstract - Beginning programmers often have difficulty interpreting exceptions and using the associated messages to pinpoint the cause of incorrect program behavior. When an interactive development environment (IDE) presents a novice developer with a runtime time exception, it generally provides with a stack trace and a limited, cryptic exception message that is hard for a beginner to interpret. This paper describes ExceptionDoctor, a Java utility that solves this problem. ExceptionDoctor intercepts exceptions thrown by student code and improves the embedded exception messages to provide levelappropriate descriptions. ExceptionDoctor also examines the source code that produced the exception (if available) in order to describe the immediate cause of the exception in student-level terms. Keywords: Java, run-time error, exception handler, explanation, debugging. 1 Introduction As entry-level programmers begin learning Java, they are quickly and abruptly introduced to the runtime exception mechanism inherent to the JVM, or Java Virtual Machine. Without proper instruction, these novice developers are forced to fight their way through understanding these runtime exceptions without the understanding of stack traces and exception messages. To understand the messages that the JVM provides the user, the student must understand what causes that exception in the first place. ExceptionDoctor rewrites Java runtime exception messages to be user friendly and more context sensitive. ExceptionDoctor s improved exception messages include three pieces of information: the offending line of code (if available), an improved exception message that both explains the meaning of the exception and the most likely cause(s), and the original exception s stack trace. The improved exception message is derived from the original exception and contains information derived from the offending line of code such as how certain variables may have contributed to the exception s cause. The improved message is presented in paragraph form with ExceptionDoctor s best guess on which portions of the source code caused the problem, how the issue can be avoided, and the name of the exception printed at the bottom. Another unique aspect of ExceptionDoctor is the redesigned architecture that allows for easy integration with existing IDEs and runtime exception handling programs. ExceptionDoctor can be added to handle any uncaught exceptions in a Java program by simply adding it to the classpath and using an additional command line argument when running the target program. The distributable library for ExceptionDoctor automatically inserts itself into JUnit tests [1], so that uncaught exceptions during testing receive benefits. If desired, ExceptionDoctor can be integrated into internal exception handlers (e.g., for caught exceptions) using a single line of code added to the targeted catch blocks. Finally, ExceptionDoctor can be added to projects easily in most interactive development environments (IDEs), including BlueJ [3] and Eclipse [2], so that students can see the new exception messages directly in the IDE s output. It can even be added onto automated grading tools. ExceptionDoctor s design uses a data-driven dispatch structure to improve efficiency and increase flexibility. As a result, once can replace specific messages and extend the system with new messages for new types of exceptions, all without modifying or even recompiling the ExceptionDoctor code. 2 Relationship to Previous Work While not many projects attempt to improve messages returned by Java exceptions, presenting the exception messages in a more readable format has been implemented by many IDEs. BlueJ [3], like many other IDEs relies on the built-in messages generated by the JVM for runtime exceptions that occur in client code. These are presented in the terminal window in the same way they would on the console if the program were run stand-alone. For exceptions that arise inside JUnit tests, BlueJ attempts hides some of the more distracting exception information from beginning users, displaying the most important information when specifically requested. When an exception occurs during a JUnit test run in BlueJ, the test results window includes a box that displays the exception s message and a shortened stack trace. BlueJ s shortened stack trace only shows the lines of code written by the student that caused the exception. This allows a student to focus on the lines of code that threw an exception; however, it does not help the student learn what they did wrong.

2 ExceptionDoctor takes a different approach to displaying exceptions. While ExceptionDoctor helps students focus on the line where the exception occurs, it also attempts to teach them what part of the line caused the exception and how it might be avoided in the future. ExceptionDoctor was inspired by and is based on the exception explanation features implemented in the Backstop project [4] created at Columbia University. Backstop provides Java execution tracing support together with a runtime exception handler to rewrite exception messages. While ExceptionDoctor has reused many of the error messages displayed by Backstop, the entire dispatch structure has been redesigned, and then packaged in a radically different deployment mechanism. While Backstop was only able to reinterpret exceptions thrown by programs passed directly to it, ExceptionDoctor is capable of integration with existing projects. This feature is an important difference between the two because it allows ExceptionDoctor to seamlessly integrate with existing professional tools with little effort (i.e., JUnit and Eclipse). By integrating ExceptionDoctor with existing tools, students are able to work in a professional environment with added training wheels. As a student progresses, they can easily remove the jar from the project and continue using the same tools. 3 ExceptionDoctor s Design ExceptionDoctor is composed of two different structures. The first structure is the handler class. Handler classes allow ExceptionDoctor to wrap exceptions with an improved exception message. The second structure is the exception map. The exception map contains all of the mappings from exception classes to their corresponding handlers. The exception map is a singleton that is empty when initialized and filled dynamically as exceptions are wrapped and new mappings are found. When ExceptionDoctor is passed an exception, it initially looks for an existing relationship in the exception map between the exception and a corresponding handler class. If no relationship exists, ExceptionDoctor looks through the class path for classes that could wrap the exception. If no classes exist, ExceptionDoctor then looks at the exception s superclass and attempts to wrap it. Because all of Java s exceptions descend from the Throwable class, which has its own default handler, there is a handler for every Java exception passed to ExceptionDoctor. Wrapped or nested Java exceptions are also handled by unwrapping first. After a successful lookup has been performed, ExceptionDoctor will then cache the exception to handler relationship in the exception map and wrap the exception with the handler that was found during the previously mentioned search. Handler classes are only able to change the message that will be displayed to the user, ExceptionDoctor enforces that a wrapped exception still has the original stack trace and has the original exception as the cause. 3.1 Dispatch Structure ExceptionDoctor includes a unique dispatch structure that allows for dynamic lookups of possible exception handler classes. When an exception is passed to ExceptionDoctor, it uses class names to determine the exception-to-handler relationship. All handlers must follow the naming convention ExceptionNameHandler, for example: IOExceptionHandler. Using this naming convention, new exception handlers can be created in the project s workspace. When ExceptionDoctor is called, the new handler is dynamically found through a search of the classpath. Assuming this class implements the ExceptionHandlerInterface, the discovered class is mapped to the exception in the exception map and used to wrap the exception in all future references. New exception handlers only need two things to be integrated with ExceptionDoctor, they must follow the previously mentioned naming convention and they must implement the ExceptionHandlerInterface. The interface ensures that the class has a wrapexception(throwable exception) method that returns a wrapped exception. The ExceptionDoctor API also includes an Abstract- ExceptionHandler that provides a set of utility functions for handler classes. These utilities help standardize the messages and exceptions that ExceptionDoctor produces. 3.2 Handlers Every handler must implement the Exception- HandlerInterface and provide a wrapexception() method that returns a newly wrapped exception. Each of the handlers can also extend the AbstractExceptionHandler, which provides methods for accessing the offending line of code. As an example, the code in Figure 1 shows a code sequence that throws an ArrayIndexOutOfBounds- Exception. When this code is run, the appropriate Array- IndexOutOfBoundsHandler will catch the exception and rework the message from -2 to the message that is displayed in Figure 2. Figure 2 was captured from eclipse s console after the uncaught exception terminated the program. The wrapped exception retains all information from the original exception. This allows the student to see what a normal exception stack trace looks like. 3.3 Dynamic Caching ExceptionDoctor implements a dynamic caching system in the map from exception classes to their handlers. Each time an exception is passed to ExceptionDoctor for

3 translation, the exception map attempts to find a matching handler. If a handler is found for the passed exception, this results in a direct link in the exception map. This allows ExceptionDoctor to dynamically populate its mappings from exceptions to handlers as needed. Dynamic caching allows for quick access to these mappings when batch processing or when integrating ExceptionDoctor into grading servers. 3.4 Running With ExceptionDoctor While ExceptionDoctor integrates itself into JUnit s test execution infrastructure, one must explicitly ask for its services to receive improved exception explanations in a plain Java program. Normally, a Java program can be run using a command like this: java <MainClassName> <args> To include ExceptionDoctor, the command should be rewritten as: java ExceptionDoctor <MainClassName> <args> This addition ensures that ExceptionDoctor will be able to access all uncaught exceptions. If ExceptionDoctor is unable to rewrite the exception message for any given exception, the original exception is passed through without change. Section 5 describes automatic configuration with JUnit tests. 4 Impact on Student Development In recent research into the most common mistakes int z = 3; int y = 5; Object x[] = new String[5]; Object k = x[z-y]; Figure 1: This code segment generates an exception. java.lang.arrayindexoutofboundsexception: In file DemoProgram.java on line 52, which reads: Object k = x[z - y]; It seems that the code tried to use an illegal value as an index to an array. The code was trying to access an element at index -2 of the array called "x". Remember, you cannot have a negative index. Be sure that the index is always positive. The variable "z - y" had the value -2 when the error occured. This error is called an ArrayIndexOutOfBoundsException. at Main.weeklyAverages(DemoProgram.java:52) at Main.computeWeatherStats(DemoProgram.java:45) at Main.main(DemoProgram.java:18) Figure 2: Example exception message rewritten in Eclipse. made by novice programmers, it was found that file I/O, null variable, and type errors were the most common exceptions to arise [5]. Each of these error types corresponds to a runtime exception that ExceptionDoctor handles. String filename = "/home/mike/notreal/foo.bar"; File f = new File(fileName); Scanner s = new Scanner(f); java.io.filenotfoundexception: In file Junit3Test.java on line 15, which reads: Scanner s = new Scanner(f); It appears that the code was trying to operate on a file called /home/mike/notreal/foo.bar. However, it seems that this file may not exist. Check that the filename is spelled correctly. Analysis shows that /home/mike/ is a valid path. The remainder of the file is invalid. This error is called an FileNotFoundException. at Junit3Test.testNoFile(Junit3Test.java:15) Figure 3: An example of a FileNotFoundException. According to Jadud [5], the most common error type is file I/O. File I/O exceptions can be caused by novices attempting to open a file or write to a file. If a student is attempting to open a file, there are generally two states that the file can be in that will result in an exception. The first state is a nonexistent file. When the file does not exist, ExceptionDoctor checks the given path for errors in the directory structure. If the directory path is correct, then the user is simply told that the file does not exist. However, if the path is incorrect, it is more likely that there is a spelling mistake in the path. Figure 3 gives an example of output in this situation. File exceptions can also occur because of bad file permissions. I/O errors are also caused by errors in writing and reading strings from input and output streams. However, IOExceptions can be complex exceptions. Because IOExceptions are so complex, ExceptionDoctor does include message support for every situation that could cause an IOException. However, through diagnostic tools such as ClockIt [6], an instructor can gather information about the types of exceptions that students are encountering and create a custom IO handler class to remind students of lessons they learned in class. The next most common kind of exception is due to an uninitialized variable. An uninitialized variable most commonly manifests itself as a NullPointerException or various data structure exceptions. When NullPointer- Exceptions are thrown, the novice has created a variable that does not refer to any object. When ExceptionDoctor encounters this, it attempts to locate the null variable in the source code. If there is only one potentially null variable in the line, the explanation simply uses that variable name in describing the cause. If there are multiple nullable variables, a list of possible null variables is included in the message instead. Figure 4 shows output for the singlevariable situation.

4 String nullstring = null; Integer i = 1; Integer j = 2; nullstring.substring(i, j); java.lang.nullpointerexception: In file Junit3Test.java on line 22, which reads: nullstring.substring(i, j); It appears that the code was trying to call a method or refer to a member variable on an object called "nullstring", which is null. Make sure the variable has been initialized in your code. Remember, declaring the variable isn't the same as initializing it. You may need to initialize the object using the keyword "new". This error is called an NullPointerException. at Junit3Test.testNull(Junit3Test.java:22) Figure 4: An example of a NullPointerException. Exceptions in initialization can be caused by data structure variables outside of the declared bounds. The simplest data structure that suffers from this error is the array. When ExceptionDoctor encounters an ArrayOutOf- BoundException, it attempts to narrow the exception cause down to three common mistakes. ExceptionDoctor can identify errors caused by negative array indexes, accessing an element in an unpopulated array, and using an array index that is greater than the array size. Object[] tenelementarray = new Object[10]; Object foo = tenelementarray[15]; java.lang.arrayindexoutofboundsexception: In file Junit3Test.java on line 27, which reads: Object foo = tenelementarray[15]; It seems that the code tried to use an illegal value as an index to an array. The code was trying to access an element at index 15 of the array called "tenelementarray". The size of the array may be less than 15. Keep in mind that if the array size is N, the biggest index you can access is N-1. This error is called an ArrayIndexOutOfBoundsException. at Junit3Test.testBadIdx(Junit3Test.java:27) Figure 5: An example of an ArrayIndexOutOfBoundException ExceptionDoctor handles an ArrayIndexOutOfBounds- Exception with only the source code line that caused the exception and the value of the index. Using only these two pieces of information, ExceptionDoctor only gives the student hints on the exception s cause based on its best guesses about the array. Figure 5 shows an array with ten elements contained and an attempt to access element fifteen, a non-existent element. Novice programmers also have issues with variable types. Variable type confusion would most likely surface in the form of a NumberFormatException. Number- FormatExceptions occur because of the different precision number types available in the Java syntax. Students might find difficultly when understanding the difference between an integer and a float. When ExceptionDoctor encounters a NumberFormatException, it attempts to parse the string multiple times using different number parsers to determine the actual format of the string. For example, Figure 6 shows the output from an attempt to perform an improper parse of a float as an integer. ExceptionDoctor attempts to parse the string as a float and reports the correct number format of the string as a float. Integer.parseInt("3.59"); java.lang.numberformatexception: In file Junit3Test.java on line 31, which reads: Integer.parseInt("3.59"); It seems that the code wants to convert a String to an integer. However, the String "3.59" appears to be a floating point value, not an integer. You may want to use a different datatype to store the value, like float. This error is called an NumberFormatException. at Junit3Test.testParse(Junit3Test.java:31) Figure 6: An example of a NumberFormatException. 5 Availability One of the key features of ExceptionDoctor is its ease of integration into a variety of different tool chains using JUnit. ExceptionDoctor is distributed as a single JAR file. Once added into a project s classpath (ahead of JUnit, if JUnit is also being used), ExceptionDoctor will automatically configure itself. Uncaught exceptions in JUnit tests (3.x or 4.x) are enhanced without any additional effort required of the student. Enhanced exception messages appear in the IDE s normal output mechanisms, whether that is a simple console window or a fancier GUI results pane (ex. Eclipse), since ExceptionDoctor does not produce any output of its own and simply redecorates the message contained within a regular Java exception object. This automatic method for integration is applicable to the Eclipse IDE. For BlueJ users, we provide a second distribution consisting of two JAR files. Adding both JAR files to BlueJ s existing lib directory is all that is required for installation. With this change, all student code run within BlueJ whether it is invoked interactively through the mouse using the object bench, is initiated by code typed into the code pad, is a stand-alone program started from a

5 class main method, or is run as a JUnit test will automatically receive ExceptionDoctor explanations on all uncaught exceptions. 6 An Evaluation of Effectiveness Murphy et al. performed an analysis of the effectiveness of the messages produced by the ExceptionDoctor framework.[4] In their study, they tested 17 students (8 male and 9 female) ability to fix a runtime error in provided source code. While solving the runtime exception, thirteen students were given ExceptionDoctor like output to aid their work. Of these students, 76% were able to find the cause of the exception within eight minutes. When asked if the messages were helpful, 13 (100%) students responded yes. Two of the students who did not complete the task said that the exception message mislead them. However, they also admitted they had not closely read the message. The four remaining students were given the same task, without the aid of ExceptionDoctor messages. These students were the highest performing members of the class and easily found the cause of the error within five minutes. Afterwards, when shown the ExceptionDoctor messages, three of the four students said that the ExceptionDoctor message would have helped them discover the cause of the error. One student felt the message would have not helped; she claimed that the message was too long (One of the reasons we reorganized the information presented to the students). Our research results do not focus on the effectiveness of the messages produced by ExceptionDoctor since their effectiveness has been proven by Murphy et al.[4] We instead focus on the tools effectiveness in identifying exceptions that students normally encounter. The code used as the basis for this evaluation consisted of all student program assignments submitted for a full semester of a CS2 course. Student work was collected using an automated grading system called Web-CAT [7]. Students were allowed to make unlimited submission attempts, and were required to write their own JUnit tests to demonstrate that their code behaved correctly. All submissions from all students that is, both work in progress and final solutions were used for this analysis. While the majority of student code did not produce runtime errors, there were a total of 465 uncaught exceptions produced by tests that students wrote, and another 4,053 uncaught exceptions produced when instructor-written reference tests were run against the student solutions for grading purposes during the course. Figure 9 shows the distribution of exception types produced by student-written tests. Of 465 uncaught exceptions, ExceptionDoctor provided useful explanations for 297 (64%). An additional 9 exceptions (2%) actually arose in library classes called by student code, rather than directly inside student code in these cases, ExceptionDoctor pinpointed the location in the student code where the library call originated, but was unable to add information to the explanation because library source was unavailable. ExceptionDoctor failed to provide explanations for the remaining exceptions (34%) because it had no appropriate handler to provide an augmented explanation (other than the default handler for any Throwable). Of these, 136 (29%) were due to user-defined exceptions specific to a particular assignment. The remaining 23 exceptions (5%) do not show up as common for students in earlier studies. Additional handlers could be written if desired. When student submissions were being more rigorously exercised by instructor-written tests, an order of magnitude more uncaught exceptions were produced. Of 4,053 exceptions, 3,882 were successfully explained (96%), and an additional 43 (1%) were due to exceptions occurring inside library code called by student code. Figure 8 shows the distribution of exception types encountered. Overall, ExceptionDoctor performed extremely well. Exceptions Arising in Instructor-written Tests NullPointerException NumberFormatException ClassCastException OutOfMemoryError ArrayIndexOutOfBounds NoSuchElementException IllegalArgumentException StackOverflowError StringIndexOutOfBounds ArithmeticException EmptyStackException AccessControlException Explained No Handler In Library Code Figure 8: A graph of exceptions found and handled in tests written by instructors on code written by students. These results are skewed towards exceptions that students could not debug themselves. Students also choose when to submit to Web-CAT for grading. Because they wrote their own tests and worked to make them pass, the exceptions occurring during grading have many of the simpler exceptions already removed. Further, problems that students could not resolve would show up repeatedly in multiple subsequent submissions, further skewing the frequency distribution.

6 Exceptions Arising in Student-written Tests NullPointerException PSEmptyStackException PSIllegalArgumentExcept StackOverflowError NumberFormatException ClassCastException FileNotFoundException MalformedURLException NoSuchElementException IllegalArgumentException ArrayIndexOutOfBounds PSArgumentMissingExce IllegalMonitorStateExcept UnsupportedOperationEx ArithmeticException StringIndexOutOfBounds Explained No Handler In Library Code Figure 9: A graph of exceptions found and handled in tests written by students on their own code. 7 Future Improvement ExceptionDoctor is easy to install with minimal overhead to use. In the current release, handlers are provided for exceptions that beginning programmers generally cause. However, the library of exception handlers could be improved over time to handle a broader group of exceptions. The algorithms used by ExceptionDoctor could also be improved. The current version of ExceptionDoctor is limited to inspecting the single source line referenced by the original exception. To improve this, ExceptionDoctor could examine a larger portion of the source code looking for variable assignments or declarations. ExceptionDoctor also struggles to identify exceptions thrown within library code. In these circumstances, the rewritten exceptions must contain messages indicating that a parameter to the library method resulted in an exception. In future research, we hope to design messages that teach students strategies for resolving exceptions they have caused indirectly. ExceptionDoctor could also be improved through use of the Java runtime debugger. It might be possible to run the test concurrently in the Java debugger in an attempt to get real-time information about variable contents and execution information, rather than solely by examining the source code. Further research needs to be conducted on student use of ExceptionDoctor. Plans have been made to integrate ExceptionDoctor into Web-CAT s Java grading plug-in so that all Web-CAT users can benefit from it. By logging exceptions that occur, more data on actual errors can be collected. Surveys could also gauge the helpfulness of exception message improvements in the eyes of students. 8 Acknowledgements This work is supported in part by the National Science Foundation under Grant No. DUE Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. 9 REFERENCES [1] JUnit home page, [2] Eclipse home page, [3] M. Kolling and J. Rosenberg, BlueJ home page, [4] C. Murphy, E. Kim, G. Kaiser, and A. Cannon. Backstop: A tool for debugging runtime errors. In Proc. 39th SIGCSE Technical Symp. Comp. Sci. Education, pages , [5] M. Jadud. A first look at novice compilation behavior using BlueJ. In Proc. 16th Workshop of the Psychology of Programming Interest Group, [6] C. Norris, F. Barry, J. B. Fenwick Jr, K. Reid, and J. Rountree. ClockIt: Collecting quantitative data on how beginning software developers really work. In Proc. 13th Conf. Innovation and Technology in Computer Science Education (ITiCSE), [7] Web-CAT home page,