Application Saver 1.2

Transcription

1 VERITAS Application Saver 1.2 Forensics Guide For Java N12025C

2

3 Table Of Contents VERITAS Application Saver Forensics Guide...1 Java Platform Edition...1 Using the Guide...3 Typefaces...3 Introduction...5 Overview of the forensics workflow...5 Features...5 Instrumentation Software...6 Forensics Viewer...6 Host Collector Handling of Forensics Files... 6 Compatibility...6 System requirements...6 Instrumentation...9 How to instrument Java bytecode...9 Static instrumentation...9 Dynamic instrumentation...11 Using instrumentation options...11 Syntax of the tfj-instr command...11 Generating traces at runtime...17 What are forensics files?...17 How forensic files are generated during runtime...17 Running instrumented programs...18 Exceptions: policy snaps...20 Manually generated files: external snaps...22 By an API call: programmatic snaps Configuring forensic settings and runtime options...24 OPT file settings...24 appsaver.conf settings...26 User-specific options file...27 For all processes in a group: group snaps...27 Using the Forensics Viewer...29 i

4 Forensics Guide for Java How to open the Forensics Viewer...29 Forensics Viewer interface...29 Docking windows...30 Workspace window...31 Trace history window...31 Source window...32 Output window...33 Data and stack window...34 Thread window...35 Chart window...35 Trace information window...37 Viewing forensic files...39 Viewing detailed information with the trace...39 Opening forensics files...40 Closing trace files...41 Initial position of the trace...41 Information captured in the trace...42 Navigating the trace and source windows...43 Changing the trace view...45 Working with multiple copies of the same trace...45 Filtering and sorting traces...46 Exporting trace results...52 Printing a trace...52 Analyzing trace results...52 Viewing general system information...52 Checking stack, data, and memory Combining trace files...55 Trace buffer size and threads in trace...56 Reproducing traces...57 Version control and viewing traces...57 Toolbar icons & shortcuts...59 Glossary...63 ii

5 Table Of Contents Index...67 iii

6

7 VERITAS Application Saver Forensics Guide Java Platform Edition The information contained in this publication is subject to change without notice. VERITAS Software Corporation makes no warranty of any kind with regard to this manual, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. VERITAS Software Corporation shall not be liable for errors contained herein or for incidental or consequential damages in connection with the furnishing, performance, or use of this manual. Copyright (c) VERITAS Software Corporation. All rights reserved. VERITAS, the VERITAS Logo and all other VERITAS product names and slogans are trademarks or registered trademarks of VERITAS Software Corporation. VERITAS and the VERITAS Logo Reg. U.S. Pat. & Tm. Off. Sun Solaris, and Java are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. or other countries. Other product names and/or slogans mentioned herein may be trademarks or registered trademarks of their respective companies. VERITAS Software Corporation 350 Ellis Street Mountain View, CA Phone Fax

8

9 Using the Guide VERITAS Application Saver has five guides: Installation and Administration - This guide will help Administrators install and configure the product. User's Guide - This guide will help Operators use the console and troubleshoot problem applications. Forensics Guide for Windows - This guide will help someone in professional services, development, or support to troubleshoot code in C/C++ applications running on Windows. Forensics Guide for Solaris - This guide will help someone in professional services, development, or support to troubleshoot code in C/C++ applications running on Solaris. Forensics Guide for Java - This guide will help someone in professional services, development, or support to troubleshoot code in Java applications running on all platforms supported by Application Saver. Typefaces Most of the text is in the font of this sentence. Programming language, filenames, command line text, function calls or parameters of a function are shown in the font of this sentence. 3

10

11 Introduction The Forensics module provides source code line-level tracing for applications developed in Java. By monitoring program execution and system information, the Forensics module pinpoints the exact source of the failure, helping you to rapidly recover from the problem, minimizing debugging time, and eliminating the need to re-create problems. This section covers: The basics of using the Forensics module Key features of this component of Application Saver System requirements Overview of the forensics workflow The Forensics module works by automatically inserting lightweight agents into Java bytecodes through a process called instrumentation. As the instrumented program runs, the Forensics module continuously monitors it, capturing relevant system information and the exact sequence of statement execution across all threads. Then, when an exception, error or termination occurs in the instrumented application, or through an external command at any time, the Forensics module generates forensics files that capture a detailed history of source statement execution. You can then use the forensics files to see exactly what led to the failure and quickly fix the problem without needing to re-create the failure in a test environment. Using the Forensics module involves the following major steps: 1. Instrument class and/or archive files such as JARs (Java archives), EARs (Enterprise archives), and WARs (Web archives). 2. Run the instrumented code and generate forensics files. 3. Analyze the forensics files using the forensics Viewer, find the source of the problem, and fix it. Details regarding these steps are covered throughout this guide. Features The Forensics module includes these main components: Instrumentation software Forensics Viewer Host Collector Handling of Forensics Files 5

12 Forensics Guide for Java Instrumentation Software The instrumentation software runs from the command line and inserts lightweight Java agents into class and/or archive files, creating new, instrumented versions. Forensics Viewer The Forensics Viewer is the graphical user interface (GUI) software used to open and analyze forensics files generated from class and/or archive files. The forensics files provide details about what happened while the instrumented module was running, including: The exact sequence of statements in all active threads Memory, stack, and variable information (if a dump file is available) Hardware and operating system information The time the forensics file was created Use the forensics file results to identify why any failures or abnormal behaviors occurred in your program, such as which thread is causing a lock in the system or which line in the code led to an exception. You can also view forensics results for normal operation of instrumented files. Host Collector Handling of Forensics Files If the Forensics module is enabled, the Host Collector on an enabled EndPoint can send the generated forensics file to the FocalPoint (or any other remote server). The Host Collector can be configured to then remove the files from the local host. The Host Collector can also compress trace files into one zipped file and deliver trace files automatically via (Windows only) or FTP. Compatibility Applications instrumented for Forensics continue to be able to be debugged with conventional debuggers. System requirements The Forensics module is designed to instrument and monitor Java applications in an environment where a version or later compliant Java virtual machine (JVM) is in use. The Forensics module only requires a Java Runtime Environment (JRE), but can also be used with a Java Software Development Kit (SDK). The Forensics Viewer graphical user interface requires a version 1.3 or later compliant JVM, regardless of which JVM version is being used to execute forensic-instrumented applications. Application Saver installs a private version of the version 1.4 JRE for this purpose. The Forensics module is designed for use with the Java 2 Platform, Standard Edition and Enterprise Edition (J2SE and J2EE). No support is currently provided for the Java 2 Micro Edition (J2ME). The Forensics module can be used with stand-alone Java applications, and J2EE components such as applets, JSPs, servlets, and Enterprise Java Beans (EJBs). In some environments (e.g., applets), 6

13 Introduction special permissions must be granted before using the Forensics module so that it is able to create disk files that contain application tracing information. Installation of the Forensics module requires approximately 39MB of disk space. Memory requirements within a JVM will increase when Java applications are instrumented with the Forensics module. The Forensics Viewer graphical user interface typically requires 30 to 100MB of memory, within a JVM, depending on the complexity and size of the instrumented application under review. Performance packs for specific operating systems are installed with the Forensics module and will automatically run to improve the performance of instrumented code. The performance packs may not work with all combinations of platform and JVM versions. 7

14

15 Instrumentation Instrumentation is the process that inserts agents into your code. These allow the Forensics module to capture the information needed to pinpoint the cause of exceptions, terminations, and other abnormal behaviors in your program. This section covers: How to instrument class and/or archive files such as JARs (Java archives) The different command options and instrumentation settings How to instrument Java bytecode There are two ways to instrument Java code: statically and dynamically. Static instrumentation is a manual command line operation that allows you to instrument class and archive files individually, while dynamic instrumentation allows you to automatically instrument a Java program upon launching it. Note that collector/lib needs to be added to the LD_LIBRARY_PATH for Java. Static instrumentation Java code must first be compiled into bytecodes before it can be instrumented. If you will be using the programmatic snap APIs for generating forensics files, you will need to include the Java runtime agent jar on your classpath. In this case, when compiling, consider using the tfjjavac command found in the bin directory distributed with the Forensics module to ensure your CLASSPATH is correctly set. If you prefer to invoke javac directly, then be sure to inspect the tfjjavac script and add any necessary information to your CLASSPATH. If you wish to use the programmatic snap API, you will need to provide access to the runtime JAR that contains that API when compiling. Add the following directory to the class-path for javac: <install-dir>/products/appsaver/collector/jar/tfj-runtime.jar To add a JAR file to the class-path of a java command, you can either add the file to a '-classpath' command-line argument (sometimes abbreviated as '-cp'), or add it to a CLASSPATH environment variable. You must have source file and line debug information in your classes for them to be eligible for forensics instrumentation. This is true by default with javac, and normally only disabled with the - g:none flag. Compile with the -g flag if you want the Forensics module to report complete dump files (including local variable names). There is an instrumentation option (-s) that you can use after compiling with - g to remove line number tables and local variable names in instrumented code. See Using instrumentation options for more details. You can statically instrument class, JAR, and compound JAR files, such as EAR and WAR files. To instrument a file, type the tfj-instr command, the desired instrumentation options, and the file name at the command line. To instrument for forensics, add the -forensics or -full options. Follow this general format: 9

16 Forensics Guide for Java tfj-instr [-option] <filename>.<extension> NOTE: You will either need to change to the directory containing the file you wish to instrument, or you will need to use the full path to the file name with the instrumentation command. The examples in this chapter assume that you are already positioned at the current directory. Type tfj-instr -h for a complete description of available options, which are also explained later in this section. EXAMPLE: To instrument sample.class without using any instrumentation options, type: tfj-instr sample.class or to instrument sample.jar, type tfj-instr sample.jar The default behavior of the tfj-instr command is to overwrite the existing file in the current directory with the instrumented version, leaving the original as a.bak file. Therefore, if -forensics or -full instrumentation options are used, the following files will be created upon instrumentation: An instrumented version of the file.tj1 file (map file).bak file (backup of the original file from which the instrumented version as created) EXAMPLE: Instrumenting sample.class without any instrumentation options results in: a new, instrumented version of sample.class sample.tj1 Note: If the tfj-instr command detects a.bak file, the command will not instrument the file. If you are creating the instrumented file in the same directory as the original, and a.bak file has previously been created, you must delete or rename the.bak file. Use wildcards (*) to instrument multiple files in the same directory at once. Use the -r instrumentation option to recursively traverse directories while instrumenting files. See Using instrumentation options for more details. On Windows, there is also a tfj-instrw.exe variant that is a GUI application, so it will run cleanly from another GUI application, such as an IDE. 10

17 Instrumentation Dynamic instrumentation The Forensics module includes a class loader that automatically launches your Java program and instruments all associated class files. This is a particularly helpful feature if you are working in a development environment because you don't have to perform instrumentation as a separate step. NOTE: The class loader currently does not operate correctly in a J2EE environment and will prevent the use of the launcher in certain other situations where non-standard class loaders are in use. Classes starting in javax and java will not be instrumented. To use the Forensics module launcher and dynamic instrumentation tool: 1. Compile the class file. The tfj-javac script is available as an alternative to compiling with javac. This will set your CLASSPATH. 2. Type this at the command line: tfj-ijava <ClientClass> where <ClientClass> is the name of the class file you want to instrument and open. Using the tfj-ijava script will also ensure the CLASSPATH is set properly. If you'd rather dynamically instrument the class without using this script, type the following at the command line, with a CLASSPATH setting that includes tfj-runtime.jar: java incert.instrument.launcher <ClientClass> NOTE: On Unix platforms, tfj-ijava is a script that does not currently support the full set of JVM options (such as the -cp flag). If you want to use specific flags, copy the necessary CLASSPATH information from the tfj-ijava script into your script. On Windows, you can pass -D and -X JVM options as well as -cp/-classpath. All class files contained within <ClientClass> will be instrumented automatically while they execute. Map (.tj1) files are saved into a directory hierarchy. Whereas static instrumentation produces a new set of class files, dynamic instrumentation eliminates the need for maintaining both an uninstrumented and instrumented code base. However, the dynamic instrumentation launcher is not recommended for production software because instrumentation occurs every time you launch the application and will affect performance. Using instrumentation options The following options are also available for instrumentation. Option flags are case sensitive. Syntax of the tfj-instr command tfj-instr [-v] [-h] [-metrics forensics] [-full] [-q] [-qi] [-s] [-nomanifest-update] [-d <out-dir>] [-label <label>] [-log <log-file>] [-p 11

18 Forensics Guide for Java 12 <policy.opt>] [-j <map-file.jar>] [-sourcepath <source-path>] [-batch <batch-file>] [-bs <backup-suffix>] [-ts <MMddyyyyHH:mm:ss>] [-tf <MMddyyyyHH:mm:ss>] {-r <dir>} *.class... *.jar... Instrumentation options: -v Print out version information and exit. -h Print out help message and exit. Instrumentation options: -metrics Instrument for metrics only (default). If neither - metrics or -forensics is specified, it will be instrumented for metrics only. -forensics -full Instrument for forensics only. Other options: Instrument for all features; i.e., both metrics and forensics -q Suppress most instrumentation progress output. -qi -no-manifest-update If you do not use the -q (quiet) option, a progress message will appear for each class that is not instrumented and each JAR file as it is instrumented. This can be distracting if you are instrumenting several files at once. Use -q to minimize the number of messages presented. Suppress already-instrumented output. -s Strip line number tables and local variable tables from output. This option is useful if you've compiled the code with -g and wish to then remove some of the added information needed for instrumentation. Do not alter archive manifests of instrumented archives (required by Weblogic 6.1 SP 2 and above). -d <out-dir> Place instrumented classes/jars and.tj1 files into a directory tree starting at <out-dir>. Defaults to the current working directory. Use this option if you do not want to overwrite existing class and JAR files with the new instrumented version. If you specify a directory that does not exist, this option will create it. Example: To instrument sample.jar and save it to the

19 Instrumentation -label <label> -log <log-file> myapp directory on the same drive, type: tfj-instr d myapp sample.jar Associates class file with a given string label. Log instrumentation progress in <log-file>. Create a log file of detailed instrumentation progress. The log file is saved to the output directory. You must specify a name of the file, which can be any format. Example: To save a log file when instrumenting webproject.jar, type: tfj-instr log webproject.log webproject.jar Details about instrumenting webproject.jar are saved in webproject.log. -p <policy.opt> Instrument classes and jar files according to the policies found in the <policy.opt> file. Instrumentation within a class can be set to the following three levels: none: No methods in the class will be instrumented. large methods: Only large methods in the class will be instrumented. all: All methods in the class will be instrumented. To use the -p (instrumentation policy) option, you must first create an.opt file containing the following line(s): instrument.class.<classname> <instrumentationlevel> where <classname> is the class defined in your Java code and <instrumentationlevel> is none, large methods, or all as previously defined. <classname> can end with.* to indicate all classes within a package. You may use = or : as well as a space separator. Example: Within webproject.jar, you want to set up the following instrumentation policies: do not instrument any methods in classone, 13

20 Forensics Guide for Java 14 instrument large methods only in classtwo, and instrument all methods in classes in package. 1. Create an.opt file called webproject.opt with the following lines: instrument.class.classone none instrument.class.classtwo large methods instrument.class.package.* all 2. Then to instrument webproject.jar using the instrumentation policies defined in webproject.opt, type: tfj-instr p webproject.opt webproject.jar Note: If -p is not in use, then by default classes starting in java.* will be excluded from instrumentation. -j <map-file.jar> Places mapfiles in <map-file.jar>. -sourcepath <source-path> -batch <batch-file> Place all map files (.tj1) created during instrumentation in a single JAR file in the output directory. You must indicate a name for the JAR (which must end in.jar). Example: To save the map files associated with instrumenting webproject.jar into a file called webmapfiles.jar, type: tfj-instr j webmapfiles.jar webproject.jar Any.tj1 files produced during the instrumentation of webproject.jar are saved to webmapfiles.jar. The map files are organized in a directory hierarchy that matches the package hierarchy of the instrumented classfiles. Note: Each time this option is used, a new map file JAR is created, overwriting any existing map file JARs with the same name. If you want to preserve existing map file JARs, be sure to give the JAR a new name. Stores given path to Java source files in mapfiles. Specifies a file containing a batch of files to instrument, one filename per line. This is useful when there are many files to instrument,

21 Instrumentation -bs <backup-suffix> -ts <MMddyyyyHH:mm:ss> -tf <MM dd DD yy yyyy HH hh mm ss> and/or the length of the command line is limited. Specifies a suffix for backing up original class files (default is ".bak"). Instrument only those classes modified after the specified time. Pass an alternate time format to use with '-ts', suitable for the Java SimpleDateFormat class. -r <dir> Instrument all class and jar files in and below <dir> (recursive instrumenting). This option is helpful if you want to instrument an entire Java application in one pass. Example: To instrument all class and JAR files found in the webproject directory (which includes several subdirectories), type: tfj-instr r webproject All the class and JAR files in the webproject directory will be overwritten with new, instrumented versions. TJ1 (map) files will be created in the same subdirectory as each instrumented class and JAR file, unless you use the -j option. Note: This option does not work with dynamic instrumentation. 15

22

23 Generating traces at runtime After instrumenting modules, the next step is running them in a development, test, or production environment. Application Saver can monitor the instrumented application for exceptions, errors, and other abnormal behaviors and produce forensics files that you can use to identify what went wrong. In addition to continuous exception monitoring, you can also control when forensics files are generated with a command line utility or with an API. And, you can further customize what the Forensics module can monitor and how it handles traces with the configuration files. Plus, you can generate forensics files for all Java class and JAR files running simultaneously with the same group name. This section covers: What are forensics files How forensics files are generated during runtime Running instrumented programs Configuring forensics settings and runtime options What are forensics files? Forensics files are a set of the following files: Forensics history file (.tjh) containing the detailed statement execution history. Forensics information file (.tjx) containing data captured at runtime, such as system properties, the time the file was generated, and runtime trace settings in place. An optional dump file (.tjd) used for viewing memory, stack, and variable information. Forensics files are saved in the results directory as defined by runtime.resultsdirectory in the default.opt configuration file (see Using configururation files to configure trace settings and runtime options ) or a custom.opt file. By default, the results directory is set to /tmp on Unixc:\winnt\temp or c:\windows\temp on Windows. The forensics file naming structure depends on how the trace was generated. Forensics files can also be compressed and saved into a single zip file (.tjz). How forensic files are generated during runtime Once an application has been instrumented, you can then run it as normal. Application Saver continuously monitors the runtime behavior of the instrumented class and JAR files, tracking source statement execution across all threads and general system data. This information can then be captured in forensics files and analyzed using the Forensics Viewer. There several main ways to generate forensics files: 17

24 Forensics Guide for Java You can configure the Forensics Agent to generate forensics files when certain exceptions are thrown in Java code, native code called from JNI, or the JVM or the process makes certain types of memory-handling errors. The instrumented application will continue to run, but the Forensics module will provide a snapshot of what was in the trace buffers at the moment the exception occurred. This is called a policy snap. You can manually generate forensics files at any time via a command line utility. This is called an external snap. You can use an API to programmatically set forensics file generation in your application. This is called a programmatic snap. In addition, to these primary methods of trace generation, you can also configure the Forensics module to always produce trace files, even if no problems are encountered during runtime. Read the following sections for details on each category of forensics file generation: Running instrumented programs Exceptions: policy snaps Manually generated files: external snaps By an API call: programmatic snaps For all processes in a group: group snaps Configuring forensics file settings and runtime options Running instrumented programs There are several recommendations for running instrumented Java programs. To launch an instrumented Java program, consider using the tfj-java command found in the bin directory distributed with Application Saver to ensure your CLASSPATH is correctly set. To make tfj-runtime.jar available, do the following: 1. Put tfj-runtime.jar in the JRE's extension directory. 2. Put it on the classpath. Either: a. Set or add the CLASSPATH environment variable OR b. Set or add to the -classpath Java command-line option, often abbreviated as -cp NOTE: On Unix platforms tfj-java is a script that does not currently support the full set of JVM options (such as the -cp flag). If you want to use specific flags, copy the necessary CLASSPATH information from the tfj-java script into a script of your own. On Windows, you can use -X and -D JVM options as well as -cp/-classpath. 18

25 Generating traces at runtime External snapping requires that you assign a port number to the executing instrumented application. If you plan on being able to externally generate a trace file for your program, use this command when launching the class or JAR: tfj-java Dtraceback.runtime.port=<portnumber> <instrumented file name> or if you don't wish to use the tfj-java script: java Dtraceback.runtime.port=<portnumber> <instrumented file name> where <portnumber> is the arbitrary port you choose for your instrumented program to monitor. You will then need to use this port number with the external snap command. Typically, usable port numbers are in the range of 1024 to Each concurrently executing instrumented application must use a different socket number. EXAMPLE: To assign a port and launch the instrumented webproject.jar, type: tfj-java Dtraceback.runtime.port=1234 webproject.jar You would then need to use port 1234 along with the external snap command as described later in this chapter. You may assign a tag name to the instrumented Java module to allow for group snaps. There are two ways to do this: either edit the runtime.runtimetag option in the.opt file used by your application or assign the name every time you launch the instrumented module. To do the latter, type: java Dtraceback.runtime.tag=<tag name> <instrumented file name> where <tag name> is the runtime tag you would like to use. Since this tag name needs to be supplied each time you run the instrumented application, it is recommended that you add this to a launch script (as well as other settings described in this section). See Using configuration files to configure trace settings and runtime options in the Installation and Administration Guide for details on creating and using.opt files. Note that by default a Java application will identify itself to the Collector as a process named java. If you have multiple Java programs you may wish to assign them different process names so they can have different.opt files. To do so, you may use the traceback.runtime.processname system property to pass a different name to the Collector. For example, if you would like to assign a specific.opt file to a webproject process, you might set java Dtraceback.runtime.processName=webproject webproject.jar and then in your appsaver.conf file specify 19

26 Forensics Guide for Java runtime.options=webproject=webproject.opt If you use the -jar flag to launch an instrumented Java program, the Forensics module runtime software will not be found in the CLASSPATH. Therefore, if you want to use the -jar flag, you will need to either: Add everything in the tfj-runtime.jar file (located in the <installdir>/jar directory) into your jar, or Install the tfj-runtime.jar file as an extension JAR. For more details, see extensions/spec.html A script is available to quickly remove all the trace files (.tjh,.tjx, and.tjd, or.tjz) generated from instrumented Java programs. At the command line, go to the directory with the files you wish to delete and run the tfj-clean script found in the INSTALL_ROOT/products/appsaver/forensics/examples/scripts directory. This script will delete all files with.tjh,.tjx,.tjd, and.tjz file extensions in that directory. Class, JAR, and map (.tj1) files will not be deleted. Exceptions: policy snaps By default, the Forensics module will produce no forensics files except when a policy or java.runtime.saveonexit are set. You can control whether additional files and dump files are generated for exceptions by establishing your own policy for individual exceptions. This is controlled by.opt files. Forensics files for exceptions have the following naming structure: gs.host.process.time.pidx.snap-nn.extension EXAMPLE: The instrumented Java program myapp is running and has just thrown a java.lang.nullpointerexception, which has an exception policy of Snap+Dump. The following files are created: gs.host.myapp pid145.snap-1.tjx gs.host.myapp pid145.snap-1.tjh gs.host.myapp pid145.snap-1.tjd Where: gs is an Application Saver identifier HOST is the name of the computer myapp is the name of the instrumented application is the date and time the process ran, formatted as YYYY-MM-DD- HH-MM-SS in local time 20

27 Generating traces at runtime 145 is the process ID number for the process running when the exception was thrown snap-1 is the number of policy snaps that occurred since the process started Traces for pure Java code Policy snap traces are controlled by settings in.opt files. By default, the Forensics module will ignore all Java exceptions. To configure the Forensics module to generate forensics files or forensics files plus a dump file, use the following line, which may be commented out, in default.opt (or a custom.opt file): java.runtime.exception.<javaexceptionname>=<policy> where <javaexcetionname> is the exception name, in this case all exceptions, and <policy> is one of three parameters: IGNORE - don't generate a forensics file SNAP - generate a forensics file SNAP+DUMP - generate a forensics file plus a memory dump. Note: The dump is most useful, however, if you compiled the Java source files with the -g flag. EXAMPLE: By default, the Forensics module ignores all Java exceptions with the following line: runtime.exception.default=ignore To generate forensics files for all Java exceptions, change ignore to snap: runtime.exception.default=snap To generate forensics files plus a dump file when java.lang.nullpointerexception is thrown: runtime.exception.java.lang.nullpointerexception=snap+dump Enter a new line in the.opt file for each exception you to monitor. The Forensics module will generate forensics files only the first time an exception is thrown from the same location in the source code. To change this, edit the policy setting: runtime.userperexceptionpolicylimit=1 If you re finding that the dump file generation is extremely slow or the dump files are extremely large, consider tuning the level at which array and object information is captured. See the runtime.dumpdepth and runtime.dumpbreadth options. 21

28 Forensics Guide for Java JVM and JNI traces For pure Java applications, the Forensics module can generate forensics files when an exception or signal occurs in the hosting JVM. If your application uses JNI to invoke native code, you can also capture both Java and native forensics files from a single application. Then, when viewing, you can open both traces in one Forensics Viewer and use the Combine Traces option to view a single history of the execution into and out of JNI and Java code. The native runtime agent can trigger snaps into Java on all platforms, but will only provide additional native forensics information on the platforms supported for native forensics, namely Windows and Solaris. To enable the native runtime to notify the Java runtime of exceptions or signals in native code, you must notify the Java runtime of the presence of the native runtime. For Windows, you must add Dtraceback.nativelib=ncrtrt to the Java process startup; for Unix, you must add Dtraceback.nativelib=incert to the Java process startup. To specify which exceptions should generate forensics files in the JVM or native code running on Solaris, edit the following line in default.opt (which is commented out, by default) or your own custom.opt file. native.runtime.usersignalpolicies=sigsegv=snap+dump; SIGPWR=SNAP;SIGHUP=IGNORE To specify which exceptions should generate forensics files in a JVM or native code running on Windows, use the configuration files similarly to edit native.runtime.userexceptionpolicies. See the Windows or Solaris Forensics manuals for more information on these configuration options. Forensics files generated from native code will have a different file extension than Java forensics files. Both Windows and Solaris forensics files have the file extensions of.tbh and.tbx. Windows dump files have a.dmp extension, while Solaris dump files have a.core extension. You can open each forensics file separately in the Forensics Viewer. Or, you can open two instances of the Forensics Viewer and open the C/C++ trace in one, and the Java trace in the other. Manually generated files: external snaps Although the Forensics module will continuously monitor your application for any exceptions that you specify, there may be instances where you want to manually generate a forensics file. You can generate a set of forensics files for an instrumented program at any time by using the external snap utility. This can be particularly helpful when your application is frozen. To run an external snap, type the following at the command line: snap <fileprefix> -<port number> where -<port number> is the port number you assigned to the Java process with the traceback.runtime.port system property (or 7782 by default), preceded by a minus sign, and <fileprefix> is the prefix you want to assign to the name of the forensics files. The naming structure for external snap forensics files is: fileprefix.portnumber.ext For example: 22

29 dhttpd.port7782.tbz You can also snap more than one process at once by using multiple ports, like this: snap <fileprefix> -<port number 1> -<port number> The snap command produces the following trace files in the directory you choose..tjh file.tjx file Generating traces at runtime However, if the Host Collector is running, the files will be compressed into a zip file with the.tjz extension. Regardless of file extension, the name of the file will be the name you specified in the command line plus the port number. When you execute the external snap command, it may take several seconds to a few minutes to save the files depending on the size of the files being compressed. You have the option of turning off compression if you want a faster snap. To turn compression off, type the following at the command line: snap -n <fileprefix> -<port number> You can use the configuration files to turn off external snapping capability as well as other settings. By an API call: programmatic snaps In addition to generating forensics files automatically for exceptions and manually with the external snap command, it is often useful to generate forensics information programmatically at one or more locations within an instrumented application. Programmatic snaps require the insertion of a single line of Java code at the point(s) in the application where a snap is desired. This instruction is: String dosnap(string message); Or, to produce a dump file in addition to the forensics files, use: String dosnapanddump(string message, String name, Object dumproot); When producing a dump, the dumproot object passed will be placed in the.dmp file with the name provided. A search for all objects referred to by dumproot will be performed out to the limits specified by the java.runtime.dumpdepth and java.runtime.dumpbreadth configuration options. The message will be written out into the.tjx files so that, later in the Forensics Viewer, the user can see additional context or information about where or why the snap was created. The dosnap() method will return the name of the.tjh file created or null if the snap failed. In order for programmatic snapping to work, the provided Snapshot.class Java class must be in the CLASSPATH of the instrumented application. Forensics files resulting from programmatic snaps have the following naming structure: gs.host.process.timestamp.portx.user-nnn.ext where user-nnn is the user snap count. Use the following function to disable or enable snapping programmatically: 23

30 Forensics Guide for Java boolean suppressalltracebackoutput(boolean suppress); Configuring forensic settings and runtime options The Forensics module has default settings for both general forensics behavior and specific handling. You can change the default settings and customize how the Forensics module works with your application by using either the default.opt file or creating a custom.opt file. For details on how to work with the configuration files, please see the "Installation and Administration Guide". OPT file settings There are several configuration options defined in default.opt. The following table lists some of the frequently used forensics options. More general settings are documented in the "Installation and Administration Guide". Note: Each setting is preceded with runtime (i.e., runtime.tracebuffersize). In addition, you can further identify the settings by using native.runtime or java.runtime if you use the Java version of the Forensics module. This setting... runtime. TraceBufferCount runtime. TraceBufferSize runtime. SaveIfNormalExit runtime. SaveIfUnhandledSignal runtime. GarbageBufferSize Does this... Sets the number of buffers used to contain the forensics trace during runtime. Each thread is assigned one buffer. The TraceBufferCount multiplied by the TraceBufferSize produces the total buffer size. For example, if the TraceBufferSize=16384 and TraceBufferCount=16, then the total buffer used is bytes. The maximum total buffer size can be 128 MB. The default is set to 64. Sets the internal circular buffer size in bytes per thread. During runtime, the buffer is used to contain the forensics trace. The default is set to The maximum total buffer size (TraceBufferCount multiplied by the TraceBufferSize) can be 128 MB. Indicates whether forensics files should be saved when the module ends normally during runtime. The default is set to false; don't save the files. (Available for native Windows traces only.) Indicates whether forensics files should be saved when unhandled signals are encountered during runtime. The default is set to true; keep the files. (Available for native Solaris traces only.) Sets the desperation buffer size. The desperation buffer is used to contain additional threads when the thread count 24

31 Generating traces at runtime runtime. SuppressSnaps runtime. SuppressProgrammaticSnaps runtime. SuppressPolicySnaps runtime. SuppressExternalSnaps runtime. SaveOnExit runtime. DumpDepth runtime. DumpBreadth runtime. exception.default or runtime. exception.<exceptionname> goes beyond the runtime.tracebuffercount. The default is set to Turns off all snap output. No programmatic, policy, or external snaps will run when this option is set to true. It overrides any individual exception settings. The default is set to false. Turns off all programmatic snaps. When this option is set to true, any snaps programmed into your instrumented software will not run. The default is set to false. Turns off all policy snaps. When this option is set to true, no snaps will occur when exceptions are encountered during runtime. This overrides any custom settings in runtime.userexceptionpolicies. The default is set to false. Turns off all external snaps. When this option is set to true, you will not be able to run the snap command to generate a forensics file, and the process will not participate in group snaps. The default is set to false. By default, the Forensics module will not save a Java forensics file upon exiting the instrumented application. Set this option to true if you want forensics files when the JVM exits. This option specifies the maximum depth of recursion during object dumping. The default value is 5, which will only dump the first five levels of objects, thereby improving the performance of dump file generation and reducing dump file size. This option specifies the maximum depth of array dumping. The default value is 100, which will only dump the first 100 elements of an array, thereby improving the performance of dump file generation and reducing dump file size. This option controls which exceptions the Forensics module will monitor in instrumented Java code. The default setting is for all exceptions (the default parameter) to be ignored. You can change runtime.exception.default (all exceptions) from ignore to either snap (generate a forensics file) or snap+dump (generate a forensics file plus a dump file). Or you can specify ignore, snap, or snap+dump status on a per-exception basis by substituting the exception name for default, i.e. runtime.exception.java.lang. 25

32 Forensics Guide for Java NullPointerException=snap. runtime. UserPerExceptionPolicyLimit runtime. UserTotalExceptionPolicyLimit runtime. UseRealTimeClock Limits the number of duplicate forensics files generated for a given exception. The default is set to 1. Limits the total number of forensics files per execution of the instrumented module. The default is set to Used to indicate whether we should emit real-time clock values, or just virtual clock values; used to aid in synchronizing combined forensics files. Default is "false". appsaver.conf settings The following chart lists the default settings for the Forensics specific options in appsaver.conf. More general settings are documented in the "Installation and Administration Guide". In addition to these settings, all of the settings discussed for use in.opt files can be added to appsaver.conf. Any settings found in appsaver.conf will then be used as the default if no.opt file is available. Also note that the reverse is not necessarily true: some of the settings defined in the following chart cannot be added to.opt files. (The .* and ftp.* settings can be set in tag-specific *.opt files.) For details on proper naming structure and usage of the runtime, native, and Java namespaces, refer to "Structure and precedence of settings in configuration files" in the "Installation and Administration Guide". This setting... runtime.runtimetag runtime.serviceport service.tracehandler .recipients .sentfrom .subject .smtpserver Does this... Processes with the same non-em ptyruntimetag setting will be snapped as a group when any one of them issues a policy or programmatic snap. This sets the TCP port used by the Forensics module. The default is localhost: This specifies the name of an executable to perform forensics file-handling actions. This can be set to ftptrace on Unix or "tracehandler" on Windows to perform the ftp and actions below. No default. List of recipients for forensics file notification. No default. Sender for forensics file notification. No default. Subject for notification. No default. Specifies the hostname of a machine running a standard SMTP mailer. (Windows only) No default. 26

33 Generating traces at runtime .attachtrace .tracemaxsize ftp.servername ftp.directory ftp.username ftp.password Requests that forensic files be attached to the directly. (Windows only) Specifies a maximum number of bytes for attached messages. (Windows only) Destination server for ftp forensics file transportation. No default. Destination directory for ftp forensics file transportation. No default. User name for ftp forensics file transportation. No default. Password for ftp forensics file transportation. No default. ftp.ftpport Specifies the FTP listener port. Default is port 21. ftp.deletetrace Delete files as they are successfully transferred. User-specific options file An options file called.appsaver is read last, and can be used to override the other options. The one exception to this rule is that a setting of runtime.useservice=false in.appsaver will prevent the runtime from attempting to contact the service at all. (i.e., this setting will take precedence over the other options files.) This option file can be used by individual users to override machine-wide settings. On Windows, this file is stored in the User Profile directory, e.g., c:/documents and Settings/USERNAME. On Unix, this file is stored in $HOME/. For all processes in a group: group snaps A group is defined as all running processes containing modules with the same, non-empty, value in the.opt file for runtime.runtimetag. For example, let s say you have five instrumented modules all using runtime.runtimetag=product. All five modules are running and an exception occurs within one of the processes. Forensics files are automatically generated not just for the process that had the exception, but also for all five processes. A forensics file is generated for the process in which the exception is thrown and any other processes running at the same time with the same RuntimeTag value. The naming structure of the forensics file for the process with the exception is as follows: gs.host.process.time.portx.tbz for example: gs.host.webserver port7782.tjz where: gs is an Application Saver identifier HOST is the name of the computer 27

34 Forensics Guide for Java webserver is the name of the module is the date and time the process ran, formatted as YYYY-MM-DD-HH- MM-SS in local time 7782 is the port for the process running when the exception was thrown The naming structure of the forensics file for all other processes running under the same configuration tag is like this: gs.host.process.time.groupn.portx.porty.tjz where PROCESS is the name of the original process. For example: gs.host.webserver group10.port7782.port123.tjz Note that there are two port numbers in this file name. The first port number is the one for the process that had the exception; the second port number represents the process that this forensics file is reporting on. NOTE: Group snap works independently of the external snap feature described previously in this chapter. If you run an external snap, the external snap command will only generate traces for the process IDs you specify in the command line. It will not generate traces for all processes running under the same tag. 28

35 Using the Forensics Viewer Forensics traces are key to discovering the cause of failures in your software. Forensics traces capture the exact sequence of statements executed during runtime, allowing you to identify the cause of signals and other failures in instrumented applications. With the Forensics module, you can use the forensics files generated during the runtime of instrumented modules to check system settings, see what led to an unhandled exception, find the thread that is causing the deadlock, and much more. This section covers: The Forensics Viewer interface Viewing forensics files Analyzing forensics results How to open the Forensics Viewer To open the Forensics Viewer, run traceviewer found in the forensics/bin/subdirectory of the directory where you installed Application Saver. Forensics Viewer interface The interface is divided into the following sections: The bottom section displays the actual trace, in the trace history window. The top right section displays child windows supporting the parent trace window at the bottom. This is where you find general system information, the source code, and stack, data, and memory information, depending on which child window is displayed. The top left section shows a list of files related to the trace and tracks what you have open in the Forensics Viewer. 29

36 Forensics Guide for Java The following sections further explain each window in the Forensics Viewer interface. For an explanation of each toolbar icon and its corresponding menu location and keyboard shortcut, see "Toolbar icons and shortcuts". NOTE: The Forensics Viewer saves the layout from the previous usage, so any windows left open (other than source file windows) the last time you viewed a forensics file will automatically be displayed the next time you open the Forensics Viewer. Docking windows The windows described in the following sections have docking behavior. That is, they can be moved around in the overall user interface, and can also be un-docked as separate main windows. Use the following actions to move windows around: Drag the title bar to move a window around within the overall interface. The outline will appear to snap to positions where the window will re-dock. Double-click the title bar to toggle between docked and un-docked states. 30

37 Using the Forensics Viewer Workspace window The workspace window shows the trace and supporting files currently open in the Forensics Viewer. It provides a map of all instrumented modules and files associated with a forensics trace. Workspace Window You can have more than one trace history file open at the same time. The active trace history file is the file name in bold in the workspace window. To open this window, click the Workspace icon Workspace. on the toolbar or from the View menu, select TIP: Use the right-mouse button while in the workspace window to set the active trace and unload, or close, traces. Double-click a trace file in the workspace window to set the active trace. Trace history window The trace window displays the trace history file, also called a trace. The trace is a record of program history. It lists an exact sequence of all functions called in all threads during the runtime of an instrumented module. The trace allows you to see exactly what was happening when the application was running, and why a failure occurred. 31

38 Forensics Guide for Java Trace History Window Traces can be filtered and sorted several ways. The initial position of the trace depends on how the trace was generated. For more information, see "Initial position of the trace". The trace history window is linked to the source window. As you select a line in the trace, the corresponding line in the source window is highlighted. You can customize the trace history window by changing which columns are displayed, resizing columns, and reordering columns. To hide any of the columns in the trace history window, right-click any column heading and select the item you wish to hide. To show the column, right-click any column heading and select it from the list. To adjust column width, move the cursor to the edge of a column heading and resize it. To change the column order, click a column heading and drag the column to another location in the trace history window. To open this window, click the trace icon on the toolbar or from the View menu, select Trace History 1. Selecting Trace History 2 or Trace History 3 will open copies of the same trace displayed in the Trace History 1 window. This allows you to concurrently view multiple locations within the trace. Only one such window is linked to the Source window (discussed in the next section) at any given time. Source window The source window displays the source code for the instrumented module. The source window is linked to the trace history window. As you select a line in the active trace history window, the corresponding line in the source window is highlighted. 32

39 Using the Forensics Viewer To open this window: Source Window Select File Open, locate the source file, then click Open. It will open automatically when a line in the trace window is selected. Output window The output window tracks what is occurring within the Forensics Viewer. It lists the status of trace history reconstruction and any successes or failures in opening files. The output window can be helpful if you are experiencing a delay in opening a trace history file. Check the messages in the output window to see if the Forensics Viewer is still in the process of opening the files or if a problem occurred. 33

40 Forensics Guide for Java Output Window To change the type of messages displayed in the output window: 1. From the View menu, select User Options. 2. In the User Options window, select Output Window. 3. Drag the bar to the appropriate Logging Level. 4. Click OK. To open this window, click the output icon on the toolbar or from the View menu, select Output. Data and stack window The data window shows the stack, local variables, global variables, and registers for the time at which the trace was captured. Use the data window to look at symbols and get addresses for use with the memory window. 34

41 Using the Forensics Viewer IMPORTANT: A dump file is required to view variables and registers. If a dump file was not generated along with the other forensics files, no variables or registers will be reported for the trace. To open this window, from the View menu, select DataOr click the Data icon on the toolbar. Thread window The thread window displays all the threads that contributed to the trace, including un-instrumented traces. Unlike the Thread List in the Trace Information window, which lists only threads that existed at the time of the snap, the Thread Window lists all threads that existing during the portion of the execution of the application captured in the trace buffers. Thread Window The thread window is used to navigate quickly between threads. For instance, clicking on a thread line will select that thread s stack to be display in the Stack window. Double-clicking will filter the Trace window to that thread. Right-clicking displays a menu of these and other options. To open this window, click the thread icon on the toolbar or from the View menu, selectthread. Chart window The Chart window presents a graphical display of thread activity. 35

42 Forensics Guide for Java Chart Window Each thread has its own line, with the thread ids on the left-hand axis. Time is represented horizontally. Use the real-time clock option to see actual time durations. The interpretation of the colors in the thread bars is as follows: Yellow: indicates that a thread has attached and is known to the runtime, but does not have execution records in the trace buffer. It may have wrapped around, or been kicked out of the trace buffers in favor of another thread. Green: indicates that there are active lines in the trace buffer. Blue: An actual line of trace history. The depth of the blue within the green indicates the depth of the call stack, i.e. how many nested calls are represented. Green with no blue, or zero call depth, may indicate a wait state. Call stack depth becomes more evident as you zoom in. The following commands are available to navigate the screen, both from the keyboard and from a right-click menu: 'i' zooms in on the current center point 'o' zooms out on the current center point 's' scrolls the current trace position into view 'f' zooms such that the whole display fits on the screen 'x' zooms in just along the x-axis 'y' zooms in just along the y-axis In addition you can also zoom in on any rectangle by rubber-banding with the left-mouse button down. 36

43 Using the Forensics Viewer To select a trace position and synchronize with any open history windows, just double-click with the left-mouse button. The Chart and Trace windows are linked, so that navigating in the Trace window will move the cursor in the Chart window. Likewise, double-clicking on a location in the Chart window will position the Trace window at that point. Since the Source window follows the Trace window, it too will be updated when the Chart window is double-clicked. The Chart window also supports tool tips, or small explanatory labels that pop up when the mouse is held stationary at a certain point for more than a second or so. These show the module, function, and line number of the selected point. To open this window, click the chart icon on the toolbar or from the View menu, selectchart. Trace information window The trace information window displays the trace information file. Whereas the actual trace shows the detailed sequence of statements executed for each thread in the program, the trace information window provides general information for further diagnosing of the problem, such as: The reason and time the forensics file was generated System information Thread and memory usage Runtime options in use Trace Information Window 37

44 Forensics Guide for Java The trace information file complements the detailed history reported in the trace history window. It provides key information for low-level troubleshooting. The following categories of information are captured in this file: Summary System Properties Memory Thread Group Policy Settings Dump File Reason the forensics file was generated. Details about the computing environment where the forensics file was generated, including OS type and version, JRE version, JVM version, user name, etc. The total memory available (to the JVM) and free memory left when the forensics file was generated. Details on each thread, reference handler, finalizer, and signal dispatcher. The values set in the.opt file. Name of the dump file (if one exists). Note that traces for C/C++ code will have the categories below reported in the trace information window. Summary Process Information Operating System Hardware Attach Time Runtime Summary information about the trace, including the reason why the trace was generated, the process id of the generating program and th local time at which the process was started. More specific information on the generating process, including the command line, and user and host names. Operating system name and version. Type of computer. Time at which the process was started. More detailed information about the program context at the time the trace was generated, including: Instrumented Module List--A list of modules in the process that were instrumented. Configuration--Application Saver policy settings. Snap List--Reasons that forensics files were generated. Process Information (Thread List, Memory Usage) Loaded Module List--Modules in process at the time of the trace. 38

45 Using the Forensics Viewer Exception Info (Exception Record, Context) Trace Statistics To open this window: Select File Open, locate the tjh or tjz file, then click Open, which will automatically open the.tjx. From the View menu, select Trace Information. For details about analyzing the data in the trace information window, see "Viewing general system information". Viewing forensic files Viewing detailed information with the trace The detailed trace history file is the key for developers to see what led to a failure. The trace history is linked to the source file. As you move through the lines in the trace, you see the corresponding lines in the source file as shown below. 39

46 Forensics Guide for Java Opening forensics files Forensics files consist of the following: Trace history file (.tjh) Trace information file (.tjx) Dump file (.tjd) (optional) Instrumented module (class and/or JAR files) Trace and Source Windows are Linked Depending on how the forensics files were generated, these files may be saved individually, or zipped together in a.tjz file. IMPORTANT: To open the forensics files, several additional files must be present on your computer to successfully reconstruct the trace history. The required files are: 40

47 Using the Forensics Viewer Map file for the instrumented module (.tj1) Source code files for the instrumented module For more information on how to make sure you have the required files, see "Reproducing traces". To open forensics files: 1. From the File menu, click Open. 2. Select the.tjz or.tjh file you wish to view and click Open. This file will be in the default directory unless you changed the ResultsDirectory setting or are viewing the result of an external snap, which will be in the directory specified in the snap command. 3. If prompted for the.tj1, module, symbol, or source files, locate the appropriate file(s) and click Open. The Locate File dialog box will prompt you for the necessary file. Look at the top of the dialog box for the type of file, the name of the file, the version number and any user label you tagged the file with during instrumentation (using the -l option). Use the version number and label information to check out the appropriate version from your source control or archival system. NOTE: If this is the first time you are opening a forensics files, you will also need to perform this step: From the View menu, click Trace History 1. After all necessary files have been located, the forensics files open and are ready for analysis. Several other windows may open along with the trace depending on what windows were left open the last time you viewed a trace. Closing trace files To close a trace file, you need to unload it, which removes it and all associated files from view. To unload a trace file: 1. In the Workspace window, right-click the trace file name. 2. Select Unload Trace from the menu that appears. Initial position of the trace When a trace is first opened, its initial position and view depends on how the trace was generated. If a trace is the result of: An unhandled exception or signal Policy snap, or Programmatic snap, 41

48 Forensics Guide for Java the trace opens in tree view and is filtered on the thread that caused the exception, signal or snap. The line with the exception or snap is highlighted. (A tree view presents a hierarchical view of the trace, with portions of the trace that correspond to a particular function call collapsible.) If the trace is the result of an external snap (also called manual snap) and is single threaded, it opens in tree view with the last line in the trace highlighted. If the trace is the result of an external snap and is multi-threaded, it opens in list view, filtered on the snap entry type. (A list view is a flat, or non-hierarchical view. Multi-thread traces can only have list views.) The last line in the trace is highlighted. Filtering on the snap entry type makes it easier to see what each thread was doing when the external snap was taken. Note that by default, the entry column is not displayed in the trace window. Right-click any column header in the trace window and select Entry type to display it. You can modify the filters initially applied to traces. For more information on filtering, see "Filtering and sorting traces". You can also change the initial view; see "Changing the trace view". Information captured in the trace The trace reports details on all functions called. The function names called while the module was executing are listed on the left of the trace window. The following details for each function are listed on the right by default: Method name Class name Source file Line number Source line Thread Begin time End time Exception Sequence number Entry type Stack depth Logical thread The name of the method called in the program. The name of the class where the method is located. Name of the Java source file. The line number in the Java source file. The line of source code from the Java source file. The name of the thread. ID or base number for the time the event in that line of the trace began (helps you identify overlapping segments). ID or base number for the time the event in that line of the trace finished. The type of exception that was thrown. The order in which the events occurred in the trace. Different lines reported in the trace: root, gap, function call, normal. How deeply nested that method is in the tree. Conceptual flow of control between separate threads and across 42

49 Using the Forensics Viewer the Java and native sides of a JNI call. Note that traces for C/C++ code will have different column names in the trace history window. See Information Captured in the Trace in the Application Saver Forensics Guide for Windows. You can add entry type, begin time, or end time to the trace view or exclude any of the default elements by hiding or showing a column. To hide and show columns: 1. Right-click a column heading (such as Thread ID or Module). 2. Select the element you wish to add or remove. Navigating the trace and source windows Use the arrow keys or the mouse to move through the trace in the trace window. The trace and source windows are linked. As you select a line in the trace window, the corresponding line in the source window will be highlighted. Using Step and Run to Cursor icons The Step and Run to Cursor icons on the toolbar provide a quick way to navigate across functions in the trace and source windows. Click the Step into button or press F7, to step into the next function in the tree. Click the Step over button or press F8, to step over the next function in the tree. Click the Step out button or press F9, to step out of the current function. Click the Step back into button or press Ctrl + F7, to move to the prior trace statement. Click the Step back over button press Ctrl + F8, to step back over the previous statement. In other words, it will automatically skip any execution trace lines associated with a function call on the previous line, and take you to the point where that function call was invoked. Click the Step back out button or press Ctrl + F9, to step back out of the current function, that is, takes you all the way back to the function call that invoked the function you are currently in. Click the Run to cursor forward button or press F4, to move forward to the line in the source containing the cursor. Click the Run to cursor backward button or press Ctrl + F4, to move backward to the line in the source containing the cursor. 43

50 Forensics Guide for Java Using the Find tool Use the Find tool when you know the exact string, function, or method you re looking for and want to quickly jump to it in the source or trace window. To use the Find tool in the source window: 1. Click in the source window to make it the active window. 2. Select Edit Find or right-click in the source window and select Find from the menu that appears. 3. Type what you want to search for and any searching options. 4. Click Find Next. To use the Find tool in the trace window: 1. Right-click in the column (not the column heading) in which you want to use the find tool and select Find from the menu that appears. 2. Type what you want to search for and any searching options. 3. Click Find Next. NOTE: Once you've searched in a specific column, the find tool will default to searching in that column until you right-click in another column or select the source window. To find the next occurrence of the same string, use the Find Next button or select Edit Find Next. To find the previous occurrence of the same string, select Edit Find Previous. Using the Go To tool The Go To tool provides a shortcut for navigating the trace using numbers in the sequence column. Each row in the trace is assigned a unique identifier, called the sequence number. As you step through the trace, you can use the sequence numbers as bookmarks. Jot down the sequence numbers for the rows you re interested in focusing on, and use the Go To tool to quickly jump from one location to another. To navigate the trace with the Go To tool: 1. Click the trace window to make that the active window. 2. From the Edit menu, select Go To. You may need to click the down arrows to see Go To as a choice on the menu. 3. Type the sequence number for the row you wish to jump to and click OK. The trace is repositioned at the row with that sequence number. 44

51 Using the Forensics Viewer Changing the trace view Traces are either presented in a list or a tree format. Viewing the trace in a tree helps you analyze the origin and sequence of function calls. The default presentation format depends on how the trace was generated. For more details on default appearance of traces, see Initial position of the trace. To change the trace view from list to tree: 1. Make sure the trace is not sorted. Right-click anywhere in the trace window except in the column headings and select Reset Sort. 2. The trace also must be single-threaded or filtered to one thread. To filter the trace to one thread, right-click the thread you wish to isolate and select Filter Thread ID=<yourThread>. 3. Right-click anywhere in the trace window except in the column headings and select Show as Tree. If the list is sorted, the Show as Tree option will not appear. This makes it easier to see the order of what happened during runtime. You may see function names listed as implied function call. An implied function call is one that was not actually found in the trace, but had to occur based on what the code indicates. NOTE: For multi-threaded traces, open additional trace windows and filter the trace on a different thread in each window. Then select the Show as Tree option in each trace. To change the trace view from tree to list, right-click anywhere in the trace window except in the column headings and select Show as List. The icon in the upper left corner of the trace window indicates whether the trace is currently in list view or tree view. TIP: If the trace is single-threaded or filtered to one thread, click the icon in the upper left corner to quickly switch between list and tree views. Working with multiple copies of the same trace By default, only one copy of the trace history is displayed when you open forensics files. You can also open multiple copies of the same trace in the Forensics Viewer. More than one copy of the same trace allows you to: Compare the results of different filters at once, such as different threads for a multi-threaded application. Compare different sorting methods at once. Display different trace columns in each window (e.g.,, Function Name, Thread ID, and Module in one window and Function Name, Source File, Line Number, and Source Line in the other). To open multiple copies of the same trace: 1. From the View menu, select Trace History 2. 45

52 Forensics Guide for Java 2. From the View menu, select Trace History 3 to open a third copy. Trace History 1 then becomes the active trace and traces history windows 2 and 3 are copies. You can filter, sort, and customize the view of all the trace windows. However, all copies are linked to each other and to the source for navigation. Therefore, as you select a line in one trace window, the position of the other trace windows changes to show the corresponding line in gray and the same line is highlighted in the source window. NOTE: When multiple copies of the same trace are open, some lines in the copies appear in red. The red color indicates those items happened within an overlapping time segment of the current item in the active trace window. The active trace window is the one that has the current selection highlighted. Click a trace window to make it the active trace. Or, select Set Active Trace from the Debug menu. Using multiple trace copies for multi-threaded applications One of the biggest benefits of working with multiple copies of the same trace is being able to focus on different threads in several windows. To view different threads in a trace for a multi-threaded application, first open multiple copies as described in the previous section. Then: 1. In the Trace History 1 window, select the line with the first thread you wish to isolate. 2. Right-click the specific thread name and select Filter Thread ID=<yourThread>, where yourthread is the name of the thread you want to isolate. 3. In the Trace History 2 window, select the line with the second thread you wish to isolate. 4. Right-click the specific thread name and select Filter Thread ID=<yourThread>. 5. In Trace History 3 window, select the line with the third thread you wish to isolate. 6. Right-click the specific thread name and select Filter Thread ID=<yourthread> The status bar at the bottom of the Forensics Viewer will display any filters that have been applied to the trace. TIP: Another helpful filter both for navigating in multiple traces and viewing multiple threads is Filter Entry Type!= to normal. (If Entry Type is not displayed in the trace view, click a column heading and select Entry Type. The Entry Type column should then appear.) In the Trace History 1 window, for example, use this filter to remove all functions with an entry type of normal and leave implied functions, function calls, etc. You can then navigate through each remaining line in Trace History 1, while viewing the information in Trace History 2 filtered in a different way, such as for a specific thread. Filtering and sorting traces Traces can be filtered and sorted to view and organize trace results in multiple ways. You can filter on specific elements, filter out elements, and sort elements. By default, some traces are filtered 46

53 Using the Forensics Viewer automatically. For more information on how traces are initially filtered, see "Initial position of the trace". Filtering on an element Filtering on an element allows you to select one element to keep and remove all the rest in that category. This is very helpful for isolating a particular element in the trace view. For example, you can check how many lines in the trace are associated with a particular function. Or, you can isolate one thread in a multi-threaded trace. To filter on an element: 1. Find the line in the trace with the element you wish to isolate. In this example, we wish to view the Handle_Request function only. 2. Right-click the specific element you wish to focus on and select Filter <ColumnName> = <value>, where <ColumnName> is the general category name and <value> is the name of the element wish to isolate. For example, to keep the Handle_Request function and 47

54 Forensics Guide for Java remove all other functions, select Filter Function Name = Handle_Request. The trace view changes to display only the lines with the element you selected. You can also have multiple layers of filters. To filter on an additional element, repeat steps 1 and 2. The status bar at the bottom of the Forensics Viewer will display any filters that have been applied to the trace. For details on how to filter and view a different thread in up to three trace history windows, see "Working with multiple copies of the same trace". Filtering out an element Filtering out an element removes that element from the trace view. This can be useful when you want to temporarily eliminate a component from the trace results. For example, perhaps there is a particular function you'd like to remove from the trace view. Or perhaps, you want to filter out one of several threads reported in the trace for a multi-threaded application. To filter out an element: 48

55 Using the Forensics Viewer 1. Find the line in the trace with the element you wish to remove from the view. In this example, we wish to remove the Handle_Request function from the trace view. 2. Right-click the specific element you wish to remove and select Filter <ColumnName>!= <value>, where <ColumnName> is the general category name and <value> is the name of the element that will be removed from the view. For example, to remove the Handle_Request function, select Filter Function Name! = Handle_Request. The element has been removed from the trace view. You can also filter out more than one element. To remove another element, repeat steps 1 and 2. The status bar at the bottom of the Forensics Viewer will display any filters that have been applied to the trace. 49

56 Forensics Guide for Java Undoing and resetting filters There are two methods for clearing or removing the filters applied to traces: remove either all filters or just the last filter set up. To remove all filters from the trace view: 1. Right-click anywhere in the trace window except the column headings. 2. Select Reset Filter to remove all filters. To remove the most recent filter: 1. Right-click anywhere in the trace window except the column headings. 2. Select Undo Filter to remove the last filter that was set up. 3. This can also serve as a multiple undo feature. Continue to select Undo Filter to remove multiple layers of filters in order from last filter applied to the first. Sorting traces In addition to filtering traces, you can also sort traces by any of the column headings, including on a single column or multiple columns. Each column can also be sorted in ascending or descending order. To sort a trace: 1. Right-click anywhere (except the column heading) in the column you wish to sort on. 2. Select Sort by <value>, where <value> is the column heading, or category, you are currently positioned in. Note that if the trace is in tree view, you will not be able to sort it. Select Show as List from the menu and then sort it. Tree view is only available for traces showing one 50

57 Using the Forensics Viewer thread. 3. To sort another column, repeat the previous steps. Notice that once a trace has been sorted, a solid gray arrow appears on the column heading that was sorted first. Columns that were not sorted first will have a hollow, white arrow. The arrow indicates whether the column is sorted in ascending or descending order. To switch between ascending and descending order, right-click in the sorted column and select Sort by Value again (where value is the column heading, or category). The status bar at the bottom of the Forensics Viewer will display any sorting that has been applied to the trace. Undoing and resetting sorting There are two methods for removing the sorting applied to traces: remove either all sorting or the most recent sorting. To remove all sorting: 1. Right-click anywhere in the trace window except the column headings. 2. Select Reset Sort to remove all sorting. To remove the most recent sort: 1. Right-click anywhere in the trace window except the column headings. 51

58 Forensics Guide for Java 2. Select Undo Sort to remove the last sort that was applied. 3. This can also serve as a multiple undo feature. Continue to select Undo Sort to remove multiple layers of sorting in order from last sort applied to the first. Exporting trace results Trace results can be exported to a file for further analysis, archival, or other activity. To export a trace: 1. Right-click anywhere in the trace window except in the column headings and select Export. 2. Type a name for the file. 3. Choose one of the three file formats: CSV (comma delimited), TSV (tab delimited), or TXT(space delimited). 4. Click Save. Printing a trace You can print the trace in the trace history window. To print a trace: 1. Select the window with the trace you wish to print. 2. Either right-click and select Print, or select File > Print, or click the Print icon. 3. Modify your printer settings as necessary and click OK. You can also preview the print job before sending it to the printer. From the File menu, select Print Preview. Analyzing trace results When a trace is generated, the Forensics module captures: General system information (the trace information file) Detailed statement execution (the trace itself) Stack, data, and memory This section covers how to use all information captured for a complete analysis of what was happening leading up to the trace and the cause of any failures. It also covers how optimized code and trace buffer settings can affect trace results. Viewing general system information Viewing general system information with the trace information file The trace information file records general information about the environment at the time the instrumented application generated the trace, such as when it was generated, on what system, what was running at the time, and what runtime options were in use. It is a helpful file that complements the detailed view the trace provides and provides key information for low-level troubleshooting. 52

59 Using the Forensics Viewer The trace information file is displayed in the trace information window. For information on opening the file, see "Trace information window". There are several categories of information recorded in the trace information file. The categories are listed in a tree structure on the left pane of the trace information window. Names and recorded values for each category are shown in the right pane. Click an item in the left pane to view the corresponding details in the right pane. Information is only displayed for children, or individual nodes, in the tree. Viewing Trace Information When a folder named... List is selected, the right pane will display a list pane, which allows sorting on each column by clicking on the column header. The following sections explain the information gathered for each category. Summary The summary category reports the high level what, when, and why behind the trace. Click Summary in the left pane of the trace information window to view the following: Reason What prompted the trace (e.g., a policy snap, manual snap, etc.) System Properties Standard system that may vary by JVM. Also includes the following: Total memory Amount of memory allocated by the Java runtime. Free memory Amount of memory currently available for allocation by the 53

60 Forensics Guide for Java Thread group Name Priority Alive Daemon Interrupted JVM. Group of threads to be acted upon collectively by single function calls. Name of thread. Priority of thread. Status of thread. Thread whose presence won't keep the JVM from exiting after the last non-daemon thread exits. Thread on which the "interrupt" method has been called. Checking stack, data, and memory Checking stack and data Rounding out the information captured, in addition to the trace history file and the trace, is stack, variable data, and memory. IMPORTANT: A dump file is required to view the stack, variables, and memory addresses. For information on producing dump files: with the snap command, see "Manually generated files: external snaps" Data The data window shows the local and global variables and registers for the stack at the time of the failure. Use the data window to look at symbols and get addresses for use with the memory window. To check the data captured, click the data icon window is linked to the stack window. Switch between the Locals, Globals, and Registers by clicking the tabs. on the toolbar or select View Data. The data 54

61 Using the Forensics Viewer Data Window Combining trace files From the Debug menu, one can select Combine Trace Files..., which displays the following dialog. This is used to integrate traces from multiple related processes. Combining Trace Files Combining traces is particularly useful when they result from associated processes, for instance Javacode and the associated JNI code, or servers and clients that form a distributed application. It is 55