Visual Profiler. User Guide

Similar documents
ClearSpeed Visual Profiler

CSX600 Runtime Software. User Guide

ENVISION. ACCELERATE.

Enterprise Architect. User Guide Series. Profiling

Enterprise Architect. User Guide Series. Profiling. Author: Sparx Systems. Date: 10/05/2018. Version: 1.0 CREATED WITH

COMP 635: Seminar on Heterogeneous Processors. Lecture 7: ClearSpeed CSX600 Processor.

file://c:\documents and Settings\degrysep\Local Settings\Temp\~hh607E.htm

Performance Analysis of Parallel Scientific Applications In Eclipse

This section covers the MIPS instruction set.

Introductory Programming Manual. The ClearSpeed Software Development Kit. Document No. 06-UG-1117 Revision: 2.E

API for Auxiliary Processing Unit

1 Preface About this Manual Intended Audience Revision History Document Conventions Version...

Introduction to the ThreadX Debugger Plugin for the IAR Embedded Workbench C-SPYDebugger

ENVISION. ACCELERATE.

To obtain the current global trace mask, call meitraceget(...). To modify the global trace mask, call meitraceset(...).

NEW CEIBO DEBUGGER. Menus and Commands

Tesla GPU Computing A Revolution in High Performance Computing

NOTE: Debug and DebugSingle are the only MPI library configurations that will produce trace output.

BASICS OF THE RENESAS SYNERGY PLATFORM

Visual Studio.NET. Although it is possible to program.net using only the command OVERVIEW OF VISUAL STUDIO.NET

Performance analysis basics

Help Volume Hewlett Packard Company. All rights reserved. Toolsets: IA Format Utility

Performance Profiling

RTOS 101. Understand your real-time applications. with the help of Percepio Tracealyzer

Trace Getting Started V8.02

EW The Source Browser might fail to start data collection properly in large projects until the Source Browser window is opened manually.

MPLAB SIM. MPLAB IDE Software Simulation Engine Microchip Technology Incorporated MPLAB SIM Software Simulation Engine

Writer Guide. Chapter 13 Working with Master Documents

Parallel Debugging. ª Objective. ª Contents. ª Learn the basics of debugging parallel programs

TDT 1.2 Release Notes and FAQ March 2002

Introduction. This tutorial introduces the following SingleStep features:

Qualcomm Snapdragon Profiler

ARM. Streamline. Performance Analyzer. Using ARM Streamline. Copyright 2010 ARM Limited. All rights reserved. ARM DUI 0482A (ID100210)

CS2: Debugging in Java

UNIT - 5 EDITORS AND DEBUGGING SYSTEMS

CSE 374 Programming Concepts & Tools. Brandon Myers Winter 2015 Lecture 11 gdb and Debugging (Thanks to Hal Perkins)

P17 System Testing Monday, September 24, 2007

COMP Superscalar. COMPSs Tracing Manual

Parallel System Architectures 2016 Lab Assignment 1: Cache Coherency

COIMBATORE EDUCATIONAL DISTRICT

Advanced Debugging with the System Profiler. Rennie Allen Cisco Field Application Engineer

BasicScript 2.25 User s Guide. May 29, 1996

Controlling the Drawing Display

System Debug. This material exempt per Department of Commerce license exception TSU Xilinx, Inc. All Rights Reserved

RVDS 4.0 Introductory Tutorial

AEMLog Users Guide. Version 1.01

BLM2031 Structured Programming. Zeyneb KURT

Pace University. Fundamental Concepts of CS121 1

Combining Arm & RISC-V in Heterogeneous Designs

Object Explorer. Atacama Large Millimeter Array

Outline. Computer programming. Debugging. What is it. Debugging. Hints. Debugging

RVDS 3.0 Introductory Tutorial

CFC. Special functions from SIMATIC S7 CFC V7.0 SP1 onwards

SEGGER J-Scope. User Guide. Document: UM08028 Software Version: 5.10 Revision: 0 Date: November 26, 2015

Noopur Gupta Eclipse JDT/UI Committer IBM India

Testing, code coverage and static analysis. COSC345 Software Engineering

Help Volume Agilent Technologies. All rights reserved. Agilent E2485A Memory Expansion Interface

ACE USER MANUAL ACE Development Team ANALOG DEVICES, INC., OCTOBER 2015

Introduction to Programming Using Java (98-388)

MCUXpresso IDE Instruction Trace Guide. Rev May, 2018 User guide

Outline. Exploiting Program Parallelism. The Hydra Approach. Data Speculation Support for a Chip Multiprocessor (Hydra CMP) HYDRA

3 TUTORIAL. In This Chapter. Figure 1-0. Table 1-0. Listing 1-0.

2 TUTORIAL. Overview. VisualDSP Getting Started Guide 2-1 for SHARC DSPs

Short Notes of CS201

UNIT- 5. Chapter 12 Processor Structure and Function

Homework 9: Software Design Considerations

Lab 0 Introduction to the MSP430F5529 Launchpad-based Lab Board and Code Composer Studio

We do not teach programming

Dataflow Editor User Guide

CS201 - Introduction to Programming Glossary By

2 TEST: A Tracer for Extracting Speculative Threads

The NetBeans IDE is a big file --- a minimum of around 30 MB. After you have downloaded the file, simply execute the file to install the software.

BASIC USER TRAINING PROGRAM Module 5: Test Case Development

Sysinternals DebugView

Monitoring and Analysis

New Features of Eli Version 4.3

JDB - QUICK GUIDE JDB - INTRODUCTION

What the CPU Sees Basic Flow Control Conditional Flow Control Structured Flow Control Functions and Scope. C Flow Control.

Libgdb. Version 0.3 Oct Thomas Lord

HOW TO USE CODE::BLOCKS IDE FOR COMPUTER PROGRAMMING LABORATORY SESSIONS

CSE 351, Spring 2010 Lab 7: Writing a Dynamic Storage Allocator Due: Thursday May 27, 11:59PM

Superscalar Processors

Project Debugging with MDK-ARM

Language Translation. Compilation vs. interpretation. Compilation diagram. Step 1: compile. Step 2: run. compiler. Compiled program. program.

Distributed Systems 8. Remote Procedure Calls

Stream Computing using Brook+

AEMLog users guide V User Guide - Advanced Engine Management 2205 West 126 th st Hawthorne CA,

The ThreadX C-SPY plugin

Resource 2 Embedded computer and development environment

OptimiData. JPEG2000 Software Development Kit for C/C++ Reference Manual. Version 1.6. from

Code::Blocks Student Manual

Reducing Miss Penalty: Read Priority over Write on Miss. Improving Cache Performance. Non-blocking Caches to reduce stalls on misses

SpiNNaker Application Programming Interface (API)

MicroBlaze TFTP Server User Guide

ENVISION. ACCELERATE.

There are different characteristics for exceptions. They are as follows:

SQL Server. Management Studio. Chapter 3. In This Chapter. Management Studio. c Introduction to SQL Server

CSCI0330 Intro Computer Systems Doeppner. Lab 02 - Tools Lab. Due: Sunday, September 23, 2018 at 6:00 PM. 1 Introduction 0.

Windows Device Driver and API Reference Manual

Evaluating Performance Via Profiling

Transcription:

Visual Profiler User Guide Version 3.0 Document No. 06-RM-1136 Revision: 4.B February 2008

Visual Profiler User Guide Table of contents Table of contents 1 Introduction................................................ 7 2 Java runtime environment.................................... 7 3 CSAPI trace................................................ 7 3.1 Enabling CSAPI trace......................................... 7 3.2 Viewing CSAPI traces........................................ 8 4 CSX system trace........................................... 9 4.1 Debugger integration........................................ 10 4.2 Configuring CSX processor trace............................... 10 4.3 Cstrace command syntax..................................... 10 4.3.1 Status................................................... 11 4.3.2 Buffer................................................... 11 4.3.3 Event................................................... 11 4.3.4 Info..................................................... 12 4.3.5 Event types.............................................. 12 4.3.6 Enable.................................................. 14 4.3.7 Disable.................................................. 14 4.3.8 Info..................................................... 14 4.3.9 Dump................................................... 14 4.3.10 Acfilter.................................................. 15 4.4 Debugger interaction........................................ 15 4.5 Viewing CSX system trace.................................... 16 4.6 Using compiler instrumented function profiling..................... 16 4.7 Profiling CSX processor code from a host application............... 17 4.8 Profiling multiple CSX processors.............................. 17 4.9 CSX system trace plugin..................................... 17 4.9.1 How to activate the CSX system trace plugin.................... 18 4.9.2 CSX system trace plugin toolbar.............................. 20 Document No. 06-RM-1136 Revision: 4.B 3

Table of contents Visual Profiler User Guide 5 CSX pipeline trace.......................................... 20 5.1 Debugger integration........................................ 21 5.2 Configuring CSX pipeline trace................................ 21 5.3 Enabling the simulator for trace capture.......................... 21 5.4 Simtrace command syntax.................................... 21 5.4.1 Enable.................................................. 22 5.4.2 Disable.................................................. 22 5.4.3 Dump................................................... 22 5.5 Debugger interaction........................................ 22 5.6 Viewing CSX pipeline trace................................... 23 6 Host instrumentation API.................................... 23 6.1 API functions.............................................. 24 6.1.1 Initialize the trace.......................................... 24 6.1.2 Finish the trace........................................... 24 6.1.3 Trace event.............................................. 24 6.1.4 Trace begin.............................................. 25 6.1.5 Trace end................................................ 25 6.1.6 Trace on................................................. 25 6.1.7 Trace off and trace reset.................................... 26 6.1.8 Trace output.............................................. 26 6.1.9 Callback helper methods.................................... 26 6.2 Host trace plugin........................................... 27 6.2.1 How to activate the host trace plugin........................... 27 6.2.2 Host trace toolbar......................................... 29 7 Visual profiler user interface................................. 30 7.1 Status bar................................................. 30 7.2 Timeline Window........................................... 31 7.2.1 Event markers............................................ 31 7.2.2 Event types.............................................. 32 7.3 Timeline toolbar............................................ 32 7.4 Source window............................................. 33 7.4.1 Source-line searching...................................... 33 7.4.2 Buffer window............................................ 33 4 Document No. 06-RM-1136 Revision: 4.B

Visual Profiler User Guide Table of contents 7.5 Window menu.............................................. 33 7.6 Main toolbar............................................... 34 7.7 Buffer toolbar.............................................. 35 7.8 Event searching............................................ 36 7.9 Opening trace files.......................................... 37 8 Visual profiler workshop overview............................ 37 9 Visual profiler file format.................................... 37 10 Bibliography.............................................. 39 Document No. 06-RM-1136 Revision: 4.B 5

Table of contents Visual Profiler User Guide 6 Document No. 06-RM-1136 Revision: 4.B

Visual Profiler User Guide Introduction 1 Introduction The ClearSpeed Visual Profiler is a generic profiling tool which can be used to visualize data contained in event-based trace files. ClearSpeed trace (.cst) files contain simple, textbased trace data and can be generated from any source, but typically come from runtime data gathered in embedded systems, simulators or host software components. The Visual Profiler can therefore be particularly useful for analyzing the performance in custom, hybrid systems where precise timing information and I/O behavior is important. 2 Java runtime environment The Visual Profiler requires a system with version 1.5 of the Java Runtime Environment (JRE) installed for it to operate correctly. Any versions prior to this will cause a failure on GUI startup. The Visual Profiler is also only verified to work correctly with the Sun JRE for Linux and Windows. Usage on other Java runtime environments is not currently supported. 3 CSAPI trace The Visual Profiler can be used to display the trace of function calls made to the CSAPI interface. This enables you to see the interaction between the host code and the Advance card. It gives visibility of every call that is made to the card API and presents them in a timeline format. This makes it easier to spot performance issues and incorrect execution. This tracing capability is built into every host application that uses the CSAPI and can be enabled without any changes to the code. 3.1 Enabling CSAPI trace The CSAPI trace is enabled using a series of environment variables. These control trace output and allow you to specify the file name to place the profiling data in. To generate trace data for the Visual Profiler the following variables need to be set: CS_CSAPI_TRACE=1 CS_CSAPI_TRACE_CSVPROF=1 CS_CSAPI_TRACE_CSVPROF_FILE= csapitrace.cst With these set, a trace file will be generated from by CSAPI when the host application calls CSAPI_delete(). With these environment variables set, you can generate trace from any CSAPI application by simply running it. You do not need to modify their application in anyway to generate the information required. Document No. 06-RM-1136 Revision: 4.B 7

CSAPI trace Visual Profiler User Guide 3.2 Viewing CSAPI traces Once the trace file has been generated it can be loaded into the Visual Profiler with as follows: csvprof csapitrace.cst The Visual Profiler will start up and parse the trace file producing a timeline display of the CSAPI activity. The activity is displayed with the individual CSAPI calls marked as events down the left hand side of the GUI in the order in which they were called. The timings for CSAPI function tracing are all done in microseconds and this is reflected by the scale set in the GUI when it opens the trace file. CSAPI function call start and finish times are marked by the colored bars in the timeline view. You can then use the features of the Visual Profiler GUI to get an overview of the board activity within the code and to time events or sections of events to investigate performance. CSAPI tracing tracks when multiple host threads are being used. When multiple threads call a CSAPI function there will be an entry in the event file for each function call on each thread. The events are tagged with the operating system thread identifier so that they can be related back to the thread in which they occurred. The trace will also display execution time outside the CSAPI so you can see how much time is spent in the CSAPI calls as a proportion of total execution time. When the Visual Profiler displays CSAPI output it should appear as shown in Figure 1. From the image you can see that the CSAPI calls are individually labelled in the event listing. 8 Document No. 06-RM-1136 Revision: 4.B

Visual Profiler User Guide CSX system trace They are also preceded by the thread ID so if you have calls to the same CSAPI function from different threads you will get one event per thread in the event list. Figure 1. CSAPI trace output 4 CSX system trace The Visual Profiler can be used to display the activity of code running on the CSX processor. This uses hardware built into the processor to capture events. The raw trace data is then post processed by the ClearSpeed debugger (csgdb) into a form that can be displayed in the Visual Profiler. Trace capture sessions can be scripted so you can generate output using the debugger non-interactively. Trace files generated in this way can also be viewed in the CSX System Trace plugin, described in Section 4.9: CSX system trace plugin on page 17. The CSX System tracing capability is only available when the debugger is connected to the hardware or the cycle accurate simulator. The functionality is not implemented in the instruction simulator. Document No. 06-RM-1136 Revision: 4.B 9

CSX system trace Visual Profiler User Guide 4.1 Debugger integration The debugger has been extended to support hardware tracing. The cstrace command provides options to allow the hardware trace port to be configured and its results output for use with the Visual Profiler GUI. The cstrace command is added to the csgdb command set when the device is connected to the debugger using the connect command. 4.2 Configuring CSX processor trace csgdb contains an extended command set for accessing the real time trace hardware of CSX processors. You can set up a trace capture session using these commands to control the capture of data to be visualized in csvprof. To set up a trace session for code running on the device, use the following steps: 1. Specify the specific event types to be traced 2. Enable trace capture from the hardware 3. Dump the data in a post-processed file suitable for use with csvprof Once these steps are complete, you will be able to view the captured data in the Visual Profiler GUI. 4.3 Cstrace command syntax This section describes the csgdb command syntax for configuring and capturing data from the CSX processor. cstrace has been added to allow symbolic access to the realtime trace hardware. Online help for the cstrace command and its subcommands can be displayed as described below. If you enter the command cstrace help in csgdb, the following output is displayed: ***** Monitor Port Trace Help ***** Use help {command} for more detailed command specific instruction. Command Arguments Description ------- --------- ----------- status View the current configuration buffer {address} {size} Configure the trace buffer event {event} {'on' 'off'} Switch a trace event on or off enable Enable monitor port trace disable Disable monitor port trace info Show information about trace events dump {format} {filename} Dump the trace data to a file acfilter {cycles} Set the cycle filter for ac events (default 2) The displayed information is the set of subcommands available to support the debugger to target the trace port. Each of these subcommands will now be described. 10 Document No. 06-RM-1136 Revision: 4.B

Visual Profiler User Guide CSX system trace 4.3.1 Status 4.3.2 Buffer The status command shows the current configuration status of the hardware trace port. This allows you to query the information about the way they have configured the hardware from the debugger. The syntax of the command as displayed by cstrace help status is as follows. ***** Monitor Port Trace Help - "status" ***** Display the current configuration of the monitor port(s). syntax : status In order to collect the information from the hardware trace port the buffer command is used to specify the location and size of trace buffer. By default, the trace collection system uses the top three quarters of the DRAM memory from the second CSX processor on a board. This means that in the majority of simple profiling cases you do not need to specify a location for the memory buffer. If you are profiling an application on the second chip or have an application which uses some of this area of the DRAM, you will have to adjust the location of your trace buffer using this command. The command cstrace help buffer displays the syntax. ***** Monitor Port Trace Help - "buffer" ***** Configure the monitor port trace buffer location. Options required are an address and size for the buffer. syntax : buffer {address} {size} 4.3.3 Event If more trace data is generated than the specified buffer size allows, the collection of events will wrap, as it is implemented as a circular buffer in the hardware. This means that you may not get all the events expected in the final trace file due to them being overwritten. There is currently no way other than visual inspection to discover that this has happened. With the event command you can control the enabling and disabling of events within the trace hardware. The syntax of the command is available by using the command cstrace help event. ***** Monitor Port Trace Help - "event" ***** Configure the monitor port events. Options required are an event type and either on or off. See info command for a description of the trace events. syntax : event {type} {'on' 'off'} Document No. 06-RM-1136 Revision: 4.B 11

CSX system trace Visual Profiler User Guide 4.3.4 Info The event type can be specified in several ways: a number, a symbol or a group of events. These event types can be listed by using the command cstrace info from the debugger. The output is as follows. ***** Monitor Port Trace Info ***** Valid monitor port trace events : Num Symbol Module Description --- ------ ------ ----------- 0 none none Trace Nothing 1 cycle mono Cycle Counter Wrap 2 branch mono Branch Taken 3 icache mono Icache Usage Activity 4 monomem mono Mono Memory Access 5 proc mono Processor Running 6 thread mono Thread Usage Activity 7 ping mono Ping Instruction Event 8 polymem poly Poly Memory Access 9 polycomp poly Poly Compute Active 10 piocdir pio PIO Controller Transfer Direction 11 piocact pio PIO Controller Active 12 piocstall pio PIO Controller Stalled 13 pioecomp pio PIO Engine Completing 14 pioesem pio PIO Engine Semaphore Wait 15 pioetrans pio PIO Engine Data Transfer 16 datalost mono Trace Data Lost Event groupings are also valid : mono poly pio all All mono processor events All poly processor events All PIO data transfer events All events 4.3.5 Event types When using the event command the number, symbol or event grouping can be passed as the type. For the base events the following text is a brief description of their use. Trace nothing This event configures the trace port so that it is does not currently have any events that are enabled for tracing. Cycle counter wrap This event defaults to on when used in conjunction with other events. If just specified separately, it will collect information when the built-in 16-bit trace hardware timer wraps around. Branch taken When the CPU executes a branch instruction, the trace hardware will record this information with this event enabled. The captured trace always shows branch destination in the final trace file so you can see where the code has branched to. Icache usage activity This event displays information about the instruction cache usage of the device. The final trace will show cache hit and miss activity for the range of code that was traced. 12 Document No. 06-RM-1136 Revision: 4.B

Visual Profiler User Guide CSX system trace Memory access valid This event captures information about the duration and location of loads and stores issued from the mono processor. In this way, you can see how long each of these events take and spot situations such as poor data cache access and data cache misses. Processor running This event allows you to see when the processor stops executing code and shows how many cycles are lost to the processor not executing. It is particularly useful for seeing interaction between the host and the CSX processor. Thread usage activity The hardware threads of the mono processor can be traced using this particular event. Once post processed by the debugger you can see how their code is running and which threads it is using. Ping instruction event The ping event is related to the insertion of the ping instruction in the address stream. When code contains ping instruction, if this event is enabled, they will be captured by the hardware trace port, otherwise they will be ignored. Poly memory access This is the first of the events relating to the poly execution unit and is responsible for tracing when the PEs are performing loads and stores from poly memory. The event collects information when the elements of the poly array are accessing memory and shows the duration of the events. Poly compute active To trace the compute activity of the PE array this event can be enabled. It shows the duration of poly instructions within the poly processor allowing you to see utilization of the execution pipeline. Note: Due to the way in which the CSX microprocessor pipeline operates (see [1]: CSX600 Core Architecture Manual), poly instructions are executed some time after they have been issued. In the context of the poly events display, when you attempt to view the source code associated with the poly events in the visual profiler, you will see that the source code information displayed relates to the current instruction being issued and not the location of the code which generated the poly event. As poly events can be combined by the trace hardware of the processor, you will have to manually inspect the source code to find the last piece of poly code that was issued. Whilst it may be difficult to locate the exact location where the poly event was generated, the functionality that is present does allow you to see which piece of code poly execution is currently overlapping. PIO controller transfer direction This event allows you to see the direction of the PIO data transfer. The direction on the memory transfer is marker in the event list when enabled as either PIOC Poly Store denoting a store to poly memory or PIOC Poly Load for a load from poly memory. PIO controller active This event shows the time period that the PIO controller is active. The controller used to set up the memory transfer and this event shows the time taken in that setup. PIO controller stalled If the PIO controller is stalled, then this event shows the length of time of the stall. Document No. 06-RM-1136 Revision: 4.B 13

CSX system trace Visual Profiler User Guide 4.3.6 Enable 4.3.7 Disable 4.3.8 Info 4.3.9 Dump PIO engine completing This event relates to the PIO engine which is responsible for doing the actual data movement. The time spent in this event shows the time that the PIO engine is waiting for all the packets it has issued on the bus to complete. PIO engine semaphore wait When the PIO engine is waiting on a semaphore, this event shows the time spent waiting on that semaphore during the transfer. PIO engine data transfer This event is the most important with regards to the PIO engine as it shows the time spent during the complete data transfer on the bus. This is what can be used to time the length of a PIO operation. Trace data lost This event occurs when the trace packets sent from the hardware monitor port to the memory cannot happen quickly enough. When this event is seen, it means that other events may have been lost and that the trace around the sections cannot be guaranteed to be accurate. This is a limitation of the real time trace hardware. When you want to enable trace collection, use the enable command. The syntax for this command can be seen within csgdb by entering the command cstrace help enable. ***** Monitor Port Trace Help - "enable" ***** Enable the generation of trace information. syntax : enable To disable the trace collection, use the disable command. The syntax for this command can be found by entering the command cstrace help disable. ***** Monitor Port Trace Help - "disable" ***** Disable the generation of trace information. syntax : disable You can use the info command to list the specific events that are available to trace. Information for using this command is also available in the csgdb help with the command cstrace help info. ***** Monitor Port Trace Help - "info" ***** Display the events which can be traced with the monitor port syntax : info Use the dump command to output the collected trace data in memory to a file that can be read by the Visual Profiler GUI. There are a variety of formats which can be used to display the trace data events in relation to actual function calls. The default case is with no 14 Document No. 06-RM-1136 Revision: 4.B

Visual Profiler User Guide CSX system trace 4.3.10 Acfilter argument and this dumps the data in its raw form. To list the syntax for this command, enter the command cstrace help dump. ***** Monitor Port Trace Help - "dump" ***** Dump the trace event buffer to a file in a specified format Format argument can be one of the following : "branch" - generate file showing time spent per function. Filename argument is the output name of the file and will automatically have.cst added. syntax : dump {format} {filename} When events toggle very quickly inside the trace port, they trigger the FIFO overflow condition to indicate that they are potentially inaccurate due to events being lost. The acfilter command allows you to set a base cycle size for triggering the PE array load/store and microcode events. With this set to 2, for example, events will only be recorded for these events when they stay idle or active for more than two cycles. The help for this command can be output from csgdb using the command cstrace help acfilter. ***** Monitor Port Trace Help - "acfilter" ***** Cycles argument is in the range 0 to 15 : syntax : acfilter {cycles} There is more detail on these particular events and their meaning in the examples contained within the CSX system trace workshops. 4.4 Debugger interaction It is possible to use the facilities of the debugger to start and stop the trace around particular function calls and to script up trace sessions using the csgdb command language. You can use this to create the trace script once and reuse it multiple times. The example csgdb script below shows how you could instrument a trace capture session around a particular function. connect break main run cstrace event all on cstrace enable break *$ret continue cstrace dump csxsystemtrace quit This file automates the trace session and captures the trace from the hardware trace port around the function main(). To use this in conjunction with csgdb, enter the following command.: csgdb x tracescript.gdb filename.csx Document No. 06-RM-1136 Revision: 4.B 15

CSX system trace Visual Profiler User Guide The script file tracescript.gdb contains the above commands and filename.csx is the code that is going to be traced. 4.5 Viewing CSX system trace Once you have generated the trace data file for the Visual Profiler, you can view it using the following command: csvprof csxsystemtrace.cst The Visual Profiler will now start up with the highest level of zoom displaying the entire trace run. 4.6 Using compiler instrumented function profiling The events that can be profiled using the CSX system trace functionality are very low level and when profiling for more than a few seconds it is possible to generate a very large file or event to wrap the trace collection buffer and write over the event in the start. To profile code for longer periods of time, minutes rather than seconds, use the ping feature of the hardware trace port. You can add the ping instruction to your code just as any other instruction in the CSX600 instruction set. When executed it generates an event on the processor hardware trace port which can be used for timing specific sections of code or reporting events. The Cn compiler has a mode implemented which allows the simple instrumentation of all user functions in a source file with the ping instruction to allow timing information to be collected which can be viewed post execution in the Visual Profiler. The instruction itself is very low latency, a single cycle, and does not change the behavior of code executing so an application will still run as expected with the instrumentation enabled. To enable this profiling from the compiler you need to pass the -pg option to cscn. This enables the compiler to instrument all Cn functions within a specific source file. To generate a file from this information for use with the profiler GUI, use the debugger as with the other hardware events in the csx system trace. The difference when using the function profiling is that you only need to enable one event (ping) and the dump command takes another option (function). When writing a debugger script to capture the trace data, the example in Section 4.4: Debugger interaction on page 15 would look as follows: connect break main run cstrace event ping on cstrace enable break *$ret continue cstrace dump function csxfunctiontrace quit This generates the file csxfunctiontrace.cst for viewing in the Visual Profiler GUI. The function style profiling is very similar to the branch trace view apart from the fact that it only records entry and exit to function calls and does not record any other branch data. When viewed in the GUI you will get a list of functions profiled as the event types and then bars in the timeline view showing how long is spent in each one. 16 Document No. 06-RM-1136 Revision: 4.B

Visual Profiler User Guide CSX system trace 4.7 Profiling CSX processor code from a host application When the code running on the CSX600 is loaded and used by an application running on the host, you must connect the debugger to the host application using the built-in debug interface rather than running the CSX code standalone. This is done by using the functionality described in the [2]: SDK Reference Manual for connecting the debugger to a host application. The following is a simple set of steps which allow you to capture a trace session from code which is under the control of a host application: 1. Set the environment variable CS_CSAPI_DEBUGGER=1. 2. Start the host application. The csgdb debugger will appear in a window connected to processor 0 on card instance 0. It will be stopped at the function _start. 3. Set a start and end breakpoint for the trace capture session in the debugger and enable the relevant trace options using the command set described above. When the end breakpoint is hit the debugger can dump out the trace data to the specified file. 4. Close the debugger. If you do not have a sensible end point for setting a breakpoint, set the CS_CSAPI_DEBUGGER_DETACH variable to 1 to make the host application stop and wait for a key press before exiting. This allows you to [Ctrl]-[C] the code in the CSX debugger and then dump the trace file as described above. The trace session can also be scripted as described above. A command script file can be passed to the debugger using the CS_CSAPI_DEBUGGER_SCRIPT environment variable. This lets you capture and generate a trace file without having to enter any commands interactively. Further details of the use of these environment variables can be found in the debugger section of the [2]: SDK Reference Manual. 4.8 Profiling multiple CSX processors Currently, profiling multiple CSX processors involves profiling each one sequentially and then viewing both generated files in parallel, using the GUI. Each of the processors can be traced individually using the debugger stand alone and loading the CSX code onto each processor or using the mechanism described above for profiling CSX code driven from a host application. 4.9 CSX system trace plugin The CSX System Trace plugin lets you observe patterns in the trace data over a greater time period than is possible in the timeline view. System events are displayed on a timebased graph where the Y-axis shows the number of cycles an event was active for in a given sample period. Trace data shown in this plugin is over the entire file and is not restricted by the buffer boundaries. Document No. 06-RM-1136 Revision: 4.B 17

CSX system trace Visual Profiler User Guide 4.9.1 How to activate the CSX system trace plugin To activate the CSX System Trace plugin, select Window -> Plugins -> CSX System Trace from the main menu. Figure 2. CSX system trace plugin from main menu The tree on the left contains two expandable items; the events list and the overlap metrics. The right panel contains the main graph. Events in the events list will be shown in the graph as they are selected multiple events can be viewed by holding the [Ctrl] key. The 18 Document No. 06-RM-1136 Revision: 4.B

Visual Profiler User Guide CSX system trace overlap metrics show difference graphs for certain combinations of system events, providing further insight into system behavior such as I/O and compute overlap. A Figure 3. CSX system trace plugin Document No. 06-RM-1136 Revision: 4.B 19

CSX pipeline trace Visual Profiler User Guide 4.9.2 CSX system trace plugin toolbar A B C D E F G H I Figure 4. CSX system plugin toolbar A B ID Field Description Move buffer to blue cursor Fit to markers Sets the buffer position to the value of the blue marker in the graph. Zooms in to the region between the blue and green markers, as in the timeline window. C Full view Zoom out to display the entire contents of the file. D E F G Decrease sample rate Increase sample rate Save chart Y-axis unit Decreases the number of samples displayed in the graph. This increases the number of cycles per sample point. Increases the number of samples displayed in the graph. This decreases the number of cycles per sample point. Opens the save dialog which allows the current graph to be saved as an image file. Allows you to switch between absolute and scaled values. Absolute shows the number of cycles compared to the maximum for each sample point, that is, utilization percent. With scaled set, the Y-axis values are scaled to fit the graph window making it easier to view events with low-utilization. H Graph type Two graph types are available, line and area. I Display buffer Determines whether or not the current buffer boundaries are highlighted on the graph. Turning this option on can make it easier to navigate around the trace file and correlate timeline regions with points in the graph. Table 1. System plugin toolbar components 5 CSX pipeline trace In conjunction with the cycle accurate simulator, it is possible for the Visual Profiler to display detailed instruction pipeline information for the CSX processor. The trace file is generated via the debugger in a similar manner to the CSX system trace. The file shows information relating to the instruction pipeline and specifically the interaction between mono and poly instructions and the overlap of the poly instruction pipelines within the PE array. If 20 Document No. 06-RM-1136 Revision: 4.B

Visual Profiler User Guide CSX pipeline trace you are writing code, this allows you to have a visual view of the instruction scheduling and use this information to optimize code for performance at the instruction level. The CSX pipeline tracing is only available when the debugger is connected to the cycle accurate simulator. The functionality is not implemented in the instruction simulator. 5.1 Debugger integration To target the pipeline trace support in the CSX cycle accurate simulator, the debugger has been extended with a new command. This command provides a set of options to allow the hardware trace port to be configured and its results output for use with the Visual Profiler GUI. The simtrace command is added into the csgdb command set when the device is connected to the debugger using the connect command. 5.2 Configuring CSX pipeline trace The ClearSpeed debugger (csgdb) contains an extended command set for accessing the pipeline trace of the CSX cycle accurate simulator. You can set up a trace capture session using this command set for the capture of data to be visualized in csvprof. To setup a trace session for code running on the simulator a user would follow these steps, 1. Enable the trace capture from the simulator. 2. Dump the data in a post processed file suitable for use with csvprof. Once these steps are complete you can view the captured data in the Visual Profiler GUI. 5.3 Enabling the simulator for trace capture To provide the debugger with the information required to produce a simulator pipeline trace file, the cycle accurate simulator must be started with the option: -T {filename} where: filename is the locate for the simulator to dump the pipeline state in a raw form. The debugger uses this file in conjunction with the debug information for the CSX file to produce a file that can be displayed in the Visual Profiler. 5.4 Simtrace command syntax This section describes the csgdb command syntax for configuring and capturing data from the CSX simulator. simtrace allows access to the simulator pipeline information. Online help for the simtrace command and its subcommands can be displayed as described below. If you enter the command simtrace help in csgdb, the following output is displayed: * **** Simulator Trace Help ***** Command Arguments Description Document No. 06-RM-1136 Revision: 4.B 21

CSX pipeline trace Visual Profiler User Guide ------- --------- ----------- enable Enable the simulator tracing disable Disable the simulator tracing dump {cst_filename} {sim_filename} Dump the trace data to a file 5.4.1 Enable 5.4.2 Disable The displayed information is the subset of commands available to support the debugger to target the simulator trace. Each of these subcommands will now be described. When you want to enable the collection of trace information inside the cycle accurate simulator, use the enable command. The command can be issued once the debugger has connected to the simulator. The syntax for this command as displayed by simtrace help enable is as follows: * **** Simulator Trace Help - "enable" ***** Enable the collection of simulator trace information. syntax : enable To disable the trace collection in the simulator, the disable command can be used. The syntax for this command can be found by entering the command simtrace help disable: ***** Simulator Trace Help - "disable" ***** Disable the collection of simulator trace information. syntax : disable 5.4.3 Dump Use the dump command to output the collected simulator trace data to a file that can be read by the Visual Profiler GUI. The command takes the simulator internal log file as the first argument and the Visual Profiler output file as the second. To list the syntax for this command, enter the command simtrace help dump: ***** Simulator Trace Help - "dump" ***** Dump the simulator trace information to a file. syntax : dump {cst_filename} {sim_filename} The workshops for the simulator pipeline trace provide examples for the usage of these commands. 5.5 Debugger interaction It is possible to use the facilities of the debugger to start and stop the trace around particular function calls and to script up trace sessions using the csgdb command language. This lets 22 Document No. 06-RM-1136 Revision: 4.B

Visual Profiler User Guide Host instrumentation API you create your own trace script once and reuse it multiple times. The example csgdb script below shows how you could instrument a trace capture session for the simulator pipeline trace around a particular function. connect -s break main run simtrace enable break *$ret continue simtrace disable simtrace dump csxpipelinetrace quit This script file will automate the collection of pipeline information around the function main(). To use this in conjunction with csgdb, enter the following command: csgdb x tracescript.gdb filename.csx The script file tracescript.gdb contains the above commands and filename.csx is the code that is going to be traced. 5.6 Viewing CSX pipeline trace Once you have generated the trace data file for the Visual Profiler, you can view it using the following command: csvprof csxpipelinetrace.cst The Visual Profiler will now start up with the highest level of zoom displaying the entire trace run. The examples in the CSX Pipeline trace workshops go through in more detail the usage of the trace information from the simulator. 6 Host instrumentation API The Visual Profiler package contains the x86 instrumentation API which allows you to profile any piece of code running on an x86 processor. This consists of a number of simple functions for enabling and collecting the trace which you can add to your code to generate a trace file which can be viewed. The functions that make up the API are described below. Examples of using the host API can be found in the Visual Profiler workshops. The header file containing the API and the library itself are provided in the installed package. Document No. 06-RM-1136 Revision: 4.B 23

Host instrumentation API Visual Profiler User Guide 6.1 API functions The API functions are described in the following. 6.1.1 Initialize the trace int CVP_trace_init(void); To initialize the trace collection API the function CVP_trace_init() should be called. This creates the required state inside the trace collection library for records to be stored. Pre : None Post : Returns : 6.1.2 Finish the trace int CVP_trace_deinit(void); When you have finished with the trace collection mechanism and want to free all of the associated state, call CVP_trace_deinit(). Pre : Post : Returns : 6.1.3 Trace event Trace collection state is allocated. CVP_SUCCESS if the state is successfully allocated otherwise CVP_ERROR. CVP_trace_init() will have been called. Trace collection state is freed. int CVP_trace_event(unsigned char *record_details, CVP_callback *callback); The function CVP_trace_event() is called to create a new trace event within the trace capture system. The user provides the name of the event to trace (which will be the event name when displayed in the GUI) and a valid CVP_callback object or NULL. This function returns the unique identifier for the trace event which should be used with API calls for generating event timing. Pre : Post : Returns : CVP_SUCCESS if the state is successfully freed otherwise CVP_ERROR. CVP_trace_init() will have been called. record_details contains a pointer to a valid event name. callback contains a valid CVP_callback pointer or NULL. Trace collection state is freed. A unique event identifier if the event is successfully created or CVP_ERROR. The callback argument is used for an event which a user wants to allow additional processing to occur once the trace collection is finished, deferring this calculation so as to have minimal impact on the executing code. A CVP_callback structure has the following members. type print_cb Which is either one of CVP_discrete or CVP_continuous. Which is a function pointer of the following type. typedef void (*CVP_trace_print_cb)(int event_id, unsigned int, event_time, unsigned char *start_args, unsigned char* end_args); 24 Document No. 06-RM-1136 Revision: 4.B

Visual Profiler User Guide Host instrumentation API 6.1.4 Trace begin int CVP_trace_begin(unsigned int type, unsigned char *args); To mark the beginning of an trace event section the code must call CVP_trace_begin(). This takes the event id type which has been allocated using CVP_trace_event() and a set of arguments to associate with this trace event. Pre : Post : Returns : 6.1.5 Trace end int CVP_trace_end(unsigned int type, unsigned char *args); To mark the end of an trace event section the code must call CVP_trace_end(). This takes the event id type which has been allocated using CVP_trace_event() and a set of arguments to associate with this trace event. Pre : Post : Returns : 6.1.6 Trace on CVP_trace_init() will have been called. type is a valid event id allocated from CVP_trace_event(). args is either NULL or an allocated string pointer, this string allows a user to tag and extra event information they want to keep along with the time of the record. It will be used output to the file and the GUI will display this extra information. The information will also be passed to the callback function if there is one associated with the current event. A marker for event beginning is allocated within the trace collection system. CVP_SUCCESS if the record is successfully added otherwise CVP_ERROR. CVP_trace_init() will have been called. type is a valid event id allocated from CVP_trace_event(). args is either NULL or an allocated string pointer, this string allows a user to tag and extra event information they want to keep along with the time of the record. It will be used output to the file and the GUI will display this extra information. The information will also be passed to the callback function if there is one associated with the current event. A marker for event end is allocated within the trace collection system. int CVP_trace_on(void); The function CVP_trace_on() is used to control the global enable of tracing. Calling this function will enable the global trace capture mechanism for all threads. By default when the trace collection API is initialized it is enabled. Pre : CVP_SUCCESS if the record is successfully added otherwise CVP_ERROR. CVP_trace_init() will have been called. Post : Global trace collection is enabled. Returns : CVP_SUCCESS is returned, no failure case is possible. Document No. 06-RM-1136 Revision: 4.B 25

Host instrumentation API Visual Profiler User Guide 6.1.7 Trace off and trace reset int CVP_trace_off(void); int CVP_trace_reset(void); The function CVP_trace_reset() is used to reset the number of records collected to 0 and free all associated information. Pre : CVP_trace_init() will have been called. Post : Record data is cleared and number of records set to 0. Returns : 6.1.8 Trace output int CVP_trace_output(unsigned char *csvprof_file); The function CVP_trace_output() is used to output the current set of trace record data to a file which can be consumed by the Visual Profiler GUI. Pre : Post : Returns : 6.1.9 Callback helper methods CVP_SUCCESS is returned, no failure case is possible. CVP_trace_init() will have been called. csvprof_file contains a pointer to a string. CVP_trace_deinit must not have been called. These functions can be used inside a callback routine to produce the correct information in the trace file for the associated metric to be picked up and displayed in the correct format for the Visual Profiler GUI. More details of their use are provided in the workshop examples for the host API. void CVP_set_cb_metric_int(int event_id, char *metric, int value); The function CVP_set_cb_metric_int() is used from within a callback method for formatting the output metric data from an integer value input. Pre : The trace record data is written out to the specified file. CVP_SUCCESS is returned if the file is successfully written otherwise CVP_ERROR is returned. event_id is a valid allocated event type. metric is a valid pointer to a string. value contains the value for the specific metric. Post : Returns : The metric information for the particular event is updated in the trace record Nothing void CVP_set_cb_metric_float(int event_id, char *metric, float value); 26 Document No. 06-RM-1136 Revision: 4.B

Visual Profiler User Guide Host instrumentation API The function CVP_set_cb_metric_float() is used from within a callback method for formatting the output metric data from an float value input. Pre : event_id is a valid allocated event type. metric is a valid pointer to a string. value contains the value for the specific metric. Post : Returns : The metric information for the particular event is updated in the trace record Nothing void CVP_set_cb_metric_double(int event_id, char *metric, double value); The function CVP_set_cb_metric_double() is used from within a callback method for formatting the output metric data from an double value input. Pre : event_id is a valid allocated event type. metric is a valid pointer to a string. value contains the value for the specific metric. Post : Returns : The metric information for the particular event is updated in the trace record Nothing 6.2 Host trace plugin Trace files generated by the host instrumentation API can be viewed in the Host Trace plugin. Like the CSX System Trace plugin, this allows the user-defined metrics to be viewed on a time-based graph where the X-axis shows the system time in microseconds, and the Y- axis represents the range of values passed in through the CVP_set_cb_metric... API calls. 6.2.1 How to activate the host trace plugin To activate the Host Trace plugin, select Window -> Plugins -> Host Trace from the main menu. Figure 5. Host trace from main menu Document No. 06-RM-1136 Revision: 4.B 27

Host instrumentation API Visual Profiler User Guide The selection tree on the left shows a list of all the user-defined event names in the trace file, grouped together by the name of the metric. Selecting an event type will bring up the corresponding plot in the main graph holding [Ctrl] whilst clicking event types which share the same metric will display multiple plots. This allows single event types to be viewed in isolation or overlaid on the same graph for comparison. Figure 6 shows the trace of a simple host program which calls sin(), cos() and tan() a number of times in a loop, logging the result of each call as a user event. The sin and cos metrics in this graph have been overlaid. Figure 6. Host trace plugin screen Note: Due to the extra information contained in host trace records, the Host Trace plugin will only show event records which are contained in the current buffer. This should not normally prove to be a problem since trace files generated by the host instrumentation API are typically much smaller than board-side traces, allowing the whole file to fit in the buffer. 28 Document No. 06-RM-1136 Revision: 4.B

Visual Profiler User Guide Host instrumentation API 6.2.2 Host trace toolbar A B C D Figure 7. Host trace toolbar A ID Field Description Fit to markers Zooms in to the region between the blue and green markers, as in the timeline window. B Full View Zoom out to display the entire contents of the file. C Save chart Opens the save dialog which allows the current graph to be saved as an image file. D Graph type Two graph types are available, line and area. Table 2. System plugin toolbar components Document No. 06-RM-1136 Revision: 4.B 29

Visual profiler user interface Visual Profiler User Guide 7 Visual profiler user interface The main components of the Visual Profiler graphical user interface (GUI) are shown in Figure 8. View toolbar Preferences toolbar buffer toolbar Buffer window Source window Events panel Status bar Figure 8. GUI overview 7.1 Status bar The status bar displays information about the currently marked event(s). A B C D E F G H I J Figure 9. Status bar 30 Document No. 06-RM-1136 Revision: 4.B

Visual Profiler User Guide Visual profiler user interface ID Field Description A1 Blue marker info Information for the event selected with the blue marker. A2 Green marker info Information for the event selected with the green marker. A3 A4 Records loaded Start time Number of records loaded against the number of records in the entire trace file. Start time (or similar unit) for the most recently selected event. A5 End time End time for the most recently selected event. A6 Event duration (end time start time) for the most recently selected event. A7 Blue marker value X-axis value of the blue marker. A8 Green marker value X-axis value of the green marker. A9 Marker difference Difference in blue and green marker values. A10 Parser progress Indicates current progress of the parser, when invoked. This information can be useful for large buffers Table 3. Status bar components 7.2 Timeline Window The events panel represents the main component within the Visual Profiler, and allows users to navigate through the content of their trace files. Event types are listed in the lefthand panel and the x-axis units are displayed at the top. 7.2.1 Event markers You can set markers on events to obtain event information (shown in the status bar and source window) and to compare any two points in the trace. Left-clicking inside an event will set the blue marker, right-clicking will set the green marker. With both markers set, the x- axis unit difference is displayed, as shown in Figure 10. Figure 10. Event markers Document No. 06-RM-1136 Revision: 4.B 31

Visual profiler user interface Visual Profiler User Guide 7.2.2 Event types You can disable events of a specific by clicking the name of the event type in the left panel. This will temporarily remove all corresponding events in the trace. Clicking the event type again will re-enable it. 7.3 Timeline toolbar A B C D E F G H I Figure 11. Timeline toolbar ID Field Description A B Scale Fits to markers Sets the view scale in the events panel. A value can be selected from the drop-down list or an arbitrary scale may be entered (minimum value of 0.01). Note: The Buffer option automatically sets the scale to display the entire contents of the current buffer. Automatically sets the scale and scroll value in the events panel to display events between the blue and green markers. Both markers must be set. C Full view Zoom out to display the entire contents of the current buffer. D Compress Y axis Compress (zoom out on) the events panel. E Expand Y axis Expand (zoom in on) the events panel. F Display blue cursor Jump to the blue marker. G Display green cursor Jump to the green marker. H Snap to event boundaries Markers will automatically snap to the beginning or end of selected events. This option is useful when precise measurements are required. I Outline Turn event outlines on or off. Table 4. Status bar components 32 Document No. 06-RM-1136 Revision: 4.B

Visual Profiler User Guide Visual profiler user interface 7.4 Source window For trace files containing references to source code, the source window will display and highlight source lines corresponding to currently marked events. With both markers set, the window will split into two sections so you can view two areas of code, possibly from two separate files. 7.4.1 Source-line searching The source window also integrates with the event search facility. Clicking a line number within a file will bring up the Find Events dialog with the search filters set to that line/file. 7.4.2 Buffer window Figure 12. Source-line searching Due to the potential size of certain trace files, the Visual Profiler provides a buffering mechanism so you can choose to only load and display a portion of a file, thereby saving memory and I/O time. The buffer window displays the loaded portion (the buffer itself) in comparison to the entire file, using the x-axis units as a reference. Furthermore, it indicates the current view within the buffer based on the scale and horizontal scrollbar position. The size of the buffer must be set in the buffer toolbar, while the position can be set either in the toolbar or by clicking somewhere within the buffer window. Current view Buffered region Boundary indicators Unbuffered region Figure 13. Buffer window 7.5 Window menu Each trace file tab inside the application contains an MDI (Multiple Document Interface) that allows you to size and position internal windows. Document No. 06-RM-1136 Revision: 4.B 33

Visual profiler user interface Visual Profiler User Guide The window menu shows the available windows for the current trace file, plus various MDI layout options. The timeline and source code windows are available by default for all trace files. Some trace files are also associated with task-specific plugins which can be accessed via the plugins sub-menu. All windows including plugins can be shown or hidden by toggling the corresponding item in the window menu. 7.6 Main toolbar The main toolbar contains controls that are global to the csvprof application. A B C D E F G H Figure 14. Main toolbar A ID Control Description Open File Opens a new trace file. Any number of trace files may be opened at one time. B Reload File Reloads the file in the current tab. C Close File Closes the current tab. D Find Events Displays the find events dialog. E F Lock Scrollbars Key Locks the horizontal and vertical scrollbars by tracking the location of the nearest event. This feature is also intended to be used only when viewing pipeline-style vertical trace files. Displays the event key associated with the current trace file. If no key is defined, this option will be disabled. G Preferences Displays the preferences dialog for default settings. H Global markers Applies mouse clicks/event selections to all tabs in the application. Table 5. Main toolbar functions 34 Document No. 06-RM-1136 Revision: 4.B

Visual Profiler User Guide Visual profiler user interface 7.7 Buffer toolbar The buffer toolbar contains controls for the buffer window. A B C D Figure 15. Buffer toolbar ID Control Description A B C D Buffer size Buffer position Forwards/back Maximum Sets the size, in x-axis units (for example, cycles) of the current buffer. Sets the position of the current buffer in the file (value represents the first x-axis unit inside the buffer). Moves the buffer forwards or backwards by the number of units specified in A. When enabled, the buffer size will be automatically adjusted to load the maximum number of records, as specified in the preferences dialog. Note: the buffer size is likely to be re-adjusted each time the buffer position changes. Table 6. View toolbar functions Document No. 06-RM-1136 Revision: 4.B 35

Visual profiler user interface Visual Profiler User Guide 7.8 Event searching The Find Events dialog (Figure 16) assists in quickly locating certain areas in a trace, using a number of search filters. A description of the filters and search options is shown in Table 7. Figure 16. Event search dialog Field Description Event type Source file Event info Start time Padding Source line direction Restricts search results to events of this type only. Restricts search results to events which contain references to the specified source file. Restricts search results to events which have info strings containing the specified substring. If the search was invoked by selecting a line number, this field will instead display the text at that line and become disabled, as in the example above. Restricts search results to events which start after the time specified. Number of x-axis units to display either side of the current event. This value will adjust the scale accordingly. If no events matching the specified source line are found, the search will automatically move on to the next line and continue until the end of the file is reached. With this field set to backwards, the search will instead move back to the previous source line, and continue until the beginning of the file is reached. Table 7. Event search dialog Filters can be combined to narrow the search criteria, for example, find an event of type x in source file y containing info z. The left and right arrow buttons specify which direction in the events panel to search in. Note: this is not to be confused with source line direction, which uses the current line number rather than the x-axis as a reference. For example, you may wish to search for 36 Document No. 06-RM-1136 Revision: 4.B