Tutorial M02. Software Debug on ARM Processors in Emulation. March 24, 2014

Transcription

1 Tutorial M02 Software Debug on ARM Processors in Emulation Tutorials Chair Franco Fummi - Università di Verona Italy March 24, 2014 Fill in, please, the tutorial evaluation form, available on the conference room desk (fac simile in the next page). Return it, please, to the front desk. This will help us to improve the quality of the Date Monday Tutorials.

2 Conference Evaluation: Monday Tutorials On behalf of the Monday Tutorial Chair, you are kindly asked to hand in a short feedback regarding the Monday Tutorial you participated at after the end of the session to the session chair. Your opinions will be most appreciated and will help the committee to maintain future DATE conferences at the highest quality and even increase the relevance of the event in the European Design and Test community. Thank you for your time and support! Monday Tutorial (please indicate the code): M Rating Please indicate: Quality of oral presentations 5 (very good) 4 (good) 3 (average) 2 (poor) 1 (very poor) Quality of written materials Level of the tutorial Global Evaluation Which topics would you like to be covered by DATE 15 Monday Tutorials? Additional Comments: Please also use the back side of the paper for any additional comments, if required. Name (optional) (optional) K.I.T. Group GmbH Dresden Münzgasse Dresden Germany Phone: Fax: info@kitdresden.de

3 Software Debug in Emulation on Veloce DATE 2014 Russell Klein Joe Hamman Agenda! Session 1 9:30 to 11:00 Options for Debug and Trace in the context of emulation Trade-offs for different debug/trace facilities! Session 2 11:30 to 13:00 Concurrent debug of multiple cores in emulation Correlation of hardware and software views Efficient utilization of emulation resources during software debug 1

4 SESSION 1 OPTIONS FOR DEBUG AND TRACE Agenda! Introduction to Emulation! Debug methods JTAG Virtual JTAG Trace based state reproduction! Trace Systems ETM (hardware trace systems) Tarmac (simulation/emulation trace systems) 2

5 Veloce-2! 2nd Generation Veloce System Improves functionality, performance and capacity over Veloce-1 Crystal-2 Chip! Configurable logic for 1 million gates! 512 megabytes on chip SRAM! Logic analyzer on a chip Trace controller Debug engine! Full visibility for all signals! 512K clocks of trace buffer For all signals! Rich routing resources Trace Controller 32 MB Macro 96K SDRAM Elements Macro Debug (500K Elements Gates Equivalent) 32 MB SDRAM Engine Virtual 1M SRAM Bytes SRAM Wire Logic 3

6 Advanced Verification Board! Employs 16 Crystal-2 chips! 16 Million gates of configurable logic! Additional 2 gigabytes on board memory! Additional routing resources High Speed Link! Data path from Veloce to host computer! Enables interaction with host based testbenches! Waveform upload! Design download! Up to 1 for each 4 AVBs 4

7 Veloce Chassis! Quattro Capacity for 16 AVBs 256 Million gate capacity when fully populated Up to 4 high speed links Up to 16 concurrent users! Maximus Capacity for 64 AVBs 1 billion gate capacity when fully populated Up to 16 high speed links Up to 64 concurrent users! Double Maximus Capacity for 128 AVBs 2 Billion gate capacity when fully populated Up to 32 high speed links Up to 128 concurrent users Veloce Chassis Quattro 256 MG; 16 Users Maximus 1 BG; 64 Users Double Maximus 2 BG; 128 Users 5

8 Software Debug with Veloce Veloce compiler flow blp RTL Design module ddr1_core (DOUT, DIN, WA, RA, WE); input [23 : 0] WA, RA; input [7 : 0] DIN; input WE; output [7 : 0] DOUT; reg [7 : 0] DOUT; reg [7 : 0] mem [ : 0]; RDFF_P RUN_N BITLINE_P CPB_I_P [3:0] Compiler Compiler (posedge WE) begin mem[wa] = DIN; end LUT 0 CPB_OUT D Q WRFF_P FREEZE_P LEG_P[2:0 ] VCLK 1 (RA) begin DOUT = mem[ra]; end endmodule Compiler Memory point Emulation Division Mentor Graphics Confidential Crystal 2 overview Slide 3 Compiler Compilation Flow! Complete flow developed by Mentor No dependence on FPGA vendor SW Fast turnaround on compile issues Encryption! Continuously enhanced Capacity utilization Runtime speed Compile speed Language support Distributed Distributed ANALYZE ANALYZE ANALYZE RTLC RTLC RTL Synthesis Incremental Distributed Distributed Partitioning & Routing (Velsyn) 25-35M gates per hour VELCC VELCC Chip Compile Scheduling Distributed SSRMODELGEN SSRMODELGEN Waveform Replay 6

9 Veloce Use Model In Circuit Emulation Transaction based Acceleration VirtuaLab Software Development Full-Chip Validation! OS and device drivers run on RTL processor model! VirtuaLAB peripheral models exercise interfaces! HW/SW debug performed with Codelink Codelink Multi-core SW/HW debug SoC PHY$ PHY$ PHY$ PHY$ CPU$ USB$ Ethernet$ SATA$ Display$ Processor$ Master$IF$ SlaveIF$ Slave$IF$ Slave$$IF$ Slave$IF$ Arbiter$ $$ Master$IF$ CPU$ Master$IF$ Software Memory Fabric$ Slave$IF$ UART$ Slave$IF$ GPIO$ Fabric$ PCI$ Express$ PHY$ 7

10 Emulation Uses! Execution of system level behavior, including software Connections to live data sources networks, peripherals, etc.! Early access for software developers Software Bring-up all code from reset to debug prompt Early driver development Including performance or throughput verification Diagnostics and low-level start-up code, boot loaders Debug and Trace Interfaces! JTAG Physical Virtual! ETM ARM embedded trace hardware! TARMAC ARM Trace ARM ACtivity logs for simulation and emulation! Codelink Trace based debug facility 8

11 Software Debug with Veloce JTAG Debug with Veloce Mentor Embedded Codebench Debugger Mentor Embedded Sourcery Probe isolve JTAG Veloce isolve JTAG Hardware Description Front of the Chassis Figure 2-1. Back of an isolve ARM Cortex JTAG Trace Chassis Emulator Connector The solution is equipped with a single 100-pin female AMP connector, which mates the VStation-legacy data cables of the Veloce emulators. The detailed pin-out is listed in the section Pinout of Veloce Connector on page 17. Front of the Chassis Figure 2-2 shows the front of the chassis. Figure 2-2. Front of the isolve ARM Cortex JTAG Trace Chassis From left to right there are: 12 Two female DSUB-9 connectors (labeled UART1 and UART2 ). A 20-pin standard ARM JTAG connector (internally labelled H1 ). JTAG Connector isolve ARM Cortex JTAG Trace User s Guide, V1.0.1 August

12 Mentor Embedded Sourcery Probes Physical JTAG Probe Embedded SW Debugger Design Memory Memory RSP CORE 0 CORE 1 PCIe / USB Peripheral Port JTAG JTAG Interface Memory Jtag Probe Memory 10

13 Probe Architecture " SW Debugger Provides the Visibility and Control # CodeBench Virtual Edition # Lauterbach TRACE32 " Virtual Probe is a Debug Server # VMAJIC + Sprite # T32MCIServer + HostMCI " CPU Provides Debug Controller # Accessed via JTAG Design Probe IP Mem CPU DBG JTAG Signal Interface Probe Logic Simplified CPU and Debug Controller Debug controller can: Jam opcodes into decoder Jam data on loads Intercept data on stores Register File Inst Pipe Inst. Dec. ALU JTAG Debug Control Ld/Str Mem Ifc 11

14 Debug Server Primitives Read a Register: Jam a store instruction Intercept the datum Return datum to the debugger Write a Register: Datum provided by the debugger Jam a load instruction Jam the datum Read Memory: Write the address to a register Jam a load (from mem) Jam a store Intercept the datum Return datum to the debugger Write Memory: Datum and address provided by debugger Write the address to a register Write the datum to a register Jam a store (to mem) Debug Server Execution Control Software Breakpoints: Read and save the opcode at BP address Write SWBP opcode CPU stops in debug mode at SWBP opcode HWBP & Watchpoints: Set an address comparator to break address Set control bits (exec, load, store, size, value, masks, ) Comparator match stops CPU in debug mode Stopping Execution: Wait for debug mode indication Save all register state Remove SWBPs and disable HWBP & WP Notify debugger Starting Execution: Install SWBP Configure HWBP and Watchpoint logic Restore all register state Exit debug mode 12

15 Software Debug with Veloce Virtual Probe Embedded SW Debugger RSP Design Virtual Probe JTAG Memory Memory CORE 0 CORE 1 PCIe / USB Peripheral Port JTAG Interface Memory Memory Virtual Probe Mentor Embedded Codebench Debugger Mentor Embedded Network Sourcery Probe Connection + Transactor isolve JTAG Veloce 13

16 Virtual Probe benefits! Virtual probe allows emulation clocks to be stopped Enables time for waveform uploads Debug, analyze hardware while system is stopped Compatibility with Transactors VirtualLAB compatibility! Design no longer tied to location of I/O card, physical probe! No need to maintain physical hardware Hardware Trace Modules! Many processor and SoC vendors have jhardware based trace facilities! Enables real-time and multi-core debug visibility Not possible with JTAG and interactive debug techniques! Generates large amounts of data As cores get faster, and there are more of them and memory bandwidth is not increasing this technique is getting less effective Large fast memories are needed for even short executions 14

17 ARM Trace Hardware! Part of the Core-Sight debug package! Trace Modules: ETM embedded trace module PTM program trace module ITM instrumentation trace module HTM AHB trace module (bus activity tracing) ARM Trace Modules! ETM Embedded Trace Module Collects register data and instruction execution Available for Cortex M, R and A5, A7, and A8! PTM Program Trace Module Collects instructions execution (branches only) Instructions between branches are assumed to execute and can be filled-in as a post processing step Available for Cortex A9 and higher processors! ITM Instrumentation Trace Module Allows program to store events (interrupt entry, lock acquisition, etc.) Provides rough timing of events! HTM AHB Trace Module Traces AHB bus infrastructure 15

18 ARM ETM example ARM ETM example! Data is collected in trace buffer (ETB) or sent to port to host system! Compressed, encoded data needs to be expanded and decoded to make use of the data! ETM is part of core-sight Core-sight needs to be implemented and configured ETM needs to implemented and configured! ETM is *very* configurable/programmable 16

19 A7 Configuration and programming registers A7 ETM Registers - continued 17

20 A7 ETM registers - continued A7 ETM registers - continued 18

21 A7 ETM Registers - continued A7 ETM Registers - continued 19

22 A7 ETM Registers the fine print ETM - Summary! ETM is complex! Newer larger processors, multi-core systems generate too much data and are migrating to PTMs (instruction only)! Once configured and programmed correctly emits compressed encoded data In emulation/simulation needs to be sent to decompress/decode program for interpretation! Debugging in emulation is preferred over debugging in silicon In emulation, hardware configuration bugs can still be fixed Many SoCs go to market with limited or non-functional ETMs due to insufficient testing! Not practical to be used early in design cycle for debug 20

23 TARMAC! Trace ARM ACtivity Simulation/emulation debug facility for tracing code execution on ARM processors! Text file which shows many artifacts of code execution Instructions executed Register updates Memory references Mode changes Exceptions! In simulation fairly straight forward to configure and use Emulation is a bit more complex Limits performance in emulation! Included in DSM models of the processor Required TARMAC module for RTL processors and emulation TARMAC example Register Regiter New Update Value Record Memory Size Operation (in access bytes) Data Instruction Physical Simulation/ Processor Opcode Disassembly (Read/Write) record of access Address state taken/skipped Physical Virtual address address and operands Emulation record time (ARM/THUMB) 21

24 TARMAC! Text based file! Generated during simulation/emulation! Multiple lines per instruction About 100 bytes per instruction 10 million instructions = ~ 1 gigabyte file! Great facility for short runs (simulation) Impractical for long runs (emulation) State trace debug Veloce Codelink Transactor! Collects data from in and around the processor GP registers Instruction/data bus Debug signals Codelink Processor Logger (SCE-MI pipe) Replay Database! Sends signal data over SCE-MI pipe to host! Host process converts it to replay database Processor/memory state change events 22

25 State trace debug Waveform View SW Debugger Codelink Replay Server Standard GDB RSP interface Replay Database! After emulation, replay database can be used to drive a virtual target Benefits of trace based debug! State of system can be reconstructed to any point in time in the emulation Enabling forward and backward debug operation! The state information can be used to provide a processor trace, like ETM or TARMAC! Debug of the design can be done off-line More on this later 23

26 Trace Debug! Compressed/encoded file used by Codelink (mentor graphic product) logs ~3 bytes per instruction! When running emulation at ~1 MHz and assuming 1 instruction per clock (optimistic) Results in 10 Gbytes per hour of emulation! Too large for long emulations Flight Recorder Mode Tracing Trigger Point Waveforms Processor State User selected buffer size May be larger/smaller than waveform buffers ~15K - ~100K instructions 24

27 Selective Tracing Waveforms Processor State Area of interest Begin log End log Summary! Debug methods JTAG Virtual JTAG Trace based state reproduction! Trace Systems ETM (hardware trace systems) Tarmac (simulation/emulation trace systems) 25

28 SESSION 1 TRADE-OFFS DEBUG AND TRACE When available in design cycle! Debug methods JTAG Virtual JTAG Trace based state reproduction! Trace Systems ETM (hardware trace systems) Tarmac (simulation/emulation trace systems) 26

29 Timeline TARMAC/Codelink JTAG/ETM Prototype FPGA prototype Emulation Simulation Timeline! Codelink/TARMAC Are available as soon as processor is running instructions No practical for FPGA prototypes or live silicon! JTAG/ETM Are available after Coresight is integrated into the design a dfull configured and debugged Often only available late in emulation cycle 27

30 Configuring Coresight! Multiple IP blocks instantiated around the processor core! Programming configuration registers! Defining ROM table Coresight Logic (Multi-core system) 28

31 Coresight ROM Table Capability! Codelink Full debugger view Instruction/data trace! Tarmac Instruction/data trace! JTAG probe Full debugger view Instruction/data trace! ETM Instruction/data trace 29

32 Performance! Codelink 20 to 25 Million instructions per second! JTAG/Probe 1 to 2 Million instructions per second Capacity! Codelink 3 bytes per instruction 1 hour of emulation = 10 Gbytes! ETM Undefined! TARMAC ~150 bytes per instruction 1 minute of emulation = 10 Gbytes! JTAG N/A 30

33 Codelink/Veloce Flight Recorder Mode Trigger Point Waveforms Processor State User selected buffer size May be larger/smaller than waveform buffers ~15K - ~100K instructions Selective Logging Waveforms Processor State Area of interest Begin log End log 31

34 ETM! Provides triggers for enabling or disabling of data collection Facility varies by processor and coresight implementation Intrusiveness! Codelink Completely non-intrusive! TARMAC Completely non-intrusive! ETM Completely non-intrusive! JTAG/Probe Stops processor execution while requiring remainder of design to be clocked Millions of clocks to perform simple operations May impact execution by varying timing of code execution 32

35 JTAG Intrusiveness! A single JTAG transaction takes 150,000 to 200,000 clock cycles Read Register Step Instruction! A high level operation can take over 1,000,000 Step 1 line of code Read several memory locations! Processor is suspended in Debug mode, design continues to be clocked State of ASIC and peripherals may advanced! JTAG introduces non-determinism When you press on a debugger button decides when the operation takes place Number of design clocks run between button pushes will vary This might impact your debug ability Trace of JTAG debugger on Veloce! 5 to 20 clocks of advancing the circuit! 150,000 to 200,000 clocks of JTAG operation 33

36 153,000 clocks of debugger overhead Design Activity Debug Overhead 18 clocks of design activity 34

37 Timeline TARMAC/Codelink JTAG/ETM Prototype FPGA prototype Emulation Simulation SESSION 2 CONCURRENT DEBUG OF MULTIPLE CORES 35

38 Multi-Core Debug! Multi-core problems are synchronization and timing problems Run and debug the code on a single core, using traditional techniques, to address all functional problems! Bare metal level Correlate program artifacts! Process/thread/task level Must resolve process details for correlation Relevant data is not on a per core basis, but is per process/ thread/task Note that processes migrate from core to core over time Time correlation! ETM trace file Time stamps are not global or absolute (unless specifically modified by SoC developer) so files from multiple cores cannot be correlated! TARMAC trace file Timestamps can be used to correlate execution artifacts 36

39 Time correlation! Codelink Automatically correlates times across cores Native debugger displays multiple cores concurrently Advances state across cores correctly Works well at bare metal level, limited support at the process level! JTAG Can stop multiple cores at the same time with cross triggers when implemented in hardware If not, there is no way to stop multiple cores at the same time Cannot correlate state with hardware (hardware advances while processor state is extracted) Most debugger front-ends can only display one core at a time Determinism! Synchronization bugs are non-deterministic! Codelink Within a given trace it will be deterministic and repeat the bug identically Allows systematic analysis of the problem! JTAG May not repeat the bug from run to run One bug may exhibit different symptoms depending on alignment of events making debug harder 37

40 Codelink Debugger View Multi-Core Debug! Debug issues will revolve around Shared data regions Synchronization primitives Locks, semaphores, atomic operations, barriers, etc! Putting watch points on shared data and sync. primitives can give insight need! Adding log (printf) events on the same data may expose patterns leading to the problem! Displaying sync events on a timeline can help 38

41 View Synchronization events over time View process data over time across cores 39

42 Debugging Multi-core! Multiple concurrent interacting state machines Same thing as debugging hardware Similar techniques can be applied! Sourcery Analyzer Shows software events on a timeline Enables hardware like debug for multi-core problems! Codelink databases can be imported into sourcery analyzer to enable multi-core debug Can be time synchronized at the source level back to source views in debugger SESSION 2 CORRELATION OF HW AND SW VIEWS 40

43 Cross Domain Views! JTAG triggering across domains! Codelink cross tool cursors! ETM time stamps! Tarmac time stamps Hardware/Software Correlation 41

44 Processor Trace (Tarmac) View! Trace files can be coordinated with other debugger views! Trace files can be generated from Codelink trace files or native tarmac files can be included Design Visibility Core0 Cache miss counter 42

45 ETM data correlation! ETM data has a 48 bit timestamp Timestamp derivation is implementation dependent By default is relative, not absolute Enables one to determine time differences, but not correlate with other views Some implementations can use higher fidelity/correlated timestamps Ask your SoC developer Tarmac time stamps! Early tarmac files had timestamps But they were not initialized correctly and cannot be used in any way! ARM has fixed this issue and timestamps in tarmac files should match emulation time Waveforms, list outputs and reports may be correlated by timestamp 43

46 SESSION 2 EFFICIENT UTILIZATION OF EMULATION RESOURCES Traditional Debug! Step through code Break on next instruction/line Run for a very short period of time Stop at next instruction/line More time spent stopped than running! Run to a breakpoint Set breakpoint Run Stop at breakpoint State of processor/software explored More time spent stopped than running 44

47 JTAG Debug with Emulation! Software debug is primarily spent examining a stopped system! When an emulator is stopped it is running at 0 MHz One user at a time Off-line Debug with Codelink! Codelink runs jobs to completion, then moves to the next job! The same emulation resource can support 10 times as many software debug sessions Many users at the same time 45

48 Logged output! Most complex software systems have some type of logging system for debug purposes! In simulation/emulation there is usually not a stdout! Codelink enables a virtual console Allowing logging to be used as a debug tool! Use a null printf to maximize performance #define printf null_printf null_printf(format_str, ) { return; } Print output! If a printf routine is part of the source code, resulting print statements can be directed to a terminal window 46

49 Logged output! Add additional printf statements through debugger No need to recompile source No need to re-run emulation! As additional logging is needed, simply add in the debugger! Re-compile and re-emulate only to determine if a code change has corrected a problem If more logging is needed, add it virtually If you need to back up, don t restart, just go backwards! Results in fewer emulations needed to debug a problem Our results show about 1/4 th as many iterations needed Overall efficency! Fewer iterations through the emulator reduces emulation demand by about 75%! Off-line debug reduces emulation demand by a factor of 5 to 10! Conservatively, one design slot on the emulator can support at least 10 software developers concurrently 47

50 Questions/Comments! Thank you for attending 48