Simplifying the Development and Debug of 8572-Based SMP Embedded Systems. Wind River Workbench Development Tools

Transcription

1 Simplifying the Development and Debug of 8572-Based SMP Embedded Systems Wind River Workbench Development Tools

2 Agenda Introducing multicore systems Debugging challenges of multicore systems Development tool requirements for multicore Multicore development solutions from Wind River Wind River Systems, Inc.

3 What Is Multicore? Multicore: A single chip containing multiple distinct processing engines, with multiple independent program counters P C BUS P C L Wind River Systems, Inc.

4 Freescale Multicore Solutions Freescale MPC8572: High-performance dual-core processor Security Acceleration XOR Pattern Matching Engine 2 x Table Lookup Unit 32 KB L1 I-Cache E500 Core 32 KB L1 I-Cache 1MB L2 Cache Coherency Module System Bus 32 KB L1 I-Cache E500 Core 32 KB L1 I-Cache DDR2/DDR3 SDRAM Controller DDR2/DDR3 SDRAM Controller Local Bus Performance Monitor DART, MPIC, 2 x PC, Timers, GPIO, Interrupt Control 4 x Gigabit Ethernet 4 x SGMII PCI Express PCI Express On-Chip Network PCI Express Serial RapidIO 4-ch. DMA 4-ch. DMA 8-lane SerDes Wind River Systems, Inc.

5 Multiprocessing Business Issues Benefit Promise of better products Performance, capability, cost The ability to provide Increased horsepower More cost-efficiency Heterogeneous systems Stronger partitioning Reliability via redundancy Cost Increased development cost, time, and risk Algorithms, partitioning Hardware and software design Software development challenges No guaranteed benefits Wind River Systems, Inc.

6 Multicore Configuration Options Symmetric Multiprocessing (SMP) Asymmetric Multiprocessing (AMP) SMP OS OS OS OS P P P P P P Identical processors/cores Uniform access to shared memory Single OS Effort required to make data or resources private Multithreaded work distributed by OS at run-time Changed programming paradigm Higher level of kernel overhead Processors/cores may be different Each one has local memory Each one has an independent OS or control software, may be different Connected for communication and synchronization Work must be partitioned at design time and assigned to processors Not easily portable to different hardware Wind River Systems, Inc.

7 Embedded Development Embedded testing requirements Higher quality needs vs. desktop software Run-time debugging Debugging code on a target embedded system Difficult issues that developers face Memory issues and leaks Timing issues with multitasking Random crashes Resource access issues, especially with multicore Side effects and unintended consequences Integration of third-party code Performance or throughput bottlenecks Wind River Systems, Inc.

8 Multicore Development Challenges Multiple interacting contexts Parallel execution of threads in SMP Independent processors in AMP Race conditions, deadlocks, memory synchronization SMP Increased debug complexity in multicore Debug can change behavior May be very difficult to reproduce Different behavior on single core vs. multiple cores Synchronization issues Start, stop, and stepping through code Stopping one core but other runs then stops Latency causes skid AMP Wind River Systems, Inc.

9 A Day in the Life of a Developer Difficulty isolating problem cause and or location Crash, slow performance/throughput, undesired effect Don t know where or how to start the debug process Debugging tricky or confusing timing problems Hardware and software interaction cause random crashes Debug methodology can change system behavior, masks issue or gives misleading results AND Multicore development and debug is hard Confusing environments with multiple interacting contexts printf does not provide sufficient debug information Wind River Systems, Inc.

10 Why Is Multicore Debug Difficult? Simple example: Two threads execute at the same time accessing shared variable Thread 1 runs on Core 1 C Code global_var_x = global_var_x + 1; printf( added 1 to global_var_x ); Assembly Code MOV global_var_x, r0 ADD #1, r0 MOV r0, global_var_x JSR 0x Thread 2 runs on Core 2 C Code global_var_x = global_var_x - 1; printf( subtracted 1 from global_var_x ) Assembly Code MOV global_var_x, r0 SUB #1, r0 MOV r0, global_var_x JSR 0x Global_var_x starts at 10 Thread 1 and 2 execute once Final value ( ) = Wind River Systems, Inc.

11 Why Is Multicore Debug Difficult? Simple example: Two threads execute at the same time accessing shared variable Thread 1 runs on Core 1 C Code global_var_x = global_var_x + 1; printf( added 1 to global_var_x ); Assembly Code MOV global_var_x, r0 ADD #1, r0 MOV r0, global_var_x JSR 0x Thread 2 runs on Core 2 C Code global_var_x = global_var_x - 1; printf( subtracted 1 from global_var_x ) Assembly Code MOV global_var_x, r0 SUB #1, r0 MOV r0, global_var_x JSR 0x Global_var_x starts at 10 Thread 1 and 2 execute once Final value ( ) = 10 No problem If thread 1 and 2 run sequentially Error If thread 1 and 2 run concurrently incorrect result will be obtained Debug method Difficult to debug with printf May work sometimes, fails sporadically Wind River Systems, Inc.

12 Why Is Multicore Debug Difficult? Simple example: Two threads execute at the same time accessing shared variable Thread 1 runs on Core 1 C Code global_var_x = global_var_x + 1; printf( added 1 to global_var_x ); Assembly Code MOV global_var_x, r0 ADD #1, r0 MOV r0, global_var_x JSR 0x Thread 2 runs on Core 2 C Code global_var_x = global_var_x - 1; printf( subtracted 1 from global_var_x ) Assembly Code MOV global_var_x, r0 SUB #1, r0 MOV r0, global_var_x JSR 0x Global_var_x starts at 10 Thread 1 and 2 execute once Final value ( ) = 10 No problem If thread 1 and 2 run sequentially Error If thread 1 and 2 run concurrently incorrect result will be obtained Debug method Difficult to debug with printf May work sometimes, fails sporadically Debug solution requirement Breakpoint setting that works across cores Ability to synchronize cores to expose bugs Example: Works in single core but not in dual core Visibility needed to all cores simultaneously Observe race conditions Wind River Systems, Inc.

13 Confusing Race Conditions in Multicore Race conditions are among the hardest multicore problems to solve. They can occur when multiple threads have simultaneous access to the same data. The cause and effect are typically millions of cycles apart. Time Processor 1 Processor 2 1 read 2 modify 3 read 4 modify 5 write 6 write Wind River tools are good for race condition debugging Wind River Systems, Inc.

14 Multicore Debug Requirements Flexible solutions Visualize system events and system mode debug Visualizing See operating system events Observe timing and sequence Spot deadlock, race, and starvation conditions System mode debug Trigger when breakpoint is hit View local and global variables, registers, cache, and system state Stop all threads and processor on all cores Permits examination of variables, register, cache, etc. Minimizes skid with synchronized stop of cores Wind River Systems, Inc.

15 Addressing Challenges with Wind River Workbench, On-Chip Debugging Edition Full-Featured Debugger for Multicore Device Software Development Common environment for multicore Hardware bring-up Platform development Application development Multicore debugging methods JTAG on-chip device debugging Workbench debugging via target agent Debugging up to 16 cores simultaneously Architecture flexibility for multicore Target OS including VxWorks, Wind River Linux, and others Wide range of target processor support Wind River On-Chip Debugging Wind River ICE Wind River Systems, Inc.

16 Wind River ICE 2 JTAG server debugs multiple devices on same scan chain Debug up to 16 devices simultaneously, connects to 128 cores Start all, stop all, and correlated multicore breakpoints Synchronize commands to multiple cores simultaneously Responsive and fast interface JTAG accelerator maximizes JTAG interface bandwidth Fast response with target system LAN Target Firmware 1 Target Firmware 2 Target Firmware 3 Target Firmware 4 JTAG Server JTAG Accelerator JTAG Connection Hardware Design Wind River Systems, Inc.

17 Workbench On-Chip Debugging Eclipse-based Single interface Built for multicore Centralized debugging Project-oriented Simplified development Scalable, robust Visualized multicore operation Wind River Systems, Inc.

18 System Viewer for Multicore Processors Synchronized Timestamping Tasks Running in Parallel Wind River Systems, Inc.

19 VxWorks SMP Debug Support On-chip debugging support For latest VxWorks 6.6 with SMP Wind River Workbench, On-Chip Debugging Edition 3.0 Breakpoint setting Coordinated across all cores Triggered regardless of which core hits the breakpoint Run control Supports all, e.g., single step, step into, step out of, etc. Moves from core to core, when thread is rescheduled Flexible system configuration SMP across all cores SMP on a subset of cores Multiple SMP instances, each on a subset of cores Wind River Systems, Inc.

20 System Mode Debugging Problem Multiple threads accessing a shared resource, e.g., a global variable Synchronized threads to access only one at a time Incorrect use, lack of use leading to problems that are some of the most difficult issues to find and fix in programming Process 1 Thread A Thread B Process 2 Thread C Thread D Wind River on-chip debugging Ability to stop all cores in system Ability to see what is running in other core(s) at time of problem Agent-based debug solutions unable to stop the system SMP OS Core 1 Core Wind River Systems, Inc.

21 Remote Debugging Breakpoint Hit on Core 1 Network Wind River ICE 2 Target Remote User Breakpoint Hit on Core 1 Better core synchronization for accurate visibility and operability Other cores stopped in short time when breakpoint hit on core 1 Closer synchronization of cores resulting in less skid Ability to see what is occurring on each core at a point in time Wind River Systems, Inc.

22 Multiple Context Debugging Simultaneous debugging Multiple targets, boards Multiple cores, processors Multiple processes, tasks, threads Source, register, and stack back-trace views in each debug context On-chip debugging with Wind River ICE 2 One scan chain, one ICE, multiple devices Start all, stop all, and multicore breakpoints with system mode debug Flexible breakpoint capabilities Hardware breakpoint support Thread qualified breakpoints Developer Host Debugger TCP Network Core 0 Debug Proxy TIPC Shared Memory Core 1 Debug Agent Wind River Systems, Inc.

23 Bay N etworks Bay N etworks Bay N etworks Multiboard Debug Scenario Debugging Scenario One Workbench instance One target manager Three concurrent connections Two target OSes Three different target processors Three different device software applications Multiple Contexts Processor, process, or thread Each context with a set of views Source Stack Registers Processes/Threads Qualify breakpoints on a process or specific thread Stop the entire process or an individual thread OA&M Card Linux-based Shelf Controller VxWorks-based Line Card ThreadX-based No Ethernet for debug Wind River Solution Benefits Single Eclipse-based user interface for debug Works across a variety of target systems, processors, and boards Wind River Systems, Inc.

24 Benefits of Wind River On-Chip Debugging Find root cause with system mode debugging Breakpoint stops entire processor Avoids skid with other running cores Identify bugs faster Clearly see the order of instruction execution on all cores Find problems quickly Work with simpler, intuitive Eclipse-based user interface One window for viewing activity on all cores Clearly identified and color-coded Speed development with remote debugging Around-the-clock-access collaborative debugging Instantaneous response time with JTAG server Wind River Systems, Inc.

25 Wind River Multiprocessing Leadership Wind River Workbench Market-Specific Middleware Integrated Horizontal Middleware VxWorks Linux Integrated Partner Hardware Broadest range of technology Operating system choice Multiprocessing architecture choice Multicore-enabled networking Interprocessor communication Debug, analysis, and diagnostics Broad processor and board support Most complete solution Extensive partner ecosystem Professional services capability Most reliable corporate partner Proven experience/industry leadership Worldwide support capability Participation in standards Large, stable, financially secure Wind River Systems, Inc.

26 Demo Block Diagram Ethernet JTAG Connection On-Chip Debugging Wind River ICE 2 MPC8572 Board Wind River Systems, Inc.

27 Wind River Systems, Inc.