TotalView on IBM PowerLE and CORAL Sierra/Summit

Transcription

1 TotalView on IBM PowerLE and CORAL Sierra/Summit Martin Bakal ScicomP 5/25/2016

2 Agenda Corporate Overview Coral Milestones TotalView New architecture Demo Questions

3 Company snapshot We are the largest independent provider of crossplatform software development tools and embedded components Founded: Headquarters: Employees: Offices Worldwide: 1989 Louisville, CO Our capabilities cover different languages, code bases, and platforms. We meet development where and how it happens.

4 Meeting customer needs with capabilities

5 Our products and services Tools Libraries Klocwork On-the-fly static code analysis for app security CodeDynamics Commercial dynamic analysis OpenLogic Support Enterprise-grade SLA support OpenLogicAudits Detailed open source license and security risk guidance TotalView for HPC Scalable debugging Zend Server Enterprise PHP app server Zend Studio PHP IDE Zend Guard PHP encoding and obfuscation SourcePro OS, database, network, and analysis abstraction for C++ Visualization Real-time data visualization at scale PV-WAVE Visual data analysis IMSL Numerical Libraries Scalable math and statistics algorithms HydraExpress SOA/C++ modernization framework HostAccess Terminal emulation for Windows Stingray MFC GUI components

6 TotalView for HPC Comprehensive multi-core and multi-threaded analysis and debug environment Thread specific breakpoints Control individual thread execution View thread specific stack and data View complex data types easily Integrated Reverse debugging Track memory leaks in running applications Supports C/C++ on Linux Allowing the business to have Predictable development schedules Less time spent debugging Platform coverage Linux, BG/Q, CUDA GPUs, Xeon Phi, Linux-PowerLE with GPUs, etc

7 LLNL/Sierra Focus Areas Collaborative work Rogue Wave, LLNL, IBM, Nvidia, RWTH Aachen Focuses on three areas OpenMP 4 + GPUs debugging MPI+GPU debugger performance and scalability EVAL (conditional breakpoints) performance and scalability DO NOT COPY OR REDISTRIBUTE WITHOUT WRITTEN PERMISSION 7

8 OpenMP 4 + GPUs Debugging OpenMP 4 debugging support (CPUs and GPUs) for Sierra Collaborate on OpenMP Debug API (OMPD) design Three phases Phase 1: TotalView/OMPD: OMP3.1/CPU, x86_64 Phase 2: TotalView/OMPD: OMP4/CPU/GPU, x86_64 Phase 3: TotalView/OMPD: OMP4/CPU/GPU, PowerLE Phase 1 progress to-date follows Draft of OMPD for OpenMP 3.1 completed RWTH Aachen implemented OMPD DLL for Intel OpenMP RTL TotalView/OMPD feature development progressing DO NOT COPY OR REDISTRIBUTE WITHOUT WRITTEN PERMISSION 8

9 OMP Control Vars & Meta Info Intel OMPD DLL currently returns no control variable information Meta information shows version #, ID, and DLL path DO NOT COPY OR REDISTRIBUTE WITHOUT WRITTEN PERMISSION 9

10 OMP Parallel Regions Parallel region hierarchy at thread, process and group widths Aggregated, process/thread list: #p:#t[dpid-range.dtid-range, ] DO NOT COPY OR REDISTRIBUTE WITHOUT WRITTEN PERMISSION 10

11 OMP Task Regions Task region display is similar to parallel region display, but shows the task relationships DO NOT COPY OR REDISTRIBUTE WITHOUT WRITTEN PERMISSION 11

12 OMP Threads Thread-centric views of information available from OMPD DO NOT COPY OR REDISTRIBUTE WITHOUT WRITTEN PERMISSION 12

13 OMP Stack Filtering Raw, unfiltered stack displays the OMP RTL stack frames OMP RTL frames typically uninteresting to users DO NOT COPY OR REDISTRIBUTE WITHOUT WRITTEN PERMISSION 13

14 OMP Stack Filtering OMPD allows the debugger to portably find and filter-out OMP RTL frames DO NOT COPY OR REDISTRIBUTE WITHOUT WRITTEN PERMISSION 14

15 OMP Master/Slave Stack Linking Stack hyperlink connects a slave s thread frame to its master s thread frame Selecting the frame jumps to the parent thread and stack frame that invoked the parallel region DO NOT COPY OR REDISTRIBUTE WITHOUT WRITTEN PERMISSION 15

16 OMP Master/Slave Stack Linking Clicking again climbs the parallel region tree, focusing on its parent DO NOT COPY OR REDISTRIBUTE WITHOUT WRITTEN PERMISSION 16

17 OMP Master/Slave Stack Linking Now at the root of the OMP parallel region tree DO NOT COPY OR REDISTRIBUTE WITHOUT WRITTEN PERMISSION 17

18 OMP Mangled Outlined-Functions OMP outlined-function name mangling is not standard DWARF could connect an outlined function to its containing function E.g., DW_AT_omp_outlined <containing-die> Instead of L_func_42 par_region0_1_2 Debugger could reliably show something like func (parallel region 1 at file.c#42) Needed A DWARF OpenMP proposal A compiler developer to produce the DWARF DO NOT COPY OR REDISTRIBUTE WITHOUT WRITTEN PERMISSION 18

19 OMP Variable Information Users have asked for OpenMP variable information E.g., private, shared, firstprivate, copyin, reduction, etc. Compile-time attributes of the variable that the compiler knows DWARF could represent these attributes DO NOT COPY OR REDISTRIBUTE WITHOUT WRITTEN PERMISSION 19

20 OMP4 + GPUs OMP4 compilers currently produce no DWARF for GPU code IBM is working on a solution OMPD currently supports only OpenMP 3.1 (no GPUs) Specification must be extended for OpenMP 4 + GPU Seeking OMP4+GPU OMPD implementation TotalView modifications DEVICE and TARGET region support CUDA/GPU support Depends on the OMP RTL execution model DO NOT COPY OR REDISTRIBUTE WITHOUT WRITTEN PERMISSION 20

21 OpenMP 4 Loose Ends Help push toward OMPD standardization OMPD for IBM/LOMP When IBM implements the DLL TotalView should be able to just use it OMP aggregated logical call tree Reassemble the structure of an executing OMP4 program into a logical call tree DO NOT COPY OR REDISTRIBUTE WITHOUT WRITTEN PERMISSION 21

22 MPI+GPU Debugging at Scale Performance, scalability, and functionality on MPI+GPU targets Two phases of Application Driven Tuning (ADT) with GPUs Phase 1: Linux-x86_64 (LULESH/RAJA, HYPRE, LAMMPS) Phase 2: Linux-PowerLE (other benchmarks) NVidia CILP allows MPI processes to share GPUs on a node CILP (hardware pre-emption) CUDA Debug API Limitations Requires creating a debug agent process per target process TotalView creates a bushier MRNet tree Future work Fix the API to support true multi-process debugging Add support for MPS debugging DO NOT COPY OR REDISTRIBUTE WITHOUT WRITTEN PERMISSION 22

23 EVAL Point Performance and Scalability Support evaluating conditional breakpoints in the debugger servers Allows the interpreter to run in parallel in the servers Client contains heavyweight stuff: symbol data, compilers, IL generator Server remains lightweight adding a small IL interpreter TV Client Lex, Parse, Compile, Generate IL IL Interpreter Broadcast IL TV Server TV TV Server Server IL TV Server Interpreter IL TV Server IL TV Server Interpreter Interpreter IL TV Server TV Server IL Interpreter IL Interpreter IL Interpreter IL Interpreter Interpreter DO NOT COPY OR REDISTRIBUTE WITHOUT WRITTEN PERMISSION 23

24 Aggregation TotalView has been enhanced to add new types of aggregation Aggregated process and thread status New root window CLI dstatus Aggregated stack back trace Graphical call tree window CLI dwhere Aggregated data CLI dprint DO NOT COPY OR REDISTRIBUTE WITHOUT WRITTEN PERMISSION 24

25 Focus on Data Aggregation TotalView dprint command Allows the ability to get data about data and array from all threads pretty easily Added support for aggregated data collection On CUDA still prints for each thread dprint -g agg_1 Variable Focus: 64:32000[ ] 0x (0) : 64:16000[0-63.1, , , , ,...] 0x (1) : 64:16000[0-63.2, , , , ,...] One line for eachunique value of the variable # of MPI rank:# of threads[mpi rank range.threads Portable across platforms and will be supported on Linux-PowerLE CORAL Sierra/Summit Data on specific value

26 Licensing Licensing is an issue Flexera doesn t support Power with their FlexNet Publisher product This means we have to time bomb the product after a year 26

27 New architecture

28 New Architecture in building UI Used to be one large application Front-End Back-End Process Qt 4 Based Front-End TotalView Debug Interface (TVDI) Transport Mechanism Communication Channel TotalView Debug Wire Protocol (TVDWP) Transport Mechanism TotalView Debug Engine Interface (TVDEI) TotalView Debugger Engine (TVDE) TotalView Debug Server Process Process

29 Why does the architecture matter This isn t short term thinking No need for XWindows on target platforms Performance Scalability Platforms More effective debugger on all platforms Easier 3rd party integrations

30 30

31 Multi-threading made easier How do you debug a problem in a 50 thread application that occurs in 1 thread? Without TotalView, Set a breakpoint in code Run and hope you hit the right thread With TotalView set thread specific breakpoints Better multithreaded debugger Understand the state of all of your threads Focus on specific threads View stack and data Built to scale to HPC and leveraging that in mainstream commercial envs

32 Viewing complex data types How do you inspect complex data types for changes? Without TotalView, Look through pointer at memory Map to the data structures Recreate the data type by hand With TotalView View the data structure directly users get to focus on debugging Complex data types support includes STL collections Large multi-dimensional arrays Boost collection classes C++ 11 specific types

33 Reverse debugging How do you isolate an intermittent a failure? Without TotalView, Set a breakpoint in code Realize you ran past the problem Re-load Set breakpoint earlier Hope it fails Keep repeating With TotalView Start recording Set a breakpoint See failure Run backwards/forwards in context of failing execution Reverse Debugging Re-creates the context when going backwards Focus down to a specific problem area easily Saves days in recreating a failure

34 Memory Analysis How do you identify buffer overflows? Runtime Memory Analysis : Eliminate Memory Errors Detects memory leaks before they are a problem Explore heap memory usage Features Detects Malloc API misuse Memory leaks Buffer overflows Low runtime overhead Easy to use Works with vendor libraries No recompilation No instrumentation

35 Regression Testing How do you make sure a bug you fixed never returns? Build a regression test Issue is it typically is time consuming What is the method to build a regression test? Use the tools that helped you find it How do you run a regression test? Invoke it during your build process Enter TotalView scripts Command line driven Access to application internals Same commands as in the debugger!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Print!! Process:!./server (Debugger Process ID: 1, System ID: 12110)! Thread:! Debugger ID: 1.1, System ID: ! Time Stamp:! :04:09! Triggered from event:! actionpoint! Results:! foreign_addr = {! sin_family = 0x0002 (2)! sin_port = 0x1fb6 (8118)! sin_addr = {! s_addr = 0x6658a8c0 ( )! }! sin_zero = ""! }!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

36 Scales to meet your need Support debugging on thousands of cores MRNet is built to multicast Aggregates data to/from cores Remote Display Client Debug on a remote machine Easy to configure Focus on your debugging