Welcomes PRACE/LinkSCEEM 2011 Winter School Jacques Philouze Vice President Sales & Marketing

Transcription

1 Welcomes PRACE/LinkSCEEM 2011 Winter School Jacques Philouze Vice President Sales & Marketing

2 Content Company Background Products in more depth Allinea OPT (Optimization and Profiling Tool) Allinea DDT (Distributed Debugging Tool) Presentation Let s practice on some examples

3 New Market Drivers Software has become the #1 roadblock Many applications will need a major redesign - IDC HPC Update, June 2010 Most ISV codes do not scale High programming costs are delaying GPU usage Development tools are a vital part of the solution

4 Allinea Software HPC tools company since 2001 Allinea DDT the scalable parallel debugger Allinea OPT the optimization tool for MPI and non-mpi Allinea DDTLite the parallel debugging plug-in for Microsoft Visual Studio Large European and US customer base 12 of top 20 systems run Allinea DDT in EMEA World's only Petascale debugger! Most scalable and cost effective debugger for Cuda Users debugging regularly at all scales at 1 or 100,000 cores.. but also easy to use on small clusters!

5 Academic Over 200 universities Clients and Partners Major research centres ANL, EPCC, CEA, Genci, Juelich, NERSC, ORNL... Aviation and Defence Airbus, AWE, BAE, Dassault, DLR, EADS... Energy CGG Veritas, IFP, Total... EDA Cadence, Intel, Synopsys Climate and Weather UK Met Office, Meteo France, NOAA

6

7 Optimization & Profiling Tool A powerful Optimization and Profiling Tool Supports MPI profiling Supports Scalar profiling Supports internal sampler, GProf and PAPI Cross-platform support Most flavours of Linux GNU, IBM, Intel, PGI and PathScale compilers x86, x86-64, ia64, PowerPC, Bluegene/P and Cell architectures Most major MPIs OpenMPI, MPICH, MPICH2, Intel, HP, Bull, LAM, SGI and SCALI Support for major scheduling systems

8 OPT Preparation Allinea OPT services need to run on physical server PostgreSQL database Logging server GUI server (client connections) Allinea OPT is a link time library Support for major MPIs or scalar OPT MPI library sits between program and MPI No need to carry out a full recompile, just re-link

9 OPT Launching Jobs Supports both GUI and CLI job launch GUI can be configured per job Launch method local, scheduler Enable sampler Sample interval Snapshot interval #procs stdin/out/err MPI functions CLI business as usual Use scheduler as normal Use your own launch script Uses opt.conf for configuration (user or system)

10 OPT Session Manager Web services interface Allows connectivity to remote GUI server Authentication via database or PAM Local Server requires no additional authentication

11 OPT Session Manager Web services interface Allows connectivity to remote GUI server Authentication via database or PAM Local Server requires no additional authentication

12 OPT Session Manager Web services interface Allows connectivity to remote GUI server Authentication via database or PAM Local Server requires no additional authentication Collect and analyse data from multiple sources Different clusters within environment Different compilers and their options Different MPIs and their options Different number of MPI processes Offers a TRUE Cloud enabled tool

13 OPT Standard Features MPI Timeline view

14 OPT Standard Features Metric Time View

15 OPT Standard Features Metric Statistics

16 OPT Standard Features Histogram

17 OPT Standard Features Job Comparison

18 OPT Bigger Picture FOCUS is the key... Too much visual information can be confusing Good parallel tools should make things simple Target the areas which consume most time! Functions which you spend most run-time in! Allinea OPT embraces this Focused approach See the big picture first Drill down successively for more information.. Don t drown users in too much data Move to a mixture of sampling and selective tracing

19 OPT Call Graph Highlights problem functions Directly linked with the time line Works with all of the following MPI function calls MPI Overview Bytes in/out Computation and communication time Profiling GProf Sampler Snapshot details (hw counters)

20 OPT Call Graph Example Select Communication and Computation time Flat Profile select function with greatest Self time Select Average, Self Seconds and Depth=3 The red cells are areas that are most significant Most of our time is spent in MPI_Alltoall Select Cumulative, see who is calling MPI_Alltoall Change depth to 4 to increase visibility Double click block_mtranspose Zoom in

21 OPT Call Graph Example Select Communication and Computation time Flat Profile select function with greatest Self time Select Average, Self Seconds and Depth=3 The red cells are areas that are most significant Most of our time is spent in MPI_Alltoall Select Cumulative, see who is calling MPI_Alltoall Change depth to 4 to increase visibility

22 OPT Call Graph Example Select Communication and Computation time Flat Profile select function with greatest Self time Select Average, Self Seconds and Depth=3 The red cells are areas that are most significant Most of our time is spent in MPI_Alltoall Select Cumulative, see who is calling MPI_Alltoall Change depth to 4 to increase visibility Double click block_mtranspose We now have bigger picture Far more Focused

23 OPT Call Graph Example Select Communication and Computation time Flat Profile select function with greatest Self time Select Average, Self Seconds and Depth=3 The red cells are areas that are most significant Most of our time is spent in MPI_Alltoall Select Cumulative, see who is calling MPI_Alltoall Change depth to 4 to increase visibility Double click block_mtranspose We now have bigger picture Far more Focused Could we have worked this out from a MPI timeline?

24 OPT Selective Logging Collecting far too much data!! Allinea OPT selective logging Logging is on by default On at MPI_Init Off by MPI_Finalize

25 OPT Selective Logging Collecting far too much data!! Allinea OPT selective logging Logging is on by default On at MPI_Init Off by MPI_Finalize Requirements C/C++ need to replace mpi.h for optmpi.h Turn ON and OFF at will In C/C++ OPT_Start_Logging(); and OPT_Stop_Logging(); In Fortran call OPT_Start_Logging and call OPT_Stop_Logging

26 OPT User Defined Regions Need to highlight a specific area of code!! Allinea OPT has user defined regions Off by default

27 OPT User Defined Regions Need to highlight a specific area of code!! Allinea OPT has user defined regions Off by default Requirements C/C++ need to replace mpi.h for optmpi.h Needs to be turned ON and OFF In C/C++ OPT_Start_Region( My Region ); and OPT_Stop_Region(); In Fortran call OPT_Start_Region('My Region') and call OPT_Stop_Region

28 Try it! Allinea OPT available for download at

29 Debugging at Target-Scale Hybrid code Lindon Locks Pre-sales engineer

30 Background Debugging: A necessary part of development Reproducing and fixing software problems Complexity of scaling and GPU architecture will introduce bugs Debuggers interactively examine processes and data Fastest way to debug with less chance of introducing more bugs Bugs at Target-scale need a debugger at Target-scale until recently debuggers limited to ~4,000-8,000 cores Bugs on GPUs need a debugger for GPUs until recently GPU software couldn't be debugged Allinea DDT is the first graphical debugger to do both

31 Problems at Target-Scale Increasing job sizes leads to unanticipated errors Regular bugs Data issues from larger data sets: i.e. garbage in..., overflow Logic issues and control flow Increasing probability of independent random error Memory errors/exhaustion: Random bugs! System problems: MPI and Operating System Pushing coded boundaries Algorithmic (performance) Hard-wired limits ( magic numbers ) Unknown unknowns

32 Bug Fixing Strategies: Part l Improved coding standards Unit tests, assertions and consistency checks Good practice: But tend to be single-process checks Parallel checks also valid and good practice Only checks for things you predict when developed Coverage is rarely perfect Unexpected problems, particularly random/system issues Often missed Debugger still required Combines well with debuggers Find why a failure occurs not just a pass/fail

33 Bug Fixing Strategies: Part ll Logging: printf and write The oldest debugger still in active use Tried and tested - as easy as hello world If you have good intuition into the problem Edit code, insert print, recompile and re-run Slow and iterative Use to log exceptions, progress or state Post-mortem analysis only Hard to establish real causal order of output of multiple processes Output can be lost by process termination Rapid growth in log output size Unscalable

34 Bug Fixing Strategies: Part lll Reproduce at a smaller scale Attempt to make problem happen on fewer nodes Often requires reduced data set the large one may not fit Didn't you already try the code at small scale? Smaller data set may not trigger the problem Does the bug even exist on smaller problems? Is it a system issue i.e. an MPI problem? Is probability stacking up against you? Example: 1 in 10,000 independent probability of error? Unlikely to spot on smaller runs without many many runs But near guaranteed to see it on a 10,000 core run What can a parallel debugger do to help? Debug at the scale of the problem - Now

35 Use a Parallel Debugger Many benefits to graphical parallel debuggers Large feature sets for common bugs Richness of user interface and real control of processes Historically all parallel debuggers hit scale problems Bottleneck at the frontend: Direct GUI nodes architectures Linear performance in number of processes Human factors limit mouse fatigue and brain overload Are tools ready for the task? Allinea DDT has changed the game

36 Distributed Debugging Tool Allinea DDT a mature and highly intuitive software tool Traditional focus has been HPC market Additional Client-Server capabilities Developing new features(multicore, GP-GPU) Cross-platform support Linux, AIX, Blue Gene/P, Super-UX, CLE GNU, Absoft, IBM, Intel, PGI, PathScale, Sun compilers x86, x86-64, ia64, PowerPC, UltraSparc, Cray XT4/5/6, NEC, NVIDIA CUDA architectures Across all MPIs OpenMP and Threads All major scheduling systems (PBS, LSF, SGE)

37 DDT : Basic Principles Sophisticated GUI helps the user to control parallel execution and helps to find and focus on potential problems User controls actions by groups.. Set breakpoints, step in / out, align stacks etc Focus in on individual threads / processes when necessary Create groups both Manually: select processes via drag and drop Automatically: by process stack, values etc

38 Start Debugging Ability to run in numerous modes Advanced interactive launch options or Simple clear interactive launch options

39 Options Select Change options

40 DDT Scheduler Launch Enhanced Queuing Submission Simple mechanism to customise options Submit non-mpi jobs to a queue Set pre-defined variables in submission script System administrator configures Alter the pre-defined variables at run-time User selectable values

41 Features Scalar Features Advanced C++ support including STL, namespaces, virtual functions and templates Advanced Fortran 90, 95 and 2003 support including modules, allocatable data, pointers and derived types

42 Features Scalar Features Advanced C++ support including STL, namespaces, virtual functions and templates Advanced Fortran 90, 95 and 2003 support including modules, allocatable data, pointers and derived types Multithreading & OpenMP features Perform actions individually or collectively

43 Features Scalar Features Advanced C++ support including STL, namespaces, virtual functions and templates Advanced Fortran 90, 95 and 2003 support including modules, allocatable data, pointers and derived types Multithreading & OpenMP features Perform actions individually or collectively MPI features Control processes Individually or by groups Visualize message queues

44 Breakpoints / Watchpoints Quickly and easily control you debug session Breakpoints Double click to add them For all processes, groups or a single process Enable/disable from window panel Add a condition Default settings (enable/disable)

45 Breakpoints / Watchpoints Quickly and easily control you debug session Breakpoints Double click to add them For all processes, groups or a single process Enable/disable from window panel Add a condition Default settings (enable/disable) Watchpoints Select from evaluate, locals or current line Set for a variable or expression Program halts if change occurs Single process only

46 Data Comparison Cross Process/Thread Comparison (CPC) Run from evaluate, locals or current line Variable or expression Create groups from results Visualize 3D OpenGL array viewer

47 Data Comparison Cross Process/Thread Comparison (CPC) Run from evaluate, locals or current line Variable or expression Create groups from results Visualize

48 Data Comparison Cross Process/Thread Comparison (CPC) Run from evaluate, locals or current line Variable or expression Create groups from results Visualize Multi Dimensional Array viewer 3D OpenGL array viewer Export csv file for post analysis

49 Memory Debugging Find memory leaks Or stop on read/write beyond end of array:

50 Scalable Process Control Parallel Stack View Find rogue processes quickly Identify classes of process behaviour Rapid grouping of processes Control Processes by Groups Set breakpoints, step, play, stop Scalable groups view: Compact group display

51 Large Array Support Browse arrays 1, 2, 3 dimensions Table view Filtering Look for an outlying value Export Save to a spreadsheet View arrays from multiple processes Search terabytes for rogue data: in parallel with v3.0

52 Checkpointing GDB Single session, no option to save/restore MPI support No thread support Still in the experimental stage

53 Checkpointing GDB Single session, no option to save/restore MPI support No thread support Still in the experimental stage OpenMPI/BLCR Prepared and ready for official OpenMPI release Ability to save/restore session Also in experimental stage

54 Checkpointing GDB Single session, no option to save/restore MPI support No thread support Still in the experimental stage OpenMPI/BLCR Prepared and ready for official OpenMPI release Ability to save/restore session Also in experimental stage

55 Petascale Debugging DDT is delivering Petascale debugging today 0,12 DDT Performance Collaborations with ORNL on Jaguar Cray XT and CEA 0,1 Logarithmic performance Many operations now faster at 220,000 than previously at 1,000 cores Time (s) 0,08 0,06 0,04 All Step All Breakpoint ~1/10 th of a second to step and gather all stacks at 220,000 cores 0, MPI Processes

56 Debugging GPU Applications

57 CUDA Debugging Options Old world printf NVIDIA SDK 3.2 allows this (new) but has limitations Fake it Run the kernel on the host x86_64 processor Languages often support targeting host CPU instead of GPU Different numeric precision different answer? Different scheduling different answer? A reasonable option for some bugs Or run on the GPU with Allinea DDT...

58 Debugging Kernels Debugging CPU and GPU concurrently Browse source, examine variables, control processes and threads Set breakpoints Or automatically stop on kernel launch Stop at a line of CUDA code Kernels stop when breakpoint reached Hover the mouse for more information Step a warp

59 Examine Thread Data At a glance display of variables Expressions, local variables and current line Also possible to edit values Displays the memory types Shared, parameter, constant, register

60 Kernel Scale Made Easy View all threads in parallel stack view At a glance, see all GPU and CPU threads together Links with thread selection Pick a tree node to select one of the CUDA threads at that location Full MPI support See GPU and CPU threads from multiple nodes

61 Current Limitations Mix of SDK and driver limitations Only one warp can be stepped per GPU at any time Rapid progress as SDKs are released Strong partnership with NVIDIA and others helping to extend capabilities Other languages and devices HMPP, PGI Accelerators, OpenCL Can only support when the compilers/drivers make it possible...

62 DDT CUDA News SDK 3.2 with DDT Released January 2011 Multi-device support Fermi and Tesla support CUDA Memcheck support for memory errors MPI and CUDA support for GPU clusters Breakpoints, thread control, and data evaluation Stop on kernel launch

63 Hands on session

64 Training at Allinea This workshop is part of Allinea regular training process Webinar Presentation Workshop Lecture Hands on On site Training

65 Expected walkthroughs Getting Started Straightforward Crashes Memory Errors Memory Leaks Incorrect Results GPU Debugging