MPI Performance Analysis TAU: Tuning and Analysis Utilities

Transcription

1 MPI Performance Analysis TAU: Tuning and Analysis Utilities Berk ONAT İTÜ Bilişim Enstitüsü 19 Haziran 2012

2 Outline TAU Parallel Performance System Hands on: How to use TAU Tools of TAU Analysis and Visualiza<ons TAU Traces and Profiles 2/36

3 TAU Parallel Performance System Features: Tuning and Analysis U<li<es 15+ year project in University of Oregon, Research Center Jülich, LANL Open source and freeware analysis u<li<es Portable performance tracing and profiling capabili<es Works with Fortran, C/C++, Java, Python Automa<c instrumenta<on of user code Own (TAU) visualiza<on tools (pprof, Paraprof) Support for other visualizing programs (Jumpshut, Vampir) 3/36

4 TAU Parallel Performance System Features: Currently only support MPI tracing Use own (TAU) compiler wrappers Can be built with several MPI implementa<ons Can also work with different networks (ethernet, infiniband, myrinet,...) Targets a general complex system computa<on model: 1. En;;es: nodes / contexts / threads 2. Mul;- level: system / so]ware / parallelism 3. Measurement and analysis abstrac;on 4/36

5 General Complex System Model Node: physically dis<nct shared memory machine Message passing node interconnec<on network Context: dis<nct virtual memory space within node Thread: execu<on threads (user/system) in context 5/36

6 TAU Parallel Performance System Targets: Mul<- level performance instrumenta<on Mul<- language automa<c source instrumenta<on Flexible and configurable performance measurement Widely- ported parallel performance profiling system Computer system architecture and opera<ng systems Different programming languages and compilers Support for mul<ple parallel programming paradigms Mul<- threading, message- passing, hybrid 6/36

7 TAU Instrumentation TAU Flexible instrumenta<on mechanisms at mul<ple levels: Source code Object code Manual (TAU API, TAU Component API) Automa<c» C, C++, F77/90/95 (Program Database Toolkit (PDT))» OpenMP (direc<ve rewri<ng (Opari), POMP spec) Pre- instrumented libraries (e.g., MPI using PMPI) Sta<cally- linked and dynamically- loaded (e.g., Python) Executable code Dynamic instrumenta<on (pre- execu<on) (DynInstAPI) Virtual machine instrumenta<on (e.g. Java using JVMPI) 7/36

8 TAU Performance System Architecture 8/36

9 TAU Configuration configure [OPTIONS] - c++=<cxx>, - cc=<cc> Speci] C++ and C compilers - pthread, - sproc Use pthread or SGI sproc threads - openmp Use OpenMP threads - jdk=<dir> Specify Java instrumenta<on(jdk) - opari=<dir> Specify loca<on of Opari OpenMP tool - papi=<dir> Specify loca<on of PAPI library - pdt=<dir> Specify loca<on of PDT tools - dyninst=<dir> Specify loca<on of DynInst Package - mpi[inc/lib]=<dir> Specify MPI library instrumenta<on - shmem[inc/lib]=<dir> Specify PSHMEM library instrumenta<on - python[inc/lib]=<dir> Specify Python instrumenta<on - or=<dir> Specify loca<on of OTF trace library - arch=<dir> Specify architecture explicitly 9/36

10 TAU Configuration configure [OPTIONS] - TRACE Generate binary TAU traces - PROFILE (default) Generate profiles (summary) - PROFILECALLPATH Generate call path profiles - PROFILEPHASE Generate phase based profiles - PROFILEMEMORY Track heap memory for each rou;ne - PROFILEHEADROOM Track memory headroom to grow - MULTIPLECOUNTERS Use hardware counters + ;me - COMPENSATE Compensate ;mer overhead - CPUTIME Use user;me + system ;me - PAPIWALLCLOCK Use PAPI s wall clock ;me - PAPIVIRTUAL Use PAPI s process virtual ;me - LINUXTIMERS Use fast x86 Linux ;mers 10/36

11 Configuring TAU for Tracing and Profiling bash: tau / $./configure - c++=icpc - cc=icc - fortran=intel - pdt=/rs/progs/pdtoolkit/3.15/ - arch=x86_64 - mpi - mpiinc=/rs/progs/intel/impi/3.1/include64 - mpilib=/rs/progs/intel/impi/3.1/lib64 - PROFILE - PROFILECALLPATH - PROFILEPARAM - PROFILEMEMORY PROFILEHEADROOM - MULTIPLECOUNTERS - PAPIVIRTUAL - LINUXTIMERS - CPUTIME - PAPIWALLCLOCK - MPITRACE - TRACE 11/36

12 Configuring TAU for Tracing and Profiling Make and Install TAU bash: tau / $ make install Will compile TAU and generate TAU Makefile under x86_64/lib/ Makefile.tau- memory- headroom- callpath- param- linux;mers- icpc- mpi- papi- perf- pdt- profile- trace- mpitrace 12/36

13 Configuring TAU for Tracing and Profiling Configuring TAU for OpenMP Profiling and Tracing(?) bash: tau / $./configure - c++=icpc - cc=icc - fortran=intel - pdt=/rs/progs/pdtoolkit/ arch=x86_64 - openmp - opari - PROFILE - PROFILECALLPATH PROFILEPARAM - PROFILEMEMORY PROFILEHEADROOM - MULTIPLECOUNTERS - PAPIVIRTUAL - LINUXTIMERS CPUTIME - PAPIWALLCLOCK - TRACE 13/36

14 Configuring TAU for Tracing and Profiling Make and Install TAU bash: tau / $ make clean bash: tau / $ make install Will compiler TAU again with new selngs and generate new TAU Makefile under x86_64/lib/ Makefile.tau- memory- headroom- callpath- param- linux;mers- icpc- pdt- papi- perf- openmp- opari- profile- trace 14/36

15 Visiual Configuration of TAU bash: $./tau_setup 15/36

16 Compiling with TAU Compiling your applica;on with TAU compiler in your makefile MPICC=tau_cc.sh export TAU_MAKEFILE=/RS/progs/tau/x86_64/lib/Makefile.tau- UYBHM_IntelTingz icpc- papi- shmem- mpi- pthread- pdt- openmp- profile- trace- mpitrace default: my_mpi_omp.x.o.c: $(MPICC) $(CFLAGS) - c $< my_mpi_omp.x: my_mpi_omp.o $(MPICC) $(CFLAGS) $^ $(LFLAGS) - o $@ 16/36

17 Compiling with TAU Auto Instrumenta;on using TAU_COMPILER $(TAU_COMPILER) stub Makefile variable in release Invokes PDT parser, TAU instrumentor, compiler through tau_compiler.sh shell script Requires minimal changes to applica<on Makefile Compila<on rules are not changed User stes TAU_MAKEFILE and TAU_OPTIONS environmental variables User renames the compilers ( CC=icc CC=tau_cc.sh) 17/36

18 How to Use TAU Login to your UHEM node using - X with ssh : bash: $ ssh - X du??@wsl- node??.uybhm.itu.edu.tr or use your PuTTY program in your Windows with X11 forwarding in SSH sec;on. Copy example file tar to your directory bash: $ cd workshop bash: $ cp /RS/users/bonat/workshop/YAZOKULU/ tar. bash: $ tar - xvf tar bash: $ cd /mpi- analyze/tau 18/36

19 How to Use TAU Selng Up Environmental Variables: Use add- ITAC- to- my- PATH.sh script. Exit and login again with - X bash: $./add- TAU- to- my- PATH.sh Adding source TAU line to your.bashrc and/or.bash_profile export PATH=/RS/progs/pdtoolkit/3.15/x86_64/bin:$PATH export PATH=/RS/progs/tau/2.19.1/x86_64/bin:$PATH export PATH=/RS/progs/papi/3.6.2/bin:$PATH export PATH=/RS/progs/perfctr/2.6.39/bin:$PATH export LD_LIBRARY_PATH=/RS/progs/papi/3.6.2/lib: $LD_LIBRARY_PATH export LD_LIBRARY_PATH=/RS/progs/perfctr/2.6.39/lib: $LD_LIBRARY_PATH 19/36

20 How to Use TAU Using TAU for Measurement bash: $ export TAU_MAKEFILE=Makefile.tau- UYBHM_IntelTingz icpc- papi- shmem- mpi- pthread- pdt- openmp- profile- trace- mpitrace bash: $ export TAU_PROFILE=1 bash: $ export TAU_COMM_MATRIX=1 Compiling your applica;on for TAU profiling and tracing bash: $ tau_cc.sh heat2d.c o heat2d.x bash: $ mpirun - np 8./heat2d.x 20/36

21 How to Use TAU Running your applica;on for performance analysis with TAU bash: $ export TAU_MAKEFILE=Makefile.tau- UYBHM_IntelTingz icpc- papi- shmem- mpi- pthread- pdt- openmp- profile- trace- mpitrace bash: $ export TAU_PROFILE=1 bash: $ export TAU_COMM_MATRIX=1 bash: $ tau_cc.sh heat2d.c - o heat2d.x Submilng your program to LSF for TAU performance analysis bash: $ bsub < tau- submit.lsf 21/36

22 How to Use TAU NODE 0;CONTEXT 0;THREAD 0: %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call int main() C void prtdat(int, int, float *, char *) C MPI_Init() MPI_Finalize() MPI_Recv() MPI_Send() void inidat(int, int, float *) C MPI_Comm_size() MPI_Comm_rank() USER EVENTS Profile :NODE 0, CONTEXT 0, THREAD NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name E E E+04 Message size received from all nodes E E E+04 Message size sent to all nodes /36 bash: $ ls profile profile profile profile profile profile profile profile bash: $ pprof Profiling with pprof and paraprof

23 How to Use TAU Profiling with pprof and paraprof FUNCTION SUMMARY (total): %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call , int main() C ,397 2, MPI_Finalize() ,345 2, MPI_Recv() ,072 1, MPI_Init() void prtdat(int, int, float *, char *) C void update(int, int, int, float *, float *) C MPI_Send() void inidat(int, int, float *) C MPI_Comm_size() MPI_Comm_rank() FUNCTION SUMMARY (mean): %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call int main() C MPI_Finalize() MPI_Recv() MPI_Init() void prtdat(int, int, float *, char *) C void update(int, int, int, float *, float *) C MPI_Send() void inidat(int, int, float *) C MPI_Comm_size() /36 Trace Analyzer & Coll MPI_Comm_rank()

24 How to Use TAU Profiling with paraprof bash: $ ls profile profile profile profile profile profile profile profile bash: $ paraprof 24/36

25 How to Use TAU Profiling with paraprof 25/36

26 How to Use TAU Profiling with paraprof 26/36

27 How to Use TAU TAU Throvle Op;on Reducing overhead of TAU by using Throrle environmental variable bash: $ export TAU_THROTTLE=1 bash: $ export TAU_THROTTLE_NUMCALLS= bash: $ export TAU_THROTTLE_PERCALL=3000 bash: $ mpirun - np 8./heat2d.x 27/36

28 How to Use TAU tau_reduce: Rule- Based Overhead Analysis for pprof dump files (pprof - d) Analyze the performance data to determine events with high (rela<ve) overhead performance measurements Create a select list for excluding those events Rule grammer: [GroupName:] Field Operator Number GroupName: indicates rule applies to events in group Field: is an event metric arribute (from profile sta<s<cs) Operator: is one of >, <, or = Number is ant number 28/36

29 How to Use TAU tau_reduce: Rule- Based Overhead Analysis File #exclude all rou<nes that are members of TAU_USER and have #less than 1000 microseconds TAU_USER:usec < 1000 #exclude all rou<nes that have higher than 1000 microseconds and are #called only once. usec < 1000 & numcalls = 1 #exclude all rou<nes that have less than 1000 usecs per call OR #have a percent higher than 5 usecs/call < 1000 percent < 5 29/36

30 How to Use TAU tau_reduce: Rule- Based Overhead Analysis Usage bash: $ tau_reduce - f filename specify filename of pprof dump file - p print out all func<ons with their arributes - o filename specify filename for select file output (default: print to screen - r filename specify filename for rule file - v verbose mode (for each rule, print out rule and all func<ons that it excludes) bash: $ tau_reduce - f app.profile - r rulefile - o new- insrile 30/36

31 How to Use TAU tau_reduce: Rule- Based Overhead Analysis File bash: $ pprof - d > app.profile bash: $ cat rulefile usecs/call > bash: $ tau_reduce - f app.profile - r rulefile - o new- insrile bash: $ cat new- insrile BEGIN_EXCLUDE_LIST MPI_Finalize() void prtdat(int, int, float *, char *) C int main() C END_EXCLUDE_LIST 31/36

32 How to Use TAU Selec;ve Instrumenta;on InsRile content: BEGIN_INSTRUMENT_SECTION sta;c ;mer name= master file= heat2d.c line=95 to line=132 sta;c ;mer name= workers file= heat2d.c line=138 to line=200 END_INSTRUMENT_SECTION Compiling bash: $ tau_cc.sh - tau_makefile=makefile- - tau_op;ons=- optverbose - opttauselectfile=insrile heat2d.c - o heat2d.x bash: $ tau- compile- with- insrile insrile bash: $ mpirun - np 8./heat2d.x bash: $ pprof 32/36

33 How to Use TAU Selec;ve Instrumenta;on pprof output: FUNCTION SUMMARY (total):! ! %Time Exclusive Inclusive #Call #Subrs Inclusive Name! msec total msec usec/call! ! , TAU application! , workers! ,472 4, MPI_Recv()! ,317 2, MPI_Finalize()! ,425 1, void update(int, int, int, float *, float *) C! MPI_Init()! master! MPI_Send()! void inidat(int, int, float *) C! MPI_Comm_size()! MPI_Comm_rank()! 33/36

34 How to Use TAU TAU Tracing Data bash: $ ls tautrace trc tautrace trc tautrace trc tautrace trc tautrace trc tautrace trc tautrace trc tautrace trc events.0.edf events.1.edf events.2.edf events.3.edf events.4.edf events.5.edf events.6.edf events.7.edf bash: $ tau_treemerge.pl tautrace trc: 394 records read. tautrace trc: 670 records read. tautrace trc: 1270 records read. tautrace trc: 1270 records read. tautrace trc: 1270 records read. tautrace trc: 1270 records read. tautrace trc: 1270 records read. tautrace trc: 670 records read. 34/36

35 How to Use TAU bash: $ tau_treemerge.pl bash: $ tau2slog2 tau.trc tau.edf - o tau.slog Records read. 95% converted 7889 Records read. 97% converted 8050 Records read. 99% converted Reached end of trace file. SLOG- 2 Header: version = SLOG NumOfChildrenPerNode = 2 TreeLeafByteSize = MaxTreeDepth = 1 MaxBufferByteSize = Categories is ) MethodDefs is 0) LineIDMaps is ) TreeRoot is ) TreeDir is ) Annota;ons is 0) Postamble is 0) Number of Drawables = 3363 ;meelapsed between 1 & 2 = 68 msec ;meelapsed between 2 & 3 = 507 msec bash: $ jumpshot tau.slog2 TAU Tracing Data 35/36

36 How to Use TAU TAU with Jumpshot 36/36

37