Module I: Measuring Program Performance

Transcription

1 Performance Programming: Theory, Practice and Case Studies Module I: Measuring Program Performance 9

2 Outline 10 Measuring methodology and guidelines Measurement tools Timing Tools Profiling Tools Process monitoring and tracing tools System monitoring tools Hardware counter measurements Monitoring tools Code instrumentation Parallel performance measurements Guidelines and recommendations Tools for parallel monitoring Summary

3 Measurement Methodology 11 Quantifying performance is the first step in the application tuning process Important to set reasonable expectations for optimization Measurements should be made repeatedly to identify parts of the program that need to be optimized Proper choice of measurement characteristics suitable for a particular application Comparison of measurements to theoretical peak values

4 Timing measurements What to Measure Wall clock time for a single job (turnaround time) Wall clock time for multiple jobs (throughput measurements) Wall clock time for parallel runs (scalability measurements) Execution and computation rates MFLOPS (million floating point operations per second) MIPS (million instructions per second) IPC (instructions per cycle) Resource utilization Memory usage I/O utilization Network usage 12

5 Benchmarking Guidelines 13 Benchmark runs should adequately represent the use of the application Preferably only one parameter changing at a time Overhead of measurement should be considered Runs from tmpfs or from a locally mounted ufs System activities should be monitored The systems should not have any other computational jobs running during benchmarking System parameters and settings should be documented together with the results of the runs.

6 Functionality Timing tools Profiling tools Monitoring tools Measurement Tools Usage requirements Tools that can operate on optimized binaries Tools that require recompilation Tools that require source code instrumentation Parallel / serial measurement tools Tools measuring serial performance Tools measuring parallel performance 14

7 Timing Entire Program Measuring the elapsed (wall- clock) time that passes during the program execution Example: Solaris time, timex, and ptime 15

8 Timing Program Portions Fortran 77: etime, dtime (both not thread safe) C, C++, Fortran 90/95: gethrtime High resolution timer (nanoseconds) Can be called via a C wrapper from Fortran 77 Can be used for multithreaded applications Platform-specific tools and methods Solaris microstate accounting Fine-grain timing measurements by accessing UltraSPARC TICK register directly.inline readtick,1 rd %tick, %o1 stx %o1, [%o0].end 16

9 Measurement Overhead Computing overhead of gethrtime() call #include<sys/time.h> time_t start, end; int i, iters = ; for (i = 0; i < iters; i++) { start = gethrtime(); end = gethrtime(); (void)printf("%lld \n", (end - start)); } Distribution Call overhead (ns) (ns)

10 Program Profiling with gprof Application profiling Special form of timing measurements that shows which functions account for large parts of application runtimes Should be used on multiple and representative test cases gprof - standard UNIX profiling utility Can be used for profiling executalbes and shared libraries Based on Program Counter (PC) sampling at periodic intervals Requires recompilation with -pg (Linux, Solaris, Tru64) or -G (HP-UX) After the run the data is collected in gmon.out file Profiling results displayed with gprof command 18

11 Output includes gprof Output Absolute time spent in a function Percentage of total run time spent in a function Number of calls to the function Average time per call Functions can be sorted by time they consume together with their descendants (commulative or inclusive time) time spent executing the function itself (self or exclusive time) % cumulative self self total time seconds seconds calls ms/call ms/call name dmmch_ [4] dmake_ [8] dgemm_ [9]... 19

12 Profiling Using Coverage Analysis Coverage analysis tools annotate source code with the number of times each line was executed Basic block profiling Results can be accumulated for multiple runs Information about hot loops in the code and branches taken Code coverage for quality assurance DO 350 L = LL, LL+ LSEC > F11 = F11 + T1( L- LL+ 1, I- II+ 1 )* $ T2( L- LL+ 1, J- JJ+ 1 ) Available on UNIX platforms Linux/GNU: gcov Solaris: tcov IRIX: cvcov, cvxcov Tru64: pixie AIX: tprof 20

13 Advanced Profiling Tools Measurement parameters and features Measurements based on hardware counters Profiling by functions basic blocks lines of high level code assembly instructions Source code annotation Capabilities to work with parallel programs synchronization overhead, load balancing monitoring Available tools Tool Vendor Platforms 21 VTune Intel NT Analyzer Sun Solaris SpeedShop SGI IRIX DCPI DEC Compaq HP Tru64, NT

14 Example: Sun Performance Analyzer (1 of 3) Profiling by function and module (no recompilation) 22

15 Example: Sun Performance Analyzer (2 of 3) Annotated source (recompilation with -g) and disassembly (no recompilation) 23

16 Example: Sun Performance Analyzer (3 of 3) Hardware counter overflow profiling 24

17 Process Monitoring Tools Tracing tools Linux: strace (ltrace for dynamic library calls) Solaris: truss (sotruss for dynamic library calls) IRIX: par Tru64: atom -tool ptrace procfs-based tools pmap: prints the address space of the program pldd: lists the dynamic shared objects linked into the process (including ones explicitly attached using dlopen) pstack: prints a stack trace for each LWP in the process pflags: prints the /proc tracing flags ptree: process trees containing specified pids or users pwait: wait for specified processes to terminate pcred: prints the credentials (effective, real, saved UIDs and GIDs) 25

18 Example: profiling system calls truss on Solaris Reports the number of system calls for a process and associated time 26

19 System Monitoring Tools Tools for various UNIX platforms vmstat, vm_stat, memvis - virtual memory and CPU statistics mpstat, mpvis - parallel memory/cpu statistics netstat, nfsstat, nfsvis - network status and statistics iostat, dkvis - I/O statistics sar - system activity report top, prstat - list of most active processes systat - system activity stats lockstat - kernel lock statistics dkstat - file status information 27

20 vmstat - Virtual Memory Statistics Available on HP-UX, Tru64, Solaris, Linux, FreeBSD, etc. Example on Alpha/Tru64 Memory Usage Paging Activity CPU Usage Idle System 28

21 Hardware Counter Measurements Hardware performance counters allow for the runtime low-overhead measurements of various hardware events Cache references Cache misses Pipeline stalls Branch misprediction statistics D-TLB (Data Translation Lookaside Buffer) misses I-TLB (Instruction Translation Lookaside Buffer) Bus statistics including DMA and cache coherency transactions on a multiprocessor systems Others Only several events can be monitored at the same time 29

22 Code Instrumentation APIs can be used directly in the code High-resolution timing of performance-critical parts of the program Access to HW performance counters Example (Solaris) if (cpc_take_sample(&before) == -1) exit(-1); for (k = 0; k < N-1; k++) sum = sum + a[k]*b[k]; if (cpc_take_sample(&after) == -1) exit(-1); Counters specified by setting PERFEVENTS environment variable example% setenv PERFEVENTS pic0=load_use,pic1=load_use_raw Works on UltraSPARC CPUs 30

23 Parallel Measurement Methodology Same guidelines as in the serial case Parallel benchmarks should be representative of typical uses of applications Benchmarking must be performed to ensure repeatable and consistent results Probe effects and tool overheads should be minimized Specifics of parallel benchmarking Parallelism vs. Concurrency Dedicated mode of benchmarking Number of processors Choice of timer and time criterion Processor-set configuration Processor allocation in clusters 31

24 Timing a Parallel Threaded Program timex can be used for parallel timing Note that the real time decreases, but the user time representing combined CPU usage stays constant 32

25 Specific Parallel Timers Timing MPI programs time or timex timers can be used in combination with MPI submitting commands (mprun, mpirun, etc.) For timing portions of an MPI program, one can use the MPI_Wtime function available in Fortran, C and C++ bindings (typically highly accurate). Threaded applications can use gethrvtime (Solaris, Tru64 with Solaris Compatibility Library) Shows the user time on a per-thread basis Can be used in combination with gethrtime, which returns the elapsed real (wallclock) time on a per-thread basis 33

26 Parallel System Monitoring CPU ID mpstat - mutliprocessor monitoring Cross calls Interrupts Context switches Thread migrations Mutex info System calls CPU usage First snapshot: average since boot Sample measurements 34

27 Kernel Lock Statistics 35 Tools that report kernel lock statistics lockstat - Solaris, IRIX, AIX, Linux lockinfo - Tru64 Allows one to specify what events to monitor spin on adaptive mutex block on read access to rwlock due to waiting writers On some platforms generates gprof-like output # lockstat -IWk example_tnf Profiling interrupt: events in seconds (1164 events/sec) Count indv cuml rcnt nsec Hottest CPU+PIL Caller % 57% cpu[12] mutex_vector_enter % 66% cpu[9]+10 disp_getwork % 74% cpu[14] mutex_tryenter % 81% cpu[5] (usermode) % 82% cpu[1] splx % 84% cpu[5]+10 _resume_from_idle % 85% cpu[9]+10 disp % 85% cpu[15]+10 setfrontdq

28 Binding a Program To a Set of Processors Process monitoring can be difficult on multiprocessor systems due to process migration Single-threaded programs One can bind to a processor For multithreaded programs One can use processor sets Commands to set up and use processor sets psrset (HP-UX, Solaris) pset (IRIX) pset_create, pset_assign_cpu, pset_assign_pid, etc. (Tru64) 36

29 Summary Monitoring performance is essential to optimization If you cannot measure it you cannot improve it Important to select benchmarks carefully and identify parameters to measure Select tools suitable for the task System-wide or process-specific? Parallel or serial? Require recompilation or instrumentation? Need source-level information? Need hardware counter information? 37