MPI Performance Analysis Trace Analyzer and Collector

Transcription

1 MPI Performance Analysis Trace Analyzer and Collector Berk ONAT İTÜ Bilişim Enstitüsü 19 Haziran 2012

2 Outline MPI Performance Analyzing Defini6ons: Profiling Defini6ons: Tracing Intel Trace Analyzer Lab: How to use ITAC

3 Performance Problems Scalability Produc6vity Efficiency Performance technology Applica6on- specific and automa6c performance tools Cluster analysis

4 Role of Programmer How should we write our programs, given that we have a good op6mizing compiler? Write simpler codes: o Easy to read, o Easy to maintain and o Ensure correctness. Do: o o o Select best algorithm Write code that s readable & maintainable Eliminate op6miza6on blockers Allow compiler to do its job Focus on inner loops Use a profiler and an analyzer to find important ones with 6me consuming

5 Definitions Profiling: Recording of summary informa6on during execu6on Inclusive, exclusive 6me, Number of calls, Hardware sta6s6cs, (hardware counters ) Reflects performance behaviour of program en66es Func6ons, Loops, Basic blocks User- defined seman6c en66es Helps to expose performance bo\lenecks and hotspots Implemented through: Sampling: periodic OS interrupts or hardware counter traps Instrumenta6on: direct inser6on of measurement code

6 Definitions Profile Terminology Rou6ne int main() Inclusive 6me 100 secs Exclusive 6me =10 secs Number of Calls 1 call Number of Subrou6nes Child rou<nes called = 3 Inclusive 6me/call 100 secs int main( ) { /* takes 100 secs */ } /* f1(); /* takes 20 secs */ f2(); /* takes 50 secs */ f1(); /* takes 20 secs */ /* other work */ Time can be replaced by counts */

7 Definitions Tracing: Recording of informa6on about significant points (events) during program execu6on Entering/exi6ng code regions (func6on, loop, block, ) Thread/process interac6ons (e.g., send/receive message) Save informa6on in event record 6mestamp CPU iden6fier, thread iden6fier Event type and event- specific informa6on Event trace is a 6me- sequenced stream of event records Can be used to reconstruct dynamic program behavior Typically requires code instrumenta6on

8 Definitions Event Tracing: Instrumenta<on, Monitor, Trace

9 Definitions Event Tracing: Timeline Visualiza<on

10 Intel Trace Analyzer and Collector Intel Trace Analyzer and Collector: provide informa6on cri6cal to understanding and op6mizing MPI cluster performance by quickly finding performance bo\lenecks with MPI communica6on Interface and Displays Metrics Tracking Scalability Instrumenta6on and Tracing Compa6bility

11 Intel Trace Analyzer and Collector Compa<bility Intel compilers and GNU* compilers Intel MPI Library MPICH (and compa6ble deriva6ves) Red Hat Enterprise Linux 3.0 or 4.0 SUSE LINUX Enterprise Server 9 or 10 SGI Al6x

12 Intel Trace Analyzer and Collector Interface and Displays Timeline Views and Parallelism Display Displays concurrent behavior of Parallel applica6ons Calculates sta6s6cs for specific 6me Intervals, processes, or func6ons Displays applica6on ac6vi6es, event Source code loca6ons, and message passing along 6me axis

13 Intel Trace Analyzer and Collector Advanced GUI Display Scalability Detailed and Aggregate Views Examines aspects of applica6on run6me behavior, grouped by func6ons or processes Easily iden6fies the amount of 6me spent in MPI communica6on Easily see the performance differences between two program runs

14 Intel Trace Analyzer and Collector Execu<on Sta<s<cs Provides subrou6ne execu6on metrics or call- tree characteris6cs Profiling Library Records distributed, event- based trace data Sta<s<cs Readability Logs informa6on for func6on calls, sent messages, and collec6ve opera6ons

15 Intel Trace Analyzer and Collector Scalability Low Overhead Provides structured trace file (STF) format for scalability Generates trace files faster Allows random access to por6ons of a trace, making it suitable for analysis of large amounts of trace data Filtering and Memory Handling Caches trace data in memory to reduce run6me overhead and memory consump6on

16 How to Use ITAC Login to your UYBHM node using - X with ssh : bash: $ ssh - X du??@wsl- node??.uhem.itu.edu.tr or use your PuTTY program in your Windows with X11 forwarding in SSH sec<on. Copy example file tar to your directory bash: $ cd workshop bash: $ cp /RS/users/bonat/workshop/YAZOKULU/ tar. bash: $ tar - xvf tar bash: $ cd /mpi- analyze/traceanalyzer

17 How to Use ITAC Seeng Up Environmental Variables: Adding source ITAC line to your.bashrc and/or.bash_profile source /RS/progs/intel/itac/7.1/bin/itacvars.sh Use add- ITAC- to- my- PATH.sh script bash: $./add- ITAC- to- my- PATH.sh

18 How to Use ITAC Collec<ng Trace Data: First create the object files: bash: $ mpiicc bujerfly.c - c Link the object file with ITC libs: bash: $ mpiicc bujerfly.o - lvt - ldwarf - lelf - lvtunwind - lnsl - lm - ldl - lpthread - L/RS/progs/intel/itac/7.1/lib/ - o bujerfly.x You can also use the given ITACcompile.sh script: bash: $./ITACcompile.sh bujerfly.c

19 How to Use ITAC Collec<ng Trace Data: First create the object files: bash: $ mpirun - np 8./bujerfly.x # Iter: 1 # Stage = 3 0 (id): I'm at the barrier 5 (id): I'm at the barrier 7 (id): I'm at the barrier 2 (id): I'm at the barrier ## Calcula<on <me for 1 itera<ons : (id): I passed the barrier 4 (id): I passed the barrier 2 (id): I passed the barrier 0 (id): I passed the barrier

20 How to Use ITAC Analyzing Trace Data: Check tracing files (.ss): bash: $ ls bujerfly.c bujerfly.x bujerfly.x.ss Link the object file with ITC libs bash: $ traceanalyzer bujerfly.x.ss

21 How to Use ITAC Event Timeline Analyzing Trace Data: Func<on Profile Message Profile 21

22 How to Use ITAC Analyzing Trace Data: 22

23 How to Use ITAC Analyzing Trace Data: 23

24