Analysis of Program Behavior

Transcription

1 Analysis of Program Behavior High Performance Computing, Visualization Lucas Mello Schnorr probably soon (LIG-CNRS INF-UFRGS) 2 nd LICIA Workshop Grenoble, France September 5th, / 25

2 Introduction High Performance Computing Supercomputers, Clusters, Multi-Clusters Dedicated interconnection, geographically centralized Distributed Systems Grids, Clouds, Volunteer Computing Shared interconnection, geographically dispersed IBM BlueGene/Q with 1,572,864 cores (#1 - Top500) 2/ 25

3 Introduction High Performance Computing Supercomputers, Clusters, Multi-Clusters Dedicated interconnection, geographically centralized Distributed Systems Grids, Clouds, Volunteer Computing Shared interconnection, geographically dispersed IBM BlueGene/Q with 1,572,864 cores (#1 - Top500) Applications Climatology, Simulation, Cryptography 2/ 25

4 Challenges and Motivation Very large applications Top500 has machines with 1.5 million cores BOINC has more than 300K active computers Low-intrusion tracing techniques Buffering, hardware support, simulation traces 3/ 25

5 Challenges and Motivation Very large applications Top500 has machines with 1.5 million cores BOINC has more than 300K active computers Low-intrusion tracing techniques Buffering, hardware support, simulation traces Space/Time trace size explosion Very detailed in time Many entities in space 3/ 25

6 Challenges and Motivation Very large applications Top500 has machines with 1.5 million cores BOINC has more than 300K active computers Low-intrusion tracing techniques Buffering, hardware support, simulation traces Space/Time trace size explosion Very detailed in time Many entities in space How to extract useful information from traces? 3/ 25

7 Approach and Objectives Approach Exploratory Trace Visualization Multi-scale graphical representation of events Data aggregation mechanisms to scale Interactivity for most of operations 4/ 25

8 Approach and Objectives Approach Exploratory Trace Visualization Multi-scale graphical representation of events Data aggregation mechanisms to scale Interactivity for most of operations Analysis of Program Behavior Validate or refute behavioral assumptions Explain anomalies, unexpected behavior Consider resource constraints 4/ 25

9 Introduction Traditional trace visualization

10 Introduction Traditional trace visualization

11 Introduction Traditional trace visualization

12 Outline 1 Why data aggregation? 2 Visualization techniques Squarified Treemap View Hierarchical Graph View 3 Tools and framework 4 Collaborations within LICIA 5 Conclusion 6/ 25

13 Trace visualization Real BOINC availability trace file Availability is either true or false One volunteer client 7/ 25

14 Trace visualization Real BOINC availability trace file Availability is either true or false One volunteer client GNUPlot to a vector file: 8-month and 12-day zoom 7/ 25

15 Trace visualization trust the rendering? Same vector file, two different views Different interpretation depending on the viewer 8/ 25

16 Trace visualization trust the rendering? Same vector file, two different views Different interpretation depending on the viewer Evince Acroread 8/ 25

17 Trace visualization trust the rendering? Same vector file, two different views Different interpretation depending on the viewer Evince Acroread Should we trust the rendering? No! We need to make choices before visualizing data 8/ 25

18 Trace visualization data aggregation 24-hour time integration 9/ 25

19 Trace visualization data aggregation 24-hour time integration Should we aggregate data without care? No! We need to make choices before aggregating data 9/ 25

20 Space/Time views Widespread, useful, intuitive, fast adoption All trace events represented, causal order Paje Vite Vampir 10/ 25

21 Space/Time views Widespread, useful, intuitive, fast adoption All trace events represented, causal order Paje Vite Vampir However... Also impacted by ever larger trace sizes Limited visualization scalability 10/ 25

22 Space/Time views limitations MPI Sweep3D 16 processes Simulated with SMPI, traced with SimGrid pj_dump ed to a csv file, loaded into R Gantt-charts by R, dumped to a vector file 11/ 25

23 Space/Time views limitations Evince 11/ 25

24 Space/Time views limitations GS (Ghostscript) 11/ 25

25 Space/Time views limitations MPI Sweep3D 16 processes Real execution on the Griffon cluster of Grid 5000 traced with TAU, converted with tau2paje Once again, Gantt-charts by R vector file 12/ 25

26 Space/Time views limitations Evince 12/ 25

27 Space/Time views limitations Acroread 12/ 25

28 So, why data aggregation? Technical need: too much data to fit on small screens 13/ 25

29 So, why data aggregation? Technical need: too much data to fit on small screens Data aggregation key for large-scale visualization 13/ 25

30 So, why data aggregation? Technical need: too much data to fit on small screens Data aggregation key for large-scale visualization Semantic: aggregated data more representative 13/ 25

31 So, why data aggregation? Technical need: too much data to fit on small screens Data aggregation key for large-scale visualization Semantic: aggregated data more representative Goals Visualization techniques for aggregated data 13/ 25

32 So, why data aggregation? Technical need: too much data to fit on small screens Data aggregation key for large-scale visualization Semantic: aggregated data more representative Goals Visualization techniques for aggregated data Analyst has full control of aggregation 13/ 25

33 Visualization Techniques 14/ 25

34 Visualization techniques Squarified Treemap View Observe outliers, differences of behavior Hierarchical aggregation B Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) C Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) D Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) E Maximum Aggregation 15/ 25

35 Visualization techniques Squarified Treemap View Observe outliers, differences of behavior Hierarchical aggregation B Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) C Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) D Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) E Maximum Aggregation Hierarchical Graph View Pin-point resource contention Correlate application behavior to network topology time slice time slice 15/ 25

36 Squarified Treemap View Scalable representation for hierarchies when compared to node-link diagrams better visualization scalability for large trees 16/ 25

37 Squarified Treemap View Scalable representation for hierarchies when compared to node-link diagrams better visualization scalability for large trees Space-filling top-down recursive layout algorithm Node value space occupied in the screen Squarified version keeps rectangles ratio close to 1 T=6 M=3 N=2 O=1 A=1 B=1 C=1 D=1 E=1 F=1 16/ 25

38 Squarified Treemap View Scalable representation for hierarchies when compared to node-link diagrams better visualization scalability for large trees Space-filling top-down recursive layout algorithm Node value space occupied in the screen Squarified version keeps rectangles ratio close to 1 T=6 M=3 N=2 O=1 T=6 A=1 B=1 C=1 D=1 E=1 F=1 16/ 25

39 Squarified Treemap View Scalable representation for hierarchies when compared to node-link diagrams better visualization scalability for large trees Space-filling top-down recursive layout algorithm Node value space occupied in the screen Squarified version keeps rectangles ratio close to 1 T=6 M=3 N=2 O=1 A=1 B=1 C=1 D=1 E=1 F=1 T=6 M=3 N=2 O=1 16/ 25

40 Squarified Treemap View Scalable representation for hierarchies when compared to node-link diagrams better visualization scalability for large trees Space-filling top-down recursive layout algorithm Node value space occupied in the screen Squarified version keeps rectangles ratio close to 1 T=6 M=3 N=2 O=1 T=6 M=3 N=2 A=1 B=1 D=1 E=1 A=1 B=1 C=1 D=1 E=1 F=1 O=1 C=1 F=1 16/ 25

41 Squarified Treemap View an example Synthetic trace with 100 thousand processes Two states, four-level hierarchy Visualization artifacts without spatial aggregation A Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) 17/ 25

42 Squarified Treemap View an example Synthetic trace with 100 thousand processes Two states, four-level hierarchy Visualization artifacts without spatial aggregation B Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) 17/ 25

43 Squarified Treemap View an example Synthetic trace with 100 thousand processes Two states, four-level hierarchy Visualization artifacts without spatial aggregation C Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) 17/ 25

44 Squarified Treemap View an example Synthetic trace with 100 thousand processes Two states, four-level hierarchy Visualization artifacts without spatial aggregation D Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) 17/ 25

45 Squarified Treemap View an example Synthetic trace with 100 thousand processes Two states, four-level hierarchy Visualization artifacts without spatial aggregation E Maximum Aggregation 17/ 25

46 Hierarchical Graph View Scalable representation for graphs Topology, with application-level metrics Identify resource bottleneck in space and time Use spatial-temporal aggregated traces Interactive force-directed layout (Barnes-Hut algorithm) 18/ 25

47 Hierarchical Graph View Scalable representation for graphs Topology, with application-level metrics Identify resource bottleneck in space and time Use spatial-temporal aggregated traces Interactive force-directed layout (Barnes-Hut algorithm) Trace metrics geometrical properties Size, shape, filling, colors Nodes: monitored entities Edges: relationship among entities time slice hosta link utilization link hostb 18/ 25

48 Hierarchical Graph View Scalable representation for graphs Topology, with application-level metrics Identify resource bottleneck in space and time Use spatial-temporal aggregated traces Interactive force-directed layout (Barnes-Hut algorithm) Trace metrics geometrical properties Size, shape, filling, colors Nodes: monitored entities Edges: relationship among entities time slice hosta link utilization link hostb 18/ 25

49 Hierarchical Graph View - an example Squares are hosts, diamonds are network links Colors represent different applications or parts of it (task type, phase) Two clusters interconnected by four network links Cluster backbones have larger bandwidth capacity Each host connected to the backbone by a private link time slice 19/ 25

50 Tools and Framework 20/ 25

51 Open-source tools PajeNG Next Generation (Space/Time views), GPL3 Only pj_dump and pj_validate (partial pajeng) Link against libpaje.so then use Paje protocol 21/ 25

52 Open-source tools PajeNG Next Generation (Space/Time views), GPL3 Only pj_dump and pj_validate (partial pajeng) Link against libpaje.so then use Paje protocol Data Liberation Front $./pj_dump input.paje > output.csv Then use your preferred visualization tool (gnuplot, R, GNU Octave,...) 21/ 25

53 Open-source tools PajeNG Next Generation (Space/Time views), GPL3 Only pj_dump and pj_validate (partial pajeng) Link against libpaje.so then use Paje protocol Data Liberation Front $./pj_dump input.paje > output.csv Then use your preferred visualization tool (gnuplot, R, GNU Octave,...) Viva (Aggregated Treemaps, Hierarchical graph), GPL3 Partial graph visualization for now (soon) Deprecates Triva, serves as research framework 21/ 25

54 Open-source tools PajeNG Next Generation (Space/Time views), GPL3 Only pj_dump and pj_validate (partial pajeng) Link against libpaje.so then use Paje protocol Data Liberation Front $./pj_dump input.paje > output.csv Then use your preferred visualization tool (gnuplot, R, GNU Octave,...) Viva (Aggregated Treemaps, Hierarchical graph), GPL3 Partial graph visualization for now (soon) Deprecates Triva, serves as research framework Some tracers and converters: SimGrid, Akypuera (tau2paje, otf2paje, otf22paje), Poti, XKaapi, EZTrace, rastro2paje, librastro, GTG, JRastro and counting... 21/ 25

55 Collaborations within LICIA 22/ 25

56 Collaborations Arnaud Legrand (LIG-Mescal) ANR Infra-SONGS Project Scalable trace analysis Trace comparison 23/ 25

57 Collaborations Arnaud Legrand (LIG-Mescal) ANR Infra-SONGS Project Scalable trace analysis Trace comparison Jean-Marc Vincent (LIG-Mescal) Evaluating data aggregation with entropy Very promising initial results already integrated with treemap 23/ 25

58 Collaborations Arnaud Legrand (LIG-Mescal) ANR Infra-SONGS Project Scalable trace analysis Trace comparison Jean-Marc Vincent (LIG-Mescal) Evaluating data aggregation with entropy Very promising initial results already integrated with treemap Guillaume Huard (LIG-Moais) SoC-Trace Project Evaluating data aggregation Using embedded applications as scenario 23/ 25

59 Collaborations Arnaud Legrand (LIG-Mescal) ANR Infra-SONGS Project Scalable trace analysis Trace comparison Jean-Marc Vincent (LIG-Mescal) Evaluating data aggregation with entropy Very promising initial results already integrated with treemap Guillaume Huard (LIG-Moais) SoC-Trace Project Evaluating data aggregation Using embedded applications as scenario Nicolas Maillard, Philippe Navaux (INF-UFRGS) expected New scenarios: parallel file systems, Java/ProActive, Many-core applications Joao Comba (INF-UFRGS) expected Propose novel information visualization techniques 23/ 25

60 Collaborations Arnaud Legrand (LIG-Mescal) ANR Infra-SONGS Project Scalable trace analysis Trace comparison Jean-Marc Vincent (LIG-Mescal) Evaluating data aggregation with entropy Very promising initial results already integrated with treemap Guillaume Huard (LIG-Moais) SoC-Trace Project Evaluating data aggregation Using embedded applications as scenario Nicolas Maillard, Philippe Navaux (INF-UFRGS) expected New scenarios: parallel file systems, Java/ProActive, Many-core applications Joao Comba (INF-UFRGS) expected Propose novel information visualization techniques Open to new collaborations 23/ 25

61 Conclusion and Future Work Data aggregation Key to scale data visualization for analysis No pre-defined or fixed parameters 24/ 25

62 Conclusion and Future Work Data aggregation Key to scale data visualization for analysis No pre-defined or fixed parameters Visualization techniques Based upon aggregated data Complementary to existing techniques Continuous evaluation of visualization scalability With larger data-sets, does it remain useful? 24/ 25

63 Conclusion and Future Work Data aggregation Key to scale data visualization for analysis No pre-defined or fixed parameters Visualization techniques Based upon aggregated data Complementary to existing techniques Continuous evaluation of visualization scalability With larger data-sets, does it remain useful? Future Work Scalable trace comparison Can we visualy compare two traces on scale? Revisit Gantt-charts Can we keep causal order analysis with aggregated data? Data aggregation and other visualization techniques 24/ 25

64 Thank you for your attention Some references VIVA Visualizing more performance data than what fits on your screen. Lucas Mello Schnorr, Arnaud Legrand. The 6th International Parallel Tools Workshop. Springer (Invited paper) Detection and Analysis of Resource Usage Anomalies in Large Distributed Systems Through Multi-scale Visualization. Lucas Mello Schnorr, Arnaud Legrand, Jean-Marc Vincent. Concurrency and Computation: Practice and Experience. Wiley A Hierarchical Aggregation Model to achieve Visualization Scalability in the analysis of Parallel Applications. Lucas Mello Schnorr, Guillaume Huard, Philippe Olivier Alexandre Navaux. Parallel Computing. Volume 38, Issue 3, March 2012, Pages More information INFRA-SONGS Project (WP-7) Simulation of Next Generation Systems WP-7: Visualization and Analysis 25/ 25