Metadata Collection for Performance Analysis

Transcription

1 Metadata Collection for Performance Analysis Karen L. Karavanic Associate Professor of Computer Science Portland State University

2 The PerfTrack Project PerfTrack is a tool for storing, exploring, and analyzing performance data Our Approach: Collect and store as much information as possible about each build and run of an application Integrate database technology into a performance analysis tool Store a wide variety of performance data Data from different measurement tools Tracing, DPCL, Paradyn, TAU, Vampir, Speedshop, HW counters, etc. Native application performance measurements

3 PerfTrack Design Design goal: No knowledge of database technology or vocabulary should be required to use PerfTrack Design goal: PerfTrack must be scalable to 1000s of program runs and 100,000s of performance results Design goal: PerfTrack must be flexible enough to store data from different measurement tools and different types of performance studies Design goal: PerfTrack must be extensible to accommodate future innovations in measurement and analysis Design goal: PerfTrack should not be limited to a specific DBMS package for its data store

4 PerfTrack Design Data Collection Scripts: Build environment Run environment Performance Data PTdataStore interface: Shelters PerfTrack user from the DBMS Perftrack Data Format (PTDF) Data navigation and analysis PerfTrack GUIs Command line interface Direct SQL query

5 PerfTrack Design Performance Result: a measured value Metric: cpu time, wall clock time, wait time Context: whole program? One function? One process? Value: result of measuring a metric in a context Everything is a resource: machine, node, process, function, execution Resource attribute: Execution - timestamp, build, machine, etc. Compiler - version Resource type Hierarchical or non-hierarchical

6 Build Application ptbuild.py --app umt2k --pathtoexe./umt2k --srcdir. -V This collects information about the build, such as the machine the application was built on, the compilers used, and environment variables that were set during the build. build operatingsystem machine compiler Time stamp Environment Variables module Path to library Size Timestamp Type Name Release Version Name Flags Include paths Libraries Path to compiler Vendor Version MPI script used MPI script flags Path to MPI script MPI script include paths MPI script libraries MPI script library paths

7 Build Attributes BUILD: Date,Time,Environment Variables BUILD/MODULE: Path, Size,Timestamp COMPILER: CompileFlags,IncludePaths,Libraries,LibraryPaths,mpiScriptCo mpilerflags,mpiscriptlibraries,mpiscriptlibrarypaths, mpiscriptname,mpiscriptpath,path ENVIRONMENT/MODULE: LibraryDynamic, Path,Size,Timestamp,Type

8 Execute Application ptrun.py --app umt2k --batchfile psub.script --inputdeck opacfile,rtin,smartin --exename./umt2k This collects information about the execution, such as the machine the application was run on, and environment variables that were set during the execution. environment operatingsystem machine execution inputdeck submission module Path to library Size Timestamp Type Name Release Version Name node Name App-specific values Permissions on exe Time stamp Number of processes Number of threads Concurrency model (MPI, OpenMP) Environment variables Languages in application User who ran application Name Mod time Time stamp Name of batch file Commands in batch file Run command Environment variables set in batch file processes Rank threads Rank

9 Execute Application performancetool metric time performanceresults Name Version Name Name Start Time End Time interval Name Start Time End Time Metric Performance tool Value Units Start time End time Execution subinterval Name Start Time End Time

10 Execution Attributes Concurrency Environment Variables Executable: GID, Name, Permissions, Size, Timestamp, UID, j Job: CompletionTime, Exit Status, Nodes, Resources Used, StartTime Languages Launch Date,Time NumberOfProcesses PageSize ProcessesPerNode, ThreadsPerProcess RunErrorMsg # any error messages from the job Username UsesMPI/OpenMP/Pthreads

11 PerfTrack Design: Generic Database Schema resource_attribute resource_item id Integer name Varchar2(255) type Varchar2(255) type_id Integer parent Integer res_id name value type Integer Varchar2(255) Varchar2(255) Varchar2(255) resource_constraint from to Integer Integer

12 PerfTrack Design: Base Resource Types Input Deck Build Time Compiler Grid Module Interval Preprocessor Metric Submission Machine Partition Node Function Code Block Subinterval Environment Operating System Processor Execution Module Performance Tool Process Function Thread Code Block

13 PerfTrack Design PerfTrack Data Format (PTdf): ResourceType resourcetypename Application appname Execution execname appname Resource resourcename resourcetypename execname Resource resourcename resourcetypename ResourceAttribute resourcename attributename attributevalue attributetype ResourceConstraint resourcename1 resourcename2 PerfResult execname resourceset perftoolname metricname value units starttime endtime

14 Flexible Schema: PERI project example Base Resource Types Grid Build Environment Compiler Time Operating System Execution Input Deck Preprocessor Performance Tool Submission Metric Custom resource types File System FileSystem/device Custom Attributes Submission: Batch queue Entry Submission: PBS resources

15 Submission Attributes batchcmd # commands in the batch file batchfile batchfiledatetime batchqueueentry # entries in the queue at the time of submit launcher launcherversion machinepartition PBS Resources runcmd # run commands in the batch file

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37 How do people want to use PerfTrack? Traditional Performance Analysis/Tuning Fully automated performance regression testing Comparative evaluation of new platforms vs. old Effects of hardware and software upgrades OS kernel performance study Organized store to replace scattered files Sharing a single data store in collaborative studies

38 What do people want to store in PerfTrack? All Performance Experiment Artifacts HW counter data, profile data, trace data, benchmark output "Barry's World": create the graph, save the graph, save the steps to create the graph Paradyn artifacts: Search History Graph, Call graph, performance data histograms Data from all common tools -- OpenSpeedshop, TAU, etc. As much description of the build and runtime environments as possible

39 Machine Data Collection "Automated System Environment Capture For PerfTrack" Capstone Project Team: Aaron Amauba, Dave Vu, Steve Wooster What to collect? When to collect? We already have this information right?? Who knows? Device model number example Is the execution timestamp enough?

40 Machine Data Collection: Host System method Idea Write scripts to run on the host system and directly measure the environment Features Modular structure. User can specify tailor made modules for the system they are interested in. Can be run anytime. Just kick off the script. Allow the user to provide resource hierarchy information when a connection to the database is not available. Status Currently implemented and tested for Linux

41 Machine Description -- Scaling Challenges Automated Host Data Collection eliminates tedious manual entry BUT When to scan? BG/L k+ nodes; update frequency?? Per Execution?? Weekly? Daily? Hourly? Who scans? Each researcher? (requires running a "scan" program on each node, can be difficult to get these types of jobs scheduled by scheduler) Do we really need all those copies?? Delay in entry -> overwrite new data with old?? names?? The best answer will involve lab support!

42 Collaborative Data Stores PERI-DB (Shirley Moore) Goal: Develop and deploy a data store for performance data sets of PERI project researchers Approach Define PERI XML schema Tools provide a mapping to/from PERI XML We extended our data collection scripts to output PERI xml In progress: translating PTdf <=> PERI xml conversion of PTdf files to PERI xml conversion of PERI xml files to PTdf input of PERI xml to PerfTrack database export of PERI xml from queries to PerfTrack database

43 Rich Data Sets Key Issues in Metadata Collection build, platform, runtime environment, performance data sparse data will impact results -- eg clustering Scalability collection frequency: each run? each experiment? each user? each boot? each upgrade? attributes for groups of resources Time: when is new knowledge created? new machine resource every year attributes of machine resource change every few weeks input data changes every few runs runtime environment may change during one execution diagnoses and comparative data during analysis

44 Key Issues in Metadata Collection Collaborative Data Stores Need to map resources and results between local sites Porting difficulties Commands for collecting metadata information vary from platform to platform Lack of common interfaces e.g. file system software version information on Linux: Lustre vs GPFS

45 The PerfTrack Project Karen Karavanic: John May: Karen L. Karavanic, John May, et al, "Integrating Database Technology with Comparison-based Parallel Performance Diagnosis: The PerfTrack Performance Experiment Management Tool," SC2005, November 2005, Seattle, WA. Kathryn Mohror, Rashawn Knapp, Nagalaxmi Karumbunathan This research supported in part by UC/LLNL subcontract #B Portions of this work were performed under the auspices of the U.S. Department of Energy by the University of California Lawrence Livermore National Laboratory under contract No. W-7405-Eng-48. Clustering interfaces: Thomas Conerly, Abraham Neben (Portland Saturday Academy Internship Program for high school students) PPerfGrid: John Hoffman (PSU masters thesis) Capstone Project: Aaron Amauba, Dave Vu, Steve Wooster (PSU undergraduate course)