Part IV. Workflow Mapping and Execution in Pegasus. (Thanks to Ewa Deelman)

Transcription

1 AAAI-08 Tutorial on Computational Workflows for Large-Scale Artificial Intelligence Research Part IV Workflow Mapping and Execution in Pegasus (Thanks to Ewa Deelman) 1

2 Pegasus-Workflow Management System Leverages abstraction for workflow description to obtain ease of use, scalability, and portability Provides a compiler to map from high-level descriptions to executable workflows Correct mapping Performance enhanced mapping Provides a runtime engine to carry out the instructions Scalable manner Reliable manner Ewa Deelman, Gaurang Mehta, Karan Vahi (USC/ISI) in collaboration with Miron Livny (UW Madison) 2

3 Underlying Grid Middleware Services Pegasus uses Globus ( grid services: Gridftp for efficient and reliable data transfer Grid proxys/certificates Pegasus uses the Condor ( job management system: CondorG for job submission to shared resources DAGman to manage job interdependencies 3

4 asic Workflow Mapping Select where to run the computations Apply a scheduling algorithm HEFT, min-min, round-robin, random The quality of the scheduling depends on the quality of information Transform task nodes into nodes with executable descriptions Execution location Environment variables initializes Appropriate command-line parameters set Select which data to access Add stage-in nodes to move data to computations Add stage-out nodes to transfer data out of remote sites to storage Add data transfer nodes between computation nodes that execute on different resources 4

5 asic Workflow Mapping Add nodes to create an execution directory on a remote site Add nodes that register the newly-created data products Add data cleanup nodes to remove data from remote sites when no longer needed reduces workflow data footprint Provide provenance capture steps Information about source of data, executables invoked, environment variables, parameters, machines used, performance Ewa Deelman, deelman@isi.edu 5

6 Pegasus Workflow Mapping Original workflow: 15 compute nodes devoid of resource assignment Resulting workflow mapped onto 3 Grid sites: 11 compute nodes (4 reduced based on available intermediate data) 12 data stage-in nodes 8 inter-site data transfers 14 data stage-out nodes to long -term storage data registration nodes (data cataloging) 60 jobs to execute

7 Some challenges in workflow mapping Automated management of data Through workflow modification Efficient mapping the workflow instances to resources Performance Data space optimizations Fault tolerance (involves interfacing with the workflow execution system) Recovery by replanning plan Mapping not a one shot thing Providing feedback to the user Feasibility, time estimates 7

8 Pegasus Deployment G R A M PS Wings Abstract Workflow (Resource -independent ) Pegasus Executable Workflow ( Resources Identified ) DAGMan Ready Tasks Job monitoring Information LSF C ondor G r idf TP H TT P Stor age LOCAL SUMIT HOST (Community resource ) Condor Queue Condor -G Condor -C Globus GRA M Condor PS LSF Resource Information and Data Location Information NMI : Globus MDS, RLS, SR Condor -G GridFTP G R AM PS LSF G R A M PS LSF G R A M PS LSF HTTP Storage C ondor G r idf TP H TT P Stor age C ondor G r idf TP H TT P Stor age C ondor G r idf TP H TT P Stor age Distributed Resources 8

9 Node clustering A A C C C C C C C C Level -based clustering D D Vertical clustering A A Arbitrary clustering cluster _1 cluster _2 C C C C C C C C D Useful for small granularity jobs D 9

10 Data Reuse When it is cheaper to access the data than to regenerate it Keeping track of data as it is generated supports workflow -level checkpointing T1 File A File 1 File 2 File 2 T2 T3 Files C and 2 are available T3 File C File D File D Original Workflow T4 Reduced Workflow T4 File E File E Mapping Complex Workflows Onto Grid Environments, E. Deelman, J. lythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, K. ackburn, A. Lazzarini, A. Arbee, R. Cavanaugh, S. Koranda, Journal USC Information of Grid Computing, Sciences Institute Vol.1, No. Yolanda 1, 2003., Gil pp (gil@isi.edu) AAAI-08 Tutorial July 13,

11 Efficient data handling Input data is staged dynamically New data products are generated during execution For large workflows 10,000+ files Similar order of intermediate and output files Total space occupied is far greater than available space failures occur Solution: Determine which data is no longer needed and when Add nodes to the workflow do cleanup data along the way Issues: minimize the number of nodes and dependencies added so as not to slow down workflow execution deal with portions of workflows scheduled to multiple sites deal with files on partition boundaries 11

12 Workflow Footprint In order to improve the workflow footprint, we need to determine when data are no longer needed: ecause data was consumed by the next component and no other component needs it ecause data was staged-out to permanent storage ecause data are no longer needed on a resource and have been stage-out to the resource that needs it Ewa Deelman, deelman@isi.edu 12

13 Cleanup Disk Space as Workflow Progresses 13

14 LIGO Workflows Full workflow: 185,000 nodes 466,000 edges 10 T of input data 1 T of output data. 166 nodes 26% improvement 56% improvement 14

15 Montage [Deelman et al 05] Small 1,200 Montage Work Montage application ~7,000 compute jobs in workflow instance ~10,000 nodes in the executable workflow same number clusters as processors speedup of ~15 on 32 processors 15

16 Pegasus + Condor + TeraGrid for Southern California Earthquake Center (SCEC) [Deelman et al 06] (nice TeraGrid folks) Executable workflow Hazard Map Condor Glide -ins Provision the resources Resource Descriptions Provenance Tracking Catalog Record Information about the Workflow Task Info Pegasus Map the Workflow onto the Grid resources Executable Workflow Run the Workflow on the Grid Resources Tasks Condor DAGMan Globus 16

17 Pegasus: Scale [Deelman et al 06] jobs Hours jobs / time in Hours Week of the year SCEC workflows run each week using Pegasus and DAGMan on the TeraGrid and USC resources. Cumulatively, the workflows consisted of over half a million tasks and used over 2.5 CPU Years, Largest workflow O(100,000) nodes Managing Large-Scale Workflow Execution from Resource Provisioning to Provenance tracking: The CyberShake Example, Ewa Deelman, Scott Callaghan, Edward Field, Hunter Francoeur, Robert Graves, Nitin Gupta, Vipin Gupta, Thomas H. Jordan, Carl Kesselman, Philip Maechling, John Mehringer, Gaurang Mehta, David Okaya, Karan Vahi, USC Information Li Zhao, Sciences e-science Institute 2006, Amsterdam, Yolanda Gil (gil@isi.edu) December 4-6, 2006, AAAI-08 best Tutorial paper July 13, 2008 award 17