Meeting the Challenges of Managing Large-Scale Scientific Workflows in Distributed Environments

Transcription

1 Meeting the Challenges of Managing Large-Scale Scientific s in Distributed Environments Ewa Deelman Yolanda Gil USC Information Sciences Institute

2 Scientific s Current workflow approaches are exploring specific aspects of the problem: Creation, reuse, provenance, performance, reliability New requirements are emerging Streaming data, from batch to interactive steering, event-driven analysis, collaborative design of workflows Need to develop a science of workflows A more comprehensive treatment of workflow lifecycle Understand current and long-term requirements from science applications reproducibility s as first-class citizens in CyberInfrastructure

3 Lifecycle Data Products Adapt, Modify and Component Libraries Template User Experiences Populate with data Data, Metadata Catalogs Data, Metadata, Provenance Information Instance Scheduling/ Execution Compute, Storage and Network Resources Execute Executable Map to available resources Planning Resource, Application Component Descriptions Ewa Deelman

4 Outline Rendering the workflow lifecycle Wings/Pegasus/DAGMan Challenges across the various aspects of workflow management User experiences Planning/Mapping Execution s-what are they good for? Research issues Conclusions

5 Lifecycle Data Products WINGS Adapt, Modify and Component Libraries Template WINGS Populate with data Data, Metadata Catalogs Data, Metadata, Provenance Information Instance Compute, Storage and Network Resources Execute DAGMan Executable Map to available resources Pegasus Resource, Application Component Descriptions Ewa Deelman

6 Entities Decimate FFT Template Instance GridFTP f1 S1-> R1 Decimate ( f1) at R1 GridFTP g1 R1-> R2 FFT(g1) at R2 GridFTP h1 R2->S1 Register h1 in RLS GridFTP f2 S1-> R1 Decimate ( f2) at R1 GridFTP g2 R1-> R2 FFT(g2) at R2 GridFTP h2 R2->S1 Register h2 in RLS GridFTP f1000 S1-> R1 Decimate ( f1000) at R1 GridFTP g1000 R1-> R2 FFT(g1000) at R2 GridFTP h1000 R2->S1 Register h1000 in RLS Executable

7 WINGS/Pegasus: Instance Generation and Selection, Using semantic technologies for workflow generation Show me workflows that generate hazard maps WINGS Selection - templates specify complex analyses sequences - instances specify data Libraries Creation Validate this workflow based on the component specs EXPERT SCIENTIST SCIENTIST Run that with the USGS data set Template Data Selection Instance Pegasus Ontologies: Domain terms, Component types, Products (OWL) Data Repositories - Preexisting data collections - execution results Executable DAGMan/ Globus Application Components Component Specification - Specifies data requirements - Specifies execution requirements SCIENTIST RESEARCHING Here is a new NEW MODELS wave propagation model, takes in a series of fault ruptures, is compiled for MPI Wings for Pegasus: A Semantic Approach to Creating Very Large Scientific s Yolanda Gil, Varun Ratnakar, Ewa Deelman, Marc Spraragen, and Jihie Kim, OWL: Experiences and Directions 2006

8 Pegasus: Planning for Execution in Grids Maps from workflow instance to executable workflow Automatically locates physical locations for both workflow components and data Finds appropriate resources to execute the components Augments the workflow with data staging and registration Reuses existing data products where applicable Publishes newly derived data products /usr/local/bin/fft /home/file1 Executable FFT filea Instance Move filea from host1:// home/filea to host2://home/file1 Ewa Deelman DataTransfer Data Registration

9 Condor DAGMan (University of Wisconsin) Follows dependencies in workflow Releases nodes to execution (to Condor Q) Provides retry capabilities executing done OK waiting

10 Challenges in user experiences Users expectations vary greatly High-level descriptions Detailed plans that include specific resources Users interactions can be exploratory Modifying portions of the workflow as the computation progresses Users need progress, failure information at the right level of detail Ewa Deelman

11 Portals, Providing high-level Interfaces Region Name, Degrees Pegasus JPL User Portal mgridexec Abstract Grid Scheduling and Execution Service ISI Concrete Condor DAGMAN mdagfiles Abstract DAGMan JPL Abstract Service Computational Grid TeraGrid Clusters SDSC m2masslist Image List mnotify IPAC 2MASS Image List Service IPAC User Notification Service NCSA ISI Condor Pool Montage: a grid portal and software toolkit for science-grade astronomical image mosaicking, J. C. Jacob, D. S. Katz, G.. erriman, J. Good, A. C. Laity, E. Deelman, C. Kesselman, G. Singh, M.-H. Su, T. A. Prince, R. Williams,, IJCSE, to appear 2006 Galactic Star Formation Region RCW 49 Ewa Deelman

12 Portals, Providing high-level Interfaces TG Science Gateway, Washington University EarthWorks Project (SCEC), lead by with J. Muench P. Maechling, H. Francoeur, and others SCEC Earthworks: Community Access to Wave Propagation Simulations, J. Muench, H. Francoeur, D. Okaya, Y. Cui, P. Maechling, E. Deelman, G. Mehta, T. Jordan TG 2006 Ewa Deelman

13 SCEC CyberShake, not a one shot workflow Needs to run before rest of the workflow is instantiated

14 Iterative workflow instantiation, mapping and execution PreSGTFD CVMFD CVMFD PreSGTFD XYZGRD CVMFD PreSGTFD CVM CVM CVM PreSGT pmvlchk PreSGT pmvlchk PreSGT oxnamecheck oxnamecheck pmvlchk oxnamecheck 127_6.txt.variation SGT-s000 -h000 SGT _7.txt.variation SGT-s000 -h000 SGT282 Wings Generated Data Files 127_6.txt.variation SGT - s000 - h000 SG T _7.txt.variation SGT- s000 - h000 SG T _11.txt.variation GT S GT - s000 - h000 SG T1 6 1 Pegasus DAGMan Globus G e nmd Se isg e n_ Li G e nmd Se isg e n_ Li... G e nmd Se isg e n_ Li Seismograms_PAS_127_6.grm Seismograms_PAS_127_7.grm Seismograms_PAS_151_11.grm Pe a kv a lca lc Pe a kv a lca lc Pe a kv a lca lc PeakVals_allPAS_127_6.bsa PeakVals_allPAS_127_7.bsa PeakVals_allPAS_151_11.bsa Results Wings for Pegasus: A Semantic Approach to Creating Very Ewa Deelman Large Scientific s Yolanda Gil, Varun Ratnakar, Ewa Deelman, Marc Spraragen, and Jihie Kim, in submission

15 Some challenges in workflow mapping Automated management of data Through workflow modification Efficient mapping the workflow instances to resources Performance Data space optimizations Fault tolerance (involves interfacing with the workflow execution system) Recovery by replanning plan Providing feedback to the user Feasibility, time estimates Ewa Deelman

16 Execution Environment Pegasus DAGMan GRAM PS LSF Condor GridF TP HTT P Storage Condor-G GRAM PS LOCAL SUMIT HOST Condor Q Condor-C Condor LSF Condor -G GridFTP G RAM PS LSF GRAM PS LSF HTTP Storage Condor Condor GridF TP HTT P Storage GridF TP HTT P Storage Ewa Deelman, deelman@isi.edu pegasus.isi.edu

17 A Node clustering A C C C C C C C C Level-based clustering D D Vertical clustering A A Arbitrary clustering cluster _1 cluster _2 C C C C C C C C D Useful for small granularity jobs D Ewa Deelman, deelman@isi.edu pegasus.isi.edu

18 Small 1,200 Montage Montage application ~7,000 compute jobs in instance ~10,000 nodes in the executable workflow same number of clusters as processors speedup of ~15 on 32 processors

19 Efficient data handling Input data is staged dynamically, new data products are generated during execution For large workflows 10,000+ files Similar order of intermediate and output files Total space occupied is far greater than available space failures occur Solution: Determine which data is no longer needed and when Add nodes to the workflow do cleanup data along the way Issues: minimize the number of nodes and dependencies added so as not to slow down workflow execution deal with portions of workflows scheduled to multiple sites deal with files on partition boundaries Ewa Deelman, pegasus.isi.edu

20 Challenges in Execution Provide fault tolerance Mask errors, Interact with the workflow planner Support resource provisioning Provide monitoring information Providing execution-level provenance Support debugging Provide workflow traces for easy replay

21 Southern California Earthquake Center (SCEC) workflows on the TeraGrid Executable workflow (nice TeraGrid folks) Hazard Map Condor Glide-ins Provision the resources Resource Descriptions VDS Provenance Tracking Catalog Record Information about the Task Info Pegasus Map the onto the Grid resources Executable Run the on the Grid Resources Tasks Abstract Condor DAGMan Globus Joint work with: R. Graves, T. Jordan, C. Kesselman, P. Maechling, D. Okaya & others

22 Gurmeet Singh et al. Application-level Resource Provisioning, Wednesday, M15, 14:30-16:00 session SCEC on the TeraGrid Fall Number of jobs per day (23 days), 261,823 jobs total, Number of CPU hours per day, 15,706 hours total (1.8 years), 10T of data JOS HRS /19 10/20 10/21 10/22 10/23 10/24 10/25 10/26 10/27 10/28 10/29 10/30 10/31 11/1 11/2 11/3 11/4 11/5 11/6 11/7 11/8 11/9 11/10

23 enefits of Scientific s (from the point of view of an application scientist) Conducts a series of computational tasks. Resources distributed across Internet. Chaining (outputs become inputs) replaces manual hand-offs. Accelerated creation of products. Ease of use - gives non-developers access to sophisticated codes. Avoids need to download-install-learn how to use someone else's code. Provides framework to host or assemble community set of applications. Honors original codes. Allows for heterogeneous coding styles. Framework to define common formats or standards when useful. Promotes exchange of data, products, codes. Community metadata. Multi-disciplinary workflows can promote even broader collaborations. E.g., ground motions fed into simulation of building shaking. Certain rules or guidelines make it easier to add a code into a workflow. Slide courtesy of David Okaya, SCEC, USC

24 s for education and sharing Application specialists design individual application components Domain experts compose workflows using application components Set correct parameters for components Pick appropriate data sets Students run sophisticated workflows on training data sets Young researchers run sophisticated workflows on data sets of interest to them Scientist share workflows across collaborations to validate a hypothesis Need to develop tools, workflow libraries, component libraries

25 Current and Future Research Resource selection Resource provisioning restructuring Adaptive computing refinement adapts to changing execution environment provenance (including provenance of the mapping process) new collaboration with Luc Moreau Management and optimization across multiple workflows debugging Streaming data workflows Automated guidance for workflow restructuring Support for long-lived and recurrent workflows Ewa Deelman, pegasus.isi.edu

26 General Conclusions s are recipes for CyberInfrastructure Need to support the dynamic nature of science Support for long-lived and recurrent workflows Many challenges and many workflow tools out there Interoperability is desired Need common representations that can be used by various workflow management systems Maybe semantic technologies? Need common provenance tracking capabilities See IPAW 06, and the Provenance Challenge To make forward progress collaboration with application scientists is essential collaboration between workflow system designers is essential Ewa Deelman

27 Scientific s a very active area Many workshops Special issues of SIGMOD 2005, JOGC 2005, SciProg 2006 (to appear) ook on e-science s (Taylor, Deelman, Gannon, Shields eds.) to appear 2006 ill Gate s SC 2005 Keynote NSF Workshop on the Challenges of Scientific s (co-chaired with Yolanda Gil), May 2006, Ewa Deelman

28 Acknowledgments Pegasus is being developed at ISI by Gaurang Mehta, Mei-Hui Su, and Karan Vahi Wings is lead by Yolanda Gil, Jihie Kim, Varun Ratnakar DAGMan is lead by Miron Livny Many application scientists made the workflows happen (GriPhyN, NVO, LIGO, Telescience, SCEC) Ewa Deelman