The Virtual Data Grid: A New Model and Architecture for Data-Intensive Collaboration

Transcription

1 The Virtual Data Grid: A New Model and Architecture for Data-Intensive Collaboration Summer Grid 2004 UT Brownsville South Padre Island Center 24 June 2004 Mike Wilde Argonne National Laboratory Mathematics and Computer Science Division

2 GriPhyN: Grid Physics Network Mission Enhance scientific productivity through discovery and processing of datasets, using the grid as a scientific workstation Virtual Data enables this approach by creating datasets from workflow recipes and recording their provenance. GriPhyN works to cross the chasm - application and computer scientists create and field-test paradigms and toolkits together 2

3 Acknowledgements: Virtual Data is a Large Team Effort The Chimera Virtual Data System is the work of Ian Foster, Jens Voeckler, Mike Wilde and Yong Zhao The Pegasus Planner is the work of Ewa Deelman, Gaurang Mehta, and Karan Vahi Applications described are the work of many people, including: James Annis, Rick Cavanaugh, Dan Engh, Rob Gardner, Albert Lazzarini, Natalia Maltsev, Marge Bardeen, and their wonderful teams 3

4 Virtual Data Scenario file1 psearch t 10 file8 simulate t 10 file2 reformat f fz file1 file1 File3,4,5 file7 conv I esd o aod file6 summarize t 10 Update workflow following changes Manage workflow; Explain provenance, e.g. for file8: psearch t 10 i file3 file4 file5 o file8 summarize t 10 i file6 o file7 reformat f fz i file2 o file3 file4 file5 conv l esd o aod i file 2 o file6 simulate t 10 o file1 file2 On-demand data generation 4

5 Virtual Data Describes analysis workflow file1 psearch t 10 file8 simulate t 10 file2 reformat f fz file1 file1 File3,4,5 file7 Requested dataset conv I esd o aod file6 summarize t 10 The recorded virtual data recipe here is: Files: 8 < (1,3,4,5,7), 7 < 6, (3,4,5,6) < 2 Programs: 8 < psearch, 7 < summarize, (3,4,5) < reformat, 6 < conv, (1,2) < simulate 5

6 Virtual Data Describes analysis workflow file1 psearch t 10 file8 simulate t 10 file2 reformat f fz file1 file1 File3,4,5 file7 Requested file conv I esd o aod file6 summarize t 10 To recreate file 8: Step 1 simulate > file1, file2 6

7 Virtual Data Describes analysis workflow file1 psearch t 10 file8 simulate t 10 file2 reformat f fz file1 file1 File3,4,5 file7 Requested file conv I esd o aod file6 summarize t 10 To re-create file8: Step 2 files 3, 4, 5, 6 derived from file 2 reformat > file3, file4, file5 conv > file 6 7

8 Virtual Data Describes analysis workflow file1 psearch t 10 file8 simulate t 10 file2 reformat f fz file1 file1 File3,4,5 file7 Requested file conv I esd o aod file6 summarize t 10 To re-create file 8: step 3 File 7 depends on file 6 Summarize > file 7 8

9 Virtual Data Describes analysis workflow file1 psearch t 10 file8 simulate t 10 file2 reformat f fz file1 file1 File3,4,5 file7 Requested file conv I esd o aod file6 summarize t 10 To re-create file 8: final step File 8 depends on files 1, 3, 4, 5, 7 psearch < file1, file3, file4, file5, file 7 > file 8 9

10 Grid3 The Laboratory Supported by the National Science Foundation and the Department of Energy. 10

11 VDL: Virtual Data Language Describes Data Transformations Transformation Abstract template of program invocation Similar to "function definition" Derivation Function call to a Transformation Store past and future: > A record of how data products were generated > A recipe of how data products can be generated Invocation Record of a Derivation execution These XML documents reside in a virtual data catalog VDC - a relational database 11

12 TR tr1(in a1, out a2) { VDL Describes Workflow via Data Dependencies argument stdin = ${a1}; argument stdout = ${a2}; } TR tr2(in a1, out a2) { argument stdin = ${a1}; argument stdout = ${a2}; } DV x1->tr1(a1=@{in:file1}, a2=@{out:file2}); DV x2->tr2(a1=@{in:file2}, a2=@{out:file3}); file1 x1 file2 x2 file3 12

13 Workflow example preprocess findrange findrange analyze Graph structure Fan-in Fan-out "left" and "right" can run in parallel Needs external input file Located via replica catalog Data file dependencies Form graph structure 13

14 Complete VDL workflow Generate appropriate derivations DV out:"f.b2"} ], ); DV left->findrange( name="left", p="0.5" ); DV right->findrange( name="right" ); DV ); 14

15 Compound Transformations Enable Functional Abstractions Compound TR encapsulates an entire sub-graph: TR rangeanalysis (in fa, p1, p2, { out fd, io fc1, io fc2, io fb1, io fb2, ) call preprocess( a=${fa}, b=[ ${out:fb1}, ${out:fb2} ] ); call findrange( a1=${in:fb1}, a2=${in:fb2}, name="left", p=${p1}, b=${out:fc1} ); call findrange( a1=${in:fb1}, a2=${in:fb2}, name="right", p=${p2}, b=${out:fc2} ); call analyze( a=[ ${in:fc1}, ${in:fc2} ], b=${fd} ); } 15

16 Derivation scripts Representation of virtual data provenance: DV d1->diamond( p2="100", p1="0" ); DV d2->diamond( p2=" ", p1="0" );... DV d70->diamond( p2="800", p1="18" ); 16

17 Invocation Provenance Completion status and resource usage Attributes of executable transformation Attributes of input and output files 17

18 Executing VDL Workflows Abstract workflow Global planner Pegasus jit planner (research) Concrete DAG Grid Info local planner DAGman / Condor-G 18

19 GriPhyN-iVDGL Applications to date ATLAS, BTeV, CMS HEP event simulation Argonne Computational Biology sequence comparison and result capture LIGO Pulsar search Sloan Digital Sky Survey cluster finding; near-earth object search planned Quarknet science education cosmic rays, HEP analysis 19

20 Genome Analysis Database Update Hit Public and Run Registered Groups Collaborators Data Flow and Storage at various levels Automatic Workflows Created as per User Request or Project A B C D Jazz/ANL B C A D End Users Interface to the Server UofWisc Grid3 Grid D B C A GADU - G C A D B Server Jetspeed Application work by Alex Rodriguez, Dina Sulakhe, Natalia Matlsev, Argonne MCS Described in GGF10 workshop paper. Chimera, Condor, Globus 20

21 Virtual Data Example: Galaxy Cluster Search DAG Sloan Data Galaxy cluster size distribution Number of Clusters Number of Galaxies Jim Annis, Steve Kent, Vijay Sehkri, Fermilab, Michael Milligan, Yong Zhao, University of Chicago. Described in SC2002 paper 21

22 Cluster Search Workflow Graph and Execution Trace Workflow jobs vs time 22

23 Virtual Data Application: mass = 200 decay = bbhigh Energy Physics Data Analysis mass = 200 mass = 200 decay = WW mass = 200 decay = ZZ mass = 200 decay = WW stability = 3 mass = 200 decay = WW stability = 1 LowPt = 20 HighPt = mass = 200 event = 8 mass = 200 plot = 1 Work and slide by Rick Cavanaugh and Dimitri Bourilkov, University of Florida Ref: CHEP 2002 paper mass = 200 decay = WW plot = 1 mass = 200 decay = WW stability = 1 mass = 200 decay = WW event = 8 mass = 200 decay = WW stability = 1 event = 8 mass = 200 decay = WW stability = 1 plot = 1 23

24 Using Virtual Data for Science Education The QuarkNet-Trillium collaboration is using Grid virtual data tools and methods to enrich science education Its an experiment to give students the means to: discover and apply datasets, algorithms, and data analysis methods collaborate by developing new ones and sharing results and observations learn data analysis methods that will ready and excite them for a scientific career And in later steps, we may actually use the Grid! 24

25 Quarknet Virtual Data Project Quarknet Virtual Data Portal Student Data, Algorithms, Results, Notes, and communications Central High School Reston, Virginia Locally Collected Data Cosmic Ray Detector Student/ Teacher Teams Virtual Data Toolkit Standard Web access Foothills High School Great Falls, Montana Locally Collected Data Cosmic Ray Detector Student/ Teacher Teams Virtual Data Catalog Yale / Middletown High Collaboration Hartford, Connecticut Locally Collected Data Cosmic Ray Detector Student teacher teams sharing data, methods, programs, and knowledge Student/ Teacher Teams Enabling collaboration-intensive science discovery with virtual data tools and methods 25

26 Detector Performance Study 26

27 Example: BTeV Event Simulation 27

28 Support for Search and Discovery Goal: make it as easy to use as Google More advanced capabilities lie below the surface (as with Google) Understand the structure and meaning of the datasets and their fields. Advanced search, using SQL-like queries Find both DATA and TRANSFORMATIONS Create datasets from queries Perform calculations on datasets, filtering results to look for patterns 28

29 Search by Metadata 29

30 Derving a new dataset to find mass of z particle: 30

31 Workflow for missing energy calculations 31

32 Virtual Provenance: list of derivations and files <job id="id000001" namespace="quarknet.hepsrch" name="ecalenergysum" level="5 dv-namespace="quarknet.hepsrch" dv-name="run1aesum"> <argument><filename file="run1a.event"/> <filename file="run1a.esm"/></argument> <uses file="run1a.esm" link="output" dontregister="false" donttransfer="false"/> <uses file="run1a.event" link="input" dontregister="false" donttransfer="false"/> </job> <job id="id000002" namespace="quarknet.hepsrch" name="ecalenergysum" level="7 dv-namespace="quarknet.hepsrch" <argument><filename file="electron10gev.event"/> <filenamefile="electron10gev.sum </job> <job id="id000014" namespace="quarknet.hepsrch" name="recontotalenergy" level="3" <argument><filename file="run1a.mis"/> <filename file="run1a.ecal"/> <uses file="run1a.muon" link="input" dontregister="false" donttransfer="false"/> <uses file="run1a.total" link="output" dontregister="false" donttransfer="false"/ <uses file="run1a.ecal" link="input" dontregister="false" donttransfer="false"/> <uses file="run1a.hcal" link="input" dontregister="false" donttransfer="false"/> <uses file="run1a.mis" link="input" dontregister="false" donttransfer="false"/> </job> <!--list of all files used --> <filename file="ecal.pct" link="inout"/> <filename file="electron10gev.avg" link="inout"/> <filename file="electron10gev.sum" link="inout"/> <filename file="hcal.pct" link="inout"/>. (excerpted for display) 32

33 Virtual Provenance in XML: control flow graph <child ref="id000003"> <parent ref="id000002"/> </child> <child ref="id000004"> <parent ref="id000003"/> </child> <child ref="id000005"> <parent ref="id000004"/> <parent ref="id00000 <child ref="id000009"> <parent ref="id000008"/> </child> <child ref="id000010"> <parent ref="id000009"/> <parent ref="id00000 <child ref="id000012"> <parent ref="id000011"/> </child> <child ref="id000013"> <parent ref="id000011"/> </child> <child ref="id000014"> <parent ref="id000010"/> <parent ref="id00001 <parent ref="id000013"/> </child> (excerpted for display ) 33

34 And writing the results up in a poster

35 Poster describing analysis 35

36 Using active data from Web Services 36

37 37

38 38

39 39

40 Levels of Interaction Skins use it like a calculator, experiment with scenarios and settings, use virtual data like a log book to document, assess, and share parameter values. Blocks re-assemble workflow pipelines using existing ones as patterns and predeveloped transforms as building blocks Code write new transforms in a variety of languages and data models 40

41 Observations A provenance approach based on interface definition and data flow declaration fits well with Grid requirements for code and data transportability and heterogeneity Working in a provenance-managed system has many fringe benefits: uniformity, precision, structure, communication, documentation The real world is messy finding the right abstractions is hard, and handling legacy applications is even harder 41

42 Vision for Provenance in the Large Universal knowledge management and production systems Vendors integrate the provenance tracking protocol into data processing products Ability to run anywhere in the Grid 42

43 Virtual Data Grid Vision discovery virtual data catalog virtual data catalog Production Manager planning Science Review workflow executor (DAGman) composition request executor (Condor-G, GRAM) workflow planner Researcher Grid Monitor request planner request predictor (Prophesy) sharing discovery derivation Data Transport virtual data index storage element replica location service simulation data simulation Data Grid storage element virtual data catalog storage element raw data Storage Resource Mgmt analysis detector Grid Operations Computing Grid 43

44 Planned Dataset Model <FORM <Title > /FORM> File Set of files Object closure XML Element Relational query or spreadsheet range New user-defined dataset type: Set of files with relational index Speculative model described in CIDR 2003 paper by Foster, Voeckler, Wilde and Zhao 44

45 Planned Dataset Type Model FileDataset Representational File FileSet MultiFileSet (Nonleaf Types are Superclasses) TarFileSet Logical EventCollection RawEventSet SimulatedEventSet MonteCarlo Simulation DiscreteEvent Simulation 45

46 Provenance Server Plans OGSA-based Grid services Discovery, security, resource management Supports code and data discovery and workflow management Object names (TR, DS, TY, DV, IV) can be used as global cross-server links Derivations can reference remote transformations and datasets Structured object namespaces & object-level access control enable large VO collaboration Generalize transforms to describe service calls, database queries and language interpreters 46

47 Provenance Hyperlinks Personal VDS DV DS TR DV DV TR TR TR DS DV DS Collaboration VDS Group VDS DV DV Personal VDS 47

48 Indexing Servers to Support Discovery Group Index Personal VDS Collaborationlevel index TR TR TR Collaboration VDS DV TR Group VDS DS DV DV DS DV DV DV DS Personal Index Personal Index Personal Index Personal VDS Collaboration-wide index 48

49 For Information and Software Virtual Data System - Chimera Virtual Data System: Overview, papers, software Grids and Grid Software - Using Grid3 - Virtual Data Toolkit The Globus Toolkit - The Condor Project Particle Physics Data Grid 49

50 Acknowledgements GriPhyN, ivdgl, and QuarkNet (in part) are supported by the National Science Foundation The Globus Alliance, PPDG, and QuarkNet are supported in part by the US Department of Energy, Office of Science; by the NASA Information Power Grid program; and by IBM 50