Dac-Man: Data Change Management for Scientific Datasets on HPC systems

Size: px

Start display at page:

Download "Dac-Man: Data Change Management for Scientific Datasets on HPC systems"

Catherine Mosley
5 years ago
Views:

1 Dac-Man: Data Change Management for Scientific Datasets on HPC systems Devarshi Ghoshal Lavanya Ramakrishnan Deborah Agarwal Lawrence Berkeley National Laboratory

Motivation Data Releases Storage Resources Compute Resources

updated limited or no provenance Longer time to scientific

due to complexity Inefficient use of compute and storage resources

2 Motivation Data Releases Storage Resources Compute Resources Scientific Discovery Large scientific datasets are frequently updated limited or no provenance Longer time to scientific discovery scientists delay updating the downstream data products due to complexity Inefficient use of compute and storage resources users often rerun data processing pipelines without understanding the impact of change 2

3 Limitations of Existing Tools Sequentially compare datasets Do not save/reuse change information Generates un-interpretable change results Lack information necessary to assess the impact of data change Unable to quantify change for scientific datasets Do not scale 3

4 Dac-Man: DAta Change MANagement A framework that identifies, captures and manages change in large scientific datasets, and enables plug-in of domain-specific change analysis with minimal user effort Is not a version control system Designed to scale on an HPC system Allows users to efficiently interpret and quantify changes 4

Architecture User interfacing components Change tracker: scans and compares data objects; manages data change Query manager: retrieves change

5 Architecture User interfacing components Change tracker: scans and compares data objects; manages data change Query manager: retrieves change information Change and metadata management Indexing area: stores indexes and filesystem metadata Caching area: stages file and data change information 5

6 Dac-Man Indexes Key: File Value: Hash(File data) Value: File Key: Hash(File data) Bi-directional indexing helps Dac-Man identify both data and metadata changes in files MPI workers build the indexes in parallel Saved on filesystem for portability and reuse 6

Comparators File comparator recursively compares files and directories, including subdirectories and symbolic links uses indexes to compare the files classifies file changes into different types

7 Comparators File comparator recursively compares files and directories, including subdirectories and symbolic links uses indexes to compare the files classifies file changes into different types modified, metadata-only, added/deleted Data comparator compares data within files Data adaptors transform different scientific data formats into Dac- Man records Dac-Man records are a collection of key-value pairs allows external scripts to be used as data comparators 7

metadata, including subdirectory changes Used for improving subsequent change

8 Dac-Man Cache Dataset-1 Datapath-1 Datapath-2 Cache-id : Cached query Dataset-2 Cache entry: Cache-id Change metadata Similar to a staging area in Git Saves file change metadata, including subdirectory changes Used for improving subsequent change retrieval queries Invalid cache entries are updated by re-indexing and recomputing the changes 8

9 Change Capture Workflow Command-line: dacman diff [options] OLD NEW scan index compare diff crawls data directories, saves directory structures and associated file metadata indexes filesystem objects uses indexes to compare files and save the change results retrieves the changes 9

10 Data Provenance in Dac-Man Metadata information captured by Dac-Man contributes to the provenance of a dataset Tracks data provenance in workflows by correlating data changes between inputs and outputs Enables using provenance information to analyze the impact of changes V1 V2 V3 10

11 Evaluation System NERSC s Cori supercomputer 32 cores per node, 128 GB DDR4 memory Datasets Sloan Digital Sky Survey (SDSS) primarily consists of FITS files multiple data releases with approx. 9.7 million files Fluxnet consists of CSV files approx files Synthetic files contain arbitrary binary data different number of files and amount of data for controlled experiments Tools: Unix diff, Git diff, Python filecmp 11

12 Comparison to Existing Tools Performs 100x better than existing diff tools 12

13 Scalability of Indexing Indexing time reduces as more resources are allocated 13

14 Amount of Data Change Performance is constant irrespective of the amount of change 14

15 Change Retrieval Speeds up change retrieval by caching change information 15

16 Conclusions Provides a scalable framework for identifying and capturing changes in large scientific datasets Generates and shares data change summaries useful for data consumers Identifies, captures and tracks changes of different types and granularities Provides a portable solution to compare remote datasets Provides meaningful change results through the use of domainspecific change metrics Can be integrated with data processing pipelines and streaming datasets for real-time change analysis 16

17 Acknowledgments Deduce Project ( U.S. DOE Contract No. DE-AC02-05CH11231 Program Manager: Rich Carlson Stephen Bailey, Boris Faybishenko, Juliane Mueller, Alex Romosan, Craig Tull 17

Seeking Supernovae in the Clouds: A Performance Study

Seeking Supernovae in the Clouds: A Performance Study Keith R. Jackson, Lavanya Ramakrishnan, Karl J. Runge, Rollin C. Thomas Lawrence Berkeley National Laboratory Why Do I Care About Supernovae? The rate