PANDA A System for Provenance and Data. Example: Sales Prediction Workflow. Example: Sales Prediction Workflow. Backward Tracing. Item Sales.

Transcription

1 PANDA A System for Provenance and Data Example: Prediction Workflow Union Predict Agg -1 s 2 Example: Prediction Workflow Backward Tracing Union Predict Agg -1 s Name Amelie Jacques Isabelle Name Address ProbDemand Paris, Amelie Cowboy Hat Hat.98 high Paris, Texas Jacques Cowboy Hat.98 Paris, Texas? Isabelle Cowboy Hat.98 Texas?? 3

2 Example: Prediction Workflow Backward Tracing Union Predict Agg -1 s Name Amelie Jacques Isabelle Address Name 65, quai d'orsay, Paris Amelie 39, rue de Bretagne, Jacques Paris 20, rue d Orsel, Paris Isabelle? Address Paris, Paris, Texas Paris, Texas Texas 4 Example: Prediction Workflow... Forward Propagation Backward Tracing Dedup Split Union Predict Agg -1 s Name Amelie Jacques Isabelle Address Name Address 65, quai d'orsay, Amelie Paris Paris, France 39, rue de Bretagne, Jacques Paris Paris, France 20, rue d Orsel, Isabelle Paris Paris, France Beret Demand high 5 Provenance (very general) Where data came from How it was derived, manipulated, combined, processed, How it has evolved over time 6

3 Uses for Provenance (very general) Explanation Sources and evolution of data; deeper understanding Verification Buggy or stale source data? Buggy processing? Auditing Recomputation Propagate changes to affected downstream data 7 Some Application Domains prediction workflows Scientific-data workflows Including human-curated data Including evolving versions of data Any analytic pipeline Extract-transform-load (ETL) processes Information-extraction pipelines 8 Third Time s a Charm Isn t provenance the same thing as lineage? Haven t you worked on it before? Yes 1. Data Warehousing project (long ago) Lineage of relational views in warehouse: formal foundations, system/caching issues Lineage in ETL pipelines: foundations & algorithms 2. Trio project (recently) Data + Uncertainty + Lineage Lineage primarily in support of uncertainty Pretty much 9

4 Panda s Ambitions Previous provenance work tends to be Panda will Either data-based or process-based Capture both: data-oriented workflows Either fine-grained or coarse-grained Cover the spectrum in a unified fashion Focused on modeling and capturing provenance Also support provenance operators and queries Geared to specific functions or domains End with a general-purpose open-source system 10 Remainder of Talk Processing nodes and provenance capture Exploiting provenance Built-in operations Ad-hoc queries Other uses Current research 11 Processing Nodes... Dedup Split Union Predict Agg -1 s Relational Structured, well-understood operations Opaque (or semi-opaque) Incomplete knowledge of internals 12

5 Provenance Capture Model Ultimate model yet to be determined From bipartite graph to Open Provenance Model Goals: Support provenance at spectrum of granularities Mesh data-oriented and process-oriented provenance Composability/transitivity Understandability For remainder of talk: think bipartite graph 13 Provenance Capture Interface... Dedup Split Union Predict Agg -1 s Processing nodes provide provenance information along with output Eager or Lazy 14 Provenance Capture Interface... Dedup Split Union Predict Agg -1 Straightforward provenance s Relational, other Worst well-understood Case: transformations Automatic No access previous to fine-grained work provenance Opaque (or semi-opaque) Provided via provenance interface, or inferred 15

6 Provenance Operations Basic Backward tracing Where did the Cowboy Hat record come from? Forward tracing Which sales predictions did Amelie contribute to? Union Predict Agg -1 s 16 Additional Functionality Forward propagation Update all affected predictions after customers move from Texas to France Union Predict Agg -1 s 17 Additional Functionality Refresh Get latest prediction for Cowboy Hat sales (only) based on modified buying patterns Backward tracing + Forward propagation Union Predict Agg -1 s 18

7 Provenance Queries How many people from each country contributed to the Cowboy Hat prediction? Which customer list contributed the most to the top 100 predicted items? Union Predict Agg -1 s 19 Provenance Queries For a specific customer list, which items have higher demand than for the entire customer set? Which customers have more duplication those processed by or by? Union Predict Agg -1 s 20 Provenance Queries For a specific customer list, which items have higher demand than for the entire customer set? Which customers have more duplication those processed by or by? Query language goals Declarative ad-hoc queries à la database systems Seamlessly combine provenance and data Amenable to optimization Many interesting optimization issues 21

8 Current Research System Workflow infrastructure Basic provenance capture & operations Algorithms For refresh problem (also foundations & implementation) Theory Decision problems: eager/lazy, intermediate data Provenance foundations for semi-opaque transformations 22 Current Research System Workflow infrastructure Basic provenance capture & operations Algorithms For refresh problem See recent (also foundations & implementation) paper Theory Decision problems: eager/lazy, intermediate data Provenance foundations for semi-opaque transformations 23 Panda System (version 0.1) Command-line Client Data s Workflow Provenance Predicate s Panda Layer ite Forward Filter s Graphical Interface File System Arbitrary acyclic workflows processing nodes automatic capture processing nodes one-one, one-many Backward-tracing and forwardpropagation 24

9 Future Work The large majority of this talk Stay tuned 25 Backup slides Optimization Issues Query-driven provenance capture Eager vs. lazy provenance capture Space-time & query-update tradeoffs Processing-node dependent Ex: Dedup, Agg Retain intermediate data sets? Fine-grained vs. coarse-grained Approximate provenance 27

10 Panda System (version 0.1) Command-line Client Graphical Interface Panda Layer Data s Workflow Provenance Predicate s ite Forward Filter s File System 28 System: Key Features Arbitrary acyclic workflows processing nodes Automatic provenance capture processing nodes One-one and one-many transformations only Backward-tracing and forward-propagation 29 Panda System (version 0.1) Command-line Client Graphical Interface Create Data s Workflow Provenance Predicate s Panda Layer ite Forward Filter s File System 30

11 Panda System (version 0.1) Command-line Client Graphical Interface Create Transformatio n Data s Workflow Provenance Predicate s Panda Layer ite Forward Filter s File System 31 Panda System (version 0.1) Command-line Client Graphical Interface Create Transformatio n Data s Workflow Provenance Predicate s Panda Layer ite Forward Filter s File System 32 Panda System (version 0.1) Command-line Client Graphical Interface Refresh Output Element Panda Layer ite Data s Workflow Provenance Predicate s Forward Filter s File System 33

12 Provenance-Based Refresh Problem Exploit provenance to efficiently compute the up-to-date value of selected output elements Challenges: See recent Formally defining provenance paper & refresh When is new value up-to-date version of old one? Provenance as key When can selective refresh be done correctly and efficiently? Properties of transformations, workflows, provenance Algorithms for wide class of transformations and workflows 34