Towards Integrating Work1low and Database Provenance

Transcription

1 IPAW 12 Interna'onal Provenance and Annota'on Workshop Towards Integrating Work1low and Database Provenance Fernando Chiriga- and Juliana Freire Polytechnic Ins4tute of NYU

2 Database Provenance Fine- grained provenance Focus: deriva4on of a piece of data in a dataset Provenance for tuples in a rela4onal database Propaga4on of provenance through queries Which parts of the database D contributed to the piece of data t according to query Q? [Cheney et al., FTDB 2009] Employee Department Name DeptName Salary And CBE $50,000 DeptName CBE Address 6 MetroTech = Chris CSE $60,000 CSE 2 MetroTech Robert CSE $55,000 Ryan ECE $40,000 Name DeptName Salary Address And CBE $50,000 6 MetroTech Chris CSE $60,000 2 MetroTech Robert CSE $55,000 2 MetroTech

3 Work1low Provenance Coarse- grained provenance A directed graph describing a computa4onal task Ver4ces = modules = processing steps + parameters Edges = connec4ons between output and input ports Execu4on order determined by flow of data from output to input ports Record of the en4re history of the deriva4on of the output [Davidson and Freire, SIGMOD 2008] head.120.vtk resampled-head-vis.png

4 Work1lows + Databases: Challenges s 12 3 input output1 output1 output2 output2 output Run 12 output (Run 1) output (Run 2)

5 Work1lows + Databases: Challenges input s 1 s 2 output Run 21 output (Run 1) output (Run 2)

6 Work1lows + Databases: Challenges Workflows are func4onal: there are no states or side effects Outputs are a func'on of the inputs O3 = vtkdatasetmapper(input=o2) O2 = vtkcontourfilter(value=57,input=o1) O1 = vtkstructuredreader(input=i1) Databases follow a stateful model The models seem to be incompa4ble: accesses to databases break the stateless scien4fic workflow model How to support reproducibility? O1 O2 I1 O3 [Davidson and Freire, SIGMOD 2008] How to properly describe the provenance of workflows that access (or modify) databases?

7 Contributions Model for integra4ng database and workflow provenance Keeps track of each state of a database leverage transac4on temporal databases [Jensen, 2000] Uniquely iden4fies a database state using transac'on 'me Supports reproducibility and provenance querying Reflects func4onality available in commercial rela4onal databases Oracle RDBMS: hdp:// op'ons/total- recall/overview/index.html DB2: hdp:// temporal/index.html?ca=drs- Proof- of- concept implementa4on VisTrails workflow system Oracle RDBMS

8 Background and De1initions Stateless workflows A module is represented by the func4on input m M f m (I m ) = < O m > output1 output2 f GetInfo (input) = < output1> f ReadDB (output1) = < output2 > f ProcessData (output2) = < output > output

9 Background and De1initions Stateful databases Use the model of transac4on temporal databases Adapt the backlog schema [Jensen, JIS 1993] A backlog keeps track of changes in a rela4on It is append- only Maintain a sequence of states S(R) of a rela4on R S(R) = {(S 1 (R),T 1 (R)),...,(S n (R),T n (R))} The difference (delta) between two states is computed as: Δ j,i (R) = S j (R) S i (R), i < j

10 Example of Backlog Relation Emp BR Emp K Name Job T Op U S 3 S 2 S 1 1 Robert Research Researcher Assistant 10 I fchiriga4 2 Claire Assistant Director 10 I fchiriga4 31 Robert Eric Administra4ve Researcher Director 15 D jfreire 1 Robert Research Assistant 15 I jfreire 3 Eric Administra4ve Director 20 I fchiriga4 S(Emp) = {(S 1 (Emp),10),(S 2 (Emp),15),(S 3 (Emp), 20)} Δ 3,1 (Emp) = 1 Robert Researcher 15 D jfreire 1 Robert Research Assistant 15 I jfreire 3 Eric Administra4ve Director 20 I fchiriga4

11 Integrating Work1low and Database Provenance Key idea: capture informa4on about the database states Previously, we had: Now, we have: f m (I m ) = < O m > f m (I m,[r,t b (R)]) = < O m,[r,t a (R)] > state before the execu4on state amer the execu4on

12 Example of Integrated Model Δ 2,1 (R) Δ 3,2 (R) Δ 4,3 (R) S 1 (R) S 2 (R) S 3 (R) S 4 (R) I 11 I 21 m 11 m 21 I 12 I 13 O 21 m 12 m 13 m 22 O 12 O 13 m 14 m 23 W1 O 14 W2 O 23 f m12 (I 12,[R,T 1 (R)]) = < O 12,[R,Tf 1 m21 (R)] (I 21 >,[R,T 4 (R)]) = < O 21,[R,T 4 (R)] > f m13 (I 13,[R,T 1 (R)]) = < O 13,[R,T 2 (R)] >

13 Reproducibility Database states are tracked during workflow execu4on To reproduce, retrieve original state using recorded transac4on 4me, if necessary Δ 2,1 (R) Δ 3,2 (R) Δ 4,3 (R) S 1 (R) S 2 (R) S 3 (R) S 4 (R) Δ 5,4 (R) S 5 (R) I 11 I 21 m 11 m 21 I 12 I 13 O 21 m 12 m 13 O 12 O 13 m 22 f m21 (I 21,[R,T 4 (R)]) = < O 21,[R,T 4 (R)] > m 14 m 23 W1 O 14 W2 O 23

14 Querying Provenance The model supports different queries Lineage of an output Lineage of a database state How- provenance see paper for details Lineage of a tuple see paper for details

15 Lineage of an Output Δ 2,1 (R) Δ 3,2 (R) Δ 4,3 (R) S 1 (R) S 2 (R) S 3 (R) S 4 (R) I 11 I 21 m 11 m 21 I 12 I 13 O 21 m 12 m 13 m 22 O 12 O 13 m 14 m 23 W1 O 14 W2 O 23 lineage(o 14 ) = {W 1, I 11,[R,T 1 (R)]}

16 Lineage of a State Δ 2,1 (R) Δ 3,2 (R) Δ 4,3 (R) S 1 (R) S 2 (R) S 3 (R) S 4 (R) I 11 I 21 m 11 m 21 I 12 I 13 O 21 m 12 m 13 m 22 O 12 O 13 m 14 m 23 W1 O 14 W2 O 23 f m13 (I 13,[R,T 1 (R)]) = < O 13,[R,T 2 (R)] > lineage(s 2 (R)) = {W 1, f m13, I 13,[R,T 1 (R)]}

17 How- Provenance The model has informa4on about states S b and S a The model can retrieve the difference Δ a,b Through this difference, we know the opera4ons performed by the module We know exactly how the module modified the rela4on

18 Lineage of a Tuple ComputeLineage(t) : lineage(t) = {} b _ times = select T, Op from R B where K = k t for each workflow instance W : for each module m M : for each T in b _ times : if (T in f m 's output) and (T not in f m 's input) : lineage(t). add([ f m,op])

19 Implementation VisTrails Workflow- based data explora4on system Provides support for provenance Oracle RDBMS Total Recall feature Tracks the changes by using a history table Similar to a backlog rela4on SELECT column_name FROM table_name AS OF time S SELECT column_name FROM table_name VERSIONS BETWEEN time_1 AND time_2 Δ

20 VisTrails Total Recall Package SELECT name FROM mountaineers WHERE AS OF country= Brazil " SCN( ) WHERE country= Brazil " T b = T a =

21 Related Work [Acar et al., TaPP 2010] Propose a common provenance graph model Use DFL to support database queries and workflow steps [Amsterdamer et al., VLDB 2011] Propose a framework to integrate fine- grained database- style provenance into workflows Modules are Pig La4n programs Transla4on to nested rela4onal calculus expressions

22 Conclusion Proposed model to integrate workflow and database provenance Leverages func4onality of transac4on temporal databases to capture database states and uniquely iden4fy them Supports reproducibility Supports a rich set of provenance queries that straddle workflows and databases Future work Querying - - efficiency and interfaces Support DDL opera4ons

23 IPAW 12 Interna'onal Provenance and Annota'on Workshop Acknowledgements Thanks to: VisTrails Team, Dieter Gawlick and Venkatesh Radhakrishnan (Oracle) and Jan Van den Bussche This work is par4ally supported by the Na-onal Science Founda-on Any ques4ons? THANK YOU!