Perm Integrating Data Provenance Support in Database Systems

Transcription

1 Perm Integrating Data Provenance Support in Database Systems Boris Glavic Database Technology Group Department of Informatics University of Zurich Gustavo Alonso Systems Group Department of Computer Science ETH Zurich

2 Overview. Introduction 2. The Perm Provenance Representation 3. Query Rewriting for Provenance Computation 4. Advanced Topics and New Stuff 5. Results 6. Conclusion 2

3 . Introduction Query Which input data item(s) contributed to which output data item(s)? Granularity Tuple Attribute Value... Contribution semantics Influence (Why) Copy (Where)... 3

4 . Introduction Application domains Datawarehousing Scientific data and curated databases Data Integration / Data Exchange Workflow-Management-Systems Accountability... 4

5 . Introduction The problem of computing this type of provenance has been solved before See e.g. [Cui, Widom ICDE 00] but... Non-relational representation of provenance data Separation of provenance and normal data Non-relational computation of provenance data Important SQL-features were not addressed Nested Subqueries 5

6 . Introduction A lot of work has been done in Data Provenance A solid theoretical foundation Several Workflow systems support provenance Some research prototypes 6

7 . Introduction Lack of real Provenance Management Systems Manage provenance information Generate on the fly Store for later use Handle external provenance Make provenance queryable Expressive provenance representation Expressive query language In connection with normal data Contribution semantics No one fits all 7

8 . Introduction Perm Provenance Extension of the Relational Model Provenance Management System Pure relational representation of provenance Query result tuples and provenance tuples are represented as a single relation 8

9 . Introduction Benefits: Provenance can be Stored in standard DBMS... Queried using SQL... Directly interpreted by a user Direct association between provenance and normal data 9

10 . Introduction Provenance computation On the fly -> Use query rewrite Given query q Generate query q+ Computes the provenance of all result tuples from q By propagation 0

11 . Introduction Benefits: Rewritten query is expressed in relational algebra Can be optimized and executed by a R-DBMS E.g. can be stored as a view Used as a subquery

12 . Introduction Extension of PostgreSQL DBMS Implemented inside of PostgreSQL Extended SQL language Perm module Implements algebraic rewrite on internal representation of a query 2

13 . Introduction SQL-PLE: SQL extension SELECT PROVENANCE... Nice benefits: CREATE VIEW x AS SELECT PROVENANCE... SELECT PROVENANCE... INTO x... SELECT... FROM (SELECT PROVENANCE... 3

14 . Introduction External provenance SELECT PROVENANCE... FROM view PROVENANCE (attr,...) Trace provenance of intermediate results SELECT PROVENANCE... FROM view BASERELATION 4

15 . Introduction Perm Browser SELECT PROVENANCE... SELECT a,b... JDBC psql Perm 5

16 . Introduction Perm Browser SELECT PROVENANCE... SELECT a,b... JDBC psql Perm 6

17 . Introduction Perm Architecture Parser & Analyser Rewriter Perm Module Planner Executor SELECT PROVENANCE... Q =... Q =... Q + =... MergeJoin (... 7

18 Overview. Introduction 2. The Perm Provenance Representation 3. Query Rewriting for Provenance Computation 4. Results 5. Conclusion 6. Demo 8

19 2. The Perm Provenance Representation What information are we representing? Tuple-level provenance Contributing tuples from base relations For a query Contribution semantics: Influence (Lineage) 9

20 2. The Perm Provenance Representation Definition of contribution semantics Why/Influence-provenance Introduced in [Cui, Widom ICDE 00] Provenance defined as a list of subsets of the input relations Defined for a single algebra operator and a single result tuple 20

21 2. The Perm Provenance Representation Definition : For a single algebra operator op with input relations T,..., Tn a list (T*,...,Tn*) of maximal subsets of the input relation is the provenance of a tuple t from the result of op iff:. op(t*,..., Tn*) = t 2. For all i and t* with t* in Ti*: op(t*,... Ti-*, t*, Ti+*,...,Tn*)!= " 2

22 2. The Perm Provenance Representation sales sname Coop Coop itemid items id 2 3 price

23 2. The Perm Provenance Representation Compute the sum of sales for each shop SELECT sname, sum(price) FROM sales, items WHERE itemid = id GROUP BY sname; 23

24 2. The Perm Provenance Representation sales sname Coop Coop itemid items id 2 3 result name Coop price Sum(price)

25 2. The Perm Provenance Representation sales sname Coop Coop itemid items id 2 3 result name Coop price Sum(price)

26 2. The Perm Provenance Representation sales sname Coop Coop itemid items id 2 3 result name Coop price Sum(price)

27 2. The Perm Provenance Representation Desired result format: Original Attributes Relation Attributes Relation n Attributes 27

28 2. The Perm Provenance Representation Original result sales items name sum(price) P(sName) P(itemId) P(id) P(price) Coop 0 Coop Coop 0 Coop

29 Overview. Introduction 2. The Perm Provenance Representation 3. Query Rewriting for Provenance Computation 4. Results 5. Conclusion 6. Demo 29

30 3. Query Rewriting for Provenance Computation Rewrite method basics Use algebra representation of the query Replace every algebra operator with an algebra statement that propagates provenance alongside with the original results -> need a rewrite rule for each relational algebra operator 30

31 3. Query Rewriting for Provenance Computation Rewrite process op op3 op2 3

32 3. Query Rewriting for Provenance Computation Rewrite process Apply Rewrite rule opa op opb opc op3 op2 op3 op2 32

33 3. Query Rewriting for Provenance Computation Rewrite process opa opb opc Apply Rewrite rules op3 op2 33

34 3. Query Rewriting for Provenance Computation Rewrite rules notations: T + P(T + ) Rewritten statement (query) Provenance attributes 34

35 3. Query Rewriting for Provenance Computation Rewrite rules example: SELECT agg, G FROM T GROUP BY G SELECT agg, G, P(T + ) FROM (SELECT agg, G FROM T GROUP BY G) AS agg LEFT OUTER JOIN (SELECT G AS G, P(T + ) FROM T + ) AS prov ON (G = G ) 35

36 3. Query Rewriting for Provenance Computation Rewrite rules example: SELECT sum(revenue) AS sum, shop FROM sales GROUP BY shop sales shop month revenue result sum shop Jan Feb 0 50 Coop Mar 0 Coop Jan 25 Coop Feb 25 36

37 3. Query Rewriting for Provenance Computation SELECT sum, shop, pshop, pmonth, prevenue FROM (SELECT sum(revenue) AS sum, shop FROM sales GROUP BY shop) AS agg LEFT OUTER JOIN (SELECT shop AS shop, pshop, pmonth, prevenue FROM sales + ) AS prov ON (shop = shop ) sum shop pshop pmonth prevenue 20 Jan Feb 0 20 Mar 0 50 Coop Coop Jan Coop Coop Feb 25 37

38 3. Query Rewriting for Provenance Computation SELECT sum, shop, pshop, pmonth, prevenue FROM (SELECT sum(revenue) AS sum, shop FROM sales GROUP BY shop) AS agg LEFT OUTER JOIN (SELECT shop AS shop, pshop, pmonth, prevenue FROM sales + ) AS prov ON (shop = shop ) sum shop pshop pmonth prevenue 20 Jan Feb 0 20 Mar 0 50 Coop Coop Jan Coop Coop Feb 25 38

39 3. Query Rewriting for Provenance Computation SELECT sum, shop, pshop, pmonth, prevenue FROM (SELECT sum(revenue) AS sum, shop FROM sales GROUP BY shop) AS agg LEFT OUTER JOIN (SELECT shop AS shop, pshop, pmonth, prevenue FROM sales + ) AS prov ON (shop = shop ) sum shop pshop pmonth prevenue 20 Jan Feb 0 20 Mar 0 50 Coop Coop Jan Coop Coop Feb 25 39

40 Overview. Introduction 2. The Perm Provenance Representation 3. Query Rewriting for Provenance Computation 4. Advanced Topics and New Stuff 5. Results 6. Conclusion 40

41 4. Copy Contribution Semantics Influence contribution semantics (I-CS) Also tuples with conditional influence Copy contribution semantics (C-CS) Tuples that have been (partially) copied to the result Subsumption: C-CS is a subset of I-CS 4

42 4. Copy Contribution Semantics Computation:. Statically analyze the query: Which attributes are copied to which attributes 2. Use query rewrite again Could use ) to filter I-CS output.....but ) can be used to heavily prune 42

43 4. Transformation Provenance Until now: Data-data provenance Which input data influenced which output data? Why not: Transformation provenance Which parts of a query influenced which output data? 43

44 4. Transformation Provenance SELECT * FROM R UNION SELECT * FROM S; SELECT * FROM R LEFT JOIN S on (a = b); 44

45 4. Transformation Provenance Representation: SELECT * FROM R LEFT JOIN <Not>S</Not> ON... <?xml version... Computation Still query rewrite Bit-sets to represent all operators of query UDF generate representation 45

46 4. Nested Subqueries Provenance for nested subqueries Important for typical provenance application domains Not addressed by other approaches 46

47 Sublinks 4. Nested Subqueries Subqueries in e.g. SELECT-clause " (R) a IN " (b=3) (S) Correlated Nested References outside attributes " (R) a IN " (b=a ) (S) Sublink that contains sublinks " (R) a IN " (b = ANY (T )) (S) 47

48 4. Nested Subqueries Sublinks play different roles in an expression Role can differ per tuple, changes provenance Computation I think you know ;-) Problems q = " a =ANY #b (S)$a=3(R) Definition breaks -> extend it How to determine role? Cannot access result of sublink query -> Produce all possible provenance tuples Simulate join by adding correlation 48

49 4. Optimizations Selection Pushdown Use case: User is interested in only the provenance of a part of the query result E.g. SELECT * FROM (SELECT PROVENANCE...) WHERE... Rationale: Reducing the size of intermediate results is even more important for provenance computation Optimization: More aggressive selection pushdown than provided by Postgres Selection sometimes allow us to transfrom outer into inner joins 49

50 4. Optimizations Un-nesting and de-correlation of sublinks Use case: Queries with sublinks Rationale: Generic strategy is very expensive Provenance computation of a join is relatively cheap and straightforward Optimization: Apply standard un-nesting and de-correlation strategies (maybe adapted for provenance computations) 50

51 Overview. Introduction 2. The Perm Provenance Representation 3. Query Rewriting for Provenance Computation 4. Advanced Topics and New Stuff 5. Results 6. Conclusion 5

52 5. Experimental Results TPC-H benchmark (normal) normal provenance 52

53 Overview. Introduction 2. The Perm Provenance Representation 3. Query Rewriting for Provenance Computation 4. Advanced Topics and New Stuff 5. Results 6. Conclusion 53

54 6. Conclusion Not covered in this talk Complete set of rewrite rules Nested subqueries Other contribution semantics Optimizations Gory implementation details Theoretical foundation 54

55 6. Conclusion Benefits Compute provenance for SQL Full SQL query power for provenance data Lazy or eager computation Reuse existing database technology Supports external provenance 55

56 6. Conclusion Future work Physical operators for more efficient provenance computation Storage compression Include transformation provenance Support different contribution semantics Support various granularities Applications: Data Integration / Data Exchange View update... 56

57 Questions 57

58 4. Handling Sublinks Provenance for nested subqueries Important for typical provenance application domains Not addressed by other approaches 58

59 Sublinks 4. Handling Sublinks Subqueries in e.g. SELECT-clause " (R) a IN " (b=3) (S) Correlated Nested References outside attributes " (R) a IN " (b=a ) (S) Sublink that contains sublinks " (R) a IN " (b = ANY (T )) (S) 59

60 4. Handling Sublinks What is the provenance of a sublink according to Definition? Sublinks can be used in different contexts Selection Projection... Sublink is used in an expression -> role of sublink in expression influences provenance 60

61 Example: 4. Handling Sublinks q = " a =ANY #b (S)$a=3(R) R a 2 3 S b 2 4 c Result a 2 3 6

62 Example: 4. Handling Sublinks q = " a =ANY #b (S)$a=3(R) Compute provenance for t = () R a 2 3 S b 2 4 c Result a

63 Example: 4. Handling Sublinks q = " a =ANY #b (S)$a=3(R) Compute provenance for t = (3) R a 2 3 S b 2 4 c Result a

64 4. Handling Sublinks How to compute the provenance according to the extended definition? Use query rewrite Generic strategy (Gen) Specialized strategies 64

65 4. Handling Sublinks Gen-strategy Correlated sublinks are problematic For queries we cannot un-nest and decorrelate. Join original query with all possible provenance tuples (base relations) 2. Rewrite the sublink query 3. Introduce additional correlation to simulate a join between ) and 2) 65

66 Overview. Introduction 2. The Perm Provenance Representation 3. Query Rewriting for Provenance Computation 4. Handling Sublinks 5. Optimizations 6. Results 7. Conclusion 66

67 4. Handling Sublinks What is the provenance of a sublink according to Definition? Sublinks can be used in different contexts Selection Projection... Sublink either Produces exactly one value Or produces a boolean value 67

68 4. Handling Sublinks Single uncorrelated ANY-sublinks in selection conditions For other Types of sublinks Correlated sublinks Nested sublinks 68

69 4. Handling Sublinks Single uncorrelated ANY-sublinks in selection conditions The result of the sublink query is fixed For a given input tuple t the sublink condition is either true or false " a =ANY " (b=3) (S)(R) 69

70 4. Handling Sublinks Some terminology The query of a sublink T sub The conditional expression of a sublink C sub q = " a =ANY #b (S)(R) T sub C sub " b (S) a = ANY " b (S) 70

71 4. Handling Sublinks Sublink condition can play different roles in a condition C of a selection (for one input tuple t): Reqtrue: the selection condition is true, iff C sub is true Reqfalse: the selection condition is true, iff C sub is false Ind: the selection condition is true indepedent of the result of C sub 7

72 4. Handling Sublinks Some more terminology All tuples from the sublink query that fulfill the unquantified sublink condition T true (t) sub All tuples from the sublink query that do not fulfill the unquantified sublink condition T sub false (t) C sub = (a = ANY " b=3 (S)) C sub = (a = b) 72

73 4. Handling Sublinks Back to ANY-sublinks in selections Proposition: " T * (t) = T true (t) # sub sub $ T sub reqtrue reqfalse,ind 73

74 Example: 4. Handling Sublinks q = " (S)(R) a =ANY #b Compute provenance for t = () R a 2 3 S b 2 4 c Result a 2 74

75 4. Handling Sublinks q = " a =ANY #b (S)(R) T sub = " b (S) C sub is reqtrue C sub = (a = b) T * true = T sub sub T sub true (t) = {()} 75

76 4. Handling Sublinks q = " a =ANY #b (S)(R) Compute provenance for t = () R a 2 3 Tsub b 2 4 C sub = (a = b) T sub true (t) = {()} 76

77 4. Handling Sublinks q = " a =ANY #b (S)(R) Compute provenance for t = () R a 2 3 S b 2 4 c Tsub b 2 4 Result a 2 R * a * Tsub b 77

78 4. Handling Sublinks Definition is ambiguous for queries with more than one sublink! q = " C #C 2 (U) C = (a =ANY R) C 2 = (a > ALL S) R b 2 S c t = (5) 5 U a 5 Result a

79 4. Handling Sublinks Definition is ambiguous for queries with more than one sublink! q = " C #C 2 (U) C = (a =ANY R) C 2 = (a > ALL S) true false R b 2 S c t = (5) 5 U a 5 Result a

80 4. Handling Sublinks q = " C #C 2 (U) C = (a =ANY R) C 2 = (a > ALL S) t = (5) Solution Solution 2 R* b 5 S* c 5 U* a 5 R* b 00 S* b U* a 5 80

81 4. Handling Sublinks q = " C #C 2 (U) C = (a =ANY R) C 2 = (a > ALL S) true false t = (5) Solution Solution 2 R* b 5 S* c 5 U* a 5 R* b 00 S* b U* a 5 8

82 4. Handling Sublinks q = " C #C 2 (U) C = (a =ANY R) C 2 = (a > ALL S) false true t = (5) Solution Solution 2 R* b 5 S* c 5 U* a 5 R* b 00 S* b U* a 5 82

83 4. Handling Sublinks Reasons for this ambiguity: The definition requires the provenance to produce the same result But not to produce the same results for the sublinks -> Definition produces false positives 83

84 4. Handling Sublinks Solution: Extend definition Add a third condition: For each sublink: If computed for one result tuple t one tuple from the provenance of the sublink Produces same sublink result as in the original query 84

85 4. Handling Sublinks q = " C #C 2 (U) C = (a =ANY R) C 2 = (a > ALL S) R* b 5 S* c 5 t = (5) Solution Solution 2 U* a 5 R* b S* b U* a