CS Hot topics in database systems: Data Provenance

Transcription

1 CS Hot topics in database systems: Data Provenance Boris Glavic August 22, 2012

2 Outline 1 Instructor 2 Course Overview 3 Course Details and Administrative Information

3 Boris Glavic Assistant Professor for Database Systems (aka the new guy) Office: Stuart Building, room 226C Office hours: Thursday, 1:00 pm - 2:00 pm Webpage: Phone: Slide 1 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

4 Outline 1 Instructor 2 Course Overview Why should I care? 3 Course Details and Administrative Information

5 CS Data Provenance Administrative Info Hours: Mon + Wed 3:15-4:30 PM Room: Stuart Building in room 106 Course Webpage: Will be linked on soon! Slide 2 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

6 Data Provenance Data Provenance Information about the creation process and origin of data Slide 3 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

7 Why do we call it Provenance? Origin of the Term From art dealing Alternative Terms Lineage Data Pedigree Slide 4 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

8 Why do we call it Provenance? Origin of the Term From art dealing Alternative Terms Lineage for kings Data Pedigree Slide 4 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

9 Why do we call it Provenance? Origin of the Term From art dealing Alternative Terms Lineage for kings Data Pedigree for dogs Slide 4 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

10 Why do we call it Provenance? Origin of the Term From art dealing for pieces of art Alternative Terms Lineage for kings Data Pedigree for dogs Slide 4 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

11 me Course Overview Course Info Provenance in Art Given a piece of art How do we know... if it is authentic? who created it? if it has been altered? Example Jan Van Eyck - Arnolfini Portrait Slide 5 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

12 me Course Overview Course Info Provenance in Art Provenance French provenir, to come from Chronology of the ownership or location of an historical object Example Painting dated by van Eyck; presumably owned by the sitters. before In possession of Don Diego de Guevara (d. Brussels 1520), a Spanish career courtier of the Habsburgs (himself the subject of a fine portrait by Michael Sittow in the National Gallery of Art). He lived most of his life in the Netherlands, and may have known the Arnolfinis in their later years. By 1516 he had given the portrait to Margaret of Austria, Habsburg Regent of the Netherlands Painting is the first item in an inventory of Margaret s paintings, made in her presence at Mechelen. The item says (in French): a large picture which is called Hernoul le Fin with his wife in a chamber, which was given to Madame by Don Diego, whose arms are on the cover of the said picture; done by the painter Johannes. A note in the margin says It is necessary to put on a lock to close it: which Madame has ordered to be done In another Mechelen inventory, a similar description, this time the name of the subject is given as Arnoult Fin In 1530 the painting was inherited by Margaret s niece Mary of Hungary, who in 1556 went to live in Spain. It is clearly described in an inventory taken after her death in 1558, when it was inherited by Philip II of Spain. A painting of two of his young daughters commissioned by Philip clearly copies the pose of the figures (Prado).[1] a German visitor saw it in the Alcazar Palace in Madrid. Now it had verses from Ovid painted on the frame: See that you promise: what harm is there in promises? In promises anyone can be rich. It is very likely that Vela zquez knew the painting, which may have influenced his Las Meninas, which shows a room in the same palace In an inventory after the death of Carlos II it was still in the palace, with shutters and the verses from Ovid Now in the Palacio Nuevo in Madrid. Jan Van Eyck - Arnolfini Portrait Slide 5 of 26 Boris Glavic The painting is now in London, in the possession of Colonel James Hay, a Scottish soldier. He claimed that after being seriously wounded at the Battle of Waterloo the previous year, the painting hung in the room where he convalesced in Brussels. He fell in love with it, and persuaded the owner to sell. More relevant to the real facts is no doubt Hay s presence at the Battle of Vitoria (1813) in Spain, where a large coach loaded by King Joseph Bonaparte with easily portable artworks from the Spanish royal collections was first plundered by British troops, before what was left was recovered by their commanders and returned to the Spanish. Hay offered the painting to the Prince Regent, later George IV of England, via Sir Thomas Lawrence. The Prince had it on approval for two years at Carlton House before eventually returning it in In another Mechelen inventory, a similar description, this time the name of the subject is given as Arnoult Fin. In 1530 the painting was inherited by Margaret s niece Mary of Hungary, who in 1556 went to live in Spain. It is clearly described in an inventory taken after her death in 1558, when it was inherited by Philip II of Spain. A painting of two of his young daughters commissioned by Philip clearly copies the pose of the figures (Prado).[1] CS Hot topics in database systems: Data Provenance

13 Provenance in Data Processing Given a piece of data How do we know... which data it is derived from? which transformations (SQL) where used to create it? who created it?... Example result shop rev t 1 Migros 125 t 2 Coop 25 Slide 6 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

14 Provenance in Data Processing Given a piece of data How do we know... which data it is derived from? which transformations (SQL) where used to create it? who created it?... Example Compute the revenue for each shop as sum of prices of items sold Example result shop rev t 1 Migros 125 t 2 Coop 25 SELECT shop, sum ( price ) AS rev FROM sales, items WHERE itemid = id GROUP BY shop sales shop itemid s 1 Migros 1 s 2 Migros 3 s 3 Coop 3 items id price i i i Slide 6 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

15 Provenance in Data Processing Given a piece of data How do we know... which data it is derived from? which transformations (SQL) where used to create it? who created it?... Definition (Data Provenance) Information about the origin and creation process of data. Example result shop rev t 1 Migros 125 t 2 Coop 25 SELECT shop, sum ( price ) AS rev FROM sales, items WHERE itemid = id GROUP BY shop sales shop itemid s 1 Migros 1 s 2 Migros 3 s 3 Coop 3 items id price i i i Slide 6 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

16 A more Complex Example Scenario You are an analyst for a garden supply shop You have to compute the first quater revenue for each shop location Datawarehouse with sales data Use SQL to compute the required information from the warehouse Slide 7 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

17 A more Complex Example Example (Input Data) Employee SSN Name WorksFor 123 Peter Peterson New York 342 Jane Janeson New York 555 Heinz Heinzmann Wuppertal Shop Location Budget New York Wuppertal Item Id Description Price 1 Lawnmower Fertilizer 32 3 Rake 9 Sales Employee Item Amount Month Slide 8 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

18 A more Complex Example Example (SalesTotal Query) CREATE VIEW SalesTotal AS SELECT Location AS Shop, Month, SSN AS Employee, Price * Amount AS Totalprice FROM Employee E, Shop H, Item I, Sales S WHERE E. WorksFor = H. Location AND E. SSN = S. Employee AND I. Id = S. Item Example (Results) SalesTotal Shop Month Employee Totalprice New York New York New York Wuppertal Slide 8 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

19 A more Complex Example Example (MonthlyRevenue Query) CREATE VIEW MonthlyRevenue SELECT Shop, Month, sum ( Totalprice ) AS Revenue FROM SalesTotal GROUP BY Shop, Month Example (Results) MonthlyRevenue Shop Month Revenue New York New York 3 18 Wuppertal 5 9 Slide 8 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

20 A more Complex Example Example (RevenueFirstQ Query) CREATE VIEW RevenueFirstQ SELECT Shop, sum ( Revenue ) AS Revenue FROM MonthlyRevenue WHERE Month < 5 GROUP BY Shop Example (Results) RevenueFirstQ Shop Revenue New York 2265 Slide 8 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

21 A more Complex Example Compute First Quarter Revenue RevenueFirstQ MonthlyRevenue SalesTotal Employee Shop Item Sales Slide 8 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

22 Example Data Example Compute First Quarter Revenue RevenueFirstQ MonthlyRevenue SalesTotal Slide 9 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

23 Example Data Example Compute First Quarter Revenue RevenueFirstQ MonthlyRevenue SalesTotal Slide 9 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

24 Example Data Example SalesTotal EmployeeShop Name WorksFor SSN Location Budget 123 Peter Peterson New York New York Jane Janeson New York New York Heinz Heinzmann Wuppertal Wuppertal Employee SSN Name WorksFor 123 Peter Peterson New York 342 Jane Janeson New York 555 Heinz Heinzmann Wuppertal Slide 9 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

25 Tracing an Error Problem One result tuple of your query looks suspicious You expect the input data to be the culprit How to know which input data affected which output data Slide 10 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

26 Tracing an Error Problem One result tuple of your query looks suspicious You expect the input data to be the culprit How to know which input data affected which output data This is Data Provenance Slide 10 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

27 Tracing an Error Problem One result tuple of your query looks suspicious You expect the input data to be the culprit How to know which input data affected which output data This is Data Provenance But how to get at the data provenance? Slide 10 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

28 Tracing an Error Problem One result tuple of your query looks suspicious You expect the input data to be the culprit How to know which input data affected which output data This is Data Provenance But how to get at the data provenance? Manually? Not reasonable for big data or complex query! Slide 10 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

29 Tracing an Error Problem One result tuple of your query looks suspicious You expect the input data to be the culprit How to know which input data affected which output data This is Data Provenance But how to get at the data provenance? Manually? Not reasonable for big data or complex query! Need system that tracks it automatically! Slide 10 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

30 Why should I care? Use Cases Debugging (tracking the sources of errors) Propagating annotations Gain deeper understanding of data and transformations Estimate quality, trust Improvement of other data processing technologies Probabilistic databases Deletion propagation Testing Slide 11 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

31 Why should I care? Application Domains Complex database queries, e.g., datawarehousing E-science and curated databases Data integration/exchange Workflow systems Slide 12 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

32 Why should I care? Application Domains Complex database queries, e.g., datawarehousing E-science and curated databases Data integration/exchange Workflow systems Application domain with complex, multi-stage data processing Map-Reduce style processing and its frontends like Pig Simulations... Slide 12 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

33 Why should I care? Debugging Example SalesTotal Slide 13 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

34 Why should I care? Annotation Propagation Example EnzymeProduce Enzyme Gene EC ALB?? EC ALB?? CREATE VIEW EnzymeProduce AS SELECT Enzyme, Name AS Gene FROM Gene G, Enzyme E WHERE G.Id = E.ProducedBy Gene Id Name 4q11-q13 ALB {a 4} 18q21.3 BCL2 {} Enzyme Enzyme Weight ProducedBy EC q11-q13 {a 1,a 2} EC q11-q13 {a 2,a 3} a 1 Necessary for red blood cells a 2 Produced in liver a 3 Unhealthy a 4 Discovered by Edmond Hillary Slide 14 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

35 Outline 1 Instructor 2 Course Overview 3 Course Details and Administrative Information

36 Course Topics We will study... Several models of database provenance Approaches for automatically tracking provenance Query languages and storage mechanism for provenance Real systems that generate provenance data Outlook into other research areas that use provenance Slide 15 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

37 Prerequisites Some database background (mainly query languages) SQL Relational algebra Datalog Ideally you have taken one of the following courses: CS 425 CS 520 CS 525 Slide 16 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

38 Course Material No text book is required Research papers that are available online Slide 17 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

39 Course Organization Course will consist of... Lectures Research paper reviews (oral presentations done by students) A major project Implementation or extension of a real system Written report Oral presentation Slide 18 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

40 Paper Reviews Papers The list of papers will be on the course website soon Topics cover the content of the course and extend it Process Students will have to pick papers to review By end of august, first week of september Some of the classes will be used for oral presentations on the topics Each presentation will be followed by a discussion Short written reviews (4-6 pages) will be due by Nov 1st Slide 19 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

41 Course Project Implementation or extension of a provenance system Topics: Adding new provenance types to existing systems Add causality based provenance to Perm Add Why-provenance to Perm Extend the How-provenance implementation of Perm to derive new information Build a visualization tool for provenance in Perm... Your topic Slide 20 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

42 Course Project cont. Organisation Choose a project by Sep 15th Meetings to discuss the projects with me Project implementation Progress reports during semester Written report: Due by Nov 15th Oral presentations: Classes at the end of the semester Slide 21 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

43 Stuff to get familiar with Get practical experience with a database system PostgreSQL: open-source, quite standard compliant, Perm based on that Use Perm which is an provenance-enabled version of Postgres Query languages: SQL Relational Algebra Datalog Programming skills: C, C++, Java *nix operating system or Cygwin for Windows users Slide 22 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

44 Detailed Course Outline Introduction to Data Provenance What is Data Provenance? Why do we need it? Understanding different types of data provenance Database Provenance Provenance Models and Systems Why-provenance Where-provenance and the DBNotes system Lineage and the WHIPS prototype Witness-list semantics and Perm Provenance semirings and Orchestra Causality and Responsibility models Storage mechanisms Query languages Slide 23 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

45 Detailed Course Outline cont. Extensions of the Provenance concept Provenance for missing answers Provenance for past queries Provenance for updates Beyond Database Provenance Scientific workflows Provenance in the operating system context Connection with Dataflow analysis in programming languages Slide 24 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

46 Grading Policies Course Project (Implementation, Report, Presentation): 60% Paper reviews: written review (15%) and oral presentation (15%) Participation in the paper discussions: (10%) Slide 25 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance

47 Questions? Further Information Office hours: Thursday, 1:00 pm - 2:00 pm, room 226C Webpage: Course webpage will be linked there soon. Slide 26 of 26 Boris Glavic CS Hot topics in database systems: Data Provenance