ON SCHEMA DISCOVERY ICDM Renée J. Miller

Size: px

Start display at page:

Download "ON SCHEMA DISCOVERY ICDM Renée J. Miller"

Hilary Hoover
6 years ago
Views:

1 ON SCHEMA DISCOVERY ICDM 2011 Renée J. Miller

2 What are Schemas? 2 Schema From the Greek "σχήμα meaning shape, or more generally, plan Structure and constraints the data (should) satisfy Attribute structure Grouping into (nested) tables Constraints Keys, functional dependencies, rules Referential constraints (foreign keys) Domain constraints More general assertions, exclusion constraints, etc. Encode some (important) data semantics

3 Where are Schemas Used? 3 Enterprise Information Designed, curated, and valuable Used in decision making A single source may be massive Foundation for Business Intelligence/Analytics Web/Personal Information Light-weight structure, little/no curation or design Convenient human memory aid Often small, but numerous, data sources Spectrum of information

4 Evolving Role of Schemas 4 Old: Prescriptive Role Time-invariant portion of data Used to ensure data consistency New: Descriptive Role Evolve as data semantics evolves Used to describe, understand, query the data

5 Goals for Talk 5 Overview some of our work on schema discovery Robust and well-studied area for enterprise data Present some new challenges for modern information Unifying theme Leveraging data semantics

6 Outline 6 Motivation Structure & Constraint Discovery Dependency Discovery Constraint Repair Schema Alignment (Data Integration) Schema Mapping Discovery Ontology Alignment Discovery Data Alignment (time permitting) New Challenges posed by Linked Open Data Conclusions and Open Problems

7 Joint Work 7 Series of Papers (including but not exclusively) IEEE DE Bulletin 03 Schema Discovery SIGMOD 04 [Andritsos, Tsaparas, M-] ICDE 06 [Andristos, Fuxman, M-] SIGMOD 07 [Udrea, Getoor, M-] VLDB 08 [Chiang, M-] ICDE 11 [Chiang, M-] Clio [Fagin, Haas, Hernandez, M-, Popa, Velegrakis] Clio: VLDB 00/02 thru to 2009 book chapter Data Exchange: [Fagin, Kolaitis, M-, Popa ICDT03, TCS05] Fei Chiang graduating Spring 2012

8 What is a Good Schema? 8 Traditional Answer: minimize redundancy Redundancy can lead to inconsistency BCNF: all functional dependencies (FDs) are keys Every attribute shall depend on the key, the whole key, and nothing but the key.... so help me Codd Information theoretic characterization Given any set of cells (attribute values) in a table, should not be able to predict the value of another cell If the only constraints allowed are FDs, we can show that BCNF minimizes redundancy [Arenas 06 Dissertation, Kolahi 07 Dissertation on 3NF]

9 Finding Good Schemas? 9 Today, can no longer assume data was designed A table many contain more than one type of entity May be result of integration or information extraction May not have constraints enforced on it Violations of constraints May be errors May be due to evolving (unknown) data semantics Need to discover/maintain schema & constraints

10 Legacy Relations 10 OiD Type Category Start End SalesAssc Time 123 Voice Unlimited 1/20/10 1/20/11 Fred10 10:17: Data Unlimited 4/20/05 3/20/09 Pat11 23:15: Data Weekend 4/20/05 4/20/06 Pat11 01:01: Voice 120HR 2/20/08 2/20/10 Fred10 23:15: Charger Motorola 2/14/10 NULL CRM 10:17: Phone MotoRZ2 12/1/10 NULL CRM 11:15: Phone BlackBerry 12/1/10 NULL CRM 12:12:22 Tables & attributes may become overloaded Service orders and product orders

11 Information Theoretic Clustering 11 We use LIMBO, a scalable algorithm that clusters categorical data [Andritsos, M-, Tsaparas SIGMOD04] Idea: Compress tuples, T, into a clustering C so that the information preserved about the attribute values is maximum [Slonim, Tishby NIPS99] Naturally finds (groups) redundant values LIMBO builds compact summaries that represent the data

12 Finding good horizontal decompositions 12 OiD Type Category Start End SalesAssc Time 123 Voice Unlimited 1/20/10 1/20/11 Fred10 10:17: Data Unlimited 4/20/05 3/20/09 Pat11 23:15: Data Weekend 4/20/05 4/20/06 Pat11 01:01: Voice 120HR 2/20/08 2/20/10 Fred10 23:15: Charger Motorola 2/14/10 NULL CRM 10:17: Phone MotoRZ2 12/1/10 NULL CRM 11:15: Phone BlackBerry 12/1/10 NULL CRM 12:12:22 Merge tuples to lose as little information about attribute values as possible

13 Applications 13 Horizontally decompose legacy tables Group tuples into semantically meaningful types [Andritsos, M-, Tsaparas SIGMOD04] Find potential duplicates tuples in data Entity-identification in relational data [Andritsos, Fuxman, M- ICDE06] Create probabilistic DBs from duplicated data Compute meaningful probabilities that reflect duplication in data [Hassanzadeh, M- VLDB Journal09]

14 Constraint Discovery 14 Functional Dependencies (includes Keys) FDEP [Flach, Savnik AICommunications99], TANE [Huhtala et al. ComputingJ.99], FastFDs [Wyss, Giannella, Roberston DaWaK01] Inclusion Dependencies (includes Foreign Keys) General Inclusion Deps [Bauckmann et al. ICDE07, De Marchi, Lopes, Petit EDBT09] and many others Foreign Keys [Zhang et al. PVLDB10] Mining may give large number of dependencies not always intuitive not all are useful for re-designing the data may be accidental Goal: find interesting dependencies efficiently

15 Conditional Rules 15 Rules may not hold over entire table FDs assume table represents a single entity type Conditional FDs [Maher TCS97, Bohannon et al. ICDE07] Applications to cleaning and understanding data Airline Status Miles Seating Boarding One World Bronze 1x Standard Standard Meridan Silver 1x Elite Standard Skyway Silver 2x Preferred Standard Skyway Gold 2x Elite First One world Gold 3x Preferred Priority Skyway Gold 2x Preferred Priority Functional Dependency [Airline,Status] [Miles] Conditional Functional Dep [Status= Gold,Seating] [Boarding] One World Silver 2x Preferred Priority

16 Discovery of Conditional FDs 16 Modification of TANE search to consider conditioning conditions [Chiang, M- VLDB08] Standard measures of rule interest Support, Entropy, Conviction,... Find approximate conditional FDs and deviants (data that form candidates for cleaning)

17 Inconsistent Data 17 Data may become inconsistent with constraints Options Drop constraints and discover new constraints that fit Assume constraints are correct and repair data [Bohannon et al. SIGMOD05], [Cong et al. VLDB07], [Kolahi, Lakshmanan ICDT09] For FD: X Y, find minimal cost changes to the data (e.g., edit distance) to the Y values Repair data and constraints to fit [Chiang, M- ICDE11]

18 Data & Constraint Repair 18 District Region Municipal AC Street City Prov PCode Brook Granville Glendale 412 Roslin Toronto ON M4N 1Y3 Brook Granville Glendale 412 Roslin Toronto OH M4N 1Y3 Brook Granville Guildwood 553 Sidney Belleville ON K8P 3Y9 Brook Granville Guildwood 553 Sidney Belleville ON K8P 1J7 Fife Parkhill Moore 725 Poth Dundee ON NOB 2E0 Fife Parkhill Moore 725 Roseville Dundee ON NOB 2E0 Fife Parkhill Napa 228 Roslin Toronto ON M4N 1Y3 Municipal F1: [District, Region] [AC] 1) Repair the data or the constraints? 2) How to find the repairs? F2: [PCode] [City, Prov]

19 Key Ideas 19 Use variance of information to select attributes for doing constraint repair Attribute with no variance of information with respect to the inconsistent data is a perfect repair Unified cost model for comparing Constraint Repair with Data Repair

20 Summary Constraint Discovery 20 Goal is to discover or repair constraints to find accurate model of data Can be used to query and understand data Maintain data consistency Clean data or correct data entry errors Semantic query optimization

21 Outline 21 Motivation Constraint Discovery Dependency Discovery Constraint Repair Schema Alignment (Data Integration) Schema Mapping Discovery Ontology Alignment Discovery Data Alignment New Challenges posed by Linked Open Data Conclusions and Open Problems

22 Discovery of Schema Mappings 22 Q S Schema S I S Μ + Q T Μ Μ + I S Q T Schema T I T Schema Mappings are declarative specifications that describe the relationship between a source schema S and a target schema T Key to both Data exchange (creating IT) aka Data Warehousing Query rewriting (creating QS) aka Data Federation

23 Mapping example 23 <a, b, c> <d, e, f> <g, h, i> P Q R source A B C D E F G H I Referential Constraint P(a,b,c) Y Z T(a,Y,Z) Q(d,e,f) X U T(X,e,U) target T A E I R(g,h,i) V W T(V,W,i) a e i a Y 0 Z 0 X 0 e U 0 V 0 W 0 i a e Z 1 V 1 W 1 i There may be many solutions for T (J, J 1, J 2, etc.) However, J seems to be more general J 1 J J 2 X 0, Y 0, Z 0 represent unknown values (or nulls ) Intuitively, J 1 and J 2 have extra information

24 Mapping example 24 Emp <Pat, b, c> Worksin <d, 100K, f> Name B C D Salary F Employee A E I Pat 100K.10 Pat Y 0 Z 0 X 0 100K U 0 V 0 W 0.10 J 1 J h 1 = {Y 0 -> 100K, Z 0 ->.10 } Dept <g, h,.10> G H Bonus Pat 100K Z 1 V 1 W 1.10 J 2 h 2 J 1 and J 2 assert extra information: Pat s salary is 100K this is not required by source or mapping Homomorphisms: h: J J 1 constants h(c) = c J tuples (X,Y,Z), J1 tuple (h(x),h(y),h(z))

25 Universal Solutions 25 Given a data exchange setting (S, T, M, Σ t ) where Σ t are constraints on the target T, and a source instance I, a universal solution is a target instance J: J is a solution for I solutions J for I, there is a homomorphism h: J J For the example, J is a universal solution there are homomorphisms h1: J J1 and h2: J J2 there are no homomorphisms from J1 or J2 J

26 Universal Solutions in Data Exchange 26 We introduced the notion of universal solutions as the best solutions in data exchange The most general solutions Foundational Results [Fagin, Kolaitis, M-, Popa, ICDT03, TCS05] Universal solutions are unique up to homomorphic equivalence; they represent the entire solution space The chase procedure produces a universal solution in polynomial time The certain answers of target conjunctive queries can be obtained by evaluation on an arbitrary universal solution; & universal solutions are only solutions with this property

27 Mapping example 27 <a, b, c> <b, e, f> <f, h, i> P Q R A B C D E F G H I Source constraints can be used to improve mapping d,e,f Q(d,e,f) a,c P(a, d,c) d,e,f Q(d,e,f) h, i R(f,h,i) M P(a,b,c) Y Z T(a,Y,Z) Q(d,e,f) X U T(X,e,U) T A E I R(g,h,i) V W T(V,W,i) J is universal solution for M a Y 0 Z 0 X 0 e U 0 V 0 W 0 i M2: P(a,b,c),Q(b,e,f),R(f,h,i) T(a,e,i) J a e i J1 J1 is universal solution for M2

28 Mapping example 28 <Pat, E1, NY> Emp Worksin <E1, 100K, D1> Dept <D1, CS,.10> Name Eid Addr Eid Salary Did Did Dname Bonus Employee Name Salary Bonus Pat 100K.10 If meaning of Employee table coincides with WorksIn relationship then mapping should be: Emp(Name,Eid,Addr),Worksin(Eid,Salary,Did),Dept (Did,Dname,Bonus) Employee(Name, Salary,Bonus) J 1 is best (universal) solution J 1

29 Schema Mapping Discovery 29 Use chase (logical inference) to infer connections in source and target schemas Potential semantic relationships Guide user in selecting correct semantic relationships to Text use in mapping Transformation code generation SQL, XQuery, XSLT transforms,... Include (skolem) terms to generate labeled nulls Technology in several IBM product lines including IBM s Infosphere Data Architect [M-, Haas, Hernandez VLDB00] thru to [Fagin+2009]

30 Schema Mapping Summary 30 Schema mapping leverages Statistical inference to infer attribute matches Logical inference to give matches a semantic interpretation As inter-schema constraints Can we extend this idea to a closer integration of approaches?

31 Ontology Mapping 31 (discoveredby, owl:inverseof, discoverer); (discoveredby, owl:type, owl:functionalproperty) (discoveredby, owl:inverseof, discoverer); (associatedwith, owl:type, owl:transitiveproperty) (resultsf rom, rdfs:subpropertyof, associatedwith)

32 Example OWL Lite Ontologies 32 An entity can be a: Class (discoveredby, owl:inverseof, discoverer); (discoveredby, owl:type, owl:functionalproperty) (discoveredby, owl:inverseof, discoverer); (associatedwith, owl:type, owl:transitiveproperty) (resultsf rom, rdfs:subpropertyof, associatedwith)

33 Example OWL Lite Ontologies 33 An entity can be a: Class Instance (discoveredby, owl:inverseof, discoverer); (discoveredby, owl:type, owl:functionalproperty) (discoveredby, owl:inverseof, discoverer); (associatedwith, owl:type, owl:transitiveproperty) (resultsf rom, rdfs:subpropertyof, associatedwith)

34 Example OWL Lite Ontologies 34 An entity can be a: Class Instance Property (discoveredby, owl:inverseof, discoverer); (discoveredby, owl:type, owl:functionalproperty) (discoveredby, owl:inverseof, discoverer); (associatedwith, owl:type, owl:transitiveproperty) (resultsf rom, rdfs:subpropertyof, associatedwith)

35 Example OWL Lite Ontologies 35 (discoveredby, owl:inverseof, discoverer) (discoveredby, owl:type, owl:functionalproperty) (associatedwith, owl:type, owl:transitiveproperty) (resultsfrom, rdfs:subpropertyof, associatedwith) Axioms

36 Computing Similarity 36 sim lexical : Jaro-Winkler and Wordnet sim structural : Jaccard for neighborhoods sim extensional : Jaccard on extensions Standard used in schema/ontology matchers [COMA++ & others] parameters: λ x, λ s, λ e different for classes, instances and properties

37 37 Similarity

38 38 Does Similarity agree with Semantics

39 39 Performing logical inference

40 Performing logical inference 40 Candidate Consequence (TheodorEscherich, owl:sameas, T.S. Escherich) is a logical consequence of the candidate (E-ColiPoisoning, owl:sameas, E-Coli)

41 Combining Evidence 41 If logical consequence(s) are similar, then increase similarity of candidate by inference similarity Candidate: pair of entities (ci, cj) asserted to be same (E-coli Poisoning same-as E-coli) similarity 0.5 Consequence: pair of entities (ei, ej) inferred to be same (Theodore same-as T. S.) similarity 0.6 Inference Similarity Greater than one if consequences are similar Less than one otherwise Multiply candidate similarity by inference similarity (E-coli Poisoning same-as E-coli) 0.5*1.5 =.75

42 Experimental Framework 42 ILIADS-tailored uses best set of parameters for each pair of ontologies ILIADS-fixed uses one set of parameters for all pairs of ontologies FCA-merge [Stumme and Maedche, IJCAI 2001] uses formal concept analysis and an external document corpus COMA++ [Aumueller et al., SIGMOD 2005] implements multiple match strategies, including robust collection of similarity functions 30 pairs of real-world ontologies (up to 20,000 triples) From a variety of domains: medical, geographical, economical, biological Ground truth provided by human reviewers

43 Precision/recall for ontologies with substantial instance data 43

44 Ontology Mapping Summary 44 Use logical constraints in schemas/ontologies Able to improve recall substantially over standard statistical inference techniques that are based on lexical, structural, semantic similarity Introduced notion of inference similarity [Udrea, Getoor, M- SIGMOD07]

45 Outline 45 Motivation Constraint Discovery Dependency Discovery Constraint Repair Schema Alignment (Data Integration) Schema Mapping Discovery Ontology Alignment Discovery Data Alignment New Challenges posed by Linked Open Data Conclusions and Open Problems

46 Vision: reconciling data 46 Linked Open Data Publish Web Data with useful semantic information LinQuer Scalable, declarative (native DBMS) support for syntactic and semantic data matching So far leveraging domain constraints (and semantics of domains), but can we do more? Applications to Linked Clinical Trials Work of Oktie Hassanzadeh [Hassanzadeh et al. CIKM09] starting soon at IBM Watson

47 Conclusions 47 Schema Discovery Need to be able to discover and maintain structural and semantic information Role in prescribing and enforcing data consistency Role in cleaning, querying, and understanding data More flexible view of schemas Support what-if style analysis Postulate constraints that match your model of a domain DBMS gives answers that are consistent with constraints If I believe that Customers are uniquely identified by their phone number, zip code, and last name, then how many customers do I have? [Fuxman 2008 ACM SIGMOD Jim Grey Dissertation Award] [Fuxman, Fazli, M- SIGMOD05]

Leveraging Data and Structure in Ontology Integration

Leveraging Data and Structure in Ontology Integration O. Udrea L. Getoor R.J. Miller Group 15 Enrico Savioli Andrea Reale Andrea Sorbini DEIS University of Bologna Searching Information in Large Spaces