Data Fusion. References

Transcription

1 Data Fusion Helena Galhardas DEI/IST References Data Fusion, Jens Bleiholder and Felix Naumann, ACM Computing Surveys, Vol. 41, N.1, Slides VLDB 2009 tutorial on Data Fusion, Luna Dong and Felix Naumann Slides Data Quality and Data Cleansing course, Felix Naumann, Winter 2014/15 2 1

2 Definition of Data Fusion Process of consolidating multiple records representing the same real-world object into a single, consistent, and clean representation Also known as: data merging, consolidation, entity resolution, finding representatives/ survivors, instance-level conflict resolution 3 Example: Data Fusion amazon.com H. Melville $3.98 " ID MAX length MIN CONCAT Herman Melville Moby Dick $5.99!! bn.com 4 2

3 Why data fusion is required? Data conflicts exist and occur due to poor data quality Errors exist when collecting or entering data Sources are not updated, etc Several types of data conflicts: representation conflicts, e.g. dollar vs. euro key equivalence conflicts, i.e. same real world objects with different identifiers attribute value conflicts, i.e. instances corresponding to same real world objects and sharing an equivalent key, differ on other attributes 5 Another Example EmpId Name Surname Salary A-ribute conflicts arpa78 John Smith 2000 smith@abc.it eugi 98 Edward Monroe 1500 monroe@abc. it ghjk09 Anthony Wite 1250 white@abc.it treg23 Marianne Collins 1150 collins@abc.it EmpId Name Surname Salary arpa78 John Smith 2600 smith@abc.it Key conflicts eugi 98 Edward Monroe 1500 monroe@abc.it ghjk09 Anthony White 1250 white@abc.it dref43 Marianne Collins 1150 collins@abc.it 6 3

4 Data Conflict Elimination Error correction Reference tables Cities, countries, products... Similarity measures Standardization Domain-knowledge (meta data) Conventions (country/region-specific spelling) Ontologies Thesauri, dictionaries for homonyms, synonyms,... Outlier detection and elimination And data fusion 7 Outline Ø Data fusion in the data integration process Completeness and conciseness Foundations of data fusion Conflict resolution strategies and functions Conflict resolution operators Existing data fusion systems 8 4

5 Data Integration Source A Source B <Titel> Federated Database Systems </Titel> <Autor> Amit Sheth </Autor> <Autor> James Larson </Autor> <publicahon> <Htle> Federated Database Systems for Managing Distributed, Heterogeneous, and Autonomous Databases </Htle> <author> Scheth & Larson </author> <year> 1990 </year> </publicahon> Matching Mapping Duplicate DetecHon Data Fusion 9 Data Integration Source A Source B <Titel> Federated Database Systems </Titel> <Autor> Amit Sheth </Autor> <Autor> James Larson </Autor> <publicahon> <Htle> Federated Database Systems for Managing Distributed, Heterogeneous, and Autonomous Databases </Htle> <author> Scheth & Larson </author> <year> 1990 </year> </publicahon> <Htle> </Htle> <author> </author> <author> </author> <year> </year> Matching IntegraHon Matching Mapping Duplicate DetecHon Data Fusion 10 5

6 Source A Source B Data Integration <Titel> Federated Database Systems </Titel> <Autor> Amit Sheth </Autor> <Autor> James Larson </Autor> <publicahon> <Htle> Federated Database Systems for Managing Distributed, Heterogeneous, and Autonomous Databases </Htle> <author> Scheth & Larson </author> <year> 1990 </year> </publicahon> TransformaHon queries or views <Htle> Federated Database Systems </Htle> <author> Amit Sheth </author> <author> James Larson </author> <Htle> Federated Database Systems for Managing Distributed, Heterogeneous, and Autonomous Databases </Htle> <author> Scheth & Larson </author> <year> 1990 </year> XQuery XQuery Matching Mapping Duplicate DetecHon Data Fusion 11 Data Integration Source A Source B <Titel> Federated Database Systems </Titel> <Autor> Amit Sheth </Autor> <Autor> James Larson </Autor> <publicahon> <Htle> Federated Database Systems for Managing Distributed, Heterogeneous, and Autonomous Databases </Htle> <author> Scheth & Larson </author> <year> 1990 </year> </publicahon> <Htle> Federated Database Systems </Htle> <author> Amit Sheth </author> <author> James Larson </author> <Htle> Federated Database Systems for Managing Distributed, Heterogeneous, and Autonomous Databases </Htle> <author> Scheth & Larson </author> <year> 1990 </year> Matching Mapping Duplicate DetecHon Data Fusion 12 6

7 Data Integration Source A Source B <Titel> Federated Database Systems </Titel> <Autor> Amit Sheth </Autor> <Autor> James Larson </Autor> <publicahon> <Htle> Federated Database Systems for Managing Distributed, Heterogeneous, and Autonomous Databases </Htle> <author> Scheth & Larson </author> <year> 1990 </year> </publicahon> <Htle> Federated Database Systems </Htle> <author> Amit Sheth </author> <author> James Larson </author> <Htle> Federated Database Systems for Managing Distributed, Heterogeneous, and Autonomous Databases </Htle> <author> Scheth & Larson </author> <year> 1990 </year> Matching Mapping Duplicate DetecHon Data Fusion 13 Data Integration Source A Source B <Htle> Federated Database Systems </Htle> <author> Amit Sheth </author> <author> James Larson </author> <Htle> Federated Database Systems for Managing Distributed, Heterogeneous, and Autonomous Databases </Htle> <author> Scheth & Larson </author> <year> 1990 </year> <Htle> Federated Database Systems for Managing Distributed, Heterogeneous, and Autonomous Databases </Htle> <author> Amit Sheth </author> <author> James Larson </author> <year> 1990 </year> Preserve lineage Matching Mapping Duplicate DetecHon Data Fusion 14 7

8 Goals of data integration To increase the completeness By adding more data sources (more objects, more attributes) so that no object is forgotten in the result the conciseness by removing redundant data, by fusing duplicate entries and merging common attributes into one no object is represented twice and the data presented to the user has no contradictions of data that is available to users and applications 15 Completeness Measures the amount of data in a dataset both in terms of the number of tuples (extensional, data level) and the number of attributes (intensional, schema level) Extensional completeness: number of unique object representations in a dataset in relation to the overall number of unique objects in the real world (in all the sources); measures the % of real-world objects covered by the dataset Intensional completeness: number of unique attributes in a dataset in relation to the overall number of unique attributes available; an increase is achieved by integrating sources with additional attributes 16 8

9 Example Extensional completeness = 4/4 = 1 in the result of combining the two sources (4 distinct IDs in the sources) Extensional completeness of ¾ in each of the sources Extensional conciseness = 4/4 =1 17 Conciseness Measures the uniqueness of object representations in a data set Extensional conciseness: number of unique objects in a dataset in relation to the overall number of object representations in the dataset Intensional conciseness: measures the number of unique attributes in a dataset in relation to the overall number of attributes 18 9

10 Data conflict A data conflict exists if, for the same realworld object, semantically equivalent attributes, from one or more sources, do not agree on their value Two kinds of data conflicts: uncertainties and contradictions 19 Uncertainty and Contradiction Uncertainty: conflict between a non-null value and one or more null values that are all used to describe the same property of an object. Easy case Contradiction Non-NULL value vs. (different) non-null value Uncertainty Contradic5on Uncertainty 20 10

11 Semantics of NULL unknown There is a value, but I do not know it. E.g.: Unknown date-of-birth not applicable There is no meaningful value. E.g.: Spouse for singles withheld There is a value, but we are not authorized to see it. E.g.: Private phone line 21 Example 22 11

12 Challenges of Data Fusion Problem Given a duplicate, create a single object representation while resolving conflicting data values. 23 The Field of Data Fusion Data Fusion Conflict types ResoluHon strategies Operators ResoluHon funchons Uncertainty ContradicHon Join-based Possible worlds SubsumpHon AggregaHon Union-based Consistent answers ComplementaHon Advanced funchons Ignorance Avoidance ResoluHon Instance-based Metadata-based Instance-based Metadata-based 24 12

13 Overview Data fusion in the integration process Completeness and conciseness Foundations of data fusion Ø Conflict resolution strategies and functions Conflict resolution operators Existing data fusion systems 25 Classification of Resolution Strategies Conflict-ignoring strategies: do not make a decision as to what to do with conflicting data Ex: presenting both x and k values for attribute B of OID 3 Conflict-avoiding strategies: acknowledge the existence of possible conflicts, but do not detect and resolve single existing conflicts. Handle conflicting data by applying a unique decision equally to all data, such as preferring data from a special source. Ex: only the null value or x when preferring objects from S over T Conflict-resolution strategies: regard all data and metadata before deciding. Deciding: choose a value from all the already present values Mediating: choose a value that does not necessarily exist among the conflicting values 26 13

14 Possible Conflict Resolution Strategies 27 Classification of Conflict Resolution Functions conflict resoluhon strategies conflict ignorance Escalate instance based Coalesce ChooseDepending Concat conflict avoidance metadata based Choose instance based conflict resoluhon metadata based deciding mediahng deciding mediahng MIN, MAX Random Vote AVG, SUM MostRecent CommonAncestor MostAbstract MostSpecific 28 14

15 Conflict Resolution Functions Func5on Descrip5on Examples Min, Max, Sum, Count, Avg Standard aggregahon NumChildren, Salary, Height Random Random choice Shoe size Longest, Shortest Longest/shortest value First_name Choose(source) Value from a parhcular source DoB (DMV), CEO (SEC) ChooseDepending(val, col) Value depends on value chosen in other column city & zip, & employer Vote Majority decision RaHng Coalesce First non-null value First_name Group, Concat Group or concatenate all values Book_reviews MostRecent Most recent (up-to-date) value Address MostAbstract, MostSpecific, CommonAncestor Use a taxonomy / ontology LocaHon Escalate Export conflichng values gender 29 Overview Data fusion in the integration process Foundations of data fusion Conflict resolution strategies and functions Ø Conflict resolution operators Existing data fusion systems 30 15

16 The Field of Data Fusion Data Fusion Conflict types ResoluHon strategies Operators ResoluHon funchons Uncertainty ContradicHon Join-based Possible worlds SubsumpHon AggregaHon Union-based Consistent answers ComplementaHon Advanced funchons Ignorance Avoidance ResoluHon Instance-based Metadata-based Instance-based Metadata-based 31 Properties of operators Value preservation An operator that does not loose any value from the objects in the sources nor creates nor duplicates values Bag-union: value preserving Set union: non value-preserving Related to completeness Uniqueness When attributes that contain unique values in the sources also contain unique values in the result Equality join: uniqueness-preserving Union: non uniqueness-preserving Related to conciseness The ideal fusion operator should preserve as many values and objects as possible while enforcing or preserving uniqueness and resolving conflicts 32 16

17 Fusing tuples Source 1(A,B,C) Source 2(A,B,D) a, b, - a, b, - a, b, -, - a, b, -, - a, b, -, - IdenHcal tuples a, b, c a, b, - a, b, c, - a, b, -, - a, b, c, - Subsumed tuples a, b, c a, b, d a, b, c, - a, b, -, d a, b, c, d ComplemenHng tuples a, b, c a, e, d a, b, c, - a, e, -, d a, f(b,e), c, d ConflicHng tuples 33 Relational Operators and extensions Identical tuples UNION, OUTER UNION Subsumed tuples (uncertainty) MINIMUM UNION Complementing tuples (uncertainty) COMPLEMENT UNION, MERGE Conflicting tuples (contradiction) Relational approaches: Match, Group, Fuse, Other approaches Possible worlds, probabilistic answers, consistent answers 34 17

18 Union: Elimination of exact duplicates Minimum Union: elimination of subsumed tuples Outer union Subsumption Minimum Union A B C a b c e f g m n o A B D a b + = e f h m p A tuple t1 subsumes a tuple t2, if it has same schema, has less NULL-values, and coincides in all non-nullvalues. A B C D a b c a b e f g e f h m n o m p R A B C D a b c e f g e f h m n o m p 35 Complement Union Proposal Elimination of complementing tuples Outer union Complementation No known SQL rewriting A B C a b c e f g m n o A B D a b + = e f h m p A tuple t1 complements a tuple t2, if it has same schema and coincides in all non-null-values. Includes duplicate removal and subsumphon A B C D a b c a b e f g e f h m n o m p R A B C D a b c e f g h m n o m p 36 18

19 Grouping and Aggregation Outer union then group by real-world ID Aggregate all other columns using conflict resolving aggregate function Efficient implementations Catches inter- and intra-source duplicates Restricted to built-in aggregate-functions MAX, MIN, AVG, VAR, STDDEV, SUM, COUNT WITH OU AS ( ( SELECT A, B, C, NULL AS D FROM U1 ) UNION (ALL) ( SELECT A, B, NULL AS C, D FROM U2 ) ), SELECT A, MAX(B), MIN(C), SUM(D) FROM OU GROUP BY A 37 FUSE BY SQL extensions to resolve uncertainties and contradictions [BN05,BBB+05] FUSE FROM implies OUTER UNION Removes subsumed and duplicate tuples by default FUSE BY declares real-world ID RESOLVE specifies conflict resolution function from catalog Default: COALESCE Implemented on top of relational DBMS XXL SELECT ID, RESOLVE(Title, Choose(IMDB)), RESOLVE(Year, Max), RESOLVE(Director), RESOLVE(Rating), RESOLVE(Genre, Concat) FUSE FROM IMDB, Filmdienst FUSE BY (ID) ON ORDER Year DESC 38 19

20 Union, Outer Union Summary of Operators Duplicates Subsumed tuples Complemen5ng tuples Contradic5ons ü û û û Minimum Union ü ü û û Full DisjuncHon ü ü ü (inter-source) Complement Union ü ü ü û Merge ü ü (intersource) MatchJoin + CTQM ü (inter-source) ü ü ü û ü Group By ü ü ü ü Fuse By ü ü ü ü û 39 FuSem Tool to query and fuse data from diverse data sources [BDN07] Based on HumMer project [BBB+05]. Explore data and find interesting subsets Execute, explore and compare five different data fusion semantics, specified in their respective syntax: SQL (and extensions, such as Subsumption) Merge MatchJoin FuseBy ConQuer 40 20

21 Overview Data fusion in the integration process Foundations of data fusion Conflict resolution strategies and functions Conflict resolution operators Ø Existing data fusion systems 41 Commercial Data Integration Tools Source: Gartner Typical ETL tools support rule-based fusion that allow to remove uncertainties but not data conflicts IIS (IBM Information Server) SSIS (Microsoft s SQL Server Integration Services) This functionality is typically named survivorship or consolidaton 42 21

22 Research Data Integration/Cleaning Systems System Conflict types Strategy class Func5on Specifica5on MulHbase Hc, data ResoluHon Choose, Avg, Min, Max, Sum, Manually, in query Hermes Hc, data ResoluHon MostRecent, Choose Manually, in mediator Fusionplex Hc, object, data ResoluHon MostRecent, Min, Max, Avg, Manually, in query HumMer Hc, object, data ResoluHon MostAbstract, Vote, Min, ChooseDepen Ajax Hc, object, data ResoluHon Various Manually, in query Manually, in workflow definihon TSIMMIS Hc, data Avoidance Choose Manually, rules in mediator SIMS/Ariadne Hc, data Avoidance Choose AutomaHcally Infomix Hc, data Avoidance onlyconsistentvalue AutomaHcally Hippo Hc, object, data Avoidance onlyconsistentvalue AutomaHcally ConQuer Hc, object, data Avoidance onlyconsistentvalue AutomaHcally Rainbow Hc, object, data Avoidance onlyconsistentvalue AutomaHcally Pegasus Hc, data Ignorance Escalate Manually Nimble Unknown Ignorance Escalate Manually Carnot Hc Ignorance Escalate AutomaHcally InfoSleuth Hc Ignorance Escalate Unknown Poler s Wheel 43 Hc Ignorance Escalate Manually, transformahon CLEENEX Next Lecture 22

23 Figure 4 a) High completeness but not concise Without schema mapping information and knowledge of object identifiers b) Intensionally concise Knowledge about common attributes given by schema mapping c) Extensionally concise Knowledge about common objects d) Intensionally and extensionally concise Knowledge about common attributes and common objects 45 23