C. Batini & M. Scannapieco Data and Information Quality Book Figures. Chapter 10: Data Quality Issues in Data Integration Systems

Transcription

1 C. Batini & M. Scannapieco Data and Information Quality Book Figures Chapter 10: Data Quality Issues in Data Integration Systems 1

2 Data Fusion Strategies conflict handling strategies conflict ignorance conflict avoidance conflict resolution instance based metadata based instance based metadata based deciding mediating deciding mediating 2

3 Mediator-wrapper architecture USER MEDIATOR WRAPPER WRAPPER WRAPPER DB1 DB2 DB3 3

4 Phases of the QP-alg approach Input User query Sources with QCAs IQ Scores Source Selection Best Sources Plan Creation All Correct Plans Output Best Plans Plan Selection 1. QCA Quality Evaluation 2. Plan Quality Evaluation 3. Plan Ranking 4

5 Example of Price computation for the Plan P i Price QS1+QS2+QS3 Price QS1+QS2 Price QS3 Price QS1 Price QS2 Plan P i 5

6 R 1 t1,1 t1,2 t1,3 t1,n1 Query result construction In DaQuinCIS t 1 1=t1,1={ <f1,1, q1,1> <f1,w, q1,w>} t 1 2=t2,n2 ={<f2,1, q2,1> <f2,w, q2,w>} t 1 3= tk,2 = {<f3,1, q3,1> <f3,w, q3,w>}... C 1 R t 1 R 2 t2,1 t2,2 t2,3 t2,n2 C 2 t 2 1=t1,2= {<f1,1, q1,1> <f1,w, q1,w>} t 2 2= t2,2= {<f2,1, q2,1> <f2,w, q2,w>} t 2 3=tk,nk = {<f3,1, q3,1> <f3,w, q3,w>}... t 2 R k tk,1 tk,2 tz 1=t1,n 1 ={<f1,1, q1,1> <f1,w, q1,w>} t C z z 2=t2,3 ={<f2,1, q2,1> <f2,w, q2,w>} t z 3= tk,1= {<f3,1, q3,1> <f3,w, q3,w>}... t z tk,3 tk,nk 6

7 Example of query fragment construction from contributing views Q Q C 1 C 2 QF 1 QF 2 null 7

8 Comparison of quality-driven processing techniques Techniques Quality Metadata Granularity of Quality Characterization QP-alg YES Source, Query Correspondences Assertions, User Queries DaQuinCIS Query Processing FusionPlex Query Processing YES Each data element of a semistructured data model Type of Mapping GLAV GAV Quality Algebra Support Preliminary YES Source GLAV Preliminary No 8

9 An example of key- and attribute-level conflicts EmployeeID Name Surname Salary arpa78 John Smith 2000 eugi98 Edward Monroe 1500 ghjk09 Anthony Wite 1250 treg23 Marianne Collins 1150 Attribute Conflicts EmployeeS1 Key Conflicts EmployeeID Name Surname Salary arpa78 John Smith 2600 eugi98 Edward Monroe 1500 ghjk09 Anthony White 1250 dref43 Mariann e Collins 1150 collins@abc.it EmployeeS2 9

10 Resolution functions as proposed in [462] Function Attribute Type Description COUNT any Counts number of conflicting values MIN any Minimum value MAX any Maximum value RANDOM any Random non null value CHOOSE(Source) any Chooses most reliable source for the particular attribute MAXIQ any Value of highest information quality GROUP any Groups all conflicting values SUM numerical Sums all values MEDIAN numerical Median value, namely having the same number of higher and lower values AVG numerical Arithmetic mean of all values VAR numerical Variance of values STDDEV numerical Standard Deviation of values SHORTEST non-numerical Minimum length value, ignoring spaces LONGEST non-numerical Maximum length value, ignoring spaces CONCAT non-numerical Concatenation of values ANNCONCAT non-numerical Annotated concatenation of values, whose purpose is to specify the source, before the actual returned value 10

11 Instance of the global relation Employee TupleID EmployeeID Name Surname Salary t 1 arpa78 John Smith 2000 smith@abc.it t 2 eugi98 Edward Monroe 1500 monroe@abc.it t 3 ghjk09 Anthony Wite 1250 white@abc.it t 4 treg23 Marianne Collins 1150 collins@abc.it t 5 arpa78 John Smith 2600 smith@abc.it t 6 eugi98 Edward Monroe 1500 monroe@abc.it t 7 ghjk09 Anthony White 1250 white@abc.it t 8 dref43 Marianne Collins 1150 collins@abc.it 11

12 Resolution of attribute conflicts TupleID EmployeeID Name Surname Salary t 1 arpa78 John Smith 2000 smith@abc.it t 2 eugi98 Edward Monroe 1500 monroe@abc.it t 3 ghjk09 Anthony White 1250 white@abc.it t 4 treg23 Marianne Collins 1150 collins@abc.it RAC(Employee,Salary(MIN), Surname(Longest), EmployeeID(Any)) 12

13 Resolution of tuple conflicts TupleID EmployeeID Name Surname Salary t 1 arpa78 John Smith 2600 smith@abc.it t 2 eugi98 Edward Monroe 1500 monroe@abc.it t 3 ghjk09 Anthony Wite 1250 white@abc.it t 4 dref43 Marianne Collins 1150 collins@abc.it RTC(Employee,ANY) 13

14 Result of the context-aware query as applied to the table EmployeeS1 EmployeeID Salary arpa EmployeeID Salary eugi EmployeeID Salary ghjk treg

15 Conflict resolution techniques Techniques Tolerance Strategies Query Model SQL-Based Conflict Resolution Aurora FusionPlex NO High Confidence, RandomEvidence, PossibleAtAll No resolution strategy, selective attribute resolution SQL Ad-hoc Conflict Tolerant Query Model Extended SQL DaQuinCIS NO Extended XML FraQL-based Conflict Resolution OO RA NO Thresholds for tolerable and intolerable conflicts Ad-hoc FraQL Ad hoc Object Oriented Extension (OO RA ) 15