Web Science. Introduction to Information Integration

Transcription

1 Web Science Introduction to Information Integration Julien Gaugaz, October 26, 2010

2 Topics 2

3 Topics 1. Information Integration 2

4 Topics 1. Information Integration 2. Web Information Retrieval 2

5 Topics 1. Information Integration 2. Web Information Retrieval 3. Entity Search 2

6 Topics 1. Information Integration 2. Web Information Retrieval 3. Entity Search 4. Web Usage 2

7 Topics 1. Information Integration 2. Web Information Retrieval 3. Entity Search 4. Web Usage 5. Collaborative Web 2

8 Topics 1. Information Integration 2. Web Information Retrieval 3. Entity Search 4. Web Usage 5. Collaborative Web 6. Web Archiving 2

9 Topics 1. Information Integration 2. Web Information Retrieval 3. Entity Search 4. Web Usage 5. Collaborative Web 6. Web Archiving 7. Medical Social Web 2

10 Why Integrating Information? Scenarios

11 Company Mergers 4

12 Company Mergers 4

13 Company Mergers 4

14 Company Mergers 4

15 Travelling Agent Agent 5

16 Booking Flights Agent 6

17 Leveraging Wikipedia Infoboxes Query 7 Data Contribution

18 Evolution 1E+06 Number of Sources 1E+05 1E+04 1E+03 1E+02 1E+01 1E Beginning of Databases Rise of Internet & Wrapping Websites Wikipedia & Social Web 8

19 What is the Problem? Kinds of discrepancies

20 Wikipedia Infoboxes [[(...) Reg. Bürgermeister]]: [[Klaus Wowereit]] [[Höhe]] : m ü. NN [[Einwohner]] : {{Metadaten Einwohnerzahl DE-BE Berlin}}[...] (rendered as: (31. Mai 2010)) leader_title = [[List of mayors of Berlin Governing Mayor]] leader = Klaus Wowereit elevation = pop_date = population = pop_metro =

21 Wikipedia Infoboxes [[(...) Reg. Bürgermeister]]: [[Klaus Wowereit]] [[Höhe]] : m ü. NN [[Einwohner]] : {{Metadaten Einwohnerzahl DE-BE Berlin}}[...] (rendered as: (31. Mai 2010)) leader_title = [[List of mayors of Berlin Governing Mayor]] leader = Klaus Wowereit elevation = pop_date = population = pop_metro =

22 Wikipedia Infoboxes [[(...) Reg. Bürgermeister]]: [[Klaus Wowereit]] [[Höhe]] : m ü. NN [[Einwohner]] : {{Metadaten Einwohnerzahl DE-BE Berlin}}[...] (rendered as: (31. Mai 2010)) leader_title = [[List of mayors of Berlin Governing Mayor]] leader = Klaus Wowereit elevation = pop_date = population = pop_metro =

23 Wikipedia Infoboxes [[(...) Reg. Bürgermeister]]: [[Klaus Wowereit]] [[Höhe]] : m ü. NN [[Einwohner]] : {{Metadaten Einwohnerzahl DE-BE Berlin}}[...] (rendered as: (31. Mai 2010)) leader_title = [[List of mayors of Berlin Governing Mayor]] leader = Klaus Wowereit elevation = pop_date = population = pop_metro =

24 Wikipedia Infoboxes [[(...) Reg. Bürgermeister]]: [[Klaus Wowereit]] [[Höhe]] : m ü. NN [[Einwohner]] : {{Metadaten Einwohnerzahl DE-BE Berlin}}[...] (rendered as: (31. Mai 2010)) leader_title = [[List of mayors of Berlin Governing Mayor]] leader = Klaus Wowereit elevation = pop_date = population = pop_metro =

25 Wikipedia Infoboxes [[(...) Reg. Bürgermeister]]: [[Klaus Wowereit]] [[Höhe]] : m ü. NN [[Einwohner]] : {{Metadaten Einwohnerzahl DE-BE Berlin}}[...] (rendered as: (31. Mai 2010)) leader_title = [[List of mayors of Berlin Governing Mayor]] leader = Klaus Wowereit elevation = pop_date = population = pop_metro =

26 Wikipedia Infoboxes [[(...) Reg. Bürgermeister]]: [[Klaus Wowereit]] [[Höhe]] : m ü. NN [[Einwohner]] : {{Metadaten Einwohnerzahl DE-BE Berlin}}[...] (rendered as: (31. Mai 2010)) leader_title = [[List of mayors of Berlin Governing Mayor]] leader = Klaus Wowereit elevation = pop_date = population = pop_metro =

27 Wikipedia Infoboxes leader_title!! = [[List of mayors of Berlin Governing Mayor]] leader!!! = Klaus Wowereit elevation! = pop_date! = population! = pop_metro! = leader_title! = [[Mayor of San Francisco Mayor]] leader_name! = [[Gavin Newsom]] ([[Democratic [...] D]]) elevation_ft!! = 52 elevation_max_ft!= 925 elevation_min_ft! = 0 population_as_of = 2008 population_total = population_metro = population_urban =

28 Wikipedia Infoboxes leader_title!! = [[List of mayors of Berlin Governing Mayor]] leader!!! = Klaus Wowereit elevation! = pop_date! = population! = pop_metro! = leader_title! = [[Mayor of San Francisco Mayor]] leader_name! = [[Gavin Newsom]] ([[Democratic [...] D]]) elevation_ft!! = 52 elevation_max_ft!= 925 elevation_min_ft! = 0 population_as_of = 2008 population_total = population_metro = population_urban =

29 Wikipedia Infoboxes leader_title!! = [[List of mayors of Berlin Governing Mayor]] leader!!! = Klaus Wowereit elevation! = pop_date! = population! = pop_metro! = leader_title! = [[Mayor of San Francisco Mayor]] leader_name! = [[Gavin Newsom]] ([[Democratic [...] D]]) elevation_ft!! = 52 elevation_max_ft!= 925 elevation_min_ft! = 0 population_as_of = 2008 population_total = population_metro = population_urban =

30 Causes of Discrepancies 12

31 Causes of Discrepancies Information sources are diverse Different cultural background Different domain of activity Different model of information 12

32 Causes of Discrepancies Information sources are diverse Different cultural background Different domain of activity Different model of information Typos and other kinds of errors 12

33 Causes of Discrepancies Information sources are diverse Different cultural background Different domain of activity Different model of information Typos and other kinds of errors Evolution over time Use, usage and users of one source may change of over time 12

34 Places of Discrepancies 13

35 Places of Discrepancies Information level where discrepancies appear: 13

36 Places of Discrepancies Information level where discrepancies appear: Semantic: meaning, sense 13

37 Places of Discrepancies Information level where discrepancies appear: Semantic: meaning, sense Representational Lexical: word / term representing the meaning Structural: how are the terms arranged to represent the meaning 13

38 Places of Discrepancies Information level where discrepancies appear: Semantic: meaning, sense Representational Lexical: word / term representing the meaning Structural: how are the terms arranged to represent the meaning Syntactic: how is the lexical and structural encoded into characters (and bits) 13

39 Places of Discrepancies Information level where discrepancies appear: Semantic: meaning, sense Representational Lexical: word / term representing the meaning Structural: how are the terms arranged to represent the meaning Syntactic: how is the lexical and structural encoded into characters (and bits) Discrepancies may concern: Schema elements (properties and structure) and values 13

40 Schema Discrepancies Semantic Einstein s full name is Albert Einstein Representational Syntactic 14

41 Schema Discrepancies Semantic Einstein s full name is Albert Einstein Einstein name first Albert last Einstein Representational Syntactic 14

42 Schema Discrepancies Semantic Einstein s full name is Albert Einstein Einstein name last first Albert Einstein Einstein full_name Albert Einstein Representational Syntactic 14

43 Schema Discrepancies Semantic Einstein s full name is Albert Einstein Einstein name last first Albert Einstein Einstein full_name Albert Einstein Representational <Einstein> <full_name> Albert Einstein. Syntactic 14

44 Schema Discrepancies Semantic Einstein s full name is Albert Einstein Einstein name last first Albert Einstein Einstein full_name Albert Einstein Representational <Einstein> <full_name> Albert Einstein. Syntactic <Einstein> <full_name>albert Einstein</full_name> </Einstein> 14

45 Schema Ambiguity Semantic Representational xyz title The Theory of Relativity xyz title Prof. Dr. techn. 15

46 Schema Ambiguity Person title Semantic Representational xyz title The Theory of Relativity xyz title Prof. Dr. techn. 15

47 Schema Ambiguity Person title Article title Semantic Representational xyz title The Theory of Relativity xyz title Prof. Dr. techn. 15

48 Value Discrepancies Einstein s full name is Albert Einstein Semantic Representational Einstein full_name Albert Einstein Albert Einstin A. Einstein Einstein, Albert 16

49 Syntactic Level Where discrepancies are addressed with standards

50 Encoding Bytes 18

51 Encoding Bytes Basic unit Universal standard: Bit (binary digit) Ternary digit (base 3, USSR 50 s, out of use) 18

52 Encoding Bytes Basic unit Universal standard: Bit (binary digit) Ternary digit (base 3, USSR 50 s, out of use) Bits into bytes Big or small endian System wise convention, easily convertible, defined in communication protocols 18

53 Encoding Characters 19

54 Encoding Characters De facto standards: UTF-8/16 19

55 Encoding Characters De facto standards: UTF-8/16 Many others exist: ASCII, ISO-8859 s, KOI-8,... 19

56 Encoding Characters De facto standards: UTF-8/16 Many others exist: ASCII, ISO-8859 s, KOI-8,... Trivial dictionary-based translation When the corresponding code exists in the target character map... 19

57 Encoding Lexico-Structural 20

58 Encoding Lexico-Structural XML, XML Schema Structured document serialization format Base for: (X)HTML SVG: Scalable Vector Graphics DOCX: Microsoft Office Word

59 RDF Resource Description Framework Encoding information

60 source: 22

61 source: <subject> <property> <object> 22

62 source: <subject> <property> <object> <subject> URI or blank node 22

63 source: <subject> <property> <object> <property> URI <subject> URI or blank node 22

64 source: <subject> <property> <object> <subject> URI or blank node 22 <property> URI <object> URI or blank node or (typed) literal

65 URI 23

66 URI URI: Universal Resource Identifiers URL s are URI s scheme:scheme-specific-part RDF encourage using URL s 23

67 URI URI: Universal Resource Identifiers URL s are URI s scheme:scheme-specific-part RDF encourage using URL s URL scheme://usr:passwd@domain:port/path? query_string#anchor 23

68 RDF 24

69 RDF Resource Description Framework 24

70 RDF Resource Description Framework Data model specialized in conceptual information modeling 24

71 RDF Resource Description Framework Data model specialized in conceptual information modeling Supported by various serialization formats: XML Notation3 (N3) Turtle... 24

72 RDF Schema (RDF/S) 25

73 RDF Schema (RDF/S) Expressed in RDF 25

74 RDF Schema (RDF/S) Expressed in RDF Types subjects and objects with classes Class hierarchy (with multiple inheritance) Type of properties of a class 25

75 RDF Schema (RDF/S) Expressed in RDF Types subjects and objects with classes Class hierarchy (with multiple inheritance) Type of properties of a class Types properties Domain: type of property s subject Range: type of property s object 25

76 RDF Schema (RDF/S) Expressed in RDF Types subjects and objects with classes Class hierarchy (with multiple inheritance) Type of properties of a class Types properties Domain: type of property s subject Range: type of property s object OWL2 is more expressive: cardinality, etc... 25

77 When to use RDF? 26

78 When to use RDF? RDF is good at Modeling information Especially when schema is unknown or changing When there is multiple schemas 26

79 When to use RDF? RDF is good at Modeling information Especially when schema is unknown or changing When there is multiple schemas RDF is not for Representing documents (XHTML, CSS) Internal data management when schema is known and fixed (Relational Databases) 26

80 Schema Matching Discrepancies between the representational and semantic levels in the schema

81 Boxer Taxpayer name boxer id weight birthdate total fights residence Trainer first name last name age address street city tax id Company Tax Office... 28

82 Boxer Taxpayer name boxer id weight birthdate total fights residence Trainer... Input: Schemas to match... first name last name age address street city tax id Company Tax Office... Possibly data instantiating those schemas 28

83 Boxer Taxpayer name boxer id weight birthdate total fights residence Trainer... Input: Schemas to match first name last name age address street city tax id Company... Tax Office... Possibly data instantiating those schemas Output: Mappings between schema elements Possibly with confidence values and alternatives Possibly with value conversion rules (matchings) 28

84 Mappings or Matching? 29

85 Mappings or Matching? Schema mapping identifies correspondences between schema elements 29

86 Mappings or Matching? Schema mapping identifies correspondences between schema elements Schema matching actually transforms an instance of one schema into an instance of another schema 29

87 How to Use Mappings? General architectures

88 Mediated Schemas Schema2 Schema1 Schema3 31

89 Mediated Schemas Mediated Schema Schema2 Schema1 Schema3 31

90 Mediated Schemas Query Mediated Schema Schema2 Schema1 Schema3 31

91 Mediated Schemas Query Mediated Schema Schema2 Schema1 Schema3 31

92 Mediated Schemas Query Mediated Schema Schema2 Schema1 Schema3 Schema2 Schema1 Schema3 31

93 Mediated Schemas Query Mediated Schema Mediated Schema Schema2 Schema1 Schema3 Schema2 Schema1 Schema3 31

94 Mediated Schemas Query Mediated Schema Mediated Schema Schema2 Schema1 Schema3 Schema2 Schema1 Schema3 31

95 Mediated Schemas Query Query Mediated Schema Mediated Schema Schema2 Schema1 Schema3 Schema2 Schema1 Schema3 31

96 Mediated Schemas Query Query Mediated Schema Mediated Schema Query Schema x Schema2 Schema1 Schema3 Schema2 Schema1 Schema3 31

97 Peer Data Management Local Source Local Mapping Peer Schema Local Schema Peer Mapping 32

98 Why not by hand? source: 33 source:

99 Why not by hand? Size and complexity of source schemas Number of schemas sources Leveraging data instance values Schemas not known in advance source: 33 source:

100 Why not by hand? Size and complexity of source schemas Number of schemas sources Leveraging data instance values Schemas not known in advance source: 33 source:

101 Why not by hand? Size and complexity of source schemas Number of schemas sources Leveraging data instance values Schemas not known in advance source: 33 source:

102 Schema Matching Features 34

103 Schema Matching Features Schema-only vs schema & instances 34

104 Schema Matching Features Schema-only vs schema & instances Representational Lexical vs structural 34

105 Schema Matching Features Schema-only vs schema & instances Representational Lexical vs structural Internal vs external 34

106 Schema Matching Features Schema-only vs schema & instances Representational Lexical vs structural Internal vs external More in: Rahm E, Bernstein PA. A survey of approaches to automatic schema matching. The VLDB Journal. 2001;10(4): Shvaiko P, Euzenat J. A Survey of Schema-Based Matching Approaches. Journal on Data Semantics IV. 2005;3730:

107 Schema Matching Techniques 35

108 Schema Matching Techniques String-based 35

109 Schema Matching Techniques String-based Language-based 35

110 Schema Matching Techniques String-based Language-based Linguistic resources 35

111 Schema Matching Techniques String-based Language-based Linguistic resources Constraint-based 35

112 Schema Matching Techniques String-based Language-based Linguistic resources Constraint-based Alignment reuse 35

113 Schema Matching Techniques String-based Language-based Linguistic resources Constraint-based Alignment reuse Upper-level formal ontologies 35

114 Schema Matching Techniques String-based Language-based Graph-based Linguistic resources Constraint-based Alignment reuse Upper-level formal ontologies 35

115 Schema Matching Techniques String-based Language-based Linguistic resources Graph-based Taxonomy-based Constraint-based Alignment reuse Upper-level formal ontologies 35

116 Schema Matching Techniques String-based Language-based Linguistic resources Constraint-based Alignment reuse Graph-based Taxonomy-based Repository of structures Upper-level formal ontologies 35

117 Schema Matching Techniques String-based Language-based Linguistic resources Constraint-based Alignment reuse Upper-level formal ontologies Graph-based Taxonomy-based Repository of structures Model-based 35

118 Schema Matching Techniques String-based Language-based Linguistic resources Constraint-based Alignment reuse Upper-level formal ontologies Graph-based Taxonomy-based Repository of structures Model-based 35

119 Schema Matching Techniques String-based Language-based Linguistic resources Constraint-based Alignment reuse Upper-level formal ontologies Graph-based Taxonomy-based Repository of structures Model-based 35

120 Schema Matching Techniques String-based Language-based Linguistic resources Constraint-based Alignment reuse Upper-level formal ontologies Graph-based Taxonomy-based Repository of structures Model-based 35

121 A String-Based Technique Leveraging lexical features

122 Edit Distance 37

123 Edit Distance String distance: measures distance between two strings 37

124 Edit Distance String distance: measures distance between two strings Edit distance: number of operations needed to transform one string into the other 37

125 Edit Distance String distance: measures distance between two strings Edit distance: number of operations needed to transform one string into the other Common basic operations: Insert, delete or substitute one character Possibly with different weights depending on the operation and characters involved 37

126 Edit Distance String distance: measures distance between two strings Edit distance: number of operations needed to transform one string into the other Common basic operations: Insert, delete or substitute one character Possibly with different weights depending on the operation and characters involved Java libraries: SecondString, SimMetrics 37

127 Levenshtein Distance S u n d a y s S a t u r d a y

128 Levenshtein Distance Edit operations: insert, delete, substitute S a t u r d a y S u n d a y s

129 Levenshtein Distance Edit operations: insert, delete, substitute Each has a weight of 1 S u n d a y s S a t u r d a y

130 Levenshtein Distance Edit operations: insert, delete, substitute Each has a weight of 1 S u n d a y s S a t u r d a y

131 Levenshtein Distance Edit operations: insert, delete, substitute Each has a weight of 1 S u n d a y s S a t u r d a y

132 Levenshtein Distance Edit operations: insert, delete, substitute Each has a weight of 1 S u n d a y s S a t u r d a y

133 Levenshtein Distance Edit operations: insert, delete, substitute Each has a weight of 1 S u n d a y s S a t u r d a y

134 Levenshtein Distance Edit operations: insert, delete, substitute Each has a weight of 1 S u n d a y s S a t u r d a y

135 Levenshtein Distance Edit operations: insert, delete, substitute Each has a weight of 1 S u n d a y s S a t u r d a y

136 Levenshtein Distance Edit operations: insert, delete, substitute Each has a weight of 1 S u n d a y s S a t u r d a y

137 Levenshtein Distance Edit operations: insert, delete, substitute Each has a weight of 1 S u n d a y s S a t u r d a y

138 Levenshtein Distance Edit operations: insert, delete, substitute Each has a weight of 1 S u n d a y s S a t u r d a y

139 Levenshtein Distance Edit operations: insert, delete, substitute Each has a weight of 1 delete from Sundays insert to Sundays substitute in Sundays S u n d a y s S a t u r d a y

140 Levenshtein Distance Edit operations: insert, delete, substitute Each has a weight of 1 delete from Sundays insert to Sundays substitute in Sundays S u n d a S a t u r d a y Sundays y s

141 Levenshtein Distance Edit operations: insert, delete, substitute Each has a weight of 1 delete from Sundays insert to Sundays substitute in Sundays S u n d a S a t u r d a y Sundays Satundays y s

142 Levenshtein Distance Edit operations: insert, delete, substitute Each has a weight of 1 delete from Sundays insert to Sundays substitute in Sundays S u n d a S a t u r d a y Sundays Satundays Satundays y s

143 Levenshtein Distance Edit operations: insert, delete, substitute Each has a weight of 1 delete from Sundays insert to Sundays substitute in Sundays S u n d a S a t u r d a y Sundays Satundays Satundays Saturdays y s

144 Levenshtein Distance Edit operations: insert, delete, substitute Each has a weight of 1 delete from Sundays insert to Sundays substitute in Sundays S u n d a S a t u r d a y Sundays Satundays Satundays Saturdays Saturdays y s

145 Levenshtein Distance Edit operations: insert, delete, substitute Each has a weight of 1 delete from Sundays insert to Sundays substitute in Sundays S u n d a S a t u r d a y Sundays Satundays Satundays Saturdays Saturdays Saturdays y s

146 Levenshtein Distance Edit operations: insert, delete, substitute Each has a weight of 1 delete from Sundays insert to Sundays substitute in Sundays S u n d a S a t u r d a y Sundays Satundays Satundays Saturdays Saturdays Saturdays y Saturdays s

147 Levenshtein Distance Edit operations: insert, delete, substitute Each has a weight of 1 delete from Sundays insert to Sundays substitute in Sundays S u n d a S a t u r d a y Sundays Satundays Satundays Saturdays Saturdays Saturdays y s Saturdays Saturdays 38

148 A Linguistic Resource WordNet

149 WordNet 40

150 WordNet Fundamental components: Synonyn Sets (Synsets) 40

151 WordNet Fundamental components: Synonyn Sets (Synsets) {car, auto, automobile, machine, motorcar} a motor vehicle with four wheels; usually propelled by an internal combustion engine 40

152 WordNet Fundamental components: Synonyn Sets (Synsets) {car, auto, automobile, machine, motorcar} a motor vehicle with four wheels; usually propelled by an internal combustion engine {car, railcar, railway car, railroad car} a wheeled vehicle adapted to the rails of railroad 40

153 Hypernyms / Hyponyms Hypernyms: superordinates, isa relationships. A synset may have more than one hypernym. Hyponyms: subordinates {motor vehicle, automotive vehicle} hypernym {car, auto, automobile, machine, motorcar} hyponyms {cab, hack, taxi, taxicab} {ambulance} 41

154 Holonym / Meronym Meronym: name of a constituent part of, the substance of, or a member of something. X is a meronym of Y if X is a part of Y. Holonym: name of the whole of which the meronym names a part. Y is a holonym of X if X is a part of Y. {car, auto, automobile, machine, motorcar} holonym meronym { accelerator, accelerator pedal, gas pedal, gas, throttle, gun} 42

155 Other relationships in WN 43

156 Other relationships in WN Antonym 43

157 Other relationships in WN Antonym Entailment (for verbs) A verb X entails Y if X cannot be done unless Y is, or has been, done. 43

158 Other relationships in WN Antonym Entailment (for verbs) A verb X entails Y if X cannot be done unless Y is, or has been, done. Attribute (for adjectives) A noun for which adjectives express values. The noun weight is an attribute, for which the adjectives light and heavy express values. 43

159 A Graph-Matching Technique Leveraging structure

160 Similarity Flooding 45

161 Similarity Flooding Uses structure of the data to help matching schemas 45

162 Similarity Flooding Uses structure of the data to help matching schemas Similarity Flooding in Melnik et al. (2002) First maps schema elements with lexical similarity Then improves matching assuming that: If two elements are similar, then the elements adjacent to them are more probable to be similar 45

163 Similarity Flooding Uses structure of the data to help matching schemas Similarity Flooding in Melnik et al. (2002) First maps schema elements with lexical similarity Then improves matching assuming that: If two elements are similar, then the elements adjacent to them are more probable to be similar Selected paper 1: Melnik S, Garcia-Molina H, Rahm E. Similarity flooding: a versatile graph matching algorithm and its application to schema matching. IEEE Comput. Soc; 2002:

164 Deduplication Detecting duplicate entries

165 Why is there Duplicates? name: Muhammad Ali boxer id: weight: 200 lb total fights: 61 residence: 17, Nicestreet Louisville, KY Sport Authorities Administrationwide database first name: Mohamed last name: Ali age: 68 address: street: Nicestreet 17 city: Wondercity tax id: # Taxes Authorities 47

166 name: Muhammad Ali boxer id: weight: 200 lb total fights: 61 residence: 17, Nicestreet Louisville, KY R M name: Muhammad Ali address: city: Cairo country: Egypt tax id: # first name: Mohamed last name: Ali age: 68 address: street: Nicestreet 17 city: Wondercity tax id: # U Input: 2 entities with matched attributes Output: M for matched or U for unmatched. Possibly R for reject between M and U for cases where supervised decision is necessary. 48

167 Deduplication Features

168 Field Distance Metrics 50

169 Field Distance Metrics Value metrics Character-based Token-based Phonetic Numeric 50

170 Field Distance Metrics Value metrics Character-based String-based metrics seen for schema matching Token-based Phonetic Numeric 50

171 Field Distance Metrics Value metrics Character-based Token-based String-based metrics seen for schema matching Similar to Information Retrieval techniques (Topic 2 next week) Phonetic Numeric 50

172 Field Distance Metrics Value metrics Character-based Token-based String-based metrics seen for schema matching Similar to Information Retrieval techniques (Topic 2 next week) Phonetic Numeric 50 Not much techniques other than considering them as strings or direct difference

173 Field Distance Metrics Value metrics Character-based Token-based String-based metrics seen for schema matching Similar to Information Retrieval techniques (Topic 2 next week) Phonetic Numeric 50 Not much techniques other than considering them as strings or direct difference

174 Phonex 1. First letter as prefix 2. Encode non-prefix consonants 3. Remove duplicate adjacent codes not separated by a vowel consonant code b, f, p, v 1 c, g, j, k, q, s, x, z 2 d, t 3 l 4 m, n 5 r 6 h, w dropped 4. Drop vowels and truncate to prefix and max 3 codes, resp. pad with zero if necessary 51

175 Phonex 1. First letter as prefix 2. Encode non-prefix consonants 3. Remove duplicate adjacent codes not separated by a vowel 4. Drop vowels and truncate to prefix and max 3 codes, resp. pad with zero if necessary consonant code b, f, p, v 1 c, g, j, k, q, s, x, z 2 d, t 3 l 4 m, n 5 r 6 h, w dropped Ashcraftson 51

176 Phonex 1. First letter as prefix 2. Encode non-prefix consonants 3. Remove duplicate adjacent codes not separated by a vowel 4. Drop vowels and truncate to prefix and max 3 codes, resp. pad with zero if necessary consonant code b, f, p, v 1 c, g, j, k, q, s, x, z 2 d, t 3 l 4 m, n 5 r 6 h, w dropped Ashcraftson 1.Ashcraftson 51

177 Phonex 1. First letter as prefix 2. Encode non-prefix consonants 3. Remove duplicate adjacent codes not separated by a vowel 4. Drop vowels and truncate to prefix and max 3 codes, resp. pad with zero if necessary consonant code b, f, p, v 1 c, g, j, k, q, s, x, z 2 d, t 3 l 4 m, n 5 r 6 h, w dropped Ashcraftson 1.Ashcraftson 2.A2 26a132o5 51

178 Phonex 1. First letter as prefix 2. Encode non-prefix consonants 3. Remove duplicate adjacent codes not separated by a vowel 4. Drop vowels and truncate to prefix and max 3 codes, resp. pad with zero if necessary consonant code b, f, p, v 1 c, g, j, k, q, s, x, z 2 d, t 3 l 4 m, n 5 r 6 h, w dropped Ashcraftson 1.Ashcraftson 2.A2 26a132o5 3.A26a132o5 51

179 Phonex 1. First letter as prefix 2. Encode non-prefix consonants 3. Remove duplicate adjacent codes not separated by a vowel 4. Drop vowels and truncate to prefix and max 3 codes, resp. pad with zero if necessary consonant code b, f, p, v 1 c, g, j, k, q, s, x, z 2 d, t 3 l 4 m, n 5 r 6 h, w dropped Ashcraftson 1.Ashcraftson 2.A2 26a132o5 3.A26a132o5 4.A261 51

180 Phonex 1. First letter as prefix 2. Encode non-prefix consonants 3. Remove duplicate adjacent codes not separated by a vowel 4. Drop vowels and truncate to prefix and max 3 codes, resp. pad with zero if necessary consonant code b, f, p, v 1 c, g, j, k, q, s, x, z 2 d, t 3 l 4 m, n 5 r 6 h, w dropped Rupert 1.Rupert 2.Ro1e63 3.Ro1e63 4.R163 Ashcraftson 1.Ashcraftson 2.A2 26a132o5 3.A26a132o5 4.A261 Robert 1.Robert 2.Ro1e63 3.Ro1e63 4.R163 51

181 Other Phonetic Codes 52

182 Other Phonetic Codes NYSIIS Developed and still in use at the New York State Division of Criminal Justice Services Encodes vowels (mostly to A) Codes are letters instead of digits Longer codes (6 instead of 4) 52

183 Other Phonetic Codes 53

184 Other Phonetic Codes Metaphone Codes are letters instead of digits No maximum code length More elaborated coding rules 53

185 Other Phonetic Codes Metaphone Codes are letters instead of digits No maximum code length More elaborated coding rules Double Metaphone Returns a secondary code to help disambiguate 53

186 Detecting Duplicates

187 Bayes Decision Rule 55

188 Bayes Decision Rule M: match, U: unmatch M if p(m x) p(u x) U otherwise 55

189 Bayes Decision Rule M: match, U: unmatch M U Using Bayes rule if p(m x) p(u x) otherwise p(m x) p(u x) p(m x) p(x) p(u x) p(x) p(m)p(x M) p(u)p(x U) l(x) = p(x M) p(x U) p(u) p(m) 55

190 Bayes Decision Rule M: match, U: unmatch M if p(m x) p(u x) U otherwise Using Bayes rule p(m x) p(u x) Decision rule: likelihood ratio l(x) = p(x M) p(x U) p(u) p(m) p(m x) p(x) p(u x) p(x) p(m)p(x M) p(u)p(x U) l(x) = p(x M) p(x U) p(u) p(m) 55

191 Bayes Decision Rule M: match, U: unmatch M U Using Bayes rule if p(m x) p(u x) otherwise p(m x) p(u x) p(m x) p(x) p(u x) p(x) p(m)p(x M) p(u)p(x U) l(x) = p(x M) p(x U) p(u) p(m) Decision rule: likelihood ratio l(x) = p(x M) p(x U) p(u) p(m) Using independence assumption p(x M) = i p(x U) = i p(x i M) p(x i U) 55

192 Bayes Decision Rule p(x i M) p(x i U) 56

193 Bayes Decision Rule Priors ( p(x i M) and p(x i U) ) can be learned on a training set 56

194 Bayes Decision Rule Priors ( p(x i M) and p(x i U) ) can be learned on a training set Other methods based on Expectation- Maximisation (EM) algorithm can estimate priors without training set 56

195 Clustering-Based Decision Selected paper 2: Chaudhuri S, Ganti V, Motwani R. Robust Identification of Fuzzy Duplicates. ICDE :

196 Clustering-Based Decision X-Means Variant of K-Means without a fixed K Chauduri et al. observed that duplicates tend Using clustering techniques with appropriate parameters 1. to have small distances from each other (compact set property), and 2. 2) to have only a small number of other neighbors within a small distance (sparse neighborhood property). Selected paper 2: Chaudhuri S, Ganti V, Motwani R. Robust Identification of Fuzzy Duplicates. ICDE :

197 Dealing with O(n 2 ) Number of comparisons 1E E+11 5E E+11 0E ' ' ' '000 1'000'000 Number of entities in repository 58

198 Canopies 59

199 Canopies Create canopies using a cheap similarity metric Overlapping clusters 59

200 Canopies Create canopies using a cheap similarity metric Overlapping clusters Compare entities pairwise using a more expensive similarity metric 59

201 Dataspaces Pay-as-you-go Information Integration

202 Dataspaces Selected paper 3: Halevy AY, Franklin M, Maier D. Principles of dataspace systems. In: PODS 06. New York, NY, USA; 2006:

203 Dataspaces Note a data integration approach per se Selected paper 3: Halevy AY, Franklin M, Maier D. Principles of dataspace systems. In: PODS 06. New York, NY, USA; 2006:

204 Dataspaces Note a data integration approach per se Data co-existence appraoch Selected paper 3: Halevy AY, Franklin M, Maier D. Principles of dataspace systems. In: PODS 06. New York, NY, USA; 2006:

205 Dataspaces Note a data integration approach per se Data co-existence appraoch Pay-as-you-go data integration Leveraging human contributions for data integration in a non-invasive manner Selected paper 3: Halevy AY, Franklin M, Maier D. Principles of dataspace systems. In: PODS 06. New York, NY, USA; 2006:

206 Relationship between Schema Matching and Deduplication first name: Mohamed last name: Ali age: 68 address: street: Nicestreet 17 city: Wondercity tax id: # name: Muhammad Ali boxer id: weight: 200 lb total fights: 61 residence: 17, Nicestreet Louisville, KY Selected paper 4: Zhou X, Gaugaz J, Balke W-T, Nejdl W. Query relaxation using malleable schemas. SIGMOD Beijing, China; 2007:

207 Relationship between Schema Matching and Deduplication Are they duplicates? first name: Mohamed last name: Ali age: 68 address: street: Nicestreet 17 city: Wondercity tax id: # name: Muhammad Ali boxer id: weight: 200 lb total fights: 61 residence: 17, Nicestreet Louisville, KY Selected paper 4: Zhou X, Gaugaz J, Balke W-T, Nejdl W. Query relaxation using malleable schemas. SIGMOD Beijing, China; 2007:

208 Relationship between Schema Matching and Deduplication Are they duplicates? To compare field values we need schema matches first name: Mohamed last name: Ali age: 68 address: street: Nicestreet 17 city: Wondercity tax id: # name: Muhammad Ali boxer id: weight: 200 lb total fights: 61 residence: 17, Nicestreet Louisville, KY Selected paper 4: Zhou X, Gaugaz J, Balke W-T, Nejdl W. Query relaxation using malleable schemas. SIGMOD Beijing, China; 2007:

209 Relationship between Schema Matching and Deduplication Are they duplicates? To compare field values we need schema matches first name: Mohamed last name: Ali age: 68 address: street: Nicestreet 17 city: Wondercity tax id: # To find schema matches we need duplicates 62 name: Muhammad Ali boxer id: weight: 200 lb total fights: 61 residence: 17, Nicestreet Louisville, KY Selected paper 4: Zhou X, Gaugaz J, Balke W-T, Nejdl W. Query relaxation using malleable schemas. SIGMOD Beijing, China; 2007:

210 Relationship between Schema Matching and Deduplication Are they duplicates? To compare field values we need schema matches first name: Mohamed last name: Ali age: 68 address: street: Nicestreet 17 city: Wondercity tax id: # To find schema matches we need duplicates etc name: Muhammad Ali boxer id: weight: 200 lb total fights: 61 residence: 17, Nicestreet Louisville, KY Selected paper 4: Zhou X, Gaugaz J, Balke W-T, Nejdl W. Query relaxation using malleable schemas. SIGMOD Beijing, China; 2007:

211 Selected Topic Papers 1. Schema Matching Melnik S, Garcia-Molina H, Rahm E. Similarity flooding: a versatile graph matching algorithm and its application to schema matching. IEEE Comput. Soc; 2002: Deduplication Chaudhuri S, Ganti V, Motwani R. Robust Identification of Fuzzy Duplicates. ICDE : Dataspaces Halevy AY, Franklin M, Maier D. Principles of dataspace systems. In: PODS 06. New York, NY, USA; 2006: Interdependence between schema matching and deduplication Zhou X, Gaugaz J, Balke W-T, Nejdl W. Query relaxation using malleable schemas. SIGMOD Beijing, China; 2007: