TABLE OF CONTENTS PAGE TITLE NO.

Transcription

1 TABLE OF CONTENTS CHAPTER PAGE TITLE ABSTRACT iv LIST OF TABLES xi LIST OF FIGURES xii LIST OF ABBREVIATIONS & SYMBOLS xiv 1. INTRODUCTION 1 2. LITERATURE SURVEY MOTIVATIONS & OBJECTIVES OF THIS WORK EXPERIMENTATION INTRODUCTION SEMI AUTOMATIC METHOD FOR STRING 32 MATCHING METRICS FOR MEASURING SIMILARITY EDIT DISTANCE AFFINE GAP METHOD NEEDLEMAN WUNSCH DISTANCE OR SELLERS 38 ALGORITHM SMITH WATERMAN DISTANCE THE JARO METRIC AND ITS 42 VARIANTS JACCARD INDEX TANIMOTO COEFFICIENT (EXTENDED JACCARD COEFFICIENT) 45 viii

2 TF / IDF (TERM FREQUENCY / INVERSE DOCUMENT 45 FREQUENCY) N-GRAMS APPROACH RABIN KARP METHOD KNUTH MORRIS PRATT METHOD BOYER MOORE APPROACH HYBRID STRING MATCHING PROCESS DATA MINING & KNOWLEDGE DISCOVERY TECHNIQUE FOR MULTIMEDIA DATA USING 62 UNSUPERVISED CONFLATION METHOD DUPLICATE DETECTION USING UNSUPERVISED CONFLATION METHOD 62 (UCM) PROBLEM DEFINITION SIMILARITY ESTIMATION UNSUPERVISED CONFLATION METHOD OVERVIEW STRING SIMILARITY FUNCTION BASED CLASSIFIER C WEIGHTED COMPONENT SIMILARITY SUMMING (WCSS) 67 CLASSIFIER C2 5. RESULTS & DISCUSSION SEMI AUTOMATIC METHOD FOR STRING MATCHING EXPERIMENTAL EVALUATION UNSUPERVISED CONFLATION METHOD EXPERIMENTAL EVALUATION DATA SETS EVALUATION METRICS EXPERIMENTAL RESULTS 77 ix

3 6. CONCLUSION 6.1 CONCLUSION SCOPE FOR FUTURE WORK 88 REFERENCES 90 APPENDICES APPENDIX I DEFINITIONS OF TERMS USED IN THIS THESIS 99 LIST OF PUBLICATIONS x

4 LIST OF TABLES TABLE PAGE TITLE 1.1 Elementary Examples of Matching Pairs of Records (Dependent on Context) Computation of Levenshtein Distance Computation of Needleman Wunsch Distance Computation of Smith-Waterman Distance IDF values Computation of scores Sample Duplicate Records from the Restaurant Database Sample Duplicate Records from the Cora Database Sample Duplicate Records from the Reasoning Database F-measures from the Experiments Structure of the table ebook Structure of the table mp Structure of the table video 76 xi

5 LIST OF FIGURES FIGURE PAGE TITLE 1.1 The general process of matching two databases Query results from Query results from Sample duplicate records from (a) A restaurant database (b) A scientific citation database Modified alignment from Advanced Dynamic Programming example Alignment from Figure 4.2 re-scored using affine gap penalties Modified alignment. Equivalent under regular gap penalty system The alignment from Figure 4.4 re-scored using affine gap penalties Computation of Jaro Metric Example for N-Grams approach Example 1 for Rabin Karp approach Example 2(a) for Rabin Karp approach Example 2(b) for Rabin Karp approach Example for KMP approach Example for KMP approach Step Example for KMP approach Step Example for KMP approach Step Example for KMP approach Step Example for KMP approach Step Example for KMP approach Step Example for KMP approach Step 7 56 xii

6 4.19 Example for KMP approach Step Example for KMP approach Step Example for KMP approach Step Example for KMP approach Step Example for KMP approach Step Example for KMP approach Step Duplicate Vector Identification Algorithm Component Weight Assignment Algorithm F-Measures from the Experiments Sample records from the ebook table Sample records from the mp3 table Sample records from the video table Domain Selection Source Selection Source Selection After Loading Calculation of Weights Record Selection Record Similarity Calculated Results Record Similarity Matching all records Three different similarity thresholds on e-book Three different similarity thresholds on mp Two different similarity thresholds on video Component weight setting based on similarity values of the fields in N Effect of the threshold in matching process 85 xiii

7 LIST OF ABBREVIATIONS & SYMBOLS AI : Artificial Intelligence DNA : Deoxyribonucleic Acid DBLP : Digital Bibliography & Library Project EM : Expectation Maximization Febrl : Freely Extensible Biomedical Record Linkage HTML : Hyper Text Markup Language ISBN : International Standard Book Number M-C : Mapping-Convergence MCMC : Markov Chain Monte Carlo NLP : Natural Language Processing OCR : Optical Character Recognition PEBL : Positive Example Based Learning PES : Post Enumeration Survey PPRL : Privacy Preserving Record Linkage RelDC : Relationships for domain independent Data Cleaning RL : Record Linkage RNA : Ribonucleic Acid SQL : Structured Query Language SVM : Support Vector Machine TF-IDF : Term Frequency Inverse Document Frequency UCM : Unsupervised Conflation Method U.S.A : United States of America WCSS : Weighted Component Similarity Summing D : Distance between two strings s : String 1 t : String 2 O : Edit Distance xiv

8 c : Cost of the edit operation x i : th i character of string x y j : j th character of string y M : Matrix G : Gap cost d : distance function P : length of the longest common prefix θ : Cosine similarity T : Tanimoto coefficient N : Non duplicate vector set C1, C2 : Classifiers S a, S b : Pair of Strings : Null set AS th : Predefined Threshold value γ : Feature Vector P(γ M) : Probabilities of observing feature vector for a matched pair (P(γ U) : Probabilities of observing feature vector for a nonmatched pair Tμ : Threshold based on desired error level for equivalent record pair Tλ : Threshold based on desired error level for nonequivalent record pair xv