RELAIS: A Record Linkage Toolkit Training course on record linkage Monica Scannapieco Istat

Transcription

1 RELAIS: A Record Linkage Toolkit Training course on record linkage Monica Scannapieco Istat scannapi@istat.it

2 RELAIS: Milestones Alfa Version (January 2007) Beta RELAIS 1.0 (February 2008) RELAIS 2.0 (June 2009) RELAIS 2.1 (June 2010) RELAIS (January 2012) RELAIS 3.0 (under development)

3 RELAIS: the Idea There is not a unique optimal solution for solving record linkage problems: for each phase the most appropriate technique should be chosen depending on application and data requirements, not only on the practitioner s skill Ad-hoc record linkage process (workflow) should be dynamically built RELAIS (REcord Linkage At IStat) is a toolkit serving such a purpose

4 Record Linkage Workflows RecLink WF Appl1 UpperLowerCase Normalization Blocking Preprocessing Normalization UpperLowerCase Schema reconciliation Search Space Reduction Blocking SNM RecLink WF Appl2 SNM Jaro Comparison Function Edit Distance Jaro Equality Equality Empirical Decision Model Probabilistic Empirical Probabilistic

5 RELAIS Features Modular structure: each phase is planned as a module of the toolkit, with an explicit interface with the other modules Top-down design: this allows to omit and/or iterate modules (phases) of the record linkage process Advantages: Dynamic composition of record linkage processes Parallel development of various techniques

6 Implementation Features RELAIS is implemented in Java and R Open Source Project (EU Public License) Java, object oriented language, suitable for data management parts R, functional language, suitable for computational intensive elaborations

7 RELAIS Architecture JAVA GUI JAVA R Application level Intermediate data res DSA MYSQL DSB M U PM RDSA RDSB

8 Principal Functionalities Input/Output Management Back-up support Residuals management Data Profiling Metadata for blocking variable selection Metadata for matching variable selection Search Space Creation and Reduction methods Cross Product Blocking Sorted Neighborhood Nested blocking Simhash (new!)

9 Principal Functionalities Comparison Functions Deterministic Decision Models Exact Rule-based Probabilistic Decision Model Fellegi-Sunter 1: 1 Reduction methods Optimal Greedy

10 Input/Output Management Input from: Text files DB Oracle DB MySQL Back-up Residuals Back-up support: allows saving intermediate and final results on a db different from the working one (but on RELAIS mysql dbms)

11 Input/Output Management Residuals management: allows processing residuals resulting from the execution of a RL process Output: GUI visualization (Utility->table display) Save to text file

12 Data Profiling Blocking and matching variables: Completeness Accuracy Consistency Categories Frequency distribution Entropy Blocking Blocking adequacy Matching Correlation

13 Search Space Creation and Reduction Methods Search Space Creation Cross product Reduction Methods: Blocking Selection of a blocking key (two or more variables) Block modality table reports information on created blocks (sizes, number of blocks, ) Reduction Method: Sorted Neighborhood Method (SNM) Selection of a sorting key (two or more variables) Choice of the window size

14 Search Space Creation and Reduction Methods Nested Blocking Choice of the blocking variables Choice of SNM variables and window size Within each block SNM is applied

15 Search Space Creation and Reduction Methods Simhash String S parsed into substrings S i (e.g. bigrams) Each substring is assigned a weight W i (i.e. how much it contributes to s) S is represented as a «feature vector» (S i, W i ) [Moses S. Charikar: Similarity Estimation Techniques from Rounding Algorithms, STOC, 2002]

16 Search Space Creation and Reduction Methods Simhash Hash function mapping each (S i, W i ) into a (0,1) vector Merge (S i, W i ) Ordering and Hamming distance between mapped vectors Comparisons within window w, i.e. Hamming distance less than w

17 One-shot Blocking Execution Automatic execution of blocks by sequencing them Possibility of executing one block at a time

18 Comparison Functions Equality Jaro Dice Jaro-Winkler Levenshtein 3-grams Soundex Numeric Comparison

19 Deterministic Decision Models: Equality Match Exact matching: relational JOIN over specified variables Useful at the initial stage of a RL process, when it makes sense to prune the pairs to compare by removing exact matches

20 Deterministic Decision Models: Rule Based Rules specified through a GUI The GUI allows specification of Variables of the formula Comparison functions Thresholds for each variable Operators to combine atomic formulas (AND/OR)

21 Deterministic Decision Models: Rule Based

22 Probabilistic Decision Model: Fellegi- Sunter Steps: 1. Choice of matching variables 2. Choice of comparison functions and thresholds (for each variable) 3. Contingency table computation 4. EM Estimation MU table result

23 estimates of frequency distributions Posterior probability f_m/(f_m+f_u) In 2.2 Version: Precision = TP/TP+FP and Recall=TP/TP+FN

24 On the Reliability of Estimation Results The model estimates can be not reliable when the conditional probabilities of at most one of the matching variables are of the kind: m(g)=0 or u(g)=1 in such conditions, the system raises a warning The system stops in case of conflicting values of m and u for a single matching variable In 2.2 version table of marginals created for analysis purposes

25 Probabilistic Decision Model: Fellegi- Sunter Steps: 1. Choice of matching variables 2. Choice of comparison functions and thresholds (for each variable) 3. Contingency table computation 4. EM Estimation MU table result 5. Threshold Selection

26 Threshold Selection Tu Tm Unmatch Possible match Match

27 Probabilistic Decision Model: Fellegi- Sunter Steps: 1. Choice of matching variables 2. Choice of comparison functions and thresholds (for each variable) 3. Contingency table computation 4. EM Estimation MU table result 5. Threshold Selection 6. Linkage Result: 1:1 Result or cluster result

28 1:1 Matching DETERMINISTIC PROBABILISTIC LP Problem (Global Optimization) LP Problem (Global Optimization) Greedy Reduction Greedy Reduction Subrules weight R weight

29 1:1 Matching Deterministic Model LP problem with input matrix: Weight associated to each atomic subrule Sum of weights Greedy: sorting of pairs by the sum of weights Choices are local

30 1:1 Matching Probabilistic Model LP problem with input matrix: r=m/u Greedy: Sorting of pairs on the basis of r Choices are local

31 Conclusions RELAIS released at: ISTAT: /analisi_dati/relais/ Joinup: e

32 Appendix: RELAIS Installation

33 Overview Introduction Installation of Java Installation of R Installation of MySQL Installation of RELAIS

34 Introduction RELAIS requires installation of : JAVA, R, MySQL JAVA JRE - JDK RELAIS R Packages: lpsolve RODBC MySQL MySQL ODBC Driver

35 Installation of Java JRE 6 JDK 1.6 ( /downloads/index.html) Modify the system variable PATH by adding the bin directory of JRE (for example: C:\Program Files\Java\jre6\bin ) to modify system variable access to: Start -> Control panel -> System -> Advanced -> Environment variables

36 Installation of R R 2.9 (or higher) ( Modify the system variable PATH by adding the bin directory of R (for example: C:\Program Files\R\R-2.9\bin ) Install packages lpsolve and RODBC choose item Install packages in menu Packages

37 Installation of MySQL database Install MySQL Server 5.1 (or higher) ( Configure MySQL Server Instance (using the MySQL Server Instance Configuration Wizard) It is necessary to create an anonymous account for instance (in the security setting form)

38 Creation of OBDC data source Install MySQL Connector ODBC driver ( wnloads/connector/odbc/) Create a ODBC source named relais for MySQL instance (Start -> Control panel -> Administration Tools -> Data Source (ODBC)) to test ODBC connection, creation of relais database is required by typing the command: create database relais; in mysql prompt

39 Installation of RELAIS Execute RELAIS Setup Wizard (Relais-2.2- setup.exe) The directory Relais 2.2 will be created in your program folder with the binary files of the toolkit It is recommended to check the folder permissions: the readwrite permissions must be enabled for all users

40 Installation of RELAIS Relais is installed, hence you can run it by executing the Relais.bat file Enjoy RELAIS!