Record Linkage. with SAS and Link King. Dinu Corbu. Queensland Health Health Statistics Centre Integration and Linkage Unit

Transcription

1 Record Linkage with SAS and Link King Dinu Corbu Queensland Health Health Statistics Centre Integration and Linkage Unit Presented at Queensland Users Exploring SAS Technology QUEST 4 June 2009

2 Basics Record linkage: searching systematically for records that refer to the same entity in one or more datasets Purposes: building longitudinal datasets for analysis, reporting, planning, operational decision making, etc

3 Typically, a data linkage project follows the steps: Blocking records to reduce the decision space, which has quadratic size n(n-1)/2 - compare Johnson with Johnston, not Johnson with Lee Comparing names, surnames, DoBs, etc using field comparison functions: - phonetic similarity: Soundex, Methaphone, NYSIIS - pattern matching: Edit distance, Jaro, Winkler - 1 for perfect match, 0 for no-match, in between for partial match Classifying the pairs of records to assess the matching/non-matching status - deterministic and/or probabilistic algorithms - biggest challenge: Grey Area (un-certain pairs)

4 Grey Area Using record-pairs comparison functions: a score is assigned for each pair Grey area : where score histograms for matches and non-matches overlap

5 My involvement in records linkage In New Zealand working for a government organisation that runs a range of data matches (listed on ) - tools: big package of tailored SAS codes, batch processed - developments: new pieces of SAS codes to answer new requirements Griffith University, the research team Justice Griffith - the package Febrl was traditionally used there: very elegant and sophisticated (highly configurable, lots of blocking methods and comparison functions), but failing sometimes when processing big datasets - had to find a rescuer: Link King that runs on SAS platform Qld Health s Health Statistic Centre, Integration and Linkage Unit - SAS, Link King, and Febrl: the core of a system we are currently building -aim: linking tens of millions of records from health system and not only

6 Looking for powerful record linkage software Tried first Wikipedia - the top of specialised page:., and (Voilà!) at the bottom of the page: Link King is a free SAS/AF application that must be run on Foundation SAS Could not find something better (and free) when searching somewhere else

7 Starting Link King

8 Main interface

9 Importing data Link King can get data in different formats: SAS, csv, SPSS.

10 Control panel for processing data

11 Classify and Classification Report This report comes after pressing the corresponding button, and it summarises:

12 Cross-referencing of the Deterministic and Probabilistic Classification System

13 Manually reviewing interface It allows to accept /reject links, and to set up and apply filters, working either pair by pair or on subsets of the Grey Area Grey Area is split in 14 subsets, depending on what variables do not match perfectly

14 Mapping the Final Results This is based on the cross-referencing deterministic and probabilistic classification system after manually reviewing the Grey Area

15 Reviewing the Final Results This allows sorting, searching, breaking links, and regrouping

16 Saving Results The Final Master Linking File only, or all final files that Link King produces can be saved in SAS and csv formats To open them later in SAS you have to run first options nofmterr;

17 Linking QHAPDC (Queensland Hospital Admitted Data Collection) with Death Data (from Births, Deaths and Marriages) We build a process that uses SAS, Link King and Febrl (due to its flexibility) to reduce or even eliminate the manual reviewing in Link King. 1. Extract QHAPDC with a SAS macro from several Oracle tables, and: - split it in appropriate subsets, clean, rearrange names - sort with nodupkey option and produce the input file for Link King 2. Running part of the Link King process and refine Grey Area in Febrl: - interrupt Link King process, and extract Grey Area with a SAS macro - refine the Grey Area with Febrl, and export it back using SAS - continue the process in Link King 3. Repeat the procedure for linking the resulting Master File with Death Data

18 Appendix A few things from what we do with SAS to prepare input data for Link King: Extracting one year of data from four Oracle tables

19 Appendix Cleaning data of records about babies not having yet a name and records with too many missing values Rearranging surnames and alternate surnames when some conditions are met

20 Thank you and launching a challenge For refining the Grey Area : We chose to use Febrl because it is more flexible, and it provides a big range of comparison functions We would welcome the creation of an application on SAS platform allowing the Grey Area to be processed automatically using more comparison than Link King has Any questions and suggestions are welcomed. Contact details: - work: dinu_corbu@health.qld.gov.au; home: dinu.corbu@yahoo.com.au;