Outline. A Probabilistic Deduplication, Record Linkage and Geocoding System. Data cleaning and standardisation (2)

Transcription

1 Outlie A Probabilistic Deduplicatio, Record Likage ad Geocodig System Peter Christe ad Tim Churches Data Miig Group, Australia Natioal Uiversity Cetre for Epidemiology ad Research, New South Wales Departmet of Health Cotact: peter.christe@au.edu.au Project web page: Fuded by the ANU, the NSW Departmet of Health, the Australia Research Coucil (ARC), ad the Australia Partership for Advaced Computig (APAC) Data cleaig ad stadardisatio Record likage / data itegratio Febrl overview Probabilistic data cleaig ad stadardisatio Blockig / idexig Record pair classificatio Parallelisatio i Febrl Data set geeratio Geocodig Outlook Peter Christe, April 2005 p./20 Peter Christe, April 2005 p.2/20 Data cleaig ad stadardisatio () Real world data is ofte dirty Missig values, icosistecies Typographical ad other errors Differet codig schemes / formats Out-of-date data Names ad addresses are especially proe to data etry errors Cleaed ad stadardised data is eeded for Loadig ito databases ad data warehouses Data miig ad other data aalysis studies Record likage ad data itegratio Name Data cleaig ad stadardisatio (2) Address Doc Peter Miller 42 Mai Rd. App. 3a Caberra A.C.T /4/986 Giveame Surame Day Moth Year doctor o. 42 peter ame miller type 42 Mai Rd. App. 3a Caberra A.C.T Uittype Uit o. ame Territory mai road apartmet 3a caberra act Postcode 2600 Remove uwated characters ad words Date of Birth 29 4 Expad abbreviatios ad correct misspelligs Segmet data ito well defied output fields 986 Peter Christe, April 2005 p.3/20 Peter Christe, April 2005 p.4/20

2 Record likage / data itegratio The task of likig together records represetig the same etity from oe or more data sources If o uique idetifier is available, probabilistic likage techiques have to be applied Applicatios of record likage Remove duplicates i a data set (iteral likage) Merge ew records ito a larger master data set Create customer or patiet orieted statistics Compile data for logitudial studies data Data cleaig ad stadardisatio are importat first steps for successful record likage Peter Christe, April 2005 p.5/20 Febrl Freely extesible biomedical record likage A experimetal platform for ew ad improved likage algorithms Modules for data cleaig ad stadardisatio, record likage, deduplicatio ad geocodig Ope source Implemeted i Pytho Easy ad rapid prototype software developmet Object-orieted ad cross-platform (Uix, Wi, Mac) Ca hadle large data sets stable ad efficietly May exteral modules, easy to exted Record likage techiques Determiistic or exact likage A uique idetifier is eeded, which is of high quality (precise, robust, stable over time, highly available) For example Medicare, ABN or Tax file umber (are they really uique, stable, trustworthy?) Probabilistic likage (Fellegi & Suter, 969) Apply likage usig available (persoal) iformatio Examples: ames, addresses, dates of birth Other techiques (rule-based, fuzzy approach, iformatio retrieval) Peter Christe, April 2005 p.6/20 Probabilistic data cleaig ad Three step approach. Cleaig stadardisatio Based o look-up tables ad correctio lists Remove uwated characters ad words Correct various misspelligs ad abbreviatios 2. Taggig Split iput ito a list of words, umbers ad separators Assig oe or more tags to each elemet of this list (usig look-up tables ad some hard-coded rules) 3. Segmetig Use either rules or a hidde Markov model (HMM) to assig list elemets to output fields Peter Christe, April 2005 p.7/20 Peter Christe, April 2005 p.8/20

3 Hidde Markov model (HMM) HMM data segmetatio 5 8 Giveame Giveame 2 Middleame Ed Middleame Ed 6 6 0% Surame 00% 7 0% Surame 00% 7 A HMM is a probabilistic fiite state machie Made of a set of states ad trasitio probabilities betwee these states I each state a observatio symbol is emitted with a certai probability distributio I our approach, the observatio symbols are tags ad the states correspod to the output fields For a observatio sequece we are iterested i the most likely path through a give HMM (i our case a observatio sequece is a tag list) The Viterbi algorithm is used for this task (a dyamic programmig approach) Smoothig is applied to accout for usee data (assig small probabilities for usee observatio symbols) Peter Christe, April 2005 p.9/20 Peter Christe, April 2005 p.0/20 Probabilistic data cleaig ad stadardisatio Example Blockig / idexig 5 8 0% Giveame 6 Surame 2 Middleame Ucleaed iput strig: Doc. peter Paul MILLER 00% Cleaed ito strig: dr peter paul miller Word ad tag lists: [ dr, peter, paul, miller ] [ TI, GM/SN, GM, SN ] Two example paths through HMM : -> (TI) -> Giveame (GM) -> Middleame (GM) -> Surame (SN) -> Ed 2: -> (TI) -> Surame (SN) -> Giveame (GM) -> Surame (SN) -> Ed 7 Ed Number of possible liks equals the product of the sizes of the two data sets to be liked Performace bottleeck i a record likage system is usually the (expesive) evaluatio of similarity measures betwee record pairs Blockig / idexig techiques are used to reduce the large amout of record comparisos Febrl cotais (curretly) three idexig methods Stadard blockig Sorted eighbourhood approach Fuzzy blockig usig -grams (e.g. bigrams) Peter Christe, April 2005 p./20 Peter Christe, April 2005 p.2/20

4 Record pair classificatio For each record pair compared a vector cotaiig matchig weights is calculated Example: Record A: [ dr, peter, paul, miller ] Record B: [ mr, pete,, miller ] Matchig weights: [0.2, 0.8, 0.0, 2.4 ] Matchig weights are used to classify record pairs as liks, o-liks, or possible liks Fellegi & Suter classifier simply sums all the weights, the uses two thresholds to classify Improved classifiers are possible (for example usig machie learig techiques) Speedup Parallelisatio Implemeted trasparetly to the user Curretly usig MPI via Pytho module PyPar Use of super-computig cetres is problematic (privacy) Alterative: I-house office clusters Some iitial performace results (o Su SMP) Step - Loadig ad idexig 20,000 records 00,000 records 200,000 records Number of processors Speedup Step 2 - Record pair compariso ad classificatio 20,000 records 00,000 records 200,000 records Number of processors Peter Christe, April 2005 p.3/20 Peter Christe, April 2005 p.4/20 Data set geeratio Difficult to acquire data for testig ad evaluatio (as record likage deals with ames ad addresses) Also, likage status is ofte ot kow (hard to evaluate ad test ew algorithms) Febrl cotais a data set geerator Uses frequecy tables for give- ad surame, street ame ad type, suburb, postcode, age, etc. Uses dictioaries of kow misspelligs Duplicate records are created via radom itroductio of modificatios (like isert/delete/traspose characters, swap field values, delete values, etc.) Data set geeratio Example Data set with 4 origial ad 6 duplicate records REC_ID, ADDRESS, ADDRESS2, SUBURB rec-0-org, wylly place, pie ret vill, taree rec-0-dup-0, wyllyplace, pie ret vill, taree rec-0-dup-, pie ret vill, wylly place, taree rec-0-dup-2, wylly place, pie ret vill, tared rec-0-dup-3, wylly parade, pie ret vill, taree rec--org, stuart street, hartford, meto rec-2-org, griffiths street, myross, kilda rec-2-dup-0, griffith sstreet, myross, kilda rec-2-dup-, griffith street, mycross, kilda rec-3-org, elleborough place, kalkite homestead, sydey Each record is give a uique idetifier, which allows the evaluatio of accuracy ad error rates for record likage Peter Christe, April 2005 p.5/20 Peter Christe, April 2005 p.6/20

5 Geocodig The process of matchig addresses with geographic locatios (logitude ad latitude) Geocodig tasks Preprocess the geocoded referece data (cleaig, stadardisatio ad idexig) d atioal address file G-NAF: Available sice early 2004 (PSMA, Source data from 3 orgaisatios (aroud 32 millio source records) Processed ito 22 ormalised database tables Clea ad stadardise the user addresses (Fuzzy) match of user addresses with the referece data Street Alias Retur locatio ad match status Street Alias Match status: address, street or locality level referece data used: G-NAF Street Adress Detail Adress Alias Adress Site Adress Site Peter Christe, April 2005 p.7/20 Peter Christe, April 2005 p.8/20 Febrl geocodig system Outlook G NAF data files Febrl clea ad stadardise Process GNAF module AustPost data GIS data Build iverted idexes User data file Iverted idex data files d user data file Geocodig module Febrl clea ad stadardise Febrl geocode match egie Web server module Web iterface iput data d Web data Oly NSW G-NAF data available (aroud 4 millio address, 58,000 street ad 5,000 locality records) Additioal Australia Post ad GIS data used (for data imputig ad to compute eighbourig regios) Several research areas Improvig probabilistic data stadardisatio New ad improved blockig / idexig methods Apply machie learig techiques for record pair classificatio Improve performaces (scalability ad parallelism) Project web page Febrl is a ideal experimetal platform to develop, implemet ad evaluate ew data stadardisatio ad record likage algorithms ad techiques Peter Christe, April 2005 p.9/20 Peter Christe, April 2005 p.20/20