Techniques for Large Scale Data Linking in SAS. By Damien John Melksham

Transcription

1 Techniques for Large Scale Data Linking in SAS By Damien John Melksham

2 What is Data Linking? Called everything imaginable: Data linking, record linkage, mergepurge, entity resolution, deduplication, fuzzy matching, etc What we mean: finding the same entity when no explicit identifier is able to do it sufficiently well(often because of data quality/real-world issues). Contrast with simple equality merges or joins. Data linking has applications in administration, health, science, tax, fraud, business, security At a bit of an intersection between statistics, computer science, math, algorithms (and maybe even philosophy).

3 Pretend Example Linking death registries to an administrative data set Such a technique might be used for insurance or health studies to allow investigations into previously hidden causes, data, and comorbidities. Death registrations tend to carry variables describing names, age, sex, geography, cause of death, etc. Please note: All details that follow are completely fictional. Any resemblance to persons living or dead (especially actual death registry entries), is purely coincidental.

4 Pretend Data d_fname d_sname d_dob d_mob d_yob d_sex d_state John Smith M NSW Jane Smoth F VIC Bob O'toole 34 M QLD a_fname a_sname a_dob a_mob a_yob a_sex a_state John Smith M NSW Jane F VIC Jeremiah Davies M VIC

5 Techniques Covered The General Philosophy of Data Linking SAS Views and Blocking Efficient Pre-Processing, Macros, and Doing Work Up-Front Custom PROC FCMP Functions: Distance and String Metrics for SAS Efficient ordering of statements to match your data I can t nearly cover everything, so I ve tried to focus on topics that might inspire you or which you can apply to other work as well, or which you can take away and use after this presentation (i.e. some free function implementations you can download).

6 Data Linking in a Nutshell Variables on two data sets provide evidence that records represent the same entity. Records on two data sets having the same value for the same variable provide evidence those records represent the same entity. Of course, there are other comparisons and relationships available apart from equality Different variables provide different amounts of evidence: i.e Sex vs First Name. It is more significant to find two Damien Melksham s than it is to find two males Total evidence combined from many variables help you find the same entities across multiple data sets. You might see terms such as deterministic/probabilistic data linking in literature for specific techniques, but language and deeper math aside, the basic ideas are pretty intuitive.

7 Pretend Example d_fname d_sname d_dob d_mob d_yob d_sex d_state John Smith M NSW a_fname a_sname a_dob a_mob a_yob a_sex a_state John Smith M NSW Total = = 9 points Looking like a match!

8 Pretend Example cont d_fname d_sname d_dob d_mob d_yob d_sex d_state John Smith M NSW a_fname a_sname a_dob a_mob a_yob a_sex a_state Jane Smoth F VIC Total = -9 points Not looking like a match!

9 SAS SQL and Data Step SQL and Data Step language are actually very well suited to data linking. SQL lets you bring records together on common variables via efficient natural/inner joins easily Data step lets you implement custom rules to customise a data linking exercise to your particular needs. SAS functions are brilliant for data cleaning and standardising before bringing records together for comparisons.

10 Computational Feasibility Data linking might not be so hard if it was computationally easy. It scales poorly 20,000,000 records compared to 20,000,00 records = 400,000,000,000,000 possible record combinations. Computers don t have enough memory or disk space to hold data that big. People don t have enough time to wait for CPU s to run calculations that big (they would probably appear in the death records before getting their answers).

11 So we cut down on comparisons Data linkers call this blocking Use a cheap method to limit record comparisons before partaking in more expensive comparisons In practice this is often an equality join between data sets on high quality variables because other ways are comparatively complicated and infeasible. (This is a relative strength of SQL) Due to errors on data sets, sometimes you have to use a few different block and comparison runs to avoid blocking on missing data, and to maximise total good comparisons. But be careful using SQL: Its hash joins just using equality with logical AND operations are efficient. Linking together several equality clauses and AND operations with logical OR operations is not (and might crash or hang your computer at the time of writing).

12 SAS Views and Why You Use Them For large data sets, you probably don t have the disk space to store all comparisons even from a restrictive equality join. But with a SAS View, you can work with data that doesn t have to be written to disk. Instead views supply record pairs one at a time for further processing, but they can be treated like a data set. A view can implement your blocking strategy, and a further data step (or SQL) can implement your more involved and expensive comparisons, which can then be output to disk if record comparisons result in points that cross a certain threshold. But remember the restrictions of views (don t sort, no random access)

13 SAS Views: Quick Syntax Examples DATA STEP VIEW: data work.block /view=work.block; run; PROC SQL VIEW; PROC SQL; CREATE VIEW work.block AS QUIT;

14 Drop lots, or keep very little There s a good chance that you don t need all variables on both data sets that you re joining for the comparison stage, so drop what you can as early as possible. Data set options wherever you reference a data set help you do this with KEEP and DROP statements. The two compliment each other well and can lead to efficient operations. Example1: work.deaths(keep= d_fname d_dob) Example2: work.deaths(drop=d_sex)

15 And while we re on data set options There s a good chance you ll want to use these data set options DROP KEEP RENAME WHERE I always had trouble remembering how these interacted with each other until someone told me these 4 are applied in alphabetical order Example: work.deaths(keep = d_fname d_dob rename = (d_fname = fname) where = (fname = John ))

16 Bring calculations forward using macro language if you can In data linking, much of your time is going to be spent waiting for the compiled code to run, so anything you can do to speed that up is usually beneficial. Use macros to bring appropriate calculations forward during macro pre-processing to write the most efficient run-time code possible. SAS is actually quite good at this, due to the historical focus on macro operations relative to functions.

17 Recode during data cleaning, not comparison Some functions are quite expensive and lend themselves more to cleaning data rather than being used than during comparisons. Phonetic algorithms are a good example. They map several names to similar encodings and deal with typos, but are expensive. Say 10 cpu cycles. I.e. James -> JMS, Jomes -> JMS Equality operation is cheap. Say 1 cpu cycle. If you have 10 records on two data sets that need comparing to each other, its cheaper to encode data then compare encodings via equality, rather than calculate encodings for each comparison and then compare via equality. Encode and then compare: (2 * 10) + (10 * 10 * 1) = 120 cpu cycles Encode during comparison: (2 * 10 * 10) + (10 * 10 * 1) = 320 cpu cycles

18 Efficient conditionals and code written for the data Lastly, if you know your data is not structured uniformly, you might be able to increase run-time efficiency by structuring conditional work in your code based on the probability of values, and relative to the amount of work each value entails. In English: If your data is 80% male, 10% female, and 5% missing, and if all three states require you to do the same amount of work, consider performing actions in the order of: if male, else if female, else.

19 SAS ships with several functions suited to data linking Soundex Spedis Complev Compged But there s a few more in the literature and I ve made my basic FCMP implementations available that you re free to use if so inclined.

20 PROC FCMP SAS Functions Caverphone 2.0 Double Metaphone Jaro/Winkler NYSIIS Chebyshev/Cityblock/Euclidean/Hamming/Minkowski Distances Fuzzy number matching Expected-unique persons You can obtain the PROC FCMP source code for these functions from: You will have to change the libraries in the PROC FCMP call and let SAS know where to find the custom functions. See PROC FCMP documentation.

21 Phonetic Algorithms Caverphone 2.0 Double Metaphone NYSIIS Soundex (SAS native function) These algorithms all try to map several names/words to simple codes. Each has its own strengths and weaknesses. Don t think they re perfect. They re also typically pretty expensive/slow in terms of CPU. I.e. d_fname_code = soundex(d_fname); a_fname_code = soundex(a_fname); Then compare d_fname_code and a_fname_code in the resulting joined data set with an equality operator, and assign points as appropriate.

22 Jaro/Winkler string similarity functions Designed to be relatively cheap, so can be used in comparisons Still far more expensive than equality or most numeric operations To be used for short strings like first names, surnames, proper names, street names. My free implementation only works up to 52 characters in length, which is not really a bad thing Reliability drops as strings get comparatively long or extremely short. Reports similarity between two strings as a number between 1 (representing equality) and 0 (representing absolute disagreement). You ll typically use these with conditions or cut-offs to assign points that deal with some fuzziness or typographical error: i.e. if jaro(d_fname, a_fname) >= 0.9 then name_point = 1; else name_point = -1;

23 There s a lot more as well You can go about as deep into data linking as you re prepared to There s a lot more techniques I haven t managed to go into today. I wrote my own system of data linking macros (ICARUS) to specialise in the topic and automate as many of these issues as possible Source is not currently publically available, but a copy of the documentation can be found at Documentations for the new functions mentioned in this presentation can be found in the ICARUS user manual at the above link.

24 Questions/Contacts/Further Information: Further questions and comments, please direct communications to: or