Patient Matching A-Z Wednesday, March 2nd 2016

Transcription

1 Patient Matching A-Z Wednesday, March 2nd 2016 Adam W. Culbertson, Innovator-in-Residence HHS, HIMSS

2 Overview Overview of Innovator-in-Residence Program Background on Patient Matching Challenges to Matching Evaluation of Matching Algorithms and Metrics

3 Innovator-in-Residence (IIR) Program Brings entrepreneurial individuals into HHS through collaboration with private and not-forprofit organizations HIMSS funded fellow working in collaboration with HHS CTOs office, IDEA LAB and the Office of the National Coordinator for Health IT Patient Matching Final Report, identified patient matching as a critical barrier to interoperability Two Year Fellowship Started August August 2016

4 High Level Goal of Interoperability

5 Simplest Model Client Server

6 Generation Storage All Pieces needed for Interoperability Structure Transport This is where standards are important Merge Governance Policy Considerations

7

8 Background

9 Significant Dates in (Patient) Matching Dunn Record Linkage Fellegi & Sunter A Theory of Record Linkage Campbell, K et al A Comparison of Link Plus, The Link King, and a Basic Deterministic Algorithm RAND Health Report Identity Crisis: An Examination of the Costs and Benefits of a Unique Patient Identifier for the US Health Care System HIMSS Patient Identify Integrity Toolkit, Patient Key Performance Indicators Winkler Matching and Record Linkage A Framework for Cross-Organizational Patient Identity Management Kho, Abel N., et al Design and Implementation of a Privacy Preserving Electronic Health Record Linkage Tool Soundex US Patent Newcombe, Kennedy, & Axford Automatic Linkage of Vital Records Grannis, et al Analysis of Identifier Performance Using a Deterministic Linkage Algorithm Grannis, et al Privacy and Security Solutions for Interoperable Health Information Exchange HIMSS Patient Identity Integrity Audacious Inquiry and ONC Patient Identification and Matching Final Report Joffe et al A Benchmark Comparison of Deterministic and Probabilistic Methods for Defining Manual Review Datasets in Duplicate Records Reconciliation Dusetzina, Stacie B., et al Linking Data for Health Services Research: A Framework and Instructional Guide HIMSS hires Innovator In Residence (IIR) focused on Patient Matching

10 Patient Matching Definition Patient matching: Comparing data from multiple sources to identify records that represent the same patient. In Healthcare involves matching varied demographic fields from different health databases to create a unified view of a patient.

11 Identity Matching / Identity Resolution Identity analysis: link analysis, data mining Structured and unstructured data sources Identity resolution: Merge/dedupe records Identity matching Measure record similarity. Search/retrieval Identity data repository Attribute matching Compare name, DOB, COB, address, etc.

12 Challenges in Matching

13 Challenges Data availability Data Quality Lack of adoption of metrics

14 Availability of Data Attributes

15 % Availability of Attributes Over Region % 90.00% 80.00% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% Site B Site A Site C 0.00% First Name Middle Name Last Name Date of Birth Birth Year Gender Social Security Number Address (full) Street Address Line 1 City State Postal Code Country Abbreviation Country Full Name Phone Number (any) Home Phone Number Cell Phone Number Work Phone Number Address Nickname Insurance Number (free text) Drivers License Number Race (OMB) Race (free text) Ethnicitiy Language Occupation Income Marital Status Height (cm) Height (m) Height (in) Height (ft) Weight (lbs) Weight (kg) Blood Type

16

17

18 Data Quality

19 Data Quality Data Quality is a Key Garbage in and Garbage out Data entry errors are compound data matching complexity Various algorithmic solutions to address these, not perfect Types of errors: Missing or Incomplete Values Inaccurate data Fat finger errors Information is out of date Transposed names Misspelled names

20 Data Quality Transposition errors Mary Sue vs Sue Marie Smith, John vs John, Smith Names change over time Marriage, Divorce More than one way to spell name Jon, John Data entry Fat-finger = typo, transposition, etc. Phonetic variation

21 Metrics for Algorithm Performance

22 Patient Matching Goal Ideal outcome of any matching exercise is correctly answering this one question hundreds or thousands of times, Are these two things the same thing? Correctly identifying all the true positives and true negatives while minimizing the number of errors, false positives and false negatives

23 Patient Matching Terminology True Positive- The two records represent the same patient True Negative- The two records don't represent the same patient

24 Patient Matching Terminology False Negative: The algorithm misses a record that should be matched False Positive: The algorithm creates a link to two records that don t actually match

25 Evaluation EHR A EHR B Truth (Gold Standard) Algorithm Match Type Jonathan Jonathan Match Match True Positive Good Jonathan Sally Non-Match Non-Match True Negative Jonathan Sally Non-Match Match False Positive Bad Jonathan Jon Match Non-Match False Negative

26 Evaluation EHR A EHR B Truth (Gold Standard) Algorithm Match Type Jonathan Jonathan Match Match True Positive Jonathan Sally Non-Match Non-Match True Negative Jonathan Sally Non-Match Match False Positive Bad Jonathan Jon Match Non-Match False Negative

27 Evaluation EHR A EHR B Truth (Gold Standard) Algorithm Match Type Jonathan Jonathan Match Match True Positive Jonathan Sally Non-Match Non-Match True Negative Jonathan Sally Non-Match Match False Positive Bad Jonathan Jon Match Non-Match False Negative

28 Evaluation Precision = True Positives / (True Positives + False Positives) Truth Positive Negative Algorithm Positive True Positive False Positive Precision Negative False Negative True Negative Recall Recall = True Positives / (True Positives + False Negatives)

29 Evaluation Calculation Precision = True Positives / (True Positives + False Positives) Recall = True Positives / (True Positives + False Negatives) Tradeoffs between Precision and Recall F Measure

30 Summary Patient matching is old problem Need to understand data attributes available Understand their quality Follow a systematic approach to evaluation Methodology to create ground truth data Metrics Precision Recall

31 Contact Information Adam W. Culbertson : aculbertson@himss.org

32 Questions?

33 Backup

34 Creating Test Data Sets

35 Development of Test Data Set Patient Database Select Potential Matches (aka Adjudication Pool) Manual Reviewer 1 Manual Reviewer 2 Manual Reviewer 3 Human-Reviewed Match Decisions (Answer Key == Ground Truth Data Set) Compare Algorithm and Test Data Set

36 Development of Ground Truth Sets Identify data set that reflects real word use case Develop potential duplicates Human adjudication review and classification Match or Non-Match Estimate truth Pooled methods using multiple matching methods

37 Issues In Establishing Ground Truth Examples B Smith Bill Smythe William Smythe W Smith??

38 Activity: Patient Names

39 Patient Names (Answers) Leigh Cramer Jean Rimbaud (OK, or John.) Alice Slawson I don t know what your neighbors names are but did you get them right? did you get the *whole* name right? Leah /li.ɑ/ /le.ɑ/ /li/ Lay Lie Leigh /lei / /lai / Laye Quoi? Li Lee Ligh

40 Identity Matching Adjudication Collector (IMAC) User Interface One screen of the Adjudication Collector continually provides questions to the adjudicator which need to be answered. These screens first ask the question with no dates provided and then again asks the question with dates shown.

41 Issues In Establishing Ground Truth Different truth for different applications Credit check Security applications Customer support De-duplication of mailing lists What is the cost of missing a match? New record entered into database Irritated customer Lives are lost Criteria for truth must be carefully established and wellunderstood by annotators Question posed to annotators must be carefully phrased

42 Issues In Establishing Ground Truth How much time / expertise is available to judge (/discount) false positives? Needs to reflect real word test use case Evaluation results are only as good as the truth on which they are based And only as appropriate as the evaluation is to the task that will be performed with the operational system Absolute recall impossible to measure without completely known test set (i.e. You don t know what you re missing. ) Estimate with pooled results

43 Issues In Establishing Ground Truth First step in evaluation is to determine why the evaluation is being conducted Different truth for different applications Security Applications vs Patient Health Record What is the cost of missing a match? Security: Lives are lost Health: Patient safety event, missed medications, allergies, etc death But this is situation today. What is the cost of wrongly identifying a match? Security : Passenger is inconvenienced / delayed Health: Patient safety event, wrong medication, treatment, liability, death Criteria for truth must be carefully established and wellunderstood E.g. Question posed to annotators must be carefully phrased Summary for Healthcare Use Case

44 The Trade-off Between False Positive and False Negative Matches As the match score threshold is increased, the number of false positives decreases, but false negatives increase. (increasing precision) As the match score threshold is lowered, the number of false negatives decreases, but false positives increase (increasing recall) Source: Grannis, S. Introduction to Record Linkage. September 27, 2012

45 Basic IR Metrics: Precision and Recall Target List : Subject : MAHMOUD ABDUL HAMEED 12/10/1945 Precision (P) = X/Y (2/4) Recall (R) = X/Z (2/3) System returns Y X MOREY APPLEBAUM MOHAMMED ABDUL HAMID MAHMOUD ABD EL HAMEED MAKMUD ABDUL HAMID MAHMOUD ABD ALHAMID False positives True Positives Z True Answers False negatives

46 Precision and Recall Inversely Related (1) Database System returns Recall Increased, but Precision Fell The Low Hanging Fruit phenomenon more false hits will come in for every true one True Hits

47 Precision and Recall Inversely Related (2) Database Precision Increased, but Recall Fell System returns More selective matching True Hits

48 What Makes a Good Evaluation? Objective gives unbiased results Replicable gives same results for same inputs Diagnostic can give information about system improvement Cost-efficient does not require extensive resources to repeat Understandable results are meaningful in some way to appropriate people Well-documented also contextualizes results in terms of purpose of the evaluation and task

49 IMAC Admin Interface An administrative screen allows the ability to manage IMAC users as well as manage the questions asked of users. This includes the ability to set the priority of questions and the number of judges to be used for each question.

50 Evaluation: Like IR Tasks Metrics F-measure - harmonic mean of precision and recall F = (β 2 + 1) P R / ( (β 2 P) + R) where P = precision = correct system responses / all system responses R = recall = correct system responses / all correct reference responses β = beta factor provides a mean to control the importance of recall over precisio Additional Measures False positives items that are identified as correct responses that are not correct responses (= 1 Precision) False negatives correct responses not identified (= 1 Recall) Fallout = non-relevant responses / all non-relevant reference responses (related to, but not directly calculable from precision / recall) Issue: Annotation Standard for Development of Ground Truth

51 Algorithm Tuning Large Affects on performance due to algorithm tuning Tuning is need specific Setting Cut-offs Upper Thresholds Feature Selection Feature Weighing Blocking

52 Algorithm Performance Data Quality Algorithm Tuning Algorithm

53 Framework for Evaluation: EAGLES 7-Step Recipe/ISLE FEMTI* 1. Define purpose of evaluation why doing the evaluation 2. Elaborate a task model what tasks are to be performed with the data 3. Define top-level quality characteristics 4. Produce detailed system requirements 5. Define metrics to measure requirements 6. Define technique to measure metrics 7. Carry out and interpret evaluation Originally developed as an evaluation framework for Machine Translation, but authors note that it should be able to be used as a generic evaluation framework. *Acronyms: EAGLES European Advisory Group on Language Engineering Standards ISLE International Standards for Language Engineering FEMTI Framework for the Evaluation of Machine Translation in ISLE (

54 Framework Applied to Patient Matching What is the question you are trying to answer? What data attributes do you have? What is the quality of the attributes? What is matching method do you want doing use? How effective is your matching method?