DATA QUALITY Information Session

Transcription

1 DATA QUALITY Information Session 5/7/2014 Data Quality 1

2 Overview Data Quality Plan Agenda HMIS P&P Participation Required Data Elements Program Descriptor, Universal, & Program-Specific Data Collection & Upload Standards Timeliness, Completeness, & Accuracy Data Quality Monitoring HMIS Reports and the Evaluation Score Create your own Data Quality Plan Data Quality 2

3 Learning Objectives At the end of this training, you will be able to: Define data quality Identify HUD required data elements Know how data quality affects reports Enhance annual evaluation score Address upload validation errors Implement your own data quality plan 5/7/2014 Data Quality 3

4 Overview U.S. Department of Housing and Urban Development (HUD) The NYC Continuum of Care (CoC) Homeless Management Information System (HMIS) NYC Department of Homeless Services (DHS) HMIS Data Quality refers to the extent that data accurately reflects information in the real world. 5/7/2014 Data Quality 4

5 Overview- HMIS Data Quality To record the most complete, accurate and timely information in order to draw reasonable conclusions about the extent of homelessness and the impact of homeless services 5/7/2014 Data Quality 5

6 Overview- Purpose of Data Quality For clarity of report analysis Reliability and validity of HMIS Data Funding applications To understand homelessness in NYC Trends, demographics & tracking progress To measure programs success Housing stability, supportive services, target populations and client eligibility 5/7/2014 Data Quality 6

7 Overview- Responsibility Outreach Teams Intake Staff Case Manager(s) Dedicated Data Quality Leader(s) Sometimes known as Quality Assurance unit(s) Executive Directors (CHO Executing Officers) NYC CoC Steering Committee NYC CoC HMIS Lead Agency (NYC DHS) 5/7/2014 Data Quality 7

8 Overview- Considerations Does your data tell the whole, most accurate, story of your clients? Program? Agency? Or your community? Do you trust the data within reports generated? Do you use reports to make decisions? Or gauge performance? Can you explain the data to others? 5/7/2014 Data Quality 8

9 Overview- Summary Activity Think about your organization and what the current data quality plan is: Who reviews data prior to monthly uploads? Who uploads the data to the NYC HMIS? Who reviews the quarterly reports? Who makes revisions to the data, if needed? Who updates each client s record annually? What can be done to improve the data quality? How can you get more involved? 5/7/2014 Data Quality 9

10 HMIS Policies and Procedures S7.1 GOAL = to receive complete, accurate, and timely data uploaded to the NYC HMIS, for documenting the CoC needs, performance, and services provided to the homeless. S7.2 HMIS Participation Thresholds S7.4 Data Collection and Upload Standards S7.4.1 Timeliness S7.4.2 Completeness S7.4.3 Accuracy S7.3 Minimum Required Data Elements S7.5 Data Quality Monitoring S7.5.1 Data Quality reporting S7.5.2 Remediation 5/7/2014 Data Quality 10

11 Participation Thresholds The NYC CoC aspires to have 100% of all projects primarily dedicated to serving homeless persons participate in HMIS. Those funded by the following are required to participate: ESG, CoC (SHP, S+C), SRO Mod Rehab, HOPWA, GDP, VA Community Contract, SSVF, and SAMHSA PATH projects. HMIS Policies & Procedures Appendix C. Participation Agreement (signed by each agency) 5/7/2014 Data Quality 11

12 HMIS Policies and Procedures S7.1 GOAL = to receive complete, accurate, and timely data uploaded to the NYC HMIS, for documenting the CoC needs, performance, and services provided to the homeless. S7.2 HMIS Participation Thresholds S7.4 Data Collection and Upload Standards S7.4.1 Timeliness S7.4.2 Completeness S7.4.3 Accuracy S7.3 Minimum Required Data Elements S7.5 Data Quality Monitoring S7.5.1 Data Quality reporting S7.5.2 Remediation 5/7/2014 Data Quality 12

13 Minimum Required Data Elements Program Descriptor Data Elements Universal Data Elements Program Specific Data Elements HMIS 5/7/2014 Data Quality 13

14 Minimum Required Data Elements PROGRAM DESCRIPTOR DATA ELEMENTS *From the HMIS Data Standards; Revised Notice- March /7/2014 Data Quality 14

15 NYC HMIS- Agency Program Information PROGRAM DESCRIPTOR DATA ELEMENTS DATA QUALITY: Data must be gathered, entered, uploaded & updated in HMIS properly! Impact reports for HUD funding applications! NYC CoC nationwide scoring! Quick Reference Card AWARDS Agency Program Information 5/7/2014 Data Quality 15

16 Minimum Required Data Elements PROGRAM DESCRIPTOR DATA ELEMENT: PROGRAM TYPE For permanent housing, NYC COC are all Permanent Supportive Housing (PSH) Choose your Grant Program Type! HPRP may only be chosen by SSVF Programs. ESG Programs must pick 1 or have separate Projects for each Type. *From the 2013 HUD HMIS CSV Format Documentation- Revised Specs.v3.03 5/7/2014 Data Quality 16

17 Minimum Required Data Elements *From the HMIS Data Standards; Revised Notice- March 2010 NOTE: Option to select NA; Not Applicable. Important to have a consistent response (matched) with the NOFA Application. 5/7/2014 Data Quality 17

18 Minimum Required Data Elements Program Descriptor Data Elements Universal Data Elements Program Specific Data Elements HMIS 5/7/2014 Data Quality 18

19 Minimum Required Data Elements UNIVERSAL DATA ELEMENTS HUD # Data Element Program Applicability Subjects When Collected Initial Entry Every Entry Exit 3.1 Name All Programs All Persons in the Household X 3.2 Social Security Number All Programs All Persons in the Household X 3.3 Date of Birth All Programs All Persons in the Household X 3.4 Race All Programs All Persons in the Household X 3.5 Ethnicity All Programs All Persons in the Household X 3.6 Gender All Programs All Persons in the Household X 3.7 Veteran Status All Programs All Adults in Household X 3.8 Disabling Condition All Programs All Persons in the Household X 3.9 Residence Prior to Program Entry All Programs All Adults in Household & Unaccompanied Youth 3.10 Zip Code of Last Permanent Address All Programs All Adults in Household & Unaccompanied Youth X X 3.11 Housing Status All Programs All Persons in the Household X X 3.12 Program Entry Date All Programs All Persons in the Household X 3.13 Program Exit Date All Programs All Persons in the Household X 3.14 Unique Person Identification Number All Programs All Persons in the Household System Generated X 3.15 Household Identification Number All Programs All Persons in the Household System Generated X *From the HMIS P&P v.2.1 Appendix I_NYC HMIS Data Dictionary- May, /7/2014 Data Quality 19

20 Minimum Required Data Elements UNIVERSAL DATA ELEMENT: VETERAN STATUS 2015 Initiative to end veterans homelessness within NYC CoC *From the HMIS P&P v.2.1 Appendix I_NYC HMIS Data Dictionary- May, /7/2014 Data Quality 20

21 Minimum Required Data Elements UNIVERSAL DATA ELEMENT: DISABLING CONDITION Required for Chronic Homeless status *From the HMIS P&P v.2.1 Appendix I_NYC HMIS Data Dictionary- May, /7/2014 Data Quality 21

22 Minimum Required Data Elements UNIVERSAL DATA ELEMENT: RESIDENCE PRIOR TO PROGRAM ENTRY Eligibility (ie. if coming from TH to PH, the client is not chronically homeless) *From the HMIS P&P v.2.1 Appendix I_NYC HMIS Data Dictionary- May, /7/2014 Data Quality 22

23 Minimum Required Data Elements Program Descriptor Data Elements Universal Data Elements Program Specific Data Elements HMIS 5/7/2014 Data Quality 23

24 Minimum Required Data Elements PROGRAM -SPECIFIC DATA ELEMENTS HUD # Data Element Program Applicability Subjects At or Near Entry At Least Every 3 Months During Enrollment When Collected At Least Once Annually During Enrollment 4.1 Income and Sources All Programs All Persons in the Household X X X 4.2 Non-Cash Benefits All Programs All Persons in the Household X X X 4.3 Physical Disability All Programs All Persons in the Household X X X 4.4 Developmental Disability All Programs All Persons in the Household X X X 4.5 Chronic Health Condition All Programs All Persons in the Household X X X 4.6 HIV/AIDS All Programs All Persons in the Household X X X 4.7 Mental Health All Programs All Persons in the Household X X X 4.8 Substance Abuse All Programs All Persons in the Household X X X 4.9 Domestic Violence All Programs All Adults & Unaccompanied Youth 4.10 Destination All Programs All Persons in the Household X X Every Exit Every Contact (as applicable) *From the HMIS P&P v.2.1 Appendix I_NYC HMIS Data Dictionary- May, Date of Contact Street Outreach Programs 4.12 Date of Engagement Street Outreach Programs 4.13 Financial Assistance Provided 4.14 Housing Relocation and Stabilization Services Provided ESG programs, PATH programs ESG programs, PATH programs All Persons in the Household X X X X All Persons in the Household X All Persons in the Household X X All Persons in the Household X X 5/7/2014 Data Quality 24

25 Minimum Required Data Elements PROGRAM- SPECIFIC DATA ELEMENTS: INCOME & SOURCES *From the HMIS P&P v.2.1 Appendix I_NYC HMIS Data Dictionary- May, /7/2014 Data Quality 25

26 Minimum Required Data Elements PROGRAM-SPECIFIC DATA ELEMENTS: NON-CASH BENEFITS *From the HMIS P&P v.2.1 Appendix I_NYC HMIS Data Dictionary- May, /7/2014 Data Quality 26

27 Minimum Required Data Elements PROGRAM SPECIFIC DATA ELEMENTS: DESTINATION HUD views a chosen destination type of permanent housing to be a favorable outcome. These include #s 3,10,11, 19, 20, 21, 22, 23 (APR - Q29) #s Vs Consider destinations (i.e. family or friends ) to be permanent unless otherwise stated by client NOTE: Local Model Safe Haven (similar to emergency shelter) differs from HUD s definition of Safe Haven (rare response, only 5 SH in NYC CoC ) 5/7/2014 Data Quality 27

28 Minimum Required Data Elements Program Descriptor Data Elements Universal Data Elements Program Specific Data Elements HMIS 5/7/2014 Data Quality 28

29 Activity (Word Search) 5/7/2014 Data Quality 29

30 HMIS Policies and Procedures S7.1 GOAL = to receive complete, accurate, and timely data uploaded to the NYC HMIS, for documenting the CoC needs, performance, and services provided to the homeless. S7.2 HMIS Participation Thresholds S7.4 Data Collection and Upload Standards S7.4.1 Timeliness S7.4.2 Completeness S7.4.3 Accuracy S7.3 Minimum Required Data Elements S7.5 Data Quality Monitoring S7.5.1 Data Quality reporting S7.5.2 Remediation 5/7/2014 Data Quality 30

31 Data Collection & Upload Standards TIMELINESS: The purpose of timeliness is to ensure access to data when it is needed: Proactively: Planning or monitoring purposes To meet reporting requirements Reactively: In response to a request for information In response to inaccurate information 5/7/2014 Data Quality 31

32 Data Collection & Upload Standards TIMELINESS: Data Uploads to NYC HMIS Data Warehouse: by the 10 th business day of the following month corrections must be resolved no later than the 20 th business day of the following month Data Entry within 3 business days of client interaction: Entry Date, Exit Date, and Contact or Service 3 mo. & 1 yr. Enrollment Date Learning of a material change (active clients) 5/7/2014 Data Quality 32

33 Data Collection & Upload Standards TIMELINESS (UPLOADS): Notification via that upload was missed Upload every month (even if notification received) Annual minimum of 9 monthly uploads required NYC CoC s Upload Report runs on the 11 th business day! Available on Select HMIS>Data Warehouse Click NYC CoC Upload Report button 5/7/2014 Data Quality 33

34 Data Collection & Upload Standards COMPLETENESS: The purpose of completeness is to ensure sufficient data on clients, their demographic characteristics, and service use to facilitate confident reporting and analysis on the extent and characteristics of homelessness, including: Unduplicated counts of clients served in the CoC Patterns of use of people entering and exiting Evaluate effectiveness of homeless systems 5/7/2014 Data Quality 34

35 Data Collection & Upload Standards COMPLETENESS: Quarterly evaluation by HMIS Lead Calculated as a percentage of all required data fields for active clients during the quarter Overall percentage of required data fields Missing (No Data), Refused, or Don t Know Expectation = No Missing Data Client s Response (not Case Manager s opinion) 5/7/2014 Data Quality 35

36 Data Collection & Upload Standards 5/7/2014 Data Quality 36

37 Data Collection & Upload Standards COMPLETENESS: Program Specific Data Elements (Room for Improvement) 5/7/2014 Data Quality 37

38 Data Collection & Upload Standards COMPLETENESS: NOTE: Entries of Don t Know should be based on clients responses! (not that the case worker does not know the answer) 5/7/2014 Data Quality 38

39 Data Collection & Upload Standards COMPLETENESS: 5/7/2014 Data Quality 39

40 Data Collection & Upload Standards DATA QUALITY REPORT (QUARTERLY ): Three Program Descriptor Data Elements Program; Agency; HUD Program Type Total Clients; Adults; Unacc. Children; Leavers All 15 Universal Data Elements First 10 Program Specific Data Elements # & % Don t Know/Refused Missing Data 5/7/2014 Data Quality 40

41 Data Collection & Upload Standards DATA QUALITY REPORT (QUARTERLY ): OLD= zip file of each Program (.pdf) NEW= spreadsheet for each Agency (.xls) 5/7/2014 Data Quality 41

42 Data Collection & Upload Standards DATA QUALITY REPORT (QUARTERLY ): High Percentage (i.e. >10%) = Red flags! Ex. Line 2 Physical Disability = 95% Missing Data * From the FY CoC Program NOFA 5/7/2014 Data Quality 42

43 Data Collection & Upload Standards ACCURACY: The purpose of accuracy is to ensure that the data housed in the NYC CoC HMIS is the best possible representation of reality as it relates to homeless people and the projects that serve them 5/7/2014 Data Quality 43

44 Data Collection & Upload Standards ACCURACY: Contributing HMIS Organization (CHO): Accuracy is determined by assessing the truthfulness by the client, the accuracy of the data collected by staff, and the accuracy of the data entered into the system by staff HMIS: Accuracy is assessed by verifying consistency across all forms of reporting 5/7/2014 Data Quality 44

45 Data Collection & Upload Standards ACCURACY = consistency across HMIS Reports: Annual Performance Report (APR) Point-In-Time (PIT) Report Housing Inventory Count (HIC) Report Notice of Funding Availability (NOFA) Annual Homeless Assessment Report (AHAR) Annual Evaluation of Grantees 5/7/2014 Data Quality 45

46 Data Collection & Upload Standards ACCURACY: Annual Performance Report A reporting tool that HUD uses to track program progress and accomplishments of grantees on an annual basis. Competition grantees are required to submit an Annual Performance Report (APR) electronically to HUD, via e-snaps every operating year. All data collection on the APR is aligned with the most recent version of the HMIS Data Standards. 5/7/2014 Data Quality 46

47 Data Collection & Upload Standards ACCURACY: Annual Performance Report (APR) Q.7 Data Quality NOTE: If client record data (i.e. income) is not updated at least 1X annually & uploaded, it will appear as missing data on the APR! 5/7/2014 Data Quality 47

48 Data Collection & Upload Standards ACCURACY: Point-in-Time (PIT) Report Provides a count of sheltered and unsheltered homeless persons Counts are further broken down into subpopulation categories counts of persons who are chronically homeless, persons with severe mental illness, chronic substance abusers, Veterans, persons with HIV/AIDS, and victims of domestic violence. 5/7/2014 Data Quality 48

49 Data Collection & Upload Standards ACCURACY: Point-In-Time (PIT) Report Households with at least 1 Adult & 1 Child Households without Children Veteran Households (above 2 criteria) Households with only Children (under 18) Homeless Subpopulations 5/7/2014 Data Quality 49

50 Data Collection & Upload Standards ACCURACY: Housing Inventory Count (HIC) Collects information about all of the beds and units in each Continuum of Care homeless system, categorized by Provider Program Types For example, Year Round Beds/Units - data gathered on ALL program types 5/7/2014 Data Quality 50

51 Data Collection & Upload Standards ACCURACY: Housing Inventory Count (HIC) HIC = Beds for ALL project types PIT = only ES & TH Program Descriptor Data Elements 5/7/2014 Data Quality 51

52 Data Collection & Upload Standards ACCURACY: Notice of Funding Availability (NOFA) HUD's homeless assistance programs are categorized as either formula (non-competitive) and competitive. HUD awards Emergency Solutions Grants by formula to states and local governments. It awards the Continuum of Care Program and other competitive funds via a national competition through a Notice of Funding Availability (NOFA). 5/7/2014 Data Quality 52

53 Data Collection & Upload Standards ACCURACY: Notice of Funding Availability (NOFA) 5/7/2014 Data Quality 53

54 Data Collection & Upload Standards ACCURACY: Notice of Funding Availability 5/7/2014 Data Quality 54

55 Data Collection & Upload Standards ACCURACY: Notice of Funding Availability 5/7/2014 Data Quality 55

56 Data Collection & Upload Standards ACCURACY: Notice of Funding Availability Assisting project participants to increase income is one way to ensure housing stability and decrease the possibility of returning to homelessness. This table is completed using APR Data. 5/7/2014 Data Quality 56

57 Data Collection & Upload Standards ACCURACY: Notice of Funding Availability 5/7/2014 Data Quality 57

58 Data Collection & Upload Standards ACCURACY: Annual Homeless Assessment Report To the U.S. Congress on the extent and nature of homelessness in America Prepared by HUD and provides nationwide estimates of homelessness demographic characteristics of homeless persons, service use patterns, and the capacity to house homeless persons based primarily on HMIS data about persons who experience homelessness during a 12-month period 5/7/2014 Data Quality 58

59 Data Collection & Upload Standards ACCURACY: Annual Homeless Assessment Report The 10 major city CoCs with the largest homeless populations in the country accounted for nearly 30% of the nation s homeless population 5/7/2014 Data Quality 59

60 Data Collection & Upload Standards ACCURACY: 2014 Evaluation Tool Program Indicator Source Max Points Utilization Rate APR 6 Chronically Homeless HMIS & Self 3 Length of Stay HMIS 3 Income, Employment, Benefit Outcomes APR 12 Housing Stabilization APR 8 Data Quality HMIS 4 Minimum 8 Uploads Per Yr -5 Participation Agreement -5 5/7/2014 Data Quality 60

61 Activity (Fill In the Blank) 5/7/2014 Data Quality 61

62 HMIS Policies and Procedures S7.1 GOAL = to receive complete, accurate, and timely data uploaded to the NYC HMIS, for documenting the CoC needs, performance, and services provided to the homeless. S7.2 HMIS Participation Thresholds S7.4 Data Collection and Upload Standards S7.4.1 Timeliness S7.4.2 Completeness S7.4.3 Accuracy S7.3 Minimum Required Data Elements S7.5 Data Quality Monitoring S7.5.1 Data Quality reporting S7.5.2 Remediation 5/7/2014 Data Quality 62

63 Monitoring DATA QUALITY REPORTING: Quarterly HMIS Data Quality Reports for review by: Agency (via ) Data Management Committee NYC CoC Steering Committee Public (via website Address issues prior to the next upload 5/7/2014 Data Quality 63

64 REMEDIATION: The Data Management Committee will review and provide recommendations, such as: Additional Training Monitoring Performance and Quality Improvement (PQI) Process To improve individual program performance, improve overall NYC CCoC performance, and align funding so that the CCoC supports the strongest programs that best achieve HUD performance goals and meet consumer needs 5/7/2014 Data Quality 64

65 HMIS Policies and Procedures S7.1 GOAL = to receive complete, accurate, and timely data uploaded to the NYC HMIS, for documenting the CoC needs, performance, and services provided to the homeless. S7.2 HMIS Participation Thresholds S7.4 Data Collection and Upload Standards S7.4.1 Timeliness S7.4.2 Completeness S7.4.3 Accuracy S7.3 Minimum Required Data Elements S7.5 Data Quality Monitoring S7.5.1 Data Quality reporting S7.5.2 Remediation 5/7/2014 Data Quality 65

66 Data Upload to HMIS HMIS Policies and Procedures: Appendix C. Participation Agreement Appendix B. Data Warehouse End User Agreement Upload Instructions 5/7/2014 Data Quality 66

67 Data Upload to HMIS HMIS Policies and Procedures: Submit Appendix F. Project Information Form to Nicketa Nusum at Prior to uploading a new project Prior to renaming projects in HMIS Prior to revising Project Type (ie. TH to PH) Prior to consolidating uploads Required Data Elements Form available on website 5/7/2014 Data Quality 67

68 Data Upload to HMIS Successful Upload = Import is Complete Message within AWARDS (NYC HMIS) inbox NOTE: in progress, is NOT yet complete 5/7/2014 Data Quality 68

69 Data Upload to HMIS Non-AWARDS Users (.CSV) Validate for Import Critical errors must be corrected prior to import Import Import error report identifies excluded records Confirm via Messages>Inbox that import is complete If errors relate to individual clients, upload will continue & exclude those problematic records 5/7/2014 Data Quality 69

70 Data Upload to HMIS VALIDATION ERRORS: Fixing validation errors will eliminate many data discrepancies before the data is uploaded 5/7/2014 Data Quality 70

71 Create Your DQ Plan Agency/Program-Level Data Quality Plan: Identify responsibilities and parties involved Specify benchmarks for timeliness, completeness, and accuracy Procedures for implementation and monitoring progress Establish a timeframe for implementing the plan Regularly monitor data quality 5/7/2014 Data Quality 71

72 Create Your DQ Plan Without good information, it is difficult to assess a client s need and determine appropriate level of services for each homeless individual or family A plan that sets data quality expectations will help case managers better understand the importance of working with their clients to gather timely, complete, and accurate data 5/7/2014 Data Quality 72

73 Create Your DQ Plan For example, knowing whether a client has served in the military helps case managers make appropriate referrals and alerts them to specific benefits the client may be eligible to receive Benefits could help client become permanently housed If the case manager does NOT know the veterans status of a client, this piece of their story is missing 5/7/2014 Data Quality 73

74 Create Your DQ Plan Looking at a particular data element and assessing how many client records have blank or missing data helps analyze how reliable the data is The more clients with incomplete information, the less valid the data is to making generalizations about the population served 5/7/2014 Data Quality 74

75 Common Errors 5/7/2014 Data Quality 75

76 Common Errors 5/7/2014 Data Quality 76

77 Summary 5/7/2014 Data Quality 77