Client and Service Information (CSI) Statewide Data Quality Best Practices Plan Report

Transcription

1 Client and Service Information (CSI) Statewide Data Quality Best Practices Plan Report Issues and Proposed Solutions for CSI Data Quality The following document and project was funded by the Mental Health Services Oversight and Accountability Commission as part of Deliverable 1 within agreement 12MHSOAC025. This report was prepared by Mental Health Data Alliance, LLC. 1/2/2014

2 Table of Contents Table of Contents Table of Contents... 1 Acknowledgements... 4 Definitions and Abbreviations... 5 Executive Summary... 6 Introduction... 6 Methods... 6 Participation... 6 Findings Background Information Introduction Overview of CSS Program History of the CSI System CSI Data Collection and Reporting Report Methods Webinars Survey Site Visits CSI Survey Survey Participation CSI County Submission Status Review CSI Data Workflow CSI Data Issues and Proposed Solutions Issues Performing Data Collection in EHR System Lack of User Training Materials Data Dictionary Complexity Data Dictionary Definitions Loss of Historic Data Quality when Upgrading County EHR Staff Not Collecting All Required CSI Data Elements Appropriately Clinicians Selecting Inappropriate Codes Required Data Omitted by Staff Difficulty Collecting Data for Services Outside County Outpatient EBP Codes Require Definition or Expansion Standardize Collected Data Sets Resource Limitations CSI Submission File Generation Issues in EHR System CSI Statewide Data Quality Best Practices Plan Report Page 1

3 Table of Contents Delays in Data Submission Resulting from Upgrading EHRs Unable to Create Submission Files in New EHR EHR Submission File Creation Issues EHR Data Elements Differ from CSI Data Elements Client Identifier Longer Than CSI Permits CSI Does Not Accept DSM-5 Codes Counties and Vendors Unclear on ICD-10 and DSM-5 Implementation EHR Software Issues EHR Unable to Track Submitted Services CSI Submission File Pre-checked for Errors Reporting to Providers Existing Pre-Checks not exhaustive CSI Pre-Checks not Yielding Correct Results CSI Submission File Uploaded to ITWS ITWS Website or CSI Submission System Inaccessible Large Files Cannot be Uploaded Counties Don t Understand the Utility of Reporting CSI Data CSI Error File Retrieved and Reviewed CSI Error File No Longer Providing Errors Uploading Staff not Trained on Error Review Summarized Error Reports not Provided Error File Difficult to Review and Interpret Identify Incorrect Data State Provider Master File not Up-To-Date Diagnosis Code Issues CSI Data Access for Counties Aggregated Data for Counties Historical Errors not Purged from Error File Insufficient Support from the State Collect Correct Data Enter Correct Data into EHR System Submit Corrected Data to ITWS Sequencing Issues Resulting in Data Correction Rejections Recommendations and Best Practices Data Collected in EHR System County Best Practices Vendor Best Practices State Best Practices CSI Statewide Data Quality Best Practices Plan Report Page 2

4 Table of Contents 4.2 CSI Submission File Generated by EHR System County Best Practices Vendor Best Practices State Best Practices CSI Submission File Pre-checked for Errors County Best Practices Vendor Best Practices State Best Practices CSI Submission File Uploaded to ITWS State Best Practices CSI Error File Retrieved and Reviewed County Best Practices State Best Practices Identify Incorrect Data County Best Practices State Best Practices Collect Correct Data Enter Correct Data into EHR System Submit Corrected Data to ITWS State Best Practices Conclusions Appendix A: Selected Survey Results... Appendix B: Example Survey... CSI Statewide Data Quality Best Practices Plan Report Page 3

5 Acknowledgements Acknowledgements We would like to acknowledge the input and materials from various people and organizations. We value the participation, input and feedback from State, county, provider, vendor and additional stakeholder staff who participated in the webinars and survey. We would especially like to thank members of the State of California staff who participated in the webinars to support the counties efforts to ensure accurate data exists in the Client Services Information (CSI) system. We would also like to thank the many county staff that raised issues and proposed solutions, which are incorporated into this document. In particular, we would like to thank the CSI staff at Orange, Tehama, and Lake Counties for facilitating on-site visits for contractors to observe existing CSI processes and/or for sharing documentation and tools for the purposes of assisting other counties to improve data quality statewide. CSI Statewide Data Quality Best Practices Plan Report Page 4

6 Definitions and Abbreviations Definitions and Abbreviations Term CCN CCR CDS CSI DCR DIG DHCS DMH DSM EBP EDI SSIS EHR FSP HIPAA HUD HMIS ICD IT IMD LCSW MFCC MHP MHSA MHSOAC OHC RN SDA State TOC XML Meaning County Client Number California Code of Regulations Client Discharge System Client Services Information Data Collection and Reporting Data Infrastructure Grant Indicator Department of Health Care Services California Department of Mental Health Diagnostic and Statistical Manual of Mental Disorders Evidence Based Practice Electronic Data Interchange SQL Server Integration Services Electronic Health Record Full Service Partnership Health Insurance Portability and Accountability Act Department of Housing and Urban Development Homeless Management Information Systems International Classification of Diseases Information Technology Institution for Mental Diseases Licensed Clinical Social Workers Marriage, Family and Child Counselors Mental Health Plan Mental Health Services Act Mental Health Services Oversight and Accountability Commission Other Health Coverage Registered Nurse Statistics and Data Analysis The State of California Table of Contents Extensible Markup Language CSI Statewide Data Quality Best Practices Plan Report Page 5

7 Executive Summary Executive Summary Introduction The Mental Health Services Oversight and Accountability Commission (MHSOAC) has sponsored a project to review the existing data submission processes for the Client and Service Information (CSI) system statewide through a series of webinars with counties to develop a report of the data reporting workflow processes, current issues with reporting structures, and best practices. Methods Information gathered and summarized in this report was obtained through a multimodal qualitative and quantitative collection approach (e.g., phone conferences, webinars, surveys, site visits and focus groups). A forefront and stakeholder driven approach was used to engage and obtain feedback from providers, counties, vendors and state representatives. Issues and best practices identified by webinar participants, survey respondents and through on-site observations were documented and presented in this report. Suggestions for remediation were included for all identified issues where possible. Participation Staff from at least 26 of the 58 counties participated in the webinar series. A total of 46 surveys were received representing 28 counties. Site visits were conducted at 2 counties. Findings As a result of insufficient documentation, training materials, and support from the State regarding CSI system (see Section 3.1.1, Section 3.1.2, Section and Section 3.6.6) in combination with identified, reported and ongoing barriers to data entry (see Section 3.1.9, Section 3.5.1, Section 3.6.1, Section 3.6.2), counties feel unsupported in the CSI submission process, and as a result counties are challenged to justify putting staff resources towards CSI data submission and data quality because the support available from the State is not adequate to sustain reliable data quality. In order to demonstrate that the State places a high value on the timely and accurate submission of CSI data, the State should place an emphasis on addressing barriers to data entry and should provide appropriate documentation, training materials and support for the CSI system. As shown in Section 1.7, a significant number of counties (25.8 percent of counties) are not currently in compliance with the 60-day reporting requirement. As a result, effective and timely data evaluation for some county-wide and statewide indicators cannot be done for recent CSI Statewide Data Quality Best Practices Plan Report Page 6

8 Executive Summary months. Therefore, it is recommended that any analysis performed on CSI data be conducted no earlier than 6 months prior to the current submission status date (e.g., any analysis for calendar year 2012 should be done using data CSI data exported on or after 6/1/2013) and should take into consideration missing data for counties significantly behind the 60-day reporting requirement. The State, counties and providers should direct attentions to bringing all counties into compliance with the 60-day reporting requirement. It is recommended that the State create documentation and a process to assist counties and vendors with continuity when counties change EHR solutions as this is the most common cause of significant data submission delays. The State should immediately address the following high priority items: 1. Ensure adequate, knowledgeable and timely support is available for counties in support of the CSI system (see Section 3.6.6) 2. Address the CSI Error File No Longer Providing Errors issue (Section 3.5.1) 3. Address Diagnosis Code Issues (Section 3.6.2) 4. Review State Provider Master File updates with counties (Section 3.6.1) 5. Provide guidance for the implementation of International Classification of Diseases version 10 (ICD-10) and Diagnostic and Statistical Manual of Mental Disorders version 5 (DSM-5) code sets (see Section and Section 3.2.7) 6. Consider providing a tool or functionality within ITWS to allow counties to exhaustively pre-check CSI files for errors; and the tool should be flexible enough to create reports by provider (see Section 3.3.1, Section 3.3.2, Section 3.3.3, Section 3.5.2, Section and Section 3.9.1) 7. Address the data failure issues due to sequencing (Section 3.9.1) 8. Make efforts to communicate to counties on how the CSI data is utilized or is planned to be utilized (see Section 3.4.3) The State should consider addressing the following priority items: 1. Provide enhanced Evidence Based Practice (EBP) code set definitions (see Section 3.1.9) 2. Address issues uploading large files for CSI data submission (see Section 3.4.2) 3. Enhance the format of the CSI error report messages (see Section 3.5.4) 4. Create an updated data dictionary, CSI system documentation and training curriculum materials (see Section 3.1.1, Section and Section 3.1.3) 5. Resume monthly or quarterly error report summarizations (see Section 3.5.3) 6. Consider implementing an automated method for counties to download historical CSI data (see Section 3.6.3) CSI Statewide Data Quality Best Practices Plan Report Page 7

9 Executive Summary 7. Consider implementing reporting functionality, such as a dashboard tool, to allow counties to view summary reports of aggregated data for counties for historical CSI data (see Section 3.6.4) Counties should consider addressing the following priority items: 1. Establish documented processes and training for CSI staff (see Section and Section 3.5.1) 2. Engage in periodic review of existing data collection processes (see Section 4.1.1) 3. Establish consortia based on EHR vendor to share EHR-specific documentation, training materials, and best practices; this type of cooperation provides a forum for communication regarding current EHR software issues, CSI data submission issues and pursuing collaborative funding of custom EHR software changes to support regulatory or other beneficial enhancements to EHR systems (see Section 4.1.1) 4. Compare elements in new or existing EHR systems with those required for submission to the CSI system (Section 3.2.4) 5. Identify the possibility of implementing a pre-check for submission errors for all CSI files (Section 4.3.1) 6. Give provider-specific error reports to providers to allow data correction (see Section 4.3.1) It was noted that reviewing the CSI Error File generated by the CSI system is the single most challenging activity related to CSI data submission process (see Section 4.5). It should be emphasized that the process of pre-checking the CSI Submission File data for errors based on the CSI validation rules prior to submission has been reported to be the single most important process that can be implemented to improve data quality and reduce the staff time required to perform data review (See Section 4.3). CSI Statewide Data Quality Best Practices Plan Report Page 8

10 1.0 Background Information 1.0 Background Information 1.1 Introduction The Mental Health Services Oversight and Accountability Commission (MHSOAC) is responsible for providing oversight for the Mental Health Services Act (MHSA) and its components. Within this role, the MHSOAC ensures accountability. As one of its oversight approaches, the MHSOAC has adopted a commitment to pursuing meaningful evaluation of the MHSA and greater community mental health system. The ability to successfully use evaluation methods to provide oversight and accountability within the MHSA is dependent upon access to valid data that is reliably reported and made available to the MHSOAC on a regular basis. The MHSOAC has identified areas within the current county-level and statewide data collection and reporting systems that are problematic and in need of improvement. The MHSOAC has begun to directly address some of these issues, although it is not clear that the MHSOAC was intended to provide this function. Nonetheless, the MHSOAC is committed to advocating for improvement of the current data collection and reporting systems since the MHSOAC is dependent on the information that is made available via these systems in order to fulfill the statutory role in evaluation of the public mental health system. Improvements in data collection and reporting systems will increase confidence in the information obtained and conclusions drawn regarding the state of the MHSA and the California community mental health system. As part of its oversight and accountability role, the MHSOAC conducts statewide evaluations of services funded under the Community Services and Supports (CSS) component of the Mental Health Services Act (MHSA). Counties, providers, and stakeholders need accurate and timely data that promote quality improvement efforts within the services that are being offered throughout the state. The MHSOAC also requires data on CSS component services and clients in order to fulfill part of its oversight and accountability role and to support quality improvement efforts of the mental health system and services within that system. Unfortunately, the data collection and reporting systems used for CSS services have not been properly maintained, which limits the quality of data collected and reported via these systems. MHSA data regarding client CSS services are primarily collected via the Client and Service Information (CSI) system, which is currently owned and maintained by the Department of Health Care Services (DHCS). In an effort to strengthen the data collected and reported via the CSI system, MHSOAC sponsored this effort to identify any existing problems with current CSI data collection and reporting processes. This report represents findings regarding CSI data CSI Statewide Data Quality Best Practices Plan Report Page 9

11 1.0 Background Information workflow process, identified issues, and contains a summary of best data collection and reporting practices. These findings were relayed to counties via webinar. 1.2 Overview of CSS Program Community Services and Supports (CSS) is the largest component of the MHSA. The CSS component is focused on community collaboration, cultural competence, client and family driven services and systems, wellness focus, which includes concepts of recovery and resilience, integrated service experiences for clients and families, as well as serving un-served and underserved populations. Housing is also a large part of the CSS component. According to Cal. Admin. Code title 9, , Community Services and Supports means the component of the Three-Year Program and Expenditure Plans that refers to service delivery systems for mental health services and supports for children and youth, transition age youth, adults, and older adults. These services and supports are similar to those found in Welfare and Institutions Code Sections 5800 et. seq. (Adult and Older Adult Systems of Care) and 5850 et. seq. (Children's System of Care). Mental Health Plans (MHPs) receive supplemental state-based funding for mental health services as a result of California Proposition 63 (now known as the Mental Health Services Act or MHSA), passed in November of MHSA provides increased funding to support California s county mental health programs. The MHSA imposes a one percent income tax on personal income in excess of $1 million to address a broad continuum of prevention, early intervention and service needs and the necessary infrastructure, technology and training elements that will effectively support this system, with the purpose of promoting recovery for individuals with serious mental illness. MHPs develop customized plans for mental health client service in accordance with numerous requirements, including that it must provide for significant local stakeholder input and involvement. Some community mental health services in the CSI database are CSS which are funded by MHSA, but MHPs are not currently required to identify MHSA funded services in the CSI database. CSI Statewide Data Quality Best Practices Plan Report Page 10

12 1.0 Background Information 1.3 History of the CSI System WIC 5610 required MHPs to report data to the State for non-duplicative client-based information including all information necessary to meet federal Medicaid reporting requirements, as well as any other state requirements established by law. There are 58 MHPs (56 counties + Yuba/Sutter combined counties + Tri-City), which will be referred to as counties hereafter. Counties currently report client-based information to the Client Service Information (CSI) system, which began July 1, 1998 and replaced the Client Discharge System (CDS). The CSI System collects data pertaining to mental health clients and the services they receive at the county level. A basic principle of the CSI system is that it reflects both Medi-Cal and non-medi- Cal clients and services provided in the County/City/Mental Health Plan program. In countystaffed providers, all clients and services must be reported. In contract providers, those clients and services provided under the contract with the county must be reported. This data is processed and stored on a database at DHCS. Counties send a CSI Submission File to DHCS monthly and are required to submit data no later than 60 days after the end of the month in which the services were provided. On July 1 st, 2006, the California Department of Mental Health (DMH) introduced a modified data format which included changes to the CSI system documentation, changes to the client record format (such as Race/Ethnicity and Data Infrastructure Grant Indicator (DIG)), services record format (such as Diagnosis and Evidence-Based Practices (EBP)), and the periodic record (such as Caregiver). The California Department of Mental Health (DMH), Statistics and Data Analysis (SDA) provided administrative statistical information and reports about county mental health programs using the Client and Service Information (CSI) system monthly by county mental health programs and state-level summarization from 1998 to In 2012, as a result of the dissolution of DMH, ownership of these systems was transferred to DHCS. DHCS now hosts and provides support for the CSI system and publishes statistical information and reports. These reports provide statewide utilization by various demographic variables and are available on the DHCS website. CSI Statewide Data Quality Best Practices Plan Report Page 11

13 1.0 Background Information 1.4 CSI Data Collection and Reporting CSI system collects data from counties regarding both Medi-Cal and non-medi-cal clients, and services provided in the County/City/Mental Health Plan program. In addition to CSI data collected and reported under county treatment programs, many counties elect to contract with private providers (hereafter providers ) for the provision of mental health services on behalf of the county. The CSI system also collects CSI data for all providers whose legal entities are reported to the County Cost Report under the category Treatment Program and the individual and group practitioners, most of which were formerly in the Fee-For-Service system. These practitioners are individual or group practice psychiatrists, psychologists, Licensed Clinical Social Workers (LCSW), Marriage, Family and Child Counselors (MFCC), and Registered Nurses (RN) as well as the Mixed Specialty group practices. CSI data for services rendered by providers are submitted to the county prior to being submitted to the State. Providers submit CSI data to the State in a variety of ways including by paper or fax which is manually entered into the county EHR by county staff; by provider staff connecting directly to the county EHR and performing their own data entry into the county EHR; and by providers utilizing their own EHR system capable of producing an export in CSI Submission File format generating their own data submission file which is manually appended to the county CSI Submission File for submission to the State. Prior to becoming certified to submit CSI data, the State requires that counties demonstrate the ability to successfully submit CSI Submission Files while achieving less than 5% overall rate of records with errors within the batch as measured by the CSI Batch Update Report, Processing Statistics. As a result, counties attempt to maintain a maximum 5% overall error rate per submission. However, compliance with the 5% overall error rate is not enforced on a per-batch submission basis. 1.5 Report Methods Information gathered and summarized in this report was obtained through a multimodal qualitative and quantitative collection approach (e.g., phone conferences, webinars, surveys, site visits and focus groups). A forefront and stakeholder driven approach was used to engage and obtain feedback from providers, counties, and state representatives. Issues and best practices identified by webinar participants, survey respondents and through on-site observations were documented and presented in this report. Suggestions for remediation were included for all identified issues where possible. All webinar participants and stakeholders were given an opportunity to review a draft of the report and provide feedback, and the final draft reflects participant feedback. CSI Statewide Data Quality Best Practices Plan Report Page 12

14 1.0 Background Information Webinars State, counties, providers, vendors and other stakeholders were invited to participate in five webinars to discuss the CSI data collection and submission process and related issues. Table lists the webinar topic, date the webinar was held, and the number of participant lines used. The exact number of participants for each webinar cannot be determined as each participant line connected to the webinar could represent multiple participants. Staff from at least 26 of the 58 counties participated in the webinar series. Although all stakeholders were invited to participate in all webinars, the second and third webinars in the series were focus groups specializing on specific aspects of the data collection and review process which may not have been applicable to all stakeholder staff. A draft of this report was made available to stakeholders on November 1 st, 2013 and presented in the Statewide Data Quality Best Practices Plan Draft Report Review webinar on November 21 st, Stakeholder feedback was incorporated into the final draft of this report. The final draft of this report was presented to stakeholders on December 20 th, Table 1.5.1: Webinars Conducted Webinar Topic Date Participant Lines Used Statewide Data Quality Best Practices Plan CSI Pre-Submission File Edit Focus Group Webinar CSI Error File Review and Remediation Webinar Statewide Data Quality Best Practices Plan Draft Report Review Statewide Data Quality Best Practices Plan Final Report Review Wednesday, September 4th (10AM to 12PM) Thursday, September 19 th (10AM to 12PM) Tuesday, September 24th (10AM 12PM) Thursday, November 21st (10AM to 12PM) Friday, December 20th (1PM to 3PM) Survey Counties, providers and vendors were asked to participate in a CSI System Data Quality Improvement Survey which requested information about data reporting workflow processes, current issues with reporting structures, best practices, and possible desired indicators from all stakeholders. Please see Appendix A for selected survey result summaries and Appendix B for a full example survey containing questions asked and response options (if applicable). CSI Statewide Data Quality Best Practices Plan Report Page 13

15 1.0 Background Information Site Visits On-site visits were conducted at two counties (Orange County and Tehama County on Monday, October 7 th and Wednesday, October 9 th respectively) to observe CSI data review and submission processes. CSI Statewide Data Quality Best Practices Plan Report Page 14

16 2.0 Data Collection Workflow 1.6 CSI Survey A 22-question survey was conducted in combination with the webinars to collect feedback from counties, providers and other stakeholders on current data collection workflow, issues, best practices and desirable indicators. A sample survey is included as Appendix B of this document. It was requested that as many as possible stakeholders participate in the survey; multiple survey responses per county and provider were encouraged to allow the gathering of as many perspectives as possible. A total of 46 surveys were received representing 28 counties. Free-text survey responses were reviewed and information regarding issues and best practices has been organized by data collection workflow step and included in Chapter 3.0 Data Collection Workflow of this document. Survey questions which could be aggregated (i.e. multiple choice selections) have been included in Appendix A of this document Survey Participation Survey responses were received from respondents identifying themselves as county and provider staff from the following 28 counties: Alameda Napa Santa Clara Butte Nevada Shasta Calaveras Orange Solano Glenn Riverside Stanislaus Humboldt Sacramento Tehama Kern San Bernardino Ventura Lake San Diego Yolo Los Angeles San Joaquin Tri-City Marin San Luis Obispo Merced San Mateo CSI Statewide Data Quality Best Practices Plan Report Page 15

17 2.0 Data Collection Workflow 1.7 CSI County Submission Status Review DHCS publishes a monthly CSI County Submission Status report on the public DHCS website which lists each county and the number of months that they are behind the 60-day reporting requirement. CSI County Submission Status reports for current and previous months were reviewed. As of December, 2013, 43 of 58 counties were in compliance with the regulatory 60-day CSI data reporting requirement. Of counties that did not meet the 60-day reporting requirement, shown in Figure 1.7.1, 10 of 58 counties were 1 to 3 months behind. As a result, no data exists in the CSI system for these counties for an approximate 3 to 5 month look-back period. Additionally, 5 of 58 counties did not meet the 60-day reporting requirement by more than 4 months indicating that no data exists in the CSI system for these counties for an approximate 6 month look-back period or more. For 2 of these 5 counties, no data exists in the CSI system for over a 1 year look-back period. The current County Submission Status report was analyzed to group counties into categories of counties meeting the 60-day reporting requirement, counties 1 to 3 months behind the 60-day reporting requirement, and counties that are 4 months or more behind the 60-day reporting requirement as shown in the graph of Counties by Submission Status Group in Figure The CSI County Submission Status report additionally documents the current issues prohibiting data submission for counties that are significantly behind the 60-day reporting requirement and includes DHCS efforts to assist counties in submitting data. A common theme among many of these counties is a change in Electronic Health Record (EHR) systems resulting in a disruption to the CSI data export and submission process. CSI Statewide Data Quality Best Practices Plan Report Page 16

18 2.0 Data Collection Workflow Figure County Months Late in 60-day Reporting Source: Note: The draft version of this report contained CSI Submission Status information as of October, Since that time, a significant reduction has occurred in the number of counties reported as late after the 60-day reporting requirement. In October, 2013, 56.9 percent of counties were identified as late in reporting. In December, 2013, the number of counties identified as late in reporting fell to 25.8 percent which represents a 31.1 percent reduction in counties reported as late over an approximate two month timeframe. The October CSI County Submission Status report can be found on the DHCS website at: CSI Statewide Data Quality Best Practices Plan Report Page 17

19 2.0 Data Collection Workflow Figure Counties by Submission Status Group Counties Meeting 60-day Reporting Requirement Counties 1 to 3 Months Behind 60-day Reporting Requirement Counties >4 Months Behind 60- day Reporting Requirement Submission Status Group Number of Counties Percentage of Counties Counties Meeting 60-day Reporting Requirement % Counties 1 to 3 Months Behind 60-day Reporting Requirement % Counties >4 Months Behind 60-day Reporting Requirement 5 8.6% CSI County Submission Status report reflects that a significant number of counties (25.8 percent of counties) are not currently in compliance with the 60-day reporting requirement. As a result, effective and timely data evaluation for some county-wide and statewide indicators cannot be done for recent months. At the end of the 3 rd month after the 60-day reporting window, indicating an approximate 5 month time period after the point in time of service rendered, 91.3 percent of counties were able to successfully submit CSI data. As a result, it is recommended that any analysis performed on CSI data be conducted no earlier than 6 months prior to the current submission status date (e.g., any analysis for calendar year 2012 could be done using CSI data exported on or after 6/1/2013) and analysis should take into consideration counties that have been unsuccessful in submitting CSI data for various reasons. CSI Statewide Data Quality Best Practices Plan Report Page 18

20 2.0 Data Collection Workflow 2.0 CSI Data Workflow The following section provides an introduction to CSI data workflow. Diagram 2.1 illustrates the relationship between each of the steps described in this section. Diagram 2.1: CSI Data Workflow CSI Statewide Data Quality Best Practices Plan Report Page 19

21 2.0 Data Collection Workflow CSI Data Workflow Description: 1. Data Collected in EHR System: County and provider staff collect CSI data elements and submit the data to an Electronic Health Record system, billing system or electronic data warehouse (hereafter referred to as EHR ). 2. CSI Submission File Generated by EHR System: Counties and providers export the data from the EHR in the standard CSI Submission File format defined in the CSI Data Dictionary. 3. CSI Submission File Pre-checked for Errors: Some counties have varying processes, reports and tools for CSI record validation prior to CSI file submission which allows for the identification of errors prior to submission. This pre-submission check is not performed by all counties. 4. CSI Submission File Uploaded to ITWS: Counties upload the exported CSI Submission File through the ITWS web interface. 5. CSI Error File Retrieved and Reviewed: The CSI system processes the files submitted and the system was designed such that counties are subsequently able to download the CSI Error File and the CSI Batch Update Report. County staff must review the CSI Error File and address data related issues. 6. Identify Incorrect Data: County staff must review the CSI Error File and CSI Batch Update Report, and perform investigation to identify data related issues which must be corrected and resubmitted. 7. Collect Correct Data: After incorrect data elements have been identified, county and provider staff must collect the correct data. 8. Enter Correct Data into EHR System: Once the correct data has been collected, county and provider staff must enter the correct data into the EHR system. 9. Submit Corrected Data to ITWS: After the correct information has been entered into the EHR system, the data must be re-exported from the EHR and again uploaded to the CSI system via the ITWS upload interface. Refer to Section 4.0 Best Practices later in this document for further detail and subcomponents of these nine steps. CSI Statewide Data Quality Best Practices Plan Report Page 20

22 3.0 CSI Data Issues and Proposed Solutions 3.0 CSI Data Issues and Proposed Solutions The following chapter summarizes the issues raised by county, provider and vendor CSI support staff (hereafter referred to as CSI staff ) with regard to data collection workflow and reporting structures for the CSI system. The issues are divided, categorized and assigned within one of the nine steps of the CSI data workflow as previously identified in Diagram 2.1. Each issue includes a list of the staff affected by the issue, a description of the issue, the consequence in the data accuracy due to the issue and proposed solutions. The affected include all of those who may potentially be affected by the issue, but not everyone in the affected category may experience the issue. Wherever possible, the affected group was narrowed to a subgrouping of CSI staff. When the affected group could not be narrowed, the term CSI staff is listed and suggests that any member of the CSI staff should be aware of the potential error and proposed solutions. In some instances, the proposed solutions may require further definition from the State, the county or from a collective agreement among stakeholders through future activities. CSI Statewide Data Quality Best Practices Plan Report Page 21

23 3.0 CSI Data Issues and Proposed Solutions 3.1 Issues Performing Data Collection in EHR System In this step, county and provider staff collect CSI data elements and submits the data to an EHR Lack of User Training Materials Affects: CSI staff Description: Available training material was reported to be insufficient for training CSI staff to collect and submit CSI data for both EHR and CSI systems. For example, several counties noted a lack of available training and documentation for the State CSI system. Some indicated that updated CSI documentation and training resources would be helpful in improving data quality, and others indicated CSI data quality improvement training from the State would be helpful. Consequence: Diminished EHR and CSI data quality Proposed Solution: Counties should consider creating or improving EHR-specific documentation for the collection of CSI data It is recommended that counties establish consortia based on EHR vendor to share EHR-specific documentation, training materials, and best practices; this type of cooperation provides a forum for communication regarding current EHR software issues, CSI data submission issues and pursuing collaborative funding of custom EHR software changes to support regulatory or other beneficial enhancements to EHR systems The State should consider improving available CSI training documentation The State should consider providing CSI staff training to counties to promote data quality CSI Statewide Data Quality Best Practices Plan Report Page 22

24 3.0 CSI Data Issues and Proposed Solutions Data Dictionary Complexity Affects: CSI staff Description: County and vendor staff have reported that the data dictionary is complex and difficult to understand. For example, one county indicated the CSI Data Dictionary is inaccessible due to its complex structure and the language used to describe data elements is difficult to understand. Consequence: Counties and vendors experience difficulty creating a CSI data collection and export process County and vendor staff experience difficulty researching data related issues Proposed Solution: The State should consider revising and enhancing the CSI Data Dictionary to make it more easily comprehensible Counties identified a number of useful aspects of the CSI Data Dictionary (such as the organization of fields by record type and data type summarization) so it is recommended that county feedback is solicited prior to modifications to this document Data Dictionary Definitions Affects: CSI staff Description: Counties have reported that the CSI Data Dictionary definitions of data elements are insufficient. For example, one county indicated that the available CSI data elements for demographic information do not accurately capture the way clients describe themselves. Another county indicated that there was a lack of clear, detailed, and timely instructions for new information and requirements such as the Institution for Mental Diseases (IMD) reporting. As another example, for the Location of Services data element, location codes and names are given but no definitions are provided for each option which leads to CSI staff confusion. Consequence: Counties and vendors experience difficulty creating a CSI data collection and export process Proposed Solution: The State should consider revising and enhancing the CSI Data Dictionary to further define and enhance available data elements CSI Statewide Data Quality Best Practices Plan Report Page 23

25 3.0 CSI Data Issues and Proposed Solutions Loss of Historic Data Quality when Upgrading County EHR Affects: Counties implementing or changing EHR solutions Description: When counties implement or change EHR systems, a data conversion is frequently performed to pull historical data forward into the new system. During this process, errors can be encountered which can result in data related errors or omissions. Examples of counties that have experienced issues during EHR upgrades are Lake, Amador, Solano and Sonoma counties. Consequence: Diminished EHR and CSI data quality Proposed Solution: Counties should perform a thorough review of data quality indicators before and after data conversion to ensure all data has been pulled forward to the new EHR successfully prior to going live with the new EHR The State should provide counties with the ability to pull, query or compare historical CSI data to support this process CSI Statewide Data Quality Best Practices Plan Report Page 24

26 3.0 CSI Data Issues and Proposed Solutions Staff Not Collecting All Required CSI Data Elements Appropriately Affects: Some CSI staff Description: Although the EHR provides the ability to collect all CSI data, not all counties are confident that forefront staff update all CSI data correctly and appropriately. For example, one county indicated that data entry training for forefront staff on the collection of accurate data would help achieve better CSI data quality. Another county indicated their biggest barrier to achieving good quality CSI data is staff performing incomplete or incorrect data collection. An additional county also indicated that maintaining education, motivation, and diligence of the staff gathering and entering CSI data into the EHR is their biggest barrier to achieving good quality CSI data. A county indicated that the volume of data collected poses a challenge to data quality and completeness, and a county indicated that their biggest barrier to achieving good quality CSI data is training staff to accurately and consistently populate the CSI admit screen in their EHR. Consequence: Diminished EHR and CSI data quality Proposed Solution: Counties should review their existing documentation and training programs to ensure forefront staff have adequate guidance in the collection of CSI data Counties should consider creating or improving EHR-specific documentation for the collection of CSI data It is recommended that counties establish consortia based on EHR vendor to share EHR-specific documentation, training materials, and best practices; this type of cooperation provides a forum for communication regarding current EHR software issues, CSI data submission issues and pursuing collaborative funding of custom EHR software changes to support regulatory or other beneficial enhancements to EHR systems Counties and vendors should ensure sufficient documentation and training are provided during EHR implementation Counties should periodically review data quality to ensure the collected CSI data appropriately reflects expected tends CSI Statewide Data Quality Best Practices Plan Report Page 25

27 3.0 CSI Data Issues and Proposed Solutions Clinicians Selecting Inappropriate Codes Affects: Some CSI staff Description: Clinicians do not always select the appropriate code as a result of relying on frequently selected codes instead of more accurate codes. For example, one county indicated that clinicians frequently select the wrong CSI codes or revert to frequently used codes as a result of the large number of available codes to choose from and staff s lack of time to look up each code to ensure accurate data entry. Consequence: Diminished EHR and CSI data quality Proposed Solution: Counties should review training documentation to ensure sufficient guidance is provided for CSI staff in selecting appropriate codes Required Data Omitted by Staff Affects: CSI Staff Description: When CSI data is required but the field is not a required field in the EHR system, it is occasionally not completed by CSI staff. For example, one county who uses the Echo EHR noted that not all CSI data is required within the EHR and as a result is occasionally not completed. Another county indicated that accuracy and completeness of CSI data collection has improved since required CSI fields were set to required fields in their EHR, Avatar. Consequence: Diminished EHR and CSI data quality Proposed Solution: Counties and vendors should ensure required CSI data element fields are set as required fields in the EHR data collection system where possible CSI Statewide Data Quality Best Practices Plan Report Page 26

28 3.0 CSI Data Issues and Proposed Solutions Difficulty Collecting Data for Services Outside County Outpatient Affects: Some CSI staff Description: Collecting some information is difficult for counties, such as crisis services information when the client is seen outside county outpatient facilities. For example, one county reported that the CSI data collection process can be cumbersome when clients are either not in the right state of mind to provide accurate data or when they simply refuse to provide the data, particularly for services rendered in the emergency room, jail and one-time crisis services provided in the office. Consequence: Diminished EHR and CSI data quality Proposed Solution: Enhance collaboration of counties using similar EHR solutions Counties should review existing procedures and training materials for collecting data in these circumstances The scope of services currently reported should be reevaluated; the State should clearly identify which services outside of outpatient facilities have the potential to be reasonably and accurately reported and the State should reference the authority for requesting such elements Service information which cannot be collected reliably within community mental health systems and supports should be evaluated for collection through alternative methods, such as through client surveys CSI Statewide Data Quality Best Practices Plan Report Page 27

29 3.0 CSI Data Issues and Proposed Solutions EBP Codes Require Definition or Expansion Affects: CSI staff Description: EBP codes need to be more flexible to allow for more codes than are currently available as the current code set is not comprehensive. For example, one county indicated that the list of EBP codes needs to be more expansive and codes require better definition to make ratings more accurate. Counties are currently able to add custom EBP codes but the presence of such custom EBP codes results in non-fatal errors. Consequence: Diminished EHR and CSI data quality Proposed Solution: The State should enhance available EBP codes and documentation The State should update the CSI Data Dictionary to include more expansive EBP codes; the State should collaborate with Counties to determine the set of EBP codes which should be added to the available CSI EBP code set The State should make it easier for Counties to add custom EBP codes by not generating non-fatal errors as a result of custom EBP codes Standardize Collected Data Sets Affects: All counties and providers Description: Counties have noted that there is variation in the data elements and formats between different systems (Full Service Partnership (FSP) Data Collection and Reporting (DCR) system, CSI, MHSA, Medi-Cal, Department of Housing and Urban Development (HUD) Homeless Management Information Systems (HMIS), Medicare, Other Health Coverage (OHC), etc.). Consequence: Data export complexity Increased administrative costs Diminished CSI data quality Proposed Solution: The State should adopt a strategy of standardizing to the degree possible the data elements and submission file formats The State should standardize data elements between datasets currently held by DHCS CSI Statewide Data Quality Best Practices Plan Report Page 28

30 3.0 CSI Data Issues and Proposed Solutions Resource Limitations Affects: Some counties Description: Counties have reported experiencing difficulty achieving good quality data as a result of budgetary and human resource limitations. For instance, one county reported that their biggest barrier to achieving very good quality CSI data was a lack of the internal resources needed to do this work. Consequence: Diminished CSI data quality Proposed Solution: Counties should ensure sufficient staff resources are available for the achievement of good quality data Counties should review data collection and data review procedures to ensure CSI staff are operating efficiently CSI Statewide Data Quality Best Practices Plan Report Page 29

31 3.0 CSI Data Issues and Proposed Solutions 3.2 CSI Submission File Generation Issues in EHR System In this step, CSI staff export the data from the EHR in the standard CSI Submission File format Delays in Data Submission Resulting from Upgrading EHRs Affects: Counties implementing or changing EHR solutions Description: When counties are implementing a new EHR or changing EHR solutions, there is frequently a delay in certification to submit CSI data files from the new system. As an example, one county upgraded to Avatar on July 1, 2013 but has not yet been able to generate CSI Submission Files. Consequence: A lag or cessation of EHR data export and submission to the CSI system leading to the inability to meet the 60-day CSI data reporting requirement Proposed Solution: Counties should ensure that adequate time and resources are allocated to ensuring the CSI data export and submission is successfully implemented The State should ensure adequate support is available to assist counties with EHR transitions CSI Statewide Data Quality Best Practices Plan Report Page 30

32 3.0 CSI Data Issues and Proposed Solutions Unable to Create Submission Files in New EHR Affects: Counties implementing or changing EHR solutions Description: After upgrading or changing EHR solutions, counties can experience issues generating CSI Submission Files. The specific issues vary by EHR and county. The following counties are currently addressing submission file creation issues after upgrading to a new EHR: Marin, Riverside, Sonoma, and Stanislaus. Consequence: A lag or cessation of EHR data export and submission to the CSI system leading to the inability to meet the 60-day CSI data reporting requirement. Proposed Solution: Counties should ensure that adequate time and resources are allocated to ensuring the CSI data export and submission is successfully implemented Vendors should ensure adequate support is available to assist counties with EHR transitions EHR Submission File Creation Issues Affects: Counties using Anasazi EHR Description: Survey respondents in at least one county reported that the process used in their EHR, Anasazi, can take a significant time (up to 13 days) to generate the CSI file. Consequence: Delays in data submission Proposed Solution: Counties should immediately contact the EHR software vendor to assist in addressing any issues It is recommended that counties establish consortia based on EHR vendor to share EHR-specific documentation, training materials, and best practices; this type of cooperation provides a forum for communication regarding current EHR software issues, CSI data submission issues and pursuing collaborative funding of custom EHR software changes to support regulatory or other beneficial enhancements to EHR systems CSI Statewide Data Quality Best Practices Plan Report Page 31

33 3.0 CSI Data Issues and Proposed Solutions EHR Data Elements Differ from CSI Data Elements Affects: Counties using certain EHRs Description: Some counties use an EHR that collects different data elements from the CSI data elements. As a result, the EHR data cannot be immediately exported from the EHR. Instead, the data must be exported from the EHR, inserted into another data warehouse tool to aggregate and transform the data, and then exported in the standard CSI Submission File format. Examples include Orange County s Cerner, Los Angeles County s CSI Electronic Data Interchange (EDI) SQL Server Integration Services (SSIS), and Tri-City s Welligent implementations. In some cases, such as Riverside County, EHR design can result in challenges collecting and submitting CSI data. In other cases, such as Los Angeles CSI-EDI SSIS, the EHR contains a richer set of data elements which must be crosswalked to a simpler set of codes allowable by CSI which reduces the accuracy and completeness of data submitted to CSI in comparison to the county EHR. Consequence: Counties must create the process to transform the data from the EHR format into CSI Submission File format which can be resource intensive Proposed Solution: Counties should evaluate EHR solutions to ensure CSI data collection format and submission file formats are compatible or allocate sufficient resources necessary to transform EHR data into CSI data elements It is recommended that counties establish consortia based on EHR vendor to share EHR-specific documentation, training materials, and best practices; this type of cooperation provides a forum for communication regarding current EHR software issues, CSI data submission issues and pursuing collaborative funding of custom EHR software changes to support regulatory or other beneficial enhancements to EHR systems A number of counties, including Orange County, have indicated that EHR systems which contain strong billing components to meet county needs may not have a strong CSI data collection module; as a result, there can be contention between the county s billing and CSI reporting needs. CSI Statewide Data Quality Best Practices Plan Report Page 32

34 3.0 CSI Data Issues and Proposed Solutions Client Identifier Longer Than CSI Permits Affects: Counties with client identifiers that are longer than 9 characters Description: Some counties have unique client identifiers (CSI County Client Numbers (CCNs)) that are longer than the acceptable length in the CSI Submission File (9 characters). As a result, these counties have needed to create an alternate, unique identifier specifically for the purposes of reporting CSI data. One such example is Orange County s implementation of Cerner. Consequence: Increased administrative costs and technical challenges Proposed Solution: The State should increase the possible length of the client identifier field The State should consider changing from flat file format to Extensible Markup Language (XML) to allow for greater flexibility with data elements CSI Does Not Accept DSM-5 Codes Affects: All Counties Description: The DSM-5 was published on May 18, 2013, superseding the DSM-IV. Consequence: Counties and vendors are unable to implement DSM-5 code sets Proposed Solution: The State should provide information about the transition to the DSM-5 code set On December 3, 2013, DHCS published MHSD Information Notice Number available which is posted at CSI Statewide Data Quality Best Practices Plan Report Page 33

35 3.0 CSI Data Issues and Proposed Solutions Counties and Vendors Unclear on ICD-10 and DSM-5 Implementation Affects: All Counties Description: Stakeholders unclear on the implementation plan for ICD-10 and DSM-5 code sets (required after October 1, 2014). Consequence: Counties and vendors are unable to prepare for the implementation of the new ICD-10 and DSM-5 code sets When one county inquired with the State regarding these new code sets, they were told it was suggested they submit "0s" for diagnosis codes Proposed Solution: The State should provide information about the transition to the ICD-10 and DSM-5 code sets On December 3, 2013, DHCS published MHSD Information Notice Number available which is posted at CSI Statewide Data Quality Best Practices Plan Report Page 34

36 3.0 CSI Data Issues and Proposed Solutions EHR Software Issues Affects: Counties using specific versions of various EHR solutions Description: Due to software bugs in EHR solutions, counties may be uncomfortable with data quality until these software issues are corrected by the EHR vendor. For example, one county has encountered bugs within their EHR which have not yet been addressed by the software vendor and are uncomfortable submitting data until those issues are resolved. Another county encountered problems with claims submissions resulting in discrepancies between their EHR, Avatar, and submitted CSI data which did not trigger warning or rejection messages when the data was submitted to CSI. Additional counties are also behind in CSI data submission as a result of EHR issues. Consequence: Delays in data submission Inability to comply with the 60-day CSI data submission requirement Proposed Solution: Counties should immediately contact the EHR software vendor to assist in addressing any issues It is recommended that counties establish consortia based on EHR vendor to share EHR-specific documentation, training materials, and best practices; this type of cooperation provides a forum for communication regarding current EHR software issues, CSI data submission issues and pursuing collaborative funding of custom EHR software changes to support regulatory or other beneficial enhancements to EHR systems CSI Statewide Data Quality Best Practices Plan Report Page 35

37 3.0 CSI Data Issues and Proposed Solutions EHR Unable to Track Submitted Services Affects: Counties using specific versions of various EHR solutions Description: Not all EHRs are able to track on a per-record basis which data has already been exported for submission to the CSI system. Instead, services are exported as a point-in-time snapshot of services rendered during a specified timeframe; this results in any subsequent changes made regarding services rendered during a previously submitted timeframe not being exported for propagation to the CSI system. As an example, if a client received a service in May 2013 but that service was not documented in the EHR until June 2013, after the CSI Submission File was created for May 2013, data for this service will not be included in the June 2013 submission file because the EHR does not track on a per-record basis which data has already been exported in previous CSI Submission Files. Some county EHR systems, like Anasazi, are able to track changes made after data submission which can be subsequently resubmitted as replace records, which is recommended. Consequence: This can result in delays in data submission as counties wait for staff to complete data entry for a certain period of time (frequently 30 to 60 days) before CSI data is exported for a given timeframe Any changes, additions or corrections to CSI data made after the CSI data has been exported for the specified timeframe will not be exported for CSI data submission Proposed Solution: Counties and EHR vendors should consider implementing submission status tracking on a per-record bases for CSI records CSI Statewide Data Quality Best Practices Plan Report Page 36

38 3.0 CSI Data Issues and Proposed Solutions 3.3 CSI Submission File Pre-checked for Errors In this step, some counties have varying processes, reports and tools which allow for checking for validation errors prior to CSI file submission. This pre-submission check is not performed by all counties Reporting to Providers Affects: Counties not currently giving CSI error reports to providers Description: Some providers have indicated that they do not receive CSI Submission File Pre-check error reports from counties. Consequence: Providers are not made aware of data related issues and are therefore unable to correct the data which can result in missing or inaccurate CSI data Proposed Solution: Counties should consider providing reports to providers to assist with the identification and remediation of data related issues The State should consider providing a tool or functionality within ITWS to allow counties to exhaustively pre-check CSI files for errors; and the tool should be flexible enough to create reports by provider Existing Pre-Checks not exhaustive Affects: Most counties Description: For counties that engage in pre-checking CSI Submission File data, a variety of methods are used including custom programs, reports, and manual data review. Example EHR reports include Avatar s Missing CSI Report, Anasazi s CSI Error Report, and Shasta County s Suspense Report. In many cases, these pre-checks do not exhaustively validate the submission data against all data submission rules. Consequence: Counties are unable to fully verify that the CSI data complies with all CSI validation rules prior to submission Proposed Solution: The State should consider providing a tool or functionality within ITWS to allow counties to exhaustively pre-check CSI files for errors; and the tool should be flexible enough to create reports by provider The State should create documentation and training materials to outline the business rules of the CSI system CSI Statewide Data Quality Best Practices Plan Report Page 37

39 3.0 CSI Data Issues and Proposed Solutions CSI Pre-Checks not Yielding Correct Results Affects: Counties using Anasazi EHR Description: The Anasazi CSI utility was misreporting data as incorrect but was correct in the system. It was misreporting the Transaction Type of a service as "Replace" when it should not have and was reporting the diagnosis code on the most recent dated Diagnosis Form instead of reporting the diagnosis code from the Diagnosis Form that covers the date of the service. Consequence: Counties experience difficulty achieving good data quality Proposed Solution: Counties should immediately contact the EHR software vendor to assist in addressing any issues CSI Statewide Data Quality Best Practices Plan Report Page 38

40 3.0 CSI Data Issues and Proposed Solutions 3.4 CSI Submission File Uploaded to ITWS In this step, counties upload the exported CSI Submission File through the ITWS interface ITWS Website or CSI Submission System Inaccessible Affects: All Counties Description: When information systems were transferred from DMH to DHCS, there was a period of time between October, 2012 and January, 2013 when the CSI submission system was inaccessible or unable to accept submissions. Consequence: Disruption to the data submission process Proposed Solution: The State should ensure reliability and availability of the ITWS and CSI submission systems to avoid future disruptions during system, platform or software changes The State should alert counties when the CSI system will be down and provide communication regarding anticipated lengths of downtime and processes to support county submissions after system is operational Large Files Cannot be Uploaded Affects: Los Angeles County Description: Due to the volume of data some counties must submit, the CSI system cannot process the entire batch at one time. Counties with large submission files must break up the data into more manageable files. Consequence: Difficulty in submitting large amounts of data Additional administrative time managing and splitting the upload file Proposed Solution: The State should modify the CSI system to accept larger files CSI Statewide Data Quality Best Practices Plan Report Page 39

41 3.0 CSI Data Issues and Proposed Solutions Counties Don t Understand the Utility of Reporting CSI Data Affects: CSI Staff Description: Throughout this project, numerous counties expressed the lack of clarity for the purpose and utility of reporting CSI data to the State. Counties wanted to know how the data was being used and why it was needed. Counties feel that they expend extensive time and resources to report the data and do not feel that their efforts are justified unless the data is being used. Consequence: Counties exert the minimum amount of effort required to meet the legal minimum for reporting CSI data, thus resulting in diminished data quality Counties are inclined to put their limited resources onto other projects rather than the CSI submission project, thus resulting in diminished data quality Proposed Solution: The State should make efforts to communicate to counties on how the CSI data is utilized or is planned to be utilized CSI Statewide Data Quality Best Practices Plan Report Page 40

42 3.0 CSI Data Issues and Proposed Solutions 3.5 CSI Error File Retrieved and Reviewed In this step, county staff must review the CSI Error File and address data related issues. Every night, the CSI system processes the submission files received by ITWS within the past 24 hours. The following day after CSI file submission, county staff can download the CSI Error File and the CSI Batch Update Report which identify and summarize both fatal errors and non-fatal warnings CSI Error File No Longer Providing Errors Affects: Some counties Description: Some counties have reported that they are no longer receiving recordspecific error messages in their CSI Error File after This has been confirmed with one county via an on-site visit (Tehama). Consequence: Counties must rely on the CSI Batch Update Report or other methods to identify the types of errors that exist in the data and must engage in a time consuming, manual process to review all submitted CSI data to identify and remediate submission errors without the aid of record-specific rejection errors Proposed Solution: The State should identify and resolve CSI system issues to restore individualized record error reports Note: A request was sent from MHDATA to DHCS for additional information regarding this issue; as of the time of the draft report, no response has been received CSI Statewide Data Quality Best Practices Plan Report Page 41

43 3.0 CSI Data Issues and Proposed Solutions Uploading Staff not Trained on Error Review Affects: Counties where Information Technology (IT) staff resources are responsible for submitting data to ITWS Description: In some counties, it has been reported that IT staff are responsible for submitting CSI data. The IT staff resources that upload the data may not be sufficiently trained to address warning and rejection messages particularly when the error rate is above 5%. In other cases, CSI data is submitted to the State by the software vendor and error files are not reviewed by county staff. Consequence: Diminished data quality as a result of potentially fixable problematic data records not being addressed Proposed Solution: It is recommended that counties assign responsibility for data quality to staff who have been trained on the CSI data collection, submission and remediation processes The county should identify a process for IT staff such that they can communicate with the appropriate subject matter experts to remedy the record errors; a different process may be needed for different types of errors The State should consider providing a tool or functionality within ITWS to allow counties to exhaustively pre-check CSI files for errors; and the tool should be flexible enough to create reports by provider The State should provide improved training materials, documentation, more user-friendly data dictionary, and more useful error messages CSI Statewide Data Quality Best Practices Plan Report Page 42

44 3.0 CSI Data Issues and Proposed Solutions Summarized Error Reports not Provided Affects: Some Counties Description: While the CSI system was under DMH, Brian Fischer used to produce a summarized report from the CSI Error File which made it easier for counties to identify and resolve errors. These are no longer being provided. Consequence: Counties are not able to identify data related errors as easily as they were previously Proposed Solution: The State should consider providing a tool or functionality within ITWS to allow counties to exhaustively pre-check CSI files for errors; and the tool should be flexible enough to create reports by provider The State should consider providing additional error report summaries similar to what was provided under DMH by Brian Fischer CSI Statewide Data Quality Best Practices Plan Report Page 43

45 3.0 CSI Data Issues and Proposed Solutions Error File Difficult to Review and Interpret Affects: CSI Staff Description: Counties have indicated that the CSI Error File is indecipherable and unusable for the purposes of error remediation. Other counties have reported that the process of manually reviewing the error file line by line is time consuming, difficult to understand, and difficult to train new staff to perform. As a result, many counties, such as Tehama and Lake Counties, review each data field in the submission file to ensure the correct data format and that each record follows the CSI submission rules. Consequence: Some counties make very few corrections to rejected records Counties may not address errors if the overall batch submission error rate is less than 5% Diminished CSI data quality Proposed Solution: The State should provide improved training materials, documentation, more user-friendly data dictionary, and more useful error messages The State should consider developing online curriculum to assist counties with training CSI staff to review errors The State should consider providing a tool or functionality within ITWS to allow counties to exhaustively pre-check CSI files for errors; and the tool should be flexible enough to create reports by provider The State should review the existing format of the CSI Error File to determine if it would be possible to add human-readable descriptive warning and rejection messages (such as The Axis I Primary is coded Y and the Axis II Primary is coded Y. Only one Axis diagnosis, either Axis I or Axis II, can be the Primary Diagnosis. ) Counties, including Lake County, indicated that certain aspects of the current CSI Error File are very useful; for instance, the inclusion of the entire CSI record which contained errors in combination with the error message allows counties to manually correct and resubmit data quickly; as a result, it is recommended that any changes made to the Error File format include opportunity for county feedback CSI Statewide Data Quality Best Practices Plan Report Page 44

46 3.0 CSI Data Issues and Proposed Solutions 3.6 Identify Incorrect Data In this step, CSI staff must review the CSI Error File line by line and investigate all errors in the submission file to identify data related issues that must be corrected and resubmitted State Provider Master File not Up-To-Date Affects: Counties with Providers Description: Counties, such as Stanislaus and Orange counties, have reported that periodically the CSI tables for the State Provider Master File is not up to date for their providers Consequence: Records containing valid state provider codes are rejected Proposed Solution: The State and counties should identify a process for updating the State Provider Master File on a frequent basis to ensure synchronization between master files to allow for successful data submission Diagnosis Code Issues Affects: All Counties Description: The diagnosis code sets have become outdated in the CSI system. Consequence: Records with valid diagnosis codes are rejected when those codes do not exist in the CSI diagnosis code library Valid codes must be replaced with less accurate but existing codes which are acceptable by the CSI system One county indicated they must spend 2 to 3 days cleaning the data before the CSI data is submitted as a result of this issue Proposed Solution: The State should update the diagnosis code sets and any additional code libraries that may be out of date CSI Statewide Data Quality Best Practices Plan Report Page 45

47 3.0 CSI Data Issues and Proposed Solutions CSI Data Access for Counties Affects: All Counties Description: Although there is a manual process in place for counties to request a full copy of their historical CSI submission data ad hoc, the request process can take weeks and counties do not have the automated ability to download or view historical data in the ITWS system. Consequence: Counties are unable to easily download and review previously submitted data to identify data related issues Some counties are not aware of the necessary steps for data retrieval capabilities Proposed Solution: The State should consider providing an automated process to allow users to download historical county CSI data via the ITWS website Aggregated Data for Counties Affects: All counties Description: Counties are currently unable to generate reports for aggregated CSI data in the CSI system. Consequence: Counties have no available reports for aggregated CSI data to assist with data quality monitoring Due to the excessive staff resources to self-generate reports for aggregated CSI data, counties do not invest much time in doing this Counties must rely on their EHR reporting capabilities to generate reports but are unable to compare these metrics with similar metrics for data in the CSI system Proposed Solution: The State should investigate options for providing an online interface capable of generating reports and metrics for aggregated CSI data elements The State should consider implementing advanced reporting to provide counties with useful aggregated report metrics: General Medical Condition (suggested by Tehama, Lake and Orange counties); changes in GAF and diagnosis over time at the countyand client- levels (suggested by San Bernardino County); Race and Ethnicity (suggested by Lake County) CSI Statewide Data Quality Best Practices Plan Report Page 46

48 3.0 CSI Data Issues and Proposed Solutions Historical Errors not Purged from Error File Affects: Some counties Description: Some counties have reported that historical errors are not being purged from the error file. Since the historical data is not in their current EHR, they are unable to correct these historical errors. Consequence: Counties have difficulty delineating current errors from historic errors, which inhibits data correction efforts Proposed Solution: The State should consider providing monthly error reports specifically focused on records submitted in the most recent batch submission in addition to the cumulative CSI Error File The State should consider providing a tool or functionality within ITWS to allow counties to exhaustively pre-check CSI files for errors; and the tool should be flexible enough to create reports by provider CSI Statewide Data Quality Best Practices Plan Report Page 47

49 3.0 CSI Data Issues and Proposed Solutions Insufficient Support from the State Affects: Some counties Description: Counties have reported support issues when contacting the State. Counties feel unsupported in the CSI submission process, and thus the CSI submission projects lack priority in many counties. For example, one county indicated that CSI support is frequently delayed, CSI support staff appear to lack training, and communication with support staff is often difficult to understand. One county who recently implemented Avatar and has been unable to submit CSI data since July, , indicated they have received very little support recently. Another county expressed a lack of knowledgeable State staff available to answer CSI-related questions. In some cases, it can take weeks to receive responses when submitting issues to the State and sometimes no response is provided; as an example, an was submitted to medccc@dhcs.ca.gov on 10/10/2013 regarding one county s CSI Error File not being updated after 2011 and no response has been received. Counties feel that there is no training or direction provided by the State. Outstanding issues (such as the EBP codes requiring further definition, diagnosis codes being out of date, the CSI Error File not being updated) which could be easily addressed have not been corrected. Consequence: Difficulty obtaining support from the State has resulted in delays or inability to address rejections Difficulty obtaining support from the State has resulted in reallocation of county resources onto other projects rather than the CSI submission project Proposed Solution: Counties should contact the State at medccc@dhcs.ca.gov for support related issues, and the State should provide responses within in an identified timeframe (e.g., one week) The State should hold quarterly technical assistance webinars with counties to address ongoing submission issues The State should provide improved training materials, documentation, more user-friendly data dictionary, and more useful error messages CSI Statewide Data Quality Best Practices Plan Report Page 48

50 3.0 CSI Data Issues and Proposed Solutions 3.7 Collect Correct Data In this step, county staff must collect the correct data. There were no issues reported with this portion of the data collection and submission process which can consist of a manual review of EHR and paper-based health records systems. 3.8 Enter Correct Data into EHR System In this step, CSI staff must enter the correct data into the EHR system. This process varies by EHR but there were no issues reported with this portion of the data collection and submission process. 3.9 Submit Corrected Data to ITWS In this step, after the correct information has been entered into the EHR system, the data must be re-exported from the EHR and again uploaded to the CSI system via the ITWS interface Sequencing Issues Resulting in Data Correction Rejections Affects: All counties Description: When multiple update records for a client are submitted in a single batch and some of the record updates are rejected while other subsequent update records are accepted by the CSI system for that client, attempts to resubmit the initially rejected update records result in errors due to the sequencing of updates. Consequence: Client data cannot be easily corrected Proposed Solution: The State should consider providing a tool or functionality within ITWS to allow counties to exhaustively pre-check CSI files for errors; and the tool should be flexible enough to create reports by provider The State should evaluate the possibility of enhancing the CSI system to reject all updates for a client if any updates for that client are rejected within the submission batch to allow for easier resubmission of correct data CSI Statewide Data Quality Best Practices Plan Report Page 49

51 4.0 Recommendations and Best Practices 4.0 Recommendations and Best Practices The following chapter summarizes the CSI data workflow and best practices. Workflow within the nine steps identified in Diagram 2.1 is further defined in this section. In some instances, the best practices require further definition from the State, the county or from a collective agreement of stakeholders. Wherever possible, proposed solutions to issues raised in Section 3 are integrated into the best practices workflow in this section. 4.1 Data Collected in EHR System The CSI data collection process varies significantly by county as a result of processes, involvement of providers and vendors, and implemented EHR solution. There are a number of strategies that counties, vendors and the State can pursue to support and improve data quality. Best practices discussed in this section relate to CSI Data Issues and Proposed Solutions in Section County Best Practices Counties should establish documented processes and training for CSI staff on the use of EHR systems to ensure proficiency with CSI data collection, submission and error remediation. Specific emphasis should be placed on training forefront staff on CSI data codes to ensure accurate, appropriate and complete data collection. Counties should ensure sufficient CSI staff resources are available to collect data, review data for accuracy and completeness, and remediate errors to the degree possible. Counties should engage in periodic review of existing data collection processes to identify any procedural changes that could be implemented which might improve the data collection process. This could include reviewing processes such as crisis services data collection when the client is seen outside county outpatient facilities (i.e. emergency room, jail, first time crisis services, etc.). Counties should establish consortia based on EHR vendor to share EHR-specific documentation, training materials and best practices. This type of cooperation provides a forum for communication regarding current EHR software issues, CSI data submission issues and pursuing collaborative funding of custom EHR software changes to support regulatory or other beneficial enhancements to EHR systems. When counties are selecting a new EHR system, they should evaluate the degree of correlation of data elements collected in the EHR with those required for submission to the CSI system. The process for data issue identification and remediation is greatly simplified by a high degree of correlation between the EHR and CSI data elements. Counties selecting EHR systems that do CSI Statewide Data Quality Best Practices Plan Report Page 50

52 4.0 Recommendations and Best Practices not closely mirror the CSI data elements should ensure adequate resources to develop and refine the process of translating EHR data elements to CSI data elements. When counties are implementing a new EHR, special consideration should be taken to ensure a smooth transition to the new system. Counties should address common issues reported during new EHR implementations such as data conversion issues, unique client identifier field length, CSI data export, and user documentation and training Vendor Best Practices Vendors should ensure that the data collection elements and business rules supporting CSI data collection mirror the CSI data submission requirements as closely as possible. This includes ensuring that all fields required for CSI data submission are set as required fields within the EHR where possible. In cases where the EHR and CSI do not share a high degree of correlation between collected data elements, vendors should ensure that counties are able to develop a defined process for transforming data elements into CSI data values for submission State Best Practices The State should provide to counties and vendors with a roadmap regarding upcoming regulatory library changes such as ICD-10 and DSM-5. The ICD-10 code set is the Health Insurance Portability and Accountability Act (HIPAA) adopted standard and required for reporting diagnosis for dates of service on and after October 1, The State should evaluate enhancing the available EBP codes to allow the CSI system to be more flexible and accurate use of this data element. The State should consider updating the current Data Dictionary to enhance usability and the creation of a comprehensive CSI User Manual to assist counties with CSI staff training. As part of enhancing or creating documentation, the State should solicit feedback from counties regarding data elements that require clarifications. The State should adopt a strategy of aligning the submission file formats and data elements collected by various reporting systems (including CSI, FSP DCR, Medi-Cal, HUD HMIS, Medicare, OHC, etc.) to reduce the complexity of overall regulatory data submission requirements and decrease administrative costs. CSI Statewide Data Quality Best Practices Plan Report Page 51

53 4.0 Recommendations and Best Practices 4.2 CSI Submission File Generated by EHR System The CSI data submission file generation process varies significantly by county as a result of implemented EHR solution. There are a number of strategies that counties, vendors and the State can pursue to support the CSI Submission File generation process. Best practices discussed in this section relate to CSI Data Issues and Proposed Solutions in Section County Best Practices Counties should ensure that sufficient resources are dedicated to both CSI staff and IT staff for submission file generation and submission. Counties experiencing technical issues with CSI Submission File creation within specific EHRs should immediately contact the EHR software vendor to address any possible issues with the data submission process. Counties implementing a new EHR solution should perform extensive testing of collecting CSI data elements and submission file creation prior to a production migration to the new EHR system Vendor Best Practices Vendors should collaborate with counties to ensure any software or submission file creation issues (such as described in Section 3.2.3) are addressed in a timely fashion to ensure counties are comfortable with their data quality and are able to maintain compliance with the 60-day reporting requirement. Vendors should provide counties with adequate support and guidance during transitions from one EHR solution to another. Counties commonly experience issues with data migration, CSI staff training, submission file creation and certification which can result in delays or cessation of CSI data submission. Vendors should consider implementing the ability to track the CSI services data that has been exported for submission to the CSI system on a per-record basis. Since many EHR solutions do not track service data export on a per-record basis, services are exported as a point-in-time snapshot of services rendered during a specified timeframe; this results in any subsequent changes made regarding services rendered during a previously submitted timeframe not being exported for propagation to the CSI system. By tracking service data export on a per-record basis, counties can be ensured that all data is submitted regardless of when EHR data entry occurs. Vendors should participate in county consortia to assist counties with EHR software issues, CSI data submission issues and pursuing collaborative funding of custom EHR software changes to support regulatory or other beneficial enhancements to EHR systems. CSI Statewide Data Quality Best Practices Plan Report Page 52

54 4.0 Recommendations and Best Practices State Best Practices The State should ensure adequate and timely support is available to assist counties with EHR transitions and consider putting a transition plan in place. Such a transition plan may include testing and readiness milestones to ensure a smooth transition to the new EHR system. The State should evaluate possible options for expanding the CCN to support EHR systems which have a client identifier that is greater than 9 digits. Counties that have client identifiers which are 10 or more characters long must create a secondary identifier which is complex and introduces excess complexity into the data submission process. It is important to note that changing the CCN number in the current CSI Submission File format would require all counties to modify their CSI data submission formats. Currently submitting counties with longer CCN identifiers have already developed processes to work around this issue. It is recommended that the State evaluate alternative submission file formats (such as delimited or XML) prior to changing the reporting format to ensure any new format requirements will be as flexible as possible as such a change would require extensive effort on behalf of all counties to implement. 4.3 CSI Submission File Pre-checked for Errors Although this step is not currently done by all counties, the process of pre-checking the CSI Submission File data for errors based on the CSI validation rules prior to submission has been reported to be the single most important process that can be implemented to improve data quality and reduce the staff time required to perform data review. The methods of prechecking submission files vary significantly between counties and EHR systems; some counties have developed very advanced pre-check tools and processes (such as Orange and Lake Counties) and some vendors have implemented very useful pre-check reports to assist counties with data review (such as Avatar and Anasazi). Best practices discussed in this section relate to CSI Data Issues and Proposed Solutions in Section County Best Practices Counties that do not currently have a process in place to pre-check CSI Submission Files or CSI data for validation errors prior to data submission should investigate whether their EHR currently has functionality to identify data related issues that will result in warnings and fatal errors prior to submitting data to the CSI system. For a number of EHRs, this functionality is in the form of a report such as the CSI Report or CSI Submission Report. If counties are unsure if their EHR provides this functionality, it is recommended that counties contact their software vendor for assistance. CSI Statewide Data Quality Best Practices Plan Report Page 53

55 4.0 Recommendations and Best Practices Some counties give provider-specific reports to providers which allow them to identify and remediate data related issues. This has been reported to be an effective method of improving overall data quality Vendor Best Practices Vendors should consider creating reports or other utilities to assist CSI staff with the identification of errors and warnings prior to CSI file submission State Best Practices The State should consider providing a tool to allow counties to exhaustively pre-check CSI files for errors prior to data submission to ensure successful initial data submission. 4.4 CSI Submission File Uploaded to ITWS It was reported by most counties that the process of CSI file submission to ITWS runs smoothly in general. The two issues raised were regarding a functional outage that occurred prior to and during the transition from DMH to DHCS and an ongoing issue related to submission file size. Best practices discussed in this section relate to CSI Data Issues and Proposed Solutions in Section State Best Practices The State should increase the maximum file size or address other issues which prohibit the upload of large files for larger counties (such as Los Angeles County.) The State should review the documentation of the CSI system to ensure that all necessary procedures exist to ensure the ITWS and CSI submission system reliability and availability. 4.5 CSI Error File Retrieved and Reviewed It was reported by many counties that the CSI Error File Review is the most challenging step in the CSI data collection and submission process. Best practices discussed in this section relate to CSI Data Issues and Proposed Solutions in Section County Best Practices Counties should review their current error file review process to ensure that CSI staff responsible for retrieving and reviewing the CSI Error File have been trained on the CSI and EHR data elements and have the necessary skills to effectively address error messages. In some cases, particularly for counties where IT staff not trained in EHR and CSI workflow are responsible for this step, it may be advisable to reassign responsibility for error file review and remediation to another subject matter expert. CSI Statewide Data Quality Best Practices Plan Report Page 54

56 4.0 Recommendations and Best Practices State Best Practices The State should investigate the CSI Error File No Longer Providing Errors issue reported in Section to determine if counties have not been receiving individualized record error reports. If an issue is discovered, a high priority should be placed on the remediation of this issue. The State should consider evaluating the format of the error report to determine what enhancements may make this error file more usable to CSI staff. Such an evaluation should include investigating the feasibility of adding human-readable warning and rejection messages to the CSI Error File(s) to reduce the difficulty CSI staff experience interpreting warning and rejection messages in the CSI Error File. Additionally, a number of counties have expressed difficulty in training CSI staff to review this error report. The State should consider enhancing the existing CSI system documentation and creating online training. Counties have indicated current training materials are not sufficient to facilitate adequate training. The State should consider resuming error report summarization similar to the summary reports provided under DMH by Brian Fischer. A number of counties indicated that this report was very helpful and greatly reduced the administrative costs and increased the effectiveness of CSI staff addressing errors. The State should consider providing a tool or functionality within ITWS to allow counties to exhaustively pre-check CSI files for errors; and the tool should be flexible enough to create reports by provider. See Section for more information. 4.6 Identify Incorrect Data Best practices discussed in this section relate to CSI Data Issues and Proposed Solutions in Section County Best Practices Counties should review the process and frequency of downloading the State Provider Master File to ensure it is synchronized with the master file in the CSI system State Best Practices The State should review the process and frequency of updating libraries (i.e. diagnosis codes, EBP codes, provider codes and similar libraries) which change periodically as a result of additional codes being added to the library. Numerous counties have reported that, when these libraries are out of date, records are being rejected as a result of invalid diagnosis code even though the diagnosis code is valid. As a result, CSI staff must select less accurate codes that the CSI system will accept. CSI Statewide Data Quality Best Practices Plan Report Page 55

57 4.0 Recommendations and Best Practices The State should review the process and frequency of updating the State Provider Master File; numerous counties have reported that when this master file is out of date that data is rejected even though the provider information submitted is valid. The State should consider implementing an automated method for counties to download their historical CSI data through the ITWS website for data correction and reporting purposes. The State should consider implementing reporting functionality, such as a dashboard tool, to allow counties to view summary reports for previously submitted CSI data to increase the value and utility of the data submitted to the CSI system and assist with the identification of data related issues. To allow counties to more easily review submission errors, it is recommended that the State consider implementing a monthly CSI Error File which focuses specifically on records submitted in the most recent batch submission in addition to the cumulative CSI Error File which reports all CSI data submission errors historically generated. 4.7 Collect Correct Data There were no issues reported with this portion of the data collection and submission process. 4.8 Enter Correct Data into EHR System There were no issues reported with this portion of the data collection and submission process. 4.9 Submit Corrected Data to ITWS This step requires a repetition of Steps 1-8 of the CSI Data Workflow and as a result issues reported in Sections of this document can be reencountered. Best practices unique to this step relate to CSI Data Issues and Proposed Solutions in Section State Best Practices After the correction of data within the EHR system, a barrier to data quality exists due to the sequencing issues (see Section 3.9.1). The State should consider providing a method for prechecking submission files for errors prior to writing the data to the database and should evaluate the possibility of rejecting all updates for clients within a submission file if any of updates are rejected with fatal errors to allow for easier resubmission of correct data. The State should consider purging historical errors from the CSI Error File or generating unique error files per CSI Submission File to enable CSI staff to more easily identify data-related issues with the current batch as counties have expressed difficulty delineating current errors from historic errors in the CSI Error File, which inhibits data correction efforts. CSI Statewide Data Quality Best Practices Plan Report Page 56

58 5.0 Conclusions 5.0 Conclusions As a result of insufficient documentation, training materials and support from the State regarding CSI system (see Section 3.1.1, Section 3.1.2, Section and Section 3.6.6) in combination with the identified, reported and ongoing barriers to data entry (see Section 3.1.9, Section 3.5.1, Section 3.6.1, Section 3.6.2), counties feel unsupported in the CSI submission process, and as a result counties are challenged to justify putting staff resources towards CSI data submission and data quality because the support available from the State is not adequate to sustain reliable data quality. In order to demonstrate that the State places a high value on the timely and accurate submission of CSI data, the State must place an emphasis on addressing barriers to data entry and should provide appropriate documentation, training materials and support for the CSI system. As shown in Section 1.7, a significant number of counties (25.8 percent of counties) are not currently in compliance with the 60-day reporting requirement. As a result, effective and timely data evaluation for some county-wide and statewide indicators cannot be done for recent months. Therefore, it is recommended that any analysis performed on CSI data be conducted no earlier than 6 months prior to the current submission status date (e.g., any analysis for calendar year 2012 should be done using data CSI data exported on or after 6/1/2013) and should take into consideration missing data for counties significantly behind the 60-day reporting requirement. The State, counties and providers should direct attentions to bringing all counties into compliance with the 60-day reporting requirement. It is recommended that the State create documentation and a process to assist counties and vendors with continuity when counties change EHR solutions as this is the most common cause of significant data submission delays. The State should immediately address the following high priority items: 1. Ensure adequate, knowledgeable and timely support is available for counties in support of the CSI system (see Section 3.6.6) 2. Address the CSI Error File No Longer Providing Errors issue (Section 3.5.1) 3. Address Diagnosis Code Issues (Section 3.6.2) 4. Review State Provider Master File updates with counties (Section 3.6.1) 5. Provide guidance for the implementation of ICD-10 and DSM-5 code sets (see Section and Section 3.2.7) 6. Consider providing a tool or functionality within ITWS to allow counties to exhaustively pre-check CSI files for errors; and the tool should be flexible enough to create reports by provider (see Section 3.3.1, Section 3.3.2, Section 3.3.3, Section 3.5.2, Section and Section 3.9.1) CSI Statewide Data Quality Best Practices Plan Report Page 57

59 5.0 Conclusions 7. Address the data failure issues due to sequencing (Section 3.9.1) 8. Make efforts to communicate to counties on how the CSI data is utilized or is planned to be utilized (see Section 3.4.3) The State should consider addressing the following priority items: 1. Provide enhanced Evidence Based Practice (EBP) code set definitions (see Section 3.1.9) 2. Address issues uploading large files for CSI data submission (see Section 3.4.2) 3. Enhance the format of the CSI error report messages (see Section 3.5.4) 4. Create an updated data dictionary, CSI system documentation and training curriculum materials (see Section 3.1.1, Section and Section 3.1.3) 5. Resume monthly or quarterly error report summarizations (see Section 3.5.3) 6. Consider implementing an automated method for counties to download historical CSI data (see Section 3.6.3) 7. Consider implementing reporting functionality, such as a dashboard tool, to allow counties to view summary reports of aggregated data for counties for historical CSI data (see Section 3.6.4) Counties should consider addressing the following priority items: 1. Establish documented processes and training for CSI staff (see Section and Section 3.5.1) 2. Engage in periodic review of existing data collection processes (see Section 4.1.1) 3. Establish consortia based on EHR vendor to share EHR-specific documentation, training materials, and best practices; this type of cooperation provides a forum for communication regarding current EHR software issues, CSI data submission issues and pursuing collaborative funding of custom EHR software changes to support regulatory or other beneficial enhancements to EHR systems (see Section 4.1.1) 4. Compare elements in new or existing EHR systems with those required for submission to the CSI system (Section 3.2.4) 5. Identify the possibility of implementing a pre-check for submission errors for all CSI files (Section 4.3.1) 6. Give provider-specific error reports to providers to allow data correction (see Section 4.3.1) It was noted that reviewing the CSI Error File generated by the CSI system is the single most challenging activity related to CSI data submission process (see Section 4.5). It should be emphasized that the process of pre-checking the CSI Submission File data for errors based on the CSI validation rules prior to submission has been reported to be the single most important CSI Statewide Data Quality Best Practices Plan Report Page 58

60 5.0 Conclusions process that can be implemented to improve data quality and reduce the staff time required to perform data review (See Section 4.3). CSI Statewide Data Quality Best Practices Plan Report Page 59

61 Appendix A: CSI Statewide Data Quality Best Practices Plan Survey Appendix A: Selected Survey Results Figure 1: CSI Monitoring and Evaluation Usefulness CSI Statewide Data Quality Best Practices Plan Survey Page 1

62 Appendix A: CSI Statewide Data Quality Best Practices Plan Survey Figure 2: EHR Accuracy and Completeness CSI Statewide Data Quality Best Practices Plan Survey Page 2

63 Appendix A: CSI Statewide Data Quality Best Practices Plan Survey Figure 3: CSI Data Accuracy CSI Statewide Data Quality Best Practices Plan Survey Page 3

64 Appendix A: CSI Statewide Data Quality Best Practices Plan Survey Figure 4: Recent Data Submission Issues CSI Statewide Data Quality Best Practices Plan Survey Page 4

65 Appendix A: CSI Statewide Data Quality Best Practices Plan Survey Figure 5: Rating CSI Staff Training CSI Statewide Data Quality Best Practices Plan Survey Page 5

66 Appendix A: CSI Statewide Data Quality Best Practices Plan Survey Figure 6: Rating Ability to Identify Incorrect CSI Data CSI Statewide Data Quality Best Practices Plan Survey Page 6

67 Appendix A: CSI Statewide Data Quality Best Practices Plan Survey Figure 7: EHR Comparison Reports CSI Statewide Data Quality Best Practices Plan Survey Page 7

68 Appendix A: CSI Statewide Data Quality Best Practices Plan Survey Figure 8: Custom CSI Data Quality Reports CSI Statewide Data Quality Best Practices Plan Survey Page 8

69 Appendix A: CSI Statewide Data Quality Best Practices Plan Survey Figure 9: EHR Penetration and Retention Rates CSI Statewide Data Quality Best Practices Plan Survey Page 9