ONC HIT Certification Program Test Results Summary for 2014 Edition EHR Certification

Transcription

1 ONC HIT Certification Program Part 1: Product and Developer Information 1.1 Certified Product Information Product Name: Clinicians Gateway Product Version: Domain: Ambulatory Test Type: Modular EHR 1.2 Developer/Vendor Information Developer/Vendor Name: Krassons, Inc. Address: 1300 Clay Street, Suite 600 Oakland, CA Website: Phone: (888) Developer/Vendor Contact: David Platton Part 2: ONC-Authorized Certification Body Information 2.1 ONC-Authorized Certification Body Information ONC-ACB Name: InfoGard Laboratories, Inc. Address: 709 Fiero Lane Suite 25 San Luis Obispo, CA Website: Phone: (805) ONC-ACB Contact: Adam Hardcastle This test results summary is approved for public release by the following ONC-Authorized Certification Body Representative: Adam Hardcastle ONC-ACB Authorized Representative EHR Certification Body Manager Function/Title Signature and Date 2/24/ InfoGard. May be reproduced only in its original entirety, without revision 1

2 2.2 Gap Certification The following identifies criterion or criteria certified via gap certification (a)(1) (a)(17) (d)(5) (d)(9) (a)(6) (b)(5)* (d)(6) (f)(1) (a)(7) (d)(1) (d)(8) *Gap certification allowed for Inpatient setting only No gap certification 2.3 Inherited Certification The following identifies criterion or criteria certified via inherited certification (a)(1) (a)(14) (c)(3) (f)(1) (a)(2) (a)(15) (d)(1) (f)(2) (a)(3) (a)(16) Inpt. only (d)(2) (f)(3) (a)(4) (a)(17) Inpt. only (d)(3) (f)(4) Inpt. only (a)(5) (b)(1) (d)(4) (f)(5) Optional & (a)(6) (b)(2) (d)(5) Amb. only (a)(7) (b)(3) (d)(6) (f)(6) Optional & (a)(8) (b)(4) (d)(7) Amb. only (a)(9) (b)(5) (d)(8) (g)(1) (a)(10) (b)(6) Inpt. only (d)(9) Optional (g)(2) (a)(11) (b)(7) (e)(1) (g)(3) (a)(12) (c)(1) (e)(2) Amb. only (g)(4) (a)(13) (c)(2) (e)(3) Amb. only No inherited certification 2016 InfoGard. May be reproduced only in its original entirety, without revision 2

3 Part 3: NVLAP-Accredited Testing Laboratory Information Report Number: R-0012 V1.3 Test Date(s): June 11, March 23, 2015 Location of Testing: InfoGard and Vendor Site 3.1 NVLAP-Accredited Testing Laboratory Information ATL Name: InfoGard Laboratories, Inc. Accreditation Number: NVLAP Lab Code Address: 709 Fiero Lane Suite 25 San Luis Obispo, CA Website: ehr@infogard.com Phone: (805) ATL Contact: Milton Padilla For more information on scope of accreditation, please reference Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory Representative: Milton Padilla ONC-ACB Authorized Representative EHR Test Body Manager Function/Title Signature and Date 2/24/ Test Information Additional Software Relied Upon for Certification Additional Software Applicable Criteria Functionality provided by Additional Software No additional software required 2016 InfoGard. May be reproduced only in its original entirety, without revision 3

4 3.2.2 Test Tools Test Tool Cypress eprescribing Validation Tool HL7 CDA Cancer Registry Reporting Validation Tool HL7 v2 Electronic Laboratory Reporting (ELR) Validation Tool HL7 v2 Immunization Information System (IIS) Reporting Valdiation To HL7 v2 Laboratory Restults Intervace (LRI) Validation Tool HL7 v2 Syndromic Surveillance Reporting Validation Tool Transport Testing Tool Direct Certificate Discovery Tool No test tools required Version Test Data Alteration (customization) to the test data was necessary and is described in Appendix [insert appendix letter ] No alteration (customization) to the test data was necessary Standards Multiple Standards Permitted The following identifies the standard(s) that has been successfully tested where more than one standard is permitted Criterion # (a)(8)(ii)(a)(2) (a)(13) (a)(15)(i) (a)(16)(ii) (b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context- Aware Information Retrieval (Infobutton) Domain (g) Network Time Protocol Version 3 (RFC 1305) Standard Successfully Tested (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context- Aware Information Retrieval (Infobutton) Domain (b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide (j) HL7 Version 3 Standard: Clinical Genomics; Pedigree (b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide (g) Network Time Protocol Version 4 (RFC 5905) 2016 InfoGard. May be reproduced only in its original entirety, without revision 4

5 (i) The code set specified at 45 CFR (a)(3) IHTSDO SNOMED CT (b)(2)(i)(a) (c)(2) (ICD-10-CM) for the International Release July 2012 indicated conditions and US Extension to SNOMED CT March 2012 Release (i) (a)(3) The code set specified at 45 CFR IHTSDO SNOMED CT (b)(7)(i) (c)(2) (ICD-10-CM) for the International Release July 2012 indicated conditions and US Extension to SNOMED CT March 2012 Release (e)(1)(i) Annex A of the FIPS Publication List encryption and hashing algorithms (e)(1)(ii)(a)(2) Common MU Data Set (15) (g) Network Time Protocol Version 3 (RFC 1305) (g) Network Time Protocol Version 4 (RFC 5905) (e)(3)(ii) Annex A of the FIPS Publication List encryption and hashing algorithms (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(2) The code set specified at 45 CFR (a)(5) (HCPCS and CPT- 4) None of the criteria and corresponding standards listed above are applicable Newer Versions of Standards The following identifies the newer version of a minimum standard(s) that has been successfully tested Newer Version Applicable Criteria No newer version of a minimum standard was tested Optional Functionality Criterion # Optional Functionality Successfully Tested (a)(4)(iii) Plot and display growth charts Receive summary care record using the standards (b)(1)(i)(b) specified at (a) and (b) (Direct and XDM Validation) (b)(1)(i)(c) Receive summary care record using the standards specified at (b) and (c) (SOAP Protocols) Transmit health information to a Third Party using the (b)(2)(ii)(b) standards specified at (a) and (b) (Direct and XDM Validation) Transmit health information to a Third Party using the (b)(2)(ii)(c) standards specified at (b) and (c) (SOAP Protocols) 2016 InfoGard. May be reproduced only in its original entirety, without revision 5

6 (f)(3) Common MU Data Set (15) Common MU Data Set (15) PCS) No optional functionality tested Ambulatory setting only Create syndrome-based public health surveillance information for transmission using the standard specified at (d)(3) (urgent care visit scenario) Express Procedures according to the standard specified at (b)(3) (45 CFR (a)(4): Code on Dental Procedures and Nomenclature) Express Procedures according to the standard specified at (b)(4) (45 CFR (c)(3): ICD Edition Certification Criteria* Successfully Tested Criteria # Version Version Criteria # TP** TD*** TP** TD*** (a)(1) (c)(3) (a)(2) (d)(1) (a)(3) (d)(2) 1.5 (a)(4) (d)(3) 1.3 (a)(5) (d)(4) 1.3 (a)(6) (d)(5) (a)(7) (d)(6) (a)(8) 1.3 (d)(7) 1.2 (a)(9) (d)(8) (a)(10) (d)(9) Optional (a)(11) 1.3 (e)(1) (a)(12) (e)(2) Amb. only (a)(13) (e)(3) Amb. only (a)(14) (a)(15) 1.2 (f)(1) (f)(2) (a)(16) Inpt. only (f)(3) (a)(17) Inpt. only (f)(4) Inpt. only (b)(1) (f)(5) Optional & (b)(2) Amb. only (b)(3) (f)(6) Optional & (b)(4) Amb. only (b)(5) 1.4 (g)(1) (b)(6) Inpt. only (g)(2) (b)(7) (g)(3) 1.3 (c)(1) (g)(4) 1.2 (c)(2) *For a list of the 2014 Edition Certification Criteria, please reference (navigation: 2014 Edition Test Method) **Indicates the version number for the Test Procedure (TP) ***Indicates the version number for the Test Data (TD) 2016 InfoGard. May be reproduced only in its original entirety, without revision 6

7 Clinical Quality Measures* Type of Clinical Quality Measures Successfully Tested: Ambulatory Inpatient No CQMs tested *For a list of the 2014 Clinical Quality Measures, please reference (navigation: 2014 Clinical Quality Measures) Ambulatory CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version 2 v v v v v v v v v Inpatient CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version InfoGard. May be reproduced only in its original entirety, without revision 7

8 3.2.8 Automated Numerator Recording and Measure Calculation Automated Numerator Recording Automated Numerator Recording Successfully Tested (a)(1) (a)(9) (a)(16) (b)(6) (a)(3) (a)(11) (a)(17) (e)(1) (a)(4) (a)(12) (b)(2) (e)(2) (a)(5) (a)(13) (b)(3) (e)(3) (a)(6) (a)(14) (b)(4) (a)(7) (a)(15) (b)(5) Automated Numerator Recording was not tested Automated Measure Calculation Automated Numerator Recording Successfully Tested (a)(1) (a)(9) (a)(16) (b)(6) (a)(3) (a)(11) (a)(17) (e)(1) (a)(4) (a)(12) (b)(2) (e)(2) (a)(5) (a)(13) (b)(3) (e)(3) (a)(6) (a)(14) (b)(4) (a)(7) (a)(15) (b)(5) Automated Measure Calculation was not tested Attestation Attestation Forms (as applicable) Appendix Safety-Enhanced Design* A Quality Management System** B Privacy and Security C *Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (b)(3), (b)(4) **Required for every EHR product 2016 InfoGard. May be reproduced only in its original entirety, without revision 8

9 Appendix A: Safety Enhanced Design 2016 InfoGard. May be reproduced only in its original entirety, without revision 9

10 EHR Usability Test Report of Clinician s Gateway, v3.6.0 Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports Clinician s Gateway v3.6.0 Date of Usability Test: June 18, 2014 Date of Report: February 7, 2015 Report Prepared By: Krassons, Inc. David Platton dplatton@krassons.com 1300 Clay Street, Suite 600 Oakland, CA EXECUTIVE SUMMARY A usability test of Clinician s Gateway, v3.6.0, Modular, was conducted on June 18, 2014 in Marin County, California by Krassons, Inc. The purpose of this test was to test and validate the usability of the current user interface, and provide evidence of usability in the EHR Under Test (EHRUT). During the usability test, 4 healthcare providers and 2 Systems Administrators matching the target demographic criteria served as participants and used the EHRUT in simulated, but representative tasks. This study collected performance data on 5 tasks typically conducted on an EHR: Computerized Provider Order Entry [ (a)(1)] Medication Lists [ (a)(6)] Medication Allergy Lists [ (a)(7)] Clinical Decision Support [ (a)(8)] Medication Reconciliation [ (b)(4)] During the [time, e.g., xx minute] one on one usability test, each participant was greeted by the administrator; they were instructed that they could withdraw at any time. Participants [had / did not have] prior experience with the EHR.4 The administrator introduced the test, and instructed participants to complete a series of tasks (given one at a time) using the EHRUT. During the testing, the administrator timed the test and, along with the data logger(s) recorded user performance data on paper and electronically. The administrator did not give the participant assistance in how to complete the task. INTRODUCTION The EHRUT(s) tested for this study was Clinician s Gateway v Designed to present medical information to healthcare providers in behavioral health settings, the EHRUT consists of a web based interface utilizing a SQL Server database, which is used on a day to day basis to track patient information. The usability testing attempted to represent realistic exercises and conditions.

11 The purpose of this study was to test and validate the usability of the current user interface, and provide evidence of usability in the EHR Under Test (EHRUT). To this end, measures of effectiveness, efficiency and user satisfaction, such as time to complete task, were captured during the usability testing. METHOD PARTICIPANTS Intended users for the product are Behavioral Health Professionals and Systems Administrators in Behavioral Health settings. This would include Nurses, Psychiatrists, and Medical Doctors with an emphasis in Behavioral Health, or who practice primarily in a Behavioral Health setting. A total of 6 participants were tested on the EHRUT(s). Participants in the test were Behavioral Health Professionals and Systems Administrators with some degree of familiarity with Clinician s Gateway. Participants were recruited by Krassons. In addition, participants had no direct connection to the development of or organization producing the EHRUT(s). Participants were not from the testing or supplier organization. Participants were given the opportunity to have the same orientation and level of training as the actual end users would have received. Recruited participants had a mix of backgrounds and demographic characteristics conforming to the recruitment screener. The following is a table of participants by characteristics, including demographics, professional experience, computing experience and user needs for assistive technology. Participant names were replaced with Participant IDs so that an individual s data cannot be tied back to individual identities. Gender Age Education Occupation/role Professional Experience Computer Experience Product Experience 1 F 50+ College Psychiatric Nurse Practicioner 20 years 2/5 4/5 2 F College Clinical RN 6 years 3/5 3/5 3 M 50+ College Systems Admin 30+ years 5/5 5/5 4 F 50+ College Medical Doctor 35+ years 4/5 4/5 5 M College Systems Admin 5 years 3/5 4/5 6 M 50+ College Psychiatrist 38 years 3/5 2/5 6 participants (matching the demographics in the section on Participants) were recruited and 6 participated in the usability test. No participants failed to show for the study. Participants were scheduled for 45 minute sessions with 15 minutes in between each session for debrief by the administrator(s) and data logger(s), and to reset systems to proper test conditions. STUDY DESIGN Overall, the objective of this test was to uncover areas where the application performed well that is, effectively, efficiently, and with satisfaction and areas where the application failed to meet the needs of the participants. The data from this test may serve as a baseline for future tests with an updated version of the same EHR and/or comparison with other EHRs provided the same tasks are used. In short, this testing serves as both a means to record or benchmark current usability, but also to identify areas where improvements must be made.

12 During the usability test, participants interacted with [number of] EHR(s). Each participant used the system in the same location, and was provided with the same instructions. The system was evaluated for effectiveness, efficiency and satisfaction as defined by measures collected and analyzed for each participant: Number of tasks successfully completed within the allotted time without assistance Time to complete the tasks Number and types of errors Path deviations Participant s verbalizations (comments) Participant s satisfaction ratings of the system Additional information about the various measures can be found in the Section on Usability Metrics. TASKS A number of tasks were constructed that would be realistic and representative of the kinds of activities a user might do with this EHR, including: For Clinical Users: Add a pending CBC lab order for today for a Client (a)(1) Add ibuprofen 800mg with a sig of 1 tablet by mouth every 6 hours for a patient (a)(6) Add an allergy to Aspirin with a reaction finding of Nausea and moderate severity for a patient (a)(7) Search for a client, view their Face Sheet, then add a medication for Heparin Sodium, Porcine 10 Unit/mL Injectable with any sig, (a)(8) confirming thereafter that the patient has an intervention for Heparin. Integrate a transition of care for a patient and process a medication allergy reconciliation for the patient, moving all externally sourced (b)(4) allergies to the internal list. For Systems Admin Users: Administer Clinical Decision Support by inactivating an intervention for Heparin (a)(8) Administer Clinical Decision Support by changing visibility rights for particular interventions, then logging in as users with appropriate (a)(8) credentials to confirm Intervention visibility. Tasks were selected based on their frequency of use, criticality of function, and those that may be the most troublesome for users. PROCEDURES Participants were instructed to perform the tasks (see specific instructions below): As quickly as possible making as few errors and deviations as possible. Without assistance; administrators were allowed to give immaterial guidance and clarification on tasks, but not instructions for use. Without using a think aloud technique.

13 For each task, the participants were given a written copy of the task. Task timing began once the administrator finished reading the question. The task time was stopped once the participant indicated they had successfully completed the task. Following the session, the administrator gave the participant the post test and thanked each individual for their participation. TEST LOCATION The test facility included a quiet testing room with a table and computer for the participant. Only the participant and administrator were in the test room. To ensure that the environment was comfortable for users, noise levels were kept to a minimum with the ambient temperature within a normal range. All of the safety instruction and evacuation procedures were valid, in place, and visible to the participants. TEST ENVIRONMENT The EHRUT would be typically be used in a healthcare office or facility. In this instance, the testing was conducted in a quiet out of the way office within a behavioral health clinic. For testing, the computer used a 1.7Ghz processor with 4GB of RAM running Windows 7 Professional. The participants used a mouse and keyboard when interacting with the EHRUT. The EHRUT used a 15 display with a resolution of 1366px by 768px. The application was set up by Krassons. The application itself was running on a laptop using a test database on a WAN connection. Technically, the system performance (i.e., response time) was representative to what actual users would experience in a field implementation. Additionally, participants were instructed not to change any of the default system settings (such as control of font size). TEST FORMS AND TOOLS The participant s interaction with the EHRUT was captured and recorded digitally with screen capture software running on the test machine. A video camera recorded each participant s facial expressions synced with the screen capture, and verbal comments were recorded with a microphone. USABILITY METRICS According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, EHRs should support a process that provides a high level of usability for all users. The goal is for users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To this end, metrics for effectiveness, efficiency and user satisfaction were captured during the usability testing. The goals of the test were to assess: 1. Effectiveness of Clinician s Gateway by measuring participant success rates and errors 2. Efficiency of Clinician s Gateway by measuring the average task time and path deviations 3. Satisfaction with Clinician s Gateway by measuring ease of use ratings RESULTS DATA ANALYSIS AND REPORTING The usability testing results for the EHRUT are detailed below (see Table). The data should yield actionable results that, if corrected, yield material, positive impact on user performance.

14 Effectiveness, Efficiency, and Ease of Use / Satisfaction Clinical Users: 1. Add a pending CBC lab order for today for a Client 2. Add ibuprofen 800mg with a sig of 1 tablet by mouth every 6 hours for a patient 3. Add an allergy to Aspirin with a reaction finding of Nausea and moderate severity for a patient 4. Search for a client, view their Face Sheet, then add a medication for Heparin Sodium, Porcine 10 Unit/mL Injectable with any sig, confirming thereafter that the patient has an intervention for Heparin. 5. Integrate a transition of care for a patient and process a medication allergy reconciliation for the patient, moving all externally sourced allergies to the internal list Systems Admin Users: 6. Administer Clinical Decision Support by inactivating an intervention for Heparin 7. Administer Clinical Decision Support by changing visibility rights for particular interventions, then logging in as users with appropriate credentials to confirm Intervention visibility Success Deviation Time # Mean (SD) (Observed / Optimal) Mean (SD) 4 4 4/ s (14.86s) 4 4 4/ s (9.91s) 4 4 4/ s (36.34s) 4 4 4/4 80s (21.21s) 4 4 4/ s (15.61s) 2 2 2/2 35.5s (12.02s) 2 2 2/2 53.5s (0.71s) Time Deviation (Observed / Optimal) Time Optimal Errors Mean (SD) Ratings (5=Very Hard) Mean (SD) 48.75/48 48s (0.577) 60.25/60 60s (0.577) 69.75/69 69s (0.5) 80/80 80s 0 2 (0.816) 101.5/ s (0.5) 35.5/35 35s 0 1 (0) 53.5/53 53s (0.71)

15 DISCUSSION OF THE FINDINGS The findings show that the majority of tasks were executed within the optimal timeframe, and that the users rated the majority of the tasks as 1 (Very Easy) or 2 (Easy). The most difficult task was Task #5, which was also estimated to be the longest optimal time. This is a new process to the system, and will likely require some modification to streamline performance. It s worth noting here that no deviations were taken during the testing procedure by the participants, nor were there any recorded failures or errors. All of the participants had some degree of familiarity with the system, and were therefore accustomed to how to perform certain tasks within the system. EFFECTIVENESS Having all successful trials with no errors or deviations, the clearest conclusion is that the system is highly effective in bringing users to whatever procedural resolution they need with relative ease. However, more data points are needed to conclude this with any certainty. Given that all of the participants had some experience with Clinician s Gateway, it cannot be ruled out easily that all of the participants were simply familiar with any eccentricities inherent in the system. EFFICIENCY Task times fell within the expected optimal time for all but one task which, combined with no deviations in timing, suggests that the steps required to execute procedures are highly efficient. Since execution steps and timing are similarly predicated on some degree of experience with the system, further information would be helpful. However, the results are nevertheless clear: the system appears to efficiently allow users to complete their procedures. SATISFACTION User satisfaction and ease of use ratings indicate that the system is highly usable and the users are satisfied with the way the system works. Procedure #5 ranks as the hardest to use, which left the users somewhat less than completely satisfied with the system. Apart from that individual procedure, however, the overall satisfaction rating is quite high. MAJOR FINDINGS Overall the findings show that the system is effective, efficient, and the users are largely satisfied with the system. Overall task times were below the optimal task times, deviations were non existent, and user satisfaction was, in general, quite high. Several participants remarked on the ease of use, especially of new features (Procedures 4 7), verbally to the test administrator after testing. The AREAS FOR IMPROVEMENT The results indicate a few areas of improvement. One large area of improvement is on the efficiency and effectiveness of the system surrounding procedure #5. Although the timing fell within the optimal timing, even with one standard deviation, it was far and away listed as the hardest procedure. While it s true that this is the most involved procedure, the participants were instructed not to rate on duration, but only ease of use. Another area for improvement is extended testing with either a larger set of participants or more participants who don t have any experience with Clinician s Gateway to ensure that the general lack of errors and deviations are indeed representative of the system, and not an artifact of population.

16 Appendix B: Quality Management System 2016 InfoGard. May be reproduced only in its original entirety, without revision 10

17

18 Appendix C: Privacy and Security 2016 InfoGard. May be reproduced only in its original entirety, without revision 11

19

20

21 Test Results Summary Document History Version Description of Change Date V1.0 V1.1 Initial release Updated Safety-Enhanced Design report March 25, 2015 February 24, 2016 END OF DOCUMENT 2016 InfoGard. May be reproduced only in its original entirety, without revision 12