ONC HIT Certification Program Test Results Summary for 2014 Edition EHR Certification. Modular EHR

Transcription

1 Test Results Summary for 2014 Edition EHR Certification R-0089-PRI Vl.O, December 24, 2014 ONC HIT Certification Program Test Results Summary for 2014 Edition EHR Certification lnfoigard Part 1: Product and Developer Information 1.1 Certified Product Information Product Name: OnCallData Product Version: 5.0 Domain: Inpatient Test Type: Modular EHR 1.2 Developer/Vendor Information Developer/Vendor Name: lnstantdx, LLC Address: 9801 Washingtonian Boulevard, Suite 240 Gaithersburg, MD Website: su Phone: (301) Developer/Vendor Contact: Call Center Part 2: ONC-Authorized Certification Body Information 2.1 ONC-Authorized Certification Body Information ONC-ACB Name: Address: Website: Phone: ONC-ACB Contact: lnfogard Laboratories, Inc. 709 Fiero Lane Suite 25 San Luis Obispo, CA (805) Adam Hardcastle This test results summary is approved for public release by the following ONC-Authorized Certification Body Representative: Mark Shin ONC-ACB Authorized Representative ACB Certifier Function/Title.tf{;.nd Date / 2 /21 /1{ f 2014 lnfogard. May be reproduced only in its original entirety, without revision 1

2 Test Results Summary for 2014 Edition EHR Certification R 0089 PRI V1.0, December 24, Gap Certification The following identifies criterion or criteria certified via gap certification (a)(1) (a)(17) (d)(5) (d)(9) (a)(6) (b)(5)* (d)(6) (f)(1) (a)(7) (d)(1) (d)(8) *Gap certification allowed for Inpatient setting only No gap certification 2.3 Inherited Certification The following identifies criterion or criteria certified via inherited certification (a)(1) (a)(14) (c)(3) (f)(1) (a)(2) (a)(15) (d)(1) (f)(2) (a)(3) (a)(16) Inpt. only (d)(2) (f)(3) (a)(4) (a)(17) Inpt. only (d)(3) (f)(4) Inpt. only (a)(5) (b)(1) (d)(4) (f)(5) Optional & (a)(6) (b)(2) (d)(5) Amb. only (a)(7) (b)(3) (d)(6) (f)(6) Optional & (a)(8) (b)(4) (d)(7) Amb. only (a)(9) (b)(5) (d)(8) (g)(1) (a)(10) (b)(6) Inpt. only (d)(9) Optional (g)(2) (a)(11) (b)(7) (e)(1) (g)(3) (a)(12) (c)(1) (e)(2) Amb. only (g)(4) (a)(13) (c)(2) (e)(3) Amb. only No inherited certification 2014 InfoGard. May be reproduced only in its original entirety, without revision 2

3 Test Results Summary for 2014 Edition EHR Certification R-0089-PRI Vl.O, December 24, 2014 lnfoigard Part 3: NVLAP-Accredited Testing Laboratory Information Report Number: R-0089 Vl.3 Test Date(s): November 14-December 24, 2014 Location of Testing: lnfogard and Vendor Site 3.1 NVLAP-Accredited Testing Laboratory Information ATL Name: lnfogard Laboratories, Inc. Accreditation Number: NVLAP Lab Code Address: 709 Fiero Lane Suite 25 San Luis Obispo, CA Website: Phone: (805) ATL Contact: Milton Padilla For more information on scope of accreditation, please reference nda rds/scopes/ htm Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory Representative: Milton Padilla ONC-ACB Authorized Representative EHR Test Body Manager Function/Title 3.2 Test Information Additional Software Relied Upon for Certification Functionality provided by Additional Software Applicable Criteria d f A ditional So tware D No additional software required 2014 lnfogard. May be reproduced only in its original entirety, without revision 3

4 Test Results Summary for 2014 Edition EHR Certification R 0089 PRI V1.0, December 24, Test Tools Test Tool Version Cypress eprescribing Validation Tool HL7 CDA Cancer Registry Reporting Validation Tool HL7 v2 Electronic Laboratory Reporting (ELR) Validation Tool HL7 v2 Immunization Information System (IIS) Reporting Valdiation T HL7 v2 Laboratory Restults Intervace (LRI) Validation Tool HL7 v2 Syndromic Surveillance Reporting Validation Tool Transport Testing Tool Direct Certificate Discovery Tool No test tools required Test Data Alteration (customization) to the test data was necessary and is described in Appendix [insert appendix letter ] No alteration (customization) to the test data was necessary Standards Multiple Standards Permitted The following identifies the standard(s) that has been successfully tested where more than one standard is permitted Criterion # (a)(8)(ii)(a)(2) (a)(13) (a)(15)(i) (a)(16)(ii) (b)(1) HL7 Version 3 Implementation Guide: URL Based Implementations of the Context Aware Information Retrieval (Infobutton) Domain (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(1) HL7 Version 3 Implementation Guide: URL Based Implementations of the Context Aware Information Retrieval (Infobutton) Domain (g) Network Time Protocol Version 3 (RFC 1305) Standard Successfully Tested (b)(2) HL7 Version 3 Implementation Guide: Context Aware Knowledge Retrieval (Infobutton) Service Oriented Architecture Implementation Guide (j) HL7 Version 3 Standard: Clinical Genomics; Pedigree (b)(2) HL7 Version 3 Implementation Guide: Context Aware Knowledge Retrieval (Infobutton) Service Oriented Architecture Implementation Guide (g) Network Time Protocol Version 4 (RFC 5905) 2014 InfoGard. May be reproduced only in its original entirety, without revision 4

5 Test Results Summary for 2014 Edition EHR Certification R 0089 PRI V1.0, December 24, 2014 (b)(2)(i)(a) (b)(7)(i) (e)(1)(i) Annex A of the FIPS Publication List encryption and hashing algorithms (g) (g) (e)(1)(ii)(a)(2) Network Time Protocol Version 3 Network Time Protocol Version 4 (RFC 1305) (RFC 5905) (e)(3)(ii) Annex A of the FIPS Publication List encryption and hashing algorithms Common MU Data Set (15) (i) The code set specified at 45 CFR (c)(2) (ICD 10 CM) for the indicated conditions (i) The code set specified at 45 CFR (c)(2) (ICD 10 CM) for the indicated conditions (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(2) The code set specified at 45 CFR (a)(5) (HCPCS and CPT 4) None of the criteria and corresponding standards listed above are applicable Newer Versions of Standards The following identifies the newer version of a minimum standard(s) that has been successfully tested Newer Version Applicable Criteria No newer version of a minimum standard was tested Optional Functionality Criterion # Optional Functionality Successfully Tested (a)(4)(iii) Plot and display growth charts Receive summary care record using the standards (b)(1)(i)(b) specified at (a) and (b) (Direct and XDM Validation) (b)(1)(i)(c) Receive summary care record using the standards specified at (b) and (c) (SOAP Protocols) Transmit health information to a Third Party using (b)(2)(ii)(b) the standards specified at (a) and (b) (Direct and XDM Validation) Transmit health information to a Third Party using (b)(2)(ii)(c) the standards specified at (b) and (c) (SOAP Protocols) 2014 InfoGard. May be reproduced only in its original entirety, without revision 5

6 Test Results Summary for 2014 Edition EHR Certification R 0089 PRI V1.0, December 24, 2014 (f)(3) Common MU Data Set (15) Ambulatory setting only Create syndrome based public health surveillance information for transmission using the standard specified at (d)(3) (urgent care visit scenario) Express Procedures according to the standard specified at (b)(3) (45 CFR (a)(4): Code on Dental Procedures and Nomenclature) Express Procedures according to the standard Common MU Data specified at (b)(4) (45 CFR (c)(3): ICD Set (15) 10 PCS) No optional functionality tested Edition Certification Criteria* Successfully Tested Criteria # Version Version Criteria # TP** TD*** TP** TD*** (a)(1) (c)(3) (a)(2) 1.2 (d)(1) (a)(3) (d)(2) 1.5 (a)(4) (d)(3) (a)(5) (d)(4) (a)(6) (d)(5) (a)(7) (d)(6) (a)(8) (d)(7) (a)(9) (d)(8) (a)(10) (d)(9) Optional (a)(11) (e)(1) (a)(12) (e)(2) Amb. only (a)(13) (e)(3) Amb. only (a)(14) (f)(1) (a)(15) (f)(2) (a)(16) Inpt. only (f)(3) (a)(17) Inpt. only (f)(4) Inpt. only (b)(1) (f)(5) Optional & (b)(2) (b)(3) Amb. only (f)(6) Optional & (b)(4) Amb. only (b)(5) (g)(1) 1.8a 2.0 (b)(6) Inpt. only (g)(2) (b)(7) (g)(3) 1.3 (c)(1) (g)(4) 1.2 (c)(2) *For a list of the 2014 Edition Certification Criteria, please reference (navigation: 2014 Edition Test Method) **Indicates the version number for the Test Procedure (TP) ***Indicates the version number for the Test Data (TD) 2014 InfoGard. May be reproduced only in its original entirety, without revision 6

7 Test Results Summary for 2014 Edition EHR Certification R 0089 PRI V1.0, December 24, Clinical Quality Measures* Type of Clinical Quality Measures Successfully Tested: Ambulatory Inpatient No CQMs tested *For a list of the 2014 Clinical Quality Measures, please reference (navigation: 2014 Clinical Quality Measures) Ambulatory CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version Inpatient CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version InfoGard. May be reproduced only in its original entirety, without revision 7

8 Test Results Summary for 2014 Edition EHR Certification R 0089 PRI V1.0, December 24, Automated Numerator Recording and Measure Calculation Automated Numerator Recording Automated Numerator Recording Successfully Tested (a)(1) (a)(9) (a)(16) (b)(6) (a)(3) (a)(11) (a)(17) (e)(1) (a)(4) (a)(12) (b)(2) (e)(2) (a)(5) (a)(13) (b)(3) (e)(3) (a)(6) (a)(14) (b)(4) (a)(7) (a)(15) (b)(5) Automated Numerator Recording was not tested Automated Measure Calculation Automated Numerator Recording Successfully Tested (a)(1) (a)(9) (a)(16) (b)(6) (a)(3) (a)(11) (a)(17) (e)(1) (a)(4) (a)(12) (b)(2) (e)(2) (a)(5) (a)(13) (b)(3) (e)(3) (a)(6) (a)(14) (b)(4) (a)(7) (a)(15) (b)(5) Automated Measure Calculation was not tested Attestation Attestation Forms (as applicable) Appendix Safety Enhanced Design* A Quality Management System** B Privacy and Security C *Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (b)(3), (b)(4) **Required for every EHR product 2014 InfoGard. May be reproduced only in its original entirety, without revision 8

9 Test Results Summary for 2014 Edition EHR Certification R 0089 PRI V1.0, December 24, 2014 Appendix A: Safety Enhanced Design 2014 InfoGard. May be reproduced only in its original entirety, without revision 9

10 EHR Usability Test Report of OnCallData Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports OnCallData v5.0 Date of Usability Test: November 10, 2014 Date of Report: November 13, 2014 Report Prepared By: Glen Ikeda, InstantDx Washingtonian Boulevard, Suite 240 Gaithersburg, MD Table of Contents 1 EXECUTIVE SUMMARY 7 2 INTRODUCTION 10 3 METHOD PARTICIPANTS STUDY DESIGN TASKS PROCEDURE TEST ENVIRONMENT PARTICIPANT INSTRUCTIONS USABILITY METRICS 17 4 RESULTS DATA ANALYSIS AND REPORTING DISCUSSION OF THE FINDINGS 20

11 EXECUTIVE SUMMARY A usability test of OnCallData v5.0 a modular EHR was conducted on November 10, 2014 in Washington DC by InstantDx, LLC. The purpose of this test was to test and validate the usability of the current user interface, and provide evidence of usability in the EHR Under Test (EHRUT). During the usability test, 2 healthcare providers matching the target demographic criteria served as participants and used the EHRUT in simulated, but representative tasks. This study collected performance data on 8 tasks typically conducted on an EHR. Prescribe a medication Check Drug formulary Check Drug/Drug and Drug/Allergy Interaction Record Medication Record Allergy Search for Patient View Medications View Allergies During the 30 minute one on one usability test, each participant was greeted by the administrator. Participants had prior experience with the EHR. The administrator introduced the test, and instructed participants to complete a series of tasks (given one at a time) using the EHRUT. During the testing, the administrator timed the test and recorded user performance data on paper. The administrator did not give the participant assistance in how to complete the task. The following types of data were collected for each participant: Number of tasks successfully completed within the allotted time without assistance Time to complete the tasks Number and types of errors Participant s verbalizations Participant s satisfaction ratings of the system All participant data was de identified no correspondence could be made from the identity of the participant to the data collected. Various recommended metrics, in accordance with the examples set forth in the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, were used to evaluate the usability of the EHRUT. Following is a summary of the performance and rating data collected on the EHRUT. Task Participant Participant Errors Time for Completion Verbalization Satisfaction Search Patient None High 0 Under 1 minute Record Medication None High 0 Under 1 minute Record Allergy None High 0 Under 1 minute View Medication None High 0 Under 1 minute

12 View Allergies None High 0 Under 1 minute Prescribe None High 0 Under 3 minutes Medication Check Drug None High 0 Under 3 minutes Formulary Check Drug/Drug and Drug/Allergy Interactions None High 0 Under 2 minutes The results from the System Usability Scale scored the subjective satisfaction with the system based on performance with these tasks to be: In addition to the performance data, the following qualitative observations were made: Major findings Application fits well into user workflow Areas for improvement Certain drug formulary information could be better highlighted 1 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149). Broadly interpreted, scores under 60 represent systems with poor usability; scores over 80 would be considered above average.

13 INTRODUCTION The EHRUT tested for this study was OnCallData v5.0, designed to present medical information to healthcare providers in medical practice office settings. The EHRUT includes electronic prescribing, medication history, patient allergies, and patient problems. The usability testing attempted to represent realistic exercises and conditions. The purpose of this study was to test and validate the usability of the current user interface, and provide evidence of usability in the EHR Under Test (EHRUT). To this end, measures of effectiveness, efficiency and user satisfaction, such as time for completion, ease of use and workflow, were captured during the usability testing. METHOD PARTICIPANTS A total of two participants were tested on the EHRUT(s). Participants in the test were nurses. Participants were recruited by InstantDx, LLC. In addition, participants had no direct connection to the development of or organization producing the EHRUT(s). Participants were not from the testing or supplier organization. Participants were given the opportunity to have the same orientation and level of training as the actual end users would have received. The following is a table of participants by characteristics, including demographics, professional experience, computing experience and user needs for assistive technology. Participant names were replaced with Participant IDs so that an individual s data cannot be tied back to individual identities. Part ID Gender Occupation/ role Professional Experience Computer Experience Product Experience 1 1 F Nurse Nurse medium high 2 2 F Nurse Nurse medium high Two participants were recruited and two participated in the usability test. Zero participants failed to show for the study. Participants were scheduled for one 45 minute session with15 minutes in between each session for debrief by the administrator and to reset systems to proper test conditions. STUDY DESIGN Overall, the objective of this test was to uncover areas where the application performed well that is, effectively, efficiently, and with satisfaction and areas where the application failed to meet the needs of the participants. The data from this test may serve as a baseline for future tests with an updated version of the same EHR and/or comparison with other EHRs provided the same tasks are used. In short, this testing serves as both a means to record or benchmark current usability, but also to identify areas where improvements must be made. During the usability test, participants interacted with one EHR. Each participant used the system in the same location, and was provided with the same instructions. The system was evaluated

14 for effectiveness, efficiency and satisfaction as defined by measures collected and analyzed for each participant: Number of tasks successfully completed within the allotted time without assistance Time to complete the tasks Number and types of errors Participant s verbalizations (comments) Participant s satisfaction ratings of the system Additional information about the various measures can be found in Section 3.9 on Usability Metrics. TASKS A number of tasks were constructed that would be realistic and representative of the kinds of activities a user might do with this EHR, including: Search for patient Record and view allergies Record and view medication Prescribe medication Check drug formulary Check drug/drug and drug/allergy interactions Tasks were selected based on their frequency of use, criticality of function, and those that may be most troublesome for users. PROCEDURES The administrator moderated the session including administering instructions and tasks. The administrator also monitored task times, obtained post task rating data, and took notes on participant comments. Participants were instructed to perform the tasks (see specific instructions below): As quickly as possible making as few errors and deviations as possible. Without assistance; administrators were allowed to give immaterial guidance and clarification on tasks, but not instructions on use. Without using a think aloud technique. Task timing began once the administrator finished reading the question. The task time was stopped once the participant indicated they had successfully completed the task. Scoring is discussed below in Section 3.9. Participants' demographic information, task success rate, time on task, errors, verbal responses, and post test questionnaire were recorded.

15 TEST ENVIRONMENT The EHRUT would be typically be used in a healthcare office or facility. In this instance, the testing was conducted in a healthcare office. For testing, the computer used a desktop computer running a Windows operating system. The participants used mouse and keyboard when interacting with the EHRUT. The application itself was running on a web browser using a test database and a WAN connection. Technically, the system performance (i.e., response time) was representative to what actual users would experience in a field implementation. Additionally, participants were instructed not to change any of the default system settings (such as control of font size). PARTICIPANT INSTRUCTIONS Participants were then given 8 tasks to complete. USABILITY METRICS According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, EHRs should support a process that provides a high level of usability for all users. The goal is for users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To this end, metrics for effectiveness, efficiency and user satisfaction were captured during the usability testing. The goals of the test were to assess: Effectiveness of OnCallData by measuring participant success rates and errors Efficiency of OnCallData by measuring the average task time Satisfaction with OnCallData by measuring ease of use ratings

16 DATA SCORING The following table (Table [x]) details how tasks were scored, errors evaluated, and the time data analyzed. Measures Effectiveness: Task Success Effectiveness: Task Failures Efficiency: Task Time Rationale and Scoring A task was counted as a Success if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis. The total number of successes were calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage. Task times were recorded for successes. Observed task times divided by the optimal time for each task is a measure of optimal efficiency. Optimal task performance time, as benchmarked by expert performance under realistic conditions, is recorded when constructing tasks. Target task times used for task times in the Moderator s Guide must be operationally defined by taking multiple measures of optimal performance and multiplying by some factor e.g. 1.25, that allows some time buffer because the participants are presumably not trained to expert performance. Thus, if expert, optimal performance on a task was 30 seconds then allotted task time performance was 30 * 1.25 seconds. This ratio should be aggregated across tasks and reported with mean and variance scores. If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion, the task was counted as an Failures. No task times were taken for errors. The total number of errors was calculated for each task and then divided by the total number of times that task was attempted. Not all deviations would be counted as errors. 11 This should also be expressed as the mean number of failed tasks per participant. On a qualitative level, an enumeration of errors and error types should be collected. Each task was timed from when the administrator said Begin until the participant said, Done. If he or she failed to say Done, the time was stopped when the participant stopped performing the task. Only task times for tasks that were successfully completed were included in the average task time analysis. Average time per task was calculated for each task. Variance measures (standard deviation and standard error) were also calculated.

17 Satisfaction: Task Rating Participant s subjective impression of the ease of use of the application was measured by administering both a simple post-task question as well as a post-session questionnaire. After each task, the participant was asked to rate Overall, this task was: on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants. 12 Common convention is that average ratings for systems judged easy to use should be 3.3 or above. To measure participants confidence in and likeability of the [EHRUT] overall, the testing team administered the System Usability Scale (SUS) post-test questionnaire. Questions included, I think I would like to use this system frequently, I thought the system was easy to use, and I would imagine that most people would learn to use this system very quickly. RESULTS DATA ANALYSIS AND REPORTING The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Participants who failed to follow session and task instructions had their data excluded from the analyses. RESULTS DATA ANALYSIS AND REPORTING The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Participants who failed to follow session and task instructions had their data excluded from the analyses. The usability testing results for the EHRUT are detailed below. The results should be seen in light of the objectives and goals outlined in Section 3.2 Study Design. The data should yield actionable results that, if corrected, yield material, positive impact on user performance. Task Participant Participant Errors Time for Completion Verbalization Satisfaction Search Patient None High 0 Under 1 minute Record Medication None High 0 Under 1 minute Record Allergy None High 0 Under 1 minute View Medication None High 0 Under 1 minute View Allergies None High 0 Under 1 minute Prescribe Medication None High 0 Under 3 minutes

18 Check Drug Formulary Check Drug/Drug and Drug/Allergy Interactions None High 0 Under 3 minutes None High 0 Under 2 minutes EFFECTIVENESS Users were able to successfully complete all tasks with no errors. EFFICIENCY No individual task took over 3 minutes. SATISFACTION Users expressed a high level of satisfaction. AREAS FOR IMPROVEMENT Certain drug formulary information could be better highlighted.

19 Appendix 1: PARTICIPANT DEMOGRAPHICS Following is a high-level overview of the participants in this study. Gender Men [0] Women [2] Total (participants) [2] Occupation/Role RN/BSN [2] Physician [0] Admin Staff [0] Total (participants) [2]

20 Test Results Summary for 2014 Edition EHR Certification R 0089 PRI V1.0, December 24, 2014 Appendix B: Quality Management System 2014 InfoGard. May be reproduced only in its original entirety, without revision 10

21

22 Test Results Summary for 2014 Edition EHR Certification R 0089 PRI V1.0, December 24, 2014 Appendix C: Privacy and Security 2014 InfoGard. May be reproduced only in its original entirety, without revision 11

23

24

25 Test Results Summary for 2014 Edition EHR Certification R 0089 PRI V1.0, December 24, 2014 Test Results Summary Document History Version Description of Change V1.0 Initial release Date December 24, 2014 END OF DOCUMENT 2014 InfoGard. May be reproduced only in its original entirety, without revision 12