ONC HIT Certification Program

Transcription

1 Test Results Summary for 2014 Edition EHR Certification R-0007-PRA V1.14, February 28, 2016 ONC HIT Certification Program Test Results Summary for 2014 Edition EHR Certification Part 1: Product and Developer Information 1.1 Certified Product Information Product Name: BRAINTREE Product Version: Domain: Ambulatory Test Type: Complete EHR 1.2 Developer/Vendor Information Developer/Vendor Name: Braintree Health Address: PO Box Corpus Christi, TX Website: Phone: (512) Developer/Vendor Contact: Sunil Reddy Part 2: ONC-Authorized Certification Body Information 2.1 ONC-Authorized Certification Body Information ONC-ACB Name: InfoGard Laboratories, Inc. Address: 709 Fiero Lane Suite 25 San Luis Obispo, CA Website: Phone: (805) ONC-ACB Contact: Adam Hardcastle This test results summary is approved for public release by the following ONC-Authorized Certification Body Representative: Adam Hardcastle ONC-ACB Authorized Representative Signature EHR Certification Body Manager Function/Title February 28, 2016 Date 2015 InfoGard. May be reproduced only in its original entirety, without revision 1

2 Test Results Summary for 2014 Edition EHR Certification R-0007-PRA V1.14, February 28, Gap Certification The following identifies criterion or criteria certified via gap certification (a)(1) (a)(17) (d)(5) (d)(9) (a)(6) (b)(5)* (d)(6) (f)(1) (a)(7) (d)(1) (d)(8) *Gap certification allowed for Inpatient setting only No gap certification 2.3 Inherited Certification The following identifies criterion or criteria certified via inherited certification (a)(1) (a)(14) (c)(3) (f)(1) (a)(2) (a)(15) (d)(1) (f)(2) (a)(3) (a)(16) Inpt. only (d)(2) (f)(3) (a)(4) (a)(17) Inpt. only (d)(3) (f)(4) Inpt. only (a)(5) (b)(1) (d)(4) (f)(5) Optional & Amb. only (a)(6) (b)(2) (d)(5) (f)(6) Optional & Amb. only (a)(7) (b)(3) (d)(6) (g)(1) (a)(8) (b)(4) (d)(7) (g)(2) (a)(9) (b)(5) (d)(8) (g)(3) (a)(10) (b)(6) Inpt. only (d)(9) Optional (g)(4) (a)(11) (b)(7) (e)(1) (a)(12) (c)(1) (e)(2) Amb. only (a)(13) (c)(2) (e)(3) Amb. only No inherited certification 2015 InfoGard. May be reproduced only in its original entirety, without revision 2

3 Test Results Summary for 2014 Edition EHR Certification R-0007-PRA V1.14, February 28, 2016 Part 3: NVLAP-Accredited Testing Laboratory Information Report Number: R-0007 Version 1.4 Test Date(s): April 8, March 3, 2015 Location of Testing: InfoGard and Vendor Site 3.1 NVLAP-Accredited Testing Laboratory Information ATL Name: InfoGard Laboratories, Inc. Accreditation Number: NVLAP Lab Code Address: 709 Fiero Lane Suite 25 San Luis Obispo, CA Website: ehr@infogard.com Phone: (805) ATL Contact: Milton Padilla For more information on scope of accreditation, please reference Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory Representative: Milton Padilla ATL Authorized Representative EHR Test Body Manager Function/Title Signature February 28, 2016 Date 3.2 Test Information Additional Software Relied Upon for Certification Additional Software Applicable Criteria Functionality provided by Additional Software DrFirst (b)(3) E-prescribing No additional software required 2015 InfoGard. May be reproduced only in its original entirety, without revision 3

4 Test Results Summary for 2014 Edition EHR Certification R-0007-PRA V1.14, February 28, Test Tools Test Tool Version Cypress eprescribing Validation Tool HL7 CDA Cancer Registry Reporting Validation Tool HL7 v2 Electronic Laboratory Reporting (ELR) Validation Tool n/a HL7 v2 Immunization Information System (IIS) Reporting Validation Tool HL7 v2 Laboratory Results Interface (LRI) Validation Tool HL7 v2 Syndromic Surveillance Reporting Validation Tool Transport Testing Tool 179 Direct Certificate Discovery Tool No test tools required Test Data Alteration (customization) to the test data was necessary and is described in Appendix [x] No alteration (customization) to the test data was necessary Standards Multiple Standards Permitted The following identifies the standard(s) that has been successfully tested where more than one standard is permitted Criterion # (a)(8)(ii)(a)(2) (a)(13) (a)(15)(i) (a)(16)(ii) (b)(1) HL7 Version 3 Implementation Guide: URL Based Implementations of the Context Aware Information Retrieval (Infobutton) Domain (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain (g) Network Time Protocol Version 3 (RFC 1305) Standard Successfully Tested (b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide (j) HL7 Version 3 Standard: Clinical Genomics; Pedigree (b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide (g) Network Time Protocol Version 4 (RFC 5905) 2015 InfoGard. May be reproduced only in its original entirety, without revision 4

5 Test Results Summary for 2014 Edition EHR Certification R-0007-PRA V1.14, February 28, 2016 Criterion # (b)(2)(i)(a) (b)(7)(i) (i) The code set specified at 45 CFR (c)(2) (ICD-10-CM) for the indicated conditions (i) The code set specified at 45 CFR (c)(2) (ICD-10-CM) for the indicated conditions Standard Successfully Tested (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (e)(1)(i) Annex A of the FIPS Publication TLS - RSA with AES and SHA (e)(1)(ii)(a)(2) (g) Network Time Protocol Version 3 (RFC 1305) (g) Network Time Protocol Version 4 (RFC 5905) (e)(3)(ii) Annex A of the FIPS Publication SHA-1 with RSA Common MU Data Set (15) (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(2) The code set specified at 45 CFR (a)(5) (HCPCS and CPT-4) None of the criteria and corresponding standards listed above are applicable Newer Versions of Standards The following identifies the newer version of a minimum standard(s) that has been successfully tested Newer Version Applicable Criteria No newer version of a minimum standard was tested Optional Functionality Criterion # (a)(4)(iii) (b)(1)(i)(b) (b)(1)(i)(c) (b)(2)(ii)(b) (b)(2)(ii)(c) (f)(3) Optional Functionality Successfully Tested Plot and display growth charts Receive summary care record using the standards specified at (a) and (b) (Direct and XDM Validation) Receive summary care record using the standards specified at (b) and (c) (SOAP Protocols) Transmit health information to a Third Party using the standards specified at (a) and (b) (Direct and XDM Validation) Transmit health information to a Third Party using the standards specified at (b) and (c) (SOAP Protocols) Ambulatory setting only Create syndrome-based public health surveillance information for transmission using the standard specified at (d)(3) (urgent care visit scenario) 2015 InfoGard. May be reproduced only in its original entirety, without revision 5

6 Test Results Summary for 2014 Edition EHR Certification R-0007-PRA V1.14, February 28, 2016 Criterion # Optional Functionality Successfully Tested Common MU Express Procedures according to the standard specified at (b)(3) Data Set (15) (45 CFR (a)(4): Code on Dental Procedures and Nomenclature) Common MU Express Procedures according to the standard specified at Data Set (15) (b)(4) (45 CFR (c)(3): ICD-10-PCS) No optional functionality tested Edition Certification Criteria* Successfully Tested Criteria # Version Version Criteria # TP** TD*** TP** TD*** (a)(1) (c)(3) (a)(2) 1.2 (d)(1) 1.2 (a)(3) (d)(2) 1.5 (a)(4) (d)(3) 1.3 (a)(5) (d)(4) 1.2 (a)(6) (d)(5) 1.2 (a)(7) (d)(6) 1.2 (a)(8) 1.2 (d)(7) 1.2 (a)(9) (d)(8) 1.2 (a)(10) (d)(9) Optional 1.2 (a)(11) 1.3 (e)(1) (a)(12) 1.3 (e)(2) Amb. only (a)(13) 1.2 (e)(3) Amb. only 1.3 (a)(14) 1.2 (f)(1) (a)(15) 1.5 (f)(2) (a)(16) Inpt. only (f)(3) (a)(17) Inpt. only (f)(4) Inpt. only (b)(1) (f)(5) Optional & Amb. only (b)(2) (f)(6) Optional & Amb. only (b)(3) (g)(1) (b)(4) (g)(2) 1.8a 2.0 (b)(5) (g)(3) 1.3 (b)(6) Inpt. only (g)(4) 1.2 (b)(7) (c)(1) (c)(2) *For a list of the 2014 Edition Certification Criteria, please reference (navigation: 2014 Edition Test Method) **Indicates the version number for the Test Procedure (TP) ***Indicates the version number for the Test Data (TD) 2015 InfoGard. May be reproduced only in its original entirety, without revision 6

7 Test Results Summary for 2014 Edition EHR Certification R-0007-PRA V1.14, February 28, Clinical Quality Measures* Type of Clinical Quality Measures Successfully Tested: Ambulatory Inpatient No CQMs tested *For a list of the 2014 Clinical Quality Measures, please reference (navigation: 2014 Clinical Quality Measures) Ambulatory CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version 2 v3 90 v3 136 v3 155 v2 22 v2 117 v2 137 v2 156 v2 50 v2 122 v2 138 v2 157 v2 52 v2 123 v2 139 v2 158 v2 56 v2 124 v2 140 v2 159 v2 61 v3 125 v2 141 v3 160 v2 62 v2 126 v2 142 v2 161 v2 64 v3 127 v2 143 v2 163 v2 65 v3 128 v2 144 v2 164 v2 66 v2 129 v3 145 v2 165 v2 68 v3 130 v2 146 v2 166 v3 69 v2 131 v2 147 v2 167 v2 74 v3 132 v2 148 v2 169 v2 75 v2 133 v2 149 v2 177 v2 77 v2 134 v2 153 v2 179 v2 82 v1 135 v2 154 v2 182 v3 Inpatient CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version InfoGard. May be reproduced only in its original entirety, without revision 7

8 Test Results Summary for 2014 Edition EHR Certification R-0007-PRA V1.14, February 28, Automated Numerator Recording and Measure Calculation Automated Numerator Recording Automated Numerator Recording Successfully Tested (a)(1) (a)(9) (a)(16) (b)(6) (a)(3) (a)(11) (a)(17) (e)(1) (a)(4) (a)(12) (b)(2) (e)(2) (a)(5) (a)(13) (b)(3) (e)(3) (a)(6) (a)(14) (b)(4) (a)(7) (a)(15) (b)(5) Automated Numerator Recording was not tested Automated Measure Calculation Automated Numerator Recording Successfully Tested (a)(1) (a)(9) (a)(16) (b)(6) (a)(3) (a)(11) (a)(17) (e)(1) (a)(4) (a)(12) (b)(2) (e)(2) (a)(5) (a)(13) (b)(3) (e)(3) (a)(6) (a)(14) (b)(4) (a)(7) (a)(15) (b)(5) Automated Measure Calculation was not tested Attestation Attestation Forms (as applicable) Safety-Enhanced Design* Quality Management System** Privacy and Security Appendix A *Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (b)(3), (b)(4) **Required for every EHR product B C 2015 InfoGard. May be reproduced only in its original entirety, without revision 8

9 Test Results Summary for 2014 Edition EHR Certification R-0007-PRA V1.14, February 28, 2016 Appendix A: Safety Enhanced Design The following required data was missing: Display size An inaccurate description of the summative usability testing measures used for Effectiveness, Efficiency, and Satisfaction was provided in the "Results" section of the report. The information provided in the table of results data did not match the results as described for measures of Effectiveness, Efficiency, and Satisfaction InfoGard. May be reproduced only in its original entirety, without revision 9

10 WCAG and Safety Enhanced Design Copyright 2014, Braintree. This document and its contents are the property of Braintree. Braintree reserves the right for distribution, use, application and reproduction. Version 3.1

11 Table Contents Table Contents Executive Summary- Safety Enhanced Design Introduction Method Usability Standard Overview Participants Study Design s a1-1cpoe medication order a1-2 CPOE Lab order a1-3 CPOE Radiology order a6 Medication List a7 Medication allergy list a5 Problem List a3 Demographics a2 Drug-drug, drug-allergy interactions checks b5a Incorporate laboratory tests and values/results a4 Vital Signs b3 Electronic prescribing b4 Clinical information reconciliation a8 Clinical decision support Procedures Test Location Testing environment TEST FORMS AND TOOLS Participant Instructions Usability Metrics Data Scoring Results Data analysis and reporting Discussions of the findings Major Findings Areas for improvement Effectiveness Satisfaction Screenshots of Braintree usability Data logger... 21

12 Braintree Data logger Admin menu scenario sheet Braintree Questionnaire page Charts WCAG Browser Support Techniques Testing Tools Web accessibility Checker Snapshot... 29

13 EHR Usability Test Report of Braintree Braintree Date of usability test: October 18, 2014 Date of Report: October25,2014 Report prepared by Braintree Health Sunil Reddy, CEO, Braintree Health 1711 W, Wheeler ave Aransas Pass, TX, 78336

14 1. Executive Summary- Safety Enhanced Design A usability test of Braintree v was conducted with test participants between October 18, 2014 and October 22, The location of the test was 1711 W. Wheeler ave Aransas Pass, TX, Zip Code The purpose of this test was to validate the usability of the current user interface and provide evidence of usability in the EHR Under test (EHRUT). During the usability test, 10 health care providers which included 5 doctors, 2 nurses, 3 staff officers matching the target demographic criteria served as participants and used the EHRUT in simulated, but representative tasks. The study collected performance data on 11 tasks conducted in EHR (a)(1) Computerized provider order entry (a)(6) Medication list (a)(7) Medication allergy list (a)(5) Problem list (a)(3) Demographics (a)(2) Drug-drug allergy interaction checks (b)(5)(a) Incorporate laboratory tests and values/results (a)(4) Vital Signs (b)(3) Electronic prescribing (a)(8) Clinical decision support (b)(4) Clinical information reconciliation During the (approximate) 4 day one-on-one usability test, each participant was greeted by the administrator and asked to review and sign an informed consent/release form they were instructed that they could withdraw any time. Participants had prior experience with the EHR. The administrator introduced the test, and instructed participants to complete a series of task (given one at a time) using the EHRUT. During the testing, the administrator time the test and, along with the data logger(s) recorded user performance data on paper and electronically. Over the course of study, no training or help materials were provided to help with the completion of the tasks. The following types of data were collected for each participant: Number of tasks successfully completed with the allotted time without assistance. Time to complete the tasks Number and types of errors. Path deviations

15 Participant s verbalizations Participant s satisfaction ratings of the system All participant data was de-identified no correspondence could be made from the identity of the participant to the data collected. Following the conclusion of the testing, participants were asked to complete a post-test questionnaire. Various metrics, in accordance with the examples set forth in the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, were used to evaluate the usability of the EHRUT. Following is a summary of the performance and rating data collected on the EHRUT The results from the System Usability Scale scored the subjective satisfaction with the system based on performance with these tasks to be: 81 In addition to the performance data, the following qualitative observations were made: Major Findings- Effectiveness, Efficiency and Satisfaction. Areas for Improvement 2. Introduction The EHRUT tested for this study was Braintree v , designed to present medical information to healthcare providers in any clinical setting. The EHRUT consists of modules to document patient appointments with completed medical information. The EHRUT has the facility to record complete medical information of the patient like patient s medical, social, surgical and family history, review of systems, diagnosis, medications, allergies, immunizations, billing, orders, lab orders and much more. The usability testing attempted to represent realistic exercises and conditions. The purpose of this study was to test and validate the usability of the current user interface, and provide evidence of usability in the EHR under test (EHRUT). The usability test focuses on the following measures Effectiveness Efficiency Satisfaction Overall system performance Average timing. 3. Method 3.1 Usability Standard Braintree is following the ISO standard ISO for all the measures.. The standard states that The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use. The standard describes how it can be applied to Specify and measure the usability of products. Specify and evaluate usability during design. Specify and measure a work system in use. The measure explains about three specific elements Specific users: The specific user for which product is being designed. Specific goals: Each user has specific goals from the product. It means the goal of the product is going to be the goal of the users.

16 Specific Context of use: The product should be designed in a way that it should work in the environment in which users will use it. The measure focuses on the following measures Effectiveness Efficiency Satisfaction 3.2 Overview As the EHRUT is a combination of different modules in an ambulatory/inpatient settings. Therefore, we are providing graphical representation of the tests conducted by the participants 3.3 Participants A total of 10 participants were tested on the EHRUTS(s) participants in the test were physicians, nurses and clinical and administrative staff. Participants were recruited by Sunil Reddy. In addition, participants had no direct connection to the development of organization producing the EHRUT(s) participants were all give the same level of training on the EMR and tasks to be completed. For the test purposes, end-user characteristics were identified and translated into a recruitment screener used to solicit potential participants. Recruited participants had a mix of backgrounds and demographic characteristics conforming to the recruitment screener. The following is a table of participants by characteristics, including demographics, professional experience, computing experience and user needs to assistive technology. Participant names were replaced with Participant IDs so that and individual s data cannot be tied back to individual identities. Information about participants represented in the following Table 1: Table 1 ID Gender Professional Experience Age Role Computer Experience Previous EHR experience Btuser1 Male 10+ years 45 Physician Yes 3 None Btuser2 Female 15+ years 50 Physician Yes 4 None Assistive Technology Needs Btuser3 Female 5 years 35 Nurse Yes 2 Moderate Btuser4 Male 7 years 42 Physician Yes 5 None Btuser5 Female 9 years 38 Physician Yes 1 None Btuser6 Female 2.5 years 32 Staff Yes 2 Moderate officer Btuser7 Male 4 years 35 Staff Officer Yes 3 None Btuser8 Male 6 years 40 Nurse Yes 4 None Btuser9 Male 3 years 30 Staff Yes 2 None

17 Officer Btuser10 Male 20+ years 55 Physician Yes 6 None 10 participants were recruited and all 10 participated in the usability test. Participants were scheduled for maximum 30 minute sessions with 15 minutes in between each session for debrief by the administrator and data logger and to reset systems to proper test conditions. A spreadsheet was used to keep track of the participant schedule, and include each participant s demographic characteristics. 3.4 Study Design Overall, the objective of this test was to uncover areas where the application performed well that is, effectively, efficiently, and with satisfaction and areas where the application failed to meet the needs of the participants. The data from this test may serve as a baseline for future tests with an updated version of the same EHR and/or comparison with other EHRs provided the same tasks are used. In short, this testing serves as both a means to record or benchmark current usability, but also to identify areas where improvements must be made. During the usability test, participants interacted with 1 EHR(s).Each participant used the system in the same location, and was provided with the same instructions. The system was evaluated for effectiveness, efficiency and satisfaction as defined by measures collected and analyzed for each participant: Number of tasks successfully completed within the allotted time without assistance Time to complete the tasks Number and types of errors Path deviations Participant s verbalizations (comments) Participant s satisfaction ratings of the system

18 Braintree WCAG and Safety Enhanced Design Version s A number of tasks were constructed that would be realistic and representative of the kinds of activities a user might do with this EHR, including: a1-1cpoe medication order Menu Navigation: Search patient -> Select patient -> erx button -> Medications -> Add/edit medication order. Steps 1. Select patient menu. 2. Search the desired patient. 3. Select the patient and go to patient chart. 4. Select the erx button. 5. Select the medications link to add the medication. 6. Search the desired medication to add. 7. Add medication to the patient record a1-2 CPOE Lab order Menu Navigation: Search patient -> Select patient -> Select CPOE -> Select Orders -> Add/edit Lab order. Steps 1. Select the patient menu. 2. Search the desired patient. 3. Select the patient and go to patient chart. 4. Select CPOE button. 5. Select orders link. 6. Select the desired lab order to be added a1-3 CPOE Radiology order Menu Navigation: Search patient -> Select patient -> Select CPOE -> Select Orders -> Add/edit Radiology Order. Steps 1. Select the patient menu. 2. Search the desired patient. 9

19 Braintree WCAG and Safety Enhanced Design Version Select the patient and go to patient chart. 4. Select CPOE button. 5. Select orders link. 6. Select the desired radiology order to be added a6 Medication List Menu Navigation: Search patient -> Select patient -> Select CPOE -> Select Medication -> Add/edit medication. Steps 1. Select the patient menu. 2. Search the desired patient. 3. Select the patient and go to patient chart. 4. Select CPOE button. 5. Select medication link. 6. The medication list appears with displaying all the medications of the patient. 7. Click on search bar for filtering different medications of the patient. 8. Add the medications using the main screen. 9. Click on medication to edit and change the status of the medication a7 Medication allergy list Menu Navigation: Search patient -> Select patient -> Select CPOE -> Select Allergy -> Add/edit allergy. Steps 1. Select the patient menu. 2. Search the desired patient. 3. Select the patient and go to patient chart. 4. Select CPOE button. 5. Select allergy link. 6. The allergy list appears with displaying all the allergies of the patient. 7. Click on search bar for filtering different allergies of the patient. 8. Add the allergy using the main screen. 9. Click on allergy to edit and change the status of the allergy a5 Problem List Menu Navigation: Search patient -> Select patient -> Select CPOE -> Select Problems -> Add/edit problems. 10

20 Braintree WCAG and Safety Enhanced Design Version 3.1 Steps 1. Select the patient menu. 2. Search the desired patient. 3. Select the patient and go to patient chart. 4. Select CPOE button. 5. Select problem link. 6. The problem list appears with displaying all the problems of the patient. 7. Click on search bar for filtering different problems of the patient. 8. Add the problems using the main screen. 9. Click on problem to edit and change the status of the problem a3 Demographics Menu Navigation: Patient -> New patient Steps 1. Click on patient menu. 2. Select new patient menu. 3. Enter the patient details with patient demographics. 4. Click on save button. 5. Select the patient and edit the patient demographics a2 Drug-drug, drug-allergy interactions checks Menu Navigation: Search patient -> Select patient -> Select erx-> Select medications -> Add medications Steps 1. Click on the patient menu 2. Search the patient. 3. Select the patient. 4. Click on erx button. 5. Click on options link for modifying the drug-drug, drug-allergy interaction. 6. Click on save button to save the changes for interactions. 7. Add medication to view the interactions check b5a Incorporate laboratory tests and values/results Menu Navigation: Search patient -> Select patient -> Select CPOE -> Select Results -> Add/edit results. Steps: 1. Click on the patient menu 11

21 Braintree WCAG and Safety Enhanced Design Version Search the patient. 3. Select the patient. 4. Click on CPOE button. 5. Select the results link. 6. Select the desired results to be added a4 Vital Signs Menu Navigation: Scheduler-> Search patient -> Schedule appointment -> Admission page-> Add pending admission-> Select patient encounter -> Select nursing record -> Select initial assessment-> Add vitals Steps 1. Click on scheduler to schedule an appointment. 2. Search the patient for which appointment needs to be scheduled. 3. Select the searched patient and schedule the appointment. 4. Go to admission page. 5. Admit the pending admission of the patient. 6. Select the encounter created of the patient. 7. Select the nursing record. 8. Select the initial assessment link to enter vitals. 9. Add/edit patient vitals b3 Electronic prescribing Menu Navigation: Search patient -> Select patient -> Select erx button -> Select prescribe -> Add/edit prescription Steps 1. Click on the patient menu 2. Search the patient. 3. Select the patient. 4. Click on erx button. 5. Select the prescribe link. 6. Add/edit the prescriptions b4 Clinical information reconciliation Menu Navigation: Search patient-> Select patient -> Select CPOE button -> Select medication link Menu Navigation: Search patient-> Select patient -> Select CPOE button -> Select allergy link Menu Navigation: Search patient-> Select patient -> Select CPOE button -> Select problem link 12

22 Braintree WCAG and Safety Enhanced Design Version 3.1 Steps 1. Click on the patient menu 2. Search the patient. 3. Select the patient. 4. Click on CPOE button. 5. Click on medication link to reconcile medications. 6. Click on allergy link to reconcile allergies. 7. Click on problem link to reconcile problems a8 Clinical decision support Menu Navigation: Search patient -> Select patient -> Click on CDS alert icon. CDS configuration Menu Navigation. Administration -> CDS Add Rule Administration -> CDS Alert config Administration -> CDS Rule group privileges. Steps 1. Click on administration and add a CDS rule. 2. Click on administration and add CDS alert configuration for the rule added. 3. Click on CDS rule group privileges and add the group privileges to the rule. 4. Click on search patient. 5. Select the patient. 6. Select the alerts icon on patient chart. 7. View alerts and reminders of the patient. s were selected based on their frequency of use, criticality of function, and those that may be most troublesome for users. s should always be constructed in light of the study objectives. 3.6 Procedures Upon arrival, participants were greeted; their identity was verified and matched with a name on the participant schedule. Participants were then assigned a participant ID. Each participant reviewed and signed an informed consent and release form. A representative from the test team witnessed the participant s signature. To ensure that the test ran smoothly, two staff members participated in this test, the usability administrator and the data logger. The usability testing staff conducting the test was experienced usability practitioners with extensive EHR knowledge, multiple years of experience, qualifying education and satisfactory qualifications. 13

23 Braintree WCAG and Safety Enhanced Design Version 3.1 The administrator moderated the session including administering instructions and tasks. The administrator also monitored task times, obtained post-task rating data, and took notes on participant comments. A second person served as the data logger and took notes on task success, path deviations, number and type of errors, and comments. Participants were instructed to perform the tasks (see specific instructions below): As quickly as possible making as few errors and deviations as possible. Without assistance; administrators were allowed to give immaterial guidance and clarification on tasks, but not instructions on use. Without using a think aloud technique. For each task, the participants were given a written copy of the task. timing began once the administrator finished reading the question. The task time was stopped once the participant indicated they had successfully completed the task Following the session, the administrator gave the participant the post-test questionnaire and thanked each individual for their participation. Participants' demographic information, task success rate, time on task, errors, deviations, verbal responses, and post-test questionnaire were recorded into a spreadsheet. 3.7 Test Location The test facility included a waiting area and a quiet testing room with a table, computer for the participant, and recording computer for the administrator. Only the participant and administrator were in the test room. All observers and the data logger remained in the same room to observe the participant complete the tasks. To ensure that the environment was comfortable for users, noise levels were kept to a minimum with the ambient temperature within a normal range. All of the safety instruction and evacuation procedures were valid, in place, and visible to the participants. 3.8 Testing environment The EHRUT would be typically be used in a healthcare office or facility. In this instance, the testing was conducted in Braintree Health Center. For testing, the computer used a DELL running Windows Server 2008 enterprise edition/windows server 2003 enterprise edition. The participants used keyboard and mouse when interacting with the EHRUT. The computer used 1600 x 900 resolutions with standard color options. The application was set up by the tester for the vendor according to the vendor s documentation describing the system set-up and preparation. The application itself was running on Internet Explorer using a test database on a LAN connection. Technically, the system performance (i.e., response time) was representative to what actual users would experience in a field implementation with some minor deviations in response to varying internet connectivity speed. Additionally, participants were instructed not to change any of the default system settings (such as control of font size) 14

24 Braintree WCAG and Safety Enhanced Design Version TEST FORMS AND TOOLS During the usability test, various documents and instruments were used, including: 1. Informed consent 2. Moderator s Guide 3. Post test questionnaire 3.10 Participant Instructions The administrator reads the following instructions aloud to the each participant Thank you for participating in this study. Your input is very important. Our session today will last about 30 minute]. During that time you will use an instance of an electronic health record. I will ask you to complete a few tasks using this system and answer some questions. You should complete the tasks as quickly as possible making as few errors as possible. Please try to complete the tasks on your own following the instructions very closely. Please note that we are not testing you we are testing the system, therefore if you have difficulty all this means is that something needs to be improved in the system. I will be here in case you need specific help, but I am not able to instruct you or provide help in how to use the application. Overall, we are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. I did not have any involvement in its creation, so please be honest with your opinions. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time. Should you feel it necessary you are able to withdraw at any time during the testing. Following the procedural instructions, participants were shown the EHR and as their first task, were given time to explore the system and make comments. Once this task was complete, the administrator gave the following instructions: For each task, I will read the description to you and say Begin. At that point, please perform the task and say Done once you believe you have successfully completed the task. I would like to request that you not talk aloud or verbalize while you are doing the tasks. I will ask you your impressions about the task once you are done. Participants were then given 13 tasks to complete 15

25 Braintree WCAG and Safety Enhanced Design Version Usability Metrics According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, EHRs should support a process that provides a high level of usability for all users. The goal is for users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To this end, metrics for effectiveness, efficiency and user satisfaction were captured during the usability testing. The goals of the test were to assess: 1. Effectiveness of Braintree v by measuring participant success rates and errors. 2. Efficiency of Braintree v by measuring the average task time and path deviations. 3. Satisfaction with Braintree v by measuring ease of use ratings. 16

26 Braintree WCAG and Safety Enhanced Design Version Data Scoring The following table details how tasks were scored, errors evaluated, and the time data analyzed. Measures Effectiveness: Success Effectiveness: Failures Efficiency: Deviations Efficiency: Time Satisfaction: Rationale and Scoring A task was counted as a Success if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis. The total number of successes were calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage. times were recorded for successes. Observed task times divided by the optimal time for each task is a measure of optimal efficiency Optimal task performance time, as benchmarked by expert performance under realistic conditions, is recorded when constructing tasks. Target task times used for task times in the Moderator s Guide must be operationally defined by taking multiple measures of optimal performance and multiplying by some factor (2.5) that allows some time buffer because the participants are presumably not trained to expert performance. Thus, if expert, optimal performance on a task was (30) seconds then allotted task time performance was (30*2.5) seconds. This ratio should be aggregated across tasks and reported with mean and variance scores. If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion, the task was counted as an Failures. No task times were taken for errors. The total number of errors was calculated for each task and then divided by the total number of times that task was attempted. Not all deviations would be counted as errors. This should also be expressed as the mean number of failed tasks per participant. On a qualitative level, an enumeration of errors and error types should be collected. The participant s path (i.e., steps) through the application was recorded. Deviations occur if the participant, for example, went to a wrong screen, clicked on an incorrect menu item, followed an incorrect link, or interacted incorrectly with an on-screen control. This path was compared to the optimal path. The number of steps in the observed path is divided by the number of optimal steps to provide a ratio of path deviation. It is strongly recommended that the task deviations be reported. Option paths (i.e. procedural steps) should be recorded when constructing tasks. Each task was timed from when the administrator said Begin until the participant said Done / if he or she failed to say Done the time was stopped when the participant stopped performing the task. Only task times for tasks that were successfully completed were included in the average task time analysis. Average time per task was calculated for each task. Variance measures (standard deviation and standard error) were also calculated. Participant s subjective impression of the ease of use of the application was 17

27 Braintree WCAG and Safety Enhanced Design Version 3.1 Rating measured by administering both a simple post-task as a post-session questionnaire. After each task, the participant was asked to rate Overall, this task was: on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants. Common convention is that average ratings for systems judged easy to use should be 3.3 or above. To measure participants confidence in and likeability of the Braintree Overall, the testing team administered the System Usability Scale (SUS) posttest questionnaire. Questions included, I think I would like to use this system frequently, I found the product unnecessarily complex, I felt very confident using this product. ID # Role 3.13 Results Data analysis and reporting The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Participants who failed to follow session and task instructions had their data excluded from the analyses. The usability testing results for the EHRUT are detailed below (see Table 2). The data should yield actionable results that, if corrected, yield material, positive impact on user performance Table 2 (s timed in minutes) Total Deviations Errors 1 Physician Physician Nurse Physician Physician Staff Officer 7 Staff Officer 8 Nurse Staff Officer 10 Physician

28 Braintree WCAG and Safety Enhanced Design Version 3.1 Mean St. Deviation CPOE medication order CPOE Lab order CPOE Radiology order Medication List Medication allergy list Problem List Demographics Drug-drug, drug-allergy interactions checks Incorporate laboratory tests and values/results Vital Signs Electronic prescribing Clinical information reconciliation Clinical decision support Table 3 (Optimal time in minutes) :00 0:50 0:50 0:45 0:45 0:45 0:30 1:10 1:00 0:30 1:15 0:45 1: The results from the SUS (System Usability Scale) scored the subjective satisfaction with the system based on performance with these tasks to be 81. Broadly interpreted, scores under 60 represent systems with poor usability; scores over 80 would be considered above average 3.14 Discussions of the findings Major Findings The analysis of the score reported mostly consistent readings. Overall feedback of the system was positive. The overall percentage of satisfaction resulted to 81% with an average score of 4.05 out of 5. All the participants rated the system very high on usability. The participants found the system easy to use once the task instructions were explained to them. The participants appreciated the ease of use for the system and provide suggestion regarding the enhancement of design of the system and enhance the overall appearance of the system according to modern designs. 19

29 Braintree WCAG and Safety Enhanced Design Version Areas for improvement Most of the participants suggestion were ideas to help increase overall usability of the system. These included providing facility to zoom the page so it is more readable, avoid unnecessary clicks. Enhance the design of the system to look more modern looking Effectiveness The overall effectiveness of the system was fully sufficient. s were completed with varying times but few deviations. No tasks were marked as Failure with all being completed within allotted time period. Experience levels with the system allowed for various effectiveness in terms of completing tasks quickly and efficiently. Prior to the study, user experts time were recorded by an individual with extensive computer skills and vast knowledge of Braintree. These times were then multiplied by a value 2.5 to more accurately represent how a regular individual would be expected to perform Satisfaction Satisfaction was vastly positive. The system was rated highly with ease of use, effectiveness and efficiency. 20

30 Braintree WCAG and Safety Enhanced Design Version Screenshots of Braintree usability Data logger Braintree Data logger Admin menu. Data logger admin screen 21

31 Braintree WCAG and Safety Enhanced Design Version scenario sheet Usability Data logger 22

32 Braintree WCAG and Safety Enhanced Design Version Braintree Questionnaire page. Braintree Questionnaire and time score page 23

33 Braintree WCAG and Safety Enhanced Design Version Charts performance graph completion graph 24

34 Braintree WCAG and Safety Enhanced Design Version 3.1 completion and confidence graph completion time graph 25

35 Braintree WCAG and Safety Enhanced Design Version 3.1 Satisfaction chart and score. 26

36 Braintree WCAG and Safety Enhanced Design Version 3.1 Satisfaction questionnaire /Question list 27

37 Braintree WCAG and Safety Enhanced Design Version WCAG 4.15 Browser Support Braintree supports for the following browsers Internet explorer 8+ Mozilla Firefox Google Chrome Opera Safari 4.16 Techniques Braintree is a web based application and it uses following techniques for implementing WCAG XHTML CSS (Cascading style sheet) 4.17 Testing Tools Braintree used following tools for implementing WCAG. Web accessibility checker Data logger for system usability scale 28

38 Braintree WCAG and Safety Enhanced Design Version Web accessibility Checker Snapshot 29

39 Test Results Summary for 2014 Edition EHR Certification R-0007-PRA V1.14, February 28, 2016 Appendix B: Quality Management System 2015 InfoGard. May be reproduced only in its original entirety, without revision 10

40

41 Test Results Summary for 2014 Edition EHR Certification R-0007-PRA V1.14, February 28, 2016 Appendix C: Privacy and Security 2015 InfoGard. May be reproduced only in its original entirety, without revision 11

42

43

44 Test Results Summary for 2014 Edition EHR Certification R-0007-PRA V1.14, February 28, 2016 Test Results Summary Document History Version Description of Change Date V1.0 Initial release May 4, 2015 V1.1 Updated Safety-Enhanced Design report February 28, 2016 END OF DOCUMENT 2015 InfoGard. May be reproduced only in its original entirety, without revision 12