ONC HIT Certification Program Test Results Summary for 2014 Edition EHR Certification

Transcription

1 Test Results Summary for 2014 Edition EHR Certification R-0079-PRI Version 1.1, February 28, 2016 ONC HIT Certification Program Test Results Summary for 2014 Edition EHR Certification Part 1: Product and Developer Information 1.1 Certified Product Information Product Name: electronic Clinical Record (ecr) Product Version: 2.8 Domain: Inpatient Test Type: Complete 1.2 Developer/Vendor Information Developer/Vendor Name: TenEleven Group Address: 6489 Transit Rd. East Amherst, NY Website: Phone: (716) Developer/Vendor Contact: Alex Alexander Part 2: ONC-Authorized Certification Body Information 2.1 ONC-Authorized Certification Body Information ONC-ACB Name: InfoGard Laboratories, Inc. Address: 709 Fiero Lane Suite 25 San Luis Obispo, CA Website: Phone: (805) ONC-ACB Contact: Adam Hardcastle This test results summary is approved for public release by the following ONC-Authorized Certification Body Representative: Adam Hardcastle ONC-ACB Authorized Representative EHR Certification Body Manager Function/Title Signature and Date 2/28/ InfoGard. May be reproduced only in its original entirety, without revision 1

2 Test Results Summary for 2014 Edition EHR Certification R-0079-PRI Version 1.1, February 28, Gap Certification The following identifies criterion or criteria certified via gap certification (a)(1) (a)(17) (d)(5) (d)(9) (a)(6) (b)(5)* (d)(6) (f)(1) (a)(7) (d)(1) (d)(8) *Gap certification allowed for Inpatient setting only No gap certification 2.3 Inherited Certification The following identifies criterion or criteria certified via inherited certification (a)(1) (a)(14) (c)(3) (f)(1) (a)(2) (a)(15) (d)(1) (f)(2) (a)(3) (a)(16) Inpt. only (d)(2) (f)(3) (a)(4) (a)(17) Inpt. only (d)(3) (f)(4) Inpt. only (a)(5) (b)(1) (d)(4) (f)(5) Optional & (a)(6) (b)(2) (d)(5) Amb. only (a)(7) (b)(3) (d)(6) (f)(6) Optional & (a)(8) (b)(4) (d)(7) Amb. only (a)(9) (b)(5) (d)(8) (g)(1) (a)(10) (b)(6) Inpt. only (d)(9) Optional (g)(2) (a)(11) (b)(7) (e)(1) (g)(3) (a)(12) (c)(1) (e)(2) Amb. only (g)(4) (a)(13) (c)(2) (e)(3) Amb. only No inherited certification 2016 InfoGard. May be reproduced only in its original entirety, without revision 2

3 Test Results Summary for 2014 Edition EHR Certification R-0079-PRI Version 1.1, February 28, 2016 Part 3: NVLAP-Accredited Testing Laboratory Information Report Number: R-0079 V1.3 Test Date(s): August 21, September 18, 2014 Location of Testing: Remote 3.1 NVLAP-Accredited Testing Laboratory Information ATL Name: InfoGard Laboratories, Inc. Accreditation Number: NVLAP Lab Code Address: 709 Fiero Lane Suite 25 San Luis Obispo, CA Website: ehr@infogard.com Phone: (805) ATL Contact: Milton Padilla For more information on scope of accreditation, please reference Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory Representative: Milton Padilla ATL Authorized Representative EHR Test Body Manager Function/Title Signature and Date 2/28/ Test Information Additional Software Relied Upon for Certification Additional Software MaxMD Direct Mail DrFirst e-prescribing Applicable Criteria (b)(1), (b)(2), (e)(1) (a)(1), (a)(2), (a)(10), (b)(3) Functionality provided by Additional Software Direct messaging Medication orders, list, erx, drug checks No additional software required 2016 InfoGard. May be reproduced only in its original entirety, without revision 3

4 Test Results Summary for 2014 Edition EHR Certification R-0079-PRI Version 1.1, February 28, Test Tools Test Tool Cypress eprescribing Validation Tool HL7 CDA Cancer Registry Reporting Validation Tool HL7 v2 Electronic Laboratory Reporting (ELR) Validation Tool HL7 v2 Immunization Information System (IIS) Reporting Valdiation T HL7 v2 Laboratory Results Interface (LRI) Validation Tool HL7 v2 Syndromic Surveillance Reporting Validation Tool Transport Testing Tool Direct Certificate Discovery Tool No test tools required Version v2.4.1 v1.0.4 v1.8.0 v1.8.0 v1.7.0 v1.8.0 v179 v Test Data Alteration (customization) to the test data was necessary and is described in Appendix No alteration (customization) to the test data was necessary Standards Multiple Standards Permitted The following identifies the standard(s) that has been successfully tested where more than one standard is permitted Criterion # (a)(8)(ii)(a)(2) (a)(13) (a)(15)(i) (a)(16)(ii) (b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context- Aware Information Retrieval (Infobutton) Domain (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context- Aware Information Retrieval (Infobutton) Domain (g) Network Time Protocol Version 3 (RFC 1305) Standard Successfully Tested (b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service- Oriented Architecture Implementation Guide (j) HL7 Version 3 Standard: Clinical Genomics; Pedigree (b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service- Oriented Architecture Implementation Guide (g) Network Time Protocol Version 4 (RFC 5905) 2016 InfoGard. May be reproduced only in its original entirety, without revision 4

5 Test Results Summary for 2014 Edition EHR Certification R-0079-PRI Version 1.1, February 28, 2016 (b)(2)(i)(a) (b)(7)(i) (e)(1)(ii)(a)(2) Common MU Data Set (15) (i) The code set specified at 45 CFR (c)(2) (ICD-10-CM) for the indicated conditions (i) The code set specified at 45 CFR (c)(2) (ICD-10-CM) for the indicated conditions (e)(1)(i) Annex A of the FIPS Publication AES 128 and SHA (g) Network Time Protocol Version 3 (RFC 1305) (e)(3)(ii) Annex A of the FIPS Publication (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (g) Network Time Protocol Version 4 (RFC 5905) (b)(2) The code set specified at 45 CFR (a)(5) (HCPCS and CPT-4) None of the criteria and corresponding standards listed above are applicable Newer Versions of Standards The following identifies the newer version of a minimum standard(s) that has been successfully tested Newer Version Applicable Criteria No newer version of a minimum standard was tested Optional Functionality Criterion # Optional Functionality Successfully Tested (a)(4)(iii) Plot and display growth charts Receive summary care record using the standards (b)(1)(i)(b) specified at (a) and (b) (Direct and XDM Validation) (b)(1)(i)(c) Receive summary care record using the standards specified at (b) and (c) (SOAP Protocols) Transmit health information to a Third Party using the (b)(2)(ii)(b) standards specified at (a) and (b) (Direct and XDM Validation) (b)(2)(ii)(c) Transmit health information to a Third Party using the standards specified at (b) and (c) (SOAP Protocols) 2016 InfoGard. May be reproduced only in its original entirety, without revision 5

6 Test Results Summary for 2014 Edition EHR Certification R-0079-PRI Version 1.1, February 28, 2016 (f)(3) Common MU Data Set (15) Ambulatory setting only Create syndrome-based public health surveillance information for transmission using the standard specified at (d)(3) (urgent care visit scenario) Express Procedures according to the standard specified at (b)(3) (45 CFR (a)(4): Code on Dental Procedures and Nomenclature) Common MU Data Express Procedures according to the standard specified Set (15) at (b)(4) (45 CFR (c)(3): ICD-10-PCS) No optional functionality tested Edition Certification Criteria* Successfully Tested Criteria # Version Version Criteria # TP** TD*** TP** TD*** (a)(1) (c)(3) (a)(2) 1.2 (d)(1) (a)(3) (d)(2) 1.5 (a)(4) (d)(3) 1.3 (a)(5) (d)(4) 1.2 (a)(6) (d)(5) (a)(7) (d)(6) (a)(8) 1.2 (d)(7) 1.2 (a)(9) (d)(8) (a)(10) (d)(9) Optional (a)(11) 1.3 (e)(1) (a)(12) 1.3 (e)(2) Amb. only (a)(13) 1.2 (e)(3) Amb. only (a)(14) 1.2 (f)(1) (a)(15) 1.5 (f)(2) (a)(16) Inpt. only (f)(3) (a)(17) Inpt. only (f)(4) Inpt. only (b)(1) (f)(5) Optional & (b)(2) Amb. only (b)(3) (f)(6) Optional & (b)(4) Amb. only (b)(5) (g)(1) (b)(6) Inpt. only (g)(2) 1.8a 2.0 (b)(7) (g)(3) 1.3 (c)(1) (g)(4) 1.2 (c)(2) *For a list of the 2014 Edition Certification Criteria, please reference (navigation: 2014 Edition Test Method) **Indicates the version number for the Test Procedure (TP) ***Indicates the version number for the Test Data (TD) 2016 InfoGard. May be reproduced only in its original entirety, without revision 6

7 Test Results Summary for 2014 Edition EHR Certification R-0079-PRI Version 1.1, February 28, Clinical Quality Measures* Type of Clinical Quality Measures Successfully Tested: Ambulatory Inpatient No CQMs tested *For a list of the 2014 Clinical Quality Measures, please reference (navigation: 2014 Clinical Quality Measures) Ambulatory CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version Inpatient CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version 9 71 v3 107 v v2 108 v v2 109 v v3 110 v2 188 v v2 190 v v v2 104 v2 114 v v InfoGard. May be reproduced only in its original entirety, without revision 7

8 Test Results Summary for 2014 Edition EHR Certification R-0079-PRI Version 1.1, February 28, Automated Numerator Recording and Measure Calculation Automated Numerator Recording Automated Numerator Recording Successfully Tested (a)(1) (a)(9) (a)(16) (b)(6) (a)(3) (a)(11) (a)(17) (e)(1) (a)(4) (a)(12) (b)(2) (e)(2) (a)(5) (a)(13) (b)(3) (e)(3) (a)(6) (a)(14) (b)(4) (a)(7) (a)(15) (b)(5) Automated Numerator Recording was not tested Automated Measure Calculation Automated Numerator Recording Successfully Tested (a)(1) (a)(9) (a)(16) (b)(6) (a)(3) (a)(11) (a)(17) (e)(1) (a)(4) (a)(12) (b)(2) (e)(2) (a)(5) (a)(13) (b)(3) (e)(3) (a)(6) (a)(14) (b)(4) (a)(7) (a)(15) (b)(5) Automated Measure Calculation was not tested Attestation Attestation Forms (as applicable) Appendix Safety-Enhanced Design* A Quality Management System** B Privacy and Security C *Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (b)(3), (b)(4) **Required for every EHR product 2016 InfoGard. May be reproduced only in its original entirety, without revision 8

9 Test Results Summary for 2014 Edition EHR Certification R-0079-PRI Version 1.1, February 28, 2016 Appendix A: Safety Enhanced Design The following required data was missing: Describe facilities Computer Type Operating System Interaction style, e.g. a mouse and keyboard Description of the display including screen size, resolution and color settings Set up of environment completed by (vendor or test laboratory) System platform Type of system (training/test database) Connection (LAN/WAN) Description of the intended users 2016 InfoGard. May be reproduced only in its original entirety, without revision 9

10 User Centered Design (a)(1) Computerized provider order entry (a)(2) Drug-drug, drug-allergy interaction checks (a)(6) Medication list (a)(7) Medication allergy list (a)(8) Clinical decision support (b)(3) Electronic prescribing (b)(4) Clinical information reconciliation Our entire ecr (electronic Clinical Record) product, including the 7 areas above, have been created using the principles of User Centered Design (UCD). Our process, Agile Scrum, is similar to that of ISO We focus on the following processes during software development: specify context of use specify user requirements produce design solutions evaluate design against requirements All ecr implementations start with a work flow study in which the customer works with TenEleven staff to document the customer s processes. This step is integral in making certain that the customer is able to utilize the software in the most efficient way possible. Enhancements are added to the system as needed, to meet customer s unique processes. TenEleven uses Agile Scrum for software development, which is an Internationally recognized development methodology. All changes and enhancements to ecr are created based upon a user story that is written using the template "As a <type of user> I want <some goal> so that <some reason>". The story is always centered on what the user wants, why they want it and how to allow them to efficiently reach the goal. We have development cycles that last 3 weeks and have a Releasable product every 4 cycles, or 12 weeks. All stories in our Product Backlog are prioritized, in regularly scheduled Agile Scrum meetings, and the highest prioritized stories are placed in the next sprint to be completed by the end of the 3 week cycle. A sprint will ordinarily consist of a mix of both customer specific stories and Product Strategy items that TenEleven has decided to do to improve our product functionality for all of our customers. At the end of each cycle we have a Potentially Releasable product that we are able to give to a customer for testing. 1 TenEleven Group Incorporated 6489 Transit Road East Amherst New York (tel) (fax) (web)

11 All of our customers are provided with a test ecr system where they can test the story, once it is completed at the end of a Sprint, to make sure the development fully meets their requirements. We will make any necessary changes based upon the customer s input and it will be released to production only after the customer has signed off on the functionality. TenEleven follows the stringent process of Agile Scrum to ensure that all functionality added to ecr meets the needs of our customers. Workflow and efficiency are always primary considerations that allow us to create a user friendly and efficient product. 2 TenEleven Group Incorporated 6489 Transit Road East Amherst New York (tel) (fax) (web)

12 EHR Usability Test Report Product: TenEleven ecr Version: 2 Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports Date of Usability Test: July 2014 Date of Report: July 2014 Report Prepared By: Alex P. Alexander Note: The following study was developed using the NISTIR 7742 template as a guide for reporting our findings: Customized Common Industry Format Template for Electronic Health Record Usability Testing. Executive Summary A usability test of the TenEleven ecr, a complete EHR, was conducted during the month of July 2014 by TenEleven Group Inc. The purpose of this test was to assess the usability of the 2014 EHR user interface, and to provide evidence of this usability. During the usability test, 5 providers (EHR users) served as participants and used the ecr in simulated, but representative tasks. This study collected performance data on tasks associated with 3 Meaningful Use criterion. These tasks were associated with the following test procedures: (a)(1) CPOE Only for Labs and Radiology orders (Med Orders covered in DrFirst document) (a)(8) Clinical Decision Support (a)(16) Electronic Medication Administration Record (b)(4) Clinical Information Reconciliation Task Detail CPOE o Med Orders See Dr. First Documentation o Lab Orders Click on Lab Orders from the Day to Day menu Select a Patient Select the lab test(s) required Fill out the Note and any other information needed Select the Send button to send the order o Radiology Orders

13 Click on Radiology Orders from the Day to Day menu Select a Patient Select the radiology test(s) required Fill out the Note and any other information needed Select the Send button to send the order Clinical Decision Support o Click on Clinical Decision Support from Day to Day menu o Create a Clinical Decision Support Rule Diagnosis, Gender - F, Age 30-40, Care Suggestion, Source then click Save o Bring up patient chart as given by administrator o Give patient Diagnosis that was in rule just created o Acknowledge Care Suggestion was delivered Electronic Medication Administration Record o Open Patient Details from Day to Day menu o Search for patient designated by administrator o click on e-prescribing tab o Prescribe 1 medication with a frequency of TID o Open patient chart and open emar Routine Meds o Administer designated medication (sign for it) o Designate a Medication Not Given then document reason o Administer a medication then designate as Signed in error then designate Error Reason o Click Save Clinical Information Reconciliation o Click on Clinical Information Reconciliation from Day to Day menu o Search for specific patient name given by administrator o Choose patient o Choose Diagnoses to add/keep in chart o Choose Medications to add/keep in chart o Choose Medication Allergies to add/keep in chart Each usability test was conducted by the user and the data logger by remote log-in to the ecr on-site testing environment. Participants of each one-on-one usability test were asked to sign a release form prior to testing. All participants were current users of the EHR software. The administrator instructed participants to complete a series of tasks, and collected the following information from each test, along with post-test data submitted via a participant questionnaire: Time to complete the task Number and types of errors Path deviations Participants verbalizations Tasks completed in the allotted time. The following is a summary of the performance and rating data collected on the EHR. Errors Deviations

14 Task Participants Task Success Time to Complete (Avg) Total Average (Total Observed/Completed Task) Total (Observed/Optimal) Average (Observed/Optimal) 1. Clinical Decision mins 3 15% 26/ Support 2. emar mins % 38/ Clinical Information Reconciliation 4. CPOE Labs & Radiology mins % 12/ min 2 8.7% 28/ In addition to the performance data, the following qualitative observations were made: 1. Major Findings 2. Areas for Improvement Introduction The EHR tested for this study was The TenEleven ecr Version 2, a complete EHR. This system is designed to represent a complete and comprehensive digital version of a patient s paper chart with real-time, patient-centered records. This system complies with all 2011 NIST certification criteria and is being submitted for verification of compliance with 2014 NIST certification criteria. The usability testing was conducted with every effort to represent realistic exercises and conditions. The purpose of this study was to test and validate the usability of the current user interface, and provide evidence of usability in the EHR. To this end, measure of effectiveness and efficiency (time to perform tasks; total number of deviations; total number of errors; etc.) were captured during the usability testing. Method Participants A total of 5 participants were tested on the ecr. Participants in the test were individuals that work within an ambulatory and inpatient healthcare environment. Participants were contacted by TenEleven. In addition, participants had no direct connection to the development of the ecr. Participants were not from TenEleven. All participants had the same level of training as all other actual end users. The following is a table of participants by characteristics, including demographics, user role, and product experience. Participant names were replaced with Participant IDs so that an individual s data cannot be tied back to individual identities. A summary of the participant demographics can be found in the Appendix. Part ID Gender Age Educa tion Occupation/ Role Professional Experience (Years) Computer Experience (Years) Product Experience (Years) Assistive Technology Needs 1 1 F BS/BA Program Director None 2 2 F BS/BA VP of Clinical Services/CCO None

15 3 3 F BS/BA Executive Vice President, Nursing Services None 4 4 F BS/BA Operations Coordinator, None Behavioral Health Services 5 5 F BS/BA IT Applications Manager None Participants were advised to allocate 60 minutes for the test but the test may end early. The added time was to ensure initial connection to the webex, provide enough time for administrator instructions and time between tasks. Study Design Overall, the objective of this test was to uncover areas where the application performed effectively, efficiently, and with satisfaction, and areas where the application failed to meet the needs of the participants. The data from this test may serve as a baseline for future tests with an updated version of the ecr provided the same tasks are used. In short, this testing serves as both a means to record or benchmark current usability, but also to identify areas where improvements must be made. During the usability test, participants interacted with the ecr. Each participant was provided the same set of instructions. The system was evaluated for effectiveness and efficiency as defined by measures collected and analyzed for each participant. Time to complete the task Number and types of errors Path deviations Participants verbalizations Tasks completed in the allotted time. A number of tasks were constructed that would be realistic and representative of the kinds of activities a user might do with this EHR, including: Add a clinical decision support rule Administer medications using the emar Perform clinical information reconciliation Order a Lab and a Radiology Test Tasks were selected based on their frequency of use, criticality of function, and those that may be most troublesome for users. Procedures Upon commencement of the test, participants were greeted, remotely logged in, and their identity was verified. Participants were then assigned a participant ID. All participants signed an informed consent form prior to the testing. The test administrator provided the instructions for each test, noted all comments from the participants; while the data logger noted all times, deviations and errors. Participants were instructed to perform the tasks: After listening to the instructions from the testing administrator As quickly as possible

16 Without assistance Task timing began after the completion of the verbal instructions from the administrator; and after an acknowledgement from the participant that they were ready to begin. The task time was stopped once the participant indicated they had successfully completed the task. Following the test, the administrator gave the participant the post-test questionnaire; and then thanked them for their time. The TenEleven staff member responsible for logging data recorded all participants demographic information, task success rate, time on task, errors, and deviations into a spreadsheet. Test Location The test was administered in a webex setting where participants were isolated from other participants in the study. Only the test administrator was on the call with the participants while the study was being administered. Test Environment The computers used for the testing were PCs running on Windows 7, Windows 8 or thin client. Users also used a mouse and keyboard while interacting with the EHR. The TenEleven application is primarily a client-server solution with some web components; and all computers were running on high speed internet connections. Test Forms and Tools None Participant Instructions The Administrator read the following instructions aloud to each participant: Thank you for participating in today s usability study of the TenEleven ecr. In a few minutes, you will be asked to perform a series of tasks and complete a user survey. Please attempt to complete the task as quickly as possible. The purpose of this study is for TenEleven to obtain information on where enhancements are needed in the application based on how quickly and easily the task is being performed in the ecr system. When it is time to perform the task, I will state the instructions and then tell you to begin. Once you have completed the task, please say Done. After you have completed the task, I will ask for feedback on the actions you had taken during the task. You will be given a specified amount of time to complete the task. This time will not be communicated to you as we are interested in seeing how long each task does take for you to perform. Usability Metrics The goals of this test were to assess: 1. The efficiency of the TenEleven ecr by measuring the length of time it takes for a user to complete the task; and the success of task completion. 2. The efficiency of the TenEleven ecr by measuring the path deviations taken by the user during the task. 3. The effectiveness of the TenEleven ecr by measuring the number and types of errors experienced by the user during the task. 4. The satisfaction of the user with the TenEleven ecr by logging their comments on the task. Data Scoring The table below details how each task was scored.

17 Measure Task Time Errors Path Deviations Rationale and Scoring Timing started when the administrator said Begin. The time ended when the participant said Done. In the event that the participant finished, and did not say Done, the administrator stopped the clock when it was clear the participant had completed the task. Task time was only counted if the participant completed the task in the allotted time. The task resulted in an error if the participant: failed to finish the task in the allotted time; or, if they became stuck and could not proceed without asking for assistance. Task time was not counted when the task resulted in an error. Path deviations were recorded as actions taken during the task that were not part of the necessary actions needed to complete the task. We calculated path deviations by taking the total number of observed deviations and dividing that number by the total number of steps taken using an optimal path. Task Success The task was considered a success if the participant completed the task in the allotted time. Results Data Analysis and Reporting The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. The testing results for the TenEleven ecr are detailed below. The table below easily identifies the tasks performed and the performance level for each task. Task Participants Task Success Time to Complete (Avg) Total Errors Average (Total Observed/Completed Task) Total (Observed/Optimal) Deviations Average (Observed/Optimal) 1. Clinical Decision mins 3 15% 26/ Support 2. emar mins % 38/ Clinical Information Reconciliation 4. CPOE Labs & Radiology mins % 12/ min 2 8.7% 28/

18 Effectiveness Participants in the study made errors in the execution of the tasks, there were no functionality errors. The bulk of the errors occurred due to the fact the piece of functionality was new to the user. These errors are expected until formal training can occur for each piece of new functionality. The Clinical Decision Support (CDS) and the Clinical Information Reconciliation (CIR) functionality are new components to the TenEleven ecr users. Both pieces of functionality are slated for future enhancements to make them more intuitive, easier to use, and require less clicks. Efficiency Participants in the study, for the most part, followed the optimal paths to complete the assigned tasks. However, there were exceptions. As mentioned above; the CDS and CIR functionality are newer components to The TenEleven ecr and it was expected that users may not be fully aware of the most optimal paths to take when performing the tasks. While reconciling the Problem list, participants were hesitant to import medical problems (Diagnoses) into their The TenEleven ecr System as this has potential to disturb their billing. Satisfaction 4 out of 5 users expressed they were Very Satisfied with the TenEleven ecr. 1 out of 5 users expressed a 2.5 out 5. Major Findings The study showed no major findings. Some of the feedback pointed to refinement of some current functionality as well as the more widespread use of our Blog so users can brainstorm together with TenEleven experts. Areas for Improvement TenEleven s plan is to implement a prioritized list of user suggestions for enhancements.

19 Appendices The following appendices include supporting data for this usability study. 1. Consent Form TenEleven ecr Usability Study Practice Consent TenEleven would like to thank you for your participation in this study. The purpose of the study is to evaluate the usability of the TenEleven ecr system. Your participation in this study will include performing a specific set of tasks within the TenEleven ecr; and completing a short survey following the study. The study should take approximately 60 minutes. The information collected by TenEleven during the study is for research purposes only. Your participation in this study is voluntary, so you are free to withdraw at any point during the study. By signing below, I agree to participate in the study. Name of Practice Name of Participant The TenEleven ecr User Role Experience w/the TenEleven ecr (Yrs) Date of Study Location Signature Date Printed Name Date

20 2. Post-Test Questionnaire TenEleven ecr Usability Study Post-Test Questionnaire 1. How would you prefer we communicate new enhancements to your practice? Members Area Page Newsletter We do not care to be notified Other 2. What is your preferred method of working with the TenEleven ecr Support department? Phone Online Chat (if this were available) Other 3. If you could change one part of the TenEleven ecr, what would you change? 4. If you could add one piece of functionality to the TenEleven ecr, what would you add? 5. On a scale of 1-5, how would rate your overall satisfaction with the TenEleven ecr? 1 (Very Dissatisfied) 2 (Somewhat Dissatisfied) 3 4 (Somewhat Satisfied) 5 (Very Satisfied)

21 3. Moderator Guide TenEleven ecr Usability Study Moderator Guide Task to perform during study Functionality Task Goal Optimal Paths Use the Clinical Decision Decision Support rule 1. Implement a Clinical Support screen to Decision Support Rule implement a new Clinical 3 minutes Perform electronic medication administration 3. Perform Clinical Information Reconciliation 4. CPOE Labs and Radiology Tests Use the Clinical Information EMAR interface to perform medication administration for a patient. Use the Clinical Information Reconciliation to choose items to include from CCDA to chart Use the Order interface for Labs and Radiology tests 4 minutes 34 2 minutes 9 1 minute Designated Task Time Task 1. Implement a Clinical Decision Support Rule 2. Perform electronic medication administration 3. Perform Clinical Information Reconciliation 4. CPOE Labs and Radiology Tests Time Designated (minutes)

22 2. Participants Demographics Gender Male 0 Female 5 The TenEleven ecr User Role Provider 1 Provider Agent 0 Clinical Staff 1 Non-Clinical Staff 3 Front Desk 0 Years of Experience w/client or Software Name <1 0 <2 3 >2 2

23 EHR Usability Test Report Product: Rcopia Version: 3 Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports Date of Usability Test: June 2013 Date of Report: June 2013 Report Prepared By: DrFirst, Inc Jason Aquilante, Product Manager jaquilante@drfirst.com 9420 Key West Ave Suite 230 Rockville, MD Note: The following study was developed using the NISTIR 7742 template as a guide for reporting our findings: Customized Common Industry Format Template for Electronic Health Record Usability Testing. DrFirst, Inc. - sales@drfirst.com (Office) (fax) 9420 Key West Avenue, Suite Rockville, MD Follow DrFirst on our Blog: Check out our videos at

24 Executive Summary A usability test of Rcopia, version 3, a modular EHR was conducted throughout the month of June 2013 by DrFirst, Inc. The purpose of this test was to test and validate the usability of the current user interface, and to provide evidence of usability. During the usability test, 10 healthcare IT users served as participants and used the EHR in simulated, but representative tasks. This study collected performance data on 11 tasks typically conducted on an EHR. The tasks conducted were related to the following: Computerized Provider Order Entry Drug-Drug/Drug-Allergy Alerts Updating a Patient s Medication List Updating a Patient s Allergy List E-Prescribing Clinical Information Reconciliation During the 45 minute one-on-one usability test, each participant was greeted by the administrator and asked to review and sign an informed consent/release form (included in Appendix). Participants had prior experience with the EHR. The administrator introduced the test, and instructed participants to complete a series of tasks (given one at a time) using the EHR. During the testing, the administrator timed the test and, along with the data logger, recorded user performance data on paper. The administrator did not give the participant assistance in how to complete the task. The following types of data were collected for each participant: Time to complete each task Number and types of errors Path deviations Participants verbalizations Tasks completed in the allotted time. All participant data was de-identified no correspondence could be made from the identity of the participant to the data collected. Following the conclusion of the testing, participants were asked to complete a post-test questionnaire. The following is a summary of the performance and rating data collected on the EHR. DrFirst, Inc. - sales@drfirst.com (Office) (fax) 9420 Key West Avenue, Suite Rockville, MD Follow DrFirst on our Blog: Check out our videos at

25 Errors Deviations Task Participant s Task Success Time to Complete (Avg) Total Avg (Total Observed/Com pleted Task) Impact Of Error Risk/Priority Total (Observed/ Optimal) Average (Observed/ Optimal) 1. Create Rx from Fav. List /8 30/ Create Rx from Drug Search % 4 1.2/4 50/ Renew 2 Active Meds /8 11/ Set preference to show all warnings 5. Prescribe medication that displays Alert 6. Add Medication through drug search % 2 2.1/2 33/ % 4.8/6 60/ % 4 1.2/4 40/ Stop 1 medication /8 21/ Add common allergy % 3.3/7 30/ Prescribe controlled Rx to EPCS pharmacy 10. Prescribe medication through Rx Report % 3 1.5/3 106/ % 3.9/5 20/ Merge two patient records % 4 2.8/1 94/ Risk Analysis and Priority Assignment To establish the risk associated with user errors, DrFirst implemented a risk analysis that developed a risk value being given to each task. Tasks with the highest risk values were prioritized in descending order. Methodology: An impact value of a user error was assigned by the scale below: Impact of Error Column: 1- No risk to patient, if error not known/re-evaluated 2- Little risk to patient, if error not known/ re-evaluated 3- Some risk to patient, if error not known/ re-evaluated 4- Significant risk to patient, if error not known/ re-evaluated DrFirst, Inc. - sales@drfirst.com (Office) (fax) 9420 Key West Avenue, Suite Rockville, MD Follow DrFirst on our Blog: Check out our videos at

26 The error average (probability of an error occurring) observed in UCD testing was multiplied by the impact value to derive a risk value. Highest priority was assigned to the highest risk value with the lowest priority assigned to the lowest risk value. In addition to the performance data, the following qualitative observations were made: 1. Major Findings 2. Areas for Improvement Introduction The EHR tested for this study was Rcopia, version 3, a modular EHR. Designed to present medical information to healthcare providers in ambulatory healthcare settings, the EHR allows providers to electronically prescribe medications to the pharmacy. The system has evolved to allow practice users to maintain a list of a patient s medications, allergies and problems. The usability testing attempted to represent realistic exercises and conditions. The purpose of this study was to test and validate the usability of the current user interface, and provide evidence of usability in the EHR. To this end, measure of effectiveness and efficiency (time to perform tasks; total number of deviations; total number of errors; etc) were captured during the usability testing. Method Participants A total of 10 participants were tested on the EHR. Participants in the test were individuals that work within an ambulatory healthcare environment. Participants were contacted by DrFirst, Inc staff to participate in the study. In addition, participants had no direct connection to the development of the EHR. Participants were not from DrFirst, Inc. All participants had the same level of training as all other actual end users. The following is a table of participants by characteristics, including demographics, user role, and product experience. Participant names were replaced with Participant IDs so that an individual s data cannot be tied back to individual identities. A summary of the participant demographics can be found in the Appendix. DrFirst, Inc. - sales@drfirst.com (Office) (fax) 9420 Key West Avenue, Suite Rockville, MD Follow DrFirst on our Blog: Check out our videos at

27 Participant ID Gender User Role Professional Experience Product Experience (Years) M Provider Agent F Provider Agent F Provider Agent F Provider Agent F Provider Agent M Provider Agent F Provider Agent F Provider Agent F Clinical Staff M Provider Agent % of all participants recruited for the test showed up to participate in the test. Participants were advised that the test would take 45 minutes; but to allocate 60 minutes for the test. The added 15 minutes was to provide enough time for administrator instructions and time between tasks. A spreadsheet was used to track participant schedules, and included each patient s demographic characteristics. Study Design Overall, the objective of this test was to uncover areas where the application performed well that is, effectively, efficiently, and with satisfaction and areas where the application failed to meet the needs of the participants. The data from this test may serve as a baseline for future tests with an updated version of the same EHR and/or comparison with other EHRs provided the same tasks are used. In short, this testing serves as both a means to record or benchmark current usability, but also to identify areas where improvements must be made. During the usability test, participants interacted with 1 EHR. Each participant was provided the same set of instructions. The system was evaluated for effectiveness and efficiency as defined by measures collected and analyzed for each participant. Time to complete each task Number and types of errors Path deviations Participants verbalizations Tasks completed in the allotted time. A number of tasks were constructed that would be realistic and representative of the kinds of activities a user might do with this EHR, including: Create Rx from Fav. List Create Rx from Drug Search Renew 2 Active Meds DrFirst, Inc. - sales@drfirst.com (Office) (fax) 9420 Key West Avenue, Suite Rockville, MD Follow DrFirst on our Blog: Check out our videos at

28 Set preference to show all warnings Prescribe medication that displays Alert Add Medication through drug search. Stop 1 medication Add common allergy Prescribe controlled Rx to EPCS pharmacy Prescribe medication through Rx Report Merge two patient records Tasks were selected based on their frequency of use, criticality of function, and those that may be most troublesome for users. Procedures Upon arrival, participants were greeted and their identity was verified and matched with the participant s name on the schedule. Participants were then assigned a participant ID. All participants signed an informed consent form prior to the testing. The test administrator witnessed each participant s signing of the consent form. To ensure that the test ran smoothly, two DrFirst staff members participated in the administration of the test. The test administrator provided the instructions for each test, and noted all comments from the participants; while the data logger noted all times, deviations and errors. Participants were instructed to perform the tasks: After listening to the instructions from the testing administrator As quickly as possible Without assistance Task timing began after the completion of the verbal instructions from the administrator; and after an acknowledgement from the participant that they were ready to begin. The task time was stopped once the participant indicated they had successfully completed the task. Following the test, the administrator gave the participant the post-test questionnaire; and then thanked them for their time. The DrFirst staff member responsible for logging data recorded all participants demographic information, task success rate, time on task, errors, and deviations into a spreadsheet. Test Location The test was administered in a setting where participants were isolated from other participants in the study. Only the test administrator and logger were with the participants while the study was being administered. To ensure that the environment was comfortable for users, noise levels were kept to a minimum with the ambient temperature within a normal range. DrFirst, Inc. - sales@drfirst.com (Office) (fax) 9420 Key West Avenue, Suite Rockville, MD Follow DrFirst on our Blog: Check out our videos at

29 Test Environment The computers used for the testing were PCs running on Windows XP or Windows 7. Users also used a mouse and keyboard while interacting with the EHR. The Rcopia application is a web-based solution; so all computers were running on high speed internet connections. All participants indicated that system performance during the test was what they were used to seeing during their typical work day. Test Forms and Tools During the usability test, various instruments and documents were used, including: Informed consent Moderator guide Post-test questionnaire Examples of these documents are to be found in the Appendix section. Participant Instructions The Administrator read the following instructions aloud to each participant: Thank you for participating in today s usability study of Rcopia. In a few minutes, you will be asked to perform a series of tasks and complete a user survey. Please attempt to complete each task as quickly as possible. The idea behind this study is for DrFirst to obtain information on where enhancements are needed in the application based on how quickly, and easily, tasks are being performed in Rcopia. When it is time to perform each task, I will state the instructions and then tell you to begin. Once you have completed the task, please say Done. After you have completed the task, I will ask for feedback on the actions you had taken during the task. You will be given a specified amount of time to complete each task. This time will not be communicated to you as we are interested in seeing how long each task does take for you to perform. Usability Metrics The goals of this test were to assess: 1. The efficiency of Rcopia by measuring the length of time it takes for a user to complete specific tasks; and the total number of tasks successfully completed during the study. 2. The efficiency of Rcopia by measuring the path deviations taken by the user during the tasks. 3. The effectiveness of Rcopia by measuring the number and types of errors experienced by the user during the tasks. 4. The satisfaction of the user with Rcopia by logging their comments on the tasks DrFirst, Inc. - sales@drfirst.com (Office) (fax) 9420 Key West Avenue, Suite Rockville, MD Follow DrFirst on our Blog: Check out our videos at

30 Data Scoring The table below details how each task was scored. Measure Task Time Errors Rationale and Scoring Timing started when the administrator said Begin. The time ended when the participant said Done. In the event that the participant finished, and did not say Done, the administrator stopped the clock when it was clear the participant had completed the task. Task times were only counted if the participant completed the task in the allotted time. The average time per task was calculated for each task. The task resulted in an error if the participant: failed to finish the task in the allotted time; or, if they became stuck and could not proceed without asking for assistance. Task time was not counted when the task resulted in an error. We calculated the error % for each task by taking the total number of errors for each task and divided that number by the total attempts at the task. Path Deviations Path deviations were recorded as actions taken during the task that were not part of the necessary actions needed to complete the task. We calculated path deviations by taking the total number of observed deviations and dividing that number by the total number of steps taken using an optimal path. Task Success A task was considered a success if the participant completed the task in the allotted time. To calculate the task success rate, we simply divided the total number of successful tasks by the total number of tasks completed. The time designated for each task was determined by taking the optimal time to complete the task and multiplying it by a factor of 1.25 to allow for those users that may not have been fully trained on the application. Results Data Analysis and Reporting The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. The testing results for Rcopia are detailed below. The table below easily identifies the tasks performed and the performance level for each task. DrFirst, Inc. - sales@drfirst.com (Office) (fax) 9420 Key West Avenue, Suite Rockville, MD Follow DrFirst on our Blog: Check out our videos at

31 Task 1. Create Rx from Fav. List 2. Create Rx from Drug Search 3. Renew 2 Active Meds 4. Set preference to show all warnings 5. Prescribe medication that displays Alert 6. Add Medication through drug search. 7. Stop 1 medication 8. Add common allergy 9. Prescribe controlled Rx to EPCS pharmacy 10. Prescribe medication through Rx Report 11. Merge two patient records Participants Task Success Time to Complete (Avg) Total Errors Average (Total Observed/Completed Task) Total (Observed/Optimal) Deviations Average (Observed/Optimal) / % 50/ / % 33/ % 60/ % 40/ / % 30/ % 106/ % 20/ % 94/ Effectiveness Participants in the study experienced errors in all but 3 tasks. However, the bulk of the errors occurred in three areas: Practice Preference Page; Pharmacy search page; and the Merge Application. Errors found in these areas were of no surprise to DrFirst. The Merge application is a newer component to DrFirst users. The practice preference page and the pharmacy search page are two areas that are focal points for future enhancements. Each of these pages present usability issues for end users; and requirements for enhancements are being drafted. Efficiency Participants in the study, for the most part, followed the optimal paths to complete the assigned tasks. However, there were two exceptions: Pharmacy Search and the Merge Application. As mentioned above; the Merge Application is a newer component to Rcopia and it was expected that users may not be fully aware of the most optimal paths to take when performing the tasks. An abnormal number of deviations were also observed when participants were asked to prescribe a DrFirst, Inc. - sales@drfirst.com (Office) (fax) 9420 Key West Avenue, Suite Rockville, MD Follow DrFirst on our Blog: Check out our videos at

32 medication to an EPCS-enabled pharmacy. Participants either elected to take extra steps to change the pharmacy; while others forgot to select a pharmacy labeled with a C to indicate it is a pharmacy that accepts Controlled Substances. Satisfaction 9 out of 10 users expressed they were Very Satisfied with Rcopia. Major Findings The study showed no major findings. Overall, the participants verbalized their happiness with the system, but many did stress they would like to see some minor enhancements. Their use of the system has not decreased over time; and most feel the functionality allows them to execute on their job functions without causing additional work. Areas for Improvement The study did verify two assumptions: enhancements are needed to optimize usability; and there needs to be better means of communication on newer features that may help improve patient safety and improve clinical workflows. As mentioned previously, we are planning to modify our practice preference (and user preference) page; as well as our pharmacy search area. We want to be sure our users can quickly (but efficiently) navigate through these areas so not to impede on their prescription writing process. Additionally, we need to find a better means of communication with our users to notify them of new features. Our post-study questionnaire revealed that 8 out of 10 participants were not aware of a particular feature within Rcopia. Improving communication with users will be discussed internally at DrFirst so we can make sure our users are aware of features that will benefit them clinically and operationally. DrFirst, Inc. - sales@drfirst.com (Office) (fax) 9420 Key West Avenue, Suite Rockville, MD Follow DrFirst on our Blog: Check out our videos at

33 Appendices The following appendices include supporting data for this usability study. 1. Sample Consent Form Rcopia Usability Study Practice Consent DrFirst would like to thank you for your participation in this study. The purpose of the study is to evaluate the usability of the Rcopia electronic prescribing system. Your participation in this study will include performing specific tasks within Rcopia; and completing a short survey following the study. The study should take approximately 30 minutes. The information collected by DrFirst during the study is for research purposes only. Your participation in this study is voluntary, so you are free to withdraw at any point during the study. By signing below, I agree to participate in the study. Name of Practice Name of Participant Rcopia User Role Experience w/rcopia (Yrs) Date of Study Location Signature Date Printed Name Date DrFirst, Inc. - sales@drfirst.com (Office) (fax) 9420 Key West Avenue, Suite Rockville, MD Follow DrFirst on our Blog: Check out our videos at

34 2. Sample Post-Test Questionnaire Rcopia User Survey 1. How would you prefer we communicate new enhancements to your practice? Members Area Page Newsletter We do not care to be notified 2. What is your preferred method of working with DrFirst Support department? Phone Online Chat (if this were available) 3. What is your preferred method for creating a prescription? Drug Search Favorites 4. How long does it take to enter a new patient in Rcopia (Demographic data only)? 0-3 minutes 3-5 minutes >5 Minutes 5. Which of the following Rcopia tasks would you use on a mobile device? Viewing a patient s record erx Updating a Patient s Med List, Allergy List, Problem List I would not use Rcopia on a Mobile device 6. If you could change one part of Rcopia, what would you change? 7. If you could add one piece of functionality to Rcopia, what would you add? 8. On a scale of 1-5, how would rate your overall satisfaction with Rcopia? 1 (Very Dissatisfied) 2 (Somewhat Dissatisfied) 3 4 (Somewhat Satisfied) 5 (Very Satisfied) 9. Which of the following functions are available through DrFirst/Rcopia? (You may check more than 1 option) Patient Education Materials Electronically Prescribe Controlled Substances Prescription Coupons for distributing to Patients Secure messaging DrFirst, Inc. - sales@drfirst.com (Office) (fax) 9420 Key West Avenue, Suite Rockville, MD Follow DrFirst on our Blog: Check out our videos at

35 3. Moderator Guide Rcopia Usability Study Computerized Provider Order Entry Tasks to perform during study: Preparation Task Goal Optimal Paths 1. No Pending Rx s 2. No Active Medications Create Rx from Favorite List 6 seconds 3 3. No Allergies 1. No Pending Rx s 2. No Active Medications Create Rx from Drug Search 34 seconds 5 3. No Allergies 1. Create 2 Active Medications Renew 2 active medications 8 seconds 1 Clinical Alerts Tasks to perform during study Preparation Task Goal Optimal Paths Set preference to have all NONE warnings be shown for Drug- Allergy interactions 27 seconds 3 Prescribe medication that is 1. Create Allergy for known to be an allergy for patient patient. 20 seconds 6 Updating Medication List Tasks to perform during study Preparation Task Goal Optimal Paths 1. No Pending Rx s Add Medication through drug 2. No Active Medications search 3. No Allergies 39 seconds 4 1. No Pending Rx s 2. No Active Medications 3. No Allergies Stop 1 medication 16 seconds 2 DrFirst, Inc. - sales@drfirst.com (Office) (fax) 9420 Key West Avenue, Suite Rockville, MD Follow DrFirst on our Blog: Check out our videos at

36 Updating Allergy List Tasks to perform during study Preparation Task Goal Optimal Paths 1. No Pending Rx s 2. No Active Medications 3. No Allergies Add common allergy 31 seconds 3 erx Tasks to perform during study Preparation Task Goal Optimal Paths Use print w/o sending to 1. No Pending Rx s prescribe controlled 2. No Active Medications 53 seconds 9 substance to EPCS enabled 3. No Allergies pharmacy. 1. Make sure at least one pending Rx for current patient Clinical Information Reconciliation Tasks to perform during study Prescribe medication through Rx Report 22 seconds 2 Preparation Task Goal Optimal Paths 1. Make sure a patient record has been sent Merge patient record from through Akario to two different sources. 120 seconds 8 provider DrFirst, Inc. - sales@drfirst.com (Office) (fax) 9420 Key West Avenue, Suite Rockville, MD Follow DrFirst on our Blog: Check out our videos at

37 4. Designated Task Times Task Time Designated (seconds) 1. Create Rx from Fav. List 6 2. Create Rx from Drug Search Renew 2 Active Meds 8 4. Set preference to show all warnings Prescribe medication that displays Alert Add Medication through drug search Stop 1 medication Add common allergy Prescribe controlled Rx to EPCS pharmacy Prescribe medication through Rx Report Merge two patient records 120 DrFirst, Inc. - sales@drfirst.com (Office) (fax) 9420 Key West Avenue, Suite Rockville, MD Follow DrFirst on our Blog: Check out our videos at

38 5. Participants Demographics Gender Male 3 Female 7 Rcopia User Role Provider 0 Provider Agent 9 Clinical Staff 1 Non-Clinical Staff 0 Front Desk 0 Years of Experience w/rcopia <1 1 <2 4 >2 5 DrFirst, Inc. - sales@drfirst.com (Office) (fax) 9420 Key West Avenue, Suite Rockville, MD Follow DrFirst on our Blog: Check out our videos at