EHR Usability Test Report of SRS EHR v10

Transcription

1 EHR Usability Test Report of SRS EHR v10 Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports SRS EHR v10 Date of Usability Test: 5/18/2017 Date of Report: 5/26/2017 Report Prepared By: Harika Prabhakar, Product Owner, SRS Health Chestnut Ridge Road, Montvale NJ, Table of Contents EXECUTIVE SUMMARY... 3 INTRODUCTION... 4 METHOD... 5 SRS User Centered Design Process... 5 PARTICIPANTS... 6 STUDY DESIGN... 6 TASKS... 7 PROCEDURES... 7 TEST LOCATION... 8 TEST ENVIRONMENT... 8 TEST FORMS AND TOOLS... 8 PARTICIPANT INSTRUCTIONS... 9 USABILITY METRICS... 9 DATA SCORING... 9 RESULTS DATA ANALYSIS AND REPORTING DISCUSSION OF THE FINDINGS EFFECTIVENESS Risk Prone Errors EFFICIENCY... 13

2 SATISFACTION MAJOR FINDINGS AREAS FOR IMPROVEMENT APPENDICES Appendix 1: SAMPLE RECRUITING SCREENER Appendix 2: PARTICIPANT DEMOGRAPHICS Appendix 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM Appendix 4: EXAMPLE MODERATOR S GUIDE Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE... 31

3 EXECUTIVE SUMMARY A usability test of SRS EHR v10 Ambulatory EHR was conducted on 18 th May 2017 in the offices of Princeton Eye Group, 419 North Harrison Street, Princeton, NJ and Orthopedic Institute of Central Jersey, 2315 Route 34 S Manasquan, NJ by SRSHealth. The purpose of this test was to test and validate the usability of the current user interface, and provide evidence of usability in the SRS EHR, the EHR under Test (EHRUT). During the usability test, 10 clinical staff matching the target demographic criteria served as participants and used the EHRUT in simulated, but representative tasks. This study collected performance data on one task typically conducted on an EHR: (a)(14) Implantable Device List During the 10 minute one-on-one usability test, each participant was greeted by the administrator and asked to review and sign an informed consent/release form (included in Appendix 3); they were instructed that they could withdraw at any time. Participants had prior experience with the EHR. The administrator introduced the test, and instructed participants to complete a series of tasks (given one at a time) using the EHRUT. During the testing, the administrator timed the test and, along with the data logger, recorded user performance data on paper and electronically. The administrator did not give the participant assistance in how to complete the task. Participant screens were recorded for subsequent analysis. The following types of data were collected for each participant: Number of tasks successfully completed within the allotted time without assistance Time to complete the tasks Number and types of errors Path deviations Participant s verbalizations Participant s satisfaction ratings of the system All participant data was de-identified no correspondence could be made from the identity of the participant to the data collected. Following the conclusion of the testing, participants were asked to complete a post-test questionnaire. Various recommended metrics, in accordance with the examples set forth in the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, were used to evaluate the usability of the EHRUT. Following is a summary of the performance and rating data collected on the SRS EHR.

4 Task Measure N Effectiveness Efficiency Satisfaction Task Success % of success Task Time Deviations (Observed / Optimal) Path Deviation Deviations (Observed / Optimal) Task Time Time (Seconds) Task Ratings Rating (1=Easy) # Searching for Implantable Device of UDI format HIBCC 10 80% Adding Implantable Device through Save&Close (UDI format HIBCC ) 10 80% Viewing Implantable Device Activity 10 Editing Implantable Device 10 Reset Implantable Device 10 Adding activity to Existing Implantable Device 10 70% % % % The results from the System Usability Scale scored the subjective satisfaction with the system based on performance with these tasks to be: In addition to the performance data, the following qualitative observations were made: - Major findings Most participants found it easy to view and update the Implantable Device list information. There were minor issues with the initial searching of the device. But as users progressed through the task they found the feature easy to use and completed tasks with ease. From a usability perspective the screen would benefit from trimming extra spaces before and after the device UDI to avoid running into errors. In addition the screen would benefit from having the necessary interfaces to automatically pre-populate the screen with the device data. Lastly visual cues to alert the user as to which data elements are missing and why the user is not able to continue with the device entry/update. - Areas for improvement As stated above, the order entry UI contains some design elements which hindered usability. For example, when saving a device the user is required to enter the state & date. From a usability standpoint these fields should have a visual indicator following a standard UI paradigm to indicate required fields. Making the Implantable Device List interoperable to reduce data entry would improve usability. INTRODUCTION The EHRUT tested for this study was SRS EHR v10. It is designed to present medical information to healthcare providers in an outpatient ambulatory environment. The SRS EHR consists of various modules

5 and interfaces design to capture, display, and modify patient clinical data. The usability testing attempted to represent realistic exercises and conditions. The purpose of this study was to test and validate the usability of the current user interface, and provide evidence of usability in the EHR under Test (EHRUT). To this end, measures of effectiveness, efficiency and user satisfaction, such as time on task, user satisfaction, and deviation from optimal paths, were captured during the usability testing. METHOD SRS User Centered Design Process SRSHealth designed and implemented its own UCD process based upon the ISO industry standard. The SRS UCD process follows the same principles of human-centered design and fits in to our overall quality management system (QMS). The system involves users in the overall design and software development process. The design team includes individuals with skills spanning across multiple disciplines and with different perspectives. The team works to address the entire user experience through a deep understanding of the tasks and workflows required by end users. The process is iterative and requirements are continually updated to better address the context of use, increase usability, and reduce user error. The main steps of the process are: Understanding/stating the context of use for the function or module Business requirements are gathered from internal and client stakeholders. These stake holders are kept up to date with product developments through Feature team meetings. Creating design solutions Business Analysis work with Feature teams to design several possible solutions and through further analysis with stakeholders the solution is refined. Creating user requirements Business analysis creates full requirements documents based on feedback from Feature teams Evaluating the solution Solution prototypes are created and vetted against the original understanding of the context of use. Performing user centered evaluations Formal summative user testing is performed and the analysis is sent back to the feature teams. The results are then used to drive future iterations of the product. The following tasks and modules were developed based on the SRS UCD design process: (a)(14) Implantable Device List

6 PARTICIPANTS The testing methods were in accordance with the SRS User Centered Design Process. A total of 10 participants were tested on the EHRUT(s). Participants in the study included medical assistants, and technicians from the orthopedic and ophthalmology specialties. In addition, participants had no direct connection to the development of or organization producing the EHRUT(s). Participants were given the opportunity to have the same orientation and level of training as the actual end users would have received. For the test purposes, end-user characteristics were identified and translated into a recruitment screener used to solicit potential participants; an example of a screener is provided in Appendix 1. Recruited participants had a mix of backgrounds and demographic characteristics conforming to the recruitment screener. The following is a table of participants by characteristics, including demographics, professional experience, computing experience and user needs for assistive technology. Participant names were replaced with Participant IDs randomly generated from the demographics collection tool so that an individual s data cannot be tied back to individual identities. ID Gender Age Occupation/Role EHR Experience (Years) IDL Module Experience Female Technician 10+ None Female Manager 10+ None Female Technician 3-5 None Female Technician 6-9 None Female Medical Assistant 10+ None Female Medical Assistant 3-5 None Female Medical Assistant 6-9 None Female Medical Assistant 3-5 None Female X-Ray Technician 3-5 None Male Medical Assistant 6-9 None Ten participants (matching the demographics in the section on Participants) were recruited and 10 participated in the usability test. Participants were scheduled for 10 minute sessions with a 5 minutes session for debrief by the administrator and data logger, and to reset systems to proper test conditions. A spreadsheet was used to keep track of the participants, and included each participant s demographic characteristics. STUDY DESIGN

7 Overall, the objective of this test was to uncover areas where the application performed well that is, effectively, efficiently, and with satisfaction and areas where the application failed to meet the needs of the participants. The data from this test may serve as a baseline for future tests with an updated version of the same EHR and/or comparison with other EHRs provided the same tasks are used. In short, this testing serves as both a means to record or benchmark current usability, but also to identify areas where improvements must be made. During the usability test, participants interacted with one EHR. Each participant used the system in the same room at the the two testing locations, and was provided with the same instructions. The system was evaluated for effectiveness, efficiency and satisfaction as defined by measures collected and analyzed for each participant: Number of tasks successfully completed within the allotted time without assistance Time to complete the tasks Number and types of errors Path deviations Participant s verbalizations (comments) Participant s satisfaction ratings of the system Additional information about the various measures can be found in Section 3.9 on Usability Metrics. TASKS A number of tasks were constructed that would be realistic and representative of the kinds of activities a user might do with this EHR, including: 1. Implantable Device List: a. Searching for Implantable Device of UDI format HIBCC b. Adding Implantable Device through Save&Close (UDI format HIBCC ) c. Viewing Implantable Device Activity d. Editing Implantable Device e. Reset Implantable Device f. Adding activity to existing Implantable Device Tasks were selected based on their frequency of use, criticality of function, and those that may be most troublesome for users. PROCEDURES Upon arrival, participants were greeted; their identity was verified and matched with a name on the participant schedule. Participants were then assigned a participant ID. Each participant reviewed and signed an informed consent and release form (See Appendix 3). A representative from the test team witnessed the participant s signature. To ensure that the test ran smoothly, two staff members participated in this test, the usability administrator and the data logger. The administrator moderated the session including administering instructions and tasks. The administrator also monitored task times, obtained post-task rating data, and took notes on participant comments. A second person served as the data logger and took notes on task success, path deviations, number and type of errors, and comments. Participants were instructed to perform the tasks (see specific instructions below): As quickly as possible making as few errors

8 Without assistance; administrators were allowed to give immaterial guidance and clarification on tasks, but not instructions on use. Without using a think aloud technique. For each task, the participants were given a written copy of the task. Task timing began once the administrator finished reading the question. The task time was stopped once the participant indicated they had successfully completed the task. Scoring is discussed below in Section 3.9. Following the session, the administrator gave the participant the post-test questionnaire (the System Usability Scale, see Appendix 5) and thanked each individual for their participation. Participants' demographic information, task success rate, time on task, errors, deviations, verbal responses, and post-test questionnaire were recorded into a spreadsheet. TEST LOCATION The tests were performed in the exam room where the EHRUT would typically be deployed and used in production. Only the participant, the administrator and the data logger were in the test room. To ensure that the environment was comfortable for users, noise levels were kept to a minimum with the ambient temperature within a normal range. All of the safety instruction and evacuation procedures were valid, in place, and visible to the participants. TEST ENVIRONMENT The EHRUT would be typically be used in a healthcare office or facility. In this instance, the testing was conducted in an examination room on a computer where interactions with the EHRUT would typically take place in real world office scenarios. For testing, the computer used a Windows PC running windows 10 with a standard mouse and keyboard. The SRS EHR v10 system was viewed on a 22/14 inch monitor. The application was set up by the SRSHealth according to standard operating procedure for client/server installation. The application itself was running on a Windows 2008 Server using a Demo/Training SQL database on a LAN connection. Technically, the system performance (i.e., response time) was representative to what actual users would experience in a field implementation. Additionally, participants were instructed not to change any of the default system settings (such as control of font size). TEST FORMS AND TOOLS During the usability test, various documents and instruments were used, including: 1. Informed Consent 2. Moderator s Guide 3. Post-test Questionnaire Examples of these documents can be found in Appendices 3-5 respectively. The Moderator s Guide was devised so as to be able to capture required data. The participant s interaction with the EHRUT was captured and recorded digitally with Goto Meeting and Goto Training software running on the test machine.

9 PARTICIPANT INSTRUCTIONS The administrator reads the following instructions aloud to the each participant (also see the full moderator s guide in Appendix 4): Thank you for participating in this study. Our session today will last about 10 minutes. During that time you will take a look at an early prototype of SRS EHR vv10. I will ask you to complete a few tasks using this system and answer some questions. We are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. You will be asked to complete these tasks on your own trying to do them as quickly as possible with the fewest possible errors or deviations. Please do not do anything more than asked. If you get lost or have difficulty I cannot answer help you with anything to do with the system. Please save your detailed comments until the end of a task or the end of the session as a whole when we can discuss freely. Since this is a test system some of the data may not make sense as it is placeholder data. We are recording the screen of our session today for our internal use only. Do you have any questions or concerns? Following the procedural instructions, participants were shown the EHR. Once this task was complete, the administrator gave the following instructions: For each task, I will read the description to you and say Begin. At that point, please perform the task and say Done once you believe you have successfully completed the task. I would like to request that you not talk aloud or verbalize while you are doing the tasks. I will ask you your impressions about the task once you are done. Participants were then given six tasks to complete. Tasks are listed in the moderator s guide in Appendix 4. USABILITY METRICS According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, EHRs should support a process that provides a high level of usability for all users. The goal is for users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To this end, metrics for effectiveness, efficiency and user satisfaction were captured during the usability testing. The goals of the test were to assess: 1. Effectiveness of SRS EHR vv10 by measuring participant success rates and errors 2. Efficiency of SRS EHR vv10 by measuring the average task time and path deviations 3. Satisfaction with SRS EHR vv10 by measuring ease of use ratings DATA SCORING The following table (Table 1) details how tasks were scored, errors evaluated, and the time data analyzed Measures Effectiveness: Task Success Rationale and Scoring A task was counted as a Success if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis.

10 The total number of successes were calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage. Task times were recorded for successes. Observed task times divided by the optimal time for each task is a measure of optimal efficiency and was recorded. Optimal task performance time, as benchmarked by expert performance under realistic conditions, is recorded when constructing tasks. Target task times used for task times in the Moderator s Guide were defined by taking multiple measures of optimal performance and multiplying each time by a factor of 1.5. This factor allows some time buffer, as participants are presumably not trained to expert performance. Thus, if expert performance on a task was [x] seconds then optimal task time performance was [x * 1.5] seconds. The ratio of reported time to optimal time was aggregated across tasks and reported with mean. Effectiveness: Task Failures If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion, the task was counted as a Failures. No task times were taken for errors. On a qualitative level, an enumeration of errors and error types was collected and is described in the narrative section below. Efficiency: Task Deviations The participant s path through the application was recorded. Deviations occur if the participant, for example, went to a wrong screen, clicked on an incorrect menu item, opened an incorrect module, or interacted incorrectly with an on-screen control. This path was compared to the optimal path. The number of steps in the observed path is divided by the number of optimal steps to provide a ratio of path deviation. Optimal paths were recorded in the moderator guide. Task deviations are discussed further in the qualitative sections. Efficiency: Task Time Satisfaction: Task Rating: Each task was timed from when the administrator said Begin until the participant said, Done. If he or she failed to say Done, the time was stopped when the participant stopped performing the task. Only task times for tasks that were successfully completed were included in the average task time analysis. Average time per task was calculated for each task. Participant s subjective impression of the ease of use of the application was measured by administering both a simple post-task question as well as a postsession questionnaire. After each task, the participant was asked to rate Overall, this task was: on a scale of 1 (Very Easy) to 5 (Very Difficult). These data are averaged across participants. Common convention is that average ratings for systems judged easy to use should be 3.3 or below. To measure participants confidence in and likeability of the SRS EHR vv10 overall, the testing team administered the System Usability Scale (SUS) post-test questionnaire. Questions included, I thought the system was easy to use and I would imagine that most people would learn to use this system very quickly. See full System Usability Score questionnaire in Appendix 5.

11 Table 1. Details of how observed data were scored. RESULTS DATA ANALYSIS AND REPORTING The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Participants who failed to follow session and task instructions had their data excluded from the analyses. There were a few minor issues which may have affected data collection. The task descriptions and the terminology used could have contributed to some hesitation on the part of the user when completing tasks. Efforts were made to use descriptions and terminology which would be familiar to users; however, some users still may have experienced some confusion. As part of testing, users were presented with entirely new data entry screens. Test participants had never used these screens to create and update implantable device list information, so it is understandable that users would have some initial difficulty navigating unfamiliar screens. Task The usability testing results for the EHRUT are detailed below (see Table 2). Measure N Effectiveness Efficiency Satisfaction Task Success % of success Task Time Deviations (Observed / Optimal) Path Deviation Deviations (Observed / Optimal) Task Time Time (Seconds) Task Ratings Rating (1=Easy) # Searching for Implantable Device of UDI format HIBCC 10 80% Adding Implantable Device through Save&Close (UDI format HIBCC ) 10 80% Viewing Implantable Device Activity 10 Editing Implantable Device 10 Reset Implantable Device 10 Adding activity to Existing Implantable Device 10 70% % % % Table 2.Usability Testing Results

12 The results from the SUS (System Usability Scale) scored the subjective satisfaction with the system based on performance with these tasks to be: Broadly interpreted, scores under 60 represent systems with poor usability; scores over 80 would be considered above average. 1 DISCUSSION OF THE FINDINGS EFFECTIVENESS Based on the findings, most users were quite effective at using the EHRUT. The average number of failures per task was 2, but the errors were primarily due to the user timing out before the task could be completed. Some users spent additional time making sure that they had typed information correctly, which contributed to timing out on the task, other users verified they had completed all intended steps before completing the last step of the task and ultimately, failure on the task. However, making certain that data is entered correctly contributes to an overall goal of this project - patient safety. Most test participants were effective and showed very few, if any path deviations. The duration taken to complete the task can be attributed to the fact that this is a new feature on the application on which no one has been trained. It would be interesting to measure effectiveness once again after training has been completed once the EHRUT has been deployed to the customers and to observe if users continue to follow an optimal path. Risk Prone Errors When discussing errors, it is important to note which tasks are more likely to result in errors and what types of errors are most likely to cause patient safety issues. Tasks which do not alter the patient s record but simply display information to the user are less likely to lead to errors. Viewing device details and reviewing the patient s device history does not involve the user entering or alerting data, so the chance of error is very low. The system will only allow ancillary information associated with the device to be updated. The actual device information itself cannot be altered and is saved as retrieved from the FDA database. So there is no chance of an incorrect device being attached with a patient record. Tasks which require the user to enter data are more prone to error. To that end every effort should be made to ensure that the user can clearly discern what has been selected on the screen and that they are given an opportunity to double check and cancel / reset actions before they are committed. Below is a prioritized list of tasks in the order of the associated risk and with a rating of the risk (1-3, where 1 is a low risk and 3 is a high risk) Task Risk Level Searching for Implantable Device of UDI format HIBCC 3 Adding Implantable Device through Save&Close (UDI format HIBCC ) 1 Viewing Implantable Device Activity 1 Editing Implantable Device 1 Reset Implantable Device 1 Adding activity to Existing Added Implantable Device 1 1 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149).

13 EFFICIENCY A few users were unable to complete the tasks in what was deemed a reasonable amount of time. Users who failed to complete the task within the maximum amount of time, as determined for each task prior to testing, had their data excluded for efficiency measures. The average deviations ratios (observed path / optimal path) in the group tested were close to 1.0 for users who could complete the task. Even users who were unable to complete a task in time were generally on the correct path and rarely deviated into a different area of the software. Thus, we conclude that users were relatively efficient when completing the tasks set before them. SATISFACTION The results from the SUS (System Usability Scale) scored the subjective satisfaction with the system based on performance with these tasks to be: Verbal feedback as well as task ratings conclude that there is a high level of comfort and overall satisfaction with the system. Specifically, users stated that the system is simple and intuitive, user friendly, and organized logically. These statements, along with other participant verbalizations, suggest a high level of usability within the system. There are of course some areas of improvement, which are discussed below, but even so the average task ease of use rating was between 1.8 and 2.3 for each task. As set forth in the data scoring section, average ratings for systems judged easy to use should be 3.3 or below. Hence, a major finding of this testing was that the modules tested are very easy to use. MAJOR FINDINGS Most of the participants were very familiar with the EHRUT so very few users had difficulty with those tasks. Most users agreed that they found that they rarely had trouble using the module. Most participants found it easy to view and update the Implantable Device list information. There were minor issues with the initial searching of the device. But as users progressed through the task they found the feature easy to use and completed tasks with ease. From a usability perspective the screen would benefit from making the visual cues to alert the user as to which data elements are missing any why the user is not able to continue with the device entry/update.

14 AREAS FOR IMPROVEMENT Generally, the feedback was very positive, but there are some areas where usability could be improved. One major takeaway from this process is that gathering user feedback earlier in the development process as well as working with less advanced/beginner users rather than super users could improve usability and overall safety. As stated above, the order entry UI contains some design elements which hindered usability. For example, when saving an order the user is required to enter the state & date. From a usability standpoint these fields should have a visual indicator following a standard UI paradigm to indicate required fields. Making the Implantable Device List interoperable to reduce data entry would improve usability.

15 APPENDICES The following appendices include supplemental data for this usability test report. Following is a list of the appendices provided: 1: Sample Recruiting Screener 2: Participant Demographics 3: Non-Disclosure Agreement (NDA) and Informed Consent Form 4: Moderator s Guide 5: System Usability Scale Questionnaire

16 Appendix 1: SAMPLE RECRUITING SCREENER The purpose of a screener to ensure that the participants selected represent the target user population as closely as possible. Rather than reaching out to individual participants, SRS opted to target the Practice Administrator and rely upon them to select qualified participants based on the functionality being tested. Recruiting Scripts: Hello, my name is, and I am calling from SRSHealth. We are recruiting participants for a usability study on the SRS EHR. Would you be willing to participate, along with staff members? This is strictly for research purposes. We will primarily be conducting tests on the usability of the new implantable device list. Could you identify users in the practice such as physicians, medical assistants, and technicians who use this module? Once you have identified those users, please send them a demographics questionnaire. Would you be able to participate between May 15 th and May 20 th? The test will take place at your office. Thank you, and we appreciate your time. Participants then filled out an online demographics survey at

17 Appendix 2: PARTICIPANT DEMOGRAPHICS Following is a high-level overview of the participants in this study. Gender Men 1 Women 9 Total (participants) 10 Age Total (participants) 10 Race/Ethnicity Caucasian 8 Asian 0 Black/African-American 1 Latino/a or Hispanic 1 Other 0 Total (participants) 10 Occupation/Role Physician 0 Medical Assistant 5 Technician 4 Manager 1 Other 0 Total (participants) 10 Years of Experience Total (participants) 10

18 Following is a full participant breakdown: ID Gender Age Occupation/Role EHR Experience (Years) IDL Module Experience Assistive Tech Needs Race /Ethnicity Female Technician 10+ None N/A Caucasian Female Manager 10+ None N/A Caucasian Female Technician 3-5 None N/A Caucasian Female Technician 6-9 None N/A Caucasian Female Medical Assistant 10+ None N/A Black/Africa n-american Female Medical Assistant 3-5 None N/A Caucasian Female Medical Assistant 6-9 None N/A Caucasian Female Medical Assistant 3-5 None N/A Latino/a or Hispanic Female X-Ray Technician 3-5 None N/A Caucasian Male Medical Assistant 6-9 None N/A Caucasian

19 Appendix 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM Non-Disclosure Agreement THIS AGREEMENT is entered into as of May 18 th 2017, between ( the Participant ) and the testing organization SRSHealth located at 155 Chestnut Ridge Road, Montvale, New Jersey, The Participant acknowledges his or her voluntary participation in today s usability study may bring the Participant into possession of Confidential Information. The term "Confidential Information" means all technical and commercial information of a proprietary or confidential nature which is disclosed by SRSHealth, or otherwise acquired by the Participant, in the course of today s study. By way of illustration, but not limitation, Confidential Information includes trade secrets, processes, formulae, data, know-how, products, designs, drawings, computer aided design files and other computer files, computer software, ideas, improvements, inventions, training methods and materials, marketing techniques, plans, strategies, budgets, financial information, or forecasts. Any information the Participant acquires relating to this product during this study is confidential and proprietary to SRSHealth and is being disclosed solely for the purposes of the Participant s participation in today s usability study. By signing this form the Participant acknowledges that s/he will not disclose this confidential information obtained today to anyone else or any other organizations. Participant s printed name: Signature: Date:

20 Informed Consent SRSHealth would like to thank you for participating in this study. The purpose of this study is to evaluate an electronic health records system. If you decide to participate, you will be asked to perform several tasks using the prototype and give your feedback. The study will last about 30 minutes. Agreement I understand and agree that as a voluntary participant in the present study conducted by SRSHealth I am free to withdraw consent or discontinue participation at any time. I understand and agree to participate in the study conducted and recorded by SRSHealth. I understand and consent to the use and release of the recording by SRSHealth. I understand that the information and recording is for research purposes only and that my name and image will not be used for any purpose other than research. I relinquish any rights to the recording and understand the recording may be copied and used by SRSHealth without further permission. I understand and agree that the purpose of this study is to make software applications more useful and usable in the future. I understand and agree that the data collected from this study may be shared with outside of SRSHealth and SRSHealth s client. I understand and agree that data confidentiality is assured, because only de-identified data i.e., identification numbers not names will be used in analysis and reporting of the results. I agree to immediately raise any concerns or areas of discomfort with the study administrator. I understand that I can leave at any time. Please check one of the following: in this study. Signature: Date:

21 Appendix 4: EXAMPLE MODERATOR S GUIDE SRS EHR Usability Test Moderator s Guide Administrator Data Logger Date Time Participant # Location Prior to testing ly Prior to each participant: Prior to each task: Goto Meeting After each participant: Goto Meeting After all testing Orientation Thank you for participating in this study. Our session today will last about 10 minutes. During that time you will take a look at an early prototype of SRS EHR vv10. I will ask you to complete a few tasks using this system and answer some questions. We are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. You will be asked to complete these tasks on your own trying to do them as quickly as possible with the fewest possible errors or deviations. Please do not do anything more than asked. If you get lost or have difficulty I cannot answer help you with anything to do with the system itself. Please save your detailed comments until the end of a task or the end of the session as a whole when we can discuss freely. Since this is a test system some of the data may not make sense as it is placeholder data. We are recording the screen of our session today for our internal use only. Do you have any questions or concerns?

22 Participant Name: Task 1: Searching for Implantable Device of UDI format HIBCC (Optimal Time: Secs) Searching for Implantable Device of UDI format HIBCC Success: Task Time: Seconds Optimal Path 1) User is logged into SRS 2) User selects a patient with appointment 3) User opens the Implantable Device List app on the Interoperability Dashboard by clicking on the green + icon 4) Enter with valid UDI format (of type HIBCC) in Search box like +B NS1/$$ LOT /SXYZ /16D C1 then Press Enter Key 5) User can see the below Implantable Device information on the screen with respect to search GMDN PT: "Cardiopulmonary bypass system filter, arterial blood line" SNOMED: Brand Name: "CAP, NON-VENTED FEMALE: BLUE NON-STERILE" Version/Model: NS Company: "NOVOSCI CORP" Lot/Batch No: LOT Serial No: XYZ MRI: "Labeling does not contain MRI Safety Information" Contains Natural Rubber: false Expiration Date: 02/02/2020 Manufactured Date: 02/02/2013 HCT/P: false

23 Observed Errors and Verbalizations: Rating: Overall, this task was: Show participant written scale: Very Easy (1) to Very Difficult (5) Administrator / Notetaker Task 2: Adding Implantable Device through Save&Close (UDI format HIBCC ) (Optimal Time: Secs) Adding Implantable Device through Save&Close (UDI format HIBCC ) Success: Task Time: Seconds Optimal Path 1) User is logged into SRS 2) User selects a patient with appointment 3) User opens the Implantable Device List app on the Interoperability Dashboard by clicking on the green + icon 4) Enter with valid UDI format (of type HIBCC) in Search box like +B NS1/$$ LOT /SXYZ /16D C1 then Press Enter Key 5) User can see the below Implantable Device information on the screen with respect to search GMDN PT: "Cardiopulmonary bypass system filter, arterial blood line"

24 SNOMED: Brand Name: "CAP, NON-VENTED FEMALE: BLUE NON-STERILE" Version/Model: NS Company: "NOVOSCI CORP" Lot/Batch No: LOT Serial No: XYZ MRI: "Labeling does not contain MRI Safety Information" Contains Natural Rubber: false Expiration Date: 02/02/2020 Manufactured Date: 02/02/2013 HCT/P: false 6) User can see the below labeled with text boxes under Expanded layout grid Friendly Name: Text Box Status: Dropdown Date: Text box(calendar) Notes: Text Box Reset and Save & Close 7) User enter the Data (Free text) in the above test box Like below Example - Friendly Name: HIBCC Device Status: Active Date: 01/01/2017 Notes: Saving the Device 8) User click on Save & Close button 9) User should able to save Implantable Device successfully and screen also closed 10) User should be see the added Implantable Device on the Interoperability Dashboard

25 11) Now user again opens the Implantable Device List app on the Interoperability Dashboard by clicking on the green + icon 12) User should able to see the saved Implantable Device in the Bottom grid and information should be displayed under columns (i.e. GMDN Pt Name, Status, Date, Brand and Notes) Observed Errors and Verbalizations: Rating: Overall, this task was: Show participant written scale: Very Easy (1) to Very Difficult (5) Administrator / Notetaker Task 3: Viewing Implantable Device Activity. (Optimal Time: Secs) Viewing Implantable Device Activity Success: Task Time: Seconds Optimal Path 1) User is logged into SRS 2) User selects a patient with appointment

26 3) User opens the Implantable Device List app on the Interoperability Dashboard by clicking on the green + icon 4) User already added Implantable device data to the Bottom grid 5) User click on + icon left to saved Implantable device in the Bottom grid 6) User should able to see the information rect Observed Errors and Verbalizations: Rating: Overall, this task was: Show participant written scale: Very Easy (1) to Very Difficult (5) Administrator / Notetaker

27 Task 4: Editing Implantable Device Editing Implantable Device (Optimal Time: 55.5 Secs) Success: Task Time: Seconds Optimal Path 1) User is logged into SRS 2) User selects a patient with appointment 3) User opens the Implantable Device List app on the Interoperability Dashboard by clicking on the green + icon 4) User already added Implantable device data to the Bottom grid 5) User click on Edit icon right side to saved Implantable device in the Bottom grid 6) User enter the updated information in the columns friendly name, date and Notes and click on save/&close 7) User should able to update successfully the columns (friendly name, status date and Notes) 8) User again click on Edit icon right side to saved Implantable device in the Bottom grid (Step 4) 9) Now user should able to see the Updated information Observed Errors and Verbalizations: Rating: Overall, this task was: Show participant written scale: Very Easy (1) to Very Difficult (5)

28 Administrator / Notetaker Task 5: Reset Implantable Device (Optimal Time: Secs) Reset Implantable Device Success: Task Time: Seconds Optimal Path 1) User is logged into SRS 2) User selects a patient with appointment 3) User opens the Implantable Device List app on the Interoperability Dashboard by clicking on the green + icon 4) User already added Implantable device data to the Bottom grid 5) User click on Edit icon right side to saved Implantable device in the Bottom grid 6) User enter the new information in the columns for friendly name, status, date and Notes and do not click on save/&close 7) User click on Reset Button 8) User should able to reset to previous information Observed Errors and Verbalizations: Rating: Overall, this task was:

29 Show participant written scale: Very Easy (1) to Very Difficult (5) Administrator / Notetaker Task 6: Adding activity to Existing Implantable Device (Optimal Time: Secs) Adding activity to Existing Implantable Device Success: Task Time: Seconds Optimal Path 1) User is logged into SRS 2) User selects a patient with appointment 3) User opens the Implantable Device List app on the Interoperability Dashboard by clicking on the green + icon 4) User already added Active Implantable device data to the Bottom grid 5) User click on Edit icon right side to saved Implantable device in the Bottom grid 6) User change the status to inactive and click on save/&close button 7) Now user again opens the Implantable Device List app on the Interoperability Dashboard by clicking on the green + icon 8) User click on + icon left to saved Implantable device in the Bottom grid User should able to see the Activity by click on + icon left to it :: Describe below Observed Errors and Verbalizations:

30 Rating: Overall, this task was: Show participant written scale: Very Easy (1) to Very Difficult (5) Administrator / Notetaker

31 Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE This questionnaire was adapted from John Brooke s SUS: a quick and dirty usability scale at Digital Equipment Corporation in the UK. 2 Participants completed the questionnaire on paper. System Usability Scale Strongly Strongly Disagree Agree 1. I think that I would like to use this system frequently I found the system unnecessarily complex I thought the system was easy to use I think that I would need the support of a technical person to be able to use this system 5. I found the various functions in this system were well integrated 6. I thought there was too much inconsistency in this system 7. I would imagine that most people would learn to use this system very quickly 8. I found the system very cumbersome to use 9. I felt very confident using the system 10. I needed to learn a lot of things before I could get going with this system Brooke, J. (1996). "SUS: a "quick and dirty" usability scale." In P. W. Jordan, B. Thomas, B. A. Weerdmeester, & A. L. McClelland. Usability Evaluation in Industry. London: Taylor and Francis.