A Usability Study of Physicians Interaction with PDA and Laptop Applications to Access an Electronic Patient Record System

Transcription

1 A Usability Study of Physicians Interaction with PDA and Laptop Applications to Access an Electronic Patient Record System Nestor J. Rodriguez, Jose A. Borges, Yajaira Soler, Viviam Murillo Institute for Computing and Informatics Studies, University of Puerto Rico- Mayagüez Daniel Z. Sands Center for Clinical Computing, Beth Israel Deaconess Medical Center, Harvard Medical School Abstract PDA (Personal Digital Assistant) technology is beginning to emerge as a viable technology for accessing and updating patient records. A handful of studies can be found in the literature on the use of PDAs for accessing patient s records. However, none have actually presented a formal usability study of user interaction with these systems. In this paper we describe a usability study in which we compared two versions (PDA and laptop) of an application to access an electronic patient record system in terms of the efficiency and satisfaction achieved by physicians while conducting typical tasks. The results of the study revealed that physicians are significantly faster on a laptop than on a PDA reading and performing tasks that require text entry. However, they are significantly faster on a PDA than on a laptop performing tasks that only require pointing and clicking. In general they are significantly more satisfied with the actions performed to complete typical tasks on the laptop than on the PDA. 1. Introduction Personal Digital Assistants or PDAs may offer tremendous benefits in helping health care professionals access, manage, and share medical information. An increasing number of physicians have learned to benefit and rely on their abilities to reference key medical information, view daily appointments, and manage tasks and notes. As the use of PDAs has become widespread and the technology proven to be a viable alternative for many applications, more PDA based studies and projects have emerged in the health care field. These includes using PDAs as a patients education tool [1], as a clinical setting messaging system [2], and as a hand-held electronic medical record for rural settings [3]. The ability of PDAs to link to a central database for accessing and updating information allows for near-unlimited potential in developing point of care applications. An example of this is Mobile Nurse [4], a PDA application tested at the Clinical Trial Center, Seoul National University Hospital of Korea. This application allows nurses to collect vital signs and intake and output measurements, perform pain assessment, enter notes, read results, handle medical orders and care plans. In [5] the authors describe the guidelines of a project named Ward-in- Hand. The main objective of this project is to provide sophisticated solutions such as handheld computers with voice input for collecting information at the point of care. A PDA based patient data and charting system for use in the University of Washington Neonatal Intensive Care Unit

2 is described in [6]. The lessons learned from this project are described in [7]. The study described in [8] compares the utility and efficiency of keyboard and pen-based data input for clinical tasks. With the exception of the study by Rodriguez et. al. [9], none of the articles found in the literature have presented a formal usability study of user interaction with PDA applications for accessing an electronic patient record system. The Rodriguez s paper describes a usability study o nurses interacting with PDA and laptop versions of a nursing documentation application. This paper presents the first usability study of physicians interacting with PDA and laptop versions of an application to access an electronic patient record system. In the next sections we describe a usability study that compares two versions (PDA and laptop) of an application to access and update electronic patient records. The study measures and compares the efficiency and satisfaction achieved by physicians while conducting typical tasks. 2. Methods 2.1 The systems Two versions of an application to access an electronic patient record system were used for the study. The data in the system was deidentified patient data from Beth Israel Deaconess Medical Center s Online Medical Record system. These versions are described in the following two sections The PDA version: The PDA version was implemented on an ipaq 3835 running the Microsoft Pocket PC operating system. The user interface is shown in figure 1. The name of the patient, the age, weight and height appear at the top of the interface followed by seven tabs (Nt, Dx, Rx, Lb, DS, C, Tx). By selecting the appropriate tab physicians can: enter a note (Nt), specify a diagnosis (Dx), Order medications (Rx), Order laboratories (Lb), order diagnostic studies (DS), Request a consult (C), and specify other treatments (Tx). Physicians can also view labs and diagnostic studies results through another user interface. The pointing device used for this version was a small stylus. Text was entered pointing with the stylus on a keyboard displayed at the bottom of the screen or using the hand writing recognition feature of the PDA The laptop version: The laptop version was implemented on a laptop with touchpad running the Windows 2000 operating system. The interface for this version is shown in figure 2. The top portion of the window provides the patient s name, age, sex, weight, height, allergies and the primary care physician. Below the top portion there are seven tabs (N, Dx, Rx, Lb, Ds, C, Tx). By selecting the appropriate tab physicians can: enter a note (N), specify a diagnosis (Dx), Order medications (Rx), Order laboratories (Lb), order diagnostic studies (Ds), Request a consult (C), and specify other treatments (Tx). Physicians can also view labs and diagnostic studies results through another user interface. The pointing device used for this version was a touchpad. Text was entered with the laptop keyboard.

3 Figure 1. The PDA graphical us er interface Figure 2. The laptop graphical us er interface

4 2.2 Participants Twenty internal medicine resident physicians from a teaching hospital in the Boston metropolitan area participated in the study. They were selected on a first-come first-serve basis from those that responded a call for participation and were compensated for their participation. They had an average of 1.8 years of experience as resident physicians. The participants had an average of 13.7 years of experience using computers. All had experience using PDAs (mean=2.7 years) and electronic patient record systems (mean=3.3 years). None of the participants had prior experience with the applications used for the study. 2.3 The tasks Each participant was asked to perform on each version the following tasks: 1. Indicate the age and weight of the patient. 2. Find the most recent vital signs. 3. Read the result of a CHEST (PA & LAT) study. 4. Order Ibuprofen 400m PO Q8H PRN 5. Order Heparin 5000 units SC Q24H 6. Order the SMG6 and CBC laboratory tests 7. Enter specified text as a note. 2.4 Research design Two versions of an application to access an electronic patient record system were used for the study. The participants were asked to perform 7 tasks on each version. The tasks were the same in both systems. Ten of the participants performed the tasks first on the PDA version and then on the laptop version. The other ten participants performed the tasks first on the laptop version and then on the PDA version. All the participants were given a short tutorial of about 6 minutes on both versions. After performing the tasks on both systems the participants were asked to fill out a subjective user satisfaction questionnaire. The questionnaire asked the participants to rate 5 activities on a 1-7 scale (1 being poor and 7 being excellent) for each version. The activities were the following: 1. looking up vital signs list 2. reading results 3. ordering medications 4. ordering laboratory tests 5. writing notes In addition the participants were asked to mention the things of each version they found easy to use and difficult to use. They were also asked to comment on anything else they wish to about each version. 2.5 Statistical analysis The dependent variables of the study were time to complete the tasks, number of task completed, and the subjective user satisfaction. A dependent-samples t test was used to compare the time to complete the tasks. A Wilcoxon sign-ranked test with an α level of.05 was used to

5 compare differences in user satisfaction. Linear regression was used to evaluate the learning effect between the two versions. 3. Results A dependent-samples t test revealed a significant difference on the average time it took the participants to complete all the tasks on the PDA version and the laptop version t(19) = 13.45, p<.05. The participants completed all the tasks in significant less time in the laptop version (mean=199.7, SD=31.5) than in the PDA version (mean=370.4, SD=58.54). Overall, the participants were 46.1% faster with the laptop version than with the PDA version. A linear regression indicated that there was a significant learning effect carried by the participants from the laptop version to the PDA version but none from the PDA version to the laptop version. With the exception of task 1 dependent-samples t tests revealed a significant difference on the average time it took the participants to complete individual tasks with the PDA version and the laptop version. For tasks 3 to 7 the participants were faster with the laptop version than with the PDA version (see table 1). For tasks 2 the participants were faster with the PDA version than with the laptop version. Table 1. Means, s tandard deviations and s peedup for completion of individual tas ks PDA Version Laptop Version Task Mean SD Mean SD Speedup % % % % % % % The relative differences in completion time for each task on each version can be appreciated in figure 3. PDA Laptop Time (seconds) Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Figure 3. Completion times for each tas k on the PDA and the laptop vers ions

6 Most of the participants were able to complete all the tasks. Table 2 shows the number of participants that completed each of the tasks for each version of the application. Table 2. Number of participants that completed each task Task PDA Laptop A Wilcoxon test revealed a significant difference in overall users satisfaction. The participants were more satisfied with the laptop version (M=5.73) than with the PDA version (M=4.65). Wilcoxon tests also revealed significant differences in user satisfaction in all individual activities except for looking up vital signs. The relative difference in satisfaction rating for each activity on each version can be appreciated in figure 4. PDA Laptop 8 Average Rating Looking up Reading Ordering Med. Ordering Lab Writing Activity Figure 4. User satisfaction rating for each activity on the PDA and the laptop versions 4. Discussion The results revealed that except for task 2 the physicians were able to complete the tasks faster on the laptop than on the PDA. On task 2 the physicians were asked to look for the most recent vital sign data. The only actions required to accomplish this task was pointing and clicking. Thus, the results for this task evidenced that it is easier to point with a stylus on a small screen than moving a pointer and clicking with a touchpad. On task 3 the physicians were asked to read a paragraph. Physicians were 11.7% faster on the laptop. The difference in completion time can be attributed to two factors. First, the font size on the PDA was smaller and second reading on the PDA required scrolling. Task 7 was the task that resulted in the largest completion time difference. For this task physicians were asked to write a 27 words note. On the PDA twelve physicians used hand writing recognition and eight used the display keypad to do this task. Even though they used different input modalities a t test revealed that there was not a significant difference between the two groups. However, any of these two input modalities are extremely slow (3.1 times slower) in comparison with the keyboard used on the laptop. Completion of tasks 4, 5 and 6 required a combination of pointing and clicking and text entry. Since pointing and clicking is faster on the PDA the differences in completion for these three tasks can be attributed to the text input modality. Thus, due to the low performance of the

7 text input modalities of the PDA physicians completed the three tasks in significantly less time on the laptop than on the PDA. In terms of task completion only one physician was not able to complete one task on the laptop. However, there were four tasks on the PDA that one or two physicians were not able to complete. These results are significant if we consider that the only knowledge the physicians had about the applications was acquired from a short tutorial (approximately 9 minutes for the PDA and 7 minutes for the laptop) given just before the test. These results suggest that the applications featured a high degree of learnability since for all practical purposes the physicians were novice users of the applications. With the exception of looking up vital signs user satisfaction ratings were consistent with tasks completion times. The activities that were completed in less time on the laptop were rated higher in user satisfaction. Writing notes was the activity that resulted in the largest difference in satisfaction. Physicians preferred looking up vital signs on the laptop than on the PDA. Although the rating difference for this activity was not significant, the results are interesting because physicians were able to complete this task faster on the PDA than on the laptop. Writing notes was the most common activity the physicians found difficult to do on the PDA. This is not a surprising result since in comparison with the laptop their performance was extremely slow and it received the lowest user satisfaction rating. Ordering medications and laboratory tests were also found to be difficult to do on both applications by a substantial number of physicians. This is consistent with the relatively low user satisfaction rating given to these activities in both systems. Using the touchpad also figured as one of the most commonly mentioned things that were difficult to use on the laptop. Most of the things the physicians found easy to do on the both system involved viewing, looking up and reading information. A substantial number of physicians found the user interface of both systems easy to use. They also found writing notes easy to do on the laptop. The most common general comment made by physicians was about the need to provide default options for ordering medication and laboratory tests. The main reason for this comment was that the user interface of the applications did not provide options for dosages when ordering medications and they lack lists of commonly ordered tests. These features are available on the system they currently use at their hospital and thus, missed them on the applications used for the study. Finally, this study was done with residents, who may have had more exposure to computers than their older counterparts in practice. Furthermore, the particular physicians who participated worked at a technologically advanced medical center. Therefore, these results may not be generalizable to all physicians. 5. Conclusion Physicians are significantly faster performing tasks that require text entry on a laptop than on a PDA. They are also significantly faster reading on a laptop than on a PDA. However, they are significantly faster on a PDA than on a laptop performing tasks that only require pointing and clicking. In general they are significantly more satisfied with the actions performed on the laptop than on the PDA. Tasks that require text entry are the most difficult ones to perform on the PDA. Considering the high percentage of task completion in both systems and the short tutorial given to each participant on each of the versions of the application used in this study we must conclude that both applications were easy to learn.

8 The results of this study suggest that limitations of the user interface of the PDA had a significant effect on the performance of the physicians and their user satisfaction. Its small screen and in particular its text input modalities are key elements of the user interface that affect physician performance and user satisfaction. Unless text input modalities are improved, user interfaces on the PDA should be designed in such a way that text entry is minimized. Tasks requiring a significant amount of text entry should be left for applications running on laptop or desktop computers. Acknowledgements This study was made possible in part by NSF grant EIA References [1] Finkelstein, J., Nambu, S., et. al, Introducing Handheld Computing for Interactive Patient Education. Proc. AMIA Symp. November 2002, page [2] Acuff, R., Fagan, L., et. al., Lightweight, Mobile for Intra-Clinic Communication, Proc. AMIA Annu Fall Symp1997, pages [3] Anantraman, V., Mikkelsen T., Khilnani, R., et. al., Open Source Handheld-based EMR for Paramedics Working in Rural Areas, Proc. AMIA Symp. November 2001, pages [4] Choi J, Hyun S, Chun J, Lee S, Shi D, Kim D. Implementation of Mobile Computing System in Clinical Environment: MobileNurse. Proc. AMIA Symp [5] Ancona, M, Dodero, G, Minuto, F, Guida, M, Gianuzzi, V,. Mobile Computing in a Hospital the Ward-in- Hand Project, Proc. of 2000 ACM Symp. on Applied Computing, 2000, Como, Italy. [6] Carroll, A, Saluja, S, Tarczy, P, Development of a Personal Digital Assistant (PDA) Based Client/Server NICU Patient Data and Charting System. Proc. AMIA Symp. November 2001, pages [7] Carroll, A, Saluja, S, Tarczy, P, The Implementation of a Personal Digital Assistant (PDA) Based Patient Record and Charting System: Lessons Learned. Proc. AMIA Symp. November 2002, pages [8] Young, P., Leung, R., et. al., An Evaluation of the Use of Hand-Held Computers for Bedside Nursing Care, International Journal of Medical Informatics, vol. 62 (2-3) 2001, pages [9] Rodriguez, N. J., Borges, J. A., Soler, Y., Murillo, V., Sands, D. Z., Bourie, T., and Colón-Rivera, C. R. PDA vs. Laptop: A Comparison of Two Versions of a Nursing Documentation Application, 16th. IEEE International Symposium on Computer-Based Medical Systems, 2003.