A Paper Prototype Usability Study of a Chronic Disease Self-management System for Older Adults

Transcription

1 A Paper Prototype Usability Study of a Chronic Disease Self-management System for Older Adults D. Tao, C. K. L. Or Department of Industrial and Manufacturing Systems Engineering, The University of Hong Kong, Pokfulam, Hong Kong (taodahuoniao@hku.hk; klor@hku.hk) Abstract - To increase the usability of a new chronic disease self-management system, we conducted a usability study of the system among older adults with chronic diseases (n=33) using a paper prototype based on the think-aloud approach. The metrics used to judge system usability were the task incompletion rate, task completion time, frequency of error, frequency of help, satisfaction, perceived usefulness, and perceived ease of use. We also interviewed the participants to elicit their comments on the system design. The quantitative data were analyzed using descriptive statistics, and the interview data were content analyzed. Overall, the participants were able to carry out the self-management tasks with the current design, but several usability issues were discovered with the system navigation, information search and interpretation, information presentation, and readability. The paper prototype study enabled us to identify these system design weaknesses and to make suitable recommendations for design modifications. Keywords Disease self-management system, older adults, paper prototype, usability evaluation I. INTRODUCTION With the advent of advanced technology, several information and communication systems have been developed and employed to expand support for the selfmanagement of chronic diseases by patients. However, many of those innovations are not regularly used in care management or have even been abandoned entirely. This issue is significant, and can be attributed to problems with the usability of the technology, such as ineffective system design (e.g., intended tasks cannot be carried out using the system); lack of ease of use and convenience of access; and a mismatch between users needs, expectations, characteristics, and the system features [1, 2]. These usability barriers result in frustration and irritation for users, inefficiency and disruption in the care management process, and a higher likelihood of committing errors [3], all of which can lead to system non-adoption. To increase the acceptance and use of any technology, designers should evaluate and ensure system usability at the early stages of system development [4]. This is especially important for healthcare technologies, as their usability can affect the quality and effectiveness of care [1]. Paper prototypes are commonly used to study usability due to their low cost and comparable effectiveness with computer-based prototypes in identifying usability problems [5, 6]. This study, which is part of a larger project to develop a computerized selfmanagement system for patients with chronic diseases, evaluated the usability problems and weaknesses of the system using a paper prototype test. The main focus was use performance and satisfaction with the system s interface design among older adults with chronic diseases. The evaluation provides guidelines and directions for the modification of the system design and for future system analysis. II. METHODOLOGY A. Self-management System Paper Prototype Our research team is working on the development of an interactive touch screen self-management system that allows patients to assess, record, and track their vital signs, including weight, blood pressure, blood glucose level, temperature, and SpO2. The assessment records can be saved in the system and retrieved for review. The system also reminds patients to take their prescribed medications at predetermined times (the time set up is carried out by the care providers, and thus no task relating to this reminder feature was tested). The creation of the system interfaces was guided by a set of human factor design principles [4, 7-9]. We used a paper prototype that consisted of a collection of color-printed screenshots of the system interface to conduct our usability study. B. Participants The participants were recruited from a non-profit medical organization that provides medical services to the community in the Hong Kong East cluster. The inclusion criteria for study participation were 1) aged 55 and older; 2) a diagnosis of chronic disease; 3) normal or correctedto-normal vision; 4) no cognitive or physical impairment; and 5) ability to read Chinese. The study received the approval of the institutional review board. Informed consent was obtained from all of the participants. C. Procedures, Materials, and Data Analysis The usability test was carried out by two trained research assistants (RAs) in a meeting room of a community health service center. Before beginning the test, one of the RAs explained the study objective and research protocol to the participants. After the participants had agreed to participate in the study and had provided informed consent, the RA gave them detailed information of the test procedures, described the purpose of the computerized self-management system, and collected

2 their basic demographic information. The participants were given two practice tasks to familiarize them with the system interface. They were then asked to perform a set of predetermined tasks (see Table I) by interacting with the color-printed paper interfaces. They were told to vocalize whatever they did, saw, and felt when performing the tasks. The experimental tasks comprised a set of navigation tasks (task 1, 4, 5, 7, and 10) and information search and simple cognitive tasks (task 2, 3, 6, 8, 9, and 11). At the beginning of the test, the participants were presented with the system s home page, which included six buttons for the six measurement modules, and were then asked to perform the experimental tasks. In the navigation tasks, the participants were told to go to the relevant measurement module. To complete the tasks, they had to search for and press the button associated with the module. In the information search and simple cognitive tasks, the participants were required to visually search for measurement values (e.g., blood glucose value) and to determine whether they were normal based on the general normal value range presented on the interfaces. The participants did not reach the information search and simple cognitive tasks if they failed to complete the preceding navigation tasks. Several performance measures were used to indicate usability: task incompletion rate, task completion time, frequency of error, and frequency of help. The task incompletion rate was defined as the percentage of participants who went through the task but were unable to complete it. completion time was the time that the participant took to perform the task. Frequency of error referred to the total number of errors made by all participants on a task (e.g., choosing the wrong button; being unable to find and explain the information correctly). Frequency of help was defined as the total number of times that all of the participants needed help on the task. The usability tests were video recorded for data extraction. Subjective outcome measures were also included. After the tasks had been completed, the RAs interviewed each of the participants and administered a questionnaire to evaluate their satisfaction with the interface design (17 items), perceived usefulness (4 items), perceived ease of use (4 items), and intention to use the system (1 item) in face-to-face interviews. The questionnaire was developed based on previous usability and technology acceptance studies [10, 11]. Except for the item intention to use the system, which was a yes-no item with a follow-up question asking the participants to explain their responses, the items were rated on a seven-point Likert scale anchored at 1 = Very bad and 7 = Very good, 1 = Strongly disagree and 7 = Strongly agree, 1 = Very unclear and 7 = Very clear, 1 = Very inappropriate and 7 = Very appropriate, or 1 = Very difficult and 7 = Very easy. At the end of the interview, two open-ended questions were asked to elicit the participants opinions about the interface design (e.g., use of font size and color and complexity) and their comments on what they liked or did not like about the system. The performance and questionnaire data were analyzed using descriptive statistics; the qualitative data (e.g., comments) were content analyzed by the two RAs. The practice task data were excluded in the data analysis. TABLE I SELF-MANAGEMENT TASKS USED IN THE USABILITY TEST Descriptions Practice Go to the SpO2 measurement module Practice Indicate the SpO2 value and determine whether it 1 Go to the blood pressure measurement module 2 Indicate the systolic pressure value and determine whether it 3 Indicate the diastolic pressure value and determine whether it 4 Go to the blood glucose measurement module 5 Select the before breakfast test time for blood glucose measurement (in this task the participants had to search for and select (by pressing ) the before-breakfast button for the measurement) 6 Indicate the blood glucose value and determine whether it is normal 7 Go to the body weight measurement module 8 Indicate the weight value 9 Indicate the body mass index (BMI) and determine whether it 10 Go to the history data page for blood pressure 11 Indicate the diastolic pressure value on a specified date on the history data chart III. RESULTS A. Participant Characteristics Forty older adults who were affiliate members of the community health service organization who met the inclusion criteria completed the usability test at the time when we wrote this paper. The first seven were pilot participants for trying out the test procedures, during which problems in the procedures were identified and fixed before the main test was performed. The other 33 adults (24 females and 9 males) participated in the main test and only their data were used for analysis. The mean age of the 33 participants was 74.1 (S.D.=9.8) years. Nine participants (27%) reported having previous experience using computers, and their average weekly use was 4.6 (S.D.=6.0) hours. None of the participants had used any type of computerized disease self-management system before. The participants reported having at least one of the following chronic conditions: hypertension (82%, 27/33), diabetes (45%, 15/33), heart disease (18%, 6/33), hypotension (3%, 1/33), prostatitis (3%, 1/33), and asthma (3%, 1/33). Seventeen (52%) of the participants reported having two or more of these chronic diseases. B. Performance Measures The data on the performance measures were analyzed with descriptive statistics. Table II shows the results. 1) incompletion rate: All of the participants

3 went through all of the navigation tasks due to the nature of the experimental design, but not every participant went through all of the information search and simple cognitive tasks due to failure to complete the preceding navigation tasks. Our analysis showed that 82 percent of all of the participants (27 of the 33) completed at least half of the 11 tasks with little to moderate guidance, whereas the other six participants (18%) experienced significant difficulty in carrying out the tasks. These six participants were able to complete only five or less of the 11 tasks. 1, 4, and 7 yielded a low task incompletion rate. 10 had the highest task incompletion rate, with 52% of the participants unable to complete the task. The task incompletion rates were moderate to high in the information search and simple cognitive tasks, ranging from 19% to 50%. For example, for task 11, 16 participants went through the task, and eight (50%) failed to complete the task. N TABLE II USER PERFORMANCE ON THE 11 TASKS Incompletion Rate Completion Time (second) Frequency of Error (n error ) Frequency of Help (n help ) Navigation s % (1) 12 (8) % (3) 7 (5) % (12) 26 (15) % (4) 7 (5) % (29) 30 (16) Information Search and Simple Cognitive s % (12) 14 (8) % (12) 5 (4) % (11) 11 (4) % (10) 9 (7) % (6) 13 (8) % (10) 10 (7) N: Total number of participants who went through the task n error : Number of participants who made errors in the task n help : Number of participants who needed help in the task 2) completion time: Among the tasks, the go to the history data page task (task 10) appeared to be the most difficult to perform and yielded the longest task completion time (57.9 seconds). 5, which required the participants to select the before breakfast test time for the blood glucose measure, had the second longest task completion time. The task completion time for the other tasks was short to medium, ranging from 5.0 to 13.2 seconds. 3) Frequency of error: The go to the history data page task (task 10) yielded the largest number of errors (62 errors) among the 11 tasks. 5 yielded 25 errors. Overall, our analysis showed that the navigation errors included choosing the wrong button when attempting to access a measurement module and incorrectly realizing that some of the icons and symbols were buttons. 4) Frequency of help: During the usability test, some of the participants inquired whether their choices were correct or asked how they should act. The RA responded to the participants with minimal support for these inquiries. As with the frequency of error, requests for help occurred most frequently for the two navigation tasks 5 and 10, again indicating their difficulty. For instance, 15 participants made 26 requests for help when choosing the test time for the blood glucose measurement (task 5). C. Satisfaction, Perceived Usefulness, and Perceived Ease of Use The analysis of the questionnaire data showed that the average rating for each of the items was at least 4 (the mid-point of the scale) (see Table III), indicating the participants positive responses to the statements in the questionnaire. The average ratings for the 17 satisfaction items ranged from 4.7 to 6.6. Of the 17 items, 13 had a mean score of 6.0 or higher. All of the perceived usefulness items and perceived ease of use items had a mean score of at least 5.9 and 6.0, respectively, showing that the participants perceived that the system was useful and easy to use. TABLE III AVERAGE RATINGS FOR THE SATISFACTION, USEFULNESS, AND EASE OF USE ITEMS Item Content Average Rating (S.D.) Satisfaction Sat1 Overall appearance of system 5.7 (1.3) Sat2 Amount of information 4.7 (1.5) Sat3 Quality of graphics 6.3 (1.1) Sat4 Size of characters 6.4 (0.9) Sat5 Readability 6.0 (1.2) Sat6 Text clarity 6.6 (0.8) Sat7 Consistency with information expectation 6.4 (0.9) Sat8 Ease of finding information 6.4 (1.1) Sat9 Mental effort in finding information 5.4 (1.5) Sat10 Helpfulness for finding health information 6.4 (1.1) Sat11 Understanding of specific health problem 5.9 (1.5) Sat12 Improvement in health knowledge 6.4 (1.0) Sat13 Improvement in knowledge of chronic 6.1 (1.5) diseases and their treatment Sat14 Efficiency in managing chronic disease 6.3 (1.1) Sat15 Encouragement to manage chronic disease 6.2 (1.2) Sat16 Improvement in managing chronic disease 6.2 (1.2) Sat17 Time saving in managing chronic disease 6.3 (1.0) Perceived Usefulness U1 Ability to manage chronic disease 6.0 (1.0) U2 Time saving in managing chronic disease 5.9 (1.1) U3 Effectiveness for disease management 6.0 (1.2) U4 Usefulness for managing chronic disease 6.3 (1.0) Perceived Ease of Use EOU1 Ease of learning the system 6.4 (0.8) EOU2 Ease of operating the system 6.0 (1.3) EOU3 Ease of being skillful at using the system 6.2 (0.9) EOU4 Ease of using the system 6.2 (1.0) D. Open-ended Question Responses The responses to the open-ended questions showed that all of the participants expressed that they liked the system. Their feedback indicated that the system was simple to use, the information on the interface was clearly presented, finding information was easy, and using the

4 system for self-management could improve their health condition. However, several system usability problems were also identified. These were grouped into four categories and are presented in Table IV. E. Intention to Use the System Twenty-one (64%) of the participants expressed their intention to use the actual system for their chronic disease self-management in the future when the system became available. Their reasons for adoption were a belief that the system would help their self-management, provide them with specific health information, save time in their selfmanagement, and improve communication with their care providers. Among those who stated that they would not use the system, cost, unfamiliarity with the technology, and limited space at home for the devices were the main reasons for non-use. TABLE IV USABILITY PROBLEMS IDENTIFIED FROM THE OPEN-ENDED QUESTION INTERVIEWS Category Navigation Information search and Interpretation Information Presentation Usability problems of the self-management system (N) Choosing the test time for the blood glucose measure was difficult and mistakes were often made in choosing the blood glucose module button (11) Entering the history data page for blood pressure was difficult and mistakes were often made in choosing the blood pressure module button (27) Navigation among different modules was complex (2) The meaning of the emoticons used to facilitate the participants information comprehension and explanation was difficult to understand (7) The reference information for normal value range was difficult to understand (14) The button for the history data page was difficult to recognize due to its inconsistency with the other buttons (10) The measurement value and measurement date on the history data chart were hard to read and identify (9) Inappropriate use of button icons (4) Complex design of the history data page interface (3) Some terminologies were hard to understand, such as SpO2 (16), the unit kg (1), and BMI (1) Readability Too small characters and too busy words (3) Low-quality graphics (9) Icons and the words on the buttons were too small (3) Low contrast of the color indicators (4) Too small decimal point symbols of the numbers (4) N: Number of participants who expressed the problems IV. DISCUSSION Our study findings indicated that all of the participants liked the system interface design, and the majority were able to perform self-management tasks with the current system design. However, we identified several usability problems and areas where the system usability could be enhanced. Conducting a usability test in the early stages of the system development process should improve the ultimate usefulness of the system, and is particularly valuable before a large-scale assessment of the system (e.g., a randomized controlled trial to examine the system effectiveness) is carried out. A. Paper Prototypes in Usability Test Although the computer-based interactive system prototypes used in usability tests allow realistic user interactions, evidence shows that paper prototypes are as effective in identifying usability problems and understanding users subjective evaluations of a system [5, 6, 12]. The findings from this study confirm the effectiveness of using this low-cost and quick-to-create prototype in usability tests. Experience from this study also shows that a paper prototype can easily evoke participants willingness to criticize the system interface. B. Findings and Recommendations for System Design Modifications The findings of this study reveal several usability problems that should be addressed in future system design. First, the navigation between different measurement modules was easy for the older adults, but the go to the history data page task (task 10) had the longest task completion time and was difficult for some participants to perform. This issue was probably due to the inconsistency between the design of the go to the history data page button and that of the other navigation buttons, which confused the participants and perhaps increased their mental effort in searching for the right button. Another possible explanation may be that this task required more steps to complete. Thus, consistent button design and minimum steps for interface navigation, as commonly recommended [4, 7], are required to improve the system usability. Second, the information search and simple cognitive tasks had moderate to high task incompletion rates. This result indicates that the information presentation on the interfaces was complex and confusing for the elderly participants. For example, although the current design of the history data chart followed a simple two-dimensional line chart design in which the measurement dates were displayed along the X-axis of the chart and the measurement values were plotted along the Y-axis, few of the participants could easily read the chart or determine the past measurement values. This suggests that a graphical representation of the measurement data that improves the older adults ability to pair the measurement dates with the corresponding measurement values should be used. Consistent with previous studies, continuous efforts are still required to consider age-related cognition decline (e.g., decline in working memory, inference formation and interpretation) in interface design for older adults [13]. More concise and intuitive methods of information presentation are also required to help older adults information retrieval and interpretation. Regarding the presentation of the information, many of the participants expressed confusion over the pictures

5 that were used to convey the purpose of the module buttons (e.g., a picture of a weight scale was used to denote weight measurement). The selection and use of those icons should be revisited and more meaningful pictures should be used to enhance the conceptual compatibility. Some of the terminologies used in the interfaces (e.g., SpO2 and BMI) were also confusing and too technical, and should be replaced by lay terms that are less ambiguous for users. The system readability was another design issue identified in the study. The font size (all of the text was in Chinese characters) was set at 18 points in the current design. However, the findings suggest that this size was too small for many of the participants to read. Similarly, the sizes of the icons and symbols were too small and the picture quality of the icons was low, which also affected their readability. It is thus suggested that the fonts, icons, and symbols should be larger to suit the elderly population, and that pictures with a high image resolution should be used for the icons. Our system design made use of redundancy to repeat content by using two formats to facilitate information comprehension among users. For instance, our design used text and redundant emoticons to convey state of health condition, which meant that users could evaluate the state of their health condition by reading the emoticons and the text. Most of the participants expressed that the redundant use of the emoticons helped them to determine whether the measurement values were normal. V. CONCLUSIONS This study provides preliminary supportive evidence of the usability of the computerized self-management system. Although none of the participants had previous experience in using any type of disease self-management system, the majority were able to use the new technology. Moreover, the low cost and quick-to-create paper prototype used in this study was able to effectively identify system usability problems. Based on these problems, recommendations for design modifications were generated and will be used to refine the system design before the future actual system analysis and the examination of the system impacts. ACKNOWLEDGMENTS We thank Priscilla Chan, social service manager of the United Christian Nethersole Community Health Service, and Professor Agnes Tiwari, Head of the School of Nursing of the University of Hong Kong, for their help with the project. This study was supported by the Seed Funding Programme for Basic Research of the University of Hong Kong (Project title: Usability analysis of a homebased electronic monitoring system in elderly patients with chronic disease using a theory-based think aloud testing protocol; PI: Calvin Or). REFERENCES [1] C. K. L. Or and B. Karsh, "A systematic review of patient acceptance of consumer health information technology," Journal of the American Medical Informatics Association, vol. 16, pp , [2] H. Jimison, et al., "Barriers and drivers of health information technology use for the elderly, chronically ill, and underserved," Evidence Report/Technology Assessment No. 175 (Prepared by the Oregon Evidence-based Practice Center under Contract No ). AHRQ Publication No. 09-E004. Rockville, MD: Agency for Healthcare Research and Quality, [3] D. Kaufman, et al., "Applying an evaluation framework for health information system design, development, and implementation," Nursing Research, vol. 55, pp , [4] J. Nielsen, Usability Engineering. San Francisco, CA: Morgan Kaufmann, [5] M. Walker, et al., "Highfidelity or low-fidelity, paper or computer? Choosing attributes when testing web prototypes," in Human Factors and Ergonomics Society 46th Annual Meeting, 2002, pp [6] R. Sefelin, et al., "Paper prototyping - what is it good for? A comparison of paper- and computer-based low-fidelity prototyping," in CHI 2003 Conference on Human Factors in Computer Systems, 2003, pp [7] B. Shneiderman and C. Plaisant, Designing the User Interface: Strategies for Effective Human-Computer Interaction. Boston, MA: Addison-Wesley, [8] M. S. Sanders and E. J. McCormick, Human Factors in Engineering and Design. New York, NY: McGraw-Hill, [9] G. Demiris, et al., "Considerations for the design of a webbased clinical monitoring and educational system for elderly patients," Journal of the American Medical Informatics Association, vol. 8, pp , [10] A. S. Grenier, et al., "Usability evaluation of an Internetbased health information/communication system for CHF patients," in 16th Triennial World Congress of the International Ergonomics Association, [11] F. D. Davis, "Perceived usefulness, perceived ease of use, and user acceptance of information technology," MIS Quarterly, vol. 13, pp , [12] M. B. Catani and D. W. Biers, "Usability evaluation and prototype fidelity: Users and usability professionals," in Human Factors and Ergonomics Society Annual Meeting, 1998, pp [13] R. W. Morrell and K. V. Echt, "Instructional design for older computer users: The influence of cognitive factors," in Aging and Skilled Performance: Advances in Theory and Application, W. A. Rogers, et al., Eds., ed Mahwah, NJ: Lawrence Erlbaum Associates, 1996, pp