Comp 15 - Usability and Human Factors. Overview. Usability. Unit 5a - Usability Evaluation Methods

Transcription

1 Comp 15 - Usability and Human Factors Unit 5a - Usability Evaluation Methods This material was developed by Columbia University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number 1U24OC Overview Usability Evaluation methods Interviews Surveys / questionnaires Usability inspection methods Heuristic evaluation Cognitive walkthrough Usability testing Component 15/Unit 5a 2 Usability Quality of a user's experience when interacting with a product or system Factors affect the user's experience: Ease of learning / learnability Efficiency of use Supports cognitive task Memorability Error frequency and severity Esthetics Subjective satisfaction 3 1

2 Why Usability Matters: The Clinical Story At least 40% of systems are either abandoned or fail to meet minimum requirements Healthcare information and management systems society consider poor usability of clinical information systems as possibly the most important factor hindering adoption Problems with adoption, productivity and patient safety 4 Why Usability Matters: The Story from Patients and Consumers ehealth interventions offer significant promise to bridge the digital divide Problems with usability and poor design disproportionately affect lower computer literacy users Exacerbate the digital divide and possibly increase health disparities Chronic illness and Aging populations have special needs More susceptible to usability problems Consequences of poor design: Lack of empowerment, reduced enthusiasm, limited adoption and nonproductive use of patient-centered systems 5 Usability Evaluation Methods Interviews/Focus groups Questionnaires/Surveys Interviews/ Questionnaires/ Ethnographic Focus Ethnographic groups Observations Surveys Observations Usability Inspection Methods Usability Usability Testing Controlled Controlled Inspection Cognitive Usability Experiments Testing Cognitive Methods Experiments 6 2

3 Usability Evaluation 7 Interviews: Four Main Types 1. Unstructured Interview Exploratory Open-ended questions What are the advantages for accessing clincial information on mobile devices? Useful for early phase design 2. Structured Ask predetermined questions (similar to questionnaire) Questions are closed: response from a limited set of alternatives. Do you visit this website: 1) daily, 2) weekly or 3) monthly? Useful when goals are clearly understood 8 Interviews: Four Main Types (cont.) 3. Semi-Structured Interview Most widely used Starts with preplanned questions Then probes the interviewee to elaborate on emergent themes Can you tell me a little more about your thoughts about redesigning this site? 4. Group Interviews/Focus groups All interviews and focus groups are typically audio recorded and transcribed 9 3

4 Phases of the Interview 1. Introduction: The interviewer introduces himself and discusses purpose of interview 2. Warm-up session: Simple questions including demographics 3. Main session: Questions are presented in a logical sequence with the more probing questions towards the end 4. Closing session: Interviewee thanks subject and signals an end to the interview 10 Usability Interview Example Usability Interview # 15 (Upstate 4) Profile: I: So I am going to ask you a few questions, if I may. I 68 years of age may start by asking you your age? Male P: 68 English Speaker I: Have you lived in this area all your life? Originally from Pulaski area P: Yes Retired toolmaker I: The greater Pulaski area, Syracuse region? Works part-time at saw sharpening shop P: I have been here 15 years, cross the road 30 years, Some college education and was born about 4 miles country side. System was his first computer, later I: Pretty close. What do you do? bought another one P: I worked as a toolmaker; I am retired now. Lives out on the route, greater Pulaski I: What was your highest level of education? area P: I got some college; I went to night school for a Lives with wife while, Nice, comfortable house I: Was this your first computer? Has exercise machine in house P: Yes Avid hunter I: Do you monitor your blood sugar daily? Computer in bedroom-easy access, but P: Yes. no chair. I: Prior to receiving this computer, did you have the Keeps devices disconnected, keeps them home monitoring kit? in a different room. P: Yes. I have had monitor ever since I have had it. Keeps diabetes diary of results and I: So you pretty well been monitoring it? values P: Yes, for the last 8 years. 11 Focus Groups Commonly used method in marketing and social science research Typically 3 to 10 participants representative of the sample population For example: attending physicians, residents, nurses, pharmacists & administrators Discussion led by trained facilitator Preset agenda to guide discussion Sufficient flexibility to pursue emergent issues Facilitator guides and prompts discussion Encourages less talkative participants to speak up Discourages more verbose participants from dominating 12 4

5 Questionnaire / Surveys Common method for collecting user demographic data and soliciting opinions Can reach a rather large group of people Questions are typically closed, though open comments may be solicited Some use a mix of negative and positive questions Require respondent to be more attentive 13 Rating Scales: Likert Mostly commonly used Measure opinions, attitudes and beliefs Widely used for evaluating user satisfaction Statement representing a range of opinions I found the system to be very easy to use Rate response on a continuum from one extreme (e.g., highest or strongest agreement) to another (lowest or strongest disagreement) 14 Sample Usability Likert Questions 15 5

6 Rating Scales: Semantic Differential Explore a range of bipolar attitudes about a particular item Participant is asked to place mark in one of a number of positions between 2 extremes to indicate agreement I found the website to be (attractive ugly) I found the materials to be (clear confusing) 16 Online Surveys: Two Types Web-based: Can be interactive and can include check boxes, pull down menus, etc. Branching logic tailor questions to specific populations Answers written to a database -based: Electronic version of paper survey Options are limited 17 Survey Monkey Illustration of online survey Focuses on user experience Different forms of interaction Check boxes for multiple selection Radio buttons for choosing among alternatives or rating items on Likert scale Configurable and flexible 18 6

7 Usability Inspection Methods: Expert Reviews Methods for examining user interface Performed by analysts or evaluators Reliance on judgment as a source of feedback Cost effective Usability expert (or team of experts) methodically steps through the system Noting usability problems and violations to usability principles Noting the severity of problems Involves assessing the usability of a system for carrying out specific tasks (i.e., scenarios) Heuristic Evaluation: Display based Cognitive Walkthrough: Task analytic 19 Heuristic Evaluation Developed by Jakob Nielsen and Rolf Molich Method for structuring the critique of a system using simple & general heuristics Several analysts critique a system to identify & diagnose usability problems Display based Each evaluator assesses the system & notes violations of usability principles In a second pass, analyst rates severity of problems Can be used in all phases of design including storyboards, prototypes and fully functioning systems 20 Usability Principles (Nielsen, 1993) Visibility of system status Match between system and the real world User control and freedom Consistency and standards Help users recognize, diagnose, and recover from errors Minimize memory load Emphasize recognition rather than recall Flexibility and efficiency Motivation and engagement 21 7

8 More Specific Heuristics (Gerhardt-Powals, 1996) Automate unwanted workload: Free cognitive resources for high-level tasks Eliminate mental calculations, estimations, comparisons, & thinking Reduce uncertainty; display data in a clear and obvious manner Reduce cognitive load by bringing together lower level data into a higher-level summation Present new information with meaningful aids to interpretation: Use a familiar framework, everyday terms, metaphors 22 More Specific Heuristics (cont.) Use names that are conceptually related to function Context-dependent Attempt to improve recall and recognition Group data consistently meaningful ways to decrease search time Limit data-driven tasks: Reduce time assimilating raw data, appropriate use of color & graphics Include only information needed by the user at a given time 23 Heuristic Evaluation Problem Description, Recommended Solution & Commentary Problem Description, Recommended Solution & Commentary Minimalism P. Description is clear, key points should Minimalism be more visually prominent. P. Description is clear, key points should be more visually prominent. S. Use bullets or bolded Text P. Picture is extraneous and eats up valuable space. S. Summary positioned above the picture. P. An excess of text S. Convey points more succinctly, use bullets and eliminate sentences Visibility S. Use bullets or bolded Text Heuristics Violated: Consistency Standards Help P. Picture is extraneous and eats up valuable space. P. User is not fully aware of scope of session and perhaps program S. Perhaps an advance organizer and a timeline would be helpful Heuristics Violated Consistency Standards P. Can t return to session from Discussion by clicking back arrow. Control/Flexibility P. Change Program is a misleading label for editing preferences. Match P. Help is not sufficiently informative and context sensitive. S. Useful if system provides some context-sensitive guidance. P. Can t return to English after switching to Spanish Reversible action/control S. Summary positioned above the picture. 24 8

9 Problem Severity Combination of 3 Factors 1. Frequency with which a problem occurs Commonly or rarely 2. Impact of the problem Easy or difficult to overcome 3. Persistence of a problem One-time problem or constant source of difficulty Nielsen s Rating Scale: 0: Don t agree there is a problem 1: Cosmetic problem only 2: Minor usability problem-low priority 3: Major usability problem 4: Usability catastrophe-fix Now! 25 9