Overview of Today s Lecture Analytical Evaluation / Usability Testing November 17, 2017 Analytical Evaluation Inspections Recapping cognitive walkthrough Heuristic evaluation Performance modelling 1 2 Reminder: Cognitive Walkthrough (Process) One or more experts: use these profiles, scenarios, and tasks, to step-by-step walk through a task with the interface (sketch, paper prototype, working system, etc.) At each step, ask 3 questions Will the user know the correct action? i.e. will the user know what to do in the interface to achieve the task? Will the user notice how to do the correct action? Can users see the button or menu item, or be expected to know the hotkey, etc., for the next action? Is it apparent when it is needed? Will the user associate and interpret the response from the action correctly? Will users know from the feedback that they have made a correct or incorrect choice of action? ex: find a book at Amazon.ca via search task: to buy a copy of a textbook from amazon.ca using search typical users: students who use the web regularly step 1. find the book using search Q: will users know what to do? A1: Yes they know that they must find search. A2: May know that they can filter search by department Q: will users see how to do it? A1: Yes they have seen website searches before, this uses standard icons. A2: The search has a standard dropdown menu, default to all. If user notices this is a menu, they will understand as a standard menu. May not notice it s a menu Q: will users understand from feedback whether the action was correct or not? step 2. purchase the book 3 Fall 2017 COMP 3020 4 1
Cognitive Walkthroughs Can be integrated throughout your whole design process A simple way to keep yourself grounded in the practicalities of the interface Heuristic Evaluation Define: heuristic a rule of thumb a principle that is a shortcut for solving a problem or making decisions Not always right/true, but cognitive shortcuts Fall 2017 COMP 3020 5 6 Design Heuristics Broad usability guidelines that can guide a developer s design efforts Derived from common design problems across many systems Several researchers and practitioners have developed different sets of heuristics (e.g. domain specific) Heuristic Evaluation Systematic inspection of an interface design to see if an interface complies with a set of usability heuristics, or usability guidelines. General process:» 3-5 inspectors (usability engineers, experts )» inspect interface in isolation (~1-2 hr for simple interfaces)» results are aggregated afterwards single evaluator catches ~35% usability problems 5 evaluators catch ~75% 7 8 2
Heuristic Evaluation Heuristic Evaluation: Background Developed by Jacob Nielsen in the early 1990s Proportion of Usability Problems Found 100% 75% 50% 25% Based on heuristics distilled from an empirical analysis of 249 usability problems These heuristics have been revised for current technology Heuristics still needed for some emerging technologies (e.g., VR, AR, Wearables, etc). 5 10 15 Number of Evaluators Design guidelines form a basis for developing heuristics 9 10 Nielsen s heuristics Nielsen s heuristics Visibility of system status - Are users kept informed at all times? Match between system and real world - Is the UI language simple? User control and freedom - Are there easy escapes from unexpected locations? Consistency and standards - Is performing similar action consistent? Help users recognize, diagnose, recover from errors - Are error messages helpful? Error prevention - Is it easy to make errors? Recognition rather than recall - Are objects, actions and options always visible? Flexibility and efficiency of use - Are there accelerators? Aesthetic and minimalist design - Is any unnecessary and irrelevant information provided? Help and documentation - Is help provided that can be easily searched? Could be too general and might need to be tailored to the environment E.g., HOMERUN suggested for corporate web site evaluation High-quality content Often updated Minimal download time Ease-of-use Relevant to users needs Unique to the online medium Netcentric corporate culture 11 12 3
Heuristic Evaluation: Advantages A few guidelines identify many common usability problems Fewer practical and ethical issues to deal with no participants Cheap and fast: a few guidelines identify many common usability problems Provides common evaluation template (to compare approaches, systems) Heuristic Evaluation: Problems Principles may be too general Subtleties involved in use Designer may not be able to overcome being defensive / experts may disagree You may actually have the wrong design altogether Can be hard to find experts False positives: does the rule always apply? Not complete: will miss problems Not a replacement for user testing 13 14 Performance Modelling Fitts Law Using mathematical models to generate quantitative predictions of certain interface actions or sequences of actions One of most tested, lasting models in HCI Models target acquisition performance T = Time A = Amplitude (distance) W = Width (size) of target a/b = empirically derived constants 15 16 4
Fitts Law What does the law tell us? How can you use the law? Hick-Hyman Law Fitts Law predicts how long it will take users to acquire targets once they know which target to select What about decision time? Hick-Hyman Law models the time it takes users to decide between n alternatives 17 18 Hick-Hyman Law Hick-Hyman Law When items are equi-probable: Applications? E.g., When certain items are more likely to be chosen than others: Deciding on menu depth vs. breadth 19 20 5
Hick-Hyman Law Keystroke Level Model Models decision time, not searching time Given a task consisting of a sequence of steps If the user is not familiar with the interface elements: How long will it take the user to perform those steps given a specific interface? Time to search through n items is linear, not logarithmic Keystroke Level Model (KLM) Models performance given a sequence of steps for an expert user 21 22 KLM Activity How is performance calculated? Individual steps described using operators: K = keystroking = 0.35s P = pointing = 1.10s (Fitts Law for greater precision) B = Button press or release (mouse) = 0.10 seconds (BB for mouse click = 0.20seconds) H = homing = 0.4s D = drawing = variable with length of line M = Mental operator = 1.35s R = response operator by system = 1.2s Sum up times for individual steps 23 How long would it take the user to replace all occurrences of a 4-letter word with a new 4-letter word 24 6
Activity Description Operation Time (sec) Reach for mouse H 0.40 Move pointer to Replace button P 1.10 Click on Replace button BB 0.2 Home on keyboard H 0.40 Type old word M, K4 2.75 Reach for mouse H 0.4 Move pointer to correct field P 1.10 Click on field BB 0.2 Home on keyboard H 0.4 Type new word M, K4 2.75 Reach for mouse H 0.4 Move pointer to Replace All P 1.10 Click Replace All BB 0.2 Total 11.4 According to this model, the time is 11.4 secs Performance Modelling: Advantages Can evaluate components of interface prior to building it Good for comparing different interface possibilities Evaluations with human subjects are expensive and experts can be hard to find Can get the kinks out of interface prior to full user testing/experimentation 25 26 Performance Modelling: Problems Difficult to model complex tasks E.g., consider designing a KLM for your complete project Most models consider only expert behaviour For really accurate predictions, coefficients need to be determined empirically Today s Messages: Analytical Evaluations Analytical evaluations are a set of structured techniques for evaluating interfaces without (necessarily) involving end users Often used to address major usability issues and inefficiencies prior to involving end users Should always complement rather than replace evaluations with end users 27 28 7
Usability test Usability Testing A usability test is a formal method for evaluating whether a design is learnable, efficient, memorable, can reduce errors, meets users expectations, etc. users are not being evaluated the design is being evaluated 29 30 Usability Test Rough Outline Bring in real users Have them complete tasks with your design, while you watch (ideally with your entire team) Measure and record things task completion, task time, error rates satisfaction, problem points, etc. Use the data to Usability Test Rough Outline Identify problems (major ones minor ones) Provide design suggestions to design/engineering team Iterate on the design, repeat use a think-aloud protocol, so you can hear what they are thinking 31 32 8
Important Considerations Testing Environments Usually takes place in a usability lab or other controlled space Major emphasis is on selecting representative users developing representative tasks 5-10 users typically selected Tasks usually last no more than 30 minutes The test conditions should be the same for every participant Informed consent form explains ethical issues 33 34 Testing Environments Testing Environments Best environment depends on pragmatic considerations, as well as what you re looking for Do you want your whole team to be able to view? Do you want to be able to review a test? How important are interruptions? What are you resources? 35 36 9
Pilot studies Practice runs Especially important for usability testing Make sure your plan is viable All the corners are checked (your script, questionnaires, tasks, etc., all work) It is worth doing several to iron out problems before doing the main study Ask colleagues if you can t spare real users 37 10