Evaluation Continued. Lecture 7 June 20, 2006

Transcription

1 Evaluation Continued Lecture 7 June 20, 2006

2 This week and next Midterm chat Anatomy of Meltdown/Review More on evaluation Case study from radiology Remember: June 27th is midterm and project day

3 Discuss Anatomy of a Meltdown Scott Adams story, and discussion ltdown.html What happened? Had a collection of 500 blog comments in a temporary holding database Clicked publish then deleted the temporary database only it wasn t temporary

4 Anatomy of Meltdown: Whose fault was it? Scott s or the designers?

5 Anatomy of a Meltdown ToG says Not Scott, here s why: Chain of 5 errors None were Scott s errors Accident waiting to happen in effect Also, no point to blaming users

6 The Scott Adams Investigation Error One: Mental (User) Model did not reflect Conceptual (Design) Model Scott: 2 databases Designers: 1 database Recall that the conceptual model is communicated via the look & feel of the interface The goal is to have the user recreate that model as his mental model through interacting with the program

7 The Scott Adams Investigation Why didn t Scott catch on? User can fail in two ways: 1. The user ends up with a fragmented model 2. The user has a complete, wrong model #2 is dangerous The users for minutes, days, or even months, systematically misterpret what the designer says

8 Example Consider the following instructions for entering a swimming pool: 1. Go to the edge 2. Confirm that the bottom is more than six feet down. 3. Hold your hands over your head 4. Begin leaning forward while bending at your knees 5. Push off the edge for maximum lift and ensure that your head points straight down for proper entry

9 The Scott Adams Investigation Users intermingle past experiences with their current experience of the interface Where there is discrepancy or ambiguity, users will misinterpret Instead Do your research! Understand users expected past experiences Then stay inline with that experience Or, clearly indicate deviations

10 The Scott Adams Investigation Error Two: Misleading Metaphor Designers rely on previous experience in attempting invoking a particular metaphor Instant deep understanding E.g. trashcan Must be true to original Features must consistent; expectations met E.g. eject by using trashcan on Mac

11 The Scott Adams Investigation Back to poor Scott Adams Publish has very specific connotations Mass replication and distribution of a document, with the original document, the draft, having little or no further usefulness after publication has been accomplished Wrt databases, publish has been drastically redefined mass replicate --> set a little flag Disaster looms

12 The Scott Adams Investigation The moral: Choose only metaphors that can be implemented in a manner sufficiently analogous to the real world and explicitly and repeatedly tell users of any necessary violations

13 The Scott Adams Investigation Error Three: Ambiguous Confirmation Dialogues Warnings must not be misterperted Scott s warning messages told him he was destroying the temporary database Instead: remove from publication destroy the only copy of this information in existence? Microsoft save changes? Evokes particular conception in new users for changing something that already exists.

14 The Scott Adams Investigation Moral: Work with a writer Ask people other than designers and engineers When you discover an ambiguity, rework and retest the dialog until ambiguity ceases to exist.

15 The Scott Adams Investigation Error Four: Confirmation Substituted for Undo Confirmations that regularly pop up are habit forming The only effect of such dialogs is to make the developers feel good: The users may be screwing up, but we warned them, so it is their own fault. Any time your user loses any work, consider it your fault Then figure out how to prevent it from happening to anyone else.

16 The Scott Adams Investigation Bizarre Can undo a single character deletion Can t recover an entire document In the case of a deletion: 1. We throw up a confirmation dialog 2. The user confirms 3. We delete the file 4. We tell the user we deleted it.

17 The Scott Adams Investigation Solution: Creating the Illusion of Deletion Leave out step 3 (delete file) Tell the user the file is gone They still can undo Later delete the file if not undone Long enough for the user to realize what he has done For the blog entries -- flag them as deleted so they do not show up, but do not delete until end of session, say Be conscious of privacy

18 The Scott Adams Investigation Moral: Offer a universal undo, and then only confirm in unusual cases

19 The Scott Adams Investigation Error Five: No Usability Evaluation Although the last four errors were inexcusable, they can be unavoidable Safety net == user testing!!!

20 The Scott Adams Investigation This disaster was preventable!!

21 Part 2: An Evaluation Framework

22 Objectives By the end of this section, you should be able to Explain key evaluation concepts & terms. Describe the evaluation paradigms & techniques used in interaction design. Discuss the conceptual and practical issues that must be considered when planning evaluations. Describe and use the DECIDE framework.

23 Evaluation Paradigms & Techniques All evaluation is affected by beliefs and bias The beliefs and practices associated with any underlying theory is called an evaluation paradigm This is different from interaction paradigm

24 4 Core Evaluation Paradigms Quick and dirty evaluation Usability testing Field studies Predictive evaluation

25 Quick and dirty evaluation Designers informally get feedback/confirmation from users or consultants Done at any stage Emphasize fast input not careful documentation

26 Usability Testing Origin 1980s Measure users performance on carefully prepared system-typical tasks Users and their software sessions are recorded Questionnaires and interviews Generally measure: Number of user errors Time to complete task Strongly controlled by evaluator Lab-like conditions

27 Usability Testing Unlike research experiments No variable manipulation Num of participants too small for stat analysis Changes in design are then engineered Recall usability engineering

28 Field Studies Done in natural settings Goal: increase understanding about what users do naturally and what impacts them Can be used to.. Identify opportunities for new technology Determine requirements for design Facilitate introduction of technology Evaluate technology

29 Field Studies Recall: Interviews Observation Ethnography Two general approaches: Outsider Observe and record while looking on Insider (or participant) E.g. ethnography

30 Predictive Evaluation Experts apply knowledge of typical users Predict usability problems No users involved Quick Inexpensive Limited E.g. Cognitive Walkthrough Heuristic Evaluation

31 Heuristic Evaluation Review by experts guided by a set of heuristics Always provide clearly marked exits Speak the user s language Called discount evaluation Report all places where heuristic is violated Design guidelines form a basis for developing heuristics Heuristics are needed that reflect the state of technology Evaluating web-based products, mobile devices, computerized toys, etc

32 Heuristic Evaluation In a study, single evaluators found only 35% of the usability problems Empirical evidence suggests that on average 5 evaluators identify 75-80% of usability problems. Heuristic evaluation is referred to as discount evaluation when 5 evaluators are used.

33 Nielsen s heuristics Visibility of system status Match between system and real world (Speak the user s language) User control and freedom (Clearly marked exits) Consistency and standards Help users recognize, diagnose, recover from errors Error prevention Recognition rather than recall Flexibility and efficiency of use Aesthetic and minimalist design Help and documentation

34 Heuristic Evaluation E.g. HOMERUN guide for web evaluation: (H)igh Quality Content (O)ften Updated (M)inimal Download Time (E)ase of Use (R)elevant to user s needs (U)nique to the online medium (N)etcentric corporate culture

35 How many evaluators? Law of Diminishing Returns 100% Proportion of Usability Problems Found (%) 75% 50% 25% 0% Number of Evaluators

36 Stages for doing heuristic evaluation Briefing session to tell experts what to do Evaluation period of 1-2 hours in which: - Each expert works separately - Take one pass to get a feel for the product - Take a second pass to focus on specific features Report results Debriefing session in which experts work together to prioritize problems

37 Heuristic Evaluation of Domain-Dependent Systems evaluators will be naïve with respect to the domain of the system: must be assisted to use the interface supply evaluators with typical usage scenarios, listing steps a user would take to perform realistic tasks constructed on the basis of a task analysis of the actual users and their work in order to be a representative as possible

38 Heuristic Evaluation: Output List of usability problems in the interface Reference violated usability principles Does not provide a systematic way to fix usability problems Does not assess the quality of any redesign Problems often easy to correct based on the guidelines for each usability principle violated

39 Heuristic Evaluation: Debriefing Session Extends heuristic evaluation to provide some design advice Participants include Evaluators Any observers of the evaluation sessions Representatives of the design team Basically a brainstorming session Focus on possible redesigns to address major usability problems A good opportunity to discuss positive aspects of the design

40 Advantages and Disadvantages Advantages Few ethical & practical issues Best experts have knowledge of application domain & users, as well as usability Disadvantages Can be difficult & expensive to find experts Biggest problems Important problems may get missed Many trivial problems are often identified

41 For more information Provides heuristics and a template so that you can evaluate different kinds of systems. More information about this is provided in the interactivities section of the idbook.com website.

42 Key Points Expert evaluation: heuristic & walkthroughs Relatively inexpensive because no users Heuristic evaluation relatively easy to learn May miss key problems & identify false ones

43 Evaluation Paradigms Summary Key aspects: Role of user Who controls Relationship btwn evaluator and user Location When used Type of data collected/how analyzed How evaluation findings are fed back Philosophy and theory underlying paradigm

44 DECIDE: A framework to guide evaluation Determine the goals the evaluation addresses. Explore the specific questions to be answered. Choose the evaluation paradigm and techniques to answer the questions. Identify the practical issues. Decide how to deal with the ethical issues. Evaluate, interpret and present the data.

45 Determine the goals Questions What are the high-level goals of the evaluation? Who wants it and why? The goals influence the paradigm for the study Some examples of goals: Identify user needs Identify the best metaphor on which to base the design. Check to ensure that the final interface is consistent. Investigate how technology affects working practices. Improve the usability of an existing product.

46 Explore the questions All evaluations need goals & questions to guide them so time is not wasted on ill-defined studies. E.g., the goal of finding out why many customers prefer to purchase paper airline tickets rather than e-tickets can be broken down into sub-questions: What are customers attitudes toward these new tickets? Are they concerned about security? Is the interface for obtaining them poor? What questions might you ask about the design of a cell phone?

47 Choose the evaluation paradigm & techniques Evaluation paradigm influences the techniques used & how data is analyzed and presented. E.g. field studies do not involve modeling

48 Identify practical issues For example, how to: select users stay on budget stay on schedule find evaluators select equipment

49 Decide on ethical issues Develop an informed consent form Participants have a right to: Know the goals of the study Know what will happen to the findings Privacy of personal information Not be quoted without their agreement Leave when they wish Be treated politely

50 Evaluate, interpret & present data Data analysis and presentation depends on paradigm and techniques The following also need to be considered: Reliability Can the study be replicated? (same results on different occasions under the same circumstances) Validity Is it measuring what you thought? Biases Is the process creating biases? Scope Can the findings be generalized? Ecological validity Is the environment of the study influencing it?

51 Pilot studies A small trial run of a study Ensures your plan is viable. Pilot studies check: That you can conduct the procedure That interview scripts, questionnaires, experiments, etc. work appropriately Also give you some initial data to help you form a better hypothesis Ask colleagues if you can t spare real users.

52 Key points An evaluation paradigm is an approach that is influenced by particular theories and philosophies. Five categories of techniques were identified: observing users, asking users, asking experts, user testing, modeling users. The DECIDE framework has six parts: Determine the overall goals Explore the questions that satisfy the goals Choose the paradigm and techniques Identify the practical issues Decide on the ethical issues Evaluate ways to analyze & present data Doing a pilot study is usually worthwhile

53 Evaluation Techniques 1. Observing users 2. Asking users their opinions 3. Asking experts their opinions 4. Testing users performance Usually in a controlled lab setting 5. Modeling users task performance Predict speed & efficiency based on #keystrokes, distance to travel, etc. Good for simple, repetitive tasks

54 Observing Users Identify needs Techniques: Notes, audio, video, interaction logs Challenge Observe but not disturb How to analyse the data Large quantities Different sources

55 Observing Users Some overlap with requirements observation techniques Different goals, though How to observe Controlled environments In the field Participant observation & ethnography

56 Observing Users Data Collection Notes plus still camera Audio recording plus still camera Video Diaries Interaction Logging

57 Observing Users Analyzing Data Qualitative analysis to tell a story for categorization Quantitative data analysis Feeding findings back into design

58 Asking Users Asking users what they think of a product Does it do what they want? Do they like it? Aesthetic appeal? Problems using it? Would they use it again? E.g. interviews, questionnaires

59 Asking Experts Software inspections/reviews Good for code and structure analysis Similar techniques for usability Heuristic evaluation Cognitive walkthrough Inexpensive Experts often have solutions to offer

60 Next Week Project Due Exam More on Phase 2 of project and presentations User Testing and Modeling