Introducing Evaluation

Transcription

1 Chapter 12 Introducing Evaluation 1

2 The aims Explain the key concepts used in evaluation. Introduce different evaluation methods. Show how different methods are used for different purposes at different stages of the design process and in different contexts. Show how evaluators mix and modify methods. Discuss the practical challenges Illustrate how data analysis methods are used in evaluation and describe some methods that are specific to evaluation. 2

3 Why, what, where and when to evaluate Iterative design & evaluation is a continuous process that examines: Why: to check users requirements and that users can use the product and they like it. What: a conceptual model, early prototypes of a new system and later, more complete prototypes. Where: in natural and laboratory settings. When: throughout design; finished products can be evaluated to collect information to inform new products. 3

4 Types of evaluation Controlled settings involving users, eg usability testing & experiments in laboratories and living labs. Natural settings involving users, eg field studies to see how the product is used in the real world. Any settings not involving users, eg consultants critique; to predict, analyze & model aspects of the interface analytics. 4

5 Usability lab 5

6 Living labs People s use of technology in their everyday lives and for long time Such evaluations are too difficult to do in a usability lab. Eg the Aware Home was embedded with a complex network of sensors and audio/video recording devices (Abowd et al., 2000). 6

7 Usability testing & field studies can complement 7

8 Evaluation case studies A. Usability test to investigate a computer game B. In the wild field study of skiers C. Crowdsourcing 8

9 A. Challenge & engagement in a collaborative immersive game Physiological measures were used. Also, user satisfaction questionnaire Result: Players were more engaged when playing against another person than when playing against a computer. 9

10 Skin response 10

11 Questionnaire high values indicate more variation Playing against computer Playing against friend Mean St. Dev. Mean St. Dev. Boring Challenging Easy Engaging Exciting Frustrating Fun Source: Mandryk and Inkpen (2004). 11

12 B. Why study skiers in the wild? Jambon et al. (2009) User experience in the wild. In: Proceedings of CHI 09, ACM Press, New York, p

13 e-skiing system components Jambon et al. (2009) User experience in the wild. In: Proceedings of CHI 09, ACM Press, New York, p

14 C. Crowdsourcing Hundreds of participants Testing on their own settings Less reliable A lot of data! E.g. testing an Amazon service with standard methodologies with crowdsourcing With crowdsourcing: same results, higher variance 14

15 Evaluation methods Method Controlled settings Natural settings Without users Observing x x Asking users x x Asking experts x x Testing x Modeling x 15

16 The language of evaluation Analytics Analytical evaluation Controlled experiment Expert review or crit Field study Formative evaluation Heuristic evaluation In the wild evaluation Living laboratory Predictive evaluation Summative evaluation Usability laboratory User studies Usability testing Users or participants 16

17 The language of evaluation Data analytics: Examining large volumes of raw data purpose: drawing inferences about that information web analytics: used to measure web site traffic through analyzing users' click data Analytical evaluation: Modeling and predicting user behavior see also: heuristic evaluation 17

18 The language of evaluation Controlled experiment: A study to test hypotheses about some aspect of an interface or other dimension aspects that are controlled typically include: task that participants are asked to perform the amount of time available to complete the tasks environment in which the evaluation study occurs Crowdsourcing: A web-based method enable large population of users to evaluate a product or take part in an experiment the crowd may be asked to perform a particular evaluation task using a new product, or to rate or comment on the product. 18

19 The language of evaluation Expert review or crit: one or more people with usability expertise and knowledge of the user population review a product looking for potential problems Field study: An evaluation study that is done in a natural environment a person's home, or in a work or leisure place Formative evaluation: it is done during design to check that the product fulfills requirements and continues to meet users needs 19

20 The language of evaluation Heuristic evaluation: knowledge of typical users is applied, often guided by heuristics, to identify usability problems In the wild study: A field study in which users are observed using products or prototypes within their everyday context Living laboratory: A place that is configured to measure and record people's everyday activity in a natural setting, such as the home usually, for a long time 20

21 The language of evaluation Predictive evaluation: Evaluation methods in which theoretically based models are used to predict user performance Summative evaluation: An evaluation that is done when the design is complete Usability laboratory: A laboratory that is specially designed for usability testing User studies: A genetic term that covers a range of evaluations involving users, including field studies and experiments 21

22 The language of evaluation Usability testing: Involves measuring users' performance on various tasks Users or participants: These terms are used interchangeably to refer to the people who take part in evaluation studies 22

23 Key points Evaluation & design are closely integrated in usercentered design. Some of the same techniques are used in evaluation as for establishing requirements but they are used differently (e.g. observation interviews & questionnaires). Three types of evaluation: laboratory based with users, in the field with users, studies that do not involve users The main methods are: observing, asking users, asking experts, user testing, inspection and modeling users task performance, analytics. Dealing with constraints is an important skill for evaluators to develop. 23