(Milano Lugano Evaluation Method) A systematic approach to usability evaluation. Part I LAB HCI prof. Garzotto - HOC-POLITECNICO DI MILANO a.a.

Transcription

1 (Milano Lugano Evaluation Method) A systematic approach to usability evaluation Part I LAB HCI prof. Garzotto - HOC-POLITECNICO DI MILANO a.a

2 ABOUT USABILITY 1. WHAT IS USABILITY? Ë Usability is the effectiveness, efficiency and satisfaction with which specified users can achieve specified goals in particular environments (ISO ) Ë What determines usability What the product is product characteristics Who is using it user characteristics What they want to achieve goals and tasks The context circumstances of use 2

3 ABOUT USABILITY 2. USABILITY S CHALLENGE Ë Applications that people want to use understand how to use are successful in using are satisfied with make fewer errors with Ë Higher levels of use conquer new visitors, make customer come back Ë Lower user support costs fewer problems and complaints Ë Contribute to trust building and branding 3

4 ABOUT USABILITY 3. EVALUATION DEPTH Ë The usability of an application can be analysed at three levels of depth: 1 : general (for any interactive digital artifact) 2 : category (e.g. web application) 3 : destination (e.g. Museum web site) 4

5 ABOUT USABILITY 4. EXISTING USABILITY METHODS / 1 Ë Two categories: Empirical testing: involves some representative users evaluators collect usability data from users simulating a session of use Problems: gap btw real sessions and simulated ones the problem of representative users effective for detecting impact of overall look&feel high costs Main techniques: Thinking aloud Contextual inquiry Focus group 5

6 ABOUT USABILITY 5. EXISTING USABILITY METHODS / 2 Inspection methods: involve usability experts ONLY evaluator systematically performs tasks on the application effective for detecting usability and design errors and breakdowns analytic, repeatable, at lower costs than empirical testing Drawback: may neglect or underestimate some problems Main techniques: Heuristic evaluation Pluralistic Walkthrough 6

7 ABOUT USABILITY 5. EXISTING USABILITY METHODS / 3 It is empirically proved that the most effective evaluation (i.e., the highest amount of usability problems discovery) is achieved by systematically combining evaluation and empirical testing 7

8 Ó MiLE methodology: principles 8

9 MiLE methodology: principles WHAT IS MILE? Ë MILE is a Systematic Usability Evaluation method systematically combining inspection method with empirical testing Ë Developed at Politecnico di Milano (Hypermedia Open Centre) and University of Lugano, Communication Sciences (TEC-lab) Ë Up to now applied to museum web sites, e- government, and e-commerce 9

10 MiLE methodology: principles MILE ACTIVITIES Ë COMBINATION OF INSPECTION AND EMPIRICAL TESTING Ë INSPECTION: evaluation of usability attributes that are independent from the specific goals and users of the application Ë EMPIRICAL TESTING: DRIVEN BY (focused by means of) THE RESULTS OF INSPECTION Ë ORGANIZED IN TWO MAIN PHASES PRELIMINARY PHASE Defining or choosing the evaluation tools (U-KIT) EXECUTION PHASE Performing the evaluation 10

11 MiLE methodology: principles KEY CONCEPTS Ë Analysis Levels Ë Usability attributes (heuristics) Ë Abstract Task Ë Scenario Ë Measurement based on Scoring and Weighting 11

12 MiLE methodology: principles Different perspectives/dimensions of analysis during inspection During inspection, MiLE emphasizes the need for: Ëseparating different perspectives of analysis: Technical perspective: The analysis of the aspects that are independent from the specific user requirements Requirement perspective The analysis of the aspects that can be judget only in relationship to the actual user requirements Ëseparating different dimensions of analysis, corresponding to different design dimensions: NAVIGATION CONTENT OPERATIONAL FUNCTIONALITY COGNITIVE ASPECTS SEMIOTIC ASPECTS GRAPHICS TECHNOLOGY (PERFORMANCE) 12

13 MiLE methodology: principles ATTRIBUTES (HEURISTICS)_1 MiLE attributes are Ë usability factors - application properties that contribute to usability, e.g. Learnability Predictability... Ë Heuristic, i.e., based on common sense, usability experience, principles from different disciplines Ë Have different levels of abstractions and generality, and should be decomposed into finer grained, more measurable sub-factors 13

14 MiLE methodology: principles ATTRIBUTES (HEURISTICS)_2 MiLE attributes can be organized as: Ë Technical attributes: Can be measured or judged largely indipendent from the user actual requirements E.g. Lay-out consistency Ë Requirements-dependent attributes: Can be measured or judged only in relationship with the user actual requirements (profile and goals of the user) E.g. Content completeness, content richness, functional completeness, learnability Ë Orthogonally, most attributes can be organized according to the different design/analyisis dimensions Navigation, content, lay-out,... (see previous slide) 14

15 MiLE methodology: principles ATTRIBUTES (HEURISTICS)_3 Evaluating usability requires evaluating the attributes by which we choose to describe usability Ë During inspection: The inspector arguments about the satisfaction/violation of some attributes; assigning a quantitative value to each attribute (evaluation ) is based on inspector s personal judgement MiLE makes inspection activities more systematic (based on scenarios and abstract tasks) but the evaluation/ranking of attributes is still subjective 15

16 MiLE methodology: principles ATTRIBUTES (HEURISTICS)_4 Ë During empirical testing Collection of empirical data about a set of MEASURABLE factors associated to some usability attribute performance measures (data about the user behavior with the system): e.g., (Efficiency: how well a user achieves goals of interest) time to complete a task, number and percentage of tasks completed correctly (resp. incorrectly) with and without assistance; time of task completation; (Orientation: the degree of user understanding on where he is in the hyperspace) number of use/request of help; number of incorrect links traversed; number of backs ;.. preference measures: data about the user opinition or user satisfaction, collected by means of questionnaires, de-briefings, user s ranking etc. 16

17 MiLE methodology: principles GUIDING THE INSPECTION AND THE EMPIRICAL TESTING PLANNING HOW TO DEFINE WHAT TO DO WITH THE APPLICATION.? Abstract tasks (Inspection) Concrete tasks derived from scenarios (Empirical Testing).. AND WHERE TO DO IT? Scenarios 17

18 MiLE methodology: principles ABSTRACT TASK / 1 Ë An abstract task (A.T) describes a pattern for an inspection activity: a (set of) parametric action(s) that the inspector should perform on the application in order to evaluate TECHNICAL attributes Ë A.T. structure: <technical attribute, (set of) inpsection action(s) > See.doc files!!! Abstract tasks: Ë capture evaluation experience Ë Enforce standardisation and uniformity, thus reducing time and cost needed for inspection So far: A.T have been defined for level 2: navigation/content/semiotics/layout/multiple media dynamics See.doc files!!! level 3: museum web sites 18

19 MiLE methodology: principles ABSTRACT TASK / 2 During inspection, the user: Ë performs the abstract tasks corresponding to the chosen attributes Ë Subjectively measure the corresponding attributes (see section: scoring and weighting) The portions of the applications to be inspected can be selected: Ë According to some previously defined scenarios To focus where to move Ë Randomly When the application is small or scenario are missing Ë Comprehensively (the entire application) When the application is small or there is enough resourses to inspect all pages 19

20 MiLE methodology: principles SCENARIOS _1 Ë Scenario: story about use (Carroll, 2002) the description of a concrete episode of use of the application (Cato, 2001) Ë A Scenario in MiLE is composed by: User profile User goal something the user wants to achieve with the application Set of Tasks something the user wants to do with the application to achieve the goal 20

21 Example (web-based learning application) SCENARIOS_2 Scenario 1 : student Goal Plan the study Tasks Know the time required to attend a course Find the ideal period to frequent a classroom session Scenario 2: Student Know course conditions Identify course goals Understand the course structure Understand how to communicate with tutors and peers Scenario 3: Instructor Know the Achieved learning level Make a test in order to verify the level of learning achieved Verify in which topics there are learning gaps 21

22 MiLE methodology: principles USER PROFILE Ë Define the main target (s) of the application ËSome criteria for creating user profiles: Person profile Age Job Degree of general culture Provenance. Background with similar applications Ability with internet Degree of web site's familiarity (first-time visitor, frequent user...)

23 MiLE methodology: principles SCENARIOS_3 Why scenarios? 1) During inspection: Ë Scenarios act as inspection focus they help inspectors to identify the portion(s) of the application that are representative for evaluation purposes (those which look more useful to fullfill the scenario s goal) The inspector carries on the abstract tasks and measures the corresponding usability attributes while trying to achieve the scenario s goal and performing the scenario s tasks 23

24 MiLE methodology: principles SCENARIOS_3 Why scenarios? (cont.) 2) During inspection Ë Scenarios define what the ispector should try to do/achieve with the application, when putting him/herself from the point of view of the user Ë Scenarios define the tasks that inspectors should try to carry on to: evaluate requirement dependent usability attributes (related to the domain and user goals/needs) which cannot be supported by ATs E.g. Content completeness,... To get an idea of what can be problematic in the user feeling 24

25 MiLE methodology: principles SCENARIOS_4 Why scenarios? (cont.) 3) During empirical testing: The user carries on (a subset of the) scenarios tasks al least those ones which helped to indentify more problems during inspection 25

26 MiLE methodology: principles THE CONCEPT OF EVALUATION KIT (U-KIT) A MiLE U-KIT includes: ËA set of Specialized Scenarios for a specific domain (e.g., museums) for a category of applications (e.g., cell phones) ËTechnical Attributes (Heuristics) coupled with Abstract Tasks ËRequirement dependend attributes The U-KIT is defined during the preparatory phase (choosing the kit elements for an existing library if available) 26

27 Ó MiLE methodology: execution process 27

28 MiLE methodology: execution process Ë MiLE execution process involves 2 main phase: INSPECTION and USER TESTING 28

29 MiLE methodology: execution process 0. THE PHASES OF INSPECTION Ë MiLE inspection involves several activities (not all of which are strictly necessary, nor are performed in a strict sequence): Ë Modeling the application under inspection (mandatorily the first step) Ë Performing abstract tasks + Scoring technical usability attributes Ë Performing scenarions tasks + Scoring requirement driven usability attributes (possibly, also higher level technical attributes) Ë Weighting the scores according to user actual requirements Ë Reporting: the evaluation activity the results descriptions of improvements, design and requirements revisions... 29

30 MiLE methodology: execution process MODELING THE APPLICATION UNDER INSPECTION Ë Goal: Understanding what the application is about and building a mental model of the application Mental model = a cognitive structure of concepts and procedures that help users to choose the appropriate actions and understand what happens with the application (Carrol) Ë Activity: The reviewer draws a high-level model either totally informally or adopting a semi-formal model (e.g. W2000) of the application under inspection. Ë Expected output: General schema of the application features e.g., structure of the content, navigational capabilities, the interface elements, services. The output varies in terms of level of detail according to the aspects of the web site that the reviewer is committed to evaluate (navigation, content, interactions, layout readability, etc...) and the required depth of the evaluation 30

31 MiLE methodology: execution process PERFORMING ABSTRACT TASKS (scenario driven approach) Ë Goal: Evaluating technical attributes Ë Activity: For each task of a salient scenario, while trying to accomplish the task the reviewer: performs the abstract tasks for the technical attributes defined in the preliminary phase; assesses whether or not the user task can be properly accomplished assignes a score to each attribute Ë Expected output: For each user task: a two-value mark (YES: can be accomplish, NO: impossible to accomplish) and a score for each technical attribute 31

32 MiLE methodology: execution process EVALUATING REQUIREMENT DEPENDENT USABILITY ATTRIBUTES Ë Goal: Evaluating requirement dependent usability attributes Ë Activity: For each task of a scenario, while trying to accomplish the task the reviewer: assesses whether or not the user task can be properly accomplished assignes a score to each relevant requirement dependent attribute Ë Expected output: For each user task: a two-value mark (YES: can be accomplish, NO: impossible to accomplish) and a score for each requirement dependent attribute Ë SEE PART II 32

33 MiLE methodology: execution process EVALUATING REQUIREMENT DEPENDENT USABILITY ATTRIBUTES (cont.) Ë Scores: the metrics can be chosen by the evaluator Ë The metrics can be quantitive (numbers) or qualitative (a judgment, e.g., bad, average, good, very good) Ë Any quantitative metrics can be converted into a numeric one and normalized according to the chosen numeric interval Ë Our suggestion: SCORE RANGE: = the attribute is totally violated 2 = insufficient fulfillment (attribute violated in most cases) 3 = in average, sufficient fulfillment 4 = good fulfillment, with minor violations 5 = excellent fulfulment (no violations detected) 33

34 MiLE methodology: execution process REPORTING THE SCORING Basic scoring schema: for each task of each scenario Scenario number Task Number: Task description: Accomplishment (Yes/No) Dimension of analysis (e.g. navigation, layout...) Score (1..5) Source of problems (brief description of a situation where violations occurred)... Source of problems (brief description of a situation where violations occurred) Attribute A1 Attribute An Dimension of analysis (e.g. navigation, layout...) Attribute H1 Attribute Hn 34

35 MiLE methodology: execution process REPORTING THE SCORING (cont.) Ë The set of all tables for all scenarions is a hypercube (a multidimension matrix) hard to represent! Ë Ë Ë Data can be grouped-by in different ways (e.g., by attribute) Several aggregated values can be derived (e.g., average score for each attribute, global score for each task, average global score for each scenario...) for further analysis The kinds of aggregations depends on which data the inspectors looks for 35

36 MiLE methodology: execution process WEIGHTING THE RESULTS Goals: Ë defining the relevance, the criticality of the various problems, i.e., how important they are in real-world situations Ë Establishing a priority for correction actions How: Ë weighting the evaluation results according to the relevance of the task the scenario Both (combination of weights) Or/and Ë weighting the evaluation results according to the relevance of the attributes In some cases, an attribute which is important for a scenario is less important for another N.B. Ë A subjective measure (to be agreed with the stakeholders and the requirements analysis) Ë To be validated (or contradicted) by means of empirical testing 36

37 MiLE methodology: execution process WEIGHTING THE RESULTS (cont.) The weighting method depends on how data are analysed For example: Ë Identification of the attributes that are more violated in more important scenarios/tasks Assignement of a priority rate to scenarios (in the range 1-3: 1= high priority; 3= low priorirty) Weight = priority See next table 37

38 MiLE methodology: execution process It is more important to find corrections for the violations here, even if A1 is more violated in Task 2!! Task 1 Task priority Score Weighted score (weight = priority) 1 (high) A1 2 2 A2 4 4 Task 2 Task priority Score Weighted score 3 (low) A1 1 3 A