User Experience Metric (UXM) and Index of Integration (IoI): Measuring Impact of HCI Activities

Transcription

1 User Experience Metric (UXM) and Index of Integration (IoI): Measuring Impact of HCI Activities Anirudha Joshi, Industrial Design Centre, IIT Bombay Design 1 Sanjay Tripathi, Tech Mahindra Ltd. Outsourced Software Development

2 UXM and IoI 1. Integrating HCI with SE TUP for Waterfall TUP for XP, RUP User experience metric Index of integration Metrics implementation in pilot projects 2. Current work 2

3 UXM and IoI 1. Integrating HCI with SE TUP for Waterfall TUP for XP, RUP User experience metric Index of integration Metrics implementation in pilot projects 2. Current work 3

4 Communication Project initiation Requirements gathering Planning Estimating Scheduling Tracking Modelling Analysis Design Construction Code Test Deployment Delivery Support Feedback 4

5 Communication Project initiation Requirements gathering Communication Project initiation User studies Ideation Product definition Evaluation and refinement Requirements specification Planning Estimating Scheduling Tracking Modelling Analysis Design Construction Code Test Deployment Delivery Support Feedback Planning Estimating Scheduling Tracking 5 Modelling Requirements analysis Software design Detailed UI prototyping Usability evaluation Construction Code Test Usability evaluation Deployment Delivery Support Feedback

6 Communication Project initiation Requirements gathering Communication Project initiation User studies Ideation Product definition Evaluation and refinement Requirements specification Planning Estimating Scheduling Tracking Modelling Analysis Design Construction Code Test Deployment Delivery Support Feedback Planning Estimating Scheduling Tracking 6 Modelling Requirements analysis Software design Detailed UI prototyping Usability evaluation Construction Code Test Usability evaluation Deployment Delivery Support Feedback

7 Activities Project initiation, stakeholder interviews, technology review Communication Outcomes Initial project brief, business goals, technology capabilities and constraints Involvement * SE HCI Communication Project initiation User studies Ideation Product definition Evaluation and refinement Requirements specification M U Plann Estimat Schedu Tracki User studies, domain analysis, contextual inquiry, focus groups, user modelling User problems, goals, opportunities and constraints Ideation, brain storming, QFD, TRIZ Design ideas, possibilities and alternatives Product definition, interaction design, information architecture Product definition options, key usage scenarios Evaluation and refinement, heuristic evaluation, iteration and refinement Selected and refined product definition with usage scenarios Requirements gathering specification 7 Redefined product brief, project scope, domain analysis, product definition, functional requirements, non-functional requirements, task flows, use cases, quality assurance

8 Activities Planning Outcomes Involvement Communication Project initiation User studies Ideation Product definition Evaluation and refinement Requirements specification Estimation, FPA, COCOMO Project costs, risk analysis SE HCI M U Plann Estimat Schedu Tracki Scheduling, (TBA) Project plan (TBA) Tracking, (TBA) (TBA) 8

9 Ideation Product definition Evaluation and refinement Requirements specification Activities Requirement analysis Modelling Outcomes Analysis models, domain classes, entity relationship models (?), data and control flow models (?), state diagrams, CRC models Involvement SE HCI M Planning Estimating Scheduling Tracking U Modelli Requirements Software de Detailed UI pro Usability eval Software design, software architecture, detailed software design Software architecture, design classes, class interfaces, collaboration diagrams, activity diagrams (?), component diagrams Detailed UI prototyping Refined UI screens coupled with use cases and task flows Usability evaluation, formative user tests, refinement of the UI prototype Usability problems, modified UI screens, UI standards 9

10 Estimating Scheduling Tracking Activities Construction Outcomes Involvement SE HCI Modelling Requirements analysis Software design Detailed UI prototyping Usability evaluation M U Constru Cod Tes Usability ev Code, integration Code releases Test Test plan, test cases, bug reports Usability evaluation, summative user tests Usability problems, modified UI screens (quick fixes) 10

11 Detailed UI prototyping Usability evaluation TUP for Waterfall Proposal 2 Construction Code Test Usability evaluation Deploy Deliv Supp Feedb 11

12 12 Communication Project initiation User studies Ideation Product definition Evaluation and refinement Requirements specification TUP for Waterfall Proposal 2 Planning Estimating Scheduling Tracking Modelling Requirements analysis Software design Detailed UI prototyping Usability evaluation What matters? How should we respond? Technical feasibility Business feasibility How should the product be designed? How are we doing? Redesign Construction Code Test Usability evaluation How are we doing? Acceptance Deployment Delivery Support Feedback

13 13

14 TUP for XP HCI needs to be one iteration ahead 14

15 15 TUP for XP expanded

16 16 RUP s Two Dimensional Process Model

17 17 TUP for RUP

18 UXM and IoI 1. Integrating HCI with SE TUP for Waterfall TUP for XP, RUP User experience metric Index of integration Metrics implementation in pilot projects 2. Current work 18

19 A Software Metric 19 a software metric is a measure of some property of a piece of software Wikipedia Lines of code Function point analysis Cyclomatic complexity Number of bugs Number of bugs found per week the process of assigning numbers or symbols to attributes of entities in the real world to describe them according to clearly defined rules Fenton and Pfleeger measures = collection of data points, metric relates the measures in some way Pressman

20 Typical Usability Metrics Seconds taken to withdraw money from the ATM Number of keystrokes to enter the word ह द Number of errors made to complete one transaction % users who abandon the shopping cart on checkout % users who say the product was easy to use % users who say they like the product % users who say they will buy the product % users who buy the product Can t compare across projects 20

21 Related Work Relative comparison of users performance between like products Lewis Single usability scale by including a reference task in a UT McGee Questionnaire based on eight HCI considerations to build a single score Lin et al Single, standardized and summated usability metric based on task time, errors, completion and satisfaction Sauro et al 21

22 In Outsource Software Companies Which project should the best, most experienced UX professionals work on? 100s of software projects 10s in the UX group Across domains, contexts, platforms How integrated are HCI activities with SE processes? Major communication gaps between HCI and SE How to guarantee a level of process compliance? 22

23 Goals To have consistent measures across a wide variety of projects Domain, context of use, platform, process models followed etc. To demonstrate the correlation between integrating HCI into SE activities and the delivered user experience To keep the additional costs of metric calculation low Minimal time Minimal extra skills Usability technique independent 23

24 User Experience Metric User experience goals Learnability Ease of use Speed of use Error-free use Retention Subjective satisfaction 24

25 User Experience Metric User experience goals Learnability Parameters Conceptual model clarity Language understandability Consistency with earlier version Consistency with other products Minimal training time Visibility of choices, data 25

26 User Experience Metric User experience goals Learnability Parameters (weightage) Conceptual model clarity (5) Language understandability (4) Consistency with earlier version (3) Consistency with other products (3) Minimal training time (3) Visibility of choices, data (2) 26

27 User Experience Metric User experience goals Learnability Parameters (weightage) Consistency with earlier version (3) Score, rating guidelines All frequent and critical tasks from earlier version are unchanged in the new version Strongly disagree (0) Weakly disagree (25) Partly agree, partly disagree (50) Weakly agree (75) Strongly agree (100) 27

28 Computing UXM: Step 1 When? Early in the project Just after user studies, when user needs, context, domain and platform are well-understood Who? HCI team member in consultation with stakeholders Does what? Identifies user experience goals Identifies parameters for each goal Specifies weightage and rating guidelines for each parameter 28

29 Computing UXM: Step 2 When? Late in the project During the last usability evaluation, when the UI is about to be frozen upon Who? One or more independent usability evaluator(s) Does what? Score each parameters of each goal A weighted average of scores of all parameters returns the UXM value 29

30 30 UXM: Demo

31 IoI: Index of integration of HCI and SE activities Process compliance metric Evaluates the integration of HCI activities with SE activities Organizational outlook 31

32 Distinctions Phases Communication Planning Modelling Construction Deployment 32

33 Distinctions Phases Communication HCI activities (from TUP) Contextual user studies and user modelling Ideation Conceptual model development Usability evaluation and refinement of product definition 33

34 Distinctions Phases Communication HCI activities (weightage) Contextual user studies and user modelling (4) Ideation (2) Conceptual model development (3) Usability evaluation and refinement of product definition (1) 34

35 Distinctions Phases Communication HCI activities (weightage) Contextual user studies and user modelling (4) Ideation (2) Conceptual model development (3) Usability evaluation and refinement of product definition (1) Score, activity rating guidelines Both organizational data gathering and user studies were done before requirements were finalized 2. User studies were done in the context of the users by the method of contextual inquiry 3. User studies were done with at least 10 users in each profile 4. User studies were done by people with experience in user studies

36 Computing IoI: Step 1 When? Organization adopts a SE process and wants to integrate HCI activities with it Who? HCI and SE process leads Does what? For given SE phases, activities and deliverables Specifies weightage and rating guidelines for each activity and deliverable Publishes the prescribed process across the organization along with the HCI activities, weightages and evaluation guidelines 36

37 Computing IoI: Step 2 When? While closing a UXM project Who? HCI team member(s) and project manager(s) Does what? Scores of each activity and deliverable of each phase A weighted average of scores of all parameters returns the IoI value 37

38 38 IoI: Demo

39 UXM and IoI 1. Integrating HCI with SE TUP for Waterfall TUP for XP, RUP User experience metric Index of integration Metrics implementation in pilot projects 2. Current work 39

40 Lessons Learnt 40 Metrics were useful both HCI designers and project stakeholders liked it HCI designer: Had we done this earlier, I would have know where to focus Project manager: I never thought we could think so much [about user experience]. Wanted to calculate metrics mid-way in the project Much customization to UXM, no changes to IoI Goal setting took much time, needed guidance There was variation among scores For parameters where evaluation guidelines not clear Guidelines helped, but more detailed ones required Narrowing steps helped reduce variation

41 Feedback about UXM and IoI Choice of goals, parameters, weightages Why measure against goals? Subjectivity Can we have pre-set goals for user segments? Product types? Bulging goal weightage Scale 0-100, 0-10, 0-2, positive, negative, partial negative Scoring Professional assessment or performance measure? Validity 41

42 UXM and IoI 1. Integrating HCI with SE TUP for Waterfall TUP for XP, RUP User experience metric Index of integration Metrics implementation in pilot projects 2. Current work 42

43 UXM Validation in the Classroom 43 = 18) Without help With help Total Average SD Average SD Hindi Text Input on Mobiles (performance test) Nokia New System 9.6 (53%) 8.0 (44%) 17.6 (98%) (44%) 2.4 (13%) 10.4 (58%) UXM by CS team (test group, scale 0-100) Sony Ericson 3.4 (19%) 4.2 (23%) 7.6 (42%) Ratings by IxD team (control group, scale 0-10) CD Writing Software (heuristic evaluation) Roxio Nero Vista cricinfo Findings by IxD team (average number of words successfully typed by each user, n=5, total words Cricket Websites (think aloud test) cricketnext cricbuzz

44 44

45 45

46 46

47 47

48 48

49 Thank You 1. Integrating HCI with SE TUP for Waterfall TUP for XP, RUP User experience metric Index of integration Metrics implementation in pilot projects 2. Current work Anirudha Joshi Sanjay Tripathi 49