Design and User Evaluation of Augmented-Reality Interfaces Topic Usability Evaluation Methods. Payel Bandyopadhyay MSc Student University of Helsinki

Transcription

1 Design and User Evaluation of Augmented-Reality Interfaces Topic Usability Evaluation Methods Payel Bandyopadhyay MSc Student University of Helsinki

2 Outline Introduction Lab based usability testing Procedure Advantages Disadvantages Questions/Comments References

3 Introduction Usability Evaluation Methods Usability testing Lab studies Field studies enables us to quantify the extent to which a product meets the needs of its intended users [3] performance measurement to determine whether usability goals are achieved [1]

4 Figure 1: Usability testing evaluation methods [6]

5 Laboratory based usability testing (1/2) Conducted in laboratories with test Not necessary to take place in dedicated laboratory [9] Controlled environment Can be conducted in various controlled environments office [9] hall way [9] simulator [9] others Mimic real life scenario The design of a laboratory usability test summarizes the issues of concern, the tasks to be observed, the questions to ask, and the criteria for screening the people who participate. The researcher creates and follows a detailed session script so that all participants receive the same instructions and error remediation, while performing the agreed-upon tasks using the same data [4]

6 Figure : A laboratory set up for conducting usability testing [1]

7 Laboratory based usability testing (2/2) Evaluate the effectiveness and ease of use of your prototype/product [2] Gain unique insights users' needs preferences observing their behavior as they perform typical activities [4] Experimenter has control of assignments of subjects treatment variables manipulation of variables Exploratory testing [4] Qualitative data video/audio recordings of user activity interviews feedback from users Performance Testing [4] Quantitative data statistics/numbers logs/ diary

8 Procedure (1/2) Lab environment should be as real as possible [10] Ensure that your prototype works exactly the way you want Define your research goal Recruit participants match your user task goal research goal Pre-screen participants (if applicable) questionnaires interviews diary Participant(s) performs a pre-defined set of tasks Explain them the exact user task to perform Show a demo

9 Procedure (2/2) Behave exactly the same way with all users no biasing Inform users (if applicable) video recordings audio recordings data logs After experiment provide participants (if applicable) questionnaires interview diary Reward the participants

10 Figure 3: A sample of how video and audio recordings can be done to evaluate your prototype/subject [8]

11 Figure 3: A sample user task definition [7]

12 Figure 3: A sample user task definition [1]

13 Advantages Opportunity to focus specific phenomena of interest [1] High-quality data video recording of the display user interaction All users experience the same setting higher quality data [5] The difficulties in conducting and collecting data are significantly reduced as compared to field based usability tests [1] Highly replicable (of real world) [1] Facilitate good data collection [1]

14 Disadvantages Less reliable data real life users Cannot address factors and issues occur at the field [1] Recordings might distract/aware the user Please the experimenter Percentage of real life scenario simulation critical factor in terms of the validity of the usability test [1]

15 Figure 2: A comparison of set-up of usability testing in real world and lab environment. This figure shows a difference between computer setting in pre-school (left) and USE lab (right) [5]

16 Figure: Difference in final results in field based usability testing and lab based usability testing [1]

17 Figure: Difference in user behaviour in field based usability testing and lab based usability testing [1]

18 Questions/Comments

19 References (1/2) [1] Henry Been-Lirn Duh, Gerald C. B. Tan, and Vivian Hsueh-hua Chen Usability evaluation for mobile device: a comparison of laboratory and field tests. In Proceedings of the 8th conference on Human-computer interaction with mobile devices and services (MobileHCI '06). ACM, New York, NY, USA, DOI= / / [2] Brenda Battleson, Austin Booth, Jane Weintrop, Usability testing of an academic library Web site: a case study, The Journal of Academic Librarianship, Volume 27, Issue 3, May 2001, Pages , ISSN , (01)00180-X. ( [3] Rosenbaum, S., "Usability evaluations versus usability testing: when and why?," Professional Communication, IEEE Transactions on, vol.32, no.4, pp.210,216, Dec 1989, doi: / URL: [4] Rosenbaum, S.; Kantner, L., "Field Usability Testing: Method, Not Compromise," Professional Communication Conference, IPCC IEEE International, vol., no., pp.1,7, 1-3 Oct. 2007, doi: /IPCC URL: [5] Razak, F.H.A.; Hafit, H.; Sedi, N.; Zubaidi, N.A.; Haron, H., "Usability testing with children: Laboratory vs field studies," User Science and Engineering (i-user), 2010 International Conference on, vol., no., pp.104,109, Dec. 2010, doi: /IUSER URL:

20 References (2/2) [6] Alshamari, M.; Mayhew, P., "Task Design: Its Impact on Usability Testing," Internet and Web Applications and Services, ICIW '08. Third International Conference on, vol., no., pp.583,589, 8-13 June 2008, doi: /ICIW URL: [7] Morten Sieker Andreasen, Henrik Villemann Nielsen, Simon Ormholt Schrøder, and Jan Stage What happened to remote usability testing?: an empirical study of three methods. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07). ACM, New York, NY, USA, DOI= / [8] Book Section D 2004@ , Mobile Human-Computer Interaction - MobileHCI V 3160, Lecture Notes in Computer Science, Brewster, Stephen, Dunlop, Mark, / _6T, Is It Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field, Springer Berlin Heidelberg, , Kjeldskov, Jesper, Skov, MikaelB. Als, BenedikteS. HÃ egh, RuneT. pp [9] A Review of MobileHCI Research Methods. / Kjeldskov, Jesper; Graham, C., Proceedings of the 5th International Mobile HCI 2003 conference, September , Udine, Italy: Lecture Notes in Computer Science. Springer, [10]