Concepts of Usability Usability Testing What is usability? How to measure it? Fang Chen ISO/IS 9241 Usability concept The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use. ISO 9241: effectiveness (% of goal achieved) + efficiency (time to complete a task, or the error rate, or the amount of effort) + satisfaction (subjective rating scale) = Usability What is context? System What is context? System Stakeholder analysis, role and benefit 1
What is context? System Stakeholder analysis, role and benefit User Experience, knowledge and skill Personal attributes What is context? System Stakeholder analysis, role and benefit User Experience, knowledge and skill Personal attributes Task What is context? Relationship System Stakeholder analysis, role and benefit User Experience, knowledge and skill Personal attributes Task Environment Technical, physical, organizational External quality Work environment user Computer system software Internal quality Quality in use Quality in use The capability of the software product to enable specialed users to achieve specialed goals with effectiveness, productivity, safety and satisfaction in specialed contexts of use. Functionality Accuracy Suitability Interoperability security Quality in use Effectiveness, productivity, safety, satisfaction Reliability Maturity Fault tolerance Recoverability availability Usability Understandability Learnability Operability attractiveness Efficiency Time behaviour Resource utilisation ISO/IEC FDIS 9126-1 1 (2000) maintainability Analysability Changability, stability, testability Portability Adaptability, installability Co-existence, replaceability 2
Satisfaction Satisfation: the level of comfort that the users feel when using a product and how acceptable the product is to users as a means of achieving their goals Satisfaction Satisfation: the level of comfort that the users feel when using a product and how acceptable the product is to users as a means of achieving their goals Subjective aspect Difficult to measure Often be strongly correlated with effectiveness and efficiency Satisfaction levels Usability testing Pleasure People s emotion Value, hope, taste, fear, etc Usability Efficiency, effectiveness, satisfaction Usability testing determines whether the users can find and use the features in the amount of the time and effort they willing to expend searching. Primary goal improve the usability Real users Do the real task Observation Analysis Functionality Fulfill the appropriate functionality The context and the environment in which it will be used Usability testing Usability is not: Quality assurance Zero defects Utility of design feature Intrinsic in products A well-trained staff in a room with nothing but paper can outdesign a poorly trained staff equiped with a state-of-the-art facility 3
Lab. Test vs. field tests Lab test Well designed Not choice aim is to test whether a product meets certain predefined usability criteria Field test acceptability of a product Usability test methods Focus on Lab. tests Usability testing lab. Usability Laboratory Usability Engineering 2007 2340 2007 2007 2340 4
Usability Engineering Methods overview Inspection method (to be carried out by cognitive expert) User testing 2341 Selecting methods Not to rely on any one method for determining the usability of product, but use a variety of methods. Learn about user Self definition (job, tasks, tools, and mental models) Learn about user Self definition (job, tasks, tools, and mental models) individual (personal, physical and cultural characteristics, as well as motivation) Learn about user Self definition (job, tasks, tools, and mental models) individual (personal, physical and cultural characteristics, as well as motivation) stage of use (How they use products over time and the choices they make about the levels of expertise they want or need to achieve) 5
Learn about user Characteristics of user populations Novices Advanced beginners Competent performers Expert performers User and task analysis What users goals are What processes they use to achieve their goals What characteristics shape the way they perform tasks and achieve goals Affect from previous experience What is the most important to user or what is most helpful to them in performing tasks What impact the environment has on their ability to perform tasks. User s goals and tasks User s tasks are not necessary user s goals User s goals and tasks User s tasks are not necessary user s goals If the task interferes with the goal, users become frustrated. User s goals and tasks Site visit User s tasks are not necessary user s goals If the task interferes with the goal, users become frustrated. the goal stack False goals (easily achieved by programmers) Corporate goals (company s benefit) Practical goals (the bridge between the company s goals and the user s goals) Personal goals (true for everyone) Follow a user for a day Question user while they work Talking after task Think aloud Critical incident technique Scenario Cued recall 6
Constraints of site visit Time constraints Budget constraints Political and ethical constraints After a site visit Report: User analysis Task analysis Environmental analysis Methodology Recommendation After a site visit Personas Daily-use scenarios (often used) Necessary-use scenarios (infrequently used) After a site visit User learning to understand how user learn the process and set different requirement to the design. Long/short term memory Different learning theories Iterative testing: Paper prototype Purpose of usability testing Good Cheap, easy, can do it often Try new ideas and concepts Team activities Enhanced communication among team members Common baseline Bad Complex products Collect response time interaction quality Team understanding Longer time for testing Different stages of product development, the test is different. 7
Planning for usability testing Team Product Goals Measurements Users Tasks Results analysis Test plan Limitations Time Money Testing conditions Who wants to know the results What is best learned from usability testing? Planning Testing condition Initial team meeting Planning meeting Results meeting Recruiting participants walkthrough Pilot test Completion of test plan Approval of the test plan Where? Field? Lab? Remote? (this is depending on the product development stages and the location of the users) Set-up for testing Mis-concept Usability test can not answer many questions, such as color preference, font size, use of particular graphics, array of products available, since the answers to such questions cannot be obtain from 5 to 8 subjects. Preparing for usability testing Screening questionnaires: The background information of users towards the use of the product. These questions shall be sent to the users and have the answer back before the test. 8
Preparing for usability testing Screening questionnaires: The background information of users towards the use of the product. These questions shall be sent to the users and have the answer back before the test. Pre-test questionnaires The questions answered before the test started. Preparing for usability testing Creating scenarios Scenarios are description of typical tasks you want the participants to perform. Avoid creating steps or instructions, focus on process Avoid using product language, use user s language Objective measurement Try to obtain if you can Post-task and post-test questionnaires Avoid bias responses Neutral questions Can be qualitative and quantitative responses Roles of the team members Important factors Based on their expertise Consistent Facilitator: give user direct instruction Camera/video operator Logger: capture the measurements Avoid bias Maintain consistency 9
Walkthrough It is a chance to test the equipment, the scenarios, the questionnaires, and the checklists for the team members. Team member s practice Intend to find the flaws in the plan Pilot test Test of the test Find out practical problems A few days earlier than real test Conducting the U- test Handling questions Team should avoid contacting with users Briefing: Any misconception can be clarified. Fill the forms. Using videotape (?) Unbias People try to please the others If, user ask a question, don t answer it directly did I do that right? what do you think should happened? or tell me what makes you wonder whether you did it right? Try to led the user shear his experience with you. Intervene Only when you see the user get lost Careful! Questions like: what are you trying to do now? or what do you think the system is doing now? Help Not to give the direction of how to solve the problem. Provide problem solving strategies. Collect data Category the data If two users have a problems, the problem has to be taking care If only one single user has that problem, depend how many users are tested! Find the explanation of the special case. Conflict data? Believe what you see! 10
Analysis Determining the causes of problems. Make sure that one solution doesn t create some other problems. Scope and severity of problems Usability report Who is going to read the report? Standard form for industry Usability report (1) Standard form for industry Usability report (2) Title Page Executive Summary Introduction Full Product Description Test Objectives Method Participants Context of Product Use in the Test Tasks Test Facility Participant s Computing Environment Test Administrator Tools Experimental Design Procedure Participant General Instructions Participant Task Instructions Standard form for industry Usability report (3) Standard form for industry Usability report (4) Usability Metrics Effectiveness Efficiency Satisfaction Results Data Analysis Data Scoring Data Reduction Statistical Analysis Presentation of the Results Performance Results Satisfaction Results Appendices References 11
Conclusion Concept of usability Quality in use Laboratory test Preparation Task study User study Carry out and measure the usbaility Documentation 12