Usability Testing: The Relation between Tasks and Issues

Transcription

1 Usability Testing: The Relation between Tasks an Issues Avinash Mairey Human-Centere Systems (HCS) Department of Computer an Information Science (IDA) MASTER THESIS-TQDT30 LIU-IDA/LITH-EX-A--12/032--SE

2 Linköping University Department of Computer an Information Science Master s Thesis Usability testing: the relation between tasks an issues by Avinash Mairey LIU-IDA/LITH-EX-A--12/032--SE Supervisor: Johan Åberg Examiner: Stefan Holmli

3 Upphovsrätt Detta okument hålls tillgängligt på Internet eller ess framtia ersättare från publiceringsatum uner förutsättning att inga extraorinära omstänigheter uppstår. Tillgång till okumentet innebär tillstån för var och en att läsa, laa ner, skriva ut enstaka kopior för enskilt bruk och att använa et oföränrat för ickekommersiell forskning och för unervisning. Överföring av upphovsrätten vi en senare tipunkt kan inte upphäva etta tillstån. All annan använning av okumentet kräver upphovsmannens megivane. För att garantera äktheten, säkerheten och tillgängligheten finns lösningar av teknisk och aministrativ art. Upphovsmannens ieella rätt innefattar rätt att bli nämn som upphovsman i en omfattning som go se kräver vi använning av okumentet på ovan beskrivna sätt samt sky mot att okumentet änras eller presenteras i såan form eller i såant sammanhang som är kränkane för upphovsmannens litterära eller konstnärliga anseene eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsia Copyright The publishers will keep this ocument online on the Internet or its possible replacement from the ate of publication barring exceptional circumstances. The online availability of the ocument implies permanent permission for anyone to rea, to ownloa, or to print out single copies for his/hers own use an to use it unchange for non-commercial research an eucational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the ocument are conitional upon the consent of the copyright owner. The publisher has taken technical an aministrative measures to assure authenticity, security an accessibility. Accoring to intellectual property law the author has the right to be mentione when his/her work is accesse as escribe above an to be protecte against infringement. For aitional information about the Linköping University Electronic Press an its proceures for publication an for assurance of ocument integrity, please refer to its www home page: Avinash Mairey. i

4 Abstract The Usability of a website can be evaluate using various methos. One of the methos is usability testing, which is wiely use an employe by the usability tester ue to its low cost an user frienliness. The results of usability testing can be affecte positively or negatively by several factors such as evaluator s role, number of users, test environment, tasks, usability problem report, usability measures, an other factors. For more than a ecae, the number of users plays a key role in usability testing. Previously one stuy [12], has shown that correlation exists between the number of user tasks an the number of issues foun. As an extension an follow-up of the recent stuies, the current work was carrie out on task esign, task number, an task coverage. Two types of task esigns were propose namely, guie tasks an unguie tasks. Consiering the task as a key factor, the remaining factors were also consiere while employing the usability test. In this stuy, two websites were chosen for performing the usability test, namely, an The usability test was carrie out with a total of 40 users. The collecte usability ata was analyze with respect to comparing means, correlation between ifferent tasks esigns, an how severe the problems foun were. The evaluation results showe that unguie task ha a greater impact on usability testing results than a guie task ha. Finally, this work conclue by recommening the usability practitioners to employ both the tasks esigns in orer to increase the usability of any given website. Key wors Usability, usability testing, guie tasks, unguie tasks, quantitative analysis, an qualitative analysis, t-test. ii

5 Acknowlegements This stuy was carrie out at Human-Centere Systems ivision at the Department of Computer an Information Science at Linköping University from September 2011 to March First an foremost, I woul like to convey my sincere gratitue to my supervisor, Dr. Johan Åberg, Linköping University, for the fruitful iscussions that were one throughout the stuy which lea me to gain in-epth knowlege in the area of usability. My sincere thanks further goes to my friens an to all my classmates in MSc Computer Systems for their encouragement throughout my stuy perio. Finally I wish to thank my parents, all my family members for giving me their moral support. iii

6 Table of Contents 1. INTRODUCTION Backgroun of Stuy Motivation Goals/Research questions Overview Disposition LITERATURE REVIEW What is Usability? Usability Evaluation Methos Usability Testing Usability Inspection Usability Inquiry What is Usability Testing? Factors affecting Usability Testing an its results Goals of Usability Testing Analyzing an Reporting Metrics for Usability Issues Frequency of Unique Issues Issues by Category Issues in Task Design for Usability Testing Conclusion from the stuies METHOD Type of stuy Evaluator s Role Characteristics of an Evaluator Participants Participants for the pilot test Participants for the main test Test Duration Targete Websites an Tasks Website Selection Task Selection Usability Issues Issues by Category Issues by Tasks iv

7 3.7 Comparing Means Correlation: Relation between variables Severity Ratings Issue Severity Rating Methoology Test Proceure/Process Pilot test proceure Main test proceure Depenent Measures RESULTS Overview Results Presentation Guie task Unguie task Comparing Means Correlation Issue Severity Rating DISCUSSION Task Completion Accuracy of the Tests in this stuy Discussion of the Statistical Results CONCLUSION FUTUREWORK REFERENCES GLOSSARY APPENDIX Ientifie Issues Usability Observation Form Introuction to Usability testing Demographic questions Usability testing for guie tasks Usability testing for unguie tasks v

8 List of Figures Figure 1: Current Issues with Usability testing Figure 2: Quantitative moel for usability Figure 3: Aopte from website Figure 4: Aopte from website Figure 5: Representation of the methoology use in this stuy Figure 6: Frequency of participants encountere for each issue category 26 Figure 7: Frequency of participants encountere for each issue category 27 Figure 8: Percentage of total number of participants encountere each type of issues category for the tasks 27 Figure 9: Percentage of total number of participants encountere each type of issues category for the tasks 28 Figure 10: Frequency of participants encountere for each issue category 29 Figure 11: Frequency of participants encountere for each issue category 29 Figure 12: Percentage of total number of participants encountere each type of issues category for the tasks 30 Figure 13: Percentage of total number of participants encountere each type of issues category for the tasks 30 Figure 14: Pie-chart representation for the ata represente in table Figure 15: Pie-chart representation for the ata represente in table Figure 16: Graphical representation of relationship between issues an user size 35 Figure 17: Graphical representation of relationship between issues an user size 35 List of Tables Table 1: List of tasks selecte to perform usability testing Table 2: Severity rating base on combination of factors (impact on users experience, frequency of users experiencing a problem) [21] Table 3: Overall Result for the guie task 25 Table 4: Overall result for the guie task 26 Table 5: Overall Result for unguie task 28 Table 6: Overall Result for unguie task 28 Table 7: Represents the number of unique issues encountere by each user 31 Table 8: t-test results for the ata represente in table Table 9: Represents the number of unique issues encountere by each user 32 Table 10: t-test results for the ata represente in table vi

9 Table 11: Represents the number of issues comes uner guie, unguie an share groups 33 Table 12: Represents the number of issues comes uner guie, unguie an share groups 33 Table 13: The number of issues share among Low, Meium, an High level severity between guie, unguie, an share groups 36 Table 14: The number of issues share among Low, Meium, an High level severity between guie, unguie, an share groups 36 Table 15: An overview of chi-square test results for comparing the task esigns in an websites Table 16: Issues that were ientifie uring usability evaluation of the both websites vii

10 1. INTRODUCTION 1.1 Backgroun of Stuy Having poor usability of a website makes the users to opt for an alternate website. The usability of a website can be increase by ientifying more number of issues an altering the website with respect to the recognize issues. Evaluating the usability of a website can be one in any one of the methos state in section 2.2. Usability testing is wiely use an employe by the usability practitioners, ue to its low cost an user frienliness. Results of a usability test will be influence by several factors, which are number of users, evaluator s role, test environment, usability problem report, usability measure, an other factors [3]. For more than a perio, number of users plays a key role in evaluating usability of a website. The research work one in [14], state that focus must be from number of users to the tasks an this stuy reveale correlation exists between the number of user tasks an number of issues encountere. As an extension an follow up of this stuy, the current work concentrates on task esign, task number, an task coverage. 1.2 Motivation Usability testing is influence positively an negatively by several factors [3]. Stuy one in [14], reveale correlation exists between the number of tasks an the number of issues encountere by the users. [1, 4] state the esign of task an its impact on usability testing. Selection of tasks is very critical an will be affecte by several factors in usability testing [24]. Selecting the wrong set of tasks will affect the usability of a website. Why task? Unertaking tasks plays an important role in usability testing, an is wiely use technique for ensuring usability of a website. They have a tremenous impact on issues which are ientifie. The tasks basically etermine what areas of the prouct are exercise an the ways in which they are exercise. Particularly with a complex prouct, this can have a major impact on what issues are uncovere [21]. Task esign will have a serious impact on the usability test [1]. All the above consierations were taken into account an work is carrie out on task esign, task number, an task coverage. 1.3 Goals/Research questions The main objective of this thesis is to contribute with knowlege about task esign an its effect on the usability test results. The esign of the tasks was one in to guie an unguie tasks. The thesis answers the following primary research question: What effects o guie an unguie tasks esigns, have on the usability test results an relation between them? The effect will be looke in terms of the number of unique issues, issues by category, severity egree, an iscovery rate per user. 1.4 Overview The esign of the tasks was classifie in to two categories i.e., guie an unguie tasks. Both these esigns were inepenent to each other. In common, they will have same set of tasks an same number of participants but iffer with respect to two ifferent groups of 1

11 participants. In guie tasks, users will perform the tasks with guielines. Following example gives overview of guie task Task: fin the epartments at Linkoping University Guielines: Browse the homepage ( Click Department on left han sie of the page Click on any one of the epartment links Come back to homepage In Unguie tasks, users were left to behave as they woul o in reality. For the above task, unguie task is as follows: User will be in Google home page Browse for Linkoping University home page Know more about the website an fin the epartments at Linkoping University 1.5 Disposition Chapter 1: Introuction This part gives the overview of the thesis goal an the area of stuy it eals with. It also gives a glance view of the problems regaring thesis goal an the methoology use to fulfill the research goal. Chapter 2: Literature review This part explains in etail regaring the usability, usability evaluation methos an factors affecting usability testing. In aition, iscussions carrie out in the relate articles, an books. Chapter 3: Metho This part eals with goals of the stuy an concepts use uring this stuy so as to get the necessary results. Chapter 4: Planning the test This part explains how the test proceure carrie out uring this stuy for collecting the usability ata. Chapter 5: Results This part presents the results rawn by analyzing the ata collecte uring the usability testing. Chapter 6: Discussions Analyzing an iscussing the results represente in chapter 5 an working towars the goal of the thesis work. 2

12 Chapter 7: Conclusion This part explains what can be conclue from the analyze ata an relating the results to the previous stuies carrie out in same area. 3

13 2. LITERATURE REVIEW 2.1 What is Usability? Usability generally refers to the metho for making proucts an systems easier to use, an matching them more closely to the user nees an requirements. The efinitions which have been use in the previous stuies were erive from a number of views an following three views were relate for measuring the usability [5]: The prouct-oriente view, measuring of usability was one in terms of ergonomic attributes of the prouct. The user-oriente view, measuring of usability was one in terms of the mental effort an attitue of the user. The user performance view, that usability can be measure by examining how the user interacts with the prouct, with particular emphasis on either: - Ease-of-use: how easy will be the prouct to use, or - Acceptability: whether the prouct will be use in the real worl. ISO stanar for software qualities (ISO 1991b) was prouct an user-oriente view: A set of attributes of software which bear on the effort neee for use an on the iniviual assessments of such use... The Usability Professional Association (UPA) efinition focuses more on the prouct evelopment process: Usability is an approach to prouct evelopment that incorporates irect user feeback throughout the evelopment cycle in orer to reuce costs an create proucts an tools that meet user nees. ISO ientifies three aspect of usability, efining it as: The extent to which a prouct can be use by specifie users to achieve specifie goals with effectiveness, efficiency, an satisfaction in a specifie context of use. The metrics such as user s effectiveness, efficiency, an satisfaction were consiere while evaluating the usability of a website. These metrics were state as follows: Effectiveness: A user s ability to successfully use a website to fin information an accomplish tasks. Efficiency: The amount of effort require by the user to complete the task. Satisfaction: The egree to which the user was happy with his/her experience while performing the task. In this work, the usability of a website was measure with respect to the user s performance view i.e., how effectively the specifie users interacts with a website to accomplish tasks. 4

14 2.2 Usability Evaluation Methos Following sections gives a brief introuction of the methos use in evaluating the usability of a website, [13]: Usability Testing Usability testing is a technique for ensuring that the intene users of a system can carry out the propose tasks efficiently, effectively an satisfactorily. Usability testing is carrie out at pre-release of a prouct so that any significant issues ientifie can be aresse. [7] Provies the following methos which are use at ifferent stages of the prouct esign in usability testing: Coaching Metho This metho is applicable at esign, coe, test, an eployment stages. While employing this metho, system expert sits along with the users to act as a coach. A user is allowe to ask the system relate questions while performing the selecte tasks an coach answers to those questions. The observer evaluates interaction between the user an the coach. The purpose of this technique is to iscover the information nees of users in orer to provie better training an ocumentation, as well as possibly reesign the interface to avoi the nee for the questions, [7]. It covers the issues with respect to effectiveness an satisfaction. Co-iscovery Learning This metho is applicable at esign, coe, test, an eployment stage. Tasks that were selecte will be performe by two users helping each other working towars a common goal. During the test observer encourages the users to explain their views about the prouct esign while performing the tasks. It covers the issues with respect to effectiveness an satisfaction. Performance Measurement This technique is applicable at esign, coe, test, an eployment stage. Through this technique quantitative ata is obtaine about the test participant s performance when they perform the tasks uring usability test. The collecte quantitative ata is most useful in oing comparative testing, or testing against preefine benchmarks. Combining this metho with a questionnaire or retrospective testing lets the usability tester to know both quantitative an qualitative ata. It covers the issues with respect to effectiveness an efficiency. Question-asking Protocol This metho is applicable at esign, coe, test, an eployment stage. While performing the tasks observer prompts participant s with questions. It covers the issues with respect to effectiveness an satisfaction. Remote Testing This metho is applicable at esign, coe, test, an eployment stage. In remote testing, the usability tester cannot observe the testing process irectly. Remote testing will be carrie out in more than one way. In one metho, usability testing can be carrie out at same-time but in 5

15 ifferent places, where the tester can observe the test user s system screen through a network an listen to tester voice through speaker telephone. Another is ifferent-time ifferent-place testing, where the users test session is starte by logging in to a test session through software an the ata will be store in a atabase. It covers the issues with respect to effectiveness, efficiency, an satisfaction. Retrospective Testing This metho is applicable at esign, coe, test, an eployment stage. While implementing this technique, user s interaction with the computer nees to be recore an replaye. After completing the test session recore vieotape will be observe by the usability expert an the participants. It covers the issues with respect to effectiveness, efficiency, an satisfaction. Shaowing Metho This metho is applicable at esign, coe, test, an eployment stage. It covers the issues with respect to effectiveness an efficiency. During the usability test, tester has an expert user sits next to him/her an explain the test user s behavior to the tester. This metho is appropriate for the test user to think alou or talk to the tester while working on the tasks. Teaching Metho This metho is applicable at esign, coe, test, an eployment stage. During the test, a user will be traine by interacting with the system to gain knowlege/experience with the prouct. After user finishing the training, tester asks the traine user to teach the new novice participant an observes the interaction between traine user an new novice user. It covers the issues with respect to effectiveness an satisfaction. Thinking Alou Protocol This metho is applicable at esign, coe, test, an eployment stage. During this technique user is aske to vocalize his/her thoughts, feelings, an opinions while interacting with the prouct. It covers the issues with respect to effectiveness an satisfaction Usability Inspection In Usability inspection approach, usability specialists, users, evelopers, professionals examine the usability relate aspects of a user interface. It is inexpensive an quick. The following one of the methos is in use for conucting usability inspection, [18]: Cognitive Walkthroughs It evaluates the esign of a prouct on how well it supports the system users in learning tasks, [23]. This type of testing is conucte by an expert in cognitive psychology an walks through the esign for ientifying potential problems using psychological principles. It involves a group of evaluators inspecting a user interface by going through a set of tasks an evaluates its unerstanability an ease of learning. This technique is best use in the esign stage of evelopment. 6

16 Feature Inspection This technique focuses on the feature set of a prouct. The inspectors were usually given use cases with the en result to be obtaine from the use of the prouct. Each feature is analyze for its availability, unerstanability, an other aspects of the usability. It covers the issues with respect to effectiveness. Heuristic Evaluation This metho involves having a small group of usability experts for evaluating a user interface using a set of guielines an noting the severity of each usability problem an where it exists, [18]. This is a metho for quick, cheap, an easy evaluation of a user interface esign. Goal of the heuristic evaluation is to fin the usability problems in the esign so that they can be attene to as part of an iterative esign process. Heuristic evaluation involves a small set of evaluators for examining the interface of a esign. Each evaluator evaluates the interface of a esign an at en of their finings communication among the evaluator s will be mae to list the ientifie problems relate to the esign. Pluralistic Walkthrough At the esign stage, when paper prototype is available, a group of users, evelopers, an human factors engineers meet together to step through a set of tasks, iscussing an evaluating the usability of a system, [23]. Group walkthroughs have the avantage of proviing a iverse range of skills an perspectives to bear on usability problems. As with any inspection, more people looking for problems, higher will be the probability of fining problems. Also, the interaction between the team uring the walkthrough helps to resolve usability issues faster. Perspective-base Inspection The goal is to unerstan the feasibility, effectiveness, an scope of perspective-base usability inspection an also by inspecting from one particular perspective at a time. In this technique inspection will be one in three perspectives. Each inspection focuses on one of the three efine perspectives: novices use, expert use, an error hanling Usability Inquiry By using this metho, usability evaluators obtain information about the user s review i.e., what were the likes, islikes, nees, an unerstaning of the system by communicating an observing the user s interaction with the system in real work, or letting them answer to the questions verbally or in written form. Following are the commonly use inquiry methos, [19]: Fiel Observation This metho is applicable at test, an eployment. Human factors engineers go to representative user s workplace an observe there work to unerstan how the users were using the system to accomplish their tasks an the kin of mental moel the users have about the system. It covers the issues relate to effectiveness an satisfaction. 7

17 Focus Groups This is a ata collecting technique where about 6 to 9 users were brought together to iscuss the usability issues relating to the system/website. A human factors engineer plays the role of a moerator, who nees to prepare the list of issues to be iscusse beforehan an seek to gather the neee information from the iscussion. This can capture spontaneous user reactions an ieas that evolve in the ynamic group process. It covers the issues relate to effectiveness an satisfaction an this metho is employe uring test an evelopment. Interviews In this technique, human factors engineers formulate questions about the prouct base on the kin of usability issues. Then they interview representative users to ask them these questions in orer to gather information esire. It is goo at obtaining etaile information as well as information that can only be obtaine from the interactive process between the interviewer an the user. In an evaluation, the interviewer reas the questions to the user, replies verbally, an the interviewer recors those responses. The methos of interviewing inclue unguie interviewing an guie interviewing. Unguie interviewing methos were use uring the earlier stages of usability evaluation. The objective of the investigator at this stage is to gather as much information as possible concerning the user's experience. The interviewer oes not have a well-efine agena an is not concerne with any specific aspects of the system. The primary objective is to obtain information on the proceures aopte by the users an on their expectations of the system. Guie interviewing has a specific, preetermine agena which inclues specific questions to help guie an irect the interview. Guie interviewing will be more of an interrogation than unguie interviewing, which is closer to a conversation. Proactive Fiel Stuy Before esigning a system, human factors engineers go to representative user's workplace an talk to them, observe there work, an ask them the questions to unerstan the user characteristics, the work flow, the system features, working environment etc. This technique shoul be use uring the requirement or early esign stage of the software evelopment. This shoul be the first step of usability work for a project. Questionnaires Questionnaires have long been use to evaluate the user interfaces. Questionnaires have also been use in electronic form. For a hanful of questionnaires specifically esigne to assess aspects of the usability, the valiity an/or reliability have been establishe, incluing some of them iscusse in [21]. 8

18 2.3 What is Usability Testing? Usability testing is a technique use to improve the user interface of a prouct. It reveals problems with in the esign of website an its functionality. The purpose of usability testing is to: Focus on specific aspects of the esign. Evaluate a esign base on preetermine usability criteria. Uncover new an previously covere problems. The most appropriate conition for carrying out usability testing is before the pre-release of a prouct to aress the ientifie issues. Usability testing is carrie out at any time or at successive intervals uring the evelopment cycle of a prouct to make the prouct user frienly with its interface esign. It is preferre to carry out usability testing early in the esign process for fining major flaws, or whenever changes were being mae. Usability testing is performe among the group of users by accomplishing the given tasks on the suggeste website. During the usability test, a moerator gathers the issues encountere by the users an feeback/suggestions to improve the usability of a website. Attributes of usability testing are: The goal is to improve a prouct/website Participants were real users an perform real tasks Participants were formally observe The ata is analyze Recommenations for improvements were mae After performing the usability test, collecte ata can be analyze in two ways: Quantitative an Qualitative analysis. Quantitative analysis represents the usability ata numerically i.e., the number of issues face, the number of tasks complete successfully, the number of clicks neee to perform the task, an time spent per task to complete. This quantitative ata is analyze statistically to compare the iterative esigns with respect to the number of problems foun. In the current stuy, quantitative ata is collecte i.e., fining the issues by observing the effectiveness of the user interaction with a website for each task esign an comparing the task esigns with respect to the ientifie issues for fining the impact of task esign on usability testing. Qualitative analysis helps to know about the users an what they were trying to accomplish. This is about the comments mae by the usability test participants to some of the questions like how frequently they were using the prouct, was the prouct efficient to use, an the satisfaction rate. 9

19 2.3.1 Factors affecting Usability Testing an its results The following are the factors that influence usability testing an its results both negatively an positively [3]. Figure 2 escribes the list of factors affecting usability testing. Usabiltiy Measures Evaluator s Role Number of users Usability Testing Test Environment Tasks Usability Problem Other factor s Figure 1: Current Issues with Usability testing Evaluator s Role The role of an evaluator was a critical issue in usability. In past research stuies, several iscussions an arguments prove that problem etection iffers noticeably. Evaluators were also criticize for a lack of methoical analysis just after the test has been finishe. Three strategies were also propose to solve the evaluator s effect [11]. Becoming a great test moerator takes four things [15] Unerstaning the basics of usability testing Interacting well with test participants Ability to establish an maintain rapport with participants Lots of practice A set of rules an the things to o an not to o while interacting with test participants, [15] makes the evaluator to get the correct usability ata Users In usability stuy, users have a major impact on usability testing results. It was very important for recruiting the correct set of users for a usability stuy. Recruiting the users will be one base on some characteristics such as age groups, gener groups, experience in usability testing etc. The next step after selecting a relevant group of users was to set the sample size of users require. Knowing the sample size of users was iscusse in a number of research papers. This was a very crucial step since size of the user s effect in fining the usability test results. It was suggeste that 5 users were enough to iscover 85% of the usability problems [16]. It was foun that only 35% of usability problems were ientifie with 5 users [14]. Base up on the type of analysis 5 an 20 users were suggeste for qualitative or quantitative ata respectively [17]. 10

20 Tasks Usability testing tasks refer to what users o in orer to achieve a goal, but even though they are user-riven, they are an important issue an can heavily influence usability evaluation. The selection of tasks was a critical activity in usability testing as some of the tasks will have a clear en state, some are open-ene, an some are self-generate by the users. What became clear in the stuy was that the selection of tasks ha a tremenous impact on the issues that were ientifie [21]. Selection of tasks helps to know which areas of the website nee to be exercise an reesigne. In particular, while testing a large prouct it will have a major impact on ientifying uncovere issues. It was suggeste of moving the focus from number users to tasks base on their research carrie out in [15]. Further research on task esign prove that it shows a severe impact on usability testing results [2] Usability Problem Report At en of the usability test, tester shoul generate a usability problem report which effectively helps the esigners an evelopers to make their ecision with regars to the reesign stage. Reporting the list of problems may not effectively help them to solve the problems. But the etaile escription along with how to eal an treat with certain problems will impact on usability test results Test Environment Usability testing will be performe in a Laboratory. The physical environment of the laboratory impacts on the testing results. It may interact irectly with the user or inirect interaction like vieo recoring, vieo call. Conucting Usability testing in natural places shows a major impact on usability testing Usability Measures an Prioritizing Problems Prior to conucting a usability test, testers shoul be aware of what to test an measure. There are three major ISO stanars for measuring usability, which are efficiency, effectiveness an user satisfaction. This moel was criticize because of it s too abstracteness. Several moels of measuring usability have been propose an criticize. Usability measures iniviually such as effectiveness, efficiency, an satisfaction [11]: Effectiveness measures can be worke out through measuring binary task completion, accuracy, completeness, quality of outcome, an others factors Efficiency can be measure through measuring input rates, task completion time, mental effort, learning, use frequency Satisfaction measure inclues stanar questionnaires, preferences, satisfaction with the interface, an others. A metho was propose in [20] for stanarizing usability metrics into a single usability metric an propose a quantitative moel for usability as shown in figure 3. 11

21 Usabiltiy Efficiency Effectiveness Satisfaction Time # of errors Completion AVG. satisfaction Other Usability Issues Figure 2: Quantitative moel for usability Artifact will be use for interaction between usability evaluation an the esign stage. It may be a prototype or a prouct that was evaluate which ha a huge impact on fining test results. The type of interaction may be a paper prototype, functional or semi functional prototype, or prouct system. The prototypes may misinterpret real system functionality. There are four challenges that can improve usability evaluation an the esign stage [12]. The first challenge was the type of prototype or a running system. The Secon challenge was the insufficient effort that was usually allocate to choosing tasks. The Thir challenge was the usability problems report. The Fourth challenge was usability problems reports neither recommens nor suggests priority of problems severity. 2.4 Goals of Usability Testing To ientify the usability problems. To make recommenations for moifying the prouct. To fin out how satisfying the prouct was. Establish benchmarks for future comparisons. 2.5 Analyzing an Reporting Metrics for Usability Issues After performing the usability test, all the ientifie issues nee to be analyze to erive some metrics relate to usability issues. Exactly how to o this will largely epen on the type of usability questions the usability tester has. Three general questions can be answere by looking at metrics relate to usability issues: [21] How is the overall usability of the prouct? This is helpful to get an overall sense of how the prouct i. Is the usability improving with the each esign iteration? Focus on this question when we nee to know how the usability is changing with the each new esign iteration. Where shoul I focus my efforts to improve the esign? The answer to this question is useful when you nee to ecie where to focus the resources. 12

22 All of the analyses that were explaine can be one with or without severity ratings. Severity ratings simply a a way to filter the issues. Sometimes it s helpful to focus on the highseverity issues. Other times it might make more sense to treat all the usability issues equally Frequency of Unique Issues The most common way to measure usability issues is to simply count the unique issues. In any esign process, analyzing the frequency of unique issues is most useful in an iterative esign process to know how the usability is changing with the each iteration or between two esigns. [21] In this stuy, guie tasks an unguie tasks were compare with respect to the number of unique issues to know which task esign reports more number of issues Issues by Category Sometimes it s helpful to know where to focus esign improvements from a tactical perspective. Perhaps we may feel that only certain areas of the prouct are causing the most usability issues, such as navigation, content, terminology, an so forth [21]. In this analysis, examining each ientifie issue an the categorizing it into a type of issue. Then look at the frequencies of issues that fall into each category. Categorizing the issues can be one in many ifferent ways an nee to make sense to you an your auience, an use a limite number of categories, typically three to eight [21]. In previous usability research on website interface esign, six usability problems categories were ientifie, namely Layout, Terminology, Feeback, Comprehension, Data Entry, Navigation [8, 22]. 2.6 Issues in Task Design for Usability Testing Stuy 1 A stuy carrie out by [14], initiates the focus of increasing the usability test performance by shifting the focus from participants to tasks coverage. It iscusses an compares the results by performing usability evaluation one among 9 teams. It also criticizes about the magic number 5. It conclues that task coverage will impact more than the participants on usability testing an suggeste further research was neee on task coverage. Results from this stuy gave basement an motivation to current work more on task coverage an some knowlege about the magic number 5. Stuy 2 The stuies carrie out by [1, 2, 3, 4], mainly explain the factors influencing usability testing an impact of task esign on the usability testing. All these papers explain the current issues of usability testing an they propose task esigns. The results from these stuies inicate positive results regaring the impacts of task esign on usability testing. Design of task an results of this stuy was consiere to carry out this thesis work. 13

23 Stuy 3: The stuies carrie out by [15, 16], mainly explains the necessary reasons regaring why to test with 5 users an also the number of users neee for quantitative stuies. User size of 20 was recommene to carry out the testing for quantitative stuy an 5 users for qualitative stuy Conclusion from the stuies The following points can be rawn from stuies: Stuy 1: Number of tasks ha shown major impact on usability test performance than number of users an lai a basement for further research on task esign, task coverage, an number of tasks. Stuy 2: Propose two types of tasks namely, guie an unguie. It showe how much a task esign can influence usability testing results. Suggeste task esign was employe in this thesis work an current stuy was carrie out regaring the number an selection criteria of tasks along with the task coverage. Stuy 3: It was recommene of using 5 users for usability testing as it will be enough to etect more number of issues in a website. Still, this enorsement has been appraise by other researchers. Depening on the type of the usability ate i.e., qualitative an quantitative this stuy suggeste to test with 5 an 20 users respectively. The type of ata collecte uring this thesis work was quantitative ata an user size of 20 was set as a benchmark for current usability testing. 14

24 3. METHOD 3.1 Type of stuy A well planne usability stuy saves time an money. The stuy was a quantitative usability testing to evaluate the effectiveness of a website i.e., ientifying the number of issues encountere by the users in a website while performing a set of tasks. In this stuy, task esign was categorize into guie an unguie task. In the guie task, the users ha to follow a set of guielines to accomplish the task in a website. Guielines will help the users to follow the correct path for accomplishing the task. In the unguie task, a user will behave like the real target user of a website an accomplish the tasks with their previous knowlege or learning by themselves. In the current stuy, an observation form was use by the moerator to note own the ientifie issues while user interacts with the website through a set of tasks. Observation form contains overview of the proceure an instructions nee to be followe by the usability tester. It also contains the set of tasks with which tester interacts with a website. Figure 5 in Section 3.10 gives an overview of the methoology employe in the current usability testing for collecting the usability ata. The inepenent variables in this stuy were testing with guie an unguie tasks. The epenent variables were the number of participants an the set of tasks selecte to perform the usability test. 3.2 Evaluator s Role In the usability test, experience of an evaluator plays a major role while collecting the usability ata. In usability testing of the current stuy, the author of this thesis work acte as an evaluator. Goal of the evaluator was to observe how effectively user interacts with a website an makes the user s to express their view an issues with in a website. Usability testing was carrie out only after the evaluator fulfills the following characteristics [15]: Having a goo knowlege about the website. Goals an the type of ate neee to be collecte uring usability testing. Prepare the test script. Conuct a pilot test. Refine the test script an materials if necessary Characteristics of an Evaluator During the usability test, the evaluator (author of this work) acte as a neutral observer with following characteristics [15]: Let s the participants speak Asks unbiase questions Neither encourages nor iscourages by emotional comments Avoi efening the prouct esign. 15

25 3.3 Participants In the current usability test, two tests were performe, namely, the pilot test an the main test. The pilot test was followe by the main test. The participants for the usability test were recruite if they meet the following pre-requisites: Sufficient browsing knowlege Previous knowlege of surfing the similar website use for testing. Willing to participate Unerstaning an juging of tasks Responsiveness: must give appropriate verbal feeback. The following steps were followe for recruiting the each participant for the current usability test: Each participant was approache an explaine about the part of the work an the role he/she has to perform in the usability test. Recruiting a participant for the usability test was one if he/she meets the minimum requirements as state in above paragraph. Total time require for finishing the usability test was also note to the participant. Without isturbing the participant s work usability testing was carrie out either: - Immeiately or, - By taking an appointment in their free time. It was very important to note that the four participants, who participate uring the pilot test, were ifferent from the forty participants who participate uring the main test. The current usability test was performe on the selecte two websites, namely, an belonging to a university an the usability test participants were recruite among the stuents at Linköping University an all were between the ages 20 to 30. The selecte two websites falls uner the same category group i.e., university website an in real worl most of the users for these websites were stuents. So in the current usability test, an average Linköping University stuent can perform well by relating with his/her previous knowlege an acts like real target users. During the test, the participants were mae comfortable to perform the usability test in presence of the evaluator Participants for the pilot test The main aim of pilot testing was to fin out if the propose methoology works efficiently for gathering the accurate usability ata an to ientify the problems encountere with the questionnaire. A total of four participants were recruite among the stuents at Linkoping University an they were categorize in to two participants for each task esign group, namely, guie an unguie. After finishing the pilot test, problems an improvements ientifie in the usability test were rectifie before proceeing to the main test. 16

26 3.3.2 Participants for the main test The total number of test participants was forty. They were recruite as the usability test participants, if they meet the minimum qualifications state in the above section. They must behave like the real target users to perform the neee tasks on a website. Unerstaning of tasks an browsing experience were the key factors for a test participant in the usability test. There were two groups of participants, namely, guie group an unguie group each consisting of twenty participants. One group of participants comes uner guie group where they perform the tasks with the given guie lines an another group of participants comes uner unguie group where they will perform the tasks similar to a real user i.e., without any assistance. Each participant was teste iniviually on a computer with same internet spee an browser. In this stuy, the number of participants playe a major role in getting reliable results, upon which significant ecision was mae. 3.4 Test Duration Each participant spent at most 40 minutes for all the tasks he/she was suppose to carry out. The evaluator carrie out the testing with 4 to 5 participants per ay. 3.5 Targete Websites an Tasks Website Selection After careful stuy the following two websites were selecte for the usability test: Both the web-sites belong to the same category an reasons for selecting the websites were as follows: Both are ynamic websites. Meets the research requirement as it has various functions, processes, an features. Users (stuents from Linköping University) can be easily recruite for this usability evaluation. Recruite users can act like real target users as they were familiar with similar kin of websites. They are unfamiliar to the test users. 17

27 Figure 3: Aopte from website Figure 4: Aopte from website 18

28 3.5.2 Task Selection Structuring of tasks plays an important role in analyzing the usability ata. Since some of the certain tasks reveal quantitative usability ata such as number of issues ientifie to finish a certain task. Other tasks can expose the qualitative ata such as user s allege ease-of-use for a given web page. In the current stuy, quantitative ata was collecte i.e., the effectiveness of a website was measure. Base on these points, the selection of tasks for the usability test was one an the chosen tasks must be: Reasonable an oable Specific an focuse The following criteria were consiere while selecting tasks: Frequency: How often user will complete the task on a website Importance: The most critical tasks Seriousness of errors: If the tasks were one incorrectly, it was a serious problem Uncertainty: Having questions about the tasks. In general, total time neee to complete a usability test on a user shoul be consiere. If the total time was too long, many of the test users will not show interest to participate in a usability test. Consiering the total time as major factor, the current usability testing was esigne to complete within minutes. A total of six tasks were selecte in this stuy to carry out the usability testing an on an average each task will take 3-6 minutes to complete. Task1 Fin the results for recently complete women's basketball events at Saluki Athletics. Task2 Fin about the "University honors program" offers at this university. Task3 Fin the events at for the visitors in this month. Task4 Fin the tuition fee for grauate stuies. Task5 How many libraries oes this university provie to stuents for accessing the books? Task6 Fin the grauate programs offere in Art & Design epartment. Task1 Fin about the news room for the meia at this university. Macquarie University offers to book a slot for IELTS exam, fin out the amount to be Task2 pai for this exam. Task3 Fin the mail-i of the person to enquire about the postgrauate scholarships. Task4 Fin about the stuent events toay. Task5 Fin about the isciplines that are offere in research of earth sciences. Task6 Fin the upcoming carrier service events for stuents to evelop their skills. Table 1: List of tasks selecte to perform usability testing 19

29 Table 1, represents the selection of tasks for both the websites an these set of tasks were selecte by following the selection criteria state above. 3.6 Usability Issues Usability issues can be efine as something which prevents the user in completion of a task or make the user to eviate the task goal in a wrong way. The ientifie issues can be analyze in many ifferent ways [21], such as frequency of unique issues with respect to the esign, frequency of issues per participant, Issues by category, Issues by task. In this thesis work Issues by category an Issues by task were use for analyzing the usability ata Issues by Category In this stuy, the issue ientifie uring usability testing will fall into one of the following issue categories: Navigation - Navigation was Guie /Unguie s perhaps the most important element of any web esign. It helps the user to reach the goal. It shows the user where they are where they can go. Terminology Unable to unerstan the terminology. Layout Inability to etect something that they nee to fin. Aesthetic problems, unnecessary information. Feeback User oes not receive relevant feeback or it was inconsistent with what the user expects. Data entry Problems with entering information. Cognitive/Content The user s interpretations, assessments an expectations. Comprehension Inability to unerstan the instructions given to them on site. By categorizing the issues, it will be very easy to know which type of issues effects the usability of a website an the areas neee to be concentrate in the next iterative esign of a website Issues by Tasks Analyzing the issues at task level will lea to know about which task nee to be prioritize. More the number of issues, higher the priority of task will be. In aition to this, the unique an common issues between tasks can be known. This analysis will help to compare the Improvements of the esign iterations. 3.7 Comparing Means Comparing means helps to know which group has more error rate or satisfactions rate compare to other group. In this work, comparison was one across two ifferent groups of participants, namely, guie group an unguie group with respect to the number of unique issues ientifie. Inepenent samples of the ata were compare between these two groups. 20