Using Pre-Test to Validate the Questionnaire for Website Usability (QWU)

Transcription

1 Using Pre-Test to Validate the Questionnaire for Website Usability (QWU) Nur Sukinah Aziz 1,2 1 Faculty of Computer, Media and Technology Management TATI University College Kemaman, Terengganu, Malaysia nursukinah@tatiuc.edu.my Adzhar Kamaludin 2 Faculty of Computer Systems & Software Engineering Universiti Malaysia Pahang Kuantan, Pahang, Malaysia adzhar@ump.edu.my Abstract Usability is one of important criteria to measure website quality. One of the approaches to measure website usability is survey using questionnaire. The aim of this study is to validate the questionnaire instrument to measure the highest education institution or university website in context of usability. Pre-test are used to validate the instrument for this study. The questionnaire instrument has 9 constructs and 51 item of questionnaire. 30 respondents were involved in this study. After analyzed the result, 7 items need to delete from the instrument. Apart from the more technical and general aspects the feedback from the respondents was useful in identifying problematic questions and it was suggested that some questions need to deleted as they were seen to have same meaning with other questions and too difficult to answer for several reason such as because the sentences of the question is difficult to understand and the level of understanding between the question and respondent. Keywords website; usability; pre-test; university website; questionnaire I. INTRODUCTION Website evaluation is to determine the quality of the website. There are many factors or characteristic to evaluate the quality of the website. Usability is one of the element that used to evaluate website [1][2]. There are many quality models that usability element such as McCall s Quality, Boehm s Quality, ISO 9126 Quality, FURPS Quality, Dromey s Quality and QUIM Quality [2]. This research used the pre - test approach as one of the method to evaluate the questionnaire that will used to evaluate the website usability. The pre-test conducted to pay more attention to questions asked to respondents. Pretesting is critical for identifying questionnaire problems. Problems with question content include confusion with the overall meaning of the question, as well as misinterpretation of individual terms or concepts. Problems with how to skip or navigate from question to question may result in missing data and frustration for both interviewers and respondents. Questionnaire formatting concerns are particularly relevant to selfadministered questionnaires, and if unaddressed, may lead to loss of vital information. II. RELATED WORK Based on ISO in Human Computer Interaction (HCI) field, usability is defined as the the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use [3][1]. In ISO , usability is defined as the capability of the software product to be understood, learned, operated, attractive to the user, and compliant to standards or guidelines, when used under specific conditions [1]. It has 6 characteristics such as functionality, reliability, usability, efficiency, maintainability and portability [4] [5] [6]. Some researchers has combine ISO 9126 and ISO 9241 attributes and develop new model that has effectiveness, efficiency, satisfaction, learnability and security as attributes [7]. Nielsen (1993) has mentioned that usability is a general concept that is difficult to measure but it is related to several parameters that can be measured. Parameters that can be measured is divided into two categories namely ; The objective behavior that measure the ability of users of the system and the reference measurement that measure how users liked the system. Refer to the definition on ISO , the criteria of usability are effectiveness, efficiency and satisfaction. It focuses on human interaction perspective for software product standard. This definition has 3 components that can divide such as specified users, achieve specified goals and specified context to use. This definition is more clearly what usability is mean and many researchers use this definition [8]. Eason is proposed by Kenneth Eason (1984) and published his model in an early issue of Behaviour and Information Technology. Eason has 3 aspect, task, user and system. Eason cannot measure usability without considering users and their target task. Eason model sees usability as the result of several interacting variables or multi - variate. [8][9]. Shackel was developed by Brian Shackel. In this model, it has 4 attributes that is effectiveness, learnability, flexibility and attitude. Shackel does not weight the dimension, recognizing that the importance of each of these may different from project to project. Shackel model /15/$ IEEE 107

2 emphasizes measurement of a number of human factors, relating to human performance and attitude [8] [9]. Nielson was developed by Jakob Nielson. The main model is system acceptability and usability is part of usefulness. Other attribute that contribute to the main model are utility, usefulness, practical acceptability and social acceptability. ISO 9241 is an international standard for guidance on usability based on process oriented. Nielson and Shneiderman are among the committee members in the development of ISO guidelines. For ISO has 3 attributes that are effectiveness, efficiency and satisfaction. ISO has objective measures of usability [10]. The disadvantage of this model is that it is to abstract [11] [12]. ISO 9126 is an international standard for the evolution of software quality model from the product perspective. ISO 9126 is an extension of previous work done by McCall (1977), Boehm (1978), FURPS and others in defining a set of software quality characteristics [13]. The internal and external metrics are functionalities, reliability, usability, effectiveness, maintainability and portability [7]. Under usability it has 5 attributes such as understandability, learnability, operability, attractiveness and usability compliance [7][14][9]. The advantage of ISO 9126 model is it provide a framework for making trade-offs between software product capabilities and the attribute are applicable to any kind of software including computer programs and provide consistent terminology for software product quality. The disadvantage of ISO 9126 was unclear architecture at the detail level of the measures, overlapping concepts, lack of a quality requirement standard, lack of guidance in assessing the results of measurement and ambiguous choice of measures [10]. QUIM or Quality in Use Integrated Measurement developed by Ahmed Seffah et al in QUIM is a consolidated model for usability measurement and metrics. It combines various standard and model such as ISO 9241 and ISO 9126 and unified into a single consolidated, hierarchical model. QUIM model consists of 10 factors and subdivided into 26 criteria or measurable criteria, and finally into specific metrics consists 127 specific metrics. The 10 factors consists Efficiency, Effectiveness, Satisfaction, Learnability, Productivity, Safety, Trustfulness Accessibility, Usefulness and Universality. The model is used to measure the actual use of working software and identifying the problem. In QUIM model associates factors with criteria and metrics in a clear and consistent way. It also usable generally and can adapt in specific context of use. The limitation of this model, it is not optimal yet and needs to be validated [11] [12] [15]. Based on previous model and studies from other researcher, 9 attributes are selected to build website usability model namely Effectiveness, Efficiency, Satisfaction, Learnability, Accessibility, Navigation, Content and Interface/design for Independent Variable (IV) factors and Intention to Use as a Dependent Variable (DV). Table 1 show the similarity between the standard usability model. 5 attributes that are common used in usability are effectiveness, efficiency, learnability, satisfaction and accessibility. Based on previous research, navigation, content and interface design are usually used to evaluate website usability. TABLE I. SIMILARITY BETWEEN THE USABILITY MODEL Eason Shackel Nielson ISO ISO 9126 QUIM Effectiveness Efficiency Learnability Satisfaction Accessibility To validate the model, a survey is conducted using the questionnaire instrument. Before distributing the questionnaire instrument, a few phases are involved to validate the instrument. One of the phases is pre-test. Pre-test is an important part of fieldwork preparation as it is a tool for checking if the questionnaire instrument measures adequately the reality of the respondents. Pretesting is a very important step in survey research. It is an absolutely necessary step to ensure all kinds of errors that are associated with survey research are reduced. It helps to improve the quality of data significantly. Pretesting is done on a small sample of respondents from the target population. After the pilot test, respondents are asked a series of questions regarding the survey as well as the process of data collection during the debriefing session. Such debriefing sessions can help detect any problem with the questionnaire design leading to ambiguity of words, misinterpretation of questions, inability to answer a question, sensitive questions, and many other problems associated with the questionnaire as well as the process of administering the survey. Table 2 show the initial list of item constructs and reference of questionnaire instrument. The questionnaire is adapted from previous studies that used in usability such as Computer System Usability Questionnaire (), WAMMI and also include a few question that refer to the item constructs that used, [25], [26]. TABLE II. THE INITIAL LIST OF QUESTIONNAIRE INSTRUMENT FOR PRE-TEST Construct Code Item Reference Efficiency EY1 When I use the Web site there is very [16] little waiting time between my actions and the Web site s response. EY2 It is easy to find the information that I need EY3 I am able to efficiently complete my work using this website EY4 I can effectively complete my work using this website EY5 I believe I became productive quickly using this website 108

3 Effectiveness EV1 On this website, it is simple to [17] accomplish the task I want to accomplish. EV2 I find it easy to get this Web site to [18] do what I want it to do EV3 I am able to complete my work quickly using this website EV4 It was simple to use this website EV5 The information is effective in helping me complete the tasks and scenarios Satisfaction S1 I feel comfortable using this website S2 This website has all the functions and capabilities I expect it to have S3 I am satisfied with how easy it is to use this web site S4 I am satisfied with this web site Learnability L1 Learning to operate the Web site is [16] easy for me. L2 I find the Web site easy to use. [17] L3 All the material is written in a way WAMMI that is easy to understand. L4 Using this website for the first time is WAMMI easy. L5 The contents provided by the website are easily understood L6 The website is designed for easy understanding L7 The information provided by the website is easy to understand L8 It was easy to learn to use this website Accessibility AC1 The website offers customization. [17] AC2 It was easy to move from one page to [20] another AC3 The text on the website is easy to [16] read AC4 It takes time to open the web page or - download the web page AC5 It has a accessibility function on the - web site ( can resize text, change the background colour etc ) AC6 The website s wording is clear and easy to understand AC7 The website uses colors and structures that are easy on the eyes AC8 The pages download quickly on this [17] website. Navigation N1 I can easily navigate this site. [21] N2 This site provides good navigation [21] facilities to information content. N3 I like the way hyperlinks are [22] embedded in this site s design N4 I feel in control when I'm using this WAMMI web site N5 I get what I expect when I click on WAMMI things on this website. N6 The navigation and labels on this Web site were clear. N7 Links are consistent and easy to [24] identify N8 The website provides multiple search features (e.g: search engine,menu bar,go-back-and-forward button, etc) to obtain the target information N9 It was easy to move from one page to [20] another Content C1 I trust the Web site to keep my [16] personal information safe. C2 I can trust this website. [21] Interface / design Intention to reuse C3 I trust the information presented on [21] this website. C4 The information provided at this site [21] is sufficient. C5 The website adequately meets my [21] information needs. C6 I find the information on this site to [21] be well organized C7 I feel this Web site clearly stated its purpose for using the site C8 The website provides up-to-date information C9 The information (such as online help, online messages, and other documentation) provided with this website is clear C10 The organization of information on the website pages is clear ID1 The website repeats the same structure, components and overall look across pages. ID2 Web pages in the website are consistently designed ID3 This web site is presented in an WAMMI; attractive way. (i.e. colors, images, [21] layout etc) ID4 The pages on this website are very WAMMI; attractive. [21] ID5 The layout of pages made tasks easier. ID6 The interface of this web site is pleasant ID7 I like using the interface of this web site ITU1 I intend to use this website again [20] ITU2 I would be willing to visit this website again ITU3 I feel this website reflects most current trend(s) and provides nice design for the site visit ITU4 I will reuse this website again [20] III. METHODOLOGY The aim was, firstly, to assess whether the questionnaire was relevant to and easily understood by the respondents, in terms of the concepts and the way they phrased in the questions; and secondly to assess the technical functioning of the questionnaire. For the instrument for this study, questionnaire from Computer System Usability Questionnaire () and WAMMI were adapted and also include a few question that refer to the item constructs that used, [25], [26]. Higher education institution websites are chosen to evaluate the proposed usability model. Evaluating website usability is of significant importance to the success of higher education websites [27]. Higher education institution websites often contain important information about academic resources, campus events, and administrative policies. As higher education websites take on significant and increasingly important roles, it is imperative that these sites be userfriendly. Before distributed to the real respondents in actual study, the instrument has gone a few evaluations. There are 3 stage of validation for the instrument before distribute for actual 109

4 study such as experts review, pre-test and pilot study. This study focusing on the analyse in pre-test. The instrument is known as Questionnaire for Website Usability (QWU). The instrument has 9 constructs namely Effectiveness, Efficiency, Satisfaction, Learnability, Accessibility, Navigation, Content and Interface/design for Independent Variable (IV) factors and Intention to Use as a Dependent Variable (DV). In this pre-test, students from the similar background with the real respondents are selected as a focus group because the real respondents in the actual study are students from higher education institution. Pre-test is conducted aimed at gaining an understanding of the answering process and finding out whether the questionnaire instrument were understood and how the real respondents will react with the instrument [28]. One of the main advantages of focus groups is the opportunity to observe a great deal of interaction on a topic in a limited period of time. Focus groups help identify variations in language, terminology, or interpretation of questions and response options [29]. About 30 students in university college were involved in this pre-test. The students were contacted and agree to participate in this study. The students divide to 6 group consist 5 students per group. The pre-test took about a week to accomplish because due the students time constraints. IV. RESULTS AND DISCUSSION After analyzing the result based on the pre-test from the respondents, 7 items from 5 construct have a few issues. The constructs that involved namely Interface/ design, Navigability, Content, Accessibility and Learnability. Apart from the more technical and general aspects the feedback from the respondents was useful in identifying problematic questions and it was suggested that some questions need to deleted as they were seen to have same meaning with other questions and too difficult to answer for several reason such as because the sentences of the question is difficult to understand and the level of understanding between the question and respondent. The items that need to delete are ID4 - Interface/ design, N2 - Navigability, C2 and C6 - Content, AC4 Accessibility, L1 - Learnability. L7 is remove to content attribute based on discussion and analyze the result from the pre-test. Most of the respondents told that the item ID4 in construct Interface/ design have same meaning with item ID3. Beside that item C6 in construct Content have the same meaning with C3, item AC4 in construct Accessibility has a same meaning with AC2 and item L7 in construct Learnability has the same meaning with L5. So after analyze the items, the researchers delete the item. Item N2 in construct Navigability have the issue the sentence is difficult to understand by respondents. Most the respondents did not understand the meaning of the sentences. A few respondents did not understand the sentences provides good navigation facilities. The item C2 in construct Content facing the issue the wording such as adequately is difficult to understand by respondents. Meanwhile the item L1 in construct Learnability, the respondents did not understand with the sentences Learning to operate the Web site is easy for me and misinterpretation of questions. Based on the respondents feedback, the item also has a same meaning with item L8. Most the respondents prefer the item L8. Based on the feedback from the respondents, a few items need to delete. The respondents also suggested that the instrument need to have translation in Bahasa Melayu for easy understanding because some student did not proficient in English. After the analysis, 45 items in the questionnaire instrument are remains. Table 2 shows the questionnaire that need to delete after analyse feedback from respondents. TABLE III. QUESTIONNAIRE DELETED AFTER ANALYSE THE RESULT FROM PRE-TEST Construct Code Before Pre-Test After Pre-Test Interface / The pages on this *Deleted have the design ID4 website are very same meaning with Navigability Content Accessibility Learnability N2 C2 C6 AC4 L1 attractive. This site provides good navigation facilities to information content. The website adequately meets my information needs. The organization of information on the website pages is clear The website s wording is clear and easy to understand Learning to operate the Web site is easy for me. The information provided by the L7 website is easy to understand * the item that need to delete ID3 sentences is difficult to understand by respondents wording such as adequately is difficult to understood *Deleted have the same meaning with C3 *Deleted has a same meaning with AC2 respondents did not understand with the sentences Remove to content attribute V. FUTURE RECOMMENDATION Some changes were made to the structure of the questionnaire. The note concerning the need to have translation in Bahasa Melayu was taken into account and these were added in the questionnaire instrument before distribute in pilot study. Another important finding which concerned revising the structure of the questionnaire (order of questions, filters etc.), deleting questions and adding new ones and rephrasing question. The order of questions was altered in some places but this mainly related to other modifications made after pre-test. After this, the pilot study will conducted to real respondents in higher education institution such as local university and college university in Malaysia focusing on university website before distribute to actual study to analyse the reliability and validity the questionnaire. 110

5 ACKNOWLEDGMENT The authors would like to extend sincere appreciation to Faculty of Computer, Media and Technology Management (FKMPT), TATI University College and Faculty of Computer System and Software Engineering (FSKKP), Universiti Malaysia Pahang for the support and encouragement in making this research a success. REFERENCES [1] A. Fernandez, E. Insfran, and S. Abrahão, Usability evaluation methods for the web: A systematic mapping study, Inf. Softw. Technol., vol. 53, no. 8, pp , Aug [2] S. K. Dubey, A. Rana, and Mridu, Analytical Comparison of Usability Measurement Methods, vol. 39, no. 15, pp , [3] M. De Marsico and S. Levialdi, Evaluating web sites: exploiting user s expectations, Int. J. Hum. Comput. Stud., vol. 60, no. 3, pp , Mar [4] Z. Balfagih and M. Mahmud, In Search of a for Evaluating the Quality of E-Commerce Web sites, Quality, no. December 2007, pp. 2 7, [5] S. K. Dubey and A. Rana, Analytical Roadmap to Usability Definitions and Decompositions, vol. 2, no. 9, pp , [6] N. Bevan, Quality and usability: A new framework, no. figure 1, pp. 1 8, [7] A. Abran, A. Khelifi, W. Suryn, and A. Seffah, Usability Meanings and Interpretations in ISO Standards, Softw. Qual. J., vol. 11, no. 4, pp , [8] L. Leventhal and J. Barnes, Usability Engineering Process, Products, and Examples. Pearson Prentice Hall, [9] A. Madan and S. K. Dubey, Usability Evaluation Methods: A Literature Review, vol. 4, no. 02, pp , [10] P.-Y. Yen, Health information technology usability evaluation: methods, models, and measures, Columbia University, [11] A. Seffah, M. Donyaee, R. B. Kline, and H. K. Padda, Usability measurement and metrics: A consolidated model, Softw. Qual. J., vol. 14, no. 2, pp , Jun [12] A. Hussain and E. Ferneley, Usability Metric for Mobile Application: A Goal Question Metric ( GQM ) Approach, pp , [13] R. E. Al-qutaish, Quality s in Software Engineering Literature: An Analytical and Comparative Study, vol. 6, no. 3, pp , [14] S. U. Services, H. Court, and L. Wcv, International Standards for HCI, no. May, pp. 1 15, [15] A. Khalili, User Interfaces for Semantic Content Authoring: A Systematic Literature Review, no. May, [16] E. Loiacono, R. Watson, and D. Goodhue, WebQual: An Instrument for Consumer Evaluation of Web Sites, Int. J. Electron. Commer., vol. 11, no. 3, pp , Apr [17] J. Wang and S. Senecal, Measuring Perceived Website Usability, J. Internet Commer., vol. 6, no. 4, pp , [18] J. W. Palmer, Web site usability, design, and performance metrics, Y. Lee and K. a. Kozar, Understanding of website usability: Specifying and measuring constructs and their relationships, Decis. Support Syst., vol. 52, no. 2, pp , Jan [20] S. Poelmans, P. Wessa, K. Milis, E. Bloemen, and C. Doom, Usability and Acceptance of E-Learning in Statistics Education, Based on the Compendium Platform, Inf. Syst. J., pp. 1 10, [21] D. Cyr, Website design, trust and culture: An eight country investigation, Electron. Commer. Res. Appl., vol. 12, no. 6, pp , Nov [22] X. Zhang, K. Keeling, and R. Pavur, Information quality of commericial web site home pages: an explorative analysis, in ICIS 00: Proceedings of the twenty first international conference on Information systems, 2000, pp C. E. Downing and C. Liu, Assessing Web Site Usability in Retail Electronic Commerce, 2011 IEEE 35th Annu. Comput. Softw. Appl. Conf., pp , Jul [24] P. J. Meyers, 25 Point Website Usability Checklist, [Online]. Available: [Accessed: 20-Apr-2015]. [25] Y.-L. Theng and J. Sin, Evaluating Usability and Efficaciousness of an E-learning System: A Quantitative, -Driven Approach, 2012 IEEE 12th Int. Conf. Adv. Learn. Technol., pp , Jul [26] K. Milis, P. Wessa, S. Poelmans, C. Doom, and E. Bloemen, The impact of gender on the acceptance of virtual learning environments, Proc. Int. Conf. Educ. Res. Innov. Int. Assoc. Technol. Educ. Dev., [27] L. L. Broberg, A Grounded Theory Approach To Examining Design And Usability Guidelines For Four-Year Tribal College Web Sites, Capella University, [28] J. K. Oostrom and M. P. Born, Using cognitive pretesting to explore causes for ethnic differences on role-plays, Int. J. Intercult. Relations, vol. 41, pp , [29] T. DeMaio, J. Rothgeb, and J. Hess, Improving survey quality through pretesting, Proc. Surv. Res.,