Measuring the Usability of Web-Based Services Exemplified by University Student Information System

Transcription

1 To appear in Proc. 15th Annual Business Congress, IMDA2006, June 18-21, 2006, Saraevo, Bosnia and Herzegovina. Measuring the of Web-Based Services Exemplified by University Student Information System Asil Oztekin, Fatih University, Turkey Alexander Nikov, Fatih University, Turkey Selim Zaim, Fatih University, Turkey Ali Riza Guner, Fatih University, Turkey For measuring the usability of web-based services a SERVQUAL model for measuring service quality extended with usability dimensions. By this tool a quantitative usability index is determined. A case study with university student information system including 179 students is carried out. By a structural equation model the weighting coefficients of the usability index are determined. By stepwise regression analysis the checklist dimensions and items with highest and lowest contribution to web-based service usability are determined. The web-based services gain increasing importance in the last years. For example Google became the main competitor of Microsoft. The development of good quality websites requires sophisticated methods for design and evaluation like (Frokjaer, Hertzum & Hornbaek, 2000), (Hornbaek, 2006), (Li, Tan & Xıe, 2002), (Nikov, Vassileva, Anguelova, Tzvetanova, & Stoeva, 2003), (Sauro, & Kindlund, 2005). For measuring the service quality, the SERVQUAL model (Parasuraman, Zeithmal & Berry, 1988) and its modification for web-based services (Li, Tan & Xie, 2002) are most widely used. These models are not considering usability dimensions. presents the capability to be used by humans easily and effectively ; quality in use ; the effectiveness, efficiency, and satisfaction with which specified users can achieve goals in particular environments (Hornbaek, 2006); how easy it is to find, understand and use the information displayed on a website (Keevil, 1998). For overcoming this drawback of SERVQUAL a SERVQUAL enhanced with usability dimensions. The performance of this tool is investigated by a study with the web-based student information system of Fatih University. Description of the Checklist Tool for Measuring of Web-Based Service SERVQUAL (Li, Tan & Xie, 2002) is measuring webbased service quality in the scale [1,5] by 28 checklist questions in 6 dimensions. For extending SERVQUAL to measure the usability of web-based services these 6 dimensions and the usability dimensions according to usability heuristics (Nielsen, 1994) and standard ISO (ISO, 1998) are studied (cf. Table 1). Only Navigation and User control and freedom in usability heuristics and Controllability in ISO are outside of SERVQUAL dimensions (cf. Figure 1). Figure 1. Comparison of Web-Based Service Quality and Construction of Checklist 29 questions were selected for the checklist tool of modified SERVQUAL. They include also 3 aggregated questions for measuring effectiveness, efficiency and satisfaction. In this study the effectiveness is measured by the indicator completion percentage of the tasks; the

2 efficiency by task completion time and the satisfaction by a 5-point semantic distance scale. In this way, objective and subjective parameters are measured. By pretests with limited number of students using the student information systems of Fatih University some questions were changed and deleted. The final checklist tool is shown on Figure 3. Index Discussions of how to measure the quality of computer systems have gone on for several decades, first under the heading of ergonomics and ease-of-use, and later under the heading usability. However, recently discussions recur on which measures of usability are suitable and on how to understand the relation between different measures of usability (Hornbaek, 2006). To increase the meaningfulness and strategic influence of usability data, there is a need to represent the entire construct of usability as a single dependent variable (usability index) without sacrificing precision (Sauro, & Kindlund, 2005). The usability index is a measure, expressed as a per cent, of how closely the features of a web site match generally accepted usability guidelines (Keevil, 1998). It is widely accepted that usability evaluation depends on efficiency, effectiveness, and satisfaction. In (Frokjaer, Hertzum & Hornbaek, 2000), effectiveness is defined as the accuracy and completeness with which users achieve certain goals. Indicators of effectiveness include quality of solution and error rates. Efficiency is the relation a) between the accuracy and completeness with which users achieve users certain goals and b) the resources expended in achieving them. Indicators of efficiency include task completion time and learning time. Satisfaction is the users comfort and positive attitudes towards the use of the system. Users satisfaction can be measured by attitude rating scales. The correlations among these usability dimensions depend in a complex way on the application domain, the user s experience, and the use context. During last three years in CHI proceedings 11 out of 19 experimental studies involving complex tasks account for only one or two dimensions of usability. When these studies make claims concerning overall usability, they rely on risky assumptions about correlations between usability dimensions. Unless domain specific studies suggest otherwise, effectiveness, efficiency, and satisfaction should be considered independent dimension of usability and all be included in usability measuring (Frokjaer, Hertzum & Hornbaek, 2000). The usability index can be determined by the aggregated evaluations of the usability dimensions effectiveness, efficiency and satisfaction (cf. Figure 2): UI 1 =a 1 *effectiveness+a 2 *efficiency+a 3 *satisfaction (1) Figure 2. Evaluation Dimensions Figure 3. Structure of Checklist Tool By Q are denoted the questions of the checklist. index 2 Controllability& Navigation Integration of Communication Quality of Information Tangibles We propose to calculate the usability index UI 2 based on the relevant usability checklist items and dimensions as shown on Figure 3. The structural equation modeling (Arbuckle, 1997) can be applied for determining the weighting coefficients a 1, a 2 and a 3. By substituting UI 2 =UI 1, the usability can be studied by regression analysis. IU = + (3) 2 b1q 1 + b2q b26q26 b27 Q26 Q22 Q21 Q19 Q18 Q16 Q15 Q13 Q12 Q9 Q8 Q5 Q4 Q2 Q1

3 To appear in Proc. 15th Annual Business Congress, IMDA2006, June 18-21, 2006, Saraevo, Bosnia and Herzegovina. Table 1. Clustering Different Checklist Formation Approaches SERVQUAL Web-based (Parasuraman, SERVQUAL Zeithmal & Berry, 1988) (Li, Tan & Xie, 2002) Heuristics (Nielsen, 1994) ISO Checklist Tool Tangibles Physical constraints Tangibles Call-back Systems Flexibility and efficiency of use Suitability for the task Consistency and standards Suitability for the task Self-descriptiveness Competence Visibility of system status (feedback) Error prevention Aesthetic and minimalist design Help users recover from errors Conformity with user expectations Error tolerance Individualization Structure of information Quality of information Quality of Information Quality of information Integration of Communication Web Assistance Recognition rather than recall Suitability for learning Integration of Communication Navigation User control and freedom Controllability Controllability & Navigation Case study The checklist tool was experimentally studied by testing and evaluating the web-based student information system of Fatih University. Data Collection 179 undergraduate students were selected from five different departments: 58 % from Management Department, 17 % from Department of International Relations, 10 % of the students in Department of Public Administration, 9 % from Department of Industrial Engineering, and 6 % from Economics Department. 40 % of the students were male and 60 % female. Participant s English knowledge was good enough to perform the tasks. To make the questions more understandable they were translated also in Turkish. All tests were carried out in the computer laboratories of Fatih University by allocating one computer to each participant. The number of the people working around and the noise level was nearly the same. The participants received checklists containing the aim of the survey, how they would proceed and some descriptive questions such as their names, departments, and gender. They were asked to perform three tasks using their personal accounts in Fatih University Student Information System. These tasks were the same for all participants. After the session they were asked to fill out the checklist. Performing the required three tasks and answering the questions of the checklist took minutes per student. Results and Discussion Most of the current methods to represent system or task usability in a single metric do not include all usability dimensions: effectiveness, efficiency and satisfaction. When researchers or developers use a narrower selection of usability measures for evaluating a system they either (a) make some implicit or explicit assumptions about relations between usability measures in the specific context, or (b)

4 run the risk of ignoring important dimensions of usability (Frokjaer, Hertzum & Hornbaek, 2000). By the structural equation model shown on Figure 4 the coefficients a 1, a 2 and a 3 of (1) were determined as follows: UI 2 =0.27*efficiency+0.04*effectiveness+0.69*satisfaction (2) Figure 4. Structural equation model According to this model the highest importance has user satisfaction for web-based service usability for this sample data gathered 69%. The contribution of efficiency is 27%. The effectiveness has only 4% contribution to overall usability. The average value of effectiveness of tasks solving is 83%. The low importance of effectiveness can be explained by the absence of information on user errors during solving tasks. This should be taken into account for improving the checklist tool. In (Sauro and Kindlund, 2005) are reported different weighting coefficients 25% for efficiency, 50% for effectiveness and 25% for satisfaction. The usability indices were calculated for all 179 students by (2). Then stepwise regression analysis was carried out by applying equation (3). Table2. Stepwise Regression Analysis Scores Index (dependent variable) R Checklist items as best predictors (P value) R 2 Q4 (0.000) Q5 Checklist items as worst predictors Q4,Q27 (0.000) Q5,Q20 Q4,Q27,Q9 (0.000) Q4,Q27,Q9,Q26 (0.000) Q5,Q20, Q7 Q5,Q20, Q7,Q13 The results of regression analysis confirm the predictive power of regression model (R2=0.48). 42% of the variance in usability index is directly explained by the most important checklist items Q4, Q27, Q9, and Q26 (cf. Table 2). In this way these four items explain 90% of the total variance. Their standardized regression coefficients are statistically significant at significance levels. The following checklist items and dimensions have the highest influence on web-based service usability and should be taken into account for designing web-based services: Q4: Is the relation between time to perform a task (planning, execution, error correction) and task complexity adequate? Q27: Is the screen visible at a range of distances and in various types of lighting? Q9: Is the use of terminology, controls, graphics and menus consistent throughout the system? Q26: Are function buttons large enough to be usable? The worst predictor variables in the model are the checklist items Q5,Q20, Q7 and Q13. They almost do not have any influence on web-based service usability. It should be reconsidered their inclusion into checklist tool or they have to be changed: Q5: Does the system guide novice users sufficiently? Q20: Is it possible to search for information (e.g. phone number) rather than entering the information directly? (Integration of Communication) Q7: Does automated or humane s responses or serving pages give users prompt service? Q13: Is the tone of messages consistently courteous? Conclusions For measuring the usability of web-based services a SERVQUAL model for measuring service quality extended with usability dimensions. By this tool a quantitative usability index is determined. A case study with university student information system including 179 students is carried out. By a structural equation model the weighting

5 coefficients of the usability index are determined. The results of stepwise regression analysis showed that the most significant checklist item influencing web-based service usability is connected with the relationship between time to perform a task and task complexity. The least important checklist item found the system guidance of novice users. Further studies on web-based service usability could include together with subjective data, measured by checklist, also objective data from biophysiological measurements. References Arbuckle, J., Amos L. (1997). User s Guide. Version 3.6, Small Waters Corp., Chicago, IL. Frick, T., Elder, M., Hebb, C., Wang, Y., and Yoon, S. (2005).Adaptive Evaluation of Complex Web Sites: How Many Tasks? Indiana University W.W.Wright Education N.Rose Ave. Bloomington,IN Frokjaer, E., Hertzum, M., and Hornbaek, K. (2000). Measuring : Are Effectiveness, Efficiency, and Satisfaction Really Correlated? CHI Papers 1-6 April 2000 Volume 2, Issue1. Hornbaek, K. (2006). Current Practice in Measuring : Challenges to Studies and Research. International Journal of Human-Computer Studies, 64 (2006), ISO. (1998). Ergonomic Requirements for Office Work With Visual Display Terminals (VDTs)-Part 11: Guidance on (ISO :1998(E)). Geneva, Switzerland. Keevil, B. (1998). Measuring the Index of Your Web Site. Keevil & Associates, Toronto, Ontario, Canada. Li, Y.N., Tan, K.C., and Xıe, M. (2002). Measuring Webbased Service Quality. Total Quality Management, Volume 13, No.5, Rouledge, Taylor&Francis Group. Nielsen J. (1994). Inspection Methods, Wiley, New York. Nikov, A., Vassileva, S., Anguelova, S., Tzvetanova, S., and Stoeva, S. (2003). WebUse: An Approach for Web Evaluation. 3 rd Symposium on Production Research, Istanbul, April 19-20, Parasuraman, A., Zeithmal, V.A. and Berry, L.L. (1988). SERVQUAL: a multi item scale for measuring consumer perceptions of service quality. Journal of Retailing. Sauro, J., & Kindlund, E. (2005). A Method to Standardize Metrics into a Single Score. CHI Papers: Methods & 2-7 April 2005, Portland, Oregon, USA