Measures of Perceived User Satisfaction with Internet Search Engines

Transcription

1 Measures of Perceived User Satisfaction with Internet Search Engines Gholamreza Torkzadeh Department of MIS University of Nevada, Las Vegas USA Christopher C. Yang Department of Systems Engineering and Engineering Management The Chinese University of Hong Kong Hong Kong Abstract Internet technology plays a significant role in the way individuals and organizations access and use information. Search engines provide the interface that shapes how individuals interact with Web sites to search, find, retrieve, store, or send information content. What users think of search engines will influence the extent and quality of Web use. This article reports on the development of an instrument for measuring perceived user satisfaction with search engines. Using a survey of 283 respondents, we conducted a factor analysis and propose a two factor 12-item instrument that measures user satisfaction with Internet search engines in terms of information content and system interface. Reliability and validity of the instrument is presented and its applications are discussed. Keywords: Internet search engine, user satisfaction, measurement, interface, Web content 1. Introduction The rapid development and use of Internet technologies are influencing the extent and quality of work in private and public organizations. Organizations invest in this technology to streamline operation, enhance communication, control quality, improve customer service, retain existing customers, gain new customers, and the like. A welldefined outcome measure that can be used in research and practice as a dependent variable to evaluate information technology success is essential (DeLone and McLean 1992). Such measures can help organizations to not only evaluate system features but also technology policies, procedures, and training programs. User satisfaction is a driving force behind adoption and use of technology and ultimately the impact that it might have on the texture of work, productivity, control, innovation, and service.

2 Measures of user satisfaction developed for traditional information systems and applications may not be appropriate for Internet search engines. These instruments have not been designed or validated for Internet search engines. There is a need for success measure that specifically addresses Internet search engines features. In this study, user satisfaction with Internet search engines is conceptualized as the affective attitude towards a specific search engine by individuals as they interact with the system. This paper describes the development of an instrument for measuring user satisfaction with Internet search engine. The intent was to develop an instrument that identifies the multidimensional nature of user satisfaction with Internet search engine, is reliable and valid, and is short and practical for use in research and practice. Psychometric metric analysis is used on 283 survey responses to purify, test, and validate a set of items for the recommended instrument of Internet user satisfaction. In the following sections, we first describe the methodology adopted in this study. Then, data analysis and results are presented followed by discussion and conclusions. 2. Literature Review 2.1 Internet Search Engines The World Wide Web is considered as a very large, unstructured, heterogeneous but ubiquitous database. Information on the Web is expanding exponentially and updating frequently. Therefore, information overloading becomes a significant problem. Searching tools are important to retrieve and filter relevant information for users. Most Internet search engines employ an on-line database indexing and searching approach [Yang and Chung, 2002, Yang et at., 2000]. The commercial Internet search engines, such as Yahoo! and Alta Vista, are composed of the fetching robot, index server, and query engine. The fetching robot gathers homepages from the WWW through the links available on the Web page and submission of Web pages. The Web sites are revisited by the fetching robots at some interval of time. Although most WWW spiders claim that they cover the entire WWW, they only cover a certain percentage of the WWW depending on the power of the fetching robots. The indexing server extracts keywords and phrases from the documents and index against the URLs. For each word and phrase, the indexing database contains the URL address of the Web page and the location of the word or phrase in the Web page. The query engine takes queries submitted from users by keywords and returns a list of URLs according to their order of relevance to the keywords. There are several popular ranking strategies. Most ranking strategies are based on word frequency, Web site popularity, location of words, connectivity of Web pages, and date of Web page being updated. Figure 1 illustrates the searching mechanism of Internet Search engines. Figure 2 are examples of the user interface of Yahoo! and Alta Vista.

3 Figure 1. Searching mechanism of Internet Search Engines Figure 2. Examples of user interface of Yahoo! and Alta Vista From the technical aspect, the success of an Internet search engine highly depends on the number of Web pages that it covers, the indexing, the retrieval and ranking mechanisms, and the user-friendliness of interface. However, from the user point of view, an Internet search engine may not be helpful to them even if its technical performance is high. The factors that affect the user satisfaction on Internet search engine include many other factors that have not been considered by the system developers. 2.2 User Satisfaction with Information Seeking Some research work on satisfaction with information seeking has been done in the past. Satisfaction with information seeking is defined as a state of mind which represents the composite of a user s material and emotional responses to the information seeking context [Applegate, 1993, Bruce, 1998]. Material satisfaction is associated with the features of information system s performance [Hiltz and Johnson, 1989, Sandore, 1990]. Such performance is assessed in relation to the relevance and utility of the information retrieved [Saracevic et al., 1988]. Emotional satisfaction is associated with feelings based on user s requirement, expectations [Applegate, 1993], goal determination, and task orientation. Bruce [Bruce, 1998] has studied the validity of magnitude estimation of user satisfaction with information seeking on the Internet in general, but not on Internet

4 search engines in particular. 37 subjects were interviewed. Magnitude estimation is a technique for measuring human perception of physical-sensory stimuli developed in the discipline of psychophysics. Its basic premise is that human subjects are able to make subjective proportional judgments of most physical sensations [Stevens, 1975]. Subjects were asked to describe two incidents of information seeking on the Internet and report the level of satisfaction for the incidents described using magnitude estimation (numeric estimation and a force of handgrip estimation). Subjects responded by providing a number and squeezing a hand dynamometer with a force for the numeric estimation and the force of handgrip estimation, respectively. No significant relationship between attending an Internet training session and satisfaction with information seeking was found but a relationship between frequency of use and satisfaction with information seeking and a signification relationship between expectation of succession and satisfaction with information were found. However, there is no literature that has described an instrument for measuring perceived user satisfaction with Internet search engine. In this article, the development of such instrument is reported. 3. Methods Because there are no widely accepted definitions of search engine satisfaction construct, we generated a list of items based on the end-user satisfaction literature and practitioners input. Those who assisted us in this study were experienced software engineers and major Web users. The end-user computing satisfaction literature suggests constructs relative to content, accuracy, format, ease of use, and timeliness. Our discussion with practitioners resulted in additional constructs related to interactivity, layout, relevancy, query generation and modification, convenience, speed, and efficiency. There were overlaps between what the end-user computing satisfaction literature proposed and what the practitioners suggested. These include, for example, timeliness and speed, relevancy and content, format and layout, and interactivity and ease of use. To ensure a comprehensive list of items, we included measures to cover all dimensions. We also included items with different wordings to better tap potential dimensions of the construct. Churchill (1979) recommends including items with slightly different shades of meaning since seemingly identical questions produce different responses and because data analysis will produce the final and preferred list of measures. The item generation process resulted in 23 items to measure user satisfaction with Web search engines. Items were not sorted and sub-headings were not used. The questionnaire used a five point Likert-type scale, where 1 = not at all; 2 = a little; 3 = moderately; 4 = much; and 5 = a great deal. We also added a global measure of perceived satisfaction to be used as a criterion for the instrument development. This global item used a five-point scale where 1 = non-existent; 2 = poor; 3 = fair; 4 = good; and 5 = excellent and asked the respondents Overall, how would you rate your satisfaction with the system? The 24 items are presented in the Appendix. The purpose of the survey was described as To evaluate the user satisfaction with Internet search engines. Internet search engine was described as The system that you use to obtain information from the Web, for

5 example MSN, Netscape Internet Search, and Yahoo. The instructions asked the respondents to circle, for each question, the response that best described their ratings. Respondents were also asked to answer demographic questions about gender, age, level of use, and reason for use. The survey questionnaire was used to gather data from individuals who attended information technology programs at two universities in the US and Hong Kong. The same version of the questionnaire was used in Hong Kong since the program was in English and respondents felt comfortable responding to the English version. Participation with the study was voluntary and respondents were given sufficient time to assess their responses. The completed questionnaires were either collected at the time by research assistants or mailed in by the respondents later. A few questionnaires were discarded because of incomplete responses. 3.1 Sample characteristics A sample of 283 usable responses was obtained representing 167 males (59%) and 116 females (41%) from the US (60%) and Hong Kong (40%). The survey asked respondents to identify a search engine and respond to questions relative to that system. Respondents used a variety of systems including Yahoo (44.9%), Google (29.7%), MSN (8.8%), AOL (4.6%), Explorer (1.8%), and Netscape (1.8%). Others used AT&T, Metacrawler, Excite, Lycos, Compuserve, Sina, and Dogpile. About 5% of the respondents used a combination of systems suggesting that once they selected a system they continued using it. Respondents fall into the following age distribution: less than 20 (1.8%), (37.3%), (9.7%), and greater than 40 (2.5%). Respondents used their systems on average 5.87 (SD = 6.6) hours per week for education (27.2%), information (24.7%), work (12.4%), entertainment (9.9%), shopping (7.1%), news (1.8%), and other (17%). 3.2 Survey method The construct validity of each item was examined to ensure that the items measured the perceived satisfaction with search engines. This was established by showing that the instrument was an appropriate operational definition of the construct it purported to measure. Two methods of construct validation cited by Kerlinger (1978) and utilized in MIS research (Ives et al. 1983, Doll and Torkzadeh 1988, 1990) were used here; corrected-item total and factor analysis. The first, correlation between total scores and item scores, assumes that the total score is valid, and the extent to which the item correlates positively with the total score is indicative of construct validity for the item. In this study each item score was subtracted from the total score in order to avoid a spurious part-whole correlation (Cohen and Cohen 1975); the result was a corrected-item total which was then correlated with the item score. A measure of criterion-related validity was also examined to identify items that may not be closely related to the user satisfaction with Internet search engine. The global item

6 measuring perceived overall satisfaction with search engines was used as a criterion scale. To the extent that each item was correlated with this, the scale provided a measure of criterion-related validity. Items should indicate a favorable or unfavorable attitude toward the object in question. Ambiguous items or items that appeared to indicate a neutral attitude were dropped (Weiss 1970). Often the investigator makes this decision. In this study, a measure of criterion-related validity was examined to identify items that did not indicate favorable or unfavorable attitudes. The domain sampling model provides a rationale for excluding items before factor analysis. The key assumption is that all items, if they belong to the domain of the concept, have an equal amount of explanatory power. If all the items in a measure are drawn from the domain of a single construct, responses to those items should be highly intercorrelated. The corrected item total correlation provides a measure of this (Churchill 1979). Churchill suggests the need to purify the measures at an early stage of research. He contends that: when factor analysis is done before purification, there seems to be a tendency to produce many more dimensions than can be conceptually identified. This is partly due to the "garbage items" that do not have the common core but do produce additional dimensions and confound the interpretation of the factor analysis. Finally, the reliability (internal consistency) of the remaining items comprising each dimension was examined using Cronbach s alpha to see if items could be eliminated without substantially lowering reliability. We wanted to maintain the reliability of at least Thus, items were eliminated if the reliability of the remaining items would be 0.90 or greater. If deleting two items would have the same impact on the reliability, the item with lower correlation with the criterion was deleted. Following these analyses, an exploratory factor analysis of the remaining items was conducted to determine the factor structure of the entire set of items. This method helped to identify factorially pure items that would facilitate the testing of more specific hypotheses (Weiss 1970), and to identify the components that make up the total measure (Campbell 1976). Bartlett's test of sphericity had a significance level at suggesting that the intercorrelation matrix contains enough common variance to make factor analysis worth pursuing. The ratio of sample size to number of items (15:1) was above the minimum (10:1) ratio suggested for factor analysis by Kerlinger. The items being factor analyzed were selected because they were closely related to each other (i.e., all items were thought to be measures of the same construct); these items may have relatively large common variance and error variance compared to their unique variance. 4. Data analysis and results Items were excluded if their correlation with the corrected item total was below 0.4 or if their correlation with the criterion scale was below 0.3; there are no accepted standards for this cutoff. The correlations with the corrected item total and the criterion item were significant at p < and similar to those used by others in MIS research. Thus, the cutoff was considered high enough to ensure that the retained items were adequate

7 measures of user satisfaction with Internet search engines. Thus, the number of items was reduced to 19 before conducting factor analysis. Factor analysis was used to identify the underlying factors or components that comprise the user satisfaction with Internet search engine construct. Items that were not factorially pure were deleted to form a modified instrument that would facilitate the testing of more specific hypotheses. The use of imprecise and ambiguous terms to label the factors was avoided (Bagozzi 1981). Using the sample of 283 responses, the data was examined using principal components analysis as the extraction technique and the varimax method of rotation. Several items had multiple loadings of above 0.4 on other (non-primary) factors. Items with many multiple loadings may be good overall measures of the construct, but including them in the scale blurs the distinction between factors. To improve the distinction between factors, items which had factor loadings greater than 0.3 on other factors were deleted from the instrument. Five more items were deleted because (1) they had lowest correlations with criterion, (2) they had lowest factor loadings, (3) the alpha level would remain about the same if these three items were deleted, and (4) descriptions of these items were similar to other items that had higher factor loadings and correlations with the criterion. These deletions resulted in a 12-item scale for measuring user satisfaction with Internet search engine. Two factors with Eigenvalues greater than one emerged explaining 60% of the variance. Each of the 12 items had a factor loading above The descriptions and loadings for the 12 items are presented in Table 1; grouped by their highest factor loading. Each of the 12 items had a corrected item total correlation (a measure of internal consistency) above 0.4 and a correlation with the criterion measure of above 0.3. The correlation for each of the 12 items was positive and significant (p = 0.01 or lower). This 12-item instrument had reliability (Cronbach's alpha) of 0.90 and a criterion-related validity of Table 2 depicts correlation matrix, means, and standard deviations for the 12-item instrument. These correlations are significant (p 0.01). In summary, based on the analysis and results, it is suggested that these 12 items provide a reliable and valid instrument for measuring user satisfaction with Internet search engine. This instrument can be used with confidence by researchers and practitioners alike. However, efforts to provide additional evidence of the instrument's validity, internal consistency, and stability are encouraged.

8 Table 1: Reliability, validity, and factor loading of measures of user satisfaction with search engines Corrected Correlation item-total with Factor Item description correlation criterion loading Information Content C1 Do you get the information you need in time?.72.55*.79 C2 Do you find the output relevant?.72.59*.78 C3 Does the system provide you with accurate information?.72.56*.77 C4 Do you get information fast enough to meet your needs?.68.65*.76 C5 Does the content of each screen meet your needs?.65.60*.74 C6 Does the system provide comprehensive information?.68.54*.72 C7 Does the system provide concise output?.66.54*.72 C8 Is the system efficient?.68.59*.72 C9 Does the system provide you with information that meet your needs?.66.52*.71 System Interface I1 Are the instructions to form a query clear?.42.33*.78 I2 Is it easy to modify your query?.49.44*.74 I3 Is it easy to correct errors using the system?.46.41*.73 * Significant at p 0.01 (2-tailed) Table 2: Correlation matrix of measures of user satisfaction with search engines C2.64 C C C C C C C I I I C1 C2 C3 C4 C5 C6 C7 C8 C9 I1 I2 I3 Mean S.D

9 5. Discussion Most researchers prefer behaviorally based measures to perceived measures for many situations. However, an actual measure of user satisfaction with Internet search engine is generally not feasible unless interaction with a specific system is closely monitored. Perceived user satisfaction is as critical as actual satisfaction in affecting the users' attitude toward system use. Perceived user satisfaction is important to studies of cause and effect relationship such as self-efficacy and system use or organizational factors and utility of Internet-based information product in decision making. The MIS research has benefited from studies that employ users' perception to operationalize behavioral constructs such as involvement (Baroudi, Olson, and Ives 1983, Franz and Robey 1984) or attitudinal constructs such as satisfaction (Bailey and Pearson 1983, Davis, Bagozzi, and Warshaw 1989). In situations where users have continual access to a system and may enhance their skills by both doing and learning, the perceptual measures may be appropriate, especially where the quality of output or the impact is perceived (Torkzadeh and Van Dyke 2002, Torkzadeh and Doll 1999). In this study, items all tap an individual's perceptions based on the experience with an Internet search engine. Therefore caution should be exercised in interpreting results since perceptions do not necessarily measure system quality and respondents may make inaccurate estimates. Furthermore, the instrument include questions that measure user perception of a system as it currently is; any major change in system content or interface may change what respondents think of that system. Therefore, the appropriateness of these measures over time should be reexamined using new data set. Although there may be reasons for additional questions to evaluate specific aspects of Internet search engines, this set of 12 items is general in nature, relate to user experience and can be used for a variety of situations. This provides a common framework for comparative analysis. This 12-item instrument may be used to evaluate and modify Internet search engines, assess user training programs, or estimate system use. Researchers may use this instrument to examine the outcome of user involvement in systems development situations. Confirmatory studies of these measures that involve different search engines and are conducted in different situations are encouraged. 6. Conclusions As the need for more focused research grows and evaluating information systems become more acute, researchers and practitioners require more reliable instruments. One limitation common to many research efforts has been the lack of reliable and valid measures. Reliable measures are critical for comparative and cumulative research findings. This paper presents progress toward the development of an instrument for measuring user satisfaction with Internet search engine. The proposed instrument is short, easy to use, and appropriate for use in both practice and research. It appears to have adequate reliability and validity.

10 7. References Applegate, R. (1993) Models of User Satisfaction: Understanding False Positives, Reference Quarterly, 32, Bagozzi, R.P. (1981) An Examination of the Validity of Two Models of Attitude," Multivariate Behavioral Research, 16, Bailey, J.E. and Pearson, S.W. (1983) Development of a Tool for Measuring and Analyzing Computer User Satisfaction, Management Science, 29, 5, Baroudi, J.J., Olson, M.H., and Ives, B. (1983) An Empirical Study of the Impact of User Involvement on System Usage and Information Satisfaction, Communications of the ACM, 29, 3, Bruce, H. (1998) User Satisfaction with Information Seeking on the Internet, Journal of the American Society for Information Science, 49, 6, Campbell, J.R. (1976) Psychometric Theory, Handbook of Industrial and Organizational Psychology, M. D. Dunnette (Ed.), Rand McNally, Chicago. Churchill, G.A., Jr. (1979) A Paradigm for Developing Better Measures of Marketing Constructs, Journal of Marketing Research, 16, 1, Cohen, J. and Cohen, P. (1975) Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences, Lawrence Erlbaum Assoc., Hillsdale, N.J. Cooper, R.P. and Zmud, R.W., Information technology implementation research a technology diffusion approach, Management Science, 34, 2, Davis, F.D., Bagozzi, R.P., and Warshaw, P.R. (1989) User Acceptance of Computer Technology: A Comparison of Two Theoretical Models, Management Science, 35, 8, Delone, W.H. and McLean, E.R. (1992) Information Systems Success: The Quest for the Dependent Variable, Information Systems Research, 3, 1, Doll, W.J. and Torkzadeh, G. (1988) The Measurement of End-User Computing Satisfaction, MIS Quarterly, 12, 2, Doll, W.J. and Torkzadeh, G. (1990) The Measurement of End-User Software Involvement, Omega, 18, 4, Franz, C.R. and Robey, D., (1984) Organizational Context, User Involvement, and the Usefulness of Information Systems, Decision Sciences, 17, 3, Hiltz, S. R., and Johnson, K. (1989) Measuring Acceptance of Computer-mediated Communication Systems, Journal of the American Society for Information Science, 40, Ives, B., Olson, M., and Baroudi, S. (1983) The Measurement of User Information Satisfaction, Communications of the ACM, 26, 10, Kerlinger, F.K. (1978) Foundations of Behavioral Research, McGraw-Hill, New York. Sandore, B. (1990) Online Searching: What Measure Satisfaction? Library and Information Science Research, 12, Saracevic, T., Kantor, P., Chamis, A. Y., and Trivison, D. (1988) A Study of Information Seeking and Retrieving. I. Background and Methodology, Journal of the American Society for Information Science, 39, Stevens, S. S. (Ed.), (1975) Psychophysics: Introduction to Its Perceptual, Neural, and Social Prospects, New York: John Wiley and Sons.

11 Torkzadeh, G. and Doll, W.J. (1999) The development of a tool for measuring the perceived impact of information technology on work, Omega, International Journal of Management Science, 27, Torkzadeh, G. and Van Dyke, T.P. (2002) Effects of training on Internet self-efficacy and computer user attitudes, Computers in Human Behavior, 18, Weiss, D.J. (1970) Factor Analysis in Counseling Research, Journal of Counseling Psychology, 17, Yang, C. C., and Chung, A. (2002) A Personal Agent for Chinese Financial News on the Web, Journal of the American Society for Information Science and Technology, Special Issue on Web Research, 53, 2, January, Yang, C. C., Yen, J., and Chen, H. (2000) Intelligent Internet Searching Agent Based on Hybrid Simulated Annealing, Decision Support Systems, Special Issue on Intelligent Agents and Digital Community, 28, 3, May, Appendix Survey Measures of User Satisfaction with Internet Search Engines 1. Do you enjoy using the system? 2. Do you think the output is presented in a useful format? 3. Is the system difficult to operate? 4. Are the instructions to form a query clear? 5. Is it easy to correct errors using the system? 6. Are you happy with the layout of the output? 7. Does the system provide you with accurate information? 8. Do you get the information you need in time? 9. Do you find the output relevant? 10. Is the system easy to use? 11. Is it easy to modify your query? 12. Is the system efficient? 13. Does the system give you redundant information? 14. Is the system troublesome? 15. Is the system convenient? 16. Is the system difficult to interact with? 17. Does the system provide you with information that meet your needs? 18. Does the system provide comprehensive information? 19. Does the system provide concise output? 20. Is it easy to query the system for your information needs? 21. Is the system user friendly? 22. Do you get information fast enough to meet your needs? 23. Does the content of each screen meet your needs? 26.* Overall, how would you rate your satisfaction with the system? * Separate scale: From 1 (Non-Existent) to 5 (Excellent).

12 About authors G. Torkzadeh is Professor and Chair of MIS at the University of Nevada, Las Vegas. He has published on management information systems issues in academic and professional journals including Information Systems Research, Management Science, MIS Quarterly, Journal of MIS, Decision Sciences, Communications of the ACM, Omega, Journal of Operational Research, Information & Management, the Journal of Knowledge Engineering, Educational and Psychological Measurement, Behaviour & Information Technology, Long Range Planning, and others. His current research interests include the impact of information technology, measuring e-commerce success, and computer selfefficacy. C. C. Yang is currently an associate professor in the Department of Systems Engineering and Engineering Management at the Chinese University of Hong Kong. From 1997 to 1999, he was an assistant professor in the Department of Computer Science and Information Systems and associate director of the Authorized Academic Java SM Campus SM at the University of Hong Kong. He received his B.S., M.S., and Ph.D. in Electrical and Computer Engineering from the University of Arizona, Tucson, AZ, in 1990, 1992, and 1997, respectively. His current research interests are digital library, Internet agent, Web retrieval, information visualization, cross-lingual information retrieval and color image processing. He has published over 80 refereed journal and conference papers.