IN GOOGLE WE TRUST: USERS DECISIONS ON RANK, POSITION AND RELEVANCY

Transcription

1 IN GOOGLE WE TRUST: USERS DECISIONS ON RANK, POSITION AND RELEVANCY Helene Hembrooke 1 Bing Pan 2* Thorsten Joachims 3 hah4@cornell.edu bp58@cornell.edu tj@cs.cornell.edu Geri Gay 1 Laura Granka 1 gkg1@cornell.edu granka@stanford.edu 1 Human-Computer Interaction Group Information Science Program Cornell University Telephone: Fax: Information Science Building, 301 College Ave, Ithaca, NY School of Business and Economics College of Charleston Telephone: Fax: George Street, Charleston, SC Department of Computer Science Cornell University Telephone: Fax: Upson Hall, Ithaca, NY *Corresponding Author These authors contributed equally to the paper September 30, 2005 Submitted for Publication in Journal of Computer-Mediated Communication Special Issue on Search Engines 1

2 IN GOOGLE WE TRUST: USERS DECISIONS ON RANK, POSITION AND RELEVANCY Abstract This research intends to find out whether a user s choice of a particular abstract in Google results pages was based on the rank of that abstract, the user s evaluation of the relevancy of that abstract, or a combination of the two. An eye tracking experiment revealed that users have substantial trust in Google s ability to rank results by their true relevance to the query. When users selected a link to follow from Google s result pages, their decisions were strongly biased towards links higher in rank even if the abstracts themselves were less relevant. While users reacted to artificially reduced retrieval quality by greater scrutiny, they failed to achieve the same success rate. This demonstrated trust in Google has implications for the search engines tremendous potential influence on culture, society, and user traffic on the Web. 2

3 IN GOOGLE WE TRUST: USERS DECISIONS ON RANK, POSITION AND RELEVANCY 1. Introduction Finding information online using search engines has become a part of our everyday life (Gorden & Pathak, 1999). Currently the most popular search engine is Google, with an index of 8 billion web pages and 250 million queries a day (Brooks, 2004; SearchEngineWatch, 2005). Google now provides search functions on handheld devices and smart phones (Google, 2005). With the ubiquitous presence of mobile devices, anytime and anywhere access to the information world has become a reality. However, most users of search engines are not aware of how they work and know little about the implications of their algorithms. Most search engines typically respond to a query with a ranked list of 10 abstracts. The ranking reflects the search engines estimated relevance of web pages to the query. Users can evaluate the abstracts before deciding whether to visit any of the suggested pages. The information search process is made possible through three parties: web authors, the search engines themselves, and the users of search engines. The web authors put their web pages online with appropriate linking to other pages; the algorithm(s) for a search engine takes advantages of the content of web pages and their link structure to rank relevant web pages; the users evaluate the result pages and make a decision on which link to click. Search engines act as an information intermediary which facilitates the information seeking process. However, how well a web page actually reflects the users search intention is hard to measure. For example, the ranking algorithm of Google uses a page s popularity as a 3

4 proxy of its quality and relevancy (Pandey, Roy, Olston, Cho, & Chakrabarti, 2005). Some have argued convincingly that algorithms such as PageRank simply set up a richget-richer loop, whereby a relatively few sites dominate the top ranks (Hindman, Tsiotsioliklis, & Johnson, 2003). The retrievability and visibility arguments represent only one aspect of the search process. We wondered what role the user plays in perpetuating this rich-get-richer dynamic. Particularly, we wondered how much of the correlation between web traffic and site popularity (Hindman, et al, 2003) is due to the alleged efficiency of these algorithms, or users tendency to simply trust the ranked output displayed by a search engine and forgo any in-depth analysis or comparison of the retrieved results. A study conducted by Granka, Joachims, and Gay (2004) indicated that most users only view and click the top two results returned by Google. However, it remains a question whether those choices were the result of trusting Google s algorithm so the users default to the top results, or if the top two were truly the most relevant ones. In order to explore the relative contribution of relevance and position, we employ eye tracking as the methodology for the current study. Eye-tracking devices are able to record eye movements and reveal subjects attention and cognitive process. In areas of cognitive psychology, Human-Computer Interaction, and marketing, eye-tracking methods have been used for decades (Rayner, 1998). We use eye-tracking to investigate how users make decisions when confronted with returned Google results following a query. We were interested in finding out whether a user s choice of a particular abstract was based on the rank of that abstract, the user s evaluation of the relevancy of that abstract, or a 4

5 combination of the two. This paper provides behavioral evidence that helps us to understand the influential factors in the evaluation process of search engine uses. 2. Related Research The hypertext nature of the Web has changed how users search and access information (Bilal & Kirby, 2002). It is imperative to understand how users access and search information on the Web in order to design better search engines. This section describes past research on user behavior and search engines. The relevant literature is organized in three parts: past research on user behavior and search engines, past eye tracking research related to web viewing behavior and information search behavior. 2.1 User behavior and Search Engines Since its introduction in 1998, Google has become the most popular search engine (SearchEngineWatch, 2005). Most of our understanding of user behavior and search engines is from web log file analysis and clickthrough analysis. These studies are descriptive in nature and report general user behavior (Bilal & Kirby, 2002). For example, most users input two or three terms in search engines, they rarely go to the second page of results, and most of time they only click on one document in the result set (Jansen, Spink, Bateman, & Saracevic, 2000). Furthermore, different user groups use search engines differently. For example, children and adults have different information search strategies and different success rates (Bilal & Kirby, 2002), and search engine users in the U.S. and Europe engage in different search behaviors (Jansen & Spink, 2004). Spink and Ozmultu also discuss the characteristics of searching using question-type queries and 5

6 identified four types of question-type queries (Spink & Ozmultu, 2002). Other researchers show the differences of search behavior over the course of time (Ozmutlu, Spink, & Ozmutlu, 2004; Jansen, Spink, & Pedersen, in press). Hsie-Yee s (2001) review of the literature summarized research on user behavior focusing on search patterns and concluded that many studies have investigated the effects of selected factors on search behavior, including information organization and presentation, type of search task, Web experience, cognitive abilities, and affective states. 2.2 Eye Tracking and the Web Understanding the processes through which users evaluate abstracts returned from a search engine and make a decision, will enable us to correctly interpret web log files as feedback data, and thus improve search engine performance (Joachims, 2002). Knowing how users evaluate result pages can also help us to understand their motivations, tasks, and cognitive processes. As a result, we may be able to design web-based information retrieval systems to better satisfy their needs. Several studies have recently been conducted regarding ocular behavior on web pages, including eye movement research on news web sites (Stanford Poynter Project, 1998), analysis of scanpaths on web pages (Josephson & Holmes, 2002), ocular behavior of web users completing tasks on a web portal page (Goldberg et al. 2002), and eye movement behavior on different types of web sites (Pan, Hembrooke, Fuesner, & Gay, 2004). The Stanford Poynter Project (1998) investigated reading behavior on news web sites and concluded that text was frequently the first-entry point for a majority of online news readers. Rayner, Rottello, Stewart, Keir, and Duffy (2001) reported similar findings in which viewers of print advertisements 6

7 spent more time on text than pictures. Josephson and Holmes (2002) studied the eye viewing behavior of on three different web pages. They concluded that users develop habitually preferred scanpaths and that features of web sites and memory might be important in determining those scanpaths. Goldberg, Stimson, Lewenstein, Scott, and Wichansky (2002) used eye tracking methods to test the performance of subjects in completing several tasks on a web portal page. Their research demonstrated the characteristics of users eye movements on that portal page. Pan et al. (2004) showed that gender, web site types, and the interaction between search sequence and web site type affects web viewing behavior. In general, as an indicator of information processing, eye movements on web pages are influenced by both individual variables and the characteristics of the stimuli. The current study combines eye tracking with clickstream data in order to make inferences regarding the impact of perceived versus actual relevancy on decision-making processes involved in information search. By doing so, we gain an in-depth understanding of how users evaluate search results and the factors that influence their choices In the eye tracking study conducted by Granka, Joachims, and Gay (2004), ten tasks were devised, ranging from informational tasks such as Who discovered the first antibiotics? to navigational tasks such as Find the homepage of Emeril - the chef who has a TV cooking program. Informational tasks require finding a particular fact while navigational tasks involve searching for a particular WWW page (Broder, 2002). The results show that the subjects viewed the top two abstracts almost equally often, and much more often than any other abstract. However, they clicked the number one ranked 7

8 abstracts significantly more often than the number two ranked abstract (Granka, Joachims, & Gay, 2004). In this research, we intend to explore whether users were simply defaulting to Google s ranked results, or if they engaged in any critical evaluation of the results at all. 3. Research Methods and Design The current work was designed to exploit Google s ranking function in order to investigate how much users rely on Google s ranking to make their decisions about relevancy. A laboratory setting was necessary to capture all aspects of the search sessions and related eye movements, so that we could systematically compare the variables across all subjects. Although some have argued that external validity is compromised in a laboratory setting, previous studies have shown that there are little or no differences in user behavior in laboratory settings and web settings on information search, especially on those tasks using keywords (Epstein, Klinkenberg, Wiley, & McKinley, 2001; Schulte- Mecklenbeck & Huber, The Subjects In this study, participants were undergraduate students of various majors (including communication, engineering, and arts and sciences) at a large university in the Northeast of the United States. All students were given extra class credit for their participation in the experiment. Twenty-two subjects were recruited and 16 complete datasets, including 11 males and 5 females, were obtained. Attrition was due to recording 8

9 difficulties and the inability of some subjects to be precisely calibrated. The average age of participating subjects was 20 years and 4 months. 3.2 Search Tasks Ten search tasks used in the study by Granka, Joachims, and Gay (2004) were included in this study. Half of the searches were navigational in nature, asking subjects to find a specific Web page or homepage. These were definitive searches, meaning that only one correct Web page would provide an acceptable answer. The other five tasks were informational, asking subjects to find a specific bit of information (Broder, 2002). The purpose was to ensure that the tasks in this experiment represented the various genres of searches that the general population uses on a regular basis, including travel, movies, current events, celebrities, and local issues. Furthermore, it is also important to recognize that these tasks were also pre-tested to ensure that the most intuitive queries would not always result in top-ranked results; therefore, the findings should be interpreted in light of the fact that these queries are on average, more difficult than a user s typical query. The following table is a brief description of the ten search tasks included in the experiment and the correct answers to these tasks (Table 1) Insert Table 1 about here Experimental Procedure The goal of this study was to explore the extent to which Google s ranking system influences users evaluation process and their information seeking decisions. All participants were required to give informed written consent prior to the start of the experiment. Before the actual experiment, the eye tracker was calibrated using a nine- 9

10 point standard calibration procedure for each subject (Duchowski, 2003). Participants were instructed to search for the 10 different tasks through the Google interface. Subjects were told to view the Web pages and search as they typically would under normal conditions, with the opportunity to scroll up and down the page at their leisure. The 10 search tasks were read aloud to the subject by the experimenter to eliminate unnecessary eye movements away from the computer monitor that could potentially hinder the accuracy of the ocular calibration. Typically, due to the monitor size, scrolling was required to view abstracts ranked 7-8 and higher on the Google results pages. To eliminate the potential bias from question order effects, all search questions were completely randomized for all subjects. The maximum time for completing each task was restricted to three minutes. After completing the 10 tasks, subjects were asked to fill out a post-task questionnaire. All questions related directly to the search tasks given in the experiment, and queried subjects about perceived task difficulty, satisfaction with the search results, and their expertise with each of the experimental search topics. Questions relating to the specific tasks were presented in the same random order that the subjects actually searched for the queries, in an attempt to facilitate memory and recall of the search experience. 3.4 Design Because Google is continuously updating its search algorithms, one specific query will not produce the same exact results on two separate occasions. Because much of the data analysis were to occur after the experimental sessions, it was necessary to cache the Web pages that the subjects actually interacted with. A proxy server was set up to 10

11 mediate the interaction between the subjects and the Google web server. The proxy script was run on the subjects computer and stored every search query typed by the subjects, as well as all links and Web pages that were viewed, along with the corresponding times that they were accessed and viewed. When the subject typed in a query, the query was sent to the proxy server, and the proxy server relayed it to Google. After receiving the results from Google, the proxy server manipulated the results, and passed on the modified results to the subject s web browser. Results were modified in two ways. First, the proxy server removed the advertisements on the Google results page to avoid distraction and ensure consistent stimulus exposure across all subjects. Second, in order to explore the relative contribution of relevance versus position to the decision making process, the results were further manipulated for each subject in one of three ways; in the Normal condition, the proxy server returned the results in their original ranked order; in the Swapped condition, the proxy server swapped the first ranked abstract with the second ranked abstract, keeping the rest of the ranking intact; in the Reversed condition, the proxy server reversed the order of the abstracts on the page: the first ranked abstract was swapped with rank 10 abstract, the rank 2 abstract was swapped with rank 9 abstract, and so on. 3.5 Eye-Tracking Apparatus The subjects eye movements were recorded using an ASL 504 commercial eye-tracker (Applied Science Technologies, Bedford, MA) which utilizes a CCD camera that employs the Pupil-Center and Corneal-Reflection method to reconstruct a subject s eye positions with a rate of 60Hz. A software application, GazeTracker accompanying the 11

12 system was used for the simultaneous acquisition of the subject s eye movements (Lankford, 2000). Web pages were displayed on a 13-inch flat panel monitor with a resolution of 1024 by 768 pixels. The tester s computer houses the ASL and head tracker software, while the participant s computer stores the Web pages to be viewed during the experiment (Lankford, 2000). 3.6 Eye Tracking Indices We classify eye movements according to the following significant indicators of ocular behavior, namely fixations, saccades, pupil dilation, and scan paths (Pan et al., 2004). Eye fixations are the most relevant metric for evaluating information processing in online search. Fixations are defined as a spatially stable gaze lasting for approximately milliseconds, during which visual attention is directed to a specific area of the visual display. Fixations represent the instances in which most information acquisition and processing occurs (Rayner, 1998). The total number of fixations is often used as an indicator of processing difficulty with fixation density related to the complexity and informativeness of the visual stimulus (DeGraef, De Troy, & d Ydewalle, 1992; Friedman, 1979; Henderson, Weeks, & Hollingsworth, 1999). As informativeness increases, so does the number of fixations in that area. 3.7 Definition of LookZones In addition to logging the clickstream and Web page data of subjects, the script also constructed LookZones around key content regions. The script utilized a feature inherent to the GazeTracker software system that automatically creates LookZones around links and pictures, which the software recognizes within the HTML tags. Thus, 12

13 the script enabled the creation of distinct LookZone regions around each of the ten displayed results (Figure 1). For the analysis, each of these displayed results on Google abstracts in rank #1, rank #2, rank 3, to rank #10 is given its own set of LookZones, from which we can then compare eye tracking behaviors across all queries, relative to these zones. LookZones were not visible during the time users were engaged in the experiment Insert Figure 1 about here Rank, Position, and Relevancy In the experiment, the authors manipulated the order of Google s returned results. In the normal condition, because Google s page rank algorithm provides the users with resources in descending order of relevancy, rank and relevancy are interchangeable. In other conditions, the abstracts of actual lower ranked could appear higher in the list. Thus, choosing the abstracts in higher ranked positions would be evidence that the user has relented to Google s expertise over the actual relevance of the abstract. Thus, we use perceived relevancy to refer to the rank order that the user is shown. The actual relevancy refers to the rank assigned to the abstract by Google. Furthermore, for all queries and results pages users encountered in the study, we gathered relevance assessments of the abstracts and the pages those abstracts represent. For each results page, we randomized the order of the abstracts and asked judges to weakly order the abstracts by how promising they looked for leading to relevant information. Each of five judges assessed all results pages for two questions, plus ten results pages from two other questions for inter-judge agreement verification. The set of abstracts/pages we asked 13

14 judges to weakly order were not limited to the (typically 10) hits from the first results page, but the set included all results encountered by a particular subject for a particular question. The inter-judge agreement on the abstracts was 82.5%. Furthermore, to address the question of how implicit feedback relates to an explicit relevance assessment of the actual Web page, we also collected relevance judgments for the actual web pages those abstracts represent from the study following the setup already described for the abstracts. The inter-judge agreement on the relevance assessment of the pages was 86.4%. In general, the objective of this research is to investigate the contribution of trust and users evaluation in their decisions on clicking behavior on Google. Specifically, we look at the success rate when the users were given lee-optimal result pages; through eye tracking metrics, we look at the degree of evaluation under less-optimal conditions; we also regress the clicking behavior on positions and relevancy to see the contribution of each factor. We anticipated dissociation between the ocular data, which would indicate some implicit conflict between the perceived rank and the actual rank, yet users would still choose what they perceived to be a higher ranked abstract based on a greater trust in Google s algorithms than their own judgment. 4. Research Results Explicit behavior in the form of clicked abstracts, and implicit behavior as evidenced by different ocular indices were analyzed on three aspects. We first looked at the success rate among three conditions; then the eye tracking metrics on the Google results page are 14

15 compared in order to explore the degree of evaluation during ; using regression, we analyzed the determinants of whether an abstract was viewed or clicked. 4.1 Success Rate A search is successful when the subject find the answer to a specific task. In overall, the success rates were significantly different among three conditions. In the Reversed condition, users completed search tasks successfully only 62% of the time, compared with 85% and 80% in the Normal and Swapped conditions respectively (p<0.05). On average 43.1% of queries resulted in reformulation of the query right away without clicking on any results. 4.2 The Degrees of Evaluation In order to explore the degree of users scrutiny and evaluation on Google results pages, we analyzed 197 web pages which were the first result pages returned by Google (containing abstracts 1-10), and on which the user clicked on at least one abstract. The results show that users in the Reversed condition spent more time checking each page (10.9 seconds vs. 6.5 seconds in the Normal condition and 5.8 seconds in the Swapped condition, P<0.05), made more fixations (30.0 fixations vs in the Normal condition and 17.9 in the Swapped condition, p<0.05), and checked more returned results (3.8 abstracts vs. 2.5 in the Normal condition and 2.7 in the Swapped condition, p<0.05). In addition, the viewing sequences of the abstracts were also calculated. A regression was defined as an instance in which a subject returns to a formally viewed abstract (see Figure 2 for an example). The number of regressions for users in the Reversed condition is significantly higher than those in the Normal and Swapped conditions (3.4, 2.2, and

16 in Reversed, Normal, and Swapped conditions respectively, p<0.05). All these indices suggest that less-optimal abstracts resulted in more scrutiny. Furthermore, after the experiments, subjects in the Swapped and Reverse condition were explicitly asked if they had any feedback, followed up by a question regarding whether everything seemed normal with the Google interface to assess whether they suspected the rouse. Users made comments such as, I didn t have much luck with several of the questions, and I just could not think of the right search terms, which reflected a tendency for them to blame themselves instead of the inferior Google results Insert Figure 2, 3, 4, 5 about here Furthermore, we looked at users eye movement metrics on each abstract on the Google result pages. First, we make comparisons between views and clicks on the 10 abstracts across three conditions; next, we compare the number of fixations and pupil dilation of certain abstracts in the three groups. In this way we were able to infer differences among the groups in how they might be processing the displayed rank of Google s output. Corresponding to the three conditions, Figures 3 to 5 illustrate which abstracts the users viewed before the first click, and where the first click was placed. In particular, the bars in the graphs show the percentage of result pages with at least one fixation or click at the respective rank. In the Normal condition, the result is similar to the study reported by Granka, Joachims, and Gay (2004): users viewed the top two ranked abstracts with the highest frequency and they clicked on the number 1 abstract most of time. In the Swapped condition, users also viewed the top 2 ranked abstracts almost equally. However, they still clicked on what they perceived to be the number 1 ranked 16

17 abstract almost 3 times more often than the abstract truly ranked highest by Google (now displayed at the number 2 position). It appears that users were heavily influence by the position of the abstract. In the Reversed condition, the percentage of clicks on the abstracts positioned at rank 1 and 2 was significantly less when compared with the other two conditions. This indicates some degree of explicit evaluation on Google results. However, the graph is far from being a mirror image of the Normal condition. A one sample T-test indicated that, similar to the Normal condition, users chose the top 5 presented abstracts significantly more often than the bottom 5 abstracts (\ p<.01). While users in the Reversed condition were somewhat more likely to view the lower ranked abstracts, this still happened fairly infrequently. Next, we compared numbers of fixations among certain abstracts. The first analyses we conducted compared the average number of fixations on the 1 st and 2 nd displayed abstracts. The outcome of this analysis indicated that there were no significant differences among the three groups in fixation numbers among the top two abstracts. Thus, even among participants in the Reversed condition who were actually viewing the less relevant abstracts in these positions, they were evaluated with equal attention. It indicates that the users contemplated these abstracts much as they would have under normal conditions. We next made comparisons among the groups on the 9 th and 10 th displayed abstracts. Here we found significant differences among the three groups, with users in the Reversed condition eliciting more fixations to both of these two abstract than in either of the other two conditions (F (1,2) = 3.53,, p<.03 and F (1,2)= 3.83, p<.02, respectively for the three conditions). Interestingly, when we compared the three groups 17

18 again on the average number of fixations for the actual number 1 and number 2 ranked abstract (displayed number 1 and 2 abstracts in the Normal condition, displayed number 2 and 1 abstract in the Swapped condition, and displayed number 10 and 9 abstract in the Reversed condition) there were significant differences. Users in the Normal condition evoked more fixations to the number 1 displayed abstracts than users in the Swapped condition exhibited to the number 2 displayed abstracts. Furthermore, fixations in the Swapped condition were significantly greater than in the Reversed Condition in response to the 10 th displayed abstract. This indicates that users in both the Swapped and Reversed conditions were tricked by the manipulated results. Finally, we looked at differences in pupil dilation for users in the Reversed condition, specifically focusing on how the dilation measure differed or not between the first two displayed abstracts and the 9 th and 10 th displayed abstract. There were no significant differences in pupil dilation between the number 1, number 2, number 9 and number 10 displayed abstracts. Again, this indicates that attention and interest for positionally higher abstracts with low actual relevancy was equivalent to the lowest abstracts with highest relevancy (Figure 6) Insert Figure 6 about here Regressions on Positions and Relevancy How strong is the influence of the position compared to the intrinsic relevance of an abstract? To address this we regressed viewing and clicking behavior on position and relevancy of the abstracts. Since Google s ranking system may not accurately reflect actual relevance in every case, we asked human judges to rate the relevance of the 18

19 abstracts for all queries and results pages encountered by the users in the study. We first used mixed model analysis to explore the significant determinants of viewing and clicking. The results show that position, relevance, condition, and query order (in the order of significance) are significant determinants of whether or not an abstract is viewed (Table 2); position, relevance, and task type (informational task or navigational task) (in the order of significance) are significant determinants of whether an abstract is clicked (Table 3). In both models, the F values of displayed position are always greater than the F value of relevancy. In other words, when all factors are considered, users trust Google s positioning more than their rational judgments based on the evaluation of different alternatives Insert Table 2 and 3 about here Conclusion, Limitations, and Future Research In summary, the findings here show that users are heavily influenced by the order in which the results are presented and, to a lesser extent, the actual relevancy of the abstracts. Users trust Google in that they click on abstracts of higher rank. When looked at in combination, the clicked choices and the ocular data indicate that while there might be some implicit awareness of the conflict between Google s rank, and their own evaluation of the abstracts, it is either not enough, or not strong enough to override the effects of displayed rank. Trust is one mechanism humans use to reduce the complexity of decision making in uncertain situations (Luhmann, 1989) and may be viewed as a fast and frugal heuristic 19

20 that exploits the regularity of the information environment (Simon, 1956; Gigerenzer & Selten, 2002). Google s information retrieval algorithm sorts the results by an estimate of the probability that it will fulfill the user s information need, thereby potentially reducing the cognitive effort and time costs for searchers. In order to determine whether trust does, in fact, lead users to the most relevant documents, we further asked human judges to rate the relevance of the actual pages instead of the abstracts on Google results page. Further analysis showed that under the Normal condition, 51% of the time users clicked on the most relevant web pages as judged by the actual web page. If the users were to have simply clicked on Google s number 1 ranked abstract, in 43% of those cases they would have resulted in finding the most relevant webpage as judged by our human judges. Thus, trust does help users to a certain degree by reducing time and effort costs to successfully locate the most relevant abstracts some of the time. However, Google search algorithm is not perfect and continues to be improved. The potential for misguided trust to exacerbate what others already fear regarding the non-egalitarian distribution of information (Hindman, et. al., 2003; Introna & Nissenbaum, 2000), whether the result of economic resources, indexing policies, or algorithms. Combining users proclivity to trust ranked results with Google s imperfect algorithm increases the chances that those already rich by virtue of nepotism, get filthy rich by robotic searchers. Smaller, less affluent, more alternative sites are thus doubly punished by ranking algorithms and lethargic searchers. Users, as a whole, are not familiar with how search engines find what they are looking for (Introna & Nissenbaum, 2000). These results, at the very least, suggest that users might benefit from having more information regarding the mechanisms by which 20

21 Google, and other search engines, crawl the web and determine how a website is ranked. Raising awareness of the limitations of search engines through design is a promising direction. In addition, recent study shows that the introduction of certain degree of randomness in the ranking of returned results actually lead to improved search (Pandey, Roy, Olston, Cho, & Chakrabarti, 2005). The limitations of this study lie primarily in its experimental nature. Research shows that information searching in a lab setting can be generalized to a large context (Epstein, Klinkenberg, Wiley, & McKinley, 2001; Schulte-Mecklenbeck & Huber, 2003). However, the subjects in the current study were all undergraduate students conducting information searches in a lab setting on artificially designed search tasks. Its applicability to a broader range of users and contexts is limited. In a follow up study we intend to look at the generalizability of these effects with different existent search engines as well as one we contrive on our own, with no known history or influence. In this way we will be able to explore the pervasiveness of this trust and whether/how much user trust transfers to other search engine contexts. In addition, we intend to recruit participants from different age groups, cultures, and search expertise. More refined analysis of the eye tracking data within the framework of information foraging theory (Pirrolli & Card, 1999), especially with respect to the cost of clicking and viewing lower ranked abstracts, as well as the value of information scent, is another promising direction. However, as the first few studies in exploring evaluation process in web search, this study provides significant theoretical and empirical applications. 21

22 6. Acknowledgments Google Inc. provided a part of funding for this research. We also would like to thank Matthew Feusner and Lori Lorigo for data preparation, discussions, and editorial comments. 22

23 References Bilal, D., & Kirby, J. (2002). Differences and similarities in information seeking: Children and adults as Web users. Information Processing & Management, 38(5), Broder, A. (2002). A taxonomy of web search. SIGIR Forum, 36(2), Brooks, T.A. (2004). The nature of meaning in the age of Google. Information Research, 9(3). Retrieved April 12, 2005, from DeGraef, P., De Troy, A., & d Ydewalle, G. (1992). Local and global contextual constraints on the identification of objects in scenes. Canadian Journal of Psychology, 46, Duchowski, A. (2003). Eye tracking methodology: Theory and practice. London: Springer. Epstein, J., Klinkenberg, W. D., Wiley, D., & McKinley, L. (2001). Insuring sample equivalence across Internet and paper-and-pencil assessments. Computers in Human Behavior, 17, Friedman, A. (1979). Framing pictures: The role of knowledge in automatized encoding and memory for gist. Journal of Experimental Psychology: General, 108, Gigerenzer, G., & Selten,R. (2002). Bounded Rationality: The Adaptive Toolbox. Cambridge, MA: MIT Press. Goldberg, J.H., Stimson, M.J., Lewenstein, M., Scott, N., & Wichansky, A.M. (2002). Eye tracking in Web search tasks: Design implications. In Eye Tracking Research & Applications (ETRA) Symposium, ACM,

24 Google Inc. (2005). Google history. Gorden, M., & Pathak, P. (1999). Finding information on the World Wide Web: The retrieval effectiveness of search engines. Information Processing and Management, 35(2), Granka, L., Joachims, T., Gay, G. (2004). Eye-tracking Analysis of User Behavior in WWW Search. Poster Abstract, Proceedings of the Conference on Research and Development in Information Retrieval (SIGIR), Henderson, J.M., Weeks, P.A., & Hollingworth, A. (1999). Effects of semantic consistency on eye movements during scene viewing. Journal of Experimental Psychology: Human Perception and Performance, 25, Hindman M., Tsioutsiouliklis K. & Johnson J.A. (2003) 'Googlearchy': How a few heavily-linked sites dominate the web. Retrieved March 15, 2004 from Hsieh-Yee, I. (2001). Research on web search behavior. Library and Information Science Research, 23, Introna, L.D., & Nissenbaum, H. (2000). Shaping the web: Why the politics of search engines matters. The Information Society, 16, Jansen, B. J., Spink, A., & Pedersen, J. (in press). A temporal comparison of AltaVista Web searching. Journal of the American Society for Information Science and Technology. Jansen, B.J., & Pooch, U. (2000). Web user studies: A review and framework for future work. Journal of American Society of Information Science and Technology, 52(3),

25 Jansen, B.J., & Spink, A. (2005). An analysis of web searching by European AlltheWeb.com users. Information Processing and Management, 41(2), Jansen, B.J., & Spink, A.B. (2004). How are we searching the World Wide Web? A comparison of nine search engine transaction logs. Information Processing and Management 40. Jansen, B.J., Spink, A., Bateman, J., & Saracevic. T. (2000). Real life, real users, and real needs: A study and analysis of user queries on the Web. Information Processing and Management, 36(2), Joachims, T. (2002). Optimizing search engines using clickthrough data. Proceedings of the ACM Conference on Knowledge Discovery and Data Mining (KDD), ACM, pp Josephson, S., & Holmes, M.E. (2002). Visual Attention to Repeated Internet Images: Testing the Scanpath Theory on the World Wide Web. In Eye Tracking Research & Applications (ETRA) Symposium, ACM, Lankford, C. (2000). Proceedings of Eye Tracking Research & Applications (ETRA) Symposium, 43. Luhmann, N. (1989). Vertrauen:. Ein Mechanismsum der Reduktion Sozialer Komplexitat. Stuttgart: Enke. Ozmutlu, S., Spink, A., & Ozmutlu, H.C. (2004). A day in the life of Web searching: An exploratory study. Information Processing and Management, 40(2), Pan, B., Hembrooke., H. Gay, G., Granka, L., Feusner, M., & Newman, J. (2004). Proeedings of Eye Tracking Research & Applications, New York: ACM. 25

26 Pandey, S., Roy, S., Olston, C., Cho, J., & Chakrabarti, S (2005). Shuffling a stacked deck: The case for partially randomized ranking of search engine results. VLDB, Pirolli, P. & Card, S.K. (1999). Information foraging. Psychological Review, 106, Rayner, K. (1998). Eye movements and information processing: 20 years of research Psychoogical. Bulletin. 124(3), 372. Rayner, K., Rottello, C.M., Stewart, A.J., Keir, J., & Duffy, S.A. (2001). Integrating text and pictorial information: Eye movements when looking at print advertisements. Journal of Experimental Psychology: Applied, 7, Schulte-Mecklenbeck, M. & Huber, O. (2003). Information search in the laboratory and on the Web: With or without an experimenter. Behavior Research Methods, Instruments & Computers, 35(2), SearchEngineWatch. (2005). Sherman, C. (2002). Why search engines fail. Retrieved April 12, 2005, from Simon, H. A. (1956). Rational choice and the structure of the environment. Psychological Review, 63, Spink, A., & Ozmultu, H.C. (2002). Characteristics of question format web queries: an exploratory study, Information Processing and Management 38, Spink, A., Ozmutlu, H.C., & Lawrence, D.P. (2004). Web searching for sexual information: An exploratory study. Information Processing and Management, 40(1),

27 Spink, A., Park, M., Jansen, B.J., & Pedersen, J. (2004). Multitasking during Web search sessions. Information Processing and Management, 40. STANFORD POYNTER PROJECT. (2000). Front Page Entry Points. Retrieved August 10th, 2003, from 27

28 Table 1. The Ten Information Search Tasks Task Type Task Correct Answer Find the homepage of Michael Jordan, the statistician. Find the page displaying the route map for Greyhound buses. Find the homepage of the 1000 Acres Dude Ranch. Navigational Find the homepage for graduate housing at Carnegie Mellon housing/ University Find the homepage of Emeril the chef who has a television cooking program. html Informational Where is the tallest mountain in New York located? With the heavy coverage of the democratic presidential primaries, you are excited to cast your vote for a candidate. When is/was democratic presidential primaries in New York? Which actor starred as the main character in the original Time Machine movie? A friend told you that Mr. Cornell used to live close to campus near University and Steward Ave. Does anybody live in his house now? If so, who? What is the name of the researcher who discovered the first modern antibiotic? The Adirondacks OR High Peaks Region March 2, 2004 Rod Taylor Members of Llenroc, the Cornell chapter of the Delta Phi Fraternity live in the mansion. Alexander Fleming 28

29 Table 2. Test of Fixed Effects in the Mixed Model of Predicting Clicks from All Abstracts Fixed Effect Numerator df Denominator df F Value Probability > F Displayed Rank <.01* Relevancy Value <.01* Query Order <.01* Condition * Gender Query Type * Significant at.05 level 29

30 Table 3. Test of Fixed Effects in the Mixed Model of Predicting Clicks from Viewed Abstracts Fixed Effect Numerator df Denominator df F Value Probability > F Displayed Rank <.001 * Relevancy Value <0.01 * Query Type <0.01 * Query Order Gender Condition * Significant at.05 level 30

31 Figure. 1. LookZone Division on a Google Result Page 31

32 Figure. 2. An Example of Regression on Google Page 32

33 Viewed Percentage of Pages Clicked Displayed Rank (numbers on the bars are Google original ranks) Figure 3. Normal Condition: Percentage of Views/Clicks on Different Ranks 33

34 Percentage of Pages Viewed Clicked Displayed Rank (numbers on the bars are Google original ranks) Figure 4. Swapped Condition: Percentage of Views/Clicks on Different Ranks 34

35 Viewed Percentage of Pages Clicked Displayed Rank (numbers on the bars are Google original ranks) Figure 5. Reversed Condition: Percentage of Views/Clicks on Different Ranks 35

36 54 Average Dilation Figure 6. Average Pupil Dilation Across Abstracts for Reversed Condition 36