Interpreting User Inactivity on Search Results

Transcription

1 Interpreting User Inactivity on Search Results Sofia Stamou 1, Efthimis N. Efthimiadis 2 1 Computer Engineering and Informatics Department, Patras University Patras, GREECE, and Department of Archives and Library Science, Ionian University Corfu, GREECE stamou@ceid.upatras.gr 2 Information School, University of Washington Seattle, WA, USA efthimis@u.washington.edu Abstract. The lack of user activity on search results was until recently perceived as a sign of user dissatisfaction from retrieval performance, often, referring to such inactivity as a failed search (negative search abandonment). However, recent studies suggest that some search tasks can be achieved in the contents of the results displayed without the need to click through them (positive search abandonment); thus they emphasize the need to discriminate between successful and failed searches without follow-up clicks. In this paper, we study users inactivity on search results in relation to their pursued search goals and investigate the impact of displayed results on user clicking decisions. Our study examines two types of post-query user inactivity: pre-determined and post-determined depending on whether the user started searching with a preset intention to look for answers only within the result snippets and did not intend to click through the results, or the user inactivity was decided after the user had reviewed the list of retrieved documents. Our findings indicate that 27% of web searches in our sample are conducted with a pre-determined intention to look for answers in the results list and 75% of them can be satisfied in the contents of the displayed results. Moreover, in nearly half the queries that did not yield result visits, the desired information is found in the result snippets. Keywords: Task-oriented search, queries without clickthrough, search abandonment, user study, interactive IR. 1 Introduction Search logs are a convenient tool for capturing some aspects of the user interaction with retrieved results. Today, the study of query logs is a popular approach for identifying user querying trends and search patterns as well as for attempting to infer the search goals users are trying to accomplish via their queries [ 1, 7, 10, 11, 15, 12, 21, 22]. Although search logs provide valuable evidence about how and what users search for, they are less revealing about the criteria upon which information seekers base their search behavioral patterns. To fill this void, researchers have carried out user studies in which searchers explicitly indicate their satisfaction from retrieval performance in relation to their underlying search goals. The commonality in these approaches is that they evaluate search tasks and retrieval effectiveness based entirely on the analysis of the user activity (i.e., clicks) on search results. This is because clickthrough data has been traditionally perceived by the information retrieval community as an indicator of implicit user feedback on the relevance of search results [ 7]. In this context, the users inactivity on search results has been generally interpreted as a sign of dissatisfaction from retrieval performance [ 19]. However, researchers [ 20, 23] have recently reported that some queries might not be followed by user clicks on search results, simply because the user information needs were

2 2 Sofia Stamou1, Efthimis N. Efthimiadis2 successfully addressed in the content of the snippets displayed in the search results page. Therefore, for those queries the lack of user activity on the returned documents should not be interpreted as a sign of decreased user satisfaction from retrieval performance. Despite the acknowledgment that some queries do not yield user visits on search results because the desired information is presented in the snippets (abstracts) of the retrieved documents [ 17], currently very few studies exist that investigate searches not followed by user visits on the results in relation to users tasks [ 20, 12, 23]. To our knowledge, none of the reported research examines the impact of retrieved but unvisited results on both the user clicking decisions and the satisfaction of their information needs. In this paper, we present a user study that investigates the intentions behind queries without clickthrough and examines the search tasks that can be successfully accomplished based entirely on the information provided on the results page. Unlike previous work on abandoned searches, i.e., queries not followed by any click or any further query within a 24-hour period [ 20], our study investigates all queries without clickthrough, that is, both those abandoned and those followed by another search. This is because we seek to understand why users choose not to visit a single page for some of their queries rather than investigate why they abandon their searches altogether. Such understanding will help us interpret the user inactivity on retrieved results in a more precise manner so as to propose more user-centric retrieval evaluation methods. Moreover, we rely on explicit user feedback in order to interpret the user inactivity on search results in a precise manner. To accomplish our study objective, we examine two types of user inactivity on search results: pre-determined and post-determined inactivity. We comparatively analyze the search tasks they pursue in order to understand why users decide not to visit results and how their decisions are influenced by the displayed result snippets. We define the user inactivity on the results as pre-determined when the user submits a query with a preset intention to look for answers in the result snippets and without following any link. On the other hand, the user inactivity is post-determined when the user issues a query with the intention to visit some results, but after reviewing the list of retrieved documents decides not to visit any of the results. The paper is organized as follows. In section 2 we present a discussion of related work. Section 3 describes in detail the methodology of the user study. In section 4 we discuss the main findings, and in section 5 we present conclusions and plans for future work. 2 Related Work The idea of utilizing the searcher activity on the returned results as an indicator of implicit relevance judgments is not new. Numerous studies exist on how the different post-query activities can be interpreted as implicit feedback signals (for an overview see [ 15]). The searchers behavior that researchers attribute as implicit measures of interest are: time spent on a page combined with the amount of scrolling on a page [ 4], duration of search and number of result sets returned [ 7], click data on and beyond the search results [ 14], use of eye-tracking methods to capture the user s visual attention on the results [ 8], repetition of result clicks across user sessions [ 25]. Although, the above measures have been proposed for inferring the user satisfaction from the results visited for some query, they have not been explored for capturing the user satisfaction from the results viewed but not visited for a query. Based on the observation that what users did not click on might reveal interesting information about their perception of the results usefulness in satisfying their search needs [ 13], a number of researchers studied the search tasks that can be achieved by viewing the result snippets and without clicking through the results [ 6, 20, 23]. The

3 Interpreting User Inactivity on Search Results 3 findings of those studies indicate that query abandonment can be good when the users find what they look for in the results list, without requiring a click on the result contents. In this paper, we build upon previous studies about the interpretation of searches without follow up clicks and we investigate via a user survey the intentions hidden behind queries lacking clickthrough, as well as the result snippets impact on the user decisions not to visit retrieved results. In particular, we concentrate on queries that yield search results but none of which are accessed by the user who issued the query and investigate whether the user started searching with a pre-determined intention to visit results or not. Our investigation aims at capturing the result snippets impact on the user post-query activities. Unlike previous work, that restricted their findings to abandoned searches [ 20], or examined the type of information needs that can be satisfied in the displayed snippets [ 6], our study relies on actual user feedback for identifying the search tasks that can be achieved without visiting search results. Moreover, we examine the results snippets impact on the nature of post-query user activity instead of merely looking at the search tasks that can be fulfilled in the contents of the displayed snippets. 3 Methodology The goals that lead people to engage in information seeking behavior affect their judgments of usefulness of the retrieved results [ 2]. This, coupled with the observation that up to 50% of the queries do not yield a single click on the results [ 5, pp.27], motivated our study on how to interpret searches not followed by user visits on the retrieved results. To that end, we carried out a user study in order to firstly identify the search tasks associated with queries lacking clickthrough and then examine for which tasks the user inactivity on results is pre-determined and for which tasks the user inactivity is post-determined (i.e., imposed) by the contents of the displayed results. Based on our findings and the feedback supplied by our participants, we try to capture the impact that the displayed results might have on both the post-query user behavior and the user satisfaction from search results. 3.1 The User Study In this section, we discuss the details of the user study we carried out in order to understand why information seekers decide not to visit any result for some of their queries. Unlike prior work that attempts to interpret searches without clicks using query log analysis, we rely on explicit user feedback in order to analyze the lack of post-query user interaction with search results. The study recruited six postgraduate computer science students (four male/ two female). Participants attended a training session where the study and its processes and procedures were explained and any questions were addressed. A browser plug-in was installed at each participant s workstation in order to collect their search trace for a time period of one week. In particular, we asked participants to conduct their web searches as they would normally do and we explained to them that we would record their HTTP requests from which we could obtain their queries and subsequent clickthrough. For every query recorded in our test set, we asked our participants to answer a set of questions (see Table 1) that we presented them via an online questionnaire. Before conducting our survey we familiarized our participants with the questions by giving them verbal explanations for every question and its candidate answers. The questions on the online questionnaire were presented incrementally to each participant and not all of them at once. We instructed our participants to open the questionnaire in a new browser window while conducting their

4 4 Sofia Stamou1, Efthimis N. Efthimiadis2 searches and answer the first two questions before issuing their queries and the next two questions after their search was completed. 3.2 Questionnaire design The questions and the selected choices for answers were pilot-tested and refined prior to data collection. As Table 1 shows, the first question aims at capturing the participants search goals and lists a number of tasks that web queries may pursue. The determination of the tasks presented to our participants relies on existing query classification schemes that have been proposed by many researches [ 3, 11, 18, 20, 23, 25]. In particular, the search tasks examined pertain to the following categories of queries: (i) informational (answer a in question #1), (ii) transactional (answer b in question #1), (iii) navigational (answer c in question #1), (iv) local (answer d in question #1), (v) person (answer e in question #1), (vi) product (answer f in question #1), (vii) quick answer (answer g in question #1), (viii) language-related (answer h in question #1), (ix) update (to answer i in question #1) and (x) repeat (answer j in question #1). Based on the above categories, we asked our participants to indicate for each of their searches the tasks that better reflected their information intentions. Note that we asked them to indicate their selections before actually submitting their queries and for every query to make a single selection depending on the task that they deemed the most suitable in describing their underlying intentions. Following the selection of a search task, the second question appeared on the screen, which again had to be answered before the submission of the query. Question #2 aims at capturing the pre-query user intention of visiting or not search results. Again, our participants had to indicate a single pre-query intention for each of their searches by selecting an appropriate answer from the three provided. In particular, answer yes to question #2 indicates that the user initiated a search with a pre-determined intention not to click through results, answer no indicates that the user started searching with a predetermined intention to click through results and answer maybe indicates that user started searching with an unclear intention about clicking through results. After answering the first two questions, we asked our participants to conduct their searches, i.e. issue their queries and interact with search results as they preferred. When they completed their searches and before proceeding with the submission of another query or the termination of search, we asked them to answer question #3, which captures the post-query user activity on search results. In particular, question #3 provides a yes/no answer and records whether participants visited or not some result for each of their queries. In case participants indicated the answer no to question #3 a fourth question appeared on the screen otherwise the survey for the recorded search was terminated. Question #4 aims at collecting feedback on the reasons why searchers did not visit any result for their corresponding queries, i.e. those not followed by clickthrough events. To capture the reasons for not clicking, we provided our participants with four possible answers. The first reason for not clicking (answer a to question#4) suggests that the searcher found what she was looking for in the search results page, therefore there was no need to click through the results. The second reason for not visiting results (answer b) suggests that the searcher did not receive the information she hoped for in the results list, therefore she assumed that the retrieved results could not satisfy her search need and as such decided not to visit any of them. The third reason for not clicking (i.e. answer c) assumes that the searcher wanted to obtain new (i.e. unseen) information for her query, but none of the retrieved results was new. Therefore, even if results seemed relevant the searcher decided not to visit any of them, since she had already seen them and the quest for her search was to find new information. Finally the fourth reason for not clicking (answer d) suggests that for some unexpected reason (i.e. network connectivity problems,

5 Interpreting User Inactivity on Search Results 5 participant s need to take an urgent break) search was interrupted right after the submission of the query and the retrieval of query results. Table 1. Survey Questionnaire. 1. What is the task of your search? a. Find detailed information about a subject of interest b. Obtain/download/ interact with a resource (e.g. game, song, greeting card) c. Find a specific URL/ web site d. Find local information (e.g. transportation timetable, movie showtimes) e. Look for a person (including myself and fiction characters) f. Find out about a product (e.g. price, features, retailers, discounts) g. Get a quick answer to my query (e.g. currency exchange rates, sport scores, stock quote) h. Look for linguistic information about my query terms (e.g. spelling, translation, definition) i. See if there is any new result retrieved since the last time I issued the query j. Re-find the information I got in one of my previous submissions of the query 2. Did you intend to visit some result(s) before issuing the query? Yes No Maybe 3. Did you actually visit some result(s)? Yes No 4. What was the reason for not visiting any result? a. I found what I was looking for in the result page b. Results seemed irrelevant c. I had already seen these results for the query d. Search was interrupted 3.3 Session Segmentation and Data De-identification Having collected our participants search traces and feedback, we firstly anonymized our data by replacing the user workstation IPs with random IDs and then we grouped the recorded queries and post-query activities into individual search sessions. To identify the distinct search sessions, we worked as follows. The first session is initiated the first time the participant issues a search request and records all her web transactions for as long as she demonstrates some search activity. We consider that a participant remains active in a search either when she views search results (i.e., she scrolls down the list of retrieved pages or moves to the next page of results), or when she accesses the retrieved documents (i.e. clicks on/opens them), or when she continues to submit queries. Under this approach, we deem that a session expires if both of the following criteria apply: (i) the period of user inactivity in a search gets longer than 10 minutes, i.e., the participant does not perform any of the above tasks within a time interval of 10 minutes, and (ii) the participant s activity that is recorded right after an idle time of 10 minutes is not a click on a retrieved result for the same query. Both criteria are suggested by [ 16]. Upon session expiration, a new session starts. After applying the above criteria to our collected search trace, we ended up with a total number of 966 queries that span to 239 search sessions. 4 Results 4.1 Query Statistics Having identified the individual search sessions of every participant and the set of queries issued in each session, the next step is to measure the fraction of queries for which

6 6 Sofia Stamou1, Efthimis N. Efthimiadis2 participants did not demonstrate any activity on the search results. In this respect, we relied on the recorded post-query user activity and we identified the queries that were not followed by result clicks. Table 2 summarizes the statistics of our experimental data. Table 2. Statistics on the experimental dataset Collection period 1 week # of unique user ID s 6 # of queries 966 # of sessions 239 # of queries without clickthrough 261 # of queries with clickthrough 705 # of clickthrough events 1,762 avg. # of sessions/user avg. # of queries/session 4.04 avg. # of clicks/query with clickthrough 2.49 According to our results, in 27.02% (261) of the queries participants did not click on any retrieved result, while in 72.98% (705) of the queries, participants visited at least one retrieved document. Having collected and processed the experimental data, we proceed with the analysis of the feedback our participants supplied for each of their reported queries. Such analysis will assist the identification of search goals that are associated with queries without clicks. Furthermore, it will help determine for which search tasks the user inactivity on search results is pre-determined, and for which tasks the user post-query inactivity is imposed by the displayed result snippets. The answers to the above issues, which we present and discuss next, will help model the user search behavior and develop user-centric retrieval evaluation models. 4.2 Classification of Queries by Search Task The distribution of the 966 queries by search tasks is given in Table 3 and generally conforms to the findings of previous work on what information seekers search for [ 11, 20, 25]. Table 3. Distribution of queries with and without clickthrough across search tasks. Search Goals Queries without Queries with clickthrough 261 clickthrough 705 Informational 2.29% 33.47% Transactional 3.84% 13.76% Navigational 3.06% 16.74% Local 21.45% 6.95% Person 8.05% 8.65% Product 6.51% 9.08% Language-related 13.03% 1.71% Update 9.97% 0.56% Repeat 4.21% 8.55% Quick answer 27.59% 0.44% Our findings indicate that for a significant fraction (i.e., 27.59%) of the searches not followed by clicks on the results, searchers aim at finding a quick answer in the result snippets, while for 33.47% of the searches that yield visits to the retrieved documents, searches aim at finding information in the contents of the retrieved documents. Next, we examine the intended user activity on search results. This will improve our understanding

7 Interpreting User Inactivity on Search Results 7 on whether the displayed results affected in any way the user decisions of clicking or not clicking on the retrieved results. 4.3 Pre-Query User Intentions across Search Tasks Based on the participants feedback, we grouped our test queries into three categories depending on whether participants issued a query with an initial intention to visit the search results, not to visit the results, or there was an unclear intention. In this respect, we relied on the answers our participants gave to question #2 and we classified the queries to the following categories: (i) searches of intended inactivity, i.e., queries that start with the intention to look for answers in the result snippets, (ii) searches of intended activity, i.e., queries that get started with the intention to look for answers in the retrieved documents, and (iii) searches with an unclear intention about where to look for answers. The results of the pre-query user intentions grouped by category are shown in Table 4. Table 4. Distribution of pre-query user intentions across search tasks. Search Goals Queries of intended inactivity Queries of intended activity Queries of unclear intention Total: 263 Total: 499 Total: 204 Informational 2.28% 39.08% 20.10% Transactional 1.52% 12.43% 20.10% Navigational 3.04% 17.64% 14.70% Local 19.39% 8.81% 4.91% Person 17.11% 3.81% 8.82% Product 6.47% 4.00% 21.57% Language-related 11.41% 2.00% 2.95% Update 9.51% 0.20% 1.96% Repeat 3.42% 11.83% 1.95% Quick answer 25.85% 0.20% 2.94% Of the 966 test queries, 499 (51.65%) were submitted with a pre-determined intention to look for answers in the contents of the retrieved documents. On the other hand, 263 (27.23%) of the queries were submitted with a pre-determined intention to seek for answers in the results list, whereas 204 (21.12%) of the queries have no pre-determined intention about where to look for the desired information. Moreover, Table 4 illustrates the distribution of our test queries across search tasks. According to our results most searches that seek the desired information in the retrieved result snippets pertain to the following search tasks: quick answer, local, person and language-related searches in that order. This practically indicates that users engaging in the above search tasks have most of the times a pre-determined intention to look for answers in the contents of the result lists and not to visit search results. On the other hand, most queries that look for information in the contents of the returned documents adhere to informational, navigational, transactional and repeat search tasks. This suggests that most of the time, users who engage in the above tasks intend to visit some of the retrieved results for satisfying their information needs. Finally, the majority of the searches for which users do not have a pre-determined intention about where to look for the desired information pertain to product, transactional and informational tasks. Results so far demonstrate that a considerable amount of searches are conducted with the intention to be satisfied in the contents of the result snippets. For those searches, we emphasize the need to evaluate retrieval effectiveness and user satisfaction from retrieval performance against the results the user views for a query. This is also shown in the work of [ 17, 20, 23], who

8 8 Sofia Stamou1, Efthimis N. Efthimiadis2 suggest that the lack of clickthrough on search results should not be interpreted as a sign of decreased user satisfaction from retrieval performance. 4.4 Post-Query User Activity across Search Tasks Another aspect of the human search behavior we examined is the correlation between the initial user intention and the demonstrated user activity on search results. This is in order to assess whether the user pre-query intentions are maintained in the post-query activity or rather they are influenced based on what the user sees in the list of search results. For our assessment, we cross-examined the answers our participants gave to questions #2 and #3. Note that question #2 captures the pre-query participant intentions, while question #3 captures the demonstrated post-query participant activity. Table 5 presents the fraction of queries across search tasks for which participants maintained initial intentions and the fraction of searches across tasks for which participants altered their initial intentions. Note that we have eliminated from our estimations queries of uncertain initial intentions (i.e. those for which our subjects selected the answer maybe to question #2). This is because for those queries participants lacked a specific initial intention about where to look for the desired information; therefore there is no evidence to judge whether their corresponding participant activity was intended or not. Under the above, the results reported in Table 5 concern the 762 queries submitted with a clear initial intention (i.e. 499 queries of intended activity and 263 queries of intended inactivity). Table 5. Distribution of queries for which participants maintained or altered their initial intentions across search tasks. Queries for which subjects maintained initial intention 557 Queries for which subjects altered initial intention Search Goals Distribution to search goals Distribution to search goals Informational 33.93% 5.86% Transactional 10.41% 3.90% Navigational 14.36% 7.81% Local 5.56% 31.22% Person 3.60% 21.46% Product 3.41% 8.78% Language-related 4.31% 7.80% Update 3.77% 2.44% Repeat 8.98% 8.78% Quick answer 11.67% 1.95% Results indicate that from the 762 queries in 73.10% (557) of the examined searches participants maintained their pre-query intentions, whereas in the remaining 26.90% (205) of the examined searches participants altered their pre-query intentions. A possible interpretation for the discrepancies between the participants pre-query intentions and postquery activity might be that the information displayed in the results list (e.g. result titles and/or snippets) influenced the participant decisions of visiting or not the search results. This might also apply to searches conducted under no pre-determined intention to visit or not search searches, which according to our findings represent 21.12% of our test queries. Given that the objective of our study is to understand the causes of user inactivity on search results, in the remainder of our analysis we concentrate on queries that were not followed by user clickthrough and investigate the impact of result snippets in satisfying their corresponding search goals. In this respect, we estimate the fraction of searches not 205

9 Interpreting User Inactivity on Search Results 9 followed by user activity on search results that were successfully addressed in the contents of the result snippets. 4.5 User Satisfaction from Searches without Clickthrough To capture the satisfaction of the participants goals pursued via searches characterized by intentional absence of clickthrough events, we relied on the answers our participants gave to question #4 (what was the reason for not visiting any result?) for their pre-determined inactive queries (i.e., the queries for which participants maintained their initial intention of not visiting search results). Table 6 reports the user perception from retrieval performance for searches that intentionally lacked user activity on the retrieved results. Results indicate that users engaging in informational, navigational, transactional and repeat searches did not satisfy their information needs in the contents of the result snippets. Therefore, for such types of searches, the absence of clickthrough on search results is a sign of user dissatisfaction from retrieval performance. On the other hand, language-related searches conducted with a pre-determined intention to look for answers in the result snippets can be successfully accomplished without the need to click through the results. Another interesting observation is that although for update searches people seek the desired information in the contents of the results list, only a small fraction of them (i.e., 28.57%) are successfully addressed in the result snippets. This might suggest that in order to better serve update queries search engines could maintain a profile for searchers or local cached copies that links their queries to viewed results. The displayed results from a new search could then display both previously seen and new results and highlight the already seen results for easier inspection by the searcher. Table 6. Post retrieval evaluation of results with respect to participants pre-determined decision to not click. Distribution to search goals Queries of pre-determined inactivity (with no clicks) % of satisfied goals Informational 0 Transactional 0 Navigational 0 Local % Person % Product % Language-related % Update % Repeat 0 Quick answer % Total: 145 Avg: 74.48% Overall, results indicate that 74.48% (i.e., 108 out of the 145) of the pre-determined inactive searches were successfully addressed in the contents of the displayed results. This further supports the argument that the lack of user activity on search results should not be interpreted as having only one negative meaning towards the results, but rather it should be validated in relation to the search tasks the user is trying to accomplish. Next, we investigate the post-determined user inactivity on search results in order to assess whether it has been implied by the contents of the displayed results.

10 10 Sofia Stamou1, Efthimis N. Efthimiadis2 4.6 Results Impact on Post-Determined User Inactivity across Search Tasks To assess how the displayed results affect participants inactivity in clickthrough, we rely on the searches our subjects initiated with the intention to visit results but did not and examine the reasons for the lack of result visitations. To interpret the post-determined user inactivity, we rely on the answers our participants gave to question #4 and we assert that the result snippets may have influenced the participant decisions to remain inactive in either a positive or negative way. Result snippets have a positive impact on user decisions to not click on the returned documents when the information sought is present in the contents of the results list. Conversely, result snippets have a negative impact on user clicking decisions when the displayed results are unsatisfactory to the user. To estimate the fraction of unintended searches without clickthrough that have been positively influenced by the displayed results, we relied on the queries submitted with the intention to visit results but ended up not doing so because according to our subjects they found what they were looking for in the result page (i.e., they selected answer a to question #4). On the other hand, to estimate the fraction of unintended searches without clickthrough that have been negatively influenced by the displayed results, we relied on the queries submitted with the intention to look for answers in the retrieved documents but did not yield document visits, because according to our subjects, results seemed irrelevant (i.e., they selected answer b to question #4). Table 7 reports the distribution of postdetermined inactive searches that have been positively and negatively influenced by the displayed results. Again, from our estimations, we have excluded queries lacking a specific initial intention about where to seek the desired information. Although, we speculate that our participants post-query behavior for searches of unclear intention may have been influenced in some way by the displayed result snippets, we have no perceptible evidence about the kind of impact (i.e., positive or negative) displayed results had on our subjects post-query activities. Table 7. Results list impact on the post-determined user inactivity on the retrieved documents. Post-determined inactive searches on which results has positive impact 43 Post-determined inactive searches on which results had negative impact Search Goals Distribution to search goals Distribution to search goals Informational 4.65% 9.09% Transactional 6.98% 2.27% Navigational 13.95% 4.55% Local 32.56% 40.91% Person 9.30% 11.36% Product 13.95% 6.82% Language-related 18.61% 0% Update 0% 2.27% Repeat 0% 20.46% Quick answer 0% 2.27% Our findings indicate that the search result snippets influence the post-query user activity in both positive and negative ways at equal rates. In particular, according to our results, 43 (49.42%) of the post-determined inactive searches represent queries for which users found the desired answers in the contents of the displayed results. Conversely, 44 (50.58%) of the post-determined inactive searches have been negatively influenced by the displayed results. This implies that there is plenty of room for further improvement before search engines can fully meet the user search goals in the information displayed on the search result lists. 44

11 5 Concluding Remarks Interpreting User Inactivity on Search Results 11 In this paper, we have studied the search tasks associated with searches lacking clickthrough events. Search abandonment can be positive or negative. We have shown that the lack in following a result is not necessarily a failed search (negative abandonment). Positive abandonment occurs when queries without clickthroughs are associated with specific goals that can be achieved in the contents of the displayed results. The main findings of our study are summarized as follows. Nearly 27% of the queries were submitted with a pre-determined intention to look for answers in the contents of the result snippets. Users engaging in intentionally inactive searches are most of the time interested in obtaining a quick answer to their queries. Nearly half of the searches that do not yield result visits manage to find the desired information in the search results list. Finally, we have shown that approximately 50% of the searches not followed by user visits on the results, although they intended to, have been negatively influenced by the information displayed on the results page in the sense that results seemed irrelevant and therefore did not trigger user visits to the retrieved data. One implication of our study is that we should not uniformly consider queries without clickthrough as failed searches but rather we should evaluate their effectiveness in relation to their underlying search tasks. Moreover, there is room for further improvement with respect to the search engines potential in serving different user goals. This is especially true for update searches in which users want to find out if there is new information from a previous submission of their query. Although the above conclusions derive from a small-scale user study in which competent searches (i.e., computer science students) were involved, a closer look at our test queries reveals that these pertain to a multitude of topics that may be of interest to the entire web population. Of course, we need to repeat our study for distinct user groups before we can generalize our findings, but still we believe that the work presented here is the first step in this direction. For future work we plan to examine additional features in the users search behavior, such as the time users spend viewing the displayed results for their pre-determined and post-determined searches and the distribution of queries without clickthrough in a search session in order to build models that can interpret the user search activity and inactivity on retrieved results. Another area for future research is how to explore the findings of our study in order to improve the search engine s effectiveness on serving diverse user needs. The development of predictive models of user activity could be used to implement search aids that would assist users in completing their search tasks successfully. Finally, we hope that this study would contribute towards the design of retrieval evaluation frameworks where search is seen holistically and incorporate multiple features for measuring user satisfaction from retrieval performance. 6 References 1. Agichtein, E., Brill, E., Dumais, S., Rango, R.: Learning user interaction models for predicting search result preferences. In Proceedings of the 29 th ACM SIGIR Conference (2006) 2. Belkin, N.: Some(what) grand challenges for information retrieval. In ACM SIGIR Forum, 42 (1): (2008) 3. Broder, A.: A taxonomy of web search. SIGIR Forum 36, 2 (Sep. 2002), 3-10 (2002) 4. Claypool, M. Le, P., Waseda, M., Brown, D.: Implicit interest indicators. In Proceedings of the International Conference on Intelligent User Interfaces, pp (2001) 5. Callan, J., Allan, J., Clarke, Ch.L.A., Dumais, S., Evans, D.A., Sanderson, M., Zhai, Ch.: Meeting of the MINDS: an information retrieval research agenda. In ACM SIGIR Forum, 41(2): (2007)

12 12 Sofia Stamou1, Efthimis N. Efthimiadis2 6. Cutrell, E., Guan, Z.: What are you looking for? an eye-tracking study of information usage in web search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp (2007) 7. Fox, S., Karnawat, K., Mydland, M., Dumais, S.,White, T.: Evaluating implicit measures to improve web search. ACM Transactions on Information Systems, 23(2): (2005) 8. Granka, L.A., Joachims, T., Gay, G.: Eye-tracking analysis of user behaviour in www results. In Proceedings of the ACM SIGIR Conference, pp (2004) 9. Huang, J., Efthimiadis, E. N.: Analyzing and Evaluating Query Reformulation Strategies in Web Search Logs. In Proceedings of the 18th ACM Conference on Information and Knowledge Management (CIKM), Hong Kong, November 2-6, 2009, pp (2009). 10. Jansen, B.J., Spink, A.: How are we searching the www: a comparison of nine search engine transaction logs. Information Processing & Management 42(1): (2006) 11. Jansen, B.J., Booth, D.L., Spink, A.: Determining the informational, navigational and transactional intent of web queries. Information Processing & Management 44: (2008) 12. Joachims, T., Granka L., Pan, B., Hembrooke H., Padlinski, F., Gay, G.: Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Transactions on Information Systems, 25(2):1-26 (2007) 13. Joachims, T., Radlinski, F.: Search engines that learn from implicit feedback. Computer, vol.40, pp (2007) 14. Jung, S., Herlocker, J.L., Webster, J.: Click data as implicit relevance feedback in web search. Information Processing & Management, 43(3): (2007) 15. Kelly, D., Teevan, J.: Implicit feedback for inferring user preference: a bibliography. In ACM SIGIR Forum, 37(2):18-28 (2003) 16. Qiu, F., Liu, Z., Cho, J.: Analysis of user web traffic with a focus on search activities. In Proceedings of the International Workshop on the Web and Databases (WebDB) (2005) 17. Radlinski, F., Kurup, M., Joachims, T.: How does clickthrough data reflect retrieval quality. In Proceedings of the CIKM Conference (2008) 18. Rose, D. E., Levinson, D.: Understanding user goals in web search. In Proceedings of the 13th International Conference on World Wide Web (WWW '04). ACM Press, New York, NY, (2004) 19. Sarma, A., Gollapudi, S., Ieong S.: Bypass rates: reducing query abandonment using negative inferences. In Proceedings of the 14 th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp (2008) 20. Scott, J.L., Huffman, B., Tokuda, A.: Good abandonment in mobile and pc internet search. In Proceedings of the 32 nd Annual ACM SIGIR Conference, Boston, MA, USA, pp (2009) 21. Sharma, H., Jansen, B.J.: Automated evaluation of search engine performance via implicit user feedback. In Proceedings of the 28 th ACM SIGIR Conference, pp (2005) 22. Spink, A.: A user centered approach to evaluating human interaction with web search engines: an exploratory study. Information Processing & Management, 38(3): (2002) 23. Stamou, S., Efthimiadis, E.N.: Queries without clicks: successful or failed searches? In Proceedings of the SIGIR Workshop on the Future of Information Retrieval Evaluation, Boston, MA, USA (2009) 24. Taksa, I., Spink, A., Goldberg, R.: A task-oriented approach to search engine usability studies. Journal of Software, 3(1): (2008) 25. Teevan, J., Adar, E., Jones, R., Potts, M.: Information re-retrieval: repeat queries in Yahoo s logs. In Proceedings of the 30 th ACM SIGIR Conference (2007)