Dissertation Research Proposal

Size: px

Start display at page:

Download "Dissertation Research Proposal"

Jemimah Oliver
5 years ago
Views:

1 Dissertation Research Proposal ISI Information Science Doctoral Dissertation Proposal Scholarship (a) Description for the research, including significance and methodology (10 pages or less, double-spaced) Please see Page

2 Evaluating Sources of Implicit Feedback in Web Searches INTRODUCTION AND BACKGROUND The search engine is becoming increasingly important as a tool to acquire information and enable self directed learning. Most of the search engines nowadays rely on searchers to represent their information needs as queries; however, it has been well documented that searchers often have a difficult time articulating their information needs and formulating effective search queries ([1], [3]). Despite the demonstrated success of many techniques to address this problem (e.g., relevance feedback for query expansion, query formulation suggest), there is an under-explored approach to further improving search engine performance and user experience, which is to capture and exploit searchers interactions with search engines. Current search engines prepare results only based on submitted queries. Even when relevance feedback techniques are used, modification of results does not happen until searchers provide explicit judgments on feedback terms. Although people examine results, and sometimes even navigate across pages of results, the information that they can see has been determined at the time of initial querying; their interactions with the search system in the result examination process are completely ignored. This is apparently a wasted opportunity for search engines to be more responsive and helpful. With IR systems becoming more interactive, the system should also actively learn from such interactions and use it to improve retrieval. If search engines can consider the initial query as the explicit representation of the information needs, and in the meantime, consider additional behavior that searchers exhibit as implicit indications of their interests, they should then try to capture searchers interactions with search engines and leverage these interactions to provide query modification suggestions and other search tips. This implicit 2

3 approach to identifying searchers interests removes the cost and the cognitive interruption to the user of providing feedback ([8], [11]). The key challenge for this approach is to find a set of evidences that (1) can possibly be captured in a natural search setting, and (2) can reliably indicate users interests and reflect their information needs. There has been some work in each aspect, but the results are not conclusive yet (c.f., [2]). The emphasis of this dissertation is to formally study the range of evidences that searcher behavior offers and understand how each kind of evidence can be useful and in what way. It differs from previous work in several ways. Firstly, previous work in this vein has been largely focused on click streams as evidence to tune search results (e.g., [5], [14]). In this dissertation, a wider variety of evidences and a wider range of granularity to support feedback and modification will be examined. In specific, the following types of behavioral evidences will be examined: view (time, number of revisits, pattern of eye movement and mouse movement, exit type), scroll (time, speed, amount, distance), mouse-over a link, click (i.e., select a link), search within page, and query modification. For view and scroll, items in parenthesis are attributes of the behaviors that may be used as measures of the searcher s interests. Secondly, all but one studies found so far study observable searcher behaviors as implicit interest indicators by somehow comparing them against explicit ratings. This approach is based on the assumption that explicit ratings give more accurate information on what a searcher finds interesting and useful. If a behavioral measure is found to correlate well with the explicit ratings, it can potentially be used in lieu of or in conjunction with the explicit feedback. The only exception is [12], using job application on a job search website as the indicator of searcher interest, but it is rare that such behavioral indicators are 3

4 available in the general Web search context. Unlike previous studies, this dissertation does not focus on whether a particular behavioral measure or combination of them correlates well with explicit measures of searchers interests; instead, it seeks to gain better understanding of the process of inferring searcher interests from behaviors. Assuming a range of behaviors is observed by a human intermediary (such as a reference librarian or a search expert), which behavior(s) will she consider as evidences of interests? Does she use a single behavior or a set of behaviors to make the inference? When does she make the inference? Why does she believe that a certain behavior is useful? Are there any rules that are commonly used? It is hoped that the answers to these question do not only provide more evidences for the usefulness of behaviors as implicit feedback measures in the context of Web search, but also will advance the understanding of why and how each type of evidence is useful. Such an understanding forms a foundation for improving search engine algorithms that exploit implicit feedback to deliver better results and create better user experience. Thirdly, a gap in the literature is that research on implicit feedback has paid little or no attention to task ([7]). Most previous studies examine behavior in an online reading environment (e.g., reading Usenet news or academic articles), but few have focused on examining the behavior during a Web search environment. This dissertation tries to fill in this gap. Web search has its distinctive features. For example, previous studies (e.g., [4]) have found that when examining search results list page searchers do not read the list linearly from top to bottom, as in other reading scenarios. In terms of observing the searcher s behavior as evidence of interests, we should at least pay attention to whether a piece of evidence is collected from behaviors on the result list page, or those on external content pages. The results list page is more uniformly 4

5 structured, so findings from this page are more easily generalized across search engines. If behaviors on external content pages are found to be more useful, then future studies will be needed to further study the features of those pages and see if the set of useful behaviors has any correlation with the feature set. Most previous studies focus on the search results list (e.g., [5], [14]), while [6] argues that it is important to collect data beyond the search results list and consider all pages visited in the entire search session. Another implication of focusing on the Web search context is that all the evidence that will be considered in the study can be automatically captured in real time in a natural setting so that the techniques arising from the study are self-contained and readily deployable in any search engines. From this perspective, the study is practice oriented and promises highly practical impact. In specific, the following research questions will be examined: Which type(s) of searcher behavior is useful evidence of the searcher s interests? How does the quality of inference on the searcher s interest evolve with more evidence available? Does more evidence lead to more reliable inferences? Does a single behavior indicate interest, or is it necessary to capture a set of behaviors? Does the genre of the page affect the behaviors that can be captured and used? In particular, are the behaviors on the search results list page more useful than those on the result content pages? Finally, why is a certain behavior useful? What are the rules to make the inference? STUDY DESIGN First phase of data collection The goal of the first phase of the study is to collect recordings of searchers behaviors during Web search 5

6 activities and create a corpus of search cases in which the searcher experiences difficulties due to underspecification problems. Different version of these recordings, showing different aspects of searchers behaviors will serve as the input for the second phase of data collection. In order to increase the chance of obtaining such search cases, efforts will be made to recruit searchers who are more likely to experience difficulties in formulating queries, including inexperienced search engine users ([9], [10]), and people who search in a new field or have complicated information needs involving multiple aspects ([1], [13]). About subjects will be recruited from UNC staff. Recruitment advertisement will be sent out on a mailing list and people interested in participating in the study will fill out a screening questionnaire, which includes demographic questions, questions on computer use and Web search experiences, as well as a few query formulation exercises. Respondents who report less search experiences and formulate underspecified queries will be selected to participate in the study. Selected participants will be notified via and scheduled for individual study sessions, which last for about 1 hour. Each participant will search on 6 tasks, all of which will involve multiple facets. To make them comparable, preferences are given to more close ended (fact finding or known item) search problems which involve about 3 facets. While the participant searches, backend logging will be performed to record the URL of each page visited as well as the time of visit. All screen activities will also be recorded into a video file to capture mouse movements, and scrolling. Finally, eye tracking will be performed to record searchers eye positions on the screen. At the end of each task, a brief semi-structured interview will be conducted to ask the participant to reflect on the search process, focusing on two questions. First, does she think her initial query clearly stated what she wanted? Second, did she learn something in the search process which made 6

7 her change her search strategy? If so, what are some of the critical instances which triggered the change? These reflections on critical instances during the search process will be compared to the analysts findings in the second phase. Second phase of data collection In this phase, search experts and information professionals (referred to as search analysts) will be recruited to examine selected recordings from the corpus and infer the interests of the searchers based on different subsets of the evidence. The focus is to capture the process in which analysts make inferences on the searcher s interests based on observed behaviors. Twelve searches cases will first be selected from those collected in the first phase. Preference will be given to cases in which the initial query underspecified the information need (missing some of the facets), but then the searcher went through multiple rounds of query modification (by adding or changing query terms) and/or browsed through many result pages before she finally successfully found the information. Searchers reflections on the search processes will also aid the selection by confirming that they indeed felt that initial queries were underspecified. Four versions of each search selected case will then be prepared as stimuli for the second phase of the study, showing different aspects of the search behavior, as summarized in Table 1. Table 1. Different versions of stimuli Version Format Which pages What information A PPT Slides Search results pages and pages 1 step from them Queries, URL, screen shots B PPT Slides A + time spent on page C Video All pages visited during the session Screen content and activities D Video C + gaze path Version A mimics the click stream data that is typically captured in the server-side log of a search engine. Version B takes into account the browsing activities in the entire session, as well as the time spent on each 7

8 page. Version C captures all the information available in Versions A and B, plus scrolling, mouse movements and other behaviors within pages. The temporal nature of video recordings should also make the time spent on each page more salient to the analysts. Finally, Version D adds the gaze path to Version C. Participants in this phase of the study are 12 search analysts who examine the recordings and infer searchers interests based on their behaviors (analysts do not know the original search problems). The expertise that is crucial to complete the task consists of experiences with observing people s Web search behaviors and knowledge on helping people improve their search strategies. People with high levels of expertise in those areas, such as reference librarians and search intermediaries, will be recruited. All sessions are conducted individually and are recorded, in which analysts will be asked to think aloud. The first group of 8 analysts will work on two cases of each version (8 in total) in a counterbalanced design. The second group of 4 participants will work on four cases of Version D (which contain all types of evidence), making the length of all sessions roughly equal (2 hours). All participants will control the speed of the replay. They will be instructed to pause the replay whenever they notice a new piece of evidence which contributes to their inference. At that point, the researcher will ask the following questions: Did you learn anything about the searcher s interest? If so, what is it? How did you learn it? What evidence(s) was it based on? What would you say about the searcher s interest now? Please summarize in a sentence. How confident are you with this guess on a 7 point scale (7 being most confident)? If you were to help the searcher find the information, are there any questions in your mind that you want to ask him or her to clarify at this point? 8

9 The researcher will force the replay to be paused if the analyst remains silent for a long time or when a major event happens in the recording. A major event is defined as submitting a query or visiting the search result list page. When the analyst finishes reading/watching a representation, she will be asked to summarize her inference and form a statement of the searcher s information need based on her best guess. She will also be given the chance to discuss any evidence that she has found and compare the usefulness of different types of behavior. Data analysis plan Content analysis will be performed on the transcribed recordings of the search analysts think aloud protocols and responses to the interview questions asked at pauses of the replay and at the end of each task. Classificatory content analysis will be based on existing classifications of the behavioral sources of implicit feedback, as discussed in [8]. The unit of observation is an attempt of inference. Each attempt of inference will be coded by the behavior or behaviors that are used, when it is used, where it is used (on search results list page or on external result content page ), and how it is used (e.g., the absence/presence of the behavior, the frequency of the behavior, the velocity of the behavior, or the duration of the behavior). Each inference will also be compared to the original search problem and scored based on how well it reflects the searcher s real interests. Analyses will be conducted at three different levels to inform the research questions. Firstly, analysis will be conducted on the search case level and can shed light on the following research questions: How much evidence is necessary before an inference on the searcher s interest can be made? Does more evidence lead 9

10 to more reliable inferences? For both of these questions, it will be most interesting to identify the critical points when the inference is significantly improved. Secondly, the unit of analysis can be the type of behavior. This kind of analysis can be used to inform the following research questions. Which type(s) of searcher behavior is useful evidence of the searcher s interests? Are the behaviors on the search results list page more useful than those on the result content pages? How is each behavior used? What are the rules to make the inference? Different types of behaviors can also be compared on how well they lead to good inferences. Thirdly, comparisons can also be made at the representation type level. Inferences based on 4 versions of search representations can be compared in terms of their effectiveness. The potential difference between Version A and B reflects the value of monitoring the additional browsing paths beyond the search results list page as well as keeping track of the time spent on each page. The difference between Version B and C attests to the usefulness of capturing searchers behavior within a page (such as scrolling and mouse movement), in addition to recording page level information, such as URLs. The difference between Version C and D will show the added value of eye tracking. SIGNIFICANCE The short-term, more theoretical motivation of this dissertation is to advance the understanding of the relationship between the types of behavior that can be captured and searchers interests. The long-term, more practical goal will be to develop techniques that can capture searchers actions in real time and provide query recommendation immediately. The techniques are self-contained and readily deployable in any search engine. Based on the analysis, the dissertation is expected to conclude with some common rules 10

11 that are used by the analysts and put forward design recommendations or algorithms that can be applied in automatic systems. REFERENCES [1] Belkin, N. (1980). Anomalous states of knowledge as a basis for information retrieval. Canadian Journal of Information Science, 5, [2] Fox, S., Karnawat, K., Mydland, M., Dumais, S., & White, T. (2005). Evaluating implicit measures to improve web search. ACM Transactions on Information Systems, 23(2), [3] Jansen, B. J., Spink, A., & Saracevic, T. (2000). Real life, real users, and real needs: a study and analysis of user queries on the web. Information Processing & Management, 36, [4] Joachims, T. (2002). Optimizing search engines using clickthrough data. In Proceedings of SIGKDD 02, [5] Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2005). Accurately Interpreting Clickthrough Data as Implicit Feedback. In Proceedings of SIGIR 2005, [6] Jung, S., Herlocker, J. L., & Webster, J. (2007). Click data as implicit relevance feedback in web search. Information Processing & Management, 43(3), [7] Kelly, D., & Belkin, N. J. (2004). Display time as implicit feedback: understanding task effects. In Proceedings of SIGIR 04, [8] Kelly, D. & Teevan, J. (2003). Implicit feedback for inferring user preference: A bibliography. SIGIR Forum, 37(2), [9] Lazonder, A. W., Biemans, H. J. A., & Wopereis, I. G. J. (2000). Differences between novice and experiences users in searching information on the World Wide Web. JASIST, 51(6), [10] Lucas, W., & Topi, H. (2002). Form and function: The impact of query term and operator usage on web search results. JASIST, 53(2), [11] Oard, D. W., & Kim, J. (2001). Modeling information content using observable behavior. In Proceedings of the 64th ASIST Annual Meeting, [12] Rafter, R., & Smyth, B. (2001). Passive profiling from server logs in an online recruitment environment. In Proceedings of ITWP 2001, [13] Schaefer A., Jordan M., Klas C., & Fuhr N. (2005). Active Support For Query Formulation in Virtual Digital Libraries: A case study with DAFFODIL. In ECDL 2005, [14] Shen, X., Tan, B., & Zhai, C. (2005). Implicit user modeling for personalized search. In Proceedings of CIKM 2005,

12 (b) Schedule of completion I am a full time doctoral student at UNC School of Information and Library Science, ranked the 1 st in the U.S.. I entered the program in Fall 2003 and expect to graduate in Spring I have completed all the course work and passed the dissertation proposal defense. The dissertation research is currently underway. I am running pilot tests this summer and will start data collection this fall. I expect to complete the research and defend the dissertation by Spring (c) Budget and budget justification for items for which financial support is sought (must be items for which no other support is available; examples of acceptable budget items are printing, computer time, fees to subjects, keypunching, statistical consulting, photography, art work, typing, and professional travel) The main cost of this study consists of two parts. The first part is the eye tracker rental (about $3000 for one month rental fee). The second part is for subject compensation. The subjects in the first phase of the study will be compensated at $10 each. Additionally, all people who filled out the screening questionnaire will enter a drawing for a $25 Best Buy gift card. The 12 subjects in the second phase of the study will be paid at $25 each. So, the total expenses on subject compensation add up to $625. (d) Other support (including scholarships, assistantships, and employment) N/A (e) Name of the dissertation advisor endorsing the proposal Dr. Gary Marchionini 12

Interpreting User Inactivity on Search Results

Interpreting User Inactivity on Search Results Sofia Stamou 1, Efthimis N. Efthimiadis 2 1 Computer Engineering and Informatics Department, Patras University Patras, 26500 GREECE, and Department of Archives