Inferring Searcher Intent

Transcription

1 , 11 July 2010 Inferring Searcher Intent Tutorial Website (for expanded and updated bibliography): Instructor contact information: Web: Tutorial Overview Part 1: Search Intent Modeling Motivation: how intent inference could help search Search intent & information seeking behavior in traditional IR Searcher models: from eye tracking to clickthrough mining Part 2: Inferring Web Searcher Intent Inferring result relevance: clicks Richer interaction models: clicks + browsing Part 3: Applications and Extensions Implicit feedback for ranking Contextualized prediction: session modeling Personalization, query suggestion, active learning 2 AAAI 2010 Tutorial: Inferring Searcher Intent 1

2 , 11 July 2010 About the Instructor (Ah-ghi-sh-tein) Research: Information retrieval and data mining Mining search behavior and interactions in web search Text mining, information extraction, and question answering Relevant experience: Assistant Professor, Summer 07: Visiting Researcher, Yahoo! Research : Postdoc, Microsoft Research : PhD student, Columbia Databases/IR 3 Outline: Search Intent and Behavior Motivation: how intent inference could help search Search intent and information seeking behavior Classical models of information seeking Web searcher intent Web searcher behavior Levels of modeling: micro-, meso-, and macro- levels Variations in web searcher behavior Click models 4 AAAI 2010 Tutorial: Inferring Searcher Intent 2

3 , 11 July 2010 Some Key Challenges for Web Search Query interpretation (infer intent) Ranking (high dimensionality) Evaluation (system improvement) Result presentation (information visualization) 5 Example: Task-Goal-Search Model car safety ratings consumer reports 6 AAAI 2010 Tutorial: Inferring Searcher Intent 3

4 , 11 July 2010 Information Retrieval Process Overview Credit: Jimmy Lin, Doug Oard, Source Selection Resource Search Engine Result Page (SERP) Query Formulation Query: car safety ratings Search Ranked List query reformulation, vocabulary learning, relevance feedback Selection Documents Examination Documents source reselection Delivery 7 Explicit Intentions in Query Logs Strohmaier et al., K-Cap 2009 Match known goals (from ConceptNet) to query logs 8 AAAI 2010 Tutorial: Inferring Searcher Intent 4

5 , 11 July 2010 Unfortunately, most queries are not so explicit 9 Outline: Search Intent and Behavior Motivation: how intent inference could help search Search intent and information seeking behavior Classical models of information seeking Web Searcher Intent Broder Rose More recent? Web Searcher Behavior Levels of modeling: micro-, meso-, and macro- levels Variations in web searcher behavior Click models Challenges and open questions 10 AAAI 2010 Tutorial: Inferring Searcher Intent 5

6 , 11 July 2010 Information Seeking Funnel Wandering:the user does not have an information seeking-goal in mind. May have a meta-goal (e.g. find a topic for my final paper. ) Exploring:the user has a general goal (e.g. learn about the history of communication technology ) but not a plan for how to achieve it. Seeking: the user has started to identify information needs that must be satisfied (e.g. find out about the role of the telegraph in communication. ), but the needs are openended. Asking:the user has a very specific information need that corresponds to a closed-class question ( when was the telegraph invented? ). D. Rose, Models of Information Seeking Information-seeking includes recognizing the information problem, establishing a plan of search, conducting the search, evaluating the results, and iterating through the process. - Marchionini, 1989 Query formulation Action (query) Review results Refine query Adapted from: M. Hearst, SUI, AAAI 2010 Tutorial: Inferring Searcher Intent 6

7 , 11 July 2010 Reviewing Results: Relevance Clues What makes information or information objects relevant? What do people look for in order to infer relevance? Topicality (subject relevance) Extrinsic (task-, goal- specific) Information Science clues research : uncover and classify attributes or criteria used for making relevance inferences 13 Information Scent for Navigation Examine clues where to find useful information Search results listings must provide the user with clues about which results to click 14 AAAI 2010 Tutorial: Inferring Searcher Intent 7

8 , 11 July 2010 Dynamic Berry Picking Model Information needs change during interactions [Bates, 1989] M.J. Bates. The design of browsing and berrypicking techniques for the online search interface. Online Review, 13(5): , Information Foraging Theory Goal: maximize rate of information gain. Patches of information websites Basic Problem: should I continue in the current patch or look for another patch? Expected gain from continuing in currentpatch, how long to continue searching in that patch Pirolli and Card, CHI 1995 Emory University 16 AAAI 2010 Tutorial: Inferring Searcher Intent 8

9 , 11 July Charnov s Marginal Value Theorem Diminishing returns: 80% of users scan only first 3 pages of search results Emory University 17 Hotel Search Goal: Find cheapest 4-star hotel in Paris. Step 1: pick hotel search site Step 2: scan list Step 3: goto1 18 AAAI 2010 Tutorial: Inferring Searcher Intent 9

10 , 11 July 2010 Example: Hotel Search (cont d) 19 Orienteering vs. Teleporting Teevan et al., CHI 2004 Orienteering: Searcher issues a quick, imprecise to get to approximately the right information space region Searchers follow known paths that require small steps that move them closer to their goal Easy (does not require to generate a perfect query) Teleporting: Issue (longer) query to jump directly to the target Expert searchers issue longer queries Requires more effort and experience. Until recently, was the dominant IR model 20 AAAI 2010 Tutorial: Inferring Searcher Intent 10

11 , 11 July 2010 Serendipity Andre et al., CHI Summary of Models Static, berry-picking, information foraging, orienteering, serendipity Classical IR Systems research mainly uses the simplest form of relevance (topicality) Open questions: How people recognize other kinds of relevance How to incorporating other forms of relevance (e.g., user goals/needs/tasks) into IR systems 22 AAAI 2010 Tutorial: Inferring Searcher Intent 11

12 , 11 July 2010 Part 1: Search Intent and Behavior Motivation: how intent inference could help search Search intent and information seeking behavior Classical models of information seeking Web Searcher Intent Broder Rose More recent? Web Searcher Behavior Levels of modeling: micro-, meso-, and macro- levels Variations in web searcher behavior Click models 23 Intent Classes (top level only) User intent taxonomy (Broder2002) Informational want to learnabout something (~40% / 65%) Navigational want togo to that page (~25% / 15%) Transactional want to do something(web-mediated) (~35% / 20%) Access a servicedownloads Shop Gray areas Find a good hub Exploratory search see what s there [from SIGIR 2008 Tutorial, Baeza-Yates and Jones] History nonya food Singapore Airlines Jakarta weather Kalimantan satellite images Nikon Finepix Car rental Kuala Lumpur 24 AAAI 2010 Tutorial: Inferring Searcher Intent 12

13 , 11 July 2010 Extended User Goal Taxonomy Rose et al., Complex Search Aula and Russel, 2008 A complex search task refers to cases where: searcher needs to conduct multiple searches to locate the information sources needed, completing the search task spans over multiple sessions (task is interrupted by other things), searcher needs to consult multiple sources of information (all the information is not available from one source, e.g., a book, a webpage, a friend), requires a combinationof exploration and more directed information finding activities, often requires note-taking (cannot hold all the information that is needed to satisfy the final goal in memory), and specificity of the goal tends to vary during the search process (often starts with exploration). 26 AAAI 2010 Tutorial: Inferring Searcher Intent 13

14 , 11 July 2010 Complex Search (Cont d) Aula and Russel, Web Search Queries Cultural and educational diversity Short queries and impatient interaction Few queries posed and few answers seen (first page) Reformulation common Smaller and different vocabulary Not expert searchers! Which box do I type in? 28 AAAI 2010 Tutorial: Inferring Searcher Intent 14

15 , 11 July 2010 Intent Distribution by Topic [from SIGIR 2008 Tutorial, Baeza-Yates and Jones] 29 Query Distribution by Demographics Education: [Weber & Castillo, SIGIR 2010] Ethnicity: Gender: 30 AAAI 2010 Tutorial: Inferring Searcher Intent 15

16 , 11 July 2010 Query Demographics 2: Demo Keywords have demographic signatures Microsoft adcenter Demographics Prediction: Quantcast adcenter[posters] 31 Domain-Specific Intents: Named Entities Named Entities (Persons, Orgs, Places) are often searched Brittany Spears? [Yin & Shah, WWW 2010] Popular phrases by entity type: 32 AAAI 2010 Tutorial: Inferring Searcher Intent 16

17 , 11 July 2010 Example Intent Taxonomy for Musicians Musicians(most popular phrases) [Yin & Shah, WWW 2010] 33 Analyzing Searches: Funneling What is the intent of customers that type such queries? Hint: What they searched before/after? Search Funnels: How can you catch customers earlier? What customers do when they leave? 34 AAAI 2010 Tutorial: Inferring Searcher Intent 17

18 , 11 July 2010 Part 1: Search Intent and Behavior Motivation: how intent inference could help search Search intent and information seeking behavior Classical models of information seeking Web Searcher Intent Broder Rose Demographics Web Searcher Behavior Levels of modeling: micro-, meso-, and macro- levels Variations in web searcher behavior Click models 35 Web Searcher Behavior Meso-level: query, intent, and session characteristics Micro-level: how searchers interact with result pages Macro-level: patterns, trends, and interests 36 AAAI 2010 Tutorial: Inferring Searcher Intent 18

19 , 11 July 2010 Levels of Understanding Searcher Behavior Micro (eye tracking): lowest level of detail, milliseconds [Daniel M. Russell, 2007] Meso(field studies): mid-level, minutes to days Macro (session analysis): millions of observations, days to months 37 Search Behavior: Scales from: Pirolli, AAAI 2010 Tutorial: Inferring Searcher Intent 19

20 , 11 July 2010 Information Retrieval Process (User view) Source Selection Resource Query Formulation Query Search Ranked List query reformulation, vocabulary learning, relevance feedback Selection Documents Examination Documents source reselection Delivery 39 People Look at Only a Few Results (Source: iprospect.comwhitepaper_2006_searchengineuserbehavior.pdf) 40 AAAI 2010 Tutorial: Inferring Searcher Intent 20

21 , 11 July 2010 Snippet Views Depend on Rank [Daniel M. Russell, 2007] Mean: 3.07 Median: Snippet Views and Clicks Depend on Rank [from Joachims et al, SIGIR 2005] 42 AAAI 2010 Tutorial: Inferring Searcher Intent 21

22 , 11 July 2010 Eyes are a Window to the Soul Eye tracking gives information about search interests: Camera Eye position Pupil diameter Seekads and fixations Reading Visual Search 43 Micro-level: Examining Results Users rapidly scan the search result page [Daniel M. Russell, 2007] What they see in lower summaries may influence judgment of higher result Spend most time scrutinizing top results 1 and 2 Trust the ranking 44 AAAI 2010 Tutorial: Inferring Searcher Intent 22

23 , 11 July 2010 POM: Partially Observable Model Wang et al., WSDM Result Examination (cont d) Searchers might use the mouse to focus reading attention, bookmark promising results, or not at all. Behavior varies with task difficulty and user expertise [K. Rodden, X. Fu, A. Aula, and I. Spiro, Eye-mouse coordination patterns on web search results pages, Extended Abstracts of ACM CHI 2008] 46 AAAI 2010 Tutorial: Inferring Searcher Intent 23

24 , 11 July 2010 Result Ex. (cont): Predicting Eye-Mouse coordination Guo & Agichtein, CHI 2010 Actual Eye-Mouse Coordination Predicted No Coordination (30%) Eye mouse distance 750 Euclidean Distance Threshold 600 Prediction Time x 10 Bookmarking (25%) Eye mouse distance Euclidean Distance Prediction Threshold Time Eye follows mouse (25%) Eye mosue distance Euclidean Distance Prediction Threshold Time 47 Macro-Level (Session) Analysis Can examine theoretical user models in light of empirical data: Orienteering? Foraging? Multi-tasking? Search is often a multi-step process: Find or navigate to a good site ( orienteering ) Browse for the answer there: [actor most oscars] vs. [oscars] Teleporting I wouldn t use Google for this, I would just go to Triangulation Draw information from multiple sources and interpolate Example: how long can you last without food? 48 AAAI 2010 Tutorial: Inferring Searcher Intent 24

25 , 11 July 2010 Users (sometimes) Multi-task [Daniel M. Russell, 2007] 49 Kinds of Search+Browsing Behavior [Daniel M. Russell, 2007] 50 AAAI 2010 Tutorial: Inferring Searcher Intent 25

26 , 11 July 2010 Parallel Browsing Behavior [Huang & White, HT 2010] 57% of all tabbed sessions are parallel browsing Can mean multitasking Common scenario: branching in exploring search results 51 Search Engine Switching Behavior White et al., CIKM % of all search sessions contained a switching event Switching events: 58.6 million switching events in 6-month period 1.4% of all Google / Yahoo! / Live queries followed by switch 12.6% of all switching events involved same query Two-thirds of switching events from browser search box Users: 72.6% of users used multiple engines in 6-month period 50% of users switched search engine within a session 52 AAAI 2010 Tutorial: Inferring Searcher Intent 26

27 , 11 July 2010 Overview of Search Engine Switching Switching is more frequent in longer sessions White et al., CIKM Overview of Switching - Survey White et al., CIKM % of survey respondents reported having switched Remarkably similar to the 72.6% observed in logs Those who did not switch: Were satisfied with current engine (57.8%) Believed no other engine would perform better (24.0%) Felt that it was too much effort to switch (6.8%) Other reasons included brand loyalty, trust, privacy Within-session switching: 24.4% of switching users did so Often or Always 66.8% of switching users did so Sometimes 54 AAAI 2010 Tutorial: Inferring Searcher Intent 27

28 , 11 July 2010 Reasons for Engine Switching White et al., CIKM 2009 Other reasons included: - Loyalty to dest. engine - Multi-engine apps. -Hope (!) Three types of reasons: Dissatisfaction with original engine Desire to verify or find additional information User preference 55 Pre-switch Behavior White et al., CIKM 2009 Most common are queries and non-serp clicks This is the action immediately before the switch What about pre-switch activity across the session? 56 AAAI 2010 Tutorial: Inferring Searcher Intent 28

29 , 11 July 2010 Pre-switch Behavior (Survey) White et al., CIKM 2009 Is there anything about your search behavior immediately preceding a switch that may indicate to an observer that you are about to switch engines? Common answers: Try several small query changes in pretty quick succession Go to more than the first page of results, again often in quick succession and often without clicks Go back and forth from SERP to individual results, without spending much time on any Click on lots of links, then switch engine for additional info Do not immediately click on something 57 Post-switch Behavior White et al., CIKM 2009 Extending the analysis beyond next action: 20% of switches eventually lead to return to origin engine 6% of switches eventually lead to use of third engine > 50% led to a result click. Are users satisfied? 58 AAAI 2010 Tutorial: Inferring Searcher Intent 29

30 , 11 July 2010 Post-Switch Satisfaction Measures of user effort / activity (# Queries, # Actions) Measure of the quality of the interaction % queries with No Clicks, # Actions to SAT (>30sec dwell) Activity # Queries # Actions Origin Destination Origin Destination All Queries Same Queries Success % NoClicks # Actions to SatAction Origin Destination Origin Destination All Queries Same Queries White et al., CIKM 2009 Users issue more queries/actions; seem less satisfied (higher %NoClicks and more actions to SAT) Switching queries may be challenging for search engines 59 Search Behavior: Expertise [White & Morris, WWW 2007] Some people are more expert at searching than others Search expertise, not domain expertise Alternative explanation: Orienteering vs. Teleporting Find characteristics of these advanced search engine users in an effort to better understand how these users search Understanding what advanced searchers are doing could improve the search experience for everyone 60 AAAI 2010 Tutorial: Inferring Searcher Intent 30

31 , 11 July 2010 Findings Post-query browsing Advanced users: Traverse trails faster Spend less time viewing each Web page Follow query trails with fewer steps Revisit pages less often Branch less often Feature p advanced 0% > 0% 25% 50% 75% Session Secs Trail Secs Display Secs Num. Steps Num. Revisits Num. Branches % Trails 72.14% 27.86%.83%.23%.05% % Users 79.90% 20.10%.79%.18%.04% Non-advanced [White & Morris, 2007] Advanced More advanced 61 Search Behavior: Demographics Gender differences: [Weber & Castillo, SIGIR 2010] Query wagner Women: Men: Education differences: 62 AAAI 2010 Tutorial: Inferring Searcher Intent 31

32 , 11 July 2010 ReFinding Behavior [From Teevan et al, 2007] 40% of the queries led to a click on a result that the same user had clicked on in a past search session. Teevan et al., 2007 What s the URL for this year s SIGIR 2010? Does not really matter, it is faster to re-find it 63 What Is Known About Re-Finding [From Teevan et al, 2007] Re-finding recent topic of interest Web re-visitation common [Tauscher & Greenberg] People follow known paths for re-finding Search engines likely to be used for re-finding Query log analysis of re-finding Query sessions [Jones& Fain] Temporal aspects [Sanderson& Dumais] 64 AAAI 2010 Tutorial: Inferring Searcher Intent 32

33 , 11 July 2010 Click on previously clicked results? [From Teevan et al, 2007] 39% 1 click > 1 click Click same and different? Click on different results? Same query issued before? Navigational 3100 (24%) 36 (<1%) 635 (5%) 485 (4%) Re-finding with different query New query? 637 (5%) 4 (<1%) 660 (5%) 7503 (57%) 65 Rank Change Degrades Re-Finding Results change rank Change in result rank reduces probability of re-click No rank change: 88% chance Rank change: 53% chance Rank change slower repeat click Compared with initial search to click No rank change: Re-click is faster Rank change: Re-click is slower [From Teevan et al, 2007] 66 AAAI 2010 Tutorial: Inferring Searcher Intent 33

34 , 11 July 2010 Aside: Mobile Search Not topic of today s tutorial Some references: M. Jones, Mobile Search Tutorial, Mobile HCI, 2009 K. Church, B. Smyth, K. Bradley, Keith and P. Cotter. A large scale study of European mobile search behaviour. Mobile HCI, 2008 Kamvar, M., Kellar, M., Patel, R., and Xu, Y. Computers and iphones and mobile phones, oh my!: a logsbased comparison of search users on different devices. WWW 2009 Kamvar, M. and Baluja, S Query suggestions for mobile search: understanding usage patterns, CHI Part 1: Summary Understanding user behavior at micro-, meso-, and macro- levels Theoretical models of information seeking Web search behavior: Levels of detail Search Intent Variations in web searcher behavior Keeping found things found 68 AAAI 2010 Tutorial: Inferring Searcher Intent 34

35 , 11 July 2010 Part 2: Inferring Web Searcher Intent Tutorial Overview Part 1: Search Intent Modeling Motivation: how intent inference could help search Web search intent & information seeking models Web searcher behavior models Part 2: Inferring Web Searcher Intent Inferring result relevance: clicks Richer interaction models: clicks + browsing Contextualizing intent models: personalization 70 AAAI 2010 Tutorial: Inferring Searcher Intent 35

36 , 11 July 2010 Part 2: Inferring Searcher Intent Inferring result relevance: clicks Richer behavior models: SERP presentation info Post-search behavior Rich interaction models for SERPs Contextualizing intent inference: Session-level models Personalization 71 Click Implicit Feedback Users often reluctant to provide relevance judgments Some searches are precision-oriented (no more like this ) They re lazy or annoyed: Was this document helpful? Can we gather relevancefeedback without requiring the user to do anything? Goal: estimate relevance from behavior 72 AAAI 2010 Tutorial: Inferring Searcher Intent 36

37 , 11 July 2010 Click Observable Behavior Minimum Scope Segment Object Class Behavior Category Examine Retain Reference Annotate View Listen Print Copy / paste Quote Mark up Select (click) Bookmark Save Purchase Delete Forward Reply Link Cite Rate Publish Subscribe Organize 73 Click Clicks as Relevance Feedback Limitations: Hard to determine the meaning of a click. If the best result is not displayed, users will click on something Presentation bias Click duration may be misleading People leave machines unattended Opening multiple tabs quickly, then reading them all slowly Multitasking Compare above to limitations of explicit feedback: Sparse, inconsistent ratings 74 AAAI 2010 Tutorial: Inferring Searcher Intent 37

38 , 11 July 2010 Strawman Click model: No Bias Naive Baseline [Craswellet al., 2008] cdi is P( Click=True Document=d, Position=i) rd is P( Click=True Document=d ) Why this baseline? We know that rd is part of the explanation Perhaps, for ranks 9 vs10, it s the main explanation It is a bad explanation at rank 1 e.g. Eye tracking Attractiveness of summary ~= Relevance of result 75 Realistic Click models Clickthrough and subsequent browsing behavior of individual users influenced by many factors Relevance of a result to a query Visual appearance and layout Result presentation order Context, history, etc. 76 AAAI 2010 Tutorial: Inferring Searcher Intent 38

39 , 11 July 2010 Click De-biasing position (first attempt) [Agichtein et al., 2006] Relative Click Frequency Higher clickthrough at top non-relevant than at top relevant document All queries PTR=1 PTR= Result Position Relative clickthrough for queries with known relevant results in position 3 (results in positions 1 and 2 are notrelevant) 77 Click Simple Model: Deviation from Expected [Agichtein et al., 2006] Relevance component: deviation from expected : Relevance(q,d)= observed -expected (p) Click frequency deviation PTR=1 PTR= Result position 78 AAAI 2010 Tutorial: Inferring Searcher Intent 39

40 , 11 July 2010 Click Simple Model: Example CD: distributional model, extends SA+N Clickthrough considered ifffrequency > εthan expected 0.4 Clickthrough Frequency Deviation Result position Click on result 2 likely by chance 4>(1,2,3,5), but not 2>(1,3) Click Click 79 Click Simple Model Results Improves precision by discarding chance clicks 80 AAAI 2010 Tutorial: Inferring Searcher Intent 40

41 , 11 July 2010 Another Formulation There are two types of user/interaction Click based on relevance Click based on rank (blindly) [Craswellet al., 2008] A.k.a. the OR model: Clicks arise from relevance OR position b i Estimate with logistic regression i 81 Linear Examination Hypothesis Users are less likely to look at lower ranks, therefore less likely to click 1 [Craswellet al., 2008] This is the AND model Clicks arise from relevance AND examination i Probability of examination does not depend on what else is in the list x i AAAI 2010 Tutorial: Inferring Searcher Intent 41

42 , 11 July 2010 Cascade Model [Taylor et al., 2008] Users examine the results in rank order At each document d Click with probability r d Or continue with probability (1-r d ) 83 Cascade Model (2) query URL 1 URL 2 URL 3 URL 4 (1-r d ) (1-r d ) (1-r d ) r 1 r 2 r 3 r 4 Relevance r d r d r d r d C 1 C 2 C 3 C 4 ClickThroughs 84 AAAI 2010 Tutorial: Inferring Searcher Intent 42

43 , 11 July 2010 Cascade Model Example 500 users typed a query 0 click on result A in rank click on result B in rank click on result C in rank 3 [Craswellet al., 2008] Cascade (with no smoothing) says: 0 of 500 clicked A ra = of 500 clicked B rb = of remaining 400 clicked C rc = Cascade Model Seems Closest to Reality [Craswellet al., 2008] Best possible: Given the true click counts for ordering BA 86 AAAI 2010 Tutorial: Inferring Searcher Intent 43

44 , 11 July 2010 Click Dynamic Bayesan Net did user examine url? O. Chapelle, & Y Zhang, A Dynamic Bayesian Network Click Model for Web Search Ranking, WWW 2009 was user satisfied by landing page? user attracted to url? 87 Click Dynamic Bayesan Net O. Chapelle, & Y Zhang, A Dynamic Bayesian Network Click Model for Web Search Ranking, WWW AAAI 2010 Tutorial: Inferring Searcher Intent 44

45 , 11 July 2010 Click Dynamic Bayesan Net (results) O. Chapelle, & Y Zhang, A Dynamic Bayesian Network Click Model for Web Search Ranking, WWW 2009 Use EM algorithm (similar to forward-backward to learn model parameters; set manually predicted relevance agrees 80% with human relevance 89 Click Clicks: Summary So Far Simple model accounts for position bias BayesNet model: extension of Cascade model shown to work well in practice Limitations? Questions? 90 AAAI 2010 Tutorial: Inferring Searcher Intent 45

46 , 11 July 2010 Click Capturing a Click in its Context [Piwowarski et al., 2009] Building query chains Simple model based on time deltas & query similarities Analysingthe chains Layered Bayesian Network (BN) model Validation of the model Relevance of clicked documents Boosted Trees with features from the BN 91 Click Overall process [Piwowarski et al., 2009] Grouping atomic sessions Time threshold Similarity threshold 92 AAAI 2010 Tutorial: Inferring Searcher Intent 46

47 , 11 July 2010 Click Layered Bayesian Network [Piwowarski et al., 2009] 93 Click The BN gives the context of a click [Piwowarski et al., 2009] Chain Probability (Chain state= / observations) = (0.2, 0.4, 0.01, 0.39, 0) Search Probability (Search state= / observations) = (0.1, 0.42, ) Page Probability (Page state= / observations) = (0.25, 0.2, ) Click Probability (Click state= / observations) = (0.02, 0.5, ) Relevance Probability ([not] Relevant / observations) = (0.4, 0.5) AAAI 2010 Tutorial: Inferring Searcher Intent 47

48 , 11 July 2010 Click Features for one click [Piwowarski et al., 2009] For each clicked document, compute features: (BN) Chain/Page/Action/Relevance state distribution (BN) Maximum likelihood configuration, likelihood Word confidence values (averaged for the query) Time and position related features This is associated with a relevance judgment from an editor and used for learning 95 Click Learning with Gradient Boosted Trees [Piwowarski et al., 2009] Use a Gradient boosted trees (Friedman 2001), with a tree depth of 4 (8 for non BN-based model) Used disjoint train (BN + GBT training) and test sets Two sets of sessions S1 and S2 (20 million chains) and two set of queries + relevance judgment J1 and J2 (about 1000 queries with behavior data) Process (repeated 4 times): learn the BN parameters on S1+J1, extract the BN features and learn the GBT with S1+J1 Extract the BN features and predict relevance assessments of J2 with sessions of S2 96 AAAI 2010 Tutorial: Inferring Searcher Intent 48

49 , 11 July 2010 Click Results: Predicting Relevance of Clicked Docs [Piwowarski et al., 2009] 97 Problem: Users click based on result Snippets [Clarke et al., 2007] Effect of Caption Features on Clickthrough Inversions, C. Clarke, E. Agichtien, S. Dumais, R. White, SIGIR AAAI 2010 Tutorial: Inferring Searcher Intent 49

50 , 11 July 2010 Clickthrough Inversions [Clarke et al., 2007] 99 Relevance is Not the Dominant Factor! [Clarke et al., 2007] 100 AAAI 2010 Tutorial: Inferring Searcher Intent 50

51 , 11 July 2010 Snippet Features Studied [Clarke et al., 2007] 101 Feature Importance [Clarke et al., 2007] 102 AAAI 2010 Tutorial: Inferring Searcher Intent 51

52 , 11 July 2010 Important Words in Snippet [Clarke et al., SIGIR 2007] 103 Extension: Use Fair Pairs Randomization Click data: [Yue et al., WWW 2010] Example result: (bars should be equal if unbiased) 104 AAAI 2010 Tutorial: Inferring Searcher Intent 52

53 , 11 July 2010 Viewing Organic Results vs. Ads Danescu-Niculescu-Mizil et al., WWW 2010 Ads and Organic results compete for user attention Navigational vs. Other Diversity v Similarity 105 Part 2: Inferring Searcher Intent Inferring result relevance: clicks Richer behavior models: SERP presentation info Richer interaction models: +presentation, +behavior 106 AAAI 2010 Tutorial: Inferring Searcher Intent 53

54 , 11 July 2010 Richer Behavior Models Behavior measures of Interest Browsing, scrolling, dwell time How to estimate relevance? Heuristics Learning-based General model: Curious Browser [Fox et al., TOIS 2005] Query + Browsing [Agichtein et al., SIGIR 2006] Active Prediction: [Yun et al., WWW 2010] 107 Curious Browser [Fox et al., 2003] 108 AAAI 2010 Tutorial: Inferring Searcher Intent 54

55 , 11 July 2010 Data Analysis [Fox et al., 2003] Bayesian modeling at result and session level Trained on 80% and tested on 20% Three levels of SAT VSAT, PSAT & DSAT Implicit measures: Result-Level Diff Secs, Duration Secs Scrolled, ScrollCnt, AvgSecsBetweenScroll, TotalScrollTime, MaxScroll TimeToFirstClick, TimeToFirstScroll Page, Page Position, Absolute Position Visits Exit Type ImageCnt, PageSize, ScriptCnt Added to Favorites, Printed Session-Level Averages of result-level measures (Dwell Time and Position) Query count Results set count Results visited End action Eugene AAAI Agichtein 2010 Tutorial: Emory Inferring University Searcher Intent 7/11/ Data Analysis, cont d [Fox et al., 2003] Eugene AAAI Agichtein 2010 Tutorial: Emory Inferring University Searcher Intent 7/11/ AAAI 2010 Tutorial: Inferring Searcher Intent 55

56 , 11 July 2010 Result-Level Findings [Fox et al., 2003] 1. Dwell time, clickthrough and exit type strongest predictors of SAT 2. Printing and Adding to Favorites highly predictive of SAT when present 3. Combined measures predict SAT better than clickthrough 111 Result Level Findings, cont d [Fox et al., 2003] Only clickthrough Combined measures Combined measures with confidence of > 0.5 (80-20 train/test split) Eugene AAAI Agichtein 2010 Tutorial: Emory Inferring University Searcher Intent 7/11/ AAAI 2010 Tutorial: Inferring Searcher Intent 56

57 , 11 July 2010 Learning Result Preferences in Rich User Interaction Space Observed and Distributional features Observed features: aggregated values over all user interactions for each query and result pair Distributional features: deviations from the expected behavior for the query Represent user interactions as vectors in Behavior Space Presentation: what a user sees before click Clickthrough: frequency and timing of clicks Browsing: what users do after the click [Agichtein et al., 2006] 113 Features for Behavior Representation [Agichtein et al., SIGIR2006] ResultPosition QueryTitleOverlap DeliberationTime ClickFrequency ClickDeviation DwellTime DwellTimeDeviation Presentation Position of the URL in Current ranking Fraction of query terms in result Title Clickthrough Seconds between query and first click Fraction of all clicks landing on page Deviation from expected click frequency Browsing Result page dwell time Deviation from expected dwell time for query Sample Behavior Features 114 AAAI 2010 Tutorial: Inferring Searcher Intent 57

58 , 11 July 2010 Predicting Result Preferences Task: predict pairwise preferences A judgewill prefer Result A > ResultB Models for preference prediction Current search engine ranking Clickthrough Full user behavior model [Agichtein et al., SIGIR2006] 115 User Behavior Model Full set of interaction features Presentation, clickthrough, browsing Train the model with explicit judgments Input: behavior feature vectors for each query-page pair in rated results Use RankNet(Burges et al., [ICML 2005]) to discover model weights [Agichtein et al., SIGIR2006] Output: a neural net that can assign a relevance score to a behavior feature vector 116 AAAI 2010 Tutorial: Inferring Searcher Intent 58

59 , 11 July 2010 Results: Predicting User Preferences [Agichtein et al., SIGIR2006] Precision SA+N Recall SA+N CD UserBehavior Baseline Baseline < SA+N < CD << UserBehavior Rich user behavior features result in dramatic improvement 117 Predicting Queries from Browsing Behavior [Cheng et al., WWW 2010] Identify Search Trigger browse-search patterns Distribution of Search-Browse patterns: URLs: movies.about.com/ nationalpriorities.org pds.jpl.nasa.gov/planets 118 AAAI 2010 Tutorial: Inferring Searcher Intent 59

60 , 11 July 2010 Summary of Part 2 Click data contains important information about the distribution of intents for a query For accurate interpretation, must model the (many) biases present: Presentation, demographics, types of intent 119 Part 3: Applications and Extensions Improving search ranking Implicit feedback Predicting Intent and Behavior Query suggestion, ads Search personalization 120 AAAI 2010 Tutorial: Inferring Searcher Intent 60

61 , 11 July 2010 Observable Behavior Minimum Scope Segment Object Class Behavior Category Examine Retain Reference Annotate View Listen Print Copy / paste Quote Mark up Bookmark Save Purchase Delete Forward Reply Link Cite Rate Publish Subscribe Organize 121 Eye Tracking Unobtrusive Relatively precise (accuracy: 1 of visual angle) Expensive Mostly used as passive tool for behavior analysis, e.g. visualized by heatmaps: We use eye tracking for immediate implicit feedback taking into account temporal fixation patterns 122 AAAI 2010 Tutorial: Inferring Searcher Intent 61

62 , 11 July 2010 Using Eye Tracking for Relevance Feedback [Buscher et al., 2008] Starting point: Noisy gaze data from the eye tracker. 2. Fixation detection and saccade classification 3. Reading (red) and skimming (yellow) detection line by line See G. Buscher, A. Dengel, L. van Elst: Eye Movements as Implicit Relevance Feedback, in CHI ' Three Feedback Methods Compared [Buscher et al., 2008] Input: viewed documents Gaze-Filter TF x IDF based on read or skimmed passages Gaze-Length- Filter Interest(t) x TF x IDF based on length of coherently read text Reading Speed Baseline ReadingScore(t) x TF x IDF based on read vs. skimmed passages containing term t TF x IDF based on opened entire documents 124 AAAI 2010 Tutorial: Inferring Searcher Intent 62

63 , 11 July 2010 Eye Tracking-based RF Results [Buscher et al., 2008] 125 Instrumenting SERP Interactions: EMU Gui, Agichtein, et al., JCDL 2009 HTTP Server HTTP Log Usage Data Train Prediction Models Data Mining & Management EMU: Firefox + LibX plugin instrumentation http log Track whitelisted sites e.g., Emory, Google, Yahoo search All SERP events logged (asynchronous http requests) 150 public use machines, 5,000+ opted-in users 126 AAAI 2010 Tutorial: Inferring Searcher Intent 63

64 , 11 July 2010 Classifying Research vs. Purchase Intent Guo & Agichtein, SIGIR subjects (grad students and staff) asked to 1. Research a product they want to purchase eventually (Research intent) 2. Search for a best deal on an item they want to purchase immediately (Purchase intent) Eye tracking and browser instrumentation performed in parallel for some of the subjects 127 Research Intent Guo & Agichtein, SIGIR AAAI 2010 Tutorial: Inferring Searcher Intent 64

65 , 11 July 2010 Purchase Intent Guo & Agichtein, SIGIR Contextualized Intent Inference Guo & Agichtein, SIGIR AAAI 2010 Tutorial: Inferring Searcher Intent 65

66 , 11 July 2010 Implementation: Conditional Random Field (CRF) Model Guo & Agichtein, SIGIR 2010 Emory University 131 Results: Ad Click Prediction 200%+ precision improvement (within task) Guo & Agichtein, SIGIR AAAI 2010 Tutorial: Inferring Searcher Intent 66

67 , 11 July 2010 Application: Learning to Rank from Click Data [ Joachims 2002 ] 133 Results [ Joachims 2002 ] Summary: Learned outperforms all base methods in experiment Learning from clickthrough data is possible Relative preferences are useful training data. 134 AAAI 2010 Tutorial: Inferring Searcher Intent 67

68 , 11 July 2010 Extension: Query Chains [Radlinski& Joachims, KDD 2005] 135 Query Chains (Cont d) [Radlinski& Joachims, KDD 2005] 136 AAAI 2010 Tutorial: Inferring Searcher Intent 68

69 , 11 July 2010 Query Chains (Results) [Radlinski& Joachims, KDD 2005] Query Chains add slight improvement over clicks 137 Richer Behavior for Dynamic Ranking [Agichtein et al., SIGIR2006] ResultPosition QueryTitleOverlap DeliberationTime ClickFrequency ClickDeviation DwellTime DwellTimeDeviation Presentation Position of the URL in Current ranking Fraction of query terms in result Title Clickthrough Seconds between query and first click Fraction of all clicks landing on page Deviation from expected click frequency Browsing Result page dwell time Deviation from expected dwell time for query Sample Behavior Features (from Lecture 2) 138 AAAI 2010 Tutorial: Inferring Searcher Intent 69

70 , 11 July 2010 Feature Merging: Details Query: SIGIR, fake results w/ fake feature values Result URL BM25 PageRank Clicks DwellTime sigir2007.org ?? Sigir2006.org acm.org/sigs/sigir/ Value scaling: Binning vs. log-linear vs. linear (e.g., μ=0, σ=1) Missing Values: [Agichtein et al., SIGIR2006] 0? (meaning for normalized feature values s.t. μ=0?) real-time : significant architecture/system problems 139 Review: NDCG Normalized Discounted Cumulative Gain Multiple Levels of Relevance DCG: y contribution of ithrank position: 2 i 1 log( i + 1) Ex: has DCG score of 1 log(2) + 3 log(3) 1 log(4) 0 log(5) 1 log(6) NDCG is normalized DCG best possible ranking as score NDCG = AAAI 2010 Tutorial: Inferring Searcher Intent 70

71 , 11 July 2010 Human Judgments Results for Incorporating Behavior into Ranking [Agichtein et al., SIGIR2006] NDCG RN Rerank-All RN+All K MAP Gain RN RN+ALL (19.13%) BM BM25+ALL AAAI 2010 Tutorial: Inferring Searcher Intent (23.71%) 7/11/ AAAI 2010 Tutorial: Inferring Searcher Intent 71

72 , 11 July 2010 Which Queries Benefit Most [Agichtein et al., SIGIR2006] Frequency Average Gain Most gains are for queries with poor original ranking 143 Extension to Unseen Queries/Documents: Search Trails [Bilenko and White, WWW 2008] Trails start with a search engine query Continue until a terminating event Another search Visit to an unrelated site (social networks, webmail) Timeout, browser homepage, browser closing 144 AAAI 2010 Tutorial: Inferring Searcher Intent 72

73 , 11 July 2010 Probabilistic Model [Bilenko and White, WWW 2008] IR via language modeling [Zhai-Lafferty, Lavrenko] Query-term distribution gives more mass to rare terms: Term-website weights combine dwell time and counts 145 Results: Learning to Rank Add Rel(q, d i ) as a feature to RankNet [Bilenko and White, WWW 2008] NDCG Baseline Baseline+Heuristic Baseline+Probabilistic 0.58 NDCG@1 NDCG@3 NDCG@10 Baseline+Probabilistic+RW 146 AAAI 2010 Tutorial: Inferring Searcher Intent 73

74 , 11 July 2010 Personalization 147 Which Queries to Personalize? [Teevan et al., 2008] Personalization benefits ambiguous queries Inter-rater reliability (Fleiss kappa) Observed agreement (P a ) exceeds expected (P e ) κ= (P a -P e ) / (1-P e ) Relevance entropy Variability in probability result is relevant (P r ) S= -ΣP r log P r Potential for personalization Ideal group ranking differs from ideal personal P4P = 1 -ndcg group Teevan, J, S.T. Dumais, and D.J. Liebling. To personalize or not to personalize: modeling queries with variation in user intent., SIGIR AAAI 2010 Tutorial: Inferring Searcher Intent 74

75 , 11 July 2010 Predicting Ambiguous Queries [Teevan et al., 2008] History Information Query Results No Query length Contains URL Contains advanced operator Time of day issued Number of results (df) Number of query suggests Query clarity ODP category entropy Number of ODP categories Portion of non-html results Portion of results from.com/.edu Number of distinct domains Yes Reformulation probability #of times query issued # of users who issued query Avg. time of day issued Avg. number of results Avg. number of query suggests Result entropy Avg. click position Avg. seconds to click Avg. clicks per user Click entropy Potential for personalization Teevan, J, S.T. Dumais, and D.J. Liebling. To personalize or not to personalize: modeling queries with variation in user intent., SIGIR Clustering Query Refinements by User Intent [Sadikov et al., WWW 2010] Mars (Candy) vs. Mars (Planet) Approach: Intent = Set of visited documents Cluster refinements using document visit distribution vectors 150 AAAI 2010 Tutorial: Inferring Searcher Intent 75

76 , 11 July 2010 Approaches to Personalization 1. Pitkowet al., Qiuet al., Jehet al., Teevan et al., Das et al., Emory 151University Figure adapted from: Personalized search on the world wide web, by Micarelli, A. and Gasparetti, F. and Sciarrone, F. and Gauch, S., LNCS 2007 personalization research Teevan et al., TOCHI 2010 Ask the searcher Is this relevant? Look at searcher s clicks Similarity to content searcher s seen before 152 AAAI 2010 Tutorial: Inferring Searcher Intent 76

77 , 11 July 2010 Ask the Searcher Teevan et al., TOCHI 2010 Explicitindicator of relevance Benefits Direct insight Drawbacks Amount of data limited Hard to get answers for the same query Unlikely to be available in a real system 153 Searcher s Clicks Teevan et al., TOCHI 2010 Implicitbehavior-based indicator of relevance Benefits Possible to collect from all users Drawbacks People click by mistake or get side tracked Biased towards what is presented 154 AAAI 2010 Tutorial: Inferring Searcher Intent 77

78 , 11 July 2010 Similarity to Seen Content Teevan et al., TOCHI 2010 Implicit content-based indicator of relevance Benefits Can collect from all users Can collect for all queries Drawbacks Privacy considerations Measures of textual similarity noisy 155 Evaluating Personalized Search Explicit judgments (offline and in situ) Evaluate components before system NOTE: What s relevant for you Deploy system Verbatim feedback, Questionnaires, etc. Measure behavioral interactions (e.g., click, reformulation, abandonment, etc.) Click biases order, presentation, etc. Interleaving for unbiased clicks Link implicit and explicit (Curious Browser toolbar) From single query to search sessions and beyond 156 AAAI 2010 Tutorial: Inferring Searcher Intent 78

79 , 11 July 2010 User Control in Personalization (RF) Emory 157University J-S. Ahn, P.Brusilovsky, D.He, and S.Y. Syn. Open user profiles for adaptive news systems: Help or harm? WWW 2007 Personalization Summary Lots of relevant content ranked low Potential for personalization high Implicit measures capture explicit variation Behavior-based: Highly accurate Content-based: Lots of variation Example: Personalized Search Behavior + content work best together Improves search result click through 158 AAAI 2010 Tutorial: Inferring Searcher Intent 79

80 , 11 July 2010 New Direction: Active Learning Goal: Learn the relevanceswith as little training data as possible. Search involves a three step process: [Radlinski& Joachims, KDD 2007] 1. Given relevance estimates, pick a ranking to display to users. 2. Given a ranking, users provide feedback: User clicks provide pairwise relevance judgments. 3. Given feedback, update the relevance estimates. Eugene AAAI Agichtein 2010 Tutorial: Emory Inferring University Searcher Intent 7/11/ Overview of Approach [Radlinski& Joachims, KDD 2007] Available information: 1. Have an estimate of the relevance of each result. 2. Can obtain pairwise comparisons of the top few results. 3. Do not have absolute relevance information. Goal: Learn the document relevance quickly. Addresses four questions: 1. How to represent knowledge about doc relevance. 2. How to maintain this knowledge as we collect data. 3. Given our knowledge, what is the best ranking? 4. What rankings do we show users to get useful data? 160 AAAI 2010 Tutorial: Inferring Searcher Intent 80

81 , 11 July : Representing Document Relevance [Radlinski& Joachims, KDD 2007] Given a fixed query, maintain knowledge about relevance as clicks are observed. This tells us which documents we are sure about, and which ones need more data : Getting Useful Data [Radlinski& Joachims, KDD 2007] Problem: could present the ranking based on current best estimate of relevance. Then the data we get would always be about the documents already ranked highly. Instead, optimize ranking shown users: 1. Pick top two docs to minimize future loss 2. Append current best estimate ranking. 162 AAAI 2010 Tutorial: Inferring Searcher Intent 81

82 , 11 July : Exploration Strategies [Radlinski& Joachims, KDD 2007] 163 Results: TREC Data [Radlinski& Joachims, KDD 2007] Optimizing for relevance estimates better than for ordering 164 AAAI 2010 Tutorial: Inferring Searcher Intent 82

83 , 11 July 2010 Tutorial Summary Understanding user behavior at micro-, meso-, and macro- levels Theoretical models of information seeking Web search behavior: Levels of detail Search Intent Variations in web searcher behavior Keeping found things found Click models 165 Tutorial Summary (2) Inferring result relevance: clicks Richer behavior models: SERP presentation info Post-search behavior Rich interaction models for SERPs Contextualizing intent inference: Session-level models Personalization 166 AAAI 2010 Tutorial: Inferring Searcher Intent 83

84 , 11 July 2010 Inferring Searcher Intent (Information) Tutorial Page: See the online version for expanded and updated bibliography Contact information for the instructor: Homepage: References and Further Reading (1) Marti Hearst, Search User Interfaces, 2009, Chapter 3 Models of the Information Seeking Process : Teevan, J., Adar, E., Jones, R. and Potts, M. Information Re-Retrieval: Repeat Queries in Yahoo's Logs, SIGIR 2007 Clarke, C, E. Agichtein, S. Dumais and R. W. White, The Influence of Caption Features on Clickthrough Patterns in Web Search, SIGIR 2007 Craswell, N., Zoeter, O., Taylor, M., Ramsey, B. An experimental comparison of click position-bias models, WSDM 2008 Dupret, G and Piwowarski, B: A user browsing model to predict search engine click data from past observations. SIGIR 2008 White, R and D. Morris, Investigating the Querying and Browsing Behavior of Advanced Search Engine Users, SIGIR AAAI 2010 Tutorial: Inferring Searcher Intent 84

85 , 11 July 2010 References and Further Reading (2) Marti Hearst, Search User Interfaces, 2009, Chapter 3 Models of the Information Seeking Process : Teevan, J., Adar, E., Jones, R. and Potts, M. Information Re-Retrieval: Repeat Queries in Yahoo's Logs, SIGIR 2007 Clarke, C, E. Agichtein, S. Dumais and R. W. White, The Influence of Caption Features on Clickthrough Patterns in Web Search, SIGIR 2007 Craswell, N., Zoeter, O., Taylor, M., Ramsey, B. An experimental comparison of click position-bias models, WSDM 2008 Dupret, G and Piwowarski, B: A user browsing model to predict search engine click data from past observations. SIGIR 2008 White, R and D. Morris, Investigating the Querying and Browsing Behavior of Advanced Search Engine Users, SIGIR References and Further Reading (3) Kelly, D. and Teevan, J. Implicit feedback for inferring user preference: a bibliography. SIGIR Forum 37, 2 (Sep. 2003) Joachims, T., Granka, L., Pan, B., Hembrooke, H., and Gay, G. Accurately interpreting clickthrough data as implicit feedback., SIGIR 2005 Agichtein, E., Brill, E., Dumais, S., and Ragno, R. Learning user interaction models for predicting web search result preferences, SIGIR 2006 Buscher, G., Dengel, A., and van Elst, L. Query expansion using gaze-based feedback on the subdocument level., SIGIR 2008 Chapelle, O, and Y. Zhang, A Dynamic Bayesian Network Click Model for Web Search Ranking, WWW 2009 Piwowarski, B, Dupret, G, Jones, R: Mining user web search activity with layered bayesiannetworks or how to capture a click in its context, WSDM 2009 Guo, Q and Agichtein, E. Ready to Buy or Just Browsing? Detecting Web Searcher Goals from Interaction Data, to appear, SIGIR AAAI 2010 Tutorial: Inferring Searcher Intent 85