Machine Learning Applications to Modeling Web Searcher Behavior Eugene Agichtein

Transcription

1 Machine Learning Applications to Modeling Web Searcher Behavior Eugene Agichtein Intelligent Information Access Lab (IRLab) Emory University

2 Talk Outline Overview of the Emory IR Lab Intent-centric Web Search Contextualized search intent detection One example medical application Eugene Agichtein, Emory University, IR Lab 2

3 Intelligent Information Access Lab (IRLab) Modeling information seeking behavior Searching the Web and social media Text and data mining for medical informatics Ablimit Aji Qi Guo Julia Kiseleva In collaboration with: - Beth Buffalo (Neurology) - Charlie Clarke (Waterloo) - Ernie Garcia (Radiology) - Phil Wolff (Psychology) - Hongyuan Zha (GaTech) Dmitry Lagun Qiaoling Liu Yu Wang Eugene Agichtein, Emory University, IR Lab 3

4 Our Approach to Intelligent Information Access Search logs: queries, clicks Data-Driven Model Discovery (machine learning/data mining) Intelligent search Information sharing Health Informatics Cognitive Diagnostics4 4 Eugene Agichtein, Emory University, IR Lab

5 Intelligent search Web-scale Text Mining Extract entities, relationships, events from text Estimate accuracy of web content DiseaseOutbreaks, The New York Times Some Applications: Incorporating extracted information into (web) search Finding implicit connections between events, entities Visualizing and exploring large text collections 18 November 2009 Eugene Agichtein, Emory 5 University, IR Lab [DL 00, ICDE 2003 best student paper, SIGMOD 2006 best paper, ]

6 Information sharing 6 6

7 Information sharing Social Media Language Analysis Social Media!= WSJ Text Text Mining/NLP Challenges: Content quality Authority/expertise User goals, Subjectivity Sentiment Temporal Sensitivity Effort and Incentives Eugene Agichtein, Emory University, IR Lab 7

8 Content Quality 88

9 Information sharing + Intelligent search Hybrid Web/Social Search Eugene Agichtein, Emory University, IR Lab 9

10 Talk Outline Overview of the Emory IR Lab Intent-centric Web Search Contextualized search intent detection One example medical application Eugene Agichtein, Emory University, IR Lab 10

11 Some Key Challenges for Web Search Query interpretation (infer intent) Ranking (high dimensionality) Evaluation (system improvement) Result presentation (information visualization) Eugene Agichtein, Emory University, IR Lab 11

12 Task-Goal-Search Model car safety ratings consumer reports Eugene Agichtein, Emory University, IR Lab 12

13 Problem Statement Given: Sequence of user actions, predictuser goal, task, and future actions Will define tasks and goal next Example applications: Predict document relevance (ranking, result presentation, summarization) Predict next query (query suggestion, spelling correction) Predict user satisfaction (market share) Eugene Agichtein, Emory University, IR Lab 13

14 Intent (Goal) Classes, top level only User intent taxonomy (Broder2002) Informational want to learn about something (~40% / 65%) Navigational want to go to that page (~25% / 15%) Transactional want to do something (web-mediated) (~35% / 20%) Access a servicedownloads Shop Gray areas Find a good hub Exploratory search see what s there Eugene Agichtein, Emory University, IR Lab History nonya food Singapore Airlines Jakarta weather Kalimantan satellite images Nikon Finepix Car rental Kuala Lumpur [from SIGIR 2008 Tutorial, Baeza-Yates and Jones]

15 Information Retrieval Process: Implementation Source Selection Resource Search Engine Result Page (SERP) Query Formulation Query: car safety ratings Search Ranked List query reformulation, vocabulary learning, relevance feedback Selection Documents Examination Documents source reselection Delivery Eugene Agichtein, Emory University, IR Lab 15

16 Search Actions Keystrokes query, scroll, CTRL-C, ) GUI: scrolling, button press, clicks Mouse: moving, scrolling, down/up, scroll Browser: new tab, close, back/forward All of these can be easily captured on SERP (javascript) Eugene Agichtein, Emory University, IR Lab 16

17 How Do We Know True User Intent? Askthe user (surveys, field studies, pop-ups) Does not scale, users get annoyed Observe user actions and guess Intent usuallyobvious to humans but not always Detect signals from user s brain (fmri, EEEG) and attempt to interpret neuron activity Adapted from [Daniel M. Russell, 2007] Eugene Agichtein, Emory University, IR Lab 17

18 What Eye Movement Can Tell Eye tracking gives information about searcher interests: Eye position Pupil diameter Saccades and fixations Reading Camera Visual Search Eugene Agichtein, Emory University, IR Lab 18

19 An Eye Tracker on Every Table Eye tracking equipment is bulky and expensive Can we infer gaze position from observable actions? Exploratory study from Google (Roddenet al.) says maybe: mouse position is sometimesrelated to eye position Eugene Agichtein, Emory University, IR Lab 19

20 Relationship Between Mouse and Gaze Position Searchers might use the mouse to focus reading attention, bookmark promising results, or not at all. Behavior varies with task difficulty and user expertise [K. Rodden, X. Fu, A. Aula, and I. Spiro, Eye-mouse coordination patterns on web search results pages, Extended Abstracts of ACM CHI 2008] Eugene Agichtein, Emory University, IR Lab 20

21 Assume Transitivity Holds Given: Gaze position ==> user intent and Mouse movement ==> gaze position Mouse movement ==> user intent Restate problem: Given user actions, infer current user s intent, focusing on Individual User s actions Eugene Agichtein, Emory University, IR Lab 21

22 Collecting Search Data: EMU HTTP Server HTTP Log Usage Data Train Prediction Models Data Mining & Management Firefox + LibXplugin Track whitelisted sites e.g., Emory, Google, Yahoo search All SERP events logged (asynchronous http requests) 150 public use machines, 5,000+ opted-in users Eugene Agichtein, Emory University, IR Lab 22

23 EMU: Querying Behavior Data Eugene Agichtein, Emory Univesity, IR Lab 23

24 Playback Example Eugene Agichtein, Emory University, IR Lab 24

25 Qi Guo Research vs. Purchase Intent 12 subjects (grad students and staff) asked to 1. Research a product they want to purchase eventually (Research intent) 2. Search for a best deal on an item they want to purchase immediately (Purchase intent) Eye tracking and browser instrumentation performed in parallel for some of the subjects EyeTech systems TM3 (rental) avoid! At reasonable resolution, samples at only ~12-15 Hz Looses calibration after a few minutes Eugene Agichtein, Emory University, IR Lab 25

26 Research Intent Eugene Agichtein, Emory University, IR Lab 26

27 Purchase Intent Eugene Agichtein, Emory University, IR Lab 27

28 Informational query: spanish wine Eugene Agichtein, Emory University, IR Lab 28

29 Mouse Features: Simple First representation: Trajectory length Horizontal range Vertical range Horizontal range Trajectory length Vertical range Eugene Agichtein, Emory University, IR Lab 29

30 Mouse Features: Full Second representation: 5 segments: initial, early, middle, late, and end Each segment: speed, acceleration, rotation, slope, etc Eugene Agichtein, Emory University, IR Lab 30

31 Classifying Search Intent Eugene Agichtein, Emory University, IR Lab 31

32 Use Support Vector Machine (SVM) Classifier SVMs maximize the marginaround the separating hyperplane. A.k.a. large margin classifiers The decision function is fully specified by a subset of training samples, the support vectors. Quadratic programming problem Seen by many as most successful current text classification method Support vectors Maximize margin Eugene Agichtein, Emory University, IR Lab 32

33 Results: Research vs. Purchase Eugene Agichtein, Emory University, IR Lab 33

34 Contextualized Intent Inference Eugene Agichtein, Emory University, IR Lab 34

35 Implementation: Conditional Random Field (CRF) Model Eugene Agichtein, Emory University, IR Lab 35

36 From HMMs to MEMMs to CRFs Eugene Agichtein, Emory University, IR Lab 36 n o n o o o s s s s,...,,..., = = v v HMM MEMM CRF S t-1 S t O t S t+1 O t+1 O t-1 S t-1 S t O t S t+1 O t+1 O t-1 S t-1 S t O t S t+1 O t+1 O t = 1 1 ) ( ) ( ), ( o t t t t t s o P s s P o s P v v v = = + 1 1, 1 1 ), ( ), ( exp 1 ), ( ) ( 1 o t k t t k k j t t j j o s o t t t t o s g s s f Z o s s P o s P t t v v v v µ λ = ), ( ), ( exp 1 ) ( o t k t t k k j t t j j o o s g s s f Z o s P v v v v µ λ Conditional Random Fields (CRFs) [from Lafferty, McCallum, Pereira 2001]

37 Application: Predict Ad Receptiveness Hypothesis: the right time to show search ads: when searcher is receptive to seeing ads Eugene Agichtein, Emory University, IR Lab 37

38 Results: Ad Click Prediction 200%+ precision improvement (within task) Eugene Agichtein, Emory University, IR Lab 38

39 Challenges Separate context from intent (e.g., smart phones) User variability: individual differences, tasks Scale of data: representation, compression Privacy: client-side data similar to other PII Obtaining realistic user data: see above EMU toolbar tracking since 2007 in Emory Libraries (biased) Eugene Agichtein, Emory University, IR Lab 39

40 Current and Future Work Detect mouse reading behavior Unsupervised intent clustering User vs. task Personalized behavior models Long-term interests/effects User mental state (frustration, cognitive impairment, ) Eugene Agichtein, Emory University, IR Lab 40

41 Dmitry Lagun Towards Web-based Visual Paired Comparison Test with Beth Buffalo, Neurology/Yerkes VPC can be used to detect MCI (years before AD), but requires eye tracking equipment Goal: develop web-based version of VPC NIH ADRC Pilot Grant, jointly with Beth Buffalo Approach: exploit connection between mouse movement and gaze position. Force usage of mouse to reveal image Or parts of image Develop robust machine learning techniques to predict cognitive impairment based on (noisy) mouse data Eugene Agichtein, Emory University, IR Lab 41

42 Initial Results Preference for novel image (59%) consistently observed Still exploring parameter space and metrics to optimize Results sensitive to Mturk worker instructions, incentives, other factors (?) Looking for advice on remote behavioral testing Eye tracking Mouse (oculus) center Eugene Agichtein, Emory University, IR Lab 42

43 Summary: From Behavior to State of Mind Approach: Machine learning methods for detecting searcher intent Calibrated and augmented with lab studies Foundational contributions: Methods to mine and integrate wide range of interactions Data-driven discovery of user state-of-mind Impact: Intelligent, intuitive search and information sharing Potential for new research tools and techniques Eugene Agichtein, Emory University, IR Lab 43

44 Main References Classifying and Characterizing Query Intent, Azin Ashkan, Charles L. A. Clarke, Eugene Agichtein, Qi Guo, In ECIR Qi Guo and Eugene Agichtein, Exploring Client-Side Instrumentation for Personalized Search Intent Inference: Preliminary Experiments, Proc. of AAAI 2008 Workshop on Intelligent Techniques for Web Personalization (ITWP 2008) Qi Guo, Eugene Agichtein, Azin Ashkan and Charles L. A. Clarke: In the Mood to Click? Inferring Searcher Advertising Receptiveness, in Proc. of WI 2009 Other papers here: Eugene Agichtein, Emory University, IR Lab 44

45 Thank you! Qi Guo, Dmitry Lagun, Beth Buffalo, and Phil Wolff Supported by: Eugene Agichtein, Emory University, IR Lab 45