Mining of Implicit User Data

Transcription

1 Introduction to Search Engine Technology Mining Implicit User Data Ronny Lempel Yahoo! Labs, Haifa Mining of Implicit User Data Implicit data: data that users enter (or leave) in the course of their interaction with the Web, without necessarily being aware that it is used for purposes beyond the immediate interaction In this class: queries, clicks, timestamps of uploaded images As opposed to explicit user data: relevance feedback, ratings, reviews, comments, tags, blog entries, forum postings Assortment of examples on how mining of implicit user data may be used to improve search and media experiences 1. Automatic construction of touristic itineraries 2. Improving automatic query suggestions 3. Pseudo-anchor text generation 4. Identification when no interaction implies user satisfaction By no means comprehensive! Search Engine Technology 2 1

2 Automatic Construction of Travel Itineraries using Social Breadcrumbs Hypertext 2010 Munmun De Choudhury* Sihem Amer-Yahia Cong Yu Yahoo! Research, NY Moran Feldman* Nadav Golbandi Ronny Lempel Yahoo! Labs, Haifa * Former intern Search Engine Technology 3 In a Nutshell Goal: construct intra-city travel itineraries automatically by tapping a latent source reflecting geo-temporal breadcrumbs left by millions of tourists Flickr images, in this case Approach Extract photo streams of individual users Associate points of interest with the photos Using the timestamps of the photos, determine: Lower bounds on the amount of time spent at a POI Upper bounds on the transit time between POIs aggregate all user photo streams into a POI graph; apply the Orienteering algorithm to construct itineraries Search Engine Technology 4 2

3 Step I Timed Paths Photo-POI Mapping Photo Streams Step II Step III Identify photos of a given city Filter out residents of a city Validate photo timestamps Extract Candidate POIs o Lonely Planet/Y! Travel to extract landmarks o Yahoo! Maps API to retrieve their geo-locations Tag & geo-based POI association Segmentation of Photo Streams o Split the stream whenever the time difference between two successive photos is large Distillation of Timed Visits <POI, start time, end time> Construction of Timed Paths Search Engine Technology o A sequence of Timed Visits5 Constructing POI Graph Given the set of timed paths, our goal is to aggregate the actions of many individual travelers into coherent itineraries while taking into consideration POI popularity. Undirected POI graph, G C (V=L C, E=L C L C ) with the following predicates: T(l L C ), the visit time at each POI l. Longest visit time for each user; take the 75 th percentile among all users T(e E), the median transit time between two POIs V(l L C ), the prize or value that an itinerary gets from visiting each POI l in L C. Defined as the product of popularity and visit duration of l (eliminates very short visit durations) Search Engine Technology 6 3

4 Itineraries and Orienteering Problem An itinerary is a path in the graph G C, where a node (POI) in the path may be visited more than once. Problem Instance: Let I be an itinerary; its prize V(I) is defined as the sum of prizes of the unique set of POIs (i.e., a POI s prize is counted only once even if it is visited multiple times) along the path. The time T(I) of the itinerary is the sum of visit times to the unique set of POIs along the path, plus the transit times along all edges on the path (including those that are traversed more than once). Objective (solved using Orienteering Problem Approximation): Find an itinerary in G C from s to t of cost (=time) at most B maximizing total node prizes. Note, B is typically whole days; s and t can be provided by the user Search Engine Technology Data Preparation Five popular cities were chosen: Barcelona, London, New York City (NYC), Paris, and San Francisco City #POIs #Timed Paths Sample POIs Barcelona 74 6,087 Museu Picasso, Plaza Reial London ,052 Buckingham Palace, Churchill Museum, Tower Bridge New York City 100 3,991 Brooklyn Bridge, Ellis Island Paris ,651 Tour Eiffel, Musee du Louvre San Francisco 80 12,308 Aquarium of the Bay, Golden Gate Bridge, Lombard Street Search Engine Technology 4

5 Example Itineraries for NYC Single day itinerary Two day itinerary Search Engine Technology 9 9 Ground Truth To compare our algorithmically constructed itineraries with reasonable baseline itineraries, we obtained itineraries provided by top tour bus companies for each city and considered them as ground truth itineraries City Barcelona London New York City Paris San Francisco Ground Truth Sources Search Engine Technology 10 5

6 User Study Design Summary Side-by-side evaluation comparing our itineraries to ground-truths Independent evaluation examining our itineraries in detail Side-by-side comparison IMP GT Questions? Which itinerary is bette POIs Transit times Visit times Independen t evaluation IMP Questions? Is the itinerary reasona POIs Transit times Visit times Search Engine Technology 11 Results Side by Side Evaluation Evaluation Metric: estimate the usefulness of the itineraries from two aspects, such as the overall utility of the itineraries and appropriateness of POIs Search Engine Technology 12 6

7 Conclusions We addressed the question of automatic generation of travel itineraries for popular touristic cities from large-scale rich media repositories with UGC (user generated content) Such repositories reveal the following patterns: What are the popular touristic points of interest? How long do tourists spend at each POI? How long does it take for tourists to transit between pairs of POIs? User studies comparing our itineraries to bus tour companies itineraries indicate that high quality itineraries can be automatically constructed from Flickr data Search Engine Technology 13 The Query Flow Graph Paolo Boldi, Francesco Bonchi, Carlos Castillo, Debora Donato, Aris Gionis, Sebastiano Vigna Yahoo! Research Barcelona, University of Milano CIKM

8 Session Query-Flow Graph ebay autotrader used fox vw s barcelona hotel barcelona rent t soccer barcelona soccer... barcelona barcelona fc Search Engine Technology 15 Methodology Partition overall query stream into user streams Divide user streams into logical sessions that pursue the same information need Logical sessions might be interleaving Signals: Timestamps of queries Edit distance between successive queries Result set similarity between successive queries more Transitions weighted by observed frequency Transitions further classified by type (next slide) Search Engine Technology 16 8

9 Transition Analysis Barcelona Luxury Barcelona hotels Brcelona Correct Specialize Generalize Parallel Move Barcelona F.C. Real Madrid Barcelona hotels Cheap Barcelona hotels Boldi, Bonchi, Castillo and Vigna, From Dango to Japanese Cakes : Query Reformulation Models and Patterns, Web Intelligence, 2009 (Best paper award) Notification: 18/Sep/ Search Engine Technology 17 Applications Intent mining: what are people really looking for when asking Q Related Query suggestions Based on most popular transitions out of current query s node When coupled with click information, can improve index quality and hence observed ranking: If session looks like XQQ QC (X: delimeter, Q: query, C:click) it means that the clicked document satisfied the information need of all previous queries Can associate the text of earlier queries with the clicked document, generating pseudo anchor text, so that in the future it will show up as a result for those preliminary queries Can expedite speed of information need satisfaction Originally proposed in Queries as anchors: selection by association, Einat Amitay, Adam Darlow, David Konopnicki, Uri Weiss, Hypertext Search Engine Technology 18 9

10 When no clicks are good news Carlos Castillo, Aris Gionis, Ronny Lempel, Yoelle Maarek Yahoo! Research Barcelona & Haifa SIGIR 2010 Industrial Track Interaction Mining for Search Behavioral signals are useful for measuring performance of retrieval systems Relevant results that satisfy the information need are clicked more often, visited (dwelled) for a longer time, lead to long-term engagement, etc. However, predicting user satisfaction accurately from search behavior signals is still an open problem Search Engine Technology 20 10

11 A (not-so-) Special Case When satisfying the user by impression, we observe a lower click-through rate (and higher abandonment) Search Engine Technology 21 Satisfaction by Impression Oneboxes and Direct Displays Oneboxes 1 and Direct Displays 2 (DD) are Very specific results answering (mostly) unambiguous queries with a unique answer directly on the SERP Displayed above regular Web results, due to their high relevance, and rendered in a differentiated format. Typical example: weather <city name> 1 : Google terminology 2 :Yahoo! terminology Search Engine Technology 22 11

12 Additional Satisfaction by Impression Examples Searches for specific stocks, movie screen times, sports results, package tracking (Fedex/UPS), etc. To the extreme, what about spell checking, arithmetic operations or currency conversion? Search Engine Technology 23 The Problem: Response Interpretation How can we distinguish between user satisfaction by impression and abandonment? Application: adjust click-based metrics for user satisfaction, avoiding mis-interpretations of CTR Search Engine Technology 24 12

13 Proposed Methodology General method Pick a class of users with a distinctive behavior Study changes in their aggregate response patterns Specific method Identify users who are Tenacious (a.k.a. pitbulls ) In queries with no DDs, display very low abandonment Will either reformulate the query or click on some results, do not let go Measure their abandonment on queries with DDs Search Engine Technology 25 Modeling users Session representation Search sessions are delimited by X Actions classes: queries (Q) and clicks (C) XQCQX means start, query, click, query, stop User representation Frequency of up to first two actions at start of session Tenacity = (XQQ+XQC)/(XQQ+XQC+XQX) In words: the frequency by which the user follows up the first query in a session by either a reformulation or a click on some result Search Engine Technology 26 13

14 (Preliminary) Experiments Segment sessions into logical goals Divide queries into two groups With direct-displays triggered above position 5 (DD) Without (NO-DD) Metric Find users with Tenacity NO-DD >= 80% (min. 10 sessions) For a DD-triggering query Q, measure (on enough tenacious users only) #abandonments / #impressions Tenacious abandonment as a good signal Ground truth user survey Ask humans do you think users querying for Q will be satisfied by impression by this DD? 1=never... 5=always Search Engine Technology 27 Weather DD Change in the tenacity of tenacious users Search Engine Technology 28 14

15 GOOD GOOD BAD BAD Change in the tenacity of tenacious users Search Engine Technology 29 GOOD 63% of bad cases 83% precision BAD Change in the tenacity of tenacious users Search Engine Technology 30 15

16 GOOD BAD Reference DD Change in the Search tenacity Engine of Technology tenacious users 31 GOOD GOOD BAD BAD Change in the Search tenacity Engine of Technology tenacious users 32 16

17 GOOD 71% of bad cases 84% precision BAD Change in the tenacity of tenacious users Search Engine Technology 33 17