Personalized Models of Search Satisfaction. Ahmed Hassan and Ryen White

Size: px

Start display at page:

Download "Personalized Models of Search Satisfaction. Ahmed Hassan and Ryen White"

Emma Cox
5 years ago
Views:

1 Personalized Models of Search Satisfaction Ahmed Hassan and Ryen White

2 Online Satisfaction Measurement Satisfying users is the main objective of any search system Measuring user satisfaction is essential for improving the system

3 Online Satisfaction Measurement Requesting feedback on searchers perceptions and search outcomes directly from users is not practical at a large scale

4 Online Satisfaction Measurement Automatically predict search satisfaction of users search tasks A search task is an atomic information need resulting in one or more queries (Jones and Klinkner, CIKM 08) Hiking; san francisco Hiking; san francisco bay area Ano nuevo state reserve Given a search task, predict whether the user has been successful or not

Online Satisfaction Measurement Previous work has focused on estimating satisfaction from implicit behavioral signals: Implicit measures to improve search (Fox et al.

5 Online Satisfaction Measurement Previous work has focused on estimating satisfaction from implicit behavioral signals: Implicit measures to improve search (Fox et al., TOIS 05) User behavior as a predictor of success (Hassan et al., WSDM 10) Predicting Different Types of Web Search Success (Ageev et al., SIGIR 11) Semi-supervised learning of success models (Hassan, SIGIR 12)

6 People Behave Differently When Satisfied People are different Satisfaction is a personal belief How users behave when they are satisfied can also differ Oliver, Richard L. Satisfaction: A behavioral perspective on the consumer. ME Sharpe, 2010

7 Individual Differences in Satisfaction/Dissatisfaction Is there a variance in behavior traditionally associated with search satisfaction? We study three behaviors commonly used to model satisfaction: Result Page Abandonment Search Task Length Click Dwell Time

8 Individual Differences in Satisfaction/Dissatisfaction Result Page Abandonment When users do not click on any of the search results returned for a query Abandonment is good Abandonment is bad

9 Num DSAT abandonments Individual Differences in Satisfaction/Dissatisfaction We plotted the number of good abandonment instances (SAT) against the number of bad abandonment instances (DSAT) (Data from Diriye et al., CIKM 12) 100 Abandonment (SAT vs. DSAT) 80 Users mostly dissatisfied when abandoning Users mostly satisfied when abandoning 0 Average over all Users Num SAT abandonments

10 Individual Differences in Satisfaction/Dissatisfaction Session/Task Length Number of queries the user issued during a search task Is a longer session considered a sign of learning or struggling? Learning Hiking; san francisco Hiking; san francisco bay area Ano nuevo state reserve Ano nuevo state reserve; mile Struggling Dinner near kendall square food kendall square cheap food kendall square kendall square food

Num. queries (DSAT) Average over all Users Individual Differences in Satisfaction/Dissatisfaction Session/Task Length Average number of queries per task for satisfied tasks (SAT) and dissatisfied

11 Num. queries (DSAT) Average over all Users Individual Differences in Satisfaction/Dissatisfaction Session/Task Length Average number of queries per task for satisfied tasks (SAT) and dissatisfied tasks (DSAT) (Data from Hassan et al., CIKM 11) 20 Num. of queries (SAT vs. DSAT) 15 fewer queries when satisfied & more queries when dissatisfied 10 5 fewer queries when dissatisfied & more queries when satisfied Num. queries (SAT)

12 Individual Differences in Satisfaction/Dissatisfaction Click Dwell Time Average duration of clicked page visits Long dwell time Learning? Long dwell time Trying to find useful information?

DSAT dwell time (secs) Average over all Users Individual Differences in Satisfaction/Dissatisfaction Click Dwell Time Average duration of clicked page visits during satisfied tasks (SAT) and

13 DSAT dwell time (secs) Average over all Users Individual Differences in Satisfaction/Dissatisfaction Click Dwell Time Average duration of clicked page visits during satisfied tasks (SAT) and dissatisfied tasks (DSAT) (Data from Hassan et al., CIKM 11) 350 Dwell time (SAT vs. DSAT) 300 long dwell time when dissatisfied long dwell time when satisfied SAT dwell times (secs)

14 Individual Differences in Satisfaction/Dissatisfaction There are large differences between users. Making generalizations about particular behaviors is risky (e.g., that abandonment is always good or bad)

15 Satisfaction Modeling Features for learning satisfaction/dissatisfaction models General Models Personalized models for groups of similar users Personalized models for individuals

16 Features for Learning Satisfaction Models Query Features Query length in terms of number of characters Query length in terms of number of words Query frequency Query click-through rate

17 Features for Learning Satisfaction Models Number of queries Session Features Number of pairs of queries with word overlap (stop words ignored) Total number of clicks Number of algorithmic, sponsored and answer clicks Number of spelling suggestion/related queries clicks Time in session so far Time to the first click Average, maximum, and minimum dwell time Average, maximum, and minimum time between queries Number of abandoned queries Average number of clicks per query Hiking; san francisco Hiking; san francisco bay area Ano nuevo state reserve Ano nuevo state reserve; mile Nature trails; san francisco China camp state park Fort funston

18 Features for Learning Satisfaction Models Search Result Page Features Number of instant answers on the SERP Number of advertisements on the SERP Result diversity: number of unique Web domains Result diversity: number of unique Topics

19 Features for Learning Satisfaction Models SearchTrail: Q 12s Click 38s Q 23s End Actions A = {Query, Click, Answer Click, Related Search, END} Patterns of Behavior Features number of transitions between every action pair a i a j for every a n A λ 1 λ 3 λ 4 number of transitions between triplets of actions a i a j a k for every a n A average time difference between every pair of actions a i a j λ 2 λ 8 λ 5 λ 6 λ 7 λ 9 (Hassan et al., WSDM 10)

20 Cohorts of Searchers A cohort is a group of searchers who share a common characteristic Not enough information to generate a personalized model for every individual user Use the profile of other similar users

21 Expertise Cohort Search behavior of expert searchers is different compared to novice searchers (Hölscher and G. Strube, WWW 00, White and Morris, SIGIR 07) Following previous work, we use advanced search syntax usage to find expert users (White and Morris, SIGIR 07)

22 Topical Interest Cohorts Searchers with similar topical interests have similar demographics (Hu et al., WWW 07) Demographics can influence search behavior (Weber and Castillo, SIGIR 10) Topical interests are inferred using the topics of clicked search results Topical labels are assigned to URLs using ODP (dmoz.org)

23 Search Engine Preference Cohorts Different engines have different user populations (comscore reports) Demographics can influence search behavior (Weber and Castillo, SIGIR 10) Different engines have different responses to queries resulting in different behaviors

24 Individuals and Cohorts Satisfaction models can be tailored for: Individuals Expertise cohort Topical interests cohorts Engine preference cohorts Other cohorts

25 Personalized Satisfaction Models Global data (all users) Personal data (individuals or cohorts) Learn a model using both general and personal data while maximizing performance on personal examples.

26 Personalized Satisfaction Models GlobalOnly: Global Data Personal Data Global Data Model Ignore personal data

27 Personalized Satisfaction Models PersonalOnly: Global Data Personal Data Personal Data Model Ignore global data

28 Personalized Satisfaction Models All: Global Data Personal Data Global Data Personal Data Model Combine global and personal data

29 Personalized Satisfaction Models Weighted: Global Data Global Data Model Personal Data Personal Data Combine global and personal data but assign larger weight to personal data

30 Personalized Satisfaction Models Re-Classify: Global Data Model Model Personal Data Build a classifier on global data and feed its output to the personal classifier as a feature

31 Personalized Satisfaction Models Prior: Global Data Model Prior on weights Personal Data Model Build a classifier on global data and use the feature weights as priors when training a classifier in the personal data

32 Accuracy Results Individual Searchers 76% 74% 72% 70% 68% 66% 64% 62% 60% 58% PersonalOnly* Weighted* Prior* GlobalOnly All Re-Classify* PersonalOnly method performs worse than GlobalOnly Limited number of data points in the individual cases Combining global and personal data performs best * Indicates statistically significant difference compared to the GlobalOnly baseline

33 Accuracy Results Expertise Cohort 83% 82% 81% 80% 79% 78% 77% 76% 75% 74% 73% GlobalOnly All PersonalOnly* Weighted* Re-Classify* Prior* PersonalOnly method achieves limited gain over the GlobalOnly When the global and the personal data are combined, we achieve much higher performance gains * Indicates statistically significant difference compared to the GlobalOnly baseline

34 Accuracy Results Topical Interests Cohorts 81% 80% 79% 78% 77% 76% 75% 74% 73% GlobalOnly All PersonalOnly* Weighted* Prior* Re-Classify* Similar trends as before PersonalOnly performs better than GlobalOnly Combining global and the personal data performs best * Indicates statistically significant difference compared to the GlobalOnly baseline

35 Accuracy Results Engine Preference Cohorts 81% 80% 79% 78% 77% 76% 75% 74% 73% 72% GlobalOnly All PersonalOnly* Weighted* Re-Classify* Prior* Same trend again PersonalOnly achieved higher gain Combining global and the personal data still performs best * Indicates statistically significant difference compared to the GlobalOnly baseline

36 % gain over GlobalOnly Effect of Cohort Size What effect does the size of the cohorts have on the performance of the models? 6.0% 5.0% 4.0% 3.0% 2.0% 1.0% 0.0% Large Cohort Global + Personal Gain Small Cohort Personal Only Gain Gain achieved on large cohort is more than gain achieved on smaller cohorts

37 % gain over GlobalOnly Effect of Cohort Size What effect does the type of the cohorts have on the performance of the models? 6.0% 5.0% 4.0% 3.0% 2.0% 1.0% 0.0% Experts Engine Topics Global + Personal Gain Personal Only Gain Gain achieved on the expertise and engine preference cohorts is more than gain achieved on the topical interest cohorts

38 Conclusions Satisfaction is a personal belief Impact of satisfaction on search behavior varies on individual basis Tailored models of search satisfaction learned from behavioral signals Personalization can be applied for individual searchers or searcher cohorts Personalized satisfaction models exceed the performance of global methods

39 Thanks! Ahmed Hassan

A Task Level Metric for Measuring Web Search Satisfaction and its Application on Improving Relevance Estimation

A Task Level Metric for Measuring Web Search Satisfaction and its Application on Improving Relevance Estimation Ahmed Hassan Microsoft Research Redmond, WA hassanam@microsoft.com Yang Song Microsoft Research