Query Session Detection as a Cascade

Transcription

1 Query Session Detection as a Cascade Matthias Hagen Benno Stein Tino Rüb Bauhaus-Universität Weimar matthias.hagen@uni-weimar.de SIR 211 Dublin, Ireland April 18, 211 Hagen, Stein, Rüb Query Session Detection as a Cascade 1

2 It s quiz time! Introduction Motivation Hagen, Stein, Rüb Query Session Detection as a Cascade 2

3 It s quiz time! Introduction Motivation What is the user searching? paris hilton Hagen, Stein, Rüb Query Session Detection as a Cascade 2

4 Without context... Introduction Motivation paris hilton source: [ Hilton 3 Crop.jpg] Hagen, Stein, Rüb Query Session Detection as a Cascade 3

5 Introduction Motivation What if you knew the previous queries? paris hotels paris marriott paris hyatt paris hilton Hagen, Stein, Rüb Query Session Detection as a Cascade 4

6 Introduction Motivation What if you knew the previous queries? paris hotels paris marriott paris hyatt paris hilton sources: [ hotel paris 2.jpg] [ [ mk logo hiltonbrandlogo.jpg] Hagen, Stein, Rüb Query Session Detection as a Cascade 4

7 Introduction Motivation Query sessions: same information need The benefits Improved understanding of user intent Improved retrieval performance via session knowledge Hagen, Stein, Rüb Query Session Detection as a Cascade 5

8 Introduction Motivation Query sessions: same information need The benefits Improved understanding of user intent Improved retrieval performance via session knowledge The minor issue Users do not announce when querying for a new information need. Hagen, Stein, Rüb Query Session Detection as a Cascade 5

9 A typical query log Introduction Motivation User Query Click domain + Click rank Time 773 istanbul en.wikipedia.org :34: istanbul archeology :2: istanbul archeology :3: istanbul archeology :24:7 773 constantinople ::4 773 constantinople :1:2 773 hurling :3:1 773 hurling en.wikipedia.org :3:5 773 liam mccarthy cup :33:4 773 liam mccarthy cup :33: liam mccarthy cup starbets.ie :42:48 Hagen, Stein, Rüb Query Session Detection as a Cascade 6

10 Introduction Motivation How to determine the break points? User Query Click domain + Click rank Time 773 istanbul en.wikipedia.org :34: istanbul archeology :2: istanbul archeology :3: istanbul archeology :24:7 773 constantinople ::4 773 constantinople :1:2 773 hurling :3:1 773 hurling en.wikipedia.org :3:5 773 liam mccarthy cup :33:4 773 liam mccarthy cup :33: liam mccarthy cup starbets.ie :42:48 Hagen, Stein, Rüb Query Session Detection as a Cascade 7

11 The key is... Introduction The Problem Automatic query session detection Hagen, Stein, Rüb Query Session Detection as a Cascade 8

12 Introduction The Problem Automatic query session detection Usual technique Check for consecutive queries whether same/new information need. Example 773 istanbul :34:17 same 773 istanbul archeology :24:7 same 773 constantinople :1:2 new 773 hurling :3:5 Hagen, Stein, Rüb Query Session Detection as a Cascade 9

13 Typical features Introduction Related Work Temporal thresholds 5 minutes [Silverstein et al., 1999] 1 15 minutes [He and Göker, 2] 3 minutes [Downey et al., 27] user specific [Murray et al., 26] Lexical similarity n-gram overlap [Zhang and Moffat, 26] Levenshtein distance [Jones and Klinkner, 28] Semantic similarity Search results [Radlinski and Joachims, 25] ESA [Lucchese et al., 211] Hagen, Stein, Rüb Query Session Detection as a Cascade 1

14 Introduction Related Work Previous methods Observations Temporal thresholds: fast but bad accuracy Feature combinations: more accurate One of the best: Geometric method (time + lexical) [Gayo-Avello, 29] Hagen, Stein, Rüb Query Session Detection as a Cascade 11

15 Previous methods Introduction Related Work Observations Temporal thresholds: fast but bad accuracy Feature combinations: more accurate One of the best: Geometric method (time + lexical) [Gayo-Avello, 29] Shortcomings All features evaluated simultaneously runtime Geometric method ignores semantics accuracy Examples Subset test suffices hurling same hurling gaa Geometric method fails hurling same mccarthy cup Hagen, Stein, Rüb Query Session Detection as a Cascade 11

16 Cascading Method The Framework We address the shortcomings in a cascade... source: [ Hagen, Stein, Ru b Query Session Detection as a Cascade 12

17 Cascading Method The Framework... well... a small 4-step cascade source: [ Cascade 4 Tier GreenL.jpg] Hagen, Stein, Ru b Query Session Detection as a Cascade 13

18 Cascading Method The Framework... well... a small 4-step cascade Step 1: Subset tests & Step 2: Geometric method & Step 3: ESA similarity. Step 4: Search results source: [ Cascade 4 Tier GreenL.jpg] Basic Idea Increased feature cost (runtime) from step to step. Expensive features only if previous steps unreliable. Hagen, Stein, Ru b Query Session Detection as a Cascade 13

19 Cascading Method Simple string comparison Step 1: Subset tests Criterion Consecutive queries q and q in same session if q sub- or superset of q. Else: Goto Step 2. Example Remarks: Repetition, specialization, or generalization. Time gap = continuing a pending session. Repetition Specialization Generalization hurling same hurling same hurling gaa same hurling hurling gaa hurling Hagen, Stein, Rüb Query Session Detection as a Cascade 14

20 Cascading Method Step 2: Geometric method Combination of temporal and lexical features [Gayo-Avello, 29] For consecutive queries q and q f temp = maximum of and 1 t 24h t is time between q and q f lex = cosine similarity of 3- to 5-grams of q and s s is session of q Hagen, Stein, Rüb Query Session Detection as a Cascade 15

21 Cascading Method Step 2: Geometric method Combination of temporal and lexical features [Gayo-Avello, 29] For consecutive queries q and q f temp = maximum of and 1 t 24h t is time between q and q f lex = cosine similarity of 3- to 5-grams of q and s s is session of q 1. Criterion (original).8 Nearly identical queries at long temporal distance Same session Consecutive queries q and q in same session if ftemp 2 + f lex 2 1. Lexical similarity New session Different queries with no temporal distance Temporal similarity Hagen, Stein, Rüb Query Session Detection as a Cascade 15

22 Cascading Method Step 2: Geometric method Performs well on standard test corpus Lexical similarity.6.4 Lexical similarity Temporal similarity Temporal similarity Same session New session Hagen, Stein, Rüb Query Session Detection as a Cascade 16

23 Cascading Method Step 2: Geometric method... but has some problems on the edge Lexical similarity Temporal similarity Major problems Similar queries, time gap (upper left) Merely a matter of opinion Diff. queries, same semantics (lower right) Incorporate semantics Hagen, Stein, Rüb Query Session Detection as a Cascade 17

24 Cascading Method Step 2: Geometric method... but has some problems on the edge Lexical similarity Temporal similarity Major problems Similar queries, time gap (upper left) Merely a matter of opinion Diff. queries, same semantics (lower right) Incorporate semantics Criterion (adapted) Original geometric method if f temp <.8 or f lex >.4. Else: Goto Step 3. Hagen, Stein, Rüb Query Session Detection as a Cascade 17

25 Cascading Method Step 3: Explicit Semantic Analysis How ESA works [Gabrilovich and Markovitch, 27] Preprocessing tf idf -weighted inverted index of Wikipedia articles term-document matrix M For consecutive queries q and q f esa = cosine similarity of M T q and M T s s is session of q Criterion Consecutive queries q and q in same session if f esa.35. Else: Goto Step 4. Hagen, Stein, Rüb Query Session Detection as a Cascade 18

26 Cascading Method Step 4: Search results Even more semantics Idea Enrich the short query strings with the results of some web search engine. Criterion Consecutive queries q and q in same session iff they share at least one of the top 1 search results. Hagen, Stein, Rüb Query Session Detection as a Cascade 19

27 Cascading Method Step 4: Search results Even more semantics Idea Enrich the short query strings with the results of some web search engine. Criterion Consecutive queries q and q in same session iff they share at least one of the top 1 search results. Remark If q and q share no top 1 result, decision should be not sure. Hagen, Stein, Rüb Query Session Detection as a Cascade 19

28 Cascading Method Experimental Results That s the complete cascade Step 1: Subset tests & Step 2: Geometric method & Step 3: ESA similarity. Step 4: Search results source: [ Cascade 4 Tier GreenL.jpg] Hagen, Stein, Ru b Query Session Detection as a Cascade 2

29 Cascading Method Experimental Results That s the complete cascade Step 1: Subset tests & Step 2: Geometric method & Step 3: ESA similarity. Step 4: Search results source: [ Cascade 4 Tier GreenL.jpg] What about accuracy and performance? Hagen, Stein, Ru b Query Session Detection as a Cascade 2

30 Accuracy and runtime Cascading Method Experimental Results Accuracy on Gayo-Avello s corpus (11 queries, 2.7 per session) Precision Recall F-Measure (β = 1.5) Geometric Cascading Performance per step on Gayo-Avello s corpus affected F-Measure time factor Step % ms 1. Step % ms 2.5 Step 3 2.5% ms 3.4 Step 4.85% ms Hagen, Stein, Rüb Query Session Detection as a Cascade 21

31 Cascading Method Experimental Results Goal: high quality session test data Our own use case Sample sessions from the AOL log as test data. AOL log (cleaned): 35.4 million interactions from 47 users. Some figures Step 4 involved on 22.5% 8 million web queries 3 ms per search 1 month Hagen, Stein, Rüb Query Session Detection as a Cascade 22

32 Cascading Method Experimental Results Goal: high quality session test data Our own use case Sample sessions from the AOL log as test data. AOL log (cleaned): 35.4 million interactions from 47 users. Some figures Step 4 involved on 22.5% 8 million web queries 3 ms per search 1 month Way out Drop Step 4 and the sessions on which it would have been invoked Remaining sessions: F-Measure =.9755 Cleaned AOL log: 27 minutes Hagen, Stein, Rüb Query Session Detection as a Cascade 22

33 Conclusion Almost the end: The take-away messages! Hagen, Stein, Rüb Query Session Detection as a Cascade 23

34 Conclusion What we have done Results Cascading method Cheap features first Beats geometric Future Work Postprocessing for multi-tasking Postprocessing for goals/missions 3 step version: simple, fast, high quality sessions Hagen, Stein, Rüb Query Session Detection as a Cascade 24

35 Conclusion What we have (not) done Results Cascading method Cheap features first Beats geometric Future Work Postprocessing for multi-tasking Postprocessing for goals/missions 3 step version: simple, fast, high quality sessions Hagen, Stein, Rüb Query Session Detection as a Cascade 24

36 Conclusion What we have (not) done Results Cascading method Cheap features first Beats geometric 3 step version: simple, fast, high quality sessions Future Work Postprocessing for multi-tasking Postprocessing for goals/missions Thank you Hagen, Stein, Rüb Query Session Detection as a Cascade 24