Modeling Rich Interac1ons in Session Search Georgetown University at TREC 2014 Session Track Jiyun Luo, Xuchu Dong and Grace Hui Yang Department of Computer Science Georgetown University
Introduc:on Session search Document retrieval for an en:re search session. TREC Session Track provides log data which records A sequence of query changes q 1,q 2 q n- 1,q n The ranked list for each past query Document clicked informa:on and dwell :me. TREC 2014 Session Track: RL1 using the last query of a session RL2 using any informa:on in current session RL3 using informa:on from other sessions We use: ClueWeb12 Category A as our corpus 2
Outline Introduc:on Methods and Approaches Ad- hoc Retrieval Model (Ad- hoc) Query Change Retrieval Model (QCM) Weighted QCM User- Click Model Clustering Session Performance Predic:on and Replacement Submissions Evalua:on Result Conclusion 3
Ad- hoc Retrieval Model (Ad- hoc) Mul:nomial Language Modeling + Dirichlet Smoothing. Term weight P(t d) as: μ is the Dirichlet smoothing parameter, and is set = 5000. 4
Query Change Retrieve Model (QCM) Idea: Query Change is an important form of user feedback Dongyi Guan, Sicong Zhang, and Hui Yang. 2013. U:lizing query change for session search. (SIGIR '13). Defining query change Δq i as the syntac:c edi:ng changes between two adjacent queries: Δq i Δq i = Added term ; Removed term Δq ; Theme term q i q i 1 + i Table 1 A example of Query Change qtheme Session Queries Query Change Q theme Q 1 = hydropower efficiency +Δq 2 = environment hydropower session 52 Q 2 = hydropower environment - Δq 2 = efficiency Q 3 = hydropower damage +Δq 3 = damage hydropower - Δq 3 = environment 5
Query Change Retrieve Model (QCM) The relevance score Increase between weights one query q i and a document d is calculated by: for theme terms Increase weights for novel added terms Score(q i, d) = log P(q i d)+αw Theme βw Add,In +εw Add,Out δw Remove Current reward/ relevance score Decrease weights for old added terms Decrease weights for removed terms 6
Query Change Retrieve Model (QCM) The relevance score between one query q i and a document d is calculated by: Score(q i, d) = log P(q i d)+αw Theme βw Add,In +εw Add,Out δw Remove The QCM model combines all queries in a session with a discount factor Υ: n i=1 Score qcm (q 1..n, d) = γ n i Score(q i, d) 7
Weighted QCM Weighted QCM combines queries based on query quality which is indicated by user click Strong SAT- Click a clicked document with dwelled :me >= 30 seconds Weak SAT- Click a clicked document with dwell :me >= 10 seconds and< 30 seconds 8
Weighted QCM Weighted QCM combines queries based on query quality which is indicated by user click Strong SAT- Click a clicked document with dwelled :me >= 30 seconds Weak SAT- Click a clicked document with dwell :me >= 10 seconds and< 30 seconds!"#$%!"#$!!..!,! =!"#$% (!!,!) +!!!"#$% (!!,!)!!!!!""#!!!!"# The good query set: Queries bringing at least one SAT- Click + the current query The bad query set: Queries bringing no SAT- Click 9
User- Click Model We boost a document s ranking score, if it is SAT- Clicked by users Session Level User- Click Model for RL2 score from QCM model boost from Session level User- Click model Ψ points for a Strong SAT- Click, θ points for a Weak SAT- Click, sum up for the whole session normaliza1on to (0,1) 10
User- Click Model Topic Level User- Click Model for RL3 boost from Topic level User- Click model similar to session level User- Click model, however calcula:on is done for the whole session cluster A session cluster is a set of sessions that sharing similar search topics 11
Clustering Topic ID is not obtainable in real search prac:ce. cluster sessions by comparing queries similarity Ø Convert all queries in one session to a term vector Ø Assign idf value as weight to each dimension Ø Cluster sessions based on the Euclidean distance of these vectors We use K- means clustering algorithm and set K = 60 12
Session Performance Predic1on and Replacement For sessions that share similar search topics predict their performance replace bad sessions results with good sessions Predict session performance Extract several features (n) from the sessions Rank sessions by formula:!"#$%!! = 1!(! #!!"!!"!!#$%!!!"#$!%&$'(!!! = TRUE! )!!!!..! 13
Session Performance Predic1on and Replacement Features Table Table&2&Features&Extracted&for&each&Session&! Feature F 1 F 2 F 3 F 4 F 5 F 6 F 7 F 8 Definition Search intent is comparison No user-click in session s!!"#$$ 5s.!!"#$$!!"!!h!!!"#!!"!!"#$$!time in a session. # of unique terms in session s 20. (!)!!"#$$_!"#_!"#!$ <!!"#$$_!"#_!"#!$ 2 Session s does not contain the most frequent search term in T(s). # of unique terms in session s 6!!!(!) #!!"!!"#!!"#!$%!!"!!"!!#$%!! #!!"!!"#!!"#!$%!!"!!"!!#$%!! <!(!) * T(s) means a session cluster including session s 14
Outline Introduc:on Methods and Approaches Ad- hoc Retrieval Model (Ad- hoc) Query Change Retrieval Model (QCM) Weighted QCM User- Click Model Clustering Session Performance Predic:on and Replacement Submissions Evalua:on Result Conclusion 15
RL1 RL2 RL3 Our Submissions GUS14RUN1 GUS14RUN2 GUS14RUN3 Weighted QCM (ω=0.65) Session Level User- Click Model Weighted QCM (ω=0.65) Topic Level User- Click Model Ad- hoc Retrieval Model Weighted QCM (ω=0.8) Session Level User- Click Model Weighted QCM (ω=0.8) Topic Level User- Click Model Weighted QCM (ω=0.8) Topic Level User- Click Model using topic ids Session Performance Predic:on and Replacement RL3 in RUN1 and RUN2 using session clusters based on query similarity RL3 in RUN3 using session cluster based on topic id Why? similar queries leads to similar retrieval list in our system. Not useful when apply session replacement strategy 16
Evalua1on Results GUS14RUN1 GUS14RUN2 GUS14RUN3 Max Med ndcg@10 P@10 ndcg@10 P@10 ndcg@10 P@10 ndcg@10 P@10 ndcg@10 P@10 RL1 0.2053 0.378 0.2053 0.378 0.2053 0.378 0.3890 0.629 0.1549 0.348 RL2 0.2458 0.426 0.2482 0.427 0.2482 0.427 0.4865 0.712 0.1626 0.372 RL3 0.2443 0.423 0.2458 0.424 0.2580 0.429 0.5111 0.744 0.1790 0.404 Ø 2 nd rank in task RL1, 1 st rank in task RL2 and RL3 Adjus:ng term weight based on query change is effec:ve Combining queries in a session is useful for Session Track User- Click is effec:ve to predicate relevance Ø A small performance drop from RL2 to RL3 in RUN1 and RUN2 cluster sessions based on query similarity may work, however need more work to refine it Ø A small increase from RL2 to RL3 in RUN3 For sessions sharing same search topics, replacing poor sessions results using good sessions is prac:cal. 17
Conclusion Achieve 20.9% increase from RL1 to RL2 by u:lizing query change feedback user click feedback Achieve 4% increase from RL2 to RL3 by Topic level User- Click Model Session performance predic:on and replacement
Thanks! Jiyun Luo, Xuchu Dong and Grace Hui Yang Department of Computer Science Georgetown University 19