Mining the Temporal Statistics of Query Terms for Searching Social Media Posts ICTIR 17 Amsterdam Oct. 1 st 2017 Jinfeng Rao Ferhan Ture Xing Niu Jimmy Lin
Task: Ad-hoc Search on Social Media domain Stream of Tweets A ranked list of tweets Interest Profiles Interest (User s Profile queries) (~topic) Example query MB001: BBC world servicestuff cut....
Background Challenges for Social Media Search Usually very short, 140 characters for tweets. Posts are written in a highly concise way, sometimes canbe quite noisy. Many abbreviations,misspellings, typos, emojis, hashtags,etc. Time is an important relevance signal Relevant posts are more likely to group together at the time shaking news happened. Example query MB001 from TREC 2011: BBC world service stuff cut distribution of relevant docs (ground truth) in below. x axis denotes the number of days prior to query time. the height of a bar denotes the number of relevant docs during that time interval.
Combine Lexical and Temporal Evidence Moving window, Dakka et al.tkde 12 [2] (pseudo trend) Kernel density estimation, Efron et al. SIGIR 14 [3] ˆf! (x) = 1 nx x xi! i K nh h i=0 Recurrent Neural Networks, Rao et al. NeuIR 17 [4] However, these work all require two-stage retrieval: Initial retrieval: estimate the ground truth distribution (pseudo trend). Second retrieval: rerank docs with the estimated pseudo trend. 0.00 0.05 0.10 0.15 0.20
Research Question Research question: can we make use of the temporalstatistics of query terms (term trends) to predict the ground truth? What is term trend? Term frequencies in the collection for each 5 minutes. An example of ground truth and term trends for query MB127 hagel nomination filibustered from TREC 2013 topic set. ground truth Strong correlation! term trends
Approach: Temporal Modeling via Regression ground truth term trends Goal: Approximate the ground truth (Y) by taking a weighted sum of all term trends (ft).
Term Importance Modeling Bursty terms can be more informative. We adopt entropy definition to measure the importance of terms. Given the counts of a particular term t (unigram/bigram) {c 1, c 2,, c n }, lower entropy = bursty term trend = moreimportant
Approach: Temporal Modeling via Regression Two questions in this non-linear regression modeling: Q1: How to model the weights of different query terms? Q2: How to differentiate the contribution from unigrams with bigrams? Q1 solution: exponential mapping from entropy to term weight Q2 solution: assume unigram weight u i, then bigram weight (1-u i ) where Ri is the difference between the maximum unigram entropy and maximum bigram entropy. Intuition: Ri > 0 => max(unigram_entropy) > max(bigram_entropy) => u i > 0.5
Approach: Temporal Modeling via Regression Problem reformulation: Objective Loss: which can be solved with gradient descent algorithm (more details in paper).
Combine Term Trend with Pseudo Trend Two ways to estimate the ground truth distribution: Document-level: pseudo trend through an initial retrieval Term-level: regression over term trends Combine term trend and pseudo trend in a linear ranking model:
Experimental Setup Topic set: TREC Microblog Track 2013 and 2014, total 115 topics. Collection: Tweets2013 (~243 million tweets) Metrics: Mean Average Precision (AP) and Precision at 30 (P30) Three data splits: Odd-even: odd numbered topics (57 topics) for training, even (58 topics) for testing Even-odd: switch train/test split Cross: 4-fold cross validation
Baselines 1. QL 2. Recency Prior, Li et al. CIKM 03 [1] 3. Moving Window, Dakka et al.tkde 12 [2] 4. Kernel Density Estimation (KDE), Efron et al. SIGIR 14 [3] Uniform-based weighting (IRDu) Score-based weighting (IRDs) Rank-based weighting (IRDr) Oracle (upper bound)
Main Results Conclusions: KDE with rank-based weights (IRDr) is the strongest baseline. Our approach (Reg-IRDr) significantly outperforms all baselines, and is even close to the upper bound in some splits.
Randomized Experiments Average improvement over QL baseline summarized over 30 random train/test splits.
Per-Topic Analysis Per-topic P30 improvement against the Query Likelihood (QL) and the best KDE baseline (IRDr).
Analysis of the Best-Performing Topic 144 How term trend signals help? red color for ground truth distribution green for pseudo trend estimated by the best KDE method (IRDr) blue for term trends. Conclusion:A combination of pseudo trend (KDE) and term trend (Our approaches) provides a more accurate estimationto the ground truth distribution.
Conclusion We are the first to study temporal statistics of query terms for social media search. Our learning to rank and regression model show this new signal is effective. For efficiency purpose, use our term trending modeling technique For effectiveness purpose, use the combination of pseudo trend and term trend modeling
Thanks for listening! Any question?
Reference 1. Xiaoyan Li and W. Bruce Cro. 2003. Time-Based Language Models. In CIKM. 469 475. 2. Wisam Dakka, Luis Gravano, and Panagiotis G. Ipeirotis. 2012. Answering General Time- Sensitive eries. TKDE 3. Miles Efron, Jimmy Lin, Jiyin He, and Arjen de Vries. 2014. Temporal Feedback for Tweet Search with Non-Parametric Density Estimation. In SIGIR. 33 42. 4. Jinfeng Rao, Hua He, Haotian Zhang, Ferhan Ture, Royal Sequiera, Salman Mohammed, and Jimmy Lin. 2017. Integrating Lexical and Temporal Signals in Neural Ranking Models for Social Media Search. In SIGIR Workshop on Neural Information Retrieval (Neu-IR)