Dynamic Embeddings for User Profiling in Twitter Shangsong Liang 1, Xiangliang Zhang 1, Zhaochun Ren 2, Evangelos Kanoulas 3 1 KAUST, Saudi Arabia 2 JD.com, China 3 University of Amsterdam, The Netherlands
Overview The Task Background and Related Work Our Method Dynamic User and Word Embedding Model (DUWE) Streaming Keyword Diversification Model (SKDM) Experiments Conclusion 2
The Task Input: A stream of tweets generated across the time Twitter Users Tweets over time Output: A set of keywords to profile the user at different point in time Given a user at time t Sport Food 3
The Task Twitter Users Tweets over time Sport Food Relevant Given a user at time t Diversified Dynamic 4
Background of User Profiling Problem Expert finding task at TREC 2005 enterprise track Given documents which describes expert candidates, answer a query with a sorted name list in a specific domain, uncovering associations between people and topics A generative language modeling approach in Balong et al (2007) Works on a Static document collection Assumes users profiling results are unchanged Need Dynamic User Profiling 5
Dynamic User Profiling Approaches ExperTime (Rybak et al 2014) A probabilistic model for learning how personal research interests evolve (Fang and Godavarthy 2014) 6
Limitations of Current User Profiling Methods Treat words as atomic units leading to a vocabulary mismatch that harms performance Represent words and users in disjoint vocabulary spaces making it difficult to measure the similarity between users and words when constructing the profile Can words and users be embedded in the same semantic space? Can their embedding be modeled in the dynamic environment? 7
Related Work in Dynamic Topic Models and Dynamic Embedding Dynamic Topic Models: modeling dynamic user interests Topic over time model (Wang et al. KDD 2006) Topic tracking model (Iwata et al. IJCAI 2009) Dynamic user clustering topic model (Liang et al. KDD 2016), etc None of them is for user profiling Dynamic Word Embedding Dynamic word embedding by separating data into time bins, and apply word2vec within each bin (Kim et al. 2014, Hamilton et al. 2016) Or based on Bayesian skip-gram model (Bamler and Mandt, 2017) All of them are for words only but not for users All of them are not for user profiling 8
Overview The Task Background and Related Work Our Method Dynamic User and Word Embedding Model (DUWE) Streaming Keyword Diversification Model (SKDM) Experiments Conclusion 9
Our Approach Dynamic User and Word Embedding Model (DUWE) Infer both users and words embeddings over time in the same semantic space Enable to measure the similarities between users and words embeddings Streaming Keyword Diversification Model Retrieve relevant keywords to profile users current interests over time Diversify the returned relevant keywords such that the keywords can cover all aspects of the users interests 10
Dynamic User and Word Embedding User Diffusion p(u t U t 1 ) / N (U t 1, 2 t 1I) N(0, 2 0 I) Observed cooccurrence of words at t-1 z t 1 y t 1 z t y t n + t 1 m + t 1 n + t m + t Observed user-word pairs at t-1 v t 1 u t 1 v t u t V U t 1 V U t Word representation at t-1 β t 1 α t 1 β t α t User representation at t Word Diffusion p(v t V t 1 ) / N (V t 1, 2 t 1I) N(0, 2 0 I) 11
Diffusion of user representation p(u t U t 1 ) / N (U t 1, 2 t 1I) N(0, 2 0 I) Gaussian Prior According to Kalman filtering, we define the variance of transition kernel for a user embedding from t-1 to t A. F F measuring the word distribution changes from previous time step t-1 to the current time step t for user u 12
Diffusion of word representation p(v t V t 1 ) / N (V t 1, Gaussian Prior 2 0 I) According to Kalman filtering, we define the variance of transition kernel for a word embedding from t-1 to t 2 t 1I) N(0, A. F F measuring the word distribution changes from t-1 to the current time step t 13
DUWE model inference Apply the skip-gram filtering for the inference (Bamler et al. 2017) and the variational inference algorithm to obtain the embeddings Posterior distribution over and conditional on the statistics information and as follows: positive and negative indicator matrices for all user-to-word pairs positive and negative indicator matrices for all word-to-word pairs where we have: model transition for users model transition for words skip-gram model for words skip-gram model for user and words 14
Streaming Keyword Diversification Model generating top-k relevant and diversified keywords for profiling users interests at time t. 15
Overview The Task Background and Related Work Our Method Dynamic User and Word Embedding Model (DUWE) Streaming Keyword Diversification Model (SKDM) Experiments Conclusion 16
Experimental Setup Datasets 1,375 users randomly sampled from Twitter 3.78 million tweets posted by the users from the beginning of their registrations up to May 31, 2015 Two types of Ground Truth: One for evaluating Relevance-oriented (RGT) performance and another for evaluating Diversity-oriented (DGT) performance. Evaluation Metrics Relevance: Pre (Precision), NDCG, MRR, MAP Their semantic version of the metrics, denoted as Pre-S, NDCG-S, MRR-S, MAP-S Diversity: Pre-IA (Intent-Aware Precision), α-ndcg, MRR-IA, MAP-IA 17
Experimental Setup Baselines Non-dynamic Embedding Models Skip-Gram Model, i.e., word2vec Model (SGM) Distributed Representations of Documents (DRD) Dynamic Traditional Profiling Model Predictive Language Model (PLM) Dynamic Topic Model User Clustering Topic model (UCT) Dynamic Embedding Models Dynamic Independent Skip-Gram model (DISG) Dynamic Pre-initialized Skip-Gram model (DPSG) Dynamic Independent Distributed Representations of documents (DIDR) Dynamic Pre-initialized Distributed Representations of documents (DPDR) 18
Overall Performance Average relevance performance on time periods of each month 19
Overall Performance Diversity performance on time periods of each month 20
An Example User s Dynamic Profiling Results over Time Top-6 keywords of an example user s dynamic profile, whose interests cover a number of aspects and dramatically change over time, from Sport, fitness, kitchen, exercise, to education. 21
Relevance and diversity performance over time Relevance performance over time Diversity performance over time 22
Performance w.r.t. embedding dimensionality 23
Overview The Task Background and Related Work Our Method Dynamic User and Word Embedding Model (DUWE) Streaming Keyword Diversification Model (SKDM) Experiments Conclusion 24
Conclusions Study the problem of dynamic user profiling in Twitter Propose a Dynamic User and Word Embedding model (DUWE) Propose a Streaming Keyword Diversification Model (SKDM) Evaluate the performance of the proposed models in real dataset, Twitter 25
Thank you for your attention! Our paper at http://www.kdd.org/kdd2018/accepted-papers/view/dynamicembeddings-for-user-profiling-in-twitter Lab of Machine Intelligence and knowledge Engineering (MINE): http://mine.kaust.edu.sa/