Interpreting Document Collections with Topic Models. Nikolaos Aletras University College London

Transcription

1 Interpreting Document Collections with Topic Models Nikolaos Aletras University College London

2 Acknowledgements Mark Stevenson, Sheffield Tim Baldwin, Melbourne Jey Han Lau, IBM Research

3 Talk Outline Introduction Topic Modelling Organising Document Collections with Topic Models Topic Coherence Topic Similarity Topic Labelling

4 Introduction Large amount of collections of documents available electronically

5 Introduction Large amount of collections of documents available electronically Unstructured

6 Introduction Large amount of collections of documents available electronically Unstructured No classification system

7 Introduction Large amount of collections of documents available electronically Unstructured No classification system Difficult to find specific information

8 Topic Models Unsupervised. Data-driven. Capturing the themes discussed within documents. Topics are probability distributions over words. Documents are represented as distributions over topics.

9 Topic Models Topic Model e.g. Latent Dirichlet Allocation (LDA; Blei et al., 2003) Topic-Word Matrix Topic-Doc Matrix

10 Topic - Word Matrix water, river, bank bank river water money interest loan Topic Topic bank, money, interest Topic representation: Top-N words with highest marginal probability.

11 Topic - Document Matrix water, river, bank Doc 1 Doc 2 Doc 3 Doc 4 Doc 5 Doc 6 Topic Topic bank, money, interest

12 Visualising Document Collections using Topic Models Topic Model Visualization Engine (TMVE; Chaney and Blei, 2012)

13 Visualising Document Collections using Topic Models Topic Model Visualization Engine (TMVE; Chaney and Blei, 2012)

14 Aim and Challenges Make topics more comprehensible and useful. Topic Coherence Topic Similarity Topic Labelling

15 Topic Coherence Topics need to present a coherent thematic subject. Why compute topic coherence? Decide which topics should be shown in topic browsers. Pre-processing step in topic labelling algorithms.

16 Topic Coherence Topics team, game, james, season, player, nba, play, knicks, coach, league model, wheel, engine, system, drive, front, vehicle, rear, speed, power privacy, andrews, elli, alexander, burke, zoo, information, chung, user, regan taylor, morris, camp, rom, elliott, romania, orange, even, gray, brantley

17 Topic Coherence Topics team, game, james, season, player, nba, play, knicks, coach, league model, wheel, engine, system, drive, front, vehicle, rear, speed, power privacy, andrews, elli, alexander, burke, zoo, information, chung, user, regan taylor, morris, camp, rom, elliott, romania, orange, even, gray, brantley

18 Topic Coherence Topics team, game, james, season, player, nba, play, knicks, coach, league model, wheel, engine, system, drive, front, vehicle, rear, speed, power privacy, andrews, elli, alexander, burke, zoo, information, chung, user, regan taylor, morris, camp, rom, elliott, romania, orange, even, gray, brantley

19 Topic Coherence Computing topic coherence: Average pairwise similarity between topic words (Newman et al., 2010). team game james season player nba play knicks coach league Sim(team, league)

20 Topic Coherence Computing Sim(wi,wj): Previous work: Pointwise Mutual Information (PMI; Newman et al., 2010). Log ratio of co-document frequency and document frequency in training corpus (Mimno et al., 2011).

21 Topic Coherence Compute topic coherence using distributional semantics (Aletras and Stevenson, 2013; IWCS). Topic words: Vectors in a semantic space. Topic word similarity: Cosine of the angles of the vectors.

22 Topic Coherence team game james season player nba play knicks coach league team game league Sim(team, game) = Cos(team, game) PMI

23 Topic Coherence Evaluation and Results 300 topics with human judgements of topic coherence. 20NG, NYT, Genomics. Spearman Correlation NYT 20NG Genomics Newman et al. (2010) Mimno et al. (2011) Dis. Sem

24 Topic Similarity Group similar topics together. Why? Decide which topics should be shown together in topic browsers.

25 Topic Similarity Topic similarity: Previous work: Compare per-topic word probability distributions: KL divergence (Li and McCallum, 2006; Wang et al., 2009; Newman et a., 2009, Kim and Oh, 2011) Jensen-Shannon Divergence (JSD; Kim and Oh, 2011) Cosine (He et al., 2009; Ramage et al., 2009; Kim and Oh, 2011) Log odds ratio (Chaney and Blei, 2012)

26 Topic Similarity Computing topic similarity: Average pairwise similarity of the words of two topics (Aletras and Stevenson, 2014; EACL). team game james season player nba play knicks coach league Sim(team, goal) world cup team soccer africa player south game match goal

27 Topic Similarity Computing Sim(w i,w j ): Word association PMI Distributional semantics Knowledge-based methods Explicit Semantic Analysis (ESA, Gabrilovich and Markovitch, 2007)

28 Topic Similarity Evaluation and Results 800 pairs of 50,100, 200 LDA topics with human judgements of topic similarity. NYT articles. Spearman Correlation Word Overlap JSD Cosine Chaney and Blei (2012) PMI Dis. Sem ESA SVR IAA

29 Topic Labelling Associate textual or image labels with topics. Why? Assist in the interpretation of the lists of words representing the topics.

30 Topic Labelling team, game, james, season, player, nba, play, knicks, coach, league

31 Topic Labelling NBA team, game, james, season, player, nba, play, knicks, coach, league

32 Topic Labelling NBA American Basketball team, game, james, season, player, nba, play, knicks, coach, league

33 Topic Labelling NBA American Basketball Basketball team, game, james, season, player, nba, play, knicks, coach, league

34 Topic Labelling NBA American Basketball Basketball team, game, james, season, player, nba, play, knicks, coach, league

35 Topic Labelling NBA American Basketball Basketball team, game, james, season, player, nba, play, knicks, coach, league

36 Topic Labelling using Text Assume a topic and a set of candidate labels, select the most appropriate. Previous work: Ranking candidate labels by combining word association measures, lexical features and an information retrieval technique into a supervised model (Lau et al., 2011).

37 Topic Labelling using Text Ranking labels using unsupervised graph-based methods (Aletras and Stevenson 2014; ACL). Retrieving text information Creating text graph Identifying important terms Scoring candidate labels

38 Topic Labelling using Text Retrieving text information TOPIC SEARCH ENGINE RESULTS METADATA team, game, james, season, player, nba, play, knicks, coach, league Metadata Title Fields: 1. LeBron James Stats 2. NBA basketball statistics 3. Cleveland Cavaliers 4. Fiba Basketball World Cup

39 Topic Labelling using Text Creating text graph w1 Edge weight: 1 - Unweighted Graph (PR) Normalised PMI - Weighted Graph (PR-NPMI) w2 w3 w4 w5 w6 w7 w8

40 Topic Labelling using Text Creating text graph Edge weight: 1 - Unweighted Graph (PR) Normalised PMI - Weighted Graph (PR-NPMI) w1 Identifying important terms w1 w3 w2 w2 w3 w4 w4 PageRank w5 w6 w7 w8 w5 w6 w7 w8

41 Topic Labelling using Text Creating text graph Edge weight: 1 - Unweighted Graph (PR) Normalised PMI - Weighted Graph (PR-NPMI) w1 Identifying important terms w1 w3 w2 w2 w3 w4 w4 PageRank w5 w7 w8 w5 w6 w6 Scoring Candidate Labels w7 Candidate Label: L = {w1, w2,, wm} Scoring Function: w8

42 Topic Labelling using Text Evaluation and Results 228 topics and 6K labels with human judgements of relevance (Lau et al., 2011). Blogs, books, news, Pubmed. Top-1 Average Rating (0-3) Blogs Books News Pubmed Lau et al. (2011)-U Lau et al. (2011)-S PR 2.05* 1.98* 2.04* 1.88** PR-NPMI 2.08* 2.01* 2.05* 1.90** Upper Bound

43 Topic Labelling using Images Identifying and ranking image labels using unsupervised graph-based methods (Aletras and Stevenson 2013; NAACL-HLT). Retrieving images Extracting textual and visual information Creating image graph Identifying important images

44 Topic Labelling using Images Retrieving text and visual information Text Features: Title and Link Metadata Fields TOPIC SEARCH ENGINE Top-20 Images team, game, james, season, player, nba, play, knicks, coach, league Visual Features: Bags of Visual Words (BOVW)

45 Creating candidate images graph Topic Labelling using Images Edge weight: Cosine of BOVW

46 Topic Labelling using Images Creating candidate images graph Ranking images PageRank Edge weight: Cosine of BOVW

47 Topic Labelling using Images Evaluation and Results 300 topics and 6K image labels with human judgements of relevance. News, Wikipedia articles Top-1 Average Rating (0-3) Random 1.79 Word Overlap 1.85 Google Image Search 1.89 PR-VIS 1.96 PR-VIS+Per(PMI) 2.00* Human Performance 2.24

48 Comparing Topic Representations Which topic representations are suitable within a document browser interface? What is the impact of different topic representations on human search effectiveness for a given query?

49 Comparing Topic Representations Retrieval Task (Aletras et al., 2014; JCDL, Aletras et al., 2015; JASIST). Aim: Identify as many documents relevant to a set of queries as possible within 3 minutes in a document collection organised using topic models.

50 Retrieval Task Document Collection: Reuters Corpus (Rose et al., 2002). 20 subject categories used as queries. 100K documents.

51 Reuters Category (Query) No. Docs Travel & Tourism 314 Domestic Politics (USA) 27,236 War - Civil War 16,615 Biographies, Personalities, People 2,601 Defence 4,224 Crime, Law Enforcement 10,673 Religion 1,477 Disasters & Accidents 3,161 International Relations 19,273 Science & Technology 1,042 Employment/Labour 2,796 Government Finance 17,904 Weather 1,190 Elections 5,866 Environment & Natural World 1,933 Arts, Culture, Entertainment 1,450 Health 1,567 European Commission Institutions 1,046 Sports 18,913 Welfare, Social Services 775

52 Retrieval Task Topic Modelling: LDA. Learned 100 topics. 84 topics after filtering out incoherent topics (Aletras and Stevenson, 2013b).

53 Retrieval Task Topic Browsing Systems: Topic Model Visualisation Engine (TMVE, Chaney and Blei, 2012). 3 topic representations: Keywords (Top-10) Textual Phrases Wikipedia article titles. Images Wikimedia images.

54 Retrieval Task Task: Two-step procedure. Given a query and a set of topics: 1. Identify all potentially relevant topics, 2. Identify relevant documents from a list of documents associated with the selected topics.

55 Retrieval Task Step 1: Identify relevant topics (Keywords).

56 Step 1: Identify relevant topics (Textual Phrases).

57 Step 1: Identify relevant topics (Images).

58 Retrieval Task Step 2: Identify relevant documents.

59 Retrieval Task Subjects and Procedure: 15 members of research staff and graduate students at the Universities of Sheffield, Melbourne and King s College. Each participant had to sign up to our on-line system. Participants had access to a personalised page where they could read instructions, see how many queries they have completed so far and perform a new query. Participants performed each of the 20 queries. Each query has been performed at least by 5 different participants.

60 Results Number of Retrieved Documents Topic Representation Total Keywords 1,086 Text 1,264 Image 1,115

61 Results Precision Topic Representation Average Keywords 0.59 Text 0.53 Image 0.56

62 Conclusions Post-processing the output of topic models can help building better exploratory search interfaces. Filtering-out junk topics. Grouping together similar topics. Assisting to topic interpretation by providing textual or image labels. If you are interested in the topic: Topic Models: Post-Processing and Applications Workshop in CIKM

63 Publications N. Aletras, T. Baldwin, J. H. Lau and M. Stevenson (2015). Evaluating Topic Representations for Exploring Document Collections. JASIST. N. Aletras and M. Stevenson (2014) Labelling Topics using Unsupervised Graph-based Methods. In ACL. N. Aletras and M. Stevenson (2014) Measuring the Similarity between Automatically Generated Topics. In EACL. N. Aletras, T. Baldwin, J. Lau and M. Stevenson (2014) Representing Topics Labels for Exploring Digital Libraries. In JCDL. N. Aletras and M. Stevenson (2013) Representing Topics Using Images. In NAACL-HLT. N. Aletras and M. Stevenson (2013) Evaluating Topic Coherence Using Distributional Semantics. In IWCS.