From Word Embeddings To Document Distances. Matt J. Kusner Yu Sun Nicholas I. Kolkin Kilian Q. Weinberger

Transcription

1 From Word Embeddings To Document Distances Matt J. Kusner Yu Sun Nicholas I. Kolkin Kilian Q. Weinberger

2 Goal: a distance between two documents?

3 Applications document classification multi-lingual document matching song identification

4 Word Embedding word2vec [Mikolov et al., 213] different from [Collobert & Weston, 28] [Mnih & Hinton, 29] word2vec is not deep! words trained on 1 billion words 3 million different words embedded R d

5 Word Embedding word2vec [Mikolov et al., 213] x i x j X 2 R d n words distance between words i and j: kx i x j k 2 is roughly their dissimilarity R d

6 Word Embedding word2vec [Mikolov et al., 213] Man King Woman Queen R d

7 How can we leverage these high quality word embeddings to compute document distances?

8 Word Mover s Distance cat run tree dog steamocean car fly race frog pattern oatpickle rock up rocket win bunnybaby

9 Goal?

10 Word Mover s Distance d d media media R d word embedding

11 Word Mover s Distance d d media media R d word embedding

12 Word Mover s Distance word mover s distance = d minimum word distance to transform mass d into d d media media R d word embedding

13 Word Mover s Distance d d media media R d word embedding

14 Word Mover s Distance d d media media R d word embedding

15 Word Mover s Distance d d media media R d word embedding

16 Word Mover s Distance d media j d T i R d word embedding

17 Word Mover s Distance d d 1/2 R d word embedding

18 Word Mover s Distance d d 1/2 R d word embedding

19 Word Mover s Distance d d 1/2 R d word embedding

20 Word Mover s Distance d WMD(d, d ), d 1/2 min T i,j=1 T ij kx i x j k 2 1/2 x j x k x i

21 Word Mover s Distance d WMD(d, d ) ),, d 1/2 min T i,j=1 T ij kx i x j k 2 s.t. T ij = d i 8i j=1 i=1 T ij = d j 8j

22 Remarks WMD(d, d ), min T ij kx i x j k 2 T s.t. s.t. i,j=1 j=1 i=1 T ij = d i F ij = d i T ij = d j i=1 F ij = d j 8i 8j 8j in CV this is the Earth Mover s Distance (EMD) [Rubner et al., 1998] an old optimal transport problem [Monge, 1781]

23 How well does WMD perform on document classification via k- nearest neighbors (k-nn)?

24 Classic Approaches bag-of-words campaign speech Washington TF-IDF LSI speech speech a a LDA [Salton & Buckley, 1988] [Deerwester et al., 199] [Blei et al., 23] topic distributions Civil War music politics sports politics topic guitar soccer Vicksburg Madonna football Washington speech

25 Results: k-nn test error % train inputs: BOW dim: k-nearest neighbor error bbcsport twitter recipe ohsumed classic reuters amazon Okapi BM25 [Robertson & Walker, 1994] TF-IDF [Jones, 1972] BOW [Frakes & Baeza-Yates, 1992] Componential Counting Grid [erina et al., 213] msda [Chen et al., 212] LDA [Blei et al., 23] LSI [Deerwester et al., 199] Word Mover's Distance news All hyper-parameters set with bayesopt.m [Gardner et al. 214]

26 Results: k-nn average error w.r.t. BOW Okapi BM25 TF-IDF BOW CCG msda LDA LSI.72 WMD

27 Computational Complexity min T F i,j=1 i,j=1 s.t. s.t. WMD(d, d ), j=1 i=1 TF ij kx i x j k 2 T ij = d i F ij = d i T ij = d j i=1 F ij = d j L with 2n constraints O(n 3 log n) 8i 8j 8j [ele & Werman, 29]

28 Computational Complexity min T F i,j=1 i,j=1 s.t. s.t. WMD(d, d ), j=1 i=1 TF ij kx i x j k 2 T ij = d i F ij = d i T ij = d j i=1 F ij = d j 8i 8j 8j approximations: [Rubner et al., 1998]; [Levina & Bickel, 21]; [Grauman & Darrell, 24]; [Shirdhonkar & Jacobs, 28]

29 Approximation 1 [Rubner et al., 1998] d d media media R d word embedding

30 Approximation 1 [Rubner et al., 1998] d d media media R d word embedding

31 d media Approximation 1 Word Centroid Distance WCD(d, d ), kxd Xd k 2 Xd Xd O(nd) [Rubner et al., 1998] media d R d word embedding

32 Faster Approximations for a random test input... distance twitter training input index amazon WCD RWMD WMD training input index

33 Approximation 2 min T F i,j=1 i,j=1 s.t. j=1 i=1 FT ij ij kx ii x jj k 22 T ij = d i F ij = d i T ij = d j i=1 F ij = d j 8i 8i 8j 8j

34 Approximation 2 min T s.t. i,j=1 j=1 i=1 T ij kx i x j k 2 T ij = d i T ij = d j 8i 8j D 1

35 Approximation 2 min T s.t. i,j=1 j=1 i=1 T ij kx i x j k 2 T ij = d i T ij = d j 8i 8j D 1 just a nearest-neighbor search!

36 Approximation 2 min T s.t. i,j=1 j=1 i=1 T ij kx i x j k 2 T ij = d i T ij = d j 8i 8j D 2 just a nearest-neighbor search!

37 Approximation 2 min T s.t. i,j=1 T ij kx i x j k 2 T ij = d i j=1 T ij = d j i=1 8i 8j min T s.t. i,j=1 T ij kx i x j k 2 T ij = d i j=1 T ij = d j i=1 8i 8j D 1 D 2 Relaxed Word Mover s Distance RWMD(d, d ), max(d 1,D 2 ) O(n 2 d)

38 Faster Approximations for a random test input... distance twitter training input index amazon WCD RWMD WMD training input index

39 Faster Approximations for a random test input... distance twitter training input index amazon WCD RWMD WMD training input index

40 Faster Approximations MD average knn error w.r.t. BOW WMD RWMD 1 c2 RWMD D 1 D 2 c1 WCD RWMD WMD

41 Other Embeddings

42 Conclusion Word Mover s Distance: media document distances from word embeddings Very accurate as it leverages high quality word2vec embedding average error w.r.t. BOW Okapi BM25 TF-IDF BOW CCG msda LDA LSI.72 WMD Fast through approximations WMD O(n 3 log n) WCD O(nd) RWMD O(n 2 d)

43 Code: Thank you. Questions?