无监督学习中的选代表和被代表问题 - AP & LLE 张响亮. Xiangliang Zhang. King Abdullah University of Science and Technology. CNCC, Oct 25, 2018 Hangzhou, China

Transcription

1 无监督学习中的选代表和被代表问题 - AP & LLE 张响亮 Xiangliang Zhang King Abdullah University of Science and Technology CNCC, Oct 25, 2018 Hangzhou, China

2 Outline Affinity Propagation (AP) [Frey and Dueck, Science, 2007] Locally Linear Embedding (LLE) [Roweis and Saul, Science, 2000] 2

3 Affinity Propagation [Frey and Dueck, Science 2007] 3

4 Affinity Propagation [Frey and Dueck, NIPS 2005] We describe a new method that, for the first time to our knowledge, combines the advantages of model-based clustering and affinity-based clustering. Mixing coefficient Component 4

5 Clustering: group the similar points together Minimize S k 2 m= 1 Sx ÎC ( xi - µ m) i m Minimize S k 2 m= 1 Sx ÎC ( xi - µ m) i m where µ = m 1 C m S x i Î C m x i where µ = { x x ÎC } m i i m K-medoids K-medians

6 Inspired by greedy k-medoids Data to cluster: The likelihood of belong to the cluster with center is the Bayesian prior probability that is a cluster center The responsibility of k-th component generating x i Assign x i with center s i Choose a new center

7 Understanding the process Message sent from x i to each center/exemplar: preference to be with each exemplar Hard decision for cluster centers/exemplars Introduce: availabilities, which is sent from exemplars to other points, and provides soft evidence of the preference for each exemplar to be available as a center for each point

8 The method presented in NIPS 05 Responsibilities are computed using likelihoods and availabilities Availabilities are computed using responsibilities, recursively Affinities 8

9 Interpretation by Factor Graph is the index of the exemplar for Should not be empty with a single exemplar Constraints: Objective function is An exemplar must select itself as exemplar 9

10 Input and Output of AP in Science07 Preference (prior) 10

11 AP: a message passing algorithm 11

12 Iterations of Message passing in AP 12

13 Iterations of Message passing in AP 13

14 Iterations of Message passing in AP 14

15 Iterations of Message passing in AP 15

16 Iterations of Message passing in AP 16

17 Iterations of Message passing in AP 17

18 Iterations of Message passing in AP 18

19 Iterations of Message passing in AP 19

20 Summary AP Xiangliang Zhang, KAUST CS340: Data Mining 20

21 Extensive study of AP 21

22 Outline Affinity Propagation (AP) [Frey and Dueck, Science, 2007] Locally Linear Embedding (LLE) [Roweis and Saul, Science, 2000] 22

23 Locally Linear Embedding (LLE) [Roweis and Saul, Science, 2000] Saul and Roweis. Think globally, fit locally: unsupervised learning of low dimensional manifolds. JMLR

24 LLE - motivations 24

25 Inspired by MDS Multidimensional Scaling (MDS), find embedding of objects in low-dim space, preserving pairwise distance Given pairwise similarity Embedding to find Eliminate the need to estimate pairwise distances between widely separated data points? 25

26 LLE general idea Locally, on a fine enough scale, everything looks linear Represent object as linear combination of its neighbors Assumption: same linear representation will hold in the low dimensional space Find a low dimensional embedding which minimizes reconstruction loss 26

27 LLE matrix representation 1.Select k nearest neighbors 2.Reconstruct x i by its K nearest neighbors Find W by minimizing e (W) = å x å i 2 i - w ijx j j 27

28 LLE matrix representation 3.Need to solve system Y = W*Y Find the embedding vectors Y i by minimizing: e (Y) = å Y where M = (I - s.t. and N å Y = i i i - å W Y W) 1 N T å Yi Yi = N i i j ij T j 2 (I - W) = å å M (Y Y ) 0 (centered on the origin) N i N I (with unit convariance) j ij i j 28

29 LLE algorithm summary 1. Find k nearest neighbors in X space 2. Solve for reconstruction weights W w = 1 C T -1 C where C jk j T = (x -h ) (x -h ), k and h is one of j x's K nearest neighbors 3. Compute embedding coordinates Y using weights W: Create a sparse matrix M = (I-W) T (I-W) Set Y to be the eigenvectors corresponding to the bottom d non-zero eigenvectors of M 29

30 Continuing, SNE Allows many-to-one mappings in which a single ambiguous object really belongs in several disparate locations in the low-dimensional space, while LLE makes one-to-one mapping. p j i is the asymmetric probability that i would pick j as its neighbor Gaussian Neighborhood in original space q j i is induced probability that point i picks point j as its neighbor Gaussian Neighborhood in lowdim space 30

31 Continuing, t-sne uses a Student-t distribution (heavier tail) rather than a Gaussian to compute the similarity between two points in the low-dimensional space symmetrized version of the SNE cost function with simpler gradients 31

32 LLE - Follow up work [Chen and Liu, 2011] LLE output strongly depends on selection of k Jing Chen and Yang Liu. Locally linear embedding: a survey. Artificial Intelligence Review (2011) 32

33 [Ting and Jordan, 2018]

34 Thank you for your attention! Lab of Machine Intelligence and knowledge Engineering (MINE):