Spectral Clustering X I AO ZE N G + E L HA M TA BA S SI CS E CL A S S P R ESENTATION MA RCH 1 6,

Transcription

1 Spectral Clustering XIAO ZENG + ELHAM TABASSI CSE 902 CLASS PRESENTATION MARCH 16,

2 Presentation based on 1. Von Luxburg, Ulrike. "A tutorial on spectral clustering." Statistics and computing 17.4 (2007): Ng, Andrew Y., Michael I. Jordan, and Yair Weiss. "On spectral clustering: Analysis and an algorithm." NIPS. Vol. 14. No Belkin, Mikhail, and Partha Niyogi. "Laplacian eigenmaps for dimensionality reduction and data representation." Neural computation 15.6 (2003):

3 Outline 1 Motivation 2 Preliminary 3 Graph Laplacian 4 Spectral Clustering algorithm 5 Three explanation 6 Practical Issue 3

4 1 Motivation # msg sent from you A B w a A you w ab w b B # msg sent to you 4

5 K-means v.s. Spectral Clustering 5

6 Why spectral clustering? A powerful tool which can produce good results if applied with care. It does not make strong assumptions on the form of the clusters. can be implemented efficiently even for large data sets, as long as we make sure that the similarity graph is sparse. Once the similarity graph is chosen, we just have to solve a linear problem, and there are no issues of getting stuck in local minima or restarting the algorithm for several times with different initializations. However, not quite a black box algorithm which automatically detects the correct clusters in any given data set. choosing a good similarity graph is not trivial, and spectral clustering can be quite unstable under different choices of the parameters for the neighborhood graphs. 6

7 Historical notes Donath and Hoffman (1973), first suggested to construct graph partitions based on eigenvectors of the adjacency matrix. Fiedler (1973) discovered that bi-partitions of a graph are closely connected with the second eigenvector of the graph Laplacian, and he suggested to use this eigenvector to partition a graph. Since then, spectral clustering has been discovered, re-discovered, and extended many times in different communities, see for example Pothen, Simon, and Liou (1990), Simon (1991), Bolla (1991), Hagen and Kahng (1992), Hendrickson and Leland (1995), Van Driessche and Roose (1995), Barnard, Pothen, and Simon (1995), Spielman and Teng (1996), Guattery and Miller (1998). A nice overview over the history of spectral clustering can be found in Spielman and Teng (1996). 7

8 Graph Clustering How do we cluster network nodes? Social network, communication network, or network built by measuring similarity or connection. Given a network, we want to find groups of nodes where there are a lot of connections between the members of the same group and few connections to the outside of the group. 8

9 2 Mathematical Preliminary Undirected Graph Input: G(V, E) Output: K clusters Maximize the number of withincluster connections Minimize the number of between-cluster connections 9

10 Clusters in a graph Adjacency matrix W Undirected Weighed graph G(V,E) d 1 = w 12 w 13 w

11 Clusters in a graph Degree matrix D Undirected Weighed graph G(V,E) d 1 0 d 2 d 3 d 0 4 d5 d A d7 11

12 Clusters in a graph = w 25 + w 46 w A = 4 Undirected Weighed graph G(V,E) w B 7 12

13 2. Constructing a graph 1. ε-neighborhood graph 2. k-nearest neighbor graph 3. fully-connected graph 13

14 ε-neighborhood graph 0.9 Connect all nodes whose pairwise distances are smaller than ε As the distances between all connected points are roughly of the same scale (at most ε), this graph is usually represented as an unweighted graph (weights are all set to 1 after connected). Suppose ε is 0.5. In this graph, the weights represent distance, not similarity. 14

15 k-nearest neighbor graph Connect vertex v i with vertex v j if v i is among the k-nearest neighbors of vertex v j Problem: v j may not be among the k-nearest neighbours of v i, which means it may be a directed graph Two solutions: 1. Connect vertex v i with vertex v j only when v i and v j are mutully among the k-nearest neighbors. 2. Connect vertex v i with vertex v j as long as one of v i and v j is among the k-nearest neighbors of the other. For k=2, v 1 s k-nearest neighbors are v 2 and v 3 v 2 s k-nearest neighbors are v 3 and v 4 15

16 Fully-connected graph Weighted adjacency matrix W Connect all points, weighted by a similarity function. Gaussian similarity function: S 12 S 13 S 14 S 15 S 16 S

17 Eigenvalue and eigenvector λ is called the eigenvalue of A, the corresponding u is called eigenvector. Au = λu A may have multiple eigenvalues. In the following slides, we assume the eigenvalues are sorted increasingly, so are their corresponding eigenvectors. λ 1 λ 2 λ 3 λ 4 17

18 3 Graph Laplacian Assume: Graph G is an undirected weighted graph with symmetric affinity matrix W (w ij 0), Degree matrix D Unnormalized Graph Laplacian is defined as L= D - W w 12 w 13 w d W D L = D - W d 2 d 3 0 d 4 d5 0 d 6 d d 1 w 12 w 13 -w d 2 d 3 d 4 d5 d 6 d7 18

19 3 Graph Laplacian L = D - W d w 21 w 12 w 13 -w 14 d 2 Indicator vector f f A = (1,1,1,1,0,0,0) T A w 31 d 3 4 -w 41 5 d 4 d5 6 7 d 6 d 7 Proof: 19

20 3 Graph Laplacian d w 21 w 31 d 2 L = D - W w 12 w 13 -w 14 d 3 d 4 d5 -w 41 Proof: L 1 = λ1 = 0 so, λ = 0 d 6 d 7 20

21 3 Graph Laplacian L = D - W d w 12 w 13 -w 14 2 w 23 w w 21 d 2 For example, if λ 1 = λ 2 = 0, then the graph is like 3 w 31 w 32 w d 3 4 -w 41 w 42 w d 4 d5 w 65 w 56 w 57 d 6 w 67 w 75 w 76 d

22 3 Normalized Graph Laplacian L = D - W d w 21 w 31 -w 41 w 12 w 13 -w 14 d 2 d 3 d 4 d5 d 6 d 7 Unnormalized graph laplacian: Normalized graph laplacian: 22

23 3 Normalized Graph Laplacian normalized graph laplacian: Properties: 23

24 3 Normalized Graph Laplacian normalized graph laplacian: Properties: 24

25 3 Normalized Graph Laplacian normalized graph laplacian: Properties: For example, if λ 1 = λ 2 = 0, then the graph is like

26 4 Unnormalized spectral clustering 26

27 4 Unnormalized spectral clustering W D L = D - W w 12 w 13 w d d 2 d d 4 d5 d 6 d d w 12 w 13 -w 14 d 2 d 3 d 4 d5 d 6 d 7 27

28 4 Unnormalized spectral clustering L = D - W d 1 w 12 w 13 -w 14 d 2 d 3 d 4 d5 d 6 d7 U = [u 1, u 2, u 3,,, u k ]

29 4 Unnormalized spectral clustering L = D - W d w 23 w w 21 w 12 w 13 -w 14 d 2 3 w 31 w 32 w d 3 4 -w 41 w 42 w d 4 d5 w 65 w 56 w 57 d 6 w 67 w 75 w 76 d7 Lu = λu u 1 is the indicator for cluster 1 u 2 is the indicator for cluster 2 U = [u 1, u 2, u 3,,, u k ] y 1 y 2 y 3 y 4 y 5 y 6 y K-means K Clusters 29

30 4 normalized spectral clustering 30

31 4 normalized spectral clustering 31

32 5 Three Explanations 1. Graph Cut point of view 2. Random walks point of view 3. Perturbation theory point of view 32

33 Spectral clustering as an approximation to graph partitioning Problem statement: Find a partition of the graph such that the edges between different groups have a very low weight (which means that points in different clusters are dissimilar from each other), and the edges within a group have high weight (which means that points within the same cluster are similar to each other). 33

34 Graph cut point of view mincut-1 For two not necessarily disjoint set A, B V, Mincut finds the partitions A 1,, A k which minimizes K=2 34

35 Graph cut point of view mincut-2 Problem: sensitive to outliers! Stoer+Wagner (1997) Solution: Balancing out the size of clusters. RatioCut (Hagen and Kahng, 1992) balances the number of vertices Ncut (Shi and Malik, 2000) balances the weight of edges. But it makes the mincut a NP hard problem! Wagner and Wagner (1993) 35

36 Graph cut point of view relaxed version APPROXIMATING NCUT LEADS TO NORMALIZED SPECTRAL CLUSTERING For k=2 Relax f to takes arbitrary values in R APPROXIMATING RATIOCUT LEADS TO UNNORMALIZED SPECTRAL CLUSTERING For k=2 Relax f to takes arbitrary values in R By trace minimization f is the second eigenvector of L rw, or equivalently the generalized eigenvector of Lu = λdu. By Rayleigh-Ritz theorem (e.g., see Section of Lu ẗkepohl, 1997), a minimizer of RatioCut is approximate by the 2 nd eigenvector of L. There is no guarantee whatsoever on the quality of the solution of the relaxed problem compared to the exact solution. See examples in Guattery and Miller (1998) 36

37 Random walk point of view Random walk: a stochastic process which randomly jumps from one vertex to another Problem statement Find a partition of the graph such that the random walk stays long within the same cluster and seldom jumps between clusters. Transition probability: Transition matrix: λ is an eigenvalue of L rw with eigenvector u if and only if 1 λ is an eigenvalue of P with eigenvector u. the largest eigenvectors of P and the smallest eigenvectors of L rw can be used to describe cluster properties of the graph. 37

38 Random walk, Ncut, and (normalized) Spectral Clustering When minimizing Ncut, we actually look for a cut through the graph such that a random walk seldom transitions from A to A and vice versa. On Spectral Clustering: Analysis and an algorithm, Ng A., Jordan M, Weiss Y. 38

39 perturbation Q: how eigenvalues and eigenvectors of a matrix A change if we add a small perturbation H, A := A+H. Ideal case: the between-cluster similarity is exactly 0 In a nearly ideal case where we still have distinct clusters, but the between-cluster similarity is not exactly 0, we consider the Laplacian matrices to be perturbed versions of the ones of the ideal case. Davis-Kahan theorem: the eigenspaces corresponding to the first k eigenvalues of the ideal matrix L and the first k eigenvalues of the perturbed matrix L are very close to each other, that is their distance is bounded by H /δ. If the perturbation H is small or the eigengap is large. 39

40 6 Practical Issue What similarity function? Which graph constructing method to use? 1. ε-neighborhood graph 2.k-nearest neighbor graph mutual k-nearest neighbor graph 3. fully-connected graph 40

41 6 Practical Issue Problem: we need to efficiently compute eigenvectors for large sparse matrix Methods: power method or Krylov subspace methods The convergence speed depends on the eigengap: The larger it is, the faster it converges Golub, Gene H., and Charles F. Van Loan. Matrix computations. Vol. 3. JHU Press,

42 6 Practical Issue (selection of k) 42

43 6 Practical Issue Which spectral clustering to use? To use the eigenvectors of L rw rather than those of L sym, because: 1. the eigenvectors of L sym are additionally multiplied with D 1 2, which might lead to undesired artifacts. 2. L sym does not have any computational advantages To use normalized rather than unnormalized spectral clustering 43

44 6 Practical Issue Why normalized spectral clustering is better? Two goals: 1 minimize the between-cluster similarity 2 maximize the within-cluster similarity For unnormalized spectral clustering (Ratiocut), only the first goal is targeted. For normalized spectral clustering(ncut), the second goal is also targeted. 44

45 Applications of spectral clustering Clustering semi-supervised learning (e.g. Chapelle, Sch ölkopf, and Zien, 2006 for an overview) manifold reconstruction (e.g., Belkin and Niyogi, 2003) Beyond partitioning graph drawing (Koren, 2005). In fact, there are many more tight connections between the topology and properties of graphs and the graph Laplacian matrices 45

46 Software 1. Python: scikit-learn 2. Matlab: Package for symmetric spectral clustering 46

47 References 1. Von Luxburg, Ulrike. "A tutorial on spectral clustering." Statistics and computing 17.4 (2007): Ng, Andrew Y., Michael I. Jordan, and Yair Weiss. "On spectral clustering: Analysis and an algorithm." NIPS. Vol. 14. No Belkin, Mikhail, and Partha Niyogi. "Laplacian eigenmaps for dimensionality reduction and data representation." Neural computation 15.6 (2003):