Machine Learning for Data Science (CS4786) Lecture 11

Transcription

1 Machine Learning for Data Science (CS4786) Lecture 11 Spectral Clustering Course Webpage :

2 Survey

3 Survey

4 Survey

5 Competition I Out! Preliminary report of 1-2 pages due Oct 4th Form your groups Download data and familiarize yourself Jot down preliminary ideas In 1/2 page mention each group members contribution so far Competition closes Oct 27th

6 T ELL ME S PECTRAL WHO YOUR C LUSTERING FRIENDS ARE... Input: Similarity matrix A Ai,j = Aj,i > 0 indicates similarity between elements xi and xj Cluster nodes in a graph. Example: Ai,j = exp( d(xi, xj )) Analysis of social network data. A is adjacency matrix of a graph

7 SPECTRAL CLUSTERING 1 if (i, j) 2 E Input: Similarity matrix A i,j = A 0 otherwise A i,j = A j,i > 0 indicates similarity between elements x i and x j n A n Example: A i,j = exp( d(x i, x j )) A is adjacency matrix of a graph

8 SPECTRAL CLUSTERING Input: Similarity matrix A A i,j = A j,i > 0 indicates similarity between elements x i and x j L = D - A nx D i,i = Example: A i,j = exp( d(x i, x j )) j=1 A i,j A is adjacency matrix of a graph

9 SPECTRAL CLUSTERING Input: Similarity matrix A A i,j = A j,i > 0 indicates similarity between elements x i and x j Cut(c) 1 2 c> Lc Minimize c > Lc s.t. c? 1 Approximately minimize cut Example: A i,j = exp( d(x i, x j )) A is adjacency matrix of a graph

10 SPECTRAL CLUSTERING ALGORITHM (UNNORMALIZED) 1 Given matrix A calculate diagonal matrix D s.t. D i,i = n j=1 A i,j 2 Calculate the Laplacian matrix L = D A 3 Find eigen vectors v 1,...,v n of L (ascending order of eigenvalues) 4 Pick the K eigenvectors with smallest eigenvalues to get y 1,...,y n R K 5 Use K-means clustering algorithm on y 1,...,y n y 1,...,y n are called spectral embedding

11 What is the Embedding? Map each node in V to R K Nodes lightly connected are farther Lets see some examples

12 Examples 4 1 1D ,2,3 4,5, D 1,2, D x y z

13 Examples D D D

14 More Examples

15 SPECTRAL CLUSTERING (UNNORMALIZED) Min-cut on a graph can be efficiently computed Why bother with the approximate algorithm Is cut even a good measure?

16 RATIO CUT Why cut is perhaps not a good measure? Fixes? Perhaps Ratio Cut CUT(C 1, C 2 ) 1 C 2 C C 2 C Set c i = 1 if i C 1 C 1 1C 2 otherwise 4 Verify that c Lc = n Ratio Cut and c 2 = n (and c 1) Relaxed solution is same as Unnormalized Spectral clustering 7

17 RATIO CUT Why cut is perhaps not a good measure? Fixes? Perhaps Ratio Cut CUT(C 1, C 2 ) 1 C 2 C C 2 C Set c i = 1 if i C 1 C 1 C 2 otherwise Verify that c Lc = n Ratio Cut and c 2 = n (and c 1) Relaxed solution is same as Unnormalized Spectral clustering

18 NORMALIZED CUT Normalized cut: Minimize sum of ratio of number of edges cut per cluster and number of edges within cluster NCUT = j CUT(C j ) Edges(C j ) Example K = 2 Edges(C i ) = degree(c i )= X t2c i D t,t CUT(C 1, C 2 ) 1 Edges(C 1 ) + 1 Edges(C 2 ) This is an NP hard problem!

19 NORMALIZED CUT Normalized cut: Minimize sum of ratio of number of edges cut per cluster and number of edges within cluster NCUT = j CUT(C j ) Edges(C j ) Example K = 2 Edges(C i ) = degree(c i )= X t2c i D t,t CUT(C 1, C 2 ) 1 Edges(C 1 ) + 1 Edges(C 2 ) This is an NP hard problem!... so relax

20 NORMALIZED CUT First note that Edges(C i ) = k xk C i D k,k Edges(C 2 ) Edges(C Set c i = 1 ) if i C 1 Edges(C 1 ) Edges(C 2 ) otherwise Verify that c Lc = E NCut and c Dc = E (and Dc 1) Hence we relax Minimize NCUT(C) to Minimize c Lc c Dc s.t. Dc 1 Solution: Find second smallest eigenvectors of L = I D 12 AD 12

21 SPECTRAL CLUSTERING Input: Similarity matrix A A i,j = A j,i > 0 indicates similarity between elements x i and x j Minimize c > Lc s.t. c? 1 = = Approximately Minimize normalized cut! ( ) Example: Solution: Find A i,j = second exp( smallest d(x i, x j )) eigenvectors of L = I D 12 AD 12 A is adjacency matrix of a graph

22 SPECTRAL CLUSTERING ALGORITHM (NORMALIZED) 1 Given matrix A calculate diagonal matrix D s.t. D i,i = n j=1 A i,j 2 Calculate the normalized Laplacian matrix L = I D 12 AD 12 3 Find eigen vectors v 1,...,v n of L (ascending order of eigenvalues) 4 Pick the K eigenvectors with smallest eigenvalues to get y 1,...,y n R K 5 Use K-means clustering algorithm on y 1,...,y n

23 Demo

24 NORMALIZED CUT: ALTERNATE VIEW If we perform random walk on graph, its the partition of graph into group of vertices such that the probability of transiting from one group to another is minimized Transition matrix: D 1 A Largest eigenvalues and eigenvectors of above matrix correspond to smallest eigenvalues and eigenvectors of D 1 L = I D 1 A For K-nearest neighbor graph (K-regular), same as normalized Laplacian