Application of Spectral Clustering Algorithm

Size: px

Start display at page:

Download "Application of Spectral Clustering Algorithm"

Kathlyn Alexander
6 years ago
Views:

1 1/27 Application of Spectral Clustering Algorithm Danielle Middlebrooks Advisor: Kasso Okoudjou Department of Mathematics University of Maryland- College Park Advance Scientific Computing II May 11, 2016

2 2/27 Outline

3 3/27 Background Information Spectral Clustering is technique that makes use of the spectrum of the similarity matrix derived from the data set in order to cluster the data set into different clusters. Implement an algorithm that groups same digits from the MNIST Handwritten digits database in the same cluster. In practice this algorithm and my code will work for any database that wants to group together similar objects.

4 4/27 Motivation Motivated by the N cut problem. min NCut(A 1,..., A k ) := min 1 2 k i=1 W (A i, Āi) vol(a i ) where A is a subset of the vertices V the compliment Ā = V \ A W (A i, A j ) = i A i,j A j w ij vol(a) = i A d i The idea is that the eigenvectors serve as indicator functions in order to easily cluster the database in a reduced dimension.

5 5/27 Implementation Personal Laptop: Macbook Pro. Matlab R2016b 4GB Memory Desktop provided by Norbert Wiener Center Matlab R2015b 128GB Memory

6 6/27 Normalized Laplacian Matrix X i X j 2 Guassian Similarity Function: s(x i, X j ) = e 2σ 2 is a parameter. { W - Adjacency matrix w ij = 1, if s(x i, X j ) > ɛ 0, otherwise D- Degree matrix Unnormalized Laplacian Matrix: L = D W Normalized Laplacian Matrix: L sym = D 1/2 LD 1/2 = I D 1/2 WD 1/2 where σ

7 7/27 Normalized Laplacian Matrix As validation we know the smallest eigenvalue of the Normalized Laplacian will be zero with eigenvector D 1/2 1 To choose the best parameters, we implement the entire algorithm a number of times, changing epsilon each time until we reach some tolerance for the total error σ = 2000 ɛ =

8 8/27 Modified B Matrix Normalized Laplacian Matrix: L sym = D 1/2 LD 1/2 = I D 1/2 WD 1/2 = I B Computing the first p eigenvalues of B using the power method give us the largest eigenvalues in magnitude. Let B mod = B + µi where µ = max(sum(b,2))

9 9/27 Computing first p Eigenvectors Using the Power Method with Deflation on B mod we compute the first p eigenvalues.

10 10/27 Computing first p Eigenvectors By changing convergence criterion and increasing max iterations we obtain λ 1 λ 2 λ 3 λ 4 r 6.90E E E E-09 r =norm( B λ v B λ v,2) (λ, v) came from power method (λ, v ) came from eigs function

11 11/27 Row Normalization Let T R nxk be the eigenvector matrix with norm 1. Set v i,j t i,j = ( p v i,p 2 )1/2 v 11 v 12 v v 1p t 11 t 12 t t 1p v i1 v i2 v i3... v ip t i1 t i2 t i3... t ip v n1 v n2 v n3... v np t n1 t n2 t n3... t np

12 12/27 K-means Clustering Let y i be the ith row of T Randomly select k cluster centroids, z j. Calculate the distance between each y i and z j. Assign the data point to the closest centroid. Recalculate centroids and distances from data points to new centroids. If no data point was reassigned then stop, else reassign data points and repeat.

13 K-means Clustering Assign the original point X i to cluster j if and only if row i of the matrix T was assigned to cluster j. 13/27

14 14/27 Cluster Classification Next we classify each cluster as a particular digit. Digit Cluster Class Run time: 23mins

15 15/27 Results Below is a table of error for each cluster on 2000 Error= Number of incorrect digits in cluster Total number of digits in cluster % 82% 48% 65% 39% 13% 69% 58% 65% 72% Total number of incorrect digits Overall Error= Total number of digits = 59% Overall Error on 1000 images=64% Overall Error on images=49%

16 16/27 Results Cluster 6 Cluster 4 Cluster 3

17 17/27 Addition of New Datapoint- Standard Method Proposition (Nystrom Method) Method for out-of-sample extension Goal: Use a similarity kernel function K(x, y) in order to embed the new data point x in the reduced dimension. Benjio, Y, et al. Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering

18 18/27 Addition of New Datapoint- Another Method? We can determine which cluster a single new datapoint belongs to without re running the entire code. Create a similarity vector, denoted as X sim of 0 s and 1 s Normalize the similarity vector by multiplying it by D 1/2 Compute the projection of the similarity vector onto the eigenvectors of the Normalized Laplacian matrix and normalize. Denoted as C sim that lives in R p. Find the centroid that is closest to C sim

19 19/27 Results Implementation on a random subset of 100 digits. Error Runtime Averaged over 100 digits 61% 12.6sec

20 20/27 Yale Face Database Contains 165 grayscale images of 15 individuals. 11 images per subject, one per different facial expression or configuration. Each image is 32x32 pixels

21 21/27 Results Using 10 subjects and 5 images per subject with σ = 2000 and ɛ = Image Cluster Class Below is a table of error for each cluster classification Error= Number of incorrect faces in cluster Total number of faces in cluster % 33% 60% 83% 0% 66% 44% 40% 60% 66% Total number of incorrect faces Overall Error= Total number of faces = 54%

22 22/27 Results Cluster 5 Cluster 4 Cluster 2

23 23/27 End of October/ Early November: Construct Similarity Graph and Normalized Laplacian matrix. End of November/ Early December: Compute first k eigenvectors validate this. February: Normalize the rows of matrix of eigenvectors and perform dimension reduction. March/April: Cluster the points using k-means and validate this step. End of Spring semester: Implement entire algorithm, optimize and obtain final results.

24 24/27 Conclusion Spectral Clustering is a relatively good clustering technique. Better performance when dataset is sufficiently large. May obtain better results by using a different Normalized Laplacian or different similarity graph.

25 25/27 [1.] Von Cybernetics, U. A Tutorial on Spectral Clustering. Statistics and Computing, 7 (2007) 4. [2.] Shi, J. and Malik J. Normalized cuts and image segmentation. IEEE Transations on Pattern Analysis and Machine Intelligence, 22 (2000) 8. [3.] Chung, Fan. Spectral Graph Theory. N.p.: American Mathematical Society. Regional Conference Series in Mathematics Ser. 92. [4.] Vishnoi, Nisheeth K.Lx = b Laplacian Solvers and their Algorithmic Applications. N.p.: Foundations and Trends in Theoretical Computer Science, [5.] Benjio, Y, Paiement, J, Vincent, P. Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering. 2003

26 26/27 Thank you

27 27/27 Proposition Let K(x i, x j ) denote a kernel function of L sym such that L sym (i, j) = K(x i, x j ). Let (v k, λ k ) be an (eigenvector,eigenvalue) pair that solves L sym v k = λ k v k. Let (f k, λ k ) be an (eigenfunction,eigenvalue) pair that solves Kf k = λ k f k. Then y k (x) is the embedding associated with a new datapoint x. λ k = 1 n λ k n n f k (x) = v ik K(x, x j ) λ k i=1 y k (x) = f k(x) = 1 n v ik K(x, x j ) n λ k i=1

Spectral Clustering on Handwritten Digits Database

October 6, 2015 Spectral Clustering on Handwritten Digits Database Danielle dmiddle1@math.umd.edu Advisor: Kasso Okoudjou kasso@umd.edu Department of Mathematics University of Maryland- College Park Advance