SPECTRAL SPARSIFICATION IN SPECTRAL CLUSTERING

Transcription

1 SPECTRAL SPARSIFICATION IN SPECTRAL CLUSTERING Alireza Chakeri, Hamidreza Farhidzadeh, Lawrence O. Hall Department of Computer Science and Engineering College of Engineering University of South Florida

2 Clustering Problem Partition the data into groups so that points in a group are similar and points in different groups are dissimilar. Feature-based: Objects are explicitly described by their attributes. Similarity-based (Graph-based): Tuple of two objects are given a similarity value. Feature-based data can be converted into similarity-based data. Gaussian kernel similarity function, cosine distance, More general representation Applications: Bioinformatics Social networks Computer Vision * 2

3 Spectral Clustering Treats clustering as a graph partitioning problem without making specific assumptions on the form of the clusters. It works with a Laplacian matrix. Common method: k-way spectral clustering Recursively use the Eigen vector with the second smallest Eigen value (Fielder vector) to bipartition the graph into two parts. cut 3

4 Cons of Spectral Clustering Computationally expensive for large datasets. It requires significant time and memory to compute eigenvectors of the Laplacian matrix. Common approaches: make the similarity matrix sparse by zeroing out some of its elements. Then use sparse Eigensolver such as SLEPc and ARPACK This is equivalent to removing some edges of the graph. Same graph but with fewer edges 4

5 Graph Sparsification Methods Common Combinatorial Sparsifiers: ϵ-neighborhood graph K nearest neighbor graph Cut sparsifier there is no approximation guarantee on preserving the spectral properties of the original Laplacian matrix. Spectral Sparsification Based on spectral properties of graphs. Strictly stronger than cut sparsifier. Spielman and Srivastava theorem: Every graph has a spectral sparsifier preserving its spectral properties. 5

6 Spectral Sparsification Observation: Graphs with similar Laplacian matrices have similar cut values, effective resistance, and other important combinatorial properties. A sparsified graph has: Similar Laplacian matrix with its original graph. Fewer edges (number of edges can be controlled by a sampling parameter). Its edges are reweighted edges of its original set. Example: Original graph How to sparsify: Sampling by effective resistance technique Sparsifier 6

7 Partitions of Sparsifier ~ Partitions of Graph Motivation: How close are the partitions of a sparsifier to the partitions of the original graph? In other words How does the Fiedler vector of a sparsifier differs from the Fiedler vector of the original graph? Observation: any vector whose Rayleigh quotient is close to the secondsmallest eigenvalue of the Laplacian matrix can also be used to find a good partition: Theorem: we prove that Fiedler vector of a sparsifier is an approximate Fielder vector of the original graph. Partitions of sparsifier approximate partitions of graph 7

8 Cluster Structure Preservation Even for a reasonably large ε, the underlying structure of a data set is preserved by the sparsifier: the sparsifier more likely contains vertices from all of the clusters of the graph. original data ε = 2 Remove data points that are not connected to any other data points in the sparsified graph ε = 10 ε = 25 8

9 Results - compatibility with the original partitions Ground truth labels: results of applying spectral clustering on the original graph Quality comparison: rand index (RI), variation of information (VOI), global consistency error (GCE). Metric ε = 1 ε = 2 ε = 3 RI GCE VOI # of edges Jain Data set* Number of instances Number of edges = = the number of edges decreases significantly (45 times) without ruining the clusters quality. * Anil K. Jain and Martin H.C. Law, Data Clustering: A User's Dilemma 9

10 Results - comparison with k-nearest neighbor sparsifier We fix ε = 3, and find k that leads to almost the same number of edges: To see why this happens: The spectral sparsifier approximates the original Fiedler vector much better than the nearest neighbor graph. 10

11 Results - sensitivity to ε Sensitivity of spectral sparsifiers to ε k-nearest neighbor to k The Fiedler vectors of spectral sparsifiers for ε = 1, 2, 3 and k-nearest neighbor for k = 10, 20, 30 Almost same number of edges Spectral sparsifier k-nearest neighbor sparsifier 11

12 CHALLENGES AND FUTURE WORK Computing and storing the entire similarity matrix, before sparsification starts: Quadratic complexity Recently, a semi-streaming setting for the spectral sparsification which runs as quick as the Spielman-Srivastava algorithm has been developed: That is as we read edges of the graph, we add them to the sparsifier. When the sparsifier gets too big, we re-sparsify it in linear time. 12

13 Summary Graph spectral clustering algorithms suffer from time and memory complexity. We adopt spectral sparsification by sampling using effective resistance to sparsify the graph Laplacian. We showed that the Fiedler vector of the sparsified graph provides a good approximation of the Fiedler vector of the original graph even when the sampling rate is low. The partitions of the sparsifier are very similar to the partitions of the original graph even when lots of sparsification is done. 13

14 Questions? Thank you 14