Kernel Spectral Clustering

Size: px

Start display at page:

Download "Kernel Spectral Clustering"

Sabrina Warner
5 years ago
Views:

1 Kernel Spectral Clustering Ilaria Giulini Université Paris Diderot joint work with Olivier Catoni

2 introduction clustering is task of grouping objects into classes (clusters) according to their similarities so that I points in a same class are highly similar I points in different classes are highly dissimilar

3 example

4 clusters I compactness I connectivity

5 clusters I compactness! k-means I connectivity! spectral clustering

6 k-means goal to partition the dataset into k clusters by assigning points to the cluster with nearest mean 1. define k starting centroids I as far as possible to each other I usually randomly chosen from the data 2. associate points to the nearest centroid I create clusters 3. calculate the new centroids I as the mean of the points in the new clusters 4. iterate step 2 and 3

7 k-means B k-means works well with compact clusters but it doesn t work well if the points are not linearly separable

8 spectral clustering given X 1,...,X n, construct a similarity matrix A ij =exp( kx i X j k 2 ) so that I A ij is big if X i and X j are similar I A ij is small if X i and X j are dissimilar ideally A is a block diagonal matrix idea to use the c first eigenvectors to get a partition in c classes in order to get a new lower-dimensional representation

9 the Ng, Jordan, Weiss algorithm input X 1,...,X n points to cluster into c classes algorithm 1. form A ij =exp( kx i X j k 2 ), A ii = 0 2. construct L ij = D 1/2 i A ij D 1/2 j where D i = P n j=1 A ij 3. compute v 1,...,v c the c largest eigenvectors of L and form X = v 1...v c n c 4. cluster the (renormalized) rows of X into c clusters (e.g. via k-means)

10 our approach setting spectral clustering in a Hilbert space points are i.i.d. in a separable Hilbert space P (unknown) and supp(p) is a union of compact connected components point of departure the NJW algorithm

11 main idea replace the projection diag( 1,..., c, 0,...,0) in the NJW algorithm by the smooth cut-off diag( m 1,..., m n ) B we don t assume to know c (automatically estimated)

12 the algorithm - the empirical version input X 1,...,X n points to cluster algorithm 1. form A ij =exp( kx i X j k 2 ) 2. construct L ij = D 1/2 i A ij D 1/2 j where D i = P n j=1 A ij 3. consider L m 4. renormalize (H m ) ij =(L m ) 1/2 ii Lij m (L m ) 1/2 jj

13 the Hilbert framework idea consider the previous matrices as empirical versions of underlying integral operators A ij =exp( kx i X j k 2 )! A(x, y) =exp kx yk 2 L = D 1/2 AD 1/2! L(x, y) =µ(x) 1/2 A(x, y)µ(y) 1/2 D i = X j A ij! µ(x) = Z A(x, z) dp(z)

14 the algorithm - the ideal version for x, y 2X 1. define A(x, y) =exp( kx yk 2 ) 2. construct L(x, y) =µ(x) 1/2 A(x, y) µ(y) 1/2 where µ(x) = R A(x, z) dp(z) 3. define K m (x, y) = Z L(y, z 1 )L(z 1, z 2 )...L(z 2m 1, x) dp(z 1 )...dp(z 2m 1 ) 4. renormalize H m (x, y) =K m (x, x) 1/2 K m (x, y) K m (y, y) 1/2

15 facts I convergence properties the empirical algorithm converges to the ideal one I number of clusters automatically estimated

16 an example goal to cluster X 1,...,X n 2 R 2 (n = 900)

17 an example note I clusters are at the vertices of a simplex I number of classes = number of vertices

18 Markov chain approach recall H m (x, y) =K m (x, x) 1/2 K m (x, y) K m (y, y) 1/2 PROPOSITION. H m (x, y) = D Rx kr x k L 2 Q, R y kr y k L 2 Q EL 2 Q where I R x = µ(x) 1/2 d dq P Z m Z 0 =x = cos Rx,R y I (Z m ) m2n the Markov chain with transitions d dp P Z m+1 Z m =x (y) =µ(x) 1 A(x, y) I Q the invariant measure with density dq dp (x) =µ(x)

19 Markov chain approach PROPOSITION. H m is almost constant on each connected component i.e. lim!1 H exp( T 2 )(x, y) = X C2C T ({x, y} C) where C T is the set of c.c. of (x, y) 2 supp(p) 2 kx yk < T 1 conclusion for a suitable m cos Rx,R y = H m (x, y) ' ( 1 x, y 2 same C 0 x, y /2 same C! points are concentrated around ON vectors 1 conjecture already proved when supp(p) is finite

20 Markov chain approach PROPOSITION. H m is almost constant on each connected component i.e. lim!1 H exp( T 2 )(x, y) = X C2C T ({x, y} C) where C T is the set of c.c. of (x, y) 2 supp(p) 2 kx yk < T 1 conclusion for a suitable m cos Rx,R y = H m (x, y) ' ( 1 x, y 2 same C 0 x, y /2 same C! points are concentrated around ON vectors 1 conjecture already proved when supp(p) is finite

21 Markov chain approach PROPOSITION. H m is almost constant on each connected component i.e. lim!1 H exp( T 2 )(x, y) = X C2C T ({x, y} C) where C T is the set of c.c. of (x, y) 2 supp(p) 2 kx yk < T 1 conclusion for a suitable m cos Rx,R y = H m (x, y) ' ( 1 x, y 2 same C 0 x, y /2 same C! points are concentrated around ON vectors 1 conjecture already proved when supp(p) is finite

22 convergence properties step 1. rewrite the algorithm in terms of Gram operators I Gram operator given a kernel k and the corresponding RKHS (H k, k) Z G k u = hu, k(z)i Hk k(z) dp(z)

23 the algorithm for x, y 2X 1. define A(x, y) =exp( kx yk 2 ) 2. construct L(x, y) =µ(x) 1/2 A(x, y) µ(y) 1/2 where µ(x) =hg A 1/2 A 1/2 (x), A 1/2 (x)i HA 1/2 3. define K m (x, y) =hg 2m 1 L L(x), L(y)i HA where L(x) =µ(x) 1/2 A(x) 4. renormalize H m (x, y) =K m (x, x) 1/2 K m (x, y) K m (y, y) 1/2

24 convergence properties step 2. provide convergence results for Gram operators Let I Gu = E [hu, Xi H X] I b Gu = 1 n P n i=1 hu, X ii H X i I apple =sup 2H E[h,Xi 4 ] E[h,Xi 2 ] 2 < +1 PROPOSITION. with probability at least 1 2, 8 2H, hg, i s hg, b apple i. C n Tr(G) max{k k 2 hg, i, } +log log(n)/ + C 0 max i kx i k 4 napple max{k k 2 hg, i, } 3 Tr(G) max{k k 2 hg, i, } +log log(n)/ where = 100appleTr(G) n/128 log log(n)/ is a suitable threshold

25 work in progress: image classification test the algorithm for sequences of connected images

26 work in progress: image classification

27 work in progress: image classification

28 work in progress: image classification

29 work in progress: image classification finalrep[, 1:2][,2] finalrep[, 1:2][,1]

30 thank you

Clustering. CS294 Practical Machine Learning Junming Yin 10/09/06

Clustering. CS294 Practical Machine Learning Junming Yin 10/09/06 Clustering CS294 Practical Machine Learning Junming Yin 10/09/06 Outline Introduction Unsupervised learning What is clustering? Application Dissimilarity (similarity) of objects Clustering algorithm K-means,