Clustering in R d. Clustering. Widely-used clustering methods. The k-means optimization problem CSE 250B

Size: px

Start display at page:

Download "Clustering in R d. Clustering. Widely-used clustering methods. The k-means optimization problem CSE 250B"

Gilbert Hodges
5 years ago
Views:

1 Clustering in R d Clustering CSE 250B Two common uses of clustering: Vector quantization Find a finite set of representatives that provides good coverage of a complex, possibly infinite, high-dimensional space. Finding meaningful structure in data Finding salient grouping in data. Widely-used clustering methods The k-means optimization problem 1 K-means and its many variants 2 EM for mixtures of Gaussians 3 Agglomerative hierarchical clustering Input: Points x 1,..., x n R d ; integer k Output: Centers, or representatives, µ 1,..., µ k R d Goal: Minimize average squared distance between points and their nearest representatives: cost(µ 1,..., µ k ) = min j x i µ j 2 Centers carve R d into k convex regions: µ j s region consists of points for which it is the closest center.

Lloyd s k-means algorithm Initialization matters NP-hard optimization problem. Heuristic: k-means algorithm. Initialize centers µ 1,..., µ k in some manner.

2 Lloyd s k-means algorithm Initialization matters NP-hard optimization problem. Heuristic: k-means algorithm. Initialize centers µ 1,..., µ k in some manner. Repeat until convergence: Assign each point to its closest center. Update each µ j to the mean of the points assigned to it. Each iteration reduces the cost convergence to a local optimum. Initializing the k-means algorithm Typical practice: choose k data points at random as the initial centers. Another common trick: start with extra centers, then prune later. A particularly good initializer: k-means++ Pick a data point x at random as the first center Let C = {x} (centers chosen so far) Repeat until desired number of centers is attained: Pick a data point x at random from the following distribution: Pr(x) dist(x, C) 2, where dist(x, C) = min z C x z Add x to C Representing images using k-means codewords How to represent a collection of images as fixed-length vectors? patch of fixed size Take all l l patches in all images. Extract features for each. Run k-means on this entire collection to get k centers. Now associate any image patch with its nearest center. Represent an image by a histogram over {1, 2,..., k}.

3 Streaming and online computation Example: sequential k-means Streaming computation: for data sets too large to fit in memory. Make one pass (or maybe a few passes) through the data. On each pass: See data points one at a time, in order. Update models/parameters along the way. There is only enough space to store a tiny fraction of the data, or a perhaps short summary. 1 Set the centers µ 1,..., µ k to the first k data points 2 Set their counts to n 1 = n 2 = = n k = 1 3 Repeat, possibly forever: Get next data point x Let µ j be the center closest to x Update µ j and n j : Online computation: an even more lightweight setup, for data that is continuously being collected. Initialize a model. Repeat forever: See a new data point. Update model if need be. µ j = n jµ j + x n j + 1 and n j = n j + 1 K-means: the good and the bad Mixtures of Gaussians The good: Fast and easy. Effective in quantization. The bad: Geared towards data in which the clusters are spherical, and of roughly the same radius. Each of the k clusters is specified by: a Gaussian distribution P j = N(µ j, Σ j ) a mixing weight π j Is there is a similarly-simple algorithm in which clusters of more general shape are accommodated? Overall distribution over R d : a mixture of Gaussians Pr(x) = π 1 P 1 (x) + + π k P k (x)

4 The clustering task We are given data x 1,..., x n R d. For any mixture model π 1,..., π k, P 1 = N(µ 1, Σ 1 ),..., P k = N(µ k, Σ k ), Pr (data π 1 P π k P k ) n = (π 1 P 1 (x i ) + + π k P k (x i )) = n k ( π j (2π) d/2 exp 1 ) Σ j 1/2 2 (x i µ j ) T Σ 1 j (x i µ j ) j=1 Optimization surface Minimize the negative log-likelihood, k ( ln π j (2π) d/2 exp 1 ) Σ j 1/2 2 (x i µ j ) T Σ 1 j (x i µ j ) j=1 Find the maximum-likelihood mixture of Gaussians: parameters {π j, µ j, Σ j : j = 1... k} maximizing this function. The EM algorithm 1 Initialize π 1,..., π k and P 1 = N(µ 1, Σ 1 ),..., P k = N(µ k, Σ k ). 2 Repeat until convergence: Assign each point x i fractionally between the k clusters: Example Data with 3 clusters, each with 100 points. w ij = Pr(cluster j x i ) = π jp j (x i ) l π lp l (x i ) Update mixing weights, means, and covariances: π j = 1 n µ j = 1 nπ j Σ j = 1 nπ j w ij w ij x i w ij (x i µ j )(x i µ j ) T k-means solution 1

Example Hierarchical clustering Data with 3 clusters, each with 100 points. Choosing the number of clusters (k) is difficult. Often: no single right answer, because of multiscale structure.

5 Example Hierarchical clustering Data with 3 clusters, each with 100 points. Choosing the number of clusters (k) is difficult. Often: no single right answer, because of multiscale structure. EM for mixture of Gaussians Hierarchical clustering avoids these problems. Example: gene expression data The single linkage algorithm Start with each point in its own, singleton, cluster Repeat until there is just one cluster: Merge the two clusters with the closest pair of points Disregard singleton clusters

6 Linkage methods Start with each point in its own, singleton, cluster Repeat until there is just one cluster: Merge the two closest clusters How to measure distance between two clusters C and C? Average linkage Three commonly-used variants: 1 Average pairwise distance between points in the two clusters dist(c, C ) = 1 C C x C x C x x 2 Distance between cluster centers dist(c, C ) = mean(c) mean(c ) Single linkage Complete linkage dist(c, C ) = dist(c, C ) = min x x x C,x C max x x x C,x C 3 Ward s method: the increase in k-means cost occasioned by merging the two clusters dist(c, C ) = C C C + C mean(c) mean(c ) 2

Clustering. Clustering in R d

Clustering. Clustering in R d Clustering Clustering in R d Two common uses of clustering: Vector quantization Find a finite set of representatives that provides good coverage of a complex, possibly infinite, high-dimensional space.