Data clustering & the k-means algorithm

Transcription

1 April 27, 2016

2 Why clustering? Unsupervised Learning Underlying structure gain insight into data generate hypotheses detect anomalies identify features Natural classification e.g. biological organisms (phylogenetic relationships) Data compression

3 k-means popular for more than 50 years simple efficient empirically successful

4 k-means Input: n points in R d, X = {x 1,..., x n } k, the number of clusters we want to partition X into

5 k-means Input: Output: n points in R d, X = {x 1,..., x n } k, the number of clusters we want to partition X into a partition the minimizes the squared error between the mean of each cluster and the points in that cluster.

6 k-means More precisely, let X = {x 1,..., x n } where x j R d for all 1 j n. Let C = {c 1,..., c k } be k clusters, each containing some of the x j s. Let µ i be the mean of cluster c i. µ i = 1 #c i x j c i x j.

7 k-means We define the squared error between µ i and the points in c i by E(c i ) = x j c i x j µ i 2.

8 k-means We define the squared error between µ i and the points in c i by E(c i ) = x j c i x j µ i 2. The sum of the squared error over all k clusters is E(C) = k i=1 x j c i x j µ i 2.

9 k-means We define the squared error between µ i and the points in c i by E(c i ) = x j c i x j µ i 2. The sum of the squared error over all k clusters is E(C) = k i=1 x j c i x j µ i 2. This is the objective function that the algorithm is designed to minimize.

10 k-means in action Example of the algorithm for points in R 2, with k = 3.

11 k-means in action (images taken from [Jain, 2010])

12 k-means in action (images taken from [Jain, 2010])

13 k-means in action (images taken from [Jain, 2010])

14 k-means in action (images taken from [Jain, 2010])

15 k-means in action (images taken from [Jain, 2010])

16 basic procedure 1 Select an initial set of k means (for example, choose k points from the dataset). 2 Assign each point to its closest mean to generate a new partition of the data. 3 Calculate the new set of k means with respect to this partition. 4 Repeat Steps 2 and 3 until cluster membership stabilizes.

17 k-means Will this procedure terminate?

18 k-means Will this procedure terminate? Monotonely decreasing sequence of sum of squared errors Finite number of clusterings for finite point set X

19 k-means Will this procedure terminate? Monotonely decreasing sequence of sum of squared errors Finite number of clusterings for finite point set X Number of steps bounded by O(n O(dk) ) Inaba et al. 1994

20 k-means Will this procedure terminate? Monotonely decreasing sequence of sum of squared errors Finite number of clusterings for finite point set X Number of steps bounded by O(n O(dk) ) Inaba et al k-means is a greedy algorithm. It may terminate in a local minumum.

21 back to example What s wrong with this example?

22 determining k Sometimes we don t know what k should be a priori.

23 determining k Sometimes we don t know what k should be a priori. Increasing k will always decrease squared error! In fact for k = n (the number of data points) the sum of squared errors is 0. So squared error does not tell us which k to use.

24 determining k Sometimes we don t know what k should be a priori. Increasing k will always decrease squared error! In fact for k = n (the number of data points) the sum of squared errors is 0. So squared error does not tell us which k to use. Idea: determine k based on the information we want from the clustering or run k-means for various k, then use some heuristic to compare the results.

25 determining k One heuristic: the elbow method. [Image from ElbowCriterion.JPG]

26 in Matlab [idx,c] = kmeans(x,k); X is an array whose rows are the points in your dataset. x 1 x 2 X =. x n

27 in Matlab k is the number of clusters. [idx,c] = kmeans(x,k);

28 in Matlab [idx,c] = kmeans(x,k); idx is a vector of length n identifying which cluster each point belongs to. cluster that x 1 is in cluster that x 2 is in idx =. cluster that x n is in

29 in Matlab [idx,c] = kmeans(x,k); C is an array containing the k centroids (means). centroid of cluster 1 centroid of cluster 2 C =. centroid of cluster k

30 References Jain, Anil K. Data Clustering: 50 Years Beyond K-means, in Pattern Recogn. Lett. 31:8, June 2010, pp Elsevier Science Inc, New York, NY, USA.