Today s lecture. Clustering and unsupervised learning. Hierarchical clustering. K-means, K-medoids, VQ

Transcription

1 Clustering CS498

2 Today s lecture Clustering and unsupervised learning Hierarchical clustering K-means, K-medoids, VQ

3 Unsupervised learning Supervised learning Use labeled data to do something smart What if the labels don t exist?

4 Some inspiration El Capitan, Yosemite National Park

5 The way we ll see it Blue Green Red

6 A new question I see classes, but How do I find them? Can I automate this? How any are there? Blue Answer: Clustering 0 Green Red

7 Clustering Discover classes in data Divide data in sensible clusters Fundamentally ill-defined problem There is often no correct solution Relies on many user choices

8 Clustering process Describe your data using features What s your objective? Define a proximity measure How is the feature space shaped? Define a clustering criterion When do samples make a cluster?

9 Know what you want Features & objective matter Which are the two classes?

10 Know what you want Features & objective matter Which are the two classes? Basketball player recruiting Player s knowledge of entomology Player s height

11 Know your space Define a sensible proximity measure

12 Know your space Define a sensible proximity measure Angle of incidence 0 2π 4π

13 Know your cluster type What forms a cluster in your space?

14 Know your cluster type What forms a cluster in your space? Planes near airports Altitude Speed

15 Know your cluster type What forms a cluster in your space? Sound bouncing off a wall Intensity Wavefront position

16 Know your cluster type What forms a cluster in your space? Ants departing colony North West East South

17 How many clusters? The deeper you look the more you ll get

18 There are no right answers! Part of clustering is an art You need to experiment to get there But some good starting points exist

19 How to cluster Tons of methods We can use step-based logical steps e.g., find two closest point and merge, repeat Or formulate a global criterion

20 Hierarchical methods Agglomerative algorithms Keep pairing up your data Divisive algorithms Keep breaking up your data

21 Agglomerative Approach Look at your data points and form pairs Keep at it

22 More formally Represent data as vectors: X = {x i,i = 1,,N } Represent clusters by: C j Represent the clustering by: R = {C j, j = 1,,m} e.g. R = {{x 1,x 3 },x 2 {x 4,x 5,x 6 }} Define a distance measure: d(c j,c i )

23 Agglomerative clustering Choose: For t = 1, R 0 = {C i = {x i },i = 1,,N } Among all clusters in R t-1, find cluster pair {C i,c j } such that: arg mind(c i,c j ) i,j Form new cluster and replace pair: C q = C i C j R t = (R t 1 {C i,c j }) {C q } Until we have only one cluster

24 Dendrogram Pretty picture version R 0 x 1 x 2 x 3 x 4 x 5 R 1 R 2 R 3 Similarity R 4

25 Venn diagram Pretty picture two x 1 x 2 x 3 x 4 x 5 R 1 R 2 R 3 R 4

26 Cluster distance? Complete linkage Merge clusters that result to the smallest diameter Single linkage Merge clusters with two closest data points Group average Use average of distances

27 What s involved At level t we have N - t clusters At level t+1 the pairs we consider are: N t 2 Overall comparisons: N 1 N t 2 t=0 (N t)(n t 1) 2 (N 1)N(N + 1) 6

28 Not good for our case El Capitan picture has 63,140 pixels How many cluster comparisons is that? N 3 N t = 41,946,968,141,536 2 t=0 Thanks, but no thanks

29 Divisive Clustering Works the other way around Start with all data in one cluster Start dividing into sub-clusters

30 Choose: For t = 1, Divisive Clustering R 0 = {X} For k = 1,,t Find least similar sub-clusters in each cluster Pick the least similar of all: arg maxd(c k,i,c k,j ) k,i,j New clustering is now: R t = (R t 1 {C t }) {C t,i,c t,j } Until each point is a cluster

31 Comparison Which one is faster? Agglomerative Divisive has a complicated search step Which one gives better results? Divisive Agglomerative makes only local observations

32 Using cost functions Given a set of data x i Define a cost function: i j J(θ,U) = u ij d(x i,θ j ) θ are the cluster parameters U {0,1} is an assignment matrix d() is a distance function

33 An iterative solution We can t use a gradient method The assignment matrix is binary-valued We have two parameters to find θ, U Fix one and find the other, repeat flip case Iterate until happy

34 Overall process Initialize θ and iterate: Estimate U Estimate θ u ij = 1, if d(x,θ ) = min d(x,θ ) i j k i k 0, otherwise i u ij d(x i,θ j ) θ j = 0 Repeat until satisfied

35 K-means Standard and super-popular algorithm Finds clusters in terms of region centers Optimizes squared Euclidean distance d(x,θ) = x θ 2

36 K-means algorithm Initialize k means µ Iterate Assign samples x i to closest mean µ Estimate µ from assigned samples x i Repeat until convergence

37 Example run step

38 Example run step

39 Example run step

40 Example run step

41 Example run step

42 How well does it work? Converges to a minimum of cost function Not for all distances though! Is heavily biased by starting positions Various initialization tricks It s pretty fast!

43 K-Means on El Capitan Blue Blue Green Red Green Red

44 K-means on El Capitan

45 K-means on El Capitan Blue Blue Green Red Green Red

46 K-means on El Capitan

47 One problem K-means struggles with outliers

48 K-medoids Medoid: Least dissimilar data point to all others Not as influenced by outliers as the mean Replace means with medoids Redesign k-means as k-medoids

49 K-medoids 40 Input data 40 k means 40 k medoids

50 Vector Quantization Use of clustering for compression Keep a codebook ( k-means) Transmit nearest codebook vector instead of current sample We transmit only the cluster index, not the entire data for each sample

51 Simple example

52 Vector Quantization in Audio Input sequence Frequency 4 Cluster Coded sequence Frequency Time

53 Recap Hierarchical clustering Agglomerative, Divisive Issues with performance K-means Fast and easy K-medoids for more robustness (but slower)