Clustering. Discover groups such that samples within a group are more similar to each other than samples across groups.

Transcription

1 Clustering 1

2 Clustering Discover groups such that samples within a group are more similar to each other than samples across groups. 2

3 Clustering Discover groups such that samples within a group are more similar to each other than samples across groups. 3

4 Clustering Discover groups such that samples within a group are more similar to each other than samples across groups. 4

5 Clustering Discover groups such that samples within a group are more similar to each other than samples across groups. Groups not known a priori (unsupervised learning) 5

6 Ingredients of clustering Dissimilarity function between samples. 6

7 Ingredients of clustering Dissimilarity function between samples. Loss function to measure goodness of clustering 7

8 Ingredients of clustering Dissimilarity function between samples. Loss function to measure goodness of clustering Algorithm for optimizing the loss function 8

9 Center-based clustering: K-means 9

10 K-means clustering Distance function = Euclidean distance (squared) Center-based clustering 10

11 K-means clustering Distance function = Euclidean distance (squared) Center-based clustering 11

12 K-means clustering How do we optimize the K-means objective function? 12

13 K-means clustering How do we optimize the K-means objective function? Block-coordinate descent (alternating minimization) Fix μ, optimize C Fix C, optimize μ What are the corresponding updates? (derive on board) 13

14

15

16

17

18

19

20

21

22

23 Application: K-means for image segmentation 23

24

25

26 Image Clusters on intensity Clusters on color K-means clustering using intensity alone and color alone

27 Image Clusters on color K-means using color alone, 11 segments

28 K-means using color alone, 11 segments.

29 Questions about K-means Is convergence guaranteed? What is the corrresponding running time per iteration? Limitations/problems? 29

30 Questions about K-means Is convergence guaranteed? What is the corrresponding running time per iteration? Limitations/problems? Sensitive to initialization K is assumed as known Cannot handle well non-convex clusters Sensitive to outliers Hard assignments 30

31 Density-based clustering: Mean-shift 31

32 Density-based clustering IDEA: Clusters are locations where data points have high density Assume data are IID samples from some underlying probability distribution Find local maxima (modes) of probability distribution 32

33 Density-based clustering IDEA: Clusters are locations where data points have high density Assume data are IID samples from some underlying probability distribution Find local maxima (modes) of probability distribution PROBLEMS TO SOLVE: What probability distribution to use? How to find its modes? 33

34 Mean Shift Theory Slides credit: Ukrainitz & Sarel

35 What is Mean Shift? A tool for: Finding modes in a set of data samples, manifesting an underlying probability density function (PDF) in R N PDF in feature space Color space Non-parametric Scale space Density Estimation Actually any feature space you can conceive Discrete PDF Representation Data Non-parametric Density GRADIENT Estimation (Mean Shift) PDF Analysis

36 Non-Parametric Density Estimation Assumption : The data points are sampled from an underlying PDF Data point density implies PDF value! Assumed Underlying PDF Real Data Samples

37 Non-Parametric Density Estimation Assumed Underlying PDF Real Data Samples

38 Non-Parametric? Density Estimation Assumed Underlying PDF Real Data Samples

39 Parametric Density Estimation Assumption : The data points are sampled from an underlying PDF PDF( x) = i c e i ( x-μ ) 2 i 2 i 2 Estimate Assumed Underlying PDF Real Data Samples

40 Kernel Density Estimation Parzen Windows - Function Forms n 1 P( x) K( x - xi) A function of some finite number of data points n i 1 x 1 x n In practice one uses the forms: Data d K( ) ck( xi ) x or K( x) ck x i1 Same function on each dimension Function of vector length only

41 Kernel Density Estimation Various Kernels n 1 P( x) K( x - xi) A function of some finite number of data points n i 1 x 1 x n Examples: Epanechnikov Kernel K E ( x) 2 c x x otherwise Data Uniform Kernel K U ( x) c x 1 0 otherwise Normal Kernel KN 1 ( x) cexp 2 x 2

42 Kernel Density Estimation Gradient n 1 P( x) K( x - x ) n i 1 i Give up estimating the PDF! Estimate ONLY the gradient Using the Kernel form: We get : 2 x - x i K( x - xi ) ck h Size of window n ig n n i c c x i1 P( x) ki gi n n i1 n i1 gi i1 x g( x) k( x)

43 Kernel Density Estimation Gradient ( ) n i i n n i i i n i i i i g c c P k g n n g x x x Computing The Mean Shift g( ) ( ) k x x

44 ( ) n i i n n i i i n i i i i g c c P k g n n g x x x Computing The Mean Shift Yet another Kernel density estimation! Simple Mean Shift procedure: Compute mean shift vector Translate the Kernel window by m(x) ( ) n i i i n i i g h g h x - x x m x x x - x g( ) ( ) k x x

45 Intuitive Description Region of interest Center of mass Objective : Find the densest region Distribution of identical billiard balls Mean Shift vector

46 Intuitive Description Region of interest Center of mass Objective : Find the densest region Distribution of identical billiard balls Mean Shift vector

47 Intuitive Description Region of interest Center of mass Objective : Find the densest region Distribution of identical billiard balls Mean Shift vector

48 Intuitive Description Region of interest Center of mass Objective : Find the densest region Distribution of identical billiard balls Mean Shift vector

49 Intuitive Description Region of interest Center of mass Objective : Find the densest region Distribution of identical billiard balls Mean Shift vector

50 Intuitive Description Region of interest Center of mass Objective : Find the densest region Distribution of identical billiard balls Mean Shift vector

51 Intuitive Description Region of interest Center of mass Objective : Find the densest region Distribution of identical billiard balls

52 Mean Shift Properties Automatic convergence speed the mean shift vector size depends on the gradient itself. Near maxima, the steps are small and refined Adaptive Gradient Ascent Convergence is guaranteed for infinitesimal steps only infinitely convergent, (therefore set a lower bound) For Uniform Kernel ( a finite number of steps ), convergence is achieved in Normal Kernel ( ) exhibits a smooth trajectory, but is slower than Uniform Kernel ( ).

53 Real Modality Analysis Tessellate the space with windows Run the procedure in parallel

54 Real Modality Analysis The blue data points were traversed by the windows towards the mode

55 Real Modality Analysis An example Window tracks signify the steepest ascent directions

56 Mean Shift Strengths & Weaknesses Strengths : Application independent tool Suitable for real data analysis Does not assume any prior shape (e.g. elliptical) on data clusters Can handle arbitrary feature spaces Weaknesses : The window size (bandwidth selection) is not trivial Inappropriate window size can cause modes to be merged, or generate additional shallow modes Use adaptive window size Only ONE parameter to choose h (window size) has a physical meaning, unlike K-Means

57 Mean Shift Applications

58 Clustering Cluster : All data points in the attraction basin of a mode Attraction basin : the region for which all trajectories lead to the same mode Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer

59 Clustering Synthetic Examples Simple Modal Structures Complex Modal Structures

60 Feature space: L*u*v representation Clustering Real Example Initial window centers Modes found Modes after pruning Final clusters

61 Clustering Real Example L*u*v space representation

62 Clustering Real Example 2D (L*u) space representation Final clusters