Clustering Distance measures K-Means. Lecture 22: Aykut Erdem December 2016 Hacettepe University

Transcription

1 Clustering Distance measures K-Means Lecture 22: Aykut Erdem December 2016 Hacettepe University

2 Last time Boosting Idea: given a weak learner, run it multiple times on (reweighted) training data, then let the learned classifiers vote On each iteration t: - weight each training example by how incorrectly it was classified - Learn a hypothesis h t - A strength for this hypothesis a t slide by Aarti Singh & Barnabas Poczos Final classifier: - A linear combination of the votes of the different classifiers weighted by their strength Practically useful Theoretically interesting 2

3 Last time.. The AdaBoost Algorithm slide by Jiri Matas and Jan Šochman 3

4 This week Distance measures K-Means Spectral clustering Hierarchical clustering What is a good clustering? 4

5 Distance measures 5

6 Distance measures In studying clustering techniques we will assume that we are given a matrix of distances between all pairs of data points: x 1 x 2 x x x x x m x 3 x 4 d(x i,x j ) slide by Julia Hockenmeier x m 6

7 What is Similarity/Dissimilarity? Hard to define! But we know it when we see it slide by Eric Xing The real meaning of similarity is a philosophical question. We will take a more pragmatic approach. Depends on representation and algorithm. For many rep.//alg., easier to think in terms of a distance (rather than similarity) between vectors. 7

8 Defining Distance Measures Definition: Let O 1 and O 2 be two objects from the universe of possible objects. The distance (dissimilarity) between O 1 and O 2 is a real number denoted by D(O 1, O 2 ). gene1 gene2 slide by Andrew Moore

9 A few examples: Euclidean distance Correlation coefficient slide by Andrew Moore ficient s(x,y) i (x i x )(y i y ) x y i d(x,y) (x i y i ) 2 Similarity rather than distance Can determine similar trends 9

10 What properties should a distance measure have? slide by Alan Fern Symmetric - D(A,B) = D(B,A) - Otherwise, we can say A looks like B but B does not look like A Positivity, and self-similarity - D(A,B) 0, and D(A,B) = 0 iff A = B - Otherwise there will different objects that we cannot tell apart Triangle inequality - D(A,B) + D(B,C) D(A,C) - Otherwise one can say A is like B, B is like C, but A is not like C at all 10

11 Distance measures Euclidean (L2) n (L 2 ) d(x, y) = (x i y i ) 2 an (L ) Manhattan (L1) an (L 1 ) d i=1 d(x, y) = x - y = (Sup) Distance L d i=1 Infinity (Sup) Distance L (Sup) Distance L d(x, y) = max 1 i d x i y i x i y i slide by Julia Hockenmeier Note that L < L1 < L2, but different distances do not induce the same ordering on points. 11

12 Distance measures x = (x 1, x 2 ) y = (x 1 2, x 2 +4) Euclidean: ( ) 1/2 = 4.47 Manhattan: = 6 Sup: Max(4, 2) = 4 4 slide by Julia Hockenmeier 2 12

13 Distance measures Different distances do not induce the same ordering on points L L 2 (a,b) (a,b) = 5 = ( /2 + ε ) = 5 + ε L (c,d) 4 L (c,d) = (4 + 4 ) = 4 2 = = /2 slide by Julia Hockenmeier 4 L (c,d) < L (a,b) L (c,d) > L 2 2 (a,b) 9 13

14 Distance measures Clustering is sensitive to the distance measure. Sometimes it is beneficial to use a distance measure that is invariant to transformations that are natural to the problem: - Mahalanobis distance: Shift and scale invariance slide by Julia Hockenmeier 14

15 Mahalanobis Distance d(x, y) = (x - y) T Σ(x y) slide by Julia Hockenmeier Σ is a (symmetric) Covariance Matrix: µ = 1 m Σ = 1 m m i=1 m i=1 x i, (average of the data) (x µ)(x µ) T, a matrix of size m m Translates all the axes to a mean = 0 and variance = 1 (shift and scale invariance) 15

16 Distance measures Some algorithms require distances between a point x and a set of points A d(x, A) This might be defined e.g. as min/max/avg distance between x and any point in A. Others require distances between two sets of points A, B, d(a, B). This might be defined e.g as min/max/avg distance between any point in A and any point in B. slide by Julia Hockenmeier 16

17 Clustering algorithms Partitioning algorithms - Construct various partitions and then evaluate them by some criterion K-means Mixture of Gaussians Spectral Clustering slide by Eric Xing Hierarchical algorithms - Create a hierarchical decomposition of the set of objects using some criterion - Bottom-up agglomerative - Top-down divisive 17

18 Desirable Properties of a Clustering Algorithm Scalability (in terms of both time and space) Ability to deal with different data types Minimal requirements for domain knowledge to determine input parameters Ability to deal with noisy data Interpretability and usability slide by Andrew Moore Optional - Incorporation of user-specified constraints 18

19 K-Means 19

20 K-Means slide by David Sontag An iterative clustering algorithm - Initialize: Pick K random points as cluster centers (means) - Alternate: Assign data instances to closest mean Assign each mean to the average of its assigned points - Stop when no points assignments change 20

21 K-Means slide by David Sontag An iterative clustering algorithm - Initialize: Pick K random points as cluster centers (means) - Alternate: Assign data instances to closest mean Assign each mean to the average of its assigned points - Stop when no points assignments change 21

22 K-Means Clustering: Example Pick K random points as cluster centers (means) Shown here for K=2 slide by David Sontag 22

23 K-Means Clustering: Example Iterative Step 1 Assign data points to closest cluster centers slide by David Sontag 23

24 K-Means Clustering: Example Iterative Step 2 Change the cluster center to the average of the assigned points slide by David Sontag 24

25 K-Means Clustering: Example Repeat until convergence slide by David Sontag 25

26 K-Means Clustering: Example slide by David Sontag 26

27 K-Means Clustering: Example slide by David Sontag 27

28 Properties of K-Means Algorithms Guaranteed to converge in a finite number of iterations Running time per iteration: 1. Assign data points to closest cluster center O(KN) time 2. Change the cluster center to the average of its assigned points O(N) time slide by David Sontag 28

29 Objective K-Means Convergence 1. Fix μ, optimize C: 2. Fix C, optimize μ: Take partial derivative of μ i and set to zero, we have slide by Alan Fern K-Means takes an alternating optimization approach, each step is guaranteed to decrease the objective thus guaranteed to converge 29

30 Demo time 30

31 K-Means Example Applications 31

32 Example: K-Means for Segmentation K=2 K =2 K=3 K =3 K=10 K = 10 Original image Goal of Segmentation is to partition an image into regions each of which has reasonably homogenous visual appearance. Original slide by David Sontag 32

33 Example: K-Means for Segmentation K=2 K =2 K=3 K =3 K=10 K = 10 Original image Original slide by David Sontag 33

34 Example: K-Means for Segmentation K=2 K =2 K=3 K =3 K=10 K = 10 Original image Original slide by David Sontag 34

35 Example: Vector quantization slide by David Sontag FIGURE Sir Ronald A. Fisher ( ) was one of the founders of modern day statistics, to whom we owe maximum-likelihood, sufficiency, and many other fundamental concepts. The image on the left is a grayscale image at 8 bits per pixel. The center image is the result of 2 2 block VQ, using 200 code vectors, with a compression rate of 1.9 bits/pixel. The right image uses only four code vectors, with a compression rate of 0.50 bits/pixel [Figure from Hastie et al. book] 35

36 Example: Simple Linear Iterative Clustering (SLIC) superpixels λ: spatial regularization parameter R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Susstrunk SLIC Superpixels Compared to State-of-the-art Superpixel Methods, IEEE T-PAMI,

37 Bag of Words model slide by Carlos Guestrin aardvark 0 about 2 all 2 Africa 1 apple 0 anxious 0... gas 1... oil 1 Zaire 0 37

38 slide by Fei Fei Li 38

39 Object Bag of words slide by Fei Fei Li 39

40 Interest Point Features Compute SIFT descriptor [Lowe 99] Normalize patch Detect patches [Mikojaczyk and Schmid 02] [Matas et al. 02] [Sivic et al. 03] slide by Josef Sivic 40

41 Patch Features slide by Josef Sivic 41

42 Dictionary Formation slide by Josef Sivic 42

43 Clustering (usually K-means) slide by Josef Sivic Vector quantization 43

44 Clustered Image Patches slide by Fei Fei Li 44

45 Visual synonyms and polysemy Visual Polysemy. Single visual word occurring on different (but locally similar) parts on different object categories. slide by Andrew Zisserman Visual Synonyms. Two different visual words representing a similar part of an object (wheel of a motorbike). 45

46 Image Representation frequency slide by Fei Fei Li codewords.. 46

47 K-Means Clustering: Some Issues How to set k? Sensitive to initial centers Sensitive to outliers Detects spherical clusters Assuming means can be computed slide by Kristen Grauman 47