Data Mining and Analysis: Fundamental Concepts and Algorithms

Transcription

1 Data Mining and Analysis: Fundamental Concepts and Algorithms dataminingbook.info Mohammed J. Zaki Wagner Meira Jr. Department of Computer Science Rensselaer Polytechnic Institute, Troy, NY, USA Department of Computer Science Universidade Federal de Minas Gerais, Belo Horizonte, Brazil Chapter 4: Hierarchical Clustering Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 4: Hierarchical Clustering / 3

2 Hierarchical Clustering The goal of hierarchical clustering is to create a sequence of nested partitions, which can be conveniently visualized via a tree or hierarchy of clusters, also called the cluster dendrogram. The clusters in the hierarchy range from the fine-grained to the coarse-grained the lowest level of the tree (the leaves) consists of each point in its own cluster, whereas the highest level (the root) consists of all points in one cluster. Agglomerative hierarchical clustering methods work in a bottom-up manner. Starting with each of the n points in a separate cluster, they repeatedly merge the most similar pair of clusters until all points are members of the same cluster. Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 4: Hierarchical Clustering / 3

3 Hierarchical Clustering: Nested Partitions Given a dataset D = {x,...,x n }, where x i R d, a clustering C = {C,...,C k } is a partition of D. A clustering A = {A,...,A r } is said to be nested in another clustering B = {B,...,B s } if and only if r > s, and for each cluster A i A, there exists a cluster B j B, such that A i B j. Hierarchical clustering yields a sequence of n nested partitions C,...,C n. The clustering C t is nested in the clustering C t. The cluster dendrogram is a rooted binary tree that captures this nesting structure, with edges between cluster C i C t and cluster C j C t if C i is nested in C j, that is, if C i C j. Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 4: Hierarchical Clustering 3 / 3

4 Hierarchical Clustering Dendrogram ABCDE The dendrogram represents the following sequence of nested partitions: AB ABCD CD Clustering C C C 3 C 4 C 5 Clusters {A},{B},{C},{D},{E} {AB},{C},{D},{E} {AB},{CD},{E} {ABCD},{E} {ABCDE} A B C D E with C t C t for t =,...,5. We assume that A and B are merged before C and D. Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 4: Hierarchical Clustering 4 / 3

5 Number of Hierarchical Clusterings The total number of different dendrograms with n leaves is given as: n (m ) = (n 3) = (n 3)!! m= (a) n = 3 (b) n = 3 3 (c) n = 3 Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 4: Hierarchical Clustering 5 / 3

6 Agglomerative Hierarchical Clustering In agglomerative hierarchical clustering, we begin with each of the n points in a separate cluster. We repeatedly merge the two closest clusters until all points are members of the same cluster. Given a set of clusters C = {C, C,.., C m }, we find the closest pair of clusters C i and C j and merge them into a new cluster C ij = C i C. Next, we update the set of clusters by removing C i and C j and adding C ij, as follows C = ( C \{C i, C j } ) {C ij }. We repeat the process until C contains only one cluster. If specified, we can stop the merging process when there are exactly k clusters remaining. Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 4: Hierarchical Clustering 6 / 3

7 Agglomerative Hierarchical Clustering Algorithm AGGLOMERATIVECLUSTERING(D, k): C {C i = {x i } x i D}// Each point in separate cluster {δ(x i, x j ): x i, x j D}// Compute distance matrix 3 repeat 4 Find the closest pair of clusters C i, C j C 5 C ij C i C j // Merge the clusters C ( C \{C i, C j } ) 6 {C ij }// Update the clustering 7 Update distance matrix to reflect new clustering 8 until C = k Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 4: Hierarchical Clustering 7 / 3

8 Distance between Clusters: Single, Complete and Average The distance between two points is typically computed using the Euclidean distance or L -norm ( d δ(x, y) = x y = (x i y i ) ) / The between-cluster distances are obtained as follows. Single Link: The minimum distance between a point in C i and a point in C j i= δ(c i, C j ) = min{δ(x, y) x C i, y C j } Complete Link: The maximum distance between points in the two clusters: δ(c i, C j ) = max{δ(x, y) x C i, y C j } Group Average: The average pairwise distance between points in C i and C j : x C δ(c i, C j ) = i y C j δ(x, y) n i n j Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 4: Hierarchical Clustering 8 / 3

9 Distance between Clusters: Mean and Ward s Mean Distance: The distance between two clusters is defined as the distance between the means or centroids of the two clusters: δ(c i, C j ) = δ(µ i,µ j ) Minimum Variance or Ward s Method: The distance between two clusters is defined as the increase in the sum of squared errors (SSE) when the two clusters are merged, where the SSE for a given cluster C i is given as δ(c i, C j ) = SSE ij = SSE ij SSE i SSE j where SSE i = x C i x µ i. After simplification, we get: ( ) ni n j µi δ(c i, C j ) = µ n i + n j j Ward s measure is therefore a weighted version of the mean distance measure. Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 4: Hierarchical Clustering 9 / 3

10 Single Link Agglomerative Clustering ABCDE δ E ABCD 3 ABCD 3 δ CD E AB 3 CD 3 CD 3 δ C D E AB 3 3 C 3 D 5 AB δ B C D E A 3 4 B 3 3 C 3 D 5 A B C D E Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 4: Hierarchical Clustering 0 / 3

11 Lance Williams Formula Whenever two clusters C i and C j are merged into C ij, we need to update the distance matrix by recomputing the distances from the newly created cluster C ij to all other clusters C r (r i and r j). The Lance Williams formula provides a general equation to recompute the distances for all of the cluster proximity measures δ(c ij, C r ) = α i δ(c i, C r )+α j δ(c j, C r ) + β δ(c i, C j )+γ δ(ci, C r ) δ(c j, C r ) The coefficients α i,α j,β, and γ differ from one measure to another. Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 4: Hierarchical Clustering / 3

12 Lance Williams Formulas for Cluster Proximity Measure α i α j β γ Single link Complete link Group average Mean distance Ward s measure n i n i +n j n i n i +n j n i +n r n i +n j +n r 0 0 n j n i +n j 0 0 n j n i +n j n j +n r n i +n j +n r n i n j (n i +n j ) 0 n r n i +n j +n r 0 Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 4: Hierarchical Clustering / 3

13 Iris Dataset: Complete Link Clustering u Contingency Table: iris-setosa iris-virginica iris-versicolor C (circle) C (triangle) 0 36 C 3 (square) u Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 4: Hierarchical Clustering 3 / 3