4. Ad-hoc I: Hierarchical clustering

Transcription

1 4. Ad-hoc I: Hierarchical clustering Hierarchical versus Flat Flat methods generate a single partition into k clusters. The number k of clusters has to be determined by the user ahead of time. Hierarchical methods generate a hierarchy of partitions, i.e. a partition P 1 into 1 clusters (the entire collection) a partition P 2 into 2 clusters a partition P n into n clusters (each object forms its own cluster) It is then up to the user to decide which of the partitions reflects actual sub-populations in the data.

2 Note: A sequence of partitions is called "hierarchical" if each cluster in a given partition is the union of clusters in the next larger partition. P4 P3 P2 P1 Top: hierarchical sequence of partitions Bottom: non hierarchical sequence

3 Hierarchical methods again come in two varieties, agglomerative and divisive. Agglomerative methods: Start with partition P n, where each object forms its own cluster. Merge the two closest clusters, obtaining P n-1. Repeat merge until only one cluster is left. Divisive methods Start with P 1. Split the collection into two clusters that are as homogenous (and as different from each other) as possible. Apply splitting procedure recursively to the clusters.

4 Note: Agglomerative methods require a rule to decide which clusters to merge. Typically one defines a distance between clusters and then merges the two clusters that are closest. Divisive methods require a rule for splitting a cluster.

5 4.1 Hierarchical agglomerative clustering Need to define a distance d(p,q) between groups, given a distance measure d(x,y) between observations. Commonly used distance measures: 1. d 1 (P,Q) = min d(x,y), for x in P, y in Q ( single linkage ) 2. d 2 (P,Q) = ave d(x,y), for x in P, y in Q ( average linkage ) 3. d 3 (P,Q) = max d(x,y), for x in P, y in Q ( complete linkage ) 4. d ( P, Q ) = x x ( centroid method ) 4 P Q P Q 5. d P Q ( Ward s method ) P Q x x 2 5 (, ) = 2 P Q + d 5 is called Ward s distance.

6 Motivation for Ward s distance: Let P k = P 1,, P k be a partition of the observations into k groups. Measure goodness of a partition by the sum of squared distances of observations from their cluster means: RSS( P ) = x x k k j P i = 1 j P i i 2 Consider all possible (k-1)-partitions obtainable from P k by a merge Merging two clusters with smallest Ward s distance optimizes goodness of new partition.

7 4.2 Hierarchical divisive clustering There are divisive versions of single linkage, average linkage, and Ward s method. Divisive version of single linkage: Compute minimal spanning tree (graph connecting all the objects with smallest total edge length. Break longest edge to obtain 2 subtrees, and a corresponding partition of the objects. Apply process recursively to the subtrees. Agglomerative and divisive versions of single linkage give identical results (more later).

8 Divisive version of Ward s method. Given cluster R. Need to find split of R into 2 groups P,Q to minimize RSS( P, Q) = xi x P + x j x i P 2 2 j Q or, equivalently, to maximize Ward s distance between P and Q. Q Note: No computationally feasible method to find optimal P, Q for large R. Have to use approximation.

9 Iterative algorithm to search for the optimal Ward s split Project observations in R on largest principal component. Split at median to obtain initial clusters P, Q. Repeat { Assign each observation to cluster with closest mean Re-compute cluster means } Until convergence Note: Each step reduces RSS(P, Q) No guarantee to find optimal partition.

10 Divisive version of average linkage Algorithm Diana, Struyf, Hubert, and Rousseuw, pp. 22

11 4.3 Dendograms Result of hierarchical clustering can be represented as binary tree: Root of tree represents entire collection Terminal nodes represent observations Each interior node represents a cluster Each subtree represents a partition Note: The tree defines many more partitions than the n-2 nontrivial ones constructed during the merge (or split) process. Note: For HAC methods, the merge order defines a sequence of n subtrees of the full tree. For HDC methods a sequence of subtrees can be defined if there is a figure of merit for each split.

12 If distance between daughter clusters is monotonically increasing as we move up the tree, we can draw dendogram: y-coordinate of vertex = distance between daughter clusters. Point set and corresponding single linkage dendogram Observations Single linkage dendogram x[,2] x[,1]

13 Standard method to extract clusters from a dendogram: Pick number of clusters k. Cut dendogram at a level that results in k subtrees.

14 4.4 Experiment Try hierarchical method on unimodal 2D datasets. Experiments suggest: Except in completely clear-cut situations, tree cutting ( cutree ) is useless for extracting clusters from a dendogram. Complete linkage fails completely for elongated clusters.

15 Needed: Diagnostics to decide whether the daughters of a dendogram node really correspond to spatially separated clusters. Automatic and manual methods for dendogram pruning. Methods for assigning observations in pruned subtrees to clusters.