CS7267 MACHINE LEARNING

Transcription

1 S7267 MAHINE LEARNING HIERARHIAL LUSTERING Ref: hengkai Li, Department of omputer Science and Engineering, University of Texas at Arlington (Slides courtesy of Vipin Kumar) Mingon Kang, Ph.D. omputer Science, Kennesaw State University

2 What is luster Analysis? Finding groups of objects such that the objects in a group will be similar (or related) to one another and different from (or unrelated to) the objects in other groups

3 What is not luster Analysis? Supervised classification Have class label information Simple segmentation Dividing students into different registration groups alphabetically, by last name Results of a query Groupings are a result of an external specification

4 Notion of a luster can be Ambiguous

5 Types of lustering A clustering is a set of clusters Important distinction between hierarchical and partitional sets of clusters Partitional lustering A division data objects into non-overlapping subsets (clusters) such that each data object is in exactly one subset Hierarchical clustering A set of nested clusters organized as a hierarchical tree

6 Partitional lustering

7 Hierarchical lustering

8 Types of lusters enter-based clusters ontiguous clusters

9 Types of lusters: enter-based enter-based A cluster is a set of objects such that an object in a cluster is closer (more similar) to the center of a cluster, than to the center of any other cluster The center of a cluster is often a centroid, the average of all the points in the cluster, or a medoid, the most representative point of a cluster

10 Types of lusters: ontiguity-based ontiguous luster (Nearest neighbor or Transitive) A cluster is a set of points such that a point in a cluster is closer (or more similar) to one or more other points in the cluster than to any point not in the cluster.

11 Similarity and Dissimilarity Similarity Numerical measure of how alike two data objects are. Is higher when objects are more alike. Often falls in the range [0,1] Dissimilarity Numerical measure of how different are two data objects Lower when objects are more alike Minimum dissimilarity is often 0 Upper limit varies Proximity refers to a similarity or dissimilarity

12 Similarity/Dissimilarity for Simple Attributes p and q are the attribute values for two data objects.

13 Hierarchical lustering Produces a set of nested clusters organized as a hierarchical tree an be visualized as a dendrogram A tree like diagram that records the sequences of merges or splits

14 Strengths of Hierarchical lustering Do not have to assume any particular number of clusters Any desired number of clusters can be obtained by cutting the dendogram at the proper level They may correspond to meaningful taxonomies Example in biological sciences (e.g., animal kingdom, phylogeny reconstruction, )

15 Hierarchical lustering Two main types of hierarchical clustering Agglomerative: Start with the points as individual clusters At each step, merge the closest pair of clusters until only one cluster (or k clusters) left Divisive: Start with one, all-inclusive cluster At each step, split a cluster until each cluster contains a point (or there are k clusters)

16 Agglomerative lustering Algorithm More popular hierarchical clustering technique Basic algorithm is straightforward 1. ompute the proximity matrix 2. Let each data point be a cluster 3. Repeat 4. Merge the two closest clusters 5. Update the proximity matrix 6. Until only a single cluster remains Key operation is the computation of the proximity of two clusters Different approaches to defining the distance between clusters distinguish the different algorithms

17 Starting Situation Start with clusters of individual points and a proximity matrix p1 p2 p3 p4 p5.. p1 p2 p3 p4 p5.... Proximity Matrix... p1 p2 p3 p4 p9 p10 p11 p12

18 Intermediate Situation After some merging steps, we have some clusters.... p1 p2 p3 p4 p9 p10 p11 p Proximity Matrix

19 Intermediate Situation We want to merge the two closest clusters (2 and 5) and update the proximity matrix Proximity Matrix... p1 p2 p3 p4 p9 p10 p11 p12

20 After Merging The question is: How do we update the proximity matrix? i.e. How do we measure proximity (distance, similarity) between two clusters? U U 5??????? U 5... p1 p2 p3 p4 p9 p10 p11 p12

21 How to Define Inter-luster Similarity Similarity? MIN (Single Linkage) MAX (omplete Linkage) Group Average Distance Between entroids Other methods driven by an objective function Ward s Method uses squared error

22 How to Define Inter-luster Similarity MIN (Single Linkage) MAX (omplete Linkage) Group Average Distance Between entroids

23 How to Define Inter-luster Similarity MIN (Single Linkage) MAX (omplete Linkage) Group Average Distance Between entroids

24 How to Define Inter-luster Similarity MIN (Single Linkage) MAX (omplete Linkage) Group Average Distance Between entroids

25 How to Define Inter-luster Similarity MIN (Single Linkage) MAX (omplete Linkage) Group Average Distance Between entroids

26 luster Similarity: MIN or Single Link Similarity of two clusters is based on the two most similar (closest) points in the different clusters Determined by one pair of points, i.e., by one link in the proximity graph. I1 I2 I3 I4 I5 I I I I I

27 Hierarchical lustering: MIN Nested lusters Dendrogram

28 Similarity vs. Distance Similarity Matrix Distance Matrix I1 I2 I3 I4 I5 I I I I I I1 I2 I3 I4 I5 I I I I I

29 Strength of MIN Original Points Two lusters an handle non-elliptical shapes

30 Limitations of MIN Original Points Two lusters Sensitive to noise and outliers

31 luster Similarity: MAX or omplete Linkage Similarity of two clusters is based on the two least similar (most distant) points in the different clusters Determined by all pairs of points in the two clusters I1 I2 I3 I4 I5 I I I I I

32 Hierarchical lustering: MAX Nested lusters Dendrogram

33 Strength of MAX Original Points Two lusters Less susceptible to noise and outliers

34 Limitations of MAX Original Points Two lusters Tends to break large clusters Biased towards globular clusters

35 luster Similarity: Group Average Proximity of two clusters is the average of pairwise proximity between points in the two clusters. pi luster i p luster proximity(p,p j j proximity(luster i, luster j) luster luster i i j j ) Need to use average connectivity for scalability since total proximity favors large clusters I1 I2 I3 I4 I5 I I I I I

36 Hierarchical lustering: Group Average Nested lusters Dendrogram

37 Hierarchical lustering: Group Average ompromise between Single and omplete Link Strengths Less susceptible to noise and outliers Limitations Biased towards globular clusters

38 Hierarchical lustering: omparison MIN MAX Group Average

39 Hierarchical lustering: Time and Space requirements O(N 2 ) space since it uses the proximity matrix. N is the number of points. O(N 3 ) time in many cases There are N steps and at each step the size, N 2, proximity matrix must be updated and searched omplexity can be reduced to O(N 2 log(n) ) time for some approaches