Clustering. Pattern Recognition IX. Michal Haindl. Clustering. Outline

Transcription

1 Clustering cluster - set of patterns whose inter-pattern distances are smaller than inter-pattern distances for patterns not in the same cluster a homogeneity and uniformity criterion no connectivity little assumptions about data - exploratory data analysis intrinsic data dimensionality Outline Pattern Recognition IX Michal Haindl Faculty of Information Technology, KTI Czech Technical University in Prague Institute of Information Theory and Automation Academy of Sciences of the Czech Republic Prague, Czech Republic optimal number of classes no optimal clustering method random data results validation Evropský sociální fond. Praha & EU: Investujeme do vaší budoucnosti MI-ROZ /Z Clustering c M. Haindl MI-ROZ /15 Outline c M. Haindl MI-ROZ /15 January 16, 2012 Outline cluster - set of patterns whose inter-pattern distances are smaller than inter-pattern distances for patterns not in the same cluster a homogeneity and uniformity criterion no connectivity little assumptions about data - exploratory data analysis intrinsic data dimensionality 1 Clustering Hierarchical Clustering 2 Cluster Validity optimal number of classes no optimal clustering method random data results validation c M. Haindl MI-ROZ /15 c M. Haindl MI-ROZ /15

2 Clustering Structures 1 Clustering compact& well-separated cluster - set of patterns whose inter-pattern distances are smaller than inter-pattern distances for patterns not in the same cluster a homogeneity and uniformity criterion no connectivity little assumptions about data - exploratory data analysis intrinsic data dimensionality unequal size optimal number of classes no optimal clustering method random data results validation c M. Haindl MI-ROZ /15 c M. Haindl MI-ROZ /15 Clustering Structures 2 Clustering Structures 1 compact& well-separated elongated concentric touching unequal size c M. Haindl MI-ROZ /15 c M. Haindl MI-ROZ /15

3 Clustering Structures 3 Clustering Structures 2 elongated clusters within clusters linearly unseparable non compact concentric touching c M. Haindl MI-ROZ /15 c M. Haindl MI-ROZ /15 Clustering Structures 3 Clustering Structures 2 clusters within clusters elongated concentric linearly unseparable non compact touching c M. Haindl MI-ROZ /15 c M. Haindl MI-ROZ /15

4 Metrics 2 Clustering Structures 3 Minkowski metric δ(r,s) = l x r,j x s,j m j=1 invariant to translations Y = X +A m = 1 city-block (Manhattan, chessboard) distance m = 2 Euclidean distance invariant to orthogonal mappings (rotations) Y = QX Q T Q = I m sup distance, Chebychev distance δ (r,s) = max i x r,i x s,i 1 m clusters within clusters linearly unseparable non compact δ(r,s) = (X r X s ) T Q(X r X s ) quadratic distance Q a PD scaling matrix c M. Haindl MI-ROZ /15 c M. Haindl MI-ROZ /15 Metrics 3 Metrics δ(r,s) = (X r X s ) T Q 1 (X r X s ) Mahalanobis distance (metric), Q covariance matrix, invariant to non-singular mappings, if Q = diag[ ] scale invariant too numerically complex Q based on all samples not on separate clusters { L ifδ(r,s) > T δ(r,s) = nonlinear distance 0 otherwise sample correlation coefficient (2 sets of data) distance function - real-valued function satisfying: 1 δ(r,s) = δ(s,r) symmetry 2 δ(r,s) 0 non-negativity 3 δ(r,r) = 0 (for similarity index δ(r,r) max s {δ(r,s)}) metric - 1.,2.,3. &: 5 If δ(r,s) = 0 X r = X s. 6 δ(r,t) δ(r,s)+δ(s,t) triangular inequality ρ r,s = n i=1 X r,ix s,i nµ r µ s nσ r σ s = Σ r,s Σr,r Σ s,s If δ is a metric so is lnδ, ln(maxδ δ) c M. Haindl MI-ROZ /15 c M. Haindl MI-ROZ /15

5 Hierarchical Clustering Data Normalization Euclidean metric requires that all variables are measured on the same scale (µ = 0,σ = 1). X i = [x i,1,...,x i,l ] T SAHN - sequential, agglomerative, hierarchical, nonoverlapping 1 set the iteration number m = 0 2 find the cluster pair such δ[(s),(t)] = min pairs {δ[(i),(j)]} 3 m = m+1 merge s,t clusters, define clustering level 4 update dissimilarity matrix (delete entry corresponding to s, t clusters, add new cluster val.) difficult for large number of patterns patterns cannot move between clusters problem where to cut the dendrogram number of classes c M. Haindl MI-ROZ /15 scaling (4 2 clusters) c M. Haindl MI-ROZ /15 Hierarchical Clustering objective to find the optimal clustering ( partitions - computationally unfeasible) K has to be known in advance K-means (Isodata) 1 Select K Θ i i = 1,...,K. 2 Compute µ j 1 i i = 1,...,K. 3 {X l,i} δ(x l,µ i ) = [(X l µ i ) T (X l µ i )] l δ(x l,µ i ) = min j {δ(x l,µ j )} X l ω i 5 stopping criterion (no of iteration, changes...) 6 j = j +1 go to 2. (7.) Apply heuristic splitting / merging criteria. patterns can move between clusters suboptimal partitions resulting from used heuristics only metric data c M. Haindl MI-ROZ /15 SAHN - sequential, agglomerative, hierarchical, nonoverlapping 1 set the iteration number m = 0 2 find the cluster pair such δ[(s),(t)] = min pairs {δ[(i),(j)]} 3 m = m+1 merge s,t clusters, define clustering level L(m) = δ[(s),(t)] 4 update dissimilarity matrix (delete entry corresponding to s, t clusters, add new cluster val.) δ[(k),(s,t)] = α s δ[(k),(s)]+α t δ[(k),(t)]+βδ[(k),(s)] +γ δ[(k),(s)] δ[(k),(t)] c M. Haindl MI-ROZ /15

6 Heuristic Partitioning 1 K = 1, i = 1 2 Assign X i into ω K. 3 i = i +1 stop if i > card(s) 4 Assign X i to the first of previously generated clusters (j) for which the first (leading) element δ(x i, j X 1 ) < threshold. 5 If there is no such a cluster and K < K max set K = K +1 go to 2., otherwise leave X i unallocated (ω 0 ) and go to 3. fast partitioning depends on data ordering c M. Haindl MI-ROZ /15 objective to find the optimal clustering ( partitions - computationally unfeasible) K has to be known in advance K-means (Isodata) 1 Select K Θ i i = 1,...,K. 2 Compute µ j 1 i i = 1,...,K. 3 {X l,i} δ(x l,µ i ) = [(X l µ i ) T (X l µ i )] l δ(x l,µ i ) = min j {δ(x l,µ j )} X l ω i 5 stopping criterion (no of iteration, changes...) 6 j = j +1 go to 2. (7.) Apply heuristic splitting / merging criteria. patterns can move between clusters suboptimal partitions resulting from used heuristics only metric data c M. Haindl MI-ROZ /15 Heuristic Partitioning 2 1 K = 0 2 K = K +1 stop if K > K max X assigned 3 Assign X i into ω K if is not a member of a cluster and its Euclidean distance from all other yet unassigned X j is maximum. X i µ K 4 Assign X i into ω K if not assigned in 3. and δ(x i,µ K ) threshold and X i closest to µ K ; otherwise go to 2. 5 Recalculate µ K, go to 4. c M. Haindl MI-ROZ /15 objective to find the optimal clustering ( partitions - computationally unfeasible) K has to be known in advance K-means (Isodata) 1 Select K Θ i i = 1,...,K. 2 Compute µ j 1 i i = 1,...,K. 3 {X l,i} δ(x l,µ i ) = [(X l µ i ) T (X l µ i )] l δ(x l,µ i ) = min j {δ(x l,µ j )} X l ω i 5 stopping criterion (no of iteration, changes...) 6 j = j +1 go to 2. (7.) Apply heuristic splitting / merging criteria. patterns can move between clusters suboptimal partitions resulting from used heuristics only metric data c M. Haindl MI-ROZ /15

7 Cluster Validity problems Are the resulting clusters real or merely an artifact of the clustering algorithm? How many clusters are present in data? How well does a hierarchy fit to data? problems with the definition of cluster and the meaning of validity Validation external - compares recovered structure to an a priori structure internal - no a priori information, tries to determine if the structure is intrinsically appropriate for the data relative - compares two structures and measures their relative merit c M. Haindl MI-ROZ /15 Cluster Validity problems Are the resulting clusters real or merely an artifact of the clustering algorithm? How many clusters are present in data? How well does a hierarchy fit to data? problems with the definition of cluster and the meaning of validity Validation external - compares recovered structure to an a priori structure internal - no a priori information, tries to determine if the structure is intrinsically appropriate for the data relative - compares two structures and measures their relative merit c M. Haindl MI-ROZ /15