Cluster Analysis. Andrew Kusiak Intelligent Systems Laboratory

Transcription

1 Cluster Aalysis Adrew Kusiak Itelliget Systems Laboratory 2139 Seamas Ceter The Uiversity of Iowa Iowa City, Iowa Two geeric modes of learig: Supervised learig Usupervised learig (clusterig)

2 Supervised Learig Goal: Determie class boudaries (desig a classifier) Supervised Learig: Classifier Desig Types of classifiers: liear piecewise liear oliear earest eighbor(s)... Implemetatio: decisio trees regressio aalysis eural etworks...

3 Usupervised Learig: Mai Classes of Clusterig Methods Hierarchical clusterig Model-based clusterig Hierarchical Clusterig Aggregate ad Divisive Methods Aggregate methods Cosider each example (object) as a separate cluster ad aggregate them successively based upo its relatioship with other objects. The aggregatio process is to optimize a performace measure. Divisive methods Work i a opposite directio by begiig with a sigle cluster ad. splittig it ito smaller clusters

4 Model-Based Clusterig Performace metric, e.g., distace is optimized i a space defied by costraits. Objects x ad y Distace Metric: Properties (i) x - y = 0 if ad oly if x = y (the objects are idetical) (ii) x - y => 0 (Noegativity (iii) x - y = y - x (Symmetry) (iii) x - z <= x - y + y - z (Triagular iequality)

5 Distace Metrics Mikowski distace measure p Σ i=1 x- y = x i -y i p Σ i=1 x -y = x i -y i x-y = Σ x i-y i 2 i=1 p = 1 Absolute p = 2 Euclidea x -y =max x i=1, 2,..., x i -y i p = ifiity Tschebyshev Mahalaobis Distace Represets relatioships betwee features Scalig is possible x-y = (x-y) T M-1(x-y) M a positive defiite it matrix

6 Mahalaobis Distace x-y = (x-y) T M -1 (x-y) For matrix M with σ i 2 as diagoal elemets, the iverse of M σ M -1 = 0 σ σ-2 x - y = Σ i=1 (x 2 i -y i ) 2 σ i Bhattaharaya distace For M = I: Euclidea distace Hierarchical Clusterig: A Aggregatio Approach Give: Data set ad a distace fuctio. 1. Begi with N clusters by assigig each object to a separate cluster 2. Proceed with this iitial cofiguratio of the clusters ad merge the closest clusters. I other words, if S ad T are the two closest clusters, form a sigle cluster {S, T} ad reduce the umber of clusters by oe. 3. Repeat Step 2 util the desired clusters have bee reached. Result: Clusters of data (partitios).

7 Hierarchical Clusterig: Dedrograms a b c d e {a}, {b}, {c, d}, {e} {a,b}, {c, d}, {e} {a,b,c,d}, {e} Hierarchical Clusterig Sigle likage algorithm Complete likage algorithm Average likage algorithm Xli likage algorithm ih

8 Hierarchical Clusterig: Sigle Likage Algorithm S, T = two clusters Similarity betwee S ad T is computed based o the miimum distace betwee the objects belogig to the correspodig clusters T T S = mi x T x y y S S T - S = mix T x - y y S Hierarchical Clusterig: Complete Likage Algorithm Maximum distace betwee the objects i the aalyzed clusters is cosidered. T T S = max x T x y y S S T - S = max x T x - y y S

9 Hierarchical Clusterig: Average Likage Algorithm Two clusters are formed based o their average distace betwee the objects i the clusters T 1 T S = card(t)card(s) x y S x T y S T - S = 1 card(s)card(t) Σ x T y S x - y Hierarchical Clusterig Sigle likage algorithm (mi distace) promotes merge of small clusters small becomes larger Complete likage algorithm (max distace) promotes creatio of large clusters large becomes larger

10 Hierarchical Clusterig Average likage algorithm the curret cluster size does ot impact compositio mix of the future clusters X likage algorithm the mix of clusters to be formed is impacted by adjustig the distace metric Hierarchical Clusterig: Computatioal Aspects a b c d e Number of clusters {a}, {b}, {c, d}, {e} {a,b}, {c, d}, {e} {a,b,c,d},,, {e} Storage of distace matrix No-iterative ti optimizatio i

11 Model-based Clusterig p-media model Geeralized p-media model s.t. mi dij xij i =1 j =1 xij = 1 for all i = 1,..., j =1 xjj = p j =1 xij <= xjj for all i = 1,...,, j = 1,..., xij = 0,1 for all i = 1,...,, j = 1,...,