Herarchcal agglomeratve Cluster Analyss Chrstne Sedle 19-3-2004 Clusterng 1
Classfcaton Basc (unconscous & conscous) human strategy to reduce complexty Always based Cluster analyss to fnd or confrm types n data to uncover relatons between objects The more enttes and the more attrbutes the more dffcultes classfyng them manually Computer-based cluster analyss Clusterng 2
Cluster analyss overvew Selecton of objects to be classfed Selecton of relevant attrbutes of these objects Calculaton of dstances between objects Cluster analyss Check of results (Modfcatons + rerun analys) Clusterng 3
Objects Selecton of objects depends on ntenton If clusters are expected: Number of objects should be balanced Many objects = large dstance matrx n ( n 1) values (e.g. 200 objects = 19900 dstance values) 2 Clusterng 4
Attrbutes Selecton of attrbutes depends on ntenton Not: The more attrbutes the surer groups wll appear Avod correlatons between attrbutes Values of attrbutes have to be comparable Treat mssng values (Weght attrbutes to nfluence clusterng) Clusterng 5
Attrbutes example 600 500 avocado parsnp fennel dandelon Poston of selected fruts/vegetables n the 2 dmensons magnesum & potassum 400 300 200 100 0 K (n mg) -10 passon frut peach water melon apple straw berry pear blueberry 0 10 20 kw frut elderberry peas 30 banana papaya 40 50 Mg (n mg) Clusterng 6
Dstance measures Based on the attrbute values the dstances between the objects have to be determned. Dstance measures have to ensure: Symmetry Trangle nequalty Dstngushablty of nondentcals Indstngushablty of dentcals d ( x, y ) = d ( y, x ) d ( x, y ) d ( x, z ) + d ( y, z f d ( x, y ) 0, then x d ( x, x ') = 0 ) y 0 Clusterng 7
Clusterng 8 Dstance measures examples Dstance measures (squared) Eucldan dstance Manhattan dstance Smlarty measures Pearson s correlaton coeffcent = = n X Y Y X 1 ), δ ( = = n X Y Y X 1 2 ) ( ), δ ( = = = = n n n Y Y X X Y Y X X Y X r 1 2 1 2 1 ) ( ) ( ) )( ( ), (
Squared Eucldan dstance example Dstances of selected fruts/vegetables based on (standardzed) content of Mg & K Proxmty Matrx Case 1:banana 2:avocado 3:parsnp 4:dandelon Ths s a dssmlarty matrx Squared Eucldean Dstance 1:banana 2:avocado 3:parsnp 4:dandelon,000 1,250 1,477,183 1,250,000,346,578 1,477,346,000 1,070,183,578 1,070,000 Clusterng 9
Cluster analyss Here dscussed (because most common): Sequental Agglomeratve Herarchcal Nonoverlappng (SAHN) Other approaches for clusterng: Herarchc dvsve Iteratve parttonng Factor analytc Clumpng... Clusterng 10
Cluster analyss Iteratve process n 1 steps necessary to cluster all objects At every step the two most smlar objects or clusters wll be merged untl all are aggregated n one cluster Clusterng 11
Cluster analyss example banana avocado parsnp dandelon banana 1.25 1.477 0.183 avocado 0.346 0.578 parsnp 1.07 dandelon d avocado, banana d avocado [ banana, dandelon ] = + 2 d avocado 1.25 2 0.578 2 [ banana, dandelon ] = + = d avocado 0.914, 2 dandelon Clusterng 12
Cluster analyss example avocado parsnp avocadoparsnp bananadandelon bananadandelon 0.914 1.2735 avocadoparsnp 1.09375 avocado parsnp 0.346 bananadandelon bananadandelon d [ banana, dandelon ], avocado d [ banana, dandelon ][ avocado, parsnp ] = + 2 0.914 2 1.2735 2 d [ banana, dandelon ][ avocado, parsnp ] = + = d [ 1.09375 banana, dandelon 2 ], parsnp Clusterng 13
Matrx updatng algorthms Several SAHN clusterng algorthms They dffer n how they calculate the dstances of new formed clusters to the other elements. Not every algorthm equally sutable for every stuaton Results can be very dfferent!! Clusterng 14
Matrx updatng algorthms Sngle lnkage Complete lnkage Unweghted average lnkage Weghted average lnkage (Un)Weghted centrod lnkage Ward s method Clusterng 15
Sngle lnkage ) d k j ) = mn( d k, d ( kj Nearest neghbor Dstance between new cluster and other elements equals the smallest n the cluster occurrng dstance to the other elements Tendency to very dfferent szed clusters (outlers!) j k Clusterng 16
Complete lnkage ) d k j ) = max( d k, d ( kj Furthest neghbor Dstance between new cluster and other elements equals the largest n the cluster occurrng dstance Clusters are only merged when dssmlarty s small. Balanced and equally szed clusters j k Clusterng 17
Unweghted average lnkage n n j d n + n n + n k [ j ] = d k + d j j UPGMA, Baverage, lnkage between groups Uses averages nstead of extreme values Number of elements n clusters s taken nto account j k Clusterng 18 kj
Weghted average lnkage d k d k [ j ] = + 2 d kj 2 WPGMA, Waverage, lnkage wthn groups Equals UPGMA but the number of elements n clusters s not takenntoaccount Can be necessary when the sze of supposed clusters or the object densty n them dffers j k Clusterng 19
(Un)Weghted centrod lnkage n n j n n j d n + n n + n ( n + n ) k [ j ] = d k + d kj d 2 j j j d d k kj d k [ j ] = + 2 2 d 4 Centrod of cluster s calculated Dstance to new cluster equals dstance to centrod j k Clusterng 20 j j
Ward s method n + n n + n n d k k j k k [ j ] = d k + d kj d n k + n + n j n k + n + n j n k + n + n j j Mnmum varance Idea: Heterogenty s not a reasonable feature of clusters Mnmze varance To be used only wth quanttatve attrbutes and squared Eucldan dstance! Clusterng 21
Matrx updatng algorthms Types of algorthms: Space-contractng (Sngle & Centrod (?) Lnkage) Unequally szed clusters Outlers vsble Space-dlatng (Complete lnkage & Ward s method) Balanced clusterng Clusters are often not easy to nterpret Space-conservng (Average lnkage) No unnaturally blown up clusters Appearng clusters are often nterpretable Clusterng 22
Space-contractng example 1 Dendrodram generated by Sngle-lnkage Clusterng 23
Space-contractng example 2 Kel Kel Rostock Rostock Hamburg Hamburg Emden Emden Bremen Bremen Berln Hannover Magdeburg Münster Cottbus Berln Hannover Magdeburg Münster Cottbus Dresden Dresden Köln Erfurt Köln Erfurt Marburg Marburg Frankfurt Frankfurt Trer Trer Nürnberg Nürnberg Saarbrücken Saarbrücken Regensburg Regensburg Stuttgart Stuttgart München München Freburg Freburg Sngle lnkage WPGMC Clusterng 24
Space-dlatng example 1 Dendrodram generated by Ward s method Clusterng 25
Space-dlatng example 2 Kel Kel Rostock Rostock Hamburg Hamburg Emden Emden Bremen Bremen Berln Hannover Hannover Magdeburg Magdeburg Münster Münster Cottbus Berln Cottbus Dresden Dresden Köln Erfurt Köln Erfurt Marburg Marburg Frankfurt Frankfurt Trer Trer Nürnberg Nürnberg Saarbrücken Saarbrücken Regensburg Regensburg Stuttgart Stuttgart München München Freburg Freburg Ward s method Complete lnkage Clusterng 26
Space-conservng example 1 Dendrodram generated by UPGMA Clusterng 27
Space-conservng example 2 Kel Kel Rostock Rostock Hamburg Hamburg Emden Emden Bremen Bremen Berln Hannover Hannover Magdeburg Magdeburg Münster Münster Cottbus Berln Cottbus Dresden Dresden Köln Erfurt Köln Erfurt Marburg Marburg Frankfurt Frankfurt Trer Trer Nürnberg Nürnberg Saarbrücken Saarbrücken Regensburg Regensburg Stuttgart Stuttgart München München Freburg Freburg UPGMA WPGMA Clusterng 28
Matrx updatng algorthms Whch should be used? Outlers shall be vsble Sngle lnkage Unequally szed clusters expected Not space-dlatng methods Dfferng object densty n expected clusters WPGMA No-dea-just-try-order: Space-conservng > space-dlatng > spacecontractng Clusterng 29
Number of clusters How many natural classes has cluster analyss generated? Subjectve decson of researcher Analyss of mergng values Large step = rather dssmlar clusters = stop Plot number of clusters aganst mergng values Graph flattens = no new nformaton = stop Ward s method: Sgnfcance test possble Clusterng 30
Valdaton of results Results should be stable Plausble nterpretaton possble Repeat cluster analyss wth dfferent samples of the same populaton Dfferent results = both nvald, but Same results = not necessarly vald and not always possble due to lack of data Cophenetc correlaton, but Normal dstrbuton (wrongly?) assumed In dendrogram fewer (dfferent) values Clusterng 31
Valdaton of results Sgnfcance tests Used attrbutes: Useless because always sgnfcant Not used (but relevant) attrbutes: Useful but only possble when knowledge about classes already exsts Monte Carlo procedures Data set s created whch has the same global propertes as orgnal data but contans no classes Both sets are clustered & results compared Sgnfcant dfferences => results vald Clusterng 32
Attenton! A lot of factors determne the results of cluster analyss Very careful selecton of objects, attrbutes, (ds)smlarty measure, cluster method and matrx updatng algorthm Cluster analyss wll always output clusters f there are natural classes or not! Clusterng 33