Hierarchical agglomerative. Cluster Analysis. Christine Siedle Clustering 1

Transcription

1 Herarchcal agglomeratve Cluster Analyss Chrstne Sedle Clusterng 1

2 Classfcaton Basc (unconscous & conscous) human strategy to reduce complexty Always based Cluster analyss to fnd or confrm types n data to uncover relatons between objects The more enttes and the more attrbutes the more dffcultes classfyng them manually Computer-based cluster analyss Clusterng 2

3 Cluster analyss overvew Selecton of objects to be classfed Selecton of relevant attrbutes of these objects Calculaton of dstances between objects Cluster analyss Check of results (Modfcatons + rerun analys) Clusterng 3

4 Objects Selecton of objects depends on ntenton If clusters are expected: Number of objects should be balanced Many objects = large dstance matrx n ( n 1) values (e.g. 200 objects = dstance values) 2 Clusterng 4

5 Attrbutes Selecton of attrbutes depends on ntenton Not: The more attrbutes the surer groups wll appear Avod correlatons between attrbutes Values of attrbutes have to be comparable Treat mssng values (Weght attrbutes to nfluence clusterng) Clusterng 5

6 Attrbutes example avocado parsnp fennel dandelon Poston of selected fruts/vegetables n the 2 dmensons magnesum & potassum K (n mg) -10 passon frut peach water melon apple straw berry pear blueberry kw frut elderberry peas 30 banana papaya Mg (n mg) Clusterng 6

7 Dstance measures Based on the attrbute values the dstances between the objects have to be determned. Dstance measures have to ensure: Symmetry Trangle nequalty Dstngushablty of nondentcals Indstngushablty of dentcals d ( x, y ) = d ( y, x ) d ( x, y ) d ( x, z ) + d ( y, z f d ( x, y ) 0, then x d ( x, x ') = 0 ) y 0 Clusterng 7

8 Clusterng 8 Dstance measures examples Dstance measures (squared) Eucldan dstance Manhattan dstance Smlarty measures Pearson s correlaton coeffcent = = n X Y Y X 1 ), δ ( = = n X Y Y X 1 2 ) ( ), δ ( = = = = n n n Y Y X X Y Y X X Y X r ) ( ) ( ) )( ( ), (

9 Squared Eucldan dstance example Dstances of selected fruts/vegetables based on (standardzed) content of Mg & K Proxmty Matrx Case 1:banana 2:avocado 3:parsnp 4:dandelon Ths s a dssmlarty matrx Squared Eucldean Dstance 1:banana 2:avocado 3:parsnp 4:dandelon,000 1,250 1,477,183 1,250,000,346,578 1,477,346,000 1,070,183,578 1,070,000 Clusterng 9

10 Cluster analyss Here dscussed (because most common): Sequental Agglomeratve Herarchcal Nonoverlappng (SAHN) Other approaches for clusterng: Herarchc dvsve Iteratve parttonng Factor analytc Clumpng... Clusterng 10

11 Cluster analyss Iteratve process n 1 steps necessary to cluster all objects At every step the two most smlar objects or clusters wll be merged untl all are aggregated n one cluster Clusterng 11

12 Cluster analyss example banana avocado parsnp dandelon banana avocado parsnp 1.07 dandelon d avocado, banana d avocado [ banana, dandelon ] = + 2 d avocado [ banana, dandelon ] = + = d avocado 0.914, 2 dandelon Clusterng 12

13 Cluster analyss example avocado parsnp avocadoparsnp bananadandelon bananadandelon avocadoparsnp avocado parsnp bananadandelon bananadandelon d [ banana, dandelon ], avocado d [ banana, dandelon ][ avocado, parsnp ] = d [ banana, dandelon ][ avocado, parsnp ] = + = d [ banana, dandelon 2 ], parsnp Clusterng 13

14 Matrx updatng algorthms Several SAHN clusterng algorthms They dffer n how they calculate the dstances of new formed clusters to the other elements. Not every algorthm equally sutable for every stuaton Results can be very dfferent!! Clusterng 14

15 Matrx updatng algorthms Sngle lnkage Complete lnkage Unweghted average lnkage Weghted average lnkage (Un)Weghted centrod lnkage Ward s method Clusterng 15

16 Sngle lnkage ) d k j ) = mn( d k, d ( kj Nearest neghbor Dstance between new cluster and other elements equals the smallest n the cluster occurrng dstance to the other elements Tendency to very dfferent szed clusters (outlers!) j k Clusterng 16

17 Complete lnkage ) d k j ) = max( d k, d ( kj Furthest neghbor Dstance between new cluster and other elements equals the largest n the cluster occurrng dstance Clusters are only merged when dssmlarty s small. Balanced and equally szed clusters j k Clusterng 17

18 Unweghted average lnkage n n j d n + n n + n k [ j ] = d k + d j j UPGMA, Baverage, lnkage between groups Uses averages nstead of extreme values Number of elements n clusters s taken nto account j k Clusterng 18 kj

19 Weghted average lnkage d k d k [ j ] = + 2 d kj 2 WPGMA, Waverage, lnkage wthn groups Equals UPGMA but the number of elements n clusters s not takenntoaccount Can be necessary when the sze of supposed clusters or the object densty n them dffers j k Clusterng 19

20 (Un)Weghted centrod lnkage n n j n n j d n + n n + n ( n + n ) k [ j ] = d k + d kj d 2 j j j d d k kj d k [ j ] = d 4 Centrod of cluster s calculated Dstance to new cluster equals dstance to centrod j k Clusterng 20 j j

21 Ward s method n + n n + n n d k k j k k [ j ] = d k + d kj d n k + n + n j n k + n + n j n k + n + n j j Mnmum varance Idea: Heterogenty s not a reasonable feature of clusters Mnmze varance To be used only wth quanttatve attrbutes and squared Eucldan dstance! Clusterng 21

22 Matrx updatng algorthms Types of algorthms: Space-contractng (Sngle & Centrod (?) Lnkage) Unequally szed clusters Outlers vsble Space-dlatng (Complete lnkage & Ward s method) Balanced clusterng Clusters are often not easy to nterpret Space-conservng (Average lnkage) No unnaturally blown up clusters Appearng clusters are often nterpretable Clusterng 22

23 Space-contractng example 1 Dendrodram generated by Sngle-lnkage Clusterng 23

24 Space-contractng example 2 Kel Kel Rostock Rostock Hamburg Hamburg Emden Emden Bremen Bremen Berln Hannover Magdeburg Münster Cottbus Berln Hannover Magdeburg Münster Cottbus Dresden Dresden Köln Erfurt Köln Erfurt Marburg Marburg Frankfurt Frankfurt Trer Trer Nürnberg Nürnberg Saarbrücken Saarbrücken Regensburg Regensburg Stuttgart Stuttgart München München Freburg Freburg Sngle lnkage WPGMC Clusterng 24

25 Space-dlatng example 1 Dendrodram generated by Ward s method Clusterng 25

26 Space-dlatng example 2 Kel Kel Rostock Rostock Hamburg Hamburg Emden Emden Bremen Bremen Berln Hannover Hannover Magdeburg Magdeburg Münster Münster Cottbus Berln Cottbus Dresden Dresden Köln Erfurt Köln Erfurt Marburg Marburg Frankfurt Frankfurt Trer Trer Nürnberg Nürnberg Saarbrücken Saarbrücken Regensburg Regensburg Stuttgart Stuttgart München München Freburg Freburg Ward s method Complete lnkage Clusterng 26

27 Space-conservng example 1 Dendrodram generated by UPGMA Clusterng 27

28 Space-conservng example 2 Kel Kel Rostock Rostock Hamburg Hamburg Emden Emden Bremen Bremen Berln Hannover Hannover Magdeburg Magdeburg Münster Münster Cottbus Berln Cottbus Dresden Dresden Köln Erfurt Köln Erfurt Marburg Marburg Frankfurt Frankfurt Trer Trer Nürnberg Nürnberg Saarbrücken Saarbrücken Regensburg Regensburg Stuttgart Stuttgart München München Freburg Freburg UPGMA WPGMA Clusterng 28

29 Matrx updatng algorthms Whch should be used? Outlers shall be vsble Sngle lnkage Unequally szed clusters expected Not space-dlatng methods Dfferng object densty n expected clusters WPGMA No-dea-just-try-order: Space-conservng > space-dlatng > spacecontractng Clusterng 29

30 Number of clusters How many natural classes has cluster analyss generated? Subjectve decson of researcher Analyss of mergng values Large step = rather dssmlar clusters = stop Plot number of clusters aganst mergng values Graph flattens = no new nformaton = stop Ward s method: Sgnfcance test possble Clusterng 30

31 Valdaton of results Results should be stable Plausble nterpretaton possble Repeat cluster analyss wth dfferent samples of the same populaton Dfferent results = both nvald, but Same results = not necessarly vald and not always possble due to lack of data Cophenetc correlaton, but Normal dstrbuton (wrongly?) assumed In dendrogram fewer (dfferent) values Clusterng 31

32 Valdaton of results Sgnfcance tests Used attrbutes: Useless because always sgnfcant Not used (but relevant) attrbutes: Useful but only possble when knowledge about classes already exsts Monte Carlo procedures Data set s created whch has the same global propertes as orgnal data but contans no classes Both sets are clustered & results compared Sgnfcant dfferences => results vald Clusterng 32

33 Attenton! A lot of factors determne the results of cluster analyss Very careful selecton of objects, attrbutes, (ds)smlarty measure, cluster method and matrx updatng algorthm Cluster analyss wll always output clusters f there are natural classes or not! Clusterng 33