Graph-based Clustering

Transcription

1 Graphbased Clusterng Transform the data nto a graph representaton ertces are the data ponts to be clustered Edges are eghted based on smlarty beteen data ponts Graph parttonng Þ Each connected component s a cluster

2 Clusterng as Graph Parttonng To thngs needed:. An obectve functon to determne hat ould be the best ay to cut the edges of a graph. An algorthm to fnd the optmal partton (optmal accordng to the obectve functon)

3 Obectve Functon for Parttonng Suppose e ant to partton the set of vertces nto to sets: and One possble obectve functon s to mnmze graph cut v v Cut( ). v 4. v 5 s eght of the edge beteen nodes and v v v 4. v 5 v v 6 v Cut. Cut.4 v 6

4 Obectve Functon for Parttonng Lmtaton of mnmzng graph cut: v.. v 3.3 v 5. v 4. Cut. v v 6 The optmal soluton mght be to splt up a sngle node from the rest of the graph! Not a desrable soluton

5 Obectve Functon for Parttonng We should not only mnmze the graph cut; but also look for balanced clusters + + d d d here ) Cut( ) Cut( ) Normalzed cut( ) Cut( ) Cut( ) Rato cut( and are the set of nodes n parttons and s the number of nodes n partton

6 Eample v v 5 v v 5. v v 4... v 3.3. v 4. v v 6 v v 6 Cut. Cut. Rato cut./ +./5. Normalzed cut./. +./.5.7 Rato cut./3 +./3.3 Normalzed cut./ +./.6.53

7 Eample If graph s uneghted (or has the same edge eght) v v 5 v v 5 v 3 v 4 v 3 v 4 v v 6 v v 6 Cut Cut Rato cut / + /5. Normalzed cut / + /9. Rato cut /3 + /3.67 Normalzed cut /5 + /5.

8 Algorthm for Graph Parttonng Ho to mnmze the obectve functon? We can use a heurstc (greedy) approach to do ths u Eample: METIS graph parttonng An elegant ay to optmze the functon s by usng deas from spectral graph theory u Ths leads to a class of algorthms knon as spectral clusterng

9 Spectral Clusterng Spectral propertes of a graph Spectral propertes: egenvalues/egenvectors of the adacency matr can be used to represent a graph There ests a relatonshp beteen spectral propertes of a graph and the graph parttonng problem

10 Spectral Propertes of a Graph Start th a smlarty/adacency matr W of a graph Defne a dagonal matr D D n ì ï í k ï î k f otherse If W s a bnary / matr then D represents the degree of node

11 Prelmnares û ù ë é W v v 3 v v 5 v 4 v 6 To blockdagonal matrces ï î ï í ì otherse f D n k k û ù ë é D To clusters

12 Graph Laplacan Matr û ù ë é W v v 3 v v 5 v 4 v 6 To block matrces W D L Laplacan û ù ë é L Laplacan also has a block structure

13 Propertes of Graph Laplacan L (D W) s a symmetrc matr L s a postve semdefnte matr Consequence: all egenvalues of L are ³

14 Spectral Clusterng Consder a data set th N data ponts. Construct an N N smlarty matr W. Compute the N N Laplacan matr L D W 3. Compute the k smallest egenvectors of L a) Each egenvector v s an N column vector b) Create a matr contanng egenvectors v v.. v k as columns (you may eclude the frst egenvector) 4. Cluster the ros n usng kmeans or other clusterng algorthms nto K clusters

15 Eample

16 Summary Spectral propertes of a graph (.e. egenvalues and egenvectors) contan nformaton about clusterng structure To fnd k clusters apply kmeans or other algorthms to the frst k egenvectors of the graph Laplacan matr

17 Mnmum Spannng Tree Gven the MST of data ponts remove the longest edge (nconsstent) and then the net longest edge.

18

19 One useful statstcs that can be estmated from the MST s the edge length dstrbuton For nstance n the case of dense clusters mmersed n a sparse set of ponts

20 Cluster aldty l l l l l Whch clusterng method s approprate for a partcular data set? Ho does one determne hether the results of a clusterng method truly characterze data? Ho do you kno hen you have a good set of clusters? Is t unusual to fnd a cluster as compact and solated as the observed clusters? Ho to guard aganst elaborate nterpretaton of randomly dstrbuted data?

21 Cluster aldty Clusterng algorthms fnd clusters even f there are no natural clusters n data to desgn D unform ne methods data ponts dffcult to KMeans; valdate K3 Cluster stablty: Perturb data by bootstrappng. Ho do clusters change over the ensemble

22 Herarchcal Clusterng Herarchcal clusterng s a method of cluster analyss hch seeks to buld a herarchy of clusters. To approaches: Agglomeratve ("bottom up ): each pont starts n ts on cluster and pars of clusters are merged as one moves up the herarchy; more popular Dvsve ("top don ): all ponts start n one cluster and splts are performed recursvely as one moves don the herarchy Ho to defne smlarty beteen to clusters or a pont and a cluster?

23 Agglomeratve Clusterng Eample Cluster s elements {a} {b} {c} {d} {e} and {f} n D; use Eucldean dstance as a smlarty Buld the herarchy from the ndvdual elements by progressvely mergng clusters Whch elements to merge n a cluster? Usually merge the to closest elements accordng to the chosen dstance

24 Snglelnk v. Completelnk Herarchcal Clusterng Suppose e have merged the to closest elements b and c to obtan clusters {a} {b c} {d} {e} and {f} To merge them further e need to take the dstance beteen {a} and {b c}. To common ays to defne dstance beteen to clusters: The mamum dstance beteen elements of each cluster (also called completelnkage clusterng): ma { d ( y ) : A y B } The mnmum dstance beteen elements of each cluster (snglelnkage clusterng): mn { d ( y ) : A y B } Stop clusterng ether hen the clusters are too far apart to be merged or hen there s a suffcently small number of clusters

25 D PCA Proecton of Irs Data

26 Mnmum Spannng Tree Clusterng of D PCA Proecton of Irs Data

27 KMeans Clusterng of Irs Data (Clusterng Assgnment shon on D PCA Proecton)

28 Snglelnk Clusterng of Irs Data

29 Completelnk Clusterng of Irs Data

30 Angkor Wat Hndu temple bult by a Khmer kng ~5AD; Khmer kngdom declned n the 5th century; French eplorers dscovered the hdden runs n late 8 s

31 Apsaras of Angkor Wat Angkor Wat contans the most unque gallery of ~ omen depcted by detaled full body portrats What facal types are represented n these portrats?

32 Clusterng of Apsara Faces Sngle Lnk 7 facal landmarks 7 landmarks Sngle Lnk clusters Ho to valdate the clusters or groups? Shape algnment

33 Ground Truth Khmer Dance and Cultural Center

34 Eploratory Data Analyss D MDS Proecton of the Smlarty matr Clusterng th large eghts assgned to chn and nose Eample devata faces from the clusters dffer largely n chn and nose thereby reflectng the eghts chosen for smlarty

35 Eploratory Data Analyss 3D MDS Proecton of the Smlarty matr

36

37 Spectral Clusterng & Graph Parttonng We have shon that the spectral propertes of the graph s related to the clusters Ho s t related to mnmzng graph cut?

38 Graph Parttonng Recall the follong obectve of graph parttonng + + ) Cut( here ) Cut( ) Cut( ) Normalzed cut( ) Cut( ) Cut( ) Rato cut( d d d

39 Rato Cut Let ndcates membershp of node v n a cluster: Also: here L s the graph Laplacan matr ï ï î ï í ì f f v v ( ) ( ) ( ) + T L

40 Rato Cut ( ) ( ) ) ( ) ( ) ( ) ( RatoCut Cut Cut Cut L T ø ö ç ç è æ ø ö ç ç è æ ø ö ç ç è æ + + ø ö ç ç è æ ø ö ç ç è æ + + ø ö ç ç è æ + ø ö ç ç è æ + +

41 Rato Cut Therefore: mn RatoCut( ) mn T L Thus e have related rato cut to Laplacan matr L But there s one ssue: u Trval soluton s s a vector of all zeros u Need to look for a nontrval soluton Look for constrants that must be satsfed by T n The soluton must be orthogonal to the vector of all s

42 Rato Cut Another constrant that must be satsfed by : L RatoCut T mn ) ( mn n n T + ø ö ç è æ + ø ö ç è æ

43 Rato Cut subect to: Ths s a constraned optmzaton problem here Instead e solve a relaaton of the problem: T L mn n T ( ) L L F n L F T T l l l Þ ï ï î ï í ì f f v v

44 Puttng It Altogether We have shon that Mnmzng graph cut s equvalent to fndng that mn T L such that T n Soluton for s gven by the egenvectors of L Thus the spectral decomposton of graph Laplacan s equvalent to the soluton of the graph parttonng problem

45 Spectral Clusterng th Rato Cut T T mn L mn l mnl But l mn th egenvector ( ) T Snce e ant a soluton here T so ¹ Instead of the smallest egenvalue e look for the egenvector correspondng to the net smallest egenvalue In summary fndng the egenvector that corresponds to the second smallest egenvalue s a relaaton of the rato cut graph parttonng problem (for k)

46 Propertes of Graph Laplacan L (D W) s a symmetrc matr L s a postve semdefnte matr For all realvalued vectors : T L ³ Consequence: all egenvalues of L are ³ ( ) ø ö ç ç è æ + N T T T T d d W D W D L ) (here ) (

47 Propertes of Laplacan Matr û ù ë é û ù ë é û ù ë é û ù ë é û ù ë é û ù ë é û ù ë é û ù ë é dd dd dd d d d d dd d d d d D D D D D D De D D D We e Suppose

48 Propertes of Laplacan Matr De We Þ ( D W ) e Þ Le Egenvalue equaton : Le le Snce e ¹ [..] T therefore l s an egenvalue of L th the correspondng egenvector e [ ] T Furthermore snce L s postve semdefnte s the smallest egenvalue of L

49 Propertes of Laplacan Matr More generally f Then u There are k egenvalues of L hch have the value u The correspondng egenvectors are: û ù ë é L k L L L û ù ë é û ù ë é û ù ë é e e e here e s [ ] T

50 Propertes of Laplacan Matr Egenvalues of L: û ù ë é L 3 3. Egenvectors of L: û ù ë é v v 3 v v 5 v 4 v 6 û ù ë é L

51 Propertes of Laplacan Matr v v 5 v 3 v 4 If e cluster the data usng only the frst egenvectors e get the to desred clusters v v 6 Egenvalues of L: Egenvectors of L: L é ë. 3 ù 3û é ë ù.4.8 û

52 Propertes of Laplacan Matr û ù ë é W W D L Laplacan û ù ë é 3 3 L v v 3 v v 5 v 4 v 6 Clusters are no longer ell separated

53 Propertes of Laplacan Matr û ù ë é 3 3 L û ù ë é û ù ë é L Egenvalues of L: Egenvectors of L: v v 3 v v 5 v 4 v 6

54 Propertes of Laplacan Matr Egenvalues of the graph Laplacan: Egenvectors of Laplacan: Can be used to obtan 3 clusters é ë ù û