Clustering Data. Clustering Methods. The clustering problem: Given a set of objects, find groups of similar objects

Transcription

1 Clusterng Data The lusterng problem: Gven a set of obets, fnd groups of smlar obets Cluster: a olleton of data obets Smlar to one another wthn the same luster Dssmlar to the obets n other lusters What s smlar? Defne approprate metrs Applatons n marketng, mage proessng, bology Clusterng Methods K-Means and K-medodsK algorthms PAM, CLARA, CLARANS [Ng and Han, VLDB 99] Herarhal algorthms CURE [Guha et al, SIGMOD 998] BIRCH [Zhang et al, SIGMOD 996] CHAMELEON [IEEE Computer, 999] Densty based algorthms DENCLUE [Hnneburg, Kem, KDD 998] DBSCAN [Ester et al, KDD 96] Subspae Clusterng CLIQUE [Agrawal et al, SIGMOD 998] PROCLUS [Agrawal et al, SIGMOD 999] ORCLUS: [Aggarwal, and Yu, SIGMOD ] DOC: [Proopu, Jones, Agarwal, and Mural, SIGMOD, ] K-Means and K-MedodsK algorthms Mnmzes the sum of square dstanes of ponts to luster representatve K k ( xk ) k E = x m Effent teratve algorthms (O(n))

2 . Ask user how many lusters they d lke. (e.g. K=5). Randomly guess K luster enter loatons *based on sldes by Padhra Smyth UC, Irvne 5 Eah data pont fnds out whh enter t s losest to. *based on sldes by Padhra Smyth UC, Irvne 6 Redefne eah enter fndng out the set of the ponts t owns *based on sldes by Padhra Smyth UC, Irvne 7

3 Problems wth K-Means K type algorthms Advantages - Relatvely effent: O(tkn), - where n s the number of obets, k s the number of lusters, and t s the number of teratons. Normally, k, t << n. n - Often termnates at a loal optmum. Problems Clusters are approxmately spheral Unable to handle nosy data and outlers Hgh dmensonalty may be a problem The value of k s an nput parameter 8 Spetral Clusterng (I) Algorthms that luster ponts usng egenvetors of matres derved from the data Obtan data representaton n the low-dmensonal spae that an be easly lustered Varety of methods that use the egenvetors dfferently [Ng, Jordan, Wess. NIPS ] [Belkn, Nyog, NIPS ] [Dhllon, KDD ] [Bah, Jordan NIPS ] [Kamvar, Klen, Mannng. IJCAI ] [Jn, Dng, Kang, NIPS 5] 9 Spetral Clusterng methods Method # Partton usng only one egenvetor at a tme Use proedure reursvely Example: Image Segmentaton Method # Use k egenvetors (k hosen by user) Dretly ompute k-way parttonng Expermentally t has been seen to be better ([Ng, Jordan, Wess. NIPS ][Bah, Jordan, NIPS ]).

4 Kernel-based k-means k lusterng (Dhllon et al., ) Data not lnearly separable Transform data to hgh-dmensonal spae usng kernel φ a funton that maps X to a hgh dmensonal spae Use the kernel trk to evaluate the dot produts: a kernel funton k (x, y) omputes φ(x) φ(y) luster kernel smlarty matrx usng weghted kernel K-Means. K The goal s to mnmze the followng obetve funton: k k ({ π } ) = α ( ) = ϕ x J where m = = x π m αϕ( ) x x π α x π Herarhal Clusterng Two bas approahes: mergng smaller lusters nto larger ones (agglomeratve), splttng larger lusters (dvsve) vsualze both va dendograms shows nestng struture merges or splts = tree nodes Step Step Step Step Step a b d e a b d e a b d e d e Step Step Step Step Step agglomeratve dvsve Herarhal Clusterng: Complexty Quadrat algorthms Runnng tme an be mproved usng samplng [Guha et al, SIGMOD 998] r usng the trangle nequalty (when t holds) *based on sldes by Padhra Smyth UC, Irvne

5 Densty-based Algorthms Clusters are regons of spae whh have a hgh densty of ponts Clusters an have arbtrary shapes Regons of hgh densty Clusterng Hgh Dmensonal Data Fundamental to all lusterng tehnques s the hoe of dstane measure between data ponts; q ( ) ( ) x x = x x D, k k= Assumpton: All features are equally mportant; Suh approahes fal n hgh dmensonal spaes Feature seleton (Dy and Brodley, ) Dmensonalty Reduton k 5 Applyng Dmensonalty Reduton Tehnques Dmensonalty reduton tehnques (suh as Sngular Value Deomposton) an provde a soluton by redung the dmensonalty of the dataset: Drawbaks: The new dmensons may be dffult to nterpret They don t mprove the lusterng n all ases 6 5

6 Applyng Dmensonalty Reduton Tehnques Dfferent dmensons may be relevant to dfferent lusters In General: Clusters may exst n dfferent subspaes, omprsed of dfferent ombnatons of features 7 Subspae lusterng Subspae lusterng addresses the problems that arse from hgh dmensonalty of data It fnds lusters n subspaes: subsets of the attrbutes Densty based tehnques CLIQUE: Agrawal, Gehrke, Gunopulos, Raghavan (SIGMOD 98) DOC: Proopu,, Jones, Agarwal,, and Mural,, (SIGMOD, ) Iteratve algorthms PROCLUS: Agrawal, Proopu,, Wolf, Yu, Park (SIGMOD 99) ORCLUS: Aggarwal,, and Yu (SIGMOD ). 8 Subspae lusterng Densty based lusters: fnd dense areas n subspaes Identfyng the rght sets of attrbutes s hard Assumng a global threshold allows bottom-up algorthms Constraned monotone searh n a latte spae salary x salary x age age 9 6

7 y Loally Adaptve Clusterng Eah luster s haraterzed by dfferent attrbute weghts (Fredman and Meulman, Domenon ) ( w x, w y ), w x > w y ( w x, w y ), w y > w x Loally Adaptve Clusterng : Example before loal transformatons after loal transformatons 9 Cluster transformed by loal weghts 8 Cluster transformed by loal weghts x LAC [C. Domenon et al SDM] Computng the weghts: X : averagesquared dstane along dmenson of ponts n w X = S S from = l e ( x) X e x S X l Exponental weghtng sheme Result : w, w, L, w k A weght vetor for eah luster 7

8 Convergene of LAC The LAC algorthm onverges to a loal mnmum of the error funton: E ( C, W) = subet to the onstrants C= k q = = w X e = [ L ] W = [ w Lw ] k k q w = EM-lke onvergene: Hdden varables: assgnments of ponts to entrods ( S ) E-step: fnd the values of S gven w, M-step: fnd w, that mnmze E ( C, W ) gven urrent estmates S. Sem-Supervsed Supervsed Clusterng Clusterng s applable n many real lfe senaros there s typally a large amount of unlabeled data avalable. The use of user nput s rtal for the suess of the lusterng proess the evaluaton of the lusterng auray. User nput s gven as Labeled data Constrants Learnng approahes that use labeled data/onstrants unlabeled data have reently attrated the nterest of researhers Motvatng sem-supervsed supervsed learnng Data are orrelated. To reognze lusters, a dstane funton should reflet suh orrelatons. Dfferent attrbutes may have dfferent degree of relevane dependng on the applaton / user requrements A lusterng algorthm does not provde the rteron to be used. Sem-supervsed algorthms: Defne lusters takng nto aount labeled data or onstrants f we have labels we wll onvert them to onstrants 5 8

9 8 6 C 8 6 a user may want the ponts n B and C to belong to the same luster Cluster A B Cluster Cluster (a) Cluster () (b) Cluster Cluster The rght lusterng may depend on the user s perspetve. Fully automat tehnques are very lmted n addressng ths problem 6 Clusterng under onstrants Use onstrants to learn a dstane funton Ponts surroundng a par of must-lnk/annot-lnk ponts should be lose to/far from eah other gude the algorthm to a useful soluton Two ponts should be n the same/dfferent lusters 7 Defnng the onstrants A set of ponts X = {x,, x n } on whh sets of must-lnk(s lnk(s) and annot-lnk onstrants(d) have been defned. Must-lnk onstrants S: {(x, x ) n X }: x and x should belong to the same luster Cannot-lnk onstrants D: {(x, x ) n X} : x and x annot belong to the same luster Condtonal onstrants δ-onstrant and ε-onstrant 8 9

10 Clusterng wth onstrants: Feasblty ssues Constrants provde nformaton that should be satsfed. Optons for onstrant-based lusterng Satsfy all onstrants Not always possble: A wth B, B wth C, C not wth A. Satsfy as many onstrants as possble 9 Clusterng wth onstrants: Feasblty ssues Any ombnaton of onstrants nvolvng annot-lnk onstrants s generally omputatonally ntratable (Davdson & Rav, ISMB ), Reduton to k-olorablty problem: Can you luster (olor) the graph wth the annot-lnk edges usng k olors (lusters)? Feasblty under ML and ε ε-onstrant: Any node x should have an ε-neghbor n ts luster (another node y suh that D(x,y) ε) S = {x S : x does not have an ε neghbor}={s 5, s 6 } Eah of these should be n ther own luster x x x x x 5 x 6 Compute the Transtve Closure on ML={CC CC r } ML(x,x ), ML(x,x ), ML(x,x 5 ) x x x x x 5 x 6 Infeasble: ff, : x CC, x S *S. Basu, I. Davdson,turoral ICDM 5

11 Clusterng based on onstrants Algorthm spef approahes Inorporate onstrants nto the lusterng algorthm COP K-Means K (Wagstaff( et al, ) Herarhal lusterng (I.( Davdson, S. Rav, 5) Inorporate metr learnng nto the algorthm MPCK-Means Means (Blenko( et al ) HMRF K-Means K (Basu( et al ) Learnng a dstane metr (Xng et al. ) Kernel-based onstraned lusterng (Kuls et al. 5) COP K-Means K (I) [Wagstaff et al, ] Sem-supervsed varants of K-Means Constrants: Intal bakground knowledge Must-lnk & Cannot-lnk onstrants are used n the lusterng proess Generate a partton that satsfes all the gven onstrants K. Wagstaff, C. Carde, S. Rogers, and S. Shroedl. Constraned k-means lusterng wth bakground knowledge. In ICML, pages ,. COP K-Means K (II) The algorthm takes n a data set (D) a set of must-lnk onstrants (Con = ) a set of annot-lnk onstrants (Con ). K-Means Clusterng based on onstrants When updatng luster assgnments, we ensure that none of the spefed onstrants are volated. Clusterng satsfyng user onstrants Assgn eah pont d to ts losest luster C. Ths wll sueed unless a onstrant would be volated. If there s another pont d = that must be assgned to the same luster as d, but that s already n some other luster, or there s another pont d that annot be grouped wth d but s already n C, then d annot be plaed n C. Constrants are never broken; f a legal luster annot be found for d, the empty partton (f g ) s returned.

12 Herarhal Clusterng based on onstrants [I. Davdson, S. Rav, 5] Instane: A set S of nodes, the (symmetr) dstane d(x,y) for eah par of nodes x and y and a olleton C of onstrants Queston: Can we reate a dendrogram for S so that all the onstrants n C are satsfed? Davdson I. and Rav, S. S. Herarhal Clusterng wth Constrants: Theory and Prate, In PKDD 5 5 Constrants and Irreduble Clusterngs A feasble lusterng C={C, C,,, C k } of a set S s rreduble f no par of lusters n C an be merged to obtan a feasble lusterng wth k- lusters. X={x, x,, x k }, Y={y, y,, y k }, Z={z, z,, z k }, W={w, w,,, w k } CL-onstrants {x, x }, {w, w }, {y, z },, If mergers are not done orretly, the dendrogram may stop prematurely Feasble lusterng wth k lusters: {x, y }, {x, y },,, {x{ k, y k }, {z, w }, {z,w },,, {z{ k, w k } But then get stuk Alternatve s: {x, w, y, y,, y k }, {x, w, z, z,, z k }, {x, w },, {x k, w k } 6 MPCK-Means Means [Blenko et al ] Inorporate metr learnng dretly nto the lusterng algorthm Unlabeled data nfluene the metr learnng proess Obetve funton Sum of total square dstanes between the ponts and luster entrods Cost of volatng the par-wse onstrants M. Blenko, S. Basu, R. Mooney. Integratng Constrants and Metr Learnng n Sem-supervsed lusterng. In Proeedngs of the st ICML Conferene, July. 7

13 Unfyng onstrants and Metr learnng J ( x mpkm, x = w f ) M x X M x µ l Al log(det( A )) ( x, x )[ l l ] w fc( x, x )[ l = l ] ( x, x ) C Generalzed K-means dstorton funton, Assumes eah luster s generated by a gaussan wth ovarane matrx A l - l Volaton must-lnk onstrants Penalty funtons Volaton annot-lnk onstrants 8 MPCK-Means Means approah Intalzaton: Use neghborhoods derved from onstrants to ntalze lusters Repeat untl onvergene (not guaranteed):. E-step: Assgn eah pont x to a luster to mnmze. M-step: dstane of x from the luster entrod onstrant volatons Estmate luster entrods C as means of eah luster Re-estmate estmate parameters A (dmenson weghts) to mnmze onstrant volatons 9 Learnng a dstane metr based on user onstrants The requrement s : learn the dstane measure to satsfy user onstrants. To smplfy the problem onsder the weghted Euldean dstane: dfferent weghts are assgned to dfferent dmensons - Other formulatons that map the ponts to a new spae an be onsdered, but are sgnfantly more omplex to optmze

14 Dstane Learnng as Convex Optmzaton [Xng et al. ] Goal: Learn a dstane metr between the ponts n X that satsfes the gven onstrants The problem redues to the followng optmzaton problem : gven that mn (x,x ) CL A (x,x ) ML x x A x x A A E. P. Xng, A. Y. Ng, M. I. Jordan, and S. Russell. Dstane metr learnng, wth applaton to lusterng wth sde-nformaton. In NIPS, Deember. Example: Learnng Dstane Funton Spae Transformed by Learned Funton Cannot-lnk Must-lnk Learnng Mahalanobs dstane Mahalanobs dstane = Euldean dstane parameterzed by matrx A T x y = ( x y ) A( x y ) A Typally A s dagonal

15 The Dagonal A Case Consderng the ase of learnng a dagonal A we an solve the orgnal optmzaton problem usng Newton-Raphson to effently optmze the followng g(a) = x x log x x A (x,x ) ML (x,x ) CL Use Newton Raphson Tehnque: x = x g(x)/g (x) A =A-g(A).J - (A) A A Kernel based Sem-supervsed lusterng A non-lnear transformaton, φ maps data to a hgh dmensonal spae the data are expeted to be more separable a kernel funton k (x, y) omputes φ(x) φ(y) The user gves onstrants The approprate kernel s reated based on onstrants [Kuls et al. 5] ( k ) = φ(x ) m w = k J{ π} = x π x,x ML x,x CL l= l l= l w Reward for onstrant satsfaton 5 Sem-Supervsed Supervsed Kernel-KMeans KMeans [Kuls et al. 5] Algorthm: Construts the approprate kernel matrx from data and onstrants Runs weghted kernel K-Means Input of the algorthm: Kernel matrx Kernel funton on vetor data or Graph affnty matrx Benefts: HMRF-KMeans and Spetral Clusterng are speal ases Fast algorthm for onstraned graph-based lusterng Kernels allow onstraned lusterng wth non-lnear luster boundares 6 5

16 Graph-based onstraned lusterng Constraned graph lusterng: mnmze ut n nput graph whle maxmally respetng a gven set of onstrants 7 Clusterng usng onstrants and luster valdty rtera Dfferent dstane metrs may satsfy the same number of onstrants One soluton s to apply a dfferent rteron that evaluates the resultng lusterng to hoose the rght dstane metr A general approah should: Learn an approprate dstane metr to satsfy the onstrants Determne the best lusterng w.r.t the defned dstane metr. 8 Cluster Valdty A problem we fae n lusterng s to defne the best parttonng of a data set,.e. number of lusters that fts a data set, apture the shape of lusters presentng n underlyng data set The lusterng results depend on the data set (data dstrbuton) Intal lusterng assumptons, algorthm nput parameters values 9 6

17 luster Seres luster Eps=, Nps= Eps=6, Nps= DBSCAN 5 luster luster luster luster luster luster (a) K-Means (b) S_Dbw luster valdty ndex [Halkd, Vazrganns, ICDM ] SDbw: a relatve algorthm-ndependent valdty ndex, based on Satterng and Densty between een lusters Man features of the proposed approah Valdty ndex S_Dbw. Based on the features of the lusters: evaluates the resultng lusterng as defned by the algorthm under onsderaton. selets for eah algorthm the optmal set of nput parameters wth regards to the spef data set. 5 S_Dbw defnton: Inter-luster Densty (ID) Dens_bw: Average densty n the area among lusters n relaton wth the densty of the lusters Dens _ bw( ) = ( ) = = densty ( u ), max{ densty ( v ), densty ( v )} n densty(u ) = f (x, u ),, f d(x, u) > stdev l f ( x, u ) = l=, otherwse where n = numberof tuples thatbelong to the lusters and,.e., xl S stdev v * *u *v 5 7

18 S_Dbw defnton: Intra-luster varane Average satterng of lusters: ( v ) σ = Sat( ) = σ ( X) where where p σ p x n = n k = ( ) p p x k x x s the p dmenson of th n X = x xk X k = k, n σ p v = n ( x v ) n p p k k = 5 S_Dbw() ) = Sat() Dens_bw() D Sat ~ & Dens_bw D D Sat & Dens_bw ~ D D5 Sat & Dens_bw Sat & Dens_bw 5 Mult-representatves vs. Sngle DS 56 a sngle representatve pont annot effently represent the shape of lusters n DS luster luster luster luster luster luster luster luster r = r = 55 8

19 ) Respetve Closest Representatve ponts For eah par of lusters (C, C ) we fnd the set of losest representatves of C wth respet to C : for eah v k n C = {(v k, v l ) v x C and mn(dst(v k, v x ))} RCR = prunng( CR p los_ rep Cluster C = (vk, v l ) CR v k v * v stdev l v shrunk v by s p u * Neghbourhood of Cluster C Respetve Closest Representatve ponts. The set of respetve representatve ponts of the lusters C and C s defned as the set of mutual losest representatves of the lusters under onern,.e. RCR = {(v k, v l ) v k = losest_rep (v l ) and v l = losest_rep (v k )}.e. RCR = CR CR Prunng mantans only the meanngful pars of representatve ponts 56 Inter luster densty Clusters separaton mples low densty among them Dens(C,C ) = RCR CR p d(los _ rep ) densty p= stdev p ( u ) Cluster C p los _ rep = (v k, v l ) Cluster C v shrunk by s v v k v Neghborhood p of u * stdev * vl Inter _ dens ( C) = max = =,.., { Dens( C, C )} 57 9