Clustering. A. Bellaachia Page: 1

Transcription

1 Clusterng. Obectves.. Clusterng.... Defntons... General Applcatons.3. What s a good clusterng? Requrements 3 3. Data Structures 4 4. Smlarty Measures Standardze data Bnary varables Nomnal Varables Ordnal Varables Rato-scaled varables Varables of med types Clusterng approaches Maor approaches. 5.. Parttonng approach. 6. The K-means clusterng method. 7. The K-medods Clusterng Method Herarchal Clusterng AGNES (Agglomeratve Nestng) Dvsve Analyss: DIANA Analyss of herarchcal clusterng Outlers Statstcal Approach Dstance-Based Approach 0 A. Bellaacha Page:

2 . Obectves Technques to group data nto related classfy datasets and provde categorcal labels, e.g., sports, technology, kd, etc. Detecton of patterns Models to predct certan future behavors.. Clusterng.. Defntons Cluster: a collecton of data obects o Smlar to one another wthn the same cluster o Dssmlar to the obects n other clusters Cluster analyss o Groupng a set of data obects nto clusters Clusterng s unsupervsed classfcaton: no predefned classes Typcal applcatons o As a stand-alone tool to get nsght nto data dstrbuton o As a preprocessng step for other algorthms.. General Applcatons o Tet mnng: Document categorzaton Detecton of topcs Summarzaton o Tet Mnng: Web log analyss Detecton of groups of smlar access patterns A. Bellaacha Page:

3 o Bo-nformatcs: Gene epresson data: detecton of cancer genes o Others: Image processng Market analyss Etc..3. What s a good clusterng? A good clusterng method wll produce hgh qualty clusters wth o Hgh ntra-class smlarty o Low nter-class smlarty The qualty of a clusterng result depends on both the smlarty measure used by the method and ts mplementaton. The qualty of a clusterng method s also measured by ts ablty to dscover some or all of the hdden patterns..4. Requrements Scalablty Ablty to deal wth dfferent types of attrbutes Dscovery of clusters wth arbtrary shape Mnmal requrements for doman knowledge to determne nput parameters Able to deal wth nose and outlers Insenstve to order of nput records Hgh dmensonalty Incorporaton of user-specfed constrants Interpretablty and usablty A. Bellaacha Page: 3

4 3. Data Structures Data Matr (two modes) n f f nf p p np Dssmlarty (or smlarty) matr 0 d(,) d(3,) : d( n,) 0 d(3,) : d( n,) 0 : 0 4. Smlarty Measures Dssmlarty/Smlarty metrc: Smlarty s epressed n terms of a dstance functon, whch s typcally metrc: d(, ) There s a separate qualty functon that measures the goodness of a cluster. The defntons of dstance functons are usually very dfferent for nterval-scaled, boolean, categorcal, ordnal and rato varables. A. Bellaacha Page: 4

5 Weghts should be assocated wth dfferent varables based on applcatons and data semantcs. It s hard to defne smlar enough or good enough o The answer s typcally hghly subectve. Type of data n clusterng analyss o Interval-scaled varables o Bnary varables o Nomnal, ordnal, and rato varables o Varables of med types 4.. Standardze data Calculate the mean absolute devaton: s f = ( n m + m + + m f f f f nf f ) Where n m = ( f f f nf ). z-score: Calculate the standardzed measurement z f = f m s f f Usng mean absolute devaton s more robust than usng standard devaton A. Bellaacha Page: 5

6 A. Bellaacha Page: 6 Computaton of data smlarty Dstances are normally used to measure the smlarty or dssmlarty between two data obects Some popular ones nclude: Mnkowsk dstance: where = (,,, p) and = (,,, p) are two p-dmensonal data obects, and q s a postve nteger. If q =, d s Manhattan dstance If q =, d s Eucldean dstance: Propertes: o d(,) 0 o d(,) = 0 o d(,) = d(,) o d(,) d(,k) + d(k,) q q p p q q d ) ( ), ( = ), ( p p d = ) ( ), ( p p d =

7 Also, one can use weghted dstance, parametrc Pearson product moment correlaton, or other dsmlarty measures 4.. Bnary varables A contngency table for bnary data 0 sum a c a+ c 0 b d b+ d sum a+ b c+ d p Smple matchng coeffcent (nvarant, f the bnary varable s symmetrc): d(, ) = b c a+ b+ + c+ d Jaccard coeffcent (nonnvarant f the bnary varable s asymmetrc): d(, ) = b c a+ + b+ c A. Bellaacha Page: 7

8 Eample: Name Gender Fever Cough Test- Test- Test-3 Test-4 Jack M Y N P N N N Mary F Y N P N P N Jm M Y P N N N N gender s a symmetrc attrbute The remanng attrbutes are asymmetrc bnary Let the values Y and P be set to, and the value N be set to d( ack, mary) = = d( ack, m) = = d( m, mary) = = Nomnal Varables A generalzaton of the bnary varable n that t can take more than states, e.g., red, yellow, blue, green Method : Smple matchng o m: # of matches, p: total # of varables A. Bellaacha Page: 8

9 d(, ) = p p m Method : use a large number of bnary varables o Creatng a new bnary varable for each of the M nomnal states 4.4. Ordnal Varables An ordnal varable can be dscrete or contnuous Order s mportant, e.g., rank Can be treated lke nterval-scaled o Replace f by ther rank: r {,, M } f f o Map the range of each varable onto [0, ] by replacng -th obect n the f-th varable by z f = r M f f Compute the dssmlarty usng methods for ntervalscaled varables A. Bellaacha Page: 9

10 4.5. Rato-scaled varables Rato-scaled varable: a postve measurement on a nonlnear scale, appromately at eponental scale, such as AeBt or Ae-Bt Methods: Treat them lke nterval-scaled varables not a good choce! (why? the scale can be dstorted) Apply logarthmc transformaton: yf = log(f) Treat them as contnuous ordnal data treat ther rank as nterval-scaled 4.6. Varables of med types A database may contan all the s types of varables Symmetrc bnary, asymmetrc bnary, nomnal, ordnal, nterval and rato One may use a weghted formula to combne ther effects: p Σ f d(, ) = Σ δ ( f ) ( f ) = p ( f ) δ f = f s bnary or nomnal: d (f) = 0 f f = f, or d (f) = o.w. f s nterval-based: use the normalzed dstance f s ordnal or rato-scaled o compute ranks rf and o treat zf as nterval-scaled z f = r f M f d A. Bellaacha Page: 0

11 5. Clusterng approaches 5.. Maor approaches Parttonng algorthms: Construct varous parttons and then evaluate them by some crteron Herarchy algorthms: Create a herarchcal decomposton of the set of data (or obects) usng some crteron Densty-based: based on connectvty and densty functons Grd-based: based on a multple-level granularty structure Model-based: A model s hypotheszed for each of the clusters and the dea s to fnd the best ft of that model to each other 5.. Parttonng approach Parttonng method: Construct a partton of a database D of n obects nto a set of k clusters Gven a k, fnd a partton of k clusters that optmzes the chosen parttonng crteron o Global optmal: ehaustvely enumerate all parttons o Heurstc methods: k-means and k-medods algorthms o k-means (MacQueen 67): Each cluster s represented by the center of the cluster o k-medods or PAM (Partton around medods) (Kaufman & Rousseeuw 87): Each cluster s represented by one of the obects n the cluster A. Bellaacha Page:

12 6. The K-means clusterng method Input: n obects (or ponts) and a number k Algorthm : o Step : Randomly place K ponts nto the space represented by the obects that are beng clustered. These ponts represent ntal group centrods. o Step : Assgn each obect to the group that has the closest centrod. o Step 3: When all obects have been assgned, recalculate the postons of the K centrods. o Repeat Steps and 3 untl the stoppng crtera s met. Algorthm : o Step : Partton obects nto k nonempty subsets o Step : Compute seed ponts as the centrods of the clusters of the current partton (the centrod s the center,.e., mean pont, of the cluster) o Step 3: Assgn each obect to the cluster wth the nearest seed pont Go back to Step, stop when no more new assgnment o Eample A. Bellaacha Page:

13 Stoppng crtera: o No change n the members of all clusters o when the squared error s less than some small threshold value α: Squared error se se = p c p m where m s the mean of all nstances n cluster c se() < α Propertes of k-means o Guaranteed to converge o Guaranteed to acheve local optmal, not necessarly global optmal. Often termnates at a local optmum. The global optmum may be found usng technques such as: determnstc annealng and genetc algorthms Analyss o Strength: Relatvely effcent: O(tkn), where n s # obects, k s # clusters, and t s # teratons. Normally, k, t << n. o Comparng: PAM: O(k(n-k) ), CLARA: O(ks + k(nk)) = o Weakness Applcable only when mean s defned, then what about categorcal data? Need to specfy k, the number of clusters, n advance Unable to handle nosy data and outlers Not sutable to dscover clusters wth non-conve shapes k A. Bellaacha Page: 3

14 Varatons of K-means method: A few varants of the k-means whch dffer n o Selecton of the ntal k means o Dssmlarty calculatons o Strateges to calculate cluster means Handlng categorcal data: k-modes (Huang 98) o Replacng means of clusters wth modes o Usng new dssmlarty measures to deal wth categorcal obects o Usng a frequency-based method to update modes of clusters o A mture of categorcal and numercal data: k-prototype method Drawbacks of k-mean method o The k-means algorthm s senstve to outlers! Snce an obect wth an etremely large value may substantally dstort the dstrbuton of the data. o K-Medods: Instead of takng the mean value of the obect n a cluster as a reference pont, medods can be used, whch s the most centrally located obect n a cluster. A. Bellaacha Page: 4

15 7. The K-medods Clusterng Method Fnd representatve obects, called medods, n clusters PAM (Parttonng Around Medods, 987) o starts from an ntal set of medods and teratvely replaces one of the medods by one of the nonmedods f t mproves the total dstance of the resultng clusterng o PAM works effectvely for small data sets, but does not scale well for large data sets CLARA (Kaufmann & Rousseeuw, 990) CLARANS (Ng & Han, 994): Randomzed samplng Focusng + spatal data structure (Ester et al., 995) A. Bellaacha Page: 5

16 8. Herarchal Clusterng Use dstance matr as clusterng crtera. Ths method does not requre the number of clusters k as an nput, but needs a termnaton condton Step 0 Step Step Step 3 Step 4 agglomeratve a a b b a b c d e c d e d e c d e dvsve Step 4 Step 3 Step Step Step AGNES (Agglomeratve Nestng) Introduced n Kaufmann and Rousseeuw (990) Implemented n statstcal analyss packages, e.g., Splus Use the Sngle-Lnk method and the dssmlarty matr. Merge nodes that have the least dssmlarty Go on n a non-descendng fashon Eventually all nodes belong to the same cluster A. Bellaacha Page: 6

17 A Dendrogram Shows How the Clusters are Merged Herarchcally o Decompose data obects nto a several levels of nested parttonng (tree of clusters), called a dendrogram. o A clusterng of the data obects s obtaned by cuttng the dendrogram at the desred level, then each connected component forms a cluster. A. Bellaacha Page: 7

18 8.. Dvsve Analyss: DIANA Introduced n Kaufmann and Rousseeuw (990) Implemented n statstcal analyss packages, e.g., Splus Inverse order of AGNES Eventually each node forms a cluster on ts own Analyss of herarchcal clusterng Maor weakness of agglomeratve clusterng methods o do not scale well: tme complety of at least O(n), where n s the number of total obects Integraton of herarchcal wth dstance-based clusterng o BIRCH (996): uses CF-tree and ncrementally adusts the qualty of sub-clusters o CURE (998): selects well-scattered ponts from the cluster and then shrnks them towards the center of the cluster by a specfed fracton o CHAMELEON (999): herarchcal clusterng usng dynamc modelng. A. Bellaacha Page: 8

19 9. Outlers What are outlers? o The set of obects are consderably dssmlar from the remander of the data o Eample: Sports: Mchael Jordon, Wayne Gretzky, Problem o Fnd top n outler ponts Applcatons: o Credt card fraud detecton o Telecom fraud detecton o Customer segmentaton o Medcal analyss 9.. Statstcal Approach Assume a model underlyng dstrbuton that generates data set (e.g. normal dstrbuton) Use dscordancy tests dependng on o Data dstrbuton o Dstrbuton parameter (e.g., mean, varance) o Number of epected outlers Drawbacks o Most tests are for sngle attrbute o In many cases, data dstrbuton may not be known A. Bellaacha Page: 9

20 9.. Dstance-Based Approach Introduced to counter the man lmtatons mposed by statstcal methods o We need mult-dmensonal analyss wthout knowng data dstrbuton. Dstance-based outler: A Outler(p, D)-outler s an obect O n a dataset T such that at least a fracton p of the obects n T les at a dstance greater than D from O Algorthms for mnng dstance-based outlers o Inde-based algorthm: Use R-tree ndeng structure. It takes O(k*n ) wthout the cost of buldng the tree. o Nested-loop algorthm: Dvde the dataset nto blocks and look for outlers n block by block. It has the same complety as nde-based algorthm. o Cell-based algorthm: Dvde the data space nto cells and look for outlers cell-by-cell rather than pont-by-pont. It takes O(n ). A. Bellaacha Page: 0