On the Efficiency of Swap-Based Clustering

Transcription

1 On the Effcency of Swap-Based Clusterng Pas Fränt and Oll Vrmaok Department of Computer Scence, Unversty of Joensuu, Fnland {frant, Abstract. Random swap-based clusterng s very smple to mplement and guaranteed to fnd the correct clusterng f terated long enough. However, ts quadratc dependency on the number of clusters can be too slow n case of some data sets. Determnstc selecton of the swapped prototype can speed-up the algorthm but only f the swap can be performed fast enough. In ths work, we ntroduce an effcent mplementaton of the swap-based heurstc and compare ts tme-dstorton effcency aganst random and determnstc varants of the swap-based clusterng, and repeated k-means. Keywords: effcency. Clusterng, prototype swap, probablstc method, k-means, 1 Introducton The clusterng can be found by a sequence of prototype swaps and by fne-tunng ther exact locaton by k-means as demonstrated n Fg. 1. A smple but effectve approach s to select the swap randomly: select the prototype to be swapped randomly and replace t to the locaton of a randomly selected data vector. Despte beng smple to mplement and effcent n practce, the quadratc dependency on the number of clusters can be a lmtng factor n cases when there are a large number of clusters. Several determnstc swap-based methods have been consdered n lterature by selectng the prototype to be swapped as the one that ncreases the cost functon value least [1], [2], or by mergng two exstng clusters [3], [4], [5] followng the sprt of agglomeratve clusterng. The new locaton of the prototype can be chosen ether by consderng locatons of all possble data vectors [3], splttng an exstng cluster [3], [6], or by usng some heurstc such as selectng the cluster wth the largest varance [1]. The swap-based approach has also been used for solvng n a related p-medan problem [7]. The man drawback of the above mentoned approaches s ther computatonal complexty. Even though the correct clusterng can be obtaned by much fewer teratons n comparson to random swap, the tme spent for the selectng the best prototype can easly overwegh the effcency of the random swap heurstc. Another drawback s that the determnstc swap may get stuck nto a locally optmal soluton f randomness s completely elmnated n the process. In terms of tme-dstorton effcency, a good compromse uses determnstc prototype selecton and random relocaton [8].

2 In ths paper, we propose a faster mplementaton for the determnstc selecton of the prototype, and compare t to the exstng swap-based varants: random swap, determnstc swap, hybrd of these two, and repeated k-means. We gve tme complexty analyss of the proposed method and show that t s as fast as the random heurstc n a sngle teraton, and s capable of fndng the same clusterng result n a fewer teratons. We compare expermentally n whch cases the proposed approach s more favorable than the random swap. One centrod, but two clusters. Two centrods, but only one cluster. Swap s made from centrod rch area to centrod poor area.. Current soluton Centrod swappng Local repartton Fne-tunng by K-means Fg. 1. Demonstraton of the swap-based clusterng for a sample ntal soluton for N=5000 data ponts from M=15 clusters. 2 Swap-based clusterng The effcency of a swap-based clusterng depends on how many teratons (swaps) are needed, and how much tme each teraton takes. In a recent study, t was shown that for a gven probablty of success, the effcency of the random swap [9], [10] has lnear dependency on the number of data vectors, quadratc dependency on the number of clusters, and nverse dependency on dmensonalty [11]. Ths s good n most cases but potentally faster method could be mplemented f the swap could be found calculated more effcently. Another advantage of the random swap method s that t s extremely smple to mplement. In comparson to k-means consstng of two steps (centrod step and

3 partton step), t has only one addtonal step: prototype swap. In most cases, ths step s ndependent on the metrc used and t s trval to mplement, whch makes t hghly useful for practcal purposes. Pseudo code of the algorthm s sketched n Fg. 2, whch produces for a gven nput data set (X), a set of prototypes (C), and the partton of the clusterng (P) as the output. Here f s the optmzaton functon to be mnmzed; typcally mean square error (MSE). Swap-based clusterng (X) C, P C Select random representatves (X); P Optmal partton(x, C); REPEAT T tmes (C new,) Swap(X, C); P new Local repartton(x, C new, P, ); C new, P new K-means(X, C new, P new ); IFf(C new, P new ) < f(c, P) THEN (C,P) C new, P new ; RETURN (C, P); Fg. 2. Structure of the general swap-based clusterng. (For detals and source code: Number of teratons At frst sght, the probablty for a successful swap seems to be rather small and a long number of teratons would therefore be needed. However, t s not necessary to fnd exactly the correct prototype for removal but selectng a neghbor cluster s suffcent. In practce, the number of teratons (T) needed depends on the probablty of falure (q), number of clusters (M), and the sze of neghborhood (α) as follows [11]: 2 M T = Θ - ln q 2 (1) The clusterng, however, can be found by much fewer swaps f the algorthm knows whch prototype should be swapped and where t should be re-located. The motvaton of ths paper s to develop a fast determnstc swappng technque that would be more effcent than the random swap, but at the same tme, avod the problem of gettng stuck nto a local mnmum, whch can easly happen when heurstc algorthms are desgned. The task seems to be straghtforward but s far from trval to mplement n practce. Methods for prototype selecton and replacement can be analytcally constructed by consderng all possble prototypes for the swap and selectng the locally optmal choce at each step. However, the challenge s to make ths wthout excessve computaton. Otherwse the method would be outperformed by the random swap technque snce t can performed much faster, and sgnfcantly larger number of canddate swaps can be tested n a much shorter tme.

4 2.2 Selectng the prototype to be swapped Swappng the prototype nvolves two desgn questons: whch prototype wll be swapped and where t should be relocated. The swappng makes one cluster to dsappear and creates a new cluster elsewhere n the data space. Several smple heurstc crtera have been consdered for the selecton: cluster wth smallest sze or smallest varance but these are too naïve to work n practce. Another approach s to merge two exstng clusters as n agglomeratve clusterng [2], or applyng splt-and-merge strategy as n [4], [5]. These are possble but operatng the clusters as entty can restrct the clusterng too much and s subect to the problem of gettng stuck nto local mnmum. Consderng the clusterng as an optmzaton problem (e.g. mnmzng mean square error), the most logcal approach s to swap the prototype whose removal from ts current locaton ncreases the cost functon least. Ths knd of local optmzaton strategy has been successfully appled n several clusterng algorthms [1], [2], [3], [4], [8], and also used n p-medan problem [7], whch dffers from the clusterng problems n the sense that the prototypes (facltes) are restrcted to a predefned set and not free parameters as n clusterng. 2.3 Effcent mplementaton of the selecton To mplement the above approach, a so-called removal cost must be calculated for each cluster as follows. Frst, for a gven data vector (x ) assocated n a cluster (p ), ts second nearest prototype (q ) s solved: q 1 m p 2 = arg mn x c (2) The removal cost for cluster () can now be estmated by summng up the dfferences f the vectors n the cluster are reparttoned to ther second nearest one (q ): D = p = ( d( x, c ) d( x, c )) [ 1 M ] q, where c denotes the prototypes, and d the dstance functon. A more accurate estmaton n [6] takes nto account that the prototypes wll be updated after the repartton, and should therefore be calculated as follows: nq D = d( x, cq ) d( x, c ) (4) p = nq + 1 Here n q refers to the sze of the secondary cluster. The drawback of these approaches s that they take N dstance calculatons for each of the M clusters. Thus, the overall tme complexty of the swap step becomes O(NM). Ths can be sgnfcantly larger than that of the random swap, whch takes only O(αN) tme per swap [11] where α s the number of neghbors. (3)

5 However, snce only a few prototypes wll change after the swap, t s more effcent to mantan both data structures (p and q ndces) after each swap as n [2, 7]. When a prototype s swapped, all vectors n the old cluster wll be reparttoned from the prmary partton (p) to ther secondary partton (q). The changes n the secondary partton (q) can be updated as descrbed n [2]. New secondary parttons must be resolved for all affected vector by a full search. Ths causes the followng necessary updates: q = p = : p q (5) : q = arg mn x c 1 m p 2 (6) Secondly, the same nformaton must also be updated after the cluster addton. The new prototype attracts vectors from the neghborng clusters. Partton of every vector that s closer to the new prototype (c ) than the one of ts current cluster, must be be updated as follows: p (7) q p (8) The full search nvolved n the argmn operaton requres O(M) tme. The cluster removed and the cluster added both have α neghbors, each consstng of N/M vectors, on average. The total tme complexty of the swap step therefore reduces to O(2α M N/M) = O(αN), whch s the same complexty as that of the random swap. 2.4 Addng new prototype We consder every possble data vector as the potental new locaton for the swapped prototype. Calculatng the cost of each of them s possble [3] but ths would be very neffcent takng O(αN 2 ) tme as there are N possble locatons, and each of them would requre O(αN) tme to perform the local repartton. To relax the tme requrements, we dvde the problem nto two sub tasks, whch wll be consdered ndependently of each other: 1. Select an exstng cluster. 2. Select the locaton wthn ths cluster. It s expected that the choce of the cluster s more mportant, and the exact locaton wthn the cluster s less sgnfcant snce k-means wll be appled to take care of the local fne-tunng of the prototype. Thus, the dea s to frst select the cluster, and then add the prototype somewhere nsde ths cluster. Ths reduces the number of choces down to M and the correspondng tme complexty to O(αNM). However, ths s stll too much. Heurstc selecton can also be consdered. The most natural s to choose the cluster havng the hghest dstorton [1]. Ths would reduce the tme complexty of ths step to O(M+N) = O(N), where the frst term orgnates from the selecton and the second

6 from the repartton. The cost (actually beneft) of addng the prototype c can be calculated as follows: E = d( x, c ) (9) p = We consder the followng heurstcs: - Current centrod + ε [1]. - Furthest vector. - Halfway of current centrod and furthest vector. - Random. These heurstcs can be performed n O(1)-O(N/M) tme plus O(αN) tme due to the repartton, whch can be performed ndependently on the selecton. The man alternatves for selectng the new locaton of the prototype are summarzed n Table 1. On the bass of ths analyss, we select the heurstc varant for the determnstc swap. Table 1. Man varants for creatng a new cluster. Varant Whch Whch Tme cluster locaton complexty Full search Try all Try all O(αN 2 ) Optmal cluster Try all Any heurstc O(αNM) Heurstc Largest dstorton Any heurstc O(αN) Random Random Random O(αN) 2.5 Demonstraton of the determnstc swap The determnstc swap s demonstrated n Fg. 3 for a random ntal soluton. The removal and addton cost for each cluster are lsted n Table 2. For the removal, clusters 1 and 2 are the best choces. Ther absence would ncrease the dstorton much less than the removal of any other cluster; the algorthm wll choose the cluster 1. For relocaton, cluster 12 causes currently the hghest dstorton by a large margn, and wll be chosen. In ths example, the furthest vector heurstc was appled.

7 One centrod, but two clusters. 1 Two centrods, but only one cluster Current soluton (MSE 1.73*10 9 ) Centrod swappng (MSE 4.89*10 9 ) Local repartton (MSE 1.59*10 9 ) Fne-tunng by K-means (MSE 1.33*10 9 ) Fg. 3. Demonstraton of determnstc swap for a gven ntal soluton usng MSE as the cost functon. Table 2. Removal and addton cost each cluster presented n Fg. 3. Removal Addton cost (D ) cost (E ) Removal Addton cost (D ) cost (E ) Combnatons of random and determnstc swap The man problem of the determnstc varant s that t ends up n sub-optmal local optma, and certan amount of randomness s stll needed n order to optmze the clusterng beyond ths lmt. The followng combnatons of random and determnstc technques are therefore consdered: - RR = random removal, random addton (=random swap). - RD = random removal, determnstc addton. - DR = determnstc removal, random addton. - DD = determnstc removal, determnstc addton.

8 Tme complextes of these varants are summarzed n Table 3. From the determnstc removal, the proposed faster mplementaton of DR and DD are denoted here as D 2 R and D 2 D, respectvely. The bottleneck of the determnstc swap s the removal, whch domnates the processng tme. Consderng the analyss n [11], the determnstc removal s more effcent than ts random counter part f M < αt, assumng that t always selects the correct cluster n 1 teraton, and random removal wthn T teratons. Wth the data set shown n Fg.3, ths s the case (M=15, α 4, T=20). The determnstc removal also stablzes the soluton faster than random removal, whch s shown by the decreased tme requred by the k-means teratons. The ncreased (theoretcal) tme complexty requred by the determnstc addton, on the other hand, s nsgnfcant n comparson to ts random counter-part. It s therefore possble that the RD varant mght be a good compromse between the random swap (RR) and determnstc swap (DD). However, the results even wth the slower method n [8] revealed the opposte: the determnstc removal and random addton (DR) provdes better tme-dstorton performance than ether DD or RD varants despte ts hgher tme complexty of sngle teraton. Ths s also the reason why the D 2 R varant s worth consderng and s proposed here. For the k-means component, we use the fast varant [12] that utlzes the nformaton of the prototype actvty from prevous teraton. Snce most of the prototypes stablze fast, t acheves a remarkable speed-up (both n theory and practce) n comparson to full search wth rather smple algorthm. Potentally further speed-up could be potentally obtaned by onng t wth kd-tree as n [13], or wth trangular nequalty rule as n [14]. Table 3. Summary of the tme complextes of the man varants. Random removal Determnstc removal Determnstc removal wth updatng data RR RD DR DD D 2 R D 2 D Removal O(1) O(1) O(MN) O(MN) O(αN) O(αN) Addton O(1) O(N) O(1) O(N) O(1) O(N) Local repartton+ O(αN) O(αN) O(αN) O(αN) O(αN) O(αN) K-means Algorthm n total O(αN) O(αN) O(MN) O(MN) O(αN) O(αN) 3 Experments In the followng, we cluster four mage data sets [2] by Hewlett-Packard Pavlon wth 2.20 GHz ADM Athlon XP processor and 512 MB memory. The vectors n Brdge are 4 4 non-overlappng blocks, n Mss Amerca 4 4 dfference blocks of two subsequent frames n a vdeo sequence. The set House conssts of RGB color values, and Europe conssts of dfferental coordnates. Tme-dstorton performance of the selected methods s compared n Fgure 4 and Table 4. The followng varants are compared: repeated k-means (RKM), random

9 swap (RS), determnstc swap wth random addton (DR) and ts tme-effcent mplementaton (D 2 R). K-means results are obtaned by the fast exact varant as proposed n [12]. The proposed algorthm (D 2 R) outperforms ts straghtforward counterpart (DR) [8] and provdes the same clusterng qualty than the random swap (RS) [9] wth a smlar or faster speed n case of all data sets except the 2-dmensonal set Europe. Ths s also the only set for whch the swap-based clusterng does not provde any beneft over the repeated k-means at all. Wth the hgher dmensonal data sets, the method works better. MSE Random Swap D 2 R Brdge Repeated k-means DR Tme Mssa1 6.1 DR MSE Repeated k-means Ramdom Swap 5.5 D 2 R Data Set Tme Fg. 4. Tme-dstorton effcency of the dfferent swap varants. Table 4. Summary of the clusterng qualty wthn a gven tme constrant. Clusterng qualty (10s) Clusterng qualty (100s) RKM RS DR D 2 R RKM RS DR D 2 R Brdge House Mss Amerca Europe

10 4 Conclusons Random swap method s very smple to mplement and fnds the correct clusterng when terated long enough. However, ts quadratc dependency on the number of clusters motvated us to develop a faster method for selectng the prototype to be swapped determnstcally. We have compared several random and determnstc varants. The methods also generalze for the case of unknown number of clusters by replacng the cost functon by a sutable cluster valdty ndex, and by havng three operatons: cluster removal, cluster addton and random swap. References 1. B. Frtzke: The LBG-U method for vector quantzaton an mprovement over LBG nspred from neural networks. Neural Processng Letters 5(1), (1997) 2. P. Fränt and O. Vrmaok: Iteratve shrnkng method for clusterng problems. Pattern Recognton 39 (5), (2006) 3. A. Lkas, N. Vlasss and J.J. Verbeek: The global k-means clusterng algorthm. Pattern Recognton 36, , (2003) 4. T. Kaukoranta, P. Fränt and O. Nevalanen: Iteratve splt-and-merge algorthm for VQ codebook generaton. Optcal Engneerng 37 (10), (1998) 5. H. Frgu and R. Krshnapuram: Clusterng by compettve agglomeraton. Pattern Recognton 30 (7), (1997) 6. P. Fränt, T. Kaukoranta and O. Nevalanen: On the splttng method for vector quantzaton codebook generaton. Optcal Engneerng 36 (11), (1997) 7. M.G.C. Resende, R.F. Werneck: A fast swap-based local search procedure for locaton problems. Ann Oper Res 150 (1), (2007) 8. P. Fränt, M. Tuononen and O. Vrmaok: Determnstc and randomzed local search algorthms for clusterng. In: IEEE Int. Conf. on Multmeda and Expo, (ICME 08), pp Hannover, Germany (2008) 9. P. Fränt and J. Kvärv: Randomsed local search algorthm for the clusterng problem. Pattern Analyss and Applcatons 3 (4), (2000) 10.T. Kanungo, D.M. Mount, N. Netanyahu, C. Patko, R. Slverman, and A.Y. Wu: A local search approxmaton algorthm for k-means clusterng. Computatonal Geometry 28 (1), (2004) 11. P. Fränt, O. Vrmaok and V. Hautamäk: Probablstc clusterng by random swap algorthm. In: IAPR Int. Conf. on Pattern Recognton (ICPR 08). Tampa, FL, USA, (2008) 12. T. Kaukoranta, P. Fränt and O. Nevalanen: A fast exact GLA based on code vector actvty detecton. IEEE Trans. on Image Processng 9 (8), (2000) 13. J.Z.C. La, Y.-C. Law: Improvement of the k-means clusterng flterng algorthm. Pattern Recognton 41 (12), (2008) 14. C. Elkan: Usng the trangle nequalty to accelerate k-means. In: Int. Conf. on Machne Leearnng (ICML 03), (2003)