A Two-Stage Algorithm for Data Clustering

Transcription

1 A Two-Stage Algorthm for Data Clusterng Abdolreza Hatamlou 1 and Salwan Abdullah 2 1 Islamc Azad Unversty, Khoy Branch, Iran 2 Data Mnng and Optmsaton Research Group, Center for Artfcal Intellgence Technology, Unverst Kebangsaan Malaysa, 436 Bang, Selangor, Malaysa Abstract - Cluster analyss s an mportant and popular data analyss technque that s used n a large varety of felds. K-means s a well-known and wdely used clusterng technque due to ts smplcty n mplementaton and hgh speed n most stuatons. However, t suffers from two major shortcomngs: t s very senstve to the ntal state of centrods and may converge to local optmum soluton. In order to overcome the shortcomngs of K-means, we present a two-stage approach, called KM-HS, whch s based on K-means and a heurstc search algorthm. At the frst stage of the proposed approach, K-means algorthm s appled to fnd an ntal soluton to the clusterng problem and then at the second stage, a heurstc search algorthm s used to mprove the qualty of the ntal soluton by searchng around t. The performance of the proposed algorthm s evaluated usng four benchmark datasets from UCI repostory. The expermental results ndcate that the KM-HS algorthm not only fnds hgh qualty clusters but also converges more quckly than other evolutonary algorthms. Keywords: Cluster Analyss; K-means; Heurstc Search Algorthm 1 Introducton Clusterng analyss parttons a set of objects nto subsets, as clusters, so that objects n the same cluster are smlar or related to each other, whle objects n dfferent clusters are dssmlar or unrelated [1, 2]. Clusterng analyss has been used n a large varety of felds and applcatons rangng from pattern recognton, web mnng, mage segmentaton, genetcs, mcrobology, geography, remote sensng, psychology, educaton, marketng and busness [3-8]. Clusterng algorthms are broadly based on two man approaches: herarchcal and parttonal [2]. Herarchcal algorthms can be dvded as agglomeratve (bottom-up) or dvsve (top-down) methods. Agglomeratve methods consder each object as a dstnct cluster and combne them sequentally n larger clusters. Dvsve methods start wth the all objects as a sngleton cluster and contnue to dvde t nto sequentally smaller clusters. Parttonal methods, on the other hand, attempt to dvde objects drectly nto a set of dsjont clusters, wthout makng a tree structure. They try to optmze an objectve functon. Typcally, the objectve functon nvolves n mnmzng the dssmlarty n the objects wthn each cluster, whle maxmzng the dssmlarty between objects n dfferent clusters. Among tradtonal clusterng algorthms, K-means s the most popular and wdely used method due to ts smplcty and computatonal effcency, wth lnear tme complexty [9]. However, t s very senstve to the ntal choce of centrods and may converge to local mnma [1]. Recently, to overcome the drawbacks of K-means algorthm, many clusterng technques based on evolutonary algorthms such as genetc algorthm (GA), tabu search (TS), smulated annealng (SA), honey bee matng optmzaton (HBMO), ant colony optmzaton (ACO) and partcle swarm optmzaton (PSO) have been proposed [11-16]. However, most of evolutonary methods are typcally very slow to fnd optmal soluton. In order to overcome the above mentoned shortcomngs, we present a novel clusterng algorthm whch combnes K-means and a heurstc search algorthm, called KM-HS. In partcular, we use K- means algorthm to fnd an ntal soluton for the clusterng problem. Afterward, we use a heurstc search algorthm to thoroughly explore around of the

2 ntal soluton so that KM-HS mght fnd global optmum. The rest of ths paper s organzed as follows. Secton 2 explans the basc prncples of cluster analyss and K- means algorthm, whle the proposed KM-HS algorthm s explaned n secton 3. Secton 4 presents expermental results usng four standard benchmark datasets. Fnally, concluson of ths work s drawn n secton 5. 2 Cluster analyss Clusterng s the process of parttonng a set of objects nto a fnte number of k clusters so that the objects wthn each cluster are smlar, whle objects n dfferent clusters are dssmlar. In most of clusterng algorthms, the crteron that s used to measure the qualty of resultng clusters s defned as n Eq.(1), whch s known as mnmzng sum of squared error [17]. f (O,C) = k l 1O Cl 2 d ( O, Z ) (1) where d( O, Z l ) specfes the dstance between object O and the cluster centrod Z l. Usually, smlarty and dssmlarty between objects are expressed through some dstance functons. The most common dstance functon s the Eucldean dstance that s defned as follows: d( O, O ) j d p 1 ( o p l o where d(o, O j ) specfes the dstance between two objects O and O j. P s the dmensonalty of the dataset. As mentoned earler, K-means s the most popular and wdely used clusterng algorthm. K-means starts wth k ntal centrods (these centrods are created randomly or derved from some heurstc approaches). Each object n the dataset s then assgned to the closest centrod. Centrods are updated by usng the mean of the objects wthn each cluster. Ths process s repeated untl a termnaton crteron s met. 3 Proposed approach The proposed approach s bult based on two man stages. At the frst stage, the K-means algorthm s appled to fnd an ntal soluton to the clusterng problem. K-means algorthm can do ths effcently, whle t s very fast. However the output of K-means possbly s far from optmal soluton and t can be mproved by some other technques. To ensure to get a p j ) 2 (2) good soluton by the K-means algorthm n the frst stage of the proposed algorthm, K-means s conducted 3 tmes and the best soluton among them wll be passed to the next stage for further mprovement. At the second stage, we have appled a heurstc search method to search around the ntal soluton found by the k-means algorthm at the frst stage. The proposed approach wll try to mprove the qualty of the ntal soluton (output of the frst stage) by searchng around t n all dmensons. The structure of the proposed heurstc search s as follows: At the frst, an ntal value wll be consdered as the ntal step of movement for the algorthm. Ths value wll be added to all features n the ntal soluton one by one. In other words, the threshold wll be added to the frst feature n the frst centrod and then the ftness value of the new produced centrod wll be calculated and compared wth the ftness value of the current centrod. If there s an mprovement n terms of ftness value, then the current centrod wll be replaced by the new centrod. Otherwse, the current centrod wll be reloaded and the search drecton changes to the other sde for the respectve feature. Whch means that, at the next teraton, the threshold value wll be subtracted from the current value of the respectve feature, and the above procedure wll be done agan. If there s no mprovement n both sdes of the consdered feature n the consdered centrod usng the current threshold, the threshold value of the respectve feature wll be dvded by two for the next teraton. Ths causes the heurstc search to act n a bnary way and the tme complexty of ths stage to be logarthmc. The above mentoned procedure wll be repeated for all features of the consdered centrod, and then for the other centrods sequentally untl the termnaton crtera are satsfed. Based on the above descrpton, the pseudo code of the KM-HS s stated as follows: Stage 1: K-means method 1.1. Select k ponts as the ntal centrods n a random way (Re)Assgn all objects to the closest centrod Recalculate the centrod of each cluster Repeat steps 1.2 and 1.3 untl a termnaton crteron s met Pass the soluton to the next stage. Stage 2: Heurstc search For all centrods =1...k do For all features j=1...d do If SD (j)==1 C(j)= C(j)+SS(j); Calculate ftness value for the new centrod. If the ftness value has been mproved Make the new centrod permanent

3 Else Reload the old centrod SD (j)=-1 Else f SD (j)==-1 C(j)= C(j)-SS(j) Calculate ftness value for the new centrod. If the ftness value has been mproved Make the new centrod permanent Else Reload the old centrod SD (j)= Else f SD (j)== SS(j)= SS(j)/2; SD (j)=1; End for End for In the above pseudo code, SD= [SD 1, SD 2,..., SD k ] s the search drecton, and SS= [SS 1, SS 2,..., SS k ] s the search step, where SD s the search drecton of the -th centrod and the length of ths array s d, whch s the dmensonalty of the test dataset. All felds n ths array are ntalzed to 1 at the begnnng, and change to -1, and 1 durng the search process. SS s the search step for the -th centrod, and t wll be set to Max(dataset) at the begnnng. Max(dataset) s an 1-dmenson array wth length of d, where each member of t contans the maxmum value of the correspondng feld n the test dataset. C=[C 1, C 2,, C k ] contans the centrods of k clusters and C (j) specfes the j-th feature n the -th cluster. To explan how the proposed heurstc search algorthm works, magne that the snapshot of the system s lke the Fg 1(a). For smplcty, we have consdered only one centrod n ths example wth four features. As seen from the Fg 1(a) at the end of the current teraton a new centrod s produced usng the current centrod, current search drecton and current search step. Improvement has happened on the frst and second features, whereas the thrd and forth features have not been changed. So, at the next teraton, search process wll contnue n the current drecton for the features 1 and 2. For the 3 rd feature, the search drecton s, meanng that, no mprovement happened for ths feature n both drectons n the prevous teratons. So, for the next teraton, the search step wll be dvded by 2 and search drecton wll be set to 1. For the last feature, the search drecton wll be set to -1.e. to search for a better soluton n the opposte drecton as there s no room for mprovement n the current drecton so, the system stuaton for the next teraton wll be lke as Fg 1(b). Current Centrod SD SS New Centrod Fnal Centrod (a) Current Centrod SD SS New Centrod Fnal Centrod???? (b) Fgure 1. An example for explanng the proposed heurstc search method 4 Expermental results Four benchmark datasets are used to assess the performance of the proposed approach, KM-HS, n comparson wth K-means [9], tabu search (TS) [11], partcle swarm optmzaton (PSO) [12], genetc algorthm (GA) [13], honey bee matng optmzaton (HBMO) [14], smulated annealng (SA) [15] and ant colony optmzaton (ACO) [16]. The used datasets are Irs, Wne, Contraceptve Method Choce (CMC) and Wsconsn Breast Cancer that are provded by UCI repostory of machne learnng databases [18]. Datasets have the followng characterstcs: Irs dataset (n=15, d=4, k=3): Ths dataset was collected by Anderson (1935). It contans three classes of 5 objects each, where each class refers to a type of rs flower. There are 15 random samples of rs flowers wth four numerc attrbutes n ths dataset. These attrbutes are sepal length and wdth n cm, petal length and wdth n cm. There s no mssng value for attrbutes. Wne dataset (n=178, d=13, k=3): Ths dataset contans the results of a chemcal analyss of wnes grown n the same regon n Italy but derved from three dfferent cultvars. Ths dataset contans 178 nstances wth 13 contnuous numerc attrbutes. There s no mssng attrbute value. Contraceptve Method Choce also denoted as CMC (n = 1473, d = 9, k = 3): Ths dataset s a subset of the 1987 Natonal Indonesa Contraceptve Prevalence Survey. The samples are marred women who ether

4 Table 1 The results for dfferent clusterng algorthms Dataset Crtera K-means GA SA TS ACO HBMO PSO KM-HS Irs Wne CMC Cancer , were not pregnant or dd not know f they were at the tme of ntervew. The problem s to predct the choce of current contraceptve method (no use has 629 objects, long-term methods have 334 objects, and shortterm methods have 51 objects) of a woman based on her demographc and socoeconomc characterstcs. Wsconsn breast cancer (n = 683, d = 9, k = 2): Ths dataset contans 683 objects, whch characterzed by nne features: clump thckness, cell sze unformty, cell shape unformty, margnal adheson, sngle epthelal cell sze, bare nucle, bland chromatn, normal nucleol, and mtoses. There are two clusters n the data: malgnant (444 objects) and bengn (239 objects). For evaluatng the effcency of clusterng algorthms, we have used two famous crtera: The frst one s the mean square error (MSE), or wthn cluster varance (Eq. 1). Obvously, a small value for the MSE ndcates hgh qualty results and vce versa. The second crteron s the number of evaluatons of the objectve functon (MSE n ths study) to ndcate that how fast the respectve algorthm can fnd the soluton of the gven dataset. Ths s shown usng abbrevaton. Clearly, the smaller value of the shows the hgh convergence to optmal soluton. In ths study, each experment s done twenty tmes and the average and the standard devaton of solutons are calculated and reported as well as the best and worst solutons. Table 1 lsts the results of the experments. The smulaton results gven n Table 1 confrm that the proposed approach, KM-HS, s robust and faster n comparson wth other algorthms. In terms of MSE, whch shows the qualty of resultng clusters, KM-HS provdes the optmum value and small standard devaton n all the test datasets. The KM-HS converges to the global optmum of, , and on the rs, wne, CMC and cancer datasets, respectvely n all of the runs, whch these are better than other approaches. The standard devaton of the solutons found by the KM-HS s for all of the test datasets, meanng that, t mght converge to optmal value n all of the runs, whereas other approaches converge to local optma n some of the runs. In terms of the number of functon evaluatons (), K-means algorthm s better than other methods. However, the qualty of the output of the K-means algorthm s not satsfactory. In compare to other methods, the KM-HS needs the least number of functon evaluatons. In bref, the results confrm that KM-HS has three sgnfcant merts n comparson to other methods. Frstly, t s a robust approach and able to fnd hgh qualty clusters n all the test datasets. Secondly, t s a vable approach that can converge to global optmum n all runs. Fnally, t s a fast algorthm and converges to optmal soluton more quckly than other methods. 5 Concluson A hybrd clusterng approach has been developed n ths work, whch s based on K-means and a heurstc search algorthm. In the proposed algorthm, the K-means s used to produce an ntal soluton to

5 the clusterng problem and after that a heurstc search algorthm has been appled to mprove the qualty of ths soluton by searchng around t. The performance of the proposed algorthm s evaluated usng a number of standard benchmark datasets. The smulaton results confrm that the proposed KM-HS algorthm able to obtan hgh qualty clusters. Moreover, the convergence speed of the proposed algorthm s more quckly than other methods n comparson. 6 References [1] J. Han, M.K.: Data Mnng: Concepts and Technques ( Academc Press, 21.) [2] Ru, X., and Wunsch, D., II: Survey of clusterng algorthms, Neural Networks, IEEE Transactons on, 16, (3), pp , 25. [3] Barn, M., and Gualter, R.: A new possblstc clusterng algorthm for lne detecton n real world magery, Pattern Recognton, 32, (11), pp , [4] Ca, W., Chen, S., and Zhang, D.: Fast and robust fuzzy c-means clusterng algorthms ncorporatng local nformaton for mage segmentaton, Pattern Recognton, 4, (3), pp , 27. [5] Cnque, L., Forest, G., and Lombard, L.: A clusterng fuzzy approach for mage segmentaton, Pattern Recognton, 37, (9), pp , 24. [6] Fan, J., Han, M., and Wang, J.: Sngle pont teratve weghted fuzzy C-means clusterng algorthm for remote sensng mage segmentaton, Pattern Recognton, 42, (11), pp , 29. [11] Al-Sultan, K.S.: A Tabu search approach to the clusterng problem, Pattern Recognton, 28, (9), pp , [12] Chng-Y, C., and Fun, Y.: Partcle swarm optmzaton algorthm and ts applcaton to clusterng analyss, n Edtor (Ed.)^(Eds.): Book Partcle swarm optmzaton algorthm and ts applcaton to clusterng analyss, pp , Vol.782, 24. [13] Cowgll, M.C., Harvey, R.J., and Watson, L.T.: A genetc algorthm approach to cluster analyss, Computers & Mathematcs wth Applcatons, 37, (7), pp , [14] Fathan, M., Amr, B., and Maroos, A.: Applcaton of honey-bee matng optmzaton algorthm on clusterng, Appled Mathematcs and Computaton, 19, (2), pp , 27. [15] Selm, S.Z., and Alsultan, K.: A smulated annealng algorthm for the clusterng problem, Pattern Recognton, 24, (1), pp , [16] Shelokar, P.S., Jayaraman, V.K., and Kulkarn, B.D.: An ant colony approach for clusterng, Analytca Chmca Acta, 59, (2), pp , 24. [17] Jan, A.K.: Data clusterng: 5 years beyond K- means, Pattern Recognton Letters, 31, (8), pp , 21. [18] C.L. Blake, C.J.M.: UCI repostory of machne learnng databases. Avalable from: < [7] Scheunders, P.: A genetc c-means clusterng algorthm appled to color mage quantzaton, Pattern Recognton, 3, (6), pp , [8] Tjh, W.-C., and Chen, L.: Possblstc fuzzy coclusterng of large document collectons, Pattern Recognton, 4, (12), pp , 27. [9] Forgy, E.W.: Cluster analyss of multvarate data: effcency versus nterpretablty of classfcatons, Bometrcs, 21, pp. 2, [1] Selm, S.Z., and Ismal, M.A.: K-Means-Type Algorthms: A Generalzed Convergence Theorem and Characterzaton of Local Optmalty, Pattern Analyss and Machne Intellgence, IEEE Transactons on, PAMI-6, (1), pp , 1984.