Automatic Clustering Using a Synergy of Genetic Algorithm and Multi-objective Differential Evolution

Transcription

1 Automatc Clusterng Usng a Synergy of Genetc Algorthm and Mult-obectve Dfferental Evoluton Debarat Kundu, Kaushk Suresh, Sayan Ghosh, Swagatam Das, Ath Abraham 2 and Youakm Badr 2 Department of Electroncs and Telecommuncaton Engneerng Jadavpur Unversty, Kolkata, Inda 2 Natonal Insttute of Appled Scences of Lyon, INSA-Lyon, Vlleurbanne, France ath.abraham@eee.org, youakm.badr@nsa-lyon.fr Abstract Ths paper apples the Dfferental Evoluton (DE) and Genetc Algorthm (GA) to the task of automatc fuzzy clusterng n a Mult-obectve Optmzaton (MO) framework. It compares the performance a hybrd of the GA and DE (GADE) algorthms over the fuzzy clusterng problem, where two conflctng fuzzy valdty ndces are smultaneously optmzed. The resultant Pareto optmal set of solutons from each algorthm conssts of a number of nondomnated solutons, from whch the user can choose the most promsng ones accordng to the problem specfcatons. A real-coded representaton of the search varables, accommodatng varable number of cluster centers, s used for GADE. The performance of GADE has also been contrasted to that of two most well-known schemes of MO.. Introducton Optmzaton-based automatc clusterng algorthms greatly rely on a cluster valdty functon (optmzaton crteron) the optma of whch appear as proxes for the unknown correct classfcaton n a prevously unhandled dataset []. Dfferent formulatons of the clusterng problem vary n the optmzaton crteron used. Most exstng clusterng methods, however, attempt to optmze ust one such clusterng crteron modeled by a sngle cluster valdty ndex. Ths often results nto consderable dscrepances observable between the solutons produced by dfferent algorthms on the same data. The sngle-obectve clusterng method may prove futle (as udged by means of expert s knowledge) n a context where the crteron employed s napproprate. In stuatons where the best soluton corresponds to a tradeoff between dfferent conflctng obectves, common sense advocates a multobectve framework for clusterng. Although there has been a plethora of papers reportng several sngle-obectve evolutonary clusterng technues (a comprehensve survey of whch can be found n [, 2]), very few research works have so far been undertaken towards the applcaton of evolutonary mult-obectve optmzaton algorthms (EMOA) for pattern clusterng [3, 4]. A state-of-the-art lterature survey ndcates that DE has already proved tself as a promsng canddate n the feld of evolutonary mult-obectve optmzaton (EMO) [5 8]. Earler t has also been successfully appled to sngleobectve parttonal clusterng [9 ]. The work reported n [3] s based on Deb et al. s celebrated NSGA (Non Domnated Sortng genetc Algorthm)-II [2] and the clusterng method descrbed n [4] s based on PESA (Pareto Evoluton based Selecton) II [3], and both the algorthms are mult-obectve varants of Genetc Algorthm (GA). However, the mult-obectve varants of DE have not been appled to the general data clusterng problems tll date, to the best of our knowledge. Snce DE, by nature, s a real-coded populaton-based optmzaton algorthm, we here

2 resort to centrod-based representaton scheme for the search varables. A MOO algorthm, n general, ends up wth a number of Pareto optmal solutons. Here we consder the Xe-Ben ndex [4] and the Fuzzy C Means (FCM) measure ( J m ) [5] as the obectve functons. The performance of GADE has also been contrasted wth two best-known EMOA-based clusterng methods tll date. The frst of these s MOCK by Handl and Knowles [4] whle the second one s based on NSGA II and was used by Bandyopadhyay et al. for pxel clusterng n remote sensng satellte mage data [3]. Here we report the results for ten representatve datasets ncludng the mcroarray Yeast sporulaton data [6]. 2. Mult-obectve Optmzaton Usng DE 2. The MO Problem In many practcal oeal lfe problems, there are many (possbly conflctng) obectves that need to be optmzed smultaneously. Under such crcumstances there no longer exsts a sngle optmal soluton but rather a whole set of possble solutons of euvalent ualty. The feld of Mult-obectve Optmzaton (MO) [7 9] deals wth smultaneous optmzaton of multple, possbly competng, obectve functons. 2.2 The Dfferental Evoluton (DE) Algorthm DE [20, 2] s a populaton-based global optmzaton algorthm that uses a floatng-pont (real-coded) representaton. It uses crossover (bnomal n ths case) and mutaton operatons to optmze a gven cost functon. For want of space, we avod mentonng the detals of the DE algorthm here and refer the reader to the aforementoned lteratures. 2.3 The Mult-obectve Varant of DE We have used the Mult-obectve DE (MODE) [4]. MODE was proposed by Xue et al. [8]. Ths algorthm uses a varant of the orgnal DE, n whch the best ndvdual s adopted to create the offsprng. A Pareto-based approach s ntroduced to mplement the selecton of the best ndvdual. If a soluton s domnated, a set of nondomnated ndvduals can be dentfed and the best turns out to be any ndvdual (randomly pcked) from ths set. 3. Mult-obectve Clusterng Scheme 3. Search-varable Representaton and Descrpton of the new algorthm In the proposed method, for n data ponts, each d-dmensonal, and for a userspecfed maxmum number of clusters K max, a chromosome s a vector of real numbers of dmenson K max + K max d. The frst K max entres are postve floatngpont numbers n [0, ], each of whch controls whether the correspondng cluster s to be actvated (.e. to be really used for classfyng the data) or not. The remanng entres are reserved for K max cluster centers, each d-dmensonal. For example, the - th vector s represented as:

3 r X (t) = T T,... T, K max, m r m r, 2... m r,k max The -th cluster center n the -th chromosome s actve or selected for parttonng the assocated dataset f T. On the other hand, ft 0, the partcular -th, =, = cluster s nactve n the -th vector n DE populaton. Thus the T, s behave lke control genes. r IF T, = THEN the -th cluster center m, s ACTIVE r ELSE s INACTIVE. () m, Conuncton of GA and DE algorthms: The Dfferental Evoluton algorthm s appled on the frst K max members of the chromosome (as actvated by the correspondng control genes), whereas, the control genes form a bnary encoded GA populaton, whch are operated by the Genetc operators of Selecton, Crossover and Mutaton. Bnary tournament selecton s employed n ths case. The dfferent GA operators are not reterated here due to space lmtatons. Smple generatonal genetc algorthm pseudo code:. Choose ntal populaton 2. Evaluate the ftness of each ndvdual n the populaton 3. Repeat untl termnaton: (tme lmt or suffcent ftness acheved). Select best-rankng ndvduals to reproduce 2. Breed new generaton through crossover and/or mutaton (genetc operatons) and gve brth to offsprng 3. Evaluate the ndvdual ftnesses of the offsprng Replace worst ranked part of populaton wth offsprng. 3.2 Selectng the Obectve Functons Conflct among the obectve functons s often benefcal snce t gudes to globally optmal solutons. In ths work we choose the Xe-Ben ndex XB and the FCM obectve functon J as the two obectves. The FCM measure J may be defned as: n k r 2 r J u d ( Z, m ), (2) = = = where s the fuzzy exponent, d ndcates a dstance measure between the -th pattern vector and -th cluster centrod, and u denotes the membershp of -th pattern n the -th cluster. The XB ndex s defned as a functon of the rato of the total varaton σ to the mnmum separaton sep of the clusters. Here σ and sep may be wrtten as:

4 k n 2 σ u d( m, Z ) (3) = = p= 2 and sep( Z) mn{ d ( m, m )} p p = (4) The XB ndex s then wrtten as: XB u σ = p= = = n sep Z) n mn k ( 2 n p d 2 ( m, Z ) { d ( Z, Z )} r Let the set of centers be denoted by { m, m2,..., m k }. The membershp value of the -th pattern n -th cluster u, =,2,... k and =,2,..., n are computed as: u = 2 (6) k d( m, Z ) p= d( m p, Z ) Note that whle computng the u s, usng euaton (2), f d m p, Z ) s eual to p ( zero for some p, then u s set to zero for all =,2,... k,, whle u p to one. Subseuently the centers encoded n a vector are updated usng: n r ( u p ) Z r = m = p n ( u p ) = 3.3 Avodng Erroneous Vectors (5) s set eual There s a possblty that n our scheme, durng computaton of the XB or J, a dvson by zero may be encountered. Ths may occur when one of the selected cluster centers n a DE-vector s outsde the boundary of dstrbutons of the data set. To avod ths problem we frst check to see f any cluster has fewer than two data ponts n t. If so, the cluster center postons of ths specal chromosome are re-ntalzed by an average computaton. 3.4 Selectng the Best Soluton from Pareto-front For choosng the most nterestng solutons from the Pareto front, we apply Tbshran et al. Gap statstc [24], a statstcal method to determne the number of clusters n a data set. 3.5 Evaluatng the Clusterng Qualty In ths work, the fnal clusterng ualty s evaluated usng two external measures. Specfcally we choose the Adusted Rand Index [25] (whch s a generalzaton of the Rand ndex [26]) and the Shouette ndex [27]. Slhouette wdth reflects the (7)

5 compactness and separaton of the clusters. Gven a set of data ponts Z Z,..., Z } C = C C,...,, the slhouette = and a gven clusterng soluton { } { n wdth s( Z r ) for each data Z r belongng to cluster confdence of belongngness, and t s defned as: r s( Z, 2 C k C ndcates a measure of the b( Z ) a( Z ) ) =. (8) max( a( Z ), b( Z )) Here a Z r ) denotes the average dstance of data pont ( ponts of the cluster to whch the data pont Z r from the other data Z r s assgned (. e. cluster C ). On the other hand, b Z r ) represents the mnmum of the average dstances of data pont ( Z r from the data ponts belongng to clusters C r, r =,2,..., k and r. The value of s Z r ) les between - and +. Large values of s Z r ) (near to ) ndcate that the ( data pont Z r s well clustered. Overall slhouette ndex s(c) of a clusterng soluton { C C } C =, 2,..., C k s defned as the mean slhouette wdth over all the data ponts: n r s( C) = s( Z ) (9) n = 4 Expermental results 4. Datasets used The expermental results showng the effectveness of mult-obectve DE based clusterng has been provded for sx artfcal and foueal lfe datasets. Table presents the detals of the datasets. The real-lfe datasets are rs, wne, breast-cancer [28] and the yeast sporulaton data. The sporulaton dataset s avalable from [3]. 4.2 Parameters for the Algorthms GADE has been used wth 40 parameter vectors n each generaton and each run of each algorthm was contnued for 00 generatons. The value of scale factor F s a random value between 0.5 and. The other parameters for the mult-obectve GA (NSGA II) based clusterng are fxed as follows: number of generatons = 00, populaton sze = 50, crossover probablty = 0.8, mutaton probablty =. Please note that GADE and the NSGA II use the same Chromosome _ length parameteepresentaton scheme. Clusterng wth MOCK was performed wth the source codes avalable from [32]. 4.3 Presentaton of Results The mean Slhouette ndex values of the best-of-run solutons provded by sx contestant algorthms over the 0 datasets have been provded n Table 2. The best entres have been marked n boldface n each row. Table 3 enlsts the adusted rand ndex values except for Yeast sporulaton data as no standard nomnal classfcaton s known for ths dataset. (

6 4.4.4 Sgnfcance and Valdaton of Mcroarray Data Clusterng Results In ths secton the best clusterng soluton provded by dfferent algorthms on the sporulaton data of yeast has been vsualzed usng the cluster profle plot (n parallel coordnates[30]) n MATLAB verson. It s a common way of vsualzng hghdmensonal geometry. Cluster profle plots (n parallel coordnates) of seven clusters for the best clusterng result (provded by GADE) on yeast sporulaton data has been shown n Fgure. The blue polylnes ndcate the member genes wthn a cluster whle the black polylne ndcates the centrod of that gene. The heatmap and fatgo results may be obtaned from [33]. Table. Detals of the datasets used. Dataset Number of Number of clusters Number of ponts Characterstcs Dataset_ Dataset _ Dataset _ Dataset _ Dataset _ Dataset_ Irs Wne Breast-Cancer Yeast Sporulaton Table 2. Mean value of sl ndex found and standard devatons (n parentheses) by contestant algorthm over 30 ndependent runs on ten datasets. Dataset Dataset_ 9.2 (.46) Dataset_ (0.65) Dataset_3 4.4 (0.36) Dataset_ (0.25) Dataset_ (3.89) Dataset_6 5.9 (0.93) Irs 2.3 (0.76) Wne 3.6 (0.46) Breast 2.08 Cancer (0.38) Yeast 7.08 Sporulaton (0.2) k Algorthms Compared GADE NSGA II MOCK Slhouette k Slhouette k Slhouette Index Index Index ( ) (.72) (0.0892) (2.8) (0.0736) (0.2360) (0.072) ( ) (.03) ( ) ( ) (0.5) ( ) (.25) ( ) (0.566) (0.46) (0.0287) (0.5) ( ) ( ) (0.939) ( ) (0.80) ( ) ( ) (.58) ( ) (0.38) ( ) ( ) (.06) ( ) (0.37) ( ) ( ) (0.67) ( ) (0.46) ( ) ( ) (0.60) ( ) (0.53) (0.0094) ( ) (0.68) (0.0483) (0.857) ( )

7 Table 3. Mean value of adusted rand ndex found and standard devatons (n parentheses) by contestant algorthm over 30 ndependent runs on ten datasets. Dataset Algorthms Compared GADE NSGA2 MOCK Dataset_ (0.0020) ( ) ( ) Dataset_ ( ) ( ) ( ) Dataset_ ( ) ( ) ( ) Dataset_ ( ) ( ) ( ) Dataset_ (0.0942) ( ) ( ) Dataset_ ( ) ( ) ( ) Irs ( ) ( ) ( ) Wne ( ) ( ) ( ) Breast Cancer (0.0053) ( ) ( ) (a) Cluster (b) Cluster 2 (c) Cluster 3 (d) Cluster 4 (e) Cluster 5 (f) Cluster 6 (g) Cluster 7 5. Conclusons Ths paper compared and contrasted the performance of GADE n an automatc clusterng framework wth two other promnent mult-obectve clusterng algorthms. The mult-obectve GADE-varant used the same varable representaton scheme. Tables 2 to 4 ndcate that GADE was usually able to produce better fnal clusterng results as compared to MOCK or NSGA II n terms of both adusted Rand ndex and Slhouette ndex when all the algorthms were let run for an eual number of generatons. Future research may extend the mult-obectve GADE-based clusterng schemes to handle dscrete chromosome representaton schemes that no longer depend on cluster centrods and thus are not based n any sense towards sphercal clusters. References [] A. K. Jan, M. N. Murty, and P. J. Flynn, Data clusterng: a revew, ACM Computng Surveys, vol. 3, no.3, (999) [2] R. Xu and D. Wunsch, Clusterng, Seres on Computatonal Intellgence, IEEE Press, [3] S. Bandyopadhyay, U. Maulk, and A. Mukhopadhyay, Multobectve genetc clusterng for pxel classfcaton n remote sensng magery, IEEE Transactons Geoscence and Remote Sensng, [4] J. Handl and J. Knowles, An evolutonary approach to multobectve clusterng, IEEE Transactons on Evolutonary Computaton, ():56-76, 2007.

8 [5] H. A. Abbass and R. Sarker, The pareto dfferental evoluton algorthm, Internatonal Journal on Artfcal Intellgence Tools, (4):53-552, [6] F. Xue, A. C. Sanderson, and R. J. Graves, Pareto-based mult-obectve dfferental evoluton, n Proceedngs of the 2003 Congress on Evolutonary Computaton (CEC 2003), volume 2, pages , Canberra, Australa, 2003, IEEE Press. [7] T. Robc and B. Flpc, DEMO: Dfferental Evoluton for Multobectve Optmzaton, In C. A. Coello Coello, A. H. Agurre, and E. Ztzler, edtors, Evolutonary Mult-Crteron Optmzaton, Thrd Internatonal Conference, EMO 2005, pages , Guanauato, Mexco, 2005, Sprnger LNCS Vol. 340, [8] A. W. Ioro and X. L, Solvng rotated mult-obectve optmzaton problems usng dfferental evoluton, n AI 2004: Advances n Artfcal Intellgence, Proceedngs, pages , Sprnger- Verlag, LNAI Vol. 3339, [9] S. Paterlna, T. Krnk, Dfferental evoluton and partcle swarm optmsaton n parttonal clusterng, Computatonal Statstcs & Data Analyss, Volume 50, Issue 5, , [0] M. Omran, A. P. Engelbrecht and A. Salman, Dfferental evoluton methods for unsupervsed mage classfcaton, Proceedngs of Seventh Congress on Evolutonary Computaton (CEC-2005). IEEE Press, [] S. Das, A. Abraham, and A. Konar, Automatc clusterng usng an mproved dfferental evoluton algorthm, IEEE Transactons on Systems Man and Cybernetcs - Part A, Vol. 38, No., pp. -20, January [2] K. Deb, A. Pratap, S. Agarwal, and T. Meyarvan, A fast and eltst multobectve genetc algorthm: NSGA-II, IEEE Transactons on Evolutonary Computaton, vol. 6, no. 2, [3] D.W. Corne, J.D. Knowles, and M.J. Oates, The pareto-envelope based selecton algorthm for multobectve optmsaton, n M. Schoenauer, K. Deb, G. Rudolph, X. Yao, E. Lutton, J. J. Merelo, and H-P. Schwefel, (eds.) Parallel Problem Solvng from Nature - PPSN VI, Sprnger Lecture Notes n Computer Scence, pp , [4] X. Xe and G. Ben, Valdty measure for fuzzy clusterng, IEEE Trans. Pattern Anal. Machne Learnng, Vol. 3, pp , (99). [5] J. C. Bezdek, Cluster valdty wth fuzzy sets, Journal of Cybernetcs, (3) 58 72, (974). [6] S. Chu et al. The transcrptonal program of sporulaton n buddng yeast, Scence, 282, , 998. [7] Y. Sawarag, H. Nakayama, and T. Tanno, Theory of multobectve optmzaton (vol. 76 of Mathematcs n Scence and Engneerng). Orlando, FL: Academc Press Inc., 985. [8] K. Deb, Mult-Obectve Optmzaton usng Evolutonary Algorthms, John Wley & Sons, 200. [9] C. A. Coello Coello, G. B. Lamont, and D. A. Van Veldhuzen, Evolutonary Algorthms for Solvng Mult-Obectve Problems, Sprnger, [20] R Storn and K Prce, Dfferental evoluton a smple and effcent heurstc for global optmzaton over contnuous spaces, Journal of Global Optmzaton, (4) (997) [2] R. Storn, K. V. Prce, and J. Lampnen, Dfferental Evoluton - A Practcal Approach to Global Optmzaton, Sprnger, Berln, [22] C. A. Mattson, A. A. Mullur, and A. Messac, Smart Pareto flter: Obtanng a mnmal representaton of multobectve desgn space, Eng. Optm., vol. 36, no. 6, pp , [23] J. Branke, K. Deb, H. Derolf, and M. Osswald, Fndng knees n mult-obectve optmzaton, n Proc. 8th Int. Conf. Parallel Problem Solvng From Nature, pp , [24] R. Tbshran, G. Walther, and T. Haste, Estmatng the number of clusters n a dataset va the Gap statstc, J. Royal Statst. Soc.: SeresB (Statstcal Methodology), vol. 63, no. 2, pp , 200. [25] L. Hubert and P. Arabe, Comparng parttons, Journal of Classfcaton, 93 28, 985. [26] W. M. Rand, Obectve crtera for the evaluaton of clusterng methods, Journal of the Amercan Statstcal Assocaton, 66, , 97. [27] P. J. Rousseeuw, Slhouettes: A graphcal ad to the nterpretaton and valdaton of cluster analyss, J. Comput. Appl. Math., vol. 20, no., pp , 987. [28] C. Blake, E.Keough and C.J.Merz, UCI repostory of machne learnng database (998). [29] S. Theodords and K. Koutroumbas, Pattern Recognton, Second Edton, Elsever Academc Press, [30] D. A. Kem and H.-P. Kregel, Vsualzaton technues for mnng large databases: a comparson, IEEE Transactons on Knowledge and Data Engneerng, v.8 n.6, p , December 996. [3] [32] [33]