An Evolvable Clustering Based Algorithm to Learn Distance Function for Supervised Environment

Transcription

1 IJCSI Internatonal Journal of Computer Scence Issues, Vol. 7, Issue 5, September 2010 ISSN (Onlne): An Evolvable Clusterng Based Algorthm to Learn Dstance Functon for Supervsed Envronment Zenab Khorshdpour, Sattar Hashem, Al Hamzeh Dept. of Computer Scence, Shraz Unversty, Iran Abstract Ths paper ntroduces a novel weght based approach to learn dstance functon to fnd the weghts that nduce a clusterng by meetng best objectve functon. Our method combnes clusterng and evolutonary algorthms for learnng weghts of dstance functon. Evolutonary algorthms, are proved to be good technques for fndng optmal solutons n a large soluton space and to be stable n the presence of nose. Experments wth UCI datasets show that employng EA to learn the dstance functon mproves the accuracy of the popular nearest neghbor classfer. Keywords: dstance functon learnng; evolutonary algorthm; clusterng algorthm; nearest neghbor. 1 Introducton Almost all learnng tasks, lke case based reasonng [1], cluster analyss and nearest neghbor classfcaton, manly depend on assessng the smlarty between objects. Unfortunately, however, defnng object smlarty measures s a dffcult and non trval task, say, they are often senstve to rrelevant, redundant, or nosy features. Many proposed methods attemp to reduce ths senstvty by parameterzng K NN's smlarty functon usng feature weghtng. The dea behnd feature weghtng s that real world applcatons nvolve wth many features; however, the objectve functon depends on few of them. The presence of nosy objects or rrelevant features n a dataset degrades the performance of machne learnng algorthms; for many cases, such n the case of k nearest neghbor machne learnng algorthm (K NN). Thus feature weghtng technque may mprove the algorthm s performance. Ths paper ntroduces a novel weght based dstance functon learnng to fnd the weghts that nduce a clusterng by meetng best objectve functon. In the recent years, dfferent approach proposed for learnng dstance functon from tranng objects. Sten and Nggemann use a neural network approach to learn weghts of dstance functons based on tranng objects [2]. Eck et.al ntroduce an approach to learn dstance functons that maxmzes the clusterng of objects belongng to the same class [3]. Objects belongng to a dataset are clustered wth respect to a gven dstance functon and the local class densty nformaton of each cluster s then used by a weght adjustment heurstc to modfy the dstance functon. Another approach, ntroduced by Kra and Rendell and Salzberg, reles on an nteractve system archtecture n whch users are asked to rate a gven smlarty predcton, and then usng a Renforcement Learnng(RL) based technques to enhance the dstance functon based on the user feedback [4], [5]. Kononenko proposes an extenson to the work by Kra and Rendell for updatng attrbute weghts based on ntracluster weghts [6]. Bagherjeran et.al propose a renforcement learnng algorthm that can ncorporate feedback and past experence to gude the search toward better cluster [7]. They use an adaptve clusterng envronment that modfes the weghts of a dstance functon based on a feedback. The adaptve clusterng

2 IJCSI Internatonal Journal of Computer Scence Issues, Vol. 7, Issue 5, September 2010 ISSN (Onlne): envronment s a non determnstc RL envronment. Its state space consst of pars of cluster representatves and dstance functon weghts. Gven a state, the learner can take an acton that ether ncreases or decreases the weght of a sngle attrbute. The search process uses the Q learnng algorthm to estmate the value of applyng a partcular weght change acton to a partcular state. Fndng the best weght n a weght based dstance functon s clearly an optmzaton problem. Ths paper presents an alternatve search space whch s more effectve than exstng approach. Our method combnes clusterng algorthm and evolutonary algorthms for learnng weghts of a dstance functon. Evolutonary algorthms proved to be good technques for fndng solutons n a large soluton space and to be stable n the presence of nose. Central dea of our approach s to use clusterng as a tool to evaluate and enhance dstance functon wth respect to an underlng class structure. In ths work, we address the problem of optmzng a weght based dstance functon by means of a clusterng algorthm and Evolutonary Strateges (ES). We show that combnng CLES DL method wth a K NN algorthm can mprove the predctve accuracy of the resultng classfer. The paper s organzed as follows: Secton 2 ntroduces clusterng algorthms. In secton 3 we revew evolutonary algorthms especally the evolutonary strateges. In secton 4, we talk about the proposed algorthm. The expermental methodology and UCI data sets used n ths work are covered n secton 5. Fnally the expermental result s descrbed n secton 6 as well as the results. The paper ends wth the conclusons. 2 Clusterng Algorthm A clusterng algorthm fnds groups of objects n a predefned attrbute space. Snce the objects have no known pror class membershp, clusterng s an unsupervsed learnng process that optmzes some explct or mplct crteron nherent to the data such as the squared summed error [8]. The man objectve of clusterng algorthm s to dvde the data nto dfferent groups called clusters n such way that the data wthn a cluster are closer to each other and data from dfferent clusters are farther from each other. Dstance crtera are the prncple component of every clusterng algorthm to approach ts objectve. Eucldean dstance s a commonly used dstance measure n most cases. Whch s defned as follow: d( x, x j ) = m 2 ( xa x ja ). (1) a=1 where x and x j are m dmensonal objects and x a s the value of attrbute "a" for a gven object x. K means and other parttonng algorthm typcally use Eucldean or Manhattan dstance metrc [8]. Many extensons to these and other parttonng algorthms attempt to employ more sophstcated dstance metrcs. On the other hand, another group of approaches attempt to learn the dstance metrc. Includng the approach explored n ths paper whch tres to learn attrbute weghts based on the followng object dstance functon d: d( x, x j ) = m 2 wa ( xa x ja ). (2) a=1 where w s weght vector whose components are non negatve and

3 IJCSI Internatonal Journal of Computer Scence Issues, Vol. 7, Issue 5, September 2010 ISSN (Onlne): w = 1 m m. It assumes that objects are descrbed by sets of attrbutes and the dssmlarty of dfferent attrbutes s measured ndependently. The dssmlarty between two objects s measured as a weghted sum of dssmlartes between ther attrbutes. To be able to do that, a weght and a dstance measure has to be provded for each attrbute. We use clusterng algorthm for learnng weght of dstance functon, and modfed dstance,.e. Equaton 2. In ths paper we combne clusterng and evolutonary algorthm for selectng best weght of the features and modfed dstance functon. In secton 3, we revew evolutonary algorthms especally the evolutonary strateges. 3 Evolutonary Algorthms Two major representatve algorthms of the evolutonary algorthms are the Genetc Algorthms (GA) and Evoluton Strateges (ES) [9], [10]. We ntroduce man characterstcs of evolutonary algorthm n the next sectons. 3.1 Chromosome Representaton and Populaton Intalzaton For any GA and ES, a chromosome representaton s needed to descrbe each chromosome n the populaton. The representaton method determnes how, a soluton s represented n the search space and also determnes the type of varaton operators such as crossover and mutaton. Each chromosome s made up of a sequence of genes from certan alphabet whch can consst of bnary dgts (0 and 1), floatng pont numbers, ntegers, symbols (.e., A, B, C, D), etc. It has been shown that more natural representatons can get more effcent and better solutons [11]. In ths paper we use ES algorthm because a real valued representaton s utlzed to descrbe the chromosome. ES strateges are typcally used for contnuous parameter optmzaton. Standard representaton of objectve varables x 1...xn s very straghtforward, where each x s represented by a floatng pont varable. Chromosome contan some strategy parameters, n partcular, parameter of mutaton operator. Detals of mutaton are treat n subsecton 3.2. Strategy parameters can be dvded nto two sets, the values and the values. The values represent the mutaton step szes, that can be a vector of sze n or only a sngleton. For any reasonable self adaptaton mechansm at least one must be present. The values whch represent nteractons between the step szes used for dfferent varables, are not always used [12]. 3.2 Mutaton In typcal EAs, Mutaton s carred out by randomly changng the value of a sngle bt (wth small probablty) to the bt strngs. The mutaton operator n ES may be appled ndependently on each objectve varable by addng a normally dstrbuted random varable wth expectaton zero and standard devaton as shown n Equaton 4, consderng n = 0. The strategy parameters are mutated usng a multplcatve, logarthmc normally dstrbuted process as shown n Equaton 3. ' =. exp( N(0,1) N (0,1)). (3) x ' = x '. N (0,1). (4) where N(0, 1) s a normally dstrbuted random varable wth zero expectaton and standard devaton of 1, N (0,1) ndcates that the random varable

4 IJCSI Internatonal Journal of Computer Scence Issues, Vol. 7, Issue 5, September 2010 ISSN (Onlne): s sampled for every objectve varable 1 ndependently. The constant ( 2 n 1 and ( 2n) can be nterpreted n the sense of "learnng rates" as n artfcal neural networks [13]. Algorthm 1 shows a basc pseudo code of ES. CLES DL method. Each chromosome contans real valued genes whch are nterpreted as the weghts of the desred dstance functon. Accordng to each chromosome, we have a new dstance functon, and so the clusterng algorthm generates new clusters from the data set. In ths algorthm, an ntal populaton of sze P can be randomly generated and les n the nterval [0, 1]. W [ w, w,..., ;,,... ]. (5) = 1 2 w n 1 2 n 4 Combng Clusterng Algorthms and ES for Dstance Learnng In ths secton, we wll gve an overvew of CLES DL approach. The key dea of our approach s to use clusterng as a tool to evaluate and enhance dstance functon wth repect to an underlyng class structure. Fgure 1: System structure of our dstance functon learnng approach (CLES-DL). Each chromosome contans weghts of dstance functon. ES generates dfferent weghts and clusterng algorthm used to evaluate the weghts. Our method combnes clusterng algorthm and evolutonary algorthms for learnng weghts of a dstance functon. ES generates dfferent weghts and clusterng algorthm used to evaluate the chromosomes. Fgure 0, shows the overall system archtecture of w = w. n w where W s a chromosome and whose th components are the normalzed and non negatve, weght of the th attrbute and s the mutaton step sze accordng to each attrbute. 4.1 Ftness Functon The ftness functon s used to defne a ftness value for each canddate soluton. Our goal s to generate a chromosome wth the best (max or mn dependng on the problem) ftness value. In ths problem, the ftness measure s the clusterng qualty provded by an algorthm whch s often a hard and subjectve to be measured. We use accuracy crtera as the ftness functon to evaluate the results. Assume that the nstances n D have been already classfed n k classes c 1, c2,..., ck. Consder a clusterng algorthm that parttons D nto k clusters cl 1, cl2,..., clk. We refer to a one to one mappng, F, from classes to clusters, such that each class c ths mapped to the cluster cl j = F( c ). The classfcaton error of the mappng s defned as: l l

5 IJCSI Internatonal Journal of Computer Scence Issues, Vol. 7, Issue 5, September 2010 ISSN (Onlne): k E = c F( c ). (6) =1 where c F( c ) measures the number of objects n class c that receved the wrong label. The optmal mappng between clusters and classes s the one that mnmzes the classfcaton error. We use E mn to denote the classfcaton error of the optmal mappng. To obtan the accuracy we compute the followng formula: Emn Accuracy = 1. D (7) The selecton process s ( ) 5.2 Descrpton of Datasets The expermental procedure s to the apply algorthm to several real world datasets from the UCI machne learnng repostory [14]. Most of these data sets have been subject of emprcal evaluaton by other researches. Ths gves the chance of comparng results among dfferent researches. A summary of characterstcs of these data sets are showed n Table 1. Table 1: Data sets used n the experments 5 Expermental Methodology In order to measure the performance of the proposed algorthms, we employed the 10 fold cross valdaton methodology. Cross valdaton randomly dvdes the avalable dataset nto k mutually exclusve subsets (folds) of approxmately equal sze, and uses the nstances of the k 1 subsets as the tranng data and the nstances of the remanng subset as the test data. The tranng dataset s used for determnng the classfer s parameters and the test dataset s used to compute the predcton error. The experment s carred out tmes to compute the predcton error of the whole dataset. Ths method s called k fold cross valdaton. In our experments we set k=10 whch s most commonly used value. 5.1 Expermental Settng Our experments were run usng standard evolutonary strategy wth tournament selecton. The reported results are based on 10 fold cross valdaton for each classfcaton task wth the followng parameter settng. Populaton sze: 50 Number of generaton: 50 Mutaton rate: Expermental Results To obtan the best dstance functon, we run the program 10 tmes ndependently. In each run, ES algorthm evolves ts populaton for 50 generaton and save the chromosomes wth the best ftness value. It s worth mentonng that result of the K means clusterng algorthm based on represented chromosome s used as the ftness value for the evolutonary process. The whole process of achevng results s depcted n Fgure 1. Table 2 shows the results acheved by the algorthm appled to the UCI datasets. CLES DL method acheve good performance n all datasets. For each dataset, the accuracy of the method wth the hghest mean accuracy s marked n bold face. The mean accuracy n all UCI data sets usng proposed algorthm s 73.87% whereas the orgnal K means reach to

6 IJCSI Internatonal Journal of Computer Scence Issues, Vol. 7, Issue 5, September 2010 ISSN (Onlne): %. It has about 8% absolute accuracy mprovement. Also to further evaluaton, 1 NN classfer algorthm s used [15]. The best chromosome over 10 ndependent run s passed as a weght vector to weghted 1 NN classfer. Table 3 shows the result of 10 fold cross valdaton over orgnal 1 NN and 1 NN wth CLES DL. The mean accuracy n 8 UCI data sets by 1 NN wth CLES DL s 80.73% and by orgnal 1 NN s 77.48%. It has about 3% absolute accuracy mprovement. When comparng mean accuraces n Table 3, we fnd that 1 NN wth CLES DL outperforms orgnal 1 NN on 8 datasets (7/8= 0.87), among them. Notably, the accuracy for the wne dataset mproved from 75.10% to over 91.50% n contrast accuracy of 1 NN wth CLES DL for breast cancer dataset s lower than orgnal 1 NN. Obvously, one may see that our approach outperforms the orgnal K NN by offerng sgnfcant mprovements. To explore the advantages of the proposed approach over well known weghtng mechansm, we conduct a new seres of experments. In these experments, we compared our weghtng approach aganst LW1NN algorthm (1 NN classfer wth attrbute weghtng) [3] and bagherjeran method [7]. Bagherjeran approach uses several parameters, accordng to these parameters four verson of ths method are represented. Table 4 shows the result of 10 fold cross valdaton over 1 NN wth CLES DL, LW1NN and bagherjeran's method. The results show the mean accuracy n UCI data sets by 1 NN wth CLES DL s better than LW1NN and bagherjeran's method. Fgure 2: Overall vew of obtanng result. We run the program 10 tmes. In each run, ES algorthm evolves populaton over 50 generaton and save chromosome wth the best ftness value. Table 2: Clusterng accuracy (and standard devaton), The comparson between K-means wth CLES-DL and orgnal K-means on 8 datasets from UCI data repostory. All standard devaton less than are not denoted. Data set K-means_CLES-DL K-means Glass Ionosphere Dabetes Sonar Vehcle Wne Breast-cancer Irs Table 3: Classfcaton accuracy (and standard devaton) on 8 datasets selected from the UCI data repostory. For each dataset, the accuracy of the method wth the hghest mean accuracy s marked n bold face. Data set 1-NN_CLES-DL 1-NN Glass Ionosphere Dabetes Sonar Vehcle Wne Breast-cancer Irs

7 IJCSI Internatonal Journal of Computer Scence Issues, Vol. 7, Issue 5, September 2010 ISSN (Onlne): Table 4: Classfcaton accuracy (and standard devaton) on 5 datasets selected from the UCI data repostory Data set 1NN_CLES-DL Bagherjeran's method LWINN Verson_(1) Verson_(2) Verson_(3) Verson_(4) Glass Ionosphere Dabetes Sonar Vehcle Averages Conclusons We present a evolutonary method for learnng dstance functon wth the man objectve of ncreasng the predctve accuracy of K NN classfer. The method s a combnaton of a clusterng algorthm and ES n order to fnd the best weght of each attrbute to be used n a K NN classfer. The proposed method n ths paper overcomes the dsadvantages of the K NN algorthm.e. ts senstvty to the presence of rrelevant features. As a future work we plan to extend the current work so as t can deal wth nose as well. Acknowledgements Ths work was supported by the Iran Tele Communcaton Research Center. References [1] P. perner. Case-based reasonng. Data mnnh on multmeda data, Lecture Notes n Artcal Intellegence, vol Sprnger,Berln, pp [2] B. Sten, O. Nggemann. Generaton of smlarty measures from dfferent sources. 14th Internatonal conference on Industral Engneerng Applcaton of Artfcal Intelgence and Expert Systems, Budapest, Hungary, Spernger, Berln, pp [3] C. Eck, A. Rouhana, A. Bagherjeran, R. Vlalta. Usng Clusterng to Learn Dstance Functons for Supervsed Smlarty Assessment. Journal of Engneerng Applcatons of Artfcal Intellgence, Vol. 19, Issue 4, pp [4] S. Salzberg. A nearest hyperrectangle learnng method, Machne Learnng 6, pp [5] K. Kra, L. A. Rendell. A practcal approach to feature selecton, Proceedngs of the Nnth Internatonal Workshop on Machne Learnng, Aberdeen, Scotland, UK, Morgan Kaufmann, Los Altos, CA, pp [6] I. Kononenko. Estmatng attrbutes: analyss and extensons of RELIEF. Proceedngs European Conference on Machne Learnng, pp [7] A. Bagherjeran, R. Vlalta, C. Eck. Adaptve Clusterng: Obtanng Beter Clusters Usng Feedback and Past Experence, Proceedngs of the Ffth IEEE Internatonal Conference On Data Mnng (ICDM 05), [8] J. M. McQueen. Some methods of classfcaton and analyss of multvarate observatons, Proc. 5th Berkeley Symp. On Mathematcal Statstcs and Probablty, pp , [9] D. Golberg. Genetc algorthms n search, optmzaton and machne learnng, Addson-Wesley Publshng Company, [10] H. P. Schwefel. Evoluton and Optmum Seekng, Wley,New York, [11] Z. Mchalewcz. Genetc Algorthms+Data Structures=Evoluton Programs. AI Seres, Sprnger, New York [12] A. E. Eben, J. E. Smth. Introducton to Evolutonary Computng, Sprnger, Natural Computng Seres 1 st edton [13] K. Mehrotra, C. K. Mohan, S. Ranka. Elements of artfcal neural networks, MIT Press [14] C. L. Blake, C. J. Merz. UCI repostory of machne learnng database, [15] S. Russel, P. Norvg. Artfcal Intellgence: A Modern Approach, Prentce Hall, 2nd edton, Zenab Khorshdpour was born n bandar-lengh, Iran. She receved her B.Sc. Degree n Computer Engneerng from shahd-behesht unversty n She s currently an M.Sc. student n Artfcal Intellgence at Shraz Unversty. Her research nterests nclude dstance functon learnng for nomnal attrbute, clusterng for categorcal data, bo-nspred algorthms. Sattar Hashem receved the PhD degree n Computer Engneerng from the Iran Unversty of Scence and Technology, n conjuncton wth Monash Unversty, Australa, n He s currently a

8 IJCSI Internatonal Journal of Computer Scence Issues, Vol. 7, Issue 5, September 2010 ISSN (Onlne): lecturer n the Electrcal and Computer Engneerng School, Shraz Unversty, Shraz, Iran. Hs research nterests nclude data stream mnng, database ntruson detecton, dmenson reducton, and adversaral learnng. Al Hamzeh receved hs Ph.D. n artfcal ntellgence from Iran Unversty of Scence and Technology (IUST) n Snce then, he has been workng as assstant professor n CSE and IT Department of Shraz Unversty. There, he s one of the founders of local CERT center whch serves as the securty and protecton servce provder n ts local area. As one of hs research nterests, he recently focuses on cryptography and steganography area and works as a team leader n CERT center to develop and break steganography method, especally n mage spatal doman. Also, he works as one of team leaders of Soft Computng group of shraz unversty workng on bo-nspred optmzaton algorthms. He s co-author of several artcles n securty and optmzaton.