Journal of Chemical and Pharmaceutical Research, 2014, 6(6): Research Article. A selective ensemble classification method on microarray data

Transcription

1 Avalable onlne Journal of Chemcal and Pharmaceutcal Research, 2014, 6(6): Research Artcle ISSN : CODEN(USA) : JCPRC5 A selectve ensemble classfcaton method on mcroarray data Tao Chen School of Mathematcs and Computer Scence, Shaanx Unversty of Technology, Hanzhong Shaanx, Chna ABSTRACT For the characterstcs of small samples and hgh dmenson of mcroarray data, ths paper proposes a selectve ensemble method teachng-learnng-based optmzaton based to classfy mcroarray data. Frstly, n order to remove rrelevant genes wth classfcaton task, releff algorthm s used to reduce orgnal gene set, and then a new tranng set s produced from orgnal tranng set accordng to top-ranked genes obtaned. Secondly, multple bootstrap tranng subsets are produced based on baggng algorthm on above obtaned tranng set to tran base classfers. Fnally, multple base classfers are selected by usng teachng-learnng-based optmzaton to buld an ensemble classfer. Expermental results on eght mcroarray datasets show our proposed method s effectve and effcent for mcroarray data classfcaton. Key words: Mcroarray data; selectve ensemble; releff; baggng; teachng-learnng-based optmzaton INTRODUCTION The dagnoss of complex genetc dseases lke cancer has conventonally been done based on the non-molecular characterstcs lke knd of tumor tssue, pathologcal characterstcs and clncal phase. DNA mcroarray technology has concerned great attenton n both the scentfc and n ndustral areas. Numerous examnatons have been presented on the usage of mcroarray gene expresson examnaton for molecular categorzaton of cancer. Several machne learnng technques have been used to classfy mcroarray data [1]. However, due to the characterstcs of small samples and hgh dmenson of mcroarray data, and many exstng rrelevant and redundant genes. It s leads to poor classfcaton performance for most machne learnng methods. In order to solve ths problem and mprove classfcaton performance, ensemble technology was ntroduced to the area of data classfcaton and obtan greatly success [2]. Ensemble learnng s a machne learnng paradgm where multple learners are traned to solve the same problem [3-5]. In contrast to ordnary machne learnng approaches whch try to learn one hypothess from tranng data, ensemble methods try to construct a set of hypotheses and combne them to use. An ensemble contans a number of learners whch are usually called base learners. The generalzaton ablty of an ensemble s usually much stronger than that of base learners. In 1995, Krogh ndcated that the generalzaton error of ensemble s equal to average generalzaton error of ndvdual mnus the average dfferences of ndvdual. Therefore, to enhance the generalzaton performance of ensemble, we should not only maxmze the generalzaton ablty of base classfers, but also ncrease the dfferences between the varous base classfers [6-7]. Baggng [8] and boostng [9] are most common ensemble algorthms and acheve hgher performance. At present, most ensemble learnng methods employ all base learners traned to buld an ensemble. However, t leads to the ncrease of storage space and computaton tme, moreover the strategy of combnng all base learners always does not acheve the best generalzaton performance. Selectve ensemble s proposed to mprove performance of ensemble method. 2860

2 Teachng-learnng-based optmzaton s a novel ntellgent optmzaton algorthm based on populaton [10-11].TLBO smulates the behavor of teachng and learnng n a class to mprove academc performance of learners. Compare wth genetc algorthm, partcle swarm, harmony Search and dfferental Evoluton, the bggest advantage of Tlbo s that t does not requre any specfc parameter to be set.moreover, t has other several merts, such as fast convergent rate, smple prncple and globe optmzaton, etc. Ths paper proposes a selectve ensemble method and t composes of three phases. The frst phase s produces a new tranng set reduced from orgnal tranng set by usng releff algorthm [12]. In the second phase, multple tranng subsets are produced by usng bootstrap technology, and then multple base classfers are traned on above every tranng subset. The three phase, a set of base classfers are selected by usng teachng-learnng-based optmzaton and combned to buld an ensemble by weghted votng. In order to evaluate effectveness of our proposed method, eght benchmark mcroarray datasets are selected and used n our experment. EXPERIMENTAL SECTION RelefF algorthm: RelefF s an extended and more robust verson of the orgnal Relef algorthm [12]. In contrast to many heurstc measures for feature selecton, RelefF does not assume condtonal ndependence of the varables. The man dea of RelefF s to estmate the qualty of features based on how good ther values dscrmnate between samples that are close. Consecutvely random samples are drawn from the data set. Each tme the k nearest neghbors of the same class and the opposte class are determned. Based on these neghborng cases the weghts of the attrbutes are adusted. As wthn the two prevous algorthms the varables are ranked and dfferent models are bult by droppng the varable wth the smallest weght. The remanng part of the selecton procedure s completely analogous to the one followed n the two prevous methods. Although the RelefF algorthm s computatonally more expensve and complex than the prevous technques, the cost of an exhaustve search s stll much hgher. Baggng algorthm: Baggng derved from bootstrap aggregaton, was the frst effectve method of ensemble learnng and s one of the smplest methods of archng [8]. The meta-algorthm, whch s a specal case of model averagng, was orgnally desgned for classfcaton and s usually appled to decson tree models, but t can be used wth any type of model for classfcaton or regresson. The method uses multple versons of a tranng set by usng the bootstrap,.e. samplng wth replacement. Each of these datasets s used to tran a dfferent model. The outputs of the models are combned by averagng (n the case of regresson) or votng (n the case of classfcaton) to create a sngle output. Baggng trans a number of base learners each from a dfferent bootstrap sample by callng a base learnng algorthm. A bootstrap sample s obtaned by subsamplng the tranng data set wth replacement, where the sze of a sample s as the same as that of the tranng data set. Thus, for a bootstrap sample, some tranng examples may appear but some may not, where the probablty that an example appears at least once s about After obtanng the base learners, Baggng combnes them by maorty votng and the most-voted class s predcted. Teachng-learnng-based optmzaton: Teachng-Learnng-Based optmzaton (TLBO) s a novel heurstc optmzaton algorthm based on nature [10-11]. The man dea of TLBO s to make use of the effect of the nfluence of a teacher on the output of learners n a class to acheve optmzaton purpose. The TLBO nclude two stages: teachng stage and learnng stage. Teachng stage s that the learners (students) learn from teacher, and learnng stage s that the learners (students) learn from one another. The bggest advantage of Tlbo s that t does not requre any specfc parameter to be set, moreover, t has other several merts, such as fast convergent rate, smple prncple and globe optmzaton, etc. In ths paper, a set of base classfers are selected from all the base classfers by usng teachng-learnng-based optmzaton and the algorthm s as follows. Algorthm: Base classfers selecton based on TLBO Input: Tranng set S, Testng set T, all the base classfers f 1, f 2,..., fd and weght of base classfers w1, w2,... w D Output: base classfers selected f, f,..., f 1 2 and ensemble classfcaton n Step 1: Intalze parameters. populaton sze NP,number of generatons G,the number of all base classfers D Step 2 :Intalze the populaton Usng the formula X = round ( rand(1, D)), we can randomly generate a populaton 2861

3 é é X ù x 1 1,1 x1,2 K x 1, D ê X x 2 ú 2,1 x2,2 x2, D ê K ú ù pop = =.where X = { x,1, x,2,..., x, D} s a bnary vector that represent the th ndvdual M M M M M X x NP NP,1 xnp,2 x êë úû êë K NP, D úû n pop, x, Î {0,1}. Each ndvdual ndcates a set of base classfers selected. If the th classfers s selected,the th poston of X s 1; whle f the th classfers s not selected, the th poston of X s 0. Step 3: Calculate the ftness of each ndvdual n pop. Accordng ndvdual X,a set of base classfers are selected and ensembled by weghted votng, and the ensemble classfcaton accuracy s expressed as f ( X ),that s ftness of the th ndvdual, so we calculate the ftness é f ( X1) ù f ( X 2) of all the ndvduals ftness =. M f ( X NP ) êë úû Step 4: For =1: G (1) Teachng phase (a) Calculate the dfference. Frst,the mean of populaton pop s calculate and expressed as M = [ m1, m2, K m D ],where NP m = å x NP ; k, k= 1 Second,fnd the best ndvdual from pop as teacher X = X ; teacher f ( X ) = max{ f ( X ), f ( X ), K f ( X )} Thrd, the dfference between M and where TF = round (1 + rand(1, D)(2-1)) Î {1,2} ; (b) For =1: NP X = X + Dfference ; calculate ftness f ( X ) ; f f ( X ) > f ( X ) X = X teacher End f End For; (2) Learnng phase For =1: NP Randomly select another ndvdual X k,such that k ¹ ; If f ( X ) > f ( X ) End If k X = X + rand(1, D) ( X - X ) k Else X = X + rand(1, D) ( X - X ) k Calculate ftness f ( X ) ; f f ( X ) > f ( X ) X = X 1 2 X s expressed as Dfference = rand(1, D) ( X - TF M ), End f; End For; End For; Step 5: Output base classfers selected and ensemble classfcaton. A new populaton s generated after G tmes teraton, we fnd the best ndvdual X = X and the best ftness ( ) = max{ ( ), ( ), K ( )} f ( X best ),where X best represent a set of base classfers selected best f X f X1 f X 2 f X NP and f ( X best ) represent ensemble classfcaton accuracy. NP teacher : Dversty among base classfers and accuracy of base classfers are key factors for affectng performance of 2862

4 ensemble learnng. The success of baggng algorthm s to produce tranng subsets wth dversty by usng bootstrap technology, therefore dversty of base classfers s obtaned by usng baggng algorthm. For mprovng accuracy of base classfers, RelefF s effectve method because t can remove rrelevant genes from orgnal genes set to mprove classfcaton performance. In addton, TLBO s employed to select a set of base classfers to buld ensemble because of advantages of TLBO. Accordng to above analyss, a selectve ensemble based on TLBO s proposed for classfyng mcroarray data. The concrete steps of our proposed method s gven as follows. Step1.Gene reducton RelefF algorthm s appled to remove rrelevant genes from orgnal gene set, and a set of genes s buld to produce a reduced tranng set from orgnal tranng set. Ths step can mprove accuracy of classfcaton because of removng of rrelevant genes. Step2. Producton of base classfers Mutple tranng subsets are obtaned by usng bootstrap technology to tran classfers. Because tranng subsets have much dversty, base classfers traned have dversty. Step3.Selecton of base classfers A set of base classfers are selected by usng TLBO to decrease storage space and computaton tme. Step4.Ensemble of base classfers selected Base classfers selected are ensemble by weghted votng to classfy new samples. Expermental data and methods: To evaluate performance of our proposed method, eght benchmark mcroarray datasets are selected and used n our experments. The nne datasets are descrbed n table 1. Table 1 nne benchmark cancer mcroarray datasets Data set classes genes samples tranng samples testng samples Colon Leukema DLBCL Glomas Leukema MLLLeukema SRBCT ALL In addton, n order to comparson superorty of our proposed method, four method (orgnal, baggng, adaboost and RelefF+baggng) are mplemented. In our experment, support vector machne wth RBF (RBF-SVM) s employed as classfer. To ensure the results of dfferent methods does not happen by chance, the experments are repeated 30 tmes ndependently, and results of 30 tmes are averaged as fnal expermental results. Expermental results and analyss: Many studes show the number of base classfers n ensemble can also affect performance of ensemble algorthm. Therefore, the number of base classfers s equal to10, 20, 30, 40 and 50 n our experment, respectvely. Table 2-6 gve the results of dfferent methods on nne datasets when number of base classfers s equal to 10,20,30,40 and 50, respectvely. The best and average of our proposed method are gven because randomness of TLBO. The best and average represents the best results and average results of 30 tmes experments. The std represents standard devaton of 30 tmes expermental results. The Num represents average number of base classfers selected by usng TLBO n 30 tmes experments. We fnd that phenomenon of reflectng from table 2-6 s smlar. It s easy to fnd that the classfcaton accuracy of our proposed method s obvously hgher than other methods. Especally, our proposed method outperforms RelefF+baggng and t ndcates selectve ensemble based on TLBO s effectve for mprovng performance of ensemble. Table 4 s analyzed as a sample and results are gven as follows. Table 4 dsplays the comparson of dfferent methods on nne datasets when number of base classfers s equal to 30.It s obvously that the classfcaton accuracy of our proposed method s the hghest n fve methods accordng to table 4 and t ndcates our proposed method s effectve for mcroarray data classfcaton. For example, for colon, classfcaton accuracy of our proposed method acheves 84.74%, whch s mproved at least 2863

5 11.06% than other methods. For Leukema1, classfcaton accuracy of our proposed method acheves 85.59%, whch s mproved at least 3.24% than other methods. For DLBCL, classfcaton accuracy of our proposed method acheves 96%, whch s mproved at least 9.33% than other methods. For Glomas, classfcaton accuracy of our proposed method acheves 86.33%, whch s mproved at least 9.66% than other methods. For Leukema2, classfcaton accuracy of our proposed method acheves 92.65%, whch s mproved at least 13.24% than other methods. For MLLLeukema, classfcaton accuracy of our proposed method acheves 92.67%, whch s mproved at least 8.23% than other methods. For ALL, classfcaton accuracy of our proposed method acheves 92.44%, whch s mproved at least 7.79% than other methods. Only on SRBCT, classfcaton accuracy of our proposed method and RelefF+baggng are same and acheve 100%, whch s mproved at least 30% than other three methods. Compare wth RelefF+baggng method, the classfcaton accuracy of our proposed method s mproved 11.06%,9.12%,9.33%,9.66%,19.12%,8.23% and 3.6% on Colon, Leukema1,DLBCL,Glomas,Leukema2, MLLLeukema,ALL, respectvely. For SRBCT, the result of the two methods are same. In general, our proposed method outperforms RelefF+baggng and t ndcates selectve ensemble based on TLBO s effectve for mprovng classfcaton performance. In table 4, avg represents summarzed result whch s calculates by averagng the accuracy over all datasets. The classfcaton accuracy of our proposed method s the hghest and acheves 91.82%, whch s 27.2%, 22.94%,15.72% and 8.76% hgh than that of four methods, respectvely. In addton, the average number of base classfers selected for 30 s 9, about 0.3 (9/30). Table 2 The results of dfferent methods (the number of all the base classfers s equal to 10) Orgnal Baggng RelefF+ baggng Best average std Num Colon Leukema DLBCL Glomas Leukema MLLLeukema SRBCT ALL avg Table 3 The results of dfferent methods (the number of all the base classfers s equal to 20) Orgnal Baggng RelefF+ baggng Best average std Num Colon Leukema DLBCL Glomas Leukema MLLLeukema SRBCT ALL avg Table 4 The results of dfferent methods (the number of all the base classfers s equal to 30) Orgnal Baggng RelefF+ baggng Best average std Num Colon Leukema DLBCL Glomas Leukema MLLLeukema SRBCT ALL avg

6 Table 5 The results of dfferent methods (the number of all the base classfers s equal to 40) Orgnal Baggng RelefF+ baggng best average std Num Colon Leukema DLBCL Glomas Leukema MLLLeukema SRBCT ALL avg Table 6 The results of dfferent methods (the number of all the base classfers s equal to 50) Orgnal Baggng RelefF+ baggng best average std Num Colon Leukema DLBCL Glomas Leukema MLLLeukema SRBCT ALL avg Fg 1 dsplays the nfluence of number of base classfers on classfcaton accuracy. We fnd accuracy does not monotonously ncrease wth number of base classfers. The classfcaton accuracy of our proposed method acheves the hghest when number of base classfers s about 20 or average classfcaton accuracy Colon Leukema1 DLBCL Glomas Leukema2 MLLLeukema SRBCT the total number of base classfers ALL Fg 1 the nfluence of the number of base classfers on classfcaton performance CONCLUSION Ths paper proposes a selectve ensemble method to classfy mcroarray data. RelefF algorthm s used to remove rrelevant genes to mprove classfcaton performance. Tranng subsets produced by bootstrap technology have large dversty and base classfers traned have dversty. TLBO s appled to select a set of base classfers to buld an ensemble. Expermental results show our proposed method not only mprove classfcaton accuracy,but also decrease computaton tme and storage space. Therefore our proposed method s effectve and effcent for mcroarray data classfcaton. Acknowledgement Ths paper s supported by Natonal Natural Scence Foundaton of Chna ( , ) and Scentfc Research Program Funded by Shaanx Provncal Educaton Department. 2865

7 REFERENCES [1] Wang Shu-Ln, L X., Zhang S, et al. Computers n Bology and Medcne, 2010, 40(2), [2] Sh L, X L., M X.,et al, Appled Soft Computng, 2011,11(8), [3] Zhao Hu. Internatonal Journal of Securty and Its Applcatons, 2013,7(5), [4] Chen Tao. Applcaton Research of Computers.2011,28(1), [5] Zhou Zh-Hua. Rough Sets, Fuzzy Sets, Data Mnng, and Granular Computng. 2003, [6] Chen Tao. Journal of Computer Applcatons,2011,31(5), [7] Chen Tao, Hong Zeng-Ln. Software Engneerng and Knowledge Engneerng: Theory and Practce, 2012, [8] Breman L. Mach. Learn. 1996, 24(1), [9] Schapre R.. Mach. Learn. 1990,l5 (2), [10] Rao R V, Savsan V J, Vakhara D P. Informaton Scences, 2012,183(1),1-15 [11] Rao R V, Savsan V J, Vakhara D P. Computer-Aded Desgn, 2011,43(3), [12]Kononenko I.Proceedngs of the European conference on machne learnng, Lecture notes n computer scence.1994, 784,