Feature Subset Selection Based on Ant Colony Optimization and. Support Vector Machine

Transcription

1 Proceedngs of the 7th WSEAS Int. Conf. on Sgnal Processng, Computatonal Geometry & Artfcal Vson, Athens, Greece, August 24-26, Feature Subset Selecton Based on Ant Colony Optmzaton and Support Vector Machne Wan-lang WANG Yong JIANG S.Y. CHEN (Collego of Software Engneerng, Zhejang Unversty of Techonology, Hangzhou 3114, Chna) Abstract: One of the sgnfcant research problems n pattern recognton s the feature subset selecton. It s appled to select a subset of features, from a much larger set, through the elmnaton of varables that produce nose or strctly correlated wth other already selected features, such that the selected subset s suffcent to perform the classfcaton task. A hybrd method usng ant colony optmzaton and support vector machne s proposed. The ant colony optmzaton searches the feature space guded by the result of the SVM. The tests on datasets show the effectveness of the method. Keywords: Feature subset selecton; Ant colony optmzaton; Support vector machne 1 Introducton The problem of classfcaton n machne learnng conssts of usng labeled examples to nduce a model that classfes objects nto a set of known classes. The objects are descrbed by a vector of features, some of whch may be rrelevant or redundant and may have a negatve effect on the accuracy of the classfer. There are two basc approaches to feature subset selecton [1,2] :wrapper [3] and flter [4,5] methods. Wrappers treat the nducton algorthm as a black box that s used by the search algorthm to evaluate each canddate feature subset. Whle gvng good results n terms of the accuracy of the fnal classfer, wrapper approaches are computatonally expensve and may be mpractcal for large data sets. Flter methods are ndependent of the classfer and select features based on propertes that good feature sets are presumed to have, such as separablty or hgh correlaton wth the target. Although flter methods are much faster that wrappers, flters may produce dsappontng results, because they completely gnore the nducton algorthm. Blum and Langley [6] argued that most exstng feature selecton algorthms consst of the followng four components: (1) Startng pont n the feature space. The search for feature subsets could start wth () no features, () all features, or () random subset of features. In the frst case, the search proceeds by addng features successvely, whle n the second case, features are successvely removed. When startng wth a random subset, features could be successvely added/removed, or reproduced by a certan procedure. In ths paper, we try all these alternatves and compare under smulatons. (2) Search procedure. Ideally, the best subset of features can be found by evaluatng all the possble subsets, whch s know as exhaustve search. However, ths becomes prohbtve as the number of features ncreases. (3) Evaluaton functon. An mportant component of any feature selecton method s the evaluaton of feature subset. Evaluaton functons measure how good a specfc subset can be n dscrmnatng between classes. Performance of classfcaton algorthms s used to select features for wrapper methods. In ths paper, the support vector machne s used as the classfer. (4) Crteron for stoppng the search. Feature.

2 Proceedngs of the 7th WSEAS Int. Conf. on Sgnal Processng, Computatonal Geometry & Artfcal Vson, Athens, Greece, August 24-26, selecton methods must decde when to stop searchng through the space of feature subsets. In ths paper, the ant colony algorthm stops when there s not mprovement on the soluton after several teratons or when n max number of teratons s reached. A number of search algorthms have been proposed n the lterature. Some of the most famous ones are the stepwse, branch-and-bound, and Genetc Algorthms (GA). Ths paper presents a hybrd approach usng ant colony algorthm and support vector machne for feature subset selecton problems. The rest of ths paper s organzed as follow. In the next secton, an ntroducton on ACO and SVM s dscussed. In sectons 3 and 4, the proposed hybrd methodology s dscussed, followed by a dscusson on the expermental tests and the results. 2 Ant colony optmzaton and support vector machne 2.1 Ant colony optmzaton The nature has always been fascnatng to human beng and t has nspred many theores to be appled to varous areas. The ant colony algorthm emulates the behavor of real ants. Ants depost a substance called pheromone on the path that they have traversed from the source to the destnaton nest ant the ants comng at a later stage apply a probablstc approach n selectng the node wth the hghest pheromone tral on the paths. Thus the ants move n an autocatalytc process (postve feedback), favorng the path along whch more ants have traveled and by and by traverse all the nodes. Derved from the behavor of real ants, Dorgo et al. [7] defned a model, whch used artfcal ant colones as an optmzaton tool. In the proposed ant systems, ants are defned as smple computatonal agents havng some memory, they are not completely blnd lke real ants and lve n an envronment where tme s dscrete. The underlyng dea was to parallelze search over several constructve computatonal threads, based on a dynamc memory structure ncorporatng nformaton on the effectveness of prevously obtaned results. The characterstcs of ACO nclude: (1) a method to construct solutons whch balances pheromone trals wth a problem-specfc heurstc, (2) a method to both renforce & evaporate pheromone, and (3) local search to mprove solutons. ACO methods have been successfully appled to dverse combnatoral optmzaton problems ncludng travelng salesman, quadratc assgnment, vehcle routng problems, telecommuncaton networks, graph colorng, constrant satsfacton, Hamltonan graphs and schedulng [8]. 1.2 Support vector machnes(svms) SVMs [9] are a knd of learnng machne based on statstcal learnng theory. The basc dea of applyng SVM to pattern classfcaton can be stated as follows: frst, map the nput vectors nto one feature space, possble n hgher space, ether lnearly or nonlnearly, whch s relevant wth the kernel functon. Then, wthn the feature space from the frst step, seek and optmzed lnear dvson, that s, construct a hyper-plane whch separates two classes. It can be extended to mult-class. SVMs tranng always seek a global optmzed soluton and avod over-fttng, so t has ablty to deal wth a large number of feature. A complete descrpton about SVMs s avalable n. In the lnear separable case, there exsts a separatng hyper-plane whose functon s w x + b = whch mples y ( w x + b = ) 1, = 1,..., N By mnmzng w subject to ths constrant, the SVM approach tres to fnd a unque separatng hyper-plane. Here w s the Eucldean norm of w, and the dstance between the hyper-plane and the nearest data ponts of each class s 2 / w. By ntroducng Lagrange multplers α, the tranng procedure amounts to solvng a convex quadratc problem. The soluton s a unque globally optmzed result, whch has the followng propertes N w = α y x Only f correspondng a >, these x are called support vectors. When SVMs are traned, the decson functon can be wrtten as

3 Proceedngs of the 7th WSEAS Int. Conf. on Sgnal Processng, Computatonal Geometry & Artfcal Vson, Athens, Greece, August 24-26, f ( x) = sgn( α y ( x x ) + b) For a lnear non-separable case, SVMs perform a nonlnear mappng of the nput vector x from the d nput space R nto determned by Kernel functon. Two typcal kernel functons are lsted as follows: d Polynomal: k( x, y) = (( x y) + coef ), d N. 2 x y RBF: k x, y) = exp{ }. ( 2 Accordng to the dfferent classfcaton problems, the dfferent kernel functon can be selected to obtan the optmal classfcaton results. The dscusson above deals wth bnary classfcaton where the class labels can take only two values: 1 and -1. In the real world problem, however, mult-class classfcaton strategy s needed. The earlest used mplementaton for SVM mult-class classfcaton s one-aganst-all (OAA) methods. It constructs k SVM models where k s the number of classes. The th SVM s traned wth all of examples n the th class wth postve labels, and all the other examples wth negatve labels. Another major method s called one-aganst-one (OAO) method. Ths method constructs k( k 1) / 2 classfers where each one s traned on data from two classes. 3 Feature subset selecton based on ACO and SVM 3.1Overvew of the approach The orgnal data set S contanng N number of features s reduced to dfferent subsets s1, s2, s3,... each havng n1, n2, n3,... number of features respectvely usng ant colony optmzaton. These subsets wll be then fed to a desgned mult-class SVM, such that the classfcaton accuracy s maxmzed. The feature subset selecton representaton exploted by artfcal ants ncludes the followng: (1) Each ant ntalzes n the frst step wll select a subset of n features from the orgnal set of N features. We try three alternatves for the value of n. ()the value of n starts from a mnmum value, and the features are added successvely tll n reaches the maxmum at a constant rate. The startng and the stoppng value of n, and the ncrement rate are all defned by the user dependng on the specfc classfcaton problem. ()the value of n starts from the mnmum value, and then features are removed, tll n reaches the maxmum at a constant rate. () Each ant randomly chooses a feature subset of m features where m s between the upper and lower lmt of value σ of n. Specfcally, for nstance a problem havng 3 features shall have a mnmum and maxmum value of n as 5 and 28. We call these methods descrbed above as () from maxmum value to mnmum value (MaxMn), () from mnmum value to maxmum value (MnMax), () random value (Random). The flow chart of the method changes a lttle wth dfferent feature number ntalzng alternatves as descrbed n Fg.1. (2) A number of artfcal ants and to search through the feature space. The number of ants ntalzed depends upon the number of features of the gven problem. Ths wll be further explaned by means of smulatons on specfc dataset. (3) τ (, j) the ntensty of pheromone tral assocate wth feature f and f j whch reflect the prevous knowledge about the mportance of whle the subset has contaned the feature f. The steps of each teraton of ACO are descrbed as follows: Step 1: the ants develop soluton consstng of n number of features each based on a probablty dstrbuton rule. Step 2: the r ants construct r dfferent solutons, each contanng a subset of n dfferent features. SVM evaluates each subset by determnng the error n predcaton of some data usng that of n features. Step 3: Once all ants have completed constructng ther subsets, a global updatng rule s appled to the soluton set whch produces the least classfcaton error. Step 4: a local pheromone updatng rule s appled to the rest of the ants. The above steps are repeated dfferently accordng to dfferent feature number ntalzed method descrbed above. The whole flow chart of the approach s showed n Fg Soluton constructon of ACO For the ACO, every feature s treated as a locaton to be vsted by the ants. For every ant, t starts n a vrtual feature locaton (such that the ant can select all features equally), and then t bulds soluton by applyng a probablstc decson polcy to move through adjacent states. In ths case each subset of feature represents a state. An ant k located n feature f selects the next feature locaton f as follows: β arg max{ τ u[ ηu ] } u J k ( ) β j = τ j[ η j ] β u τ J u [ ηu ] k ( ) f q q otherwse j f j (1)

4 Proceedngs of the 7th WSEAS Int. Conf. on Sgnal Processng, Computatonal Geometry & Artfcal Vson, Athens, Greece, August 24-26, Intalze the algorthm Intalze the algorthm For features=start to fnsh For generaton=1 to max For generaton=1 to max ACO soluton constructon wth the No. of features=(features) ACO soluton constructon wth the No. of features=random number between maxmum and mnmum value Recordng the local best subset of feature Recordng the global best subset of feature Recordng the global best subset of feature Predctng the output for the global subset of features usng SVM Predctng the output for the global subset of features usng SVM End End a. Flow chart of the method usng MaxMn/MnMax b. Flow chart of the method usng Random Fg.1. Flow chart of the proposed method For a partcular ant k, η u ( η u =1 n ths paper) s a heurstc assocated wth the feature f and u J k s the set of features, whch are not a part of the soluton set, developed by ant k. β s a parameter, whch determnes the relatve mportance of pheromone versus heurstc. q s a random number unformly dstrbuted n between [ 1]. The parameter q, explotaton probablty factor, determnes the relatve mportance of explotaton versus exploraton. In explotaton, ants select those features whch have a maxmum of the product descrbed n Equ.1, where as n based exploraton the probablty of each feature to be selected by ants corresponds to the value of the above-mentoned product. Ths helps the ants to keep explorng new states whch are close to the optmal soluton. 1.3 Pheromone update rule of ACO As descrbed above, when all ants construct ther solutons, and every subset s assocated wth a classfcaton error. Then, the pheromone trals of the best ant tour (the best feature subset) s updated as τ 1 ρ) τ + ρσ, f,f S bs, j j ( j j bs Where S s the best feature subset, σ = Q / Err s bs, Errs s the classfcaton error assocated wth the subset s fed nto the SVM, Q and ρ are parameters. The purpose of the global updatng rule s to encourage the ants to produce subset wth least classfcaton error. Global updatng rule s only appled to that subset of feature, whch has produced the lease error n the current teraton. Besdes, the pheromone levels of all features pars of other subsets are adjusted accordng to the followng formula: τ 1 κ) τ +, f,f S k, j j ( j κτ j Where < κ < 1s a parameter and τ s the ntal pheromone level at the begnnng of the problem. 4 Expermental tests and results 4.1 Datasets used In order to evaluate the approach dscussed n the prevous sectons, datasets (WDBC, Image Segment, Dermatology, Hypothyrod and Wne) from UCI [1] were tested. Detal nformaton about the datasets s avalable at ts webste. The SVM classfer was realzed based on the SVM-Lb [11]. 4.2 Parameters settng (1) Number of ants The selecton of the rght number of ants s a very crtcal ssue affectng the performance of the algorthm. The number of ants must be suffcent to explore all potental states, whle expendng the least possble tme. Each ant ntalzed the features number randomly as descrbed above (comparson of the three alternatves for ntal feature number wll be dscussed below) and the performance of the algorthm was tested usng 5, 1, 15, 2, 25 ants. Other basc parameters of ACO are: q =.4, Q =.1, ρ =.8, κ =. 9.The ACO stopped when a user-defned soluton number s max =3 was reached. For the gven data sets, t was observed that the algorthm gave best results for 15 ants wthn a specfed tme.

5 Proceedngs of the 7th WSEAS Int. Conf. on Sgnal Processng, Computatonal Geometry & Artfcal Vson, Athens, Greece, August 24-26, Error n % Feature number Effect of number of ants Error n % Feature number Number of ants Fg.2. Effect of the number of ants on performance for WDBA (2) Explotaton probablty factor q The parameter q determnes the relatve mportance of explotaton versus exploraton. By settng.8 < q < 1, t can speed up the convergence of the algorthm, but also makes the algorthm fall nto local optmal pont easly. Fg. 3 shows that the gven datasets, the algorthm performance s better when q <. 6. Other parameters of ACO were set as follows: 15 ants, Q =.1, ρ =.8, κ =. 9. Error n % NO.of features The effect of parameter q Error n % NO. of features q Fg.3. Effect of parameter q on the performance for WDBA 4.3 Number of features ntalzng methods Three methods for ntalzng the number of features are presented above. For the MaxMn and MnMax methods, t s hard to determne the number of generatons and the addng/removng rate. What s more, two other nterestng topcs should be focused: (1) search effcency. Suppose the number of features of the best subset s n for a gven problem, then the algorthm can have the chance to get the optmal subset when the ntal number of the features s set to n. As dscussed above, the MaxMn/MnMax method adds/removes the feature at a user-defned rate, thus when the number of features ntalzed s smaller/bgger than n, the left search seems to be useless cause the search space doesn t contan the optmal soluton. () The Random method has stronger robustness. The state s transted accordng to the pheromone level between the feature locatons n ACO, so to great extent the current best soluton s determned by the knowledge exploted by the ants (pheromone levels). For a specfc problem, the number of features of the best subset, n may be close to the maxmum/mnmum value, whch makes the exploted knowledge lmted when the ntal number of features s n. Table 1 shows the comparson of the performance usng the MnMax and Random methods wth dfferent maxmum and mnmum value. Fg.4 shows that the algorthm wth Random method performed better than the algorthms wth the other two methods. Table 1 Comparson of the performance for WDBA usng the MnMax and Random No. of features /Classfcaton accuracy n % Mnmum-Maxmum MnMax Random /95% 9/97% /95% 8/96% 4.4 Expermental results The results obtaned are presented n Table 2. Feature subset selecton usng the proposed approach can mprove the performance of the patter classfer snce feature selecton s not only concerned wth reducng the number of features but also elmnatng the varables that produce nose or, are correlated wth other already selected varables. The basc parameters of the ACO are set as follows: 15 ants, q =.4, Q =.1, ρ =.8, κ =. 9. Error n % No. of features 1 Random 5 MnMax MaxMn No. of solutons 2 Random 1 MnMax MaxMn No. of solutons Fg.4. Comparson of performance for WDBA usng dfferent No. of features ntalzng methods

6 Proceedngs of the 7th WSEAS Int. Conf. on Sgnal Processng, Computatonal Geometry & Artfcal Vson, Athens, Greece, August 24-26, Table 2 Results of the SVM predcton usng the reduced subset and the set of complete features Dataset Set sze Tranng/Test No. of features Reduced subset Accuracy(all features) Accuracy(reduced features) Subset WDBC 3/ % 96.% 1,2,6,8,11,18,21,27 Image Seg 21/ % 79.57% 3-5,8,12,16,18,19 Dermatology 266/ % 95.% 4,5,8,13,15,2,31,32 Hypothyrod 3/ % 96.% 1,3,8,9,11,12,14,15,17,21,22 Wne 12/ % 94.83% 1,2,3,7,9, Fg.5. Graph of the actual output aganst the SVM predcted output usng the reduces subset for Wne dataset 5 Conclusons In ths paper, we presented a novel feature selecton method based on ant colony optmzaton and support vector machne. The smulaton results show that the method offers an attractve approach to solve the feature subset selecton problem. In addton, comparson of number of features ntalzng methods s dscussed by means of expermental smulatons. The potental future work ncludes developng and testng performance of the method for very large state spaces. Acknowledgments Ths project s supported by the Natonal Natural Scence Foundaton of Chna (NSFC No , ). References [1] John G., Kohav, R., Phleger K.: Irrelevant features and the feature subset problem[c]. In: Proceedngs of the 11 th Internatonal Conference on Machne Learnng, Morgan Kaufmann(1994): [2] Dash M., Lu H. Feature selecton for classfcaton[j]. Intellgent Data Analyss, An Internatonal Journal, Elsever, 1997, 1(3). [3] QIAO Lyan, PENG Xyuan, PENG Yu. BPSO-SVM wrapper for feature subset selecton[j]. ACTA ELECTRONICA SINICA, 26, 34(3): [4] Koller D., Saham M. Toward optmal feature selecton[c]. In: Proceedngs of Internatonal Conference on Machne Learnng, [5] WU Zhfeng, CHEN Dongxa.An algorthm of feature subset selecton based on genetc algorthm[j]. Journal of the Hebe Academy of Scences, 26,23(2):48-5. [6] Blum A.L.,P. Langley. Selecton of relevant features and examples n machne learnng[j]. Artfcal Intellgence, 1997, 97: [7] Dorgo M., Manezzo V.,and Colorn A. Ant system: Optmzaton by a colony of cooperatng agents[j]. IEEE Transactons on Systems, Man and Cybernetcs-Part B, 1996, 26: [8] WU Qd, WANG Le. Intellgent Ant colony algorthm and ts applcaton[m]. Shangha: Shangha scentfc and Technologcal Educaton Publshng, 24. [9] Cotes C, Vapnk V. Support vector networks [J]. Machne Learnng,1995, 2 (3) : [1]Blake C. L., Merz C.J. UCI repostory of machne learnng database, [11]Chang Chh-Chung, Ln Chh-Jen, LIBSVM: a lbrary for support vector machnes, 21.Software avalable at