A Powerful Feature Selection approach based on Mutual Information

Transcription

1 6 IJCN Internatonal Journal of Computer cence and Network ecurty, VOL.8 No.4, Aprl 008 A Powerful Feature electon approach based on Mutual Informaton Al El Akad, Abdelall El Ouardgh, and Drss Aboutadne GCM-LRIT, Faculty of cences, Mohammed V Unversty, Rabat, Morocco MCE, GCM-LRIT, Faculty of Economc cences, Hassan I Unversty, ettat, Morocco GCM-LRIT, Faculty of cences, Mohammed V Unversty, Rabat, Morocco ummary Feature selecton ams to reduce the dmensonalty of patterns for classfcatory analyss by selectng the most nformatve nstead of rrelevant and/or redundant features. In ths paper we propose a novel feature selecton measure based on mutual nformaton and takes nto consderaton the nteracton between features. The proposed measure s used to determne relevant features from the orgnal feature set for a pattern recognton problem. We use a upport Vector Machne (VM classfer to compare the performance of our measure wth recently proposed nformaton theoretc crtera. Very good performances are obtaned when applyng ths method on handwrtten dgtal recognton data. Key words: Feature selecton, Feature nteracton, Mutual Informaton, Interacton Gan, Handwrtten Dgt recognton. Introducton Feature selecton s a very mportant step n classfcaton snce the ncluson of rrelevant and redundant features often degrade the performance of a classfcaton algorthm both n speed and predcton accuracy. In the case of a pattern recognton problem, the obectve of feature selecton s to fnd the smallest subset of features that maxmzes the pattern recognton ablty. Ideally, ths can be acheved by examnng all possble subsets and fndng the one that satsfes the above crteron. Ths approach s known as exhaustve feature selecton. Even wth a moderate number of features, the exhaustve selecton s mpractcal because of ts computatonal requrements. Other feature selecton methods were developed to reduce computatonal complexty by compromsng performance. All feature selecton methods need to use an evaluaton functon together wth a search procedure to obtan the optmal feature set. The evaluaton functon measures how good a specfc subset can be n dscrmnatng between classes and can be dvded nto two man groups: flters and wrappers. Flters measure the relevance of feature subsets ndependently of any classfer, whereas wrappers use the classfer s performance as the evaluaton measure. earch procedures on the other hand, are methods that only consder small porton of all possble subsets. In ths paper, our obectve s to develop an evaluaton functon that can be used wth any search procedure. We wll consder flter evaluaton measures because they are faster than wrapper and can handle large datasets []. A varety of flter-based measures have already been proposed n the lterature. The most popular fall under the followng three categores: dstance measures, consstency measures and nformaton measures. Ths paper wll focus on the nformaton measure that s based on the concept of mutual nformaton. The drawback of the recently proposed nformaton measures s that they don t take nto consderaton the nteracton between features. Indeed, a sngle feature can be consdered rrelevant based on ts correlaton wth the class; but when combned wth other features, t becomes very relevant. Unntentonal removal of these features can result n a loss of useful nformaton and thus may cause poor classfcaton performance. Ths s studed n [3] as attrbute nteracton. We wll propose a new nformaton based evaluaton functon called IGF (Interacton Gan for Feature electon that overcomes the drawbacks of the other functons. Our proposed method wll be used to determne the relevant features from the orgnal feature set for a pattern recognton problem and wll be compared wth three recently proposed nformaton theoretc crtera. The rest of the paper s organzed as the follows: ome nformaton theoretc notons for feature selecton and the state of the art about the recently proposed nformaton theoretc crtera for feature selecton are addressed n the secton two. ecton three presents our proposed evaluaton functon based on Interacton Gan (IGF. Expermental results on handwrtten dgtal recognton data and comparson n term of classfcaton accuracy between our proposed method and three recently proposed nformaton theoretc crtera s presented n secton four. The last secton summarzes the fndng and gves some perspectves that can follow up on ths work. Manuscrpt receved Aprl 8, 008. Manuscrpt revsed Aprl 0, 008.

2 IJCN Internatonal Journal of Computer cence and Network ecurty, VOL.8 No.4, Aprl Informaton theoretc for Feature electon. Defntons and measurements ( Mutual nformaton and condtonal mutual nformaton: The frst goal of a predcton model s to mnmze the uncertanty on the dependent varable. A good formalzaton of the uncertanty of a random varable s gven by hannon and Weaver s [4] nformaton theory. Whle frst developed for bnary varables, t has been extended to contnuous varables. Let and Y be two random varables (they can have real or vector values. We denote μ the ont probablty densty functon of, Y and Y. We recall that the margnal densty functons are gven by: ( x = μ, Y y dy ( y = μ, Y y dx μ ( μ ( Y Let us now recall some elements of nformaton theory. The uncertanty on Y s gven by ts entropy defned as: μ H( = ( ylog μ ( y dy (3 Y Y If we get knowledge on Y ndrectly by knowng, the resultng uncertanty on Y knowng s gven by ts condtonal entropy, that s: H( Y / = μ ( x μy ( y / = xlog μy ( y / = x dydx (4 The ont uncertanty of the (, Y par s gven by the ont entropy, defned as: μ H, = y log y dxdy (5 (, Y μ, Y The mutual nformaton between and Y can be consdered as a measure of the amount of knowledge on Y provded by (or conversely on the amount of knowledge on provded by Y. Therefore, t can be defned as [5]: I( ; = H( H( Y / (6 Whch s exactly the reducton of the uncertanty of Y when s known. If Y s the dependant varable n a predcton context, the mutual nformaton s thus partcularly suted to measure the pertnence of n a model for Y [6]. Usng the propertes of the entropy, the mutual nformaton can be rewrtten nto: I( ; = H( + H( H(, (7 That s, accordng to the prevously recalled defntons, nto [7]: μ, Y y I( ; = μ Y y log dxdy (8, μ ( x μ ( y The condtonal mutual nformaton s defned as: I( ; Y / = H ( / H ( / Y, (9 = I / Y, I( / ( Y Ths value quantfes how much nformaton s shared between and Y, gven the value of. Another way to see t, as t s decomposed above, s as the dfference between the nformaton requred to descrbe gven, and the nformaton to descrbe gven both and Y. If Y and carry the same nformaton about, the two terms on the rght are equal, and the condtonal mutual nformaton s zero. On the opposte, f both Y and brng nformaton, and f those nformatons are complementary, the dfference s large. ( Feature nteracton and Interacton Gan: Feature selecton s one effectve mean to remove rrelevant features [8]. Optmal feature selecton requres an exponentally large search space ( O ( * N, where N s the number of features [9]. Researchers often resort to varous approxmatons to determne relevant features (e.g., relevance s determned by correlaton between ndvdual features and the class [0], []. However, a sngle feature can be consdered rrelevant based on ts correlaton wth the class; but when combned wth other features, t becomes very relevant. An llustraton of feature nteracton s gven by the well-known OR problem [], [3]: We see that and have null mutual nformaton wth the output, once they are taken ndvdually (.e I ( ; = 0, I ( ; = 0. However, when they are taken together, the mutual nformaton Y

3 8 IJCN Internatonal Journal of Computer cence and Network ecurty, VOL.8 No.4, Aprl 008 I (, ; = H( > 0 of the subset s postve. Interacton explans why an apparently rrelevant combnaton of varables can eventually perform effcently n a learnng task. To decde, whether there s nteracton between two attrbutes, [4] propose an heurstc test, called nteracton gan. It s based on the well-known dea of nformaton gan. Informaton gan can be regarded as a measure of the strength of a -way nteracton between an attrbute and the class Y. In ths sprt, we can generalze t to 3-way nteractons by ntroducng the nteracton gan [4]: I( ; ; = I(, ; I( ; I( ; (0 Interacton gan can be understood as the dfference between the actual decrease n entropy acheved by the ont attrbute and the expected decrease n entropy wth the assumpton of ndependence between attrbutes and. The hgher the nteracton gan, the more nformaton was ganed by onng the attrbutes n the Cartesan product, n comparson wth the nformaton ganed from sngle attrbutes. It s qute easy to see that when nteracton gan s negatve, context decreased the amount of dependence. When the nteracton gan s postve, context ncreased the amount of dependence. When the nteracton gan s zero, context dd not affect the dependence between the two attrbutes. Interacton gan s dentcal to the noton of nteracton nformaton [3] and mutual nformaton among three random varables [4], [5]. In the followng secton, we wll proceed to a crtcal survey of nformaton theoretc approaches exstng n lterature, by stressng when and where the noton of nteracton s taken nto account.. tate of the Art As mutual nformaton can measure relevance, ths quantty s currently used n lterature for performng feature selecton. One of the man reasons for adoptng t s ts low complexty computatonal complexty cost ( O ( d * N where d s the number of varables and N s the number of samples n the case of dscrete varables. The followng sectons wll sketch three state-of-the-art flter approaches that use ths quantty. ( Varable Rankng (RANK: The rankng method returns a rankng of varables on the bass of ther ndvdual mutual nformaton wth the output. Ths means that, gven n nput varables, the method frst computes n tmes the quantty I( ;, = Kn, then ranks the varables accordng to ths quantty and eventually dscards the least relevant ones [6]. The man advantage of the method s ts rapdty of executon. Indeed, only n computatons of mutual nformaton are requred for a resultng complexty ( O ( n* * N. The man drawback derves from the fact that possble redundances between varables s not taken nto account. Indeed, two redundant varables, yet hghly relevant taken ndvdually, wll be both well ranked. As a result, a model that uses these two varables s dangerously prone to an ncreased varance wthout any gan n terms of bas reducton. On the contrary, two varables can be complementary to the output (.e. hghly relevant together whle each of them appears to be poorly relevant once taken ndvdually. As a consequence, these varables could be badly ranked, or worse elmnated, by the rankng flter. Although the varable rankng algorthm s reputed to be fast, t may be poorly effcent as t only reles on ndvdual relevance. Recently, new algorthms that combne relevance and redundancy analyss offer a good compromse between accuracy and computatonal load as the Fast Correlaton Based Flter [6]. Also, some heurstc search methods such as the best frst search (also known as the forward selecton can be combned effcently wth nformaton theoretc crtera n order to select the best varable gven a prevously selected subset. In the next sectons, two theoretc crtera exstng n the lterature and that can be easly combned wth the forward selecton, are presented. ( Mnmum Redundancy - Maxmum Relevance crteron (MRMR: The mnmum redundancy - maxmum relevance crteron [7] conssts n selectng the varable among the not yet selected features that maxmzes u z where u s a relevance term and z s a redundancy term. More precsely, u s the relevance of to the output Y alone, and z s the mean redundancy of to each varable already selected. u = I( ; ( z = I( ; d ( MRMR = arg max ( u z (3 At each step, ths method selects the varable whch has the best compromse relevance-redundancy. Ths selecton crteron s fast and effcent. At step d of the forward search, the search algorthm computes n d evaluatons

4 IJCN Internatonal Journal of Computer cence and Network ecurty, VOL.8 No.4, Aprl where each evaluaton requres the estmaton of d + b-varate denstes (one for each already selected varables plus one wth the output. It has been shown n [7] that the MRMR crteron s an optmal frst order approxmaton of the condtonal relevance crteron. Furthermore, MRMR avods the estmaton of multvarate denstes by usng multple b-varate denstes. Note that, although the method ams to address the ssue of redundancy between varables through the term z, t s not able to take nto account the nteractons between varables. ( Condtonal Mutual Informaton Maxmzaton Crteron (CMIM: Ths approach [8] proposes to select the feature whose mnmal condtonal relevance I ( ; Y / among the selected features, s maxmal. Ths requres the computaton of the mutual nformaton of and the output Y, condtonal on each feature prevously selected. Then, the mnmal value s retaned and the feature that has a maxmal mnmal condtonal relevance s selected. The varable returned accordng to the CMIM crteron s: CMIM = arg max ( mn ( I( ; Y / (4 Ths selecton crteron s powerful. It selects relevant varables, t avods redundancy, t avods estmatng hgh dmensonal multvarate denstes and unlke the prevous method, t does not gnore varable nteracton. However, t wll not necessary select an nteractng varable wth the already selected varables. Indeed, a varable that has a hgh nteracton wth an already selected varable wll be characterzed by a hgh condtonal mutual nformaton wth that varable but not necessarly by a hgh mnmal condtonal nformaton. In terms of complexty, note that at th the d step of the forward search, the algorthm computes n d evaluatons where each evaluaton followng CMIM requres the estmaton of d tr-varate denstes (one for each prevously selected varable. 3. Interacton Gan Based Feature electon (IGF The new proposed evaluaton measure for a gven feature wll be based on the ndvdual Mutual Informaton and a compromse between features redundancy and features nteracton. The compromse s made by the mean of Interacton Gan. In formal notaton, the varable returned accordng to the IGF crteron s: IGF = arg max ( I( ; + I( ; ; (5 d The man advantage n usng ths crteron for selectng varables s that an nteractng varable of an already selected one has a much hgher probablty to be selected than wth other crtera. The relevance of each feature can be ndcated by ts Mutual Informaton wth class labels I( ;. The second term makes a compromse between redundancy and nteracton. A negatve Interacton Gan ndcates that the features are redundant and a postve one ndcates that the features work well together. 4. Expermental Results In ths secton, we perform comprehensve experments on handwrtten dgtal recognton dataset to compare the IGF selecton algorthm wth the three state of the art approaches dscussed above: The Rankng algorthm, the Mnmum Redundancy Maxmum Relevance crteron and the Condtonal Mutual Informaton Maxmzaton crteron. 4. Dataset Descrpton We have used the dataset of handwrtten numeral recognton from UCI Machne Learnng Repostory [9]. It conssts of 649 features on handwrtten numerals ( 0 9. These 649 features dstrbute over the followng feature sets: 76 Fourer coeffcents of the character shapes, 6 profle correlatons, 64 Karhunen-Love coeffcents, 40 pxel averages n x3 wndows, 47 Zernke moments, 6 morphologcal features. There are 00 patterns per class (for a total of,000 patterns. 4. Classfer Descrpton VM (upport Vector Machne s a relatvely new and promsng classfcaton method [0]. It s a margn classfer that draws an optmal hyper-plane n the feature vector space; ths defnes a boundary that maxmzes the margn between data samples n two classes, therefore leadng to good generalzaton propertes. A key factor n VM s to use kernels to construct nonlnear decson boundary. In ths expermentaton, we used the Weka [] verson of LIBVM [] whch allow us to drectly construct a multclass VM wth exponental kernel. 4.3 Assessment measure We assessed classfcaton performance usng K-fold cross valdaton. In ths assessment method the orgnal sample s parttoned nto k sub-samples. Of the k sub-samples, a

5 0 IJCN Internatonal Journal of Computer cence and Network ecurty, VOL.8 No.4, Aprl 008 sngle sub-sample s retaned as the valdaton data for testng the model, and the remanng k sub-samples are used as tranng data. The cross-valdaton process s then repeated k tmes (the folds, wth each of the k sub-samples used exactly once as the valdaton data. The k results from the folds then can be averaged (or otherwse combned to produce a sngle estmaton. Cross valdaton accuracy provdes more realstc assessment of classfers whch generalze well to unseen data. We used 0-fold cross valdaton [3], [4]. 4.4 Results Each selecton method stops after those thrty varables have been selected. Then, the evaluaton of the selecton s done by usng a 0-fold cross valdaton wth a VM learnng algorthm. The accuracy of classfcaton (recognton rate relatvely to the step by step ntroducton of the varables s computed and the evoluton of the recognton rate usng dfferent feature selecton algorthm s reported n the fg.. Fg. Evoluton of the 0-fold cross valdaton accuracy of the VM learnng algorthm The above graph show the strength of our proposed measure compared wth the three well known feature selecton algorthms. In addton IGF s better than the other algorthm by at least % of the recognton rate. The analyss of ths graph allowed us to take out the followng results: The measures based on the mutual nformaton can be used for performng feature selecton for the problem of pattern recognton; The analyss of the nteracton between features must be taken nto consderaton when selectng features for pattern recognton. 4. Concluson and future work In ths paper, we proposed a new evaluaton functon, called IGF, based on the concept of mutual nformaton and nteracton gan. The functon takes nto consderaton the nteracton between features. When the functon was used wth the stepwse selecton procedure n the problem of pattern recognton, t mproves classfcaton accuracy wth a lesser number of features compared to the other methods. The man advantage of the proposal measure s that t takes nto account dfferent features nteracton wthout ncreasng the computatonal complexty. Further experments wll focus on other pattern recognton problems. Moreover, other search strateges than the forward selecton n order to valdate the crteron n a wder range of domans. References [] Dash, M., Lu, H. Feature selecton for classfcaton. Intellgent Data Analyss pp 3-56 (997. [] Isabelle Guyon Andr Elsseeff, An Introducton to Varable and Feature electon Journal of Machne Learnng Research 3 pp 57-8 (003. [3] Aleks Jakuln and Ivan Bratko. Analyzng attrbute dependences. In PKDD, 003. [4] C.E. hannon, W. Weaver, The Mathematcal Theory of Communcaton, Unversty of Illnos Press, Urbana, IL, 949. [5] T.M. Cover, J.A. Thomas, Elements of Informaton Theory,Wley, New York, 99. [6] Yu, L., Lu, H.: Effcent feature selecton va analyss of relevance and redundancy. Journal of Machne Learnng Research 5 ( [7] C.H. Chen, tatstcal Pattern Recognton, partan Books, Washngton, DC, 973. [8] A. L. Blum and P. Langley. electon of relevant features and examples n machne learnng. Artfcal Intellgence, 97:457, 997. [9] H. Almuallm and T. G. Detterch. Learnng boolean concepts n the presence of many rrelevant features. Artfcal Intellgence, 69(- :79305, 994. [0] M. A. Hall. Correlaton-based feature selecton for dscrete and numerc class machne learnng. In ICML, 000. [] L. Yu and H. Lu. Feature selecton for hghdmensonal data: a fast correlaton-based flter soluton. In ICML, 003. [3] Kohav, R., John, G.H.: Wrappers for feature subset selecton. Artfcal Intellgence 97(- ( [3] McGll, W.J.: Multvarate nformaton transmsson. Psychometrka 9 ( [4] Jakuln, A.: Attrbute nteractons n machne learnng. Master s thess, Unversty of Lublana, Faculty of Computer and Informaton cence (003. [5] Yeung, R.W.: A new outlook on hannon s nformaton measures. IEEE Transactons on Informaton Theory 37 ( [6] Duch, W., Wnarsk, T., Besada, J., Kachel, A.: Feature selecton and rankng flters. In: Internatonal Conference on Artfcal Neural Networks (ICANN and Internatonal

6 IJCN Internatonal Journal of Computer cence and Network ecurty, VOL.8 No.4, Aprl 008 Conference on Neural Informaton Processng (ICONIP. ( [7] Peng, H., Long, F.: An effcent max-dependency algorthm for gene selecton. In: 36th ymposum on the Interface: Computatonal Bology and Bonformatcs. (004. [8] Fleuret, F.: Fast bnary feature selecton wth condtonal mutual nformaton. Journal of Machne Learnng Research 5 ( [9] C.L. Blake and C.J. Merz. UCI repostory of machne learnng databases, 998. [0] Vapnk V, The Nature of tatstcal Learnng Theory, New York: prnger, 995. [] Ian H. Wtten and Ebe Frank. Data mnng:practcal machne learnng tools and technques wth Java mplementatons. Morgan Kaufman, an Francsco, CA, UA, [] Yasser EL-Manzalawy (005. WLVM. URLhttp:// yasser/wlsvm/. [3] Moore, A.W. and Lee, M.., Effcent algorthms for mnmzng cross valdaton error. In: Proceedngs of Eleventh Internatonal Conference on Machne Learnng, Morgan Kaufmann, New Brunswck, New Jersey, 90-98, (994. [4] Thomas G. Detterch Approxmate tatstcal Tests for Comparng upervsed Classfcaton Learnng Algorthms Neural Computaton 0, (998. [5] T. Mtchell, Machne Learnng McGraw-Hll (997.