Feature Selection Method Based on Adaptive Relief Algorithm

Transcription

1 2010 3rd Internatonal Conference on Computer and Electrcal Engneerng (ICCEE 2010) IPCSIT vol. 53 (2012) (2012) IACSIT Press, Sngapore DOI: /IPCSIT.2012.V53.No.2.10 Feature Selecton Method Based on Adaptve Relef Algorthm Fan Wenbng +, Wang Quanquan and Zhu Hu Zhengzhou Unversty Zhengzhou, Chna Abstract It s a very sgnfcant task that how to select nformatve features from the feature space for pattern recognton. Relef s consdered one of the most successful algorthms for evaluatng the qualty of features. In ths paper, we provde a vald proof for the frst tme, whch demonstrates a blnd selecton problem n the prevous Relef algorthm. Then we propose an adaptve Relef (A-Relef) algorthm to allevate the defcences of Relef by dvdng the nstance set adaptvely. Last, several experments are reported by A-Relef and other feature selecton methods. The expermental results llustrate the effcency of A-Relef algorthm proposed n ths paper. Keywords-Relef algorthm; feature selecton; pattern recognton; 1. Introducton Feature selecton s one of the mportant aspects of pattern recognton. A better feature selecton algorthm, whch elmnates the redundant feature effectvely n feature space, can fnd a feature subset whch s most relevant to models n current applcaton. Not only can ts proper desgn reduce system complexty, but t can also decrease processng tme. Feature selecton s wdely used n mage processng, feature reducton and machne learnng as well as artfcal ntellgence, and t plays a crtcal role n many other cases. Wth lmted tranng samples, selectng useful features for these knds of problems poses a serous challenge to the exstng feature selecton algorthms. Among the extant feature selecton algorthms, the Relef algorthm s consdered one of the most successful ones due to ts smplcty and effectveness. Relef algorthm was frst proposed n [1]. The key dea of Relef s to teratvely estmate feature weghts accordng to ther ablty to dscrmnate between neghborng models. Then, n [2] Relef was extended to handle nosy and mssng data and solve multclassfcaton ssues whch the orgnal Relef algorthm can not deal wth. Subsequently, n order to explore the framework of expectaton maxmzaton, Iteratve-Relef s put forward n [3]. Nevertheless, the defcency of blnd selecton was not dscovered n prevous research. Ths paper proposed a novel feature selecton algorthm called Adaptve Relef (termed A-Relef). Ths proposed algorthm can dvde the tranng set adaptvely accordng to the pecularty of these features. These features brng about blnd selecton when they are processng by former algorthms. Consequently, through handlng these features by A-Relef, the authentc connecton between features and models s reflected. It offers effectve nformaton for the further dentfcaton. 2. Scarcty of Relef Algorthm Relef algorthm assesses the correlaton between features and models by means of feature weght value, and yet n actual practce, there are stll shortcomngs n Relef algorthm, e.g., when all model types nvolved n the present problem are already defnte, certan features stll nclude a certan model type whch s not + Correspondng author. E-mal address: ewbfan@zzu.edu.cn

2 referred to n the current ssue. In ths case, these features, whch are straghtway substtuted n Relef, are accounted to have an ntmate relatonshp wth model types, regardless of whether they related to model types. Therefore, Relef sometmes performs blnd selecton, whch s not expected to occur. In the followng, we provde a thorough nterpretaton of blnd selecton, whch s never dscovered n the anteror research. The procedure of Relef algorthm s represented n Fg. 1 [1]. In each teraton, an nstance x s randomly selected and then two nearest neghbors of x are found, one from the same classfcaton (termed the nearest ht or NH) and the other from a dssmlar classfcaton (termed the nearest mss or NM). The weght of the th feature s then updated: w = w + x NM (x) - x NH (x) (1) N R= {(x n, y n)} n=, set w = 0, 1 I, number of teraton T ; t = T (1) Intalzaton: gven 1 (2) for 1: (3) Randomly select a nstance x from R ; (4) Fnd the nearest ht NH(x) and mss NM(x) of x; (5) for = 1: I (6) Compute: w = w + x NM (x) - x NH (x) (7) end (8) end Fg 1. Procedure of Relef algorthm The Relef algorthm was ncpently desgned to deal wth bnary problems. Afterward, Relef-F was proposed n [4] to dspose multclass problems by perfectng the weght update rule (lne 6 of Fg. 1) as: Pc () w = w + { x NM ( x) x NH ( x)} (2) c c ( c Y, c y( x)) 1 Pc ( ) where Y = {1,..., C} s the model type space, NM c () x s the nearest mss of x from class c, and Pc () s the a pror probablty of class c. From the above analyss, we fnd that Relef algorthm s a feature weghtng algorthm that utlzes the performance of a hghly nonlnear classfer n search for nformatve features. In order to convenently llustrate the major drawback of Relef, we provde two defntons as follows: Defnton 1: f feature η contans one or more model types, whch model type space does not nclude n the problem to be resolved, we desgnate η as bogus feature. Defnton 2: a sort of model type, whch does not exst n model type space n real applcaton and s represented by bogus feature, s denoted as connotatve classfcaton (termed CC). Compared wth other features, bogus features usually perform some specal partculartes whch can be summarzed as: Regardless of whether there s strong correlaton between bogus features and model type, terated by Relef algorthm, the weghts of bogus features acheve a larger value. Accordngly, the bogus feature s regarded as an nformatve feature whch has a remarkable correlaton wth model type. Adopted a feature subset comprsng the bogus feature, pattern recognton wll deterorate classfcaton performance. The descrpton of bogus feature s presented n Fg 2. Fg 2 reveals the dstrbuton of nstance set of feature η. It s the ultmate purpose that the nstance x can be accurately dstngushed between model class A and model class B. Consequently, model classc and model class D are unexpected model types whch need not be transacted n ths case (.e., class C and classd are CC), and then feature η possesses the dosyncrasy of bogus feature.

3 η class A x classb class C d k 2 NH(x) d k1 NM(x) class D We defne the margn for a nstance x as Fg 2. Dstrbuton of feature η λ = d(x NM(x )) d(x NH(x )) (3) n n n n n where NM(x) and NH(x) are the nearest mss and ht of nstance x, respectvely, and d() s the dstance functon. For a random nstance x, f x classa, then t s evdent that x classc or x classd ; n ths case, assumng x classc,.e., x classa classc, we have λ = dk1 dk2 > 0.If x classb, smlarly we can conclude that d k1 s larger than d k 2, thus λ > 0. Hence, under the crcumstance, for each nstance x, d(xn NM(x n)) d(xn NH(x n)) > 0 s establshed. In the teraton procedure, substtuted a postve value of λ n nto (1), the value of feature weght w n keeps on ncreasng. Ultmately the feature η has been mstakenly consdered as an nformatve feature due to the larger value of w n, and obvously n Fg 2, there s few dscrepancy between class A and class B. Nevertheless, the bogus feature η, selected by Relef algorthm, partcpates n pattern recognton. It s reported that rrelevant features can deterorate classfcaton performance [5]. Therefore, blnd selecton n orgnal Relef algorthm s a grevous mstake n the feature selecton aspect, and then for classfcaton purposes, removng rrelevant features s necessary. 3. Adaptve Relef Algorthm Bogus features are unavodable n several applcatons such as DNA mcroarray. Some researchers have ndcated that the recognton of a small gene subclass wth good predctve ablty may not be suffcent to afford sgnfcant perspcacty nto the understandng and modelng of the connecton between certan dseases and genes [6]. In ths secton, A-Relef algorthm wll be proposed to solve the blnd selecton problem n real applcaton. Blnd selecton problems can not be easly settled through conventonal optmzaton technques. By compartmentalzng the nstance subset adaptvely accordng to the connotatve classfcaton presented n bogus features, the proposed algorthm mplements an onlne algorthm that solves the blnd selecton problem. Accordngly, to dscover bogus features s the major assgnment. Before gvng the descrpton of A-Relef algorthm we provde an nterpretaton of ths algorthm as follows: Accordng to the descrpton of bogus features n Defnton 1, the judgment fundament, whch dentfes bogus features, s whether the feature can contan the nformaton of connotatve classfcaton. In the proposed algorthm, each feature wll be nspected deeply to detect the bogus feature, before the feature s traned through A-Relef.

4 If the present feature s a bogus feature, the nstance subset wll be dvded by a threshold value ξ, whch can be acqured by experences or expert knowledge. For other features, they can be substtuted nto Relef algorthm straghtway. The procedure of A-Relef algorthm s presented n Fg. 3. Prompt: T number of teratons I number of features N number of nstances (obvously T N ) Th a vector of threshold value η l the lth feature Φ ( η l ) the set of the values of feature η l n all nstances. num a I N matrx, the th row of num (.e., num(,:) ) s put nto the ndex of a new subset, whch s dvded by Th. N (1) Intalzaton: gven R= {(x n, y n)} n= 1, set w = 0, 1 I ; (2) for l = 1: I (3) Compute ndexes: num(,:) l = { Φ ( ηl ) > Th()} l ; (4) Dvde R nto dfferent groups: Ml = { Ml1,..., MlC} (5) end (6) for j = 1: I (7) f mn_ ds( M j, M j* ) > 3.5 max_ ds( M j) (8) Confrm: η j s a bogus feature; (9) for k = 1: C (10) for t = 1: T (11) Randomly select an nstance x from M jk ; (12) Fnd the nearest ht NH(x) and mss NM( x ) of x n M jk ; j j j j (13) Compute: wj = wj + x NM (x) - x NH (x) (14) end (15) end (16) else for t = 1: T (17) Randomly select an nstance x from R ; (18) for = 1: I (19) Compute: w = w + x NM (x) - x NH (x) (20) end (21) end f (22) end Fg 3. Procedure of Adaptve Relef algorthm The weght update rule (step 13 and 19 of Fg. 3) can be modfed by (2) for the sake of handlng multclass problems. Obvously, compared wth the prevous Relef algorthm, the A-Relef algorthm adopts an approach that we dfferentate the nstances set by the connotatve classfcaton before tranng the feature. Then, for the bogus feature, the nearest ht NH( x ) and mss NM( x ) of x hal from not R but M whch can been obtaned by Th (step 4 n Fg. 3). M = { class class, class class, class class, class class } (4) C A C B C A C B In Fg. 2, based on the prevous analyss, wth traned by the orgnal Relef algorthm, the feature η was consdered one of optmzaton features. However, wth the transformaton of subset, whch nstance set was transformed from R nto M, η was regarded as an rrelevant feature owng to the dversfcaton of tranng nstance set by usng A-Relef. Ths s n accord wth the facts. In concluson A-Relef successvely performs onlne learnng and solves the blnd selecton problems conssted n the orgnal Relef algorthms.

5 4. Implementng and Analyss In order to verfy the effectveness and effcency of the Adaptve Relef for the blnd selecton problem, some mages of tran part are tested n ths paper. It s our ultmate goal to dentfy fault mages from all samples, whch are shown n Fg Calculatng mage features We extract mage features by the grey level co-occurrence matrx (termed GLCM), whch s mentoned n [7]. The way of calculatng the GLCM of mage I s shown n Fg. 5, and there are four formats of GLCM, as showed n Fg. 6. For every format of GLCM, we can acqure sx features as follows: contrast, dssmlarty, homogenety, entropy, energy, correlaton [8]. The abbrevaton of contrast0 n Table I means feature contrast s obtaned through the 0 o drecton GLCM n Fg. 6. Consequently, we should compute 24 features aggregately n ths case. In ths mage classfcaton, we performed three sets experments: 120 samples, 350 samples and 500 samples Results analyss The Relef, Relef-F, I-Relef and A-Relef clusterng approaches are appled on the three sets, respectvely. Then, through a nonlnear classfer [9], features, selected by the three methods, were separately used to dstngush fault mage from another set wthn 100 samples and 200 samples, n whch samples were rrelevant to samples n the prevous tranng sets. Accordng to the prncple of the A-Relef algorthm n Fg.3, the weght of feature, terated by the proposed algorthm, s the same as the weght traned by the orgnal algorthms n addton to the weght of a bogus feature. Table I shows results of weghts of feature contrast, whch s nconsstent. Besdes, feature contrast s the unque one n the feature space. We can make a generalzaton that the feature contrast s a bogus feature. On further analyss, we detected that the feature contrast nvolved a connotatve classfcaton--exposure dscrepancy. Owng to the complexty of shootng condton, t s neluctable. There are abundant mages exposed overly, as showed n the thrd pcture of Fg. 4. Then by means of a nonlnear classfer, usng the feature spaces chose respectvely by the three methods, we evaluate the performance of each algorthm. Table II demonstrates the expermental results by usng Relef, Relef-F, I-Relef and A-Relef. Through the comparson of the experment n Table II, we can see that the ameloratve feature selecton algorthm, A-Relef, mproved the accuracy of the classfcaton effectvely. However, due to the presence of a bogus feature contract n feature space acheved by usng Relef, Relef-F and I-Relef, ther recognton results are extraordnary poor. Fg 4. Some examples of tran mages 90 ο [1,0] 135 ο [1,1] 45 ο 0 ο [ 1,1] [0,1] Fg.5. The way of calculatng GLCM Fg.6. Four formats of GLCM

6 TABLE I. THE RESULT OF A BOGUS FEATURE TRAINED BY RELIEF, RELIEF-F, I-RELIEF AND A-RELIEF Features Algorthm Relef Relef-F I-Relef A-Relef Sample number Contrast Contrast Contrast Contrast TABLE II. CLASSIFICATION ACCURACY BY USING A-RELIEF AND THE PREVIOUS RELIEF ALGORITHM Algorthm Relef Relef-F I-Relef A-Relef Result Sample number rght wrong Classfcaton Accuracy 26% 23.5% 60% 59% 75% 71% 92% 93% 5. Concluson In ths paper, we present an exhaustve nterpretaton of the blnd selecton problem exsted n the orgnal Relef algorthm, and the mathematcal proof s provded. Then a novel feature selecton has been proposed. We have adopted a technque based on dfferentatng the nstances set adaptvely n the proposed Adaptve Relef algorthm. Fnally, we dealt wth the tran mages by usng the Adaptve Relef and the prevous Relef algorthm. Expermental results llustrate that the amendatory Adaptve Relef algorthm mproved the accuracy of the classfcaton effectvely and resolved the blnd selecton problem n the orgnal algorthm drastcally. How to decrease the complexty of ths algorthm wll be the further task of ths paper. 6. References [1] Kra K, Rendell L.A practcal approach to feature selecton[c].proc 9th Internatonal Workshop on Machne Learnng,1992: [2] Robnk Skonjam, Kononenko I.Theoretcal and Emprcal analyss of RelefF and RRelefF. Machne Learnng, 2003,53(1): [3] Sun Y Jun. Iteratve relef for feature weghtng algorthms, theores, and applcatons. IEEE Trans on Pattern Analyss and Machne Intellgence, 2007,29(6): [4] Kononenko I. Analyss and extensons of Relef. European Conference on Machne Learnng,1994: [5] R. Kohav and G. H. John. Wrappers for Feature Subset Selecton. Artfcal Intellgence, vol. 97, nos. 1-2, pp , [6] T. Jenssen and E. Hovg, Gene-Expresson Proflng n Breast Cancer. Lancet, vol. 365, pp , 200. [7] Gustavo Carnero, Allan Jepson. Flexble spatal confguraton of local mage features. IEEE Transactons on Pattern Analyss and Machne Intellgence,2007,29(12): [8] TATSUHIKO HIRUKAWA, SATOSHI KOMADA. Image feature based navgaton of Nonholonomc Moble Robots wth actve camera. SICE Annual Conference,2007: [9] R. Duda, P. Hart, and D. Stork, Pattern Classfcaton. J. Wley, 2000.