Fuzzy Rough Neural Network and Its Application to Feature Selection

Transcription

1 70 Internatonal Journal of Fuzzy Systems, Vol. 3, No. 4, December 0 Fuzzy Rough Neural Network and Its Applcaton to Feature Selecton Junyang Zhao and Zhl Zhang Abstract For the sake of measurng fuzzy uncertanty and rough uncertanty of real datasets, the fuzzy rough membershp functon (FRMF) defned n fuzzy rough set s ntroduced. A new fuzzy rough neural network (FRNN) s constructed based on neural network mplementaton of FRMF. FRNN has the merts of quck learnng and good classfcaton performance. And then a new neural network feature selecton algorthm based on FRNN s desgned. The nput nodes of FRNN are pruned accordng to the descent of classfcaton accuracy; thereby the search of optmal feature subset s realzed wth reference to resdual nput nodes. The test results on UCI datasets show that the algorthm s quck and effectve, and has better selecton precson and generalzaton capablty than RBF feature selecton. Keywords: Fuzzy rough membershp functon, Fuzzy rough neural network, Feature selecton, RBF neural network.. Introducton In the applcaton feld of pattern recognton, the mass features under analyss are often rrelevant or redundant. Therefore, the collected data n practce should be preprocessed to remove redundant features before utlzng these datasets. Feature selecton s one of the most mportant preprocessng methods n the research felds such as data mnng and pattern recognton. Feature selecton pcks a feature subset wth excellent capablty from the ntal hgh-dmensonal feature set based on certan evaluaton crteron, and ams to preserve or even mprove the classfcaton performance of the data set. Orgnally, feature selecton attracts not enough attenton, but wth the development of computer technology and the explosve of nformaton, feature selecton gets ncreasngly mportance to nformaton processng. Some famous feature selecton algorthms as B&B, Focus, Relef, LVF and CFS are proposed. Neural network adapts to deal wth nose data for ts Correspondng Author: Junyang Zhao s wth an Research Inst. Of H-tech Hongqng Town, an 7005, Chna. E-mal: zhaoy80@sna.com. Manuscrpt receved 6 Oct. 0; revsed 5 Dec. 0; accepted Dec. 0. merts of strong approachng ablty and good fault-tolerance performance. So the combnaton of neural network and feature selecton can further enhance the capablty of complex data processng. There are two knds of combnatve modes: one s preprocessng data frstly by feature selecton, and then nputtng the data to neural network for further research; the other s utlzng drectly the confguraton character and classfcaton performance of neural network to realze feature selecton. The artcle emphaszes partcularly on the second mode. Ths mode was ntally mplemented n Neural Network Feature Selector (NNFS) proposed by Setono and Lu n 997 [], whch realzed feature selecton by prunng a three-layer feedforward neural network to remove redundant network connectons. Subsequently, Basak [] descrbed a new feature selecton method by examnng the parameters of a traned radal bass functon network. The connecton weghts of hdden and output layers ndcate the relatve mportance of clusters and classes, and the ntraclass and nterclass dstances are calculated to evaluate feature subsets. Sankar K. Pal [3] proposed a neuro-fuzzy approach for unsupervsed feature evaluaton by tranng network weghts whch ndcate the mportance of correspondng features to rank these features n 000. A. Verkas [4] presents a new neural network tranng method wth an augmented cross-entropy error functon based on NNFS, and a new prunng approach for dentfyng salent features accordng to classfcaton accuracy n 00. Fuzzy uncertanty and rough uncertanty are exstent n real data. In order to completely measure data uncertanty, a new four-layer feedforward fuzzy-rough neural network s constructed based on fuzzy rough membershp functon of fuzzy rough set, and a new feature selecton method s realzed based on ths network. The artcle wll frst ntroduce the notons of fuzzy rough set and fuzzy rough membershp functon n secton ; and then the structure and nput-output connectons of FRNN are presented n secton 3. nally, a new feature selecton algorthm based on FRNN s proposed n secton 4, and the performance of the algorthm s tested by UCI datasets and compared wth RBF feature selecton method.. Fuzzy Rough Membershp Functon rstly, let us ntroduce fuzzy rough set. Fuzzy rough 0 TFSA

2 Junyang Zhao and Zhl Zhang: Fuzzy Rough Neural Network and Its Applcaton to Feature Selecton 7 set can be defned by a par of fuzzy lower approxmaton and fuzzy upper approxmaton of fuzzy approxmaton space as follows [5]. Defnton.: Let ( U, R) be a fuzzy approxmaton space, U s a non-empty set of fnte obects, x U, R s a fuzzy equvalence relaton on U, f( U), the lower approxmaton and upper approxmaton denoted by R and R are defned as: μ ( x) = nf max{ μ ( y), μ ( x, y)} () R R y U μ ( x) = supmn{ μ ( y), μ ( x, y)} () R y U Defnton.: Let ( U, R) be a fuzzy approxmaton space, U s a non-empty set of fnte obects, x U, R s a fuzzy equvalence relaton on U, U / R = { F, F,, F } H, f( U), the degree of fuzzy rough membershp [6] of x relatve to s defned by ( x) 0 H ρ = H = R μ ( x) τ ( x) f μ ( x) > 0 (3) else where, μ ( x) s the degree of fuzzy membershp of x relatve to F, and the second term n the formula mn( μ ( x), μ ( x)) F x U τ ( x) = = s the degree F μ ( x) x U of rough membershp of x relatve to, H s the number of fuzzy classes contanng x and μ ( x) > Fuzzy Rough Neural Network A. The Structure of FRNN In lterature [7], Sarkar once proposed a three-layer fuzzy rough neural network based on fuzzy rough membershp functon, and appled the network to vowel classfcaton. Ths network can dspose the rough uncertanty ssue of fuzzy clusterng, but has the lmtatons of low adaptaton to new data and low classfcaton accuracy. Therefore, Zhang dongbo proposed an mproved method, but ths method needs a great deal of tme for tranng weghts [8]. To reduce tranng tme, a four-layer feedforward fuzzy-rough neural network ncludng nput layer, cluster layer, membershp layer and output layer s desgned as showed n gure. The neuron number of nput layer les on the dmensons of nput pattern; the neuron number of cluster layer les on the cluster number of nput data whch clustered by unsupervsed FCM algorthm; the neuron number of membershp layer and output layer are equal to the output classes. Let net and O be the nput and output sgnals of pq the th neuron n the th layer, and ω be the connecton weght comng from the th neuron n the p th layer to the th neuron n the q th layer. The detaled nput-output relatons of each layer wll be descrbed below: () Input layer: The orgnal data s mported to the nput layer wthout any transformaton, and each node s correspondng to a feature of nput sample. net = x =,,, N (4) O = net (5) where, x s the th feature of nput sample x = ( x, x,, x ) T N, N s the number of features. () Cluster layer: Cluster layer calculates the fuzzy membershps of each nput sample to clusters by Gauss functon. gure. Structure of mproved fuzzy rough neural network. O net = x = ( x, x,, x ) T (6) T ( x m ) ( x m ) = exp σ where, m andσ are the cluster center and devaton of the th cluster. If F represents the th cluster, then O can be expressed as μ ( x). F N (7) (3) Membershp layer: The nput of membershp layer denotes the value of fuzzy rough membershp as formula (3), and the actuaton functon adopts tansg, then net H 3 3 ω O = = (8)

3 7 Internatonal Journal of Fuzzy Systems, Vol. 3, No. 4, December 0 O 3 = + net 3 exp( ) H s the neuron number of membershp layer. The 3 connecton weghts ω of layer two to layer three can be calculated by formula (0), namely μ ( x) 3 F C x C ω = = (0) H F H μ ( x) x C where, C s the th output class. (4) Output layer: The output layer fnally calculates the membershps of nput samples to each output class, and the samples wll be classfed to the class whch has the maxmum membershp value. net c ω O = 4 4 O net (9) = () = () where, c s the output class number. The connecton 34 weghts ω of layer three to layer four can be traned by gradent descendng method, or calculated drectly. The gradent descendng method needs a lot of tme, and can t ensure the convergence, so drectly n the artcle. Let S be the target outputs, then W O weght matrx s calculated as 34 3 W O S 34 ω wll be calculated 34 3 = S, the = ( ) (3) B. The Generalzaton of FRNN The generalzaton ablty of neural network means the ablty of traned neural network expressng correct nput-output connectons on test samples of the same dataset. The hgher tranng precson does not mean the hgher test precson. Sometmes, the neural network wth better approachng ablty on tranng data has weaker generalzaton capablty to new sample data for ts over-fttng problem. The structure and learnng algorthm of neural network both affect the network generalzaton capablty. In order to evaluate the generalzaton ablty of FRNN, FRNN s compared to the radal bass functon (RBF) neural network based on the dfferences of classfcaton precson. Therefore, sx common UCI datasets are ntroduced to test FRNN, whch are breast_cancer, breast_cancer_wsconsn, Cleveland, Glass, Wne and Wdbc [9]. For the convenence of wrtng, breast_cancer and breast_cancer_wsconsn are shortened as B.C. and B.C.W. These datasets contan not only dscrete features, but also contnuous features, and need to deal wth two-class or mult-class problems. Table. The tranng and test precsons of FRNN and RBF. Datasets All features Tranng precson RBF Tranng precson FRNN B. C B. C. W Cleveland Glass Wne Wdbc Ave As showed n table, the RBF tranng precsons on the sx datasets are relatvely hgh, but the test precsons are relatvely lower. The average test precson s only 79.6% of the tranng precson, and the dfference of tranng precson and test precson changes greatly wth dfferent datasets. Such as, the test precsons are hgher than the tranng precson by 0.59% on B. C. W, but lower on Glass by 46.86%. The FRNN tranng precsons on the sx datasets are lower than RBF by average 7.5%, even 35.54% on Cleveland, but the tranng precsons and the test precsons are comparatvely consstent. The test precsons are hgher than the tranng precsons on most datasets. Such as, the FRNN average test precson on the sx datasets s hgher than the FRNN average tranng precson by 0.39%, even 4.49% on Breast Cancer. At the same tme, the FRNN test precson s hgher than the RBF test precson by average 3.09%, even 0.0% on Glass whch RBF performances worst. The above analyss llustrate that FRNN s adaptve to dfferent data samples and has a good mert of achevng hgher test precson on most datasets. FRNN has better generalzaton ablty than RBF network. 4. Feature Selecton Algorthm based on FRNN A. The Essence of the Algorthm In ths secton, FRNN s appled to feature selecton by heurstc backward search strategy. Orgnally, all features set A s mported to FRNN whch wll be traned to satsfy desred precson n the tranng set S, and then the precson varatons of the network n the valdatng set S after deletng each feature of set A are compared. The feature whch has the least precson varaton s chosen to delete and correspondngly FRNN s trmmed. If the precson descendng level of the trmmed FRNN n the valdatng set S s controlled under the pre-desgned threshold, then ths feature s not mportant and wll be deleted, else retaned. The process s repeated untl precson descendng levels of all nput

4 Junyang Zhao and Zhl Zhang: Fuzzy Rough Neural Network and Its Applcaton to Feature Selecton 73 features remaned n the network exceed the threshold, and all resdual features can t be deleted. As descrbed above, the crteron to evaluate features s the network precson descendng level after deletng features. In the artcle, the network precson s defned accordng to the network classfcaton accuracy to nput samples: Accu = / (4) where, Accu s the number of correctly classfed samples, s all nput samples. All B. The Realzaton of the Fuzzy Rough Neural Network Feature Selecton Algorthm (FRNN_FS) Algorthm: FRNN_FS P Construct a four-layer feedforward fuzzy-rough neural network, let A = { A, A,, A } N be the set of all features, tran network Q to satsfy desred precson, and calculate network precson R and R n the tranng set S and valdatng set S, Δ R s permtted precson descendng level; P To all nput nodes k =,,, N a) Delete the kth nput node of network Q and all weghts connected to ths node to construct a new network Q ; k Accu b) Calculate the classfcaton precson R k of n the valdatng set S ; c) Calculate precson descendng rato δ = ( R R )/ R k k of Q k ; P3 Rank δ k, let All Q k δ mn be the rato of most unapparent feature; P4 If δ <Δ R mn, then delete ths feature and correspondng nput node, N = N, return to step P; Else, end. Ths feature selecton algorthm adopts heurstc backward search strategy, and doesn t need exhaustve search or complete search. Let the number of condtonal features be N, then the tme complexty of the algorthm s ON ( log N ). 5. Expermental Analyss by UCI Data Before experments, the parameter settng of FRNN should be frstly taken nto account especally the number of cluster layer nodes. The set of cluster layer nodes s an mportant parameter whch drectly affects the performance of FRNN, but stll has no unfed settng method. To meet the need of clusterng, the number of cluster layer nodes should be at least twce the number of decson classes. Therefore, some tests have been carred out to dscover the nfluence of cluster layer nodes to FRNN performance on four datasets. Let multple factor (Mf) be the dvson of the number of cluster layer nodes by the number of decson classes. gure denotes the relatons of Mf, tranng precson and test precson on these four datasets. As showed n gure, the tranng precsons and test precsons both decrease wth the ncrease of Mf at most stuaton, but the test precsons on Wdbc ncrease apprecably. Some experences can be concluded that t does not mean more cluster layer nodes nduce much better network performance. The augment of cluster layer nodes greatly ncrease the network complexty. gure 3 ndcates the relatons of Mf and FRNN run-tme on the datasets. Wth the augment of Mf, the network run-tme ncrease rapdly whch reflects the much more complex FRNN. Takng the above analyss nto account synthetcally, t s reasonable to set multple factor Mf be two (a) Breast Cancer Wsconsn 0.94 Tran Accu (c) Wne Tran Accu (b) Cleveland Tran Accu (d) Wdbc 0.94 Tran Accu gure. Relatons of multple factor and classfcaton (a) Breast Cancer Wsconsn (c) Wne (b) Cleveland (d) Wdbc gure 3. Relatons of multple factor and run-tme.

5 74 Internatonal Journal of Fuzzy Systems, Vol. 3, No. 4, December 0 Table. Classfcaton accuraces of orgnal feature sets and subsets selected by FRNN_FS. Evaluaton Indexes Cleveland Wne Wdbc All features Selected features All features Selected features All features Selected features Average feature number 3(0.00) 3(0.47) 3(0.00) 5.5(.7) 30(0.00) 3.4(0.97) Average tranng precson 59.5(.9) 6.43(0.77) 98.3(0.00) 94.77(.48) 9.38(0.00) 93.70(0.95) Average test precson 57.5(.57) 6.73(.3) 98.59(0.00) 98.73(.3) 9.94(0.4) 95.88(0.59) Datasets Tranng precson Table 3. Classfcaton accuraces of feature v selected by FRNN and RBF. RBF_FS Feature number Tme(s) Tranng precson FRNN_FS Feature number B. C. 9.44(0.78) 57.0(.05) 4(0.00) (.86) 73.5(.93) 3.4(.7) 0.8 B. C. W 98.57(0.86) 93.9(0.34) 4(0.00) (0.98) 97.8(0.55) 3.9(.0) 0.78 Cleveland 97.5(0.65) 47.(0.5) 7.8(0.3) (0.77) 6.73(.3) 3.0(0.47).5 Glass 95.3(.8) 54.65(.8) 4(0.00) (3.7) 6.5(.9) 4.3(0.95) 0.73 Wne 98.3(0.85) 98.59(0.76) 8(0.00) (.48) 98.73(.3) 5.5(.7) 0.95 Wdbc 98.83(0.88) 96.05(0.58) 8.9(0.85) (0.95) 95.88(0.59) 3.4(0.97) 4.54 Ave Tme(s) In order to valdate the feasblty and performance of the algorthm FRNN_FS, the sx datasets n secton 3 are also adopted n ths secton. Besdes, ths algorthm s compared to RBF feature selecton algorthm (RBF_FS) wth backward prunng strategy whch has good approachng capablty. The RBF_FS algorthm takes the same feature prunng procedure as FRNN_FS, the only dfference s RBF_FS adopts orthogonal least squares (OLS) method [0] to tran RBF network. The OLS method has the merts of hgh tranng precson, fast convergence velocty and self-determnaton of hdden neurons. In these experments, the neuron number of cluster layer s defned twce the number of decson classes, each newly constructed network s traned at least 30 tmes, precson descendng rato Δ R s set to Table shows the classfcaton precson comparsons of orgnal feature sets and subsets selected by FRNN on Cleveland, Wne and Wdbc datasets. The tranng and test precsons of the selected subsets on these three datasets are all heghtened, except the tranng precson on Wne. Besdes, the feature number of the subsets decreases greatly. For example, the subset feature number on Wdbc s only.3% of orgnal features; synchronously the average test precson mproves 3.%. Another character s that the test precson of FRNN s hgher than the tranng precson regardless orgnal feature sets or subsets, and mproves 4% on Wne, only except on orgnal Cleveland. It ndcates that FRNN has favorable generalzaton capablty. Table 3 shows the performance comparsons of subsets selected by RBF and FRNN on the sx datasets, such as classfcaton precson, feature numbers of subsets, run tme. Accordng to the No Free Lunch theory, dfferent algorthms have respectve merts and faults. There s no algorthm absolutely excellent than another algorthm, but also t s true that one algorthm can be synthetcally excellent than another algorthm. From the data n table, FRNN_FS obvously performs well than RBF_FS n most ndexes. The only excepton s that the tranng precson of FRNN s not as good as RBF. As showed on the results of these sx datasets, the average tranng precson of RBF s 96.76%, and that the average tranng precson of FRNN s 79.78%, but the performances of these two networks on test precson are qute dfferent. The average test precson of RBF s only 74.56%, and the average test precson of FRNN s 8.70% wth 9.6 percent mprovement. It llustrates the generalzaton capablty of FRNN s better than RBF. Ths character represents much obvously on B.C., Cleveland and Glass. Takng B.C. as an example, the tranng precson of FRNN s lower than RBF by.%, but the test precson of FRNN s hgher than RBF by 6.49%. Another mportant obectve of feature selecton s to select features as less as possble whle ensurng class-

6 Junyang Zhao and Zhl Zhang: Fuzzy Rough Neural Network and Its Applcaton to Feature Selecton 75 fcaton precson. The average feature number of subsets selected by FRNN s 3.9, whch s only 64.% of the average feature number of subsets selected by RBF. Especally on Cleveland, the subset feature number of FRNN s less than RBF by 6.5%, but the test precson of FRNN s hgher than RBF by 5.6%. Besdes, the run tme of FRNN_FS averagely reduces than RBF by 9.5%. As we know, the reducton extent s certanly related to each dataset. It does not mean that the run tme of FRNN_FS can be reduced greatly to all datasets, but at least t ndcates the runnng effcency of FRNN_FS s relatvely hgh. As syntheszed from the above analyss, FRNN has the merts of hgh classfcaton precson, fast tranng velocty and good generalzaton capablty. The feature selecton algorthm based on FRNN can select subsets wth smaller features, at the same tme the subsets possess hgh average classfcaton precson. Therefore, the algorthm FRNN_FS s stable and effectve. 6. Conclusons In order to deal wth fuzzy uncertanty and rough uncertanty of real data, the concept of fuzzy rough membershp functon defned n fuzzy rough set s ntroduced. An mproved fuzzy rough neural network combnng FRMF and neural network s constructed. show that FRNN_FS can deal wth complex real data, select feature subsets effectvely, and also wth good generalzaton capablty. Smultaneously, the data mported to FRNN_FS need to be clustered beforehand, and the number of cluster layer neurons can t be determned drectly, but need to be presented accordng to experences. These dsadvantages should be amelorated n the future.ths new FRNN s further appled to feature selecton by repeatedly trmmng network nput nodes to mprove classfcaton precson, and the FRNN_FS algorthm s proposed based on ths method. Datasets experments. References [] R. Setono and H. Lu, Neural-Network Feature Selector, IEEE Trans. on Neural Network, vol. 8, no. 3, pp ,997. [] J. Basak and S. Mtra, Feature Selecton Usng Radal Bass Functon Networks, Neural Comput & Applc, vol. 8, pp , 999. [3] K. P. Sankar, K. D. Raat, and J. Basak, Unsupervsed Feature Evaluaton: A Neuro-Fuzzy Approach, IEEE Trans. on Neural Network, vol., no., pp , 000. [4] A. Verkas and M. Bacauskene, Feature selecton wth neural networks, Pattern Recognton Letters, vol. 3, pp , 00. [5] D. Dubos and H. Prade, Rough fuzzy sets and fuzzy rough sets, Int. J. Gen. Syst., vol. 7, pp. 9-09, 990. [6] M. Sarkar and B. Yegnanarayana Fuzzy-rough Membershp Functons, In Proc. IEEE Int. Conf. on Systems, Man and Cybernetcs, Calforna, USA, pp , 998. [7] M. Sarkar and B. Yegnanarayana, Fuzzy-Rough Neural Network for Vowel Classfcaton, In Proc. IEEE Int. Conf. on Systems, Man and Cybernetcs, Calforna, USA, pp , 998. [8] D. B. Zhang and Y. N. Wang, Fuzzy-rough Neural Network and Its Applcaton to Vowel Recognton, Control and Decson, vol., no., pp. -4, 006. [9] [0] S. Chen, C. F. N. Cowan, and P. M. Grant, Orthogonal Least Squares Learnng Algorthm for Radal Bass Functon Networks, IEEE Trans. On Neural Networks, vol., no., pp , 99. Junyang Zhao s currently workng at an Research Inst. Of H-tech, an, Chna. He has a M.S. and a Ph.D. degree n Control Engneerng obtaned n 005 and 009, respectvely, from an Research Inst. Of H-tech. He serves as a program commttee member of IEEE WCCI 0. Hs research nterests nclude fuzzy sets, rough sets, computatonal ntellgence and uncertanty nference. Zhl Zhang s now a full professor at an Res- earch Inst. Of H-tech, an, Chna. He has a M.S. and a Ph.D. degree n Control Engneerng obtaned n 99 and 00, respectvely, from an Research Inst. Of H-tech. He has publshed more than 00 refereed ournal and conference papers. Hs research nterests nclude fault dagnoss and system smulaton.