A HYBRID FEATURE SELECTION METHOD BASED ON FISHER SCORE AND GENETIC ALGORITHM

Transcription

1 Journa of Mathematica Sciences: Advances and Appications Voume 37, 2016, Pages Avaiabe at DOI: A HYBRID FEATURE SELECTION METHOD BASED ON FISHER SCORE AND GENETIC ALGORITHM Schoo of Information Science and Technoogy Jinan University Guangzhou P. R. China e-mai: @163.com Abstract Fisher score and genetic agorithm are widey used for feature seection. However, some redundant features wi be seected by Fisher score, and the convergence properties may be worse if the initia popuation of genetic agorithm is generated by a random manner. To improve the performance of feature seection by Fisher score and genetic agorithm, we propose a hybrid feature seection method, which merging the advantages of Fisher score and genetic agorithm together. It aims at utiizing the features' Fisher score to generate the initia popuation of genetic agorithm. To begin with, the Fisher scores of a the features wi be mapped into a specific interva by a inear function, and then the rescaed Fisher scores wi be utiized to generate the initia popuation of genetic agorithm. Finay, the initia popuation wi be used in the subsequent procedure of genetic agorithm to perform feature seection with eitist strategy for reference. In this paper, we choose four data sets of Sonar, WDBC, Arrhythmia, and Hepatitis to test the performance of our agorithm. Feature subsets of the four data sets wi be seected by our agorithm, and then the dimensionaity of data sets wi be reduced according to the seected feature subsets, respectivey. 1-NN cassifier is used to cassify the dimensionaity reduced data sets, and respectivey, achieving the cassification 2010 Mathematics Subject Cassification: 68T10. Keywords and phrases: feature seection, Fisher score, genetic agorithm, eitist strategy. Received January 21, Scientific Advances Pubishers

2 52 accuracy of 72.36%, 95.64%, 72.04%, and 87.83% with ten-fod cross vaidation method. The experiment resuts show that, compared to the performance of Fisher score, genetic agorithm and Fisher score genetic agorithm, our agorithm is fit for eiminate redundant features, and can seect discriminative features. Above a, our method is effective in feature seection. 1. Introduction In many research fieds such as machine earnin pattern recognition, biomedicine and so on, researchers need to anaysis vast amounts of data, which is described by high-dimensiona vectors. Every component of a vector represents a kind of feature of the highdimensiona data. In reaity, there are some meaningess and redundant features in the high-dimensiona vectors. To improve the efficiency of subsequent procedure (such as cassification and prediction) and save the dedicated space of device, we need to eiminate the meaningess and/or redundant features from those high-dimensiona vectors, and seect the features which are most reevant for our probems to form the optima feature subset, the procedure is caed to be feature seection, or feature subset seection (FSS) [1, 2]. Superficiay, exhaustive search can seect the optima feature subset, however, it is a NP-hard probem that we adopt exhaustive search to seect features, because each dataset has 2 n feature subsets. In the past decades, a number of feature seection methods have been proposed, and they are generay divided into two categories: fiter-based methods and wrapper-based methods. Fiter-based methods rank the features according to a predefined criterion independent of the actua generaization performance of the earning machine, and seect those features with high ranking scores, so a faster speed can usuay be obtained. Mutua information (MI) [3]; Fisher score (FS) [4]; Reieff [5]; Lapacian score [6]; Hibert Schmidt independence criterion (HSIC) [7]; and Trace ratio criterion [8] or so can be regarded as the criterion for fiter-based feature seection, among which Fisher score is one of the most widey used criteria for fiter-based feature seection due to its genera good performance. The specific process of feature seection method based on Fisher score is ike this: cacuating

3 A HYBRID FEATURE SELECTION METHOD 53 the Fisher scores of a the features, and for a given threshod θ, feature f i is seected if F ( f i ) > θ, otherwise, if F ( f i ) θ, feature f i wi not be seected. Seecting features by Fisher score can improve the accuracy of subsequent procedure (such as cassification and prediction), and the process is simpe, feasibe and time saving. However, there are aso some probems about Fisher score: (1) How to set the threshod θ such that an optima feature subset is seected? (2) We cannot eiminate redundant features via Fisher score. For exampe, both the scores of feature f i and feature f j are very high, but they are highy correated. In this case, the fiter-based agorithm wi seect both f i and f j, whie either f i or f j shoud be eiminated without any oss in the subsequent cassification performance. (3) Since the fiter-based agorithm computes the Fisher score of each feature individuay, it negects the combination of features, which means evauating two or more than two features together. For instance, it coud be the case that the scores of feature f i and feature f j are both ow, but the score of the combination of f i and f j is very high. In this case, the fiter-based agorithm wi discard both f i and f j, athough they shoud be seected. Wrapper-based methods use a predictor as a back box and the predictor performance as the objective function to evauate the feature subset. A wide range of wrapper-based methods have been used incuding forward seection [9]; backward eimination [10]; hi-cimbing [11]; branch and bound agorithms [12]; simuated anneaing and genetic agorithms (GAs) [13, 14, 15, 16]. Kudo and Skansky [17] made a comparison among many of the feature seection agorithms and expicity recommended that GAs shoud be used for arge-scae probems with more than 50 candidate variabes. In many practica appications, the genetic agorithm can be reformed in different ways according to different situations. However, when the GA is adopted to seect features, it is unreasonabe that the initia popuation is generated by a random manner. Because every feature has the same chance to be seected into the initia popuation, the convergence properties of GA may be worse.

4 54 Yet, we just want to seect the best features rapidy. Since our purpose is to seect the best features rapidy, why not seect the features of high Fisher score by setting a threshod θ to generate the initia popuation, and then adopt the genetic agorithm to achieve the optima feature subset? It is reasonabe to a extent, however, features of owest Fisher score may have itte chance to be seected. To overcome the above probems, we consider that if the features of higher Fisher score have more probabiities to be seected to generate the initia popuation than features of ower Fisher score, and some chromosomes of best fitness are reserved to the next generation without any genetic operations, the cassification accuracy after feature seection wi be improved. The basic idea of this paper is just about this. In this paper, we present a hybrid feature seection method based on Fisher score and genetic agorithm. Our method utiize features rescaed Fisher score to generate the initia popuation of genetic agorithm. In the first pace, the Fisher scores of a the features wi be mapped into a specific interva by a inear function, and then we utiize the rescaed Fisher scores to generate the initia popuation of genetic agorithm. Finay, the initia popuation wi be used in the subsequent procedure of genetic agorithm to perform feature seection with eitist strategy for reference. Experiments on four benchmark data sets of Sonar, WDBC, Arrhythmia, and Hepatitis indicate that the proposed method outperforms Fisher score method and genetic agorithm, and it does we in eiminating redundant features. Features seected by our method are more discriminative than that some state of the art feature seection methods. The remainder of this paper is organized as foows. In Section 2, we briefy review Fisher score and genetic agorithm. We present the hybrid feature seection method based on Fisher score and genetic agorithm in Section 3. The experiments on four data sets of Sonar, WDBC, Arrhythmia, and Hepatitis are demonstrated in Section 4. Finay, we draw a concusion in Section 5.

5 A HYBRID FEATURE SELECTION METHOD A Brief Review of Fisher Score and Genetic Agorithm 2.1. Fisher score, N i y i i= 1 Given dataset { x }, where M x i R and y i { 1, 2,, c} represents the cass which x i beongs to. N and M are the number of sampes and features, respectivey. f 1, f2,, fm denote the M features. Redundant and meaningess features may be incuded in the feature set when M is arge, thus we want to seect a subset from the M features in order to make the data set more productive. For exampe, we hope the seected features improve the cassification accuracy for a cassification probem. If a feature is discriminative, the between-cass variance of the feature shoud be arge, whie the within-cass variance of the feature shoud be sma. Let µ denote the mean of feature f, and f i mean of feature f i in cass k. Fisher score [4] is defined as foows: i k µ be the f i F ( f ) = i c nk k = 1 c k = 1 yj = k k ( µ µ ) fi k ( f µ ) j, i fi fi 2 2, (2.1) where n k is the number of sampes in cass k, and f j, i is the vaue of feature f i in sampe x j. From the Equation (2.1), we can concude that features with higher Fisher score are more discriminative to some extent Genetic agorithm The genetic agorithm (GA) [18] was originay deveoped by Hoand [3] and his associates. The basic idea of GA is to construct a fitness function according to the objective function of the probem, and utiize the fitness function to evauate a set of candidate soutions (every soution responds to a chromosome) caed a popuation. Based on the Darwinian

6 56 principe of surviva of the fittest, the GA obtains the optima soution after a series of iterative computations. GA generates successive popuations of aternate soutions that are represented by a chromosome, i.e., a soution to the probem, unti acceptabe resut is obtained. Genetic agorithm is a method that based on group optimization, as described beow: Step 1: Generating initia popuation. The generating of initia popuation is random, and the specific way reays on the encoding of chromosomes. The 0-1 codes are generated as beow: et 0, I R denotes the -th chromosome of the initia popuation, for 0, 0, 0, 0, I = ( I, I,, I ), generating a random foating number 1 2 M ξ i U( 0, 1), if ξ i > 0.5, 0, then I i = 1; otherwise, if ξ i 0.5, then 0, Ii = 0 ( i = 1, 2,, M ). The setting of popuation size N reays on the computing abiity of a computer and the compexity of a agorithm. Generay, N is set to be It is the 0-th generation at present. Step 2: Estimating stopping criterion. We adopt the number of maximum generation as the stopping criterion. The agorithm stops when the number of generations reaches the preset maximum generation. Step 3: Computing the fitness. Choosing a proper fitness function f ( I ) based on the objective function of optimizing issue, and cacuating the fitness of every chromosome, respectivey. The higher fitness a chromosome have, the more probabiity it wi be seected to take part in the subsequent genetic operation. Step 4: Seecting the chromosome with high fitness. The rouette-whee seection scheme wi be used to seect N chromosomes to generate a new popuation. Step 5: Genetic operation. The seected chromosomes need to undergo genetic operations, such as crossover and mutation. (1) Crossover: On the basis of the crossover rate P c, the crossover operation generates new M

7 A HYBRID FEATURE SELECTION METHOD 57 chromosomes (offspring) out of their parents. (2) Mutation: The mutation is appied to the offspring. According to the mutation rate P m, we perform the 1-0 and 0-1 conversions. Step 6: Updating the popuation. We obtain the offspring after the crossover and mutation process, and the agorithm enters to the next generation. Return to Step A Hybrid Feature Seection Method Based on Fisher Score and Genetic Agorithm To overcome the probems of Fisher score and genetic agorithm, we propose a hybrid feature seection method based on Fisher score and F f F f,, F f genetic agorithm. We cacuate the Fisher score [ ( ) ( ) ( )] 1, 2 of every feature, and then map a the Fisher scores into a cosed interva [ ε 1 ε], by a inear function, where ε is a reativey sma positive rea vaue. We utiize the rescaed Fisher scores to generate the initia popuation of genetic agorithm, and make a itte modification in the subsequent operator of genetic agorithm. We aim to seect a more optima subset of features than Fisher score and genetic agorithm Generating initia popuation Suppose we have Fisher scores [ F ( f ), F ( f2 ),, F ( fm )] features, utiize a inear function L( F ( f )) to rescae ( ) [ ε 1 ε] i M 1 for a F f into range, for i = 1, 2,, M, and the inear function L( F ( f )) is given as beow: 1 2ε ε( Fmax + Fmin ) Fmin L( F ( f i )) = F ( fi ) +, (3.1) F F F F max min where ε is a reativey sma positive rea vaue, 0 < ε < 0.1, and we ascertain the vaue of ε according to the practica situation of probems. Fmax = max { F ( f1 ), F ( f2 ),, F ( fm )}, Fmin = min { F ( f1 ), F ( f2 ),, F ( fm )}. max min i i

8 58 We adopt 0-1 coding scheme to represent individua in every generation. Let M I R denote the -th individua in the g-th generation. Then I = ( I1, I2,, I M ), where I i = 1 if feature f is seected, otherwise, I = 0, for i = 1, 2,, M. i i Unike genetic agorithm, we mainy utiize the rescaed Fisher scores L( F ( f )) to generate initia popuation in GA agorithm. i Generating M rea vaue of η i by computer with a random manner, where η i U( 0, 1), if ( ( )) 1 L F f i > η i, 0, then I i = 1; otherwise, if L( F ( f )) η, then I = 0, for i = 1, 2,, M. i 3.2. Fitness function i 0, i The seection of a fitness function has a direct infuence on the convergence rate of GA, and it aso concerns whether the optima soution wi be found. Generay speakin the fitness function reates to the objective function of a optimization probem. Therefore, the fitness function denoted by f ( I ) is defined as cassification accuracy of 1-NN cassifier with 10-fods cross vaidation, and there is an additiona term of M g I, i i= 1 λ to penaize the number of features. Then f ( I ) can be expressed as: M i i= 1 f ( I ) = accuracy( I ) λ I, (3.2) where λ is a positive rea parameter Eitism strategy In traditiona genetic agorithm, after a series of seection, crossover and mutation procedures, we get N (where N means popuation size) new chromosomes, and the N new chromosomes wi repace a of their parents to form the next generation. Different from the traditiona GA,

9 A HYBRID FEATURE SELECTION METHOD 59 eitism strategy [19] considers two situations: if the new chromosome with best fitness is superior to the best fitness individua in the ast popuation, a the individuas in the ast popuation wi be repaced; otherwise, the most inferior new chromosome is repaced by the best chromosome in the ast generation. Eitism strategy guarantees the consistency of the optima soution at present and in history, and makes the agorithm to be goba convergence [20]. We take the idea of eitism strategy for reference, seecting ( N m) chromosomes from the ast popuation to participate in genetic operation, and retain those chromosomes corresponding to top m fitness vaues in the next generation without participating in any genetic operations. The specific operations are exempified in Figure 3.1. This operator does not obstruct the creation of new chromosomes, and ensures the best fitness chromosome in the next generation never inferior to the one in the ast generation, the process of evoution becomes a optimizing process. Figure 3.1. The eitism strategy operation in this paper Seection strategy Let P( I ) denotes the probabiity that define P( I ) as foows: I is seected. Then we f ( I ) P( I ) =, (3.3) N = 1 f ( I ) where N means popuation size. Rouette whee seection scheme is appied to seect individuas for genetic operations. Let

10 60 PP 0 = 0, (3.4) PP j = P( I ). (3.5) j= 1 Generate a random variabe ξ k ~ U( 0, 1), for k = 1, 2,, N; if PP < 1 ξk PP, then the individua process is repeated ( N m) times Genetic operator and stopping criterion I is seected. The seection Genetic operator mainy incudes two procedures: crossover and mutation. The ( N m) seected chromosomes wi take part in the crossover and mutation procedure according to the probabiity of crossover and mutation, respectivey. The crossover operation generates two new chromosomes (offspring) out of their parents, and the mutation operation sighty perturbs the offspring. The GA stops when the number of generations reaches the preset maximum generation Maxiters Crossover The crossover operator is performed according to the crossover probabiity denoted by P c. The probabiity of crossover contros the frequency of crossover operator, and a higher probabiity is good for open up a new searching area, whie a ower probabiity may ead to a buntness state of GA agorithm. To get a considerabe resut, in this paper, we set the crossover probabiity as a function of the iteration times, and the expression of the crossover probabiity is given beow: 1 P c = n( iters) + 0.7, (3.6) 10 n( Maxiters) where iters denotes the current iteration times and Maxiters is the maximum iteration times. Figure 3.2 shows the curve of crossover probabiity as a function of iteration times in 500 times of iteration.

11 A HYBRID FEATURE SELECTION METHOD 61 Figure 3.2. P c changing with iteration times. We adopt the singe-point crossover manner, and seect two chromosomes t t t t of = ( I, I,, I ) I and = 1 2 M t+1 t+ 1 t+ 1 t+ 1 I ( 1, I2,, I M ) I by Rouette whee, choosing a cutting-point randomy and exchanging the part on the right of cutting-point to get two new chromosomes of g + 1, t g + 1, t g + 1, t ( I, I,, I ) g + 1, t g + 1, t+ 1 g + 1, t+ 1 g + 1, t+ 1 I = 1 2 M and I = ( I1, I2,, g + 1, t+ 1 I M ). A specific exampe of singe-point crossover is exempified in Figure 3.3. Figure 3.3. A specific exampe of singe-point crossover Mutation The mutation operator is performed according to the mutation probabiity denoted by P m. The mainy purpose of mutation is to

12 62 maintain the diversities of popuation. In genera, a ower mutation probabiity means ess ikey to ose important genes; whie a higher mutation probabiity wi ead the agorithm to be a random research process, so we need to carefuy contro the number of 1-0 and 0-1 conversions. In this paper, we set the mutation probabiity as a function of the iteration times, and the expression of the crossover probabiity is given beow: ( 1) 0.04 cos π iters P m = , (3.7) 2( Maxiters 1) where iters denotes the current iteration times and Maxiters is the maximum iteration times. Figure 3.4 shows the curve of mutation probabiity as a function of iteration times in 500 iterations. Figure 3.4. P m changing with iteration times. Based on the mutation probabiity, we seect three gene sites, and perform the 1-0 and 0-1 conversions. A specific exampe of mutation is exempified in Figure Figure 3.5. A specific exampe of mutation.

13 A HYBRID FEATURE SELECTION METHOD Experiments 4.1. Datasets To test the efficiency of our proposed agorithm adequatey, we use a subset of UCI machine earning benchmark data set [21], and they are Sonar, WDBC, Arrhythmia, and Hepatitis dataset, respectivey. Tabe 4.1 summarizes the characteristics of the data sets used in our experiments. A datasets are standardized to be zero-mean and normaized by standard deviation for each dimension. Tabe 4.1. Description of the data sets used in our experiments Data sets Size Number of features Casses Sonar WDBC Arrhythmia Hepatitis Sonar dataset contains 208 vectors, each vector incudes 60 components, which means the dataset has 60 features. We acquire the data from the feedback information of sonar. There are two casses in Sonar dataset, data of rock and data of minera. After feature seection, 1-nearest neighbour (1-NN) cassifier is used for cassification, and ascertain whether a vector beongs to minera according to the resut of cassification. WDBC dataset contains 569 vectors, each vector incudes 30 components, which means the dataset has 30 features. There are two casses in WDBC dataset, benign breast cancer and maignant breast cancer. After feature seection, 1-nearest neighbour (1-NN) cassifier is used for cassification, and ascertain whether a vector beongs to benign or maignant according to the resut of cassification.

14 64 Arrhythmia dataset contains 452 vectors, each vector incudes 279 components, which means the dataset has 279 features. There are two casses in Arrhythmia dataset, norma heart rate and abnorma heart rate. After feature seection, 1-nearest neighbour (1-NN) cassifier is used for cassification, and ascertain whether a vector beongs to norma heart rate or abnorma heart rate according to the resut of cassification. Hepatitis dataset contains 155 vectors, each vector incudes 19 components, which means the dataset has 19 features. There are two casses in Hepatitis dataset, hepatitis patients and heathy person. After feature seection, 1-nearest neighbour (1-NN) cassifier is used for cassification, and ascertain whether a vector beongs to hepatitis patients or not according to the resut of cassification Experiments and parameters In our experiments, we compare the proposed method to the state-ofart feature seection methods: Fisher score (FS), Genetic agorithm (GA), and Fisher score genetic agorithm (FSGA). We perform four experiments on every dataset respectivey, and the parameters of 16 experiments are summarized in Tabe 4.2.

15 A HYBRID FEATURE SELECTION METHOD 65 Tabe 4.2. The parameters in 16 experiments Data sets Sonar WDBC Arrhythmia Hepatitis Parameters Agorithms λ N Maxiters θ ε m FS 0.01 GA FSGA * * HFSGA * FS 0.01 GA FSGA HFSGA * FS 0.01 GA FSGA HFSGA FS GA FSGA HFSGA Experiment 1. Feature seection by Fisher score (FS). Whether a feature is seected or not is just reated to the feature s Fisher score and the setting of threshod. When a threshod θ is settin C θ features of F ( f i ) > θ wi be seected to construct a feature subset A, and then we reduce the dimensionaity of origina data set according to the seected feature subset A. 1-NN cassifier is used to cassify the dimensionaity reduced data set. To ensure the comparabiity with other three methods, we aso construct a fitness function of f 0 ( A), which is expressed as the subtraction of the cassification accuracy accuracy(a) with ten-fod cross vaidation method and a penaty item of λc θ. The fitness of a feature subset A is defined as f ( A) = accuracy( A) λc, (4.1) 0 θ

16 66 where parameter λ is the same as that in formua (3.2). In the experiment of Sonar, WDBC, and Hepatitis, we set λ = 0.01, whie in the experiment of Arrhythmia data set which has the most features, we set λ = From formua (4.1), we can concude that the feature subset with higher fitness is better. Experiment 2. Feature seection by genetic agorithm (GA). When adopt GA to perform feature seection in the four data sets respectivey, we set the popuation size N = 100, and λ = in the experiments of Sonar, WDBC and Hepatitis, whie λ = in the experiment of Arrhythmia. In the experiments of Sonar, WDBC, and Arrhythmia, we set Maxiters = 500, and Maxiters = 1000 in the Hepatitis data set experiment. Experiment 3. Feature seection by Fisher score genetic agorithm (FSGA). The specific process of FSGA is ike this: in the first pace, set a threshod θ to seect features whose F ( f ) > θ, then utiize the seected features to generate initia popuation of genetic agorithm by random manner (the features whose Fisher score is ower than threshod θ wi not be seected to generate initia popuation), finay, the feature subset is seected by traditiona genetic agorithm. We set the threshod to be 0.15, 1.5, 0.5, and 0.28, respectivey, in data sets of Sonar, WDBC, Arrhythmia, Hepatitis, and the setting of parameter N, λ, and Maxiters are the same as Experiment 2. Experiment 4. Feature seection by the proposed hybrid feature seection method based on Fisher score and genetic agorithm (HFSGA). We set N = 100, ε = 0.05, m = 2 in experiments of data sets Sonar, WDBC, Arrhythmia, and Hepatitis, and the setting of parameter N, λ, and Maxiters are the same as Experiment 2. i

17 A HYBRID FEATURE SELECTION METHOD Experimenta resuts anaysis The resuts of four experiments in Sonar, WDBC, Arrhythmia, and Hepatitis data sets are shown in Figure 4.1, Figure 4.2, Figure 4.3, and Figure 4.4, respectivey, and Tabe 4.3 shows the cassification accuracy by ten-fod cross vaidation method Experimenta resuts on Sonar From Figure 4.1 and Tabe 4.3, we can concude that our method achieved the highest cassification accuracy of 72.36% in Sonar data set, and seected 4 features, so it can save the storage space and operation time in subsequent processing. In addition, our method reached the best fitness of just requiring 40 iterations, whie the FSGA needs 75 iterations to get the best fitness of , and the GA needs 338 iterations to get the best fitness of 0.6, thus, our method has the fastest convergence rate. In terms of cassification accuracy, number of seected features and convergence rate, our proposed method is most effective for Sonar data set to perform feature seection.

18 68 (a) Resuts of feature seection by FS (b) Resuts of feature seection by GA

19 A HYBRID FEATURE SELECTION METHOD 69 (c) Resuts of feature seection by FSGA (d) Resuts of feature seection by HFSGA Figure 4.1. Four experimenta resuts on Sonar data set Experimenta resuts on WDBC Figure 4.2 and Tabe 4.3 show that the cassification accuracy of performing feature seection on WDBC data set by the four methods are 92.70%, 94.10%, 94.05%, and 95.66%, respectivey, which indicates that the discrimination between reevant features and irreevant features is obvious. The FS seected 5 features, whie other three methods seected 3

20 70 features respectivey, we can infer that there may be 2 redundant features with high Fisher score. Our method achieved the highest cassification accuracy of 95.66%, which demonstrated that the proposed method does we in eiminating redundant features. (a) Resuts of feature seection by FS (b) Resuts of feature seection by GA

21 A HYBRID FEATURE SELECTION METHOD 71 (c) Resuts of feature seection by FSGA (d) Resuts of feature seection by HFSGA Figure 4.2. Four experimenta resuts on WDBC data set Experimenta resuts on Arrhythmia From Figure 4.3(a), when Fisher score is adopted to feature seection, in the beginnin the fitness becomes higher and higher with the increase of threshod, and when the threshod is higher than 0.5, the fitness becomes ower and ower with the increase of threshod. We can infer that, on Arrhythmia data set, there are more redundant features and ess

22 72 meaningess features. The cassification accuracy of proposed method is 70.54%, ower than the FSGA method of 72.34%, whie higher than genetic agorithm and Fisher score of 66.57% and 67.49%, respectivey. This resut indicates that, on the one hand, Fisher score is a favorabe criterion to determine the reevance of features for Arrhythmia data set; on the other hand, the FAGA method does we in eiminating irreevant features, whie the proposed HFAGA method is good at eiminating redundant features. (a) Resuts of feature seection by FS (b) Resuts of feature seection by GA

23 A HYBRID FEATURE SELECTION METHOD 73 (c) Resuts of feature seection by FSGA (d) Resuts of feature seection by HFSGA Figure 4.3. Four experimenta resuts on Arrhythmia data set Experimenta resuts on Hepatitis From Figure 4.4 and Tabe 4.3, we can concude that our method achieved the highest cassification accuracy of 87.83% on Hepatitis data set, and seected 5 features, so it can save the storage space and

24 74 operation time in subsequent processing. Furthermore, our method has the fastest convergence rate. Above a, our proposed method is most effective for Hepatitis data set to perform feature seection. (a) Resuts of feature seection by FS (b) Resuts of feature seection by GA

25 A HYBRID FEATURE SELECTION METHOD 75 (c) Resuts of feature seection by FSGA (d) Resuts of feature seection by HFSGA Figure 4.4. Four experimenta resuts on Hepatitis data set.

26 76 Tabe 4.3. Cassification accuracy and number of seected features of 16 experiments Data sets Agorithms Best fitness Cassification Accuracy Number of seected features FS Sonar (60) GA FSGA HFSGA FS WDBC (30) GA FSGA HFSGA FS Arrhythmia (279) GA FSGA HFSGA FS Hepatitis (19) GA FSGA HFSGA Discussion The proposed HFSGA method is superior to other three feature seection methods of FS, GA, and FSGA methods in terms of cassification accuracy, and it does we in eiminating redundant features with a fast convergence rate. From the experiments on Arrhythmia data set, we can discover that the FSGA method is good at eiminating irreevant features. In a word, our method is effective in feature seection. Certainy, our proposed HFSGA method requires to set many parameters, so we can do some further researches to expore the infuence of parameters on experiment resuts, and then determine the

27 A HYBRID FEATURE SELECTION METHOD 77 optima parameters. In addition, the seection of cassifier aso infuences the cassification accuracy, thus, the match of cassifier and feature seection method is worthy of further research. 5. Concusion In this paper, we presented a hybrid feature seection method based on Fisher score and genetic agorithm. It utiizes features' rescaed Fisher score to generate the initia popuation of genetic agorithm. On the one hand, our method avoids the conundrum of how to set a threshod, and it does we in eiminating redundant features, thus, it can seect the feature subset with better performance. On the other hand, unike genetic agorithm generating initia popuation with a random manner, our method has a faster convergence rate. Contrast experiments on four benchmark data sets of Sonar, WDBC, Arrhythmia, and Hepatitis indicate that the proposed method outperforms Fisher score method and genetic agorithm, and it is effective in feature seection. References [1] I. Guyon and A. Eisseeff, An introduction to variabe and feature seection, Journa of Machine Learning Research 3 (2003), [2] Y. Yang and J. O. Pedersen, A comparative study on feature seection in text categorization, In ICML (1997), [3] D. Koer and M. Sahami, Toward optima feature seection, In ICML (1996), [4] P. E. H. R. O. Duda and D. G. Stork, Pattern Cassification, Wiey-Inter Science Pubication, [5] M. Robnik-Sikonja and I. Kononenko, Theoretica and Empirica Anaysis of ReiefF and RReiefF, Machine Learning 53(1-2) (2003), [6] X. He, D. Cai and P. Niyogi, Lapacian score for feature seection, In NIPS, [7] L. Son A. J. Smoa, A. Gretton, K. M. Borgwardt and J. Bedo, Supervised feature seection via dependence estimation, In ICML (2007), [8] F. Nie, S. Xian Y. Jia, C. Zhang and S. Yan, Trace ratio criterion for feature seection, In AAAI (2008),

28 78 [9] R. Battiti, Using mutua information for seecting features in supervised neura net earnin IEEE Trans. Neura Networks 5(4) (1994), [10] C. M. Bishop, Neura Networks for Pattern Recognition, Oxford University Press, [11] R. Caruana and D. Freita Greedy Attribute Seection, In: Proc. of the 11th Internet. Conf. on Machine Learn., New Brunswick, NJ, USA, (1994), [12] P. Somo, P. Pudi and J. Kitter, Fast branch and bound agorithms for optima feature seection, IEEE Trans. Pattern Ana. Machine Inte. 26(7) (2004), [13] J. H. Yang and V. Honavar, Feature subset seection using a genetic agorithm, IEEE Inte. Systems 13(2) (1998), [14] M. L. Raymer, W. F. Punch, E. D. Goodman, L. A. Kuhn and A. K. Jain, Dimensionaity reduction using genetic agorithms, IEEE Trans. Evout. Comput. 4(2) (2000), [15] B. Bhanu and Y. Lin, Genetic agorithm based feature seection for target detection in SAR images, Image Vision Comput. 21(7) (2003), [16] F. Zhu and S. Guan, Feature seection for moduar GA-based cassification, App. Soft Comput. J. 4(4) (2004), [17] M. Kudo and J. Skansky, Comparison of agorithms that seect features for pattern cassifiers, Pattern Recognition 33(1) (2000), [18] John H. Hoand, Adaptation in Natura and Artificia Systems, [19] K. A. De Jon An Anaysis of the Behavior of a Cass of Generic Adaptive Systems, University of Michigan, [20] B. Foge David, Evoutionary Computation: Toward a New Phiosophy of Machine Inteigence, 2nd Edition, Wiey-IEEE Press, [21] A. Asuncion and D. Newman, UCI Machine Learning Repository, g