Feature Selection using Modified Imperialist Competitive Algorithm

Size: px

Start display at page:

Download "Feature Selection using Modified Imperialist Competitive Algorithm"

Sharleen Shelton
5 years ago
Views:

Feature Selection using Modified Imperialist Competitive Algorithm S. J. Mousavirad Department of Computer and Electrical Engineering University of Kashan Kashan, Iran jalalmoosavirad@gmail.

1 Feature Selection using Modified Imperialist Competitive Algorithm S. J. Mousavirad Department of Computer and Electrical Engineering University of Kashan Kashan, Iran Abstract Feature selection process is one of the main steps in data mining and nowledge discovery. Feature selection is a process to remove redundant and irreverent features without reducing the classification accuracy. This paper tries to select the best features set using imperialist competitive algorithm. Imperialist competitive algorithm is a novel population based algorithm which is inspired sociopolitical process of imperialist competition. In this paper, a modified imperialist competitive algorithm is presented and then this proposed algorithm is applied to feature selection process. To verify the effectiveness of the proposed approach, experiments carried out on some datasets. Results showed the features set selected by the imperialist competitive algorithm provide the better classification performance compared to the other methods. Keywords-feature selection, impeirliast compeitive algorithm, population based algorithm, data mining, nowledge discovery I. INTRODUCTION In the process of data classification, set of features may include redundant, noisy or even irrelevant information. This information may even reduce the classification accuracy. Select of the appropriate features or feature selection is one of the main steps in data mining and nowledge discovery. Finding the appropriate features with N number of features need to evaluate 2 N possible subset. This approach is an exhausting and time consuming method. There are other methods based on heuristic and random search that try to reduce computational complexity. Population based optimization algorithms such as genetic algorithm (GA)[1-3], particle swarm optimization (PSO)[4-5], and ant colony algorithm (ACO)[6-7] have been considered as effective wrapper feature selection approach. Imperialist competitive algorithm(ica) is a new population based optimization algorithm that is inspired by imperialist competition[8]. It has been used extensively to solve different inds of optimization problem. T.Ninam et al., [9] presented a novel algorithm that is based on -means and ICA. The experimented results showed that this algorithm presented better results from ACO, PSO, simulated annealing (SA), GA, tabu search (TA), and honey bee mating optimization (HBMO). In another research, M.T. Mahmoudi et al.,[10] evolved artificial neural networ using grammar encoding and imperialist competitive algorithm. The proposed method was compared with other five methods. In all four datasets, the proposed method outperforms its competitors. S.J. Mousavirad et al., [11] tried to select the H. Ebrahimpour-Komleh* Department of Computer and Electrical Engineering University of Kashan Kashan, Iran ebrahimpour@gmail.com {Corresponding Author} best features set for classification of rice varieties based on image of bul samples using ICA. The Results showed the ICA provides the better performance compared to GA in dataset obtained from rice images but efficiency of this approach is not evaluated on standard benchmars. The objective of this paper is as follows: Presentation of a discrete modified ICA, Apply the proposed version of ICA to feature selection problem, Evaluation of the proposed approach on standard benchmars. The rest of this paper is organized as follows. First, a brief description of ICA has been demonstrated. Then the proposed approach for feature selection using modified ICA has been described. In the next section, the experiment and results are presented. Finally, several conclusion have been concluded. II. THE BASIC PRINCIPAL OF ORIGINAL IMPERIALIST COMPETITIVE ALGORITHM ICA is a new population based optimization algorithm that has recently been introduced for dealing with different inds of optimization problem[8, 12]. This algorithm is based on sociopolitical process of imperialist competition. Similar other population based algorithms, it starts with an initial population called countries. Most powerful countries are selected as imperialists and the rest form the colonies of these imperialists. Figure 1 shows the initial population of each empire. Figure 1. Generating the initial empires[8] After dividing colonies between imperialists, each colony moves toward their related imperialist countries. This movement is based on assimilation policy. In ICA, revolution is a sudden change in a part of socio-political characteristics. It increases the exploration of the algorithm. The total power of an empire depends on both the power of the imperialist country and the power of its

colonies. Imperialist competition among empires forms the basis of ICA. According to the imperialist competition, the most powerful empires tend to increase their power while wea empires collapse.

2 colonies. Imperialist competition among empires forms the basis of ICA. According to the imperialist competition, the most powerful empires tend to increase their power while wea empires collapse. In imperialist competition, all empires try to tae possession of colonies of other empires and control them. This is modeled by just picing some of the weaest colony of the weaest empires and maing a competition among all empires to posses these colonies[8]. Figure 2 shows a picture of the modeled imperialist competition. Start Initialize the empires Assimilate colonies End Stop condition satisfied Revolve some colonies Unite similar empires Eliminate the empires Is there a colony in empire which has lower cost than that of the imperialist? Yes No No Yes Is there an empire with no colonies? Figure 2.Imperialist competition. The more powerful an empire is, the more liely it will possess the weaest colony of the weaest empire[8]. At last after an iterative process, the most powerful empire will tae the possession of other empires and wins the competition. Figure 3 shows the flowchart of the ICA. III. THE PROPOSED APPROACH In this section, a new feature selection approach using modified ICA algorithm is presented. The steps of the proposed approach are demonstrated in details in the following subsections. A. Genertaing Initial empires In a N dimensional problem, a country is a 1 N array. In the proposed approach, each country is a string of binary numbers. When value of a cell from country is 1, the feature is selected and when it is 0, the feature is not selected. Figure 4 Shows the feature representation as a country from initial empires. The cost value of a country is defined as the classification accuracy of KNN. The algorithm starts with N initial countries in the population size, and the best of them,, is chosen as the imperialist. N imp To form the initial empires, the colonies are divided among imperialists based on their power. For this purpose, the normalized cost of an imperialist is defined by C c max( c ) i where c is the cost of th imperialist and C is its normalized cost. Then, the power of each imperialist as follows: pop Figure 3. The flowchart of original ICA P C Nimp On the other hand, the normalized power of an imperialsit shows the approximate number of colonies that should be possessed by the imperialsits. Thus the initial number of colonies of the th empire will be where Exchange the position of that the imperialist and the colony empire and Compute the total cost of all empires.. i1 C N. C. round{ p. N } col NC is the initial number of colonies of the th Ncol B. Assimilation is the number of all colonies. After creating the initial empire, colonies start moving toward their imperialist which is based on assimilation policy. The original version of ICA operates on continuous optimization problems. Since feature selection is a discrete problem, in the current paper a new approach for assimilation is presented. This operator is as below in Figure 5. i Imperialist competition

3 F 1 F 2 F 3 F 4 F n-2 F n-1 F n Country Features Subset= {F 2,F 4,,F n-2,f n} Figure 4. An example of feature representation using ICA For each imperialist and their colonies do Calculate city bloc distance (D) between colony and their imperialist Create a binary string(s) of length N with initial value of zero Assign 1 to some array cells proportional to D. Copy the cells from the imperialist correspond to location of the 1 s in the S to the same position in the colony. End Figure 5. the proposed approach for assimilation C. Revolution Revolution in this algorithm caused a country to abrupt changing its socio-political characteristics. This operator increases the exploration of the algorithm and prevents the early convergence of countries to local minimum. In ICA, the revolution rate defines the percentage of colonies in each empire that undergoes the revolution process. This rate is a constant number for each empire in the original version; but in the real world, The more the powerful empires, the less the revolution probability. Thus revolution rate is intended to dynamically as follows: r c Where and are positive constants, c is complement of the total power of an empire (in this problem, c is classification error), and r is revolution rate. Plots of r versus c for various values of are shown in figure 6. Figure 6. plots of the equation r ( 1 in all cases) c for various values of D. Imperialist updating While moving toward the imperialist, a colony may reach to a position with lower cost than that of imperialists. In this case, the colony and the imperialist change the position. E. Computing total cost of an empire Total power an empire is depends on both the power of the imperialist country and power of its colonies. This fact is modeled as follows: T. C. Cost{ imperialist } mean{ Cost{ colonies of empire }} where TC.. is the total cost of the th empire and is a positive number which is considered to be less than 1. F. Imperialist competition In the imperialist competition, all empires try to tae possession of colonies of other empires. This struggle gradually results a decrease in the power of weaer empires and an increase in the power of more powerful empires. This is modeled by just picing some of the weaest colonies of the weaest empire and maing a competition among all empire to possess these colonies. To start the competition, the possession probability of each empire based on its total power is calculated. The normalized total cost of an empire is obtained by where N. T. C. T. C. max{ T. C. } TC and N. T. C.. i i are the total cost and normalized total cost of the th empire, respectively. The possession probability of each empire is given by P emp N. T. C. Nimp i1 N. T. C. Next, roulette wheel method[13] is used for assigning the mentioned colonies to empire. G. Elimination the powerless empires An empire collapsed when it loses all of its colony. In this case, imperialist considered as a colony and it assigned to the other empire. H. Stopping criteria After a while, all the empires except the most powerful one will collapse and all the colonies will be under the central of this unique empire. In such a situation, all the colonies will have the same cost and same position. In such a condition, we put on end to the imperialist competition and stop the algorithm. IV. RESULTS AND DISCUSSIONS In this section, results of experiments to evaluate the effectiveness of the proposed approach and to compare i

4 with other methods are presented. To evaluate the proposed approach, classification accuracy is computed using -nearest neighbor algorithm[14] after the feature selection procedure. Five well nown datasets have been selected as follows: Iris: in this dataset each class refers to a type of iris plant. One class is linearly separable from the other 2; the latter are NOT linearly separable from each other[15]. Wine: it includes data from a chemical analysis of wine grown in the same region in Italy but derived from three different cultivars[15]. Pima Indians Diabetes: it includes Pima Indians diabetes analysis that belongs to classes of healthy and diabetics. Glass identification: in this dataset, each class refers to whether the glass are a type of float glass or not. Breast cancer: Features in this dataset are computed from a digitized image of a fine needle aspirate (FNA) of a breast mass. They describe characteristics of the cell nuclei present in the image[15]. Table 1 shows characteristics of the used datasets. Some of selected datasets contain missing values. A method based on nearest neighbor algorithm for replacing missing data with substituted values was applied[16]. In this method, missing values substitute with the corresponding feature value from the nearest neighbor instance. Nearest neighbor instance is the nearest instance in Euclidean distance. In order to obtain comparative results, fold cross validation method[14] has been used. The average K results from this method was computed to produce a single estimator. The proposed algorithm was tested with different values. The results showed that the best performance is achieved when parameters shown in table 2 are used. An example of the process of the imperialist competition based feature selection searching for optimal solution is given in Figure 7 and 8. It can be seen that the classification error are decreased which it is indicating the convergence of the proposed algorithm. Figure 7 shows the best and average cost for each iteration. Figure 8 presents the evolution of the search for best number of features. Table 3 shows the result of ICA based feature selection. The results for the table 3 represented the average of 10 fold in fold cross validation method. Results of the proposed approach was compared with three feature selection method: Sequential forward selection (SFS)[17], Sequential bacward selection (BFS)[17], and genetic based feature selection(ga-fs).according to the table 3, feature selection techniques has been showed a considerable increase in classification accuracy compared to when all feature set is used. In addition, ICA based feature selection creates higher accuracy compared with other algorithms tested in all datasets. A B C D Figure 7. Best and mean cost for each iteration in A) wine, B)glass identification, C) iris, D) pima Indians diabetes, and E) breast cancer datasets E

5 Table 1. Description of the datasets used. NO. Dataset used Number of features Number of instance Number of class Missing value 1 Wine No 2 Iris No 3 Glass identification No 4 Pima Yes 5 Breast cancer Yes Table 2. Parameter setting for ICA based feature selection Datasets N pop N imp Max Decades Wine Iris Pima Indians Diabetes Glass Identification Breast cancer A B C D Figure 8. Best number of feature for each iteration in A) wine B) glass identification C) iris D) pima Indians diabetes, and E) breast cancer datasets Table 3. Classification results using the proposed approach No. of the Accuracy Dataset Name original features KNN Without FS* FFS BFS GA-FS ICA-FS Wine Iris Glass identification Pima Breast cancer * Feature selection E

6 I. CONCLUSION AND FUTURE WORK In this paper, a new approach feature selection based on imperialist competitive algorithm (ICA) was presented. In the proposed approach, features are encoded to binary string as country in ICA. The ICA based feature selection was evaluated on five well nown datasets. The obtained results indicate the high performance to find the superior features leads to high classification accuracy. The proposed algorithm can be combined with other population based algorithms in future wor. References [1] J. Yang and V. Honavar, "Feature subset selection using a genetic algorithm," in Feature extraction, construction and selection, ed: Springer, 1998, pp [2] H. Uğuz, "A two-stage feature selection method for text categorization by using information gain, principal component analysis and genetic algorithm," Knowledge-Based Systems, vol. 24, pp , [3] R. Leardi, "Application of a genetic algorithm to feature selection under full validation conditions and to outlier detection," Journal of Chemometrics, vol. 8, pp , [4] X. Wang, J. Yang, X. Teng, W. Xia, and R. Jensen, "Feature selection based on rough sets and particle swarm optimization," Pattern Recognition Letters, vol. 28, pp , [5] A. Unler and A. Murat, "A discrete particle swarm optimization method for feature selection in binary classification problems," European Journal of Operational Research, vol. 206, pp , [6] M. H. Aghdam, N. Ghasem-Aghaee, and M. E. Basiri, "Text feature selection using ant colony optimization," Expert Systems with Applications, vol. 36, pp , [7] A.-A. Ahmed, "Feature subset selection using ant colony optimization," [8] E. Atashpaz-Gargari and C. Lucas, "Imperialist competitive algorithm: an algorithm for optimization inspired by imperialistic competition," in Evolutionary Computation, CEC IEEE Congress on, 2007, pp [9] T. Ninam, E. Taherian Fard, N. Pourjafarian, and A. Rousta, "An efficient hybrid algorithm based on modified imperialist competitive algorithm and K- means for data clustering," Engineering Applications of Artificial Intelligence, vol. 24, pp , [10] M. T. Mahmoudi, F. Taghiyareh, N. Forouzideh, and C. Lucas, "Evolving artificial neural networ structure using grammar encoding and colonial competitive algorithm," Neural Computing and Applications, pp. 1-16, [11] S. MousaviRad, F. A. Tab, and K. Mollazade, "Application of Imperialist Competitive Algorithm for Feature Selection: A Case Study on Bul Rice Classification," International Journal of Computer Applications, vol. 40, [12] E. A. Gargari, F. Hashemzadeh, R. Rajabioun, and C. Lucas, "Colonial competitive algorithm: a novel approach for PID controller design in MIMO distillation column process," International Journal of Intelligent Computing and Cybernetics, vol. 1, pp , [13] A. E. Eiben and J. E. Smith, Introduction to evolutionary computing vol. 2: Springer Berlin, [14] C. M. Bishop and N. M. Nasrabadi, Pattern recognition and machine learning vol. 1: springer New Yor, [15] A. Asuncion and D. J. Newman, "UCI machine learning repository," ed, [16] T. Hastie, R. Tibshirani, G. Sherloc, M. Eisen, P. Brown, and D. Botstein, "Imputing missing data for gene expression arrays," ed: Stanford University Statistics Department Technical report, [17] J. Kittler, "Feature selection and extraction," Handboo of pattern recognition and image processing, pp , 1986.

AN EFFICIENT COST FUNCTION FOR IMPERIALIST COMPETITIVE ALGORITHM TO FIND BEST CLUSTERS

AN EFFICIENT COST FUNCTION FOR IMPERIALIST COMPETITIVE ALGORITHM TO FIND BEST CLUSTERS 1 MOJGAN GHANAVATI, 2 MOHAMAD REZA GHOLAMIAN, 3 BEHROUZ MINAEI, 4 MEHRAN DAVOUDI 2 Professor, Iran University of Science