IMPROVED FUZZY CLUSTERING METHOD BASED ON INTUITIONISTIC FUZZY PARTICLE SWARM OPTIMIZATION

Transcription

1 Journal of Theoretial and Applied Information Tehnology IMPROVED FUZZY CLUSTERING METHOD BASED ON INTUITIONISTIC FUZZY PARTICLE SWARM OPTIMIZATION V.KUMUTHA, 2 S. PALANIAMMAL D.J. Aademy For Managerial Exellene, Department of Computer Siene, Coimbatore, Sri Krishna College Of Tehnology, Department of Siene & Humanities, Coimbatore, kumuthav@gmail.om, 2 splvlb@yahoo.om ABSTRACT Reent advanes in tehnology has led to huge growth in generating high dimensional data sets by apturing millions of fats in various fields, time phases, loalities and brands. Miroarray data ontains gene expression from thousands of genes (features) from only tens of hundreds of samples. The rih soure of information generated from miroarray experiments often onsist of inomplete and/or inonsistent data. Data mining is a powerful tehnology that automates the proess of disovering hidden patterns. Traditional fuzzy lustering approahes are available whih laks to proess effiiently in ase of inomplete or inonsistent data. It has high influene over the resulting partitions. In this proposed approah, the degree of membership to indeterminay is extended by adopting the onept of generalization of fuzzy logi, whih is known as intuitionisti fuzzy logi. This paper proposes a hybrid approah for lustering high dimensional data set using FCM and Intuitionisti Fuzzy Partile Swarm Optimization (IFPSO) to overome the loal onvergene problem. To find similarity among objets and luster enters intuitionisti based similarity measure is used. Intuitionisti fuzzy partile swarm optimization optimizes the working of the Fuzzy - means algorithm. Experimental results of proposed approah shows better results when ompared with the existing methods. Keywords: Fuzzy Clustering; Intuitionisti Fuzzy; Partile Swarm Optimization; Gene Expression Data; Yeast Data; Degree Of Membership. INTRODUCTION The most ommon popular Data mining tehniques disussed are lustering and lassifiation[4]. Classifiation is a supervised learning method, whereas lustering is an unsupervised learning method. The goal of lustering is to asertain new set of ategories[2]. Clustering tehnique groups objets of similar pattern into one partition. Clustering tehniques are broadly lassified into hard and soft partition [7]. The traditional hard partitioning methods allow one objet to lie in only one luster at a time. The hard partition gives undesirable results, i) while fixing an objet that almost lie between two lusters and ii) plaing an outlier. This adverse situation an be fixed by fuzzy lustering. Fuzzy lustering allows one data item to belong to several lusters onurrently with different membership degrees. The assigning to a partition is determined by the membership degree that lies between 0 and [6]. Fuzzy -means (FCM) is the most ommon fuzzy lustering algorithm. The algorithm uses objetive funtion to measure the desirability of partitions. Fuzzy -means is an effetive algorithm, whereas the random seletions of enter point make iterative proess falling into loal optima solution hene different initializations may lead to different results. Unertainty is one of the major hallenges posed by real-world lustering appliations in the loalization of the feature vetors. In miroarray data, the number of samples is very limited while the volume of genes is very large; suh data sets are very sparse in high-dimensional gene spae. Moreover most of the genes olleted may not neessarily be of interest. Unertainty about whih genes are relevant makes it diffiult to selet informative genes[5]. To handle this problem intuitionisti fuzzy approah is used. Intuitionisti Fuzzy Sets (IFSs)[] are generalized fuzzy sets, whih are useful in oping with the hesitany originating from imperfet or impreise information. Membership and non-membership 8

2 Journal of Theoretial and Applied Information Tehnology value are elements involved in this sets. The degree of membership denotes the validity or trueness of the element to the set, whereas the non-validity of falseness of the element to the set denotes the nonmembership value. Apart from validity and nonvalidity of the element, another element named hesitany or indeterminay or unertainty poses diffiulty in determining the validness of the membership of the element to the group. Reent researh indiates that applying intuitionisti fuzzy sets to high dimensional data provides optimal lustering results. Partile Swarm Optimization (PSO) is a population based searh tehnique that share similar harateristis to Geneti Algorithm. It is an adaptive algorithm based on soial-psyhologial metaphor; a population of individuals adapts by returning stohastially toward previously suessful regions [8]. In this paper the FCM algorithm is ombined with intuitionisti fuzzy partile swarm optimization taking the merits of both to give effiient results. The paper is organized as follows. Setion 2 briefs about the related works. Setion 3 desribes the materials and methods. Setion 4 elaborates about the proposed work. Experimental results on data sets are given in setion 5 and setion 6 onludes the work. 2. RELATED WORK There are many PSO based Fuzzy lustering methods available in the literature. [5] in their proposed work used the distane between the sample and luster enters to distribute the membership in order to meet the onstraints of FCM. The optimum partile has been direted to lose the group in an optimized way. A haoti partile swarm fuzzy lustering algorithm based on haoti partile swarm and gradient method was proposed in [3]. Adaptive inertia weight fator and iterative haoti map with infinite ollapses are used. The method uses haoti PSO to searh the fuzzy lustering model, using the searh apability of fuzzy -means and thereby avoided the loal onvergene problem. The gradient operator is superior over the FCM algorithm. An effiient hybrid method based on fuzzy partile swam optimization (FPSO) and Fuzzy C- Means (FCM) algorithms, to solve the fuzzy lustering problem, espeially for large data sets was presented in [9]. The performane was improved by seeding the initial swarm with the result of the -means algorithm. The experiment indiates that the omputation times and solution quality of FPSO for large datasets was better than FCM. In [0], PSO algorithm and fuzzy methods were ombined to avoid loal peaks and find global optimal solution. This approah uses global searh apaity to overome FCM defiits. It finds optimal loation of lusters enters for input dataset and finally finds the member omponents of eah luster. A hybrid approah, in whih fuzzy -means lustering method and artifiial neural networks were used in fuzzy time series to get more aurate foreasts were presented in [6]. Fuzzifiation step in FCM removes problems aused by partition of disourse of universe and fuzzy relationships defined by artifiial neural networks avoids use of diffiult matrix operations. Two methods for minimizing the reformulated objetive funtions of the fuzzy -means lustering model by partile swarm optimization: PSO V and PSO-U. In PSO V eah partile represents a omponent of a luster enter, and in PSO U eah partile represents an unsaled and unnormalized membership value were presented in [3]. The approah was ompared with alternating optimization and ant olony optimization methods. 3. MATERIALS AND METHODS 3. Fuzzy -means algorithm Instead of assigning an objet to a single luster, the iterative method, Fuzzy C-Means algorithm (FCM) uses the onept of fuzzy membership, where eah objet will have different membership values on eah luster. It partitions set of n objets in R d dimensional [2] spae into ( < < n) O={o,o 2, o n } fuzzy lusters with Z={z,z 2,.z n } luster enters or entroids. The fuzzy lustering of objets is desribed by a fuzzy matrix with n rows and olumns in whih n is the number of data objets and is the number of lusters,, the element in the ith row and jth olumn in, point out the degree of assoiation or membership funtion of the ith objet with the jth luster. The haraters of are as follows: ε [0,] j =, 2,..., () =, j =,2,..., n (2) i= n 0 < < =, 2,... (3) i= 9

3 Journal of Theoretial and Applied Information Tehnology The objetive funtion of FCM algorithm is to minimize the Eq. 4: J n m 2 m i j d = = = m< α (4) where d = o i z j (5) where m (m>) is a salar termed the weighting exponent and ontrols the fuzziness of the resulting lusters and d is the Eulidean distane from objet o i to the luster enter z j. The z j, entroid of the jth luster, is obtained using Eq. (6). z n m i o = i j = n m i = Algorithm. Fuzzy -means (6). Selet m (m>) and initialize the membership funtion values, i=,2, n, j=,2 2. Compute the luster enters Z j, j =,2,..., by using Eq. (6) 3. Compute Eulidian distane, d, i =,2,..., n; j=,2,..., 4. Update the membership funtion, i =, 2... n; j=,2,..., by using below equation = k= d d ik 2 m If not onverged, go to step PSO (7) Partile swarm optimization (PSO) is a population-based stohasti optimization tehnique inspired by bird floking and fish shooling [8] whih is based on iterations/generations. The algorithmi flow in PSO starts with a population of partiles whose positions represent the potential solutions for the studied problem, and veloities are randomly initialized in the searh spae. In eah iteration the searh for optimal position is performed by updating the partile veloities and positions. In eah iteration, the fitness value of eah partile s position is determined using a fitness funtion. The veloity of eah partile is updated using two best positions, personal best position and global best position. The personal best position, pbest, is the best position the partile has visited and gbest is the best position the swarm has visited sine the first time step. A partile s veloity and position are updated as follows. V(t+)=wV(t)+ r (pbest(t)-x(t))+ 2 r 2 (gbest(t)- X(t)); (8) X(t+)=X(t)+V(t+) (9) where, X and V are position and veloity of partile respetively. w is inertia weight, and 2 are positive onstants, alled aeleration oeffiients whih ontrol the influene of pbest and gbest on the searh proess,p is the number of partiles in the swarm, r and r 2 are random values in range [0, ] Fuzzy PSO Algorithm A partile swarm optimization with fuzzy set theory is alled fuzzy partile swarm optimization (FPSO) []. Using fuzzy relation between variables, FPSO redefines the position and veloity of partiles and its also applied for lustering problem. In this method X is the position of partile, the fuzzy relation for the set of data objets O={o,o 2, o n }, to set of lusters enters Z={z,z 2,.z n }an be expressed as follows... X = n... n (0) Here, is the membership funtion of the ith objet with the jth luster with onstraints [0,] =, 2... n =, 2... () i i j j= = =,2... n i (2) therefore it is known that the position matrix of eah partile is the same as fuzzy matrix in FCM algorithm. Also the veloity of eah partile is stated using a matrix with the size n rows and olumns, the elements of whih are in range between - and. The equations (3) and (4) are used for updating the positions and veloities of the partiles based on the matrix. (3) X ( t + ) = X ( t) V ( t + ) (4) After updating the position matrix, it may violate the onstraints given in () and (2) sine it is ompulsory to normalize the position matrix. First all the negative elements in matrix are set to zero. If all elements in a row of the matrix are zero, they need to be reevaluated using series of random numbers within the interval between 0 and, and then the matrix undergoes the following transformation without violating the following onstraints: 0

4 Journal of Theoretial and Applied Information Tehnology... j j j= j= Xnormal = n... n j j = j= j (5) This tehnique uses the following equation as fitness funtion for evaluating the solutions. K f ( X ) = (6) J m Here, K is a onstant and J m is the objetive funtion of FCM algorithm. The smaller is J m, the better is the lustering effet and the higher is the individual fitness f (X). The termination ondition in this method is the maximum number of iterations or no improvement in gbest in a number of iterations. The FCM algorithm is quiker than the FPSO algorithm beause it need not inur as muh of funtion evaluations, but it normally go down into loal optima. FCM algorithm inorporated with FPSO algorithm to form a hybrid lustering algorithm alled FCM-FPSO whih maintains the merits of both FCM and FPSO algorithms. Algorithm 2. Fuzzy PSO Input : Dataset Output : Objetive Values Step. Initialize the parameters inluding population size P,, 2, w and the maximum iterative ount. Step 2. Create a swarm with P partiles (X, pbest, gbest and V are n matries). Step 3.Initialize X, V, pbest for eah partile and gbest for the swarm. Step 4. Calulate the luster enters for eah partile using z n m i o = i j = n m i = (7) Step 5. Calulate the fitness value of eah partile using Eq. (6) Step 6. Calulate pbest for eah partile. Step 7. Calulate gbest for the swarm. Step 8. Update the veloity matrix for eah partile using Eq. (3) Step 9. Update the position matrix for eah partile usingeq. (4) Step 0. If terminating ondition is not met, go to step Intuitionisti Fuzzy Set The Intuitionisti Fuzzy Set (IFS) was defined as an extension of the ordinary Fuzzy Set [2] [4]. As opposed to a fuzzy set in X, given by: A= {( x, A (x)) x ε X } (8) where A (x) [0,] is the membership funtion of the fuzzy set A, an intuitionisti fuzzy set B is given by: B ={ x, B (x)), ν B(x) x ε X} (9) where B (x) [0,] and ν B (x) [0,] are suh that: 0 B (x)), ν B(x) (20) and B (x),ν B (x) [0,] denote degrees of membership and non-membership of x B, respetively. For eah intuitionisti fuzzy set B in X, hesitation margin (or intuitionisti fuzzy index ) of x B is given by: π B (x) = (x ) ν B (x ) (2) whih expresses a hesitation degree of whether x belongs to B or not. It is obvious that 0 π B (x), for eah x X. To desribe an intuitionisti fuzzy set ompletely, it is neessary to use any two funtions from the triplet: membership funtion; non-membership funtion; and hesitation margin. 4. PROPOSED APPROACH 4. Intuitionisti Fuzzy Partile Swarm Optimization (IFPSO) Existing approahes works fine for the data-sets whih are not orrupted with noise but if the dataset is noisy or distorted then it wrongly lassifies noisy pixels beause of its abnormal feature data and results in an inorret membership and improper lustering. The above said problem was also faed by the fuzzy partile swarm optimization approah. To overome the problem of abnormal features that exist among the partile lustering an be overwhelmed by introduing the onept of intuitionisti fuzzy based partile swarm optimization whih is the generalization of fuzzy based partile swarm optimization. In this approah eah partile is onerned not only with the membership funtion but the in deterministi degree is also taken into onsideration for handling the abnormality problem. The abnormality problem arises due to the inonsisteny of the partiles position information.

5 Journal of Theoretial and Applied Information Tehnology Data set Preproessing Applying IFPSO Applying FCM Generating luster patterns Figure : Framework of Proposed Work This proposed approah IFPSO introdues another degree into the onsideration for handling the unertainty problem among the partiles using hesitation degree whih is also known as in deterministi degree.... X = n... n Algorithm 3. IFPSO algorithm (22) Input : Dataset Output : Objetive Values Step. Initialize the parameters inluding population size P,, 2, w and the maximum iterative ount. Step 2. Create a swarm with P partiles (X, pbest, gbest and V are n matries). Step 3.Initialize X, V, pbest for eah partile and gbest for the swarm. Step 4. Calulate the luster enters for eah partile using * n ik xik Z = (23) j k = n * k = ik Step 5. Calulate the fitness value of eah partile using Eq. (6) Step 6. Calulate pbest for eah partile. Step 7. Calulate gbest for the swarm. Step 8. Update the veloity matrix for eah partile using Eq. (3) Step 9. Update the position matrix for eah partile using Eq. (4) Step 0. If terminating ondition is not met, go to step Hybrid Fuzzy C Means and Intuitionisti Fuzzy Partile Swarm Optimization The FCM algorithm is quiker than the IFPSO algorithms beause it uses few funtion evaluations, but it normally go down into loal optima. Rather, FCM algorithm inorporated with IFPSO algorithm to form a hybrid lustering algorithm alled FCM- IFPSO whih maintains the merits of both FCM and IFPSO algorithms. In FCM-IFPSO algorithm, FCM is applied to the partiles in the swarm every number of iterations/generations suh that the fitness value of eah partile is improved. The algorithm 4 illustrate hybrid FCM-IFPSO. Algorithm 4. FCM-IFPSO algorithm Input : Dataset Output : Objetive Values Step. Initialize the parameters of IFPSO and FCM inluding population size P,, 2, w, and m. Step 2. Create a swarm with P partiles (X, pbest, gbest and V are n matries). Step 3. Initialize X, V, pbest for eah partile and gbest for the swarm Step 4. IFPSO algorithm 4. Calulate the luster enters for eah partile using by (23) 4.2 Calulate the fitness value of eah partile using by (6) 4.3 Calulate pbest for eah partile. 4.4 Calulate gbest for the swarm. 4.5 Update the veloity matrix for eah partile using by (3) 4.6 Update the position matrix for eah partile using by (4) 4.7 If terminating ondition is not met, go to step 4 Step 5. FCM algorithm 5. Compute the luster enters j z, j =,2,...,, by using (3) 5.2 Compute Eulidean distane, d, i =,2,..., n; j=,2,..., 5.3 Update the membership funtion, i =,2 n = j=,2,..., 2 k= d d ik m (24) 5.4 Calulate pbest for eah partile. 5.5 Calulate gbest for the swarm If FCM terminating ondition is not met, go to step 5. 2

6 Journal of Theoretial and Applied Information Tehnology Step 6. If FCM-IFPSO terminating ondition is not met, go to Step EXPERIMENTAL RESULTS The algorithms disussed in the previous setion have been implemented using MATLAB. For evaluating the performane of the proposed work, four different benhmark datasets are taken into onsideration. 5. Parameter settings The optimized performane of the IFPSO and FCM-IFPSO, fine tuning has been exeuted and best values for their parameters are hosen. The experimental results based on these algorithms ahieve best under the following settings:,2, the value is3.0 - population is 2, and weight values are: the minimum of weight value is 0.2 and maximum of value is 0.8 and weighting exponent omponent m value is 2 whih is ommon to all the algorithms. The FPSO and IFPSO terminating ondition is till it reahes the maximum iteration value when the algorithm annot improve the gbest in 500 onseutive iterations. Also the FCM IFPSO terminating ondition is met, when the algorithm annot improve the gbest in 2 onseutive iterations Figure 2:Results for Iris data set Worst Average Best The experimental results of over 00 independent runs for FCM and 0 independent runs for IFPSO and FCM IFPSO are shown in the figures 2,3,4 & 5. Figure 2 shows the best, average and worst ases of objetive funtion values obtained applying over iris data set for FCM, FPSO, IFPSO and FCM- IFPSO algorithms. The proposed hybrid approah FCM-IFPSO method depits better results over the other existing methods Figure 3:Results for yeast data set Figure 3 shows the best, average and worst ases of objetive funtion values obtained applying over yeast data set for FCM, FPSO, IFPSO and FCM- IFPSO algorithms. Worst Average Best Figure 4:Results for olon aner data set Figure 4 shows the best, average and worst ases of objetive funtion values obtained applying over olon aner data set for FCM, FPSO, IFPSO and FCM-IFPSO algorithms. The proposed approah depits better results when ompared to the existing approahes and it an flee from loal optima. Figure 5shows the best, average and worst ases of objetive funtion values obtained applying over leukemia data set for FCM, FPSO, IFPSO and FCM-IFPSO algorithms. The proposed approah depits better results when ompared to the existing approahes Worst Average Best 3

7 Journal of Theoretial and Applied Information Tehnology Worst Average Best Figure 5:Results for leukemia data set Table desribes the number of samples and genes of the miroarray dataset namely, yeast, olon aner and leukemia datasets. Table 2 desribes the number of attributes and instanes of the benhmark iris dataset. Table. Gene expression dataset Data set No. of samples No. of genes Yeast Colon aner Leukemia Table 2. Benhmark dataset Data set No. of attributes No. of instanes Iris 4 50 Figure 6: Average umulative hange for 29 generations Figure 7:Mean and best sore value for 29 generations The above figure shows the various generations runs of proposed FCM-IFPSO algorithm with the mean and best sore value. The figure 6 shows Average umulative hange in value of the fitness funtion over 50 generations less than e-006 and onstraint violation less than e-006, after 29 generations. Figure 7 shows the mean and best sore over 29 generations ( Final best point: [ ]). 6. CONCLUSION Data mining poses a vital issue in lustering of high dimensional gene expression data. Several algorithms have been addressed in the literature for handling suh data. The fuzzy -means algorithm is sensitive to initialization and is easily trapped in loal optima. On the other hand, the fuzzy partile swarm algorithm is a global stohasti tool whih ould be implemented and applied easily to solve various funtion optimization problems. Both of them fail to handle the unertainty ondition and they left the onept of indeterminay when there is a presene of vagueness or inompleteness in lustering dataset. In this paper, an optimization approah is put forward in order to overome the shortomings of the fuzzy -means and fuzzy partile swarm optimization. The proposed method removes the indeterminay thereby a good performane of the desired lusters is obtained. Instead of just onsidering the membership value of eah objet in the luster, the proposed approah takes into aount the in- deterministi value as an important fator in the ase of inompleteness in the dataset lustering. Experimental results over well-known data sets, Iris, Yeast, Colon aner, and Leukemia, 4

8 Journal of Theoretial and Applied Information Tehnology show that the proposed hybrid method is profiient and an reveal very enouraging results in term of quality of solution found. A new hybrid method ombining fuzzy -means and Intuitionisti fuzzy Partile swarm optimization algorithm have been applied suessfully for real world datasets. The omputational results show that the performane of the proposed algorithm is better than the other existing algorithms. Our future work would be direted towards Intuitionisti fuzzy lustering ombined with intuitionisti fuzzy partile swarm optimization in order to ahieve better results for miroarray data. REFRENCES: [] Atanassov, K., Intuitionisti fuzzy sets: past, present and future, Proeedings of the 3rd Conferene of the European Soiety for Fuzzy Logi and Tehnology, 2003, pp [2] Atanassov, K.T., Intuitionisti Fuzzy Sets Theory and Appliations, Studies in Fuzziness and Soft Computing, 999, Phisia-Verlag. [3] Chaoshun, L., Jianzhong, Z., Pangao, K., Jian, X., A novel haoti partile swarm optimization based fuzzy lustering algorithm, Neuro omputing, Vol. 83, No.5, 202, pp [4] Chen, M.S., Han, J., Yu, P.S., Data mining: An Overview from Database Perspetive, IEEE Trans. Knowl. Data Eng., 996. [5] Daxin, J. C., Tang A. Z., Cluster Analysis for Gene Expression Data: A Survey, IEEE Trans. Knowl. Data Eng., Vol. 6, No., 2004, pp [6] Erol, E., Cagdas, H. A., Ufuk, Y., Fuzzy time series foreasting with a novel hybrid approah ombining fuzzy -means and neural networks, Expert Systems with Appliations, Vol. 40, 203, pp. 54. [7] Jain, A.K., Mutry, M.N., Flynn, P.J., Data Clustering: A Review, ACM Computing Surveys, Vol. 3, No. 3, 999, pp [8] Kennedy, J., Eberhart, R., Swarm Intelligene, Morgan Kaufmann Publishers, In., 200, San Franiso, CA. [9] Mehdizadeh, E., Sadi-Nezhad, S., Tavakkoli- Moghaddam, R., Optimization Of Fuzzy Clustering Criteria By A Hybrid PSO And Fuzzy C-Means Clustering Algorithm, Iranian Journal of Fuzzy Systems, Vol. 5, No. 3, 2008, pp. -4. [0] Mir, M., Tadayon Tabrizi, G., Improving Data Clustering Using Fuzzy Logi and PSO Algorithm, 20 th Iranian Conferene on Eletrial Engineering, 202, Tehran, Iran [] Pang, W., Wang, K., Zhou, C., Dong, L. Fuzzy disrete partile swarm optimization for solving traveling salesman problem, Proeedings of the fourth international onferene on omputer and information tehnology, IEEE CS Press 2004, pp [2] Rokah. L., Maimon, O., Clustering methods, Data Mining and Knowledge Disovery Handbook, Springer, New York, 2005, pp [3] Runkler, T. A., Katz, C., Fuzzy lustering by partile swarm optimization, IEEE international onferene on fuzzy systems, 2006, pp [4]Shanthi, S., Murali Bhaskaran, V. Intuitionisti Fuzzy C-Means and Deision Tree Approah for Breast Caner Detetion and Classifiation, European Journal of Sientifi Researh, Vol.66 No.3, 20, pp [5] Qiang, N., Xinjian, H., An Improved Fuzzy C- means Clustering Algorithm based on PSO, Journal of Software, Vol. 6, No. 5, 20. [6] Zadeh, L.A. Fuzzy sets, Information Control, Vol. 8, 965, pp