FUZZY C-MEANS ALGORITHM BASED ON PRETREATMENT OF SIMILARITY RELATIONTP

Transcription

1 Dynamics of Continuous, Discrete and Impulsive Systems Series B: Applications & Algorithms 14 (2007) Copyright c 2007 Watam Press FUZZY C-MEANS ALGORITHM BASED ON PRETREATMENT OF SIMILARITY RELATIONTP Zhiping Jia 1, Chenghui Zhang 2 and Rupeng Sun 1 1 School of Computer Science and Technology Shandong University, Jinan, ,China 2 School of Control Science and Engineering Shandong University, Jinan, ,China Abstract. In order to make up some deficiencies of the fuzzy c-means clustering algorithm, a new FCM algorithm based on pretreatment of similarity relation between samples is proposed in the paper, which is utilized to estimate the fuzzy clustering centers and the weight coefficient of samples effecting on the fuzzy clustering centers during iteration process. The new FCM algorithm makes the clustering quicker and more accurate. Finally, a simulation experiment is given in the paper to demonstrate the new FCM algorithm can avoid the limitations of the traditional FCM algorithm and the improvement is very effective. Keywords. Fuzzy Set; Clustering Analysis; Fuzzy C-Means Algorithm; Similarity Relation. AMS (MOS) subject classification: 62H30. 1 Introduction Clustering is a process of dividing some datum into several categories, and the data object in the same category has more similarity. In the course of the clustering analysis, the fuzzy theory was used to instruct the clustering process, and so a new clustering algorithm-fuzzy c-means clustering algorithm(fcm) was proposed. The FCM algorithm classifies data objects into some different categories, and it can express the undefined degree of each object belonging to each category. Therefore, the FCM algorithm can reflect the actual instances more externally and gradually it is becoming the mainstream of clustering analysis and application([1,2,3,4]). Many clustering methods based on the fuzzy theory have been proposed, such as the transitive closure clustering algorithm based on the fuzzy equal relationship, the largest tree algorithm based on fuzzy graph theory, and the clustering algorithm based on objective function and etc. The FCM algorithm is discussed in this paper, which belongs to an algorithm based on objective function. This clustering algorithm has a good real-time property and is suitable for the situation with a great deal of datum. Meanwhile, it is simply designed and easy to be realized on computer. So it has become the most popular clustering algorithm at present. Deep researches on the

2 104 Z. Jia,C. Zhang and R. Sun traditional FCM algorithm are firstly done in the paper, based on which improvement to remedy its deficiency is given. The experimental result shows that the improved algorithm has better performances not only in validity but also in efficiency([5]). 2 Fuzzy C-Means Algorithm With the developing of the fuzzy theory, the fuzzy c-means clustering algorithm based on Ruspini fuzzy clustering theory was proposed in 1980s. It is an extended hard c-means algorithm belonging to the traditional c-means algorithm with its goal to find the optimum fuzzy clustering to minimize the value of the objective function ([6,7,8]). Let X = {x 1, x 2, x 3,..., x n } R m stand for a given datum set; n is the total number of samples; a sample x i includes m features, as like x i = {x i1, x i2, x i3,..., x im };the sample set X is divided into c(2 c n)categories.u = (u ij ) n c denotes a fuzzy c partition matrix of the sample set X,and an element u ij represents the relative degree between the ith sample and the jth category. The fuzzy partition of X subjects to the following restrictions. 1. i, j u ij [0, 1]; c 2. i j=1 u ij = 1; (1) 3. j i=1 u ij > 0. In above formula (1), condition 1 qualifies the elements in fuzzy partition matrix range in [0, 1]; condition 2 shows that the sum of its relative degrees for any one sample in datum set X should be one; condition 3 shows that for anyone of c fuzzy categories, there exists some samples whose relative degrees between themselves are greater than zero to belong to this category. The condition 3 qualifies each c fuzzy set not to be null. Let V = {v 1, v 2, v 3,..., v c } be a clustering center vector set of c fuzzy sets, where v j R S is the clustering center vector of the jth category. J.C.Bezdek defines the objective function of fuzzy c partition as follows: J m (U, V ) = i=1 j=1 (u ij ) m (d ij ) 2 (2) Where d ij = x i v j denotes the distance v j between the sample x i and the clustering center of the jth category. Here d ij can be defined by different distance formulas according to actual needs. m is called to be fuzzy weight exponent or smoothing parameter, and it decides the fuzzy degree of fuzzy partition matrix U. Pal and Bezdek concluded the reasonable value range[1.5, 2.5] of m from the clustering validity. The objective function

3 Improved Fuzzy C-Means Algorithm 105 J m (U, V ) is defined to be the sum of the square of the weight distances between each sample data and all the clustering centers. In order to obtain the optimization of the fuzzy clustering, J.C.Bezdek reduced the fuzzy c partition problem to be how to seek the minimum of the objective function given by formula(2) under the restriction of formula(1). Because every column of the fuzzy matrix U is independent, thus it comes the corresponding change as follows: min{j m (U, V )} = min{ i=1 j=1 (u ij ) m (d ij ) 2 } = min{ (u ij ) m (d ij ) 2 } The above formula can be resolved via the Lagrange multiplier method and changed formula is as follows: F = j=1 i=1 (u ij ) m (d ij ) 2 + λ( u ij 1) Where the one level necessary conditions of the optimization are as follows: j=1 j=1 F λ = { (u ij ) 1} = 0 (3-1) j=1 F u kl = [m(u kl ) m 1 (d kl ) 2 λ] = 0 (3-2) Formula (4) can be concluded from formula (3-2): λ u kl = [ m(d kl ) 2 ] 1 m 1 (4) Further it can draw forth as follows according to formula(4) and formula(3-1). u tl = t=1 t=1 ( λ m ) 1 1 m 1 )[ (d tl ) 2 ] 1 m 1 λ = ( m ) 1 1 m 1 ){ [ (d tl ) 2 ] 1 m 1 } = 1 t=1 ( λ m ) 1 m 1 ) = 1 c t=1 [ 1 1 (d tl ) ] 2 m 1 The above two formulas are added to formula(4) and it can conclude: 1 u kl = c t=1 [ d kl 2 d tl ] m 1

4 106 Z. Jia,C. Zhang and R. Sun Considering d ij may be zero, it should be discussed in two different aspects. Then the u ij to minimize the objective function J m (U, V ) is as follows: u ij = 1/ c k=1 ( d ij d kj ) 2 m 1 d ij 0; u ij = 1, u ik = 0 d ij, 1 k c and k j. In formula(5), when d ij = 0, i.e. the distance between sample i and the center vector of the jth category is zero, sample i is considered to belong to the jth category, whose relative degree is one and the possibility of belonging to other categories is zero. Using the same computation method, v j can be acquired to minimize the objective function J m (U, V ). and further: i=1 v j J m (U, V ) = 0 (u jk ) m [(x i x j ) T ((x i x j )] = 0 v j v j (u jk ) m (x i x j ) = 0 i=1 and ultimately it can be concluded as follows: (5) v j = i=1 (u ij) m x i i=1 (u ij) m (6) Fuzzy partition matrix U and clustering center vector set V can be computed according to formula(5) and (6). The FCM algorithm is a continuous iterative process to minimize the objective function J m (U, V ). Formula(5) and (6) are used in each iterative computation to obtain more accurate fuzzy partition matrix U and clustering center V and smaller objective function J m (U, V ). After some iterative times, the value of the objective function will converge at a certain infinitesimal point and then the iteration will stop. At the same time, the optimization clustering of fuzzy matrix U and clustering center V will also be gained. 3 Deficiencies and Improvement of FCM Algorithm The traditional FCM algorithm has two distinct deficiencies.

5 Improved Fuzzy C-Means Algorithm 107 (a)initially, the FCM algorithm selects clustering centers randomly, and this random selection may result in the FCM algorithm to converge at a local infinitesimal point rather than to achieve the holistic optimization. (b)it can be seen from formula(6) that all samples have the same effect on the selection of FCM clustering centers. Therefore, a few abnormal samples may have worse impacts on the selection of FCM clustering centers. In order to solve the deficiency(a), the simulation anneal algorithm was adopted in document[9] to optimize the all and the one, which needed too much time and was short of real-time; a new method based on the limited distance among categories was proposed in document[10] to pre-cluster the samples, but with its shortage that there was no good standards to select the limited distance. An improved method was given in document[11] to remedy the second deficiency, but it is too complex and not suitable for the applications with a great deal of datum. A fuzzy c-means algorithm based on pretreatment of similarity relation (SR-FCM) is proposed in this paper. Firstly it needs to establish fuzzy similarity relation matrix for data set X, and computes the average similarity relations between each sample to get intensity of sample distribution. It uses the average similarity coefficient as effect weight for clustering centers to shrivel abnormal samples influence for clustering centers. According to similarity relation matrix, pre-partition is done, where the samples having bigger similarity coefficient are divided into c categories (c is the set number of categories). The average value of all samples attributes in one category is used as its initial clustering centers, which avoids the problem of selecting clustering centers randomly in the FCM algorithm. By using the pre-treated samples effect weight and the 5 initial clustering centers closing to the final one, the FCM iteration operations can notably reduce iteration times and avoid algorithm trapping easily into local minimum. Detail steps of the algorithm are as follows: (1)Establish fuzzy similarly relation matrix S n n for the dataset. According to three principles that should be obeyed during the process of constructing fuzzy similarly relations, that is correctness, invariability and discriminable, and the similarity matrix is constructed by using the method of absolute values reciprocal, which is shown as follow(7): { 1 i = j r ij = P a m k=1 x ik x jk i j. (7) where r ij is similarity coefficient of sample i and j, m is amount of sample index attributes,and a is adjustment constant of r ij s range. (2)Compute average similarity coefficient s i between every sample and others. Computational Method is expression (8): s i = 1 n 1 j=1,j i r ij (8)

6 108 Z. Jia,C. Zhang and R. Sun According to average similarity coefficient s i, Band the effect weight w i of every sample for the clustering center is computed using expression (9): w i = a s i k=1 s k (9) where s i is average similarity coefficient of sample i and a is adjustment constant of w i. (3)Pre-partition dataset into c categories in terms of fuzzy similarity relations. Set t = n/c(/ is exact division operation), which represents average amount of samples in one category. Rules of partition are as follows. 1 Find out sample i max with the maximum of average similarity coefficient, then select first t 1 samples (excluding i max ) which have the maximum similarity coefficient with sample i max. Remove all samples whose similarity coefficient with i max is less than s i (to make sure correct partition categories with less samples in the context of sampling distribution odds), then let the left p samples (p t 1) and i max belong to one category. The clustering center vector of this category v is computed using the formula as expression(10): v = 1 p+1 x k (10) p Update new category amount c = c 1. If c = 0, forward to step 3 ; Otherwise remove similarity coefficient of the samples in categories of step 1 from matrix S n n to get a new similarity matrix S n n (here: n equals n p in step 1 ).Now the average amount of samples in one category is (n p) c. Go back to step 1 using the updated S n n, c, t. 3 End up pre-partition of categories based on fuzzy similarity relations, and get vector sets of initial clustering center V. (4)Modify expression (6) and get expression (6 ) as follows: k=1 v j = i=1 w iu m ij x i i=1 w iu m ij (6 ) where w i is effect weight of sample i for clustering center. (5)For given dataset X,and amount of category c, establish smoothing factor m, iteration convergence threshold valueand maximum of iteration times T max. Together with effect weight w i got from step (2) and initial clustering center V got from step (3), use expressions (5) and (6 ) to do iterate according to rules of the FCM algorithm, and obtain the final clustering results. 4 Experimental Results In order to validate the performance of the improved algorithm, FCM and SR- FCM algorithms are respectively used to cluster the datasets X 20 7 below(20

7 Improved Fuzzy C-Means Algorithm 109 samples in total, 7 attributes per sample) in MatLab X 20 7 = In this experiment, FCM and SR-FCM algorithms are respectively and repeatedly run 50 times under the conditions of clustering amount c = 3, smoothing factor m = 2, convergence threshold 7 value of iteration ε = Statistics of clustering results, times of iteration and objective function are listed as follows (Assuming clustering results are correct when objective function has optimum solution): Table 1: Contrast of results Algorithm Cluster results Average times Average value of iteration of objective function FCM correct 40 times Incorrect 10 times SR-FCM Correct:50 times Figure.1 illustrates variational graph of iteration times in the first 20 times for FCM and SR-FCM algorithms. As we can see, iteration times of the FCM algorithm is rather unstable, even varies every time. Although it is less than the SR-FCM algorithm between times, when that happens it always traps into local minimum of objective function and causes clustering failure (e.g.figure.1 Situations of FuncVal equals and ). Even if clustering is successful, variation span of iteration times is rather large. In

8 110 Z. Jia,C. Zhang and R. Sun the SR-FCM algorithm, however, because of the preprocess using similarly relations, definition of initial clustering centers and the weight effect given to every sample for clustering centers, the successful rate and iteration times of algorithm are rather stable, and iteration times is one forth less than the average value of the FCM algorithm. Since considering samples weight effect for clustering centers, the convergence of the SR-FCM algorithm is a little bigger than the optimum solution. Figure 1: Variation graph of iteration times of algorithms 5 Conclusion Aiming at two deficiencies of the conventional FCM algorithm, which are, on one hand, the random selection of initial clustering centers that may cause unable to get globally optimum solution, and on the other hand, the same weight effect of all samples for clustering centers during iterations that may make determination of clustering centers more difficult because of abnormal samples, similar relation matrix is used in this paper to preprocesses all samples at first, then the FCM algorithm is used to implement clustering. This method takes less cost to get the stabilization of the whole algorithm and decreases the computational complexity. In the end, experimental results further validate the efficiency of improved algorithm proposed in this paper. 6 Acknowledgements This article is supported by National Natural Science Foundation of P.R.China ( ) and Key Scientific and Technological Project of Shandong Province

9 Improved Fuzzy C-Means Algorithm 111 ( ),P.R.China. 7 References [1] Xin-bo Gao, Fuzzy cluster analysis and its applications, Xian Electronic Technology University Press, [2] Jian Yu, Hou-kuan Huang, A new weighting fuzzy c-means algorithm, Fuzzy Systems, 2003.FUZZ 03.The 12th IEEE International Conference on,2, (2003) [3] Ming-Chuan Hung,Don-Lin Yang, An efficient fuzzy c-means clustering algorithm, Data Mining, 2001, ICDM 2001, Proceedings IEEE International Conference on,12, (2001) [4] Mehmed Kantardzic, Data mining concepts, models, methods, and algorithms, TsingHua Unversity Press, Beijing, [5] R.N.Dave,S. Sen, Robust fuzzy clustering of relational data,fuzzy Systems, 10(6), (2002) [6] J.M. Leski,Generalized weighted conditional fuzzy clustering, Fuzzy Systems, 11(6), (2003) [7] N.R.Pal,K.Pal,J.M.Keller,A possibilistic fuzzy c-means clustering algorithm,fuzzy Systems, 13(4), (2005) [8] J.F.Kolen,T.Hutecheson,Reducing the time complexity of the fuzzy c-means algorithm,fuzzy Systems, 10(2), (2002) [9] S.Z.Selim,K.Alsultan,A simulated annealing algorithm for the clustering problem, Pattern Recognition,24(10),(1991) [10] Xin-Bo Zhang,Two phrases fuzzy c-means clustering algorithm,journal of Circuits and Systems, 10(2), (2005) [11] A.Bensaid,L.OHall,J.C.Bezdek,Validity-guided (re)clustering with applications to image segmentation,fuzzy Systems,4(2), (1996) Received November 2005; revised June journal@monotone.uwaterloo.ca journal/