An Alternative Approach to the Fuzzifier in Fuzzy Clustering to Obtain Better Clustering Results

Transcription

1 An Alternative Approah to the Fuzziier in Fuzzy Clustering to Obtain Better Clustering Results Frank Klawonn Department o Computer Siene University o Applied Sienes BS/WF Salzdahlumer Str. 46/48 D Wolenbuettel Germany.klawonn@h-wolenbuettel.de Frank Höppner Department o Eonomis University o Applied Sienes BS/WF Robert-Koh-Platz 0-4 D Wolsburg Germany.hoeppner@h-wolenbuettel.de Abstrat The most ommon uzzy lustering algorithms are based on the minimization o an objetive untion that evaluates (uzzy) luster partitions. The generalisation step rom hard lustering to risp lustering requires the introdution o an additional parameter, the so alled uzziier. This uzziier does not only ontrol, how muh lusters may overlap, but has also some undesired onsequenes. For example, data have (almost) always non-zero membership degrees to all lusters, no matter how ar they are away rom a luster. Keywords: Fuzzy lustering, noise lustering, uzziier. Introdution The most ommon uzzy lustering tehniques aim at minimizing an objetive untion whose (main) parameters are the membership degrees and the parameters determining the loalisation as well as the shape o the lusters. Although the extension rom risp to uzzy lustering seems to be an obvious onept, it turns out that to atually obtain membership degrees between zero and one, it is neessary to introdue a so-alled uzziier in uzzy lustering. Usually, the uzziier is simply used to ontrol how muh lusters may overlap. In this paper, we provide a deeper understanding o the underlying onept o the uzziier and derive a more general approah that leads to improved results in uzzy lustering. Setion 2 briely reviews the neessary bakground in objetive untion-based uzzy lustering. The purpose, bakground and the onsequenes o the additional parameter in uzzy lustering the uzziier is examined in setion 3. Based on these onsiderations and on a more general understanding o the uzziier, we propose an improved alternative to the uzziier in setion 4 and outline possible other approahes in the inal onlusions. 2 Objetive Funtion-Based Fuzzy Clustering Fuzzy lustering is suited or inding strutures in data. A data set is divided into a set o lusters and in ontrast to hard lustering a datum is not assigned to a unique luster. In order to handle noisy and ambiguous data, membership degrees o the data to the lusters are omputed. Most uzzy lustering tehniques are designed to optimise an objet untion with onstraints. The most ommon approah is the so alled probabilisti lustering with the objetive untion i j u m i j d i j and the onstraints i onstrained by i () or all j n (2) that should be minimized. It is assumed that the number o lusters is ixed. We will not disuss the issue o determining the number o lusters here and reer or an overview to [2, 5]. The set o data to be lustered is x x n IR p.

2 is the membership degree o datum x j to the ith luster. d i j is some distane measure speiying the distane between datum x j and luster i, or instane the (quadrati) Eulidean distane o x j to the ith luster entre. The parameter m, alled uzziier, ontrols how muh lusters may overlap. The onstraints lead to the name probabilisti lustering, sine in this ase the membership degree an also be interpreted as the probability that x j belongs to luster i. The parameters to be optimised are the membership degrees and the luster parameters that are not given expliitly here. They are hidden in the distanes d i j. Sine this is a non-linear optimisation problem, the most ommon approah to minimize the objetive untion () is to alternatingly optimise either the membership degrees or the luster parameters while onsidering the other parameter set as ixed. In this paper we are not interested in the great variety o luster shapes (spheres, ellipsoids, lines, quadris, ) that an be ound by hoosing suitable luster parameters and an adequate distane untion. (For an overview we reer again to [2, 5].) We only onentrate on the aspet o the membership degrees. Taking the onstraints in equation (2) into aount by Lagrange untions, the minimum o the objetive untion () w.r.t. the membership degrees is obtained at [] (3) m di j k d k j when the luster parameters, i.e. the distane values d i j, are onsidered to be ixed. (I d i j 0 or one or more lusters, we deviate rom (3) and assign x j with membership degree to the or one o the lusters with d i j 0 and hoose 0 or the other lusters i.) I the lusters are represented by simple prototypes v i IR p and the distanes d i j are the squared Eulidean distanes o the data to the orresponding luster prototypes as in the uzzy -means algorithm, the minimum o the objetive untion () w.r.t. the luster prototypes is obtained at [] v i n j um i j x j (4) n j um i j when the membership degrees are onsidered to be ixed. The prototypes are still the luster entres. The luster prototypes are simply luster weighted entres based on the membership degrees. The uzzy lustering sheme using alternatingly equations (3) and (4) is alled uzzy -means algorithm (FCM). As mentioned beore, more ompliated luster shapes an be deteted by introduing additional luster parameters and a modiied distane untion. Our onsiderations apply to all these shemes, but it would lead too ar to disuss them in detail. However, we should mention that there are alternative approahes to uzzy lustering than only probabilisti lustering. Noise lustering [3] maintains the priniple o probabilisti lustering, but an additional noise luster is introdued. All data have a ixed (large) distane to the noise luster. In this way, data that are near the border between two lusters, still have a high membership degree to both lusters as in probabilisti lustering. But data that are ar away rom all lusters will be assigned to the noise luster and have no longer a high membership degree to other lusters. Our investigations and our alternative approah it also peretly to noise lustering. We do not over possibilisti lustering [7] where the probabilisti onstraint is ompletely dropped and an additional term in the objetive untion is introdued to avoid the trivial solution 0 or all i j. However, the aim o possibilisti lustering is atually not to ind the global optimum o the orresponding objetive untion, sine this is obtained, when all lusters are idential [8]. Another approah that emphasizes a probabilisti interpretation in uzzy lustering is desribed in [4] where membership degrees as well as membership probabilities are used or the lustering. In this way, some o the problems o the standard FCM an be avoided as well. However, this approah assumes the use o the Eulidean or a Mahalanobis distane and is not suitable or arbitrary luster shapes as in shell lustering. 3 Eets o the Fuzziier The update equation (3) or the membership degrees derived rom the objetive untion () an lead to undesired or ounterintuitive results, beause zero membership degrees never our (exept in the extremely rare ase, when a data vetor oinides with a luster entre). No matter, how ar away a data vetor is rom

3 to have their omputed luster entres drawn into the diretion o the entre o the data set. These eets an be amended, when a small uzziier is hosen. The prie or this is that we end up more or less with hard lustering again and even neighbouring lusters beome artiiially well separated, although there might be ambiguous data between these lusters. 4 Replaing the Fuzziier Funtion Figure : Clusters with varying density a luster and how well it is overed by another luster, it will still have non-zero membership degrees to all other lusters. Figure shows an undesired side-eet o the probabilisti uzzy lustering approah. There are obviously two lusters. However, the right-hand luster has a muh higher data density than the other one. This single dense luster attrats the other luster prototype so that the prototype o the let-hand luster migrates ompletely into the dense luster. In the igure we have also indiated or whih luster a data vetor has the highest membership degree. Another ounterintuitive eet o probabilisti uzzy lustering ours in the ollowing situation. Assume we have a data set that we have lustered already. Then we add more data to the data set in the orm o a new luster that is ar away rom all other lusters. I we reluster this enlarged data set with one more luster as the original data set, we would expet the same result, exept that the new data are overed by the additional luster, i.e., we would assume that the new, well separated luster has no inluene on the old ones. However, sine we never obtain zero membership degrees, the new data (luster) will inluene the old lusters. This means also that, i we have many lusters, lusters ar away rom the entre o the whole data set tend Viewing the objetive untion () rom a more general point o view, the uzziier arries out a transormation t : 0 0 u u m o the membership degrees. This transormation should be inreasing and t 0 0 and t should hold. This is deinitely satisied by the transormation t u u m. However, there might be better hoies. Let us onsider an arbitrary transormation t satisying the aore mentioned onditions and let us take a loser look whih eet the transormation t has on the objetive untion. Assume, we want to minimize the objetive untion i j t d i j (5) under the onstraints (2) w.r.t. the values, i.e., we onsider the distanes as ixed. Assuming that the transormation t is dierentiable, the onstraints lead to the Lagrange untion L i j t d i j and the partial derivatives n j λ j i L t d i j λ j (6) At a minimum o the objetive untion the partial derivatives must be zero, i.e. λ j t d i j. Sine λ j is independent o i, we must have t d i j t u k j d k j (7) or all i k at a minimum. This atually means that these produts must be balaned during the minimization proess. Considering the standard transormation t u used in uzzy lustering, it yields very small values or u m

4 the derivative near zero and even zero at u 0. This is the reason, why zero membership degrees nearly never our. We thereore propose to use another transormation t that does not assume the value zero or the derivative at zero. In order not to end up with risp membership degrees again, we should make sure that the derivative t is inreasing, but not starting at the value zero at zero. When we onsider a data vetor x and the luster i nearest to x and another luster k urther away rom x, taking (7) into aount, we an see that the quotient t 0 t indiates whih value the quotient d i j d k j (8) must exeed, in order to yield a non-zero membership or the luster k urther away rom x. The larger the value (8) is, the more will the lustering tend to preer risp lusters. Although (8) yields always zero or the standard uzzy lustering with t u u m, we an still use a similar measure, when we replae the derivative at zero in (8) by a value o the derivative at ε 0 near zero. t ε t gets larger or a smaller uzziier m, when we hoose t u u m. A more detailed analysis o this problem an be ound in [6], where also an alternative quadrati transormation t is disussed. In this paper we propose to replae the transormation t by the ollowing one: t α : 0 0 u e α eαu (9) We now derive the update equations or our new lustering approah. We have to minimize the objetive untion i j e α eα d i j (0) under the onstraints (2) and the onstraints 0. Computing the partial derivatives o the Lagrange untion L i j e α eα d i j and solving or we obtain n j λ j i λ α ln j e α () α d i j i 0. Using k:uk j 0 u k j, we an ompute λ j e α ĉ e α α d k j ĉ k:u k j 0 where ĉ is the number o lusters to whih data vetor x j has non-zero membership degrees. Replaing λ j in (), we inally obtain the update equation or αĉ α k:u k j 0 ln dk j d i j (2) We still have to determine whih are zero. Sine we want to minimize the objetive untion (0), we an make the ollowing observation. I 0 and d i j d t j, then at a minimum o the objetive untion, we must have u t j 0 as well. Otherwise we ould redue the value by setting u t j to zero and letting assume the original value o u t j. This implies the ollowing. For a ixed j we an sort the distanes d i j in dereasing order. Without loss o generality let us assume d j d j. I there are zero membership degrees at all, we know that or minimizing the objetive untion the -values with larger distanes have to be zero. (2) does not apply to these -values. Thereore, we have to ind the smallest index i 0 to whih (2) is appliable, i.e. or whih it yields a positive value. For i i 0 we have 0 and or i i 0 the membership degree is omputed aording to (2) with ĉ i 0. When using our modiied algorithm, the update equations or the luster prototype remain the same or instane, in the ase o FCM luster entres as in (4) exept that we have to replae u m i j by t α e α eα In addition to using the slightly more ompliated update equation (2) instead o (3), we also have to sort

5 the distanes to the luster entres or eah data vetor in every iteration step. However, rom the point o view o omputational omplexity, this does not lead to a signiiant derease in perormane, sine we only have to sort as many distanes as we have lusters eah time. And the number o lusters is usually quite small so that we mainly have a linear inrease o the omputational omplexity in the number o data. For reasons o eiieny, we also reommend to ompute (2) in the ollowing way: ĉ αĉ ln d k j αln d i j (3) k:u k j 0 For a ixed data vetor x j only the last term ln d i j varies or the lusters, the rest needs only to be omputed one in eah iteration step. With this modiied algorithm the data set in igure is lustered orretly (or instane with α 0 5). Espeially, when our modiied approah is oupled with noise lustering, most o the undesired eets o uzzy lustering an be avoided and the advantages o a uzzy approah an be maintained. The value α 0 in our approah has a similar eet as the uzziier m in standard uzzy lustering: The smaller α is hosen, the risper the uzzy partition tends to be. This is also obvious rom the quotient (8). We obtain t α 0 e α t α [3] R.N. Davé. Charaterization and Detetion o Noise in Clustering. Pattern Reognition Letters 2 (99), [4] A. Flores-Sintas, J.M. Cadenas, F. Martin. Membership Funtions in the Fuzzy -Means Algorithm. Fuzzy Sets and Systems 0 (999), [5] F. Höppner, F. Klawonn, R. Kruse, T. Runkler. Fuzzy Cluster Analysis. Wiley, Chihester (999). [6] F. Klawonn, F. Höppner. What is Fuzzy About Fuzzy Clustering? Understanding and Improving the Conept o the Fuzziier. In: Proeedings Intelligent Data Analysis Springer, Berlin (to appear). [7] R. Krishnapuram, J. Keller. A Possibilisti Approah to Clustering. IEEE Trans. on Fuzzy Systems (993), [8] H. Timm, C. Borgelt, R. Kruse. A Modiiation to Improve Possibilisti Cluster Analysis. IEEE Intern. Con. on Fuzzy Systems, Honululu (2002) 5 Conlusions We have proposed a new approah to uzzy lustering that overomes the problem in uzzy lustering that all data tend to inluene all lusters. As a uture work, we will extend our approah to more general transormations t other than the exponential one proposed in this paper and the quadrati approah presented in [6]. Reerenes [] J.C. Bezdek. Pattern Reognition with Fuzzy Objetive Funtion Algorithms. Plenum Press, New York (98). [2] J.C. Bezdek, J. Keller, R. Krishnapuram, N.R. Pal. Fuzzy Models and Algorithms or Pattern Reognition and Image Proessing. Kluwer, Boston (999).