Fuzzy Clustering in Parallel Universes

Transcription

1 Fuzzy Clustering in Parallel Universes Bern Wisweel an Michael R. Berthol ALTANA-Chair for Bioinformatics an Information Mining Department of Computer an Information Science, University of Konstanz Konstanz, Germany Abstract We present an extension of the fuzzy c-means algorithm, which operates simultaneously on ifferent feature spaces so-calle parallel universes an also incorporates noise etection. The metho assigns membership values of patterns to ifferent universes, which are then aopte throughout the training. This leas to better clustering results since patterns not contributing to clustering in a universe are completely or partially) ignore. The metho also uses an auxiliary universe to capture patterns that o not contribute to any of the clusters in the real universes an therefore liely represent noise. The outcome of the algorithm are clusters istribute over ifferent parallel universes, each moeling a particular, potentially overlapping, subset of the ata an a set of patterns etecte as noise. One potential target application of the propose metho is biological ata analysis where ifferent escriptors for molecules are available but none of them by itself shows global satisfactory preiction results. Key wors: Fuzzy Clustering, Objective Function, Multiple Descriptor Spaces, Noise Hanling Introuction In recent years, researchers have wore extensively in the fiel of cluster analysis, which has resulte in a wie range of fuzzy) clustering algorithms [9,]. Most of the methos assume the ata to be given in a single mostly numeric) feature space. In some applications, however, it is common to have multiple representations of the ata available. Such applications inclue biological ata analysis, in which, e. g. molecular similarity can be efine in various ways. Fingerprints are the most commonly use similarity measure. A fingerprint in aress: {wisweel,berthol}@inf.uni-onstanz.e Bern Wisweel an Michael R. Berthol). Preprint submitte to Elsevier Science 3 October 25

2 a molecular sense is usually a binary vector, whereby each bit inicates the presence or absence of a molecular feature. The similarity of two compouns can be expresse base on their bit vectors using the Tanimoto coefficient for example. Other escriptors encoe numerical features erive from 3D maps, incorporating the molecular size an shape, hyrophilic an hyrophobic regions quantification, surface charge istribution, etc. [6]. Further similarities involve the comparison of chemical graphs, inter-atomic istances, an molecular fiel escriptors. However, it has been shown that often a single escriptor fails to show satisfactory preiction results [6]. Other application omains inclue web mining where a ocument can be escribe base on its content an on anchor texts of hyperlins pointing to it [4]. 3D objects as use in CAD-catalogues, virtual reality applications, meicine an many other omains can be escribe, for instance, by various so-calle feature vectors, i. e. vector of scalars whose carinalities can easily reach couple of hunres. Feature vectors can rely on ifferent statistics of the 3D object, projection methos, volumetric representations obtaine by iscretizing the object s surface, 2D images, or topological matchings. Bustos et al. [5] provie a survey of feature-base similarity measures of 3D objects. In the following we enote these multiple representations, i. e. ifferent escriptor spaces, as Parallel Universes [4], each of which having representations of all objects of the ata set. The challenge that we are facing here is to tae avantage of the information encoe in the ifferent universes to fin clusters that resie in one or more universes each moeling one particular subset of the ata. In this paper, we evelop an extene fuzzy c-means FCM) algorithm [] with noise etection that is applicable to parallel universes, by assigning membership values from objects to universes. The optimization of the objective function is similar to the original FCM but also inclues the learning of the membership values to compute the impact of objects to universes. In the next section, we iscuss in more etail the concept of parallel universes; section 3 presents relate wor. We formulate our new objective function in section 4, introuce the clustering algorithm in section 5 an illustrate its usefulness with some numeric examples in section 6. 2 Parallel Universes We consier parallel universes to be a set of feature spaces for a given set of objects. Each object is assigne a representation in each single universe. Typically, parallel universes encoe ifferent properties of the ata an thus lea to ifferent measures of similarity. For instance, similarity of molecular com- 2

3 pouns can be base on surface charge istribution or a specific fingerprint representation.) Note, ue to these iniviual measurements they can also show ifferent structural information an therefore exhibit istinctive clustering. This property iffers from the problem setting in the so-calle Multi-View Clustering [3] where a single universe, i. e. view, suffices for learning but the aim is on bining ifferent views to improve the classification accuracy an/or accelerating the learning process. As it often causes confusion, we want to emphasize the ifference of the concept of parallel universes to feature selection methos [2], feature transformation such as principle component analysis an singular value ecomposition), an subspace clustering [3,8], whose problem efinitions soun similar at first but are very ifferent from what we iscuss here. Feature selection methos attempt to iscover attributes in a ata set that are most relevant to the tas at han. Subspace clustering is an extension of feature selection that sees to ientify ifferent subspaces, i. e. subsets of input features, for the same ataset. These algorithms become particularly useful when ealing with high-imensional ata, where often, many imensions are irrelevant an can mas existing clusters in noise. The main goal of such algorithms is therefore to uncover subsets of attributes subspaces), on which subsets of the ata are self-similar, i. e. buil subspace-clusters, whereas the clustering in parallel universes is given the efinition of semantically meaningful universes along with representations of all ata in them an the goal is to exploit this information. The objective for our problem efinition is on ientifying clusters locate in ifferent universes whereby each cluster moels a subset of the ata base on some unerlying property. Since stanar clustering techniques are not able to cope with parallel universes, one coul either restrict the analysis to a single universe at a time or efine a escriptor space comprising all universes. However, using only one particular universe omits information encoe in the other representations an the construction of a joint feature space an the erivation of an appropriate istance measure are cumbersome an require great care as it can introuce artifacts or hie an loose clusters that were apparent in a single universe. 3 Relate Wor Clustering in parallel universes is a relatively new fiel of research an was first mentione in [4]. In [], the DBSCAN algorithm is extene an applie to parallel universes. DBSCAN uses the notion of ense regions by means of core objects, i. e. objects that have a minimum number of objects in their ɛ-) neighborhoo. A cluster is then efine as a set of connecte) ense regions. The authors exten this concept in two ifferent ways: They efine an object 3

4 as a neighbor of a core object if it is in the ɛ-neighborhoo of this core object either ) in any of the representations or 2) in all of them. The cluster size is finally etermine through appropriate values of ɛ an. Case ) seems rather wea, having objects in one cluster even though they might not be similar in any of the representational feature spaces. Case 2), in comparison, is very conservative since it oes not reveal local clusters, i. e. subsets of the ata that only group in a single universe. However, the results in [] are promising. Another clustering scheme calle Collaborative fuzzy clustering is base on the FCM algorithm an was introuce in [5]. The author proposes an architecture in which objects escribe in parallel universes can be processe together with the objective of fining structures that are common to all universes. Clustering is carrie out by applying the c-means algorithm to all universes iniviually an then by exchanging information from the local clustering results base on the partitioning matrices. Note, the objective function, as introuce in [5], assumes the same number of clusters in each universe an, moreover, a global orer on the clusters which is very restrictive ue to the ranom initialization of FCM. A supervise clustering technique for parallel universes was given in [4]. It focuses on a moel for a particular minor) class of interest by constructing local neighborhoo histograms, so-calle Neighborgrams for each object of interest in each universe. The algorithm assigns a quality value to each Neighborgram an greeily inclues the best Neighborgram, no matter from which universe it stems, in the global preiction moel. Objects that are covere by this Neighborgram are finally remove from consieration in a sequential covering manner. This process is repeate until the global moel has sufficient preictive power. Although the algorithm is powerful to moel a minority class, it suffers from computational complexity on larger ata sets. Blum an Mitchell [4] introuce co-training as a semi-supervise proceure whereby two ifferent hypotheses are traine on two istinct representations an then bootstrap each other. In particular they consier the problem of classifying web pages base on the ocument itself an on anchor texts of inboun hyperlins. They require a conitional inepenence of both universes an state that each representation shoul suffice for learning if enough labele ata were available. The benefit of their strategy is that inexpensive) unlabele ata augment the expensive) labele ata by using the preiction in one universe to support the ecision maing in the other. Other relate wor inclues reinforcement clustering [8] an extensions of partitioning methos such as -Means, -Meois, an EM an hierarchical, agglomerative methos, all in [3]. 4

5 4 Objective Functions In this section, we introuce all necessary notation, review the FCM [,7] algorithm an formulate two new objective functions that are suitable to be use for parallel universes. The first one is a generic function that, similar to the stanar FCM, has no explicit noise hanling an therefore forces a cluster membership preiction for each pattern while the secon objective function also incorporates noise etection an, hence, allows patterns to not participate to any cluster. The technical etails, i. e. the erivation of the objective functions, can be foun in the appenix. In the following, we consier U, u U, parallel universe, each having representational feature vectors for all objects x u) i = x u) i,,..., x u i,a,... x u i,a u ) with A u inicating the imensionality of the u-th universe. We epict the overall number of objects as T, i T. We are intereste in ientifying K u clusters in universe u. We further assume appropriate efinitions of istance functions for each universe u) w u), xu) u) i where w = w u),,..., wu),a,... wu),a u ) enotes the -th prototype in the u-th universe. We confine ourselves to the Eucliean istance in the following. In general, there are no restrictions to the istance metrics other than ifferentiability. In particular, they o not nee to be of the same type in all universes. This is important to note, since we can use the propose algorithm in the same feature space, i. e. x u ) i = x u 2) i for some u an u 2, but ifferent istance measures in these universes. 4. Objective Function with No Noise Detection The stanar FCM algorithm relies on one feature space only an minimizes the accumulate sum of istances between patterns x i an cluster centers w, weighte by the egree of membership to which a pattern belongs to a cluster. Note that we omit the subscript u here, as we consier only one universe: T K J m = vi, m w, x i. ) i= = The coefficient m, ) is a fuzzyfication parameter, an v i, the respective value from the partition matrix, i. e. the egree to which pattern x i belongs to cluster. 5

6 This function is subject to minimization uner the constraint K i : v i, =, 2) = requiring that the coverage of any pattern i nees to accumulate to. The above objective function assumes all cluster caniates to be locate in the same feature space an is therefore not irectly applicable to parallel universes. To overcome this, we introuce a matrix z i,u ) i T, u U encoing the membership of patterns to universes. A value z i,u close to enotes a strong contribution of pattern x i to the clustering in universe u, an a smaller value, a respectively lesser egree. The new objective function is given by J m,m = T U i= u= K u z i,u = v u) i, u) w u), xu) i 3) Parameter m, ) allows analogous to m) to have impact on the fuzzyfication of z i,u : The larger m, the more equal the istribution of z i,u, giving each pattern an equal impact to all universes. A value close to will strengthen the composition of z i,u an assign high values to universes where a pattern shows goo clustering behavior an small values to those where it oes not. Note, we now have U, u U, ifferent partition matrices v u) ) i, to i T, K u assign membership egrees of objects to cluster prototypes in each universe. As in the stanar FCM algorithm, the objective function has to fulfill sie constraints. The coverage of a pattern among the partitions in each universe must accumulate to : i, u : K u = v u) i, =. 4) This is similar to the constraint of the single universe FCM in 2) an is require for each universe iniviually. Aitionally, the membership of a pattern to ifferent universes z i,u has to satisfy stanar requirements for membership egrees: it must accumulate to for each object consiering all universes an must be in the unit interval, i. e. U i : z i,u =. 5) u= 6

7 The minimization is one with respect to the parameters v u) i,, z i,u, an w u). The erivation of objective function 3) can be foun in the appenix, the final upate equations are given by A.2), A.7), an A.4). 4.2 Objective Function with Noise Detection The objective function as introuce in the previous section has one major rawbac: Patterns that o not contribute to any of the clusters in any universe still have a great impact on the cluster formation as the cluster memberships for each iniviual pattern nee to sum to one. This is not avantageous since ata sets in many real worl applications, if not all, contain outliers or noisy patterns. Particularly in the presente application omain it may happen that certain structural properties of the ata are not capture by any of the given semantically meaningful!) universes an therefore this portion of the ata appears to be noise. The ientification of these patterns is important for two reasons: First, as note above, these patterns influence the cluster formation an can lea to istorte clusters. Seconly, noise patterns may lea to insights on which properties of the unerlying ata are not well moele by any of the universe efinitions an therefore give hints as to what nees to be aresse when efining new universes or similarity measures. In orer to incorporate noise etection we nee to exten our objective function such that it also allows the explicit notion of noise. We aopt an extension introuce by Davé [7], which wors on the single universe FCM. The objective function accoring to Davé is given by: T T K J m = vi, m w, x i K + noise v i,. 6) i= = i= = This equation is similar to ) except for the last term. It serves as a noise cluster; all objects have a fixe, user-efine istance noise to this noise cluster. Objects that are not close to any cluster center w can therefore be etecte as noise. The constraint 2ust be soften: K i : v i,, 7) = requiring that the coverage of any pattern i nees to accumulate to at most the remainer to represents the membership to the noise cluster). Similar to the last term in 6), we a a new term to our new objective function 3) whose role is to localize the noise an place it in a single auxiliary 7

8 universe: T J m,m = U i= u= K u z i,u = T + noise i= v u) i, u) w u) U z i,u. u=, xu) i 8) By assigning patterns to this noise universe, we eclare them to be outliers in the ata set. The parameter noise reflects the fixe istance between a virtual cluster in the noise universe an all ata points. Hence, if the minimum istance between a ata point an any cluster in one of the universes is greater than noise, the pattern is labele as noise. The optimization splits into three parts: optimization of the partition values v u) i, for each universe; etermining the membership egrees of patterns to universes z i,u an finally the aaption of the center vectors of the cluster representatives w u). The upate equations of these parameters are given as follows. For the partition values v i,, we get v u) i, = K u u) = u) w u), xu) i w u), xu) i m. 9) Note, this equation is inepenent of the values z i,u an is therefore ientical to the upate expression in the single universe FCM. The optimization with respect to z i,u yiels z i,u = U ū= Ku v u) = i, K ū = v ū) i, u) ū) w u), xu) i w ū), xū) i + noise m, ) an finally the upate equation for the aaption of the prototype vectors w u) is of the form w u) = ) i= z i,u u u) v i, x i ) T i= z i,u u. ) v i, T 8

9 Thus, the upate of the prototypes epens not only on the partitioning value v u) i,, i. e. the egree to which pattern i belongs to cluster in universe u, but also to z i,u representing the membership egrees of patterns to the current universe of interest. Patterns with larger values z i,u will contribute more to the aaption of the prototype vectors, while patterns with a smaller egree accoringly to a lesser extent. Equippe with these upate equations, we can introuce the overall clustering scheme in the next section. 5 Clustering algorithm Similar to the stanar FCM algorithm, clustering is carrie out in an iterative manner, involving three steps: ) Upate of the partition matrices v u) i, ) 2) Upate of the membership egrees z i,u ) 3) Upate of the prototypes w u) ) More precisely, the clustering proceure is given as: ) Given: Input pattern set escribe in U parallel universes: x u) i, i T, u U. 2) Select: A set of istance metrics u),, an the number of clusters for each universe K u, u U, efine parameter m an m. 3) Initialize: Partition parameters v u) i, with ranom values an the cluster prototypes by rawing samples from the ata. Assign equal weights to all membership egrees z i,u =. U 4) Train: 5) Repeat 6) Upate partitioning values v u) i, accoring to 9) 7) Upate membership egrees z i,u accoring to ) 8) Compute prototypes w u) i using ) 9) until a termination criterion has been satisfie. The algorithm starts with a given set of universe efinitions an the specification of the istance metrics to be use. Also, the number of clusters in each universe nees to be efine in avance. The membership egrees z i,u are initialize with equal weight line 3)), thus having the same impact on 9

10 all universes. The optimization phase in line 5) to 9) is in comparison to the stanar FCM algorithm extene by the optimization of the universe membership egrees, line 7). The possibilities for the termination criterion in line 9) are manifol, as is also the case in the stanar FCM. One can stop after a certain number of iterations or use the change of the value of the objective function 3) between two successive iterations as stopping criteria. There are also more sophisticate approaches, for instance the change to the partition matrices uring optimization. Just lie the FCM algorithm, this metho suffers from the fact that the user has to specify the number of prototypes to be foun. Furthermore, our approach even requires the efinition of cluster counts per universe. There are numerous approaches to suggest the number of clusters in the case of the stanar FCM, [9,7,2] to name but a few. Although we have not yet stuie their applicability to our problem efinition we o believe that some of them can be aapte naturally to be use in our context as well. 6 Experimental Results In orer to emonstrate the propose approach, we generate synthetic ata sets with ifferent numbers of parallel universes. For simplicity an in orer to visualize the results we restricte the size of a universe to 2 imensions an generate 2 Gaussian istribute clusters per universe. We use 4 patterns to buil groupings by assigning each object to one of the universes an rawing its features in that universe accoring to the istribution of the cluster ranomly picing one of the two). The features of that object in the other universes were rawn from a uniform istribution, i. e. they liely represent noise in these universes unless they fall, by chance, into one of the clusters). Figure shows an example ata set with three universes. The top figures show only the objects that were generate to cluster in the respective universe. The bottom figures show all patterns, i. e. also the patterns that cluster in the other universes. They efine the reference clustering. In this example, when looing solely at one universe, about 2/3 of the ata oes not contribute to clustering an therefore are noise in that universe. To compare the results we applie the FCM algorithm [] to the joint feature space of all universes an set the number of esire clusters to the overall number of generate clusters. The cluster membership ecision for the single-universe FCM is base on the highest value of the partition values, i. e. the cluster to a pattern i is etermine by = arg max K {v i, }.

11 Universe Universe 2 Universe Fig.. Three universes of a synthetic ata set. The top figures show only objects that were generate within the respective universe using two clusters per universe). The bottom figures show all patterns; note that most of them i. e. the ones from the other two universes), are noise in this particular universe. For clarification we use ifferent shapes for objects that originate from ifferent universes. Fig. 2. Clustering quality for 3 ifferent ata sets. The number of universes ranges from 2 to 4 universes. Note how the cluster quality of the joint feature space rops sharply whereas the parallel universe approach seems less affecte. An overall ecline of cluster quality is to be expecte since the number of clusters to be etecte increases. When the universe information is taen into account, a cluster ecision is base on the memberships to universes z i,u an memberships to clusters v u) i,. The winning universe is etermine by ū = arg max u U {z i,u } an the corresponing cluster in ū is calculate as = arg max Kū{v ū) i, }. We use the following quality measure to evaluate the clustering outcome an

12 compare it to the reference clustering []: Q K C) = C i C C i T entropy KC i )), where K is the reference clustering, i. e. the clusters as generate, C the clustering to evaluate, an entropy K C i ) the entropy of cluster C i with respect to K. This function is if C equals K an if all clusters are completely mixe such that they all contain an equal fraction of the clusters in K or all points are preicte to be noise. Thus, the higher the value, the better the clustering. Figure 2 summarizes the quality values for 3 experiments. The number of universes ranges from 2 to 4. The left bar for each experiment in figure 2 shows the quality value when using the new objective function as introuce in section 4., i. e. with incorporating the nowlege of parallel universes but no explicit noise etection. The right bar shows the quality value when applying the stanar FCM to the joint feature space. Clearly, for this ata set, our algorithm taes avantage of the information encoe in ifferent universes an ientifies the major parts of the original clusters much better. In a secon experiment, we artificially ae 6 noise patterns in orer to test the ability of noise etection. The patterns features were rawn from a ranom istribution in all universes, hence, they liely represent noise. We then applie our new algorithm in parallel universes with an without noise etection an compare the results to the extene FCM algorithm with noise etection [7] applie to the joint feature space. The crisp cluster membership was base on the egree of membership of a pattern to the auxiliary noise cluster: Whenever this value was higher than the maximum membership to any of the real clusters, the pattern was labele as noise, i. e. max K {v i, } < K = v i,. Similarly, in case of the algorithm in parallel universes, a pattern is etecte as noise when the egree of membership to the auxiliary noise universe is higher than to any real universe, max u U {z i,u } < U u= z i,u. Figure 3 summarizes the quality values for this experiment. Clearly, when allowing the algorithm to label patterns as noise, the quality value increases. However, when applying FCM to the joint feature space right most bar), most of the ata was labele as noise. It was noticeable, that the noise etection 3% of the ata was generate ranomly such that it shoul not cluster in any universe) ecrease when there were more universes, since the number of clusters an therefore the chance to hit one of them when rawing the features of a noise object increase for this artifical ata. As a result, the ifference in quality between the clustering algorithm which allows noise etection an the clustering algorithm that forces a cluster preiction eclines The overall number of patterns is therefore 2 patterns. 2

13 Fig. 3. Results on the artificial ataset with 6 patterns being noise, i. e. not contributing to any cluster. When using our new algorithms the two left bars for each experiment) the quality values are always greater than the value for the FCM with noise cluster [7] applie to the joint feature space. when there are more universes. This effect occurs no matter how carefully the noise istance parameter noise is chosen. However, if we have only few universes, the ifference is quite obvious. Figure 4 visually emonstrates the clusters from the foregoing example as they are etermine by the fuzzy c-means algorithm in parallel universes: The top figures show the outcome when using the objective function introuce in section 4., i. e. without noise etection, an the bottom figures show the clusters when allowing noise etection section 4.2). The figures show only the patterns that are part of clusters in the respective universe; other patterns, either covere by clusters in the remaining universes or etecte as noise, are filtere out. Note how the clusters in the top figures are sprea an contain patterns that obviously o not mae much sense for this clustering. This is ue to the fact that the algorithm is not allowe to iscar such patterns as noise: each pattern must be assigne to a cluster. The bottom figures, in comparison, show the clusters as well-shape, ense regions. Patterns that in the top figures istort the clusters are not inclue here. It shows nicely that the algorithm oes not force a cluster preiction an will recognize these patterns as being noise. We chose this in of ata generation to test the ability to etect clusters that are blurre by noise. Particularly in biological ata analysis it is common to have noisy ata for which ifferent escriptors are available an each by itself exhibits only little clustering power. 3

14 Universe Universe 2 Universe Fig. 4. The top figures show the clusters as they are foun when applying the algorithm with no noise etection. The bottom figures show the clusters foun by the algorithm using noise etection. While the clusters in the top figures contain patterns that o not appear natural for this clustering, the clustering with noise etection reveals those patterns an buils up clear groupings. 7 Conclusion We consiere the problem of unsupervise clustering in parallel universes, i. e. problems where multiple representations are available for each object. We evelope an extension of the fuzzy c-means algorithm with noise etection that uses membership egrees to moel the impact of objects to the clustering in a particular universe. By incorporating these membership values into the objective function, we were able to erive upate equations which minimize the objective function with respect to these values, the partition matrices, an the prototype center vectors. In orer to moel the concept of noise, i. e. patterns that apparently are not containe in any of the clusters in any universe, we introuce an auxiliary noise universe that has one single cluster to which all objects have a fixe, pre-efine istance. Patterns that are not covere by any of the clusters are assigne a high membership to this universe an can therefore be reveale as noise. The clustering algorithm itself wors in an iterative manner similar to the stanar FCM using the above upate equations to compute a localinimum. The result are clusters locate in ifferent parallel universes, each moeling only a subset of the overall ata an ignoring ata that o not contribute to clustering in a universe. We emonstrate that the algorithm performs well on a synthetic ata set an nicely exploits the information of having ifferent universes. Further stuies will concentrate on the overlap of clusters. The propose ob- 4

15 jective function rewars clusters that only occur in one universe. Objects that cluster well in more than one universe coul possibly be ientifie when having balance membership values to the universes but very unbalance partitioning values for the cluster memberships within these particular universes. Other stuies will continue to focus on the applicability of the propose metho to real worl ata an heuristics that ajust the number of clusters per universe. Acnowlegments This wor was partially supporte by DFG Research Training Group GK-42 Explorative Analysis an Visualization of Large Information Spaces. A Appenix In orer to compute a minimum of the objective function 3) with respect to 4) an 5), we exploit a Lagrange technique to merge the constraine part of the optimization problem with the unconstraine one. Note, we sip the extra notation of the noise universe in 8); it can be seen as an aitional universe, i. e. the number of universe is U +, that has one cluster to which all patterns have a fixe istance of noise. The erivation can then be applie as follows. It leas to a new objective function F i F i = U K u z i,u u) v i, u= = U + λ K u u u= = u) w u) ) v u) i, + λ, xu) i A.) U z i,u ), u= which we minimize iniviually for each pattern x i. The parameters λ an λ u, u U, enote the Lagrange multipliers to tae 4) an 5) into account. The necessary conitions leaing to local minima of F i rea as F i z i,u =, F i v u) i, =, F i λ =, F i λ u u U, K u. =, A.2) 5

16 In the following we will erive upate equations for the z an v parameters. Evaluating the first erivative of the equations in A.2) yiels the expression F i = m K u z i,u z i,u = v u) i, u) w u), xu) i λ =, an hence z i,u = λ m ) m Ku = v u) i, u) w u), xu) i m. A.3) We can rewrite the above equation λ m ) m Ku = zi,u = v u) i, u) w u), xu) i ) m. A.4) From the erivative of F i w. r. t. λ in A.2), it follows F i λ = U U z i,u =, u= u= z i,u = A.5) which returns the normalization conition as in 5). Using the formula for z i,u in A.3) an integrating it into expression A.5) we compute U λ u= λ m m ) m ) m U u= Ku = Ku = v u) i, v u) i, u) u) w u) w u), xu) i, xu) i m m = =. A.6) We mae use of A.4) an substitute λ m in A.6). Note, we use ū as parameter inex of the sum to aress the fact that it covers all universes, whereas u enotes the current universe of interest. It follows m ) Ku = z i,u = v u) i, u) w u), xu) i ) m 6

17 U ū= Kū = v ū) i, ū) w ū), xū) i m, which can be simplifie to U = z i,u ū= Ku = Kū = ) u v i, u) w u), xu) i ) ū v i, ū) ū) w, xū) i m, an returns an immeiate upate expression for the membership z i,u of pattern i to universe u: z i,u = u) Ku U v u) = i, ū= K ū v ū) = i, ū) w u), xu) i w ū), xū) i m. A.7) Analogous to the calculations above we can erive the upate equation for value v u) i, which represents the partitioning value of pattern i to cluster in universe u. From A.2) it follows F i v u) i, = z i,u m v u) i, u) w u), xu) i λ u =, an thus λ u m z i,u v u) i, = ) m = v u) i, m z i,u u) w u) λ u u) w u), xu) i m, xu) i ) m, A.8). A.9) Zeroing the erivative of F i w. r. t. λ u will result in conition 4), ensuring that the partition values sum to, i. e. F i λ u = K u = v u) i, =. A.) We use A.8) an A.) to come up with 7

18 K u = = m z i,u = λ u m z i,u ) m λ u u) u) w K u = m,, xu) i m, xu) i u) w u). A.) Equation A.9) allows us to replace the first multiplier in A.). We will use the notation to point out that the sum in A.) consiers all partitions in a universe an to enote one particular cluster coming from A.8), = v u) i, u) w u) = v u) i, K u =, xu) i u) w u) u) w u) ) m Finally, the upate rule for v u) i, K u =, xu) m i, xu) i arises as: u) w u) m, xu) i v u) i, = K u u) = u) w u), xu) i w u), xu) i m. A.2) For the sae of completeness we also erive the upate rules for the cluster prototypes w u). We confine ourselves to the Eucliean istance here, assuming the ata is normalize 2 : u) w u), A u xu) i = a= w u),a xu) i,a, A.3) with A u the number of imensions in universe u an w u),a the value of the prototype in imension a. x u) i,a is the value of the a-th attribute of pattern i in universe u, respectively. The necessary conition for a minimum of the objective function 3) is of the form u) w J =. Using the Eucliean istance as given in A.3) we obtain 2 The erivation of the upates using other than the Eucliean istance wors in a similar manner. 8

19 T i= J m,m w u),a! = ) 2 z i,u u u) v i, w,a ) xu) i,a = w u),a T i= ) T z i,u u v i, = i= T w u),a = ) z i,u u u) v i, x i,a ) i= z i,u u u) v i, x i,a ) T i= z i,u u. A.4) v i, References [] J. C. Beze. Pattern Recognition with Fuzzy Objective Function Algorithms. Plenum Press, New Yor, 98. [2] James C. Beze an Richar J. Hathaway. VAT: a tool for visual assessment of cluster) tenency. In Proceeings of the 22 International Joint Conference on Neural Networs IJCNN 2), pages , 22. [3] Steffen Bicel an Tobias Scheffer. Multi-view clustering. In Proceeings of the Fourth IEEE International Conference on Data Mining ICDM 4), pages 9 26, 24. [4] Avrim Blum an Tom Mitchell. Combining labele an unlabele ata with co-training. In Proceeings of the eleventh annual Conference on Computational Learning Theory COLT 98), pages 92. ACM Press, 998. [5] Benjamin Bustos, Daniel A. Keim, Dietmar Saupe, Tobias Schrec, an Dejan V. Vranić. An experimental effectiveness comparison of methos for 3D similarity search. International Journal on Digital Libraries, Special issue on Multimeia Contents an Management in Digital Libraries, 25. To appear. [6] G. Cruciani, P. Crivori, P.-A. Carrupt, an B. Testa. Molecular fiels in quantitative structure-permeation relationships: the VolSurf approach. Journal of Molecular Structure, 53:7 3, 2. [7] Rajesh N. Davé. Characterization an etection of noise in clustering. Pattern Recognition Letters, 2: , 99. [8] Jerome H. Frieman an Jacqueline J. Meulmany. Clustering objects on subsets of attributes. Journal of the Royal Statistical Society, 664), 24. [9] Davi J. Han, Heii Mannila, an Pahraic Smyth. Principles of Data Mining. MIT Press, 2. [] Fran Höppner, Fran Klawoon, Ruolf Kruse, an Thomas Runler. Fuzzy Cluster Analysis. John Wiley, Chichester, Englan,

20 [] Karin Kailing, Hans-Peter Kriegel, Alexey Pryahin, an Matthias Schubert. Clustering multi-represente objects with noise. In PAKDD, pages , 24. [2] Huan Liu an Hiroshi Motoa. Feature Selection for Knowlege Discovery & Data Mining. Kluwer Acaemic Publishers, 998. [3] Lance Parsons, Ehtesham Haque, an Huan Liu. Subspace clustering for high imensional ata: a review. SIGKDD Explor. Newsl., 6):9 5, 24. [4] Davi E. Patterson an Michael R. Berthol. Clustering in parallel universes. In Proceeings of the 2 IEEE Conference in Systems, Man an Cybernetics. IEEE Press, 2. [5] Witol Perycz. Collaborative fuzzy clustering. Pattern Recognition Letters, 234): , 22. [6] Ansgar Schuffenhauer, Valerie J. Gillet, an Peter Willett. Similarity searching in files of three-imensional chemical structures: Analysis of the bioster atabase using two-imensional fingerprints an molecular fiel escriptors. Journal of Chemical Information an Computer Sciences, 42):295 37, 2. [7] N. B. Venateswarlu an P. S. V. S. K. Raju. Fast ISODATA clustering algorithms. Pattern Recognition, 253): , 992. [8] Jiong Wang, Hua-Jun Zeng, Zheng Chen, Hongjun Lu, Li Tao, an Wei-Ying Ma. ReCoM: Reinforcement clustering of multi-type interrelate ata objects. In In Proceeings of the 26th annual international ACM SIGIR conference on research an evelopment in information retrieval SIGIR 3), pages , 23. [9] R. R. Yager an D. P. Filev. Approximate clustering via the mountain metho. IEEE Trans. Systems Man Cybernet., 248): , August