A new Fuzzy Noise-rejection Data Partitioning Algorithm with Revised Mahalanobis Distance

Transcription

1 A new Fuzzy ose-reecton Data Parttonng Algorth wth Revsed Mahalanobs Dstance M.H. Fazel Zarand, Mlad Avazbeg I.B. Tursen Departent of Industral Engneerng, Arabr Unversty of Technology Tehran, Iran Departent of Industral Engneerng, TOBB Econoy and Technology Unversty Anara, Turey Eal: Abstract Fuzzy C-Means (FCM and hard clusterng are the ost coon tools for data parttonng. However, the presence of nosy observatons n the data ay cause generaton of copletely unrelable parttons fro these clusterng algorths. Also, applcaton of the Eucldean dstance n FCM only produces sphercal clusters. In ths paper, a new nose-reecton clusterng algorth based on Mahalanobs dstance s presented whch s able to detect the nose and outler data and also ellpsodal clusters. Unle the tradtonal FCM, the proposed clusterng tool provdes uch effcent data parttonng capabltes n the presence of nose and outlers. For valdaton of the proposed odel, the odel s appled to dfferent nosy data sets. Keywords Cluster Valdty Index (CVI, Fuzzy C-Means (FCM, Possblstc C-eans (PCM, Revsed Gustafson-Kessel (GK, Revsed Mahalanobs Dstance. Introducton Clusterng ethods have been extensvely used n coputer vson and pattern recognton. Fuzzy clusterng ethods have shown spectacular ablty to detect not only volue clusters, but also clusters whch are actually thn shells,.e. curves and surfaces. Most analytc fuzzy clusterng approaches are derved fro the fuzzy C-eans (FCM algorth. FCM uses the probablstc constrant that the ebershp of a data pont across classes sus to. The constrant s used to generate the ebershps update equatons for an teratve algorth. The ebershps resultng fro FCM and ts dervatve however, do not always correspond to the ntutve concept of belongng or copatblty. Moreover, the algorths have consderable trouble n nosy envronents. Reference [] suarzes the an probles of classc FCM as follows: In order to get the optal partton, ntal locatons of the cluster centers should be assgned. The FCM algorth always converges to a local extree. Dfferent choces of ntal cluster centers ay lead to a dfferent extrea. The scentfc bass for the choce of, the weghtng exponent, s stll not clear. The optu nuber of clusters n the data s assgned a pror. There should be a crteron to assgn the optal nuber of clusters. To overcoe these probles and drawbacs, frst, [] ntroduces a new ethod for fuzzy clusterng called possblstc fuzzy clusterng. Ther approach dffers fro the prevous clusterng ethods n that the resultng partton of the data can be nterpreted as a possblstc partton, and the ebershp values ay be nterpreted as degrees of possblty of the ponts belongng to the classes,.e., the copatbltes of the ponts wth the class prototypes. They construct an approprate obectve functon whose nu wll characterze a good possblstc partton of the data, and derve the ebershp and prototype update equatons fro necessary condtons for nzaton of the related crteron functon. ext nonated wor s []. Mele et al. n [] show how ther PCM addresses the entoned probles of the FCM for soe exaple cases. However, ther ethod has soe ltatons whch are the concentraton of ths paper. The Rest of the paper s organzed as follows: Secton dscusses the drawbacs of the prevous PCM s. Then, n secton 3 the proposed ethod s presented n order to overcoe these ltatons. Secton 4 apples the ethod for dfferent data sets to verfy and valdate the proposed ethod. Ltatons of the tradtonal PCM s Reference [] uses Eucldean dstance n all data parttonng steps. However, clearly, Eucldean dstance usually fals to recognze the approprate shape of the clusters n coplex data sets. Especally, when data nclude ellpsodal shapes, Eucldean dstance loses sght to those data whch should be consdered n a cluster fro data whch should not be ncluded n that cluster. Moreover, the cluster valdty ndex (CVI appled n [], suffers fro lnear behavour of the Eucldean dstance. There are any saples of data sets whch the appled CVI s not able to recognze the nuber of clusters correctly. Another ltaton s about nose dentfcaton procedure. Applcaton of Eucldean dstance can slead the nose reecton procedure. Ths proble s dscussed n the related secton of the proposed algorth. Whle Eucldean dstance ples the above entoned ltatons, Mahalanobs dstance can tgate or n soe cases can overcoe the cted ltatons copletely. Another advantage of the Mahalanobs dstance s that the ISB:

2 Mahalanobs dstance can dentfy both sphercal and ellpsodal clusters correctly. 3 Iproveents to the tradtonal PCM s As quoted n ntroducton, the proposed ethod apples the Mahalanobs dstance for nose reecton proble. However, when applyng ahalanobs dstance, soe dffcultes ay occur wth covarance atrx. In ths secton, frst, the proble wth calculaton of the Covarance Matrx n Mahalanobs dstance s dscussed. An estaton of Covarance Matrx s ntroduced to tgate the proble. Then, new Mahalanobs dstance and revsed Gustafson-Kessel Clusterng are presented based on the estaton. ext, the revsed Gustafson-Kessel Clusterng ethod wth the new Mahalanobs dstance s ntegrated nto the PCM whch s presented n []. Moreover, a new CVI s appled whch s based on proxty of two fuzzy sets. The new CVI s ndependent fro the dstance type and therefore can be ntegrated to the ethod wth Mahalanobs dstance. Ths CVI s used n [3]. Fg. shows an exaple of a data set whch cannot be clustered wth standard GK. Usng new estaton for covarance atrx, the nuercal probles would be resolved and GK can approprately fnd the correct nuber of clusters (exaple fro [4]. The above odfcaton prevents the GK algorth fro runnng nto nuercal probles. However, as a result one can get clusters that are extreely long n the drecton of the largest egenvalues and have lttle relatonshp wth real dstrbuton of data. Ths can cause over fttng of the data and consequently one obtans a poor odel [4]. Ths proble occurs anly when the nuber of data ponts n a cluster becoes too low. In such a case, the coputed covarance atrx s not a relable estate of the underlyng data dstrbuton [7]. One way to tacle ths proble s to 3. Revsed GK wth revsed Mahalanobs In statstcs, Mahalanobs dstance s a dstance easure ntroduced by P. C. Mahalanobs n 936. It s based on correlatons between varables by whch dfferent patterns can be dentfed and analyzed. It s a useful way of deternng slarty of an unnown saple set to a nown one. It dffers fro Eucldean dstance n that t taes nto account the correlatons of the data set and s scale-nvarant,.e. not dependent on the scale of easureents. Mahalanobs dstance can be defned as dsslarty easure between two rando vectors x, y of the sae dstrbuton wth the covarance atrx S : T DM, µ = µ S µ The egenvalues and egenvectors of the covarance atrx descrbe the shape and orentaton of the clusters n GK ethod of clusterng whch apples ahalanobs dstance nstead of Eucldean one. When an egenvalues s zero or when the raton between the axal and nal egenvalues,.e., the condton nuber of F, s very large the atrx s nearly sngular. In such a case, the nverse of covarance atrx cannot be calculated. Also the noralzaton to a fxed volue fals, as the deternant (the volue of the covarance atrx becoes zero and the followng forula thus cannot be appled n GK ethod [4]: / n det( F F ( A straghtforward way to avod nuercal probles s to constran the rato between the axal and nal egenvalues such that t s saller than soe predefned threshold. When ths threshold exceeds, the nal egenvalues s ncreased such that the rato equals to the threshold and the covarance s reconstructed by [4]: F = ΦΛΦ ( where, s the dagonal atrx contanng the lted egenvalues and s a atrx whose coluns are the correspondng egenvectors. Fgure : Lnear Clusters lt the rato between axal and nal egenvalues even further than descrbed n the prevous secton. Ths wll prevent the extree elongaton of the clusters. Another way s to add a scaled dentty atrx to the covarance atrx. Reference [7] and [8] descrbe several dfferent ethods to prove the covarance estaton. Inspred by these ethods, [4] proposes the followng estate for the GK algorth and calculaton of Mahalanobs dstance: new / n F = ( γ F + γ det( F0 I (3 where [0,] s the tunng paraeter and F 0 s the covarance atrx of the whole data set. Dependng on, the clusters are forced to have a ore or less equal shape. When s, all covarance atrces are equal and have the sae sze, whch of course lts the possblty of the algorth to properly dentfy clusters. For the coplete descrpton of the revsed GK, readers can refer to [4]. The an steps of the algorth are as follows [4]: Repeat for l =,,, Ite# Step : Copute cluster prototypes (eans ( l ( µ Z ( l = v =, <= <= K ( l ( µ = Step : Copute the cluster covarance atrces (4 ISB:

3 F = = ( µ ( l ( l ( l T ( Z v ( Z v = ( µ ( l, K Add a scaled dentty atrx: / n F = ( γ F + γ det( F0 I, K (6 Extract egenvalues and egenvectors fro F. Fnd ax =ax and set: λ λ ax λ = for whch β λ ax > β (7 Reconstruct F by F = [ φ... φn ] dag( λ,... λn [ φ... φn] Step 3: Copute the dstances ( l T / n ( l DA = ( Z v [ ρ det( F F ]( Z v (8 where <=<= and <=<=. Step 4: Update the partton atrx for f D A > 0 for K, ( l µ = K (9 ( D / D = A A Otherwse ( l µ = 0 f D > 0, and ( l µ [0,] (0 K A ( l wth µ = otherwse. = ( ( l Untl U l U < ε. Iproved GK clusterng does not have the probles that standard GK ay face n soe cases. In the next secton, appled cluster valdty ndex s descrbed. 3. Cluster Valdty Index (CVI Ths paper uses a cluster valdty ndex proposed by [9]. Defnton: The relatve slarty Srel (x :Ap,Aq between two fuzzy sets A p and A q at x s defned as: f(x :Ap Aq ( f(x :Ap Aq + f(x :Ap Aq + f(x :Aq Ap whle f(x :Ap Aq = ua ua and s the p q nu operator. Moreover for functon of deference: f(x :Ap Aq = Max( 0, u A u A ( P q Defnton: The relatve slarty between two fuzzy sets A p and A q s defned as: n S (Ap,Aq = S (x : Ap,Aq h( x (3 rel rel = where c h( x = u log( u (4 p= AP AP (5 Here, h s the entropy of datu x and u A s the P ebershp value of x to cluster A p. Defnton3: The cluster valdty ndex s as follows: c V ( U, V; X = S rel (Ap,Aq (5 c( c p q The optal nuber of clusters s obtaned by nzng V over the range of c values (uber of clusters. Moreover, soe other popular CVI s such as Xe and Ben CVI [5] and Known CVI [6] are evaluated. Results show that the new CVI has better results and s able to fnd the nuber of clusters correctly n any coplex stuatons. The nputs to the CVI used n the proposed odel are obtaned by the revsed GK. 4 The proposed ethod In ths secton, we present our ethod base on the odfcatons presented n secton 3. For the selecton of the weght exponent (, t s suggested to be chosen far fro ts both extrees so as to ensure that the cluster valdty ndex shows the optu nuber of fuzzy clusters. A fuzzy total scatter atrx s defned n [0] as: S T = K T ( ( u υ υ (6 = = The trace of the fuzzy total scatter atrx decreases onotoncally fro a constant value z to zero as vares fro one to nfnty. For data parttonng, a sutable value for s that whch gves a value for trace (s T equal to z/ []. The constant value z s defned as: T z = trace( [( x x x ] (7 = = = Usng the trace value, we deterne the value of ( whch s the degree of fuzzness of the syste. ext, usng the ntroduced CVI and acheved (, the approprate nuber of clusters would be obtaned. After deternaton of and nuber of clusters, we repeat the followng procedure, teratvely: We frst pleent the revsed GK on the data set. To choose the ntal cluster centers of the revsed GK, we apply sple FCM for ntalzaton of our clusterng ethod. Other ethods le [] have appled AHC. The GK s senstve to the ntal cluster centers and ntal ebershp functon values. Our experents show that unle FCM, AHC s not a good ethod for the ntalzaton of the revsed GK. ext, n order to fnd the data ponts that are too far fro all cluster centers, [3] proposes the followng ndex for each data pont x : W = C x = υ h A (8 where, =,,...,, c s the nuber of clusters, and s the nuber of data. The ndex W s the suaton of the dstance of the data pont x to all cluster centers. Ths gves a easure of how far each data pont s fro the dfferent cluster centers assgned n the frst step of the algorth. The nose s dentfed through the data ponts that have large ISB:

4 values of W and therefore, a threshold X s assgned to tr these outlers fro the data set. The value of the threshold depends on the range of the nput data to the algorth. Whle [] apples (8 to fnd the W, we proposed the followng equaton n order to fnd the W : C W = Mean( WGHT x υ h = A (9 where, WGHT s a weght assgned to the -th dstance after that we sorted the dstances of x fro all cluster centers (c. Reference [] does not consder such weght. It s clear that sply the suaton of dstances can not provde a good easure for detecton of the outlers. Instead the proposed odel can assgn the largest weght to the sallest dstance and the sallest weght to the largest dstance vce versa usng (9. Weghts are the nput to the odel. After choosng the threshold, [] coputes: ηn z = (0 where, n s the nuber of nose ponts and s the total nuber of data. The percentage of good data ponts,.e., nlers can then be calculated as: zˆ = z ( After dentfyng the percentage of nlers n the data, we copute the correspondng ch-square data dstrbuton value [3]. Then, we calculate the cut-off dstance: u FC cut = υ χ ( where, υ s a resoluton paraeter that depends on the nuber of clusters, and χ s the ch-square value coputed by (. By nowng the new cut-off dstance, the optu nuber of clusters, the degree of fuzzness, and the ntal locaton of the clusters centers, we calculate the ebershp atrx through (3: u = (3 d, υ + { } υ It should be noted that (0, (, ( and (3 are presented n []. In the proposed PCM, we use these equatons wth revsed Mahalanobs dstance. 5 Results of Experents In ths secton, the proposed odel s appled for handlng four dfferent cases. The paraeters of the proposed odel for these cases are suarzed n table : OC: uber of Clusters s the degree of fuzzness of syste obtaned through 6 and 7. s the cut-off dstance Beta and Gaa are the paraeters of (3 and (7. #Itr: The nuber of teraton whch PCM goes on. W: WGHT n (9. Error: If the changes n the value of ebershp functons were saller than Error, the algorth stops. In all cases, Beta s.00e+6, #Itr s 7 and Error s.00e- 07. It should be noted that the data of the cases and ther related confguraton are very slar to [9] and [3]. However, we regenerated the data by ourselves. Table : Paraeters of the algorth for dfferent cases Paraeter OC Gaa W Case E-07 [6 ] Case E-09 [8 3 ] Case E-0 [8 3 ] Case E-07 [8 4 3 ] Case E-0 [8 4 3 ] It should be entoned that the experents can t be appled for the prevous nose reecton ethods n the lterature. Ths s because of the ellpsodal fors of data whch can t be handled usng Eucldean dstance. To ae ths clear, n Fg., the clusterng ethods based on Eucldean dstance apparently fal to recognze the shapes of clusters and noses correctly. That s why, only experents are appled for the proposed ethod. 5. Case Study I Every data cloud ncludes 500 data and 50 data are randoly generated as nose. Man data and noses both have Gaussan dstrbuton. Fgure : Data before clusterng Fgure 3: Fndng the approprate n case study I To deterne the exponent of fuzzness (, the total scatter atrx s calculated for dfferent values of and dfferent ISB:

5 nuber of clusters as s shown n (6. Then usng z n (7, the approprate s chosen. The trace value calculated as 5.835e+004. We use half of trace value (.957e+004 to deterne the optu (Fg. 3. to clusterng and choosng the approprate nuber of clusters are presented. 5. Case Study II To choose the correct nuber of clusters, we need to calculate the CVI for dfferent nuber of clusters. Fg. 4 shows the appled CVI for case study I. As the fgure shows, the optu nuber of clusters s equal to four. Base on Fg. 3 and Fg. 4, we consder fuzzness exponent ( equal to.5 and the nuber of clusters equal to 4. The result of the clusterng s shown n Fg. 5. The result shows that the proposed ethod can dentfy the noses fro the an data accurately. The expanson of the eclpse shape clusters can be lted by as an nput to the odel. Fgure 6: Data before clusterng Fgure 4: Identfcaton of the optu nuber of clusters Fgure 7: Identfcaton of the optu nuber of clusters Ths case s presented to show the strength of the used CVI and the effect of. As Fgure 8 shows, the cobnaton of these two eleents n the proposed odel, enable the odel to dentfy the nuber of clusters, the shapes of the clusters and fnally the noses accurately. Fgure 5: Data after Clusterng It s clear that, the ethod presented n [] can not dentfy the correct shape of the clusters because of the sphercal behavour of the data. In Fg. 5, data whch are shown wth star *, have a axu ebershp functon value lower than 0., hence are dentfed as nose. In the other cases, each data cloud ncludes 500 data and 50 data are randoly generated as nose. Man data and noses both have Gaussan dstrbuton. Also only the fgures related Fgure 8: Data after Clusterng 5.3 Case Study III Fg 9 shows the data set and noses. ISB:

6 5.4 Case Study IV Fgure 9: Data before clusterng Fgure : Data before clusterng As Fg. 0 shows, the optu nuber of clusters s equal to fve. Ths case s presented to valdate that whether the ethod s able to fnd the sphercal and ellpsodal shapes sultaneously or s not. As Fgure shows, the ethod could fnd the crcle at the center of the fgure and four eclpses around the crcle accurately. Also the noses are detected accurately as shown n Fgure. Fgure 3: Identfcaton of the optu nuber of clusters Fgure 0: Identfcaton of the optu nuber of clusters Fgure 4: Data after Clusterng Fgure : Data after Clusterng In ths case, fve eclpses are spread n dfferent drectons as shown n Fgure. The clusterng results show that the ethod can fnd the coplex behavors of data n dfferent drectons accurately as well as sple cases. 6 Conclusons In ths paper, a new nose reecton clusterng ethod base on Mahalanobs dstance and a dfferent cluster valdty ndex s presented. The new ethod can be dstngushed fro the exstng ethods n the lterature fro the followng ponts: ISB:

7 - The nose reecton clusterng ethods exstng n the lterature anly use Eucldean dstance n ther clusterng ethod. The proposed ethod, apples Mahalanobs dstance whch enables the ethod to dentfy the ellpsodal behavor of data besdes sphercal behavor. - In order to fnd the approprate nuber of clusters, the ethod uses a well-defned cluster valdty ndex whch s ndependent fro dstance type. - A new weghtng syste s attached to the nose reecton ethod, whch helps for better detecton of noses. Method s appled to dfferent cases and the results show that the ethod s capable of dentfcaton of nose and outlers wthn sphercal and ellpsodal data. References [] W.W. Mele, A.A. Goldenberg, M.R. Ea, A fuzzy nose-reecton data parttonng algorth, Internatonal Journal of Approxate Reasonng, 38: 7, 005. [] R. Krshnapura, J.M. Keller, A Possblstc Approach to Clusterng, IEEE Transactons on Fuzzy Systes, (: 98-0, 993. [3] M.H. Fazel Zarand, B. Rezaee, I.B. Tursen, E. eshat, A type- fuzzy rule-based expert syste odel for stoc prce analyss. Expert Systes wth Applcaton, 36:39-54, 009. [4] R. Babusa, R.J. van der Veen, U. Kaya, Iproved Covarance Estaton for Gustafson-Kessel Clusterng. IEEE, , 00. [5] X.L. Xe, G. Ben, A valdty easure for fuzzy clusterng, IEEE Transactons on Pattern Analyss and Machne Intellgence, 3(8: , 99. [6] S.H. Kwon, Cluster valdty ndex for fuzzy clusterng. Electroncs Letters, 34(: 76 77, 998. [7] S. Tadudn and D.A. Landgrebe. Covarance estaton wth lted tranng saples. IEEE Transactons on Geoscences and Reote Sensng, 37(4: 3-8, July 999. [8] J.F. Fredan, Regularze dscrnate analyss, J.R. Statst. Soc., 84:7-4, 989. [9] K, Y.I., K, D.W., Lee, D., Lee, K.H., A cluster valdaton ndex for GK cluster analyss based on relatve degree of sharng, Inforaton Scence, 68:5-4, 004. [0] M. Sugeno, T. Yasuawa, A fuzzy-logc-based approach to qualtatve odelng, IEEE Transactons on Fuzzy Systes, (, 993. [] M.R. Ea, I.B. Tursen, A.A. Goldenberg, Developent of a systeatc ethodology of fuzzy logc odelng, IEEE Transactons on Fuzzy Systes, 6 (3: , 998. [] J.H. Ward, Herarchcal groupng to optze an obectve functon, Journal of Aercan Statstcs Assocaton, 58: 36 44, 963. [3] W.W. Mele, eurofuzzy Control of Modular and Reconfgurable Robots, PhD. Dssertaton, Unversty of Toronto, Canada, 00. ISB: