K-means Clustering Algorithm in Projected Spaces

Transcription

1 K-means Clusterng Algorthm n Projecte paces Alssar NAER, Dens HAMAD.A.. -U..C.O 50 rue F. Busson, BP 699, 68 Calas, France Emal: nasser@lasl.unv-lttoral.fr Chaban NAR ebanese Unversty E.F Rue Al-Arz, rpol - ebanon. chnasr@eee.org Abstract Clusterng has been known as a popular technque for pattern recognton, mage processng, an ata mnng. Unfortunately, all known clusterng algorthms ten to break own n hgh mensonal spaces; ths s ue to the nherent sparsty of the ponts. We nvestgate, n ths paper, the use of lnear an nonlnear prncpal manfols for learnng lowmensonal representatons for clusterng. everal leang methos: PCA, KPCA, ammon, an CCA are examne an teste n clusterng experments usng synthetc an real atasets from the UCI Databases. We compare the clusterng performance of the K-means algorthm on ata projecte by these projecton methos. he expermental results show that K-means clusterng on ata projecte by KPCA outperforms those projecte by the three other methos. Keywors: clusterng, projecton methos, K-means, clusterng accuracy. Introucton Clusterng s one of the well-stue technques, whch concerns the parttonng of smlar objects nto clusters such that objects n the same cluster share some unque propertes. ome algorthms use computatonal power of computers to automatcally fn tght groups or zones of hgh enstes. However, clusterng algorthms often show lmte performances n a hgh mensonal space ue to the nherent sparsty of the ata. In hgh mensonal applcatons, t s lely that for any gven par of ponts there exst at least a few mensons on whch the ponts are far apart from one another. o a clusterng algorthm s often precee by feature selecton. he goal s to fn the partcular mensons on whch the ata ponts are correlate. Prunng away the remanng mensons may reuce the nose n the ata an mprove the classfcaton performance. hereby, mensonalty reucton an especally nonlnear projecton technques receve more an more nterest n fferent omans: nexaton an mage retreval [], ata mnng [], supervson an process agnoss [3], etc. o, several projecton methos are avalable n pattern recognton an machne learnng lterature [4-]. Dmensonalty reucton can be one n two ways: ( selectng a small but mportant subset of features or ( generatng (extractng new features by projectng the ata nto a lower mensonal space whch preserves the stngushng characterstcs of the orgnal hghmensonal ata. Dmensonalty reucton can be useful for other exploratory ata analyss. It can help n both clusterng tenency assessment an prove a means to ece the number of clusters n the ata by lookng at the scatter plot of the lower mensonal space. In the context of clusterng, a goo feature extracton scheme shoul mantan an enhance those features of the nput ata whch make stnct pattern classes separate from each other. We present, n ths paper, four methos for mensonalty reucton: Prncpal Components Analyss (PCA, Kernel PCA (KPCA, ammon an Curvlnear Components Analyss (CCA. hese methos are then examne an teste n clusterng experments usng synthetc an real atasets from the UCI Databases. We compare the clusterng performance of the K-means algorthm n the corresponng he remanng part of the paper s organze as follows. ecton escrbes the four mensonalty reucton methos an K-means clusterng algorthm n projecte space s presente n secton 3. Expermental results are scusse n secton 4, whle secton 5 conclues the paper. Dmensonalty reucton methos For the sake of comparng, what follows brefly ntrouces four methos for mensonalty reucton an conser, for ths purpose, a set of centere observatons x l, where x l n R N an l =.. Prncpal Components Analyss (PCA PCA s a powerful technque for extractng structure from a hgh-mensonal space. It s a lnear projecton metho from the N-mensonal nput space to M-mensonal output space ( M N by solvng an Egenvalues- Egenvectors problem: Cv = λv ( Where C s the covarance matrx of centre ata: C = ( x l.( xl ( = l λ an v are the egenvalue an egenvector respectvely of C.

2 = m vm by stackng the M largest egenvectors n columns. he prncpal components Y are then efne by: Y = W X (3 he new prncpal axes capture the maxmal varance n such a way the new ata or prncpal components n the output space are uncorrelate an sperse. hs metho of projecton accorng to the axs of maxmal sperson oesn t take always nto account non-lnearty relatonshp between nput ata [0]. We form the matrx W { v, K, v,, }. Kernel Prncpal Components Analyss (KPCA Real worl ata are often nonlnear, n whch case lnear technques, le PCA, are not approprate. o overcome ths problem, KPCA metho, frst ntrouce n [6], has been wely use for non-lnear feature extracton an ata projecton. he non-lnearty s ntrouce by mappng the ata from the nput space R N to a feature space F. KPCA utlzes kernel trck to perform operaton n the new feature space where ata samples are more separable. Accorng to Cover s theorem, the nonlnear ata structure n the nput space s more lely to be lnear after hgh-mensonal nonlnear mappng []. By usng a nonlnear kernel functon nstea of the stanar ot prouct, we mplctly perform PCA n a hgh-mensonal space F whch s non-lnearly relate to nput space. Consequently, KPCA prouces features whch capture the nonlnear structure n the ata better than lnear PCA. Gven the mappng functon: φ : R x N F φ( l x l (4 he correlaton matrx n the feature space F s efne as: C = = l φ( x φ( (5 l x l KPCA metho s base on solvng egenvector system on the transforme space: C v = λ v (6 Where λ an v are the egenvalue an egenvector respectvely of C. v les n the span of φ, K, φ( x, thus t s a lnear ( x combnaton of φ x elements. hus, v can be wrtten as: ( l v = aφ( x (7 = Defnng the kernel functon by: K( xl, x = ( φ( xl φ( x (8 In orer to extract prncpal components of any pont x we have to project the mage φ(x of ths pont on the M obtane egenvectors v m : ( = = j ym = vmφx amjk( xj, x (9 he Egenvectors { y, y,, } n F are calle m y M nonlnear prncpal components. Note that, KPCA can extract more nonlnear features than corresponng number of lnear features n orgnal feature space. A number of fferent kernels have alreay been use n many areas of Kernel Machnes, of polynomal, Gaussan, an sgmo type..3 ammon s algorthm ammon s algorthm ams to fn a confguraton of ponts n the M mensonal projecton output space from an stance matrx (proxmty matrx where M < N. It s base on mnmzng the ammon stress, n a manner that local geometrcal structure of observatons s well preserve n output space [4]. et s enote: = ( x, x the stance between ata ponts x an x j. j δ = ( y, y the stance between projecte ata ponts j y an y j. ammon tress s efne as follows: Where D = E = < j D. < j ( δ (0 hs algorthm uses the metho of steepest escent for mnmzng E. herefore, the aaptaton equaton of y s gven by: E E y = α ( y y Where α, 0 < α < s the learnng step an: E y E y = = D < k D < k ( ( δ δ (y y [( δ ( y ykj ( _δ δ + kj

3 .4 Curvlnear Components Analyss (CCA he ea of Curvlnear Components Analyss metho s to preserve stances n the nput an output spaces; all the possble stances between ponts n the nput space shoul match the respectve stances n the output space. However, preservaton of larger stances many not be possble n the case of nonlnear ata, as a global unfolng of the manfol s requre to reuce the menson. In ths case, t s mportant that at least local stances shoul be preserve. For ths reason, CCA uses a neghborhoo functon whch ensures the conton of stance matchng s satsfe for smaller stances whle t s relaxe for larger stances [8]. ECCA = ( δ G( δ ( < j Where G s a monotone postve ecreasng functon of δ that ams to favor local stances. E CCA s mnmze by the followng aaptaton rule: δ y = β(t [G( δ ( δ G'( δ ](y y j (3 where β (t s the step sze of the graent an G s the ervatve of G. 3 K-means clusterng algorthm n projecte ata space. We propose to apply the K-means clusterng algorthm on ata projecte by the four mensonalty reucton methos n orer to compare ther powerfulness. In the next, we escrbe the K-means algorthm an the evaluaton crteron []. 3. K-means algorthm K-means algorthm parttons the ataset nto a selecte number K of clusters by mnmzng a formal objectve means-square-error storton ME: terms = { u } λ where the Nx mensonal vector N has elements of value /N, λ an u are the egenvalues/egenvectors ecomposton of the Kernel matrx [3]. Although t s an unsupervse problem, labelle ataset from UCI atabase are only use to calculate the clusterng accuracy of K-means. 4 Expermental results We apply the projecton methos on four examples; two synthetc atasets an two real atasets from the UCI atabases. For KPCA metho an for all experments we choose the Gaussan kernel. he problem s n choosng the value of the Gaussan wth σ whch n our experments has been tune. 4. wo spheres he synthetc ataset we nvestgate s rownng from two generator strbutons; t conssts of 800 ponts n three mensons. 400 ponts are selecte ranomly wthn a hemsphere of raus 0.6 an the rest 400 from a shell efne by two hemspheres of raus an.03. he value of σ equals 0. for KPCA metho. he omnant terms estmate clusters; ths value can be easly estmate by lookng at the projecte ata by KPCA. herefore the results of K-means algorthms ntalze by K equals an apple to the orgnal an projecte ataset are summarze n the fgure 3. KPCA metho gves the best result wth zero error corresponng to an equvalent clusterng accuracy of 00%. Followe by CCA (8.%, PCA (78.87 whch s the same result obtane by applyng K-means to orgnal ataset, an fnally ammon s algorthm gves the poor results for a clusterng accuracy equals 77.87%. Note that, for the four methos we only use -mensonal projecte space for the purpose of clusterng. Plots of corresponng - mensonal projecte spaces of the four projecton methos are shown n fgures -4. K ME = x c (4 k= x Ck k c k s the center of the class C k, k=,, K. Due to ts smplcty to mplement, we choose the K- means clusterng problem n our experments to compare an evaluate the performance of the mensonalty reucton methos scusse n secton. herefore, the clusterng accuracy can gve us a straghtforwar metrcs evaluatng the capactes of methos to possess the ata for clusterng. K-means algorthm s thus apple to ata projecte by these projecton methos. hese methos can prove a pre-processng phase for the problem of clusterng n hgh mensonal space. Instea we cluster ata n ther corresponng projecte subspace. he possble number of clusters to ntalze K-means algorthm wthn the ata can be estmate by vsualzng the projecte ata an by choosng the most omnant Fgure : wo spheres ataset: Data parttonng n ther PCA

4 Plots of corresponng -mensonal projecte spaces of the four projecton methos are shown n fgures 5-8. Fgure : wo spheres ataset: Data parttonng n ther KPCA Fgure 5: Irs ataset: Data parttonng n ther PCA Fgure 3: wo spheres ataset: Data parttonng n ther ammon Fgure 6: Irs ataset: Data parttonng n ther KPCA Fg 4: wo spheres ataset: Data parttonng n ther CCA 4. Irs ataset Irs ata s a well known ataset (UCI atabases. he ata we analyze consst of 50 samples for each of the three classes present n the ata, Irs etosa, Verscolor an Vergnca. Each atum s four mensonal an conssts of measures of the plants morphology. For KPCA apple on ths ataset, the value of σ equals.. he omnant terms prove 3 clusters. herefore, the results of K-means algorthms ntalze by K equals 3 an apple to the orgnal an projecte ataset are summarze n the fgure 3. KPCA gves the best result wth sx errors corresponng to an equvalent clusterng accuracy of 96%. Followe by CCA (89.33%, ammon s algorthm (88, K-means to orgnal ata (84% an fnally by PCA ( Here agan, we only use - mensonal projecte space for the purpose of clusterng. Fgure 7: Irs ataset: Data parttonng n ther ammon Fgure 8: Irs ataset: Data parttonng n ther CCA

5 4.3 Wne ataset As a fnal example we present results from a wne recognton problem from the UCI atabases. he ataset conssts of 78 samples of 3-mensonal whch are the results of a chemcal analyss of wnes grown n the same regon n Italy but erve from three fferent cultvars. he analyss etermne the quanttes of 3 consttuents foun n each of the three types of wnes. he four projecton methos are apple to the ata. he plot onto the frst two prncpal components of KPCA for σ equals 0.4 shows that the number of clusters s three. hs result s approve by the estmate number gven by omnant terms. here are only three sgnfcant contrbutors of thus ncatng the presence of 3 clusters. Once agan K- means algorthm apple to ata projecte by KPCA gves the best results wth 8 errors corresponng to an equvalent clusterng accuracy of 95.5% followe by CCA (5.%, ammon s algorthm (50,56%. Fnally PCA an K-means apple to orgnal ataset gve the same results (49.43%. Plots of corresponng -mensonal projecte spaces of the four projecton methos are shown n fgures 9-. Fgure : Wne ataset: Data parttonng n ther ammon Fgure : Wne ataset: Data parttonng n ther CCA 4.4 Dscusson Fgure 9: Wne ataset: Data parttonng n ther PCA Fgure 0: Wne ataset: Data parttonng n ther KPCA Fgure 3 summarzes the clusterng accuracy obtane by applyng K-means on orgnal an projecte ata by the four methos. We can see that KPCA outperforms the three other methos; t gves the best clusterng results for all examples, especally for complex an nonlnearly separable ata. hs s ue to the nonlnear transformaton of KPCA whch maps the nput space to a hgh mensonal feature space where the transforme ata coul be lnearly separable. We have to note that, we only use two prncpal components of KPCA, talkng more components may greatly mprove the clusterng process. Dataset Data PCA KPCA ammon CCA -pheres Irs Wnes ,56 5. Fgure 3: Accuracy comparsons of the K-means clusterng algorthm on the three orgnal an projecte ataset by the four projecton methos. 5 Concluson We nvestgate the use of lnear an nonlnear prncpal manfols for learnng low-mensonal representatons for clusterng. PCA, KPCA, ammon, an CCA methos are compare an teste by means of clusterng accuracy of K-means algorthm. In orer to compare the qualty of ata projecte by these methos, we apple K-means algorthm on the projecte ata of synthetc an real

6 examples taken from the UCI atabases. he results show that K-means apple to ata projecte by KPCA an keepng only the two frst components outperforms those projecte by the other three methos for the same number of components. hs s ue to the fact that, KPCA nonlnearly maps the nput ata nto a hgh-mensonal space n whch the separablty s lnear an the clusters are straghtforwar to be entfe. As we see for the example of two spheres n 3-mensonal space, the corresponng prncpal components n -mensonsal KPCA space are perfectly separable an K-means algorthm successfully stngushe the two clusters; whereas the other mensonalty reucton methos gve poor results. For the other atasets, KPCA methos prove the approprate ata for K-means clusterng. he number of clusters for K-means algorthm s estmate by lookng at the scatterplots of the lower mensonal ata an/or by takng the few omnant terms of λ { u }. o summarze, we note that KPCA s more sutable to preprocess the ata for clusterng than the three other methos PCA, ammon an CCA wth fferent complexty of atasets. [9] J. Mao an A. K. Jan, Artfcal neural networks for features extracton an multvarate ata projecton, IEEE rans. Neural Networks, vol. 6, no., pp , 995. [0] I.. Jollffe, Prncpal Component Analyss. prnger-verlag, 986. []. Haykn, Neural Networks. Prentce-Hall, Englewoo Clffs, NJ, 999. [] J. McQueen. ome methos for classfcaton an analyss of multvarate observatons. In Proceengs of the Ffth Berkeley ymposum on Mathematcal tatstcs an Probablty, pp. :8-97, 967. [3] M. Grolam, Mercer kernel-base clusterng n feature space, Neural Networks, IEEE ransactons on, Volume 3, Issue 3, pp , May 00. References [] A.K. Jan an J. Mao, "Artfcal Neural Network for Nonlnear Projecton of Multvarate Data", Proc. IEEE Int. Jont. Conf. on Neural Networks, Vol. 3, pp , Baltmore-Marylan, 99. [] R. Agrawal an al, Automatc ubspace Clusterng of Hgh Dmenonal Data for Data Mnng Applcatons, Proc. ACM IGMOD conf., pp , 998. [3] Q. Chen, R.J. Wynne, P.Goulng, D. anoz, he Applcaton of Prncpal Component Analyss an Kernel Densty Estmaton to Enhance Process Montorng, Control Eng. Prac., , 000. [4] J.W. ammon, "A non lnear mappng for ata structure analyss". IEEE ransactons on computers, Vol. C-8, No. 5, pp , 969. [5] W. eleck, K. elecka an J. lansky, "An overvew of mappng technques for exploratory analyss". Pattern Recognton, Vol., No. 5, pp. 4-49, 988. [6] B. hölkopf, A.J. mola, earnng wth Kernels: upport Vector Machnes, Regularzaton, Optmzaton an Beyon, the MI Press, Cambrge, Massachusetts, onon, Englan, 00 [7] W. eleck, K. elecka an J. lansky, "Experments on mappng technques for exploratory pettern recognton". Pattern Recognton, Vol., pp , 988. [8] P. Demartnes an J. Hérault, Curvlnear Component Analyss: A self-organzng Neural Network for Nonlnear Mappng of Data ets, IEEE rans. Neural Networks, vol. 8, no., January 997.