A NEW LINEAR APPROXIMATE CLUSTERING ALGORITHM BASED UPON SAMPLING WITH PROBABILITY DISTRIBUTING

Transcription

1 A NEW LINEAR APPROXIMATE CLUSTERING ALGORITHM BASED UPON SAMPLING WITH PROBABILITY DISTRIBUTING CHANG-AN YUAN,, CHANG-JIE TANG, CHUAN LI, JIAN-JUN HU, JING PENG College of Computer, Schuan unversty, Chengdu, Schuan 60064, Chna Department of Informaton Technology, Guangx Teachers Educaton Unversty, Nannng, Guangx 53000, Chna Abstract: Clusterng s an mportant research drecton n knowledge dscovery. As the classcal method n Clusterng, the k-medan algorthm s wth serous defcency such as low effcency, bad adaptablty for large data set etc. To solve ths problem, a new method named LCPD (Lnear Clusterng Based on Probablty Dstrbutng) s proposed n ths paper. The man contrbuton ncludes:() Parttons the buckets by usng the space of equal probablty n the m-dmenson super-cube to make the number of data tems n each layer ( namely the bucket of Hash) approxmate equal, gets the layerng samplng wth the small cost; () The samples under the new algorthms s wth suffcent representatve power for total data set; (3)Proves that the complexty of the new algorthm s O(n); (4) By the comparng experment shows that the performance of LCPD s magntude hgher than tradtonal wth the number of data set near to 0000, and the clusterng quantty s ncrease 55% wth number of data set near to Keywords: k-medan algorthm; Clusterng; Probablty Dstrbutng; Hash functon; Samplng. Introducton and related work Clusterng s one of the most mportant drectons n knowledge dscovery, whch ams at groupng data ponts to be processed and gets the resultng groups. And n these groups, data ponts are most smlar and data ponts at dfferent groups dffer most. Dfferent algorthms exst on dfferent data characterstcs. Generally speakng, there are fve dfferent classes []. The major parttonng method s: for a gven database wth n objects or records, dvde t nto k classes. The earlest method s k-means method []. Ths method has good scalablty to bg scale data set. The man problem s()each attrbute mean value of the object should be predefned;() The method s senstve to nosy data. A varant of k-dvdng method s k-centre method. The PAM ( Parttonng around Medod ) method by L.Kaufman and P.J.Rousseeuw s a typcal one [3]. The merts nclude: ()It has good capablty to prevent nose and outlners and s more robust than k-means method; () The clusterng results are ndependent to the order of nput data; (3)The results reman unchanged after conductng movement or transferrng. Therefore, centre-based methods are the bases of all clusterng method. The defects are they have hgher runnng cost. In order to adapt k-centre method to large dataset, L.Kaufman and P.J.Rousseeuw proposed CLARA (Clusterng Large Applcatons)algorthm [3],.e. samplng based method. Apply PAM algorthm n each applcaton sample (the number of sample s s), each nteractve step has the complexty O (ks +k (n-k)). The effectveness of CLARA s subject to the sze of the sample, and the qualty of clusterng s subject to the representablty of samples. R.Ng and J.Han. proposed CLARANS(Clusterng Large Applcaton based upon Randomzed search) algorthm [4], whch combnes samplng technques and PAM. In each step of the searchng process, a sample s extracted whch mproves the clusterng qualty. The defect of ths method s that the complexty s O (n ). The techncal dffculty of enhancng qualty and effcency of clusterng gathers n the samplng technque of CLARA method, namely, makng the samples representng the real statstcal characterstc of the total. Tradtonal CLARA method adopts smple samplng, whch reflects the statstcs features of data codng, does not reflect ts data dstrbuton statstcs; especally when data skews, the sample data can not represent the total. At that tme, level-dvson can be a good choce. Whle tradtonal dvson needs related knowledge. Ths paper proposes and mplements a model LCPD (Lnear Clusterng Based Probablty Dstrbutng) under large dataset, whch constructs probablty hash functon based upon dstrbuton, and mplements herarchcal samplng wth Hash functon, realzes hgh effcency and hgh qualty. Ths model ams at () keepng /05/$ IEEE 58

2 the complexty of K-centre ponts clusterng algorthm wthn the lnear tme of n; () promsng the clusterng qualty. Notng that tradtonal samplng needs knowledge about layers n advance, we ntroduce hash functon based samplng technque and utlze hash bucket to dvde levels, and then do samplng, whch fully represents the statstcs statstcal characterstc of data. The man contrbutons nclude()proposng a probablty dstrbuton based Hash functon constructon, realzng the level dvson sample model LCPD, whch makes each layer (namely, hash bucket) havng smlar data ponts and thus have good level-dvson effect;()the new algorthm promses that the extracted samples can fully represent the total;(3)the new algorthm realzes the fast k-centre ponts clusterng of n lnear tme complexty;(4)comparng ths algorthm wth other related algorthms by experments. The rest of ths paper s organzed as follows: The Secton gves symbols and defntons; The Secton3 ntroduces the probablty dstrbuton based hash functon constructon and level-dvson realzaton technque; The Secton4 ntroduces probablty dstrbuton based hash functon constructon and the level-dvson samplng clusterng method; The Secton5 conducts experments; and the Secton6 performs concluson.. Symbols and defntons of LCPD model In order to formally descrbe LCPD model to make t easy to read, the followng table lsts all the symbols n ths paper. Table Lst of sgns Symbols A Cost functon s the techncal ponts of LCPD. And t s an mportant evaluatng functon of clusterng qualty. Its formal defnton s the followng: Defnton (Cost functon)suppose n ndvduals are classfed nto A={A =,,,k}. Then k C(A)= d( X ( j), med( )) s sad to be the cost = x( j) A( ι ) functon of class A. Where med() s the centre pont of class A ( =,,,k ), X(j) s the ndvdual of A (=,,,k). d s the dstance between X(j) and med(). Drectly, the am of K-centre pont clusterng s to fnd a class A, whch makes C(A)reach mnmum value. Defnton (The clusterng termnaton value ß) Suppose n the PAM algorthm, the classes from the I-th cycle and the (I+)-th cycle are A I and A I+, f C( A C( A I I + ) -<ß, ) the cycle termnates. And A I+ becomes the fnal clusterng. Then ß(0 ß<) s called the termnaton value. The clusterng termnaton value wll be used n algorthm later. The man functon s to rase the clusterng effcency. Usng an approxmate clusterng algorthm, reducng a lttle qualty, s mportant to large scale dataset. 3. Hash functon based samplng 3.. Basc assumpton Ths paper uses the total probablty dstrbuton functon to construct hash functon owng to(a)the calculaton amount to completely compute the precse total Meanng dstrbuton s too bg ( b ) The total dstrbuton s approxmate owng to data nose, even to deal wth data of X j Denotes the jth varable of th object n the whole number. Thus LCPD uses some propertes of random total samplng. It uses the total evaluaton dstrbuton functon x j Denotes the jth varable s value of th to substtute ts precse dstrbuton accordng to some object n the sample propertes of random samplng. Xmean Denotes the means of varable X Suppose the data set s: X=(X j ) n m, namely, t has m S x Denotes the standard varance of varable X varants, n rows data. To smplfy the problem,wthout n Denotes the number of ndvduals of the losng ts generalty, we set the followng assumptons: total Assumpton there are the followng types n the m m Denotes the number of varables k Dvarants: enotes the number of classes med Denotes the th centre pont Denotes the th class a) Contnuous type, e.g. weght and heght, etc. The dstance s usually calculated wth Eucld dstance or Manhattan dstance. b) Bnary type, namely, the varable has only two states, e.g. gender. c) Nomnal type, the generalzaton of the bnary type, whose states are more than, e.g. color. d) All the other types can be regarded as the specal 59

3 cases of the above three types. Assumpton In the m varables, x,,x m are contnuous type, x m+,,x m+m are bnary type, x m+m+,, x m+m+m3 are nomnal type. m +m +m 3 =m. Namely, m contnuous varables, m bnary varables and m 3 nomnal varables. Regardng the bnary varables, the dstance of the two objects, j s usually denoted as the matchng coeffcent: d(,j)=f/m, of whch f s number of the dfferent states between objects and j about the m bnary varables. Concernng the nomnal varable, the dstance of objects and j s usually denoted as ther matchng coeffcent: d(,j)=m 3 -g/m 3, of whch g s the number of the same states between objects and j about the m 3 nomnal varables. 3.. The evaluaton of the probablty dstrbuton functons LCPD uses smple random samplng method to evaluate the probablty dstrbuton functons of the varable. Supposes the number of sample data s ssmp. To evaluate the probablty dstrbuton functons of all knds of varables, we gve the followng Propertes (See reference [5]). Property (non-bas evaluaton) ()The sample means x mean s the non-bas evaluaton of total means X mean. ()x total =nx mean s the non-bas evaluaton of total X total. n x = (3)Sample varance s = (x-x mean ) /(ssmp-) s the non-bas evaluaton of total varance S n = (X-X mean ) /(n-). Property (approxmate dstrbuton about all knds of varables) () For contnuous random varable x, ts evaluaton dstrbuton functon s approxmate normal dstrbuton N (x mena, s x ). The dstrbuton functon s: x ( y xmean) F(x)= exp[ ] dy () s s π x () For bnary varable x, suppose ts states to be 0 and. Of the ssmp samples, the number of 0 states s ssmp0, and the number of state s ssmp. Let p= ssmp0/ssmp, the evaluaton dstrbuton functon s: x X = p x = 0 F(x)= () x = (3) For nomnal varable x, suppose ts states to be sta, sta, sta t, notated as,, t. The number of states of extracted samples s ksta, ksta ksta t, let p =ksta /ssmp (=,, t). Its evaluaton dstrbuton functon s: p <=, j (-< x =,,...,t) F(x)= (3) j= Property 3 The number of smple random samplng samples ssmp s decded on the followng formula: µ ssmp= α S X where µ s the α rx mean double α-quantle of the standard normal dstrbuton, r s the relatve error The probablty dstrbuton based hash functon constructon LCPD takes the followng steps n constructng Hash functon: () Extractng smple random samplng from the total. And the samplng s performed accordng to each dmensonal varable. () Get the approxmate dstrbuton of each dmensonal varable accordng to formula(), (), and(3). Construct the Hash functon as follows: H(x,x,,x m )=F(x )F(x ) F(x m ) (4) The above mentoned method assumes that each varable s ndependent wth one other. For total data, f one varable s multcollnearty wth another, we could transform data by usng the factor analyss method to remove the multcollnearty. The complexty s O(n). Please refer to the proof of Proposton n reference [6]. Proposton Let x, x, x m be ndependent wth each other, H(x,x,,x m ) s the jont dstrbuton of varable X=(x,x,,x m ). Proof:It follows from the property of jont dstrbuton of ndependent varables Level-dvson samplng LCPD allocates data nto dfferent buckets wth Hash functon, namely, dvdng data nto dfferent levels and then extract a stochastc ndvdual from each bucket and hence realze the level dvson samplng. 50

4 () Accordng to the requrement of clusterng () Do smple samplng aganst each column; qualty, settng the number of samplng samples SLAYER, dvdng equally [0,] SLAYER nterval, the dvson ponts are as follows: () Evaluate the dstrbuton functons accordng to formula()()(3); (3) Construct Hash functon H accordng to formula 0= 0,,,, slayer-, slayer =, then q - q- =/ (4); SLAYER (q=,,, slayer) (4) Classfy n ndvduals nto SLAYER buckets () Puttng the n data nto slayer buckets. The accordng to(5); method s the followng: (5) For (nt =; <= sk;++) If the j s row satsfes: (6) { q- <=H(x j, x j,, x jm )< q (q=,, slayer-) (7) Randomly extract an ndvdual from a { q- j j jm q then the j s row belongs to the q s bucket. Proposton (The data characterstc of each bucket)in the context of the bucket dvson method above, the number of data n each bucket s equal n probablty. Proof. By Proposton, H(x, x,, x m ) s the jont probablty dstrbuton functon of varable X=(x,x,,x m ), regardng (x j,x j,,x mj ) s a pont n the m-dmenson super-cube. Snce bucket s dvded accordng to dstrbuton functon equal probablty.(note that t s not dvded by the volume of these super-cube), namely, the super-cube are dvded nto SLAYER equal probablty spaces,.e. SLAYER equal probablty Hash bucket. From the frequency meanng of probablty functon, the frequency of each bucket havng pont s, thus the slayer number of data n each bucket s equal n probablty. Property 4. The precson of level dvson samplng s better than smple random samplng. Namely, the evaluaton varance of level dvson s less than smple random samplng. See reference [7] 4. Hash functon based clusterng method and approxmate methods The PD-Hash-PAM method and PD-Hash-PAM-ß method of LCPD frstly performs smple samplng, evaluatng dstrbuton functon and then construct hash functon and samples the data accordng to the hash functon. After the typcal samples are got, the PAM method s carred out. Detal algorthm s the followng: Algorthm (PD-Hash-PAM Algorthm): Input:X j (=,,,n, j=,,,m), k: the number of classes, sk: the number of samples and SLAYER: the number of ndvduals n each sample. Output: Classfes n ndvduals nto A ={A (=,,,k)},satsfyng:mn(c(a)) Begn bucket and form a sample wth SLAYER ndvduals; (8) Realze PAM method to selected samples and fnd the k-centre ponts of each sample; (9) For each ndvdual X j of the whole dataset, judge whch s closer to X j among the k centre ponts. And get the class A ={ A, A,, A }; (0) Calculate the cost functon of ths cyclng classfcaton C(A ), f C(A ) = mn(c(a ), C(A - ), C(A - ),, C(A )), the k centre ponts are the set of best centre ponts; () } End. In the mplementaton of PAM method, the cost of cyclng wll decrease quckly, but wll tend to become stable untl unchanged. And n the followng cycles, cost functon may decrease lttle. So we utlze the formula n defnton and gve the followng approxmate algorthm. Algorthm Approxmate PD-Hash-PAM Algorthm (PD-Hash-PAM-ß Algorthm) Input:X j (=,,,n, j=,,,m), k: the number of classes, sk: the number of samples and SLAYER: the number of ndvduals n each sample. Output: Classfy the n ndvduals nto A ={A ( =,,,k )}, satsfyng the termnaton crtera n defnton. Begn. Random samples the data n each column;. Evaluate each column s dstrbuton functon accordng to ()()(3); 3. Construct Hash functon accordng to(4); 4. Classfy n ndvduals nto SLAYER buckets accordng to (5); 5. For (nt =; <= sk; I++) 6. { 7. Randomly extract an ndvdual from each bucket, form a sample wth SLAYER samples; 8. Conduct RAM algorthm to all selected samples and fnd the k centre ponts for each sample. The termnaton crtera of PAM algorthm adopts what s k 5

5 defned n defnton ; 9. For each ndvdual X j, n the whole dataset, judge whch centre s closest to X j, and obtan a class A ={,,, A }; A A k 0. Calculate the cost functon, C(A ), of each cyclng classfcaton result. If C(A )=mn(c(a ), C(A - ), C(A - ),, C(A )), the k centre ponts of the th sample are the best centre ponts so far;. } End. Experence. Accordng to practce experence n reference [], sk=5,slayer=40+k optmzes the results. Proposton 3 Let n be the number of ndvduals of the data set. The complexty of PD-Hash-PAM algorthm and PD-Hash-PAM-ß algorthm s O(n), namely, the lnear tme complexty along wth n. Proof Suppose the nteractve tme s t, t s obvous that t, sk, m, k, ssmp, SLAYER<<n The cost n the frst step s ssmp; The second step: The thrd step: The forth step: n From the 5 th to 9 th step: O(t(k*slayer )+k(n-k)) So the cost of the whole algorthm s: ssmp++n+o(sk(t(k*slayer +k(n-k)))=o(n). So the complexty of the algorthm s lnear along wth n. 5. Expermental analyss Ths paper manly conducts the followng comparatve experments: () Tme varaton of PD-Hash-PAM algorthm, PD-Hash-PAM-ß algorthm, PAM algorthm, CLARA algorthm, and CLARANS algorthm along wth dataset sze () Qualty of PD-Hash-PAM algorthm, PD-Hash-PAM-ß algorthm, PAM algorthm, CLARA algorthm, and CLARANS algorthm along wth dataset sze Test Data: got through the randomzed method provded n [8] Platform: CPU=PIV.8G,OS=Wndows000,VC++ Man parameters: m=4,ß=%,k=8 Experment results are shown n Fgure and Fgure. The result show that wth the growth of dataset sze, the runnng tme of PAM algorthm and CLARANS algorthm ncrease greatly on n scale and the spared tme of LCARANS aganst PAM s lmted. Whereas the other 3 methods ncrease on factor of n. And PD-Hash-PAM- % costs the least tme. Fgure shows that n the data numbers of dfferent levels, the cost functon wth mnmum values s the PAM algorthm, the maxmum values belong to CLARA algorthm. Fgure also shows that PD-Hash-PAM algorthm and PD-Hash-PAM-ß algorthm greatly mprove the clusterng qualty. tme (seconds) PAM CLAR CLARANS Hash-PAM Hash-PAM- % 4000 number of dataset Fgure. Runnng tme of PD-Hash-PAM, PD-Hash-PAM-%,PAM, CLARA, CLARANS cost functon number of dataset clar Hash-PAM Hash-PAM- % PAM CLARANS Fgure. Cost functon of PD-Hash-PAM, PD-Hash-PAM-%, PAM, CLARA, CLARANS 6. Conclusons and future work The classcal method n Clusterng, the k-medan algorthm s wth serous defcency n the aspect of effcency and adaptablty for large data se. To solve ths problem, ths paper does the followng work: 5

6 () Proposes a probablty dstrbuton based hash functon and usng ths functon to realze abstractng model LCPD; () Proposes and realzng LCPD model based lnear clusterng method PD-Hash-PAM and approxmate lnear method PD-Hash-PAM- ß; (3) Improves the tme complexty. It ncreasng along wth number of objects s obvous lower than PAM method and CLARANS method. When the number of data ponts s around 0000, t s magntudes slower. It s smlar to CLARA algorthm. Whle the performance of PD-Hash-PAM- % algorthm s lower than CLARA algorthm; (4) Improves the clusterng qualty. It s obvously hgher than CLARA algorthm. When PD-Hash-PAM algorthm works on 8000 szed data, ts clusterng qualty s 55% hgher than CLARA algorthm. The performance of PD-Hash-PAM- % algorthm s 40% hgher than CLARA algorthm; (5) Proves theoretcally that the complexty of the algorthm s O (n). As the problem s complex and mportant, n the future work, we wll study clusterng by usng GEP (Gene Expresson Programmng) and other ntellgent methods [9,0] to mproves the clusterng qualty. Acknowledgements Supported by the Natonal Scence Foundaton of Guangx Grant # , Natonal Basc Research 973 Program of Chna #00CB504, and the Natonal Scence Foundaton of Chna Grant# References [] Jawe Han, Mchelne Kamber. Data Mnng: Concepts and Technques. Morgan Kaufmann Publshers, 00. [] J.MacQueen J.Some methods for classfcaton and analyss of multvarate observatons. Proc.5 th Berkeley Symp. Math. Statst, Prob, pp.8-97, 967. [3] Kaufman L and Rousseeuw P.J. Fndng Groups n Data: An ntroducton to cluster anayss. New youk: john wley&sons,990. [4] Ng R and Han. J. Effcent and effectve clusterng method for spatal data mnng. In Proc. 994 Int. Conf. Very Large Data Base(VLDB 94), Santago, Chle, pp.44-55, Sept.994. [5] Murray R. Spegel, Larry J. Stephens, Schaum s Outlne of Theory and Problems of Statstcs, Thrd Edton. McGraw-Hll Companes, Inc. 999 [6] Yuan Chang-an,Tang Chang-je, Xe Fang-jun et al. Implementaton of Mnng Algorthm for the Regresson Model Based on Spatal Multcollnearty Data. Journal of Schuan Unversty (Natural Scence Edton), vol 4, No., pp , 004. [7] Lesle Ksh, Survey samplng, John Wley & Sons. Inc. 985 [8] Jan,A.K.,Dubes,R.C. Algorthms for clusterng data. Prentce-Hall(988). [9] Yuan Chang-an,Tang Chang-je, Zuo Je et al. Functon Mnng Based on Gene Expresson Programmng Convergency Analyss and Remnant-Guded Evoluton Algorthm. Joural of Schuan Unversty (Engneerng Scence Edton), Vol 36, No.6, pp.00-05, 004. [0] Ru-Lang Wang, Wang We, Yong-Qng Lu, A Quanttatve Study on the Stablty of Delay Dscrete Sngular Systems. Journal of Mathematcal Analyss and Applcatons, Vol.8, pp ,