A Simple Methodology for Database Clustering. Hao Tang 12 Guangdong University of Technology, Guangdong, , China

Transcription

1 for Database Clusterng Guangdong Unversty of Technology, Guangdong, 0503, Chna E-mal: Me Zhang Guangdong Unversty of Technology, Guangdong, 0503, Chna Database clusterng s a preprocess technology for mult-database mnng. Exstng algorthms for database clusterng are successful n terms of havng a cluster, but the tme complexty s hgh or excessve pursut n respect of the non-trval complete clusterng, whch may lead to a bad clusterng or applcaton-dependent. The practcal applcaton of these algorthms have hgh tme nstablty and complexty. In ths artcle, we put forward the applcaton-ndependent database clusterng methodology by usng herarchcal clusterng method to avod nstablty and reduce tme complexty. Ths methodology s called Database Herarchcal Clusterng. We frstly construct a mult-obectve optmzaton problem, and then use herarchcal clusterng algorthm to fnd the optmal cluster structure thought. We also use the cophenetc correlaton coeffcent to evaluate the best cluster. Experments on the synthetc databases and the real-world databases show that our method of clusterng stablty features lower tme complexty than that of the BestClassfcaton whle also hghlghtng strong generalzaton ablty. PoS(CENet05)09 CENet05-3 September 05 Shangha, Chna Speaer Correspongdng Author Copyrght owned by the author(s) under the terms of the Creatve Commons Attrbuton-NonCommercal-NoDervatves 4.0 Internatonal Lcense (CC BY-NC-ND 4.0).

2 . Introducton As we now, there are many large companes whch have dfferent database dstrbutons n dfferent branches therefore, mult-database mnng s an mportant branch of data mnng whch has become more and more mportant. In order to reduce the search cost, we need to determne whch database s assocated wth our data mnng. Ths mportant step we call the database selecton []. Furthermore, let s thn a need to handle the multple large database of Mult-Natonal Corporaton. Ths company may need to fnd the non-proft assocaton analyss proect (product).the ultmate goal s to dentfy those wthout much proft nor other products to promote proftable products. The correlaton analyss may fnd such products, thereby the company may termnate the transactons of products. The nature of the analyss may need to dentfy smlar databases. We note that, f the two databases contan many smlar transactons, the two databases are consdered to be smlar f the two transactons contan many of the same goods, the two transactons are smlar. In ths sense, the more same frequent temsets two databases contan, the more smlar they are []. We could cluster mult-databases accordng to the smlarty between two databases. Then, carry out mnng n the same class n the database. It s a sgnfcant procedure n terms of the analyss of explorng patterns, groupng, decsonmang and machne-learnng. In ths artcle, we manly dscuss the transacton database clusterng.. Related Wors As to the mult database mnng, the frst dea (sngle database mnng) s that a number of data n the database together consttutes a sngle data set[]. Lu et al. put forward a search related database of mult database mnng technology[3]. Ther man researches on the dentfcaton were assocated wth the database applcaton. Zhang et al.desgned a new mult-database mnng process to mne the mult-databases[4]. The database clusterng was the frst step, and then fnd the local mode. The paper desgned a smlarty by usng the tems of transacton or hgh-frequency rules. Furthermore, they put forward algorthms called GreedyClass and BestClassfcaton, n whch, a database was chose to be one class frstly and then udg whether or not the next of database s ncorporated nto the nown classes or consdered as a new class teself. H. L et al. [5] desgned an mproved method to reduce the tme complexty of BestClassfcaton. The mproved algorthm can obtan the best classfcaton correctly for m gven databases and the tme complexty has been reduced from 4 3 O( n m m ) to O( n m m ). Anmesh Adhar et al.proposed two smlarty measures by the frequent tem sets of databases and desgned an arthmetc to cluster the data set. In order to mprove the effcency of cluster., the multple data clusterng method s proposed based on hgh-class coheson and low couplng between classes of applcaton-ndependent. There are other effectve researches. Wu et al.proposed a pattern recognton method of weghted by a mult database[6]. Yn and Han put forward a new strategy for hgh dmensonal heterogeneous database, ths strategy may not apply to the aggregaton transacton database[7]. Yn et al.proposed two scalable hgh classfcaton algorthms: CrossMne-Rule (based on the assocaton rules) and CrossMne-Tree (based on the decson tree)[8]. Bandyopadhyay et al.,based on the aggregaton of K-means algorthm sutable for the mage sensor networ, proposed the somorphsm of envronmental data technology [9]. PoS(CENet05)09 3. Clusterng Multple Databases 3. Smlarty measures Defnton : appont D { D, D,, Dm} as a set for all database. In the case of D, ths smlarty matrx s DSM ( D, ) as descrbed by the measure of smlarty sm. Ths dstance

3 matrx s DDM ( D, ) expressed by the measure of dstance -sm, they are square matrx, whose (, ) element are DSM, ( D, ) sm ( D, D, ) and DDM, ( D, ) sm( D, D, ) respectvely, whle D, D D,,,,, m. As to m databases, the two n one group, there are C m m( m ) databases. As to not a group of database, we calculated the smlarty of them. In case of hgh smlarty or dstance, the two database may be assgned to the same class. Defnton: there have m data pool D, D,, D m. appont D s a set for m whle C={ c, c,..., c n },( n m ) s a canddate cluster (partton) of D, D,, D m,f () c, for n. () D c c, (3) c c,for,, n. where, c ( n ) means a class of C. Defnton3: gven D s a assembly of m databases, that s D,,D m. C={ c, c,..., },( n m ) s a canddate cluster of D. The smlarty between databases c and c under threshold α s defned as follows: Itemsets sm( c, c, ) Itemsets c c, Itemsets, Itemsets c c,, Defnton 4: D specfes a dataset for m databases, that D,D,,D m. C={ c, c,..., },( n m ) s a canddate cluster of D. The class dstance matrx CDM ( ) of C expressed by the measure of dstance -sm, s a n-order matrx, The frst (, ) th element CDM ( ) sm( c, c, ),,,,, n., 3. Relevance of Databases Defnton 5: D s a collecton of databases m and C s a canddate cluster of D, D={D,D,,D m }. Under measure sm, t s defned as the sum of dstances as follows: dst ( c) ( sm( D, D, )) (3.) D, D c from the database D and D s expressed as sm(d,d,α). The shorter the dstance between two databases s, the hgher the sum of smlarty of a class wll be. Defnton 6: desgnate a data set D, and C={ c, c,..., },( n m ) s a canddate cluster of D, D={D, D,, D m }. The sum of dstance of C under threshold α s sum - dst( ) dst ( c) (3.3) The above defnton s to reveal the sum of dstance of a cluster based on the dstance of all classes. Lower value means hgher coheson whch s an mportant parameter to a good cluster. Defnton 7. D s a partcular data set and C={ c, c,..., } s a canddate cluster of D, D={D, D,, D m }. The couplng of cluster C s couplng( ) sm( c, c, ) (3.4) c, c C The defnton couplng represents the relevance of each par of classes n a canddate cluster. cc (3.) PoS(CENet05)09 3

4 3.3 Fndng the Best Clusterng A good cluster of multple database must be of hgh coheson, low couplng and as fewer classes as possble[][][4].the best cluster s selected among alternatve canddate clusters. The tas of our clusterng s to fnd a clusterng functon g : D C by usng the database smlarty matrx DSM ( D, ) and the database dstance matrx DDM ( D, ) of D so that the expected features sum - dst( ), couplng( ) and C are taen to a mnmum. We can wrte: mn sum - dst( ) mn couplng( ),subect to all feasble g. (3.5) mn C Such ssues refer to the more obectve (standard) optmzaton problem [0]. Here our problem s three-crtera optmzaton. Our lnear weghted the sum method [] and put t nto a sngle obectve optmzaton problem, as follows: mn l sum- dst( a) + l couplng( a) + l C 3 (3.6) subect to all feasble g, (3.6), where 0,(,,3 ), s the crteron weght. And then we can use the lnear functon as follows: x MnValue y,( MnValue x MaxValue) (3.7) MaxValue MnValue To transform the obectve functons nto the range of 0~ then we have the normalzed obectve functons as follows: sum - dst( ) 0 sum - dst( ) sum - dst( ), m( m ) 0 m( m ) couplng( ) 0 couplng( ) couplng( ), (3.8) m( m ) 0 m( m ) C C. m Accordng to Formulas (3.6)(3.8), we gve the best cluster defntons: Defnton 8. The goodness of a canddate cluster C={ c, c,..., } use sm smlarty that s defned such as : goodness( C) mn sum dst C couplng - (, ) ( ) 3 C (3.9) When,(,,3), t s nown as an unform calculaton weght, whch means that all obectves are equally mportant. As to our tas, we treat the sum-dst, couplng and C symmetrcally n the methodology. Let F ( ) sum - dst( ) couplng( ) C,then we have goodness( C) mn F( ) (3.0) PoS(CENet05)09 4. Algorthm of the Clusterng Snce the number of the canddate clusters s much larger because there are not a few databases needed to be clustered, t s mpossble to obtan all of them. As we now, the databases wth hgh smlarty should be clustered nto one class, so we can cluster databases based on the smlartes herarchcally. The procedure Database Herarchcal Clusterng (DHC) for generatng canddate clusters and dentfyng the best one s adopted n Procedure. 4

5 Procedure. Database Herarchcal Clusterng(DHC) begn Input: D (m):databases, α: threshold value Output: C best : the best cluster () fnd the frequent tem sets of each D under threshold α () construct the database smlarty matrx DSM ( D, ) and the database dstance matrx DDM ( D, ) (3) let = (4) construct a canddate cluster C. C ={ c, c,, cm } where c ={D }, m (5)let CDM ( C, )= DDM ( D, ),calculate F( C, ) sum - dst ( C, ) coupln g( C, ) C (6)let goodness (C best )= F ( C, ), Cbest C (7)whle ( m ) do begn (7.)fnd the par of classes ( c, ), the dstance value CDM p,q ( C, c p q mnmum n the upper trangle of CDM( C, ) (7.) let =+ (7.3) let c c p c q )of whch s the 7.4) let C = C { c p }{ c q }+{ c } 7.5) (7.5) f m begn construct CDM( C, ) and calculate F ( C, ) else calculate F ( C, ) end f (7.6) f F ( C, ) < goodness (C best ) begn let goodness (C best )= F ( C, ), Cbest C end f end for (8) output the best cluster C best End procedure Procedure Herarchcal Clusterng generates m canddate clusters and obtans the best cluster accordng to goodness. When the allocaton threshold α, step ntalzes Procedure (), () and (3), Procedure(4) constructs a canddate cluster that each database to be one class. Step (5) and (6) ntalze the goodness. Step (7) fnds the mnmum goodness and the best cluster by usng herarchcal process. Step (8) outputs the best cluster C best.5. ExpermentsWe conducted a seres of experments to verfy the valdty of our approach. We use a synthetc database T0I4D00K to splt nto ten database experments on the effectveness of our proposed algorthm. The T0I4D00K splt nto ten databases, the basc characterstcs of the ten databases as shown n Table. As to a database, accordng to the threshold of change, we get dfferent optmal clusterngs, more small. The algorthm runs longer, as shown n Table. We stll use smlarty of the two-dmensonal tables (Table 3). Compare our algorthm wth BestClassfcaton, the results are gven n table 4. As for other values of BestClassfcaton, whch are not the most clusterng, we only gve two value comparson algorthm. From Table 4, our algorthm s superor to DHC BestClassfcaton. PoS(CENet05)09 5

6 Database DB DB DB3 DB4 DB5 DB6 DB7 DB8 DB9 DB0 quantty of transactons 0,000 0,000 0,000 0,000 0,000 0,000 0,000 0,000 0,000 0,000 Table : Enter the Database Features Datast Set D Mm support Table : Enter the Database Features Transacton s average length Best cluster The average number of frequent tem sets Item number set {{DB,DB 3,DB 7,DB 5,DB 8,DB 0},{DB,DB 4,DB 5,DB 9,DB 6}, {{DB 7,DB 3,DB },{DB,DB 4},{DB 5,DB 9,DB 6},{DB 8,DB 0}} {{DB,DB 3},{DB 4,DB },{DB 6,DB 9,DB 5},{DB 7},{DB 8},{DB 0}} {{DB,DB 3},{DB,DB 5},{DB 4,DB 9},{DB 7},{DB 9},{DB 3},{DB 0}} {{DB 4,DB },{DB,DB 5},{DB 4,DB 9},{DB 9},{DB 4},{DB 5},{DB 0}} {{DB 3},{DB,DB 5}{DB },{DB 3,DB 7},{DB 4},{DB },{DB 5},{DB 0}} {{DB 5},{DB 3,DB 6},{DB },{DB 5,DB 4},{DB 8},{DB 9},{DB 8},{DB 0}} {{DB },{DB 5},{DB 7},{DB },{DB 5,DB 6},{DB 3},{DB 9},{DB },{DB 0}} {{DB 0},{DB 9},{DB 8},{DB 7},{DB 6},{DB 5},{DB 4},{DB 3},{DB },{DB } sm DB DB DB 3 DB 4 DB 5 DB 6 DB 7 DB 8 DB DB DB 3 DB 4 DB 5 DB 6 DB 7 DB Tme 5mn 5mn mn 0mn 7mn 5mn mn 68s 45s PoS(CENet05)09 Table 3: Smlarty Telaton Algorthm Best cluster Tme (ms) BestClassfcaton 0.6 {{DB 7},{DB,DB 4},{DB }{DB 3},{DB 6},{DB 5},{DB 8}} 0 DHC {{DB 6,DB 3,DB 5},{DB 7},{DB 8},{DB 6,DB 4,DB }} 4 BestClassfcaton {{DB 8},{DB 7},{DB 6},{DB 5},{DB 4},{DB 3},{DB }, {DB }} 5 DHC {{DB },{DB },{DB 3},{DB 4},{DB 5},{DB 6},{DB 7}, {DB 8}} 0 Table 4: Expermental Results on 3 Smlarty 6

7 5. Concluson Ths paper presents a mult database classfcaton method based on hgh coheson and low couplng. The defnton of dstance used to measure the coheson and the defnton of couplng degree s used to measure the Mult-target optmzaton problem. Afterwards, the herarchcal clusterng algorthm s used to fnd the deologcal structure for optmal clusterng. The we use an artfcal database T0I4D00K and a real database smlarty n two-dmensonal tables shows effectveness of the algorthm model then we carry on the contrast experment wth the BestClassfcaton algorthm. The experments show that our method of clusterng stablty s strong and the tme complexty s lower than that of BestClassfcaton whle featurng strong generalzaton ablty. References [] X.Wu,C.Zhang,S. Zhang. Database classfcaton for mult-database mnng, Informaton Systems. 30 (), 7 88(005). [] A. Adhar, P.R. Rao. Effcent clusterng of databases nduced by local patterns. Decson Support Systems. 44(4),95-945(008). [3] H. Lu, H. Lu, J. Yao. Toward multdatabase mnng: dentfyng relevant databases, IEEE Transactons on Knowledge and Data Engneerng 3(4): (00). [4] D.Yuan, H. Fu, Z. L, H. Wu. An applcaton-ndependent database classfcaton method based on hgh coheson and low couplng, Journal of Informaton & Computatonal Scence 7(),- 6(0). [5] H. L, X. Hu, and Y. Zhang. An mproved database classcaton algorthm for mult-database mn-ng. Proc. of Fronters of Algorthmcs Worshop n LNCS. Hefe, Chna, 009: [6] X.D.Wu, S.C.Zhang. Syntheszng hgh-frequencyrules from dfferent data sources, IEEE Trans. Knowledge Data Eng. 5 (): (003). [7] X. Yn, J. Han. Effcent classfcaton from m ultple heterogeneous databases. In:Proceedngs of 9-th European Conf. on Prncples and Practce of Knowledge covery n Databases,Lecture Notes n Computer Scence Volume 37,pp,404 46(005) [8] X. Yn, J. Han, J. Yang, PS. Yu. Effcent classfcaton across multple database relatons: A crossmne approach. IEEE Transactons on Knowledge and Data E ngneerng 8(6): (006) PoS(CENet05)09 [9] S.Bandyopadhyay, C. Gannella, U.Maul, H. Kargupta, K.Lu, S. Datta. Clusterng dstrbuted data streams n peer-to-peer envronments. Informaton Scences 76(4): (006) [0] C. Hllermeer. Nonlnear multobectve optmzaton. Brhäuser Basel.pp,35,00. [] L. Zadeh. Optmalty and non-scalar-valued performance crtera. IEEE TA 8():59 60(963). 7