MINIMIZATION OF THE VALUE OF DAVIES-BOULDIN INDEX

Transcription

1 MIIMIZATIO OF THE VALUE OF DAVIES-BOULDI IDEX ISMO ÄRÄIE ad PASI FRÄTI Departmet of Computer Scece, Uversty of Joesuu Box, FI-800 Joesuu, FILAD ABSTRACT We study the clusterg problem whe usg Daves-Bould dex as the optmzato crtero. The problem s to partto a gve data set of vectors to M clusters so that the value of the Daves-Bould dex s mmzed. The dex dffers from the mea square error that t also taes to accout the dstace betwee code vectors. Ths leads to the problem that settg the code vectors as the cetrods of the clusters does ot gve the optmal placemet for mmzg the Daves-Bould dex. We derve formula for optmzg the locato of the cluster cetrods. eywords: patter recogto, algorthms, cluster aalyss, umber of clusters.. ITRODUCTIO Usupervsed classfcato s a mportat part of may applcatos mage processg ad aalyss. The ma goal s to obta a hgh qualty clusterg for a set of put vectors mmzg a gve optmzato crtero []. Clusterg s a combatoral optmzato problem where the am s to partto a set of data vectors to a umber of classes. Data vectors wth smlar features should be grouped together ad vectors wth dfferet features to dfferet groups. There are several establshed methods for geeratg a clusterg [,, 3]. The methods are usually desged to mmze the mea square error betwee the data vectors ad ther cluster cetrod. Mea square error, however, s ot sutable for determg the umber of clusters, sce t decreases as the umber of clusters creases. I fact, t would result a stuato where the umber of clusters equals to the umber of data vectors, whch yelds to error value of zero. We are therefore forced to use other crtera ad ths may lead to chages the algorthm tself. Geeralzed Lloyd algorthm (GLA) [4] s a wdely used algorthm that sees local optmum by teratvely chagg betwee two represetatos. The GLA s also used as a tegral part of several more sophstcated algorthms, such as stochastc relaxato [5], determstc aealg [6], geetc algorthm [7], ad radomzed local search [8]. The optmzato steps of the GLA are bascally desged to the mmzato of the mea square error. However, there s o guaratee that these steps would be optmal for other crtera, such as Daves-Bould dex (DBI) [9]. I ths study, we show that the cetrod of the cluster s ot the optmal selecto of the cluster represetatve (code vector) for mmzg DBI. We the study the problem for the pot of vew of a sgle cluster. We show that the code vector of a cluster should be moved from the cetrod to the opposte drecto from the earest eghbor cluster. We also derve formulas that gve the exact amout of the movemet. As a result, we obta a method for optmzg the evaluato crtero drectly. We the apply the proposed method wth the clusterg algorthm preseted [8]. Fgure : Example of the partto of a data set (the small blac dots). The cluster cetrods are represeted by the large gray dots ad the cluster boudares by Voroo dagram.. CLUSTERIG PROBLEM We use the followg otatos: umber of data vectors. umber of vectors cluster. M umber of clusters. umber of attrbutes. X Set of data vectors X = {x,..., x }. P Set of cluster dces P = {p,..., p }. C Set of M code vectors, C = {c,..., c M }. d(a, b) Dstace betwee vectors a ad b.

2 Mea square error () s the average parwse dstace betwee data vectors ad the correspodg cluster represetatve (code vector). Usually d s Eucldea dstace, but other metrcs are also used. Gve the set of code vectors (C), ad the set of cluster dces (P) for each trag vector, ca be calculated as d( x, c p ) () There exst several clusterg algorthms, of whch we ow recall two. Both of the algorthms tae the data set (X), the umber of clusters (M), ad a tal soluto (e.g. radom clusterg) as put, ad produce the partto (P), ad/or the set of code vectors (C) as the output.. Geeralzed Lloyd algorthm, also ow as LBG or -meas, starts wth a tal codeboo, whch s teratvely mproved by usg two optmalty codtos tur. I the frst step, each data vector x s mapped to the earest code vector c. I the secod step, a ew set of code vectors s obtaed by replacg each code vector as the cetrod (average) of the data vectors the cluster. The ew soluto s ever worse tha the prevous oe [3]. The soluto coverges to the earest local optmum. The algorthm s easy to mplemet but very sestve to the tal soluto due to the fact that oly local chages are made to the soluto. Pseudocode of the GLA s preseted Fgure. GLA(X,C) retur C,P REPEAT C prevous C; FOR all [, ] DO p such that x s closest to c ; FOR all [, M] DO c Average of x, whose p = ; UTIL C = C prevous ; Retur C,P; Fgure : Pseudocode for GLA. Radomzed local search (RLS) [8] overcomes the sestvty to the tal soluto. It starts wth a radom soluto, whch s the mproved by a predefed umber of teratos. At each step, a ew caddate soluto s costructed as follows. The clusterg structure of the curret soluto s frst modfed by replacg a radomly chose code vector by a radomly chose put vector. The partto of the ew soluto s the adusted respect to the modfed set of clusters. Two teratos of the GLA are appled to fe-tue the tral soluto. The caddate s evaluated ad accepted f t mproves the prevous soluto. The algorthm s terated for a fxed umber of teratos. Pseudocode of the RLS algorthm s preseted Fgure 3. LS(X,M) retur C, P C Set of radomly chose data vectors. FOR all [, ] DO p such that x s earest to c ; FOR a TO umberofiteratos DO C ew C; c Radomly chose data vector C ew, P ew GLA(X,C ew ); IF (X, C ew, P ew ) < (X, C, P) THE C C ew ; P P ew ; ED IF ED FOR retur C, P; Fgure 3: Pseudocode for local search. 3. SOLVIG THE UMBER OF CLUSTERS I order to determe the correct umber of clusters, we must have a crtero that taes to accout the umber of clusters ad a algorthm that vares the umber of clusters. otable property of s that t's value decreases as the umber of clusters creases. I fact, the optmal soluto would be = M, ad the = 0. Ths maes the usutable for determg the umber of clusters. We use Daves-Bould dex [9] as the crtero because, our expermets, we have foud t our expermets qute relable amog several alteratve measures; wth regard to potg out the correct umber of clusters. Daves-Bould dex taes to accout both the error caused by represetg the data vectors wth ther cluster cetrods (tra cluster dversty) ad the dstace betwee clusters (ter cluster dstace). The tra cluster dversty of a cluster s calculated as p d( c, x ) () The ter cluster dstace of the cluster ad s measured as the dstace betwee ther cetrods c ad c. Wth these two measures we ca the calculate the closeess of the two clusters as the sum of ther -values dvded by the dstace of ther cetrods: R, c c (3) The closeess of the cluster s proportoal to ther -values ad drectly proportoal the dstace of ther cetrods. Small values of R, dcate that the clusters are separated ad large values that the clusters are close to each other. I order to calculate DBI-value, we tae for each cluster the maxmum value of (3) ad deote t as R :

3 R max R, (4) The overall Daves-Bould dex s defed as: DBI M M R x, c, (5) The smplest algorthm that determes the correct umber of clusters ust vares M over a rage of values [M m..m max ] a loop ad geerates clusterg for every umber of M usg ay algorthm. We use the radomzed local search (RLS) for geeratg the clusterg for each value of M. We refer ths approach as brute force local search. The pseudocode for the algorthm s preseted Fgure 4. BFLS(X, M m, M max ) retur C, P C best, P best LS(X, M m ); FOR M m + TO M max DO C,P RLS(X, ); IF DBI(X,C,P) <DBI(X, C best, P best ) THE C best C; P best P; ED FOR retur C best, P best ; Fgure 4: pseudocode for brute force local search. 4. SEEIG FOR OPTIMAL DBI The data requred to evaluate the clusterg s the same for both ad DBI. Both measures eed the cluster represetatves (code vectors) ad the partto, ad ths formato s already avalable the GLA ad the RLS algorthms. The optmal partto ca also be geerated the same way for both measures. Parttog s performed by mappg each vector to the cluster that has the closest code vector. p argmd (6) M It s therefore straghtforward to apply DBI stead of the BFLS algorthm. The ma problem les the fact that whle we are usg DBI the BFLS algorthm for evaluatg the solutos we are stll usg the RLS ad GLA to refe the solutos. Mmzg certaly mmzes the deomator of (3), but t does ot do the same for the omator. I.e., the code vectors may move closer to each other whe we apply the GLA, but for DBI, t mght be better to move the code vectors away from each other by a small amout. Although movg the code vector may crease, the move s beefcal because the offset the crease of the dstace betwee code vectors offsets ths crease, resultg a overall decrease of DBI. We propose to use DBI ot oly to select amog the several clusterg wth varous umber of clusters, but also for optmal placemet the code vectors. I order to do ths, we develop a method to place the code vectors so that the crease does ot offset the crease the dstace betwee the code vectors. We must therefore determe the drecto ad the magtude of the shft. We wll preset formulas for local optmzatos oly;.e., the case of each cluster separately. The globally optmal settlemet of the vectors s much more dffcult to obta because the shft of a sgle code vector may cause chages the optmal placemet for aother vectors. Moreover, t s also possble that after movg a code vector of a cluster away from ts eghborg cluster, the cluster wll o loger be the earest cluster for. We wll show later that the drecto of the movemet does't affect the crease of, provded that the orgal placemet of the code vector s the cetrod of the cluster. We wll also show that the optmal drecto of the movemet for mmzg DBI s the opposte from the drecto to the cetrod of the earest cluster. We presume that the dstace fucto use s the Eucldea dstace. Lemma : The drecto of the movemet s the opposte from the cetrod of the earest eghbor cluster. Lemma : The magtude of the optmal movemet,.e. the legth of the vector to be added to the code vector s: c c ( ) c c (7) We prove the lemmas the Chapter 5. The effect of the movemet s llustrated Fgure 5. The movemet s mplemeted sde the GLA as a addtoal fe-tug step, as show Fgure 6. I ths case, the code vectors are moved after each GLA terato. Aother approach would be to mae the fe-tug outsde the GLA the ma loop of the RLS algorthm. However, ths s oly a matter of mplemetato as we expect the effect to be the same both approaches. Fgure 5: The drecto of the movemet..

4 GLA(X,C) retur C,P REPEAT C prevous C; FOR all [, ] DO p such that x s closest to c ; FOR all [, M] DO c Average of x, whose p = ; FeTueCodeVectors(C,P,X); UTIL C = C prevous ; Retur C,P; Fgure 6: pseudocode for modfed GLA. The route FeTueCodeVectors refes the locato of the code vectors. However, t s ot clear how the fe-tug should be mplemeted as there are a umber of practcal problems. For M clusters, we ca draw a graph that shows the relatoshps of the clusters, the sese how they cotrbute to the R -values of other clusters. The graph mght loo le that of Fgure 7. A C B D Fgure 7: earest eghbor graph of the clusters demostratg the cotrbutors for R for each cluster. For each cluster, we face the problem of whch oe to move. I some cases the selecto s easy. For example, whe cosderg Fgure 7, t s better to move A, sce movg B mght move t closer to D ad thus worse the stuato there. Usg the same logc, t s better to move E, H ad G tha to try to move F, sce most lely F would move towards some other code vector. If we th of the umber of edges comg to ad leavg a ode, we ca costruct a heurstc that the odes wth less edges should be moved before others. Ths seems to be a good rule, but t stll does ot guaratee a optmal placemet of the vectors. For example, t s possble that the code vector moves too far from ts eghbor, ad as a cosequece, the R, decreases so much that t wll o loger be the maxmum. For example, movg C mght cause t to become closer to A, whch meas that we should move C the amout where R C,D = R C,A ad ot ay further. We mplemeted two dfferet methods. The frst method (smple method) ust taes to accout the code vector of the earest cluster (wth whch the E F G H maxmum of R, ). The secod method (averagg method) taes to accout all clusters that had ther maxmum R, obtaed wth the cluster questo. The dsplacemet s the calculated for all cluster pars ad the the average of these vectors was tae as the fal dsplacemet. We also expermeted wth varous ways to wegh the vectors order to get at least the locally optmal soluto but we dd ot foud ay feasble way to compute the globally optmum placemet (foud wth exhaustve search our tests). Amog the expermeted methods we used () per-cluster s as defed by equato () ad () the R -values (4) for the eghbor clusters questo. These two expermetal methods produced a offset vector that had more correct magtude ad more correct drecto tha the averagg method. Thus, the use of a more complex algorthm dd ot seem to be ustfed. 5. PROOF OF LEMMAS AD Frst we show that the crease does ot deped o the drecto of the movemet. The we determe the drecto to whch the code vector should be moved, ad fally, we solve the magtude of the movemet. Lemma 3: Icrease depeds oly o the magtude of the movemet vector, provded that the orgal code vector s the average of the trag vectors cluster. Proof of lemma 3: Let be the per-cluster wth the code vector c ad let be the movemet vector that s to be added to c. The for the cluster s: c p p p x ( c ( c x x ) c x ) (8) Usg ths, we derve formula for the chage of whe the code vector s moved to pot c + from the pot c :

5 ( p p p c c p ) x c ( c x c x x c p ) x x c x p (9) By the assumpto that c equals to the average of the pots ( x ) the cluster, we get: (0) Thus, the drecto of has o effect o the ad the oly affectg factor s the magtude of the vector. Proof of lemma : I order to mmze R, we must maxmze the dstace betwee the code vectors c ad c stat. We presume that oly oe code vector (c ) s beg moved ad the (c stat ) other remas statoary. Logcally, should pot away from c stat. Let g be the vector c - c stat. We determe the agle betwee the vectors ad g order to maxmze the dstace r, see Fgure 8. The legth of ca be assumed to be fxed. r g c stat Fgure 8: Elemets for determg the agle of. c c + Sce the dstace s always o-egatve we ca solve the maxma of the square dstace betwee c + ad c stat : r g cos r g cos g cos s g g cos s g cos g cos s () The legths of the orgal dstace vector g ad the movemet vector are postve costats. The oly varable equato () s, ad therefore t s easy to see that the maxma s obtaed whe cos =. Thus, the optmal value s =0, whch proves the lemma. Proof of lemma : The ext questo s to determe how much the code vector should be moved. Ths depeds o the per-cluster s. We must fd that mmzes the R, whe c s moved uts away from c. The drecto of s ow fxed ad we eed to determe ts legth oly. The crease s accordg to (0), ad the crease the cluster dstace s, sce g ad have the same drecto. Thus, the ew R value after the movemet s: R ' g () To fd out the that mmzes (3) we tae ts dervatve ad fd ts zero-crossg pots. Ths yelds to: g ( ) ' R d (3) g The dvsor s o-zero because the code vectors are ot detcal Therefore, we eed to fd the zerocrossg pots of the deomator. Sce t s a secod-order polyomal wth regards to, we ca solve t usg formula: g g 4 g g 4( ) (4) Sce egatve vector legths mae o sese, we get (7), ad the lemma s therefore prove. 6. Test results We have tested the proposed methods wth two dfferet trag sets. The frst set s the oe show Fgure, ad the secod set s Fgure 9. The data sets are artfcally geerated so that they cota 5 clusters. The clusters are clearly vsble the frst set. I the secod set, the clusters overlap ad there are o clear boudares, but t s stll possble to detfy the clusters. We ra the BFLS algorthm 00 tmes wth ad wthout the optmzatos related to the DBI. The same settgs wth regard to the umber of teratos ad the rage for the umber of clusters were used all tests. For each umber of clusters, we appled the local search algorthm usg 5000 teratos ad for

6 each terato, two teratos of the GLA were performed. The DBI-values of the orgal method are summarzed Fgures 0 ad for the rage from 3 to 7 clusters. The correct umber of clusters (5) was foud all the 00 rus whe appled to the frst data set. Thus, the method ca dstgush the correct umber of clusters qute easly wth trag sets where the clusters are more clearly separated. I the case of the secod set, the dstcto s ot as clear as the DBI-values are vrtually equal for M=4 ad M=5, see Fgure. The method fds the correct umber of clusters oly 8 tmes out of the 00 test rus. The resultg clusterg s show Fgure 9, where the mssg cluster should be the mddle. DBI-value umber of clusters Max Avg M Fgure : DBI-values for the test set usg the orgal method. The results are from 00 test rus. The reaso for poor performace mght have bee the low umber of GLA-teratos. We therefore tred to mprove the method by creasg the umber of the GLA-teratos to see whether t would solve the problem. We appled 0 GLA-teratos for each local search teratos. The results are show Fgure, where the blac bars represet the modfed method. The result clearly shows that the modfcato dd ot help but, o the cotrary, t made the problem eve worse. The results of the ew methods (smple method ad averagg method) are summarzed Fgure 3. They show that wthout the mprovemets, oly 8 rus out of 00 foud the correct umber. Whe applyg the smple optmzato, the results do ot chage much from the orgal. The correct umber of clusters s foud 6 tmes. Whe applyg the more averagg method, the correct umber of clusters s foud 7 tmes. DBI-value Fgure 9: The secod data set. Max M umber of clusters % umber of clusters Orgal 0 GLAteratos Fgure : The umber of tmes the dfferet umbers of clusters are foud wth the orgal method by creasg the umber of the GLA teratos. Fgure 0: DBI-values for the test set usg the orgal method. The results are from 00 test rus.

7 % umber of clusters Orgal Smple Averagg 6 7 Fgure 3: The umber of tmes the dfferet umbers of clusters ar foud wth dfferet optmzatos Orgal Smple Averagg Fgure 4: Average DBI-values wth dfferet optmzatos. The results of the optmzatos are studed further Fgure 4. It shows that the optmzed method ca fd eve a lower DBI at the mmum pot, but stll for the correct (4) umber of clusters. The results for the correct umber of clusters (5) are eve worse although the dfferece s eglgble. Ths dcates that the problem s more related to DBI tself tha to the exact optmzato of the lcoato of the code vectors. 7. COCLUSIOS We have derved formula for optmzg the locato of the code vectors usg Daves-Bould dex. The formula was the appled the clusterg whe solvg the correct umber of clusters. The results were theoretcally well argued but ther mpact the practcal applcato was ot sgfcat. From the expermets, we ca draw the followg tetatve coclusos. Frstly, t seems that the DBI wors well whe the clusters are clearly separable. The suboptmalty of the cetrod locato seems ot to be a serous problem whe we use the brute force search strategy the clusterg. The problem arses oly extreme stuatos whe the clusters are greatly overlappg. Secodly, t s possble that the formula gvg the locally optmal locato of a sgle code vector s ot suffcet. Istead, globally optmum placemet of the code vectors could resolve the problem but t s computatoally ot feasble. Moreover, t s ucerta whether the DBI tself s the correct measure as t has some heurstc elemets. For example, t cosders for each cluster oly the earest cluster the measuremet. Further studes would therefore be eeded to reach a coclusve aswer to the problem. REFERECES:. B.S. Evertt: Cluster Aalyss. 3rd edto, (Lodo: Edward Arold / Halsted Press, 993).. L. aufma ad P.J. Rousseeuw, Fdg Groups Data: A Itroducto to Cluster Aalyss, (ew Yor: Joh Wley Sos, 990). 3. A. Gersho ad R.M. Gray, Vector Quatzato ad Sgal Compresso. (Dordrecht: luwer Academc Publshers, 99). 4. Y. Lde, A. Buzo, R.M. Gray: A algorthm for vector quatzer desg. IEEE Trasactos o Commucatos 8 (), 980, Zeger ad A. Gersho: Stochastc relaxato algorthm for mproved vector quatzer desg, Electrocs Letters, 5 (4), 989, Rose, Determstc aealg for clusterg, compresso, classfcato, regresso, ad related optmzato problems, Proc. of the IEEE 86 (), ov. 998, P. Frät, J. värv, T. auorata, O. evalae: Geetc algorthms for large scale clusterg problems, The Computer Joural 40 (9), 997, pp P. Frät ad J. värv: Radomzed local search algorthm for the clusterg problem, Patter Aalyss ad Applcatos, 000 (to appear). 9. D.L. Daves ad D.W. Bould: A cluster separato measure, IEEE Trasactos o Patter Aalyss ad Mache Itellgece (), 979, 4-7.