A Scatter Search Approach for the Minimum Sum-of- Squares Clustering Problem

Transcription

1 A Scatter Search Approach for the Mnmum Sum-of- Squares Clusterng Problem Joaquín A. Pacheco Department of Appled Economcs, Unversty of Burgos, Plaza Infanta Elena s/n BURGOS 09001, SPAIN Tf: ; fx ; emal: Latest verson: September 19, 003 Abstract A metaheurstc procedure based on the Scatter Search approach s proposed for the non-herarchcal clusterng problem under the crteron of mnmum Sum-of-Squares Clusterng. Ths algorthm ncorporates procedures based on dfferent strateges, such as Local Search, GRASP, Tabu Search or Path Relnkng. The am s to obtan qualty solutons wth short computaton tmes. A seres of computatonal experments has been performed. The proposed algorthm obtans better results than prevously reported methods, especally wth small numbers of clusters. Keywords: Clusterzaton, Metaheurstcs, Scatter Search, Local Search, GRASP, Tabu Search, Path Relnkng 1. Introducton Consder a set X = {x 1, x,..., x N } of N ponts n R q and let m be a predetermned postve nteger. The Mnmum Sum-of-Squares Clusterng (MSSC) problem s to fnd a partton of X nto m dsont subsets (clusters) so that the sum of squared dstances from each pont to the centrod of ts cluster s mnmum. Specfcally, let P m denote the set of all the parttons of X n m sets, where each partton PA P m s defned as PA = {C 1, C,..., C m } and where C denotes each of the clusters that forms PA. Thus, the problem can be expressed as: where the centrod x s defned as The problem can be wrtten as PA m mn x x, P x m = 1 x C l xl C l = 1 xl, wth n = C. n N mn x x k, l= 1 where k l s the cluster to whch pont x l belongs. The desgn of clusters s a well known exploratory Data Analyss ssue called Pattern Recognton. The am s to fnd whether a gven set of cases X has some structure and, n f so, to dsplay t n the form of a partton. Ths problem belongs to the area of Non- Herarchcal cluster desgn, whch has many applcatons n economcs, socal and natural scences. It s known to be NP-Hard [5]. l l

2 Varous exact methods for MSSC can be found n the lterature (see, for example [17] and [7]), some of them, such as the method proposed by du Merle et al. [8], have succeeded n resolvng problems wth up to 150 ponts. For larger-szed problems the use of heurstc algorthms s stll necessary. The most popular are those based on Local Search methods, such as the well-known K-Means [15] and H-Means [14] procedures. In a recent work, Hansen and Mladenovc [13] propose a new Local Search procedure, J-Means, along wth varants H-Means+ or HK-Means. In recent years algorthms usng Metaheurstc strateges have been desgned, such as Smulated Annealng [16], Tabu Search [], Genetc Algorthms [3] or most recently Varable Neghborhood Search or VNS [8], [13] and Memetc Algorthms [19]. An algorthm that s able to obtan good solutons n short tmes s proposed for ths problem. Ths method s based n the a recent Metaheurstc strategy named Scatter Search (SS). Ths method also ncorporates others procedures based n others methods, such as Local Search, Tabu Search, GRASP and Path Relnkng. Ths Scatter Search approach s analyzed and compared wth other recents technques. In all cases, our proposed technque gves adequate solutons, compared wth others recent technques, n reasonable tme, especally wth small values of m.. Soluton Approach The soluton approach that we have developed for the MSSC problem conssts of an adaptaton of scatter search. Scatter Search s an nstance of the so-called evolutonary methods, whch s not based solely on randomzaton as the man mechansm for searchng. SS has been successfully mplemented n a varety of settngs ncludng combnatoral optmzaton and nonlnear optmzaton n contnuous varables. Scatter search embodes prncples and strateges that are stll not emulated by other evolutonary methods, and that prove advantageous for solvng a varety of complex optmzaton problems. More about the orgn and multple applcatons of scatter search can be found n Glover [11], Glover, Laguna and Martí [1] and Laguna [18]. SS uses a set of solutons named Reference Set, (RefSet). RefSet s composed by the best solutons by qualty (subset RefSet1) and dversty (subset RefSet). In each teraton new solutons are generated from those of Reference Set, and then Reference Set s updated wht these new solutons. A statc verson s desgned for ths problem. An outlne of ths mplementaton s shown below. Procedure Statc_Scatter_Search Step 1. Generate an ntal set of solutons P by usng a Dversfcaton- Generaton Method Step. Improve these solutons by an Improvement Method Step 3. Wth these solutons buld an ntal RefSet Step 4. Repeat 4.1. Obtan all subsets of pars from RefSet 4.. Combne these subsets and obtan new solutons 4.3. Improve these solutons by the Improvement Method 4.4. Update RefSet wth these news solutons untl RefSet s stable (.e. no new solutons have been ncluded) {fnal 4.}

3 Step 5. If max_ter teratons (Steps 1-4) elapse wthout mprovement stop else return Step 1 Fgure 1. SS mplementaton Let s denote by n_pob the sze of P (Step 1). Also let s denote by b 1 and b the szes of RefSet1 and RefSet. For buldng the ntal RefSet (Step 3) frst the best solutons (by qualty) are taken from P. Then, the next solutons are added to RefSet by dversty. For that, the followng measure of dversty s used. Let λ be a soluton n P \ RefSet, we defne δ mn (λ) = mn {df(λ,λ ) / λ RefSet}; where df(λ,λ ) = number of assgnments n λ that are dfferent from λ. We then select the canddate soluton λ that maxmzes δ mn (λ). The updatng of the RefSet s based only on qualty. That s, only the new solutons that mprove the qualty of the worst soluton n RefSet are added, (Step 4.4). We now provde descrptons of the dversfcaton, mprovement and combnaton methods..1 Dversfcaton Method Our dversfcaton method s based on GRASP constructons. GRASP, or greedy randomzed adaptve search procedure, s a heurstc that constructs solutons wth controlled randomzaton and a greedy functon. Most GRASP mplementatons also nclude a local search that s used to mprove upon the solutons generated wth the randomzed greedy functon. GRASP was orgnally proposed n the context of a set coverng problem [9]. Detals of the methodology and a survey of applcatons can be found n Feo and Resende [10] and Ptsouls and Resende [0]. The method proposed n ths paper conssts of two stages. In the frst, a subset S wth ponts from X and suffcently far from each other (seed-ponts) s bult. Ths s denoted as S = {x s1, x s,, x sm }. We also use m one-pont clusters, correspondng to the seed ponts,.e., C = {x s }, =1,,m. In the second stage, the remanng ponts are assgned to dfferent clusters accordng to the dstance to ther centrods. The set S s determned as follows: - Determne x *, the farthest pont from the centrod of X, and do S = {x * }. - Whle S < m do: 1 Calculate = mn{ x - x l : x l S}, x S.. Calculate max = max { : x S } 3. Buld L = {x / α max } 4. Choose x * L randomly and do S = S {x * }. The α parameter (0 α 1) controls the level of randomzaton for the greedy selectons. Randomzaton decreases as the value of α ncreases. Ths controlled randomzaton results n a samplng procedure where the best soluton found s typcally 3

4 better than the one found by settng α = 1. A udcous selecton of the value of α provdes a balance between dversfcaton and soluton qualty. The frst tme that the dversfcaton method s employed (Step 1 n Fgure 1), there s no hstory assocated wth the number of tmes each pont has been selected as seed pont. However, ths nformaton s valuable when the method s appled to rebuld the reference. The nformaton s stored n the followng array: freq() = number of tmes that pont x has been selected to belong to S n the executon of Step 0 and prevous executons of Step 8 of Fgure 1. The nformaton accumulated n freq() s used to modfy the values n the applcaton of the frst phase of dversfcaton method. The modfed evaluaton s: = β max freq freq () max where freq max = max { freq() : }. The modfed values are used to calculate max and execute the dversfcaton method. Large values of β encourage the selecton of seed ponts that have not been frequently made. The use of frequency nformaton wthn a dversfcaton method s nspred by Campos, et al. [6]. In phase of our dversfcaton method, the ponts of X \ S are assgned as follows: 1. Set A = X \ S, (A = Set of unassgned ponts). For each pont x A and for each cluster C, = 1,, m, calculate the ncrease n obectve functon that results from assgnng pont x to cluster C. Ths value Γ s calculated as follows: n Γ = x x n + 1 where x s the centrod of C and n = C. 3. Calculate Γ ** = mn { Γ : x A, = 1,, m } 4. Assgn x * to C * and set A = A { x * } 5. If A return to, else stop..3 Improvement Method As an mprovement method the H-Means+ algorthm [13] s used together wth a smple Tabu Search procedure. The H-means+ algorthm s a varant of the well-known H- Means algorthm [14]. The Tabu Search procedure has been proposed by Pacheco and Valenca [19] and s descrbed shortly below. Ths Tabu Search procedure uses the neghborng moves employed n K-Means. These moves consst at each step n the movement of an entty from a cluster to a dfferent one. In order to avod repettve cyclng when a move whch conssts n movng pont x from cluster C l to cluster C s performed, pont x s 4

5 prevented from returnng to the cluster C l for a certan number of teratons. Specfcally, defne Matrx_tabu (l, ) = the number of the teraton n whch pont x leaves cluster C l. The Tabu Search method s descrbed below, where P denotes a ntal soluton wth a value f. The parameter τ ndcates the number of teratons durng whch a pont s not allowed to return to the leavng cluster. The parameter κ ndcates the maxmum number of unmproved teratons. In our case we use the stop crteron κ = 100. After dfferent tests, τ was set as m. Tabu Search 1 Do Matrx_tabu(,) = τ, =1,...,m, = 1,,N Do δ = 0 and P* = P, f* = f and η =0; 3 Repeat (3.1) δ = δ + 1 (3.) Determne v ** = mn {v / =1,...,m; = 1,...,N ; x C verfyng nter > Matrx_tabu (,) + τ or f + v < f* ( aspraton crteron )} (3.3) Reassgn x * to C * ; (3.4) Do Matrx_tabu (l*,*) = δ (l* beng the prevous cluster of x * ); (3.5) If f (the value of the current soluton P) < f* then do: P* = P, f* = f and η = δ ; untl (δ η > κ) or another termnaton crteron Where v s the change n the value of the obectve functon when x s reassgned to C. The followng formula s obtaned from Späth [] to smplfy the calculatons n K- Means. Let C l be the cluster to whch x belongs, then the value of v s calculated as follows: n nl v = x x x l x. n + 1 n 1.4 Combnaton Method New solutons are generated from combnng pars of reference set solutons. The number of solutons generated from each combnaton depends on the relatve qualty of the solutons beng combned. Let λ and λ be the reference solutons beng combned, where <. Assume, as before, that the reference set s ordered n a way that λ 1 s the best soluton and λ b s the worst. Then, the number of solutons generated from each combnaton s: 3 f b 1 and b 1, f b 1 and > b 1 and 1 f > b 1 and > b 1. Each subset wth two elements of RefSet s used to generate new solutons. To do so, we use a strategy called Path Relnkng. The basc dea s as follows: a path to on the two ntal solutons s bult. A number of ntermedate ponts (solutons) from the path' or chan are selected as new solutons. The am s for the ntermedate ponts or solutons to be as equdstant as possble from each other. The mprovement method descrbed n Sect.. s then appled to these ntermedate solutons. Fgure depcts ths dea. l 5

6 λ λ Improvement λ* λ** Fgure.- Generaton of New Solutons by usng Path Relnkng From every par of solutons of the reference set, n fgure λ and λ, a path to on them s bult. Solutons n these ntermedate preselected postons wthn the path are selected and mproved. In ths way new solutons are generated, (n fgure λ* y λ**). In order to buld the path that ons λ and λ, ponts wth dfferent clusters are selected n the dfferent steps and the shft s carred out. At each step the best possble shft s selected. In ths way, the ntermedate solutons n each step have another element n common wth λ. Path Relnkng s a strategy tradtonally assocated wth the ntensfcaton phase of the Tabu Search. The underlyng dea s that n the path between two good solutons there should be solutons of smlar qualty (n some cases, even better solutons). See Glover, Laguna and Martí [1] for more detals. 3. Parameter Fne Tunng One of the most tme consumng tasks n the development of metaheurstc procedures for optmzaton s the tunng of search parameters. Adenso-Daz and Laguna [1] propose an automated parameter tunng system called CALIBRA that employs statstcal analyss technques and a local search procedure to create a systematc way of fne-tunng algorthms. The goal of CALIBRA s to provde an automated system to fne-tune algorthms, where a user needs only to specfy a tranng set of nstances and a measure of performance. CALIBRA works wth a set of problem nstances and a range of values for each search parameter. The procedure utlzes desgn of experments and heurstcs to search for the best parameter values. The qualty of a set of values s tested on the specfed set of problem nstances. CALIBRA s avalable at opalo.etsg.unov.es/~adenso/fle_d.html, where a user s manual can also be found. Our complete set of test problems conssts of 15 nstances, as we pont out n Secton 4. We selected a relatvely small tranng set of 3 problems, because we determned that ths sample was representatve and that the parameter values found wth CALIBRA would also perform well when appled to the entre test set. The parameters to be adusted were α and β n the (0.1, 0.9) range and b 1 and b n the (3, 7) range. The values for MaxIter and PSze were set to and 0, respectvely. CALIBRA obtaned 6

7 the followng values, after runnng for approxmately 5 hours on a Pentum III machne at 600 MHz: α = 0.8, β = 0.5, b 1 = 5 and b = 5. Although a computatonal tme of 5 hours may seems excessve, t actually represents a very reasonable effort when compared to manual parameter fne-tunng. We use these parameter values for all of the experments reported n Secton Computatonal Experments To compare the effcences of our Scatter Search algorthm and those of other recent strateges, a seres of tests s performed. Next the results of ths set of computatonal experments usng these algorthms are shown. The followng algorthms are tested: Memetc-HK: Memetc Algorthm proposed n Pacheco and Valenca [19], HybMem: Hybrd Algorthm proposed n Pacheco and Valenca [19], VNS-HK: the VNS Algorthm, proposed n Hansen and Mladenovc [13], wth HK-Means as the Local Search method VNS-J: the VNS Algorthm, wth J-Means as the Local Search method. SS: Scatter Search Algorthm proposed n Secton. For ths work we have used our own mplementaton of every algorthm. The HybMem algorthm uses an ntal populaton generated by the greedy-random method descrbed n Beltrán and Pacheco (001). The VNS-HK and VNS-J algorthms use the best of these solutons as the ntal soluton. The sets of problems nstances used n testng are: () the 575 () 1060 and () 3038 ponts of the plane taken from TSPLIB [1] data base. All the tests n the current work are performed on a personal computer wth a Pentum III 600 MHz processor. All the algorthms have been mplemented n Pascal usng Borland Delph 5.0. In () the runnng tme s lmted to 300 seconds for all the algorthms. We set m = 5, 10, 15, 0, 30, 40, and 100. In tables 1 the solutons obtaned for each algorthm are presented. Table 1. Results for 575 TSPLIB m Memetc HybMem VNS-HK VNS-J SS , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,4 7

8 In (), we set m = 10, 0, 30,..., 150. These test data were prevously used n Hansen and Mladenovc [13], where the best soluton known for every value of m, (except m = 40) s reported. These were obtaned on a SUN Ultra I System workstaton wth 10 mnutes computaton tme. The runnng tme s lmted to 600 seconds for all the algorthms. In table the solutons obtaned for each algorthm and the percent devaton wth respect to the best-known soluton (dvt) are presented. Table. Results and percent desvaton respect to the prevous best know soluton. m Memetc HybMem VNS-HK VNS-J SS dvt : 0 0 0,185 0, ,181 0,05 1,757 1, ,899 1,48 0,79,54 0, ,89,503,19,43 0, ,47 1,773 0,71 1, ,09 3,398 1,173 4,91 0, ,318 3,77 0,353 0, , , , , ,7 3,516 3,34 0,304,644 0, , , , ,3 4,33,711 0,13 4,18 0, , , , ,7 4,03 4,03 0,77 3,08 1, , , ,6 4,383 4,383 1,513 3,943 1, , , , , ,4 5,5 5,086 1,68,531 0, , , , , ,8 5,18 4,35 0,366 3,643 0,511 In (), we set m = 10, 0,, 50, 100, 150,..., 400. These test data were prevously used n Hansen and Mladenovc [13], where the best soluton known for every value of m, s reported. These were obtaned on a SUN Ultra I System workstaton wth 50 mnutes computaton tme. The runnng tme s lmted to 600 seconds for all the algorthms. In table 9 the solutons obtaned for each algorthm and the percent devaton wth respect to the best-known soluton (dvt) are presented. Table 3. Results and percent desvaton respect to the prevous best know soluton. m Memetc HybMem VNS-HK VNS-J SS

9 m Memetc HybMem VNS-HK VNS-J SS , , ,109 0,94 0,709 0, ,17 0 0,346 0,46-0, , , ,6-0,001-0,083-0,068-0,01-0, , , , ,7,604,109 0,937,564 0, , , , , ,4,665,665 0,419,539 0, , , , , ,1 3,54 3,54 0,397 3,3 1, , , , , ,3 3,769 3,769 1,151 3,446, , , , , ,4 3,49 3,49 0,808,946, , , , , ,3 4,6 4,144 1,384 3,955, , , , , ,94 3,988 3,988 1,766 3,757 3, , , , , ,37 3, 3, 1,567 3,18 1, , , , , ,5,39,39 1,05 1,97 1,44 Tables 1-3 yelds the follow observatons: - The HybMem and Memetc-HK algorthms yeld good solutons when the number of clusters, m, s low. However, the qualty of ther solutons deterorates as the value of m ncreases. - The VNS-J, and specally the VNS-HK algorthms become more compettve than the other algorthms as the value of m ncrease. For hgh values of m (m 50 n (), m 90 n (), m 00 n ()), the VNS-HK algorthm yelds the best soluton out of all the strateges proposed n 14 cases. - In 6 out of 41 cases, the SS algorthm yelds the best soluton. Furthermore, t s especally effcent wth lower m values (m < 50 n (), m < 90 n (), m < 00 n ()). In such cases, t always yelds the best soluton (except m = 150 n ()). Besdes n () SS equals the earler best known soluton for m =10, 0, 30, and 70 (for m=40 no soluton was known). For m = 50, 60, and 80 the devatons from the earler best known soluton are very small. In () SS equals the earler best known soluton for m=10, 0, 30 and mproves that soluton for m = 40 and For hgher values of m, SS stll proves to be very compettve,.e., n sx out of twenty cases t yelds the best soluton (m = 130 and 140). In the other ffteen nstances, t holds second place, after the VNS-HK algorthm. Devatons n relaton to the earler best solutons are stll low. - Globally speakng, the VNS-HK and the SS algorthms are the best strateges n these tests. SS seems to be better, as t yelds the best soluton n most cases (6 compared to ). 9

10 - Another nterestng aspect s the robustness that SS shows n comparson to other strateges. In () no case s the best known soluton yelded by SS further than 1.350%. VNS-HK goes up to.19%. The other strateges show greater dstances: up to 5.5% for Memetc-HK, 5.086% for HybMem, and 4.91% for VNS-J. Fgure 3 shows the evoluton of the soluton found for the dfferent algorthms, accordng to the computatonal tme for N = 1060 and m = Computatonal Tme n Seconds Memetc HybMem VNS-HK VNS-J SS Fgure 3.- Evoluton of the soluton value found for the dfferent strateges accordng to the computatonal tme As shown n fgure 3, VNS-HK, and especally SS, have a very greedy character. After a few seconds, they yeld solutons that are better than those produced by other strateges after usng up the maxmum computatonal tme (600 seconds). 5.- Conclusons The contrbuton of our work s the development of a specalzed and sophstcated scatter search procedure for the soluton of the cluster desgn problem. Ths contrbuton s mportant, because our dversfcaton, mprovement and combnaton methods ntroduce novel features that can be adapted to other stuatons. Our method brngs together several search strateges and mechansms, such as GRASP constructons, local search, tabu search and Path Relnkng wthn the framework of scatter search. Usng nstances from the lterature, we were able to show the mert of our SS desgn. In partcular, our experments show that our method obtans the best overall solutons aganst others recent methods. Our method s specally effectve for small values of m. Wth hgher values of m t s only surpassed by the VNS algorthm proposed by Hansen and Mladenovc (001), wth HK-Means as the Local Search Procedure. However, the followng s worth notng: n ther work, VNS s used wth J-Means and J-Means+ as a local search procedure, whereas n our work the use of J-Means n VNS leads to worse solutons than VNS wth HK-Means and also to worse solutons than the methods we propose. 10

11 Fnally our scatter search approach shows two mportant features: t s a greedy method (t found good solutons n very short calculaton tme) and s able to evolve (mprove these solutons wth more calculaton tme). References [1] Adenso-Díaz B and. Laguna M. Automated Fne Tunng of Algorthms wth Taguch Fractonal Expermental Desgns and Local Search. Unversty of Colorado at Boulder; 001. [] Al-Sultan KH. A Tabu Search Approach to the Clusterng Problem. Pattern Recognton 1995;8: [3] Babu GP and Murty MN. A Near-Optmal Intal Seed Value Selecton n K-means Algorthm usng Genetc Algorthms. Pattern Recognton Letters 1993;14: [4] Beltrán M and Pacheco J. Nuevos métodos para el dseño de cluster no erárqucos. Una aplcacón a los muncpos de Castlla y León. Estadístca Española. Insttuto Naconal de Estadístca 001;43, 148:09-4. (avalable n [5] Brucker P. On the Complexty of Clusterng Problems. Lecture Notes n Economcs and Mathematcal Systems 1978;157: [6] Campos V, Glover F, Laguna M and Martí R. An Expermental Evaluaton of a Scatter Search for the Lnear Orderng Problem. Journal of Global Optmzaton 001; 1: [7] Dehr G. Evaluaton of a Branch and Bound Algorthm for Clusterng. SIAM J.Sc.Statst.Comput. 1985; 6: [8] du Merle, O., Hansen, P., Jaumard, B. and Mladenovc, N. An Interor Pont Algorthm for Mnmum Sum of Squares Clusterng. SIAM Journal on Scentfc Computng 000; 1(4): [9] Feo, T.A. and Resende, M.G.C. A Probablstc heurstc for a computatonally dffcult Set Coverng Problem. Operatons Research Letters 1989; 8: [10] Feo, T.A. and Resende, M.G.C. Greedy Randomzed Adaptve Search Procedures. Journal of Global Optmzaton 1995;:1-7. [11] Glover, F. A Template for Scatter Search and Path Relnkng. n Artfcal Evoluton, Lecture Notes n Computer Scence, 1363, J.-K. Hao, E. Lutton, E. Ronald, M. Schoenauer and D. Snyers (Eds.) Sprnger 1998, pp [1] Glover, F., Laguna, M. and Martí, R. Fundamentals of Scatter Search and Path Relnkng. Control and Cybernetcs 000; 39,3: [13] Hansen, P. and Mladenovc, N. J-Means: A new Local Search Heurstc for Mnmum Sum-of- Squares Clusterng. Pattern Recognton 001; 34(): [14] Howard, R. Classfyng a Populaton nto Homogeneous Groups. In Lawrence, J.R. (eds.), Operatonal Research n the Socal Scences. Tavstock Publ., London [15] Jancey, R.C. Multdmensonal Group Analyss. Australan J. Botany 1966;14: [16] Klen, R.W. and Dubes, R.C. Experments n Proecton and Clusterng by Smulated Annealng. Pattern Recognton 1989; : [17] Koontz, W.L.G., Narendra, P.M. and Fukunuga, K. A Branch and Bound Clusterng Algorthm. IEEE Transactons on Computers 1975; C-4: [18] Laguna, M. Scatter Search, n Handbook of Appled Optmzaton, P. M. Pardalos and M. G. C. Resende (Eds.), Oxford Unversty Press, New York 00, pp [19] Pacheco, J. and Valenca, O. Desgn of Hbrds for the Mnmun Sum-of-Squares Clusterng Problem. Computatonal Statstcs and Data Analyss 003; 43,: [0] Ptsouls, L.S. and Resende, M.G.C. Greedy Randomzed Adaptve Search Procedures n Handbook of Appled Optmzaton, P. M. Pardalos and M. G. C. Resende (Eds.), Oxford Unversty Press 00, pp [1] Renelt, G. TSPLIB: A Travellng Salesman Problem Lbrary. ORSA Journal on Computng 1991; 3: [] Späth, H. Cluster Analyss Algorthms for Data Reducton and Classfcaton of Obects. Ells Horwood, Chchester