Performance Comparisons of PSO based Clustering

Transcription

1 Performace Comparisos of PSO based Clusterig Suresh Chadra Satapathy, 2 Guaidhi Pradha, 3 Sabyasachi Pattai, 4 JVR Murthy, 5 PVGD Prasad Reddy Ail Neeruoda Istitute of Techology ad Scieces, Sagivalas,Vishaapatam Dist 2 Bhubaaada Orissa School of Egieerig, Cuttac 3 FM Uiversity, Balasore 4 JNTU College of Egieerig, Kaida, 5 College of Egieerig, AU, Vishaapatam, Abstract I this paper we have ivestigated the performace of PSO Particle Swarm Optimizatio based clusterig o few real world data sets ad oe artificial data set. The performaces are measured by two metric amely quatizatio error ad iter-cluster distace. The K meas clusterig algorithm is first implemeted for all data sets, the results of which form the basis of compariso of PSO based approaches. We have explored differet variats of PSO such as gbest, lbest rig, lbest voeuma ad for compariso purposes. The results reveal that PSO based clusterig algorithms perform better compared to K meas i all data sets. Keywords - K-Meas, Particle Swarm Optimizatio, Fuctio Optimizatio, Data Clusterig. INTRODUCTION Data clusterig is the process of groupig together similar multi-dimesioal data vectors ito a umber of clusters or bis. Clusterig algorithms have bee applied to a wide rage of problems, icludig exploratory data aalysis, data miig [], image segmetatio [2] ad mathematical programmig [3,4] Clusterig techiques have bee used successfully to address the scalability problem of machie learig ad data miig algorithms. Clusterig algorithms ca be grouped ito two mai classes of algorithms, amely supervised ad usupervised. With supervised clusterig, the learig algorithm has a exteral teacher that idicates the target class to which a data vector should belog. For usupervised clusterig, a teacher does ot exist, ad data vectors are grouped based o distace from oe aother. This paper focuses o usupervised clusterig. May usupervised clusterig algorithms have bee developed oe such algorithm is K-meas which is simple, straightforward ad is based o the firm foudatio of aalysis of variaces. The mai drawbac of the K-meas algorithm is that the result is sesitive to the selectio of the iitial cluster cetroids ad may coverge to the local optima. This is solved by PSO as it performs globalized search for solutios. So this paper explores the applicability of PSO ad its variats to cluster data vectors. I the process of doig so, the objective of the paper is: to show that the stadard PSO algorithm ca be used to cluster arbitrary data, ad to compare the performace of PSO ad its variats with stadard K-meas algorithm. The rest of the paper is orgaized as follows: Sectio 2 presets a overview of K-meas algorithm. The basic PSO ad its variats are discussed i sectio 3. Fuctio optimizatio usig PSO models are give i sectio 4. How Clusterig is doe with PSO is discussed i sectio 5. Experimetal results are summarized i sectio 6 ad Coclusio ad further wor is emphasized i sectio K-Meas Clusterig Oe of the most importat compoets of a clusterig algorithm is the measure of similarity used to determie how close two patters are to oe aother. K- meas clusterig group data vectors ito a predefied umber of clusters, based o Euclidea distace as similarity measure. Data vectors withi a cluster have small Euclidea distaces from oe aother, ad are associated with oe cetroid vector, which represets the "midpoit" of that cluster. The cetroid vector is the mea of the data vectors that belog to the correspodig cluster. For the purpose of this paper, followig symbols are defied: N d deotes the iput dimesio, i.e. the umber of parameters of each data vector No deotes the umber of data vectors to be clustered Nc deotes the umber of cluster cetroids (as provided by the user), i.e. the umber of clusters to be formed

2 IterJRI Computer Sciece ad Networig, Vol., Issue, July z p deotes the p th data vector m j deotes the cetroid vector of cluster j j is the umber of data vectors i cluster j C j, is the subset of data vectors that form cluster j. Usig the above otatio, the stadard K-meas algorithm is summarized as. Radomly iitialize the N c cluster cetroid vectors 2. Repeat (a) For each data vector, assig the vector to the class with the closest cetroid vector, where the distace to the cetroid is determied usig N 2 d z m z m d p, j p j ---- () Where subscripts the dimesio (b) Recalculate the cluster cetroid vectors, usig m j z p j z p C j util a stoppig criterio is satisfied, (2) The K-meas clusterig process ca be stopped whe ay oe of the followig criteria are satisfied: whe the maximum umber of iteratios has bee exceeded, whe there is little chage i the cetroid vectors over a umber of iteratios or whe there are o cluster membership chages. For the purposes of this study, the algorithm is stopped whe a user-specified umber of iteratios have bee exceeded. the particle positio that results i the best evaluatio of a give fitess (objective) fuctio. Each particle represets a positio i N d dimesioal space, ad is: flow through this multidimesioal search space, adjustig its positio toward both the particle's best positio foud thus far. ad the best positio i the eighborhood of that particle. Each particle i maitais the followig iformatio: x i : The curret positio of the particle; v i : The curret velocity. of the particle; y i : The persoal best positio of the paicle. Usig the above otatio. a particle's positio is adjusted accordig to ^ vi, t wvi, t cr, t y i, t xi, t c2r2, t yi t xi t, (3) r t x t v t xi i i (4) t r t ~ 0,, j, 2, j U ad =,.., N d c,c Where w is the iertia weight, 2 are the acceleratio costats ad r is the radom umber geerated for avoidig ad biasig effect to social ad cogitive compoets. The velocity is thus calculated based o three cotributios: () a fractio of the previous velocity, (2) the cogitive compoet which is a fuctio of the distace of the particle from its persoal best positio, ad (3) the social compoet which is a fuctio of the distace of the particle from the best particle foud thus far (i.e. the best of the persoal bests) The persoal best positio of particle i is calculated as 3. Particle Swarm Optimizatio ad its variats Particle swarm optimizatio (PSO) is a populatio-based stochastic search process, modeled after the social behavior of a bird floc [5,6]. The algorithm maitais a populatio of particles, where each particle represets a potetial solutio to a optimizatio problem. I the cotext of PSO, a swarm refers to a umber of potetial solutios to the optimizatio problem, where each potetial solutio is referred to as a particle. The aim of the PSO is to fid Two basic approaches to PSO exist based o the iterpretatio of the eighborhood of particles. Equatio (3) reflects the gbest versio of PSO where, for each particle, the eighborhood is simply the etire swarm. The social compoet the causes particles to be draw towards the best particle i the swam. I the lbest PSO model, the swam is divided ito overlappig eighborhoods, ad the best particle of each eighborhood is determied. For the lbest PSO model, the social compoet of equatio (3) chages to.

3 20 Performace Comparisos of PSO ^ Where y j is the best particle i the eighborhood of the i th particle. The PSO is usually executed with repeated applicatio of equatios (3) ad (4) util a specified umber of iteratios have bee exceeded. Alteratively, the algorithm ca be termiated whe the velocity updates are close to zero over a umber of iteratios. lbest-rig is oe of the variat of PSO i which the pbest is determied with respect to the eighborig adjacet particles as show i figure. 4 PSO a tool for Fuctio Optimizatio PSO ca be applied to umber of real world problems lie optimizatio which has bee expadig i all directios at a astoishig rate. So the optimizatio of several complex fuctios is doe with PSO. We have applied the differet variatios of PSO amely lbest (rig ad vo-neuma architectures) [7,8] ad gbest for optimizig some stadard Bechmar fuctios give i the Table I [7], with rage of search ad maximum velocities i Table II, ad correspodig results are give i table III. fuctio Sphere fuctio RoseB roc fuctio Rastrig ri fuctio Table I: Bechmars for simulatios Mathematical represetatio 2 f ( x) x i i ( ) f x [00( xi xi ) ( xi ) ] i 2 f3( x) 0 [ x i 0cos(2 xi )] i Table II: Rage of search ad Maximum Velocity Figure - Rig architecture I Vo-Neuma architecture the particles are cosidered to be i two dimesioal matrix. pbest of the particle is determied with respect to four eighborig adjacet particles as show i figure 2. Fuctio Rage of search Maximum Velocity (-00,00) 00 (-00,00) 00 (-0,0) 0 X X 2 * * X i- X i X 2 X 22 * * X 2i- X 2i X 3 X 32 * * X 3i- X 3i * * * * * * * * * * * * X j X j2 * * X ji- X ji Figure 2 - Vo-Neuma architecture

4 IterJRI Computer Sciece ad Networig, Vol., Issue, July Fuc tio f f2 f3 Type of solvig gbest lbest-rig lbest- VoNeuma gbest lbest-rig lbest- VoNeuma gbest lbest-rig lbest- VoNeuma Table-III: Results Dimesi o Iteratio s Best fitess Mea Stadard deviatio e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e+00 From the above results it ca be see that the PSO is a very good cadidate for solvig optimizatio problems. So the data clusterig problem is a sort of optimizatio problem where i the objective is to fid a similar data objects ito a specific group. I our wor the PSO is used for ivestigatig this objective. 5. PSO Clusterig I the cotext of clusterig, a sigle particle represets the N c cluster cetroid vectors. That is, each particle xi is costructed as follows: where refers to the j-th cluster cetroid vector of the i-th particle i cluster C ij. Therefore, a swarm represets a umber of cadidate clusters for the curret data vectors. The fitess of particles is easily measured as the quatizatio error, Where d is defied i equatio (), ad is the umber of data vectors belogig to cluster i.e. the frequecy of that cluster. This sectio presets a stadard PSO for clusterig data ito a give umber of clusters. 5. PSO Cluster Algorithm Usig the stadard gbest PSO, data vectors ca be clustered as follows:. Iitialize each particle to cotai N c, radomly selected cluster cetroids. 2. For do (a) For each particle i do

5 22 Performace Comparisos of PSO (b) For each data vector z p d z p, m i, j to i) Calculate the Euclidea distace all cluster cetroids ii) Assig z p to cluster = such that iii) Calculate the fitess usig equatio (8) (c) Update the global best ad local best positios (d) Update the cluster cetroids usig equatios (3) ad (4) Where t,,, is the maximum umber of iteratios. The populatio-based search of the PSO algorithm reduces the effect that iitial coditios have, as opposed to the K-meas algorithm; the search starts from multiple positios i parallel. Sectio 6 shows that the PSO algorithm performs better tha the K-meas algorithm i terms of quatizatio error. X X 2 X X i X m X 2 X 22 X X 2i X 2m X 3 X 32 X X 3i X 3m X X 2 X X i X m 6. Data Set ad Experimetal Results This sectio compares the results of the K-meas ad PSO algorithms o five clusterig problems. The mai purpose is to compare the quality of the respective clusterig, where quality is measured accordig to the followig two criteria: the quatizatio error as defied i equatio (8); the iter-cluster distaces, i.e. the distace betwee the cetroids of the clusters, where the objective is to maximize the distace betwee clusters. coefficiets c ad c2 are fixed at.042 to esure good covergece [0]. The clusterig problems used for the purpose of this paper are: Iris plats database: This is a well-uderstood database with 4 iputs, 3 classes ad 50 data vectors. Wie: This is a classificatio problem with "well behaved class structures. There are 3 iputs, 3 classes ad 78 data vectors. Hayes Roth which has 32 data vectors with 3 classes ad 5 iputs. Diabetes data set has 768 data vectors havig 2 classes ad 8 iputs. Artificial: This problem follows the followig classificatio rule; class if ( z ad( z2 class 2 0 Otherwise 0.7) or(( z 0.2 z 0.3) A total of 400 data vectors are radomly created betwee (-,). Table IV summarizes the results obtaied from the five clusterig algorithms for the problems cited above. The values reported are averages over 30 simulatios, with stadard deviatios to idicate the rage of values to which the algorithms coverge. First, cosider the fitess of solutios, i.e. the quatizatio error, for all data sets PSO based clusterig is better tha K-meas. However, lbest_vonuuma provides better fitess values i terms of quatizatio error ad iter_cluster distace for all data sets except for Wie. For Wie ad Hayes Roth, gives good result. The lbest_voeuuma gives worst quatizatio error but comparatively good iter_cluster distace measure for these data sets. The stadard deviatios (std) foud to be very close, thereby idicatig the covergece of algorithms to better results. )) For all the results reported, averages over 30 simulatios are give. All algorithms are ru for 000 fuctio evaluatios, ad the PSO algorithms used 0 particles. The taes the seed from result of K-meas clusterig. This seed is cosidered as oe particle i swarm of particles i PSO. For PSO, w is varyig as per the paper [9]. The iitial weight is fixed at 0.9 ad the fial weight at 0.4. The acceleratio

6 IterJRI Computer Sciece ad Networig, Vol., Issue, July CONCLUSION This paper ivestigates the applicatio of the PSO to cluster data vectors. Five algorithms were tested, amely a stadard K-meas, gbest PSO, lbest_rig, lbest_voeumma ad. The PSO approaches are compared agaist K-meas clusterig, which showed that the PSO approaches have better covergece to lower quatizatio errors, ad i geeral, larger iter-cluster distaces. Future studies will ivolve more elaborate tests o higher dimesioal problems-ad large umber of patters. The PSO clusterig algorithms will also be exteded to dyamically determie the optimal umber of clusters. REFERENCES Table IV: Results of clusterig Data Sets Algorithm Quatizatio error,std Iris K meas lbest_rig lbest_voeuma Hayes Roth Wie Diabetes Artificial K meas lbest_rig lbest_voeuma K meas lbest_rig lbest_voeuma K meas lbest_rig lbest_voeuma K meas lbest_rig lbest_voeuma [] IE Evagelou. DG Hadjimitsis, AA Lazaidou, CClayto, Data Miig ad Kowledge Discovery i Complex Image Data usig Artificial Neural Networs, Worshop o Complex Reasoig a Geographical Data,Cyprus, 200. [2] T Lillesad, R Keifer, Remote Sesig ad Image Iterpretatio,Joh Wiley & Sos [3] HC Adrews. Itroductio to Mathematical Techiques i Patter Recogitio, Joh Wiley & Sos, New Yor , , , , , , , , , , , , , , , , , , , , , , , , , Iter cluster distace,std , , , , , , , , , , , , , , , , , , , , , , , , , [4] MR Rao, Cluster Aalysis ad Mathematical Programmig,Joural of the America Statistical Associatio,Vol. 22, pp , 97 I. [5] J Keedy, RC Eberhart, Particle Swarm Optimizatio, Proceedigs of the IEEE Iteratioal Joit Coferece o Neural Networs, Vol. 4, pp , 995. [6] JKeedy, RC Eberhart, Y Shi, Swarm Itelligece, Morga Kaufma, [7] T. PhaiKumar et. al Fuctio Optimizatio Usig Particle Swarm Optimizatio at ICSCI-07,Hyderabad. [8] B.Naga VSSV Prasada Rao et. al Swarm Itelliget Ucostraied Fuctio Optimizer at Techozio 2007 NIT, NIT Waragal [9] Emperical study of Particle Swarm Optimizatio, Proc.IEEE, Iteratioal Cogress, Evolutioary Computatio, vol.3 999,pp.0-06 [0] F va de Bergh, A Aalysis of Particle Swarm Optimizers, PhD Thesis, Departmet of Computer Sciece,Uiversity of Pretoria, Pretoria, South Africa, 2002.