Complexity Analysis of Problem-Dimension Using PSO

Transcription

1 Proceedngs of the 7th WSEAS Internatonal Conference on Evolutonary Computng, Cavtat, Croata, June -4, 6 (pp45-5) Complexty Analyss of Problem-Dmenson Usng PSO BUTHAINAH S. AL-KAZEMI AND SAMI J. HABIB, Department of Computer Engneerng Kuwat Unversty P.O. Box 5969 Safat, 36 Kuwat KUWAIT Abstract: -Ths work analyzes the nternal behavor of partcle swarm optmzaton (PSO) algorthm when the complexty of the problem ncreased. The mpact of number of dmensons for three well-known benchmark functons, DeJong, Rosenbrock and Rastrgn, were tested usng PSO. A Problem-Specfc Dstance Functon (PSDF) was defned to evaluate the ftness of ndvdual solutons and test the dversty n neghborng ndvduals. The PSDF started wth a large value, but converged to the optmum n few generatons, rrespectve of complexty of problem or objectve benchmark functon. The smulaton llustrates that all parameters n any dmenson behave n smlar pattern and we can expect smlar behavor for addtonal complexty n the problem. Key-Words: - Evolutonary Computaton, Genetc Algorthm, Partcle Swarm Optmzaton, Dstance Functon, Parameter Predcton.. Introducton The ablty of Partcle Swarm Optmzaton (PSO) algorthms, to fnd near optmal solutons to an optmzaton problem has been wdely demonstrated []. However, one of the major concerns n applyng these methods s the behavor of ths algorthm when the number of varables or the dmenson of the problem s ncreased. The numbers of unknowns have a sgnfcant mpact on the algorthm s behavor, and therefore greatly affect the qualty of the fnal soluton found, as well as the tme taken to fnd that soluton [] [3]. These results demonstrate that by understandng the behavor of the algorthm when usng few unknowns, and the relatonshps between them, the computaton requred can be reduced sgnfcantly [4]. The objectve of ths paper s to demonstrate the effectveness of the behavor analyss of PSO algorthm by varyng complexty of problem-doman. We observe that as the complexty of problem ncreases, there s not much change n the way an ndvdual soluton behaves. We tested and justfed the behavor by comparng the computatonal performance of the PSO usng a set of three mathematcal benchmark objectve functons namely, DeJong F, Rosenbrock and Rastrgn [5]. We tred to analyze the behavor of ndvdual elements and formulate a relaton between consecutve ncrements n the number of unknowns from to 6. Also we analyzed each dmenson and observed the behavor of ndvduals n a partcular dmenson by varyng the problem complexty. We have used the dstance functon to evaluate the ndvduals accordng to ther poston from the best ndvdual so far. The dstance functon (DF) s usually a measure of how far two solutons are from each other [6]. We have classfed the dstance functons nto two groups: an algorthmc dstance functon ADF and a problem-specfc dstance functon PSDF. In ths paper, we use the PSDF. Ths paper s organzed n 5 sectons. Secton descrbes the Partcle Swarm Optmzaton Algorthm and defnes the Dstance Functons. In Secton 3, the benchmark objectve functons are dscussed. Secton 4 and 5 are a detaled explanaton of our experment and t ncludes PSDF value analyss for three benchmark functons uder study. In Secton 6, we conclude our observatons.. Partcle Swarm Optmzaton and Dstance Functons. PSO

2 Proceedngs of the 7th WSEAS Internatonal Conference on Evolutonary Computng, Cavtat, Croata, June -4, 6 (pp45-5) Evolutonary Computatonal Algorthms (ECA) are ganng popularty especally when dffcult optmzaton problems are studed n depth [7]. It complements the study of tradtonal computatonal systems [8]. There are several types of ECAs n use today. The four tradtonal algorthms are: the Genetc Algorthm (GA), Evolutonary Programmng (EP), Evoluton Strateges (ES), and Genetc Programmng (GP). More recently, new ECAs have arsen, such as Partcle Swarm Optmzaton (PSO), Ant Colony (ACO), and the Shuffled Frog Leapng Algorthm (SFLA). The PSO s a populaton based stochastc searchng technque, whch was developed by Kennedy and Eberhart n 995. It s one of the most popular methods for optmzaton of contnuous nonlnear functons. One of the mportant merts of PSO Algorthms s ther ablty to tackle real-world problems, whch are very hard or even unsolvable by tradtonal optmzaton technques. Accordng to [9], PSO s a bologcal motvated optmzaton paradgm whch nvolves socal nteracton between ndvduals of a populaton.these ndvduals, also called partcles, can move through a multdmensonal search-space n order to fnd an optmum. Partcles can change ther movement drectons and veloctes accordng to the nformaton whch they themselves and the whole swarm have gathered so far from the ftness landscape. Each partcle remembers ts own best poston t has encountered by then whle all partcles know about the overall best poston whch has been found by all partcles of the swarm. From ths nformaton and each partcle's current velocty vector new velocty vectors are calculated and the postons of the partcles are updated. Dfferent parameter settngs can nfluence the behavor of the swarm, for example ts ablty to converge. PSO has been nspred by the socal behavor of flock of brds or school of fshes. It uses populaton of ndvduals, where they are tryng to reach the best soluton []. In each generaton, the swarm s flyng wth a certan velocty, whch s used to change ther postons. The changng of the poston not only depends on the velocty but also on the current poston of the partcle tself n that nstance and to the best ndvdual s poston so far. There s only one pece of food n the area beng searched. All the brds don t know where the food s; however, the brds know how far the food s n each generaton. Therefore, the effectve strategy to fnd the food s to follow the brd that s the nearest to the food. In PSO, each sngle soluton s a brd n the search space and t s called a partcle. All of partcles have ftness and velocty values. The ftness value s evaluated by the ftness functon to be optmzed; moreover, the velocty value drects the flyng of the partcle through the problem space by followng the current best partcle. A modfed verson of PSO wth nerta weghts was ntroduced by Sh and Eberhart [4], as depcted n Fgure. Partcle Swarm Optmzaton: begn t =; 3 ntalze partcles P (t); 4 evaluate partcles P (t); 5 whle (termnaton condtons are unsatsfed) 6 begn 7 t = t + ; 8 update weghts 9 select pbest for each partcle select gbest from P (t-); calculate partcle velocty P (t); update partcle poston P (t); 3 evaluate partcles P (t); 4 end 5 end Fg : The basc structure of partcle swarm optmzaton. PSO algorthm executes two Equatons - for updatng the velocty and poston of every partcle n the populaton. V[] = w * V[] + C rand() ( pbest[] present[] ) ( C rand() ( gbes [ ] present[] )) () present [] = present[] + V[] () In the equaton and equaton, V[] and present[] represent the velocty and poston of th partcle respectvely. The pbest[] represents the present best for the th partcle and gbest[] represents the global best among the populaton so far. The functon rand () generates random number unformly. w s the nerta weght C and C are constants assgned value most of the tme.. Dstance Functons The dstance functons (DF) are a set of measurements, whch should be collected at each generaton, to pont out the nternal behavor (smlarty and dfference) between the current optmal-soluton to the rest of the populaton [] []. Such nformaton can help us n +

3 Proceedngs of the 7th WSEAS Internatonal Conference on Evolutonary Computng, Cavtat, Croata, June -4, 6 (pp45-5) understandng of the convergence process wthn PSO, and gude us on how to better use such algorthms. Also usng the dstance functon we can evaluate and test the effect of ncrease n the complexty of problem and test the behavor of PSO algorthm [3]. We have classfed the dstance functons nto two groups: an algorthmc DF and a problem-specfc DF. ) Algorthmc Dstance Functons: The algorthmc dstance functons (ADF) are a set of measurements that are drectly assocated wth the evolutonary approaches, such as the convergence rate, executng tme, and memory allocaton. In ths paper, we are more nterested n the problem-specfc dstance functon. ) Problem-Specfc Dstance Functons: The problem-specfc dstance functons (PSDF) are a set of measurements that are drectly assocated wth the problems to be solved by an evolutonary approach. Therefore, a set of PSDF may not be used to evaluate another problem. The PSDF defned for our research s X-Dff (read as delta X t), whch ndcates the root of the sum of the mean square dfference between the varable X of the th ndvdual wthn the t generaton to ts best X varable wthn the same t generaton, as stated n Equaton 3.The term PS represents the populaton sze. (3) PS X = = t t ( X X ) best PS, =,,3 PS t Equaton 3 states the dfference per ndvdual; however, a good PSDF should gve an nsght behavor of the entre populaton. The data collected from Equatons 6 gve us frst-order nformaton on how PSO perform at each generaton, and how both algorthms are drftng toward the best/optmum [4]. 3. Benchmark Functons A set of benchmark test functons s selected wth the goal of representng ftness landscapes rangng from functons n whch a global optmum can be found easly usng constrcton for partcle velocty control, to the functons n whch t s dffcult to fnd the global optmum. We have used three benchmark optmzaton functons, whch are DeJong F, Rosenbrock and Rastrgn. Each functon has a number of nstances dependng on the number of unknowns or dmenson,. Here we have expermented wth fve nstances for rangng from to 6 per functon. The mplementatons are scalable,.e., the dmensons of the functons are adjustable va a sngle parameter used n the functon. The followng functons have been used for evaluaton. 3. De Jong s F functon (Crcle Functon) Ths functon s known as crcle functon when = and sphere functon n case of =3. It s contnuous, convex and unmodal. Mn F ( X ) X + X Subject (4) to = X Global mnmum: x = ; f = Rosenbrock s Valley (Banana functon) Rosenbrock s valley s a classc optmzaton problem and t s notorous for ts slow convergence to the optmum value. The global optmum s nsde a long, narrow, parabolc shaped valley. To fnd the valley s trval, however convergence to the global optmum s dffcult. Hence ths problem s often used n assessng the performance of the optmzaton algorthms. It s a unmodal functon and has sngle local maxmum or mnmum. Mn F ( X ) ( X X ) + ( X ) Subject (5) to = X Rastrgn Functon Ths functon s a typcal example of nonlnear multmodal functon that may have local as well as global optma. It s farly complex due ts complcated search space and large number of local mnma. Mn Subjet to (6) F ( X ) n = ( X = - 5. cos(πx ) + ) X Expermental Settng We have mplemented the basc structure of partcle swarm optmzaton algorthm (as depcted n Fgure

4 Proceedngs of the 7th WSEAS Internatonal Conference on Evolutonary Computng, Cavtat, Croata, June -4, 6 (pp45-5) ) usng C language. The DeJong F, Rosenbrock and Rastrgn functons are used as our examples (as shown n Equatons 4, 5, and 6) wth 5 ndvduals n a populaton and 3 generatons for all three functons. The range of each unknown s as shown earler n equatons 4, 5, and 6. The maxmum velocty was set always equal to the maxmum postve range for each problem. C and C were set to. The number of unknowns s ncreased from up to 6 unknowns for each problem. 5. Results 5. PSDF Analyss DeJong F Functon We start wth mnmum number of unknowns.e. unknowns for DeJong F functon whch s a crcle functon. Gradually we ncrement the dmenson to observe the behavor pattern. Fgures, 3, 4, and 5 dsplay the results obtaned for DeJong F functon for 3 generatons, from to 6 dmensons. As observed from the PSDF values n Fgure, the ndvduals' performance s smlar for the second unknown n all varatons obtaned by alterng the total number of unknowns. Notce that all ndvduals start wth large dversty but there s huge drop n dversty after say 75 generatons and graph s steady plateau. Consderng the second dmenson as seen from Fgure 3, we plot the values of the second unknown when there are, 4, and 6 unknowns respectvely. In case of unknowns, the PSDF value of the second unknown s.54 n the begnnng and after generatons t becomes.. When number of unknowns s 4, the value for the second unknown starts wth.38 and comes to lower than.5 wthn n generatons. The ndvduals behave n smlar manner when number of unknowns s 6. The PSDF value for the second unknown was ntally. then t became.8 n generatons. When we analyze the graphs for thrd and fourth dmensons n Fgures 4 and 5 respectvely, the behavor pattern for PSDF values shows smlar curves as t was observed for frst and second dmensons respectvely. Thus the PSDF values show that the dversty n ndvduals s hgh n the begnnng of searchng then t gradually reduces, whch ndcates less dversty. Fgure 6 shows the behavor of three unknowns from three dfferent dmensons, namely,, 5, and 6. From ths fgure, we notce that the three unknowns behave n the same way. They all start wth unstable and dverse acton, and after few generatons, all of the unknowns become more stable and less dverse. Mean X Mean X Mean X Mean X for to 6 Unknowns Mean X for Unknwns Mean X for 4 Unknwns Mean X for 6 Unknwns Fg Mean X for DeJong F Functon Mean X for to 6 Unknowns Mean X for Unknwns Mean Xfor 4 Unknwns Mean X for 6 Unknwns Fg 3 Mean X for DeJong F Functon. Mean X3 for to 6 Unknowns Mean X3 for 4 Unknowns Mean X3 for 6 Unknowns Fg 4 Mean X3 for DeJong F Functon.

5 Proceedngs of the 7th WSEAS Internatonal Conference on Evolutonary Computng, Cavtat, Croata, June -4, 6 (pp45-5) Mean X4 Mean X Mean X4 for to 6 Unknowns Mean X4 for 4 Unknowns Mean X4 for 6 Unknowns Fg 5 Mean X4 for DeJong F Functon. Mean X for 6 Unknowns for DeJong F Mean X for 6 Unknwn Mean X5 for 6 Unknwn Mean X6 for 6 Unknwn Fg 6 Mean X for DeJong F Functon n 6th dmenson. 5. PSDF Analyss Rosenbrock Functon In the Rosenbrock functon, we start wth smlar setup. We have generated the PSDF values for second dmenson and then ncreasng the complexty of problem, we move to further dmensons. Fgures 7, 8, 9, and plots the PSDF values for Rosenbrock functon where the numbers of unknowns range from to 6. From these fgures, we fnd that the behavor pattern of ndvdual solutons s n a partcular pattern. From Fgure 7, we observed that for frst dmenson, when number of unknowns s, the PSDF value was.9 and gradually the dversty between ndvdual solutons reduced and became.6 n generatons. Also the ntal values were respectvely.67 and.6 for 4 and 6 number of unknowns whch became.4 and.3 respectvely n generatons. Thus the PSO algorthm gans stablty n generatons for frst dmenson of Rosenbrock functon. As demonstrated n Fgure 8, the values are plotted for second dmenson for Rosenbrock functon, when number of unknowns s ncremented from to 6. In case of unknowns, the value started wth.54 and came down to. n just generatons. Smlarly, ntal values for 4 unknowns and 6 unknowns were.47 and. respectvely, whch decayed to.3 and. respectvely, n generatons. As observed from Fgures 9 and, n case of thrd and fourth dmensons, t s showng reducton n devaton between ndvdual values and stablty after generatons. For thrd dmenson the ntal values were.49 and.35 respectvely for 4 and 6 unknowns whch after generatons became. and. respectvely. However for the fourth dmenson, the PSO algorthm generates the values.8 for 4 unknowns and.87 for 6 unknowns n the begnnng. After generatons, fourth dmenson values become.9 and.37 for 4 and 6 number of unknowns, thus ganng stablty n fewer generatons. Thus, the devaton n ndvdual postons s more ntally, but more or less PSDF has same value for number of unknowns equal to 3, 4, 5 or 6, and ths devaton drops to approxmately whch s optmum value n less than just 5 generatons. Thus rrespectve of the number of unknowns for the gven problem the plot s same. In fgure, observaton of the behavor pattern of PSO algorthm when appled to Rosenbrock s shown. We plot the graph for 5 th and 6 th dmensons, and analyze t. From the analyss, t s observed that the ndvduals gradually converge wth reducton n dversty n both dmensons. Mean X Mean X for to 6 Unknowns Mean X for Unknwns Mean X for 4 Unknwns Mean X for 6 Unknwns Fg 7 Mean X for Rosenbrock Functon.

6 Proceedngs of the 7th WSEAS Internatonal Conference on Evolutonary Computng, Cavtat, Croata, June -4, 6 (pp45-5) Mean X Mean X for to 6 Unknowns Mean X for Unknwns Mean Xfor 4 Unknwns Mean X for 6 Unknwns Fg 8 Mean X for Rosenbrock Functon. Mean X Mean X for 6 Unknowns for Rosenbrock Mean X for 6 Unknwn Mean X5 for 6 Unknwn Mean X6 for 6 Unknwn Fg Mean X for Rosenbrock Functon n 6 th dmenson. Mean X3 Mean X Mean X3 for to 6 Unknowns Mean X3 for 4 Unknow ns Mean X3 for 6 Unknow ns Fg 9 Mean X3 for Rosenbrock Functon. Mean X4 for to 6 Unknowns Mean X4 for 4 Unknow ns Mean X4 for 6 Unknow ns Fg Mean X4 for Rosenbrock Functon. 5.3PSDF Analyss Rastrgn Functon We started wth frst dmenson and consequently analyzed second, thrd and fourth dmensons, n case of Rastrgn functon. Fgures, 3, 4 and 5 are plot of PSDF values for parameters rangng from to 6 unknowns for the benchmark functon of Rastrgn. As observed, though the ntal value s large for all the three cases, the optmum value s almost reached n less than generatons, rrespectve of varaton n number of unknowns. The PSO algorthm gans stablty soon n approxmately generatons as was the smlar case wth DeJong F functon and Rosenbrock functon. As observed from Fgure, wthn only generatons, the values converge, even though they started at values as hgh as.35,.47 and. for, 4, and 6 unknowns respectvely. The dversty n ndvdual vales reduces drastcally and come down to.6,.6 and.3 n generatons. Fgure 3 shows the second dmenson values for, 4, and 6 unknowns. The graph starts wth values.,.47, and.99 for, 4, and 6 unknowns respectvely. After generatons, the same values for, 4, and 6 generatons are.3,.3, and.3 respectvely. From Fgure 4, we get the values for thrd dmenson, whch are.3 and.87 respectvely for 4 and 6 unknowns ntally. The values decay to.6 and.36 for 4 and unknowns respectvely n generatons. As observed from Fgure 5, after generatons, the values for fourth dmenson for 4 unknowns s.9 whch was ntally.8, and for 6 unknowns t s.37 whch was ntally.86. Thus

7 Proceedngs of the 7th WSEAS Internatonal Conference on Evolutonary Computng, Cavtat, Croata, June -4, 6 (pp45-5) the graph s steady plateau and ndvduals are not devated at all after few generatons. Fgure 6 also justfes our observaton that when values are graphed for Rastrgn functon, wth as much complexty as n 6th dmenson, the values for frst, ffth and sxth dmensons are plotted and t s showng a constant decay and fnally forms a plateau n less than generatons where dversty s less. 3.5 Mean X Analyss for to 6 Unknowns Mean X Mean X4 Analyss for to 6 Unknowns Mean X4 for 4 Unknwns Mean X4 for 6 Unknwns Mean X.5 Mean X for Unknwns Mean X for 4 Unknwns Mean X for 6 Unknwns.4. Fg 5 Mean X4 for Rastrgn Functon. Mean X for 6 Unknowns for Rastrgn Fg Mean X for Rastrgn Functon. Mean X Mean X for 6 Unknwn Mean X5 for 6 Unknwn Mean X6 for 6 Unknwn Mean X Mean X Analyss for to 6 Unknowns Mean X for Unknwns Mean X for 4 Unknwns Mean X for 6 Unknwns Fg 6 Mean X for Rastrgn Functon n 6 th dmenson Fg 3 Mean X for Rastrgn Functon. Mean X3 Analyss for to 6 Unknowns Mean X3 for 4 Unknw ns Mean X3 for 6 Unknw ns Fg 4 Mean X3 for Rastrgn Functon 6. concluson Ths work analyzes nternal behavor of the PSO algorthm, wth respectve ncrement n the number of unknowns whch means the complexty of problemdoman, usng the values of Problem Specfc Dstance Functon (PSDF) for the three well-known mathematcal benchmark functons, such as DeJong F, Rastrgn and Rosenbrock. The crteron for analyss was varaton n the number of unknowns for these benchmark functons from unknowns to 6 unknowns. We observed that ndvduals behave n smlar manner rrespectve of the complexty of the problem. The mean dstance functon started wth a large value, but t converged to optmum n few generatons. That means, once we know the results for or more number of unknowns, we can predct the expected behavor for ncremental number of unknowns. We have justfed the behavor usng all the three objectve benchmark problems under study.

8 Proceedngs of the 7th WSEAS Internatonal Conference on Evolutonary Computng, Cavtat, Croata, June -4, 6 (pp45-5) The parameters V max and the range of varables of PSO are fxed durng the entre experment. Thus ncreasng the complexty of a problem, and wth the fxed settngs for PSO parameters, the algorthm behaves n smlar manner whch means that PSO algorthm s stable. In addton, the result was that the algorthm always start wth bg collecton of ndvduals and brng all the ndvduals together near the optmum value, thus becomng less dverse. Ths was true for all the dmensons, whch means that no matter how complex the problem becomes, PSO behaves the same. As a concluson, f we have very complex problem wth many unknowns, testng t for few unknowns wll gve us enough predcton whether PSO s able to solve ths problem or not. Thus we can use PSDF as a measure whether PSO can solve the problem n hand or not. Ths wll be done by applyng the PSDF analyss to less complex problem and observe the behavor of PSO to t. If PSO was able to converge to the optmum, then we can drectly use t for more complex cases. From ths experment, we can observe that PSO has a convergence rate whch s qute hgh, the stablty s acheved n fewer generatons, and the dversty of ndvduals reduces sgnfcantly. In addton, PSDF values helped n predctng whether PSO wll be able to solve more complex cases or not. In future work, we wll try to fnd the actual values of unknowns n more complex dmenson usng PSO, f we have the values of unknowns n less complex problem. References [] P. J. Angelne, Evolutonary Optmzaton versus Partcle Swarm Optmzaton: Phlosophcal and Performance Dfferences, n Proc. Evolutonary Programmng VII (EP98,) LNCS 447, pp. 6-6, 998. [] A. Czarn, C. MacNsh, K. Vjayan, B. Turlach, and R. Gupta, Statstcal Exploratory Analyss of Genetc Algorthms, IEEE Transactons on Evolutonary Computaton, vol. 8(4), August 4, pp. 45-4, 4 [3] B. Cestnk, Estmatng probabltes:a Crucal Task n Machne Learnng, n Proc. The European Conference n Artfcal Intellgence, pp [4] Y. Sh and R. Eberhart, Parameter Selecton n Partcle Swarm Optmzaton, Proc. EP98, 998. [5] B. Jung and B. Karney, "Benchmark Tests of Evolutonary Computatonal Algorthms, Envronmental Informatcs Acheves, vol., pp , 4. [6] W. Chang, A. Sutclffe and R. Nevlle, A Dstance Functon-Based Mult-Objectve Evolutonary Algorthm, n James Foster (edtors), Genetc and Evolutonary Computaton Conference. Late-Breakng Papers, AAAI, Chcago, Illnos, USA, pp , 3. [7] S. Lous, Genetc Learnng for Combnatonal Logc Desgn, Journal of Soft Computng manuscrpt, vol. 9(), pp , 4. [8] J. Holland, Adaptaton n Natural and Artfcal Systems, The Unversty of Mchgan Press, 975. [9] X. Hu, PSO Tutoral, [] J. Kennedy and R. Eberhart, Partcle Swarm Optmzaton, n Proc. the IEEE Internatonal Conference on Neural Networks, 995. [] S. Manay and A. Yezz, A Second-Order PDE Technque to Construct Dstance Functons wth More Accurate Dervatves, n Proc. IEEE Internatonal Conference on Image Processng, 3, vol., pp. I , 3. [] X. Yao, How Powerful/Effcent Is Your Evolutonary Algorthm, A keynote Presentaton at 4 IEEE Congress on Evolutonary Computatons, Portland, Oregon, USA, 4. [3] Y. Sh and R. Eberhart, A Modfed Partcle Swarm Optmzer, n Proc. the IEEE Internatonal Conference on Evolutonary Computaton, Anchorage, Alaska, pp , 998. [4] S. Habb and B. Al-kazem, Comparatve Study Between the Internal Behavor of GA and PSO Through Problem-Specfc Dstance Functons, n Proc. 5 IEEE Congress on Evolutonary Computaton, 5. [5] K. McAloon and C. Tretkoff, Optmzaton and Computatonal Logc, Wley Interscence Seres n Mathematcs and Optmzaton. Wley Press, New York, 996.