Problem Defntons and Evaluaton Crtera for the CEC 15 Competton on Learnng-based Real-Parameter Sngle Objectve Optmzaton J. J. Lang 1, B. Y. Qu, P. N. Suganthan 3, Q. Chen 4 1 School of Electrcal Engneerng, Zhengzhou Unversty, Zhengzhou, Chna School of Electrc and Informaton Engneerng, Zhongyuan Unversty of Technology, Zhengzhou, Chna 3 School of Electrcal and Electronc Engneerng, Nanyang Technologcal Unversty, Sngapore 4 Faclty Desgn and Instrument Insttute, Chna Aerodynamc Research and Development Center, Chna langjng@zzu.edu.cn, qby1984@hotmal.com, epnsugan@ntu.edu.sg, chenqn198@gmal.com Techncal Report 1411A, Computatonal Intellgence Laboratory, Zhengzhou Unversty, Zhengzhou Chna And Techncal Report, Nanyang Technologcal Unversty, Sngapore November 14
1. Introducton Sngle objectve optmzaton algorthms are the bass of the more complex optmzaton algorthms such as mult-objectve, nchng, dynamc, constraned optmzaton algorthms and so on. Research on sngle objectve optmzaton algorthms nfluence the development of the optmzaton branches mentoned above. In the recent years, varous knds of novel optmzaton algorthms have been proposed to solve real-parameter optmzaton problems. Ths specal sesson s devoted to the approaches, algorthms and technques for solvng real parameter sngle objectve optmzaton wthout knowng the exact equatons of the test functons (.e. blackbox optmzaton). We encourage all researchers to test ther algorthms on the CEC 15 test sutes. The partcpants are requred to send the fnal results(after submttng ther fnal paper verson n March 15)n the format specfed n ths techncal report to the organzers. The organzers wll present an overall analyss and comparson based on these results. We wll also use statstcal tests on convergence performance to compare algorthms that eventually generate smlar fnal solutons. Papers on novel concepts that help us n understandng problem characterstcs are also welcome. Results of 1D and 3D problems are acceptable for the frst revew submsson. However, other dmensonal results as specfed n the techncal report should also be ncluded n the fnal verson, f space permts. Thus, fnal results for all dmensons n the format ntroduced n the techncal report should be zpped and sent to the organzers after the fnal verson of the paper s submtted. Please note that n ths competton error values smaller than 1-8 wll be taken as zero. You can download the C, JAVA and Matlab codes for CEC 15 test sute from the webste gven below: http://www.ntu.edu.sg/home/epnsugan/ndex_fles/cec15/cec15.htm Ths techncal report presents the detals of benchmark sute used for CEC 15 competton on learnng based sngle objectve global optmzaton. 1
1.1 Introducton to Learnng-Based Problems As a relatvely new solver for the optmzaton problems, evolutonary algorthm has attracted the attenton of researchers n varous felds. When testng the performance of a novel evolutonary algorthm, we always choose a group of benchmark functons and compare the proposed new algorthm wth other exstng algorthms on these benchmark functons. To obtan far comparson results and to smplfy the experments, we always set the parameters of the algorthms to be the same for all test functons. In general, specfyng dfferent sets of parameters for dfferent test functons s not allowed. Due to ths approach, we lose the opportunty to analyze how to adjust the algorthm to solve a specfed problem n the most effectve manner. As we all know that there s no free lunch and for solvng a partcular real-world problem, we only need one most effectve algorthm. In practce, t s hard to magne a scenaro whereby a researcher or engneer has to solve hghly dverse problems at the same tme. In other words, a practcng engneer s more lkely to solve numerous nstances of a partcular problem. Under ths consderaton and by the fact that by shftng the poston of the optmum and mldly changng the rotaton matrx wll not change the propertes of the benchmark functons sgnfcantly, we propose a set of learnng-based benchmark problems. In ths competton, the partcpants are allowed to optmze the parameters of ther proposed (hybrd) optmzaton algorthm for each problem. Although a completely dfferent optmzaton algorthm mght be used for solvng each of the 15 problems, ths approach s strongly dscouraged, as our objectve s to develop a hghly tunable algorthm to solve dverse nstances of real-world problems. In other words, our objectve s not to dentfy the best algorthms for solvng each of the 15 synthetc benchmark problems. To test the generalzaton performance of the algorthm and assocated parameters, the competton has two stages: Stage 1: Infnte nstances of shfted optma and rotaton matrxes can be generated. The partcpants can optmze the parameters of ther proposed algorthms for each problem wth these data and wrte the paper. Adaptve learnng methods are also allowed.
Stage : A dfferent testng set of shfted optma and rotaton matrces wll be provded to test the algorthms wth the optmzed parameters n Stage 1. The performance on the testng set wll be used for the fnal rankng. 1. Summary of the CEC 15 Learnng-Based Benchmark Sute TableI. Summary of the CEC 15 Learnng-Based Benchmark Sute No. Functons F *=F (x*) Unmodal Functons Smple Multmodal Functons Hybrd Functons 1 Rotated Hgh Condtoned Ellptc Functon 1 Rotated Cgar Functon 3 Shfted and Rotated Ackley s Functon 3 4 Shfted and Rotated Rastrgn s Functon 4 5 Shfted and Rotated Schwefel s Functon 5 6 Hybrd Functon 1 (N=3) 6 7 Hybrd Functon (N=4) 7 8 Hybrd Functon 3(N=5) 8 9 Composton Functon 1 (N=3) 9 1 Composton Functon (N=3) 1 Composton Functons 11 Composton Functon 3 (N=5) 11 1 Composton Functon 4 (N=5) 1 13 Composton Functon 5 (N=5) 13 14 Composton Functon 6 (N=7) 14 15 Composton Functon 7 (N=1) 15 Search Range: [-1,1] D *Please Note: 1. These problems should be treated as black-box problems. The explct equatons of the problems are not to be used.. These functons are wth bounds constrants. Searchng beyond the search range s not allowed. 3
1.3 Some Defntons: All test functons are mnmzaton problems defned as followng: D: dmensons. o 1 1 D Mn f(x), x [ x, x,..., x ] 1 T [ o, o,..., o ] : the shfted global optmum (defned n shft_data_x.txt ), whch s randomly dstrbuted n [-8,8] D. Each functon has a shft data for CEC 14. All test functons are shfted to o and scalable. For convenence, the same search ranges are defned for all test functons. Search range: [-1,1] D. M : rotaton matrx. Dfferent rotaton matrces are assgned to each functon and each basc functon. The varables are dvded nto subcomponents randomly. The rotaton matrx for each subcomponents are generated from standard normally dstrbuted entres by Gram-Schmdt ortho-normalzaton wth condton number c that s equal to 1 or. D T 1.4 Defntons of the Basc Functons 1) Hgh Condtoned Ellptc Functon D 1 1 6 D1 f1( x) (1 ) x (1) ) Cgar Functon ( ) 1 D f x x x () 6 1 3) Dscus Functon f ( x ) 1 x x (3) D 6 3 1 4
4) Rosenbrock s Functon 1 4 D 1 1 f ( x ) (1( x x ) ( x 1) ) (4) 5) Ackley s Functon 1 1 f x x x e (5) D D 5( ) exp(. ) exp( cos( )) D 1 D 1 6) Weerstrass Functon 6 D kmax kmax k k k k 1 k k f ( x ) ( [ a cos( b ( x.5))]) D [ a cos( b.5)] (6) a=.5, b=3, kmax= 7) Grewank s Functon x x f7( x ) cos( ) 1 (7) D D 1 4 1 8) Rastrgn s Functon 8( ) D ( 1cos( ) 1) 1 f x x x (8) 9) Modfed Schwefel s Functon 9 D 1 f ( x) 418.989 D g( z ), z x +4.9687467536e+ 1/ zsn( z ) f z 5 ( z 5) gz ( ) (5 mod( z,5))sn( 5 mod( z,5) ) f z 5 1D ( z 5) (mod( z,5) 5) sn( mod( z,5) 5 ) f z 5 1D 1) Katsuura Functon x x round( x ) (1) D 3 j j 1 1 1. 1 D 1( ) (1 ) j D 1 j1 D f (9) 5
11) HappyCat Functon 11 D 1/4 D D 1 1 1 f ( x ) x D (.5 x x )/ D.5 (11) 1) HGBatFuncton 1 D D 1/ D D 1 1 1 1 f ( x ) ( x ) ( x ) (.5 x x )/ D.5 (1) 13) Expanded Grewank s plus Rosenbrock s Functon f ( x) f ( f ( x, x )) f ( f ( x, x ))... f ( f ( x, x )) f ( f ( x, x ))(13) 13 7 4 1 7 4 3 7 4 D1 D 7 4 D 1 14) Expanded Scaffer s F6 Functon (sn ( x y ).5) Scaffer s F6 Functon: gxy (, ).5 (1.1( x y )) f ( x ) g( x, x ) g( x, x )... g( x, x ) g( x, x ) (14) 14 1 3 D1 D D 1 1.5 Defntons of the CEC 15Learnng-Based Benchmark Sute A. Unmodal Functons: 1) Rotated Hgh Condtoned Ellptc Functon F ( x) f ( M ( xo )) F * (15) 1 1 1 1 1 Fgure 1.3-D map for -D functon 6
Propertes: Unmodal Non-separable Quadratc ll-condtoned ) Rotated Cgar Functon F ( x) f ( M ( xo )) F * (16) Fgure. 3-D map for -D functon Propertes: Unmodal Non-separable Smooth but narrow rdge B. Multmodal Functons 3) Shfted and Rotated Ackley s Functon F ( x) f ( M ( xo )) F * (17) 3 5 3 3 3 7
Fgure 3. 3-D map for -D functon Propertes: Mult-modal Non-separable 4) Shfted and Rotated Rastrgn s Functon 5.1( x o4) F4( x ) f8( M4( )) F4* (18) 1 Propertes: Fgure 4. 3-D map for -D functon Mult-modal Non-separable Local optma s number s huge 5) Shfted and Rotated Schwefel s Functon 1( x o5 ) F5( x ) f9( M5( )) F5* (19) 1 8
Fgure 5(a). 3-D map for -D functon Propertes: Fgure 5(b).Contour map for -D functon Mult-modal Non-separable Local optma s number s huge and second better local optmum s far from the global optmum. C. Hybrd Functons Consderng that n the real-world optmzaton problems, dfferent subcomponents of the varables may have dfferent propertes [8]. In ths set of hybrd functons, the varables are randomly dvded nto some subcomponents and then dfferent basc functons are used for dfferent subcomponents. F x g M z g M z g M z F x () * ( ) 1( 1 1) ( )... N( N N) ( ) F(x): g (x): hybrd functon th basc functon used to construct the hybrd functon 9
N: number of basc functons z = [ z, z,..., z ] 1 N z [ y, y,..., y ], z [ y, y,..., y ],..., z [ y, y,..., y ] 1 S1 S Sn S 1 n11 Sn 1 Sn 1n N S N1 SN1 SD n1 n 1 k1 y x- o, S randperm(1: D ) p : used to control the percentage of g (x) n : dmenson for each basc functon N 1 n D,,...,, N n pd n p D n p D n D n 1 1 N1 N1 N 1 Propertes: Mult-modal or Unmodal, dependng on the basc functon Non-separable subcomponents Dfferent propertes for dfferent varables subcomponents 1 6) Hybrd Functon 1 N= 3 p = [.3,.3,.4] g 1 : Modfed Schwefel's Functon f 9 g : Rastrgn's Functon f 8 g 3 : Hgh Condtoned Ellptc Functon f 1 7) Hybrd Functon 3 N= 4 p =[.,.,.3,.3] g 1 : Grewank's Functon f 7 g : Weerstrass Functon f 6 g 3 : Rosenbrock's Functon f 4 g 4 : Scaffer's F6 Functon f 14 1
8) Hybrd Functon 5 N= 5 p = [.1,.,.,.,.3] g 1 : Scaffer s F6 Functon f 14 g : HGBat Functon f 1 g 3 : Rosenbrock s Functon f 4 g 4 : Modfed Schwefel s Functon f 9 g 5 : Hgh Condtoned Ellptc Functon f 1 D. Composton Functons N 1 F( x) { *[ g ( x ) bas ]} F * (1) F(x): g (x): composton functon th basc functon used to construct the composton functon N: number of basc functons o : new shfted optmum poston for each g (x), defne the global and local optma s poston bas : defnes whch optmum s global optmum : used to control each g (x) s coverage range, a small gve a narrow range for that g (x) : w : used to control each g (x) s heght weght value for each g (x), calculated as below: w D j1 1 ( x o ) j j D ( xj oj) j1 exp( ) D () Then normalze the weght / n w w 1 So when x o, 1 j j j for j 1,,..., N, f ( x) bas f * 11
The local optmum whch has the smallest bas value s the global optmum. The composton functon merges the propertes of the sub-functons better and mantans contnuty around the global/local optma. Functons F =F-F * are used as g. In ths way, the functon values of global optma of g are equal to for all composton functons n ths report. For some composton functons, the hybrd functons are also used as the basc functons. Wth hybrd functons as the basc functons, the composton functon can have dfferent propertes for dfferent varables subcomponents. 9) Composton Functon 1 N = 3 = [,,] = [1, 1, 1] bas =[, 1, ]+F 9 * g 1 Schwefel's Functon g, g 3: Rotated Rastrgn s Functon Rotated HGBat Functon x 1 4 3.5 1.5 1.5-1 -1-8 -6-4 - - -4-6 -8 4 Fgure6(a). 3-D map for -D functon (example) 1
4 - -4-6 -8-1 -1-8 -6-4 - Propertes: Fgure6(b).Contour map for -D functon (example) Mult-modal Non-separable Dfferent propertes around dfferent local optma The basc functon of whch the global optmum belongs to s fxed. The sequence of the other basc functons can be randomly generated. 1) Composton Functon N = 3 = [1, 3,5] = [1, 1, 1] bas =[, 1, ]+F 1 * g 1, g, g 3 : Hybrd Functon 1 Hybrd Functon Hybrd Functon 3 Propertes: Mult-modal Non-separable Asymmetrcal Dfferent propertes around dfferent local optma Dfferent propertes for dfferent varables subcomponents 13
The sequence of the basc functons can be randomly generated. 11) Composton Functon 3 N = 5 = [1, 1, 1,, ] = [1, 1,.5, 5,1e-6] bas =[, 1,, 3, 4]+F 11 * g 1: Rotated HGBat Functon g, g 3,g 4,g 5: Rotated Rastrgn s Functon Rotated Schwefel's Functon Rotated Weerstrass Functon Rotated Hgh Condtoned Ellptc Functon x 1 7 1 1 8 6 4-1 -5 5 1 1 5-5 -1 Fgure 8(a). 3-D map for -D functon (example) 1 8 6 4 - -4-6 -8-1 -1-8 -6-4 - 4 6 8 1 Fgure8(b).Contour map for -D functon (example) 14
Propertes: Mult-modal Non-separable Asymmetrcal Dfferent propertes around dfferent local optma The basc functon of whch the global optmum belongs to s fxed. The sequence of the other basc functons can be randomly generated. 1) Composton Functon 4 N = 5 = [1,,,3,3] = [.5, 1, 1e-7, 1, 1] bas =[, 1, 1,, ]+F 1 * g 1, g, g 3,g 4,g 5: Rotated Schwefel's Functon Rotated Rastrgn s Functon Rotated Hgh Condtoned Ellptc Functon Rotated Expanded Scaffer s F6 Functon Rotated HappyCat Functon 4 3 1-1 -5 5-5 5 1 1-1 Fgure9(a). 3-D map for -D functon (example) 15
1 8 6 4 - -4-6 -8-1 -1-8 -6-4 - 4 6 8 1 Fgure9(b).Contour map for -D functon (example) Propertes: Mult-modal Non-separable Asymmetrcal Dfferent propertes around dfferent local optma Dfferent propertes for dfferent varables subcomponents The sequence of the basc functons can be randomly generated 13) Composton Functon 5 N = 5 = [1, 1, 1,, ] = [1, 1, 1, 5, 1] bas =[, 1,, 3, 4]+F 13 * g 1, g, g 3, g 4, g 5: Hybrd Functon 3 Rotated Rastrgn s Functon Hybrd Functon 1 Rotated Schwefel's Functon Rotated Expanded Scaffer s F6 Functon 16
Propertes: Mult-modal Non-separable Asymmetrcal Dfferent propertes around dfferent local optma The sequence of the basc functons can be randomly generated 14) Composton Functon 6 N = 7 = [1,, 3, 4, 5, 5, 5] = [1,.5,.5, 1,1e-6,1e-6, 1] bas =[, 1,, 3,3,4, 4]+F 14 * g 1: Rotated HappyCat Functon g, g 3, g 4, g 5, g 6, g 7: Rotated Expanded Grewank s plus Rosenbrock s Functon Rotated Schwefel's Functon Rotated Expanded Scaffer s F6 Functon Rotated Hgh Condtoned Ellptc Functon Rotated Cgar Functon Rotated Rastrgn s Functon x 1 4 3 1-1 5 1-5 5 1-1 -5 Fgure 1(a). 3-D map for -D functon (example) 17
1 8 6 4 - -4-6 -8-1 -1-8 -6-4 - 4 6 8 1 Propertes: Fgure 1(b).Contour map for -D functon (example) Mult-modal Non-separable Asymmetrcal Dfferent propertes around dfferent local optma The basc functon of whch the global optmum belongs to s fxed. The sequence of the other basc functons can be randomly generated. 15) Composton Functon 7 N = 1 = [1, 1,,, 3, 3, 4, 4, 5, 5] = [.1,.5e-1,.1,.5e-, 1e-3,.1, 1e-5, 1,.5e-, 1e-3] bas =[, 1, 1,,, 3, 3, 4, 4, 5]+F 15 * g 1, g, g 3, g 4, g 5, g 6, g 7, g 8, g 9, g 1: Rotated Rastrgn s Functon Rotated Weerstrass Functon Rotated HappyCat Functon Rotated Schwefel's Functon Rotated Rosenbrock's Functon Rotated HGBat Functon Rotated Katsuura Functon 18
Rotated Expanded Scaffer s F6 Functon Rotated Expanded Grewank s plus Rosenbrock s Functon Rotated Ackley Functon x 1 5.5 1.5 1.5 1 5-5 -1-1 -5 5 1 Fgure 11(a). 3-D map for -D functon (example) 1 8 6 4 - -4-6 -8-1 -1-8 -6-4 - 4 6 8 1 Propertes: Fgure 11(b).Contour map for -D functon (example) Mult-modal Non-separable Asymmetrcal Dfferent propertes around dfferent local optma The sequence of the basc functons can be randomly generated 19
.Evaluaton Crtera.1 Expermental Settng Problems: 15 mnmzaton problems Dmensons: D=1, 3, 5, 1 (Results only for 1D and 3D are acceptable for the ntal submsson; but 5D and 1D should be ncluded n the fnal verson) Runs / problem:51 (Do not run many 51 runs to pck the best run) MaxFES: 1*D (Max_FES for 1D= 1; for 3D=3; for 5D = 5; for 1D = 1) SearchRange: [-1,1] D Intalzaton: Unform random ntalzaton wthn the search space. Random seed s based on tme, Matlab users can use rand('state', sum(1*clock)). Global Optmum: All problems have the global optmum wthn the gven bounds and there s no need to perform search outsde of the gven bounds for these problems. F( x*) F( o ) F * Termnaton: Termnate when reachng MaxFES or the error value s smaller than 1-8..1 Results Record 1) Record functon error value (F (x)-f (x*)) after (.1,.1,.1,.,.3,.4,.5,.1,.,.3,.4,.5,.6,.7,.8,.9, 1.)*MaxFES for each run. In ths case, 17 error values are recorded for each functon for each run. Sort the error values acheved after MaxFES n 51 runs from the smallest (best) to the largest (worst) and present the best, worst, mean, medan and standard varance values of functon error values for the 51 runs. Please Notce: Error value smaller than 1-8 wll be taken as zero. ) Algorthm Complexty a) Run the test program below: for =1:1
x=.55 + (double); x=x + x; x=x/; x=x*x; x=sqrt(x); x=log(x); x=exp(x); x=x/(x+); end Computng tme for the above=t; b) Evaluate the computng tme just for Functon 1. For evaluatons of a certan dmenson D, t gves T1; c) The complete computng tme for the algorthm wth evaluatons of the same D dmensonal Functon 1 s T. d) Execute step c fve tmes and get fve T values. T =Mean(T) The complexty of the algorthm s reflected by: T,T1, T, and ( T -T1)/T The algorthm complextes are calculated on 1, 3, 5 and 1 dmensons, to show the algorthm complexty s relatonshp wth dmenson. Also provde suffcent detals on the computng system and the programmng language used. In step c, we execute the complete algorthm fvetmes to accommodate varatons n executon tme due adaptve nature of some algorthms. Please Note: Smlar programmng styles should be used for all T, T1 and T. (For example, f m ndvduals are evaluated at the same tme n the algorthm, the same style should be employed for calculatng T1; f parallel calculaton s employed for calculatng T, the same way should be used for calculatng T and T1. In other word, the complexty calculaton should be far.) 3) Parameters Partcpants are allowed to search for a dstnct set of parameters for each problem. Please provde detals on the followng whenever applcable: a) All parameters to be adjusted; b) Correspondng dynamc ranges; c) Gudelnes on how to adjust the parameters; d) Estmated cost of parameter tunng n terms of number of FEs; e) Actual parameter values used for each problem. 1
4) Results Format The partcpants are requred to send the fnal results as the followng format to the organzers and the organzers wll present an overall analyss and comparson based on these results. Create one txt document wth the name AlgorthmName_FunctonNo._D.txt for each test functon and for each dmenson. For example, PSO results for test functon 5 and D=3, the fle name should be PSO_5_3.txt. Then save the results matrx (the gray shadowng part) as Table II n the fle: Table II. Informaton Matrx for D Dmensonal Functon X ***.txt Run 1 Run Run 51 Functon error values when FES=.1*MaxFES Functon error values when FES=.1*MaxFES Functon error values when FES=.1*MaxFES Functon error values when FES=.*MaxFES Functon error values when FES=.3*MaxFES Functon error values when FES=.4*MaxFES Functon error values when FES=.5*MaxFES Functon error values when FES=.9*MaxFES Functon error values when FES=MaxFES Thus 15*4(1D, 3D, 5D and 1D)fles (each fle contans a 17*51matrx.) and a lst of the parameters used for each functon should be zpped and sent to the organzers. Notce: All partcpants are allowed to mprove ther algorthms further after submttng the ntal verson of ther papers to CEC14. And they are requred to submt ther results n the ntroduced format to the organzers after submttng the fnal verson of paper as soon as possble.
.3ResultsTemple Language: Matlab 13a Algorthm: Partcle Swarm Optmzer (PSO) Results Notce: Consderng the length lmt of the paper, only Error Values Acheved wth MaxFES are need to be lsted. Whle the authors are requred to send all results (15*4 fles descrbed n secton.) to the organzers for a better comparson among the algorthms. Table III. Results for 1D Func. Best Worst Medan Mean Std 1 3 4 5 6 7 8 9 1 11 1 13 14 15 Table IV. Results for 3D Table V. Results for 5D Table VI. Results for 1D 3
Algorthm Complexty D=1 D=3 D=5 D=1 Table VII. Computatonal Complexty T T1 T ( T -T1)/T Parameters a) All parameters to be adjusted b) Correspondng dynamc ranges c) Gudelnes on how to adjust the parameters d) Estmated cost of parameter tunng n terms of number of FES e) Actual parameter values used. References [1] Q. Chen, B. Lu and Q. Zhang, J. J. Lang, P. N. Suganthan, B. Y. Qu Problem Defntons and Evaluaton Crtera for CEC 15 Specal Sesson on Computatonally Expensve Sngle Objectve Optmzaton, Techncal Report, 14. [] B. Y. Qu, J. J. Lang, P. N. Suganthan, Q. Chen, "Problem Defntons and Evaluaton Crtera for the CEC 15 Specal Sesson and Competton on Sngle Objectve Mult-Nche Optmzaton", Techncal Report1411B,Computatonal Intellgence Laboratory, Zhengzhou Unversty, Zhengzhou Chna and Techncal Report, Nanyang Technologcal Unversty, Sngapore, November 14 [3] P. N. Suganthan, N. Hansen, J. J. Lang, K. Deb, Y.-P. Chen, A. Auger & S. Twar, "Problem Defntons and Evaluaton Crtera for the CEC 5 Specal Sesson on Real-Parameter Optmzaton," Techncal Report, Nanyang Technologcal Unversty, Sngapore, May 5 and KanGAL Report #55, IIT Kanpur, Inda, 5. [4] J. J. Lang, B. Y. Qu, P. N. Suganthan, Alfredo G. Hernández-Díaz, "Problem Defntons and Evaluaton Crtera for the CEC 13 Specal Sesson and Competton on Real-Parameter Optmzaton", Techncal Report 11, Computatonal Intellgence 4
Laboratory, Zhengzhou Unversty, Zhengzhou Chna and Techncal Report, Nanyang Technologcal Unversty, Sngapore, January 13. [5] J. J. Lang, B-Y. Qu, P. N. Suganthan, "Problem Defntons and Evaluaton Crtera for the CEC 14 Specal Sesson and Competton on Sngle Objectve Real-Parameter Numercal Optmzaton",Techncal Report1311,Computatonal Intellgence Laboratory, Zhengzhou Unversty, Zhengzhou Chna and Techncal Report, Nanyang Technologcal Unversty, Sngapore, December 13 [6] Joaqun Derrac, Salvador Garca, Sheldon Hu, Francsco Herrera, Ponnuthura N. Suganthan, "Statstcal analyss of convergence performance throughout the search: A case study wth SaDE-MMTS and Sa-EPSDE-MMTS," IEEE Symp. On DE 13, IEEE SSCI13, Sngapore. [7] Nkolaus Hansen, Steffen Fnck, Raymond Ros and Anne Auger, "Real-Parameter Black-Box Optmzaton Benchmarkng 1: Noseless Functons Defntons" INRIA research report RR-689,March 4,1. [8] Xaodong L, Ke Tang, Mohammad N. Omdvar, Zhenyu Yang, and Ka Qn, Benchmark Functons for the CEC 13 Specal Sesson and Competton on Large-Scale Global Optmzaton, Techncal Report,13 5