Performance Evaluation of an Advanced Local Search Evolutionary Algorithm

Transcription

1 Anne Auger and Nikolau Hanen Performance Evaluation of an Advanced Local Search Evolutionary Algorithm Proceeding of the IEEE Congre on Evolutionary Computation, CEC 2005 c IEEE

2 Performance Evaluation of an Advanced Local Search Evolutionary Algorithm Anne Auger CoLab Computational Laboratory, ETH Zürich, Switzerland Nikolau Hanen CSE Lab, ETH Zürich, Switzerland Abtract- One natural quetion when teting performance of global optimization algorithm i: how performance compare to a retart local earch algorithm. One purpoe of thi paper i to provide reult for uch comparion. To thi end, the performance of a retart (advanced local-earch trategy, the CMA-ES with mall initial tep-ize, are invetigated on the 25 function of the CEC 2005 real-parameter optimization tet uit. The econd aim i to clarify the theoretical background of the performance criterion propoed to quantitatively compare the earch algorithm. The theoretical analyi allow u to generalize the criterion propoed and to define a new criterion that can be applied more appropriate in a different context. 1 Introduction Thi paper introduce the retart verion of the o-called (µ, λ-cma-es (referred to a LR-CMA-ES and applie it to the CEC 2005 real-parameter optimization benchmark function uit [7]. The LR-CMA-ES i a quai parameter free, comparatively imple global optimization algorithm exploiting the advanced local earch propertie of the (µ, λ-cma-es. Beyond the interet of the LR-CMA-ES a a global optimization algorithm, the purpoe of invetigating a retart trategy of a competitive local earch evolutionary algorithm i to provide baeline reult for the CEC 2005 real-parameter optimization benchmark function uit. The econd contribution of thi paper i the theoretical analyi of the (ucce performance criterion propoed for reporting the reult [7]. Thi analyi allow u to clarify and generalize the criterion. The CMA-ES wa originally introduced to improve the local earch performance of evolution trategie [5]. It reduce the number of function evaluation to olve badly caled quadratic problem by everal order of magnidute [6]. Compared to other evolutionary algorithm, an important property of the CMA-ES i it invariance againt linear tranformation of the earch pace. Surpriingly, alo global earch propertie can be improved by the CMA [6], and, depending on the population ize, the CMA-ES even reveal competitive global earch performance [4]. In order to tre the local earch characteritic of the CMA-ES, we ue a hundred time maller initial tep-ize than i recommended a default. Moreover we tick to the default population ize (between 10 and 15 for the earch pace dimenion in thi paper, which lead to competitive local earch performance. The reulting algorithm can be then regarded a an advanced local earch, becaue (a the complete covariance matrix of the earch ditribution i efficiently adapted to the local topography of the objective function and (2 tep-ize adaptation can reult in comparatively large tep even when the initial tep-ize i choen to be mall. The remainder of thi paper i organized a follow: Section 2 analyze and generalize the ucce performance criterion given in [7], Section 3 preent the bottom line of the algorithm, Section 4 explain the experimental procedure and Section 5 preent the experimental reult. 2 Succe performance a the random variable meauring the running time (here, number of function evaluation for unucceful run of A topped after a reaonable topping criterion and TA the number of evaluation for a ucceful run of A. Let T be the random variable meauring the overall running time (here: overall number of function evaluation until the ucce criterion i met by independent retart of A, Comparing the performance of different algorithm on multi-modal problem implie to take into account that ome algorithm may have a mall probability of ucce but converge fat wherea other may have a larger probability of ucce but be lower. Thereby, one way to meaure (ucce performance of an Algorithm A i to invetigate the expected number of function evaluation to reach a certain function value (the ucce criterion by conducting independent retart of A. e conider that A ha a probability of ucce p (0, 1] (or ucce rate and define T u A N 1 T = (TA u k + TA, (1 k=1 where (TA u k are independent random variable with the ame ditribution a TA u and N the random variable meauring the number of run of A (N 1 unucceful, and 1 ucceful run. The random variable N follow a geometric ditribution with parameter p. To compute the expectation E(T of T, we firt take the conditional expectation of Eq. 1 with repect to the random variable N and ue the fact that (TA u k are i.i.d a TA u E(T N = (N 1E(T u A + E(T A Taking now the expectation again, we obtain the general expreion for the expectation of T E(T = (E(N 1E(TA u + E(T A ( 1 p = E(TA u + E(T p A (2

3 where we have ued the property that the expectation of a geometric ditribution of parameter p i equal to 1/p. e can in the ame way derive the variance of T tarting from Eq. 1. e omit the intermediate tep and give the final expreion: ( 1 p var(t = var(ta u p ( 1 p ( + E p 2 (T u A 2 + var(t A (3 where we ue the notation var to denote the variance of a random variable. From the general expreion Eq. 2 for the expectation of T we now derive two pecific (ucce performance criteria for tochatic earch algorithm. Moreover from Eq. 3 we will derive the variance of the econd ucce performance criterion. Firt, making the aumption that the expected number of evaluation for ucceful and unucceful run i the ame, i.e E(TA u = E(T A, the RHS of Eq. 2 implifie to the following expreion that correpond to the performance criterion defined in [7] (already propoed in [4] SP1 = E(T A p. (4 Second, we ue the fact that the algorithm A invetigated in thi paper i a retart trategy, and that the maximum number of function evaluation i given a FE max = n 10 4 [7]. Then, becaue A proceed with a retart whenever a topping criterion i met, any unucceful run of A reache the maximum number of function evaluation allowed. Therefore, E(TA u = FE max and var(ta u = 0 where var denote the variance of a random variable. Thee expreion lead to a different implification of Eq. 2, and to the definition of the econd ucce performance: ( 1 p SP2 = FE max + E(T p A (5 with it variance ( 1 p var(sp2 = (FE max 2 + var(ta, (6 derived from Eq. 3. p 2 Etimating the ucce performance The etimator of SP1 propoed in [7] can be derived by firt etimating the probability of ucce p a p = Nbr. ucceful run. Nbr. run Thi etimator i a maximum likelihood etimator for p and i unbiaed. Second the etimation of the expected number of function evaluation for ucceful run i Ê(TA Nbr. of evaluation for ucceful run = Nbr. ucceful run hen p 0, an etimator ŜP 1 for SP1 i then ŜP 1 = Ê(T A p. (7 Thi etimator i aymptotically convergent. Aymptotically convergent etimator for SP2 and var(sp2 can be derived uing a well p : ( 1 p ŜP2 = FE max + p Ê(T A (8 ( var(sp2 1 p = 2 (FE max 2 + var(t A p (9 q var(sp2 Nucce with var(t A the claical unbiaed etimator for the variance of the number of ucceful run. A more natural etimator for E(N = 1 p conit in ampling a fixed number of geometric random variable of parameter p and computing the empirical mean. In particular thi etimator i unbiaed. A ampling a geometric random variable of parameter p mean retarting the algorithm A until ucce, ampling N ucce geometric random variable implie doing run of A until a fixed number of uccee N ucce i reached. Therefore the number of run of A performed i not a fixed number (like for ŜP 1 and ŜP 2 but a random variable. Thi random variable depend on the probability of ucce and will increae in expectation for decreaing probability of ucce. The drawback of uch an etimator in practice i that the number of run performed i not fixed. However, ince uch an etimator i the empirical mean of independent identically ditributed random variable, aymptotic confidence interval can be derived from the Central Limit Theorem. Let SP 2 denote the etimator, with a probability of 0.95 we have aymptotically that SP2 SP 2 ± 1.96, where N ucce i the number of geometric random variable ampled (correponding to doing run of A until N ucce ucce are reached. Thi confidence interval ugget that confidence interval for ŜP 1 and ŜP 2 cale like 1. bp Nbr. run 3 The retart CMA-ES The (µ, λ-cma-es In thi paper we ue the (µ, λ- CMA-ES thoroughly decribed in [4]. e outline the general principle of the algorithm in hort and refer to [4] for the detail. For generation g + 1, λ offpring are ampled independently according to ( N x (g, (σ(g 2 C (g for k = 1,..., λ x (g+1 k where N ( m, C denote a normally ditributed random vector with mean m and covariance matrix C. The µ bet offpring are recombined into x (g+1 = µ i=1 w i x (g+1 i:λ, where the poitive weight w i R um to one. The equation for updating the remaining parameter of the normal ditribution are given in [4]: Eq. 2 and 3 for the covariance

4 matrix C, Eq. 4 and 5 for the tep-ize σ (cumulative path length control. 1 On convex quadratic function, the adaptation mechanim for σ and C allow to achieve log-linear convergence 2 after an adaptation time which can cale between 0 and n 2. The default parameter for the trategy are given in [4], Eq The default population ize grow with log n and equal to λ = 10, 14, 15 for n = 10, 30, 50. Only x (0 and σ (0 have to be et depending on the problem. The local retart (µ, λ-cma-es (LR-CMA-ES For the retart trategy the (µ, λ-cma-es i topped whenever a topping criterion i met, and a retart i launched. The new run ue the ame trategy parameter and the ame initialization procedure, and it i independent of all other run. To decide when to retart, the following topping criteria are ued. 3 Stop if the range of the bet objective function value of the lat n/λ generation i zero (equalfunvalhit, or the range of thee function value and all function value of the lat generation i below Tolfun= Stop if the tandard deviation of the normal ditribution i maller than TolX in all coordinate and if σ p c (the evolution path from Eq. 2 in [4] i maller than TolX in all component. e et TolX= σ (0. Stop if adding a 0.1-tandard deviation vector in a principal axi direction of C (g doe not change x (g (noeffectaxi. More formally, top if x (g equal to x (g +0.1 σ(g λ i u i, where i = (g mod n + 1, and λ i and u i are repectively the ith eigenvalue and eigenvector of C (g, with u i = 1. Stop if adding 0.2-tandard deviation in each coordinate doe change x (g (noeffectcoord. Stop if the condition number of the covariance matrix exceed (conditioncov. The ditribution of the tarting point x (0 and the initial tep-ize σ (0 are problem dependent and their etting i decribed in the next ection, a well a the overall topping criteria for the LR-CMA-ES. 4 Experimental procedure The LR-CMA-ES ha been invetigated on the 25 tet function decribed in [7] for dimenion 10, 30 and 50. For each function a bounded ubet [A, B] n of R n i precribed. The 1 A more elaborated algorithm decription can be acceed via 2 On a log cale the performance i linear with repect to the number of function evaluation. 3 Thee topping criteria were developed before the benchmark function uit ued in thi paper wa aembled. Table 1: Meaured CPU-econd, according to [7], uing MATLAB 7.0.1, Red Hat Linux 2.4, 1GByte RAM, Pentium 4 3GHz proceor. Time T2 i the CPU-time for running the retart CMA-ES until function evaluation on function 3. The maller number for T2 for n = 30 compared to n = 10 i caued by the 1.4 time larger population ize for n = 30. Becaue each population i evaluated (erially within a ingle function call the number of function call to reach function evaluation i maller T0 T1 T2 n = n = n = initial tarting point x (0 for each retart are ampled uniformly within thi ubet and the initial tep-ize σ (0 for each retart i equal to 10 2 (B A/2. The overall topping criteria for the algorithm precribed in [7] are: top before n 10 4 function evaluation or top if the error in the function value i below The boundary handling i done according to the tandard implementation of CMA- ES and conit in penalizing the individual in the infeaible region. 4 For each tet function, 25 run are performed. All performance criteria were evaluated baed on the ame run. In particular, the time when to meaure the objective function error value (namely at 10 3, 10 4, 10 5 function evaluation were not ued a input parameter to the algorithm (e.g., to et the maximum number of function evaluation to adjut an annealing rate. Tet function The complete definition of the tet uit i available in [7]. The definition of function 1 to 12 i baed on claical benchmark function, that we will refer in the equel alo by their name. Function 1 to 5 are unimodal and function 6 to 12 are multi-modal. Function 13 to 25 reult from the compoition of everal function. To prevent exploitation of ymmetry of the earch pace and of the typical zero value aociated with the global optimum, the local optimum i hifted to a value different from zero and the function value of the global optima are non zero. 5 Reult Figure 1 preent the convergence graph of objective function error value. The tep in the graph are caued by the retart that improve the performance on the noiy function 4 and on the multi-modal function 11 13, and According to the requirement, Table 1 report CPUtime meaurement, Table 2 give the number of function evaluation to reach the ucce criterion (if ucceful, the ucce rate, and the ucce performance a defined in Section 2. The objective function error value after 10 3, 10 4, 10 5 and n 10 4 function evaluation are preented in Table 3, 4, and 5. 4 For detail refer to the ued MATLAB code, cmae.m, Verion 2.35, ee

5 f f f f f function evaluation function evaluation function evaluation function evaluation function evaluation Figure 1: Bet objective function error value (log cale veru number of function evaluation for the 25 benchmark function in dimenion n = 30. For each run the bet individual found until the current generation i conidered and hown i the median value of 25 run at each generation. The repective problem number i given in the legend. In ome cae (function 13, n = 10; function 21, n = 10, 50, and function 23, n = 50, the LR-CMA-ES ignificantly outperform IPOP-CMA-ES [1], a trategy with ucceively increaed population ize. 5 Furthermore, the diverity between the run i often larger for the LR-CMA- ES, and the bet of 25 run i (ligthly better in mot cae on function 15, 18 23, and 25. The performance on function 1 3, and 5 7 i highly competitive (beide n = 50 for function 5. Even on the multi-modal Griewank function 7 the ucce rate i 100% (ee Table 2. Here we oberve an initial increae of the tep-ize by about two order of magnitude almot up to a tep-ize that i typically ued a initial tep-ize. If the upper bound for the tep-ize i et to the initial tep-ize, reult on the Griewank function become ignificantly wore (not hown. Thi clearly indicate that, due to it tepize adaptation, the LR-CMA-ES allow yet to earch more globally than a pure local earch method. On the remaining multi-modal function LR-CMA-ES fail to locate the global optimum. hile on Ratrigin function 9 and 10 the CMA-ES with large population ize i able to locate the global optimum [4, 1], for the compoite function the feaibility of locating the global optimum ha yet to be hown. The failure on the noiy function 4 can be explained by the mall initial tep-ize. In a highly noiy environment, the cumulative tep-ize adaptation of the CMA fail to enlarge the tep-ize [2]. 5 The tatitical tet table for the non-parametric ilcoxon rank um tet for different median value are given in [1]. To judge the ignificance level, we apply the (mot conervative Bonferroni correction for multiple teting. 6 Summary and concluion In thi paper empirical reult on the CEC 2005 real parameter optimization benchmark function uit are preented for the LR-CMA-ES, a local retart trategy of a competitive local earch evolutionary algorithm. The purpoe of the reult i to be a baeline for comparion. On the non-noiy unimodal and on few multi-modal function the LR-CMA- ES reveal competitive performance. The reult on the remaining multi-modal function erve a a benchmark that any ambitiou global earch algorithm ha to beat. Second, we have introduced a general performance criterion baed on the ucce rate and the running time of ucceful and unucceful run (e.g. in term of number of function evaluation. Thi criterion reveal a meaningful ingle number that can be ued to quantitatively compare earch algorithm on function, where at leat one run reache a given ucce criterion (e.g. a given function value. Finally we emphaize two important apect that have to be taken into account when judging the performance of earch algorithm. Firt, the LR-CMA-ES i quai parameter free: 6 in the preented experiment only the initial earch 6 Remark that the number of parameter in the decription of an algorithm i omewhat arbitrary: the more general the decription, the more parameter appear. Therefore, the exitence or abence of parameter in the algorithm decription cannot have influence on the aement of the number of parameter that need to be (empirically or heuritically determined each time the algorithm i applied. For the CMA-ES, trategy parameter have been choen in advance, baed on principle algorithmic conideration and in-depth empirical invetigation on a few imple tet function. To our experience the trategy parameter (e.g. a learning rate, a time horizon, or a damping factor motly depend on algorithmic internal conideration and on the earch pace dimenion, and to a much leer extend on

6 Table 2: Performance meaure for uccefully optimized problem. Prob.: Problem number; Tol: ucce criterion on function value error; 3rd 9th column: number of function evaluation (minimal, 7 th, median, 19 th, maximal, mean and tandard deviation to reach the ucce criterion Tol; 10th 14th column: Empirical etimator for the ucce probability p, for the ucce performance criteria SP1 and SP2, and for td(sp2 n = 10 n = 30 n = 50 Prob. Tol min 7 th median 19 th max mean td bp SP1 d SP2 d qvar( SP2 d 1 1e e e e e e e e e e e+2 2 1e e e e e e e e e e e+2 3 1e e e e e e e e e e e+2 4 1e e e e e e e e+5 5 1e e e e e e e e e e e+2 6 1e e e e e e e e e e e+3 7 1e e e e e e e e e e e+3 8 1e e+0-9 1e e e e e e e e e e e e e e+5 Prob. Tol min 7 th median 19 th max mean td bp SP1 d SP2 d qvar( SP2 d 1 1e e e e e e e e e e e+2 2 1e e e e e e e e e e e+2 3 1e e e e e e e e e e e+2 4 1e e+0-5 1e e e e e e e e e e e+4 6 1e e e e e e e e e e e+4 7 1e e e e e e e e e e e+3 8 1e e+0-9 1e e e e e e e e+0 - Prob. Tol min 7 th median 19 th max mean td bp SP1 d SP2 d qvar( SP2 d 1 1e e e e e e e e e e e+2 2 1e e e e e e e e e e e+2 3 1e e e e e e e e e e e+3 4 1e e+0-5 1e e e e e e e+6 6 1e e e e e e e e e e e+4 7 1e e e e e e e e e e e+2 8 1e e+0-9 1e e e e e e e e+0 - region wa defined problem dependent and no (further parameter tuning wa performed. Second, the LR-CMA-ES ha everal invariance propertie [3], like invariance againt order preerving tranformation of the objective function value and invariance againt linear tranformation of the earch pace [6]. Invariance propertie are highly deirable, becaue they imply uniform performance on clae of function and therefore allow for generalization of the empirical reult. Invariance and the procedure to adjut parameter need to be carefully regarded for a concluive performance evaluation of earch algorithm. Acknowledgment The author would like to greatly thank Stefan Kern for hi contribution to the paper, the pecific objective function the algorithm i applied to. Neverthele, it i poible to improve the performance by tuning trategy parameter and topping criteria on mot (all? function. Bibliography [1] A. Auger and N. Hanen. A retart CMA evolution trategy with increaing population ize. In Proceeding of the IEEE Congre on Evolutionary Computation, [2] H-G. Beyer and D. Arnold. Qualm regarding the optimality of cumulative path length control in CSA/CMAevolution trategie. Evol. Comput., 11(1:19 28, [3] N. Hanen. Invariance, elf-adaptation and correlated mutation in evolution trategie. In M. Schoenauer, K. Deb, G. Rudolph, X. Yao, E. Lutton, J.J. Merelo, and H.-P. Schwefel, editor, Parallel Problem Solving from Nature - PPSN VI, page Springer, [4] N. Hanen and S. Kern. Evaluating the CMA evolution trategy on multimodal tet function. In Xin Yao et al., editor, Parallel Problem Solving from Nature - PPSN VIII, LNCS 3242, page Springer, [5] N. Hanen and A. Otermeier. Adapting arbitrary normal mutation ditribution in evolution trategie: The

7 Table 3: Bet objective function error value reached after 10 3, 10 4 and 10 5 function evaluation (FES repectively (row on the 25 tet problem (column in dimenion n = 10. Given are minimum, 7 th, median, 19 th, and maximum value from 25 run, a well a mean and tandard deviation. A run i topped whenever the objective function error value drop below 10 8 and it final value i ued for all larger FES FES Prob min 1.35e e e e e e e e e e e e+1 7 th 3.41e e e e e e e e e e e e+3 med. 6.39e e e e e e e e e e e e+4 1e3 19 th 1.49e e e e e e e e e e e e+4 max 7.71e e e e e e e e e e e e+5 mean 1.45e e e e e e e e e e e e+4 td 1.94e e e e e e e e e e e e+4 min 1.81e e e e e e e e e e e e 9 7 th 3.83e e e e e e e e e e e e+2 med. 5.39e e e e e e e e e e e e+4 1e4 19 th 6.58e e e e e e e e e e e e+4 max 8.59e e e e e e e e e e e e+5 mean 5.14e e e e e e e e e e e e+4 td 1.82e e e e e e e e e e e e+4 min 1.81e e e e e e e e e e e e 9 7 th 3.83e e e e e e e e e e e e 9 med. 5.39e e e e e e e e e e e e+1 1e5 19 th 6.58e e e e e e e e e e e e+1 max 8.59e e e e e e e e e e e e+3 mean 5.14e e e e e e e e e e e e+2 td 1.82e e e e e e e e e e e e+2 FES Prob min 2.61e e e e e e e e e e e e e+2 7 th 3.38e e e e e e e e e e e e e+2 med. 3.70e e e e e e e e e e e e e+2 1e3 19 th 4.11e e e e e e e e e e e e e+3 max 5.92e e e e e e e e e e e e e+3 mean 3.87e e e e e e e e e e e e e+3 td 7.95e e e e e e e e e e e e e+2 min 5.08e e e e e e e e e e e e e+2 7 th 6.87e e e e e e e e e e e e e+2 med. 8.24e e e e e e e e e e e e e+2 1e4 19 th 9.16e e e e e e e e e e e e e+2 max 1.60e e e e e e e e e e e e e+3 mean 8.79e e e e e e e e e e e e e+2 td 3.23e e e e e e e e e e e e e+2 min 1.88e e e e e e e e e e e e e+2 7 th 4.16e e e e e e e e e e e e e+2 med. 4.79e e e e e e e e e e e e e+2 1e5 19 th 5.60e e e e e e e e e e e e e+2 max 8.17e e e e e e e e e e e e e+3 mean 4.94e e e e e e e e e e e e e+2 td 1.38e e e e e e e e e e e e e+2 covariance matrix adaptation. In Proceeding of the 1996 IEEE Conference on Evolutionary Computation (ICEC 96, page , [6] N. Hanen and A. Otermeier. Completely derandomized elf-adaptation in evolution trategie. Evol. Comput., 9(2: , [7] P. N. Suganthan, N. Hanen, J. J. Liang, K. Deb, Y.- P. Chen, A. Auger, and S. Tiwari. Problem definition and evaluation criteria for the cec 2005 pecial eion on real-parameter optimization. Technical report, Nanyang Technological Univerity, Singapore, May

8 Table 4: Bet objective function error value reached in dimenion n = 30, ee caption of Table 3 for detail FES Prob min 1.06e e e e e e e e e e e e+5 7 th 3.49e e e e e e e e e e e e+5 med. 4.74e e e e e e e e e e e e+5 1e3 19 th 5.43e e e e e e e e e e e e+5 max 9.11e e e e e e e e e e e e+5 mean 4.54e e e e e e e e e e e e+5 td 1.83e e e e e e e e e e e e+5 min 2.98e e e e e e e e e e e e+3 7 th 4.81e e e e e e e e e e e e+4 med. 5.40e e e e e e e e e e e e+4 1e4 19 th 5.82e e e e e e e e e e e e+4 max 7.06e e e e e e e e e e e e+5 mean 5.28e e e e e e e e e e e e+4 td 9.82e e e e e e e e e e e e+4 min 2.98e e e e e e e e e e e e+3 7 th 4.81e e e e e e e e e e e e+3 med. 5.40e e e e e e e e e e e e+4 1e5 19 th 5.82e e e e e e e e e e e e+4 max 7.06e e e e e e e e e e e e+5 mean 5.28e e e e e e e e e e e e+4 td 9.82e e e e e e e e e e e e+4 min 2.98e e e e e e e e e e e e+0 7 th 4.81e e e e e e e e e e e e+3 med. 5.40e e e e e e e e e e e e+3 3e5 19 th 5.82e e e e e e e e e e e e+4 max 7.06e e e e e e e e e e e e+4 mean 5.28e e e e e e e e e e e e+4 td 9.82e e e e e e e e e e e e+4 FES Prob min 2.98e e e e e e e e e e e e e+3 7 th 6.37e e e e e e e e e e e e e+3 med. 1.32e e e e e e e e e e e e e+3 1e3 19 th 2.15e e e e e e e e e e e e e+3 max 1.19e e e e e e e e e e e e e+3 mean 2.06e e e e e e e e e e e e e+3 td 2.54e e e e e e e e e e e e e+1 min 2.61e e e e e e e e e e e e e+2 7 th 3.05e e e e e e e e e e e e e+2 med. 3.67e e e e e e e e e e e e e+2 1e4 19 th 4.19e e e e e e e e e e e e e+3 max 5.02e e e e e e e e e e e e e+3 mean 3.64e e e e e e e e e e e e e+2 td 7.27e e e e e e e e e e e e e+2 min 2.09e e e e e e e e e e e e e+2 7 th 2.52e e e e e e e e e e e e e+2 med. 2.82e e e e e e e e e e e e e+2 1e5 19 th 3.01e e e e e e e e e e e e e+3 max 3.82e e e e e e e e e e e e e+3 mean 2.84e e e e e e e e e e e e e+2 td 4.69e e e e e e e e e e e e e+2 min 1.48e e e e e e e e e e e e e+2 7 th 2.10e e e e e e e e e e e e e+2 med. 2.24e e e e e e e e e e e e e+2 3e5 19 th 2.52e e e e e e e e e e e e e+3 max 2.98e e e e e e e e e e e e e+3 mean 2.32e e e e e e e e e e e e e+2 td 3.46e e e e e e e e e e e e e+2

9 Table 5: Bet objective function error value reached in dimenion n = 50, ee caption of Table 3 for detail FES Prob min 2.02e e e e e e e e e e e e+6 7 th 3.25e e e e e e e e e e e e+6 med. 3.57e e e e e e e e e e e e+6 1e3 19 th 4.09e e e e e e e e e e e e+6 max 7.02e e e e e e e e e e e e+6 mean 3.82e e e e e e e e e e e e+6 td 1.02e e e e e e e e e e e e+6 min 4.34e e e e e e e e e e e e+4 7 th 5.71e e e e e e e e e e e e+5 med. 6.23e e e e e e e e e e e e+5 1e4 19 th 6.74e e e e e e e e e e e e+5 max 7.50e e e e e e e e e e e e+6 mean 6.20e e e e e e e e e e e e+5 td 8.04e e e e e e e e e e e e+5 min 4.34e e e e e e e e e e e e+3 7 th 5.71e e e e e e e e e e e e+5 med. 6.23e e e e e e e e e e e e+5 1e5 19 th 6.74e e e e e e e e e e e e+5 max 7.50e e e e e e e e e e e e+6 mean 6.20e e e e e e e e e e e e+5 td 8.04e e e e e e e e e e e e+5 min 4.34e e e e e e e e e e e e+3 7 th 5.71e e e e e e e e e e e e+4 med. 6.23e e e e e e e e e e e e+4 5e5 19 th 6.74e e e e e e e e e e e e+5 max 7.50e e e e e e e e e e e e+5 mean 6.20e e e e e e e e e e e e+4 td 8.04e e e e e e e e e e e e+4 FES Prob min 2.01e e e e e e e e e e e e e+3 7 th 1.08e e e e e e e e e e e e e+3 med. 1.88e e e e e e e e e e e e e+3 1e3 19 th 2.59e e e e e e e e e e e e e+3 max 8.60e e e e e e e e e e e e e+3 mean 2.74e e e e e e e e e e e e e+3 td 2.55e e e e e e e e e e e e e+1 min 4.60e e e e e e e e e e e e e+2 7 th 6.53e e e e e e e e e e e e e+2 med. 7.33e e e e e e e e e e e e e+2 1e4 19 th 8.54e e e e e e e e e e e e e+3 max 2.01e e e e e e e e e e e e e+3 mean 8.16e e e e e e e e e e e e e+3 td 3.08e e e e e e e e e e e e e+2 min 4.25e e e e e e e e e e e e e+2 7 th 4.99e e e e e e e e e e e e e+2 med. 5.80e e e e e e e e e e e e e+2 1e5 19 th 6.47e e e e e e e e e e e e e+2 max 7.68e e e e e e e e e e e e e+3 mean 5.73e e e e e e e e e e e e e+2 td 9.62e e e e e e e e e e e e e+2 min 3.76e e e e e e e e e e e e e+2 7 th 4.37e e e e e e e e e e e e e+2 med. 4.56e e e e e e e e e e e e e+2 5e5 19 th 5.05e e e e e e e e e e e e e+2 max 5.60e e e e e e e e e e e e e+3 mean 4.70e e e e e e e e e e e e e+2 td 5.09e e e e e e e e e e e e e+2