Nelder-Mead Method with Local Selection Using Neighborhood and Memory for Stochastic Optimization

Transcription

1 Journal of Computer Science, 9 (4): , 03 ISSN oi:0.3844/cssp Publishe Online 9 (4) 03 ( Neler-Mea Metho with Local Selection Using Neighborhoo an Memory for Stochastic Optimization Noocharin Tippayawannakorn an Juta Pichitlamken Department of Inustrial Engineering, Faculty of Engineering, Kasetsart University, Bangkok, 0900, Thailan eceive , evise ; Accepte ABSTACT We consier the Neler-Mea (NM) simple algorithm for optimization of iscrete-event stochastic simulation moels. We propose new moifications of NM to reuce computational time an to improve quality of the estimate optimal solutions. Our means inclue utilizing past information of alreay seen solutions, epaning search space to their neighborhoo an using aaptive sample sizes. We compare performance of these etensions on si test functions with 3 levels of ranom variations. We fin that using past information leas to reuction of computational efforts by up to 0%. The aaptive moifications nee more resources than the non-aaptive counterparts for up to 70% but give better-quality solutions. We recommen the aaptive algorithms with using memory with or without neighborhoo structure. Keywors: Neler-Mea Simple, Aaptive Neler-Mea Simple, Continuous Stochastic Optimization, Neighborhoo Search, Local Selection. INTODUCTION An Optimization via Simulation (OvS) is the problem of fining possible set of input variables or ecision variables that give maimum or minimum obective function values. In aition, a simulation optimization also aims at minimizing computational resources spent while maimizing the information obtaine in a simulation eperiment (Carson an Maria, 997). We are intereste in the OvS problems that have stochastic obective functions an continuous ecision variables (Alon et al., 005; Henerson an Nelson, 006; Olafsson an Kim, 00; Swisher et al., 004) for OvS surveys. Many OvS tools are evelope for unconstraine continuous problems. Most of them are base on the ranom search metho that takes obective function values from a set of sample points an uses that information to select the net points. Various techniques iffer in the choice of sampling strategies (Anraottir, 006). A point-base strategy involves sampling points in a neighborhoo of the current solution, e.g., the Stochastic uler (Alrefaei an Anraottir, 005) an the Simulate Annealing (Press et al., 007). A set-base strategy generates a set of caniate solutions from a subset of the feasible region, e.g., the Neste Partitions Metho (Shi an Olafsson, 009) an the Neler-Mea Simple (Neler an Mea, 965). A population-base strategy creates a collection of caniate solutions using some properties of the previously visite solutions; for eample, the Genetic Algorithm (Hollan, 000) an the Evolutionary Strategies (Beyer an Schwefel, 00). We focus on the Neler-Mea (NM) simple algorithm (Neler an Mea, 965), which is originally evelope for unconstraine eterministic optimization. It emonstrates wie versatility an ease of use such that it is implemente in MATLAB as a function fminsearch. The NM is also robust with respect to small ranom variations in the observe obective function values; therefore, it is use for optimizing stochastic problems as well (Tomick, 995; Humphrey an Wilson, 000) However, in the case that Corresponing Author: Noocharin Tippayawannakorn, Department of Inustrial Engineering, Faculty of Engineering. Kasetsart University, Bangkok, 0900, Thailan 463

2 Noocharin Tippayawannakorn an Juta Pichitlamken / Journal of Computer Science 9 (4): , 03 variability in the obective function values are sufficiently large, the NM may terminate before reaching the global optima. For this, Barton an Ivey (996) propose the algorithm S9 that improve NM performance for stochastic problems by increasing the shrink parameter an recalculating every point of shrink simple. In this stuy, we propose variants of the NM by utilizing past information an/or proimate points. Specifically, we incorporate: Information collecte since the search begins an Search neighborhoo With numerical eperiments, we show that our algorithms provie better solutions while requiring less computational efforts than the original NM. Generally, an OvS problem can be efine as follows: The obective is to etermine an optimal solution, *, that minimizes the unknown obective function, µ: Θ over a continuous feasible region, Θ * ; that is, = arg min µ (), where, Θ is a vector Θ of ecision variables an is calle a solution. The obective function µ() cannot be observe irectly; thus, it is estimate with stochastic simulation, i.e. Equation : µ () = E G(, ξ ) ξ () where, G (,ξ ) is a simulation output evaluate at an ξ is an unbiase ranom element with E[ ξ ] = 0 an V[ ξ ] =σ. The estimate of µ(), µ ˆ(), is a sample mean of m inepenent simulation outputs: µ ˆ() = G() = G(, ξ ) m i () m i = This stuy is organize as follows: Section introuces the original form of the NM simple algorithm, its eisting variants, our etensions an escribes esign of numerical eperimens. Section 3 shows of numerical results. Section 4 iscusses the results. Ultimately, we conclue in section 5. aacent to any given vertices. The original simple consists of the reflection of one verte through the centroi of the opposite face. Sometimes a sequence of reflections brings the search back to where it starts. Neler an Mea (965) a epansion an contraction moves to accelerate the search an a shrink step is introuce to ecrease the lengths of eges aacent to the current best verte by half, in case that none of the steps brings acceptable improvement to the original simple. Figure illustrates -imensional trial points for a simple consisting of 0, an. The soli lines simple is initialize. Other linestyle simplees show various simple operations, e.g., an epansion point is, a reflection point is E, internal i an eternal contraction points are C Ce an, respectively an C is the centroi of the best points. By the efault setting of fminsearch (the NM implementation in MATLAB), a single simulation output (m = in Equation ) is an estimate of an obective function µ ˆ(). The overall logical steps of the NM algorithm are shown in Fig. an it can be eplaine in more etails as follows. Initialization: Create an Initial Simple Select a starting point 0 Θ, a vector of imensions Form an initial simple of + points, by efining: = [,, K, + s, K, ],,, K,, (3) i i i where, s i are the user-specifie initial step sizes. Estimate the obective function values at each of the + simple points from m inepenent simulation outputs by computing its sample averages via Equation to get µ ˆ ( ˆ ˆ 0 ), µ ( ), K, µ ( ) then initialize the iteration number = an a number of observation of simulation outputs, count = m(+). MATEIALS AND METHODS.. The Neler-Mea Simple Algorithm... Original NM The first of the simple methos is ue to Spenley et al. (96) for eterministic problems. They assume that any point in the omain of search can be constructe by taking a linear combination of the eges 464 Fig.. All simple operations of the Neler-Mea simple

3 Noocharin Tippayawannakorn an Juta Pichitlamken / Journal of Computer Science 9 (4): , 03 Fig.. Flow chart for the th iteration of the Neler-Mea simple algorithm e-orer these points in a non-ecreasing orer so that µ ˆ( ˆ ˆ 0) µ ( ) K µ ( ) While ma ˆ ˆ i 0 ε an ma µ ˆ( ˆ ˆ ˆ i) µ ( 0) ε µ ˆ,,,..., <N search an count < N µ ˆ are true.,,..., Step : Calculate the eflection Point A worst point on the simple (recall that we consier a minimization problem, so the worst point is one with the highest sample mean ) is replace with another point which has a lower obective function. Let be the reflection of the worst point an passes through the centroi C of the -best points. These points are compute as C= an = 0 = C +α(c ) where, α <0 is a reflection parameter; typically. Then the obective function value µ ˆ( ) is estimate via m simulation outputs then count = count+m. Step : Upate the Simple Figure 3 shows an initial simple with ash lines an an upate simple with soli lines. The upate simple epens on the relationship between µ ˆ( ) an µ ˆ ( ), µ ˆ ( ), K, ˆ( ) 0 µ ; that is: If µ ˆ ( ˆ ˆ 0 ) µ ( ) < µ ( ), set an µ ˆ ( ) µ ˆ( ) as shown in Fig. 3a. Then go to Step 4. If µ ˆ ( ) <µ ˆ ( ), the search continues in the same 0 E irection by calculating the epansion point, = C+ γ( C ) where γ>0 is an epansion parameter, E µ is estimate from m typically. Then ˆ ( ) simulation outputs by Equation, then count = count + m. The epansion point is accepte when it improves over the best point in the simple, 0, as shown in Fig. 3b; otherwise, the reflection point is accepte. Go to Step

4 Noocharin Tippayawannakorn an Juta Pichitlamken / Journal of Computer Science 9 (4): , 03 (a) (b) (c) Fig. 3. Operations of the NM simple metho () If µ ˆ ( ) µ ˆ ( ), the search reuces the simple size by calculating the contraction point, C = C+ β(% C), where, β>0 is a contract parameter (generally 0.5), an % is if µ ˆ ( ˆ ) <µ ( ) an C otherwise. The obective function µ ˆ( ) is estimate C then count = count+m. If µ ˆ( ˆ ) µ ( ), go to Step 3; otherwise, the contraction point is accepte, i.e., C. The upate simple can be one of two soli-line simplees in Fig. 3c epening on which * * as the optimal ˆ an µ ˆ( ˆ ), respectively, when i when the search terminates. Contraction point ( c or c e ) is use. Go to Step 4. Step 3: Shrink the Simple If the reflection point an contraction point provie no improvement, then the simple is shrunk towar the best point 0 as shown in Fig. 3. Compute the new simple as follows: [, τ + ( τ), τ + ( τ), K, τ + ( τ) ] (4) where, τ is a shrink parameter, typically 0.5. The obective function values of these new points are estimate count = count +m. Then go to Step 4. Step 4: e-orer the Simple Points in Ascening Orers Then let = +. En while * * eturn ˆ an µ ˆ( ˆ ).... Barton an Ivey Stochastic Moification of NM an its Variant (S9 an ANS9) Barton an Ivey (996) aapt the NM algorithm to accommoate stochastic variations in the obective function values. From empirical results, they see that because the NM algorithm relies on the ranks of the obective function values at the simple vertices, it can make progress in presence of relatively small ranomness which oes not change the rank of the function value at the simple points. However, if the variations in the function value are large enough, it affects the relative rank of the simple vertices an misleas the algorithm. Barton an Ivey (996) recommen the shrinkage coefficient (in Equation 4) of 0.9 instea of the usual 0.5 to increase the etent of reuction after shrink. This change improves the performance effectively for the cases where the original NM fails. esampling the best point after shrink reuces the frequency of contraction, but this strategy is not effective in improving algorithm performance. 466

5 Noocharin Tippayawannakorn an Juta Pichitlamken / Journal of Computer Science 9 (4): , 03 Moreover, Barton an Ivey (996) apply another stopping criterion for stochastic problems, as suggeste by Dennis an Woos (987). The search terminates when the simple size is sufficiently small: i 0 ε ma(, ) 0 (5) where, µ ˆ( ˆ ˆ 0) <µ ( ) < K <µ ( ), an is the Eucliean norm, i.e., = + + L +. Besies the stopping criteria in Equation 5, their so-calle S9 is the NM with the following moifications: The obective function is estimate with 6 inepenent simulation outputs (m = 6 in Equation ); every solution is resample every time it is encountere (no search history is kept) an the shrinkage coefficient τ is 0.9. The rescaling operations of the NM algorithm can lea to a too-early termination at a non-optimum if noise is present. Tomick et al. (995) moify the S9 further to allow the sample sizes to aust aaptively to the observe noise in the solution space, calle ANS. Suppose that m is the minimum number of observations taken at each new trial point uring the th iteration an m 0 = 6: m bm if S / ( σ ) χ = m otherwise +, α (6) where, b =.5 is a factor to increase the sample size, is a size of the ecision variable, is the largest integer smaller than : S = + m + m µ ˆ( ) µ ˆ( ik) ik k= k= m (+ )m is the mean square treatments from Analysis of Variance (ANOVA), σ is the variance of white noise, ξ in Equation an χ,α is an α upper percentile of the chisquare istribution with egree of freeom... New Variants of the Neler-Mea Simple Algorithm We are motivate by several general-purpose optimization algorithms for eterministic problems that are base on a neighborhoo search; for eample, the very large scale neighborhoo search (Pichitlamken an Nelson, 003), neighborhoo search base on tabu search an complete local search with memory for solving the uncapacitate facility location problem (Pichitlamken et al., 006). At each iteration, the search iteratively moves from the current solution to one of its neighbors which is better than itself an any other solutions in the neighbor-hoo. Neighborhoo search strategy an statistical selection of the best are use in OvS in Tomick et al. (995) followe by a framework for OvS in More et al. (98). First, we efine an alreay-visite solution as that is not too far from the one alreay seen v, v = ma v = e, where, i i v i { } is the uniform norm, = ma,, K,, an the neighbor istance 0 e ε where ε =0-4, similar to the terminating criterion of the fminsearch. We select the neighbor v which provies the minimum e v as istinguishable from. For an eample of -imensional problem, suppose = (.0008,.500), v = (.0008,.500) an v = (.00,.500), v 3 = (.0005,.6000) then e = ma { , }= 9 0-5, e = ma ( ,.500 ) = 7 0-5, e 3 = ma( , ) = 0.. Since e 3 is greater than ε, v 3 oes not belong to N() an we select v to represent because e ε an it has the smallest uniformnorm istance from. We efine the neighborhoo of solution = [,,, ] as N() consisting of all alreay-seen solutions which lie insie the region of: [ ε,, K, ],[ +ε,, K, ],[, ε, K, ], [, +ε, K, ], K,[,, K, +ε] (7) where, ε is the user-specifie maimum neighborhoo istance. We eclue any neighbors that lie outsie the feasible space Θ, or ones which are further than ε from any given. The aim of using past information is to save simulation effort by avoiing sampling at every encounter. In our implementation of the NM search, we compare the sample averages of all caniates an select the one with the smallest average as the winner. We propose two NM-base algorithms with memory as follows.... The Neler-Mea Selection with Memory (NMSM) Simulation outputs that have been obtaine for revisite solutions an their caniate solutions are kept in a atabase an they replace new sampling. Nevertheless, for alreay-seen solutions, NMSM as one simulation output every time it is encountere so as to protect the search from unusually goo or ba history. 467

6 Noocharin Tippayawannakorn an Juta Pichitlamken / Journal of Computer Science 9 (4): , 03 Fig. 4. NMSMN algorithm Let V be a collection of visite solutions uring the th iteration where V 0 = φ. If V then V = V -. The NM is change as follows: before generating new simulation outputs G(,ξ i ) for { 0,,,,,, E, C }, we check whether they have been visite. If V, we use their past simulation outputs an generate only one new observation. The historical an one aitional observations are use to calculate a new µ ˆ() of a revisite solution. Otherwise, the NMSM generates new m observations for calculating new µ ˆ() then registers into the visite set.... The Neler-Mea Selection with Memory an using Neighborhoo (NMSMN) The NMSMN is the NMSM integrate with a neighborhoo. It constructs a neighborhoo for every verte of the simple an estimates their obective function values. The best solution in the neighborhoo replaces the original verte. The avantages of the NMSMN are that they utilize past information of previous encounters an it also augments the search area to their neighborhoo. Let N( ) be a neighbor set of with ε = 0.0 in Equation 7. Thus V is a collection of visite solutions an their neighborhoo uring the th iteration where V 0 = φ. If V, V = V - N( ). The NM are change as follows: For { 0,,,,,, E, C }, before generating new simulation outputs, we check whether they are alreay visite solutions by the algorithm in Fig The Aaptive Neler-Mea Selection with Memory (ANSM) The ANSM is NMSM, but when search reaches the step of upating the simple, the µ ˆ ( ), µ ˆ ( ), K, µ ˆ ( + ) are estimate by the NMSM where Step 4 is moifie as follows: Step 4: e-orer the Simple Points In ascening orers then compare all upate epecte obective functions of simple points for etermining the m + by Equation 6. Let = The Aaptive Neler-Mea Selection with Memory an using Neighborhoo (ANSMN) This moification applies ANSM, to the NMSMN. It combines the avantage of utilizing memory of revisite solution an their neighbors an efficiently spening resources to ensure further progress approaching to minimum obective function value..3. Numerical Eperiments In section.3., we escribe a set of test functions an their starting solutions. Section.3. eplains the main figures of merit that we use to evaluate an compare the performance of many moifications of the NM. Section.3.3 iscusses the empirical test setup..3.. Test Functions We test the unconstraine optimization algorithm on a set of si eterministic test functions; that is: G(, ξ ) = g() +ξ, (8) as ξ is an unbiase ranom element with E [ξ ] = 0 an V[ ξ ] =σ an g() is eterministic test problems. These test functions are imensional ( = ). Our stanar eviations, σ, are 0.75,.00 an.5 times g(*) as efine in Equation 8. Common ranom numbers are use. Some of the selecte functions have appeare in previous stuies. For eample, test functions -5 are classical test functions prouce by More et al. (98). They were also use in Humphrey an Wilson (000) an Barton an Ivey (996) for optimization of noisy responses. Test function 6 is aapte from Neermeier et al. (000). Each of these eterministic test functions has a unique optimum Test Function : Paraboloi Function The paraboloi function is efine as: 468

7 Noocharin Tippayawannakorn an Juta Pichitlamken / Journal of Computer Science 9 (4): , 03 g() = + The starting point is given by = [,,,]. The optimal of function value g* = is achieve at point * = [0,..,0]. Figure 5 epicts the polynomial function for case =. This function is concave, symmetric an having only one minimum point. It is easy to optimize if no noise eists. However, when noise is present, optimization is ifficult Test Function : Variably Dimensione Function The variably imensione function is given by: i + g() = [f i()] + where f i () = i - for i =,,, + f () = ( ) an = The starting point is given by = = f + () = ( ). [,,, ], where =-(/), =,,,. The optimal function value g* = is achieve at point * = [,,]. Figure 6 epicts the variably imensione function for =. The search area is U-curve, which is a crosse flat area. There are numerous local minima in the region of flat area but only one unique global minima eist Test Function 3: Trigonometric Function The trigonometric function is efine as: g() = [f i()] + where for,...,, f i() = cos( ) + i[ cos( i -)] -sin( i -). The starting point is = [/,,/]. The optimal of function value g* = is achieve at point * = [+πk,..,πk ] where k = 0±, ±, for =,..,. Figure 7 illustrates the trigonometric function for =. This function is a sine curve an multi-moal minima. = Test Function 4: Etene osenbrock The etene osenbrock function is efine as: g() = [f i()] + where, for,..,/, f i () = 0(i i ) an f i () = (- i- ). The starting point is = [-.,,, -.,]. The optimal of function value g* = occurs at * = [,,]. Figure 8 epicts the etene osenbrock function for the case of =. This function is a non-conve function. The global minimum is insie a long, narrow, parabolic shape flat valley. To fin the valley is trivial. However, it is ifficult to converge to the global minimum Test Function 5: Brown s Almost-Linear Function where The Brown s almost-linear function is given by: i i = g() = [f i()] + f () = + (+ ) for i =,...,- an f () =. The solution = [/,,/] is use = as the starting point. The optimal function value g* = is achieve at the point * = [λ,..,λ,λ - ] where λ satisfies λ -(+) λ - + = 0. Humphrey an Wilson (000) compute the value of λ is 0.5 for =. Figure 9 illustrates the Brown s almost-linear function for =. The function is not linearly separable an has the basic form of a nonlinear least squares problem Test Function 6: Symmetrical Gaussian Function The symmetrical Gaussian function is efine as: g() = ep [f i()] 5000 i = where, f i () =00- i for i =,..,. The starting point is = [70,,70]. The optimal of function value g* = is achieve at the point * = [00,..,00]. Figure 0 epicts the symmetrical Gaussian function for =. If any starting point is in area of blene curve, it converges to a unique global minimum point. On the other han, if any staring point is in flat area, it is ifficult to reach the minimum point..3.. Search Performance Measures When the search terminates, optimal solutions can be estimate in at least 3 ways: The solution on-han, the most frequently visite solution, or the solution with the best cumulative averages (Banks, 998; Anraottir, 999). Our preliminary eperiments fin that the solutions on han outperform other estimates for optimal solutions. Motivate by Humphrey an Wilson (000), we evaluate the search performance via the average of the following performance measures over many replications: 469

8 Noocharin Tippayawannakorn an Juta Pichitlamken / Journal of Computer Science 9 (4): , 03 Fig. 5. Paraboloi function for = Fig. 6. Variably imensione function for = 470

9 Noocharin Tippayawannakorn an Juta Pichitlamken / Journal of Computer Science 9 (4): , 03 Fig. 7. Trigonometric function for = Fig. 8. Etene rosenbrock function for = 47

10 Noocharin Tippayawannakorn an Juta Pichitlamken / Journal of Computer Science 9 (4): , 03 Fig. 9. Brown s almost-linear function for = Fig. 0. Symmetrical gaussian function for = 47

11 Noocharin Tippayawannakorn an Juta Pichitlamken / Journal of Computer Science 9 (4): , Logarithm of the Total Number of Simulation Outputs To measure the computation work performe by a simulation optimization proceure, we compute: L ln(total number of simulation outputs) It provies at best a rough inication of the total computational work require by a simulation proceure Deviation of the Best Estimate Optimal Function Value from the True Optimal Value For a measure of accuracy of the best result elivere by a simulation optimization proceure, we consier Equation 9: D = µ ˆ µ * * ( ) ( ) * µ ( ) This measure cannot be employe for Test Function since each coorinate of the true optimum for the paraboloi function is equal to zero Empirical Test Setup Our implementations are run on MATLAB by moifying fminsearch function. The NM coefficients are as follow: α =, γ =, β = 0.5 an τ = 0.5. The initial step size, s i as shown in Equation 3, is 0 4. Minimum eviation ε an ε ˆµ are 0 4. Maimum buget consumption N µ ˆ an N search are 0 5. To estimate the obective function, the sample size m is 6. Maimum neighborhoo istance ε is 0.0. Three level of stanar eviation of ranom noise ξ is {0.75 g (*),.00g (*),.5 g (*)}. The factor of increasing simulation size b in Tomick (995) is.5. We perform 0 macroreplications (i.e., eperiments) for each test problem on 0 search algorithms as follows: (9) NM-The original Neler-Mea Simple S9-The Barton an Ivey stochastic moification ANS9-Aaptive Neler-Mea moification NMSM-The Neler-Mea selection with memory NMSM+S9-The NMSM with S9 NMSMN-The NMSM with using neighborhoo NMSMN+S9-The NMSMN with S9 ANSM-Aaptive Neler-Mea selection with memory ANSM+S9-ANSM with S9 ANSMMN-ANSM with using neighborhoo ANSMMN+S9-ANSMMN with S9 3. ESULTS Table shows the buget consumption of each algorithm until computation buget is ehauste or until the search is unable to get any improvements. Table contains the average eviation in estimating the obective function values as efine in (9). As epecte, when the egree of ranomness increases, a given test problem becomes more ifficult. Most algorithms fare worse an their estimate optimal solutions are further away from the optimal solutions because all algorithms give smaller D at lower * ranom noise. That means µ ( ˆ ) is not far from µ (*). On the contrary, Test Function 4 is the most ifficult to optimize even when the level of ranomness is low. Moreover, using the aaptive strategy with memory gives the smallest D although the aaptive feature requires more computational effort. For eample, for Test Function an at all ranom noise levels, ANS9 consumes more computational resource than S9, but it rewars with better estimate optimal solutions, i.e., smaller D. Similar results can be observe between ANSM+S9 an ANSMN+S9 in comparison with NMSM an NMSMN, respectively. Consiering D, no algorithms ecisively wins at all noise levels, but almost wining algorithms involve the aaptive metho. Similarly, when we consier L, no algorithms outperforms completely at all noise levels. For better comparison, we compare relative ratio of both, L an D, between pairs of algorithms. The ata is ivie into 3 sets. Firstly, comparing between the aaptive an non-aaptive methos, e.g., ANS9/S9, ANSM+S9/NMSM +S9 an ANSMN+S9/NMSMN, for most test functions an almost all noise levels, the aaptive methos spens 79% more computation effort than the non-aaptive methos, but they give better estimates of the optimal solutions by reucing D, for up to 50%. For eample at σ =.00 g(*) for Test Function, all aaptive algorithms, ANS9, ANSM+S9 an ANSMN+ S9, yiel the estimates of the optimal solutions closer to the true optimum than the non-aaptive algorithms, S9, NMSM+S9 an NMSMN+S9 respectively; D, is reuce approimately by 30% although they spen more computational effort. Ecept for σ = 0.75g(*) of Test Function 3, the aaptive algorithms with using memory (e.g., ANSM+S9 an ANSMN+S9) are slightly less than the corresponing the non-aaptive algorithms (i.e., NMSM+S9 an NMSMN+S9, respectively); an for Test Function 4 with σ = {0.75 g(*),.00g (*)}, the aaptive algorithms with using memory (e.g., ANSM+S9 an ANSMN+S9) the true optimum, i.e., D increases. For Test Functions, 3, 5 an 6, D increases when the stanar eviation of ranom noise goes up, across all algorithms. For Test Function, almost every algorithms also ehibit this pattern ecept ANSMS9. Moreover, the results show that for Test Functions -3 an 5-6, it is not ifficult to fin o not give better improvement than their counterparts. 473

12 Noocharin Tippayawannakorn an Juta Pichitlamken / Journal of Computer Science 9 (4): , 03 Table. Logarithm of computational effort (L) Test function Factor σ Algorithm NM S NMSM NMSM+S NMSMN NMSMN+S ANS ANSM+S ANSMN+S NM S NMSM NMSM+S NMSMN NMSMN+S ANS ANSM+S ANSMN+S NM S NMSM NMSM+S NMSMN NMSMN+S ANS ANSM+S ANSMN+S Seconly, comparing between memory an nonmemory methos, e.g., NMSM/NM, NMSM+S9/S9 an ANSM+S9/ANS9, the results show that memory eployment saves on resource consumption, spening on average 77% less than non-memory counterparts for all test functions an noise levels, e.g., NMSM+S9 an ANSM+S9 consume less resources for about 30%, than S9 an ANS9, respectively. This is because solutions that are less than ε apart are classifie to be the same. If the search revisits the alreay seen parts of the solution space, it may use the sampling ata from the previous visits, instea of resampling anew. For most test functions an noise levels, on average, utilizing memory reuces eviation D by up to 80% of nonutilizing memory, asie from Test Function 4 that involves the aaptive methos. For eample, Test Function 5 for all noise levels, NMSM, NMSM+S9 an ANSMN+S9 give better optimal solutions by up to 6% to 49% of NM, S9 an ANS9, respectively. The rest of comparing is between neighbor an nonneighbor methos, e.g., NMSMN/NMSM, NMSMN+S9 /NMSM+S9 an ANSMN+S9/ANSM+S9. egaring the memory-utilizing property, for all noise levels an almost all test functions, ecept Test Function 4, the nonaaptive methos which incorporate the neighborstructure neither saves computation effort nor improves estimates of the optimal solution, e.g., on Test Functions an 6, NMSMN an NMSMN+ S9 give inistinguishable results on D an L from NMSM an NMSM+S9, respectively. In other wors, using neighbor-structure is greey an misle to a non-optimum compare to the algorithms without neighbor-structure. On the other han, ANSMN+S9 provies a better optimal solution an spens less computation effort than ANSM+S9 by 3% an 9%, respectively. These results show that using neighbor-structure on aaptive algorithms provie improve estimate optimal solutions. The search with goo performance when there is no limitation on computational resource is AMSMN+ S9 because it gives the least D at all noise levels for most test functions, ecept Test Function 4. If computational computational resource is limite, NMSM+S9 performs better. 474

13 Noocharin Tippayawannakorn an Juta Pichitlamken / Journal of Computer Science 9 (4): , 03 Table. Average eviation of function value of algorithm from true function value (D) Test function Factor σ Algorithm NM S NMSM NMSM+S NMSMN NMSMN+S ANS ANSM+S ANSMN+S NM S NMSM NMSM+S NMSMN NMSMN+S ANS ANSM+S ANSMN+S NM S NMSM NMSM+S NMSMN NMSMN+S ANS ANSM+S ANSMN+S DISCUSSION We show that the Neler-Mea algorithm which is esigne for eterministic optimization can be moifie to accommoate stochastic outputs. Using past information generally ecreases computational effort an not eoparizes the performance significantly. Although all aaptive-with-memory algorithms spens more computational resources, they give better optimal solutions comparing with their corresponing non-aaptive ones. Eploiting neighborhoo an utilizing memory are helpful for aaptive algorithms, but not for non-aaptive algorithms. The search performance is improve on either ANSM+S9 an ANSMN+S9 for unconstraine resource consumption, or NMSM an NMSMN for economical resource consumption. 5. CONCLUSION Utilizing past information in continouse optimization saves computational resource. To improve an estimate optimal solutions without limitation of computational effort, uses past information corporates on aaptive metho. For our future work, we eten our algorithm to iscrete search space an apply simulation to ecision making such as queuing or inventory problem. 6. ACKNOWLEDGEMENT This stuy was supporte by the Inustrial Engineering Department, Faculty of Engineering an the Grauate School of Kasetsart University, Thailan. The authors woul like to thank Dr. Walailuck Chavanasporn, Mathematics epartment, King Mongkut University of Technology North Bangkok, for her eitorial comments. Some parts of this work were presente previously at the Asia Simulation Conference 0 in Seoul, Korea (Tippayawannakorn an Pichitlamken, 0). 7. EFEENCES Alon,., S.W. Feigelson, E. Manevich, D.M. ose an J. Schmitz et al., 005. Alpha4beta-epenent ahesion strengthening uner mechanical strain is regulate by paillin association with the alpha4- cytoplasmic omain. J. Cell. Biol., 7: PMID:

14 Noocharin Tippayawannakorn an Juta Pichitlamken / Journal of Computer Science 9 (4): , 03 Alrefaei, M.H. an S. Anraottir, 005. Discrete stochastic optimization using variants of the stochastic ruler metho. Naval es. Logistics, 5: DOI: 0.00/nav.0080 Anraottir, S., 999. Accelerating the convergence of ranom search methos for iscrete stochastic optimization. Assoc. Comput. Mach. Trans. Moel. Comput. Simulat., 9: DOI: 0.45/ Anraottir, S., 006. Chapter 0 An Overview of Simulation Optimization via anom Search. In: Hanbooks in Operations esearch an Management Science, Henerson, G.S. an B.L. Nelson (Es.), pp: Banks, J., 998. Chapter 9 simulation optimization. Principles, Methool. Av. Applic. Practice, Barton,.. an J.S. Ivey, 996. Neler-Mea simple moifications for simulation optimization. Manage. Sci., 4: DOI: 0.87/mnsc Beyer, H.G. an H.P. Schwefel. 00. Evolution strategies-a comprehensive introuction. J. Natural Comp., : 3-5. DOI: 0.03/A: Carson, Y. an A. Maria, 997. Simulation optimization: Methos an applications. Proceeings of the 9th Conference on Winter Simulation, Dec. 07-0, IEEE Computer Society Washington, DC, USA., pp: 8-6. DOI: 0.45/ Dennis, J.E. an D.J. Woos, 987. Optimization on Microcomputers: The Neler-Mea Simple Algorithm. In: Society for Inustrial an Applie Mathematics, Wouk, A. (E.), pp: 6-. Henerson, S.G. an B.L. Nelson, 006. Simulation. st En., Springer, Dorrecht, pp: 693. Hollan, J., 000. Builing blocks, cohort genetic algorithms, an hyperplane-efine functions. Evolut. Comput., 8: DOI: 0.6/ Humphrey, D.G. an J.. Wilson, 000. A revise simple search proceure for stochastic simulation response surface optimization. INFOMS J. Comput. Fall, : DOI: 0.87/ioc More, J.J., B.S. Garbow an K.E. Hillstrom, 98. Testing unconstraine optimization software. ACM Trans. Mathemat. Software, 7: 7-4. DOI: 0.45/ Neermeier, H.G., G.J. Oortmarssen, N. Piersma,. Dekker an J.D.F. Habbema, 000. Aaptive etensions of the neler an mea simple metho for optimization of stochastic simulation moels. Econometric Institute eport, Erasmus University otteram, Econometric Institute. Neler, J.A. an. Mea, 965. A simple metho for function minimization. Comput. J., 7: DOI: 0.093/comnl/ Olafsson, S. an J. Kim, 00. Simulation optimization: Simulation optimization. Proceeings of the 34th Conference on Winter Simulation: Eploring New Frontiers, (WSC 0), ACM Press, pp: Pichitlamken, J. an B.L. Nelson, 003. A combine proceure for optimization via simulation. ACM Trans. Moel. Comput. Simulat., 3: DOI: 0.45/ Pichitlamken, J., B.L. Nelson an L.J. Hong, 006. A sequential proceure for neighborhoo selection-ofthe-best in optimization via simulation. Eur. J. Operat. es., 73: DOI: 0.06/.eor Press, W.H., S.A. Teukolsky, W.T., Vetterling an B.P. Flannery, 007. Section 0.. Simulate Annealing Methos In Numerical ecipes: The Art of Scientific Computing. 3r En., Cambrige University Press, New York, ISBN-0: pp: Shi, L. an S. Olafsson, 009. Neste Partitions Optimization. In: Encyclopeia of Optimization, Christooulos, A.F. an P.M., Panos, (Es.), pp: Spenley, W., G.. Het an F.. Himsworth, 96. Sequential application of simple esigns in optimisation an evolutionary operation. Technometrics, 4: Swisher, J.., P.D. Hyen, S.H. Jacobson an L.W. Schruben, 004. A survey of recent avances in iscrete input parameter iscrete-event simulation optimization. IIE Trans., 36: DOI: 0.080/ Tippayawannakorn, N. an J. Pichitlamken, 0 Neler-mea metho with local selection using neighbor hoo an memory for optimization via simulation. Proceeings of the Avance Methos, Techniques an Applications in Moeling an Simulation, (AMTAMS ), pp: Tomick, J.J., 995. On convergence of the neler-mea simple algorithm for unconstraine stochastic optimization. Pennsylvania State University. Tomick, J.J., S.F. Arnol an.. Barton, 995. Sample size selection for improve Neler-Mea performance. Proceeings of Winter Simulation Conference, Dec. 3-6, IEEE Xplore Press, Arlington, VA., pp: DOI: 0.09/WSC