A Revised Simplex Search Procedure for Stochastic Simulation Response Surface Optimization

Transcription

1 272 INFORMS Journal on Computing $05.00 Vol. 12, No. 4, Fall INFORMS A Revise Simplex Search Proceure for Stochastic Simulation Response Surface Optimization DAVID G. HUMPHREY Nortel Networks, Operations Research Department, Research Triangle Park, NC 27713, humphre@nortelnetworks.com JAMES R. WILSON Department of Inustrial Engineering, North Carolina State University, Raleigh, NC 27695, jwilson@eos.ncsu.eu, Web: (Receive: September 1997; revise: April 2000; accepte: May 2000) We evelop a variant of the Neler-Mea (NM) simplex search proceure for stochastic simulation optimization that is esigne to avoi many of the weaknesses encumbering similar irect-search methos in particular, excessive sensitivity to starting values, premature termination at a local optimum, lack of robustness against noisy responses, an computational inefficiency. The Revise Simplex Search (RSS) proceure consists of a three-phase application of the NM metho in which: (a) the ening values for one phase become the starting values for the next phase; (b) the step size for the initial simplex (respectively, the shrink coefficient) ecreases geometrically (respectively, increases linearly) over successive phases; an (c) the final estimate optimum is the best of the ening values for the three phases. To compare RSS versus NM an proceure RS S9 ue to Barton an Ivey, we summarize a simulation stuy base on four selecte performance measures compute for six test problems that inclue aitive white-noise error, with three levels of problem imensionality an noise variability use in each problem. In the selecte test problems, RSS yiele significantly more accurate estimates of the optimum than NM or RS S9, an both RSS an RS S9 require roughly four times as many function evaluations as NM. S tochastic simulation optimization can be viewe as fining a combination of (eterministic) input parameters (factor levels or esign variables) that yiels the optimal expecte value of a user-specifie (ranom) output response generate by the simulation moel. Let the -imensional esign point x [x 1,...,x ] represent the factor-level combination specifying the policy governing operation of the simulation moel uner the associate scenario (or alternative system configuration). Thus the components of x together with a (possibly infinite) stream of ranom numbers constitute the full set of inputs to the simulation moel; an we let Y(x) [Y 1 (x),...,y p (x)] enote the p-imensional ranom vector of output responses generate on one run of the simulation at esign point x. With respect to optimization of system performance, we assume that one of the components of Y(x), say the initial element Y 1 (x), is the primary response of interest; an we let (x) E[Y 1 (x)] enote the response surface function to be optimize. We efine the region of interest for the optimization proceure, x R x efines feasible system operating conitions, (1) where R enotes -imensional Eucliean space. Assuming that the primary performance measure is expecte total cost an thus shoul be minimize, we seek to etermine * min x an x* arg min x, x the minimum cost an the optimal esign point efining the minimum-cost system configuration. In this article we formulate, implement, an evaluate a stochastic simulation optimization proceure that incorporates many esirable properties of the well-known Neler- Mea (NM) simplex search proceure (Neler an Mea 1965) while avoiing some of the critical weaknesses of this proceure in particular, excessive sensitivity to starting values, premature termination at a local optimum, lack of robustness against noisy responses, an computational inefficiency (Parkinson an Hutchinson 1972, Barton an Ivey 1996). In Section 2 we give a formal algorithmic statement of the Revise Simplex Search (RSS) proceure. In Section 3 we summarize the main figures of merit that we use to evaluate an compare proceures for stochastic simulation optimization, an we analyze the significant factors that affect the performance of simplex-search-type proceures. Section 4 contains a summary of a comprehensive Monte Carlo comparison of proceure RSS versus the classical proceure NM as well as proceure RS S9, a variant of NM that was evelope by Barton an Ivey (1996). Finally, in Section 4 we recapitulate the main finings of this work, an we present recommenations for future research. Although this paper is base on Humphrey (1997), some of our results were also presente in Humphrey an Wilson (1998). 1. Revise Simplex Search (RSS) Proceure In this section we escribe the operation of the RSS proceure, an we introuce the symbolism require to specify precisely the steps of the proceure. RSS operates in three x Subject classifications: Simulation, esign of experiments: irect-search optimization techniques Other key wors: Simplex search proceures.

2 273 A Revise Simplex Search Proceure for Stochastic Simulation Response Surface Optimization phases inexe by the phase counter, an within each phase, each aitional stage q involves generating a new simplex from the current simplex via the operations of reflection, expansion, contraction, or shrinkage (as escribe below) until the termination criterion for the current phase (also escribe below) is satisfie. In the current phase an stage q of proceure RSS, we let x i [x i,1,...,x i, ] enote the ith vertex of the latest simplex generate by RSS for i 1,..., 1, q 0,1,...,an 1, 2, 3. (Although x i (q, ) might be a more complete notation for the ith vertex of the qth simplex generate in phase, we suppress the exponent (q, ) for simplicity since no confusion can result from this usage.) In the initial phase of operation of RSS, the user provies the initial vertex x 1 [x 1,1,...,x 1, ] that efines the starting point for the overall search proceure. In terms of the step size parameter, the initial step size 1 for the first phase is 1 max{1, x 1,j j 1,...,}. The remaining vertices of the initial simplex are given by x i 1 x 1 1 e i for i 1,...,, where e i is the -imensional unit vector with one in the ith component an zeros elsewhere. In the initial phase of operation of RSS, the coefficient for the shrinkage operation has the value originally recommene by Neler an Mea (1965). With respect to the current (latest) simplex generate in phase of the operation of proceure RSS, we let ˆ(x i ) enote the simulation-base estimate of the objective function value (x i ) at vertex x i for i 1,..., 1, an we let x max enote the vertex of the current simplex yieling ˆ max ˆ x max max ˆ x i 1 i 1. (2) In similar fashion, we efine x min an ˆmin ˆmin (x min ), an we let x ntw enote the vertex of the current simplex yieling ˆntw ˆ(x ntw ), the next-to-worst (secon largest) of the response surface estimates observe at the vertices of the current simplex. When it is not important to emphasize the vertex upon which quantities like ˆ(x max ) epen, we will use the alternative notation ˆmax for simplicity. For q 0, 1,...,theqth stage within phase of proceure RSS begins by computing the centroi of all the vertices in the current simplex except x max, x cen 1 1 i x max x. (3) i 1 Phase of proceure RSS ens when RSS generates a new simplex that is sufficiently small to satisfy the termination criterion. Then the phase counter is incremente by one an proceure RSS is restarte, provie 3. At the beginning of phase of proceure RSS for 2 an 3, we take the initial step size to be so that the initial step size ecreases geometrically over successive phases. Similarly, we take for 2 an 3 so that the shrink coefficient increases linearly over successive phases until it reaches the value 0.9 recommene by Barton an Ivey (1996) for optimization of noisy functions. For 1, 2, 3, we let xˆ*( ) enote the final estimate of the optimal solution elivere in phase. Then we take as the final estimate optimum the best of the ening values for all three phases. A flow chart of RSS is epicte in Figure 1, an a formal statement of the algorithm is given below. The basis for the esign of proceure RSS is etaile in Section 2 below. Steps of Proceure RSS 0. Set Up Phase 1. Initialize the following: the phase counter 4 1; the iteration (stage, simplex) counter, q 4 0; the shrink coefficient use in phase 1, ; the initial step size use in phase 1, 1 4 max x 1,j j 1,...,, if x 1 0, 1, otherwise; an the other vertices of the initial simplex in phase 1, (4) x i 1 4 x 1 1 e i for i 1,...,. (5) Go to step Attempt Reflection. Form a new simplex by reflecting x max through the centroi x cen of the remaining vertices of the current simplex to obtain the reflecte point x refl 4 x cen x cen x max, where 1.0 is the reflection coefficient of Neler an Mea (1965). If ˆ min ˆ refl ˆ ntw, (6) that is, if the reflecte point x refl yiels a response no worse (no larger) than the next-to-worst vertex x ntw in the current simplex but oes not yiel a better (smaller) response than the best vertex x min, then replace the worst vertex x max in the current simplex by the reflecte point x refl, x max 4 x refl ; (7) an go to step 6 to test the termination criterion. If the conition (6) for accepting the reflection is not satisfie, then go to step Attempt Expansion. If ˆ refl ˆ min (8) so that the reflecte point x refl is better than the best vertex x min in the current simplex, then exten the search in the irection x refl x cen to yiel the expansion point x exp 4 x cen x refl x cen, where 2.0 is the expansion coefficient of Neler an Mea (1965). If ˆexp ˆmin, then accept the expansion an replace x max by x exp in the current simplex, x max 4 x exp ; an go to step 6. If ˆexp ˆmin, then reject the attempte expansion an replace x max by x refl in the current simplex, x max 4 x refl ; an go to step 6. Finally if the conition (8) for attempting expansion is not satisfie, then go to step 3.

3 274 Humphrey an Wilson Figure 1. Flow chart of proceure RSS. 3. Set Up Attempte Contraction. If ˆ refl ˆ ntw so that the reflecte point x refl yiels a worse (larger) response than the next-to-worst vertex x ntw of the current simplex, then reuce the size of the current simplex either by a contraction or a more rastic shrinkage. To set up this reuction in the size of the simplex, upate the worst vertex in the current simplex as follows: Compute the contraction point x cont 4 x cen x max x cen, where 0.5 is the contraction coefficient of Neler an Mea (1965). 4. Contract Simplex in One Direction. If ˆ cont ˆ max, if ˆ refl ˆ max, then x max 4 x refl ˆ max 4 ˆ refl. so that the contracte point x cont yiels a response no worse (no larger) than the worst vertex x max of the current simplex,

4 275 A Revise Simplex Search Proceure for Stochastic Simulation Response Surface Optimization then replace x max in the current simplex by x cont, x max 4 x cont ; an go to step 6; otherwise go to step Shrink Entire Simplex. If the contracte point x cont yiels a worse (larger) response than every vertex in the current simplex incluing x max so that the shrinkage conition ˆ cont ˆ max is satisfie, then reuce the lengths of all eges of the current simplex with common enpoint x min by the shrinkage factor, yieling a new simplex with vertices x i 4 x min x i x min for i 1,..., 1; (9) an go to step Test Termination Criterion for Current Phase. After each reflection, expansion, contraction, or shrinkage, apply the termination criterion max 1 i 1 x i x min 1 x min, if x min 0, 2, otherwise, (10) where 1 an 2 are user-specifie tolerances an the maximum is taken over all vertices in the current simplex. If the termination conition (10) is not satisfie, then increment the iteration counter q 4 q 1 an go to step 1. If the termination conition (10) is satisfie, then go to step Terminate Current Phase. Recor the termination point of the current phase xˆ* 4 x min, (11) increment the phase counter, 4 1, an go to step Test Final Termination Criterion. If 3, then compute the final estimate xˆ* of the global optimum accoring to * 4 arg min ˆ xˆ* 1, 2, 3 an xˆ* 4 xˆ* * ; finally eliver xˆ* an ˆ(xˆ*) an stop. If 3, then go to step Set Up Next Phase. Initialize the following: the iteration counter q 4 0; the first vertex of the initial simplex in the current phase, x 1 4 xˆ* 1 ; (12) the initial step size for the current phase, ; (13) the shrink coefficient for the current phase, ; (14) an the other vertices of the initial simplex in the current phase, x i 1 4 x 1 e i for i 1,...,. (15) Go to step Development of Proceure RSS In this section we formulate the principal performance measures that we use to evaluate an compare proceures for stochastic simulation optimization, an we summarize our preliminary analysis of the significant factors affecting the performance of simplex-search-type proceures. This analysis forme the basis for the esign of proceure RSS. 2.1 Formulation of Performance Measures for Simulation Optimization We use four figures of merit to evaluate stochastic simulation optimization proceures: (a) logarithm of the number of function evaluations; (b) absolute percentage eviation of the estimate optimal function value from the true optimal function value; (c) maximum over all coorinates of the absolute percentage eviation of the estimate optimum from the true optimum taken with respect to each coorinate separately; an () average over all coorinates of the absolute percentage eviation of the estimate optimum from the true optimum taken with respect to each coorinate separately Logarithm of Number of Function Evaluations To measure the computational work performe by a simulation optimization proceure, we compute the (natural) logarithm of the total number of function evaluations require by the proceure before it terminates an elivers the final estimates ˆ(xˆ*) an xˆ*: L ln total number of function evaluations require. (16) The logarithmic transformation in (16) is use to obtain approximately normal observations with a common variance to which we can apply stanar statistical techniques such as analysis of variance an multiple comparisons proceures; see Anerson an McLean (1974). Although L is wiely use in experimental comparisons of simulation optimization proceures (Barton an Ivey 1996), it shoul be recognize that in the optimization of a large-scale stochastic simulation moel, each function evaluation represents a separate simulation run, an ifferent runs may require substantially ifferent amounts of execution time to eliver the corresponing function values. Thus in general L provies at best a rough inication of the total computational work require by a simulation optimization proceure Final Function Value Provie that the optimal function value * 0, we use the absolute percentage eviation D ˆ * xˆ* * * (17) as a measure of the accuracy of the final result elivere by a simulation optimization proceure. When average over inepenent replications of each proceure applie to a given test problem, the quantity (17) provies a imensionless figure of merit that allows us to compare the perfor-

5 276 Humphrey an Wilson mance of simulation optimization proceures across ifferent test problems. All the test problems use in this work were specifically constructe to have nonzero optimal function values Coorinatewise Maximum Absolute Percentage Deviation from Global Optimum The thir performance measure is the maximum over all j (for 1 j ) of the absolute percentage eviation of xˆ* j (the jth coorinate of the estimate optimum xˆ*) from x* j (the jth coorinate of the true optimum x*), provie that each x* j 0: B max xˆ * j x* j x* j. (18) 1 j When there are multiple optima, we evaluate the right-han sie of (18) for each optimum, an we take the smallest of these quantities as the final value of B. When average over inepenent replications of each proceure applie to a given test problem, the quantity (18) provies another imensionless figure of merit that allows us to compare the performance of simulation optimization proceures across ifferent test problems. All of the test problems use in this work were specifically constructe to have optima with all coorinates having nonzero values Coorinatewise Average Absolute Percentage Deviation from Global Optimum The final performance measure is the average compute over all j (for 1 j ) of the absolute percentage eviation of xˆ* j (the jth coorinate of the estimate optimum xˆ*) from x* j (the jth coorinate of the true optimum x*), provie that each x* j 0: A 1 j 1 xˆ * j x* j x* j. (19) When there are multiple optima, we evaluate the right-han sie of (19) for each optimum an take the smallest of these quantities as the final value of A. We believe that A provies the best overall characterization of the accuracy with which a simulation optimization proceure estimates the true optimum. No single performance measure can tell the entire story about the performance of a particular search proceure, but we believe that (16) (19) provie meaningful information that can be aggregate over ifferent test problems to yiel a comprehensive basis for comparison of selecte simulation optimization proceures. 2.2 Significant Factors Affecting Performance of Simplex-Search-Type Proceures In seeking to formulate a simplex-search-type proceure that avois some of the rawbacks of the Neler-Mea proceure when it is applie to optimization of noisy response functions, we ientifie three significant factors affecting the performance of such proceures: sizing the initial simplex, restarting the search, an ajusting the shrink coefficient. Each of these factors will be iscusse briefly; a more etaile analysis of these factors is given in Humphrey (1997) an in Humphrey an Wilson (2000) Sizing the Initial Simplex Our preliminary experimentation showe that starting a simplex-search-type proceure with a larger initial simplex generally improve the performance of the proceure. The iea behin starting with a larger initial simplex is straightforwar. A smaller simplex starting far from the true optimum will have to iterate many times (mostly through reflections an expansions) in orer to move into a neighborhoo of the optimum in which the response surface is well behave. Along the way to such a neighborhoo, any errant contractions or shrinkages of the current simplex will significantly slow the proceure s progress towar the optimum. By comparison, a larger simplex starting far from the optimum can make much faster progress towar the optimum by covering more groun with each reflection or expansion; an in this situation any errant contractions or shrinkages will have a less severe effect on the proceure s progress towar the optimum. Although Parkinson an Hutchinson (1972) observe similar effects in their extensive numerical evaluation of the performance of simplex-searchtype proceures for optimization of eterministic response functions, the situation is much less clear-cut when such proceures are applie to noisy responses. Base on a preliminary simulation stuy similar to that escribe in Section 3 below, we foun that taking the initial step size parameter 4.0 in (4) appeare to yiel the best overall performance for proceure RSS Restarting the Search Our preliminary experimentation also reveale that to guar against premature termination at a false optimum, the most effective action was to step away from the current termination point, restart the search proceure with a new initial simplex, an compare the resulting alternative termination points. Parkinson an Hutchinson (1972) observe similar effects with eterministic response functions. Base on a preliminary simulation stuy similar to that escribe in Section 3 below, we foun that significantly improve performance of RSS was obtaine by restarting the proceure twice thus in effect we esigne RSS to operate in three phases an finally eliver the best solution taken over all three phases. Displays (12) (15) specify the restart step for phase of proceure RSS as it epens on the results of phase 1 for 2 an 3. Notice that on each successive phase of operation of proceure RSS, the initial step size is reuce by 50% compare to the initial step size use in the previous phase Ajusting the Shrink Coefficient Another change incorporate into proceure RSS involves the shrink coefficient. Every time a shrinkage is performe,

6 277 A Revise Simplex Search Proceure for Stochastic Simulation Response Surface Optimization each ege of the simplex is rescale by the factor ; an since 0 1, the overall size of the simplex is substantially reuce. Base on a preliminary simulation stuy similar to that escribe in Section 3 below, we obtaine better performance in the first phase of operation of proceure RSS by using the shrink coefficient value that was originally recommene by Neler an Mea (1965); but in the later phases of operation of RSS, we obtaine greater protection against premature termination with the larger shrink coefficient values an In proceure RS S9, Barton an Ivey (1996) fixe the shrink coefficient at the value 0.9 to reuce the likelihoo of premature termination; moreover, after each shrinkage operation (9) is performe, proceure RS S9 requires resampling the response at the anchor point x min an then reranking an relabeling the vertices of the new simplex (that is, x max, x ntw, x min, etc.) before attempting the next reflection operation. The motivation for the shrink-coefficient assignments 1 0.5, 2 0.7, use in proceure RSS is that uring the earlier, hill-climbing phases of the operation of RSS, the current simplex is usually far from the optimum, an the likelihoo of an errant shrinkage shoul be relatively low since the topology of the response surface often has a larger effect on the behavior of the search proceure than the noise in the sample responses. If uring its first phase of operation proceure RSS etects what appears to be nonconvex behavior in the responses observe at the vertices of the current simplex so that a shrinkage operation shoul be performe, then the smaller value of 1 shoul enable the shrink operation to be more effective in positioning the simplex in a locally convex neighborhoo of the optimum. If several errant shrinkages are performe uring the earlier phases of the search an the simplex becomes too small to make effective progress towar the optimum, then proceure RSS attempts to compensate for this through the formation of a new initial simplex at the start of each of the later phases of the search. Moreover in the later phases of the search, the simplex is usually in a subregion of the region of interest (1) where the response surface is relatively flat so that the noise in the sample responses typically has a larger effect on the behavior of the search proceure; consequently the likelihoo of performing an errant shrinkage shoul be higher than it was in the earlier phases of the search. Because shrinkages rastically reuce the size of the simplex, we attempt to protect against errant shrinkages (an consequently premature termination) in the later phases of the search by increasing the value of the shrink coefficient for successive values of the phase counter. 3. Experimental Performance Evaluation In this section we escribe the problems use in testing proceure RSS an comparing its performance with that of proceures NM an RS S9. We also provie a summary an analysis of the experimental results. 3.1 Description of Test Problems We selecte six problems to serve as a test-be for comparing the performance of proceure RSS with that of proceures NM an RS S9. Similar problems were use in the experimental performance evaluation of Parkinson an Hutchinson (1972) for optimization of eterministic response functions an in the stuy of Baron an Ivey (1996) for optimization of noisy response functions. To mimic the behavior of responses generate by a stochastic simulation moel, we took each sample response to have the form ˆ(x) (x), where ( ) is one of the test functions escribe below an the aitive white-noise error term is ranomly sample from a normal istribution with a mean of zero an a stanar eviation that is systematically varie to examine the effect of increasing levels of noise variability on the selecte simplex-search-type proceures. For all three proceures, we use the common termination criterion (10) with an to provie an equitable basis for comparing the performance of these proceures. For each test problem escribe below, we specify the function to be minimize, the starting point use by each search proceure, the optimal function value, an the point(s) corresponing to the optimal function value. A complete escription of all test problems is given in Humphrey (1997) Test Problem 1: Variably Dimensione Function The variably imensione function is efine as 2 x f i x 2 1, i 1 where f i x x i 1 for i 1,...,, f 1 x j x j 1, j 1 an f 2 x j x j 1 2. j 1 The initial point is given by x 1 [x 1,1, x 1,2,...,x 1, ], where x 1,j 1 (j/), j 1,...,. The optimal function value of * 1 is achieve at the point x* [1,..., 1]. Figure 2 epicts the variably imensione function for the case Test Problem 2: Trigonometric Function The trigonometric function is efine as x f i x 2 1, i 1 where f i x cos x j 1 i 1 cos x i 1 j 1 sin x i 1, i 1,...,.

7 278 Humphrey an Wilson Figure 2. for 2. Test problem 1: variably imensione function Figure 4. for 2. Test problem 3: extene Rosenbrock function where 2 f 2i 1 x 10 x 2i x 2i 1 f 2i x 1 x 2i 1 for i 1,...,/2. The initial point is given by x 1 [ 1.2, 1,..., 1.2, 1], an the optimal value of * 1 occurs at x* [1,...,1].Figure 4 epicts the extene Rosenbrock function for the case Test Problem 4: Extene Powell Singular Function Figure Trigonometric function (test problem 2) for The extene Powell singular function is efine as x f i x 2 1, i 1 We use the starting point x 1 [1/,...,1/]. The optimal value of * 1 is achieve at every point in the lattice of points given by x* k1k 2 k 1 2 k 1,...,1 2 k, where k j 0, 1, 2,..., for j 1,...,. (20) Figure 3 epicts the trigonometric function for 2. When a given search proceure terminates on this test problem, we etermine which of the optimal points specifie by (20) is closest in Eucliean istance to the final estimate xˆ*, an we use that optimum for calculating performance measures A an B as specifie in isplays (19) an (18), respectively Test Problem 3: Extene Rosenbrock Function The extene Rosenbrock function is efine as x f i x 2 1, i 1 where f 4i 3 x f 4i 2 x x 4i 3 10x 4i x 4i 1 x 4i f 4i 1 x x 4i 2 2x 4i f 4i x 10 x 4i 3 x 4i 2 for i 1,...,/4, so that the imensionality of the input vector x must be a multiple of 4. The starting point is x 1 [3, 1,0,1,...,3, 1, 0, 1], an the optimal value of * 1 is achieve at the point x* [1,...,1] Test Problem 5: Brown s Almost-Linear Function Brown s almost-linear function is efine as x f i x 2 1, i 1

8 279 A Revise Simplex Search Proceure for Stochastic Simulation Response Surface Optimization Table I. Optimal Points of Brown s Almost-Linear Function (Test Problem 5) Table II. Coefficients {c i 1 i } of Corana Function (Test Problem 6) Optimal Points i c i i c i i c i i c i 2 x* 1 [1, 1], x* 2 [1/2, 2] 10 x* 1 [1,...,1], x* 2 [ ,..., , ] 18 x* 1 [1,...,1], x* 2 [ ,..., , ] Test Problem 6: Corana Function In some respects, the Corana function (Corana et al. 1987) represents the most ifficult test problem use in our experimental performance evaluation. We take x R a i x i a i for i 1,...,, where a i 10 4 for i 1,...,, an we efine a set of pockets within as follows: k1,...,k x k i s i t i x i 1 k i s i t i Figure 5. Test problem 5: Brown s almost-linear function for 2. where f i x x i x j 1 for i 1,..., 1, j 1 an f x j x 1. j 1 The initial point is given by x 1 [1/2,..., 1/2], an the optimal function value is * 1, which occurs at two ifferent points for all values of. Moré et al. (1981) show that (x*) 1atx* (,...,, 1 ), where satisfies (21) Notice that 1 is a solution of (21) for every value of.we compute the following aitional real solutions of (21): 1 for 2; for 10; an for Table I specifies the optimal points x* 1, x* 2 for the values of use in our experimental performance evaluation. Figure 5 epicts Brown s almost linear function for the case 2. When a given search proceure terminates on this test problem, we etermine which of the two optimal points is closer in Eucliean istance an we use that optimal point for calculating performance measures A an B. for i 1,...,, where k 1,...,k are integers, the vectors t (t 1,...,t ) an s (s 1,...,s ) are compose of positive real numbers, an t i s i /2 for i 1,...,. From this we efine to be the family of open, isjoint, rectangular subomains of R within efine as follows: k 1 k1,...,k 0,...,0. k The Corana function is efine by 1 x i 1 c i x i 1 2, 1 i 1 c i z i 1 2, for x, for x, where z i k is i t i, if k i 0, 0, if k i 0, k i s i t i, if k i for i 1,...,. 0, The initial point is given by x 1 [2,...,2],antheoptimal function value * 1 is achieve at the point x* [1,...,1]. We use s i 0.2, t i 0.05 for i 1,...,, an 0.15 as is usually one in applications of the Corana function. Table II specifies the coefficients {c i 1 i } use in our experimentation with the Corana function. Figure 6 epicts the Corana function for the case that 2 an c 1 c 2 1. Note that a value of c (as is use in our analysis, but not shown in Figure 6) causes the response surface to be extremely steep in the secon coorinate irection. 3.2 Summary of Experimental Results In the experimental performance evaluation, we sought to inclue low, meium, an high levels of imensionality an noise variability. For the low level of imensionality, 2 is the natural choice. Since the literature inicates that simplex-search type proceures ten to perform well

9 280 Humphrey an Wilson Figure Plot of Corana function (test problem 6) for for 10 (Neler an Mea 1965, Barton an Ivey 1996), we took 10 as the meium level of imensionality for all test problems except problem 4, where we took 8to satisfy the requirements of the extene Powell singular function. Base on our previous computational experience with the Neler-Mea proceure in a wie variety of statistical-estimation problems involving minimization of functions of up to 20 inepenent variables (Wagner an Wilson 1996, Kuhl an Wilson 2000), we took 18 as the high level of imensionality for all test problems except problem 4, where we took 16. To gauge the effect of increasing levels of noise variability on the selecte simplex-search-type proceures, we took each sample response to have the form ˆ(x) (x), where ( ) is one of the selecte test functions escribe in Section 3.1 an is ranomly sample from a normal istribution with a mean of zero an a stanar eviation of 0.75, 1.0, or 1.25 times the magnitue of the optimal response *. This arrangement provie low, meium, an high levels of variation aroun the true unerlying response surface relative to the optimal function value * 1 that was common to all six test problems. Our stuy of the ith problem (1 i 6) constitute a complete factorial experiment in which there were three factors each at three levels as efine below: P j jth level of optimization proceure Q k kth level of problem imensionality NM for j 0, RSS for j 1, RS S9 for j 2; 2 4 in problem 4 for k 1, 10 8 in problem 4 for k 2, in problem 4 for k 3; Figure 7. Normal probability plot of estimate resiuals in the ANOVA moel (22) for performance measure L of test problem 1. an N l lth level of noise stanar eviation 0.75 * for l 1, 1.00 * for l 2, 1.25 * for l 3. Within the ith experiment an for each of the selecte performance measures that were observe on the mth replication of the treatment combination (P j, Q k, N l ), we postulate a statistical moel of the form Z ijklm 0 P W Pj Q W Qk N W Nl PQ W Pj W Qk PN W Pj W Nl QN W Qk W Nl ijklm, (22) where 1 i 6, 1 m 9, an the coe inepenent variables W Pj, W Qk, an W Nl are efine as follows: W Pj W Qk 1, for j 0, 0, for j 1, 1, for j 2; 1, for k 1, 0, for k 2, 1, for k 3; an W Nl 1, for l 1, 0, for l 2, 1, for l 3. We use the statistical moel (22) to perform analysis of variance (ANOVA) an appropriate follow-up multiple comparisons proceures for each of the performance measures L, D, B, an A; an for these performance measures, the epenent variable Z ijklm is given by L ijklm, D ijklm, B ijklm, or A ijklm, respectively. To assess the valiity of the statistical moel (22) on which our experimental performance evaluation is base, we examine the estimate resiuals for this moel using normal probability plots an the Shapiro-Wilk test for normality (Shapiro an Wilk 1965). Figures 7 to 10, respectively, isplay normal probability plots for the resiuals corresponing to the performance measures L, D, B, an A in test problem 1. The P-values for the Shapiro-Wilk test statistics corresponing to Figures 7 through 10 are 0.99, 0.98, 0.99,

10 281 A Revise Simplex Search Proceure for Stochastic Simulation Response Surface Optimization Figure 8. Normal probability plot of estimate resiuals in the ANOVA moel (22) for performance measure D of test problem 1. Figure 10. Normal probability plot of estimate resiuals for ANOVA moel (22) of performance measure A in test problem 1. Figure 9. Normal probability plot of estimate resiuals in the ANOVA moel (22) for performance measure B in test problem 1. an 0.98, respectively. These results are representative of the other 20 cases iscusse in Humphrey (1997). We believe these results provie substantial visual an statistical evience that the resiuals associate with the ANOVA moel (22) are approximately normally istribute with a constant variance across all levels of the inepenent variables P j, Q k, an N l. For a more etaile iscussion of the valiation of (22) incluing formal statistical tests for homogeneity of the response variances, see Humphrey an Wilson (2000). We compute average performance measures for each problem i (1 i 6) an optimization proceure j (0 j 2) as follows: L ij k 1 3 l 1 9 L ijklm ; m 1 an D ij, B ij, an A ij are efine similarly. For problem i separately (1 i 6), we use the ANOVA proceure of SAS (SAS Institute 1989) to compare L i0, L i1, an L i2 via a Ryan-Einot-Gabriel-Welsch multiple comparisons F-test (Einot an Gabriel 1975) with level of significance This test looks for significant ifferences among the means, an groups the means accoringly (where means not significantly ifferent from each other are place within the same group). This type of test was performe 24 times so that each of the three optimization proceures was compare against the other two proceures with respect to each of the four performance measures on all six test problems. It shoul be recognize that the overall level of significance 0.05 for the multiple-comparisons tests iscusse in the next section applies to each combination of test problem an performance measure separately. Because the performance of the selecte optimization proceures iffere so rastically between test problems, it was necessary to analyze the results for each test problem as a separate experiment. Table III summarizes the results of these F-tests, an Section 3.3 below contains an analysis of these results. 3.3 Analysis of Experimental Results Most of the analysis of this section is base irectly on the information presente in Table III. We consier each of the performance measures separately over the six problems stuie. Humphrey (1997) provies complete etails on the analysis of the experimental results ANOVA Results The ANOVA results provie aitional evience of the aequacy of the statistical moel (22) for the purposes of this stuy. All R 2 values are above 0.93, an most are above For each test problem, the ANOVA reveals two significant main effects problem imensionality an optimization proceure. As the imensionality of each test problem increase, optimization of the associate function became more ifficult; an this phenomenon resulte in large F- values for the imensionality factor (Q k ). For each test problem, the corresponing ANOVA for (22) also reveals that optimization proceure (P j ) is a highly significant main effect. As suggeste by Table III an elaborate in the next four subsections, the principal source of this significant effect was the generally superior performance of proceure RSS versus proceures NM an RS S9 with respect to the performance measures A, B, an D.

11 282 Humphrey an Wilson Table III. Results of Multiple Comparisons Tests on Proceures NM, RS S9, an RSS for Level of Significance 0.05 Performance Measure Prob i L ij D ij B ij A ij Proc j Value Gr* Proc j Value Gr* Proc j Value Gr* Proc j Value Gr* 1 RS S NM NM NM RSS RS S RS S RS S NM RSS RSS RSS RS S NM NM NM RSS RSS RS S RS S NM RS S RSS RSS RS S NM NM RS S RSS RS S RS S NM NM RSS RSS RSS RSS NM NM NM RS S RS S RS S RS S NM RSS RSS RSS RS S NM RS S RS S RSS RS S NM NM NM RSS RSS RSS RS S RS S RSS NM RSS NM NM RS S NM RSS RS S RSS *Grouping of proceures with nonsignificant ifferences in performance base on Ryan-Einot-Gabriel-Welsch multiple comparison proceure. Table IV. Proceure Relative Computational Effort of Proceures Problem Of the two-factor interactions represente in (22), the only significant effect is the interaction of problem imensionality with search proceure. Unfortunately, we have been unable to raw any general conclusions about the relative avantages or isavantages of the three search proceures with increasing imensionality. We believe that this issue shoul be the subject of future investigation Number of Function Evaluations Avg NM RSS RS S In terms of the logarithm of the number of function evaluations performe, Table III shows that proceures RSS an RS S9 were roughly comparable, while proceure NM require significantly less work than RSS or RS S9 to eliver a final answer. If we take the number of function evaluations for proceure NM as a baseline, then from Table IV we see that proceures RSS an RS S9 generally require about four times as many function evaluations as proceure NM Final Function Value at Estimate Optimum The results presente in Table III for the performance measure D warrant further iscussion. In every test problem except problem 2, proceure RSS yiele an average value of D that is significantly smaller than the average D-values prouce by either NM or RS S9. In problem 2 (that is, the trigonometric function with the lattice (20) of optimal points), proceures RSS an RS S9 yiele results that are not statistically istinguishable from each other but are significantly better than the results of proceure NM. Moreover, notice that with respect to the performance measure D, proceure RSS performe much better than either RS S9 or NM on two of the six problems (namely, problems 1 an 4) Maximum Relative Component Deviation from Global Optimum With respect to the performance measure B, Table III shows that proceure RSS significantly outperforme both proceures NM an RS S9. In problem 1, proceure RSS has a B -value of about 0.38 while the corresponing B -values for proceures RS S9 an NM are each about The results are less ramatic for problems 2 5, but they still clearly favor

12 283 A Revise Simplex Search Proceure for Stochastic Simulation Response Surface Optimization RSS. In problem 6, however, there is no clear-cut istinction between the performances of the three proceures Average Relative Component Deviation from Global Optimum With respect to the performance measure A, Table III shows that proceure RSS significantly outperforme proceures NM an RS S9 in the first four problems. In problems 5 an 6 there are no significant ifferences in the performances of the three proceures. 4. Conclusions an Recommenations for Future Research 4.1 Conclusions The results of our experimental performance evaluation of proceures NM, RS S9, an RSS show that in the six test problems, proceure RSS require roughly as much work as proceure RS S9 an about four times as much work as proceure NM. However, in four of the six test problems, proceure RSS significantly outperforme RS S9 an NM with respect to all measures of convergence to the optimum; an in the other two test problems, RSS consistently elivere results at least as goo as the results for proceures NM an RS S9. Although such experimental results are extremely ifficult to generalize, they o suggest that significant improvements in the performance of simplex-searchtype proceures can be achieve by exploiting the principal features of proceure RSS namely, a multiphase approach in which (a) the search is restarte in the secon an subsequent phases; (b) in successive phases the size of the initial simplex is progressively reuce while the shrink coefficient is progressively increase to provie aequate protection against premature termination; an (c) in the en the best solution is taken over all phases of the search proceure. 4.2 Recommenations for Future Research The analysis in Section 4 raises questions an issues that merit consieration for future work. The suite of six test problems shoul be enlarge to provie for analysis on a collection of test problems that encompasses an even broaer range of the following factors: egree of ifficulty, imensionality, an response surface geometry. The experimental performance evaluation shoul also be expane to inclue other variants of proceure NM. Another promising area for future research is a more etaile stuy of the effects of imensionality on the performance of proceure RSS. While our stuy looke at imensionalities 2, 10, 18 ( 4, 8, 16 for test problem 4), a more etaile examination of imensionalities within an above this range coul probably provie aitional insight into how proceure RSS performs as the number of esign variables changes. Finally, an effort shoul be mae to formulate some rules of thumb for the use of proceure RSS in general applications. Particular issues of interest are how to set the starting point x 1 an the initial step size parameter for the search proceure. We believe that future progress in the evelopment of effective an efficient simplex-search-type proceures will epen critically on the evelopment of generally applicable, robust techniques for ajusting these quantities to the problem at han. References Anerson, V.L., R.A. McLean Design of Experiments: A Realistic Approach. Marcel Dekker, Inc., New York. Barton, R.R., J.S. Ivey, Jr Neler-Mea simplex moifications for simplex optimization. Management Science Corana, A., M. Marchesi, C. Martini, S. Riella Minimizing multimoal functions of continuous variables with the simulate annealing algorithm. ACM Transactions on Mathematical Software Einot, I., K.R. Gabriel A stuy of the powers of several methos of multiple comparisons. Journal of the American Statistical Association Humphrey, D.G A revise simplex search proceure for stochastic simulation response-surface optimization. Ph.D. Dissertation, Department of Inustrial Engineering, North Carolina State University, Raleigh, NC. Humphrey, D.G., J.R. Wilson A revise simplex search proceure for stochastic simulation response-surface optimization. D.J. Meeiros, E.F. Watson, J.S. Carson, M.S. Manivannan, es. Proceeings of the 1998 Winter Simulation Conference. Institute of Electrical an Electronics Engineers, Piscataway, NJ Humphrey, D.G., J.R. Wilson A revise simplex search proceure for stochastic simulation response-surface optimization. Technical Report, Department of Inustrial Engineering, North Carolina State University, Raleigh, NC. ftp://ftp.ncsu.eu/pub/ eos/pub/jwilson/rssv4c.pf [accesse April 5, 2000]. Kuhl, M.E., J.R. Wilson Least squares estimation of nonhomogeneous Poisson processes. Journal of Statistical Computation an Simulation ftp://ftp.ncsu.eu/pub/eos/pub/ jwilson/jscs30.pf [accesse April 5, 2000]. Moré, J.J., B.S. Garbow, K.E. Hillstrom Testing unconstraine optimization software. ACM Transactions on Mathematical Software Neler, J.A., R. Mea A simplex metho for function minimization. Computer Journal J.M. Parkinson, D. Hutchinson An investigation into the efficiency of variants on the simplex metho. F.A. Lootsma, e. Numerical Methos for Non-linear Optimization. Acaemic Press, Lonon SAS Institute, Inc., SAS/STAT User s Guie, Version 6, Fourth Eition. SAS Institute Inc., Cary, NC. Shapiro, S.S., M.B. Wilk An analysis of variance test for normality. Biometrika Wagner, M.A.F., J.R. Wilson Using univariate Bézier istributions to moel simulation input processes. IIE Transactions