PARALLELIZATION OF THE NELDER-MEAD SIMPLEX ALGORITHM

Transcription

1 PARALLELIZATION OF THE NELDER-MEAD SIMPLEX ALGORITHM Scott Wu Montgomery Blair High School Silver Spring, Maryland Paul Kienzle Center for Neutron Research, National Institute of Standards and Technology Gaithersburg, Maryland

2 ABSTRACT The Nelder-Mead Simplex algorithm, proposed by John Nelder and Roger Mead, is an iterative multidimensional, downhill search algorithm which uses a simplex to perform a search over a function. It is frequently used in numerical optimization because of its simplicity and effectiveness, but has room for improvement. As a mainly serial algorithm, the Nelder-Mead Simplex algorithm fails to take full advantage of the parallel processing capabilities available among modern computers. In order to improve the performance of the Nelder-Mead Simplex algorithm, a parallel version of the algorithm, described in Lee and Wiswall (2007), was implemented, allowing it to effectively utilize parallel processing. Additionally, another parallel variation of the algorithm was written by extending the number of simplex vertices. Each version of the algorithm was tested with three optimization problems: minimizing the quadratic function, minimizing the Rosenbrock function and fitting a polynomial. The results of these tests indicated linear speedup in the parallel algorithm when it was run in a parallel environment. Even without use of parallel processing however, the algorithm still showed increased robustness, successfully converging more often, at the cost of a slight increase in work. The parallel Nelder-Mead Simplex algorithm can be used for faster and more efficient optimization by utilizing parallel processing techniques, or by employing its increased robustness. 1

3 INTRODUCTION Optimization is a very important technique in everyday life. Businesses want to optimize the difference between profits and production costs. Engineers want to optimize the performance of their products. Researchers want to optimize data to fit models. There exist a variety of optimization algorithms with different purposes; some can find multiple solutions to a problem, others find solutions extremely quickly. However, for more complex problems, optimization algorithms require longer run times to find a solution, or perhaps fail to find one at all. The purpose of this research is to increase the performance of one common optimization algorithm, the Nelder-Mead Simplex algorithm, through parallelization, and compare its performance to existing algorithms. The Nelder-Mead Simplex algorithm, developed by John Nelder and Roger Mead in 1965, is a quick and simple multidimensional optimization algorithm. Multidimensionality is important for fitting models, which may contain dozens of unknown coefficients. Given the general form and calculated test points of a model, the Nelder-Mead Simplex algorithm can find specific coefficients by optimizing the difference between the estimated model and the points. The Nelder-Mead Simplex algorithm uses a simplex to perform a downhill search on a function. A simplex in n dimensions is a figure with n + 1 vertices such that all vertices cannot be contained within any lower dimension. To traverse the function, each point is sorted by its value, equal to the function evaluation at that point. The worst point, the point with the highest value, is transformed about the centroid of the remaining points, producing a new point. Transformations, as shown in Figure 1.1, include the reflection, expansion, outer contraction and inner contraction points. This constitutes one iteration of the algorithm. The algorithm is repeated until the simplex converges onto a single point. 2

4 Figure 1.1 Each point on the line containing the worst vertex and the centroid are different possible points for transforming the worst vertex. The centroid is not a transformation point because otherwise the simplex might lose a dimension of freedom. Parallel processing environments are now common in modern technology, allowing multiple computations to be performed simultaneously. The original algorithm, however, is mainly serial. Parallelizing the algorithm can improve the overall speed of the algorithm linearly with the number of parallel processors available. The Nelder-Mead algorithm can be parallelized by taking k worst vertices to transform rather than just a single worst vertex. Each of the vertices can be transformed simultaneously, one on each parallel process. Consequently, the result will return in the same time it takes a single vertex to be computed. Three versions of the algorithm were used in this research: the original Serial Simplex algorithm, the Parallel Simplex algorithm, and the Extended Simplex algorithm. The algorithms were also run on three different environments: a single core computer, a multi-core computer and a computer cluster. Each algorithm was tested with various problems, ranging from simple to complex. The results indicated that both the parallel and extended versions of the algorithm showed a linear speedup in number of iterations needed up to a certain point. Wall clock time followed this linear speedup up until all the possible parallel processes were used. The results of this research can be used to solve optimization problems quicker and more efficiently. 3

5 MATERIALS AND METHODS Hardware: The hardware used in this research consisted of 27 computers, one gigabit Ethernet switch, and a set of proper computer accessories for each computer. Computer accessories include one power supply and one Ethernet cable. One computer, called the head node, was a regular computer with a hard disk drive. All other computers, called worker nodes, had dual core processors, 2 GB of RAM and no hard disk drive. These "diskless" computers were placed on a computer rack for centralized cooling and organization. Connecting all the computers together formed the computer cluster used to run programs in a parallel environment. Software: The software used in this research was divided into those needed to setup the computer cluster and those needed to implement the Nelder-Mead Simplex algorithm. Ubuntu was the primary operating system used in the computer cluster. Additional packages were installed on the head node to configure the cluster. A Dynamic Host Configuration Protocol (DHCP) server established the head node to act as a router, creating an internal network solely between it and the connected worker nodes. A Pre-boot Execution Environment Boot (PXE Boot) server provides instructions for the worker nodes on startup through the network, allowing them to download and boot to a custom built Ubuntu kernel through a Trivial File Transfer Protocol (TFTP) server. A Network File System (NFS) server allowed worker nodes to access a file system mounted over the network on the head node. Ganglia allowed computers on the cluster to be monitored through a web interface. 4

6 The Nelder-Mead Simplex algorithms were programmed in Python. Python includes the standard Python library as well as additional libraries such as NumPy, SciPy, MatPlotLib and PyLab. Open Message Passing Interface (OpenMPI) and its Python wrapper, MPI4py, were also used to communicate between worker nodes within the program. Bayesian Uncertainty Modeling for Parametric Systems (BUMPS) is a fitting engine written in Python, and contains a modified version of the Serial Nelder-Mead Simplex algorithm found in the SciPy library, as well as Differential Evolution, another optimization algorithm. Setting up the Cluster: The computer cluster provided the parallel environment needed to run parallel programs. To setup the computer cluster, all computers were attached to the gigabit switch using Ethernet cables. The switch was attached to the head node, which ran a DHCP server to configure and manage the cluster network. A custom Ubuntu kernel was built to minimize size while including all necessary components, such as network drivers, a temporary file system stored in memory, and an NFS mounted from the head node. Because the worker nodes were diskless, this kernel needed to be sent on startup in order for the worker nodes to boot. In order to run OpenMPI on the cluster, all worker nodes require a password-less remote shell connection. Adding a generated SSH key to an account on the NFS allowed remote connections to log in without a password. This enabled OpenMPI to log in to each worker node and run programs without user input. Because the worker nodes were located on an isolated network, security was not a concern in enabling this feature. 5

7 Programming the Algorithm: Using the implementation in BUMPS and the original paper Nelder and Mead (1965) as references, a standalone version of the original Nelder-Mead Simplex algorithm was created in Python. The algorithm's parameters included the target function to optimize, various starting and stopping criteria, and returned a set of results containing the solution and performance statistics. Modifying the Algorithm: Multiple branches of this program were derived from the original algorithm. The main modification was the parallel implementation of the algorithm, described in Lee and Wiswall (2007). The Parallel Simplex algorithm, accepted an additional argument, which specified the degree of parallelism. In the original algorithm, only the worst vertex in the simplex is transformed. The degree of parallelism specifies multiple vertices to transform. For example, with a degree of 2, each iteration transforms the two worst vertices. A degree of 1 is the same as the Serial Simplex algorithm. The transformation of these worst vertices can be parallelized since transformations are independent of each other. The Extended Simplex algorithm is an extension of the Parallel Simplex algorithm. Instead of using the traditional definition of a simplex, the Extended Simplex algorithm uses additional vertices. The number of additional vertices ranges anywhere from 1 to the number of dimensions. For example, in two dimensions, an extended simplex could be a square instead of a triangle, or, in three dimensions, a cube instead of a tetrahedron. At one additional vertex, the Extended Simplex algorithm is equivalent to the Parallel Simplex algorithm. Another modification to the Parallel Simplex algorithm utilized a mapper to perform function evaluations in parallel, either using Python's multiprocessing library or MPI. Given a 6

8 collection of points to transform, the mapper assigns each point to a processor or worker node. If there are more points than processors, the mapper waits until one processor finishes and then sends it the next point to transform. Test Problems: Three test problems were used to measure the performance of the algorithms. The Quadratic function and the Rosenbrock function are unimodal and multidimensional functions that are optimized by finding their minimums. The Polynomial Fitting problem attempts to fit a certain degree polynomial given a generated set of points. The function itself takes polynomial coefficients and returns the sum of the residuals squared at each of the points. The Quadratic function, defined by the equation in Figure 2.1, is a simple bowl whose minimum is at the origin. Because all sides point downhill, convergence is both quick and trivial. N f(x 1, x 2,, x N ) = x i 2 i=1 Figure 2.1 The Quadratic function in three dimensional space and its equation for a multidimensional point. 7

9 The Rosenbrock function, defined by the equation in Figure 2.2, is a more difficult function to optimize, due to the hill and curved valley. Its minimum lies at the point (1, 1,, 1). N 1 f(x 1, x 2,, x N ) = (1 x i ) (x i+1 x i 2 ) 2 i=1 Figure 2.2 The Rosenbrock function in three dimensional space and its equation for a multidimensional point. The Polynomial Fitting problem is an example of fitting data by minimizing the deviation of given points from a predicted function. First, an N - 1 degree polynomial is created with random coefficients. Next, N points are generated by taking the function evaluation of the polynomial at different points. The optimization algorithm is able to access only these given points and not the original polynomial. Because N points define a unique polynomial of degree N 1, there exists a unique set of N degree polynomial coefficients that fit the given set of points. The Polynomial Fitting function takes N coefficients as inputs and creates the predicted polynomial. The residuals at each of the given points are calculated from the predicted polynomial. The sum of the residuals squared is returned as the function value. When the predicted polynomial is equal to the original polynomial, all points fit the predicted polynomial, and the sum of the residuals squared will equal zero. 8

10 Given points p x1, p y1, p x2, p y2,, p xk, p yk to fit an N 1 degree polynomial K f(c 1, c 2,, c N ) = p yi g p xi 2 i=1 where g(x) = c 1 + c 2 x + c 3 x c N x N 1 Figure 2.3 Fitting three points with a cubic function. Residuals are shown with dotted lines. The sum of these residuals squared gives the function evaluation. Performing Tests: The performance of each algorithm was measured in terms of function evaluations, function iterations and rate of failure. Function evaluations measures the total amount of work an algorithm does. Ideally, the execution time of the algorithm itself is negligible compared to that of the function evaluation. For a serial environment, this statistic measures the amount of time needed to run the algorithm. For a parallel environment, this statistic only measures the work across all processors, since many evaluations are done at the same time. Function iterations measures performance of the algorithm in a parallel environment. Assuming there are a sufficient number of processors, each iteration takes the same amount of time, since no more than two serial evaluations are performed on any given processor. 9

11 The rate of failure measures the robustness of an algorithm by counting the number of times the algorithm fails to optimize. Failure can result from converging to a false minimum, exceeding the maximum number of iterations, or failing to converge at all. As the difficulty of the problem increases, it is important that the algorithm not only performs quickly, but consistently and successfully. These three statistics were collected by running performance tests using every combination of problem and algorithm. The Serial Simplex algorithm and Differential Evolution ran 100 trials with random initial conditions for each problem. The Parallel Simplex algorithm ran 100 trials for each problem for every degree of parallelism. Degrees of parallelism ranged from 1 to one less than the total number of vertices. Corresponding runs of different degrees of parallelism begin with the same seeded random initial conditions. The Extended Simplex algorithm ran 10 trials for every degree of parallelism and for every number of extra vertices. Extra vertices ranged from 1 to double the number of parameters. The Quadratic function was tested with 100 dimensions, the Rosenbrock function with 25 dimensions, and the Polynomial Fitting problem with 10 dimensions. 10

12 RESULTS Performance statistics were collected through capturing data on each of the algorithms. The results were categorized into three groups for comparison and analysis, with some results placed in more than one group for different comparisons. The first group of results compared the performance of the Parallel Simplex algorithm to that of the Serial Simplex algorithm on each problem. Figure 3.1 shows the average number of iterations versus degree of parallelism for the Quadratic, Rosenbrock and Polynomial Fitting problems. Figure 3.2 shows the average number of function evaluations versus degree of parallelism for the same problems. Figure 3.3 shows the rate of failure versus degree of parallelism for the Rosenbrock function. Degrees of parallelism are labeled as a percentage of the total parameters since each problem differed in the number of vertices. At 1 degree of parallelism, the Parallel Simplex algorithm transforms a single vertex, making it equivalent to the Serial Simplex algorithm. Figure 3.1 At 1 degree of parallelism, the Parallel Simplex is equivalent to the Serial Simplex. Linearity on the log-log plot indicates linear speedup in a parallel environment. All three problems demonstrate linear speedup up to about a 30% degree of parallelism. 11

13 Figure 3.2 A near horizontal lines indicate little change in the amount of work done. Again, the number of evaluations is nearly constant, or linearly decreasing in the case of the polynomial fit, until about a 30% degree of parallelism. Figure 3.3 The rate of failure demonstrates the difficulty of the problems. The Polynomial Fit failed often while the Quadratic function never failed to converge. The rate of failure is also minimal around a 30% degree of parallelism. 12

14 The second group of results compared the Parallel Simplex algorithm to the Serial Simplex algorithm, Extended Simplex algorithm and Differential Evolution. Tables 3.4, 3.5 and 3.6 present statistics of the four algorithms on the Quadratic, Rosenbrock and Polynomial Fitting problems respectively. For the Parallel Simplex algorithm, the degree of parallelism used in comparison is selected by the result that gives produces the best result. Quadratic Function Average Iterations Average Function Evaluations Failure Rate Serial Simplex % Parallel Simplex (29% degree of parallelism) Parallel Simplex (95% degree of parallelism) Extended Simplex (55% degree of parallelism with 100 extra vertices) % % % Differential Evolution % Table 3.4 The Quadratic function showed no instance of failure throughout the tests. The Parallel and extended Simplex algorithm list their best performance. Differential Evolution was run with default parameters, resulting in non-ideal performance. However, this may be compared to a non-ideal Parallel Simplex test. Rosenbrock Function Average Iterations Average Function Evaluations Failure Rate Serial Simplex % Parallel Simplex (32% degree of parallelism) Parallel Simplex (60% degree of parallelism) Extended Simplex (30% degree of parallelism with 25 extra vertices) % % % Differential Evolution %* Table 3.5 The Rosenbrock function proves to be a more difficult problem, requiring many more iterations, evaluations and showing higher rates of failure at the best performance. * Differential Evolution, a global optimizer, occasionally converged to another minimum at a saddle point, which accounts for its high failure rate. 13

15 Polynomial Fitting Average Iterations Average Function Evaluations Failure Rate Serial Simplex % Parallel Simplex (40% degree of parallelism) Parallel Simplex (80% degree of parallelism) Extended Simplex (50% degree of parallelism with 10 extra vertices) % % % Differential Evolution* Table 3.6 Polynomial fitting is the most difficult of the three problems, with the highest rates of failure. The Parallel Simplex shows a large decrease in evaluations, as opposed to a small increase in the other problems. The Extended Simplex shows even further decreases in all three categories. * Due to unmodified fitting parameters, Differential Evolution failed to converge at all within the iteration limit. The third group of results compares the performance of the Parallel Simplex algorithm to the Extended Parallel Simplex algorithm, with double the vertices, on various problems. Figures 3.7 and 3.8 compare the two on the Quadratic function for iterations and evaluations, respectively. Figures 3.9 focus specifically on the Extended Parallel Simplex algorithm. The color coded graphs represent the number of iterations (a) or evaluations (b) versus the number of extra simplex vertices versus the degree of parallelism. Since the degree of parallelism cannot exceed the total number of vertices, unobtainable points are colored white. 14

16 Figure 3.7 The Extended Simplex algorithm was run with twice the number of vertices for the Quadratic function. The linearity of the Extended Simplex algorithm extends further than that of the Parallel Simplex algorithm. The peak performance of the Extended Simplex algorithm is at 55% degree of parallelism as opposed to 29% in the Parallel Simplex algorithm. Figure 3.8 The horizontal line for the Extended Simplex also extends further than that of the Parallel Simplex in terms of evaluations. Although the Extended Simplex starts with more evaluations than the Parallel Simplex, the Parallel Simplex surpasses the Extended Simplex after reaching the peak. 15

17 Figure 3.9 The Extended Simplex algorithm was tested by varying both the degree of parallelism and the number of extra vertices. White colored points indicate that the degree of parallelism exceeded the total number of vertices. The best performance of iterations occurs at around 100 extra vertices and 100 degrees of parallelism. The best performance of evaluations occurs near the opposite end of the graph, at around 5 extra vertices and 5 degrees of parallelism. 16

18 DISCUSSION AND CONCLUSION Measures of Performance: Iterations and functions evaluations are measures of performance under different situations. In an environment with sufficient parallel processes available, iterations are a direct measure of performance, since the average computation time is ideally the same for each iteration. On a log-log scale, a linear plot corresponds to linear speedup, where the speed of the algorithm is directly proportional to degree of parallelism. Function evaluations indicate the total amount of work being done by all processors. At each iteration, each parallel process requires one or two function evaluations. In a parallel environment, this measure is irrelevant to performance since all evaluations are performed at the same time. In a serial environment however, function evaluations are the direct measure of performance. Rate of failure determine the robustness of the algorithm's optimization capabilities. An algorithm with a higher rate of failure requires more tests on a problem before a solution may be determined. A high rate of failure may even counteract the increased performance on individual runs. Similarly, a low rate of failure further increases the performance of an algorithm. Serial Simplex Algorithm vs Parallel Simplex Algorithm: The Parallel Simplex algorithm observes peak performance at approximately a 30% degree of parallelism, for all optimization problems. Up until this peak, parallel processing effectively allows for linear speedup, as shown by the linear decrease in iterations (Figure 3.1). The number of evaluations increases slightly up until the peak (Figure 3.2). After this peak, iterations and evaluations both increase significantly. Additionally, Parallel Simplex algorithm 17

19 shows a decrease in the rate of failure. On the Rosenbrock function, the algorithm would occasionally converge to a false minimum, and on the Polynomial Fitting problem, the algorithm failed to fit within the iteration limit. As the degree of parallelism increased to the peak, the rate of failure decreased to almost no failures on both problems. Compared to the Serial Simplex algorithm at 1 degree of freedom, the peak of the Parallel Simplex algorithm demonstrates linear speedup at the cost of a slight increase in evaluations. With a sufficient number of parallel processes, the evaluations are negligible. However, even without parallel processing, the peak still exhibits increased robustness at the cost of the slight increase in evaluations. Beyond the peak, there is a clear decrease in overall performance. Extended Simplex Algorithm: The Extended Simplex algorithm expands upon the Parallel Simplex algorithm by adding additional vertices, thus allowing the degrees of parallelism to increase beyond the number of parameters. On the Quadratic function and Polynomial fitting problems, its peak extends up to 55% degrees of parallelism, which shows an even further decrease in iterations compared to that of the Parallel Simplex algorithm (Table 3.4, 3.6). Evaluations and rate of failure also follow a similar pattern. The number of evaluations increases slightly on the Quadratic function, and decreases on the Polynomial fitting as the degree of parallelism approaches the peak. The rate of failure on the Polynomial fitting becomes 0%, compared to the 3% with the Parallel Simplex algorithm. On the Rosenbrock function, however, the peak still lies at approximately a 30% degree of parallelism (Table 3.5). The number of evaluations again increases slightly up to the peak, but the rate of failure is greater than that of the Parallel Simplex algorithm. On these failures, the 18

20 Extended Simplex algorithm converged to a false minimum, much like Differential Evolution. When calculating the centroid of the Extended Simplex, the extra vertices contribute more to the direction the transformation, which points to either the true minimum or a false minimum. Compared to the Parallel Simplex, the Extended Simplex is less likely to change directions of its search and search that direction more aggressively. While this increases the rate at which the algorithm converges, it also increases the chance of converging to false minimums. The Extended Simplex algorithm shows promising results, but depends on the given problem. Given a sufficient number of parallel processers, the Extended Simplex algorithm may increase its performance over the Parallel Simplex and Serial Simplex algorithm, but may also encounter decreased performance, as seen on the Rosenbrock function. Unlike the Parallel Simplex algorithm, an Extended Simplex with 1 degree of parallelism does not necessarily increase robustness, and yet comes with the cost of a much greater increase of evaluations, making it impractical for serial environments. Parallel Simplex Algorithm vs Differential Evolution: Differential evolution is a multimodal optimization algorithm that works by transforming a population of points. Unlike the Nelder-Mead Simplex algorithm, points are transformed per dimension, and are based on a series of random selections. Because iterations in Differential Evolution are calculated differently, they cannot be compared to the Nelder-Mead Simplex iterations (Table 3.4, 3.5). Additionally, Differential Evolution contains parameters for fine tuning the optimization such as the population size, the expansion factor and the crossover constant described in Storn and Price (1997). Since these factors were left at the default values specified in the BUMPS 19

21 software, evaluations cannot be compared either, since these values are not optimized for best performance. Performance results for the Parallel Simplex algorithm under non-optimal conditions were also included to show that the Parallel Simplex can also exhibit poor performance under flawed conditions. Whereas the Nelder-Mead Simplex algorithm is a local optimizer, Differential Evolution is a global optimizer, and will converge to any minimum. On the Rosenbrock function, Differential Evolution did not fail to converge, as shown in Table 3.5, but instead converged to another minimum on the Rosenbrock function. Upon further investigation, this point was determined to be approximately f( 1,1,,1) = 4. Differential Evolution requires further adjustments in order to be properly compared to the Parallel Simplex algorithm. Future Work: The Parallel and Extended Simplex algorithms both show promising results for improved optimization performance. Additional modification of the algorithms may further enhance performance and flexibility. The simplex vertex transformations listed in Figure 1.1 are ones proposed in the original algorithm. Implementing different sets of transformations, or adaptive transformations may increase the performance of the algorithm. The extra vertices in the Extended Simplex algorithm may be utilized for purposes other than search. Implementing the probability of an incorrect transformation could allow the simplex to escape a false convergence; otherwise the incorrect transformation is corrected in the next iteration. This would maintain the performance of the Parallel Simplex algorithm while increasing robustness on difficult problems. 20

22 REFERENCES Lee, D., & Wiswall, M. (2007). A Parallel Implementation of the Simplex Function Minimization Routine. Computational Economics, 30(2), Retrieved August 9, 2013, from simplex_edit_2_8_2007.pdf Nelder, J., & Mead, R. (1965). A Simplex Method for Function Minimization. The Computer Journal, 7(4), Retrieved August 9, 2013, from Storn, R., & Price, K. (1997). Differential Evolution A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces. Journal of Global Optimization, 11,