Surrogate-assisted Self-accelerated Particle Swarm Optimization

Surrogate-assisted Self-accelerated Particle Swarm Optimization Kambiz Haji Hajikolaei 1, Amir Safari, G. Gary Wang ±, Hirpa G. Lemu, ± School of Mechatronic Systems Engineering, Simon Fraser University, V3T0A3 Vancouver, BC Canada, Mechanical and Structural Engineering Department, University of Stavanger, 4036 Stavanger, Norway Surrogate-assisted self-accelerated particle swarm optimization () is a major modification of an original PSO which uses all previously evaluated particles aiming to increase the computational efficiency. A newly in-house developed metamodeling approach named high dimensional model representation with principal component analysis (PCA- HDMR), which was specifically established for so called high-dimensional, expensive, blackbox (HEB) problems, is used to approximate a function using all particles calculated during the optimization process. Then, based on the minimum of the constructed metamodel, a term called metamodeling acceleration is added to the velocity update formula in the original PSO algorithm. The proposed optimization algorithm performance is investigated using several benchmark problems with different number of variables and the results are also compared with original PSO results. Preliminary results show a considerable performance improvement in terms of number of function evaluations as well as achieved global optimum specifically for high-dimensional problems. Nomenclature = Surrogate-assisted self-accelerated particle swarm optimization RS-HDMR = Random sampling high dimensional model representation PCA-HDMR = High dimensional model representation with principal component analysis MDO = Multi-disciplinary design optimization HEB = High-dimensional, expensive, black-box D = Dimension m = number of used basis functions N = number of sample points I. Introduction omputational performance of optimization methods and strategies plays a major role in design optimization of C engineering problems with multidisciplinary nature, i.e., multidisciplinary design optimization (MDO). MDO is a field of engineering optimization that simultaneously addresses a number of disciplines to solve real design problems. In this area, however, a majority of problems involve so called high-dimensional, expensive, black-box (HEB) functions, which typically need complex finite element analyses (FEA) and/or computational fluid dynamics (CFD) for analyses/simulations. To find the optimal values of the design variables for such problems, searching through a considerably high-dimensional design space is required, resulting in difficulties called curse of dimensionality. On the other hand, the high dimensionality of design variables presents an exponential difficulty for both problem modeling and optimization 1. Research Assistant, Product Design & Optimization Laboratory, School of Mechatronic Systems Engineering, Simon Fraser University, V3T0A3 Vancouver, BC Canada PhD Candidate, Mechanical Design & Simulation Research Group, Mechanical and Structural Engineering Department, University of Stavanger, 4036 Stavanger, Norway ± Professor, School of Mechatronic Systems Engineering, Simon Fraser University, V3T0A3 Vancouver, BC Canada Associate Professor, Mechanical and Structural Engineering Department, University of Stavanger, 4036 Stavanger, Norway 1

At the same time, there are a variety of approaches attempt to tackle the mentioned kind of challenge dealing with optimization of high-dimensional problems from both modeling and optimization points of view. Some techniques are single level (hierarchical) methods, and other techniques are considered as multi-level (non-hierarchical) approaches 2. Among all these methods, random sampling high dimensional model representation (RS-HDMR) is one of the most commanding and efficient methods has been developed to build an HDMR model from random sample points with a linear combination of specified basis functions 3. In terms of optimization strategies, nature-inspired algorithms such as genetic algorithm (GA), particle swarm optimization (PSO), and ant colony optimization (ACO) have recently demonstrated their success as well as popularity in practice. Nevertheless, they cannot fully satisfy the design optimization need for the complex, high dimensional and expensive problems so that some minor/major modifications are demanded depending on the nature of difficulty. Among them, PSO is becoming a hotspot as a population-based technique to meet requirements on both computational efficiency and self-improvement capability. Hence, many researchers have strived to modify its feature by using various development strategies 4. In recent times, a large number of studies have paid attention to use metamodels for particle evaluation during PSO process in different applications 5-9. In this way, they have constructed surrogate-based approximations and utilized them in conjuction with PSO in an inexact pre-evaluation procedure so that by using this mixed evaluations (metamodels/high-fidelity functions) computational cost can be meaningfully reduced. This study, however, introduces a novel approach of surrogate-assisted PSO algorithms to internally develop its performance and to automatically make itself more appropriate specially for HEB problems. Application of high dimensional model representation with principal component analysis (PCA-HDMR) as a means of metamodeling of formerly evaluated particles during a PSO procedure is addressed in this work. On the one hand, there are totally random particles evaluated in initial population as well as different iterations of PSO so that applying RS-HDMR concept is appropriate. Because of the limited number of particles and non-controllable sampling (which would make it non-uniform), on the other hand, using a modified approach in approximating HEB problems named PCA- HDMR seems to be more feasible. The approach uses principal component analysis to help identifying the coefficients of basis functions in a way that minimizes the variation from the underlying (black-box) function 10. The metamodel constructed from unemployed evaluated particles is finally used to generate an extra term for the existing velocity update formula at each iteration. With application of this new framework, PSO accelerates and reaches to the global minimum by using significantly fewer number of function evaluations. The proposed method is applied to five benchmark functions and its good performance can be seen when compared with the original PSO. II. Overview of PCA-HDMR High Dimensional Model Representation (HDMR) is an approach suitable for high dimensional metamodeling problems, first inotroduced by Sobol 12 with the general form of:,,,, (1),,, where is a constant representing the zero-th order effecr of. and, represent first-order effect of variable and second-order effect variables,, respectively. In general, describes the l-th order effect of variables,,,. There are different types of HDMR such as ANOVA-HDMR 13, Cut-HDMR 13, RBF-HDMR 14, RS-HDMR 3, and so on, and there are pros and cons in using them. For this paper, the recently developed type, PCA-HDMR, is selected because it works with random sampling, even if the sampling is non-uniform. This property is vital in because of non-uniformity of sampling during the optimization iterations. PCA-HDMR has the general form of:, (2) 2

where and, are families of linearly independent bases for uni-variable and bi-variable functions on the 0,1 and 0,1, repectively. The form is common between RS-HDMR and PCA-HDMR and the difference is in the way of calculating the coefficients. PCA-HDMR matrix is a 1 matrix shown in Eq. 3. The basis functions,, are named for simplicity. The basis functions are calculated in all sample points and are subtracted by their average values over the sample points to construct the PCA-HDMR matrix. The last column of the matrix is related to the original black-box function value in the sample points. They are rescaled to be in 0,1 () and subtracted by their average values. 1 1, 2 2,,, (3) After applying singular value decomposition (SVD), the last linear transformation that corresponds to the minimum variation is selected. Considering that,,, as the vector of coefficients related to the transformation, the scaled approximation of the original function is found as: 1 1 1 2 2 1 (4) By rescaling the function to the real range of black-box the function, an approximation of the function is obtained that is called PCA-HDMR metamodel. III. Methodology The first part of this section briefly reviews the original PSO. The proposed modified PSO is then presented in the second part. A. Original PSO PSO is one of the most well-known swarm algorithms which are inspired by the behaviour of organisms that interrelate in nature within the large groups. PSO originally developed by Kennedy and Eberhart in 1995 11. In general, the PSO algorithm consists of three main steps as follows 11 : Generating positions and velocities of particles Update the velocities Update the positions Each particle refers to a point in a multi-dimensional space whose dimensions are related to the numbers of design variables. The positions of these particles are changed in iterations corresponding to the velocities which are updated at each step. In an original PSO algorithm, the particles are manipulated according to the equations specified by dashed box in Figure 1, where: w is the inertia factor, v is velocity of i th particle in the current motion, c is self confidence factor, and 1 c 2 is swarm confidence factor. Also x i, k i k i P, and g Pk are position of i th particle in current motion, the best position of i th particle in current and all previous moves, and position of the particle with the best global fitness at current move k, respectively. B. PCA-HDMR driven accelerated PSO () Since it is pretty straightway to amend PSO by directly appending other mechanisms such as proper metamodels to simultaneously evaluate some of the particles without need to the expensive function evaluation, PSO has the potential to become a self-modified intelligent structure for enhancing the computational efficiency by itself. As shown in Figure 1, the proposed surrogate-assisted self-accelerated PSO () algorithm is a paradigm constructed based on using the previous experience of the algorithm to define an extra velocity updating term. Based on that, the PCA-HDMR builds an appropriate metamodel from all previous particles evaluated by expensive function from the beginning of the optimization process until the current step. Then, the algorithm calculates new velocities to move the particles from positions in time k to new positions in time k+1 by using four terms instead of three previously defined parameters. In other words, new algorithm needs another value to calculate updated velocities: M so-called metamodel s minima. Its relevant coefficient is named metamodeling g k 3

acceleration factor. The coefficients indicate the effect of the current motion, particle own memory, swarm influence, and metamodel driven minima. The algorithm is repeated until a stop criterion reaches (which can be maximum epochs or a defined precision). It should be noted that no more function evaluation is needed for the new algorithm and the previously used sample points are reused for acceleration. Note that the PCA-HDMR needs a minimum number of function evaluations (NOC) for building the model 10. Therefore, and PSO are the same until the number of evaluated points reaches the NOC value. After that the new term is added to the PSO that makes. In the following section, five chosen problems which are a class of test problems with different dimension sizes are addressed to investigate the ability of the proposed optimization algorithm. Figure 1. Flowchart of the proposed algorithm IV. Results and Discussion In this section, the results obtained from is compared with those of the original PSO using five different benchmark functions. The functions are selected with different shapes and with different number of input variables, which are shown in Appendix. Every benchmark function is optimized using and PSO five times and the convergences are shown in the same graph. Figures 2-6 show the results related to the functions # 1-5, respectively. The curves with empty squares show the PSO convergence that are compared to curves with solid square representing approach convergence. The population size is equal to the number of input variables times four. Almost in all the runs related to all functions, shows better performance that PSO. The interesting results obtained from the figures is that applying the new approach has more significant effect in high dimensional 4

problems. When the number of input variables increases, the distances between empty square curves and solid square curves are increasing. This is a promising result and can be an opening to optimizing high dimensional, expensive and black-box (HEB) problems. 4 Objective Value 3 2 1 0-1 Figure 2. The convergence comparison of different PSO approaches: Function # 1, 2-D test function 10 2 Objective Value 10 1 10 0 10-1 Figure 3. The convergence comparison of different PSO approaches: Function # 2, 5-D test function 5

10 3 Objective Value 10 2 10 1 10 0 10-1 10-2 Figure 4. The convergence comparison of different PSO approaches: Function # 3, 10-D test function 10 1 Objective Value 10 0 10-1 10-2 Figure 5. The convergence comparison of different PSO approaches: Function # 4, 20-D test function 6

Objective Value 10 5 10 4 Figure 6. The convergence comparison of different PSO approaches: Function # 5, 30-D test function For better illustration of the effect of using surrogate model acceleration in PSO performance, two random runs from the previously shown optimizations are presented in detail in Fig. 7. The runs are related to the function # 4, 20-D function. As shown in flowchart (Fig. 1), surrogate acceleration starts working after the number of evaluated sample points reachs the NOC. The highlighted sections in Fig. 7 show the transition from PSO to SASA- PSO for two independent runs. After completing iteration #12, the number of evaluated functions is reached to the related NOC and the results has a significant jump to better optiumum point at iteration #13. Figure 7. Impact of metamodeling acceleration term on the PSO trend (): Two sample runs of Function # 4, 20-D test function 7

V. Conclusion In this paper, a new approach to HDMR-based self-acceleration of an original PSO was introduced. The approach comes from the capability of PCA-HDMR algorithm in building metamodels with random non-uniform sampling. In the proposed Surrogated Assisted Self Accelerated PSO () method, the previously evaluated sample points that are disregarded in the original PSO are used for making the accelerator and no additional function evaluations is needed than the original PSO. The results show considerable improvement of the convergence rate and global optimization searching ability. The approach is more effective in high dimensional problems. More tests and applications will be performed to further test the new algorithm for HEB problems. References 1 Shan, S., and Wang, G., "Survey of modeling and optimization strategies to solve high-dimensional design problems with computationally-expensive black-box functions", Struct Multidisc Optim, vol. 41, pp. 219-241, 2010. 2 Yi, S., Shin, J., and Park, G., "Comparison of MDO methods with mathematical examples", Structural and Multidisciplinary Optimization, vol. 35, pp. 391-402, 2008. 3 Alis, O. F., and Rabitz, H., Efficient implementation of high dimensional model representations, J. of Math. Chem., Vol. 29, No. 2, pp. 127-142, 2001. 4 Chan, F. T. S., and K. Tiwari, M., Swarm intelligence: Focus on ant and particle swarm optimization, I-Tech Education and Publishing, Vienna, Austria, 2007. 5 Praveen, C., and Duvigneau, R., Metamodel-assisted particle swarm optimization and application to aerodynamic shape optimization, Research project report at Unité de recherche INRIA Sophia Antipolis, France, 2007. 6 Tang, Y., Chen, J., and Wei, J., A surrogate-based particle swarm optimization algorithm for solving optimization problems with expensive black box functions, Engineering Optimization, vol. 45, Issue 5, pp. 557-576, 2012. 7 Im, J-B, Ro, Y-H, Lee, S-Y, and Park, J., Hybrid simulated annealing with particle swarm optimization applied krigging meta model, Proceedings of the 48 th AIAA Structural Dynamics and Materials Conference, Honolulu, Hawaii, USA, April 2007. 8 Im, J-B, and Park, J., Stochastic structural optimization using particle swarm optimization, surrogate models and Bayesian statistics, Chinese Journal of Aeronautics, Vol. 26, No. 1, pp. 112-121, 2013. 9 Santana, L. V., Coello, C. A., Hernandez, A. G., and Moises, J., Surrogate-based multi-objective particle swarm optimization, Proceedings of IEEE Swarm Intelligence Symposium, USA, September 2008. 10 Hajikolaei, K. H., and Wang G. G., High Dimensional Model Representation with Principal Component Analysis: PCA- HDMR, Submitted to ASME Journal of Mechanical Design, 2012. 11 Kennedy, J., and Eberhart, R., "Particle Swarm Optimization", Proceedings of IEEE International Conference on Neural Networks IV, pp. 1942 1948, 1995. 12 Sobol, I. M., "Sensitivity estimates for nonlinear mathematical models ", Math. Model. Comp., Vol. 1, Issue 4, pp. 407-414, 1993. 13 Rabitz, H., and Alis, O. F., " General foundation of high dimensional model representation ", J. of Math. Chem., Vol. 25, pp. 197-233, 1999. 14 Shan, S., and Wang, G. G., " Metamodeling for High Dimensional Simulation-based Design Problems ", ASME Trans., J. of Mech. Des.., Vol. 132, pp. 1-11, 2010. Appendix No. D Function Variable Ranges 1 2 4 2.1 1 3 4 4 2 2, 2 5 1 1 1 2 2, 3 10 1 10 1 4 20 1 10 3 3, 0 5, 5 30 1,2,3,,30 2 2, 8