NEWTON BASED HOMOTOPY OPTIMIZATION METHOD FOR SOLVING GLOBAL OPTIMIZATION PROBLEM AHMAD ZHARIF SALAMI BIN MOHD TAHA

NEWTON BASED HOMOTOPY OPTIMIZATION METHOD FOR SOLVING GLOBAL OPTIMIZATION PROBLEM AHMAD ZHARIF SALAMI BIN MOHD TAHA A dissertation submitted in partial fulfillment of the requirements for the award of the degree of Master of Science (Mathematics) Faculty of Science Universiti Teknologi Malaysia JANUARY 2013

To my beloved loved ones and my beloved family iii

iv ACKNOWLEDGEMENTS Alhamdulillah, with the resources and guidance of Allah S.W.T, I managed to complete this project as scheduled. I would like to take this opportunity to convey my sincere appreciation to my supervisor, Mr. Ismail Bin Kamis for his excellent supervision, motivation and valuable advice given through the course of this project even though he has a tight schedule. I am particularly indebted to Assoc. Prof. Dr. Ismail Mohamad, the mathematics lecturer for the course MSM1280 for his thoughtful and constructive input towards this project. Not forgotten to Assoc. Prof. Dr. Rohanin Ahmad, a lecturer in mathematics for MSM1613 courses. This class teaches optimization techniques, including homotopy. On the basis of what she taught in class, I successfully applied the homotopy in this study. Without a doubt, my part to make this report a reality would not be possible without the support of my parents. To my entire friend especially Nur Intan Liyana and whomever I interact regularly, Thank you so much. Finally, I hope that my work will be beneficial to all and can be used as a reference in the future.

v ABSTRACT Optimization method is widely used in mechanics and engineering, economics, operations research and engineering controls. Various methods have been introduced to solve the optimization problem and mostly it will get a local optimum. One of the most commonly used method is the Newton-Raphson method. In this method, there are some circumstances where it is unable to solve the optimization problem. With the help of homotopy, the problems faced by the Newton-Raphson method can be overcome and thus solve the optimization problem. Therefore, the aim of this study is to investigate the Newton-Raphson method as the basis for the homotopy optimization method for finding local minimum and also the global minimum. There are several auxiliary homotopy functions that should be selected and this project using the Newton Homotopy and Fixed-Point Homotopy. The ability for these two functions are compared in solving optimization. To strengthen these findings, the project is programmed using MATLAB to implement the Newton s based Homotopy Optimization Method. The four functions of univariate and multivariate are provided for illustrative purposes. This project has succeeded to compare the ability of these two auxiliary homotopy functions in solving global optimization method.

vi ABSTRAK Kaedah pengoptimuman digunakan secara meluas dalam mekanik dan kejuruteraan, ekonomi, penyelidikan operasi dan kawalan kejuruteraan. Pelbagai kaedah telah diperkenalkan untuk menyelesaikan masalah pengoptimuman dan kebanyakannya ia akan mendapatkan optimum tempatan. Salah satu kaedah yang paling biasa digunakan adalah kaedah Newton-Raphson. Dalam kaedah ini, terdapat beberapa keadaan di mana ia tidak dapat menyelesaikan masalah pengoptimuman. Dengan bantuan homotopi, masalah yang dihadapi oleh kaedah Newton-Raphson boleh diatasi dan seterusnya menyelesaikan masalah pengoptimuman. Oleh itu, matlamat kajian ini adalah untuk menyiasat kaedah Newton-Raphson sebagai asas bagi kaedah pengoptimuman homotopi untuk mencari minimum tempatan dan juga minimum sejagat. Terdapat beberapa fungsi tambahan homotopi yang perlu dipilih dan projek ini menggunakan Newton homotopi dan Titik-Tetap homotopi. Keupayaan untuk kedua-dua fungsi dibandingkan dalam menyelesaikan pengoptimuman. Untuk menguatkan penemuan ini, projek ini diprogramkan menggunakan MATLAB untuk melaksanakan kaedah pengoptimuman homotopi berasaskan Newton. Empat fungsi univariat dan multivariat disediakan untuk tujuan ilustrasi. Projek ini telah berjaya untuk membandingkan keupayaan kedua-dua fungsi tambahan homotopi dalam menyelesaikan kaedah pengoptimuman sejagat.

vii TABLE OF CONTENTS CHAPTER TITLE PAGE DECLARATION DEDICATION ACKNOWLEDGEMENTS ABSTRACT ABSTRAK TABLE OF CONTENTS LIST OF TABLES LIST OF FIGURES ii iii iv v vi vii x xii 1 INTRODUCTION 1 1.1 Introduction 1 1.2 Background of the Study 3 1.3 Statement of the Problem 3 1.4 Objectives of the Study 4 1.5 Scope of the Study 4 1.6 Significance of the Study 4 2 LITERATURE REVIEWS 5 2.1 Introduction 5 2.2 Local Optimization 5 2.3 Global Optimization 7 2.4 Newton-Raphson Method 8 2.4.1 Newton-Raphson Algorithm 11 2.4.2 Stopping Criteria 11 2.4.3 Drawbacks of Newton-Raphson Method 12

viii 2.5 Newton-Raphson Method in Optimization 14 2.5.1 Hessian Matrix 15 2.6 Homotopy Theory 15 2.6.1 Homotopy Equivalence 16 2.6.2 Deformation Retraction 17 2.7 Linear Homotopy 18 2.8 Newton Homotopy 19 2.9 Fixed-Point Homotopy 20 2.10 Homotopy Continuation Method 21 2.11 Homotopy Optimization Method 22 2.12 Minimum, Maximum, and Saddle points 24 2.12.1 2D Model of Stationary Point 24 2.12.2 3D Model of Stationary Point 25 3 RESEARCH METHODOLOGY 28 3.1 Introduction 28 3.2 Research Design and Procedure 28 3.3 Homotopy Optimization Method Procedure 29 3.3.1 Homotopy Equation Formulation 29 3.4 Newton-Raphson Method for Optimization 30 3.5 Newton Based Homotopy Optimization Method 31 3.6 Global Homotopy Optimization Method 31 3.7 Flowchart of Global Homotopy Optimization Method 32 4 EXPERIMENTATION 33 4.1 Introduction 33 4.2 Numerical Example 33 5 RESULT AND DISCUSSION 37 5.1 Introduction 37 5.2 One Variable Test Function 37 5.3 Multiple Variable Test Function 43

ix 5.4 Discussion 48 5.5 Conclusion 52 5.6 Recommendation 53 REFERENCES 54 Appendix A 57 Appendix B 59

x LIST OF TABLES TABLE NO. TITLE PAGE 4.1 Value of x at different value t 36 5.1 Result in finding the global minimum for f(x) = 1 12 x4 1 6 x3 3x 2 + x. 38 5.2 Result in finding the global minimum for f(x) = sin(3x) + (1 x). 39 5.3 Result in finding the global minimum for f(x) = x sin(1/x). 40 5.4 Result in finding the global minimum for f(x) = 0.940249612 + exp( 0.1x 2 ) sin(10x) cos(8x). 41 5.5 Result in finding global minimum for f(x, y) = xy exp( x 2 y 2 ). 44 5.6 Result in finding global minimum for f(x, y) = (x 2) 2 (x + 1) 2 + (y + 1) 2 (y 2) 2. 45 5.7 Result in finding global minimum for f(x, y) = (4 2.1x 2 + x 4 /3)x 2 + xy + ( 4 + 4y 2 )y 2 46 5.8 Result in finding global minimum for f(x, y) = cos(πx) + sin(πy) + exp(xy). 47 5.9 Output for calculation of Fixed-Point Homotopy using Microsoft Excel 2007 48 5.10 Output for calculation of Newton Homotopy using Microsoft Excel 2007 50 5.11 Output using another function of start system g(x) 51

xi 5.12 Output using another function of start system g(x) with influence of x 0 52

xii LIST OF FIGURES FIGURE NO. TITLE PAGE 2.1 Runaway 12 2.2 Flatspot 13 2.3 Cycle 13 2.4 2D minimum point 24 2.5 2D maximum point 25 2.6 2D saddle point 25 2.7 3D minimum point 26 2.8 3D maximum point 26 2.9 3D saddle point 27 3.1 Flowchart for finding the global minimum using Homotopy Method 32 5.1 Graph of f(x) = 1 12 x4 1 6 x3 3x 2 + x. 38 5.2 Graph of f(x) = sin(3x) + (1 x). 39 5.3 Graph of f(x) = x sin(1/x). 40 5.4 Graph of f(x) = 0.940249612 + exp( 0.1x 2 ) sin(10x) cos(8x). 41 5.5 Movement of initial value at x = 1.2 until global minimum 42 5.6 Movement of initial value at x = 0 until global minimum 42 5.7 Graph of f(x, y) = xy exp( x 2 y 2 ). 43 5.8 Graph of f(x, y) = (x 2) 2 (x + 1) 2 + (y + 1) 2 (y 2) 2. 44

xiii 5.9 Graph of f(x, y) = (4 2.1x 2 + x 4 /3)x 2 + xy + ( 4 + 4y 2 )y 2. 46 5.10 Graph of f(x, y) = cos(πx) + sin(πy) + exp(xy). 47

CHAPTER 1 INTRODUCTION 1.1 Introduction Optimization theory is one of the oldest branches of mathematics. Rothlauf (2011) wrote that with the emergence of computer around the 1940 s, the first optimization method is solved using computer was introduced by Gauss which is the Gaussian elimination method. In recent years, optimization has received enormous attention and it is an active area in the field of operational research, computational and applied mathematics. This is due to the rapid advances in computer technology, for example in computer hardware component such as high-speed processors, large capacity random-access memory (RAM) and others. Not forgetting the development computer software application, we now have a variety of user-friendly software. For example of the computer hardware, this research used quad-core processor that has four independent actual central processing units (called cores ), which are the units that read and execute program instructions. Mathematical software used is MATLAB R2011a where it is a high-level language and interactive environment for numerical computation, visualization and programming. Optimization studies the mathematical problem of minimize and maximize a given objective function. It can be divided into unconstrained optimization and constrained optimization. Therefore optimization is focusing on providing the best possible solutions to systems described by the mathematical model.

2 Nowadays, optimization techniques are widely used in areas of industrial operations, engineering design and control, business and financial management, data analysis, medical imaging and treatment, to mention just a few. The people who work in this area have always been interested to design optimization. For example, the optimization is used to help make decisions in financial portfolio management where the objective is to maximize profit, while a constraint is to keep a certain measurement of investment risks below some given tolerable level. From this example, we can model the problem into equations and plot the graph so that we can easily find the maximum or minimum value. But in real-life problems, there are many variables that are included and the situation cannot be formulated in linear form. Therefore it usually has a large number of local minimum and maximum. Finding an arbitrary local optimum is simply achieved by using local optimization methods. The general criteria to find the local minimum is by setting the gradient equal to zero and Hessian matrix is positive definite. But in global optimization there is no such general criterion for declaring that global minimum has been reached. Therefore, finding the global maximum or minimum of a function is a lot more challenging. Finding a global minimizer of a function is one of the most interesting areas in nonlinear problems. Determine the minimum point among the local minima in the area interest is the goal of global minimization (Liberti, 2008). Method that first used in global optimization is deterministic techniques, mostly based on the divideand-conquer principle. This was introduced in the late 1950 s with the advent of the computers into the research environment. Stochastic techniques based on adaptive random search appeared between the 1970 s and early 1980 s. The slow pace progress in continuous global optimization due to computational method was very expensive at that time until the 1990 s where computer hardware with the necessary power becomes available. Since the beginning of the 1990 s, the optimization research community has witnessed an explosion of papers, books, algorithms, software packages and resources concerning deterministic and stochastic global optimization.

3 1.2 Background of the Study Global homotopy optimization methods have been developed to find all local minimizers of a function (Diener, 1995); however, due to the amount of computation required in these methods, they are typically only applicable to problems with a small number of local minimizers. In case of any real life problems one should employ global numerical methods to avoid initial value problems. Using numerical algorithms to solve polynomial systems with tools originating from algebraic geometry is the main activity in the so called Numerical Algebraic Geometry. This is a new developing field for the crossroads of algebraic geometry, numerical analysis, computer science and engineering. Homotopy continuation method is a global numerical method to solve not only polynomial systems, but also a nonlinear system in general. Many analytical approaches are local search that finds local minimum and continues to the global minimum. In order to find global minimum, one needs to find the local minimum first. Local search have the tendency to be stuck in local minima because they greatly depend on initial solution. Therefore to find the global optimum, researchers try to find tools or methods to help the local searches. In this research, the local search method that used is Newton-Raphson method (also known as Newton s method). This method is a powerful technique due to the basis for the most effective procedures in linear and nonlinear programming. Newton- Raphson method used to find the local minimum and then injected into the homotopy optimization method in order to find the global minimum. 1.3 Statement of the Problem In this project, the homotopy optimization method is applied in solving nonlinear unconstrained minimization problem. This study focused on examining ability of the variants of homotopy function in optimization to locate the global minimizer.

4 1.4 Objectives of the Study The main objectives of this study are: (i) To determine the ability of Newton Homotopy and Fixed-Point Homotopy in solving optimization problems. (ii) To apply the properties of the homotopy optimization method to locate the local extremum and then determine the global minimizer. (iii) To run and solve optimization for one variable and two variable test function by using MATLAB software. (iv) To analyze the results of simulation. 1.5 Scope of the Study In this study, the method used for optimization is the homotopy optimization method. This method need any local minimization method to minimize the function. Therefore the chosen method is Newton-Raphson method. The variants of homotopy were focusing on Newton Homotopy and Fixed-Point Homotopy. Several test functions were tested using the homotopy optimization method to optimize and locate the local extremum and then determine the global minimum. By using MATLAB R2011a, computer programming performed for the homotopy method and Newton- Raphson method. 1.6 Significance of the Study Usually Homotopy used to overcome divergence in Newton-Raphson method for finding root. Therefore, this study is useful in applying homotopy in solving optimization and examine the ability of the variants of homotopy function. The result from this study can be references for future study in optimization.

54 REFERENCES Basener, W. (2011). Topology and Its Applications. Pure And Applied Mathematics. Wiley. Bector, C., Chandra, S. and Dutta, J. (2005). Principles of optimization theory. Alpha Science International Ltd. Brandimarte, P. (2006). Numerical Methods in Finance and Economics: A MATLAB- Based Introduction. Statistics in Practice. Wiley. Butt, R. (2009). Introduction to Numerical Analysis Using MATLAB. Infinity Science Series. Jones & Bartlett Learning. Cheney, E. and Kincaid, D. (2012). Numerical Mathematics and Computing. Brooks/Cole. Chong, E. and Zak, S. (2011). An Introduction to Optimization. Wiley Series in Discrete Mathematics and Optimization. Wiley. Diener, I. (1995). Trajectory methods in global optimization. Handbook of Global optimization. 2: 649 668. Diwekar, U. (2008). Introduction to Applied Optimization. Springer Optimization and Its Applications. 2 Edition. Springer. Dunlavy, D. M. and O Leary, D. P. (2005). Homotopy Optimization Methods for Global Optimization. Technical Report SAND2005-7495. Sandia National Laboratories. Gomez, S. and Levy, A. (1982). The tunnelling method for solving the constrained global optimization problem with several non-connected feasible regions. Numerical analysis. 909: 34 47.

55 Gray, B. (1975). Homotopy Theory: An Introduction to Algebraic Topology. Pure and Applied Mathematics. Academic Press. Hasan, T. (2009). The Application of Homotopy Method In Solving Electrical Circuit Design Problem. World Academy of Science, Engineering and Technology. 49: 952 954. Judd, K. (1998). Numerical Methods in Economics. Mit Press. Lawson, T. (2006). Topology: A Geometric Approach. Oxford Graduate Texts in Mathematics Series. Oxford University Press. Li, T. (1997). Numerical solution of multivariate polynomial systems by homotopy continuation methods. Acta numerica. 6(1): 399 436. Liberti, L. (2008). Introduction to global optimization. Technical Report. LIX, École Polytechnique. Martí, R. (2003). Multi-start methods. Handbook of metaheuristics. 57: 355 368. Michalewicz, Z. and Fogel, D. (2004). How to Solve It: Modern Heuristics. Springer. Monahan, J. (2011). Numerical Methods of Statistics. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press. Roberts, C. (2010). Ordinary Differential Equations: Applications, Models, and Computing. Textbooks in Mathematics Series. Taylor & Francis Group. Rothlauf, F. (2011). Design of Modern Heuristics: Principles and Application. Natural Computing Series. Springer. Shick, P. (2011). Topology: Point-Set and Geometric. Pure and Applied Mathematics: A Wiley Series of Texts, Monographs and Tracts. Wiley. Wu, T. (2005). A study of convergence on the Newton-homotopy continuation method. Applied mathematics and computation. 168(2): 1169 1174. Yamamura, K. and Kuroki, W. (2006). An efficient and globally convergent homotopy method for finding DC operating points of nonlinear circuits. In Proceedings of

56 the 2006 Asia and South Pacific Design Automation Conference. IEEE Press. 408 415. Yang, W., Cao, W., Chung, T. and Morris, J. (2005). Applied Numerical Methods Using MATLAB. Wiley. Yang, X. (2008). Introduction To Computational Mathematics. World Scientific Pub. Žilinskas, J. (2009). Parallel global optimization in multidimensional scaling. Parallel Scientific Computing and Optimization. 27: 69 82.