Multivariate Numerical Optimization
|
|
- Emmeline Abigail Cook
- 6 years ago
- Views:
Transcription
1 Jianxin Wei March 1, 2013
2 Outline 1 Graphics for Function of Two Variables 2 Nelder-Mead Simplex Method 3 Steepest Descent Method 4 Newton s Method 5 Quasi-Newton s Method 6 Built-in R Function 7 Linear Programming
3 Graphics for Function of Two Variables Outline 1 Graphics for Function of Two Variables 2 Nelder-Mead Simplex Method 3 Steepest Descent Method 4 Newton s Method 5 Quasi-Newton s Method 6 Built-in R Function 7 Linear Programming
4 Graphics for Function of Two Variables Built-in R Functions persp() Draws perspective plots. You can see the function from different angles by changing the values of theta and phi contour() Create a contour plot or add contour lines to an existing plot.
5 Graphics for Function of Two Variables f <- function(x,y) 0.5*x^2+2.5*y^2 x <- seq(-6,6,length=51) y <- seq(-3,3,length=51) z <- outer(x,y,f) persp(x, y, z, theta = 30, phi =0, expand = 0.5, col = "lightblue") z x f(x)=0.5*x^2+2.5*y^2 y
6 z x y z x z Multivariate Numerical Optimization Graphics for Function of Two Variables theta=30,phi=0 theta=90,phi=0 x y theta=90,phi=45 theta=45,phi= 30 z y y x
7 Graphics for Function of Two Variables The contour plot for f(x)=0.5*x^2+2.5*y^2 f <- function(x,y){ 0.5*x^2+2.5*y^2 } x <- seq(-6,6,length=51) y <- seq(-3,3,length=51) z <- outer(x,y,f) contour(x,y,z)
8 Graphics for Function of Two Variables colored contour plot of 0.5x^2+2.5y^2 f <- function(x,y){ 0.5*x^2+2.5*y^2 } x <- seq(-6,6,length=51) y <- seq(-3,3,length=51) z <- outer(x,y,f) filled.contour(x,y,z, color = terrain.colors)
9 Graphics for Function of Two Variables Plot the function in (interactive)3d Plot the function in (interactive)3d: 1 Library the package rgl 2 open3d() 3 persp3d(x,y,z)
10 Nelder-Mead Simplex Method Outline 1 Graphics for Function of Two Variables 2 Nelder-Mead Simplex Method 3 Steepest Descent Method 4 Newton s Method 5 Quasi-Newton s Method 6 Built-in R Function 7 Linear Programming
11 Nelder-Mead Simplex Method Outline of Nelder-Mead simplex Method 1 For an p dimensional function, it starts with p + 1 points x 1,..., x p+1, arranged so that they enclose a nonzero volume. For example, in two dimensions, it starts with three points, and the three points should form a triangle. 2 The p + 1 points are labeled in order from smallest to largest values of f(x i ), so that f(x 1 ) f(x 2 ) f(x p+1 ) 3 Drop x p+1 and replace it with a point that gives a smaller value. The new point is chosen from the four proposals according the value of f(z 1 ), where z 1 is the reflection point of x p+1 through x mid.
12 Nelder-Mead Simplex Method Nelder-Mead simplex in two dimensions If f(z 1 ) < f(x 1 ), i.e. f(z 1 ) falls in Region A, calculate z 2 by reflection and expansion: If f(z 2 ) < f(z 1 ), replace x 3 with z 2 (Reflection and Expansion). If f(z 2 ) f(z 1 ), replace x 3 with z 1 (Reflection).
13 Nelder-Mead Simplex Method Nelder-Mead simplex in two dimensions If f(z 1 ) < f(x 1 ), i.e. f(z 1 ) falls in Region A, calculate z 2 by reflection and expansion: If f(z 2 ) < f(z 1 ), replace x 3 with z 2 (Reflection and Expansion). If f(z 2 ) f(z 1 ), replace x 3 with z 1 (Reflection). If f(x 1 ) f(z 1 ) < f(x 2 ), i.e. f(z 1 ) falls in Region B, replace x 3 with z 1 (Reflection).
14 Nelder-Mead Simplex Method Nelder-Mead simplex in two dimensions If f(z 1 ) < f(x 1 ), i.e. f(z 1 ) falls in Region A, calculate z 2 by reflection and expansion: If f(z 2 ) < f(z 1 ), replace x 3 with z 2 (Reflection and Expansion). If f(z 2 ) f(z 1 ), replace x 3 with z 1 (Reflection). If f(x 1 ) f(z 1 ) < f(x 2 ), i.e. f(z 1 ) falls in Region B, replace x 3 with z 1 (Reflection). If f(x 2 ) f(z 1 ) < f(x 3 ), i.e. f(z 1 ) falls in Region C, Swap z 1 with x 3.
15 Nelder-Mead Simplex Method Nelder-Mead simplex in two dimensions If f(z 1 ) < f(x 1 ), i.e. f(z 1 ) falls in Region A, calculate z 2 by reflection and expansion: If f(z 2 ) < f(z 1 ), replace x 3 with z 2 (Reflection and Expansion). If f(z 2 ) f(z 1 ), replace x 3 with z 1 (Reflection). If f(x 1 ) f(z 1 ) < f(x 2 ), i.e. f(z 1 ) falls in Region B, replace x 3 with z 1 (Reflection). If f(x 2 ) f(z 1 ) < f(x 3 ), i.e. f(z 1 ) falls in Region C, Swap z 1 with x 3. If f(z 1 ) f(x 3 ), i.e. f(z 1 ) falls in Region D, calculate z 3 by contraction 1: If f(z 3 ) < f(x 3 ), replace x 3 with z 3 (Contraction 1). If f(z 3 ) f(x 3 ), replace x 2, x 3 with x 4, x 5 (Contraction 2).
16 Nelder-Mead Simplex Method Remarks for Nelder-Mead Simplex Method Does not use the derivatives Makes no use of the objective function values other than to compare them (Golden section search share this property) May be rather slow Convergence is not guaranteed Effective when p is small The default method in built-in R function optim()
17 Steepest Descent Method Outline 1 Graphics for Function of Two Variables 2 Nelder-Mead Simplex Method 3 Steepest Descent Method 4 Newton s Method 5 Quasi-Newton s Method 6 Built-in R Function 7 Linear Programming
18 Steepest Descent Method Steepest descent Steepest descent method is one of the oldest and simplest methods for multidimensional optimization. The method can be outlined as: Choose a staring point, search in the direction which the function value goes downhill, f(x). Find the minimum in that direction, calculate the f(x), and search in the new direction. Algorithm: Steepest Descent x 0 =initial guess for k = 0, 1, 2,... s k = f(x k ) {Compute negative Gradient} Choose α k to minimize f(x k + α k s k ){perform line search} x k+1 = x k + α k s k {update solution} end
19 Steepest Descent Method steepest descent method
20 Steepest Descent Method Remarks Very reliable in that it can always make progress provided the gradient is nonzero. Not effective, the convergence rate is only linear.
21 Newton s Method Outline 1 Graphics for Function of Two Variables 2 Nelder-Mead Simplex Method 3 Steepest Descent Method 4 Newton s Method 5 Quasi-Newton s Method 6 Built-in R Function 7 Linear Programming
22 Newton s Method Newton s Method Idea: Approximate the function by a quadratic function. Algorithm: Newton s Method x 0 =initial guess for k = 0, 1, 2,... x k+1 = x k H 1 f (x k) f(x k ) {update solution} end
23 Quasi-Newton s Method Outline 1 Graphics for Function of Two Variables 2 Nelder-Mead Simplex Method 3 Steepest Descent Method 4 Newton s Method 5 Quasi-Newton s Method 6 Built-in R Function 7 Linear Programming
24 Quasi-Newton s Method Quasi-Newton Methods Variants of Newton s method have been developed to reduce its overhead or improve its reliability, or both. Quasi-Newton methods have the general form x k+1 = x k α k B 1 k f(x k), where α k is a line search parameter and B k is some approximation of the Hessian matrix. Many quasi-newton methods are more robust than the pure Newton method and have considerably lower overhead per iteration, yet remain superlinearly convergent.
25 Quasi-Newton s Method BFGS method A quasi-newton method: builds up approximation to Hessian matrix from gradients at start and finish of steps Update term for the Hessian approximation is due to Broyden, Fletcher, Goldfarb and Shanno (proposed separately by all four in 1970) Uses derivative information, calculated either from a user-supplied function or by finite differences If dimension is large, the matrix stored may be very large
26 Quasi-Newton s Method Algorithm: BGFS Method for Unconstrained Optimization x 0 =initial guess B 0 =initial Hessian approximation for k = 0, 1, 2,... x k+1 = x k B 1 f (x k) f(x k ) {update solution} y k = f(x k+1 ) f(x k ) B k+1 = B k + (y k yk T )/(yt k s k) {update approximate Hessian} (B k s k s T k B k)/(s T k B ks k ) end
27 Quasi-Newton s Method CG method A conjugate gradient method: chooses successive search directions that are analogous to axes of an ellipse Does not store a Hessian matrix Suitable for very large problems. Less robust than BFGS method Uses derivative information, calculated either from a user-supplied function or by finite differences
28 Quasi-Newton s Method Algorithm: Conjugate Gradient Method for Unconstrained Optimization x 0 =initial guess g 0 = f(x 0 ) s 0 = g 0 for k = 0, 1, 2,... Choose α k to minimize f(x k + α k s k ) { perform line search} x k+1 = x k + α k s k { update solution } g k+1 = f(x k+1 ) {Compute new gradient} β k+1 = (gk+1 T g k+1)/(gk T g k) s k+1 = g k+1 + β k+1 s k {modify gradient} end
29 Quasi-Newton s Method L-BFGS-B method A limited memory version of BFGS Does not store a Hessian matrix, only a limited number of update steps for it Uses derivative information, calculated either from a user-supplied function or by finite differences Can restrict the solution to lie within a box, the only method of optim() that can do this
30 Built-in R Function Outline 1 Graphics for Function of Two Variables 2 Nelder-Mead Simplex Method 3 Steepest Descent Method 4 Newton s Method 5 Quasi-Newton s Method 6 Built-in R Function 7 Linear Programming
31 Built-in R Function General Optimization R has several different functions: most flexible is optim() which includes several different algorithms. Algorithm of choice depends on how easy it is to calculate derivatives for the function and the dimensions of the objective function. Usually better to supply a function to calculate derivatives, but may be unnecessary extra work. Experiment with different methods...
32 Built-in R Function optim() example We want to minimize f(x 1, x 2 ) = 0.5x x2 2 f <- function (x) 0.5*x[1]^2+2.5*x[2]^2 Gr <- function(x) c(x[1],5*x[2]) optim(c(1,2),f) optim(c(1,2),f,gr=gr, method="bfgs") optim(c(1,2),f,method="bfgs") optim(c(1,2),f,method="cg")... Read the help document of optim().
33 Built-in R Function constroptim() Example Minimize f(x 1, x 2 ) = 0.5x x2 2, subject to x 1 + x 2 > 1 f <- function (x) x[1]^2+x[2]^2 constroptim(c(1,2),f,gr=null,ui=matrix(c(1,1),1,2),ci=1) The feasible region is defined by ui % %theta ci >= 0. The starting value must be in the interior of the feasible region, but the minimum may be on the boundary. ui is the constraint matrix ci is the constraint vector theta is the vector of function variables
34 Linear Programming Outline 1 Graphics for Function of Two Variables 2 Nelder-Mead Simplex Method 3 Steepest Descent Method 4 Newton s Method 5 Quasi-Newton s Method 6 Built-in R Function 7 Linear Programming
35 Linear Programming Standard form for linear programming Minimize subject to the constraints C(x) = c 1 x c k x k a 11 x a 1k b 1 a 21 x a 2k b 2... a m1 x a mk b m and the nonnegativity conditions x 1 0,..., x k 0.
36 Linear Programming Linear programming in R Consider the linear programming problem: Minimize subject to the constraints and x 1, x 2 0. In R, this can be solved as 5x 1 + 8x 2 x 1 + x 2 2 x 2 + 2x 2 3 install.packages("lpsolve") library(lpsolve) lp(objective.in=c(5,8),const.mat=matrix(c(1,1,1,2),2,2), const.rhs=c(2,3),const.dir=c(">=",">="))
Lecture 6 - Multivariate numerical optimization
Lecture 6 - Multivariate numerical optimization Björn Andersson (w/ Jianxin Wei) Department of Statistics, Uppsala University February 13, 2014 1 / 36 Table of Contents 1 Plotting functions of two variables
More informationModern Methods of Data Analysis - WS 07/08
Modern Methods of Data Analysis Lecture XV (04.02.08) Contents: Function Minimization (see E. Lohrmann & V. Blobel) Optimization Problem Set of n independent variables Sometimes in addition some constraints
More informationConstrained and Unconstrained Optimization
Constrained and Unconstrained Optimization Carlos Hurtado Department of Economics University of Illinois at Urbana-Champaign hrtdmrt2@illinois.edu Oct 10th, 2017 C. Hurtado (UIUC - Economics) Numerical
More informationIntroduction to Optimization Problems and Methods
Introduction to Optimization Problems and Methods wjch@umich.edu December 10, 2009 Outline 1 Linear Optimization Problem Simplex Method 2 3 Cutting Plane Method 4 Discrete Dynamic Programming Problem Simplex
More informationToday. Golden section, discussion of error Newton s method. Newton s method, steepest descent, conjugate gradient
Optimization Last time Root finding: definition, motivation Algorithms: Bisection, false position, secant, Newton-Raphson Convergence & tradeoffs Example applications of Newton s method Root finding in
More informationNumerical Optimization
Numerical Optimization Quantitative Macroeconomics Raül Santaeulàlia-Llopis MOVE-UAB and Barcelona GSE Fall 2018 Raül Santaeulàlia-Llopis (MOVE-UAB,BGSE) QM: Numerical Optimization Fall 2018 1 / 46 1 Introduction
More informationIntroduction to Optimization
Introduction to Optimization Second Order Optimization Methods Marc Toussaint U Stuttgart Planned Outline Gradient-based optimization (1st order methods) plain grad., steepest descent, conjugate grad.,
More informationEllipsoid Algorithm :Algorithms in the Real World. Ellipsoid Algorithm. Reduction from general case
Ellipsoid Algorithm 15-853:Algorithms in the Real World Linear and Integer Programming II Ellipsoid algorithm Interior point methods First polynomial-time algorithm for linear programming (Khachian 79)
More informationLogistic Regression
Logistic Regression ddebarr@uw.edu 2016-05-26 Agenda Model Specification Model Fitting Bayesian Logistic Regression Online Learning and Stochastic Optimization Generative versus Discriminative Classifiers
More informationIntroduction to optimization methods and line search
Introduction to optimization methods and line search Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi How to find optimal solutions? Trial and error widely used in practice, not efficient and
More informationOptimization. (Lectures on Numerical Analysis for Economists III) Jesús Fernández-Villaverde 1 and Pablo Guerrón 2 February 20, 2018
Optimization (Lectures on Numerical Analysis for Economists III) Jesús Fernández-Villaverde 1 and Pablo Guerrón 2 February 20, 2018 1 University of Pennsylvania 2 Boston College Optimization Optimization
More informationA Study on the Optimization Methods for Optomechanical Alignment
A Study on the Optimization Methods for Optomechanical Alignment Ming-Ta Yu a, Tsung-Yin Lin b *, Yi-You Li a, and Pei-Feng Shu a a Dept. of Mech. Eng., National Chiao Tung University, Hsinchu 300, Taiwan,
More informationTheoretical Concepts of Machine Learning
Theoretical Concepts of Machine Learning Part 2 Institute of Bioinformatics Johannes Kepler University, Linz, Austria Outline 1 Introduction 2 Generalization Error 3 Maximum Likelihood 4 Noise Models 5
More information25. NLP algorithms. ˆ Overview. ˆ Local methods. ˆ Constrained optimization. ˆ Global methods. ˆ Black-box methods.
CS/ECE/ISyE 524 Introduction to Optimization Spring 2017 18 25. NLP algorithms ˆ Overview ˆ Local methods ˆ Constrained optimization ˆ Global methods ˆ Black-box methods ˆ Course wrap-up Laurent Lessard
More informationExperimental Data and Training
Modeling and Control of Dynamic Systems Experimental Data and Training Mihkel Pajusalu Alo Peets Tartu, 2008 1 Overview Experimental data Designing input signal Preparing data for modeling Training Criterion
More informationAPPLIED OPTIMIZATION WITH MATLAB PROGRAMMING
APPLIED OPTIMIZATION WITH MATLAB PROGRAMMING Second Edition P. Venkataraman Rochester Institute of Technology WILEY JOHN WILEY & SONS, INC. CONTENTS PREFACE xiii 1 Introduction 1 1.1. Optimization Fundamentals
More informationEfficient Tuning of SVM Hyperparameters Using Radius/Margin Bound and Iterative Algorithms
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 13, NO. 5, SEPTEMBER 2002 1225 Efficient Tuning of SVM Hyperparameters Using Radius/Margin Bound and Iterative Algorithms S. Sathiya Keerthi Abstract This paper
More informationNewton and Quasi-Newton Methods
Lab 17 Newton and Quasi-Newton Methods Lab Objective: Newton s method is generally useful because of its fast convergence properties. However, Newton s method requires the explicit calculation of the second
More informationMachine Learning for Signal Processing Lecture 4: Optimization
Machine Learning for Signal Processing Lecture 4: Optimization 13 Sep 2015 Instructor: Bhiksha Raj (slides largely by Najim Dehak, JHU) 11-755/18-797 1 Index 1. The problem of optimization 2. Direct optimization
More informationIntroduction. Optimization
Introduction to Optimization Amy Langville SAMSI Undergraduate Workshop N.C. State University SAMSI 6/1/05 GOAL: minimize f(x 1, x 2, x 3, x 4, x 5 ) = x 2 1.5x 2x 3 + x 4 /x 5 PRIZE: $1 million # of independent
More informationComputational Methods. Constrained Optimization
Computational Methods Constrained Optimization Manfred Huber 2010 1 Constrained Optimization Unconstrained Optimization finds a minimum of a function under the assumption that the parameters can take on
More informationOptimization. Industrial AI Lab.
Optimization Industrial AI Lab. Optimization An important tool in 1) Engineering problem solving and 2) Decision science People optimize Nature optimizes 2 Optimization People optimize (source: http://nautil.us/blog/to-save-drowning-people-ask-yourself-what-would-light-do)
More informationEnergy Minimization -Non-Derivative Methods -First Derivative Methods. Background Image Courtesy: 3dciencia.com visual life sciences
Energy Minimization -Non-Derivative Methods -First Derivative Methods Background Image Courtesy: 3dciencia.com visual life sciences Introduction Contents Criteria to start minimization Energy Minimization
More informationChapter Multidimensional Gradient Method
Chapter 09.04 Multidimensional Gradient Method After reading this chapter, you should be able to: 1. Understand how multi-dimensional gradient methods are different from direct search methods. Understand
More informationParameters Estimation of Material Constitutive Models using Optimization Algorithms
The University of Akron IdeaExchange@UAkron Honors Research Projects The Dr. Gary B. and Pamela S. Williams Honors College Spring 2015 Parameters Estimation of Material Constitutive Models using Optimization
More informationLecture 12: convergence. Derivative (one variable)
Lecture 12: convergence More about multivariable calculus Descent methods Backtracking line search More about convexity (first and second order) Newton step Example 1: linear programming (one var., one
More informationOptimization in Scilab
Scilab sheet Optimization in Scilab Scilab provides a high-level matrix language and allows to define complex mathematical models and to easily connect to existing libraries. That is why optimization is
More informationMATH3016: OPTIMIZATION
MATH3016: OPTIMIZATION Lecturer: Dr Huifu Xu School of Mathematics University of Southampton Highfield SO17 1BJ Southampton Email: h.xu@soton.ac.uk 1 Introduction What is optimization? Optimization is
More informationLaboratory exercise. Laboratory experiment OPT-1 Nonlinear Optimization
Fachgebiet Simulation und Optimale Prozesse Fakultät für Informatik und Automatisierung Institut für Automatisierungsund Systemtechnik Laboratory exercise Laboratory experiment OPT-1 Nonlinear Optimization
More informationDavid G. Luenberger Yinyu Ye. Linear and Nonlinear. Programming. Fourth Edition. ö Springer
David G. Luenberger Yinyu Ye Linear and Nonlinear Programming Fourth Edition ö Springer Contents 1 Introduction 1 1.1 Optimization 1 1.2 Types of Problems 2 1.3 Size of Problems 5 1.4 Iterative Algorithms
More informationGradient, Newton and conjugate direction methods for unconstrained nonlinear optimization
Gradient, Newton and conjugate direction methods for unconstrained nonlinear optimization Consider the gradient method (steepest descent), with exact unidimensional search, the Newton method and the conjugate
More informationConvex Optimization CMU-10725
Convex Optimization CMU-10725 Conjugate Direction Methods Barnabás Póczos & Ryan Tibshirani Conjugate Direction Methods 2 Books to Read David G. Luenberger, Yinyu Ye: Linear and Nonlinear Programming Nesterov:
More informationCS281 Section 3: Practical Optimization
CS281 Section 3: Practical Optimization David Duvenaud and Dougal Maclaurin Most parameter estimation problems in machine learning cannot be solved in closed form, so we often have to resort to numerical
More informationOptimization with Scipy
Lab 15 Optimization with Scipy Lab Objective: The Optimize package in Scipy provides highly optimized and versatile methods for solving fundamental optimization problems. In this lab we introduce the syntax
More informationIntroduction to unconstrained optimization - derivative-free methods
Introduction to unconstrained optimization - derivative-free methods Jussi Hakanen Post-doctoral researcher Office: AgC426.3 jussi.hakanen@jyu.fi Learning outcomes To understand the basic principles of
More informationNumerical Optimization: Introduction and gradient-based methods
Numerical Optimization: Introduction and gradient-based methods Master 2 Recherche LRI Apprentissage Statistique et Optimisation Anne Auger Inria Saclay-Ile-de-France November 2011 http://tao.lri.fr/tiki-index.php?page=courses
More informationA Brief Look at Optimization
A Brief Look at Optimization CSC 412/2506 Tutorial David Madras January 18, 2018 Slides adapted from last year s version Overview Introduction Classes of optimization problems Linear programming Steepest
More informationTested Paradigm to Include Optimization in Machine Learning Algorithms
Tested Paradigm to Include Optimization in Machine Learning Algorithms Aishwarya Asesh School of Computing Science and Engineering VIT University Vellore, India International Journal of Engineering Research
More informationA large number of user subroutines and utility routines is available in Abaqus, that are all programmed in Fortran. Subroutines are different for
1 2 3 A large number of user subroutines and utility routines is available in Abaqus, that are all programmed in Fortran. Subroutines are different for implicit (standard) and explicit solvers. Utility
More information06: Logistic Regression
06_Logistic_Regression 06: Logistic Regression Previous Next Index Classification Where y is a discrete value Develop the logistic regression algorithm to determine what class a new input should fall into
More information10.7 Variable Metric Methods in Multidimensions
10.7 Variable Metric Methods in Multidimensions 425 *fret=dbrent(ax,xx,bx,f1dim,df1dim,tol,&xmin); for (j=1;j
More informationINTRODUCTION TO LINEAR AND NONLINEAR PROGRAMMING
INTRODUCTION TO LINEAR AND NONLINEAR PROGRAMMING DAVID G. LUENBERGER Stanford University TT ADDISON-WESLEY PUBLISHING COMPANY Reading, Massachusetts Menlo Park, California London Don Mills, Ontario CONTENTS
More informationOPTIMIZATION FOR AUTOMATIC HISTORY MATCHING
INTERNATIONAL JOURNAL OF NUMERICAL ANALYSIS AND MODELING Volume 2, Supp, Pages 131 137 c 2005 Institute for Scientific Computing and Information OPTIMIZATION FOR AUTOMATIC HISTORY MATCHING Abstract. SHUGUANG
More informationClassical Gradient Methods
Classical Gradient Methods Note simultaneous course at AMSI (math) summer school: Nonlin. Optimization Methods (see http://wwwmaths.anu.edu.au/events/amsiss05/) Recommended textbook (Springer Verlag, 1999):
More informationM. Sc. (Artificial Intelligence and Machine Learning)
Course Name: Advanced Python Course Code: MSCAI 122 This course will introduce students to advanced python implementations and the latest Machine Learning and Deep learning libraries, Scikit-Learn and
More information10.6 Conjugate Gradient Methods in Multidimensions
420 Chapter 10. Minimization or Maximization of Functions CITED REFERENCES AND FURTHER READING: Brent, R.P. 1973, Algorithms for Minimization without Derivatives (Englewood Cliffs, NJ: Prentice- Hall),
More informationOptimization. 1. Optimization. by Prof. Seungchul Lee Industrial AI Lab POSTECH. Table of Contents
Optimization by Prof. Seungchul Lee Industrial AI Lab http://isystems.unist.ac.kr/ POSTECH Table of Contents I. 1. Optimization II. 2. Solving Optimization Problems III. 3. How do we Find x f(x) = 0 IV.
More informationMulti Layer Perceptron trained by Quasi Newton learning rule
Multi Layer Perceptron trained by Quasi Newton learning rule Feed-forward neural networks provide a general framework for representing nonlinear functional mappings between a set of input variables and
More informationAlgoritmi di Ottimizzazione: Parte A Aspetti generali e algoritmi classici
Identificazione e Controllo Intelligente Algoritmi di Ottimizzazione: Parte A Aspetti generali e algoritmi classici David Naso A.A. 2006-2007 Identificazione e Controllo Intelligente 1 Search and optimization
More informationA Derivative-Free Approximate Gradient Sampling Algorithm for Finite Minimax Problems
1 / 33 A Derivative-Free Approximate Gradient Sampling Algorithm for Finite Minimax Problems Speaker: Julie Nutini Joint work with Warren Hare University of British Columbia (Okanagan) III Latin American
More informationNMath Analysis User s Guide
NMath Analysis User s Guide Version 2.0 CenterSpace Software Corvallis, Oregon NMATH ANALYSIS USER S GUIDE 2009 Copyright CenterSpace Software, LLC. All Rights Reserved. The correct bibliographic reference
More informationShort Reminder of Nonlinear Programming
Short Reminder of Nonlinear Programming Kaisa Miettinen Dept. of Math. Inf. Tech. Email: kaisa.miettinen@jyu.fi Homepage: http://www.mit.jyu.fi/miettine Contents Background General overview briefly theory
More informationMinima, Maxima, Saddle points
Minima, Maxima, Saddle points Levent Kandiller Industrial Engineering Department Çankaya University, Turkey Minima, Maxima, Saddle points p./9 Scalar Functions Let us remember the properties for maxima,
More informationTitle. Syntax. optimize( ) Function optimization. S = optimize init() (varies) optimize init which(s [, { "max" "min" } ] )
Title optimize( ) Function optimization Syntax S = optimize init() (varies) optimize init which(s [, { "max" "min" } ] ) (varies) optimize init evaluator(s [, &function() ] ) (varies) optimize init evaluatortype(s
More informationLecture 12: Feasible direction methods
Lecture 12 Lecture 12: Feasible direction methods Kin Cheong Sou December 2, 2013 TMA947 Lecture 12 Lecture 12: Feasible direction methods 1 / 1 Feasible-direction methods, I Intro Consider the problem
More information16. LECTURE 16. I understand how to find the rate of change in any direction. I understand in what direction the maximum rate of change happens.
6. LETURE 6 Objectives I understand how to find the rate of change in any direction. I understand in what direction the maximum rate of change happens. So far, we ve learned the definition of the gradient
More informationCharacterizing Improving Directions Unconstrained Optimization
Final Review IE417 In the Beginning... In the beginning, Weierstrass's theorem said that a continuous function achieves a minimum on a compact set. Using this, we showed that for a convex set S and y not
More informationComparison of Interior Point Filter Line Search Strategies for Constrained Optimization by Performance Profiles
INTERNATIONAL JOURNAL OF MATHEMATICS MODELS AND METHODS IN APPLIED SCIENCES Comparison of Interior Point Filter Line Search Strategies for Constrained Optimization by Performance Profiles M. Fernanda P.
More informationA Scaled Gradient Descent Method for. Unconstrained Optimiziation Problems With A. Priori Estimation of the Minimum Value
A Scaled Gradient Descent Method for Unconstrained Optimiziation Problems With A Priori Estimation of the Minimum Value A SCALED GRADIENT DESCENT METHOD FOR UNCONSTRAINED OPTIMIZIATION PROBLEMS WITH A
More informationPackage ihs. February 25, 2015
Version 1.0 Type Package Title Inverse Hyperbolic Sine Distribution Date 2015-02-24 Author Carter Davis Package ihs February 25, 2015 Maintainer Carter Davis Depends R (>= 2.4.0),
More informationThe Simplex Algorithm
The Simplex Algorithm April 25, 2005 We seek x 1,..., x n 0 which mini- Problem. mizes C(x 1,..., x n ) = c 1 x 1 + + c n x n, subject to the constraint Ax b, where A is m n, b = m 1. Through the introduction
More informationLECTURE 13: SOLUTION METHODS FOR CONSTRAINED OPTIMIZATION. 1. Primal approach 2. Penalty and barrier methods 3. Dual approach 4. Primal-dual approach
LECTURE 13: SOLUTION METHODS FOR CONSTRAINED OPTIMIZATION 1. Primal approach 2. Penalty and barrier methods 3. Dual approach 4. Primal-dual approach Basic approaches I. Primal Approach - Feasible Direction
More informationOptimization. there will solely. any other methods presented can be. saved, and the. possibility. the behavior of. next point is to.
From: http:/ //trond.hjorteland.com/thesis/node1.html Optimization As discussed briefly in Section 4.1, the problem we are facing when searching for stationaryy values of the action given in equation (4.1)
More informationSimplex of Nelder & Mead Algorithm
Simplex of N & M Simplex of Nelder & Mead Algorithm AKA the Amoeba algorithm In the class of direct search methods Unconstrained (although constraints can be added as part of error function) nonlinear
More informationAlgorithms for convex optimization
Algorithms for convex optimization Michal Kočvara Institute of Information Theory and Automation Academy of Sciences of the Czech Republic and Czech Technical University kocvara@utia.cas.cz http://www.utia.cas.cz/kocvara
More information5 Machine Learning Abstractions and Numerical Optimization
Machine Learning Abstractions and Numerical Optimization 25 5 Machine Learning Abstractions and Numerical Optimization ML ABSTRACTIONS [some meta comments on machine learning] [When you write a large computer
More informationParallel Implementation of Nudged Elastic Band Method
Parallel Implementation of Nudged Elastic Band Method.33 Final Project Anubhav Sinha December, Background Methods in computational chemistry give potential energy surfaces (PES). Purpose: find the transition
More informationNEW CERN PROTON SYNCHROTRON BEAM OPTIMIZATION TOOL
16th Int. Conf. on Accelerator and Large Experimental Control Systems ICALEPCS2017, Barcelona, Spain JACoW Publishing NEW CERN PROTON SYNCHROTRON BEAM OPTIMIZATION TOOL E. Piselli, A. Akroh CERN, Geneva,
More informationADDENDUM TO THE SEDUMI USER GUIDE VERSION 1.1
ADDENDUM TO THE SEDUMI USER GUIDE VERSION 1.1 IMRE PÓLIK 1. Introduction The main goal of this reference guide is to give a summary of all the options in SeDuMi. The default value of the options is satisfactory
More informationProgramming, numerics and optimization
Programming, numerics and optimization Lecture C-4: Constrained optimization Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428 June
More informationPARAMETER ESTIMATION OF GROUND THERMAL PROPERTIES
PARAMETER ESTIMATION OF GROUND THERMAL PROPERTIES by NAGENDRA KUMAR JAIN Bachelor of Science Indian Institute of Technology Bombay, India 1997 Submitted to the Faculty of the Graduate College of the Oklahoma
More informationContents. I Basics 1. Copyright by SIAM. Unauthorized reproduction of this article is prohibited.
page v Preface xiii I Basics 1 1 Optimization Models 3 1.1 Introduction... 3 1.2 Optimization: An Informal Introduction... 4 1.3 Linear Equations... 7 1.4 Linear Optimization... 10 Exercises... 12 1.5
More informationTitle. Description. stata.com
Title stata.com optimize( ) Function optimization Description Syntax Remarks and examples Conformability Diagnostics References Also see Description These functions find parameter vector or scalar p such
More informationIterative Algorithms I: Elementary Iterative Methods and the Conjugate Gradient Algorithms
Iterative Algorithms I: Elementary Iterative Methods and the Conjugate Gradient Algorithms By:- Nitin Kamra Indian Institute of Technology, Delhi Advisor:- Prof. Ulrich Reude 1. Introduction to Linear
More informationConvexity Theory and Gradient Methods
Convexity Theory and Gradient Methods Angelia Nedić angelia@illinois.edu ISE Department and Coordinated Science Laboratory University of Illinois at Urbana-Champaign Outline Convex Functions Optimality
More information3.3 Optimizing Functions of Several Variables 3.4 Lagrange Multipliers
3.3 Optimizing Functions of Several Variables 3.4 Lagrange Multipliers Prof. Tesler Math 20C Fall 2018 Prof. Tesler 3.3 3.4 Optimization Math 20C / Fall 2018 1 / 56 Optimizing y = f (x) In Math 20A, we
More informationHST.582J / 6.555J / J Biomedical Signal and Image Processing Spring 2007
MIT OpenCourseWare http://ocw.mit.edu HST.582J / 6.555J / 16.456J Biomedical Signal and Image Processing Spring 2007 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.
More informationIterative Methods for Solving Linear Problems
Iterative Methods for Solving Linear Problems When problems become too large (too many data points, too many model parameters), SVD and related approaches become impractical. Iterative Methods for Solving
More informationA CONJUGATE DIRECTION IMPLEMENTATION OF THE BFGS ALGORITHM WITH AUTOMATIC SCALING. Ian D Coope
i A CONJUGATE DIRECTION IMPLEMENTATION OF THE BFGS ALGORITHM WITH AUTOMATIC SCALING Ian D Coope No. 42 December 1987 A CONJUGATE DIRECTION IMPLEMENTATION OF THE BFGS ALGORITHM WITH AUTOMATIC SCALING IAN
More informationOpenGL Graphics System. 2D Graphics Primitives. Drawing 2D Graphics Primitives. 2D Graphics Primitives. Mathematical 2D Primitives.
D Graphics Primitives Eye sees Displays - CRT/LCD Frame buffer - Addressable pixel array (D) Graphics processor s main function is to map application model (D) by projection on to D primitives: points,
More informationA Brief Overview of Optimization Problems. Steven G. Johnson MIT course , Fall 2008
A Brief Overview of Optimization Problems Steven G. Johnson MIT course 18.335, Fall 2008 Why optimization? In some sense, all engineering design is optimization: choosing design parameters to improve some
More informationNeural Networks: Optimization Part 1. Intro to Deep Learning, Fall 2018
Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2018 1 Story so far Neural networks are universal approximators Can model any odd thing Provided they have the right architecture We must
More informationGeneric descent algorithm Generalization to multiple dimensions Problems of descent methods, possible improvements Fixes Local minima
1 Lecture 10: descent methods Generic descent algorithm Generalization to multiple dimensions Problems of descent methods, possible improvements Fixes Local minima Gradient descent (reminder) Minimum of
More informationTraining of Neural Networks. Q.J. Zhang, Carleton University
Training of Neural Networks Notation: x: input of the original modeling problem or the neural network y: output of the original modeling problem or the neural network w: internal weights/parameters of
More informationConstrained Optimization COS 323
Constrained Optimization COS 323 Last time Introduction to optimization objective function, variables, [constraints] 1-dimensional methods Golden section, discussion of error Newton s method Multi-dimensional
More informationDelaunay-based Derivative-free Optimization via Global Surrogate. Pooriya Beyhaghi, Daniele Cavaglieri and Thomas Bewley
Delaunay-based Derivative-free Optimization via Global Surrogate Pooriya Beyhaghi, Daniele Cavaglieri and Thomas Bewley May 23, 2014 Delaunay-based Derivative-free Optimization via Global Surrogate Pooriya
More informationA new mini-max, constrained optimization method for solving worst case problems
Carnegie Mellon University Research Showcase @ CMU Department of Electrical and Computer Engineering Carnegie Institute of Technology 1979 A new mini-max, constrained optimization method for solving worst
More informationAccelerating the Hessian-free Gauss-Newton Full-waveform Inversion via Preconditioned Conjugate Gradient Method
Accelerating the Hessian-free Gauss-Newton Full-waveform Inversion via Preconditioned Conjugate Gradient Method Wenyong Pan 1, Kris Innanen 1 and Wenyuan Liao 2 1. CREWES Project, Department of Geoscience,
More informationarxiv: v1 [cs.na] 28 Dec 2018
arxiv:1812.10986v1 [cs.na] 28 Dec 2018 Vilin: Unconstrained Numerical Optimization Application Marko Miladinović 1, Predrag Živadinović 2, 1,2 University of Niš, Faculty of Sciences and Mathematics, Department
More informationBackward facing step Homework. Department of Fluid Mechanics. For Personal Use. Budapest University of Technology and Economics. Budapest, 2010 autumn
Backward facing step Homework Department of Fluid Mechanics Budapest University of Technology and Economics Budapest, 2010 autumn Updated: October 26, 2010 CONTENTS i Contents 1 Introduction 1 2 The problem
More informationCPSC 340: Machine Learning and Data Mining. Robust Regression Fall 2015
CPSC 340: Machine Learning and Data Mining Robust Regression Fall 2015 Admin Can you see Assignment 1 grades on UBC connect? Auditors, don t worry about it. You should already be working on Assignment
More informationMethods of Optimization for Numerical Algorithms
faculty of science and engineering mathematics and applied mathematics Methods of Optimization for Numerical Algorithms Bachelor s Project Mathematics July 2017 Student: S.J. Petersen First supervisor:
More informationIntroduction to Design Optimization: Search Methods
Introduction to Design Optimization: Search Methods 1-D Optimization The Search We don t know the curve. Given α, we can calculate f(α). By inspecting some points, we try to find the approximated shape
More informationLinear Discriminant Functions: Gradient Descent and Perceptron Convergence
Linear Discriminant Functions: Gradient Descent and Perceptron Convergence The Two-Category Linearly Separable Case (5.4) Minimizing the Perceptron Criterion Function (5.5) Role of Linear Discriminant
More informationLinear and Integer Programming :Algorithms in the Real World. Related Optimization Problems. How important is optimization?
Linear and Integer Programming 15-853:Algorithms in the Real World Linear and Integer Programming I Introduction Geometric Interpretation Simplex Method Linear or Integer programming maximize z = c T x
More informationMaximum Likelihood estimation: Stata vs. Gauss
Maximum Likelihood estimation: Stata vs. Gauss Index Motivation Objective The Maximum Likelihood Method Capabilities: Stata vs Gauss Conclusions Motivation Stata is a powerful and flexible statistical
More informationSolving Optimization and Inverse Problems in Remote Sensing by using Evolutionary Algorithms
Technical University Munich Faculty for civil engineering and land surveying Remote Sensing Technology Prof. Dr.-Ing. Richard Bamler Solving Optimization and Inverse Problems in Remote Sensing by using
More informationA direct search method for smooth and nonsmooth unconstrained optimization
ANZIAM J. 48 (CTAC2006) pp.c927 C948, 2008 C927 A direct search method for smooth and nonsmooth unconstrained optimization C. J. Price 1 M. Reale 2 B. L. Robertson 3 (Received 24 August 2006; revised 4
More informationComputational Optimization. Constrained Optimization Algorithms
Computational Optimization Constrained Optimization Algorithms Same basic algorithms Repeat Determine descent direction Determine step size Take a step Until Optimal But now must consider feasibility,
More informationLINE SEARCH DESCENT METHODS FOR UNCONSTRAINED MINIMIZATION
Chapter 2 LINE SEARCH DESCENT METHODS FOR UNCONSTRAINED MINIMIZATION 2.1 General line search descent algorithm for unconstrained minimization Over the last 40 years many powerful direct search algorithms
More information