Introduction to optimization methods and line search
|
|
- Christine Small
- 6 years ago
- Views:
Transcription
1 Introduction to optimization methods and line search Jussi Hakanen Post-doctoral researcher
2 How to find optimal solutions? Trial and error widely used in practice, not efficient and high possibility to miss good solutions Better to use a systematic way to find optimal solution Typically we know only function value(s) at the current trial point possibly gradients at the current trial point How can we know which solution is optimal? How can we find optimal solutions?
3 Optimality conditions How can we know that a solution is optimal? One way is to utilize optimality conditions Necessary optimality conditions = conditions that an optimal solution has to satisfy (does not guarantee optimality) Sufficient optimality conditions = conditions that guarantee optimality when satisfied 1. order conditions (1. order derivatives) and 2. order conditions (2. order derivatives)
4 Global vs. local minimizers A solution x S is a global minimizer if f x f x for all x S A solution x S is a local minimizer if there exists an ε > 0 s.t. f x f(x) for all x S where x x < ε Convexity: a local minimizer is a global minimizer Global minimizers are preferred, local minimizers are usually easier to identify
5 Solving an optimization problem Find optimal values x for the variables Problems that can be solved analytically min x 2, when x 3 x = 3 Usually impossible to solve analytically Must be solved numerically approximation of the solution In mathematical optimization a starting point is iteratively improved
6 Numerical solution Modelling mathematical model of the problem Numerical methods numerical simulation model for the mathematical model Optimization method solve the problem utilizing the numerical simulation model SO modelling simulation optimization
7 Optimization method Algorithm: a mathematical description 1. Choose a stopping parameter ε > 0, starting point x 1 and a symmetric positive definite n n matrix D 1 (e.g. D 1 = I). Set y 1 = x 1 and h = j = If f(y j ) < ε, stop. Otherwise, set d j = D f(y j ). Let λ j be a solution of min f(y j + λd j ), s.t. λ 0. Set y j+1 = y j + λ j d j. If j = n, set y 1 = x h+1 = y n+1, h = h + 1, j = 1 and repeat (2). 3. Compute D j+1. Set j = j + 1 and go to (2). Method: numerical methods included Software: a method implemented as a computer programme
8 Structure of optimization methods Typically Constraint handling converts the problem to (a series of) unconstrained problems In unconstrained optimization a search direction is determined at each iteration The best solution in the search direction is found with line search Constraint handling method Unconstrained optimization Line search
9 Local optimization methods Find a (closest) local optimum Fast Usually utilize derivatives Mathematical convergence For example Direct search methods (pattern search, Hooke & Jeeves, Nelder & Mead, ) Gradient based methods (steepest descent, Newton s method, quasi-newton method, conjugate gradient, SQP, interior point methods )
10 Global optimization methods Try to get as close to global optimum as possible No mathematical convergence Do not assume much of the problem Slow, use lots of function evaluations Heuristic, contain randomness Most well known are nature-inspired methods (TIES451 Selected topics in soft computing) based on improving a population of solutions at a time instead of a single solution
11 Hybrid methods Combination of global and local methods Try to combine the benefits of both rough estimate with a global method, fine tune with a local method Challenge: how the methods should be combined? e.g. when to switch from global to local? (speed vs. accuracy)
12 Line search What did you find out about line search?
13 Line search The idea of line search is to optimize a given function with respect to a single variable Optimization algorithms for multivariable problems generate iteratively search directions in which better solutions are found Line search is used to find these! Exact minimum is not required but an approximation of it which is within a given tolerance ε > 0 enough to know that x [a, b ] where b a < ε
14 Optimality conditions Necessary: Let f: R R be differentiable. If x is a local minimizer, then f x = 0. In addition, if f is twice continuously differentiable and x is a local minimizer, then f x 0. Sufficient: Let f: R R be twice continuously differentiable. If f x = 0 and f x > 0, then x is a strict local minimizer.
15 Examples f x = (x 2) 2 4 f x = 2x 4 f x = 2 If x = 2, then both the necessary and sufficient optimality conditions are satisfied f x = (x 2) 2 4
16 Examples f x = (x 2) 3 4 f x = 3 x 2 2 f x = 6x 12 If x = 2, then the necessary optimality conditions are satisfied although x = 2 is not a local minimizer It is a saddle point Sufficient optimality conditions are not satisfied in x = 2 f x = (x 2) 3 4
17 Note on optimality conditions If f is not differentiable, then local minimizer can be in a point where f is 1) not differentiable or 2) discontinuous f x = x
18 Finding a unimodal interval Most line search methods assume that the search is started from a unimodal interval [a, b] f is unimodal in [a, b] if there is exactly one x [a, b] s.t. for all x 1, x 2 [a, b] for which x 1 < x 2 holds If x 2 < x, then f x 1 > f(x 2 ) and If x 1 > x, then f x 1 < f(x 2 )
19 Search with fixed steps Let (A, B) be the interval where we want to find a minimum for f Compute values for f in P equally spaced points x i in (A, B) x i = A + i (B A), P+1 i = 1,, P When points x j, x j+i and x j+2 are found s.t. f x j > f x j+1 < f(x j+2 ), we know that there exist at least one local minimizer in (x j, x j+2 ) The interval can be further reduced
20 Line search methods Assume that f is unimodal in [a, b] General idea is to start reducing the interval [a, b] s.t. the minimizer is still included in it An approximation of the minimizer is found when the length of the interval is smaller than a pre-determined tolerance Line search methods can be divided into Elimination methods Interpolation methods (often use derivatives)
21 The method of bisection Elimination method 1) Choose small but significant constant 2ε > 0 and an allowable length L > 0 for the final interval. Let [a 1, b 1 ] be the original (unimodal) interval. Set h = 1. 2) If b h a h < L, stop. Minimizer x [a h, b h ]. Otherwise, compute values of f in x h = ah +b h ε and y h = ah +b h + ε. 2 3) If f x h < f(y h ), set a h+1 = a h and b h+1 = y h. Otherwise, set a h+1 = x h and b h+1 = b h. Set h = h + 1 and go to step 2). 2 a h x h y h 2ε b h
22 The method of bisection (cont.) Efficiency: Length of the interval after h iterations is 1 1 h b a + 2ε h Number of iterations required if the final length should be L is (why?) h = ln ( L 2ε )/ ln 2 b a 2ε For each iteration, the objective function is evaluated 2 times (in x h and y h ) in total 2h evaluations
23 Golden section Assume that we want to separate a sub interval (length y) from an interval of length L such that y Then, y = L 0.618L L y = L y It is said that now the interval is divided in the ratio of golden section Theorem Divide an interval [a, b] in the ratio of golden section first from right (point d) and then from left (point c). Then point c divides the interval [a, d] in the ratio of golden section and point d does the same for [c, b]. a c d b
24 Golden section search Elimination method, known also as Fibonacci search. Let C = 1) Choose an allowable length L > 0 for the final interval. Let [a 1, b 1 ] be the original (unimodal) interval. Set x 1 = a C b 1 a 1 = b 1 C(b 1 a 1 ) and y 1 = a 1 + C b 1 a 1. Compute f(x 1 ) and f(y 1 ). Set h = 1. 2) If b h a h < L, stop. Minimizer x [a h, b h ]. Otherwise, if f x h f(y h ) go to step 4). 3) Set a h+1 = x h and b h+1 = b h. Further set x h+1 = y h and y h+1 = a h+1 + C(b h+1 a h+1 ). Compute f(y h+1 ) and go to step 5). 4) Set a h+1 = a h and b h+1 = y h. Further set y h+1 = x h and x h+1 = a h+1 + (1 C)(b h+1 a h+1 ). Compute f(x h+1 ). 5) Set h = h + 1 and go to step 2). a h x h y h b h
25 Golden section search (cont.) Efficiency Length of the interval after h iterations is C h (b a) Number of iterations required if the final length should be L is (why?) h = ln ( L )/ ln C b a For each iteration (except the last), the objective function is evaluated one time (in x h+1 or y h+1 ) plus in the beginning in two points (x 1 and y 1 ) in total h + 1 evaluations
26 Quadratic interpolation Idea is to approximate f with a quadratic polynomial whose minimizer is known Taylor s second order polynomial is used: p x = f x h + f x h x x h f x h x x h 2 If f x h 0, then p(x) has a critical point in x h+1 when p (x h+1 ) = 0 x h+1 = x h f (xh ) f (x h ) Newton s method for solving f x = 0!!! Interpolation can also be applied in the case where no derivatives are available (find out the idea by yourself)
27 Programming assignment Form the pairs!!! Start programming by implementing some line search method Any programming language is ok Test your implementation with some optimization problems where you know the minimizer
28 Topic of the lectures on January 20 th & 22 nd Mon, Jan 20 th : unconstrained optimization with multiple variables, optimality conditions and methods that don t utilize gradient information (=direct search methods) Wed, Jan 22 nd : methods that utilize gradient information Study this before the lecture! Questions to be considered What kind of optimality conditions there exist? What kind of techniques direct search methods use to find a local minimizer? How gradient information is utilized?
Introduction to unconstrained optimization - derivative-free methods
Introduction to unconstrained optimization - derivative-free methods Jussi Hakanen Post-doctoral researcher Office: AgC426.3 jussi.hakanen@jyu.fi Learning outcomes To understand the basic principles of
More informationCharacterizing Improving Directions Unconstrained Optimization
Final Review IE417 In the Beginning... In the beginning, Weierstrass's theorem said that a continuous function achieves a minimum on a compact set. Using this, we showed that for a convex set S and y not
More informationToday. Golden section, discussion of error Newton s method. Newton s method, steepest descent, conjugate gradient
Optimization Last time Root finding: definition, motivation Algorithms: Bisection, false position, secant, Newton-Raphson Convergence & tradeoffs Example applications of Newton s method Root finding in
More informationOptimization. Industrial AI Lab.
Optimization Industrial AI Lab. Optimization An important tool in 1) Engineering problem solving and 2) Decision science People optimize Nature optimizes 2 Optimization People optimize (source: http://nautil.us/blog/to-save-drowning-people-ask-yourself-what-would-light-do)
More informationModern Methods of Data Analysis - WS 07/08
Modern Methods of Data Analysis Lecture XV (04.02.08) Contents: Function Minimization (see E. Lohrmann & V. Blobel) Optimization Problem Set of n independent variables Sometimes in addition some constraints
More informationIntroduction to Optimization Problems and Methods
Introduction to Optimization Problems and Methods wjch@umich.edu December 10, 2009 Outline 1 Linear Optimization Problem Simplex Method 2 3 Cutting Plane Method 4 Discrete Dynamic Programming Problem Simplex
More informationConstrained and Unconstrained Optimization
Constrained and Unconstrained Optimization Carlos Hurtado Department of Economics University of Illinois at Urbana-Champaign hrtdmrt2@illinois.edu Oct 10th, 2017 C. Hurtado (UIUC - Economics) Numerical
More informationShort Reminder of Nonlinear Programming
Short Reminder of Nonlinear Programming Kaisa Miettinen Dept. of Math. Inf. Tech. Email: kaisa.miettinen@jyu.fi Homepage: http://www.mit.jyu.fi/miettine Contents Background General overview briefly theory
More informationMATH3016: OPTIMIZATION
MATH3016: OPTIMIZATION Lecturer: Dr Huifu Xu School of Mathematics University of Southampton Highfield SO17 1BJ Southampton Email: h.xu@soton.ac.uk 1 Introduction What is optimization? Optimization is
More informationLecture 12: convergence. Derivative (one variable)
Lecture 12: convergence More about multivariable calculus Descent methods Backtracking line search More about convexity (first and second order) Newton step Example 1: linear programming (one var., one
More informationDavid G. Luenberger Yinyu Ye. Linear and Nonlinear. Programming. Fourth Edition. ö Springer
David G. Luenberger Yinyu Ye Linear and Nonlinear Programming Fourth Edition ö Springer Contents 1 Introduction 1 1.1 Optimization 1 1.2 Types of Problems 2 1.3 Size of Problems 5 1.4 Iterative Algorithms
More informationAPPLIED OPTIMIZATION WITH MATLAB PROGRAMMING
APPLIED OPTIMIZATION WITH MATLAB PROGRAMMING Second Edition P. Venkataraman Rochester Institute of Technology WILEY JOHN WILEY & SONS, INC. CONTENTS PREFACE xiii 1 Introduction 1 1.1. Optimization Fundamentals
More informationM. Sc. (Artificial Intelligence and Machine Learning)
Course Name: Advanced Python Course Code: MSCAI 122 This course will introduce students to advanced python implementations and the latest Machine Learning and Deep learning libraries, Scikit-Learn and
More informationOptimization. 1. Optimization. by Prof. Seungchul Lee Industrial AI Lab POSTECH. Table of Contents
Optimization by Prof. Seungchul Lee Industrial AI Lab http://isystems.unist.ac.kr/ POSTECH Table of Contents I. 1. Optimization II. 2. Solving Optimization Problems III. 3. How do we Find x f(x) = 0 IV.
More informationIntroduction to Optimization
Introduction to Optimization Second Order Optimization Methods Marc Toussaint U Stuttgart Planned Outline Gradient-based optimization (1st order methods) plain grad., steepest descent, conjugate grad.,
More informationSolution Methods Numerical Algorithms
Solution Methods Numerical Algorithms Evelien van der Hurk DTU Managment Engineering Class Exercises From Last Time 2 DTU Management Engineering 42111: Static and Dynamic Optimization (6) 09/10/2017 Class
More informationToday s class. Roots of equation Finish up incremental search Open methods. Numerical Methods, Fall 2011 Lecture 5. Prof. Jinbo Bi CSE, UConn
Today s class Roots of equation Finish up incremental search Open methods 1 False Position Method Although the interval [a,b] where the root becomes iteratively closer with the false position method, unlike
More informationINTRODUCTION TO LINEAR AND NONLINEAR PROGRAMMING
INTRODUCTION TO LINEAR AND NONLINEAR PROGRAMMING DAVID G. LUENBERGER Stanford University TT ADDISON-WESLEY PUBLISHING COMPANY Reading, Massachusetts Menlo Park, California London Don Mills, Ontario CONTENTS
More informationMultivariate Numerical Optimization
Jianxin Wei March 1, 2013 Outline 1 Graphics for Function of Two Variables 2 Nelder-Mead Simplex Method 3 Steepest Descent Method 4 Newton s Method 5 Quasi-Newton s Method 6 Built-in R Function 7 Linear
More informationA Brief Look at Optimization
A Brief Look at Optimization CSC 412/2506 Tutorial David Madras January 18, 2018 Slides adapted from last year s version Overview Introduction Classes of optimization problems Linear programming Steepest
More informationLECTURE NOTES Non-Linear Programming
CEE 6110 David Rosenberg p. 1 Learning Objectives LECTURE NOTES Non-Linear Programming 1. Write out the non-linear model formulation 2. Describe the difficulties of solving a non-linear programming model
More informationCS281 Section 3: Practical Optimization
CS281 Section 3: Practical Optimization David Duvenaud and Dougal Maclaurin Most parameter estimation problems in machine learning cannot be solved in closed form, so we often have to resort to numerical
More informationNumerical Optimization: Introduction and gradient-based methods
Numerical Optimization: Introduction and gradient-based methods Master 2 Recherche LRI Apprentissage Statistique et Optimisation Anne Auger Inria Saclay-Ile-de-France November 2011 http://tao.lri.fr/tiki-index.php?page=courses
More informationComputational Methods. Constrained Optimization
Computational Methods Constrained Optimization Manfred Huber 2010 1 Constrained Optimization Unconstrained Optimization finds a minimum of a function under the assumption that the parameters can take on
More informationNumerical Optimization
Numerical Optimization Quantitative Macroeconomics Raül Santaeulàlia-Llopis MOVE-UAB and Barcelona GSE Fall 2018 Raül Santaeulàlia-Llopis (MOVE-UAB,BGSE) QM: Numerical Optimization Fall 2018 1 / 46 1 Introduction
More information16.410/413 Principles of Autonomy and Decision Making
16.410/413 Principles of Autonomy and Decision Making Lecture 17: The Simplex Method Emilio Frazzoli Aeronautics and Astronautics Massachusetts Institute of Technology November 10, 2010 Frazzoli (MIT)
More informationA Brief Overview of Optimization Problems. Steven G. Johnson MIT course , Fall 2008
A Brief Overview of Optimization Problems Steven G. Johnson MIT course 18.335, Fall 2008 Why optimization? In some sense, all engineering design is optimization: choosing design parameters to improve some
More informationContents. I Basics 1. Copyright by SIAM. Unauthorized reproduction of this article is prohibited.
page v Preface xiii I Basics 1 1 Optimization Models 3 1.1 Introduction... 3 1.2 Optimization: An Informal Introduction... 4 1.3 Linear Equations... 7 1.4 Linear Optimization... 10 Exercises... 12 1.5
More informationCMU-Q Lecture 9: Optimization II: Constrained,Unconstrained Optimization Convex optimization. Teacher: Gianni A. Di Caro
CMU-Q 15-381 Lecture 9: Optimization II: Constrained,Unconstrained Optimization Convex optimization Teacher: Gianni A. Di Caro GLOBAL FUNCTION OPTIMIZATION Find the global maximum of the function f x (and
More informationLecture 12: Feasible direction methods
Lecture 12 Lecture 12: Feasible direction methods Kin Cheong Sou December 2, 2013 TMA947 Lecture 12 Lecture 12: Feasible direction methods 1 / 1 Feasible-direction methods, I Intro Consider the problem
More informationGradient Descent. Wed Sept 20th, James McInenrey Adapted from slides by Francisco J. R. Ruiz
Gradient Descent Wed Sept 20th, 2017 James McInenrey Adapted from slides by Francisco J. R. Ruiz Housekeeping A few clarifications of and adjustments to the course schedule: No more breaks at the midpoint
More information5 Machine Learning Abstractions and Numerical Optimization
Machine Learning Abstractions and Numerical Optimization 25 5 Machine Learning Abstractions and Numerical Optimization ML ABSTRACTIONS [some meta comments on machine learning] [When you write a large computer
More informationClassical Gradient Methods
Classical Gradient Methods Note simultaneous course at AMSI (math) summer school: Nonlin. Optimization Methods (see http://wwwmaths.anu.edu.au/events/amsiss05/) Recommended textbook (Springer Verlag, 1999):
More informationNonlinear Programming
Nonlinear Programming SECOND EDITION Dimitri P. Bertsekas Massachusetts Institute of Technology WWW site for book Information and Orders http://world.std.com/~athenasc/index.html Athena Scientific, Belmont,
More informationAlgorithms for convex optimization
Algorithms for convex optimization Michal Kočvara Institute of Information Theory and Automation Academy of Sciences of the Czech Republic and Czech Technical University kocvara@utia.cas.cz http://www.utia.cas.cz/kocvara
More information1. Introduction. performance of numerical methods. complexity bounds. structural convex optimization. course goals and topics
1. Introduction EE 546, Univ of Washington, Spring 2016 performance of numerical methods complexity bounds structural convex optimization course goals and topics 1 1 Some course info Welcome to EE 546!
More informationAlgoritmi di Ottimizzazione: Parte A Aspetti generali e algoritmi classici
Identificazione e Controllo Intelligente Algoritmi di Ottimizzazione: Parte A Aspetti generali e algoritmi classici David Naso A.A. 2006-2007 Identificazione e Controllo Intelligente 1 Search and optimization
More informationDelaunay-based Derivative-free Optimization via Global Surrogate. Pooriya Beyhaghi, Daniele Cavaglieri and Thomas Bewley
Delaunay-based Derivative-free Optimization via Global Surrogate Pooriya Beyhaghi, Daniele Cavaglieri and Thomas Bewley May 23, 2014 Delaunay-based Derivative-free Optimization via Global Surrogate Pooriya
More informationProgramming, numerics and optimization
Programming, numerics and optimization Lecture C-4: Constrained optimization Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428 June
More informationDS Machine Learning and Data Mining I. Alina Oprea Associate Professor, CCIS Northeastern University
DS 4400 Machine Learning and Data Mining I Alina Oprea Associate Professor, CCIS Northeastern University September 20 2018 Review Solution for multiple linear regression can be computed in closed form
More informationContents. Hilary Term. Summary of Numerical Analysis for this term. Sources of error in numerical calculation. Solving Problems
Contents Hilary Term 1 Root Finding 4 11 Bracketing and Bisection 5 111 Finding the root numerically 5 112 Pseudo BRACKET code 7 113 Drawbacks 8 114 Tips for success with Bracketing & Bisection 9 115 Virtues
More informationA Brief Overview of Optimization Problems. Steven G. Johnson MIT course , Fall 2008
A Brief Overview of Optimization Problems Steven G. Johnson MIT course 18.335, Fall 2008 Why optimization? In some sense, all engineering design is optimization: choosing design parameters to improve some
More informationAn evolutionary annealing-simplex algorithm for global optimisation of water resource systems
FIFTH INTERNATIONAL CONFERENCE ON HYDROINFORMATICS 1-5 July 2002, Cardiff, UK C05 - Evolutionary algorithms in hydroinformatics An evolutionary annealing-simplex algorithm for global optimisation of water
More informationAn Evolutionary Algorithm for Minimizing Multimodal Functions
An Evolutionary Algorithm for Minimizing Multimodal Functions D.G. Sotiropoulos, V.P. Plagianakos and M.N. Vrahatis University of Patras, Department of Mamatics, Division of Computational Mamatics & Informatics,
More informationNatasha S. Sharma, PhD
Revisiting the function evaluation problem Most functions cannot be evaluated exactly: 2 x, e x, ln x, trigonometric functions since by using a computer we are limited to the use of elementary arithmetic
More informationfor Approximating the Analytic Center of a Polytope Abstract. The analytic center of a polytope P + = fx 0 : Ax = b e T x =1g log x j
A Lagrangian Relaxation Method for Approximating the Analytic Center of a Polytope Masakazu Kojima y Nimrod Megiddo z Shinji Mizuno x September 1992 Abstract. The analytic center of a polytope P + = fx
More informationComputational Optimization. Constrained Optimization Algorithms
Computational Optimization Constrained Optimization Algorithms Same basic algorithms Repeat Determine descent direction Determine step size Take a step Until Optimal But now must consider feasibility,
More informationCPSC 340: Machine Learning and Data Mining. Robust Regression Fall 2015
CPSC 340: Machine Learning and Data Mining Robust Regression Fall 2015 Admin Can you see Assignment 1 grades on UBC connect? Auditors, don t worry about it. You should already be working on Assignment
More informationIntroduction to Design Optimization
Introduction to Design Optimization First Edition Krishnan Suresh i Dedicated to my family. They mean the world to me. ii Origins of this Text Preface Like many other textbooks, this text has evolved from
More informationOptimization. (Lectures on Numerical Analysis for Economists III) Jesús Fernández-Villaverde 1 and Pablo Guerrón 2 February 20, 2018
Optimization (Lectures on Numerical Analysis for Economists III) Jesús Fernández-Villaverde 1 and Pablo Guerrón 2 February 20, 2018 1 University of Pennsylvania 2 Boston College Optimization Optimization
More informationIntroduction. Optimization
Introduction to Optimization Amy Langville SAMSI Undergraduate Workshop N.C. State University SAMSI 6/1/05 GOAL: minimize f(x 1, x 2, x 3, x 4, x 5 ) = x 2 1.5x 2x 3 + x 4 /x 5 PRIZE: $1 million # of independent
More information25. NLP algorithms. ˆ Overview. ˆ Local methods. ˆ Constrained optimization. ˆ Global methods. ˆ Black-box methods.
CS/ECE/ISyE 524 Introduction to Optimization Spring 2017 18 25. NLP algorithms ˆ Overview ˆ Local methods ˆ Constrained optimization ˆ Global methods ˆ Black-box methods ˆ Course wrap-up Laurent Lessard
More informationMachine Learning for Software Engineering
Machine Learning for Software Engineering Single-State Meta-Heuristics Prof. Dr.-Ing. Norbert Siegmund Intelligent Software Systems 1 2 Recap: Goal is to Find the Optimum Challenges of general optimization
More informationGradient, Newton and conjugate direction methods for unconstrained nonlinear optimization
Gradient, Newton and conjugate direction methods for unconstrained nonlinear optimization Consider the gradient method (steepest descent), with exact unidimensional search, the Newton method and the conjugate
More informationISM206 Lecture, April 26, 2005 Optimization of Nonlinear Objectives, with Non-Linear Constraints
ISM206 Lecture, April 26, 2005 Optimization of Nonlinear Objectives, with Non-Linear Constraints Instructor: Kevin Ross Scribe: Pritam Roy May 0, 2005 Outline of topics for the lecture We will discuss
More informationFAQs on Convex Optimization
FAQs on Convex Optimization. What is a convex programming problem? A convex programming problem is the minimization of a convex function on a convex set, i.e. min f(x) X C where f: R n R and C R n. f is
More informationKey points. Assume (except for point 4) f : R n R is twice continuously differentiable. strictly above the graph of f except at x
Key points Assume (except for point 4) f : R n R is twice continuously differentiable 1 If Hf is neg def at x, then f attains a strict local max at x iff f(x) = 0 In (1), replace Hf(x) negative definite
More informationMaximizing an interpolating quadratic
Week 11: Monday, Apr 9 Maximizing an interpolating quadratic Suppose that a function f is evaluated on a reasonably fine, uniform mesh {x i } n i=0 with spacing h = x i+1 x i. How can we find any local
More informationIntroduction to Modern Control Systems
Introduction to Modern Control Systems Convex Optimization, Duality and Linear Matrix Inequalities Kostas Margellos University of Oxford AIMS CDT 2016-17 Introduction to Modern Control Systems November
More informationNumerical Methods Lecture 7 - Optimization
Numerical Methods Lecture 7 - Optimization Topics: numerical optimization - Newton again - Random search - Golden Section Search READING : text pgs. 331-349 Optimization - motivation What? Locating where
More informationFinding optimal configurations Adversarial search
CS 171 Introduction to AI Lecture 10 Finding optimal configurations Adversarial search Milos Hauskrecht milos@cs.pitt.edu 39 Sennott Square Announcements Homework assignment is out Due on Thursday next
More informationKey Concepts: Economic Computation, Part II
Key Concepts: Economic Computation, Part II Brent Hickman Fall, 2009 The purpose of the second section of these notes is to give you some further practice with numerical computation in MATLAB, and also
More informationINTRODUCTION AN INTRODUCTION TO DIRECT OPTIMIZATION METHODS. SIMPLE EXAMPLE: COMPASS SEARCH (Con t) SIMPLE EXAMPLE: COMPASS SEARCH
AN INTRODUCTION TO DIRECT OPTIMIZATION METHODS B Arnaud Bistoquet 03/30/04 INTRODUCTION Goal: minimization of a real-valued function f(), R n f() is differentiable + derivative-based f() can be computed
More informationIntroduction to Optimization
Introduction to Optimization Constrained Optimization Marc Toussaint U Stuttgart Constrained Optimization General constrained optimization problem: Let R n, f : R n R, g : R n R m, h : R n R l find min
More informationMachine Learning for Signal Processing Lecture 4: Optimization
Machine Learning for Signal Processing Lecture 4: Optimization 13 Sep 2015 Instructor: Bhiksha Raj (slides largely by Najim Dehak, JHU) 11-755/18-797 1 Index 1. The problem of optimization 2. Direct optimization
More informationConvex Optimization CMU-10725
Convex Optimization CMU-10725 Conjugate Direction Methods Barnabás Póczos & Ryan Tibshirani Conjugate Direction Methods 2 Books to Read David G. Luenberger, Yinyu Ye: Linear and Nonlinear Programming Nesterov:
More informationEfficient Tuning of SVM Hyperparameters Using Radius/Margin Bound and Iterative Algorithms
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 13, NO. 5, SEPTEMBER 2002 1225 Efficient Tuning of SVM Hyperparameters Using Radius/Margin Bound and Iterative Algorithms S. Sathiya Keerthi Abstract This paper
More informationFast oriented bounding box optimization on the rotation group SO(3, R)
Fast oriented bounding box optimization on the rotation group SO(3, R) Chia-Tche Chang 1, Bastien Gorissen 2,3 and Samuel Melchior 1,2 chia-tche.chang@uclouvain.be bastien.gorissen@cenaero.be samuel.melchior@uclouvain.be
More informationA Provably Convergent Multifidelity Optimization Algorithm not Requiring High-Fidelity Derivatives
A Provably Convergent Multifidelity Optimization Algorithm not Requiring High-Fidelity Derivatives Multidisciplinary Design Optimization Specialist Conference April 4, 00 Andrew March & Karen Willcox Motivation
More informationAssignment in Optimization 2016
Assignment in Optimization 2016 You can choose one of the following two alternatives Alternative 1 Nonlinear least-square fitting using the Gauss-Newton method The task is to solve an optimization problem
More informationConvexity Theory and Gradient Methods
Convexity Theory and Gradient Methods Angelia Nedić angelia@illinois.edu ISE Department and Coordinated Science Laboratory University of Illinois at Urbana-Champaign Outline Convex Functions Optimality
More informationRecent Developments in Model-based Derivative-free Optimization
Recent Developments in Model-based Derivative-free Optimization Seppo Pulkkinen April 23, 2010 Introduction Problem definition The problem we are considering is a nonlinear optimization problem with constraints:
More informationOPTIMIZATION FOR AUTOMATIC HISTORY MATCHING
INTERNATIONAL JOURNAL OF NUMERICAL ANALYSIS AND MODELING Volume 2, Supp, Pages 131 137 c 2005 Institute for Scientific Computing and Information OPTIMIZATION FOR AUTOMATIC HISTORY MATCHING Abstract. SHUGUANG
More informationMEI STRUCTURED MATHEMATICS. MEI conference University of Hertfordshire June C3 COURSEWORK
MEI STRUCTURED MATHEMATICS MEI conference University of Hertfordshire June 29 2009 C3 COURSEWORK What is this coursework designed to do and how do we prepare students for it? Presenter: Val Hanrahan The
More informationLecture 9. Introduction to Numerical Techniques
Lecture 9. Introduction to Numerical Techniques Ivan Papusha CDS270 2: Mathematical Methods in Control and System Engineering May 27, 2015 1 / 25 Logistics hw8 (last one) due today. do an easy problem
More informationTheoretical Concepts of Machine Learning
Theoretical Concepts of Machine Learning Part 2 Institute of Bioinformatics Johannes Kepler University, Linz, Austria Outline 1 Introduction 2 Generalization Error 3 Maximum Likelihood 4 Noise Models 5
More informationCS 6210 Fall 2016 Bei Wang. Review Lecture What have we learnt in Scientific Computing?
CS 6210 Fall 2016 Bei Wang Review Lecture What have we learnt in Scientific Computing? Let s recall the scientific computing pipeline observed phenomenon mathematical model discretization solution algorithm
More informationInformed search algorithms. (Based on slides by Oren Etzioni, Stuart Russell)
Informed search algorithms (Based on slides by Oren Etzioni, Stuart Russell) The problem # Unique board configurations in search space 8-puzzle 9! = 362880 15-puzzle 16! = 20922789888000 10 13 24-puzzle
More informationSolution Methodologies for. the Smallest Enclosing Circle Problem
Solution Methodologies for the Smallest Enclosing Circle Problem Sheng Xu 1 Robert M. Freund 2 Jie Sun 3 Tribute. We would like to dedicate this paper to Elijah Polak. Professor Polak has made substantial
More informationConstrained optimization
Constrained optimization A general constrained optimization problem has the form where The Lagrangian function is given by Primal and dual optimization problems Primal: Dual: Weak duality: Strong duality:
More informationLecture 6 - Multivariate numerical optimization
Lecture 6 - Multivariate numerical optimization Björn Andersson (w/ Jianxin Wei) Department of Statistics, Uppsala University February 13, 2014 1 / 36 Table of Contents 1 Plotting functions of two variables
More informationSTRUCTURAL & MULTIDISCIPLINARY OPTIMIZATION
STRUCTURAL & MULTIDISCIPLINARY OPTIMIZATION Pierre DUYSINX Patricia TOSSINGS Department of Aerospace and Mechanical Engineering Academic year 2018-2019 1 Course objectives To become familiar with the introduction
More informationGraphing Techniques. Domain (, ) Range (, ) Squaring Function f(x) = x 2 Domain (, ) Range [, ) f( x) = x 2
Graphing Techniques In this chapter, we will take our knowledge of graphs of basic functions and expand our ability to graph polynomial and rational functions using common sense, zeros, y-intercepts, stretching
More informationIntroduction to Design Optimization: Search Methods
Introduction to Design Optimization: Search Methods 1-D Optimization The Search We don t know the curve. Given α, we can calculate f(α). By inspecting some points, we try to find the approximated shape
More informationHandout 2 - Root Finding using MATLAB
Handout 2 - Root Finding using MATLAB Middle East Technical University MATLAB has couple of built-in root finding functions. In this handout we ll have a look at fzero, roots and solve functions. It is
More informationTraining of Neural Networks. Q.J. Zhang, Carleton University
Training of Neural Networks Notation: x: input of the original modeling problem or the neural network y: output of the original modeling problem or the neural network w: internal weights/parameters of
More informationME 555: Distributed Optimization
ME 555: Distributed Optimization Duke University Spring 2015 1 Administrative Course: ME 555: Distributed Optimization (Spring 2015) Instructor: Time: Location: Office hours: Website: Soomin Lee (email:
More information6 BLAS (Basic Linear Algebra Subroutines)
161 BLAS 6.1 Motivation 6 BLAS (Basic Linear Algebra Subroutines) 6.1 Motivation How to optimise programs that use a lot of linear algebra operations? Efficiency depends on but also on: processor speed
More informationOptimization III: Constrained Optimization
Optimization III: Constrained Optimization CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Doug James (and Justin Solomon) CS 205A: Mathematical Methods Optimization III: Constrained Optimization
More informationModule 1 Lecture Notes 2. Optimization Problem and Model Formulation
Optimization Methods: Introduction and Basic concepts 1 Module 1 Lecture Notes 2 Optimization Problem and Model Formulation Introduction In the previous lecture we studied the evolution of optimization
More informationImage Registration Lecture 4: First Examples
Image Registration Lecture 4: First Examples Prof. Charlene Tsai Outline Example Intensity-based registration SSD error function Image mapping Function minimization: Gradient descent Derivative calculation
More informationModule 4 : Solving Linear Algebraic Equations Section 11 Appendix C: Steepest Descent / Gradient Search Method
Module 4 : Solving Linear Algebraic Equations Section 11 Appendix C: Steepest Descent / Gradient Search Method 11 Appendix C: Steepest Descent / Gradient Search Method In the module on Problem Discretization
More informationAMS527: Numerical Analysis II
AMS527: Numerical Analysis II A Brief Overview of Finite Element Methods Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao SUNY Stony Brook AMS527: Numerical Analysis II 1 / 25 Overview Basic concepts Mathematical
More informationarxiv: v2 [math.oc] 12 Feb 2019
A CENTRAL PATH FOLLOWING METHOD USING THE NORMALIZED GRADIENTS Long Chen 1, Wenyi Chen 2, and Kai-Uwe Bletzinger 1 arxiv:1902.04040v2 [math.oc] 12 Feb 2019 1 Chair of Structural Analysis, Technical University
More informationME 261: Numerical Analysis Lecture-12: Numerical Interpolation
1 ME 261: Numerical Analysis Lecture-12: Numerical Interpolation Md. Tanver Hossain Department of Mechanical Engineering, BUET http://tantusher.buet.ac.bd 2 Inverse Interpolation Problem : Given a table
More informationAnimation Lecture 10 Slide Fall 2003
Animation Lecture 10 Slide 1 6.837 Fall 2003 Conventional Animation Draw each frame of the animation great control tedious Reduce burden with cel animation layer keyframe inbetween cel panoramas (Disney
More informationGeneric descent algorithm Generalization to multiple dimensions Problems of descent methods, possible improvements Fixes Local minima
1 Lecture 10: descent methods Generic descent algorithm Generalization to multiple dimensions Problems of descent methods, possible improvements Fixes Local minima Gradient descent (reminder) Minimum of
More informationConvex Optimization MLSS 2015
Convex Optimization MLSS 2015 Constantine Caramanis The University of Texas at Austin The Optimization Problem minimize : f (x) subject to : x X. The Optimization Problem minimize : f (x) subject to :
More informationK-Means and Gaussian Mixture Models
K-Means and Gaussian Mixture Models David Rosenberg New York University June 15, 2015 David Rosenberg (New York University) DS-GA 1003 June 15, 2015 1 / 43 K-Means Clustering Example: Old Faithful Geyser
More informationChapter Multidimensional Gradient Method
Chapter 09.04 Multidimensional Gradient Method After reading this chapter, you should be able to: 1. Understand how multi-dimensional gradient methods are different from direct search methods. Understand
More information