Introduction to optimization methods and line search

Size: px
Start display at page:

Download "Introduction to optimization methods and line search"

Transcription

1 Introduction to optimization methods and line search Jussi Hakanen Post-doctoral researcher

2 How to find optimal solutions? Trial and error widely used in practice, not efficient and high possibility to miss good solutions Better to use a systematic way to find optimal solution Typically we know only function value(s) at the current trial point possibly gradients at the current trial point How can we know which solution is optimal? How can we find optimal solutions?

3 Optimality conditions How can we know that a solution is optimal? One way is to utilize optimality conditions Necessary optimality conditions = conditions that an optimal solution has to satisfy (does not guarantee optimality) Sufficient optimality conditions = conditions that guarantee optimality when satisfied 1. order conditions (1. order derivatives) and 2. order conditions (2. order derivatives)

4 Global vs. local minimizers A solution x S is a global minimizer if f x f x for all x S A solution x S is a local minimizer if there exists an ε > 0 s.t. f x f(x) for all x S where x x < ε Convexity: a local minimizer is a global minimizer Global minimizers are preferred, local minimizers are usually easier to identify

5 Solving an optimization problem Find optimal values x for the variables Problems that can be solved analytically min x 2, when x 3 x = 3 Usually impossible to solve analytically Must be solved numerically approximation of the solution In mathematical optimization a starting point is iteratively improved

6 Numerical solution Modelling mathematical model of the problem Numerical methods numerical simulation model for the mathematical model Optimization method solve the problem utilizing the numerical simulation model SO modelling simulation optimization

7 Optimization method Algorithm: a mathematical description 1. Choose a stopping parameter ε > 0, starting point x 1 and a symmetric positive definite n n matrix D 1 (e.g. D 1 = I). Set y 1 = x 1 and h = j = If f(y j ) < ε, stop. Otherwise, set d j = D f(y j ). Let λ j be a solution of min f(y j + λd j ), s.t. λ 0. Set y j+1 = y j + λ j d j. If j = n, set y 1 = x h+1 = y n+1, h = h + 1, j = 1 and repeat (2). 3. Compute D j+1. Set j = j + 1 and go to (2). Method: numerical methods included Software: a method implemented as a computer programme

8 Structure of optimization methods Typically Constraint handling converts the problem to (a series of) unconstrained problems In unconstrained optimization a search direction is determined at each iteration The best solution in the search direction is found with line search Constraint handling method Unconstrained optimization Line search

9 Local optimization methods Find a (closest) local optimum Fast Usually utilize derivatives Mathematical convergence For example Direct search methods (pattern search, Hooke & Jeeves, Nelder & Mead, ) Gradient based methods (steepest descent, Newton s method, quasi-newton method, conjugate gradient, SQP, interior point methods )

10 Global optimization methods Try to get as close to global optimum as possible No mathematical convergence Do not assume much of the problem Slow, use lots of function evaluations Heuristic, contain randomness Most well known are nature-inspired methods (TIES451 Selected topics in soft computing) based on improving a population of solutions at a time instead of a single solution

11 Hybrid methods Combination of global and local methods Try to combine the benefits of both rough estimate with a global method, fine tune with a local method Challenge: how the methods should be combined? e.g. when to switch from global to local? (speed vs. accuracy)

12 Line search What did you find out about line search?

13 Line search The idea of line search is to optimize a given function with respect to a single variable Optimization algorithms for multivariable problems generate iteratively search directions in which better solutions are found Line search is used to find these! Exact minimum is not required but an approximation of it which is within a given tolerance ε > 0 enough to know that x [a, b ] where b a < ε

14 Optimality conditions Necessary: Let f: R R be differentiable. If x is a local minimizer, then f x = 0. In addition, if f is twice continuously differentiable and x is a local minimizer, then f x 0. Sufficient: Let f: R R be twice continuously differentiable. If f x = 0 and f x > 0, then x is a strict local minimizer.

15 Examples f x = (x 2) 2 4 f x = 2x 4 f x = 2 If x = 2, then both the necessary and sufficient optimality conditions are satisfied f x = (x 2) 2 4

16 Examples f x = (x 2) 3 4 f x = 3 x 2 2 f x = 6x 12 If x = 2, then the necessary optimality conditions are satisfied although x = 2 is not a local minimizer It is a saddle point Sufficient optimality conditions are not satisfied in x = 2 f x = (x 2) 3 4

17 Note on optimality conditions If f is not differentiable, then local minimizer can be in a point where f is 1) not differentiable or 2) discontinuous f x = x

18 Finding a unimodal interval Most line search methods assume that the search is started from a unimodal interval [a, b] f is unimodal in [a, b] if there is exactly one x [a, b] s.t. for all x 1, x 2 [a, b] for which x 1 < x 2 holds If x 2 < x, then f x 1 > f(x 2 ) and If x 1 > x, then f x 1 < f(x 2 )

19 Search with fixed steps Let (A, B) be the interval where we want to find a minimum for f Compute values for f in P equally spaced points x i in (A, B) x i = A + i (B A), P+1 i = 1,, P When points x j, x j+i and x j+2 are found s.t. f x j > f x j+1 < f(x j+2 ), we know that there exist at least one local minimizer in (x j, x j+2 ) The interval can be further reduced

20 Line search methods Assume that f is unimodal in [a, b] General idea is to start reducing the interval [a, b] s.t. the minimizer is still included in it An approximation of the minimizer is found when the length of the interval is smaller than a pre-determined tolerance Line search methods can be divided into Elimination methods Interpolation methods (often use derivatives)

21 The method of bisection Elimination method 1) Choose small but significant constant 2ε > 0 and an allowable length L > 0 for the final interval. Let [a 1, b 1 ] be the original (unimodal) interval. Set h = 1. 2) If b h a h < L, stop. Minimizer x [a h, b h ]. Otherwise, compute values of f in x h = ah +b h ε and y h = ah +b h + ε. 2 3) If f x h < f(y h ), set a h+1 = a h and b h+1 = y h. Otherwise, set a h+1 = x h and b h+1 = b h. Set h = h + 1 and go to step 2). 2 a h x h y h 2ε b h

22 The method of bisection (cont.) Efficiency: Length of the interval after h iterations is 1 1 h b a + 2ε h Number of iterations required if the final length should be L is (why?) h = ln ( L 2ε )/ ln 2 b a 2ε For each iteration, the objective function is evaluated 2 times (in x h and y h ) in total 2h evaluations

23 Golden section Assume that we want to separate a sub interval (length y) from an interval of length L such that y Then, y = L 0.618L L y = L y It is said that now the interval is divided in the ratio of golden section Theorem Divide an interval [a, b] in the ratio of golden section first from right (point d) and then from left (point c). Then point c divides the interval [a, d] in the ratio of golden section and point d does the same for [c, b]. a c d b

24 Golden section search Elimination method, known also as Fibonacci search. Let C = 1) Choose an allowable length L > 0 for the final interval. Let [a 1, b 1 ] be the original (unimodal) interval. Set x 1 = a C b 1 a 1 = b 1 C(b 1 a 1 ) and y 1 = a 1 + C b 1 a 1. Compute f(x 1 ) and f(y 1 ). Set h = 1. 2) If b h a h < L, stop. Minimizer x [a h, b h ]. Otherwise, if f x h f(y h ) go to step 4). 3) Set a h+1 = x h and b h+1 = b h. Further set x h+1 = y h and y h+1 = a h+1 + C(b h+1 a h+1 ). Compute f(y h+1 ) and go to step 5). 4) Set a h+1 = a h and b h+1 = y h. Further set y h+1 = x h and x h+1 = a h+1 + (1 C)(b h+1 a h+1 ). Compute f(x h+1 ). 5) Set h = h + 1 and go to step 2). a h x h y h b h

25 Golden section search (cont.) Efficiency Length of the interval after h iterations is C h (b a) Number of iterations required if the final length should be L is (why?) h = ln ( L )/ ln C b a For each iteration (except the last), the objective function is evaluated one time (in x h+1 or y h+1 ) plus in the beginning in two points (x 1 and y 1 ) in total h + 1 evaluations

26 Quadratic interpolation Idea is to approximate f with a quadratic polynomial whose minimizer is known Taylor s second order polynomial is used: p x = f x h + f x h x x h f x h x x h 2 If f x h 0, then p(x) has a critical point in x h+1 when p (x h+1 ) = 0 x h+1 = x h f (xh ) f (x h ) Newton s method for solving f x = 0!!! Interpolation can also be applied in the case where no derivatives are available (find out the idea by yourself)

27 Programming assignment Form the pairs!!! Start programming by implementing some line search method Any programming language is ok Test your implementation with some optimization problems where you know the minimizer

28 Topic of the lectures on January 20 th & 22 nd Mon, Jan 20 th : unconstrained optimization with multiple variables, optimality conditions and methods that don t utilize gradient information (=direct search methods) Wed, Jan 22 nd : methods that utilize gradient information Study this before the lecture! Questions to be considered What kind of optimality conditions there exist? What kind of techniques direct search methods use to find a local minimizer? How gradient information is utilized?

Introduction to unconstrained optimization - derivative-free methods

Introduction to unconstrained optimization - derivative-free methods Introduction to unconstrained optimization - derivative-free methods Jussi Hakanen Post-doctoral researcher Office: AgC426.3 jussi.hakanen@jyu.fi Learning outcomes To understand the basic principles of

More information

Characterizing Improving Directions Unconstrained Optimization

Characterizing Improving Directions Unconstrained Optimization Final Review IE417 In the Beginning... In the beginning, Weierstrass's theorem said that a continuous function achieves a minimum on a compact set. Using this, we showed that for a convex set S and y not

More information

Today. Golden section, discussion of error Newton s method. Newton s method, steepest descent, conjugate gradient

Today. Golden section, discussion of error Newton s method. Newton s method, steepest descent, conjugate gradient Optimization Last time Root finding: definition, motivation Algorithms: Bisection, false position, secant, Newton-Raphson Convergence & tradeoffs Example applications of Newton s method Root finding in

More information

Optimization. Industrial AI Lab.

Optimization. Industrial AI Lab. Optimization Industrial AI Lab. Optimization An important tool in 1) Engineering problem solving and 2) Decision science People optimize Nature optimizes 2 Optimization People optimize (source: http://nautil.us/blog/to-save-drowning-people-ask-yourself-what-would-light-do)

More information

Modern Methods of Data Analysis - WS 07/08

Modern Methods of Data Analysis - WS 07/08 Modern Methods of Data Analysis Lecture XV (04.02.08) Contents: Function Minimization (see E. Lohrmann & V. Blobel) Optimization Problem Set of n independent variables Sometimes in addition some constraints

More information

Introduction to Optimization Problems and Methods

Introduction to Optimization Problems and Methods Introduction to Optimization Problems and Methods wjch@umich.edu December 10, 2009 Outline 1 Linear Optimization Problem Simplex Method 2 3 Cutting Plane Method 4 Discrete Dynamic Programming Problem Simplex

More information

Constrained and Unconstrained Optimization

Constrained and Unconstrained Optimization Constrained and Unconstrained Optimization Carlos Hurtado Department of Economics University of Illinois at Urbana-Champaign hrtdmrt2@illinois.edu Oct 10th, 2017 C. Hurtado (UIUC - Economics) Numerical

More information

Short Reminder of Nonlinear Programming

Short Reminder of Nonlinear Programming Short Reminder of Nonlinear Programming Kaisa Miettinen Dept. of Math. Inf. Tech. Email: kaisa.miettinen@jyu.fi Homepage: http://www.mit.jyu.fi/miettine Contents Background General overview briefly theory

More information

MATH3016: OPTIMIZATION

MATH3016: OPTIMIZATION MATH3016: OPTIMIZATION Lecturer: Dr Huifu Xu School of Mathematics University of Southampton Highfield SO17 1BJ Southampton Email: h.xu@soton.ac.uk 1 Introduction What is optimization? Optimization is

More information

Lecture 12: convergence. Derivative (one variable)

Lecture 12: convergence. Derivative (one variable) Lecture 12: convergence More about multivariable calculus Descent methods Backtracking line search More about convexity (first and second order) Newton step Example 1: linear programming (one var., one

More information

David G. Luenberger Yinyu Ye. Linear and Nonlinear. Programming. Fourth Edition. ö Springer

David G. Luenberger Yinyu Ye. Linear and Nonlinear. Programming. Fourth Edition. ö Springer David G. Luenberger Yinyu Ye Linear and Nonlinear Programming Fourth Edition ö Springer Contents 1 Introduction 1 1.1 Optimization 1 1.2 Types of Problems 2 1.3 Size of Problems 5 1.4 Iterative Algorithms

More information

APPLIED OPTIMIZATION WITH MATLAB PROGRAMMING

APPLIED OPTIMIZATION WITH MATLAB PROGRAMMING APPLIED OPTIMIZATION WITH MATLAB PROGRAMMING Second Edition P. Venkataraman Rochester Institute of Technology WILEY JOHN WILEY & SONS, INC. CONTENTS PREFACE xiii 1 Introduction 1 1.1. Optimization Fundamentals

More information

M. Sc. (Artificial Intelligence and Machine Learning)

M. Sc. (Artificial Intelligence and Machine Learning) Course Name: Advanced Python Course Code: MSCAI 122 This course will introduce students to advanced python implementations and the latest Machine Learning and Deep learning libraries, Scikit-Learn and

More information

Optimization. 1. Optimization. by Prof. Seungchul Lee Industrial AI Lab POSTECH. Table of Contents

Optimization. 1. Optimization. by Prof. Seungchul Lee Industrial AI Lab  POSTECH. Table of Contents Optimization by Prof. Seungchul Lee Industrial AI Lab http://isystems.unist.ac.kr/ POSTECH Table of Contents I. 1. Optimization II. 2. Solving Optimization Problems III. 3. How do we Find x f(x) = 0 IV.

More information

Introduction to Optimization

Introduction to Optimization Introduction to Optimization Second Order Optimization Methods Marc Toussaint U Stuttgart Planned Outline Gradient-based optimization (1st order methods) plain grad., steepest descent, conjugate grad.,

More information

Solution Methods Numerical Algorithms

Solution Methods Numerical Algorithms Solution Methods Numerical Algorithms Evelien van der Hurk DTU Managment Engineering Class Exercises From Last Time 2 DTU Management Engineering 42111: Static and Dynamic Optimization (6) 09/10/2017 Class

More information

Today s class. Roots of equation Finish up incremental search Open methods. Numerical Methods, Fall 2011 Lecture 5. Prof. Jinbo Bi CSE, UConn

Today s class. Roots of equation Finish up incremental search Open methods. Numerical Methods, Fall 2011 Lecture 5. Prof. Jinbo Bi CSE, UConn Today s class Roots of equation Finish up incremental search Open methods 1 False Position Method Although the interval [a,b] where the root becomes iteratively closer with the false position method, unlike

More information

INTRODUCTION TO LINEAR AND NONLINEAR PROGRAMMING

INTRODUCTION TO LINEAR AND NONLINEAR PROGRAMMING INTRODUCTION TO LINEAR AND NONLINEAR PROGRAMMING DAVID G. LUENBERGER Stanford University TT ADDISON-WESLEY PUBLISHING COMPANY Reading, Massachusetts Menlo Park, California London Don Mills, Ontario CONTENTS

More information

Multivariate Numerical Optimization

Multivariate Numerical Optimization Jianxin Wei March 1, 2013 Outline 1 Graphics for Function of Two Variables 2 Nelder-Mead Simplex Method 3 Steepest Descent Method 4 Newton s Method 5 Quasi-Newton s Method 6 Built-in R Function 7 Linear

More information

A Brief Look at Optimization

A Brief Look at Optimization A Brief Look at Optimization CSC 412/2506 Tutorial David Madras January 18, 2018 Slides adapted from last year s version Overview Introduction Classes of optimization problems Linear programming Steepest

More information

LECTURE NOTES Non-Linear Programming

LECTURE NOTES Non-Linear Programming CEE 6110 David Rosenberg p. 1 Learning Objectives LECTURE NOTES Non-Linear Programming 1. Write out the non-linear model formulation 2. Describe the difficulties of solving a non-linear programming model

More information

CS281 Section 3: Practical Optimization

CS281 Section 3: Practical Optimization CS281 Section 3: Practical Optimization David Duvenaud and Dougal Maclaurin Most parameter estimation problems in machine learning cannot be solved in closed form, so we often have to resort to numerical

More information

Numerical Optimization: Introduction and gradient-based methods

Numerical Optimization: Introduction and gradient-based methods Numerical Optimization: Introduction and gradient-based methods Master 2 Recherche LRI Apprentissage Statistique et Optimisation Anne Auger Inria Saclay-Ile-de-France November 2011 http://tao.lri.fr/tiki-index.php?page=courses

More information

Computational Methods. Constrained Optimization

Computational Methods. Constrained Optimization Computational Methods Constrained Optimization Manfred Huber 2010 1 Constrained Optimization Unconstrained Optimization finds a minimum of a function under the assumption that the parameters can take on

More information

Numerical Optimization

Numerical Optimization Numerical Optimization Quantitative Macroeconomics Raül Santaeulàlia-Llopis MOVE-UAB and Barcelona GSE Fall 2018 Raül Santaeulàlia-Llopis (MOVE-UAB,BGSE) QM: Numerical Optimization Fall 2018 1 / 46 1 Introduction

More information

16.410/413 Principles of Autonomy and Decision Making

16.410/413 Principles of Autonomy and Decision Making 16.410/413 Principles of Autonomy and Decision Making Lecture 17: The Simplex Method Emilio Frazzoli Aeronautics and Astronautics Massachusetts Institute of Technology November 10, 2010 Frazzoli (MIT)

More information

A Brief Overview of Optimization Problems. Steven G. Johnson MIT course , Fall 2008

A Brief Overview of Optimization Problems. Steven G. Johnson MIT course , Fall 2008 A Brief Overview of Optimization Problems Steven G. Johnson MIT course 18.335, Fall 2008 Why optimization? In some sense, all engineering design is optimization: choosing design parameters to improve some

More information

Contents. I Basics 1. Copyright by SIAM. Unauthorized reproduction of this article is prohibited.

Contents. I Basics 1. Copyright by SIAM. Unauthorized reproduction of this article is prohibited. page v Preface xiii I Basics 1 1 Optimization Models 3 1.1 Introduction... 3 1.2 Optimization: An Informal Introduction... 4 1.3 Linear Equations... 7 1.4 Linear Optimization... 10 Exercises... 12 1.5

More information

CMU-Q Lecture 9: Optimization II: Constrained,Unconstrained Optimization Convex optimization. Teacher: Gianni A. Di Caro

CMU-Q Lecture 9: Optimization II: Constrained,Unconstrained Optimization Convex optimization. Teacher: Gianni A. Di Caro CMU-Q 15-381 Lecture 9: Optimization II: Constrained,Unconstrained Optimization Convex optimization Teacher: Gianni A. Di Caro GLOBAL FUNCTION OPTIMIZATION Find the global maximum of the function f x (and

More information

Lecture 12: Feasible direction methods

Lecture 12: Feasible direction methods Lecture 12 Lecture 12: Feasible direction methods Kin Cheong Sou December 2, 2013 TMA947 Lecture 12 Lecture 12: Feasible direction methods 1 / 1 Feasible-direction methods, I Intro Consider the problem

More information

Gradient Descent. Wed Sept 20th, James McInenrey Adapted from slides by Francisco J. R. Ruiz

Gradient Descent. Wed Sept 20th, James McInenrey Adapted from slides by Francisco J. R. Ruiz Gradient Descent Wed Sept 20th, 2017 James McInenrey Adapted from slides by Francisco J. R. Ruiz Housekeeping A few clarifications of and adjustments to the course schedule: No more breaks at the midpoint

More information

5 Machine Learning Abstractions and Numerical Optimization

5 Machine Learning Abstractions and Numerical Optimization Machine Learning Abstractions and Numerical Optimization 25 5 Machine Learning Abstractions and Numerical Optimization ML ABSTRACTIONS [some meta comments on machine learning] [When you write a large computer

More information

Classical Gradient Methods

Classical Gradient Methods Classical Gradient Methods Note simultaneous course at AMSI (math) summer school: Nonlin. Optimization Methods (see http://wwwmaths.anu.edu.au/events/amsiss05/) Recommended textbook (Springer Verlag, 1999):

More information

Nonlinear Programming

Nonlinear Programming Nonlinear Programming SECOND EDITION Dimitri P. Bertsekas Massachusetts Institute of Technology WWW site for book Information and Orders http://world.std.com/~athenasc/index.html Athena Scientific, Belmont,

More information

Algorithms for convex optimization

Algorithms for convex optimization Algorithms for convex optimization Michal Kočvara Institute of Information Theory and Automation Academy of Sciences of the Czech Republic and Czech Technical University kocvara@utia.cas.cz http://www.utia.cas.cz/kocvara

More information

1. Introduction. performance of numerical methods. complexity bounds. structural convex optimization. course goals and topics

1. Introduction. performance of numerical methods. complexity bounds. structural convex optimization. course goals and topics 1. Introduction EE 546, Univ of Washington, Spring 2016 performance of numerical methods complexity bounds structural convex optimization course goals and topics 1 1 Some course info Welcome to EE 546!

More information

Algoritmi di Ottimizzazione: Parte A Aspetti generali e algoritmi classici

Algoritmi di Ottimizzazione: Parte A Aspetti generali e algoritmi classici Identificazione e Controllo Intelligente Algoritmi di Ottimizzazione: Parte A Aspetti generali e algoritmi classici David Naso A.A. 2006-2007 Identificazione e Controllo Intelligente 1 Search and optimization

More information

Delaunay-based Derivative-free Optimization via Global Surrogate. Pooriya Beyhaghi, Daniele Cavaglieri and Thomas Bewley

Delaunay-based Derivative-free Optimization via Global Surrogate. Pooriya Beyhaghi, Daniele Cavaglieri and Thomas Bewley Delaunay-based Derivative-free Optimization via Global Surrogate Pooriya Beyhaghi, Daniele Cavaglieri and Thomas Bewley May 23, 2014 Delaunay-based Derivative-free Optimization via Global Surrogate Pooriya

More information

Programming, numerics and optimization

Programming, numerics and optimization Programming, numerics and optimization Lecture C-4: Constrained optimization Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428 June

More information

DS Machine Learning and Data Mining I. Alina Oprea Associate Professor, CCIS Northeastern University

DS Machine Learning and Data Mining I. Alina Oprea Associate Professor, CCIS Northeastern University DS 4400 Machine Learning and Data Mining I Alina Oprea Associate Professor, CCIS Northeastern University September 20 2018 Review Solution for multiple linear regression can be computed in closed form

More information

Contents. Hilary Term. Summary of Numerical Analysis for this term. Sources of error in numerical calculation. Solving Problems

Contents. Hilary Term. Summary of Numerical Analysis for this term. Sources of error in numerical calculation. Solving Problems Contents Hilary Term 1 Root Finding 4 11 Bracketing and Bisection 5 111 Finding the root numerically 5 112 Pseudo BRACKET code 7 113 Drawbacks 8 114 Tips for success with Bracketing & Bisection 9 115 Virtues

More information

A Brief Overview of Optimization Problems. Steven G. Johnson MIT course , Fall 2008

A Brief Overview of Optimization Problems. Steven G. Johnson MIT course , Fall 2008 A Brief Overview of Optimization Problems Steven G. Johnson MIT course 18.335, Fall 2008 Why optimization? In some sense, all engineering design is optimization: choosing design parameters to improve some

More information

An evolutionary annealing-simplex algorithm for global optimisation of water resource systems

An evolutionary annealing-simplex algorithm for global optimisation of water resource systems FIFTH INTERNATIONAL CONFERENCE ON HYDROINFORMATICS 1-5 July 2002, Cardiff, UK C05 - Evolutionary algorithms in hydroinformatics An evolutionary annealing-simplex algorithm for global optimisation of water

More information

An Evolutionary Algorithm for Minimizing Multimodal Functions

An Evolutionary Algorithm for Minimizing Multimodal Functions An Evolutionary Algorithm for Minimizing Multimodal Functions D.G. Sotiropoulos, V.P. Plagianakos and M.N. Vrahatis University of Patras, Department of Mamatics, Division of Computational Mamatics & Informatics,

More information

Natasha S. Sharma, PhD

Natasha S. Sharma, PhD Revisiting the function evaluation problem Most functions cannot be evaluated exactly: 2 x, e x, ln x, trigonometric functions since by using a computer we are limited to the use of elementary arithmetic

More information

for Approximating the Analytic Center of a Polytope Abstract. The analytic center of a polytope P + = fx 0 : Ax = b e T x =1g log x j

for Approximating the Analytic Center of a Polytope Abstract. The analytic center of a polytope P + = fx 0 : Ax = b e T x =1g log x j A Lagrangian Relaxation Method for Approximating the Analytic Center of a Polytope Masakazu Kojima y Nimrod Megiddo z Shinji Mizuno x September 1992 Abstract. The analytic center of a polytope P + = fx

More information

Computational Optimization. Constrained Optimization Algorithms

Computational Optimization. Constrained Optimization Algorithms Computational Optimization Constrained Optimization Algorithms Same basic algorithms Repeat Determine descent direction Determine step size Take a step Until Optimal But now must consider feasibility,

More information

CPSC 340: Machine Learning and Data Mining. Robust Regression Fall 2015

CPSC 340: Machine Learning and Data Mining. Robust Regression Fall 2015 CPSC 340: Machine Learning and Data Mining Robust Regression Fall 2015 Admin Can you see Assignment 1 grades on UBC connect? Auditors, don t worry about it. You should already be working on Assignment

More information

Introduction to Design Optimization

Introduction to Design Optimization Introduction to Design Optimization First Edition Krishnan Suresh i Dedicated to my family. They mean the world to me. ii Origins of this Text Preface Like many other textbooks, this text has evolved from

More information

Optimization. (Lectures on Numerical Analysis for Economists III) Jesús Fernández-Villaverde 1 and Pablo Guerrón 2 February 20, 2018

Optimization. (Lectures on Numerical Analysis for Economists III) Jesús Fernández-Villaverde 1 and Pablo Guerrón 2 February 20, 2018 Optimization (Lectures on Numerical Analysis for Economists III) Jesús Fernández-Villaverde 1 and Pablo Guerrón 2 February 20, 2018 1 University of Pennsylvania 2 Boston College Optimization Optimization

More information

Introduction. Optimization

Introduction. Optimization Introduction to Optimization Amy Langville SAMSI Undergraduate Workshop N.C. State University SAMSI 6/1/05 GOAL: minimize f(x 1, x 2, x 3, x 4, x 5 ) = x 2 1.5x 2x 3 + x 4 /x 5 PRIZE: $1 million # of independent

More information

25. NLP algorithms. ˆ Overview. ˆ Local methods. ˆ Constrained optimization. ˆ Global methods. ˆ Black-box methods.

25. NLP algorithms. ˆ Overview. ˆ Local methods. ˆ Constrained optimization. ˆ Global methods. ˆ Black-box methods. CS/ECE/ISyE 524 Introduction to Optimization Spring 2017 18 25. NLP algorithms ˆ Overview ˆ Local methods ˆ Constrained optimization ˆ Global methods ˆ Black-box methods ˆ Course wrap-up Laurent Lessard

More information

Machine Learning for Software Engineering

Machine Learning for Software Engineering Machine Learning for Software Engineering Single-State Meta-Heuristics Prof. Dr.-Ing. Norbert Siegmund Intelligent Software Systems 1 2 Recap: Goal is to Find the Optimum Challenges of general optimization

More information

Gradient, Newton and conjugate direction methods for unconstrained nonlinear optimization

Gradient, Newton and conjugate direction methods for unconstrained nonlinear optimization Gradient, Newton and conjugate direction methods for unconstrained nonlinear optimization Consider the gradient method (steepest descent), with exact unidimensional search, the Newton method and the conjugate

More information

ISM206 Lecture, April 26, 2005 Optimization of Nonlinear Objectives, with Non-Linear Constraints

ISM206 Lecture, April 26, 2005 Optimization of Nonlinear Objectives, with Non-Linear Constraints ISM206 Lecture, April 26, 2005 Optimization of Nonlinear Objectives, with Non-Linear Constraints Instructor: Kevin Ross Scribe: Pritam Roy May 0, 2005 Outline of topics for the lecture We will discuss

More information

FAQs on Convex Optimization

FAQs on Convex Optimization FAQs on Convex Optimization. What is a convex programming problem? A convex programming problem is the minimization of a convex function on a convex set, i.e. min f(x) X C where f: R n R and C R n. f is

More information

Key points. Assume (except for point 4) f : R n R is twice continuously differentiable. strictly above the graph of f except at x

Key points. Assume (except for point 4) f : R n R is twice continuously differentiable. strictly above the graph of f except at x Key points Assume (except for point 4) f : R n R is twice continuously differentiable 1 If Hf is neg def at x, then f attains a strict local max at x iff f(x) = 0 In (1), replace Hf(x) negative definite

More information

Maximizing an interpolating quadratic

Maximizing an interpolating quadratic Week 11: Monday, Apr 9 Maximizing an interpolating quadratic Suppose that a function f is evaluated on a reasonably fine, uniform mesh {x i } n i=0 with spacing h = x i+1 x i. How can we find any local

More information

Introduction to Modern Control Systems

Introduction to Modern Control Systems Introduction to Modern Control Systems Convex Optimization, Duality and Linear Matrix Inequalities Kostas Margellos University of Oxford AIMS CDT 2016-17 Introduction to Modern Control Systems November

More information

Numerical Methods Lecture 7 - Optimization

Numerical Methods Lecture 7 - Optimization Numerical Methods Lecture 7 - Optimization Topics: numerical optimization - Newton again - Random search - Golden Section Search READING : text pgs. 331-349 Optimization - motivation What? Locating where

More information

Finding optimal configurations Adversarial search

Finding optimal configurations Adversarial search CS 171 Introduction to AI Lecture 10 Finding optimal configurations Adversarial search Milos Hauskrecht milos@cs.pitt.edu 39 Sennott Square Announcements Homework assignment is out Due on Thursday next

More information

Key Concepts: Economic Computation, Part II

Key Concepts: Economic Computation, Part II Key Concepts: Economic Computation, Part II Brent Hickman Fall, 2009 The purpose of the second section of these notes is to give you some further practice with numerical computation in MATLAB, and also

More information

INTRODUCTION AN INTRODUCTION TO DIRECT OPTIMIZATION METHODS. SIMPLE EXAMPLE: COMPASS SEARCH (Con t) SIMPLE EXAMPLE: COMPASS SEARCH

INTRODUCTION AN INTRODUCTION TO DIRECT OPTIMIZATION METHODS. SIMPLE EXAMPLE: COMPASS SEARCH (Con t) SIMPLE EXAMPLE: COMPASS SEARCH AN INTRODUCTION TO DIRECT OPTIMIZATION METHODS B Arnaud Bistoquet 03/30/04 INTRODUCTION Goal: minimization of a real-valued function f(), R n f() is differentiable + derivative-based f() can be computed

More information

Introduction to Optimization

Introduction to Optimization Introduction to Optimization Constrained Optimization Marc Toussaint U Stuttgart Constrained Optimization General constrained optimization problem: Let R n, f : R n R, g : R n R m, h : R n R l find min

More information

Machine Learning for Signal Processing Lecture 4: Optimization

Machine Learning for Signal Processing Lecture 4: Optimization Machine Learning for Signal Processing Lecture 4: Optimization 13 Sep 2015 Instructor: Bhiksha Raj (slides largely by Najim Dehak, JHU) 11-755/18-797 1 Index 1. The problem of optimization 2. Direct optimization

More information

Convex Optimization CMU-10725

Convex Optimization CMU-10725 Convex Optimization CMU-10725 Conjugate Direction Methods Barnabás Póczos & Ryan Tibshirani Conjugate Direction Methods 2 Books to Read David G. Luenberger, Yinyu Ye: Linear and Nonlinear Programming Nesterov:

More information

Efficient Tuning of SVM Hyperparameters Using Radius/Margin Bound and Iterative Algorithms

Efficient Tuning of SVM Hyperparameters Using Radius/Margin Bound and Iterative Algorithms IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 13, NO. 5, SEPTEMBER 2002 1225 Efficient Tuning of SVM Hyperparameters Using Radius/Margin Bound and Iterative Algorithms S. Sathiya Keerthi Abstract This paper

More information

Fast oriented bounding box optimization on the rotation group SO(3, R)

Fast oriented bounding box optimization on the rotation group SO(3, R) Fast oriented bounding box optimization on the rotation group SO(3, R) Chia-Tche Chang 1, Bastien Gorissen 2,3 and Samuel Melchior 1,2 chia-tche.chang@uclouvain.be bastien.gorissen@cenaero.be samuel.melchior@uclouvain.be

More information

A Provably Convergent Multifidelity Optimization Algorithm not Requiring High-Fidelity Derivatives

A Provably Convergent Multifidelity Optimization Algorithm not Requiring High-Fidelity Derivatives A Provably Convergent Multifidelity Optimization Algorithm not Requiring High-Fidelity Derivatives Multidisciplinary Design Optimization Specialist Conference April 4, 00 Andrew March & Karen Willcox Motivation

More information

Assignment in Optimization 2016

Assignment in Optimization 2016 Assignment in Optimization 2016 You can choose one of the following two alternatives Alternative 1 Nonlinear least-square fitting using the Gauss-Newton method The task is to solve an optimization problem

More information

Convexity Theory and Gradient Methods

Convexity Theory and Gradient Methods Convexity Theory and Gradient Methods Angelia Nedić angelia@illinois.edu ISE Department and Coordinated Science Laboratory University of Illinois at Urbana-Champaign Outline Convex Functions Optimality

More information

Recent Developments in Model-based Derivative-free Optimization

Recent Developments in Model-based Derivative-free Optimization Recent Developments in Model-based Derivative-free Optimization Seppo Pulkkinen April 23, 2010 Introduction Problem definition The problem we are considering is a nonlinear optimization problem with constraints:

More information

OPTIMIZATION FOR AUTOMATIC HISTORY MATCHING

OPTIMIZATION FOR AUTOMATIC HISTORY MATCHING INTERNATIONAL JOURNAL OF NUMERICAL ANALYSIS AND MODELING Volume 2, Supp, Pages 131 137 c 2005 Institute for Scientific Computing and Information OPTIMIZATION FOR AUTOMATIC HISTORY MATCHING Abstract. SHUGUANG

More information

MEI STRUCTURED MATHEMATICS. MEI conference University of Hertfordshire June C3 COURSEWORK

MEI STRUCTURED MATHEMATICS. MEI conference University of Hertfordshire June C3 COURSEWORK MEI STRUCTURED MATHEMATICS MEI conference University of Hertfordshire June 29 2009 C3 COURSEWORK What is this coursework designed to do and how do we prepare students for it? Presenter: Val Hanrahan The

More information

Lecture 9. Introduction to Numerical Techniques

Lecture 9. Introduction to Numerical Techniques Lecture 9. Introduction to Numerical Techniques Ivan Papusha CDS270 2: Mathematical Methods in Control and System Engineering May 27, 2015 1 / 25 Logistics hw8 (last one) due today. do an easy problem

More information

Theoretical Concepts of Machine Learning

Theoretical Concepts of Machine Learning Theoretical Concepts of Machine Learning Part 2 Institute of Bioinformatics Johannes Kepler University, Linz, Austria Outline 1 Introduction 2 Generalization Error 3 Maximum Likelihood 4 Noise Models 5

More information

CS 6210 Fall 2016 Bei Wang. Review Lecture What have we learnt in Scientific Computing?

CS 6210 Fall 2016 Bei Wang. Review Lecture What have we learnt in Scientific Computing? CS 6210 Fall 2016 Bei Wang Review Lecture What have we learnt in Scientific Computing? Let s recall the scientific computing pipeline observed phenomenon mathematical model discretization solution algorithm

More information

Informed search algorithms. (Based on slides by Oren Etzioni, Stuart Russell)

Informed search algorithms. (Based on slides by Oren Etzioni, Stuart Russell) Informed search algorithms (Based on slides by Oren Etzioni, Stuart Russell) The problem # Unique board configurations in search space 8-puzzle 9! = 362880 15-puzzle 16! = 20922789888000 10 13 24-puzzle

More information

Solution Methodologies for. the Smallest Enclosing Circle Problem

Solution Methodologies for. the Smallest Enclosing Circle Problem Solution Methodologies for the Smallest Enclosing Circle Problem Sheng Xu 1 Robert M. Freund 2 Jie Sun 3 Tribute. We would like to dedicate this paper to Elijah Polak. Professor Polak has made substantial

More information

Constrained optimization

Constrained optimization Constrained optimization A general constrained optimization problem has the form where The Lagrangian function is given by Primal and dual optimization problems Primal: Dual: Weak duality: Strong duality:

More information

Lecture 6 - Multivariate numerical optimization

Lecture 6 - Multivariate numerical optimization Lecture 6 - Multivariate numerical optimization Björn Andersson (w/ Jianxin Wei) Department of Statistics, Uppsala University February 13, 2014 1 / 36 Table of Contents 1 Plotting functions of two variables

More information

STRUCTURAL & MULTIDISCIPLINARY OPTIMIZATION

STRUCTURAL & MULTIDISCIPLINARY OPTIMIZATION STRUCTURAL & MULTIDISCIPLINARY OPTIMIZATION Pierre DUYSINX Patricia TOSSINGS Department of Aerospace and Mechanical Engineering Academic year 2018-2019 1 Course objectives To become familiar with the introduction

More information

Graphing Techniques. Domain (, ) Range (, ) Squaring Function f(x) = x 2 Domain (, ) Range [, ) f( x) = x 2

Graphing Techniques. Domain (, ) Range (, ) Squaring Function f(x) = x 2 Domain (, ) Range [, ) f( x) = x 2 Graphing Techniques In this chapter, we will take our knowledge of graphs of basic functions and expand our ability to graph polynomial and rational functions using common sense, zeros, y-intercepts, stretching

More information

Introduction to Design Optimization: Search Methods

Introduction to Design Optimization: Search Methods Introduction to Design Optimization: Search Methods 1-D Optimization The Search We don t know the curve. Given α, we can calculate f(α). By inspecting some points, we try to find the approximated shape

More information

Handout 2 - Root Finding using MATLAB

Handout 2 - Root Finding using MATLAB Handout 2 - Root Finding using MATLAB Middle East Technical University MATLAB has couple of built-in root finding functions. In this handout we ll have a look at fzero, roots and solve functions. It is

More information

Training of Neural Networks. Q.J. Zhang, Carleton University

Training of Neural Networks. Q.J. Zhang, Carleton University Training of Neural Networks Notation: x: input of the original modeling problem or the neural network y: output of the original modeling problem or the neural network w: internal weights/parameters of

More information

ME 555: Distributed Optimization

ME 555: Distributed Optimization ME 555: Distributed Optimization Duke University Spring 2015 1 Administrative Course: ME 555: Distributed Optimization (Spring 2015) Instructor: Time: Location: Office hours: Website: Soomin Lee (email:

More information

6 BLAS (Basic Linear Algebra Subroutines)

6 BLAS (Basic Linear Algebra Subroutines) 161 BLAS 6.1 Motivation 6 BLAS (Basic Linear Algebra Subroutines) 6.1 Motivation How to optimise programs that use a lot of linear algebra operations? Efficiency depends on but also on: processor speed

More information

Optimization III: Constrained Optimization

Optimization III: Constrained Optimization Optimization III: Constrained Optimization CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Doug James (and Justin Solomon) CS 205A: Mathematical Methods Optimization III: Constrained Optimization

More information

Module 1 Lecture Notes 2. Optimization Problem and Model Formulation

Module 1 Lecture Notes 2. Optimization Problem and Model Formulation Optimization Methods: Introduction and Basic concepts 1 Module 1 Lecture Notes 2 Optimization Problem and Model Formulation Introduction In the previous lecture we studied the evolution of optimization

More information

Image Registration Lecture 4: First Examples

Image Registration Lecture 4: First Examples Image Registration Lecture 4: First Examples Prof. Charlene Tsai Outline Example Intensity-based registration SSD error function Image mapping Function minimization: Gradient descent Derivative calculation

More information

Module 4 : Solving Linear Algebraic Equations Section 11 Appendix C: Steepest Descent / Gradient Search Method

Module 4 : Solving Linear Algebraic Equations Section 11 Appendix C: Steepest Descent / Gradient Search Method Module 4 : Solving Linear Algebraic Equations Section 11 Appendix C: Steepest Descent / Gradient Search Method 11 Appendix C: Steepest Descent / Gradient Search Method In the module on Problem Discretization

More information

AMS527: Numerical Analysis II

AMS527: Numerical Analysis II AMS527: Numerical Analysis II A Brief Overview of Finite Element Methods Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao SUNY Stony Brook AMS527: Numerical Analysis II 1 / 25 Overview Basic concepts Mathematical

More information

arxiv: v2 [math.oc] 12 Feb 2019

arxiv: v2 [math.oc] 12 Feb 2019 A CENTRAL PATH FOLLOWING METHOD USING THE NORMALIZED GRADIENTS Long Chen 1, Wenyi Chen 2, and Kai-Uwe Bletzinger 1 arxiv:1902.04040v2 [math.oc] 12 Feb 2019 1 Chair of Structural Analysis, Technical University

More information

ME 261: Numerical Analysis Lecture-12: Numerical Interpolation

ME 261: Numerical Analysis Lecture-12: Numerical Interpolation 1 ME 261: Numerical Analysis Lecture-12: Numerical Interpolation Md. Tanver Hossain Department of Mechanical Engineering, BUET http://tantusher.buet.ac.bd 2 Inverse Interpolation Problem : Given a table

More information

Animation Lecture 10 Slide Fall 2003

Animation Lecture 10 Slide Fall 2003 Animation Lecture 10 Slide 1 6.837 Fall 2003 Conventional Animation Draw each frame of the animation great control tedious Reduce burden with cel animation layer keyframe inbetween cel panoramas (Disney

More information

Generic descent algorithm Generalization to multiple dimensions Problems of descent methods, possible improvements Fixes Local minima

Generic descent algorithm Generalization to multiple dimensions Problems of descent methods, possible improvements Fixes Local minima 1 Lecture 10: descent methods Generic descent algorithm Generalization to multiple dimensions Problems of descent methods, possible improvements Fixes Local minima Gradient descent (reminder) Minimum of

More information

Convex Optimization MLSS 2015

Convex Optimization MLSS 2015 Convex Optimization MLSS 2015 Constantine Caramanis The University of Texas at Austin The Optimization Problem minimize : f (x) subject to : x X. The Optimization Problem minimize : f (x) subject to :

More information

K-Means and Gaussian Mixture Models

K-Means and Gaussian Mixture Models K-Means and Gaussian Mixture Models David Rosenberg New York University June 15, 2015 David Rosenberg (New York University) DS-GA 1003 June 15, 2015 1 / 43 K-Means Clustering Example: Old Faithful Geyser

More information

Chapter Multidimensional Gradient Method

Chapter Multidimensional Gradient Method Chapter 09.04 Multidimensional Gradient Method After reading this chapter, you should be able to: 1. Understand how multi-dimensional gradient methods are different from direct search methods. Understand

More information