Introduction. Optimization
|
|
- Flora Wilson
- 6 years ago
- Views:
Transcription
1 Introduction to Optimization Amy Langville SAMSI Undergraduate Workshop N.C. State University SAMSI 6/1/05
2 GOAL: minimize f(x 1, x 2, x 3, x 4, x 5 ) = x 2 1.5x 2x 3 + x 4 /x 5 PRIZE: $1 million # of independent variables = z = f(x 1, x 2, x 3, x 4, x 5 ) lives in R?
3 GOAL: minimize f(x 1, x 2, x 3, x 4, x 5 ) = x 2 1.5x 2x 3 + x 4 /x 5 PRIZE: $1 million # of independent variables = z = f(x 1, x 2, x 3, x 4, x 5 ) lives in R? Suppose you know little to nothing about Calculus or Optimization, could you win the prize? How?
4 GOAL: minimize f(x 1, x 2, x 3, x 4, x 5 ) = x 2 1.5x 2x 3 + x 4 /x 5 PRIZE: $1 million # of independent variables = z = f(x 1, x 2, x 3, x 4, x 5 ) lives in R? Suppose you know little to nothing about Calculus or Optimization, could you win the prize? How? Trial and Error, repeated function evaluations
5 Calculus III Review local min vs. global min vs. saddle point CPs and horizontal T. planes Local Mins and 2nd Derivative Test Global Mins and CPs and BPs Gradient = Direction of?
6 Constrained vs. Unconstrained Opt. Unconstrained min f(x, y) = x 2 + y 2 Constrained min f(x, y) = x 2 + y 2 s.t. x 0, y 0 min f(x, y) = x 2 + y 2 s.t. x > 0, y > 0 min f(x, y) = x 2 + y 2 s.t. 1 x 2,0 y 3 EVT min f(x, y) = x 2 + y 2 s.t. y = x + 2
7 Gradient Descent Methods Hillclimbers on Cloudy Day: max f(x, y) = min f(x, y) Initializations 1st-order and 2nd-order info. from partials: Gradient + Hessian Matlab function: gd(α, x 0 )
8 Iterative Methods Issues Convergence Test: what is it for gd.m? Convergence Proof: is gd.m guaranteed to converge to local min? For α > 0? For α < 0? Rate of Convergence: how many iterations? How do starting points x 0 affect number of iterations? Worst starting point for α = 4? Best?
9 Convergence of Optimization Methods global vs. local vs. stationary point vs. none Most optimization algorithms cannot guarantee convergence to global min, much less local min. However, some classes of optim. problems are particularly nice. Convex objective EX: z =.5(α x 2 + y 2 ), α > 0 Every local min is global min! Even for particularly tough optim. problems, sometimes the most popular, successful algorithms perform well on many problems, despite lack of convergence theory. Must qualify statements: I found best global min to date.
10 Your Least Squares Problem how many variables/unknowns n =? z = f(x 1, x 2,..., x n ) lives in R? can we graph z?
11 Nonsmooth, Nondifferentiable Surfaces Can t compute gradient f can t use GD Methods Line Search Methods Method of Alternating Variables (Coordinate Descent): solve series of 1-D problems what would these steps look like on contour map?
12 fminsearch and Nelder-Mead maintain basis of n + 1 points where n = # variables form simplex using these points; convex hull idea: move in direction away from worst of these points EX: n = 2, so maintain basis of 3 points living in xy-plane simplex is triangle create new simplex by moving away from worst point: reflect, expand, contract, shrink steps
13 PROPERTIES OF NELDER MEAD 117 x 3 x 3 x x x r x r x e Fig. 1. Nelder Mead simplices after a reflection and an expansion step. The original simplex is shown with a dashed line. x 3 x 3 x cc x x x 1 x c x r Fig. 2. Nelder Mead simplices after an outside contraction, an inside contraction, and a shrink. The original simplex is shown with a dashed line. then x (k+1) 1 = x (k) 1. Beyond this, whatever rule is used to define the original ordering may be applied after a shrink. We define the change index k of iteration k as the smallest index of a vertex that differs between iterations k and k + 1: (2.8) k = min{ i x (k) i x (k+1) i }. (Tie-breaking rules are needed to define a unique value of k.) When Algorithm NM terminates in step 2, 1 <k n; with termination in step 3, k = 1; with termination in step 4, 1 k n + 1; and with termination in step 5, k = 1 or 2. A statement that x j changes means that j is the change index at the relevant iteration. The rules and definitions given so far imply that, for a nonshrink iteration,
14 N-M Algorithm
15 N-M Algorithm
16 .! / #! $% ( % 10 0 SN1939A SN1939A, Residual norm as a function of λ 1 and λ Residual norm luminosity days λ λ 2 6 % 10 0 SN1939A SN1939A, Residual norm as a function of λ 1 and λ Residual norm luminosity days λ λ 2 6)
17 8 % 10 0 SN1939A SN1939A, Residual norm as a function of λ 1 and λ Residual norm luminosity days λ λ 2 Q % 10 0 SN1939A SN1939A, Residual norm as a function of λ 1 and λ Residual norm luminosity days λ λ 2 6
18 % 10 0 SN1939A SN1939A, Residual norm as a function of λ 1 and λ Residual norm luminosity days λ λ 2 68 % 10 0 SN1939A SN1939A, Residual norm as a function of λ 1 and λ Residual norm luminosity days λ λ 2 8(
19 Q(! % 10 0 SN1939A SN1939A, Residual norm as a function of λ 1 and λ Residual norm Starting guess λ 0 luminosity Optimizer λ * days λ λ 2 7 / 7, ( % * " - ( ) - * +( " $ $ " - * - * ( $,* * ( $,* ("/ - * ( $,* * ( $,* ("/ --"- ( $,* - * - * *+ ( "-$ --"- - * - ( ) + ( " $ $ " ( $,* * ( $,* * *+ ( "-$ 86
20 N-M Algorithm not proven to converge in general but, widely used easy to implement inexpensive: usually only 1-2 function evaluations/iteration no derivatives needed makes good progress at beginning of iteration history Assignments: Display N-M steps using options.display= iter ; fminsearch(fun,[x0],options); Write nested for loops in Matlab to generate grid of starting points (and later random starting points) for fminsearch to find best global min
21 Genetic/Evolutionary Algorithms at each iteration either mate or mutate possible solution vectors based on fitness of possible solution vectors as measured by objective function
Simplex of Nelder & Mead Algorithm
Simplex of N & M Simplex of Nelder & Mead Algorithm AKA the Amoeba algorithm In the class of direct search methods Unconstrained (although constraints can be added as part of error function) nonlinear
More informationToday. Golden section, discussion of error Newton s method. Newton s method, steepest descent, conjugate gradient
Optimization Last time Root finding: definition, motivation Algorithms: Bisection, false position, secant, Newton-Raphson Convergence & tradeoffs Example applications of Newton s method Root finding in
More informationIntroduction to unconstrained optimization - derivative-free methods
Introduction to unconstrained optimization - derivative-free methods Jussi Hakanen Post-doctoral researcher Office: AgC426.3 jussi.hakanen@jyu.fi Learning outcomes To understand the basic principles of
More informationCMU-Q Lecture 9: Optimization II: Constrained,Unconstrained Optimization Convex optimization. Teacher: Gianni A. Di Caro
CMU-Q 15-381 Lecture 9: Optimization II: Constrained,Unconstrained Optimization Convex optimization Teacher: Gianni A. Di Caro GLOBAL FUNCTION OPTIMIZATION Find the global maximum of the function f x (and
More informationNumerical Optimization
Numerical Optimization Quantitative Macroeconomics Raül Santaeulàlia-Llopis MOVE-UAB and Barcelona GSE Fall 2018 Raül Santaeulàlia-Llopis (MOVE-UAB,BGSE) QM: Numerical Optimization Fall 2018 1 / 46 1 Introduction
More informationA Derivative-Free Approximate Gradient Sampling Algorithm for Finite Minimax Problems
1 / 33 A Derivative-Free Approximate Gradient Sampling Algorithm for Finite Minimax Problems Speaker: Julie Nutini Joint work with Warren Hare University of British Columbia (Okanagan) III Latin American
More informationIntroduction to Optimization Problems and Methods
Introduction to Optimization Problems and Methods wjch@umich.edu December 10, 2009 Outline 1 Linear Optimization Problem Simplex Method 2 3 Cutting Plane Method 4 Discrete Dynamic Programming Problem Simplex
More informationMultivariate Numerical Optimization
Jianxin Wei March 1, 2013 Outline 1 Graphics for Function of Two Variables 2 Nelder-Mead Simplex Method 3 Steepest Descent Method 4 Newton s Method 5 Quasi-Newton s Method 6 Built-in R Function 7 Linear
More informationConstrained and Unconstrained Optimization
Constrained and Unconstrained Optimization Carlos Hurtado Department of Economics University of Illinois at Urbana-Champaign hrtdmrt2@illinois.edu Oct 10th, 2017 C. Hurtado (UIUC - Economics) Numerical
More informationLecture 25 Nonlinear Programming. November 9, 2009
Nonlinear Programming November 9, 2009 Outline Nonlinear Programming Another example of NLP problem What makes these problems complex Scalar Function Unconstrained Problem Local and global optima: definition,
More informationFast oriented bounding box optimization on the rotation group SO(3, R)
Fast oriented bounding box optimization on the rotation group SO(3, R) Chia-Tche Chang 1, Bastien Gorissen 2,3 and Samuel Melchior 1,2 chia-tche.chang@uclouvain.be bastien.gorissen@cenaero.be samuel.melchior@uclouvain.be
More informationModern Methods of Data Analysis - WS 07/08
Modern Methods of Data Analysis Lecture XV (04.02.08) Contents: Function Minimization (see E. Lohrmann & V. Blobel) Optimization Problem Set of n independent variables Sometimes in addition some constraints
More informationIntroduction to optimization methods and line search
Introduction to optimization methods and line search Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi How to find optimal solutions? Trial and error widely used in practice, not efficient and
More informationAlgoritmi di Ottimizzazione: Parte A Aspetti generali e algoritmi classici
Identificazione e Controllo Intelligente Algoritmi di Ottimizzazione: Parte A Aspetti generali e algoritmi classici David Naso A.A. 2006-2007 Identificazione e Controllo Intelligente 1 Search and optimization
More informationNelder-Mead Enhanced Extreme Learning Machine
Philip Reiner, Bogdan M. Wilamowski, "Nelder-Mead Enhanced Extreme Learning Machine", 7-th IEEE Intelligent Engineering Systems Conference, INES 23, Costa Rica, June 9-2., 29, pp. 225-23 Nelder-Mead Enhanced
More information1. Introduction. performance of numerical methods. complexity bounds. structural convex optimization. course goals and topics
1. Introduction EE 546, Univ of Washington, Spring 2016 performance of numerical methods complexity bounds structural convex optimization course goals and topics 1 1 Some course info Welcome to EE 546!
More informationNEW CERN PROTON SYNCHROTRON BEAM OPTIMIZATION TOOL
16th Int. Conf. on Accelerator and Large Experimental Control Systems ICALEPCS2017, Barcelona, Spain JACoW Publishing NEW CERN PROTON SYNCHROTRON BEAM OPTIMIZATION TOOL E. Piselli, A. Akroh CERN, Geneva,
More informationComputational Methods. Constrained Optimization
Computational Methods Constrained Optimization Manfred Huber 2010 1 Constrained Optimization Unconstrained Optimization finds a minimum of a function under the assumption that the parameters can take on
More informationCPSC 340: Machine Learning and Data Mining. Robust Regression Fall 2015
CPSC 340: Machine Learning and Data Mining Robust Regression Fall 2015 Admin Can you see Assignment 1 grades on UBC connect? Auditors, don t worry about it. You should already be working on Assignment
More informationMathematical Programming and Research Methods (Part II)
Mathematical Programming and Research Methods (Part II) 4. Convexity and Optimization Massimiliano Pontil (based on previous lecture by Andreas Argyriou) 1 Today s Plan Convex sets and functions Types
More informationLecture 12: Feasible direction methods
Lecture 12 Lecture 12: Feasible direction methods Kin Cheong Sou December 2, 2013 TMA947 Lecture 12 Lecture 12: Feasible direction methods 1 / 1 Feasible-direction methods, I Intro Consider the problem
More informationIntroduction to Optimization
Introduction to Optimization Second Order Optimization Methods Marc Toussaint U Stuttgart Planned Outline Gradient-based optimization (1st order methods) plain grad., steepest descent, conjugate grad.,
More informationNon-Derivative Optimization: Mathematics or Heuristics?
Non-Derivative Optimization: Mathematics or Heuristics? Margaret H. Wright Computer Science Department Courant Institute of Mathematical Sciences New York University Kovalevskaya Colloquium Berlin Mathematical
More informationDS Machine Learning and Data Mining I. Alina Oprea Associate Professor, CCIS Northeastern University
DS 4400 Machine Learning and Data Mining I Alina Oprea Associate Professor, CCIS Northeastern University September 20 2018 Review Solution for multiple linear regression can be computed in closed form
More informationDelaunay-based Derivative-free Optimization via Global Surrogate. Pooriya Beyhaghi, Daniele Cavaglieri and Thomas Bewley
Delaunay-based Derivative-free Optimization via Global Surrogate Pooriya Beyhaghi, Daniele Cavaglieri and Thomas Bewley May 23, 2014 Delaunay-based Derivative-free Optimization via Global Surrogate Pooriya
More informationProgramming, numerics and optimization
Programming, numerics and optimization Lecture C-4: Constrained optimization Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428 June
More informationMATH3016: OPTIMIZATION
MATH3016: OPTIMIZATION Lecturer: Dr Huifu Xu School of Mathematics University of Southampton Highfield SO17 1BJ Southampton Email: h.xu@soton.ac.uk 1 Introduction What is optimization? Optimization is
More information/ Approximation Algorithms Lecturer: Michael Dinitz Topic: Linear Programming Date: 2/24/15 Scribe: Runze Tang
600.469 / 600.669 Approximation Algorithms Lecturer: Michael Dinitz Topic: Linear Programming Date: 2/24/15 Scribe: Runze Tang 9.1 Linear Programming Suppose we are trying to approximate a minimization
More informationLecture 2 September 3
EE 381V: Large Scale Optimization Fall 2012 Lecture 2 September 3 Lecturer: Caramanis & Sanghavi Scribe: Hongbo Si, Qiaoyang Ye 2.1 Overview of the last Lecture The focus of the last lecture was to give
More informationOptimization. Industrial AI Lab.
Optimization Industrial AI Lab. Optimization An important tool in 1) Engineering problem solving and 2) Decision science People optimize Nature optimizes 2 Optimization People optimize (source: http://nautil.us/blog/to-save-drowning-people-ask-yourself-what-would-light-do)
More informationPARALLELIZATION OF THE NELDER-MEAD SIMPLEX ALGORITHM
PARALLELIZATION OF THE NELDER-MEAD SIMPLEX ALGORITHM Scott Wu Montgomery Blair High School Silver Spring, Maryland Paul Kienzle Center for Neutron Research, National Institute of Standards and Technology
More informationLecture 19: Convex Non-Smooth Optimization. April 2, 2007
: Convex Non-Smooth Optimization April 2, 2007 Outline Lecture 19 Convex non-smooth problems Examples Subgradients and subdifferentials Subgradient properties Operations with subgradients and subdifferentials
More informationLecture 3: Convex sets
Lecture 3: Convex sets Rajat Mittal IIT Kanpur We denote the set of real numbers as R. Most of the time we will be working with space R n and its elements will be called vectors. Remember that a subspace
More informationAn Evolutionary Algorithm for Minimizing Multimodal Functions
An Evolutionary Algorithm for Minimizing Multimodal Functions D.G. Sotiropoulos, V.P. Plagianakos and M.N. Vrahatis University of Patras, Department of Mamatics, Division of Computational Mamatics & Informatics,
More informationChapter 14 Global Search Algorithms
Chapter 14 Global Search Algorithms An Introduction to Optimization Spring, 2015 Wei-Ta Chu 1 Introduction We discuss various search methods that attempts to search throughout the entire feasible set.
More informationChapter 3 Numerical Methods
Chapter 3 Numerical Methods Part 1 3.1 Linearization and Optimization of Functions of Vectors 1 Problem Notation 2 Outline 3.1.1 Linearization 3.1.2 Optimization of Objective Functions 3.1.3 Constrained
More informationKey points. Assume (except for point 4) f : R n R is twice continuously differentiable. strictly above the graph of f except at x
Key points Assume (except for point 4) f : R n R is twice continuously differentiable 1 If Hf is neg def at x, then f attains a strict local max at x iff f(x) = 0 In (1), replace Hf(x) negative definite
More informationVariations on Regression Models. Prof. Bennett Math Models of Data Science 2/02/06
Variations on Regression Models Prof. Bennett Math Models of Data Science 2/02/06 Outline Steps in modeling Review of Least Squares model Model in E & K pg 24-29 Aqualsol version of E&K Other loss functions
More informationEllipsoid Algorithm :Algorithms in the Real World. Ellipsoid Algorithm. Reduction from general case
Ellipsoid Algorithm 15-853:Algorithms in the Real World Linear and Integer Programming II Ellipsoid algorithm Interior point methods First polynomial-time algorithm for linear programming (Khachian 79)
More informationAlgorithms for convex optimization
Algorithms for convex optimization Michal Kočvara Institute of Information Theory and Automation Academy of Sciences of the Czech Republic and Czech Technical University kocvara@utia.cas.cz http://www.utia.cas.cz/kocvara
More informationB553 Lecture 12: Global Optimization
B553 Lecture 12: Global Optimization Kris Hauser February 20, 2012 Most of the techniques we have examined in prior lectures only deal with local optimization, so that we can only guarantee convergence
More informationConvex Optimization Lecture 2
Convex Optimization Lecture 2 Today: Convex Analysis Center-of-mass Algorithm 1 Convex Analysis Convex Sets Definition: A set C R n is convex if for all x, y C and all 0 λ 1, λx + (1 λ)y C Operations that
More informationConvex Optimization MLSS 2015
Convex Optimization MLSS 2015 Constantine Caramanis The University of Texas at Austin The Optimization Problem minimize : f (x) subject to : x X. The Optimization Problem minimize : f (x) subject to :
More informationCharacterizing Improving Directions Unconstrained Optimization
Final Review IE417 In the Beginning... In the beginning, Weierstrass's theorem said that a continuous function achieves a minimum on a compact set. Using this, we showed that for a convex set S and y not
More informationLocalSolver 4.0: novelties and benchmarks
LocalSolver 4.0: novelties and benchmarks Thierry Benoist Julien Darlay Bertrand Estellon Frédéric Gardi Romain Megel www.localsolver.com 1/18 LocalSolver 3.1 Solver for combinatorial optimization Simple
More informationAn evolutionary annealing-simplex algorithm for global optimisation of water resource systems
FIFTH INTERNATIONAL CONFERENCE ON HYDROINFORMATICS 1-5 July 2002, Cardiff, UK C05 - Evolutionary algorithms in hydroinformatics An evolutionary annealing-simplex algorithm for global optimisation of water
More informationHST.582J / 6.555J / J Biomedical Signal and Image Processing Spring 2007
MIT OpenCourseWare http://ocw.mit.edu HST.582J / 6.555J / 16.456J Biomedical Signal and Image Processing Spring 2007 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.
More informationUnconstrained Optimization Principles of Unconstrained Optimization Search Methods
1 Nonlinear Programming Types of Nonlinear Programs (NLP) Convexity and Convex Programs NLP Solutions Unconstrained Optimization Principles of Unconstrained Optimization Search Methods Constrained Optimization
More informationICS 161 Algorithms Winter 1998 Final Exam. 1: out of 15. 2: out of 15. 3: out of 20. 4: out of 15. 5: out of 20. 6: out of 15.
ICS 161 Algorithms Winter 1998 Final Exam Name: ID: 1: out of 15 2: out of 15 3: out of 20 4: out of 15 5: out of 20 6: out of 15 total: out of 100 1. Solve the following recurrences. (Just give the solutions;
More informationMachine Learning for Signal Processing Lecture 4: Optimization
Machine Learning for Signal Processing Lecture 4: Optimization 13 Sep 2015 Instructor: Bhiksha Raj (slides largely by Najim Dehak, JHU) 11-755/18-797 1 Index 1. The problem of optimization 2. Direct optimization
More informationA Direct Search Algorithm for Global Optimization
Article A Direct Search Algorithm for Global Optimization Enrique Baeyens, *, Alberto Herreros and José R. Perán 2 Instituto de las Tecnologías Avanzadas de la Producción (ITAP), Universidad de Valladolid,
More informationKey Concepts: Economic Computation, Part II
Key Concepts: Economic Computation, Part II Brent Hickman Fall, 2009 The purpose of the second section of these notes is to give you some further practice with numerical computation in MATLAB, and also
More informationf xx (x, y) = 6 + 6x f xy (x, y) = 0 f yy (x, y) = y In general, the quantity that we re interested in is
1. Let f(x, y) = 5 + 3x 2 + 3y 2 + 2y 3 + x 3. (a) Final all critical points of f. (b) Use the second derivatives test to classify the critical points you found in (a) as a local maximum, local minimum,
More information5. GENERALIZED INVERSE SOLUTIONS
5. GENERALIZED INVERSE SOLUTIONS The Geometry of Generalized Inverse Solutions The generalized inverse solution to the control allocation problem involves constructing a matrix which satisfies the equation
More informationOptimizing the TracePro Optimization Process
Optimizing the TracePro Optimization Process A TracePro Webinar December 17, 2014 Presenter Presenter Dave Jacobsen Sr. Application Engineer Lambda Research Corporation Moderator Mike Gauvin Vice President
More information25. NLP algorithms. ˆ Overview. ˆ Local methods. ˆ Constrained optimization. ˆ Global methods. ˆ Black-box methods.
CS/ECE/ISyE 524 Introduction to Optimization Spring 2017 18 25. NLP algorithms ˆ Overview ˆ Local methods ˆ Constrained optimization ˆ Global methods ˆ Black-box methods ˆ Course wrap-up Laurent Lessard
More informationAn Improvement of Incremental Conductance MPPT Algorithm for PV Systems Based on the Nelder Mead Optimization
An Improvement of Incremental Conductance MPPT Algorithm for PV Systems Based on the Nelder Mead Optimization Tomislav Pavlović, and Željko Ban E-mail: tomislav.pavlovic@fer.hr University of Zagreb Faculty
More informationA Short SVM (Support Vector Machine) Tutorial
A Short SVM (Support Vector Machine) Tutorial j.p.lewis CGIT Lab / IMSC U. Southern California version 0.zz dec 004 This tutorial assumes you are familiar with linear algebra and equality-constrained optimization/lagrange
More informationMultidimensional Minimization
C. C. Kankelborg, 1999-11-18 (rev. 2009-Mar-24) I. INTRODUCTION Multidimensional Minimization This lecture is devoted to the task of minimization in N dimensions, which may be stated as follows: For a
More informationConvex Optimization CMU-10725
Convex Optimization CMU-10725 Ellipsoid Methods Barnabás Póczos & Ryan Tibshirani Outline Linear programs Simplex algorithm Running time: Polynomial or Exponential? Cutting planes & Ellipsoid methods for
More informationOptimization and least squares. Prof. Noah Snavely CS1114
Optimization and least squares Prof. Noah Snavely CS1114 http://cs1114.cs.cornell.edu Administrivia A5 Part 1 due tomorrow by 5pm (please sign up for a demo slot) Part 2 will be due in two weeks (4/17)
More informationFeature Detectors and Descriptors: Corners, Lines, etc.
Feature Detectors and Descriptors: Corners, Lines, etc. Edges vs. Corners Edges = maxima in intensity gradient Edges vs. Corners Corners = lots of variation in direction of gradient in a small neighborhood
More informationCS281 Section 3: Practical Optimization
CS281 Section 3: Practical Optimization David Duvenaud and Dougal Maclaurin Most parameter estimation problems in machine learning cannot be solved in closed form, so we often have to resort to numerical
More informationBilinear Programming
Bilinear Programming Artyom G. Nahapetyan Center for Applied Optimization Industrial and Systems Engineering Department University of Florida Gainesville, Florida 32611-6595 Email address: artyom@ufl.edu
More informationPackage optimsimplex
Package optimsimplex February 15, 2013 Type Package Title R port of the Scilab optimsimplex module Version 1.0-4 Date 2011-03-30 Author Sebastien Bihorel, Michael Baudin (author of the original module)
More informationShort Reminder of Nonlinear Programming
Short Reminder of Nonlinear Programming Kaisa Miettinen Dept. of Math. Inf. Tech. Email: kaisa.miettinen@jyu.fi Homepage: http://www.mit.jyu.fi/miettine Contents Background General overview briefly theory
More informationLecture 6 - Multivariate numerical optimization
Lecture 6 - Multivariate numerical optimization Björn Andersson (w/ Jianxin Wei) Department of Statistics, Uppsala University February 13, 2014 1 / 36 Table of Contents 1 Plotting functions of two variables
More informationLecture 12: convergence. Derivative (one variable)
Lecture 12: convergence More about multivariable calculus Descent methods Backtracking line search More about convexity (first and second order) Newton step Example 1: linear programming (one var., one
More informationSimplicial Global Optimization
Simplicial Global Optimization Julius Žilinskas Vilnius University, Lithuania September, 7 http://web.vu.lt/mii/j.zilinskas Global optimization Find f = min x A f (x) and x A, f (x ) = f, where A R n.
More informationGradient LASSO algoithm
Gradient LASSO algoithm Yongdai Kim Seoul National University, Korea jointly with Yuwon Kim University of Minnesota, USA and Jinseog Kim Statistical Research Center for Complex Systems, Korea Contents
More informationLECTURE NOTES Non-Linear Programming
CEE 6110 David Rosenberg p. 1 Learning Objectives LECTURE NOTES Non-Linear Programming 1. Write out the non-linear model formulation 2. Describe the difficulties of solving a non-linear programming model
More informationSolving Smart Grid Operation Problems Through Variable Neighborhood Search
1 Solving Smart Grid Operation Problems Through Variable Neighborhood Search Leonardo H. Macedo, São Paulo State University-Ilha Solteira (Brazil) John F. Franco, São Paulo State University-Rosana (Brazil)
More informationConvexity Theory and Gradient Methods
Convexity Theory and Gradient Methods Angelia Nedić angelia@illinois.edu ISE Department and Coordinated Science Laboratory University of Illinois at Urbana-Champaign Outline Convex Functions Optimality
More informationParameter Estimation of DC Motor using Adaptive Transfer Function based on Nelder-Mead Optimisation
Indonesian Journal of Electrical Engineering and Computer Science Vol. 9, No. 3, March 2018, pp. 696~702 ISSN: 2502-4752, DOI: 10.11591/ijeecs.v9.i3.pp696-702 696 Parameter Estimation of DC Motor using
More informationGeneric descent algorithm Generalization to multiple dimensions Problems of descent methods, possible improvements Fixes Local minima
1 Lecture 10: descent methods Generic descent algorithm Generalization to multiple dimensions Problems of descent methods, possible improvements Fixes Local minima Gradient descent (reminder) Minimum of
More informationConstrained Optimization COS 323
Constrained Optimization COS 323 Last time Introduction to optimization objective function, variables, [constraints] 1-dimensional methods Golden section, discussion of error Newton s method Multi-dimensional
More informationM. Sc. (Artificial Intelligence and Machine Learning)
Course Name: Advanced Python Course Code: MSCAI 122 This course will introduce students to advanced python implementations and the latest Machine Learning and Deep learning libraries, Scikit-Learn and
More informationLearning via Optimization
Lecture 7 1 Outline 1. Optimization Convexity 2. Linear regression in depth Locally weighted linear regression 3. Brief dips Logistic Regression [Stochastic] gradient ascent/descent Support Vector Machines
More informationSparse Optimization Lecture: Proximal Operator/Algorithm and Lagrange Dual
Sparse Optimization Lecture: Proximal Operator/Algorithm and Lagrange Dual Instructor: Wotao Yin July 2013 online discussions on piazza.com Those who complete this lecture will know learn the proximal
More informationConvex Optimization - Chapter 1-2. Xiangru Lian August 28, 2015
Convex Optimization - Chapter 1-2 Xiangru Lian August 28, 2015 1 Mathematical optimization minimize f 0 (x) s.t. f j (x) 0, j=1,,m, (1) x S x. (x 1,,x n ). optimization variable. f 0. R n R. objective
More informationHAM: A HYBRID ALGORITHM PREDICTION BASED MODEL FOR ELECTRICITY DEMAND
HAM: A HYBRID ALGORITHM PREDICTION BASED MODEL FOR ELECTRICITY DEMAND 1 WAHAB MUSA, 2 SALMAWATY TANSA 1 Assoc. Prof., Department of Electrical Engineering, UNIVERSITAS NEGERI GORONTALO 2 Lektor., Department
More informationOptimization. 1. Optimization. by Prof. Seungchul Lee Industrial AI Lab POSTECH. Table of Contents
Optimization by Prof. Seungchul Lee Industrial AI Lab http://isystems.unist.ac.kr/ POSTECH Table of Contents I. 1. Optimization II. 2. Solving Optimization Problems III. 3. How do we Find x f(x) = 0 IV.
More informationThe Mesh Adaptive Direct Search Algorithm for Discrete Blackbox Optimization
The Mesh Adaptive Direct Search Algorithm for Discrete Blackbox Optimization Sébastien Le Digabel Charles Audet Christophe Tribes GERAD and École Polytechnique de Montréal ICCOPT, Tokyo 2016 08 08 BBO:
More informationRevisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization. Author: Martin Jaggi Presenter: Zhongxing Peng
Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization Author: Martin Jaggi Presenter: Zhongxing Peng Outline 1. Theoretical Results 2. Applications Outline 1. Theoretical Results 2. Applications
More informationCS 188: Artificial Intelligence
CS 188: Artificial Intelligence Constraint Satisfaction Problems II and Local Search Instructors: Sergey Levine and Stuart Russell University of California, Berkeley [These slides were created by Dan Klein
More information2. Optimization problems 6
6 2.1 Examples... 7... 8 2.3 Convex sets and functions... 9 2.4 Convex optimization problems... 10 2.1 Examples 7-1 An (NP-) optimization problem P 0 is defined as follows Each instance I P 0 has a feasibility
More informationMaster of Science in Applied Geophysics Research Thesis. Triangulation for seismic modelling with optimization techniques.
Master of Science in Applied Geophysics Research Thesis Triangulation for seismic modelling with optimization techniques Weijun Wang Triangulation for seismic modelling with optimization techniques Master
More informationAssignment 2 Master Solution. Part (a) /ESD.77 Spring 2004 Multidisciplinary System Design Optimization. (a1) Decomposition
16.888/ESD.77 Spring 004 Multidisciplinary System Design Optimization Assignment Master Solution Part (a) (a1) Decomposition The following system of equations must be solved: xx x + = 0 (1) 1 3 x + 3x
More informationCS 435, 2018 Lecture 2, Date: 1 March 2018 Instructor: Nisheeth Vishnoi. Convex Programming and Efficiency
CS 435, 2018 Lecture 2, Date: 1 March 2018 Instructor: Nisheeth Vishnoi Convex Programming and Efficiency In this lecture, we formalize convex programming problem, discuss what it means to solve it efficiently
More information16.410/413 Principles of Autonomy and Decision Making
16.410/413 Principles of Autonomy and Decision Making Lecture 17: The Simplex Method Emilio Frazzoli Aeronautics and Astronautics Massachusetts Institute of Technology November 10, 2010 Frazzoli (MIT)
More informationA Brief Look at Optimization
A Brief Look at Optimization CSC 412/2506 Tutorial David Madras January 18, 2018 Slides adapted from last year s version Overview Introduction Classes of optimization problems Linear programming Steepest
More informationComputational Optimization. Constrained Optimization Algorithms
Computational Optimization Constrained Optimization Algorithms Same basic algorithms Repeat Determine descent direction Determine step size Take a step Until Optimal But now must consider feasibility,
More informationComposite Self-concordant Minimization
Composite Self-concordant Minimization Volkan Cevher Laboratory for Information and Inference Systems-LIONS Ecole Polytechnique Federale de Lausanne (EPFL) volkan.cevher@epfl.ch Paris 6 Dec 11, 2013 joint
More informationIntroduction to Machine Learning Spring 2018 Note Sparsity and LASSO. 1.1 Sparsity for SVMs
CS 189 Introduction to Machine Learning Spring 2018 Note 21 1 Sparsity and LASSO 1.1 Sparsity for SVMs Recall the oective function of the soft-margin SVM prolem: w,ξ 1 2 w 2 + C Note that if a point x
More informationUser s Guide to Climb High. (A Numerical Optimization Routine) Introduction
User s Guide to Climb High (A Numerical Optimization Routine) Hang Qian Iowa State University Introduction Welcome to Climb High, a numerical routine for unconstrained non-linear optimization problems
More informationCamera calibration. Robotic vision. Ville Kyrki
Camera calibration Robotic vision 19.1.2017 Where are we? Images, imaging Image enhancement Feature extraction and matching Image-based tracking Camera models and calibration Pose estimation Motion analysis
More informationIterative Methods for Solving Linear Problems
Iterative Methods for Solving Linear Problems When problems become too large (too many data points, too many model parameters), SVD and related approaches become impractical. Iterative Methods for Solving
More informationOptimization Plugin for RapidMiner. Venkatesh Umaashankar Sangkyun Lee. Technical Report 04/2012. technische universität dortmund
Optimization Plugin for RapidMiner Technical Report Venkatesh Umaashankar Sangkyun Lee 04/2012 technische universität dortmund Part of the work on this technical report has been supported by Deutsche Forschungsgemeinschaft
More informationEXTRA-CREDIT PROBLEMS ON SURFACES, MULTIVARIABLE FUNCTIONS AND PARTIAL DERIVATIVES
EXTRA-CREDIT PROBLEMS ON SURFACES, MULTIVARIABLE FUNCTIONS AND PARTIAL DERIVATIVES A. HAVENS These problems are for extra-credit, which is counted against lost points on quizzes or WebAssign. You do not
More informationHardware-Efficient Parallelized Optimization with COMSOL Multiphysics and MATLAB
Hardware-Efficient Parallelized Optimization with COMSOL Multiphysics and MATLAB Frommelt Thomas* and Gutser Raphael SGL Carbon GmbH *Corresponding author: Werner-von-Siemens Straße 18, 86405 Meitingen,
More information