Recent Developments in Model-based Derivative-free Optimization
|
|
- Patricia Glenn
- 6 years ago
- Views:
Transcription
1 Recent Developments in Model-based Derivative-free Optimization Seppo Pulkkinen April 23, 2010
2 Introduction Problem definition The problem we are considering is a nonlinear optimization problem with constraints: min x R n f : Rn R l i x i u i, i = 1,..., n, Ax b. We also assume that the objective function is nonconvex and not necessarily differentiable. is expensive to evaluate.
3 A Motivating Example: Image Matching Problem Given two consequent images and a region in the first image, find a matching region from the second image: Practical considerations: Difficult to find invariant measures between the images. The transformations between the images can be large. The images may be contaminated with noise.
4 Image Matching Problem: A Simple Mathematical Formulation Problem definition The aim is to find transformation parameters giving the best fit between the matched regions by solving the problem min p x Ω x I 1 (x ) I 2 (x + T (x, p)) 2. This is a nonconvex nonlinear optimization problem having a large number of local minima. nondifferentiable objective function. possibly constraints enforcing the smoothness of the solution. Implication: Local gradient-based methods are not really usable for such problems.
5 Problems With Noisy Data Smooth Noisy Consider a difference approximation of the form f f (x + he i) f (x). x i h Clearly, any local gradient approximations become unusable in the presence of noise.
6 The Traditional Approach - Taylor Series Approximations Most gradient-based algorithms employ the quadratic Taylor series approximation m(x + s) = f (x) + f (x) T s st H(x)s. This can expressed in a more generic form, that is m(x + s) = c + b T s st As, where c R, b R n and A R n n. Problem: Can the model parameters c, b and A be estimated without evaluating derivatives?
7 Interpolation-based Methods An alternative approach: Determine the model parameters c, b and A from interpolation equations m(x + y i ) = f (x + y i ), i = 1,..., Y, where Y = {y 1,..., y m } is the set of interpolation points. A model defined by the above equations only requires that f can be evaluated at the given points: no derivatives are needed. is not restricted to the small neighbourhood of x.
8 Limitations of Quadratic Models Quadratic interpolation models have several limitations: The need to solve O(n 2 ) parameters from interpolation equations. Updating the model parameters has complexity of O(n 4 ). The interpolation set must be well-poised, which leads to complex geometric conditions. Quadratic models are essentially local: they cannot model multimodal behaviour of a nonconvex function. Problem: Is there a model function that requires only O(n) parameters. requires only mild conditions for well-poisedness. can approximate functions with multiple local minima.
9 Improvement: Underdetermined Quadratic Models An alternative approach is to determine only the diagonal elements of the matrix A from interpolation equations. The off-diagonal elements of the matrix A are approximated by using the minimum Frobenius norm method (Powell, 2004, Wild, 2008). Requires only 2n + 1 model parameters. The amount of work per iteration is O(n 3 ) (Powell, 2004). Analogous to quasi-newton methods. However... this approach still has the limitations of quadratic models: it gives only local approximations.
10 A Novel Approach - Radial Basis Function Models RBF Model Function A typical radial basis function model is of the form Y m(x + s) = λ i φ( s y i ) + p(s), i=1 where λ i are weighting coefficients and p is a low-order polynomial. Such a model function addresses the questions posed above: The minimum number of interpolation points is n + 2. Can use an arbitrary number of interpolation points. Ideal for approximating functions with multiple minima.
11 Radial Basis Functions: Overview The choice of the radial basis function φ is crucial for the accuracy and numerical stability of the approximation. Commonly used radial basis functions: φ(r) = r linear φ(r) = r 3 cubic φ(r) = r 2 log r thin plate φ(r) = (γr 2 + 1) 3 2 multiquadric φ(r) = exp γr 2 gaussian r 0. Other important applications of radial basis functions are solving partial differential equations. neural networks.
12 An Illustrative Example RBF Interpolation with 30 randomly chosen interpolation points, Rastrigin function: 1.0 Function RBF model
13 An Illustrative Example Function RBF model Model Radial basis function models yield global approximations of the objective function. increasingly accurate approximations as the number of interpolation points increases.
14 Limiting Functions of Flat RBF Models (1) Examples of RBF models with adjustable shape parameter: φ(r) = (γr 2 + 1) 3 2 φ(r) = e γr 2 multiquadric gaussian The limit γ 0 (Fornberg et al., 2004): When Y = (n+1)(n+2) 2, the limit γ 0 yields under certain conditions a quadratic polynomial, i.e. Y lim γ 0 i=1 λ i φ( s y i, γ) + p(s) = 1 2 st As + b T s + c. Implication: RBF models yield accurate local approximations by letting γ 0 near a minimum.
15 Limiting Functions of flat RBF Models (2) Function Multiquadric RBF model (γ=5) Multiquadric RBF model (γ=0.05) 180 Quadratic model
16 Geomeric Conditions for RBF Interpolation (1) We are particularly interested in multiquadric RBF models Y m(x + s) = λ i (γ s y i 2 + 1) g T s + c, i=1 where the linear polynomial tail guarantees an unique interpolant (Powell, 1992). provides an estimate for the function gradient. The interpolation equations uniquely determining the model parameters are: m(x + y i ) = f (x + y i ), i = 1,..., Y Y λ i p j (y i ) = 0, j = 1,..., n + 1 i=1
17 Geomeric Conditions for RBF Interpolation (2) We denote the linear tail p(s) by n+1 i=1 c ip i (s), where {p 1,..., p n+1 } span the linear polynomial space. The interpolation equations in matrix form are [ ] [ ] Φ Π λ Π T = F, 0 c where λ = λ 1. λ Y, c = c 1. c n+1, F = f (x + y 1 ). f (x + y Y ). and Φ ij = φ( y i y j ), Π ij = p i (y j ).
18 Geomeric Conditions for RBF Interpolation (3) Necessary conditions A necessary condition for an unique solution of the interpolation equations is that rank(π) n + 1, or equivalently, at a least subset consisting of n + 1 points is linearly independent. Two approaches for ensuring this poisedness condition can be found in the literature: 1 Apply correction steps for improving the quality of the model (Powell, 2004, Scheinberg et al., 2009). 2 Avoid inserting any bad interpolation points (Marazzi and Nocedal, 2002).
19 The Trust Region Framework (1) Idea: Define a region in which the model can be considered reliable. Contour lines of f An iterative algorithm: Each new iterate x k+1 is defined by x k+1 = x k + s k, where s k minimizes the model within the current trust region.
20 The Trust Region Framework (2) Mathematical formulation: Solve the minimization problem s = arg min s {m k (x k + s) x k + s B k }, where the spherical trust region B k is defined as B k = {x F x x k < k }, and F is the set of feasible points. Also adjust the trust region radius k, if necessary: If the step s leads to sufficiently smaller function value, increase the radius, set k+1 > k. Otherwise, shrink the trust region, set k+1 < k.
21 Updating the Model Under Geometric Constraints The Constraint Condition: S = span({y 1,..., y n+1 } \ {y }). Compute vector ˆn that is orthonormal to S. The feasible region containing sufficiently linearly independent points is defined by infeasible region F = {x B k x T ˆn > γ x }. The Idea of The Algorithm: Replace some interpolation point y with a better point y +, for example, y + = s k.
22 The Special Structure of RBF Models Motivation RBF models are linear combinations of convex and concave functions. Hence, it seems natural to express the model function in the decomposed form m(x) = g(x) h(x), where g and h are convex. Implications This special structure allows developing efficient d.c. (diff-convex) algorithms for minimizing the RBF model function.
23 Diff-convex Decompositions of RBF Models The following decompositions of RBF models have been proposed in the literature (Hoai An, Vaz and Vicente, 2009): Separation of convex and concave terms: g(x) = λ i 0 λ i φ( x y i ) + p(x), h(x) = λ i <0( λ i )φ( x y i ) Regularization approach: g(x) = ρ 2 x 2 + p(x), Y h(x) = ρ 2 x 2 λ i φ( x y i ) i=1
24 The d.c. Algorithm: Preliminaries g(x) g(x)-h(x) -h(x) x The Idea of the d.c. Algorithm: Replace the concave term h(x) of f (x) = g(x) h(x) with its linear approximation, let f (x) g(x) (h(x 0 ) + h(x 0 )(x x 0 )).
25 The d.c. Algorithm: Mathematical Formulation Statement of the Algorithm: Iteratively solve the problem x k+1 = arg min x F {g(x) (h(x k) + (x x k ) T y k )}, where y k = h(x k ). Using this formulation can be beneficial, if the new problem is easier to solve can be solved more efficiently than the original problem.
26 Convexification: An Illustrative Example Idea: Convexify the function by adding a convex term ρ 2 x 2 with a large enough parameter ρ to it.
27 The d.c. Algorithm: Regularization Approach With the regularized d.c. decomposition, solving the linearized minimization problem x k+1 = arg min x F {g(x) (h(x k) + (x x k ) T y k )}, is equivalent to solving x k+1 = arg min x (x O + y k g ), x F ρ which is the projection of the term x O + y k g ρ to the set F. We have a gradient descent method requiring no line search. a convenient way to handle constraints.
28 How to determine the Regularization Parameter ρ? The sufficient condition for convexity of h The convexity of h within the trust region B is guaranteed, if ρ max x B 2 h (x), where Y h (x) = λ i φ( x y i ). i=1 It is possible to derive an upper bound for the minimum ρ that ensures convexity. When ρ gives an accurate estimate, the algorithm converges rapidly.
29 Thank you! Questions?
COMPUTATIONAL INTELLIGENCE (INTRODUCTION TO MACHINE LEARNING) SS18. Lecture 2: Linear Regression Gradient Descent Non-linear basis functions
COMPUTATIONAL INTELLIGENCE (INTRODUCTION TO MACHINE LEARNING) SS18 Lecture 2: Linear Regression Gradient Descent Non-linear basis functions LINEAR REGRESSION MOTIVATION Why Linear Regression? Simplest
More informationCOMPUTATIONAL INTELLIGENCE (CS) (INTRODUCTION TO MACHINE LEARNING) SS16. Lecture 2: Linear Regression Gradient Descent Non-linear basis functions
COMPUTATIONAL INTELLIGENCE (CS) (INTRODUCTION TO MACHINE LEARNING) SS16 Lecture 2: Linear Regression Gradient Descent Non-linear basis functions LINEAR REGRESSION MOTIVATION Why Linear Regression? Regression
More informationA Random Variable Shape Parameter Strategy for Radial Basis Function Approximation Methods
A Random Variable Shape Parameter Strategy for Radial Basis Function Approximation Methods Scott A. Sarra, Derek Sturgill Marshall University, Department of Mathematics, One John Marshall Drive, Huntington
More informationCS 450 Numerical Analysis. Chapter 7: Interpolation
Lecture slides based on the textbook Scientific Computing: An Introductory Survey by Michael T. Heath, copyright c 2018 by the Society for Industrial and Applied Mathematics. http://www.siam.org/books/cl80
More informationProgramming, numerics and optimization
Programming, numerics and optimization Lecture C-4: Constrained optimization Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428 June
More informationConvexization in Markov Chain Monte Carlo
in Markov Chain Monte Carlo 1 IBM T. J. Watson Yorktown Heights, NY 2 Department of Aerospace Engineering Technion, Israel August 23, 2011 Problem Statement MCMC processes in general are governed by non
More informationOptimization. Industrial AI Lab.
Optimization Industrial AI Lab. Optimization An important tool in 1) Engineering problem solving and 2) Decision science People optimize Nature optimizes 2 Optimization People optimize (source: http://nautil.us/blog/to-save-drowning-people-ask-yourself-what-would-light-do)
More informationOptimization for Machine Learning
Optimization for Machine Learning (Problems; Algorithms - C) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html
More informationLearning a classification of Mixed-Integer Quadratic Programming problems
Learning a classification of Mixed-Integer Quadratic Programming problems CERMICS 2018 June 29, 2018, Fréjus Pierre Bonami 1, Andrea Lodi 2, Giulia Zarpellon 2 1 CPLEX Optimization, IBM Spain 2 Polytechnique
More informationLocally Weighted Least Squares Regression for Image Denoising, Reconstruction and Up-sampling
Locally Weighted Least Squares Regression for Image Denoising, Reconstruction and Up-sampling Moritz Baecher May 15, 29 1 Introduction Edge-preserving smoothing and super-resolution are classic and important
More informationSYSTEMS OF NONLINEAR EQUATIONS
SYSTEMS OF NONLINEAR EQUATIONS Widely used in the mathematical modeling of real world phenomena. We introduce some numerical methods for their solution. For better intuition, we examine systems of two
More informationComputational Methods. Constrained Optimization
Computational Methods Constrained Optimization Manfred Huber 2010 1 Constrained Optimization Unconstrained Optimization finds a minimum of a function under the assumption that the parameters can take on
More informationPerformance Evaluation of an Interior Point Filter Line Search Method for Constrained Optimization
6th WSEAS International Conference on SYSTEM SCIENCE and SIMULATION in ENGINEERING, Venice, Italy, November 21-23, 2007 18 Performance Evaluation of an Interior Point Filter Line Search Method for Constrained
More informationMathematical Programming and Research Methods (Part II)
Mathematical Programming and Research Methods (Part II) 4. Convexity and Optimization Massimiliano Pontil (based on previous lecture by Andreas Argyriou) 1 Today s Plan Convex sets and functions Types
More informationSolution Methods Numerical Algorithms
Solution Methods Numerical Algorithms Evelien van der Hurk DTU Managment Engineering Class Exercises From Last Time 2 DTU Management Engineering 42111: Static and Dynamic Optimization (6) 09/10/2017 Class
More informationComparison of Interior Point Filter Line Search Strategies for Constrained Optimization by Performance Profiles
INTERNATIONAL JOURNAL OF MATHEMATICS MODELS AND METHODS IN APPLIED SCIENCES Comparison of Interior Point Filter Line Search Strategies for Constrained Optimization by Performance Profiles M. Fernanda P.
More informationmorphology on binary images
morphology on binary images Ole-Johan Skrede 10.05.2017 INF2310 - Digital Image Processing Department of Informatics The Faculty of Mathematics and Natural Sciences University of Oslo After original slides
More informationIE598 Big Data Optimization Summary Nonconvex Optimization
IE598 Big Data Optimization Summary Nonconvex Optimization Instructor: Niao He April 16, 2018 1 This Course Big Data Optimization Explore modern optimization theories, algorithms, and big data applications
More informationInteger Programming Theory
Integer Programming Theory Laura Galli October 24, 2016 In the following we assume all functions are linear, hence we often drop the term linear. In discrete optimization, we seek to find a solution x
More informationConstrained and Unconstrained Optimization
Constrained and Unconstrained Optimization Carlos Hurtado Department of Economics University of Illinois at Urbana-Champaign hrtdmrt2@illinois.edu Oct 10th, 2017 C. Hurtado (UIUC - Economics) Numerical
More information(1) Given the following system of linear equations, which depends on a parameter a R, 3x y + 5z = 2 4x + y + (a 2 14)z = a + 2
(1 Given the following system of linear equations, which depends on a parameter a R, x + 2y 3z = 4 3x y + 5z = 2 4x + y + (a 2 14z = a + 2 (a Classify the system of equations depending on the values of
More informationCHAPTER 6 IMPLEMENTATION OF RADIAL BASIS FUNCTION NEURAL NETWORK FOR STEGANALYSIS
95 CHAPTER 6 IMPLEMENTATION OF RADIAL BASIS FUNCTION NEURAL NETWORK FOR STEGANALYSIS 6.1 INTRODUCTION The concept of distance measure is used to associate the input and output pattern values. RBFs use
More informationLearning from Data Linear Parameter Models
Learning from Data Linear Parameter Models Copyright David Barber 200-2004. Course lecturer: Amos Storkey a.storkey@ed.ac.uk Course page : http://www.anc.ed.ac.uk/ amos/lfd/ 2 chirps per sec 26 24 22 20
More informationApproximation Methods in Optimization
Approximation Methods in Optimization The basic idea is that if you have a function that is noisy and possibly expensive to evaluate, then that function can be sampled at a few points and a fit of it created.
More informationKernels and representation
Kernels and representation Corso di AA, anno 2017/18, Padova Fabio Aiolli 20 Dicembre 2017 Fabio Aiolli Kernels and representation 20 Dicembre 2017 1 / 19 (Hierarchical) Representation Learning Hierarchical
More informationBilevel Sparse Coding
Adobe Research 345 Park Ave, San Jose, CA Mar 15, 2013 Outline 1 2 The learning model The learning algorithm 3 4 Sparse Modeling Many types of sensory data, e.g., images and audio, are in high-dimensional
More informationApproximation of a Fuzzy Function by Using Radial Basis Functions Interpolation
International Journal of Mathematical Modelling & Computations Vol. 07, No. 03, Summer 2017, 299-307 Approximation of a Fuzzy Function by Using Radial Basis Functions Interpolation R. Firouzdor a and M.
More informationA C 2 Four-Point Subdivision Scheme with Fourth Order Accuracy and its Extensions
A C 2 Four-Point Subdivision Scheme with Fourth Order Accuracy and its Extensions Nira Dyn School of Mathematical Sciences Tel Aviv University Michael S. Floater Department of Informatics University of
More informationAlgorithms for convex optimization
Algorithms for convex optimization Michal Kočvara Institute of Information Theory and Automation Academy of Sciences of the Czech Republic and Czech Technical University kocvara@utia.cas.cz http://www.utia.cas.cz/kocvara
More informationConvex Optimization MLSS 2015
Convex Optimization MLSS 2015 Constantine Caramanis The University of Texas at Austin The Optimization Problem minimize : f (x) subject to : x X. The Optimization Problem minimize : f (x) subject to :
More informationDelaunay-based Derivative-free Optimization via Global Surrogate. Pooriya Beyhaghi, Daniele Cavaglieri and Thomas Bewley
Delaunay-based Derivative-free Optimization via Global Surrogate Pooriya Beyhaghi, Daniele Cavaglieri and Thomas Bewley May 23, 2014 Delaunay-based Derivative-free Optimization via Global Surrogate Pooriya
More informationLecture 25: Bezier Subdivision. And he took unto him all these, and divided them in the midst, and laid each piece one against another: Genesis 15:10
Lecture 25: Bezier Subdivision And he took unto him all these, and divided them in the midst, and laid each piece one against another: Genesis 15:10 1. Divide and Conquer If we are going to build useful
More informationChemnitz Scientific Computing Preprints
Roman Unger Obstacle Description with Radial Basis Functions for Contact Problems in Elasticity CSC/09-01 Chemnitz Scientific Computing Preprints Impressum: Chemnitz Scientific Computing Preprints ISSN
More informationIll-Posed Problems with A Priori Information
INVERSE AND ILL-POSED PROBLEMS SERIES Ill-Posed Problems with A Priori Information V.V.Vasin andalageev HIV SPIII Utrecht, The Netherlands, 1995 CONTENTS Introduction 1 CHAPTER 1. UNSTABLE PROBLEMS 1 Base
More informationTwo-phase matrix splitting methods for asymmetric and symmetric LCP
Two-phase matrix splitting methods for asymmetric and symmetric LCP Daniel P. Robinson Department of Applied Mathematics and Statistics Johns Hopkins University Joint work with Feng, Nocedal, and Pang
More informationSurrogate Gradient Algorithm for Lagrangian Relaxation 1,2
Surrogate Gradient Algorithm for Lagrangian Relaxation 1,2 X. Zhao 3, P. B. Luh 4, and J. Wang 5 Communicated by W.B. Gong and D. D. Yao 1 This paper is dedicated to Professor Yu-Chi Ho for his 65th birthday.
More informationEnsemble methods in machine learning. Example. Neural networks. Neural networks
Ensemble methods in machine learning Bootstrap aggregating (bagging) train an ensemble of models based on randomly resampled versions of the training set, then take a majority vote Example What if you
More informationChapter 1 BACKGROUND
Chapter BACKGROUND. Introduction In many areas of mathematics and in applications of mathematics, it is often necessary to be able to infer information about some function using only a limited set of sample
More informationA fast algorithm for sparse reconstruction based on shrinkage, subspace optimization and continuation [Wen,Yin,Goldfarb,Zhang 2009]
A fast algorithm for sparse reconstruction based on shrinkage, subspace optimization and continuation [Wen,Yin,Goldfarb,Zhang 2009] Yongjia Song University of Wisconsin-Madison April 22, 2010 Yongjia Song
More informationAspects of Convex, Nonconvex, and Geometric Optimization (Lecture 1) Suvrit Sra Massachusetts Institute of Technology
Aspects of Convex, Nonconvex, and Geometric Optimization (Lecture 1) Suvrit Sra Massachusetts Institute of Technology Hausdorff Institute for Mathematics (HIM) Trimester: Mathematics of Signal Processing
More informationFunction approximation using RBF network. 10 basis functions and 25 data points.
1 Function approximation using RBF network F (x j ) = m 1 w i ϕ( x j t i ) i=1 j = 1... N, m 1 = 10, N = 25 10 basis functions and 25 data points. Basis function centers are plotted with circles and data
More informationLagrangian methods for the regularization of discrete ill-posed problems. G. Landi
Lagrangian methods for the regularization of discrete ill-posed problems G. Landi Abstract In many science and engineering applications, the discretization of linear illposed problems gives rise to large
More informationParallel and Distributed Sparse Optimization Algorithms
Parallel and Distributed Sparse Optimization Algorithms Part I Ruoyu Li 1 1 Department of Computer Science and Engineering University of Texas at Arlington March 19, 2015 Ruoyu Li (UTA) Parallel and Distributed
More informationIntroduction to Optimization Problems and Methods
Introduction to Optimization Problems and Methods wjch@umich.edu December 10, 2009 Outline 1 Linear Optimization Problem Simplex Method 2 3 Cutting Plane Method 4 Discrete Dynamic Programming Problem Simplex
More informationCPSC 340: Machine Learning and Data Mining. Robust Regression Fall 2015
CPSC 340: Machine Learning and Data Mining Robust Regression Fall 2015 Admin Can you see Assignment 1 grades on UBC connect? Auditors, don t worry about it. You should already be working on Assignment
More informationIncremental Gradient, Subgradient, and Proximal Methods for Convex Optimization: A Survey. Chapter 4 : Optimization for Machine Learning
Incremental Gradient, Subgradient, and Proximal Methods for Convex Optimization: A Survey Chapter 4 : Optimization for Machine Learning Summary of Chapter 2 Chapter 2: Convex Optimization with Sparsity
More informationContents. I Basics 1. Copyright by SIAM. Unauthorized reproduction of this article is prohibited.
page v Preface xiii I Basics 1 1 Optimization Models 3 1.1 Introduction... 3 1.2 Optimization: An Informal Introduction... 4 1.3 Linear Equations... 7 1.4 Linear Optimization... 10 Exercises... 12 1.5
More informationINTRODUCTION TO LINEAR AND NONLINEAR PROGRAMMING
INTRODUCTION TO LINEAR AND NONLINEAR PROGRAMMING DAVID G. LUENBERGER Stanford University TT ADDISON-WESLEY PUBLISHING COMPANY Reading, Massachusetts Menlo Park, California London Don Mills, Ontario CONTENTS
More informationClustering K-means. Machine Learning CSEP546 Carlos Guestrin University of Washington February 18, Carlos Guestrin
Clustering K-means Machine Learning CSEP546 Carlos Guestrin University of Washington February 18, 2014 Carlos Guestrin 2005-2014 1 Clustering images Set of Images [Goldberger et al.] Carlos Guestrin 2005-2014
More informationCLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS
CLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS CHAPTER 4 CLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS 4.1 Introduction Optical character recognition is one of
More informationCalibration by Optimization Without Using Derivatives
Calibration by Optimization Without Using Derivatives Markus Lazar 1, Fakultät für Ingenieurwissenschaften University of Applied Sciences, Rosenheim, Germany Florian Jarre 1, Mathematisches Institut, University
More informationTheoretical Concepts of Machine Learning
Theoretical Concepts of Machine Learning Part 2 Institute of Bioinformatics Johannes Kepler University, Linz, Austria Outline 1 Introduction 2 Generalization Error 3 Maximum Likelihood 4 Noise Models 5
More informationREAL-CODED GENETIC ALGORITHMS CONSTRAINED OPTIMIZATION. Nedim TUTKUN
REAL-CODED GENETIC ALGORITHMS CONSTRAINED OPTIMIZATION Nedim TUTKUN nedimtutkun@gmail.com Outlines Unconstrained Optimization Ackley s Function GA Approach for Ackley s Function Nonlinear Programming Penalty
More informationCharacterizing Improving Directions Unconstrained Optimization
Final Review IE417 In the Beginning... In the beginning, Weierstrass's theorem said that a continuous function achieves a minimum on a compact set. Using this, we showed that for a convex set S and y not
More informationIntroduction to Computer Graphics. Modeling (3) April 27, 2017 Kenshi Takayama
Introduction to Computer Graphics Modeling (3) April 27, 2017 Kenshi Takayama Solid modeling 2 Solid models Thin shapes represented by single polygons Unorientable Clear definition of inside & outside
More informationIntroduction to optimization methods and line search
Introduction to optimization methods and line search Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi How to find optimal solutions? Trial and error widely used in practice, not efficient and
More informationLinear Methods for Regression and Shrinkage Methods
Linear Methods for Regression and Shrinkage Methods Reference: The Elements of Statistical Learning, by T. Hastie, R. Tibshirani, J. Friedman, Springer 1 Linear Regression Models Least Squares Input vectors
More informationA projected Hessian matrix for full waveform inversion Yong Ma and Dave Hale, Center for Wave Phenomena, Colorado School of Mines
A projected Hessian matrix for full waveform inversion Yong Ma and Dave Hale, Center for Wave Phenomena, Colorado School of Mines SUMMARY A Hessian matrix in full waveform inversion (FWI) is difficult
More informationChapter 3 Numerical Methods
Chapter 3 Numerical Methods Part 1 3.1 Linearization and Optimization of Functions of Vectors 1 Problem Notation 2 Outline 3.1.1 Linearization 3.1.2 Optimization of Objective Functions 3.1.3 Constrained
More informationA Truncated Newton Method in an Augmented Lagrangian Framework for Nonlinear Programming
A Truncated Newton Method in an Augmented Lagrangian Framework for Nonlinear Programming Gianni Di Pillo (dipillo@dis.uniroma1.it) Giampaolo Liuzzi (liuzzi@iasi.cnr.it) Stefano Lucidi (lucidi@dis.uniroma1.it)
More informationSupport Vector Machines.
Support Vector Machines srihari@buffalo.edu SVM Discussion Overview 1. Overview of SVMs 2. Margin Geometry 3. SVM Optimization 4. Overlapping Distributions 5. Relationship to Logistic Regression 6. Dealing
More information1.7.1 Laplacian Smoothing
1.7.1 Laplacian Smoothing 320491: Advanced Graphics - Chapter 1 434 Theory Minimize energy functional total curvature estimate by polynomial-fitting non-linear (very slow!) 320491: Advanced Graphics -
More informationEfficient Tuning of SVM Hyperparameters Using Radius/Margin Bound and Iterative Algorithms
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 13, NO. 5, SEPTEMBER 2002 1225 Efficient Tuning of SVM Hyperparameters Using Radius/Margin Bound and Iterative Algorithms S. Sathiya Keerthi Abstract This paper
More informationNumerical experience with a derivative-free trust-funnel method for nonlinear optimization problems with general nonlinear constraints by Ph. R. Sampaio and Ph. L. Toint Report naxys-3-25 August 2, 25
More informationModern Methods of Data Analysis - WS 07/08
Modern Methods of Data Analysis Lecture XV (04.02.08) Contents: Function Minimization (see E. Lohrmann & V. Blobel) Optimization Problem Set of n independent variables Sometimes in addition some constraints
More informationIterative Algorithms I: Elementary Iterative Methods and the Conjugate Gradient Algorithms
Iterative Algorithms I: Elementary Iterative Methods and the Conjugate Gradient Algorithms By:- Nitin Kamra Indian Institute of Technology, Delhi Advisor:- Prof. Ulrich Reude 1. Introduction to Linear
More informationAdaptive Node Selection in Periodic Radial Basis Function Interpolations
Adaptive Node Selection in Periodic Radial Basis Function Interpolations Muhammad Shams Dept. of Mathematics UMass Dartmouth Dartmouth MA 02747 Email: mshams@umassd.edu December 19, 2011 Abstract In RBFs,
More informationProjection-Based Methods in Optimization
Projection-Based Methods in Optimization Charles Byrne (Charles Byrne@uml.edu) http://faculty.uml.edu/cbyrne/cbyrne.html Department of Mathematical Sciences University of Massachusetts Lowell Lowell, MA
More informationNeural Networks. CE-725: Statistical Pattern Recognition Sharif University of Technology Spring Soleymani
Neural Networks CE-725: Statistical Pattern Recognition Sharif University of Technology Spring 2013 Soleymani Outline Biological and artificial neural networks Feed-forward neural networks Single layer
More informationRadial Basis Function Networks: Algorithms
Radial Basis Function Networks: Algorithms Neural Computation : Lecture 14 John A. Bullinaria, 2015 1. The RBF Mapping 2. The RBF Network Architecture 3. Computational Power of RBF Networks 4. Training
More informationBayesian Methods in Vision: MAP Estimation, MRFs, Optimization
Bayesian Methods in Vision: MAP Estimation, MRFs, Optimization CS 650: Computer Vision Bryan S. Morse Optimization Approaches to Vision / Image Processing Recurring theme: Cast vision problem as an optimization
More informationCOMPUTATIONAL INTELLIGENCE
COMPUTATIONAL INTELLIGENCE Radial Basis Function Networks Adrian Horzyk Preface Radial Basis Function Networks (RBFN) are a kind of artificial neural networks that use radial basis functions (RBF) as activation
More informationDerivative Free Optimization Methods: A Brief, Opinionated, and Incomplete Look at a Few Recent Developments
Derivative Free Optimization Methods: A Brief, Opinionated, and Incomplete Look at a Few Recent Developments Margaret H. Wright Computer Science Department Courant Institute of Mathematical Sciences New
More informationIntroduction to unconstrained optimization - derivative-free methods
Introduction to unconstrained optimization - derivative-free methods Jussi Hakanen Post-doctoral researcher Office: AgC426.3 jussi.hakanen@jyu.fi Learning outcomes To understand the basic principles of
More informationSearch direction improvement for gradient-based optimization problems
Computer Aided Optimum Design in Engineering IX 3 Search direction improvement for gradient-based optimization problems S Ganguly & W L Neu Aerospace and Ocean Engineering, Virginia Tech, USA Abstract
More informationA C 2 Four-Point Subdivision Scheme with Fourth Order Accuracy and its Extensions
A C 2 Four-Point Subdivision Scheme with Fourth Order Accuracy and its Extensions Nira Dyn Michael S. Floater Kai Hormann Abstract. We present a new four-point subdivision scheme that generates C 2 curves.
More informationCollocation and optimization initialization
Boundary Elements and Other Mesh Reduction Methods XXXVII 55 Collocation and optimization initialization E. J. Kansa 1 & L. Ling 2 1 Convergent Solutions, USA 2 Hong Kong Baptist University, Hong Kong
More informationIterative methods for use with the Fast Multipole Method
Iterative methods for use with the Fast Multipole Method Ramani Duraiswami Perceptual Interfaces and Reality Lab. Computer Science & UMIACS University of Maryland, College Park, MD Joint work with Nail
More informationPost-Processing Radial Basis Function Approximations: A Hybrid Method
Post-Processing Radial Basis Function Approximations: A Hybrid Method Muhammad Shams Dept. of Mathematics UMass Dartmouth Dartmouth MA 02747 Email: mshams@umassd.edu August 4th 2011 Abstract With the use
More informationA Brief Look at Optimization
A Brief Look at Optimization CSC 412/2506 Tutorial David Madras January 18, 2018 Slides adapted from last year s version Overview Introduction Classes of optimization problems Linear programming Steepest
More informationMVE165/MMG630, Applied Optimization Lecture 8 Integer linear programming algorithms. Ann-Brith Strömberg
MVE165/MMG630, Integer linear programming algorithms Ann-Brith Strömberg 2009 04 15 Methods for ILP: Overview (Ch. 14.1) Enumeration Implicit enumeration: Branch and bound Relaxations Decomposition methods:
More informationNonlinear Programming
Nonlinear Programming SECOND EDITION Dimitri P. Bertsekas Massachusetts Institute of Technology WWW site for book Information and Orders http://world.std.com/~athenasc/index.html Athena Scientific, Belmont,
More informationModule 1 Lecture Notes 2. Optimization Problem and Model Formulation
Optimization Methods: Introduction and Basic concepts 1 Module 1 Lecture Notes 2 Optimization Problem and Model Formulation Introduction In the previous lecture we studied the evolution of optimization
More informationExperimental Data and Training
Modeling and Control of Dynamic Systems Experimental Data and Training Mihkel Pajusalu Alo Peets Tartu, 2008 1 Overview Experimental data Designing input signal Preparing data for modeling Training Criterion
More informationAM205: lecture 2. 1 These have been shifted to MD 323 for the rest of the semester.
AM205: lecture 2 Luna and Gary will hold a Python tutorial on Wednesday in 60 Oxford Street, Room 330 Assignment 1 will be posted this week Chris will hold office hours on Thursday (1:30pm 3:30pm, Pierce
More informationSID-PSM: A PATTERN SEARCH METHOD GUIDED BY SIMPLEX DERIVATIVES FOR USE IN DERIVATIVE-FREE OPTIMIZATION
SID-PSM: A PATTERN SEARCH METHOD GUIDED BY SIMPLEX DERIVATIVES FOR USE IN DERIVATIVE-FREE OPTIMIZATION A. L. CUSTÓDIO AND L. N. VICENTE Abstract. SID-PSM (version 1.3) is a suite of MATLAB [1] functions
More informationIntroduction to Optimization
Introduction to Optimization Second Order Optimization Methods Marc Toussaint U Stuttgart Planned Outline Gradient-based optimization (1st order methods) plain grad., steepest descent, conjugate grad.,
More informationNumerical Optimization: Introduction and gradient-based methods
Numerical Optimization: Introduction and gradient-based methods Master 2 Recherche LRI Apprentissage Statistique et Optimisation Anne Auger Inria Saclay-Ile-de-France November 2011 http://tao.lri.fr/tiki-index.php?page=courses
More informationLecture 19: November 5
0-725/36-725: Convex Optimization Fall 205 Lecturer: Ryan Tibshirani Lecture 9: November 5 Scribes: Hyun Ah Song Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These notes have not
More informationFast Radial Basis Functions for Engineering Applications. Prof. Marco Evangelos Biancolini University of Rome Tor Vergata
Fast Radial Basis Functions for Engineering Applications Prof. Marco Evangelos Biancolini University of Rome Tor Vergata Outline 2 RBF background Fast RBF on HPC Engineering Applications Mesh morphing
More informationNotes on Robust Estimation David J. Fleet Allan Jepson March 30, 005 Robust Estimataion. The field of robust statistics [3, 4] is concerned with estimation problems in which the data contains gross errors,
More informationConvex Optimization / Homework 2, due Oct 3
Convex Optimization 0-725/36-725 Homework 2, due Oct 3 Instructions: You must complete Problems 3 and either Problem 4 or Problem 5 (your choice between the two) When you submit the homework, upload a
More informationLearning via Optimization
Lecture 7 1 Outline 1. Optimization Convexity 2. Linear regression in depth Locally weighted linear regression 3. Brief dips Logistic Regression [Stochastic] gradient ascent/descent Support Vector Machines
More informationRBF Interpolation with CSRBF of Large Data Sets
RBF Interpolation with CSRBF of Large Data Sets Vaclav Skala University of West Bohemia, Plzen, Czech Republic. www.vaclavskala.eu Abstract This contribution presents a new analysis of properties of the
More information3 Nonlinear Regression
CSC 4 / CSC D / CSC C 3 Sometimes linear models are not sufficient to capture the real-world phenomena, and thus nonlinear models are necessary. In regression, all such models will have the same basic
More informationConvex or non-convex: which is better?
Sparsity Amplified Ivan Selesnick Electrical and Computer Engineering Tandon School of Engineering New York University Brooklyn, New York March 7 / 4 Convex or non-convex: which is better? (for sparse-regularized
More informationarxiv: v1 [math.na] 20 Sep 2016
arxiv:1609.06236v1 [math.na] 20 Sep 2016 A Local Mesh Modification Strategy for Interface Problems with Application to Shape and Topology Optimization P. Gangl 1,2 and U. Langer 3 1 Doctoral Program Comp.
More informationlecture 10: B-Splines
9 lecture : -Splines -Splines: a basis for splines Throughout our discussion of standard polynomial interpolation, we viewed P n as a linear space of dimension n +, and then expressed the unique interpolating
More informationMatching. Compare region of image to region of image. Today, simplest kind of matching. Intensities similar.
Matching Compare region of image to region of image. We talked about this for stereo. Important for motion. Epipolar constraint unknown. But motion small. Recognition Find object in image. Recognize object.
More informationThe Pre-Image Problem in Kernel Methods
The Pre-Image Problem in Kernel Methods James Kwok Ivor Tsang Department of Computer Science Hong Kong University of Science and Technology Hong Kong The Pre-Image Problem in Kernel Methods ICML-2003 1
More information