DC Programming: A brief tutorial. Andres Munoz Medina Courant Institute of Mathematical Sciences
|
|
- Suzan Glenn
- 6 years ago
- Views:
Transcription
1 DC Programming: A brief tutorial. Andres Munoz Medina Courant Institute of Mathematical Sciences munoz@cims.nyu.edu
2 Difference of Convex Functions Definition: A function exists f is said to be DC if there g, h convex functions such that f = g h Definition: A function is locally DC if for every there exists a neighborhood U such that f U is DC. x
3 Contents Motivation. DC functions and properties. DC programming and DCA. CCCP and convergence results. Global optimization algorithms.
4 Motivation Not all machine learning problems are convex anymore Transductive SVM s [Wang et. al 2003] Kernel learning [Argyriou et. al 2004] Structure prediction [Narasinham 2012] Auction mechanism design [MM and Muñoz]
5 Notation Let g be a convex function. The conjugate of g is defined as g g (y) =supy, x g(x) x For >0, g(x 0 ) denotes the subdifferential of g at, i.e. x 0 g(x 0 )={v R n g(x) g(x 0 )+x x 0,v g(x 0 ) will denote the exact subdifferential.
6 DC functions A function f : R R is DC iff is the integral of a function of bounded variation. [Hartman 59] A locally DC function is globally DC. [Hartman 59] All twice continuously differentiable functions are DC. Closed under sum, negation, supremum and products.
7 DC programming DC programming refers to optimization problems of the form. min x g(x) h(x) More generally, for f i (x) DC functions min x g(x) h(x) subject to f i (x) 0
8 Global optimality conditions. A point x is a global solution if and only if h(x ) g(x ) Let w =infg(x) h(x), then a point x is a global solution if and only if x 0=inf x { h(x)+t g(x) t w }
9 Global optimality conditions. A point x is a global solution if and only if h(x ) g(x ) Let w =infg(x) h(x), then a point x is a global solution if and only if x 0=inf x { h(x)+t g(x) t w }
10 Global optimality conditions. A point x is a global solution if and only if h(x ) g(x ) Let w =infg(x) h(x), then a point x is a global solution if and only if x 0=inf x { h(x)+t g(x) t w }
11 Local optimality conditions If x verifies h(x ) int g(x ), then is a strict local minimizer of g h
12 DC duality By definition of conjugate function inf x g(x) h(x) =inf x =inf x =inf y g(x) (supx, y h (y)) inf y This is the dual of the original problem y h (y)+g(x) x, y h (y) g (y)
13 DC algorithm. We want to find a sequence x k that decreases the function at every step. Use duality. If y h(x 0 ) then h (y) g (y) =x 0,y h(x 0 ) sup(x, y g(x)) x =inf x g(x) h(x)+x 0 x, y g(x 0 ) h(x 0 )
14 DC algorithm Solve the partial problems S(x ) inf{h (y) g (y) :y h(x )} T (y ) inf{g(x) h(x) :x g (y )} Choose y and. k S(x k ) x k+1 T (y k ) Solve concave minimization problems. Simplified DCA y k h(x k ) x k g (y k )
15 DCA as CCCP If the function is differentiable, the simplified DCA becomes y k = h(x k ) and x k+1 g (y k ) Equivalent to
16 DCA as CCCP If the function is differentiable, the simplified DCA becomes y k = h(x k ) and x k+1 g (y k ) Equivalent to
17 DCA as CCCP If the function is differentiable, the simplified DCA becomes y k = h(x k ) and x k+1 g (y k ) x k+1 argmin g(x) x, h(x k ) Equivalent to
18 DCA as CCCP If the function is differentiable, the simplified DCA becomes y k = h(x k ) and x k+1 g (y k ) x k+1 argmin g(x) x, h(x k ) Equivalent to x k+1 argmin g(x) h(x k ) x x k, h(x k )
19 CCCP as a majorization minimization algorithm To minimize f, MM algorithms build a majorization function F such that f(x) F (x, y) x, y f(x) =F (x, x) x Do coordinate descent on F In our scenario F (x, y) =g(x) h(y) x y, h(y)
20 Convergence results Unconstrained DC functions: Convergence to a local minimum (no rate of convergence). Bound depends on moduli of convexity. [PD Tao, LT Hoai An 97] Unconstrained smooth optimization: Linear or almost quadratic convergence depending on curvature [Roweis et. al 03] Constrained smooth optimization: Convergence without rate using Zangwill s theory. [Lanckriet, Sriperumbudur 09]
21 Global convergence NP-hard in general: Minimizing a quadratic function with one negative eigenvalue with linear constraints. [Pardalos 91] Mostly branch and bound methods and cutting plane methods [H. Tuy 03, Horst and Thoai 99] Some successful results with low rank functions.
22 Cutting plane algorithm[h. Tuy 03] All limit points of this algorithm are global minimizers Finite convergence for piecewise linear functions (Conjecture). Keeps track of exponentially many vertices.
23 Cutting plane algorithm[h. Tuy 03] All limit points of this algorithm are global minimizers Finite convergence for piecewise linear functions (Conjecture). Keeps track of exponentially many vertices.
24 Cutting plane algorithm[h. Tuy 03] All limit points of this algorithm are global minimizers Finite convergence for piecewise linear functions (Conjecture). Keeps track of exponentially many vertices.
25 Cutting plane algorithm[h. Tuy 03] All limit points of this algorithm are global minimizers Finite convergence for piecewise linear functions (x 0,t 0 ) (Conjecture). Keeps track of exponentially many vertices.
26 Cutting plane algorithm[h. Tuy 03] All limit points of this algorithm are global minimizers Finite convergence for piecewise linear functions (x 0,t 0 ) (Conjecture). Keeps track of exponentially many vertices.
27 Cutting plane algorithm[h. Tuy 03] All limit points of this algorithm are global minimizers Finite convergence for piecewise linear functions (x 0,t 0 ) (Conjecture). Keeps track of exponentially many vertices.
28 Cutting plane algorithm[h. Tuy 03] All limit points of this algorithm are global minimizers Finite convergence for piecewise linear functions (Conjecture). Keeps track of exponentially many vertices.
29 Cutting plane algorithm[h. Tuy 03] All limit points of this algorithm are global minimizers Finite convergence for piecewise linear functions (x 1,t 1 ) (Conjecture). Keeps track of exponentially many vertices.
30 Cutting plane algorithm[h. Tuy 03] All limit points of this algorithm are global minimizers Finite convergence for piecewise linear functions (x 1,t 1 ) (Conjecture). Keeps track of exponentially many vertices.
31 Cutting plane algorithm[h. Tuy 03] All limit points of this algorithm are global minimizers Finite convergence for piecewise linear functions (x 1,t 1 ) (Conjecture). Keeps track of exponentially many vertices.
32 Cutting plane algorithm[h. Tuy 03] All limit points of this algorithm are global minimizers Finite convergence for piecewise linear functions (Conjecture). Keeps track of exponentially many vertices.
33 Cutting plane algorithm[h. Tuy 03] All limit points of this algorithm are global minimizers Finite convergence for piecewise linear functions (x 2,t 2 ) (Conjecture). Keeps track of exponentially many vertices.
34 Cutting plane algorithm[h. Tuy 03] All limit points of this algorithm are global minimizers Finite convergence for piecewise linear functions (x 2,t 2 ) (Conjecture). Keeps track of exponentially many vertices.
35 Cutting plane algorithm[h. Tuy 03] All limit points of this algorithm are global minimizers Finite convergence for piecewise linear functions (x 2,t 2 ) (Conjecture). Keeps track of exponentially many vertices.
36 Cutting plane algorithm[h. Tuy 03] All limit points of this algorithm are global minimizers Finite convergence for piecewise linear functions (Conjecture). Keeps track of exponentially many vertices.
37 Cutting plane algorithm[h. Tuy 03] All limit points of this algorithm are global minimizers Finite convergence for piecewise linear functions (x 3,t 3 ) (Conjecture). Keeps track of exponentially many vertices.
38 Cutting plane algorithm[h. Tuy 03] All limit points of this algorithm are global minimizers Finite convergence for piecewise linear functions (x 3,t 3 ) (Conjecture). Keeps track of exponentially many vertices.
39 Cutting plane algorithm[h. Tuy 03] All limit points of this algorithm are global minimizers Finite convergence for piecewise linear functions (x 3,t 3 ) (Conjecture). Keeps track of exponentially many vertices.
40 Branch and bound.[hoarst and Thoai 99] Main idea is to keep upper and lower bounds of g h on simplices. S k Upper bound: Evaluate g(x k ) h(x k ) for x k S k. Use DCA as subroutine for better bounds. Lower bound: If (v i ) n+1 n+1 and. Solve x = i=1 α i v i are the vertices of i=1 S k min α g i α i v i α i h(v i )
41 Low rank optimization and polynomial guarantees[goyal and Ravi 08] A function f : R n R has rank k n if there exists g : R k R and α 1,...,α k R n such that f(x) =g(α 1 x,..., α k x) Most examples in Economy literature. For a quasi-concave function to solve min f(x). x C f we want Can always transform DC programs to this type of problem.
42 Algorithm Let g satisfy the following conditions. The gradient g(y) 0 g(λy) λ c g(y) for all λ>1 and some c α i x>0 for all x P There is an algorithm that finds with f(x) (1 + )f(x. ) in O c k k x
43 Further reading Farkas type results and duality for DC programs with convex constraints. [Dinh et. al 13] On DC functions and mappings [Duda et. al 01]
44 Open problems Local rate of convergence for constrained DC programs. Is there a condition under which DCA finds global optima. For instance g h might not be convex but h g might. Finite convergence of cutting plane methods.
45 References Dinh, Nguyen, Guy Vallet, and T. T. A. Nghia. "Farkas-type results and duality for DC programs with convex constraints." (2006). Goyal, Vineet, and R. Ravi. "An FPTAS for minimizing a class of lowrank quasi-concave functions over a convex set." Operations Research Letters 41.2 (2013): Tao, Pham Dinh, and Le Thi Hoai An. "Convex analysis approach to DC programming: theory, algorithms and applications." Acta Mathematica Vietnamica 22.1 (1997): Horst, R., and Ng V. Thoai. "DC programming: overview." Journal of Optimization Theory and Applications (1999): Tuy, H. "On global optimality conditions and cutting plane algorithms." Journal of optimization theory and applications (2003): Yuille, Alan L., and Anand Rangarajan. "The concave-convex procedure."neural Computation 15.4 (2003):
46 References Salakhutdinov, Ruslan, Sam Roweis, and Zoubin Ghahramani. "On the convergence of bound optimization algorithms." Proceedings of the Nineteenth conference on Uncertainty in Artificial Intelligence. Morgan Kaufmann Publishers Inc., Hartman, Philip. "On functions representable as a difference of convex functions." Pacific J. Math 9.3 (1959): Hiriart-Urruty, J-B. "Generalized Differentiability/Duality and Optimization for Problems Dealing with Differences of Convex Functions." Convexity and duality in optimization. Springer Berlin Heidelberg,
Bilinear Programming
Bilinear Programming Artyom G. Nahapetyan Center for Applied Optimization Industrial and Systems Engineering Department University of Florida Gainesville, Florida 32611-6595 Email address: artyom@ufl.edu
More informationNesterov's Smoothing Technique and Minimizing Differences of Convex Functions for Hierarchical Clustering
Portland State University PDXScholar Mathematics and Statistics Faculty Publications and Presentations Fariborz Maseeh Department of Mathematics and Statistics 3-017 Nesterov's Smoothing Technique and
More informationDavid G. Luenberger Yinyu Ye. Linear and Nonlinear. Programming. Fourth Edition. ö Springer
David G. Luenberger Yinyu Ye Linear and Nonlinear Programming Fourth Edition ö Springer Contents 1 Introduction 1 1.1 Optimization 1 1.2 Types of Problems 2 1.3 Size of Problems 5 1.4 Iterative Algorithms
More informationThe Maximum Ratio Clique Problem: A Continuous Optimization Approach and Some New Results
The Maximum Ratio Clique Problem: A Continuous Optimization Approach and Some New Results Mahdi Moeini Chair of Business Information Systems and Operations Research (BISOR), Technical University of Kaiserslautern,
More informationPrimal and Dual Methods for Optimisation over the Non-dominated Set of a Multi-objective Programme and Computing the Nadir Point
Primal and Dual Methods for Optimisation over the Non-dominated Set of a Multi-objective Programme and Computing the Nadir Point Ethan Liu Supervisor: Professor Matthias Ehrgott Lancaster University Outline
More informationConvexity Theory and Gradient Methods
Convexity Theory and Gradient Methods Angelia Nedić angelia@illinois.edu ISE Department and Coordinated Science Laboratory University of Illinois at Urbana-Champaign Outline Convex Functions Optimality
More informationA PARAMETRIC SIMPLEX METHOD FOR OPTIMIZING A LINEAR FUNCTION OVER THE EFFICIENT SET OF A BICRITERIA LINEAR PROBLEM. 1.
ACTA MATHEMATICA VIETNAMICA Volume 21, Number 1, 1996, pp. 59 67 59 A PARAMETRIC SIMPLEX METHOD FOR OPTIMIZING A LINEAR FUNCTION OVER THE EFFICIENT SET OF A BICRITERIA LINEAR PROBLEM NGUYEN DINH DAN AND
More informationcontrol polytope. These points are manipulated by a descent method to compute a candidate global minimizer. The second method is described in Section
Some Heuristics and Test Problems for Nonconvex Quadratic Programming over a Simplex Ivo Nowak September 3, 1998 Keywords:global optimization, nonconvex quadratic programming, heuristics, Bezier methods,
More informationAn FPTAS for minimizing the product of two non-negative linear cost functions
Math. Program., Ser. A DOI 10.1007/s10107-009-0287-4 SHORT COMMUNICATION An FPTAS for minimizing the product of two non-negative linear cost functions Vineet Goyal Latife Genc-Kaya R. Ravi Received: 2
More informationConvexity and Optimization
Convexity and Optimization Richard Lusby Department of Management Engineering Technical University of Denmark Today s Material Extrema Convex Function Convex Sets Other Convexity Concepts Unconstrained
More informationConvex Optimization MLSS 2015
Convex Optimization MLSS 2015 Constantine Caramanis The University of Texas at Austin The Optimization Problem minimize : f (x) subject to : x X. The Optimization Problem minimize : f (x) subject to :
More informationShiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 2. Convex Optimization
Shiqian Ma, MAT-258A: Numerical Optimization 1 Chapter 2 Convex Optimization Shiqian Ma, MAT-258A: Numerical Optimization 2 2.1. Convex Optimization General optimization problem: min f 0 (x) s.t., f i
More informationNonlinear Programming
Nonlinear Programming SECOND EDITION Dimitri P. Bertsekas Massachusetts Institute of Technology WWW site for book Information and Orders http://world.std.com/~athenasc/index.html Athena Scientific, Belmont,
More informationConvexity and Optimization
Convexity and Optimization Richard Lusby DTU Management Engineering Class Exercises From Last Time 2 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017 Today s Material Extrema
More informationConvex Analysis and Minimization Algorithms I
Jean-Baptiste Hiriart-Urruty Claude Lemarechal Convex Analysis and Minimization Algorithms I Fundamentals With 113 Figures Springer-Verlag Berlin Heidelberg New York London Paris Tokyo Hong Kong Barcelona
More informationSurrogate Gradient Algorithm for Lagrangian Relaxation 1,2
Surrogate Gradient Algorithm for Lagrangian Relaxation 1,2 X. Zhao 3, P. B. Luh 4, and J. Wang 5 Communicated by W.B. Gong and D. D. Yao 1 This paper is dedicated to Professor Yu-Chi Ho for his 65th birthday.
More informationCharacterizing Improving Directions Unconstrained Optimization
Final Review IE417 In the Beginning... In the beginning, Weierstrass's theorem said that a continuous function achieves a minimum on a compact set. Using this, we showed that for a convex set S and y not
More informationLagrangian Relaxation: An overview
Discrete Math for Bioinformatics WS 11/12:, by A. Bockmayr/K. Reinert, 22. Januar 2013, 13:27 4001 Lagrangian Relaxation: An overview Sources for this lecture: D. Bertsimas and J. Tsitsiklis: Introduction
More informationIntroduction to Modern Control Systems
Introduction to Modern Control Systems Convex Optimization, Duality and Linear Matrix Inequalities Kostas Margellos University of Oxford AIMS CDT 2016-17 Introduction to Modern Control Systems November
More informationConvex sets and convex functions
Convex sets and convex functions Convex optimization problems Convex sets and their examples Separating and supporting hyperplanes Projections on convex sets Convex functions, conjugate functions ECE 602,
More informationConvex sets and convex functions
Convex sets and convex functions Convex optimization problems Convex sets and their examples Separating and supporting hyperplanes Projections on convex sets Convex functions, conjugate functions ECE 602,
More informationINTRODUCTION TO LINEAR AND NONLINEAR PROGRAMMING
INTRODUCTION TO LINEAR AND NONLINEAR PROGRAMMING DAVID G. LUENBERGER Stanford University TT ADDISON-WESLEY PUBLISHING COMPANY Reading, Massachusetts Menlo Park, California London Don Mills, Ontario CONTENTS
More informationBilinear Modeling Solution Approach for Fixed Charged Network Flow Problems (Draft)
Bilinear Modeling Solution Approach for Fixed Charged Network Flow Problems (Draft) Steffen Rebennack 1, Artyom Nahapetyan 2, Panos M. Pardalos 1 1 Department of Industrial & Systems Engineering, Center
More informationProgramming, numerics and optimization
Programming, numerics and optimization Lecture C-4: Constrained optimization Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428 June
More informationCOMS 4771 Support Vector Machines. Nakul Verma
COMS 4771 Support Vector Machines Nakul Verma Last time Decision boundaries for classification Linear decision boundary (linear classification) The Perceptron algorithm Mistake bound for the perceptron
More informationIntroduction to optimization methods and line search
Introduction to optimization methods and line search Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi How to find optimal solutions? Trial and error widely used in practice, not efficient and
More informationof Convex Analysis Fundamentals Jean-Baptiste Hiriart-Urruty Claude Lemarechal Springer With 66 Figures
2008 AGI-Information Management Consultants May be used for personal purporses only or by libraries associated to dandelon.com network. Jean-Baptiste Hiriart-Urruty Claude Lemarechal Fundamentals of Convex
More informationConvex Optimization Lecture 2
Convex Optimization Lecture 2 Today: Convex Analysis Center-of-mass Algorithm 1 Convex Analysis Convex Sets Definition: A set C R n is convex if for all x, y C and all 0 λ 1, λx + (1 λ)y C Operations that
More informationContents. I Basics 1. Copyright by SIAM. Unauthorized reproduction of this article is prohibited.
page v Preface xiii I Basics 1 1 Optimization Models 3 1.1 Introduction... 3 1.2 Optimization: An Informal Introduction... 4 1.3 Linear Equations... 7 1.4 Linear Optimization... 10 Exercises... 12 1.5
More informationSparse Optimization Lecture: Proximal Operator/Algorithm and Lagrange Dual
Sparse Optimization Lecture: Proximal Operator/Algorithm and Lagrange Dual Instructor: Wotao Yin July 2013 online discussions on piazza.com Those who complete this lecture will know learn the proximal
More informationAspects of Convex, Nonconvex, and Geometric Optimization (Lecture 1) Suvrit Sra Massachusetts Institute of Technology
Aspects of Convex, Nonconvex, and Geometric Optimization (Lecture 1) Suvrit Sra Massachusetts Institute of Technology Hausdorff Institute for Mathematics (HIM) Trimester: Mathematics of Signal Processing
More informationLecture 19: Convex Non-Smooth Optimization. April 2, 2007
: Convex Non-Smooth Optimization April 2, 2007 Outline Lecture 19 Convex non-smooth problems Examples Subgradients and subdifferentials Subgradient properties Operations with subgradients and subdifferentials
More informationIE598 Big Data Optimization Summary Nonconvex Optimization
IE598 Big Data Optimization Summary Nonconvex Optimization Instructor: Niao He April 16, 2018 1 This Course Big Data Optimization Explore modern optimization theories, algorithms, and big data applications
More informationConvex Optimization. Lijun Zhang Modification of
Convex Optimization Lijun Zhang zlj@nju.edu.cn http://cs.nju.edu.cn/zlj Modification of http://stanford.edu/~boyd/cvxbook/bv_cvxslides.pdf Outline Introduction Convex Sets & Functions Convex Optimization
More informationWalheer Barnabé. Topics in Mathematics Practical Session 2 - Topology & Convex
Topics in Mathematics Practical Session 2 - Topology & Convex Sets Outline (i) Set membership and set operations (ii) Closed and open balls/sets (iii) Points (iv) Sets (v) Convex Sets Set Membership and
More informationResearch Interests Optimization:
Mitchell: Research interests 1 Research Interests Optimization: looking for the best solution from among a number of candidates. Prototypical optimization problem: min f(x) subject to g(x) 0 x X IR n Here,
More informationA Linear Programming Approach to Concave Approximation and Value Function Iteration
A Linear Programming Approach to Concave Approximation and Value Function Iteration Ronaldo Carpio Takashi Kamihigashi May 18, 2015 Abstract A basic task in numerical computation is to approximate a continuous
More informationNonsmooth Optimization and Related Topics
Nonsmooth Optimization and Related Topics Edited by F. H. Clarke University of Montreal Montreal, Quebec, Canada V. F. Dem'yanov Leningrad State University Leningrad, USSR I and F. Giannessi University
More informationKernels and Constrained Optimization
Machine Learning 1 WS2014 Module IN2064 Sheet 8 Page 1 Machine Learning Worksheet 8 Kernels and Constrained Optimization 1 Kernelized k-nearest neighbours To classify the point x the k-nearest neighbours
More informationRecent Developments in Model-based Derivative-free Optimization
Recent Developments in Model-based Derivative-free Optimization Seppo Pulkkinen April 23, 2010 Introduction Problem definition The problem we are considering is a nonlinear optimization problem with constraints:
More informationLECTURE 13: SOLUTION METHODS FOR CONSTRAINED OPTIMIZATION. 1. Primal approach 2. Penalty and barrier methods 3. Dual approach 4. Primal-dual approach
LECTURE 13: SOLUTION METHODS FOR CONSTRAINED OPTIMIZATION 1. Primal approach 2. Penalty and barrier methods 3. Dual approach 4. Primal-dual approach Basic approaches I. Primal Approach - Feasible Direction
More informationAn FPTAS for Optimizing a Class of Low-Rank Functions Over a Polytope
An FPTAS for Optimizing a Class of Low-Rank Functions Over a Polytope Shashi Mittal Andreas S. Schulz Abstract We present a fully polynomial time approximation scheme (FPTAS) for optimizing a very general
More informationConstrained and Unconstrained Optimization
Constrained and Unconstrained Optimization Carlos Hurtado Department of Economics University of Illinois at Urbana-Champaign hrtdmrt2@illinois.edu Oct 10th, 2017 C. Hurtado (UIUC - Economics) Numerical
More informationAM 221: Advanced Optimization Spring 2016
AM 221: Advanced Optimization Spring 2016 Prof. Yaron Singer Lecture 2 Wednesday, January 27th 1 Overview In our previous lecture we discussed several applications of optimization, introduced basic terminology,
More informationOptimization. 1. Optimization. by Prof. Seungchul Lee Industrial AI Lab POSTECH. Table of Contents
Optimization by Prof. Seungchul Lee Industrial AI Lab http://isystems.unist.ac.kr/ POSTECH Table of Contents I. 1. Optimization II. 2. Solving Optimization Problems III. 3. How do we Find x f(x) = 0 IV.
More informationProbabilistic Graphical Models
School of Computer Science Probabilistic Graphical Models Theory of Variational Inference: Inner and Outer Approximation Eric Xing Lecture 14, February 29, 2016 Reading: W & J Book Chapters Eric Xing @
More informationOptimization for Machine Learning
Optimization for Machine Learning (Problems; Algorithms - C) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html
More informationTutorial on Convex Optimization for Engineers
Tutorial on Convex Optimization for Engineers M.Sc. Jens Steinwandt Communications Research Laboratory Ilmenau University of Technology PO Box 100565 D-98684 Ilmenau, Germany jens.steinwandt@tu-ilmenau.de
More informationIntroduction to Optimization Problems and Methods
Introduction to Optimization Problems and Methods wjch@umich.edu December 10, 2009 Outline 1 Linear Optimization Problem Simplex Method 2 3 Cutting Plane Method 4 Discrete Dynamic Programming Problem Simplex
More informationCME307/MS&E311 Theory Summary
CME307/MS&E311 Theory Summary Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/~yyye http://www.stanford.edu/class/msande311/
More informationA Brief Overview of Optimization Problems. Steven G. Johnson MIT course , Fall 2008
A Brief Overview of Optimization Problems Steven G. Johnson MIT course 18.335, Fall 2008 Why optimization? In some sense, all engineering design is optimization: choosing design parameters to improve some
More informationConvexity: an introduction
Convexity: an introduction Geir Dahl CMA, Dept. of Mathematics and Dept. of Informatics University of Oslo 1 / 74 1. Introduction 1. Introduction what is convexity where does it arise main concepts and
More informationMathematical Programming and Research Methods (Part II)
Mathematical Programming and Research Methods (Part II) 4. Convexity and Optimization Massimiliano Pontil (based on previous lecture by Andreas Argyriou) 1 Today s Plan Convex sets and functions Types
More informationIntroduction to Constrained Optimization
Introduction to Constrained Optimization Duality and KKT Conditions Pratik Shah {pratik.shah [at] lnmiit.ac.in} The LNM Institute of Information Technology www.lnmiit.ac.in February 13, 2013 LNMIIT MLPR
More informationMachine Learning for Signal Processing Lecture 4: Optimization
Machine Learning for Signal Processing Lecture 4: Optimization 13 Sep 2015 Instructor: Bhiksha Raj (slides largely by Najim Dehak, JHU) 11-755/18-797 1 Index 1. The problem of optimization 2. Direct optimization
More informationA Brief Overview of Optimization Problems. Steven G. Johnson MIT course , Fall 2008
A Brief Overview of Optimization Problems Steven G. Johnson MIT course 18.335, Fall 2008 Why optimization? In some sense, all engineering design is optimization: choosing design parameters to improve some
More informationLECTURE NOTES Non-Linear Programming
CEE 6110 David Rosenberg p. 1 Learning Objectives LECTURE NOTES Non-Linear Programming 1. Write out the non-linear model formulation 2. Describe the difficulties of solving a non-linear programming model
More informationCPSC 340: Machine Learning and Data Mining. Robust Regression Fall 2015
CPSC 340: Machine Learning and Data Mining Robust Regression Fall 2015 Admin Can you see Assignment 1 grades on UBC connect? Auditors, don t worry about it. You should already be working on Assignment
More informationCOM Optimization for Communications Summary: Convex Sets and Convex Functions
1 Convex Sets Affine Sets COM524500 Optimization for Communications Summary: Convex Sets and Convex Functions A set C R n is said to be affine if A point x 1, x 2 C = θx 1 + (1 θ)x 2 C, θ R (1) y = k θ
More informationSolution Methods Numerical Algorithms
Solution Methods Numerical Algorithms Evelien van der Hurk DTU Managment Engineering Class Exercises From Last Time 2 DTU Management Engineering 42111: Static and Dynamic Optimization (6) 09/10/2017 Class
More informationConvex Optimization and Machine Learning
Convex Optimization and Machine Learning Mengliu Zhao Machine Learning Reading Group School of Computing Science Simon Fraser University March 12, 2014 Mengliu Zhao SFU-MLRG March 12, 2014 1 / 25 Introduction
More informationApplied Lagrange Duality for Constrained Optimization
Applied Lagrange Duality for Constrained Optimization Robert M. Freund February 10, 2004 c 2004 Massachusetts Institute of Technology. 1 1 Overview The Practical Importance of Duality Review of Convexity
More informationmorphology on binary images
morphology on binary images Ole-Johan Skrede 10.05.2017 INF2310 - Digital Image Processing Department of Informatics The Faculty of Mathematics and Natural Sciences University of Oslo After original slides
More informationAffine function. suppose f : R n R m is affine (f(x) =Ax + b with A R m n, b R m ) the image of a convex set under f is convex
Affine function suppose f : R n R m is affine (f(x) =Ax + b with A R m n, b R m ) the image of a convex set under f is convex S R n convex = f(s) ={f(x) x S} convex the inverse image f 1 (C) of a convex
More informationConvexization in Markov Chain Monte Carlo
in Markov Chain Monte Carlo 1 IBM T. J. Watson Yorktown Heights, NY 2 Department of Aerospace Engineering Technion, Israel August 23, 2011 Problem Statement MCMC processes in general are governed by non
More informationIntroduction to Optimization
Introduction to Optimization Second Order Optimization Methods Marc Toussaint U Stuttgart Planned Outline Gradient-based optimization (1st order methods) plain grad., steepest descent, conjugate grad.,
More informationCME307/MS&E311 Optimization Theory Summary
CME307/MS&E311 Optimization Theory Summary Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/~yyye http://www.stanford.edu/class/msande311/
More informationOptimization. Industrial AI Lab.
Optimization Industrial AI Lab. Optimization An important tool in 1) Engineering problem solving and 2) Decision science People optimize Nature optimizes 2 Optimization People optimize (source: http://nautil.us/blog/to-save-drowning-people-ask-yourself-what-would-light-do)
More informationLecture 4: Convexity
10-725: Convex Optimization Fall 2013 Lecture 4: Convexity Lecturer: Barnabás Póczos Scribes: Jessica Chemali, David Fouhey, Yuxiong Wang Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:
More informationCMU-Q Lecture 9: Optimization II: Constrained,Unconstrained Optimization Convex optimization. Teacher: Gianni A. Di Caro
CMU-Q 15-381 Lecture 9: Optimization II: Constrained,Unconstrained Optimization Convex optimization Teacher: Gianni A. Di Caro GLOBAL FUNCTION OPTIMIZATION Find the global maximum of the function f x (and
More informationFACES OF CONVEX SETS
FACES OF CONVEX SETS VERA ROSHCHINA Abstract. We remind the basic definitions of faces of convex sets and their basic properties. For more details see the classic references [1, 2] and [4] for polytopes.
More informationNatural Language Processing
Natural Language Processing Classification III Dan Klein UC Berkeley 1 Classification 2 Linear Models: Perceptron The perceptron algorithm Iteratively processes the training set, reacting to training errors
More informationRevisiting the Upper Bounding Process in a Safe Branch and Bound Algorithm
Revisiting the Upper Bounding Process in a Safe Branch and Bound Algorithm Alexandre Goldsztejn 1, Yahia Lebbah 2,3, Claude Michel 3, and Michel Rueher 3 1 CNRS / Université de Nantes 2, rue de la Houssinière,
More information1. Introduction. performance of numerical methods. complexity bounds. structural convex optimization. course goals and topics
1. Introduction EE 546, Univ of Washington, Spring 2016 performance of numerical methods complexity bounds structural convex optimization course goals and topics 1 1 Some course info Welcome to EE 546!
More informationA Comparison of Mixed-Integer Programming Models for Non-Convex Piecewise Linear Cost Minimization Problems
A Comparison of Mixed-Integer Programming Models for Non-Convex Piecewise Linear Cost Minimization Problems Keely L. Croxton Fisher College of Business The Ohio State University Bernard Gendron Département
More informationConvex Optimization / Homework 2, due Oct 3
Convex Optimization 0-725/36-725 Homework 2, due Oct 3 Instructions: You must complete Problems 3 and either Problem 4 or Problem 5 (your choice between the two) When you submit the homework, upload a
More informationNew Methods for Solving Large Scale Linear Programming Problems in the Windows and Linux computer operating systems
arxiv:1209.4308v1 [math.oc] 19 Sep 2012 New Methods for Solving Large Scale Linear Programming Problems in the Windows and Linux computer operating systems Saeed Ketabchi, Hossein Moosaei, Hossein Sahleh
More informationConvex Programs. COMPSCI 371D Machine Learning. COMPSCI 371D Machine Learning Convex Programs 1 / 21
Convex Programs COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Convex Programs 1 / 21 Logistic Regression! Support Vector Machines Support Vector Machines (SVMs) and Convex Programs SVMs are
More informationIE 521 Convex Optimization
Lecture 4: 5th February 2019 Outline 1 / 23 Which function is different from others? Figure: Functions 2 / 23 Definition of Convex Function Definition. A function f (x) : R n R is convex if (i) dom(f )
More informationEfficient Evolutionary Algorithm for Single-Objective Bilevel Optimization
Efficient Evolutionary Algorithm for Single-Objective Bilevel Optimization Ankur Sinha, Pekka Malo Dept. of Information and Service Economy Aalto University School of Business, Finland Firstname.Lastname@aalto.fi
More informationLecture 5: Duality Theory
Lecture 5: Duality Theory Rajat Mittal IIT Kanpur The objective of this lecture note will be to learn duality theory of linear programming. We are planning to answer following questions. What are hyperplane
More informationSupport Vector Machines.
Support Vector Machines srihari@buffalo.edu SVM Discussion Overview. Importance of SVMs. Overview of Mathematical Techniques Employed 3. Margin Geometry 4. SVM Training Methodology 5. Overlapping Distributions
More informationTowards a practical simplex method for second order cone programming
Towards a practical simplex method for second order cone programming Kartik Krishnan Department of Computing and Software McMaster University Joint work with Gábor Pataki (UNC), Neha Gupta (IIT Delhi),
More informationConvergence of Nonconvex Douglas-Rachford Splitting and Nonconvex ADMM
Convergence of Nonconvex Douglas-Rachford Splitting and Nonconvex ADMM Shuvomoy Das Gupta Thales Canada 56th Allerton Conference on Communication, Control and Computing, 2018 1 What is this talk about?
More informationMath 5593 Linear Programming Lecture Notes
Math 5593 Linear Programming Lecture Notes Unit II: Theory & Foundations (Convex Analysis) University of Colorado Denver, Fall 2013 Topics 1 Convex Sets 1 1.1 Basic Properties (Luenberger-Ye Appendix B.1).........................
More informationUnconstrained Optimization Principles of Unconstrained Optimization Search Methods
1 Nonlinear Programming Types of Nonlinear Programs (NLP) Convexity and Convex Programs NLP Solutions Unconstrained Optimization Principles of Unconstrained Optimization Search Methods Constrained Optimization
More informationEfficient Tuning of SVM Hyperparameters Using Radius/Margin Bound and Iterative Algorithms
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 13, NO. 5, SEPTEMBER 2002 1225 Efficient Tuning of SVM Hyperparameters Using Radius/Margin Bound and Iterative Algorithms S. Sathiya Keerthi Abstract This paper
More informationA Deterministic Global Optimization Method for Variational Inference
A Deterministic Global Optimization Method for Variational Inference Hachem Saddiki Mathematics and Statistics University of Massachusetts, Amherst saddiki@math.umass.edu Andrew C. Trapp Operations and
More informationGlobal Minimization via Piecewise-Linear Underestimation
Journal of Global Optimization,, 1 9 (2004) c 2004 Kluwer Academic Publishers, Boston. Manufactured in The Netherlands. Global Minimization via Piecewise-Linear Underestimation O. L. MANGASARIAN olvi@cs.wisc.edu
More informationChapter 3 Numerical Methods
Chapter 3 Numerical Methods Part 1 3.1 Linearization and Optimization of Functions of Vectors 1 Problem Notation 2 Outline 3.1.1 Linearization 3.1.2 Optimization of Objective Functions 3.1.3 Constrained
More informationOptimization Plugin for RapidMiner. Venkatesh Umaashankar Sangkyun Lee. Technical Report 04/2012. technische universität dortmund
Optimization Plugin for RapidMiner Technical Report Venkatesh Umaashankar Sangkyun Lee 04/2012 technische universität dortmund Part of the work on this technical report has been supported by Deutsche Forschungsgemeinschaft
More informationLecture 25 Nonlinear Programming. November 9, 2009
Nonlinear Programming November 9, 2009 Outline Nonlinear Programming Another example of NLP problem What makes these problems complex Scalar Function Unconstrained Problem Local and global optima: definition,
More informationSupport Vector Machines. James McInerney Adapted from slides by Nakul Verma
Support Vector Machines James McInerney Adapted from slides by Nakul Verma Last time Decision boundaries for classification Linear decision boundary (linear classification) The Perceptron algorithm Mistake
More informationConic Duality. yyye
Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 1 Conic Duality Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/
More informationON CONCAVITY OF THE PRINCIPAL S PROFIT MAXIMIZATION FACING AGENTS WHO RESPOND NONLINEARLY TO PRICES
ON CONCAVITY OF THE PRINCIPAL S PROFIT MAXIMIZATION FACING AGENTS WHO RESPOND NONLINEARLY TO PRICES Shuangjian Zhang This is joint work with my supervisor Robert J. McCann University of Toronto April 11,
More informationCS675: Convex and Combinatorial Optimization Fall 2014 Convex Functions. Instructor: Shaddin Dughmi
CS675: Convex and Combinatorial Optimization Fall 2014 Convex Functions Instructor: Shaddin Dughmi Outline 1 Convex Functions 2 Examples of Convex and Concave Functions 3 Convexity-Preserving Operations
More informationA fast algorithm for sparse reconstruction based on shrinkage, subspace optimization and continuation [Wen,Yin,Goldfarb,Zhang 2009]
A fast algorithm for sparse reconstruction based on shrinkage, subspace optimization and continuation [Wen,Yin,Goldfarb,Zhang 2009] Yongjia Song University of Wisconsin-Madison April 22, 2010 Yongjia Song
More informationFebruary 2017 (1/20) 2 Piecewise Polynomial Interpolation 2.2 (Natural) Cubic Splines. MA378/531 Numerical Analysis II ( NA2 )
f f f f f (/2).9.8.7.6.5.4.3.2. S Knots.7.6.5.4.3.2. 5 5.2.8.6.4.2 S Knots.2 5 5.9.8.7.6.5.4.3.2..9.8.7.6.5.4.3.2. S Knots 5 5 S Knots 5 5 5 5.35.3.25.2.5..5 5 5.6.5.4.3.2. 5 5 4 x 3 3.5 3 2.5 2.5.5 5
More informationCPSC 340: Machine Learning and Data Mining. Principal Component Analysis Fall 2016
CPSC 340: Machine Learning and Data Mining Principal Component Analysis Fall 2016 A2/Midterm: Admin Grades/solutions will be posted after class. Assignment 4: Posted, due November 14. Extra office hours:
More informationLecture 2 September 3
EE 381V: Large Scale Optimization Fall 2012 Lecture 2 September 3 Lecturer: Caramanis & Sanghavi Scribe: Hongbo Si, Qiaoyang Ye 2.1 Overview of the last Lecture The focus of the last lecture was to give
More information