Convex Optimization MLSS 2015
|
|
- Lauren York
- 5 years ago
- Views:
Transcription
1 Convex Optimization MLSS 2015 Constantine Caramanis The University of Texas at Austin
2 The Optimization Problem minimize : f (x) subject to : x X.
3 The Optimization Problem minimize : f (x) subject to : x X. What can this model? When can we solve it?
4 What Can We Model? Optimization: a frame of mind...
5 What Can We Model? Max Margin Classification Figure: Given labeled examples, find a classifier with the biggest margin of separation. Is this an optimization problem?
6 What Can We Model? Image Denoising Figure: Given the noisy camera man, can the denoising problem be cast as an optimization?
7 What Can We Model? Matrix Completion Users F i l m s Figure: Recover a low-rank matrix from a few of its entries. This is a starting point for many recommendation engines. Is this an optimization problem?
8 What Can We Model? Optimal Inequalities in Probability. X an integer-valued random variable. Given some moment constraints: µ i = E[X i ], i = 1,2,3,4,5, find the best upper and lower bounds for P{X [5,15]}
9 What Can We Model?...and what can we solve?
10 Convex Optimization minimize : f (x) subject to : x X. f (x) a convex function X a convex set
11 Convex Sets Definition A set, X, is called a convex set if and only if the convex combination of any two points in the set belongs to the set, i.e. X R n is convex if x 1,x 2 X and λ [0,1], λx 1 + (1 λ)x 2 X. Definition A convex combination of points x 1,...,x k is described by k i=0 θ ix i, where θ θ k = 1 and θ i 0.
12 Convex Sets Figure: A convex set can be easily determined by examining whether the line segment between any two points in the set are in the set. Thus the figure on the left (circle) is in the set wheras the figure on the right (star) is not.
13 Convex Functions Definition The domain of a function f : R n R is denoted dom(f ), and is defined as the set of points where a function f is finite: dom(f ) = {x R n : f (x) < }.
14 Definition Convex Functions: Definition 1 A function f : R n R is convex if for any x 1,x 2 dom(f ) R n, λ [0,1], we have: λf (x 1 ) + (1 λ)f (x 2 ) f (λx 1 + (1 λ)x 2 ). Figure: Convex functions
15 Convex Functions: Definition 2 Definition Suppose a function f : R n R is differentiable. Then it is convex if and only if f (y) f (x) + f (x) (y x). f ( y) ( x, f ( x) ) f ( x) + f ( x) T ( y x)
16 Convex Functions: Definition 3 Definition Suppose that a function f : R n R is twice differentiable. Then f is convex iff its Hessian is positive semidefinite: 2 f (x) 0, x dom(f ) Note that the equivalences of the three definitions are proved in the lecture notes(proposition 1 and 2). We would just leave it here without the proofs.
17 Examples of Convex Functions Exponential f (x) = e ax, a R Powers f (x) = x a is convex on R ++ when a 1 or a 0, concave otherwise. Negative Logarithm f (x) = logx is convex on R ++ Norms The L p norms on R n are convex: x p = ( x i p ) 1/p, (1 p ) Max Function f (x) = max{x 1,x 2,,x n } is convex on R n Some Matrix Functions The sum of the k largest singular values.
18 Intuition: Convex Optimization Easy Figure: Gradient Descent on convex functions: Rolling down hill will lead to convergence to the global optimum
19 Intuition: Non-Convex Optimization Hard Figure: Gradient Descent on non-convex functions: Rolling down hill may lead us to a local minimum, which can be far from the global minimum. Many problems have massive numbers of highly suboptimal local optima.
20 Outline From Here Modeling so how do we model some of the problems mentioned above? Algorithms how do we solve them? Theory what can we prove?
21 Optimal Inequalities in Probability X an integer-valued random variable. Given some moment constraints: µ i = E[X i ], i = 1,2,3,4,5, find the best upper and lower bounds for P{X [5,15]}
22 Convex Modeling of Optimal Inequalities in Probability Let P j = P{X = j}. Then we formulate the optimization problem for finding upper/lower bounds as, max/min : s.t. : 15 P j j=5 P j 0, for any j P j = 1 j P j j i = µ i, i = 1,2,3,4,5. j f ( ) =? X =?
23 Image Denoising Figure: Given the noisy camera man, can the denoising problem be cast as an optimization?
24 Convex Modeling of Image Denoising Domain-specific insight: Natural images have structure: sharp edges with areas of near-constant intensity. Denote image by pixel intensity map: X : [0,1] [0,1] R. So X (a,b) is intensity of pixel (a,b). Let X clean and X noisy denote the clean and noisy images. We denoise by finding an image close to the noisy image but with smooth areas and sharp edges: min : (X (a,b) X noisy (a,b)) 2 + λ X (a,b) 2 2. X [0,1] 2 [0,1] 2 f ( ) =? X =?
25 Matrix Completion Users F i l m s Figure: Recover a low-rank matrix from a few of its entries. This is a starting point for many recommendation engines. Is this an optimization problem?
26 Convex Modeling of Matrix Completion Direct Minimization for the Rank min rank(x ) s.t. X ij = M ij, for observed (i,j) However, rank-minimization is a non-convex problem.
27 Convex Relaxations A simple idea with far-reaching consequences: if a problem is non-convex, solve the closest convex problem.
28 Convex Modeling of Matrix Completion Nuclear Norm Convex Relaxation of the Rank min X s.t. X ij = M ij, for observed (i,j) here, X is called the nuclear norm. It is the sum of singular values of X. Exercise. Show that f (X ) = X is convex.
29 Exercises and Software Try out these examples! Optimal Probability inequalities: try using linprog in Matlab. More general convex solver: CVX free download:
30 Outline From Here Modeling so how do we model some of the problems mentioned above? Algorithms how do we solve them? Theory what can we prove?
31 Outline From Here Modeling so how do we model some of the problems mentioned above? Algorithms how do we solve them? Modern problems in machine learning are increasingly characterized by their massive size. We need iterative algorithms that have good convergence guarantees. Theory what can we prove? Many interesting problems sparse regression, matrix completion, etc. are inherently non-convex, but ideas of convex relaxation as above, can be used. When can we prove that the solution of the convex problem is useful?
32 Algorithms for Convex Optimization min : f (x) s.t. : x X R n. Want: ˆx such that ˆx is close to x, or f (ˆx) is close to f (x ). What can we expect? How hard must we work? Answer depends on f ( ), X, n, and error tolerance, ε.
33 Algorithms for Convex Optimization min : Question: which is better? f (x) s.t. : x X R n. Algorithm (A1) produces an ε-accuracy solution in time O(n 2 log(1/ε)); Algorithm (A2) produces an ε-accuracy solution in time O(n/ε 2 ).
34 Second Order and Interior Point Methods
35 First Order Methods Oracle Model: given x, oracle produces (f (x), f (x)), and Π X (x). How many calls to the oracle do we need to produce an ε-accurate solution? Discussion.
36 First Order Methods Oracle Model: given x, oracle produces (f (x), f (x)), and Π X (x). How many calls to the oracle do we need to produce an ε-accurate solution? Basic iterative algorithm: smooth convex optimization x + = x η f (x) unconstrained optimization x + = Π X (x η f (x)) constrained optimization Assumptions on f ( )? Comp. per iteration? No. of iterations?
37 Convergence of Gradient Descent min : f (x) s.t. : x X R n. Assumption: f ( ) has L-Lipschitz gradients. f (x) f (y) L x y. Upper bound on curvature of f.
38 Convergence of Gradient Descent Recall the definition of f convex: Definition Suppose a function f : R n R is differentiable. Then it is convex if and only if f (y) f (x) + f (x) (y x). f ( y) ( x, f ( x) ) f ( x) + f ( x) T ( y x)
39 Convergence of Gradient Descent Now, in addition to: f (y) f (x) + f (x) (y x). we have Lemma If f L-Lipschitz, then f (y) f (x) + f (x) (y x) + L 2 y x 2.
40 Convergence of Gradient Descent Proof of Lemma: First note that the function is convex. g(x) = L 2 x 2 f (x), Exercise. If f ( ) has second derivatives, 2 g(x) = L I 2 f (x) 0. Prove g(x) is convex without that assumption.
41 Convergence of Gradient Descent From the lemma, g(x) convex means, by definition, g(y) g(x) + g(x) (y x). Rearranging, we get the statement of the lemma.
42 Convergence of Gradient Descent From the Lemma: f (y) f (x) + f (x) (y x) + L 2 y x 2 we have: f (x η f (x)) f (x) + f (x) ( η f (x)) + L η f (x) 2 }{{} 2 y ( ) L = f (x) + 2 η2 η f (x) 2. Corollary Choosing η < 1/L, f (x η f (x)) f (x) η 2 f (x) 2.
43 Convergence of Gradient Descent Now for x (i+1) = x (i) η f (x (i) ), we have f (x (i+1) ) f (x (i) ) η 2 f (x (i) ) 2 f (x ) + f (x (i) ) (x (i) x ) η 2 f (x (i) ) 2 = f + 1 2η ( x (i) x 2 x (i) x η f (x (i) ) 2 ) = f + 1 2η ( x (i) x 2 x (i+1) x 2 ).
44 Convergence of Gradient Descent Summing over k iterations of the algorithm: k i=1 f (x (i) ) f 1 2η f (x (k) ) f 1 k ( k Theorem k i=1 ( x (i) x 2 x (i+1) x 2 ) = 1 2η ( x (0) x 2 x (k) x 2 ) 1 2η x (0) x 2 i=1 f (x (i) ) f ) 1 k ( ) 1 2η x (0) x 2. Under the above assumptions, gradient descent converges at a rate of O(1/k); i.e., it has error ε in O(1/ε) iterations.
45 Convergence of Gradient Descent Discussion: This analysis is for the unconstrained setting. The result is the same for the constrained setting. Key to proof: using convexity and upper bound on curvature. Result: dimension independent!
46 What about Different Assumptions Given a function f (x) and error target ε such that f (ˆx) f ε, and using gradient descent: Under the assumption that f (x) is smooth (upper bound on curvature) and convex: O( 1 ε ) iterations are needed. What if f (x) is not smooth? For example, f (x) = Ax b 2 + x 1.
47 Subgradients and Subdifferential f ( ) convex but not differentiable. Still have basic definition of convexity: f (y) f (x) + g x (y x). Now (possibly) many under-estimates. Define subdifferential: f (x) = {g : f (y) f (x) + g x (y x), y}
48 Subgradient Algorithm Starting at x, given any g x f (x), and step size η, x + = x ηg x. Convergence guarantees: We still have convexity inequality. Do we have something like??? f (x η f (x)) f (x) η f (x) 2 2
49 Unfortunately Not: Example The subgradient method is not a descent method. f (x 1,x 2 ) = x x 2. current position is given by (x 1,x 2 ) = (10,0) the two extreme subgradients at (10,0) are (1,10) and (1, 10) then the subgradient set is obtained as f (10,0) = {(1,10v) for 1 v 1}
50 x 2 Extreme Subgradient New point (10,0) x 1 Extreme Subgradient Consider g x = (1, 1) f (10,0) Then x + = x tg x. The resulting point increases the function value.
51 Convergence of Subgradient Method Assume f is convex, and g G for any g f (x), x. For x (i+1) = x (i) ηg x (i) we have: x (k+1) x 2 = x (i) ηg x (k) x 2 = x (k) x 2 2ηg x (k) (x (k) x ) + η 2 g x (k) 2 x (k) x 2 2η(f (ˆx) f ) + η 2 G 2 x (0) x 2 2kη(f (ˆx) f ) + η 2 G 2 k.
52 Convergence of Subgradient Method Rearranging, gives f (ˆx) f x (0) x 2 + G 2 η 2 k. 2kη Minimizing over η we find best step size η = 1/ k, and we get f (ˆx) f 1 k. Theorem Under the above assumptions, the subgradient method converges at a rate of O(1/ k); i.e., it has error ε in O(1/ε 2 ) iterations.
53 What about Different Assumptions Given a function f (x) and error target ε such that f (ˆx) f ε, and using (sub)gradient descent: Under the assumption that f (x) is convex: O( 1 ε 2 ) iterations are needed. Under the assumption that f (x) is smooth (upper bound on curvature) and convex: O( 1 ε ) iterations are needed. Under the assumption that f (x) is smooth and strongly convex (also lower bound on curvature): O(ln( 1 ε )) iterations are needed. Is this the best we can do?
54 First Order Convergence Guarantees For (sub)gradient algorithm, the above rates are the best i.e., analysis cannot be improved. There are, however, other first-order algorithms.
55 Proximal Algorithm Suppose we want to minimize: f (x) = g(x) + h(x), where g(x) is smooth, and h(x) is simple. Example: l 1 -regularized regression min : Ax b λ x 1. Ax b 2 2 is smooth, and x 1 is simple. Can we do better?
56 Proximal Algorithm Briefly, the answer is yes. If we can easily evaluate the Prox function: Prox ηh (y) = arg min x : h(x) + 1 2η x y 2. Proximal algorithm: Convergence rate: O(1/k) x + = Prox ηh (x η g(x)).
57 Accelerated Algorithms (Lower Bounds) Accelerated algorithms: x (k+1) = x (k) α f (x (k) ) + β(x (k) x (k 1) ) Still fits in our oracle model. Convergence: If f is smooth, then for error ε, we need O(1/ ε) iterations, hence O(1/k 2 ) convergence. Proximal analogs for the case: f (x) = g(x) + h(x).
58 Outline From Here Modeling so how do we model some of the problems mentioned above? Algorithms how do we solve them? Theory what can we prove?
59 Duality Theory Figure: A convex set can be represented in two ways: a convex hull of extreme points, or an intersection of half-spaces that contain it.
60 Duality Theory Figure: Consider optimizing in a given direction over a convex set.
61 Duality Theory Figure: The optimal point is the point with the best value.
62 Duality Theory Figure:...but there is also a half-space that says you cannot go further. This is called a certificate of optimality. Non-convex optimization problems do not always have such easy certificates of optimality.
63 Duality Theory We can search for the best point (search over points). Or (and) we can search for the certificate of optimality (search over half-spaces).
64 Provably Close to Optimality Figure: If we solve a relaxation, sometimes we can characterize via duality theory, how close the optimal point on the non-convex set will be to the optimal point on the convex relaxation. Sparse regression and compressed sensing. Low-rank matrix completion. Many other examples.
65 Summary and Directions Modeling with Convex Optimization: Intuition and creativity are absolutely essential. Successful problem modeling comes from understanding the problem and hence what is important (e.g., boundary/smoothness of natural images, or approximate low-rank of rankings and preferences), and also the theory of convex optimization which problems can be solved quickly/efficiently, and what can we say about convex approximations to non-convex problems? Algorithms: Today we discussed first order methods. These are suited for very large-scale problems, as we often see in large-scale ML/data mining. Not all methods are applicable or best for all problems. Understanding the demands of your application, and the performance of convex optimization algorithms in different settings, is very important make or break in large-scale applications.
66 Summary and Directions Theory and Duality: We discussed this least today, but it is important not only for analysis, but also for algorithmic development. We discussed only algorithms that search for optimal points. Dual algorithms search over half-spaces. Different problems may yield to better solution or approximation in one domain or another.
67 Some Useful References Convex Optimization, by Stephen Boyd and Lieven Vandenberghe (see also the slides from their courses). Convex Optimization Algorithms, by Dimitri Bertsekas. Optimization Models and Applications, by Guiseppe Calafiore and Laurent El Ghaoui. Introductory Lectures on Convex Optimization, by Yurii Nesterov. Lectures on Modern Convex Optimization, by Aharon Ben-Tal and Arkadi Nemirovski
68 The End Thanks, and feel free to contact me with questions:
Introduction to Modern Control Systems
Introduction to Modern Control Systems Convex Optimization, Duality and Linear Matrix Inequalities Kostas Margellos University of Oxford AIMS CDT 2016-17 Introduction to Modern Control Systems November
More informationLecture 19: Convex Non-Smooth Optimization. April 2, 2007
: Convex Non-Smooth Optimization April 2, 2007 Outline Lecture 19 Convex non-smooth problems Examples Subgradients and subdifferentials Subgradient properties Operations with subgradients and subdifferentials
More informationCMU-Q Lecture 9: Optimization II: Constrained,Unconstrained Optimization Convex optimization. Teacher: Gianni A. Di Caro
CMU-Q 15-381 Lecture 9: Optimization II: Constrained,Unconstrained Optimization Convex optimization Teacher: Gianni A. Di Caro GLOBAL FUNCTION OPTIMIZATION Find the global maximum of the function f x (and
More information1. Introduction. performance of numerical methods. complexity bounds. structural convex optimization. course goals and topics
1. Introduction EE 546, Univ of Washington, Spring 2016 performance of numerical methods complexity bounds structural convex optimization course goals and topics 1 1 Some course info Welcome to EE 546!
More informationOptimization for Machine Learning
Optimization for Machine Learning (Problems; Algorithms - C) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html
More informationConvex Optimization / Homework 2, due Oct 3
Convex Optimization 0-725/36-725 Homework 2, due Oct 3 Instructions: You must complete Problems 3 and either Problem 4 or Problem 5 (your choice between the two) When you submit the homework, upload a
More informationConvex Optimization Lecture 2
Convex Optimization Lecture 2 Today: Convex Analysis Center-of-mass Algorithm 1 Convex Analysis Convex Sets Definition: A set C R n is convex if for all x, y C and all 0 λ 1, λx + (1 λ)y C Operations that
More informationLecture 2 September 3
EE 381V: Large Scale Optimization Fall 2012 Lecture 2 September 3 Lecturer: Caramanis & Sanghavi Scribe: Hongbo Si, Qiaoyang Ye 2.1 Overview of the last Lecture The focus of the last lecture was to give
More informationConvexity Theory and Gradient Methods
Convexity Theory and Gradient Methods Angelia Nedić angelia@illinois.edu ISE Department and Coordinated Science Laboratory University of Illinois at Urbana-Champaign Outline Convex Functions Optimality
More informationSection 5 Convex Optimisation 1. W. Dai (IC) EE4.66 Data Proc. Convex Optimisation page 5-1
Section 5 Convex Optimisation 1 W. Dai (IC) EE4.66 Data Proc. Convex Optimisation 1 2018 page 5-1 Convex Combination Denition 5.1 A convex combination is a linear combination of points where all coecients
More informationProximal operator and methods
Proximal operator and methods Master 2 Data Science, Univ. Paris Saclay Robert M. Gower Optimization Sum of Terms A Datum Function Finite Sum Training Problem The Training Problem Convergence GD I Theorem
More informationMathematical Programming and Research Methods (Part II)
Mathematical Programming and Research Methods (Part II) 4. Convexity and Optimization Massimiliano Pontil (based on previous lecture by Andreas Argyriou) 1 Today s Plan Convex sets and functions Types
More informationConvex Optimization M2
Convex Optimization M2 Lecture 1 A. d Aspremont. Convex Optimization M2. 1/49 Today Convex optimization: introduction Course organization and other gory details... Convex sets, basic definitions. A. d
More informationAspects of Convex, Nonconvex, and Geometric Optimization (Lecture 1) Suvrit Sra Massachusetts Institute of Technology
Aspects of Convex, Nonconvex, and Geometric Optimization (Lecture 1) Suvrit Sra Massachusetts Institute of Technology Hausdorff Institute for Mathematics (HIM) Trimester: Mathematics of Signal Processing
More informationSparse Optimization Lecture: Proximal Operator/Algorithm and Lagrange Dual
Sparse Optimization Lecture: Proximal Operator/Algorithm and Lagrange Dual Instructor: Wotao Yin July 2013 online discussions on piazza.com Those who complete this lecture will know learn the proximal
More informationTutorial on Convex Optimization for Engineers
Tutorial on Convex Optimization for Engineers M.Sc. Jens Steinwandt Communications Research Laboratory Ilmenau University of Technology PO Box 100565 D-98684 Ilmenau, Germany jens.steinwandt@tu-ilmenau.de
More informationLagrangian Relaxation: An overview
Discrete Math for Bioinformatics WS 11/12:, by A. Bockmayr/K. Reinert, 22. Januar 2013, 13:27 4001 Lagrangian Relaxation: An overview Sources for this lecture: D. Bertsimas and J. Tsitsiklis: Introduction
More informationAdvanced Operations Research Techniques IE316. Quiz 1 Review. Dr. Ted Ralphs
Advanced Operations Research Techniques IE316 Quiz 1 Review Dr. Ted Ralphs IE316 Quiz 1 Review 1 Reading for The Quiz Material covered in detail in lecture. 1.1, 1.4, 2.1-2.6, 3.1-3.3, 3.5 Background material
More informationLecture 1: Introduction
CSE 599: Interplay between Convex Optimization and Geometry Winter 218 Lecturer: Yin Tat Lee Lecture 1: Introduction Disclaimer: Please tell me any mistake you noticed. 1.1 Course Information Objective:
More informationShiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 2. Convex Optimization
Shiqian Ma, MAT-258A: Numerical Optimization 1 Chapter 2 Convex Optimization Shiqian Ma, MAT-258A: Numerical Optimization 2 2.1. Convex Optimization General optimization problem: min f 0 (x) s.t., f i
More informationIE 521 Convex Optimization
Lecture 4: 5th February 2019 Outline 1 / 23 Which function is different from others? Figure: Functions 2 / 23 Definition of Convex Function Definition. A function f (x) : R n R is convex if (i) dom(f )
More informationConvexity: an introduction
Convexity: an introduction Geir Dahl CMA, Dept. of Mathematics and Dept. of Informatics University of Oslo 1 / 74 1. Introduction 1. Introduction what is convexity where does it arise main concepts and
More information15.082J and 6.855J. Lagrangian Relaxation 2 Algorithms Application to LPs
15.082J and 6.855J Lagrangian Relaxation 2 Algorithms Application to LPs 1 The Constrained Shortest Path Problem (1,10) 2 (1,1) 4 (2,3) (1,7) 1 (10,3) (1,2) (10,1) (5,7) 3 (12,3) 5 (2,2) 6 Find the shortest
More informationLecture 19 Subgradient Methods. November 5, 2008
Subgradient Methods November 5, 2008 Outline Lecture 19 Subgradients and Level Sets Subgradient Method Convergence and Convergence Rate Convex Optimization 1 Subgradients and Level Sets A vector s is a
More informationLecture 2 - Introduction to Polytopes
Lecture 2 - Introduction to Polytopes Optimization and Approximation - ENS M1 Nicolas Bousquet 1 Reminder of Linear Algebra definitions Let x 1,..., x m be points in R n and λ 1,..., λ m be real numbers.
More informationAM 221: Advanced Optimization Spring 2016
AM 221: Advanced Optimization Spring 2016 Prof. Yaron Singer Lecture 2 Wednesday, January 27th 1 Overview In our previous lecture we discussed several applications of optimization, introduced basic terminology,
More informationCS 435, 2018 Lecture 2, Date: 1 March 2018 Instructor: Nisheeth Vishnoi. Convex Programming and Efficiency
CS 435, 2018 Lecture 2, Date: 1 March 2018 Instructor: Nisheeth Vishnoi Convex Programming and Efficiency In this lecture, we formalize convex programming problem, discuss what it means to solve it efficiently
More informationConvexity I: Sets and Functions
Convexity I: Sets and Functions Lecturer: Aarti Singh Co-instructor: Pradeep Ravikumar Convex Optimization 10-725/36-725 See supplements for reviews of basic real analysis basic multivariate calculus basic
More informationProjection-Based Methods in Optimization
Projection-Based Methods in Optimization Charles Byrne (Charles Byrne@uml.edu) http://faculty.uml.edu/cbyrne/cbyrne.html Department of Mathematical Sciences University of Massachusetts Lowell Lowell, MA
More informationCharacterizing Improving Directions Unconstrained Optimization
Final Review IE417 In the Beginning... In the beginning, Weierstrass's theorem said that a continuous function achieves a minimum on a compact set. Using this, we showed that for a convex set S and y not
More informationLecture 5: Properties of convex sets
Lecture 5: Properties of convex sets Rajat Mittal IIT Kanpur This week we will see properties of convex sets. These properties make convex sets special and are the reason why convex optimization problems
More informationEE/AA 578: Convex Optimization
EE/AA 578: Convex Optimization Instructor: Maryam Fazel University of Washington Fall 2016 1. Introduction EE/AA 578, Univ of Washington, Fall 2016 course logistics mathematical optimization least-squares;
More informationMath 5593 Linear Programming Lecture Notes
Math 5593 Linear Programming Lecture Notes Unit II: Theory & Foundations (Convex Analysis) University of Colorado Denver, Fall 2013 Topics 1 Convex Sets 1 1.1 Basic Properties (Luenberger-Ye Appendix B.1).........................
More informationRevisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization. Author: Martin Jaggi Presenter: Zhongxing Peng
Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization Author: Martin Jaggi Presenter: Zhongxing Peng Outline 1. Theoretical Results 2. Applications Outline 1. Theoretical Results 2. Applications
More informationA primal-dual framework for mixtures of regularizers
A primal-dual framework for mixtures of regularizers Baran Gözcü baran.goezcue@epfl.ch Laboratory for Information and Inference Systems (LIONS) École Polytechnique Fédérale de Lausanne (EPFL) Switzerland
More informationConvex Optimization. Lijun Zhang Modification of
Convex Optimization Lijun Zhang zlj@nju.edu.cn http://cs.nju.edu.cn/zlj Modification of http://stanford.edu/~boyd/cvxbook/bv_cvxslides.pdf Outline Introduction Convex Sets & Functions Convex Optimization
More informationIntroduction to Convex Optimization. Prof. Daniel P. Palomar
Introduction to Convex Optimization Prof. Daniel P. Palomar The Hong Kong University of Science and Technology (HKUST) MAFS6010R- Portfolio Optimization with R MSc in Financial Mathematics Fall 2018-19,
More informationIntroduction to Constrained Optimization
Introduction to Constrained Optimization Duality and KKT Conditions Pratik Shah {pratik.shah [at] lnmiit.ac.in} The LNM Institute of Information Technology www.lnmiit.ac.in February 13, 2013 LNMIIT MLPR
More informationOutlier Pursuit: Robust PCA and Collaborative Filtering
Outlier Pursuit: Robust PCA and Collaborative Filtering Huan Xu Dept. of Mechanical Engineering & Dept. of Mathematics National University of Singapore Joint w/ Constantine Caramanis, Yudong Chen, Sujay
More informationNumerical Optimization
Convex Sets Computer Science and Automation Indian Institute of Science Bangalore 560 012, India. NPTEL Course on Let x 1, x 2 R n, x 1 x 2. Line and line segment Line passing through x 1 and x 2 : {y
More informationLecture 4: Convexity
10-725: Convex Optimization Fall 2013 Lecture 4: Convexity Lecturer: Barnabás Póczos Scribes: Jessica Chemali, David Fouhey, Yuxiong Wang Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:
More informationMTAEA Convexity and Quasiconvexity
School of Economics, Australian National University February 19, 2010 Convex Combinations and Convex Sets. Definition. Given any finite collection of points x 1,..., x m R n, a point z R n is said to be
More informationProgramming, numerics and optimization
Programming, numerics and optimization Lecture C-4: Constrained optimization Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428 June
More informationConvexization in Markov Chain Monte Carlo
in Markov Chain Monte Carlo 1 IBM T. J. Watson Yorktown Heights, NY 2 Department of Aerospace Engineering Technion, Israel August 23, 2011 Problem Statement MCMC processes in general are governed by non
More informationCOMS 4771 Support Vector Machines. Nakul Verma
COMS 4771 Support Vector Machines Nakul Verma Last time Decision boundaries for classification Linear decision boundary (linear classification) The Perceptron algorithm Mistake bound for the perceptron
More informationLecture 4 Duality and Decomposition Techniques
Lecture 4 Duality and Decomposition Techniques Jie Lu (jielu@kth.se) Richard Combes Alexandre Proutiere Automatic Control, KTH September 19, 2013 Consider the primal problem Lagrange Duality Lagrangian
More informationIntroduction to optimization
Introduction to optimization G. Ferrari Trecate Dipartimento di Ingegneria Industriale e dell Informazione Università degli Studi di Pavia Industrial Automation Ferrari Trecate (DIS) Optimization Industrial
More informationConvexity and Optimization
Convexity and Optimization Richard Lusby Department of Management Engineering Technical University of Denmark Today s Material Extrema Convex Function Convex Sets Other Convexity Concepts Unconstrained
More informationSolution Methods Numerical Algorithms
Solution Methods Numerical Algorithms Evelien van der Hurk DTU Managment Engineering Class Exercises From Last Time 2 DTU Management Engineering 42111: Static and Dynamic Optimization (6) 09/10/2017 Class
More informationHomework 1 (a and b) Convex Sets and Convex Functions
Homework 1 (a and b) Convex Sets and Convex Functions CMU 10-725/36-725: Convex Optimization (Fall 2017) OUT: Sep 1 DUE: Prob 1-3 Sep 11, 5:00 PM; Prob 4 Sep 15, 5:00 PM START HERE: Instructions Collaboration
More informationLinear Programming. Larry Blume. Cornell University & The Santa Fe Institute & IHS
Linear Programming Larry Blume Cornell University & The Santa Fe Institute & IHS Linear Programs The general linear program is a constrained optimization problem where objectives and constraints are all
More informationLecture 2: August 31
10-725/36-725: Convex Optimization Fall 2016 Lecture 2: August 31 Lecturer: Lecturer: Ryan Tibshirani Scribes: Scribes: Lidan Mu, Simon Du, Binxuan Huang 2.1 Review A convex optimization problem is of
More informationLecture 2: August 29, 2018
10-725/36-725: Convex Optimization Fall 2018 Lecturer: Ryan Tibshirani Lecture 2: August 29, 2018 Scribes: Adam Harley Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These notes have
More information15. Cutting plane and ellipsoid methods
EE 546, Univ of Washington, Spring 2012 15. Cutting plane and ellipsoid methods localization methods cutting-plane oracle examples of cutting plane methods ellipsoid method convergence proof inequality
More informationApplied Lagrange Duality for Constrained Optimization
Applied Lagrange Duality for Constrained Optimization Robert M. Freund February 10, 2004 c 2004 Massachusetts Institute of Technology. 1 1 Overview The Practical Importance of Duality Review of Convexity
More informationAdvanced Operations Research Techniques IE316. Quiz 2 Review. Dr. Ted Ralphs
Advanced Operations Research Techniques IE316 Quiz 2 Review Dr. Ted Ralphs IE316 Quiz 2 Review 1 Reading for The Quiz Material covered in detail in lecture Bertsimas 4.1-4.5, 4.8, 5.1-5.5, 6.1-6.3 Material
More informationORIE 6300 Mathematical Programming I September 2, Lecture 3
ORIE 6300 Mathematical Programming I September 2, 2014 Lecturer: David P. Williamson Lecture 3 Scribe: Divya Singhvi Last time we discussed how to take dual of an LP in two different ways. Today we will
More informationConvex optimization algorithms for sparse and low-rank representations
Convex optimization algorithms for sparse and low-rank representations Lieven Vandenberghe, Hsiao-Han Chao (UCLA) ECC 2013 Tutorial Session Sparse and low-rank representation methods in control, estimation,
More informationEC 521 MATHEMATICAL METHODS FOR ECONOMICS. Lecture 2: Convex Sets
EC 51 MATHEMATICAL METHODS FOR ECONOMICS Lecture : Convex Sets Murat YILMAZ Boğaziçi University In this section, we focus on convex sets, separating hyperplane theorems and Farkas Lemma. And as an application
More information16.410/413 Principles of Autonomy and Decision Making
16.410/413 Principles of Autonomy and Decision Making Lecture 17: The Simplex Method Emilio Frazzoli Aeronautics and Astronautics Massachusetts Institute of Technology November 10, 2010 Frazzoli (MIT)
More informationConvex Optimization - Chapter 1-2. Xiangru Lian August 28, 2015
Convex Optimization - Chapter 1-2 Xiangru Lian August 28, 2015 1 Mathematical optimization minimize f 0 (x) s.t. f j (x) 0, j=1,,m, (1) x S x. (x 1,,x n ). optimization variable. f 0. R n R. objective
More informationConic Duality. yyye
Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 1 Conic Duality Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/
More information25. NLP algorithms. ˆ Overview. ˆ Local methods. ˆ Constrained optimization. ˆ Global methods. ˆ Black-box methods.
CS/ECE/ISyE 524 Introduction to Optimization Spring 2017 18 25. NLP algorithms ˆ Overview ˆ Local methods ˆ Constrained optimization ˆ Global methods ˆ Black-box methods ˆ Course wrap-up Laurent Lessard
More informationACCELERATED DUAL GRADIENT-BASED METHODS FOR TOTAL VARIATION IMAGE DENOISING/DEBLURRING PROBLEMS. Donghwan Kim and Jeffrey A.
ACCELERATED DUAL GRADIENT-BASED METHODS FOR TOTAL VARIATION IMAGE DENOISING/DEBLURRING PROBLEMS Donghwan Kim and Jeffrey A. Fessler University of Michigan Dept. of Electrical Engineering and Computer Science
More informationLecture 5: Duality Theory
Lecture 5: Duality Theory Rajat Mittal IIT Kanpur The objective of this lecture note will be to learn duality theory of linear programming. We are planning to answer following questions. What are hyperplane
More informationMathematical and Algorithmic Foundations Linear Programming and Matchings
Adavnced Algorithms Lectures Mathematical and Algorithmic Foundations Linear Programming and Matchings Paul G. Spirakis Department of Computer Science University of Patras and Liverpool Paul G. Spirakis
More informationUnconstrained Optimization Principles of Unconstrained Optimization Search Methods
1 Nonlinear Programming Types of Nonlinear Programs (NLP) Convexity and Convex Programs NLP Solutions Unconstrained Optimization Principles of Unconstrained Optimization Search Methods Constrained Optimization
More informationDiscrete Optimization 2010 Lecture 5 Min-Cost Flows & Total Unimodularity
Discrete Optimization 2010 Lecture 5 Min-Cost Flows & Total Unimodularity Marc Uetz University of Twente m.uetz@utwente.nl Lecture 5: sheet 1 / 26 Marc Uetz Discrete Optimization Outline 1 Min-Cost Flows
More informationCalifornia Institute of Technology Crash-Course on Convex Optimization Fall Ec 133 Guilherme Freitas
California Institute of Technology HSS Division Crash-Course on Convex Optimization Fall 2011-12 Ec 133 Guilherme Freitas In this text, we will study the following basic problem: maximize x C f(x) subject
More informationConvex Optimization. Erick Delage, and Ashutosh Saxena. October 20, (a) (b) (c)
Convex Optimization (for CS229) Erick Delage, and Ashutosh Saxena October 20, 2006 1 Convex Sets Definition: A set G R n is convex if every pair of point (x, y) G, the segment beteen x and y is in A. More
More informationDM545 Linear and Integer Programming. Lecture 2. The Simplex Method. Marco Chiarandini
DM545 Linear and Integer Programming Lecture 2 The Marco Chiarandini Department of Mathematics & Computer Science University of Southern Denmark Outline 1. 2. 3. 4. Standard Form Basic Feasible Solutions
More informationA Brief Look at Optimization
A Brief Look at Optimization CSC 412/2506 Tutorial David Madras January 18, 2018 Slides adapted from last year s version Overview Introduction Classes of optimization problems Linear programming Steepest
More informationAlgorithms for convex optimization
Algorithms for convex optimization Michal Kočvara Institute of Information Theory and Automation Academy of Sciences of the Czech Republic and Czech Technical University kocvara@utia.cas.cz http://www.utia.cas.cz/kocvara
More informationLecture 18: March 23
0-725/36-725: Convex Optimization Spring 205 Lecturer: Ryan Tibshirani Lecture 8: March 23 Scribes: James Duyck Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These notes have not
More informationConvexity and Optimization
Convexity and Optimization Richard Lusby DTU Management Engineering Class Exercises From Last Time 2 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017 Today s Material Extrema
More informationLecture 2: August 29, 2018
10-725/36-725: Convex Optimization Fall 2018 Lecturer: Ryan Tibshirani Lecture 2: August 29, 2018 Scribes: Yingjing Lu, Adam Harley, Ruosong Wang Note: LaTeX template courtesy of UC Berkeley EECS dept.
More informationNonlinear Programming
Nonlinear Programming SECOND EDITION Dimitri P. Bertsekas Massachusetts Institute of Technology WWW site for book Information and Orders http://world.std.com/~athenasc/index.html Athena Scientific, Belmont,
More informationLecture 2 Optimization with equality constraints
Lecture 2 Optimization with equality constraints Constrained optimization The idea of constrained optimisation is that the choice of one variable often affects the amount of another variable that can be
More informationLecture 2. Topology of Sets in R n. August 27, 2008
Lecture 2 Topology of Sets in R n August 27, 2008 Outline Vectors, Matrices, Norms, Convergence Open and Closed Sets Special Sets: Subspace, Affine Set, Cone, Convex Set Special Convex Sets: Hyperplane,
More informationConvex Optimization and Machine Learning
Convex Optimization and Machine Learning Mengliu Zhao Machine Learning Reading Group School of Computing Science Simon Fraser University March 12, 2014 Mengliu Zhao SFU-MLRG March 12, 2014 1 / 25 Introduction
More informationAffine function. suppose f : R n R m is affine (f(x) =Ax + b with A R m n, b R m ) the image of a convex set under f is convex
Affine function suppose f : R n R m is affine (f(x) =Ax + b with A R m n, b R m ) the image of a convex set under f is convex S R n convex = f(s) ={f(x) x S} convex the inverse image f 1 (C) of a convex
More informationIDENTIFYING ACTIVE MANIFOLDS
Algorithmic Operations Research Vol.2 (2007) 75 82 IDENTIFYING ACTIVE MANIFOLDS W.L. Hare a a Department of Mathematics, Simon Fraser University, Burnaby, BC V5A 1S6, Canada. A.S. Lewis b b School of ORIE,
More informationA Multilevel Proximal Gradient Algorithm for a Class of Composite Optimization Problems
A Multilevel Proximal Gradient Algorithm for a Class of Composite Optimization Problems Panos Parpas May 9, 2017 Abstract Composite optimization models consist of the minimization of the sum of a smooth
More information2. Optimization problems 6
6 2.1 Examples... 7... 8 2.3 Convex sets and functions... 9 2.4 Convex optimization problems... 10 2.1 Examples 7-1 An (NP-) optimization problem P 0 is defined as follows Each instance I P 0 has a feasibility
More information14.5 Directional Derivatives and the Gradient Vector
14.5 Directional Derivatives and the Gradient Vector 1. Directional Derivatives. Recall z = f (x, y) and the partial derivatives f x and f y are defined as f (x 0 + h, y 0 ) f (x 0, y 0 ) f x (x 0, y 0
More informationLecture 12: Feasible direction methods
Lecture 12 Lecture 12: Feasible direction methods Kin Cheong Sou December 2, 2013 TMA947 Lecture 12 Lecture 12: Feasible direction methods 1 / 1 Feasible-direction methods, I Intro Consider the problem
More informationConvex Sets (cont.) Convex Functions
Convex Sets (cont.) Convex Functions Optimization - 10725 Carlos Guestrin Carnegie Mellon University February 27 th, 2008 1 Definitions of convex sets Convex v. Non-convex sets Line segment definition:
More informationLecture 25 Nonlinear Programming. November 9, 2009
Nonlinear Programming November 9, 2009 Outline Nonlinear Programming Another example of NLP problem What makes these problems complex Scalar Function Unconstrained Problem Local and global optima: definition,
More informationIn this chapter we introduce some of the basic concepts that will be useful for the study of integer programming problems.
2 Basics In this chapter we introduce some of the basic concepts that will be useful for the study of integer programming problems. 2.1 Notation Let A R m n be a matrix with row index set M = {1,...,m}
More informationA More Efficient Approach to Large Scale Matrix Completion Problems
A More Efficient Approach to Large Scale Matrix Completion Problems Matthew Olson August 25, 2014 Abstract This paper investigates a scalable optimization procedure to the low-rank matrix completion problem
More informationThe Alternating Direction Method of Multipliers
The Alternating Direction Method of Multipliers With Adaptive Step Size Selection Peter Sutor, Jr. Project Advisor: Professor Tom Goldstein October 8, 2015 1 / 30 Introduction Presentation Outline 1 Convex
More informationLecture 3: Convex sets
Lecture 3: Convex sets Rajat Mittal IIT Kanpur We denote the set of real numbers as R. Most of the time we will be working with space R n and its elements will be called vectors. Remember that a subspace
More informationConvex Optimization. Stephen Boyd
Convex Optimization Stephen Boyd Electrical Engineering Computer Science Management Science and Engineering Institute for Computational Mathematics & Engineering Stanford University Institute for Advanced
More informationAlternating Projections
Alternating Projections Stephen Boyd and Jon Dattorro EE392o, Stanford University Autumn, 2003 1 Alternating projection algorithm Alternating projections is a very simple algorithm for computing a point
More informationPROJECTION ONTO A POLYHEDRON THAT EXPLOITS SPARSITY
PROJECTION ONTO A POLYHEDRON THAT EXPLOITS SPARSITY WILLIAM W. HAGER AND HONGCHAO ZHANG Abstract. An algorithm is developed for projecting a point onto a polyhedron. The algorithm solves a dual version
More informationCPSC 340: Machine Learning and Data Mining. Robust Regression Fall 2015
CPSC 340: Machine Learning and Data Mining Robust Regression Fall 2015 Admin Can you see Assignment 1 grades on UBC connect? Auditors, don t worry about it. You should already be working on Assignment
More informationBilinear Programming
Bilinear Programming Artyom G. Nahapetyan Center for Applied Optimization Industrial and Systems Engineering Department University of Florida Gainesville, Florida 32611-6595 Email address: artyom@ufl.edu
More informationProbabilistic Graphical Models
School of Computer Science Probabilistic Graphical Models Theory of Variational Inference: Inner and Outer Approximation Eric Xing Lecture 14, February 29, 2016 Reading: W & J Book Chapters Eric Xing @
More information4 Integer Linear Programming (ILP)
TDA6/DIT37 DISCRETE OPTIMIZATION 17 PERIOD 3 WEEK III 4 Integer Linear Programg (ILP) 14 An integer linear program, ILP for short, has the same form as a linear program (LP). The only difference is that
More informationOptimization. Industrial AI Lab.
Optimization Industrial AI Lab. Optimization An important tool in 1) Engineering problem solving and 2) Decision science People optimize Nature optimizes 2 Optimization People optimize (source: http://nautil.us/blog/to-save-drowning-people-ask-yourself-what-would-light-do)
More information