Shiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 2. Convex Optimization
|
|
- Gladys Hall
- 5 years ago
- Views:
Transcription
1 Shiqian Ma, MAT-258A: Numerical Optimization 1 Chapter 2 Convex Optimization
2 Shiqian Ma, MAT-258A: Numerical Optimization Convex Optimization General optimization problem: min f 0 (x) s.t., f i (x) b i, i = 1,..., m. x = (x 1,..., x n ): optimization variables f 0 : R n R: objective function f i : R n R, i = 1,..., m: constraint functions
3 Shiqian Ma, MAT-258A: Numerical Optimization 3 General optimization problem very difficult to solve methods involve some compromise: long computation time, or not always find the global solution exceptions: certain problem classes can be solved efficiently and reliably linear programming problems convex optimization problems We focus on convex optimization problems in this course. many tricks to transforming problems into convex form surprisingly many problems can be solved via convex optimization
4 Shiqian Ma, MAT-258A: Numerical Optimization 4 Convex optimization min f 0 (x) s.t., f i (x) b i, i = 1,..., m objective and constraint functions are convex f i (λx + (1 λ)y) λf i (x) + (1 λ)f i (y) for all x, y R n, and λ [0, 1]. i.e., objective function is a convex function, feasible region is a convex set examples: LP, QP, SDP
5 Shiqian Ma, MAT-258A: Numerical Optimization Convex sets Affine set Line through x 1, x 2 : all points x = θx 1 + (1 θ)x 2, (θ R) Affine set: contains the line through any two distinct points in the set example: solution set of linear equations {x Ax = b}
6 Shiqian Ma, MAT-258A: Numerical Optimization 6 Convex set Line segment between x 1 and x 2 : all points x = θx 1 + (1 θ)x 2, 0 θ 1 convex set: contains line segment between any two points in the set x 1, x 2 C, 0 θ 1 θx 1 + (1 θ)x 2 C convex hull (conv(s)): set of all convex combinations of points in S: k x = θ 1 x 1 + θ 2 x θ k x k, θ i = 1, θ i 0. i=1
7 Shiqian Ma, MAT-258A: Numerical Optimization 7 Convex cone: set that contains all conic combinations of points in the set. conic combination of x 1 and x 2 : any point of the form x = θ 1 x 1 + θ 2 x 2, θ 1 0, θ 2 0. hyperplane: set of form {x a x = b}, (a 0) halfspace: set of the form {x a x b}, (a 0) a is the normal vector hyperplanes are affine and convex; halfspaces are convex Euclidean ball with center x c and radius r: B(x c, r) = {x x x c 2 r} = {x c + ru u 2 1}
8 Shiqian Ma, MAT-258A: Numerical Optimization 8 ellipsoid: set of the form {x (x x c ) P 1 (x x c ) 1} with P S n ++ (i.e., P symmetric positive definite) norm ball with center x c and radius r: {x x x c r} norm cone: {(x, t) x t} Euclidean norm cone is called second-order cone Polyhedra: solution set of finitely many linear inequalities and equalities Ax b, Cx = d. Positive semidefinite cone S n denotes the set of symmetric n n matrices
9 Shiqian Ma, MAT-258A: Numerical Optimization 9 S n + = {X S n X 0} denotes positive semidefinite n n matrices X S n + z Xz 0, for all z S n + is a convex cone S n ++ = {X S n X 0}: positive definite n n matrices
10 Shiqian Ma, MAT-258A: Numerical Optimization 10 Operations that preserve convexity practical methods for establishing convexity of a set C apply definition x 1, x 2 C, 0 θ 1 θx 1 + (1 θ)x 2 C show that C is obtained from simple convex sets (hyperplanes, halfspaces, norm balls,...) by operations that preserve convexity intersection affine functions perspective function linear-fractional functions
11 Shiqian Ma, MAT-258A: Numerical Optimization 11 Affine function suppose f : R n R m is affine (f(x) = Ax + b with A R m n, b R m ) the image of convex set under f is convex S R n convex f(s) = {f(x) x S} convex the inverse image f 1 (C) of a convex set under f is convex Examples: C R m convex f 1 (C) = {x R n f(x) C} convex scaling, translation, projection solution set of linear matrix inequality {x x 1 A x m A m B} (with A i, B S p )
12 Shiqian Ma, MAT-258A: Numerical Optimization 12 Perspective and linear-fractional functions Perspective function P : R n+1 R n : P (x, t) = x/t, dom P = {(x, t) t > 0} images and inverse images of convex sets under perspective are convex linear-fractional function f : R n R m : f(x) = Ax + b c x + d, dom f = {x c x + d > 0} images and inverse images of convex sets under linear-fractional are convex
13 Shiqian Ma, MAT-258A: Numerical Optimization 13 Basic properties of convexity Separation hyperplane theorem if C and D are disjoint convex sets, then there exists a 0, b such that a x b for x C, a x b for x D the hyperplane {x a x = b} separates C and D supporting hyperplane to set C at boundary point x 0 : {x a x = a x 0 } where a 0 and a x a x 0 for all x C supporting hyperplane theorem: if C is convex, then there exists a supporting hyperplane at every boundary point of C
14 Shiqian Ma, MAT-258A: Numerical Optimization Convex functions
15 Shiqian Ma, MAT-258A: Numerical Optimization 15 Definition f : R n R is convex if dom f is a convex set and f(θx + (1 θ)y) θf(x) + (1 θ)f(y) for all x, y dom f, 0 θ 1 f is concave if f is convex f is strictly convex if dom f is a convex set and f(θx + (1 θ)y) < θf(x) + (1 θ)f(y) for all x, y dom f, x y, 0 < θ < 1
16 Shiqian Ma, MAT-258A: Numerical Optimization 16 Examples on R Convex: affine: ax + b on R, for any a, b R exponential: e ax, for any a R powers: x α on R ++, for α 1 or α 0 powers of absolution value: x p on R, for p 1 negative entropy: x log x on R ++ Concave: affine: ax + b on R, for any a, b R powers: x α on R ++, for 0 α 1 logarithm: log x on R ++
17 Shiqian Ma, MAT-258A: Numerical Optimization 17 Examples on R n and R m n affine functions are convex and concave; all norms are convex examples on R n affine function f(x) = a x + b norms: x p = ( n i=1 x i p ) 1/p for p 1; x = max k x k examples on R m n affine function spectral norm f(x) = Tr (A X) + b = m i=1 n A ij X ij + b j=1 f(x) = X 2 = σ max (X) = (λ max (X X)) 1/2
18 Shiqian Ma, MAT-258A: Numerical Optimization 18 Restriction of a convex function to a line f : R n R is convex if and only if the function g : R R, g(t) = f(x + tv), dom g = {t x + tv dom f} is convex (in t) for any x dom f, v R n can check convexity of f by checking convexity of functions of one variable example. f : S n R with f(x) = log det X, dom f = S n ++ g(t) = log det(x + tv ) = log det X + log det(i + tx 1/2 V X 1/2 ) = log det X + n i=1 log(1 + tλ i) where λ i are the eigenvalues of X 1/2 V X 1/2 g is concave in t (for any choice of X 0, V ); hence f is concave
19 Shiqian Ma, MAT-258A: Numerical Optimization 19 Extended-value extension extended-value extension f of f is f(x) = f(x), x dom f, f(x) =, x / dom f often simplifies notation; for example, the condition 0 θ 1 f(θx + (1 θ)y) θ f(x) + (1 θ) f(y) dom f is convex for x, y dom f, 0 θ 1 f(θx + (1 θ)y) θf(x) + (1 θ)f(y)
20 Shiqian Ma, MAT-258A: Numerical Optimization 20 First-order condition f is differentiable if dom f is open and the gradient ( f(x) f(x) =, f(x),..., f(x) ) x 1 x 2 x n exists at each x dom f First-order condition: differentiable f with convex domain is convex iff f(y) f(x) + f(x) (y x) for all x, y dom f First-order approximation of f is global underestimator
21 Shiqian Ma, MAT-258A: Numerical Optimization 21 Second-order conditions f is twice differentiable if dom f is open and the Hessian 2 f(x) S n, 2 f(x) ij = 2 f(x) x i x j, i, j = 1,..., n, exists at each x dom f Second-order conditions: for twice differentiable f with convex domain f is convex if and only if 2 f(x) 0 for all x dom f if 2 f(x) 0 for all x dom f, then f is strictly convex
22 Shiqian Ma, MAT-258A: Numerical Optimization 22 Examples quadratic function: f(x) = (1/2)x P x + q x + r (with P S n ) f(x) = P x + q, 2 f(x) = P convex if P 0 least-squares objective: f(x) = Ax b 2 2 f(x) = 2A (Ax b), 2 f(x) = 2A A convex (for any A) quadratic-over-linear: f(x, y) = x 2 /y 2 f(x, y) = 2 ( ) ( ) y y 0 y 3 x x convex for any y > 0
23 Shiqian Ma, MAT-258A: Numerical Optimization 23 log-sum-exp: f(x) = log n k=1 exp x k is convex 2 f(x) = 1 1 z diag(z) 1 (1 z) 2zz (z k = exp x k ) to show 2 f(x) 0, we must verify that v 2 f(x)v 0 for all v: v 2 f(x)v = ( k z kv 2 k )( k z k) ( k v kz k ) 2 ( k z k) 2 0 since ( k v kz k ) 2 ( k z kv 2 k )( k z k) (from Cauchy-Schwarz inequality) geometric mean: f(x) = ( n k=1 x k) 1/n on R n ++ is concave (similar proof as for log-sum-exp)
24 Shiqian Ma, MAT-258A: Numerical Optimization 24 α-sublevel set of f : R n R: Epigraph and sublevel set C α = {x dom f f(x) α} sublevel sets of convex functions are convex (converse is false) epigraph of f : R n R: epi f = {(x, t) R n+1 x dom f, f(x) t} f is convex if and only if epi f is a convex set
25 Shiqian Ma, MAT-258A: Numerical Optimization 25 Jensen s inequality if f is convex, then for any random variable z f(ez) Ef(z)
26 Shiqian Ma, MAT-258A: Numerical Optimization 26 Operations that preserve convexity practical methods for establishing convexity of a function verify definition (often simplified by restricting to a line) for twice differentiable functions, show 2 f(x) 0 show that f is obtained from simple convex functions by operations that preserve convexity nonnegative weighted sum composition with affine function pointwise maximum and supremum composition minimization perspective
27 Shiqian Ma, MAT-258A: Numerical Optimization 27 Composition with affine function composition with affine function: f(ax + b) is convex if f is convex examples: log barrier for linear inequalities m f(x) = log(b i a i x), dom f = {x a i x < b i, i = 1,..., m} i=1 (any) norm of affine function: f(x) = Ax + b
28 Shiqian Ma, MAT-258A: Numerical Optimization 28 Pointwise maximum if f 1,..., f m are convex, then f(x) = max{f 1 (x),..., f m (x)} is convex examples: piecewise-linear function: f(x) = max i=1,...,m (a i x + b i) is convex sum of r largest components of x R n : f(x) = x [1] + x [2] x [r] is convex (x [i] is the i-th largest component of x) proof: f(x) = max{x i1 + x i x ir 1 i 1 < i 2 <... < i r n}
29 Shiqian Ma, MAT-258A: Numerical Optimization 29 Pointwise supremum if f(x, y) is convex in x for each y A, then is convex examples: g(x) = sup f(x, y) y A support function of a set C: S C (x) = sup y C y x is convex distance to farthest point in a set C: f(x) = sup x y y C maximum eigenvalue of symmetric matrix: for X S n, λ max (X) = sup y Xy y 2 =1
30 Shiqian Ma, MAT-258A: Numerical Optimization 30 Composition with scalar functions composition of g : R n R and h : R R: f(x) = h(g(x)) f is convex if g convex, h convex, h nondecreasing f is convex if g concave, h convex, h nonincreasing proof (for n = 1, differentiable g,h) examples f (x) = h (g(x))g (x) 2 + h (g(x))g (x) exp g(x) is convex if g is convex 1/g(x) is convex if g is concave and positive
31 Shiqian Ma, MAT-258A: Numerical Optimization 31 Vector composition composition of g : R n R k and h : R k R: f(x) = h(g(x)) = h(g 1 (x), g 2 (x),..., g k (x)) f is convex if g i convex, h convex, h nondecreasing in each argument f is convex if g i concave, h convex, h nonincreasing in each argument proof (for n = 1, differentiable g,h) examples f (x) = g (x) 2 h(g(x))g (x) + h(g(x)) g (x) m i=1 log g i(x) is concave if g i are concave and positive log m i=1 exp g i(x) is convex if g i are convex
32 Shiqian Ma, MAT-258A: Numerical Optimization 32 Minimization if f(x, y) is convex in (x, y) and C is a convex set, then is convex examples g(x) = inf f(x, y) y C f(x, y) = x Ax + 2x By + y Cy with ( ) A B B 0, C 0 C minimizing over y gives g(x) = inf y f(x, y) = x (A BC 1 B )x g is convex because Schur complement A BC 1 B 0 distance to a set: dist(x, S) = inf y S x y is convex if S is convex
33 Shiqian Ma, MAT-258A: Numerical Optimization 33 Perspective the perspective of a function f : R n R is the function g : R n R R, g(x, t) = tf(x/t), dom g = {(x, t) x/t dom f, t > 0} g is convex if f is convex examples f(x) = x x is convex; hence g(x, t) = x x/t is convex for t > 0 negative logarithm f(x) = log x is convex; hence relative entropy g(x, t) = t log t t log x is convex on R 2 ++ if f is convex, then g(x) = (c x + d)f((ax + b)/(c x + d)) is convex on {x c x + d > 0, (Ax + b)/(c x + d) dom f}
34 Shiqian Ma, MAT-258A: Numerical Optimization 34 the conjugate of a function f is The conjugate function f (y) = f is convex (even if f is not) examples negative logarithm f(x) = log x sup (y x f(x)) x dom f f (y) = { sup x>0 (xy + log x) 1 log( y) if y < 0 = otherwise strictly convex quadratic f(x) = (1/2)x Qx with Q S n ++ f (y) = sup(y x (1/2)x Qx = (1/2)y Q 1 y x
35 Shiqian Ma, MAT-258A: Numerical Optimization Convex optimization problem standard form convex optimization problem min f 0 (x) s.t. f i (x) 0, i = 1,..., m a i x = b i, i = 1,..., p f 0, f 1,..., f m are convex; equality constraints are affine often written as min f 0 (x) s.t. f i (x) 0, i = 1,..., m Ax = b feasible set of a convex optimization problem is convex
36 Shiqian Ma, MAT-258A: Numerical Optimization 36 Local and global optima any locally optimal point of a convex problem is (globally) optimal proof: suppose x is locally optimal and y is optimal with f 0 (y) < f 0 (x) x locally optimal means there is an R > 0 such that z feasible, z x 2 R f 0 (z) f 0 (x) consider z = θy + (1 θ)x with θ = R/(2 y x 2 ) y x 2 > R, so 0 < θ < 1/2 z is a convex combination of two feasible points, hence also feasible z x 2 = R/2 and f 0 (z) θf 0 (x) + (1 θ)f 0 (y) < f 0 (x) which contradicts our assumption that x is locally optimal
37 Shiqian Ma, MAT-258A: Numerical Optimization 37 Optimality conditions for differentiable f 0 x is optimal if and only if it is feasible and f 0 (x) (y x) 0 for all feasible y if nonzero, f 0 (x) defines a supporting hyperplane to feasible set X at x unconstrained problem: x is optimal if and only if x dom f 0, f 0 (x) = 0
38 Shiqian Ma, MAT-258A: Numerical Optimization 38 Lagrangian standard form problem (not necessarily convex) min f 0 (x) s.t. f i (x) 0, i = 1,..., m h i (x) = 0, i = 1,..., p variable x R n, domain D, optimal value p Lagrangian: L : R n R m R p R, with dom L = D R m R p L(x, λ, ν) = f 0 (x) + m λ i f i (x) + i=1 Lagrange dual function: g : R m R p R, p ν i h i (x) g(λ, ν) = inf x D L(x, λ, ν) = inf x D (f 0 (x) + m i=1 λ if i (x) + p i=1 ν ih i (x)) i=1
39 Shiqian Ma, MAT-258A: Numerical Optimization 39 g is concave, can be for some λ, ν lower bound property: if λ 0, then g(λ, ν) p proof: if x is feasible and λ 0, then f 0 ( x) L( x, λ, ν) inf L(x, λ, ν) = g(λ, ν) x D minimizing over all feasible x gives p g(λ, ν)
40 Shiqian Ma, MAT-258A: Numerical Optimization 40 Lagrange dual and conjugate function dual function min f 0 (x) s.t. Ax b, Cx = d g(λ, ν) = inf x dom f0 (f 0 (x) + (A λ + C ν) x b λ d ν) = f 0 ( A λ C ν) b λ d ν
41 Shiqian Ma, MAT-258A: Numerical Optimization 41 Lagrange dual problem finds best lower bound on p The dual problem max g(λ, ν) s.t. λ 0 a convex optimization problem; optimal value denoted by d
42 Shiqian Ma, MAT-258A: Numerical Optimization 42 weak duality: d p Weak and strong duality always holds (for convex and nonconvex problems) can be used to find nontrivial lower bounds for difficult problems strong duality: d = p does not hold in general (usually) holds for convex problems conditions that guarantee strong duality in convex problems are called constraint qualifications
43 Shiqian Ma, MAT-258A: Numerical Optimization 43 Slater s constraint qualification strong duality holds for a convex problem if it is strictly feasible, i.e., min f 0 (x) s.t. f i (x) 0, i = 1,..., m Ax = b x int D : f i (x) < 0, i = 1,..., m, Ax = b also guarantees that the dual optimum is attained (if p > ) there exist many other types of constraint qualifications
44 Shiqian Ma, MAT-258A: Numerical Optimization 44 Karush-Kuhn-Tucker (KKT) conditions the following 4 conditions are called KKT conditions (for a problem with differentiable f i,h i ): primal constraints: f i (x) 0, i = 1,..., m, h i (x) = 0, i = 1,..., p dual constraints: λ 0 complementary slackness: λ i f i (x) = 0, i = 1,..., m gradient of Lagrangian wrt x vanishes: m p f 0 (x) + λ i f i (x) + ν i h i (x) = 0 i=1 i=1
45 Shiqian Ma, MAT-258A: Numerical Optimization 45 KKT conditions for convex problem if x, λ, ν satisfy KKT for a convex problem, then they are optimal: from complementary slackness: f 0 ( x) = L( x, λ, ν) from 4th condition (and convexity): g( λ, ν) = L( x, λ, ν) hence f 0 ( x) = g( λ, ν) if Slater s condition is satisfied: x is optimal iff there exist λ,ν that satisfy KKT conditions recall that Slater implies strong duality, and dual optimum is attained generalizes optimality condition f 0 (x) = 0 for unconstrained problem
Affine function. suppose f : R n R m is affine (f(x) =Ax + b with A R m n, b R m ) the image of a convex set under f is convex
Affine function suppose f : R n R m is affine (f(x) =Ax + b with A R m n, b R m ) the image of a convex set under f is convex S R n convex = f(s) ={f(x) x S} convex the inverse image f 1 (C) of a convex
More informationConvex sets and convex functions
Convex sets and convex functions Convex optimization problems Convex sets and their examples Separating and supporting hyperplanes Projections on convex sets Convex functions, conjugate functions ECE 602,
More informationConvex sets and convex functions
Convex sets and convex functions Convex optimization problems Convex sets and their examples Separating and supporting hyperplanes Projections on convex sets Convex functions, conjugate functions ECE 602,
More informationCOM Optimization for Communications Summary: Convex Sets and Convex Functions
1 Convex Sets Affine Sets COM524500 Optimization for Communications Summary: Convex Sets and Convex Functions A set C R n is said to be affine if A point x 1, x 2 C = θx 1 + (1 θ)x 2 C, θ R (1) y = k θ
More information2. Convex sets. affine and convex sets. some important examples. operations that preserve convexity. generalized inequalities
2. Convex sets Convex Optimization Boyd & Vandenberghe affine and convex sets some important examples operations that preserve convexity generalized inequalities separating and supporting hyperplanes dual
More informationConvexity Theory and Gradient Methods
Convexity Theory and Gradient Methods Angelia Nedić angelia@illinois.edu ISE Department and Coordinated Science Laboratory University of Illinois at Urbana-Champaign Outline Convex Functions Optimality
More informationCS675: Convex and Combinatorial Optimization Fall 2014 Convex Functions. Instructor: Shaddin Dughmi
CS675: Convex and Combinatorial Optimization Fall 2014 Convex Functions Instructor: Shaddin Dughmi Outline 1 Convex Functions 2 Examples of Convex and Concave Functions 3 Convexity-Preserving Operations
More informationTutorial on Convex Optimization for Engineers
Tutorial on Convex Optimization for Engineers M.Sc. Jens Steinwandt Communications Research Laboratory Ilmenau University of Technology PO Box 100565 D-98684 Ilmenau, Germany jens.steinwandt@tu-ilmenau.de
More information2. Convex sets. x 1. x 2. affine set: contains the line through any two distinct points in the set
2. Convex sets Convex Optimization Boyd & Vandenberghe affine and convex sets some important examples operations that preserve convexity generalized inequalities separating and supporting hyperplanes dual
More informationConvex Optimization. Convex Sets. ENSAE: Optimisation 1/24
Convex Optimization Convex Sets ENSAE: Optimisation 1/24 Today affine and convex sets some important examples operations that preserve convexity generalized inequalities separating and supporting hyperplanes
More information2. Convex sets. affine and convex sets. some important examples. operations that preserve convexity. generalized inequalities
2. Convex sets Convex Optimization Boyd & Vandenberghe affine and convex sets some important examples operations that preserve convexity generalized inequalities separating and supporting hyperplanes dual
More informationIntroduction to Modern Control Systems
Introduction to Modern Control Systems Convex Optimization, Duality and Linear Matrix Inequalities Kostas Margellos University of Oxford AIMS CDT 2016-17 Introduction to Modern Control Systems November
More informationLecture 4: Convexity
10-725: Convex Optimization Fall 2013 Lecture 4: Convexity Lecturer: Barnabás Póczos Scribes: Jessica Chemali, David Fouhey, Yuxiong Wang Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:
More informationConvex Sets. CSCI5254: Convex Optimization & Its Applications. subspaces, affine sets, and convex sets. operations that preserve convexity
CSCI5254: Convex Optimization & Its Applications Convex Sets subspaces, affine sets, and convex sets operations that preserve convexity generalized inequalities separating and supporting hyperplanes dual
More informationConvexity I: Sets and Functions
Convexity I: Sets and Functions Lecturer: Aarti Singh Co-instructor: Pradeep Ravikumar Convex Optimization 10-725/36-725 See supplements for reviews of basic real analysis basic multivariate calculus basic
More informationLecture 2: August 31
10-725/36-725: Convex Optimization Fall 2016 Lecture 2: August 31 Lecturer: Lecturer: Ryan Tibshirani Scribes: Scribes: Lidan Mu, Simon Du, Binxuan Huang 2.1 Review A convex optimization problem is of
More informationConvex Sets (cont.) Convex Functions
Convex Sets (cont.) Convex Functions Optimization - 10725 Carlos Guestrin Carnegie Mellon University February 27 th, 2008 1 Definitions of convex sets Convex v. Non-convex sets Line segment definition:
More informationIntroduction to Convex Optimization. Prof. Daniel P. Palomar
Introduction to Convex Optimization Prof. Daniel P. Palomar The Hong Kong University of Science and Technology (HKUST) MAFS6010R- Portfolio Optimization with R MSc in Financial Mathematics Fall 2018-19,
More informationMathematical Programming and Research Methods (Part II)
Mathematical Programming and Research Methods (Part II) 4. Convexity and Optimization Massimiliano Pontil (based on previous lecture by Andreas Argyriou) 1 Today s Plan Convex sets and functions Types
More informationLecture 2: August 29, 2018
10-725/36-725: Convex Optimization Fall 2018 Lecturer: Ryan Tibshirani Lecture 2: August 29, 2018 Scribes: Adam Harley Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These notes have
More informationCS675: Convex and Combinatorial Optimization Spring 2018 Convex Sets. Instructor: Shaddin Dughmi
CS675: Convex and Combinatorial Optimization Spring 2018 Convex Sets Instructor: Shaddin Dughmi Outline 1 Convex sets, Affine sets, and Cones 2 Examples of Convex Sets 3 Convexity-Preserving Operations
More informationConvex Optimization M2
Convex Optimization M2 Lecture 1 A. d Aspremont. Convex Optimization M2. 1/49 Today Convex optimization: introduction Course organization and other gory details... Convex sets, basic definitions. A. d
More informationConvex Optimization. 2. Convex Sets. Prof. Ying Cui. Department of Electrical Engineering Shanghai Jiao Tong University. SJTU Ying Cui 1 / 33
Convex Optimization 2. Convex Sets Prof. Ying Cui Department of Electrical Engineering Shanghai Jiao Tong University 2018 SJTU Ying Cui 1 / 33 Outline Affine and convex sets Some important examples Operations
More informationLecture: Convex Sets
/24 Lecture: Convex Sets http://bicmr.pku.edu.cn/~wenzw/opt-27-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/24 affine and convex sets some
More informationConvex Optimization - Chapter 1-2. Xiangru Lian August 28, 2015
Convex Optimization - Chapter 1-2 Xiangru Lian August 28, 2015 1 Mathematical optimization minimize f 0 (x) s.t. f j (x) 0, j=1,,m, (1) x S x. (x 1,,x n ). optimization variable. f 0. R n R. objective
More informationLecture 5: Properties of convex sets
Lecture 5: Properties of convex sets Rajat Mittal IIT Kanpur This week we will see properties of convex sets. These properties make convex sets special and are the reason why convex optimization problems
More information60 2 Convex sets. {x a T x b} {x ã T x b}
60 2 Convex sets Exercises Definition of convexity 21 Let C R n be a convex set, with x 1,, x k C, and let θ 1,, θ k R satisfy θ i 0, θ 1 + + θ k = 1 Show that θ 1x 1 + + θ k x k C (The definition of convexity
More informationCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 4: Convex Sets. Instructor: Shaddin Dughmi
CS599: Convex and Combinatorial Optimization Fall 2013 Lecture 4: Convex Sets Instructor: Shaddin Dughmi Announcements New room: KAP 158 Today: Convex Sets Mostly from Boyd and Vandenberghe. Read all of
More informationConvexity: an introduction
Convexity: an introduction Geir Dahl CMA, Dept. of Mathematics and Dept. of Informatics University of Oslo 1 / 74 1. Introduction 1. Introduction what is convexity where does it arise main concepts and
More informationLecture 2: August 29, 2018
10-725/36-725: Convex Optimization Fall 2018 Lecturer: Ryan Tibshirani Lecture 2: August 29, 2018 Scribes: Yingjing Lu, Adam Harley, Ruosong Wang Note: LaTeX template courtesy of UC Berkeley EECS dept.
More informationChapter 4 Convex Optimization Problems
Chapter 4 Convex Optimization Problems Shupeng Gui Computer Science, UR October 16, 2015 hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, 2015 1 / 58 Outline 1 Optimization problems
More informationApplied Lagrange Duality for Constrained Optimization
Applied Lagrange Duality for Constrained Optimization Robert M. Freund February 10, 2004 c 2004 Massachusetts Institute of Technology. 1 1 Overview The Practical Importance of Duality Review of Convexity
More informationConvex Optimization. Erick Delage, and Ashutosh Saxena. October 20, (a) (b) (c)
Convex Optimization (for CS229) Erick Delage, and Ashutosh Saxena October 20, 2006 1 Convex Sets Definition: A set G R n is convex if every pair of point (x, y) G, the segment beteen x and y is in A. More
More informationLecture 19: Convex Non-Smooth Optimization. April 2, 2007
: Convex Non-Smooth Optimization April 2, 2007 Outline Lecture 19 Convex non-smooth problems Examples Subgradients and subdifferentials Subgradient properties Operations with subgradients and subdifferentials
More informationIE 521 Convex Optimization
Lecture 4: 5th February 2019 Outline 1 / 23 Which function is different from others? Figure: Functions 2 / 23 Definition of Convex Function Definition. A function f (x) : R n R is convex if (i) dom(f )
More informationAspects of Convex, Nonconvex, and Geometric Optimization (Lecture 1) Suvrit Sra Massachusetts Institute of Technology
Aspects of Convex, Nonconvex, and Geometric Optimization (Lecture 1) Suvrit Sra Massachusetts Institute of Technology Hausdorff Institute for Mathematics (HIM) Trimester: Mathematics of Signal Processing
More informationCMU-Q Lecture 9: Optimization II: Constrained,Unconstrained Optimization Convex optimization. Teacher: Gianni A. Di Caro
CMU-Q 15-381 Lecture 9: Optimization II: Constrained,Unconstrained Optimization Convex optimization Teacher: Gianni A. Di Caro GLOBAL FUNCTION OPTIMIZATION Find the global maximum of the function f x (and
More informationConvex Optimization. Lijun Zhang Modification of
Convex Optimization Lijun Zhang zlj@nju.edu.cn http://cs.nju.edu.cn/zlj Modification of http://stanford.edu/~boyd/cvxbook/bv_cvxslides.pdf Outline Introduction Convex Sets & Functions Convex Optimization
More informationConvex Optimization Lecture 2
Convex Optimization Lecture 2 Today: Convex Analysis Center-of-mass Algorithm 1 Convex Analysis Convex Sets Definition: A set C R n is convex if for all x, y C and all 0 λ 1, λx + (1 λ)y C Operations that
More informationOptimization under uncertainty: modeling and solution methods
Optimization under uncertainty: modeling and solution methods Paolo Brandimarte Dipartimento di Scienze Matematiche Politecnico di Torino e-mail: paolo.brandimarte@polito.it URL: http://staff.polito.it/paolo.brandimarte
More informationIntroduction to Constrained Optimization
Introduction to Constrained Optimization Duality and KKT Conditions Pratik Shah {pratik.shah [at] lnmiit.ac.in} The LNM Institute of Information Technology www.lnmiit.ac.in February 13, 2013 LNMIIT MLPR
More informationSimplex Algorithm in 1 Slide
Administrivia 1 Canonical form: Simplex Algorithm in 1 Slide If we do pivot in A r,s >0, where c s
More informationMTAEA Convexity and Quasiconvexity
School of Economics, Australian National University February 19, 2010 Convex Combinations and Convex Sets. Definition. Given any finite collection of points x 1,..., x m R n, a point z R n is said to be
More informationEE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 6
EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 6 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory April 19, 2012 Andre Tkacenko
More informationLecture 2 Convex Sets
Optimization Theory and Applications Lecture 2 Convex Sets Prof. Chun-Hung Liu Dept. of Electrical and Computer Engineering National Chiao Tung University Fall 2016 2016/9/29 Lecture 2: Convex Sets 1 Outline
More informationNumerical Optimization
Convex Sets Computer Science and Automation Indian Institute of Science Bangalore 560 012, India. NPTEL Course on Let x 1, x 2 R n, x 1 x 2. Line and line segment Line passing through x 1 and x 2 : {y
More informationLecture 5: Duality Theory
Lecture 5: Duality Theory Rajat Mittal IIT Kanpur The objective of this lecture note will be to learn duality theory of linear programming. We are planning to answer following questions. What are hyperplane
More informationDM545 Linear and Integer Programming. Lecture 2. The Simplex Method. Marco Chiarandini
DM545 Linear and Integer Programming Lecture 2 The Marco Chiarandini Department of Mathematics & Computer Science University of Southern Denmark Outline 1. 2. 3. 4. Standard Form Basic Feasible Solutions
More informationLecture 2 - Introduction to Polytopes
Lecture 2 - Introduction to Polytopes Optimization and Approximation - ENS M1 Nicolas Bousquet 1 Reminder of Linear Algebra definitions Let x 1,..., x m be points in R n and λ 1,..., λ m be real numbers.
More informationProgramming, numerics and optimization
Programming, numerics and optimization Lecture C-4: Constrained optimization Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428 June
More informationUnconstrained Optimization Principles of Unconstrained Optimization Search Methods
1 Nonlinear Programming Types of Nonlinear Programs (NLP) Convexity and Convex Programs NLP Solutions Unconstrained Optimization Principles of Unconstrained Optimization Search Methods Constrained Optimization
More informationConic Duality. yyye
Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 1 Conic Duality Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/
More informationCME307/MS&E311 Theory Summary
CME307/MS&E311 Theory Summary Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/~yyye http://www.stanford.edu/class/msande311/
More informationIntroduction to Convex Optimization
Introduction to Convex Optimization Lieven Vandenberghe Electrical Engineering Department, UCLA IPAM Graduate Summer School: Computer Vision August 5, 2013 Convex optimization problem minimize f 0 (x)
More informationConvex Optimization and Machine Learning
Convex Optimization and Machine Learning Mengliu Zhao Machine Learning Reading Group School of Computing Science Simon Fraser University March 12, 2014 Mengliu Zhao SFU-MLRG March 12, 2014 1 / 25 Introduction
More informationRevisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization. Author: Martin Jaggi Presenter: Zhongxing Peng
Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization Author: Martin Jaggi Presenter: Zhongxing Peng Outline 1. Theoretical Results 2. Applications Outline 1. Theoretical Results 2. Applications
More informationCharacterizing Improving Directions Unconstrained Optimization
Final Review IE417 In the Beginning... In the beginning, Weierstrass's theorem said that a continuous function achieves a minimum on a compact set. Using this, we showed that for a convex set S and y not
More informationLecture 7: Support Vector Machine
Lecture 7: Support Vector Machine Hien Van Nguyen University of Houston 9/28/2017 Separating hyperplane Red and green dots can be separated by a separating hyperplane Two classes are separable, i.e., each
More informationLinear methods for supervised learning
Linear methods for supervised learning LDA Logistic regression Naïve Bayes PLA Maximum margin hyperplanes Soft-margin hyperplanes Least squares resgression Ridge regression Nonlinear feature maps Sometimes
More informationLecture 2. Topology of Sets in R n. August 27, 2008
Lecture 2 Topology of Sets in R n August 27, 2008 Outline Vectors, Matrices, Norms, Convergence Open and Closed Sets Special Sets: Subspace, Affine Set, Cone, Convex Set Special Convex Sets: Hyperplane,
More informationLinear programming and duality theory
Linear programming and duality theory Complements of Operations Research Giovanni Righini Linear Programming (LP) A linear program is defined by linear constraints, a linear objective function. Its variables
More informationLecture 10: SVM Lecture Overview Support Vector Machines The binary classification problem
Computational Learning Theory Fall Semester, 2012/13 Lecture 10: SVM Lecturer: Yishay Mansour Scribe: Gitit Kehat, Yogev Vaknin and Ezra Levin 1 10.1 Lecture Overview In this lecture we present in detail
More informationLecture 2 September 3
EE 381V: Large Scale Optimization Fall 2012 Lecture 2 September 3 Lecturer: Caramanis & Sanghavi Scribe: Hongbo Si, Qiaoyang Ye 2.1 Overview of the last Lecture The focus of the last lecture was to give
More informationMath 5593 Linear Programming Lecture Notes
Math 5593 Linear Programming Lecture Notes Unit II: Theory & Foundations (Convex Analysis) University of Colorado Denver, Fall 2013 Topics 1 Convex Sets 1 1.1 Basic Properties (Luenberger-Ye Appendix B.1).........................
More informationCME307/MS&E311 Optimization Theory Summary
CME307/MS&E311 Optimization Theory Summary Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/~yyye http://www.stanford.edu/class/msande311/
More informationNonlinear Programming
Nonlinear Programming SECOND EDITION Dimitri P. Bertsekas Massachusetts Institute of Technology WWW site for book Information and Orders http://world.std.com/~athenasc/index.html Athena Scientific, Belmont,
More informationEC 521 MATHEMATICAL METHODS FOR ECONOMICS. Lecture 2: Convex Sets
EC 51 MATHEMATICAL METHODS FOR ECONOMICS Lecture : Convex Sets Murat YILMAZ Boğaziçi University In this section, we focus on convex sets, separating hyperplane theorems and Farkas Lemma. And as an application
More informationA Short SVM (Support Vector Machine) Tutorial
A Short SVM (Support Vector Machine) Tutorial j.p.lewis CGIT Lab / IMSC U. Southern California version 0.zz dec 004 This tutorial assumes you are familiar with linear algebra and equality-constrained optimization/lagrange
More information11 Linear Programming
11 Linear Programming 11.1 Definition and Importance The final topic in this course is Linear Programming. We say that a problem is an instance of linear programming when it can be effectively expressed
More informationLinear Programming. Larry Blume. Cornell University & The Santa Fe Institute & IHS
Linear Programming Larry Blume Cornell University & The Santa Fe Institute & IHS Linear Programs The general linear program is a constrained optimization problem where objectives and constraints are all
More informationConvex Sets. Pontus Giselsson
Convex Sets Pontus Giselsson 1 Today s lecture convex sets convex, affine, conical hulls closure, interior, relative interior, boundary, relative boundary separating and supporting hyperplane theorems
More informationLec 11 Rate-Distortion Optimization (RDO) in Video Coding-I
CS/EE 5590 / ENG 401 Special Topics (17804, 17815, 17803) Lec 11 Rate-Distortion Optimization (RDO) in Video Coding-I Zhu Li Course Web: http://l.web.umkc.edu/lizhu/teaching/2016sp.video-communication/main.html
More informationMathematical and Algorithmic Foundations Linear Programming and Matchings
Adavnced Algorithms Lectures Mathematical and Algorithmic Foundations Linear Programming and Matchings Paul G. Spirakis Department of Computer Science University of Patras and Liverpool Paul G. Spirakis
More informationThis lecture: Convex optimization Convex sets Convex functions Convex optimization problems Why convex optimization? Why so early in the course?
Lec4 Page 1 Lec4p1, ORF363/COS323 This lecture: Convex optimization Convex sets Convex functions Convex optimization problems Why convex optimization? Why so early in the course? Instructor: Amir Ali Ahmadi
More information13. Cones and semidefinite constraints
CS/ECE/ISyE 524 Introduction to Optimization Spring 2017 18 13. Cones and semidefinite constraints ˆ Geometry of cones ˆ Second order cone programs ˆ Example: robust linear program ˆ Semidefinite constraints
More information(1) Given the following system of linear equations, which depends on a parameter a R, 3x y + 5z = 2 4x + y + (a 2 14)z = a + 2
(1 Given the following system of linear equations, which depends on a parameter a R, x + 2y 3z = 4 3x y + 5z = 2 4x + y + (a 2 14z = a + 2 (a Classify the system of equations depending on the values of
More informationAdvanced Operations Research Techniques IE316. Quiz 2 Review. Dr. Ted Ralphs
Advanced Operations Research Techniques IE316 Quiz 2 Review Dr. Ted Ralphs IE316 Quiz 2 Review 1 Reading for The Quiz Material covered in detail in lecture Bertsimas 4.1-4.5, 4.8, 5.1-5.5, 6.1-6.3 Material
More informationSection 5 Convex Optimisation 1. W. Dai (IC) EE4.66 Data Proc. Convex Optimisation page 5-1
Section 5 Convex Optimisation 1 W. Dai (IC) EE4.66 Data Proc. Convex Optimisation 1 2018 page 5-1 Convex Combination Denition 5.1 A convex combination is a linear combination of points where all coecients
More informationContents. I Basics 1. Copyright by SIAM. Unauthorized reproduction of this article is prohibited.
page v Preface xiii I Basics 1 1 Optimization Models 3 1.1 Introduction... 3 1.2 Optimization: An Informal Introduction... 4 1.3 Linear Equations... 7 1.4 Linear Optimization... 10 Exercises... 12 1.5
More informationAMS : Combinatorial Optimization Homework Problems - Week V
AMS 553.766: Combinatorial Optimization Homework Problems - Week V For the following problems, A R m n will be m n matrices, and b R m. An affine subspace is the set of solutions to a a system of linear
More information2. Optimization problems 6
6 2.1 Examples... 7... 8 2.3 Convex sets and functions... 9 2.4 Convex optimization problems... 10 2.1 Examples 7-1 An (NP-) optimization problem P 0 is defined as follows Each instance I P 0 has a feasibility
More informationCS 435, 2018 Lecture 2, Date: 1 March 2018 Instructor: Nisheeth Vishnoi. Convex Programming and Efficiency
CS 435, 2018 Lecture 2, Date: 1 March 2018 Instructor: Nisheeth Vishnoi Convex Programming and Efficiency In this lecture, we formalize convex programming problem, discuss what it means to solve it efficiently
More informationConvex Optimization MLSS 2015
Convex Optimization MLSS 2015 Constantine Caramanis The University of Texas at Austin The Optimization Problem minimize : f (x) subject to : x X. The Optimization Problem minimize : f (x) subject to :
More informationWeek 5. Convex Optimization
Week 5. Convex Optimization Lecturer: Prof. Santosh Vempala Scribe: Xin Wang, Zihao Li Feb. 9 and, 206 Week 5. Convex Optimization. The convex optimization formulation A general optimization problem is
More informationAdvanced Operations Research Techniques IE316. Quiz 1 Review. Dr. Ted Ralphs
Advanced Operations Research Techniques IE316 Quiz 1 Review Dr. Ted Ralphs IE316 Quiz 1 Review 1 Reading for The Quiz Material covered in detail in lecture. 1.1, 1.4, 2.1-2.6, 3.1-3.3, 3.5 Background material
More informationIntroduction to optimization
Introduction to optimization G. Ferrari Trecate Dipartimento di Ingegneria Industriale e dell Informazione Università degli Studi di Pavia Industrial Automation Ferrari Trecate (DIS) Optimization Industrial
More informationminimise f(x) subject to g(x) = b, x X. inf f(x) = inf L(x,
Optimisation Lecture 3 - Easter 2017 Michael Tehranchi Lagrangian necessity Consider the problem Let minimise f(x) subject to g(x) = b, x X. L(x, λ) = f(x) + λ (b g(x)) be the Lagrangian. Notice that for
More informationLocally convex topological vector spaces
Chapter 4 Locally convex topological vector spaces 4.1 Definition by neighbourhoods Let us start this section by briefly recalling some basic properties of convex subsets of a vector space over K (where
More informationLocal and Global Minimum
Local and Global Minimum Stationary Point. From elementary calculus, a single variable function has a stationary point at if the derivative vanishes at, i.e., 0. Graphically, the slope of the function
More informationof Convex Analysis Fundamentals Jean-Baptiste Hiriart-Urruty Claude Lemarechal Springer With 66 Figures
2008 AGI-Information Management Consultants May be used for personal purporses only or by libraries associated to dandelon.com network. Jean-Baptiste Hiriart-Urruty Claude Lemarechal Fundamentals of Convex
More informationConvexity and Optimization
Convexity and Optimization Richard Lusby DTU Management Engineering Class Exercises From Last Time 2 DTU Management Engineering 42111: Static and Dynamic Optimization (3) 18/09/2017 Today s Material Extrema
More informationPrograms. Introduction
16 Interior Point I: Linear Programs Lab Objective: For decades after its invention, the Simplex algorithm was the only competitive method for linear programming. The past 30 years, however, have seen
More informationConvexity and Optimization
Convexity and Optimization Richard Lusby Department of Management Engineering Technical University of Denmark Today s Material Extrema Convex Function Convex Sets Other Convexity Concepts Unconstrained
More informationConvex Optimization. Chapter 1 - chapter 2.2
Convex Optimization Chapter 1 - chapter 2.2 Introduction In optimization literatures, one will frequently encounter terms like linear programming, convex set convex cone, convex hull, semidefinite cone
More informationCOMS 4771 Support Vector Machines. Nakul Verma
COMS 4771 Support Vector Machines Nakul Verma Last time Decision boundaries for classification Linear decision boundary (linear classification) The Perceptron algorithm Mistake bound for the perceptron
More informationIntroduction to Optimization
Introduction to Optimization Constrained Optimization Marc Toussaint U Stuttgart Constrained Optimization General constrained optimization problem: Let R n, f : R n R, g : R n R m, h : R n R l find min
More informationOptimization III: Constrained Optimization
Optimization III: Constrained Optimization CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Doug James (and Justin Solomon) CS 205A: Mathematical Methods Optimization III: Constrained Optimization
More information3. The Simplex algorithmn The Simplex algorithmn 3.1 Forms of linear programs
11 3.1 Forms of linear programs... 12 3.2 Basic feasible solutions... 13 3.3 The geometry of linear programs... 14 3.4 Local search among basic feasible solutions... 15 3.5 Organization in tableaus...
More informationmaximize c, x subject to Ax b,
Lecture 8 Linear programming is about problems of the form maximize c, x subject to Ax b, where A R m n, x R n, c R n, and b R m, and the inequality sign means inequality in each row. The feasible set
More informationLecture 2 Optimization with equality constraints
Lecture 2 Optimization with equality constraints Constrained optimization The idea of constrained optimisation is that the choice of one variable often affects the amount of another variable that can be
More information