Affine function. suppose f : R n R m is affine (f(x) =Ax + b with A R m n, b R m ) the image of a convex set under f is convex

Transcription

1 Affine function suppose f : R n R m is affine (f(x) =Ax + b with A R m n, b R m ) the image of a convex set under f is convex S R n convex = f(s) ={f(x) x S} convex the inverse image f 1 (C) of a convex set under f is convex C R m convex = f 1 (C) ={x R n f(x) C} convex examples scaling, translation, projection solution set of linear matrix inequality {x x 1 A x m A m B} (with A i,b S p ) hyperbolic cone {x x T Px (c T x) 2,c T x 0} (with P S n +) Convex sets 2 13

2 Perspective and linear-fractional function perspective function P : R n+1 R n : P (x, t) =x/t, dom P = {(x, t) t>0} images and inverse images of convex sets under perspective are convex linear-fractional function f : R n R m : f(x) = Ax + b c T x + d, dom f = {x ct x + d>0} images and inverse images of convex sets under linear-fractional functions are convex Convex sets 2 14

3 Generalized inequalities a convex cone K R n is a proper cone if K is closed (contains its boundary) K is solid (has nonempty interior) K is pointed (contains no line) examples nonnegative orthant K = R n + = {x R n x i 0,i=1,...,n} positive semidefinite cone K = S n + nonnegative polynomials on [0, 1]: K = {x R n x 1 + x 2 t + x 3 t x n t n 1 0 for t [0, 1]} Convex sets 2 16

4 generalized inequality defined by a proper cone K: x K y y x K, x K y y x int K examples componentwise inequality (K = R n +) x R n + y x i y i, i =1,...,n matrix inequality (K = S n +) X S n + Y Y X positive semidefinite these two types are so common that we drop the subscript in K properties: many properties of K are similar to on R, e.g., x K y, u K v = x + u K y + v Convex sets 2 17

5 Minimum and minimal elements K is not in general a linear ordering: wecanhavex K y and y K x x S is the minimum element of S with respect to K if y S = x K y x S is a minimal element of S with respect to K if y S, y K x = y = x example (K = R 2 +) x 1 is the minimum element of S 1 x 2 is a minimal element of S 2 x1 S 2 S 1 x 2 Convex sets 2 18

6 Separating hyperplane theorem if C and D are disjoint convex sets, then there exists a 0, b such that a T x b for x C, a T x b for x D a T x b a T x b D C a the hyperplane {x a T x = b} separates C and D strict separation requires additional assumptions (e.g., C is closed, D is a singleton) Convex sets 2 19

7 Supporting hyperplane theorem supporting hyperplane to set C at boundary point x 0 : {x a T x = a T x 0 } where a 0and a T x a T x 0 for all x C C x 0 a supporting hyperplane theorem: if C is convex, then there exists a supporting hyperplane at every boundary point of C Convex sets 2 20

8 Vector optimization general vector optimization problem minimize (w.r.t. K) f 0 (x) subject to f i (x) 0, i =1,...,m h i (x) 0, i =1,...,p vector objective f 0 : R n R q, minimized w.r.t. proper cone K R q convex vector optimization problem minimize (w.r.t. K) f 0 (x) subject to f i (x) 0, i =1,...,m Ax = b with f 0 K-convex, f 1,...,f m convex Convex optimization problems 4 40

9 Optimal and Pareto optimal points set of achievable objective values O = {f 0 (x) x feasible} feasible x is optimal if f 0 (x) is the minimum value of O feasible x is Pareto optimal if f 0 (x) is a minimal value of O O f 0 (x po ) O f 0 (x ) x is optimal x po is Pareto optimal Convex optimization problems 4 41

10 Multicriterion optimization vector optimization problem with K = R q + f 0 (x) =(F 1 (x),...,f q (x)) q different objectives F i ; roughly speaking we want all F i s to be small feasible x is optimal if y feasible = f 0 (x ) f 0 (y) if there exists an optimal point, the objectives are noncompeting feasible x po is Pareto optimal if y feasible, f 0 (y) f 0 (x po ) = f 0 (x po )=f 0 (y) if there are multiple Pareto optimal values, there is a trade-off between the objectives Convex optimization problems 4 42

11 Regularized least-squares minimize (w.r.t. R 2 +) ( Ax b 2 2, x 2 2) 25 F2(x) = x O F 1 (x) = Ax b 2 2 example for A R ; heavy line is formed by Pareto optimal points Convex optimization problems 4 43

12 Risk return trade-off in portfolio optimization minimize (w.r.t. R 2 +) ( p T x, x T Σx) subject to 1 T x =1, x 0 x R n is investment portfolio; x i is fraction invested in asset i p R n is vector of relative asset price changes; modeled as a random variable with mean p, covariance Σ p T x = E r is expected return; x T Σx = var r is return variance example 15% 1 mean return 10% 5% allocation x 0.5 x(4) x(3) x(2) x(1) 0% 0% 10% 20% standard deviation of return 0 0% 10% 20% standard deviation of return Convex optimization problems 4 44

13 Dual cones and generalized inequalities dual cone of a cone K: K = {y y T x 0 for all x K} examples K = R n +: K = R n + K = S n +: K = S n + K = {(x, t) x 2 t}: K = {(x, t) x 2 t} K = {(x, t) x 1 t}: K = {(x, t) x t} first three examples are self-dual cones dual cones of proper cones are proper, hence define generalized inequalities: y K 0 y T x 0 for all x K 0 Convex sets 2 21

14 Minimum and minimal elements via dual inequalities minimum element w.r.t. K x is minimum element of S iff for all λ K 0, x is the unique minimizer of λ T z over S minimal element w.r.t. K x S if x minimizes λ T z over S for some λ K 0, thenx is minimal λ 1 x 1 S x 2 λ 2 if x is a minimal element of a convex set S, then there exists a nonzero λ K 0 such that x minimizes λ T z over S Convex sets 2 22

15 optimal production frontier different production methods use different amounts of resources x R n production set P : resource vectors x for all possible production methods efficient (Pareto optimal) methods correspond to resource vectors x that are minimal w.r.t. R n + example (n =2) fuel x 1, x 2, x 3 are efficient; x 4, x 5 are not P x 1 x 2 x5 x 4 λ x 3 labor Convex sets 2 23

16 3. Convex functions Convex Optimization Boyd & Vandenberghe basic properties and examples operations that preserve convexity the conjugate function quasiconvex functions log-concave and log-convex functions convexity with respect to generalized inequalities 3 1

17 Definition f : R n R is convex if dom f is a convex set and f(θx +(1 θ)y) θf(x)+(1 θ)f(y) for all x, y dom f, 0 θ 1 (x, f(x)) (y, f(y)) f is concave if f is convex f is strictly convex if dom f is convex and f(θx +(1 θ)y) <θf(x)+(1 θ)f(y) for x, y dom f, x y, 0 <θ<1 Convex functions 3 2

18 Examples on R convex: affine: ax + b on R, foranya, b R exponential: e ax,foranya R powers: x α on R ++,forα 1 or α 0 powers of absolute value: x p on R, forp 1 negative entropy: x log x on R ++ concave: affine: ax + b on R, foranya, b R powers: x α on R ++,for0 α 1 logarithm: log x on R ++ Convex functions 3 3

19 Examples on R n and R m n affine functions are convex and concave; all norms are convex examples on R n affine function f(x) =a T x + b norms: x p =( n i=1 x i p ) 1/p for p 1; x =max k x k examples on R m n (m n matrices) affine function f(x) =tr(a T X)+b = m i=1 n A ij X ij + b j=1 spectral (maximum singular value) norm f(x) = X 2 = σ max (X) =(λ max (X T X)) 1/2 Convex functions 3 4

20 Restriction of a convex function to a line f : R n R is convex if and only if the function g : R R, g(t) =f(x + tv), dom g = {t x + tv dom f} is convex (in t) foranyx dom f, v R n can check convexity of f by checking convexity of functions of one variable example. f : S n R with f(x) = log det X, dom f = S n ++ g(t) = log det(x + tv ) = logdetx + log det(i + tx 1/2 VX 1/2 ) = n log det X + log(1 + tλ i ) where λ i are the eigenvalues of X 1/2 VX 1/2 g is concave in t (for any choice of X 0, V ); hence f is concave i=1 Convex functions 3 5

21 Extended-value extension extended-value extension f of f is f(x) =f(x), x dom f, f(x) =, x dom f often simplifies notation; for example, the condition 0 θ 1 = f(θx +(1 θ)y) θ f(x)+(1 θ) f(y) (as an inequality in R { }), means the same as the two conditions dom f is convex for x, y dom f, 0 θ 1 = f(θx +(1 θ)y) θf(x)+(1 θ)f(y) Convex functions 3 6

22 First-order condition f is differentiable if dom f is open and the gradient f(x) = exists at each x dom f ( f(x) x 1, f(x),..., f(x) ) x 2 x n 1st-order condition: differentiable f with convex domain is convex iff f(y) f(x)+ f(x) T (y x) for all x, y dom f f(y) f(x) + f(x) T (y x) (x, f(x)) first-order approximation of f is global underestimator Convex functions 3 7

23 Second-order conditions f is twice differentiable if dom f is open and the Hessian 2 f(x) S n, 2 f(x) ij = 2 f(x) x i x j, i,j =1,...,n, exists at each x dom f 2nd-order conditions: for twice differentiable f with convex domain f is convex if and only if 2 f(x) 0 for all x dom f if 2 f(x) 0 for all x dom f, thenf is strictly convex Convex functions 3 8

24 Examples quadratic function: f(x) =(1/2)x T Px+ q T x + r (with P S n ) f(x) =Px+ q, 2 f(x) =P convex if P 0 least-squares objective: f(x) = Ax b 2 2 f(x) =2A T (Ax b), 2 f(x) =2A T A convex (for any A) quadratic-over-linear: f(x, y) =x 2 /y 2 2 f(x, y) = 2 y 3 [ y x ][ y x ] T 0 f(x, y) convex for y>0 1 y 0 2 x 0 Convex functions 3 9

25 log-sum-exp: f(x) =log n k=1 exp x k is convex 2 f(x) = 1 1 T z diag(z) 1 (1 T z) 2zzT (z k =expx k ) to show 2 f(x) 0, we must verify that v T 2 f(x)v 0 for all v: v T 2 f(x)v = ( k z kv 2 k )( k z k) ( k v kz k ) 2 ( k z k) 2 0 since ( k v kz k ) 2 ( k z kv 2 k )( k z k) (from Cauchy-Schwarz inequality) geometric mean: f(x) =( n k=1 x k) 1/n on R n ++ is concave (similar proof as for log-sum-exp) Convex functions 3 10

26 Epigraph and sublevel set α-sublevel set of f : R n R: C α = {x dom f f(x) α} sublevel sets of convex functions are convex (converse is false) epigraph of f : R n R: epi f = {(x, t) R n+1 x dom f, f(x) t} epi f f f is convex if and only if epi f is a convex set Convex functions 3 11

27 Jensen s inequality basic inequality: if f is convex, then for 0 θ 1, f(θx +(1 θ)y) θf(x)+(1 θ)f(y) extension: if f is convex, then f(e z) E f(z) for any random variable z basic inequality is special case with discrete distribution prob(z = x) =θ, prob(z = y) =1 θ Convex functions 3 12

28 Operations that preserve convexity practical methods for establishing convexity of a function 1. verify definition (often simplified by restricting to a line) 2. for twice differentiable functions, show 2 f(x) 0 3. show that f is obtained from simple convex functions by operations that preserve convexity nonnegative weighted sum composition with affine function pointwise maximum and supremum composition minimization perspective Convex functions 3 13

29 Positive weighted sum & composition with affine function nonnegative multiple: αf is convex if f is convex, α 0 sum: f 1 + f 2 convex if f 1,f 2 convex (extends to infinite sums, integrals) composition with affine function: f(ax + b) is convex if f is convex examples log barrier for linear inequalities f(x) = m log(b i a T i x), i=1 dom f = {x a T i x<b i,i=1,...,m} (any) norm of affine function: f(x) = Ax + b Convex functions 3 14

30 Pointwise maximum if f 1,...,f m are convex, then f(x) =max{f 1 (x),...,f m (x)} is convex examples piecewise-linear function: f(x) =max i=1,...,m (a T i x + b i) is convex sum of r largest components of x R n : f(x) =x [1] + x [2] + + x [r] is convex (x [i] is ith largest component of x) proof: f(x) =max{x i1 + x i2 + + x ir 1 i 1 <i 2 < <i r n} Convex functions 3 15