IE 521 Convex Optimization

Transcription

1 Lecture 4: 5th February 2019

2 Outline 1 / 23

3 Which function is different from others? Figure: Functions 2 / 23

4 Definition of Convex Function Definition. A function f (x) : R n R is convex if (i) dom(f ) := {x R n : f (x) < } is a convex set; (ii) x, y dom(f ) and λ [0, 1], f (λx + (1 λ)y) λf (x) + (1 λ)f (y). Figure: Convex function Geometrically, the line segment between (x, f (x)), (y, f (y)) sits above the graph of f. 3 / 23

5 Brotherhood Definition. (Strict and Strong Convex) A function is called strictly convex if (ii) holds with strict sign, i.e., λ (0, 1), f (λx + (1 λ)y) < λf (x) + (1 λ)f (y). A function is called α-strongly convex if f (x) α 2 x 2 2 is convex, i.e., λ [0, 1], f (λx+(1 λ)y) λf (x)+(1 λ)f (y) α 2 λ(1 λ) x y 2 2. Note that strongly convex = strictly convex = convex Definition. (Concave/Strictly Concave/Strongly Concave) A function is called concave if f (x) is convex. Similarly for strictly concavity and strongly concavity. 4 / 23

6 of Example 1. Simple univariate functions: Even powers: x p, p is even Exponential: e ax, a R Negative logarithmic: log x Absolute value: x Negative entropy: x log(x) Example 2. Affine functions: f (x) = a T x + b both convex & concave, but not strctly convex/concave 5 / 23

7 of Example 3. Norms: l p -norm on R n : x p := ( n x i p ) 1/p (p 1) i=1 Q-norm on R n : x Q := x T Qx (Q 0) Frobenius norm on R m n : A F = ( m i=1 n Spectral and nuclear norms on R m n : j=1 A ij 2 ) 1/2 A = max σ i(a) i=1,...,min{m.n} A = σ i (A) i=1,...,min{m.n} 6 / 23

8 of Example 4. Some quadratic functions: f (x) = 1 2 x T Qx + b T x + c convex if and only if Q 0 is positive semi-definite strictly convex if and only if Q 0 is positive definite 7 / 23

9 of Example 5. Indicator function: { 0, x C I C (x) =, x C I C (x) is convex if the set C is a convex set. (why?) Example 6. Supporting function: I C (x) = sup x T y y C IC (x) is always convex for any set C. (why?) 8 / 23

10 of Example 7. More examples Piecewise linear functions: max(a T 1 x + b 1,..., a T k x + b k) Log of exponential sums: log( k i=1 eat i x+b i ) Negative log of determinant: log(det(x )) Q. How to show convexity of these functions? 9 / 23

11 Convexity-Preserving Operators Taking conic combination; Taking affine composition; Taking pointwise maximum and supremum; Taking convex monotone composition; Taking partial minimization; Taking the perspective transformation; 10 / 23

12 Taking Conic Combination Proposition. If f i (x), i I are convex functions and α i 0, i I, then so is g(x) = i I α i f i (x). Remark. (Extension to integrals) If f (x, ω) is convex in x and α(ω) 0, ω Ω, then so is g(x) = α(ω)f (x, ω)dω Ω Example 8. If η is a well-defined random variable on Ω, and f (x, η(ω)) is convex, ω Ω, then E η [f (x, η)] is convex. 11 / 23

13 Taking Affine Composition Proposition. If f (x) is convex and A(y) : y Ay + b is an affine mapping, then so is g(y) := f (Ay + b). Example 9. The following functions are convex: f (x) = Ax b 2 2, f (x) = i eat i x b i, f (x) = n i=1 log(at i x b i ). 12 / 23

14 Taking Pointwise Maximum and Supremum Proposition. If f i (x), i I are convex, then so is g(x) := max i I f i (x). Remark. (Extension to pointwise supremum) If f (x, ω) is convex in x, for ω Ω, then so is Example 10. g(x) := sup ω Ω f (x, ω). The following functions are convex: g(x) = max(a T 1 x + b 1,..., a T k x + b k) I C (x) = sup y C x T y d max (x, C) = max y C y x 2 λ max (X ) = max y 2 =1 y T Xy 13 / 23

15 Taking Convex Monotone Composition Proposition. If f i (x), i = 1,..., m are convex and F (y 1,..., y m ) is convex and component-wise non-decreasing, then so is g(x) = F (f 1 (x),..., f m (x)). Remark. Taking pointwise maximum is a special case of the above rule by setting F (y 1,..., y m ) = max(y 1,..., y m ), Example 11. F (f 1 (x),..., f m (x)) = e f (x) is convex if f is convex max f i(x). i=1,...,m log f (x) is convex if f is concave log( k i=1 ef i ) is convex if f i are convex. 14 / 23

16 Taking Convex Monotone Composition Proposition. If f i (x), i = 1,..., m are convex and F (y 1,..., y m ) is convex and component-wise non-decreasing, then so is g(x) = F (f 1 (x),..., f m (x)). Proof. By convexity of f i, we have f i (λx + (1 λ)y) λf i (x) + (1 λ)f i (y), i, λ [0, 1]. Hence, we have for any x, y dom(g), λ [0, 1], g(λx + (1 λ)y) = F (f 1 (λx + (1 λ)y),..., f m (λx + (1 λ)y)) F (λf 1 (x) + (1 λ)f 1 (y),..., λf m (x) + (1 λ)f m (y)) λf (f 1 (x),..., f m (x)) + (1 λ)f (f 1 (x),..., f m (x)) = λg(x) + (1 λ)g(y) 15 / 23

17 Taking Partial Minimization Proposition. If f (x, y) is convex in (x, y) R n and Y is a convex set, then so is g(x) = inf f (x, y). y Y Example 12. The following are convex: d(x, C) = min y C x y 2, where C is convex; g(x) = inf y {h(y) Ay = x}, where h is convex. 16 / 23

18 Taking Partial Minimization Proposition. If f (x, y) is convex in (x, y) R n and Y is a convex set, then so is g(x) = inf f (x, y). y Y Proof. dom(g) = {x : (x, y) dom(f ) and y C} is a projection of dom(f ), hence is convex. Given any x 1, x 2, by definition, for any ɛ > 0, y 1, y 2 Y s.t. f (x 1, y 1 ) g(x 1 ) + ɛ/2, f (x 2, y 2 ) g(x 2 ) + ɛ/2 By convexity of f (x, y), this implies f (λx 1 +(1 λ)x 2, λy 1 +(1 λ)y 2 ) λg(x 1 )+(1 λ)g(x 2 )+ɛ. ɛ > 0, g(λx 1 + (1 λ)x 2 ) λg(x 1 ) + (1 λ)g(x 2 ) + ɛ. Letting ɛ 0 leads to the convexity of g. 17 / 23

19 Taking Perspective Function Proposition. If f is convex, then so is the perspective function g(x, t) = tf (x/t), where dom(g) = {(x, t) : x/t dom(f ), t > 0}. Example 13. g(x, t) = x T x/t is convex on R n R ++ ; g(x, t) = t log t t log x is convex on R 2 ++; D KL (x, y) = i [x i log( x i y i ) x i + y i ] on R n ++ R n / 23

20 Taking Perspective Function Proposition. If f is convex, then so is the perspective function g(x, t) = tf (x/t), where dom(g) = {(x, t) : x/t dom(f ), t > 0}. Proof. dom(g) is the inverse image of dom(f ) under the perspective function P(x, t) := x/t for t > 0. So it is convex. (why?) Consider (x, t), (y, s) dom(g), and λ (0, 1). g(λx + (1 λ)y, λt + (1 λ)s) ( ) λx + (1 λ)y) =(λt + (1 λ)s)f λt + (1 λ)s λtf (x/t) + (1 λ)sf (y/s) =λg(x, t) + (1 λ)g(y, s). 19 / 23

21 Quick Check Which of the following function is convex: A. f (x 1, x 2 ) = x 1 x 2 on R 2 + B. f (x 1, x 2 ) = min(x 1, x 2 ) on R 2 C. f (x 1, x 2 ) = x 1 x 2 on R 2 ++ D. f (x 1, x 2 ) = x2 1 x 2 on R R / 23

22 Quick Check Which of the following function is not convex: A. f (x) = x B. f (x) = x 2 C. f (x) = x 3 D. f (x) = log( x ) 21 / 23

23 Application: Inventory Model Consider a single period inventory system. Let x denotes the inventory level and d denote the random demand in that period following distribution D. Suppose that the vendor suffer either a holding cost of h dollars per unit for excess inventory or a penalty cost of p dollars per unit for lost demands. What s the expected total cost f (x) as a function of x? Is it a convex function? The cost function f (x) = E d D [h max(x d, 0) + p max(d x, 0)] is a convex function. 22 / 23

24 References Boyd & Vandenberghe, Chapter / 23