Convexity Theory and Gradient Methods

Transcription

1 Convexity Theory and Gradient Methods Angelia Nedić ISE Department and Coordinated Science Laboratory University of Illinois at Urbana-Champaign

2 Outline Convex Functions Optimality Principle Projection Theorem Gradient Methods 1

3 IMA, Minnesota June 1 6, 2014 Convex Set Line segment [x1, x2] Rn is the set of all points xα = αx1 + (1 α)x2 for α [0, 1]. A set C is convex when for all x1, x2 C, the segment [x1, x2] is contained in the set C. 2

4 Convex Function Let f be a function from R n to R, f : R n R Informally: f is convex when for every segment [x 1, x 2 ], as x α = αx 1 +(1 α)x 2 varies over the line segment [x 1, x 2 ], the points (x α, f(x α )) lie below the segment connecting (x 1, f(x 1 )) and (x 2, f(x 2 )) The domain of f is a set in R n defined by dom(f) = {x R n f(x) is well defined (finite)} Def. A function f is convex if (1) Its domain dom(f) is a convex set in R n and (2) For all x 1, x 2 dom(f) and α [0, 1] f(αx 1 + (1 α)x 2 ) αf(x 1 ) + (1 α)f(x 2 ) The function is strictly convex if the inequality is strict whenever x 1 x 2 3

5 Examples on R Convex: Affine: ax + b over R for any a, b R Exponential: e ax over R for any a R Power: x p over (0, + ) for p 1 or p 0 Powers of absolute value: x p over R for p 1 Negative entropy: x ln x over (0, + ) Concave: Affine: ax + b over R for any a, b R Powers: x p over (0, + ) for 0 p 1 Logarithm: ln x over (0, + ) 4

6 Examples: Affine Functions and Norms Affine functions are both convex and concave Norms are convex Examples on R n Affine function f(x) = a x + b with a R n and b R Euclidean, l 1, and l norms General l p norms x p = ( n i=1 x i p ) 1/p for p 1 5

7 Examples on R m n The space R m n is the space of m n matrices Affine function f(x) = tr(a X) + b = m n i=1 j=1 a ij x ij + b Spectral (maximum singular value) norm f(x) = X 2 = σ max (X) = λ max (X X) where λ max (A) denotes the maximum eigenvalue of a matrix A 6

8 Verifying Convexity of a Function We can verify that a given function f is convex by Using the definition Applying some special criteria Second-order conditions First-order conditions Reduction to a scalar function Showing that f is obtained through operations preserving convexity 7

9 Second-Order Conditions Let f be twice differentiable and let dom(f) = R n required that dom(f) is open] [in general, it is The Hessian 2 f(x) is a symmetric n n matrix whose entries are the second-order partial derivatives of f at x: [ 2 f(x) ] ij = 2 f(x) x i x j for i, j = 1,..., n 2nd-order conditions: For a twice differentiable f with convex domain f is convex if and only if 2 f(x) 0 for all x dom(f) f is strictly convex if 2 f(x) 0 for all x dom(f) 8

10 Examples Quadratic function: f(x) = (1/2)x P x + q x + r with a symmetric n n matrix P f(x) = P x + q, 2 f(x) = P Convex for P 0 Least-squares objective: f(x) = Ax b 2 with an m n matrix A Convex for any A f(x) = 2A (Ax b), Quadratic-over-linear: f(x, y) = x 2 /y 2 f(x) = 2A A Convex for y > 0 2 f(x, y) = 2 y 3 [ y x ] [ y x ] T 0 9

11 Verifying Convexity of a Function We can verify that a given function f is convex by Using the definition Applying some special criteria Second-order conditions First-order conditions Reduction to a scalar function Showing that f is obtained through operations preserving convexity 10

12 First-Order Condition f is differentiable if dom(f) is open and the gradient f(x) = ( f(x) x 1, f(x),..., f(x) ) x 2 x n exists at each x domf 1st-order condition: differentiable f is convex if and only if its domain is convex and f(x) + f(x), z x f(z) for all x, z dom(f) A first order approximation is a global underestimate of f Very important property used in algorithm designs and performance analysis 11

13 Restriction of a convex function to a line f is convex if and only if domf is convex and the function g : R R, g(t) = f(x + tv), dom(g) = {t x + tv dom(f)} is convex (in t) for any x domf, v R n Checking convexity of multivariable functions can be done by checking convexity of functions of one variable Example f : S n R with f(x) = ln det X, dom(f) = S n ++ g(t) = ln det(x + tv ) = ln det X ln det(i + tx 1/2 V X 1/2 ) = ln det X n i=1 ln(1 + tλ i ) where λ i are the eigenvalues of X 1/2 V X 1/2 g is convex in t (for any choice of V and any X 0); hence f is convex 12

14 Operations Preserving Convexity Positive Scaling Sum Composition with Affine Mapping Special Compositions Point-wise Maximum Point-wise Supremum Partial Minimization 13

15 Scaling, Sum, & Composition with Affine Function Positive multiple For a convex f and λ > 0, the function λf is convex Sum: For convex f 1 and f 2, the sum f 1 + f 2 is convex (extends to infinite sums, integrals) Composition with affine function: For a convex f and affine g [i.e., g(x) = Ax + b], the composition f g is convex, where (f g)(x) = f(ax + b) Examples Log-barrier for linear inequalities f(x) = m i=1 ln(b i a ix), domf = {x a ix < b i, i = 1,..., m} (Any) Norm of affine function: f(x) = Ax + b 14

16 Composition with Scalar Functions Composition of g : R n R and h : R R with dom(g) = R n dom(h) = R: f(x) = h(g(x)) f is convex if and (1) g is convex, h is nondecreasing and convex (2) g is concave, h is nonincreasing and convex Examples e g(x) is convex if g is convex 1 g(x) is convex if g is concave and positive 15

17 Composition with Vector Functions Composition of g : R n R p and h : R p R with dom(g) = R n and dom(h) = R p : f(x) = h(g(x)) = h(g 1 (x), g 2 (x),..., g p (x)) f is convex if (1) each g i is convex, h is convex and nondecreasing in each argument (2) each g i is concave, h is convex and nonincreasing in each argument Example m i=1 e g i(x) is convex if g i are convex 16

18 Pointwise maximum For convex functions f 1,..., f m, the pointwise-max function F (x) = max {f 1 (x),..., f m (x)} is convex (What is domain of F?) Examples Piecewise-linear function: f(x) = max i=1,...,m (a T i x + b i ) is convex Sum of r largest components of a vector x R n : f(x) = x [1] + x [2] + + x [r] is convex (x [i] is i-th largest component of x) f(x) = max (i 1,...,i r ) I r {x i1 + x i2 + + x ir } I r = {(i 1,..., i r ) i 1 <... < i r, i j {1,..., m}, j = 1,..., n} Pointwise supremum - later 17

19 Extended-Value Functions: Epigraph A function f is an extended-value function if f : R n R {, + } Example: consider f(x) = inf y 0 xy for x R Def. The epigraph of a function f over R n is the following set in R n+1 : epif = {(x, w) R n+1 x R n, f(x) w} Theorem: [Convex Function - Convex Epigraph ] A function f is convex if and only if its epigraph epif is a convex set in R n+1 This allows us to use the convexity of the epigraph as the definition of convexity (often done). These are equivalent in view of the theorem. For an f with domain domf, we associate an extended-value function f defined by f(x) if x domf f(x) = + otherwise domf is the projection of epif on R n ; convexity of f by letting w = f(x) 18

20 Pointwise Supremum Let A R p and f : R n R p R. Let f(x, z) be convex in x for each z A. Then, the supremum function over the set A is convex: Examples g(x) = sup z A f(x, z) Set support function is convex for a set C R n, S C : R n R, S C (x) = sup z C z x Set farthest-distance function is convex for a set C R n, f : R n R, f(x) = sup z C x z Maximum eigenvalue function of a symmetric matrix is convex λ max : S n R, λ max (X) = sup z =1 z Xz 19

21 Minimization Let C R p be a nonempty convex set Let f : R n R p R be a convex function [in (x, z) R n R p ]. Then g(x) = inf f(x, z) z C is convex Example Distance to a set: for a nonempty convex C R n, dist(x, C) = min z C x z is convex Proof for the case when g is finite: Let x 1, x 2 R n and α (0, 1) be arbitrary. Let ɛ > 0 be arbitrarily small. Then, there exist z 1 and z 2 such that (x 1, z 1 ), (x 2, z 2 ) C with f(x 1, z 1 ) g(x 1 ) + ɛ and f(x 2, z 2 ) g(x 2 ) + ɛ. Consider f(αx 1 + (1 α)x 2, αz 1 + (1 α)z 2 ) and use the convexity of f and C. 20

22 Level Sets and Convex Functions Def. Given a scalar c R and a function f, a (lower) level set of f associated with c is given by L c (f) = {x R n f(x) c} Examples: f(x) = x 2 for x R n, f(x 1, x 2 ) = e x 1 Every level set of a convex function is convex Converse is false: Consider f(x) = e x for x R Recall definition of a concave function: g is concave when g is convex Every (upper) level set of a concave function is convex 21

23 Optimality Principle for Differentiable f In the following, unless otherwise stated, the function f is assumed to be differentiable and convex, with domf = R n Let f be a differentiable convex function and let C be a nonempty closed convex set Theorem A vector x is optimal if and only if x C and f(x ), z x 0 for all z C 22

24 Unconstrained Optimization minimize subject to f(x) x R n A vector x is optimal if and only if f(x ) = 0 Follows from f(x ), z x 0 for all z R n 23

25 Linear Equality Constrained Problem minimize subject to f(x) Ax = b with A R m n and b R m When does an optimal solution exist? A vector x is optimal if and only if f(x )y = 0 for all y N A Using N A = ImA, we have that x is optimal if and only if there exists λ R m such f(x ) + A λ = 0 This is Primal-Dual (Lagrange Multiplier) Optimality Condition 24

26 Minimization over the Nonnegative Orthant minimize subject to f(x) x 0, x R n When does an optimal solution exist? A vector x is optimal if and only if f(x ), x = 0 This known as Complementarity Condition in Lagrangian duality. Again it follows from optimality principle: f(x ), z x 0 for all z 0 25

27 Projection Theorem Let C R n be a nonempty closed convex set and ˆx R n be arbitrary (a) There is a unique solution to the following problem minimize z ˆx 2 subject to z C (b) A vector z C is the solution if and only if z ˆx, z z 0 for all z C The solution is said to be the projection of ˆx on C in the Euclidean norm, denoted by Π C [ˆx] 26

28 Proof of the Projection Theorem (a) The objective function is strongly convex since its Hessian is equal to 2I. Therefore, the optimal solution exists and it is unique. (b) By the first-order optimality condition, we have z C is the solution if and only if f(z ), z z 0 for all z C Since f(z) = 2(z ˆx), the result follows. Function f has a positive definite Hessian 2 f(x) everywhere 27

29 Projection Properties Th. Let C R n be a nonempty closed convex set (a) The projection mapping Π C : R n C is non-expansive, i.e., Π C [x] Π C [y] x y for all x, y R n (b) The set distance function d : R n R given by dist(x, C) = Π C [x] x is convex 28

30 Proof of Projection Property (a) (a) The relation evidently holds for any x and y with Π C [x] = Π C [y]. Consider now arbitrary x, y R n with Π C [x] Π C [y]. By Projection Theorem (b), we have Π C [x] x, z Π C [x] 0 for all z C (1) Π C [y] y, z Π C [y] 0 for all z C (2) Using z = Π C [y] in Eq. (1) and z = Π C [x] in Eq. (2), and adding the resulting inequalities, we obtain Π C [y] y + x Π C [x], Π C [x] Π C [y] 0 implying that y x, Π C [x] Π C [y] Π C [x] Π C [y] 2 Since Π C [x] Π C [y], it follows that y x Π C [x] Π C [y] 29

31 Proof of Projection Property (b) (b) Note that the distance function is equivalently given by dist(x, C) = min z C x z for all x Rn The function h(x, z) = x z is convex in (x, z) over R n R n. The set C is convex, hence dist(x, C) is convex (see the lecture on operations preserving convexity of functions) 30

32 Fejer Para-contraction Property Th. Let C R n be a nonempty closed convex set The projection mapping is para-contraction with respect to the set C, i.e., Π C [x] y 2 y x 2 Π C [x] x 2 for all x R n, y C 31

33 Optimality Property: Fixed-Point Interpretation Let f be a differentiable convex function and let C be a nonempty closed convex set Theorem A vector x is optimal if and only if Proof By optimality principle x = Π C [x α f(x )] for all α > 0 f(x ), z x 0 for all z C α f(x ), z x 0 for all z C and any α > 0 x (x α f(x )), z x 0 for all z C and any α > 0 By Projection Theorem a vector z = Π C [ˆx] if and only if Hence, the relation z ˆx, z z 0 for all z C x (x α f(x )), z x }{{} 0 for all z C and any α > 0 ˆx is equivalent to x = Π C [x α f(x )] for all α > 0 32

34 Gradient Methods: Solve Fixed-Point Formulation Recursion x = Π C [x α f(x )] for all α > 0 x(t + 1) = Π C [x(t) α t f(x(t))] If T : R n R n is a map that also has x C as a fixed point, we can write another fixed point relation (whose fixed points are optimal for f over C) x = βt (x ) + (1 β)π C [x α f(x )] for all α > 0, β [0, 1] Since the solutions to min x C f(x) are typically not known in advance: Method T = I x(t + 1) = β t x(t) + (1 β t )Π C [x(t) α t f(x(t))] 33

35 Gradient Method: Basic Relation Recursion/Algorithm x(t + 1) = Π C [x(t) α t f(x(t))] Using para-contractive property of the projection Π C [x] y 2 y x 2 Π C [x] x 2 for all x R n, y C we obtain for all y C and all t: x(t + 1) y 2 x(t) α t f(x(t)) y 2 x(t + 1) x(t) 2 Open the quadratic term x(t+1) y 2 x(t) y 2 2α t f(x(t)), x(t) y +α 2 t f(x(t)) 2 x(t+1) x(t) 2 By convexity of the function so we have f(x(t)), x(t) y f(x(t)) f(y) x(t + 1) y 2 x(t) y 2 2α t (f(x(t)) f(y)) + α 2 t f(x(t)) 2 x(t + 1) x(t) 2 34

36 Gradient Methods - Another Way This is equivalent to x(t + 1) = argmin y C which is equivalent to x(t + 1) = Π C [x(t) α t f(x(t))] {α t f(x(t), y x(t) + 12 y x(t) 2 } { x(t + 1) = argmin f(x(t), y x(t) + 1 y x(t) 2 y C 2α t This view is suitable when norms other than the Euclidean are used! Even- semi norms can be used { x(t + 1) = argmin f(x(t), y x(t) + 1 } D(y, x(t)) y C 2α t with a Bregman-distance function D(y, x(t)). Analysis of the behavior starts by establishing a basic relation throughout the use of the optimality principle. } 35