FAQs on Convex Optimization

Transcription

1 FAQs on Convex Optimization. What is a convex programming problem? A convex programming problem is the minimization of a convex function on a convex set, i.e. min f(x) X C where f: R n R and C R n. f is a convex function and c a convex set. Usually C is described as follows where C = { x: g i ' s g i (x) 0, i=...m, h j (^)=0, j=...m} are convex function and h j ' s are affine function.. What is the importance of convex optimization problems? The major importance of convex programming or convex optimization arises from the fact that every local minimum is a global minimum. Let us consider minimizing f: R n R or C R n where f is a convex function and C is a convex set. Let be a local minimum of f on C. thus Ǝδ>0 such that z ( ) C, f(z) f( ). Let X C( take it outside B δ ( ) C). Join x & B δ ( ) C)using a line segment. Let Z λ = λ x + (- λ) λ Thus Ǝ 0 (o,) such that (o, λ 0, Thus for λ λ (o, 0 Z λ B δ ( ) C f (zλ) ( )

2 f(λx + (-λ) ) f ( ) By convexity of λf(x) +(-λ) f() f( ) => λ(f(x) - f( => (f(x) - f( )) o )) o, as λ >0 Since x is arbitrary we have as the global minimum. 3. What can we tell about the continuity and differentiality of a convex function? If f: R n R is convex then f is continuous and even locally Lipschitz, i.e; for any x Rn and K 0 such that for all y,z B δ (x) we have f (y)- f (z) Kǁ y-zǁ, If f: C R is convex and C is a closed convex set then, f is continuous on the interior of C. If f: R n R is convex, then it is differentiable almost everywhere, i.e.; the set of points in R n at which f is not differentiable forms a set of measure zero. A differentiable function f: R n R is convex if and only if; for all x, y in R n. f(y) -f(x) < f ( x ), y x Thus if (x Rn ) be such that minimizer at x. f =0, then f has a global 4. If f: R n R is differentiable then can we detect it. If f is twice continuously differentiable then there is at least a theoretical way to detect it. A function f is convex if and only if the Hersian matrix f(x) is positive for all x Rn semi-definitely. If f(x) is positive definite for all x Rn, then f is strictly converse. The converse need not be true. Example : f(x) = X 4, X R If f is strongly convex then f(x) is always positive definite.

3 Let f be a p-strongly convex function. since f is twice continuously differentiable, it is differentiable and hence f(y) - f(x) f (x), y x +p ǁy-xǁ, P>0 Now by Taylor's theorem for any λ>0, & w Rn f x+ λw ( = f x ( + λ f (x),w + λ w, f(x) w λ +0 ) Now by strong convexity λ w, f(x) w λ +0 ) Pλ ǁwǁ => w, f(x) w + 0 ( λ ) λ P ǁwǁ Now as λ 0 (i.e; λ 0 we have w, f(x) w P ǁwǁ i.e; w, f(x) w P ǁwǁ Thus f(x) is positive definite. 5. What are the major classes of convex optimization problems? a) Linear Programming problem b) Conic Programming problem c) Semi-definite Programming d) Quadratic convex programming under linear constraints e) Quadratic convex programming under quadratic constraint Linear Programming : min < ax > Ax = b x o where C Rn, A is a m n matrix, b R m, & x 0 x R n This is called linear programming in the standard form. Important feature: If a lower bound exists a minimizer exists. Conic Programming : min < ax >

4 Ax = b x K where K is a pointed convex cone. The cone is called pointed if K (-K) = {0} K for example could be the ice-cream cone or Lorenz-cone. K= { x Rn } : x +x +...+x n... x n ; x n 0 case the above conic problem is called the second-order conic programming problem (SOCP for short). Lorenz cone: Lorenz cone is not a polyhedral cone. Semi- definite Programming : S n + n S : set of nχn systematic matrices : set of nχn, systematic and positive semidefinite matrices S n ++ : { X S n + : X is positive definite} + n S Inner product in S n : min is a convex cone but not polyhedral X, Y > C, X > A i, X > trace (X,Y) = b i X + n S Semi definite programming or SDD for short is not a linear programming problem in matrices.

5 Quadratic convex programming with linear constraints. min <x,qx + c, + d Ax = b x 0 Q S n +, c R n, d R, A is a m n matrix and x + n 0 x R Important fact : If a lower bound exists, then a minimizer exists. This is the celebrated Frank-Wolfe theorem. Quadratic convex programming with linear constraints min x,q 0 + C 0, + d 0 x,q i + C i, + d i 0 i=,...,m where Q 0, Q,..., Q m are positive semi-definite matrices, C 0, C, C m are vectors in R n and d 0, d... d m are elements in R. 6. What are saddle point conditions? Consider the convex optimization problem (CP) min f(x) g i (x) 0, i-,,...m Construct the Lagrangian as follows L (x, λ ) = f(x) + λ g (x) + λ g (x)+...+ λ m g m (x) where λ = ( λ λ m) R i.e; λ i 0, for all i=,...m A vector is (, λ ) R n R is called a saddle point if L (, λ ) (, λ ) L ( x, λ ), for all x R n, and λ R

6 If solves convex optimization problem and slater condition holds, i.e; there exists x R n s.t. g i ( x) 0. i=,...,m then there exists λ R s.t. i) L (, λ ) (, λ ) L ( x, λ ), for all x R n, and λ R ii) λ g i ( ) = 0, i=,...,m If there exists a pair of (, λ ) R n R such that i) & ii) hold then solves (CP).