Chapter 4 Convex Optimization Problems

Transcription

1 Chapter 4 Convex Optimization Problems Shupeng Gui Computer Science, UR October 16, 2015 hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

2 Outline 1 Optimization problems 2 Convex optimization problem 3 Quasiconvex optimization 4 Linear program(lp) 5 Quadratic program(qp) 6 Generalized inequality constraints 7 Vector optimization hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

3 Basic terology Optimization problem in standard form f 0 (x) s.t. f i (x) 0, i = 1, 2,..., m h i (x) = 0, i = 1, 2,..., p x R n is the optimization variable f 0 : R n R is the objective or cost function f i : R n R, i = 1, 2,..., m, are the inequality constraint functions h i : R n R are the equality constraint functions hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

4 Optimal value optimal value p = inf{f 0 (x) f i (x) 0, i = 1, 2,..., m, h i (x) = 0, i = 1,..., p} p = if problem is infeasible (no x satisfies the constraints) p = if problem is unbounded below inf means the infimum. hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

5 Optimal value Example What is the optimal value here? f 0 (x) = (x x 0 ) 2 s.t. x 6 x 10 Figure 1: f 0 which mentioned above,x 0 = 4 hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

6 Optimal value Example What is the optimal value here? 1 f 0 (x) = (x x 0 ) 2 s.t. x = x 0 Figure 2: f 0 which mentioned above,x 0 = 4 hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

7 Optimal Value Example What is the optimal value here? f 0 (x) = log(x) s.t. x > 0 Figure 3: f 0 which mentioned above hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

8 Optimal and locally optimal points x is feasible if x dom f 0 and it satisfies the constraints a feasible x is optimal if f 0 (x) = p ; X opt is the set of optimal points X opt = {x f i (x) 0, i = 1,..., m, h i (x) = 0, i = 1,...p, f 0 (x) = p } x is locally optimal if there is an R > 0 such that x is optimal for f 0 (z) s.t. f i (z) 0, i = 1,..., m h i (z) = 0, i = 1,..., p z x 2 R hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

9 Optimal and locally optimal points Figure 4: Local optimal points hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

10 Optimal and locally optimal points example (with n=1, m=p=0) What is the optimal value and how about the optimal points? f 0 (x) = 1 x, dom f 0 = R ++ Figure 5: f 0 (x) = 1/x hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

11 Optimal and locally optimal points example (with n=1, m=p=0) What is the optimal value and how about the optimal points? f 0 (x) = log(x), dom f 0 = R ++ Figure 6: f 0 (x) = log(x) hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

12 Optimal and locally optimal points example (with n=1, m=p=0) What is the optimal value and how about the optimal points? f 0 (x) = x log(x), dom f 0 = R ++ Figure 7: f 0 (x) = x log(x) hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

13 Optimal and locally optimal points example (with n=1, m=p=0) What is the optimal value and how about the optimal points? f 0 (x) = x log(x), dom f 0 = R ++ Figure 8: f 0 (x) = x log(x) p = 1/e, x = 1/e is optimal. hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

14 Optimal and locally optimal points example (with n=1, m=p=0) What is the optimal value and how about the optimal points? f 0 (x) = x 3 3x Figure 9: f 0 (x) = x 3 3x hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

15 Implicit constraints the standard form optimization problem has an implicit constraint x D = m p dom f i dom h i i=0 i=1 Here include f 0. we call D the domain of the problem. the constraints f i (x) 0, h i (x) = 0 are explicit constraints. a problem is unconstrained if it has no explicit constraints (m=p=0); Example f 0 (x) = k log(b i a T i x) is an unconstrained problem with implicit constraints a T i x < b i, i=1 hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

16 Feasibility problem find s.t. x f i (x) 0, i = 1,..., m h i (x) = 0, i = 1,..., p can be considered a special case of the general problem with f 0 (x) = 0 0 s.t. f i (x) 0, i = 1,..., m h i (x) = 0, i = 1,..., p p = 0 if constraints are feasible; any feasible x is optimal p = if constraints are infeasible hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

17 Convex optimization problem standard form convex optimization problem f 0 (x) s.t. f i (x) 0, i = 1, 2,..., m a T i x = b i, i = 1, 2,..., p f 0, f 1,..., f m are convex; equality constraints are affine problem is quasiconvex if f 0 is quasiconvex(and f 0, f 1,..., f m convex) hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

18 standard form convex optimization problem Sometimes written as f 0 (x) s.t. f i (x) 0, i = 1, 2,..., m Ax = b important property: feasible set of a convex optimization problem is convex. hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

19 standard form convex optimization problem Feasible set of a convex optimization problem is convex. Proof Recall the intersection of (any number of) convex sets is convex Chapter 2 It is the intersection of the domain of the problem D = m i=0 dom f i which is a convex set, with m (convex) sublevel sets {x f i (x) 0} and p hyperplanes {x a T i x = b i} hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

20 standard form convex optimization problem Example f 0 (x) = x x 2 2 s.t. f 1 (x) = x 1 /(1 + x 2 2) 0 h i (x) = (x 1 + x 2 ) 2 = 0 f 0 is convex; feasible set {(x 1, x 2 ) x 1 = x 2 0} is convex not a convex problem (according to our definition): f 1 is not convex, h i is not affine. equivalent (but not identical) to the convex problem x x 2 2 s.t. x 1 0 x 1 + x 2 = 0 hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

21 standard form convex optimization problem Figure 10: f 0 (x) = x x 2 2 hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

22 Local and global optima Any locally optimal point of a convex problem is (globally) optimal Proof: suppose x is locally optimal, but there exists a feasible y with f 0 (y) < f 0 (x) x locally optimal means there is an R > 0 such that z feasible, z x 2 R f 0 (z) f 0 (x) Consider z = θy + (1 θ)x with θ = R/(2 y x 2 ) y x 2 > R, so 0 < θ < 1/2 z is a convex combination of two feasible points, hence also feasible z x 2 = R/2 and f 0 (z) θf 0 (y) + (1 θ)f 0 (x) < f 0 (x) which contradicts our assumption that x is locally optimal hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

23 Local and global optima Figure 11: f 0 (x) = x 2 hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

24 Optimality criterion for differentiable Suppose that f 0 (x) is differentiable in a convex optimization problem, x, y dom f 0 f 0 (y) f 0 (x) + f 0 (x) T (y x) (first order condition) Feasible set X opt = {x f i (x) 0, i = 1,..., m, a T i x = b i, i = 1,..., p} Criterion x is optimal if and only if it is feasible and f 0 (x) T (y x) 0 for all feasible y If nonzero, f 0 (x) defines a supporting hyperplane to feasible set X at x. hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

25 Optimality criterion for differentiable Figure 12: Supporting Hyperplane hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

26 Optimality criterion for differentiable Proof 1 Suppose x X and satisfies f 0 (x) T (y x) 0 for all feasible y. Then if y X, we have f 0 (y) f 0 (x), by the first order condition, f 0 (y) f 0 (x) + f 0 (x) T (y x). This shows x is an optimal point for the problem. 2 Suppose x is optimal, but the condition f 0 (x) T (y x) 0 for all feasible y does not hold, i.e., for some y X we have f 0 (x) T (y x) < 0 Consider z(t) = tx + (1 t)y, t [0, 1]. Since z(t) is on the line segment between x and y, and the feasible set is convex. z(t) is feasible. hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

27 Optimality criterion for differentiable Claim that for small positive 1 t, we have f 0 (z(t)) < f 0 (x), since which x is not optimal. d dx f 0(z(t)) t=1 = f 0 (x) T (y x) < 0 so for small positive 1 t, we have f 0 (z(t)) < f 0 (x) hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

28 Optimality criterion for differentiable unconstrained problem: x is optimal if and only if x dom f 0, f 0 (x) = 0 Proof Suppose x is optimal, which means that x dom f 0, and for all feasible y we have f 0 (x) T (y x) 0. Since f 0 is differentiable, its domain is open, so all y sufficiently close to x are feasible. Take y = x t f 0 (x), where x R is a parameter. For t small and positive, y is feasible, and so f 0 (x) T (y x) = f 0 (x) T ( t f 0 (x)) = t f 0 (x) Therefore, x is optimal if and only if x dom f 0, f 0 (x) = 0 hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

29 Optimality criterion for differentiable equality constrained problem: f 0 (x) s.t.ax = b x is optimal if and only if there exists a v such that x dom f 0, Ax = b, f 0 (x) + A T v = 0 Proof y, Ay = b. Since x is feasible, every feasible y has the form y = x + v for some v N (A). Then f 0 (x) T v 0, v N (A). If a linear function is nonnegative on a subspace, then it must be zero on the subspace, so it follows that f 0 (x) T v = 0, v N (A). In other words, f 0 (x) N (A). Using the fact N (A) = R(A T ), f 0 (x) R(A T ), there exists a v R p such that f 0 (x) + A T v = 0 hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

30 Optimality criterion for differentiable imization over nonnegative orthant f 0 (x) s.t. x 0 x is optimal if and only if x dom f 0, x 0, f 0 (x) 0, x i ( f 0 (x)) i = 0, i = 1,..., n hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

31 Equivalent convex problems two problems are (informally) equivalent if the solution of one is readily obtained from the solution of the other, and vice-versa Some common transformations that preserve convexity. eliating equality constraints is equivalent to f 0 (x) s.t. f i (x) 0, i = 1, 2,..., m Ax = b where F and x 0 are such that z f 0 (F z + x 0 ) s.t. f i (F z + x 0 ) 0, i = 1, 2,..., m Ax = b x = F z + x 0 for some z hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

32 Equivalent convex problems introducing equality constraints f 0 (A 0 x + b 0 ) s.t. f i (A i x + b i ) 0, i = 1, 2,..., m is equivalent to x,y i f 0 (y 0 ) s.t. f i (y i ) 0, i = 1, 2,..., m y i = A i x + b i, i = 0, 1,..., m hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

33 Equivalent convex problems introducing slack variables for linear inequalities is equivalent to x,s f 0 (x) s.t. a T i x b i, i = 1, 2,..., m f 0(x) s.t. a T i x + s i = b i, i = 1, 2,..., m s i 0, i = 1,..., m hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

34 Equivalent convex problems epigraph form x,t t s.t. f 0 (x) t 0 f i (x) 0, i = 1, 2,..., m Ax = b hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

35 Equivalent convex problems imizing over some variables x,t f 0 (x 1, x 2 ) s.t. f i (x 1 ) 0, i = 1, 2,..., m is equivalent to x,t f 0(x 1 ) s.t. f i (x 1 ) 0, i = 1, 2,..., m where f 0 (x 1) = inf x2 f 0 (x 1, x 2 ) hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

36 Quasiconvex optimization x,t f 0 (x) s.t. f i (x) 0, i = 1, 2,..., m Ax = b with f 0 : R n R quasiconvex, f 1,..., f m convex This problem can have locally optimal points that are not (globally) optimal Figure 13: Quasiconvex optimization hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

37 Quasiconvex optimization convex representation of sublevel sets of f 0 if f 0 is quasiconvex, there exists a family of functions Φ t such that: Φ t (x) is convex in x for fixed t t-sublevel set of f 0 is 0-sublevel set of Φ t, i.e., f 0 (x) t Φ t (x) 0 example f 0 (x) = p(x) q(x) with p convex, q concave, and p(x) 0, q(x) > 0 on dom f 0 Take Φ t (x) = p(x) tq(x): for t 0, Φ t convex in x p(x)/q(x) t if and only if Φ t (x) 0 hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

38 Linear program(lp) s.t. c T x + d Gx h Ax = b convex problem with affine objective and constraint functions feasible set is a polyhedron hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

39 Linear program(lp) Formulate the LPs to standard form. s.t. c T x + d Gx + s = h Ax = b s 0 Hints: x +, positive part of x; x, negative part of x. x = x + x. s.t. c T x + c T x + d Gx + Gx + s = h Ax + Ax = b s 0, x + 0, x 0 hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

40 Linear-fractional program linear-fractional program s.t. f 0 (x) Gx h Ax = b f 0 (x) = ct x + d e T + f, dom f 0(x) = {x e T x + f > 0} a quasiconvex optimization problem; can be solved by visection also equivalent to the LP (variables y,z) s.t. c T y + dz Gy hz Ay = bz e T y + fz = 1, z 0 hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

41 Generalized linear-fractional program f 0 (x) = max i=1,...,r c T i x + d i e T i x + f, domf 0 (x) = {x e T i x + f i, i = 1,..., r} i a quasiconvex optimization problem; can be solved by bisection. hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

42 Quadratic program(qp) s.t. (1/2)x T P x + q T x + r Gx h Ax = b P S n +, so objective is convex quadratic imize a convex quadratic function over a polyhedron Example: least-squares Ax b 2 2 x T A T Ax 2b T Ax + b T b hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

43 Quadratically constained quadratic program(qcqp) (1/2)x T P 0 x + q0 T x + r 0 s.t. (1/2)x T P i x + qi T x + r i 0, i = 1,..., m Ax = b P i S n +; objective and constraints are convex quadratic if P 1,..., P m S n ++, feasible region is intersection of m ellipsoids and an affine set. hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

44 Second-order cone programg s.t. f T x A i x + b i 2 c T i x + d i, i = 1,..., m F x = g A i R ni n, F R p n inequalities are called second-order cone (SOC) constraints: (A i x + b i, c T i x + d i ) second order cone inr n i+1 for A i = 0, reduces to an LP; if c i = 0, reduces to a QCQP more general than QCQP and LP hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

45 Robust linear programg the parameters in optimization problems are often uncertain, e.g., in an LP s.t. c T x a T i x b i, i = 1,..., m there can be uncertainty in c, a i, b i two common approaches to handling uncertainty(in a i, for simplicity) deteristic model: constraints must hold for all a i ε s.t. c T x a T i x b i, a i ε i, i = 1,..., m stochastic model: a i is random variable; constraints must hold with probability η s.t. c T x prob(a T i x b i ) η, i = 1,..., m hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

46 deteristic approach via SOCP choose an ellipsoid as ε i : ε i = {ā i + P i u u 2 1} (ā i R n, P i R n n ) center is ā i, semi-axes detered by singular values/vectors of P i robust LP s.t. c T x a T i x b i, a i ε i, i = 1,..., m is equivalent to the SOCP s.t. c T x ā T i x + P T i x 2 b i, i = 1,..., m hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

47 Stochastic approach via SOCP assume a i is Gaussian with mean ā i, covariance Σ i (a i N (ā i, Σ i )) a T i x is Gaussian r.v. with mean at i x, variance xt Σ i x; hence ( prob(a T bi ā T i i x b i ) = Φ x ) Σ 1/2 i x 2 where Φ(x) = (1/ 2π) x e t2 /2 dt is CDF of N (0, 1) robust LP s.t. c T x prob(a T i x b i ), i = 1,..., m with η 1/2, is equivalent to the SOCP c T x s.t. ā T i x + Φ 1 (η) Σ 1/2 i x 2 b i, i = 1,..., m hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

48 Geometric programg monomial function f(x) = cx a 1 1 xa xa n n, dom f = R n ++ with c > 0; exponent a i can be any real number posynomial function: sum of monomials f(x) = K k=1 c k x a 1k 1 x a 2k 2...x a nk n, dom f = R n ++, dom f = R n ++ hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

49 Geometric programg geometric program (GP) s.t. f 0 (x) f i (x) 1, i = 1,..., m h i (x) = 1, i = 1,..., p with f i posynomial, h i monomial hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

50 Geometric program in convex form change variables t oy i = log x i, and take logarithm of cost, constraints monomial f(x) = cx a 1 1 xa xa n n transforms to log f(e y 1,..., e y n ) = a T y + b (b = log c) posynomial f(x) = K k=1 c kx a 1k 1 x a 2k 2...x a nk n transforms to ( K ) log f(e y 1,..., e y n ) = log e at k y+b k (b k = log c k ) k=1 geometric program transforms to convex problem K log( exp(a T 0k y + b 0k)) s.t. log( k=1 K exp(a T ik y + b ik)) 0, i = 1,..., m k=1 Gy + d = 0 hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

51 Generalized inequality constraints convex problem with generalized inequality constraints s.t. f 0 (x) f i (x) Ki 0, i = 1,..., m Ax = b f 0 : R n R convex;f i : R n R k i K i -convex w.r.t proper cone K i same properties as standard convex problem (convex feasible set, local optimum is global, etc.) conic form problem: special case with affine objective and constraints c T x s.t. F x + g K 0 Ax = b hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

52 Semidefinite program (SDP) c T x s.t. x 1 F 1 + x 2 F x n F n + G 0 Ax = b with F i, G S k inequality constraint is called linear matrix inequality (LMI) includes problems with multiple LMI constraints: for example, x 1 ˆF x n ˆFn + Ĝ 0, x 1 F x n Fn + G 0 is equivalent to single LMI [ ] [ F1 0 F2 0 x 1 +x 2 0 F1 0 F2 ] +...+x n [ Fn 0 0 Fn ] [ G G ] 0 hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

53 LP and SOCP as SDP LP and equivalent SDP LP: s.t. c T x Ax b SDP: c T x s.t. diag(ax b) 0 hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

54 LP and SOCP as SDP SOCP and equivalent SDP SOCP: s.t. f T x A i x + b i 2 c T i x + d i, i = 1,..., m SDP: s.t. f T x [ (c T i x + d i )I A i x + b i (A i x + b i ) T c T i x + d i ] 0, i = 1,..., m hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

55 Vector optimization general vector optimization problem (w.r.t.k) s.t. f 0 (x) f i (x) 0, i = 1,..., m h i (x) = 0, i = 1,..., p vector objective f 0 : R n R q, imized w.r.t. proper cone K R q convex vector optimization problem (w.r.t.k) s.t. f 0 (x) f i (x) 0, i = 1,..., m h i (x) = 0, i = 1,..., p with f 0 K-convex, f 1,..., f m convex. hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

56 Optimal and Pareto optimal points set of achievable objective values O = {f 0 (x) x feasible} feasible x is optimal if f 0 (x) is the imum value of O feasible x is Pareto optimal if f 0 (x) is a imal value of O hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

57 Multicriterion optimization vector optimization problem with K = R q + f 0 (x) = (F 1 (x),..., F q (x)) q different objectives F i : roughly speaking we want all F i s to be small feasible x is optimal if y feasible f 0 (x ) f 0 (y) if there exists an optimal point, the objectives are noncompeting feasible x po is Pareto optimal if y feasible, f 0 (y) f 0 (x po ) = f 0 (y) if there are multiple Pareto optimal values, there is a trade-off between the objectives hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58

58 Scalarization to find Pareto optimal points: choose λ K 0 and solve scalar problem s.t. λ T f 0 (x) f i (x) 0, i = 1,..., m h i (x) = 0, i = 1,..., p if x is optimal for scalar problem, then it is Pareto-optimal for vector optimization problem. For convex vector optimization problems, can find (almost) all Pareto optimal points by varying λ K 0 hupeng Gui (Computer Science, UR) Convex Optimization Problems October 16, / 58