CME307/MS&E311 Theory Summary

Transcription

1 CME307/MS&E311 Theory Summary Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A

2 Optimization Problems A set of decision variables,, in vector or matri form with dimension n A continuous and sometime differentiable objective function f() A feasible region where can be in min s.t. One can smooth them by reformulation as constrained optimization: ma min i { f i (),i=1,,n} > ma α s.t. α f i () 0, for i=1,,n 2

3 Function, Gradient Vector and Hessian Matri The Gradient Vector of f at n ) ( f f f f 3 A function f of in R n Taylor s Epansion Theorem The Hessian Matri of f at n n 2 1 n 2 n ) ( f f f f f

4 Conve and Concave Functions f() f() f( 2 ) f( 1 )+(1- )f( 2 ) f( 1 ) f( 1 +(1- ) 2 ) 1 1 +(1-) 2 2 f() is a conve function if and only if for any given two points 1 and 2 in the function domain and for any constant 0 1 f( 1 +(1 ) 2 ) f( 1 )+(1 )f( 2 ) Strictly conve if 1 2, f( ) < 0.5f( 1 )+0.5f( 2 ) 4

5 Conve Quadratic Functions f()= T Q+c T is a conve function if and only if Hessian matri Q is positive semi definite (PSD). f()= T Q+c T is a strictly conve function if and only if Q is positive definite (PD). Q is PSD if and only if T Q 0 for all. A 22 matri is PSD (or PD) if and only if two diagonal entries and the determinant are nonnegative (or positive) 5

6 Conve Sets A set is conve if every line segment connecting any two points in the set is contained entirely within the set E polyhedron E ball An etreme point of a conve set is any point that is not on any line segment connecting any other two distinct points of the set The intersection of conve sets is a conve set A set is closed if the limit of any convergent sequence of the set belongs to the set 6

7 Properties of Conve Function f() b If f() is a conve function, then the lower level set {: f() b} is a conve set for any constant b. The graph of a conve function lies above its tangent line (planes). The Hessian matri of a conve function is positive semi definite. 7

8 Optimization Problem Classes Unconstrained Optimization Conve or Nonconve Constrained Optimization min s.t. Conic Linear Optimization (CLO) Conve Constrained Optimization (CCO) Feasible region/set conve; objective general Generally Constrained Optimization (GCO) Conve Optimization (CO) Minimize a conve function over a conve feasible set Maimize a concave function over a conve feasible set 8

9 Optimization Problem Forms min c T min f () s.t. A b = 0, s.t. h i () = 0, i=1,,m X K c i () 0, i=1,,p Conic Linear Optimization (CLO) A: an m n matri c: objective coefficient K: a closed conve cone This is conve optimization Generally Constrained Optimization (GCO) Each function can be continuous, continuously differentiable (C 1 ), or twice continuously differentiable (C 2 ) It is CCO if c i are all concave, and h i are all linear/affine functions. In addition, if f is conve, it is CO. 9

10 Why do we care about conve optimization? It guarantees that every local optimizer is a global optimizer It guarantees that every (first order) KKT (or stationary) point/solution is a global optimizer This is significant because all of our numerical optimization algorithms search/generate a KKT point/solution Sometime the problem can be convefied : min c T, s.t. 2 =1 min c T, s.t

11 Optimization Theory: Mathematical Foundations Taylor s Epansion Theorem Implicit Function Theorem Separating Hyperplane Theorem Supporting Hyperplane Theorem Caratheodory s Theorem Duality and KKT Optimality Conditions Alternative Linear System/Farkas Lemma 11

12 Theory: Feasibility Conditions Feasibility Conditions or Farkas Lemmas are developed to characterize and certify feasibility or infeasibility of a feasible region Alternative Systems X and Y: X has a feasible solution if and only if Y has no feasible solution X and Y cannot both have feasible solution Eactly one of them has a feasible solution They can be viewed as special cases of Linear Programming primal and dual pairs 12

13 Alternative Systems and CLO Pairs I A b = 0, X K System X A: an m n matri b: m dimension vector K: a closed conve cone b T y=1(>0) A T y + s = 0, s K* System Y K* is the dual cone p*=min 0 T d*=ma b T y s.t. A b = 0, s.t. A T y + s = 0, X K s K* 13

14 Alternative Systems and CLO Pairs II c T = 1(<0) A = 0, X K A T y + s c = 0, s K* System X A: an m n matri c: n dimension vector K: a closed conve cone System Y K* is the dual cone p*=mi n s.t. c T A = 0, d*=ma s.t. 0 T y A T y + s c = 0, X K* s K 14

15 Feasibility Test Machine Yes Is system X feasible? No Is system Y feasible? Not under any circumstances Is system Y feasible? Yes under certain conditions of cone K and data matri A: a) K is a polyhedron cone, or b) A or A T y has an interior solution 15

16 Theory: Optimality Conditions Optimality (KKT) Conditions are developed to characterize and certify possible minimizers Feasibility of original variables Optimality conditions consist of original variables and Lagrange multipliers Zero order, First order, Second order, necessary, sufficient They may not lead directly to a very efficient algorithm for solving problems, but they do have a number of benefits: They give insight into what optimal solutions look like They provide a way to set up and solve small problems They provide a method to check solutions to large problems The Lagrange multipliers can be seen as sensitivities of the constraints A minimizers may not satisfy optimality conditions unless certain constraint qualifications hold. 16

17 KKT Optimality Condition Test Machine Passed KKT Optimality Condition Test Failed Is a (local) optimizer? Is not a (local) optimizer? Yes only under certain circumstances Higher Order Test Not under certain constraint qualifications: a) Feasible region has an interior, or b) is a regular point on the hypersurface of active constraints 17

18 p*=min s.t. Duality Theorems for CLO c T d*=ma A b = 0, s.t. X K Primal Problem A: an m n matri c: objective coefficient K: a closed conve cone Weak Duality Theorem Dual Problem K* is the dual cone 0 Order Condition: p* =d* Sufficient! Strong Duality Theorem: They must equal? Yes under certain conditions of cone K and data matri A,b,c: a) K is a polyhedron cone, or b) either one has an interior feasible solution 18

19 min s.t. The Lagrange Function of GCO f () Restriction on multipliers y i, c i ()(,=, ) 0, i=1,,m y i (, free, ) 0, i=1,,m The Largrange Function L(,y) = f() i y i c i () The Lagrange function can be interpreted as a penalized aggregated objective function: y i free: can be penalized either way y i 0 : can be penalized when c i () 0 y i 0 : can be penalized when c i () 0 19

20 The Lagrangian Duality for GCO p*=min f () s.t. c i ()(,=, ) 0, i=1,,m Weak Duality Theorem P* d* Let ɸ(y) = inf L(,y) Strong Duality Theorem They must equal? Not necessarily! d*=ma ɸ(y) s.t. y i (, free, ) 0, i=1,,m 0 Order Condition: p* = d* Sufficient! 20

21 Zero Order Optimality Test for CLO and GCO Passed 0 order Optimality Test: 0 duality gap? Failed Yes under any circumstances Is an optimizer? Higher order test Zero order condition is sufficient Is not a (local) optimizer? a) Not for sure if K is a polyhedral cone in CLO; or b) Not when Feasible region has an interior in CCO; otherwise c) Inconclusive in GCO. 21

22 1 and 2 order Conditions: Unconstrained Problem: Minimize f(), where is a vector that could have any values, positive or negative First Order Necessary Condition (min or ma): f() = 0 ( f/ i = 0 for all i) is the first order necessary condition for optimization Second Order Necessary Condition: 2 f() is positive semidefinite (PSD) 2 f/ i2 > 0 [ d T 2 f()d 0 for all d ] Second Order Sufficient Condition f/ i = 0 f (Given FONC satisfied) 2 f() is positive definite (PD) [d T 2 f()d > 0 for all d 0 ] i 22

23 1 Order KKT Condition for GCO Recall the Largrange Function L(,y) = f() i c i ()y i c i () (,=, ) 0, y i (, free, ) 0, i=1,,m 23

24 Optimality Test for CCO Passed 1 order KKT Optimality Test Failed Is a (local) optimizer? Is not a (local) optimizer? Yes if f is also (locally) conve Not when the feasible region has an interior 2 order test 24

25 Optimality Test for GCO Passed 1 order KKT Optimality Test Failed Is a (local) optimizer? Yes if it is a (locally) conve problem 2 order test Is not a (local) optimizer? Not when is a regular point on the hypersurface of active constraints 25

26 2 Order KKT Condition for Equality CO This can be done by checking positive semi definiteness (or definiteness) of the projected Hessian of the Lagrange function 26

27 Applications: Optimality Conditions The market equilibrium theory Fisher market, Arrow Debreu market Duality and optimality lead to equilibrium conditions Sensor localization SOCP: KKT conditions eplain observations SDP: Duality eplains localizability Offline and Online LP Learning optimal dual solution helps to make primal decisions online Non conve regularization L p norm regulation function for unconstrained or constrained minimization KKT conditions establish a desired thresh holding properties at any KKT solution (first or second order) 27