Computational Optimization. Constrained Optimization Algorithms

Transcription

1 Computational Optimization Constrained Optimization Algorithms

2 Same basic algorithms Repeat Determine descent direction Determine step size Take a step Until Optimal But now must consider feasibility, e.g. Pick a feasible descent directions Choose step size to maintain feasibility Need dual multipliers to check optimality

3 Builds on Prior Approaches Unconstrained: min f ( x) Lots of approaches Linear Equality Constrained: min f ( x) st.. Ax= b Convert to unconstrained and solve

4 Prior Approaches (cont) Linear Inequality Constrained: min f ( x) st.. Ax b Identify active constraints Solve subproblems Nonlinear Inequality Constrained: min f( x) st.. g( x) 0 Linearize constraints Solve subproblems

5 Feasible Direction Methods Use feasible descent directs. Consider 1 min xqx ' + cx ' 2 st.. Ax= b If we have feasible point ˆx then all x = xˆ + Zp are feasible.

6 Reduced Problem 1 min p ( x ˆ + Zp) ' Q( x ˆ + Zp) + c'( x ˆ + Zp) min ' ' ( ˆ ) ' ˆ ˆ ' ˆ p p Z QZp + Qx + c Zp + xqx + c x min ' ' ( ˆ p pzqzp+ Qx+ c) ' Zp 2 ZQZ ' : = Reduced Hessian Z'( Q( xˆ + Zp) + c) : = Reduced Gradient

7 Reduced Newton s Equation For general problems Z 2 ' f( x) Z : = Reduced Hessian Z' f( x) : = Reduced Gradient Reduced Newton Equations Z f x Z v Z f x 2 ' ( ) = - ' ( ) v = Z f x Z Z f x 2 1 ( ' ( ) ) ' ( ) in reduced space 2 1 p = Zv = Z( Z' f( x) Z ) Z' f( x) in original space Then apply usual Newton s Method

8 Reduced Steepest Descent Use reduced steepest descent direction v = Z' f( x) in reduced space d = Zv = ZZ' f( x) in original space Convergence rate depends on condition number of reduced Hessian Cond Z f x Z Cond Z Z Cond f x 2 2 ( ' ( ) ) ( ' ) ( ( )) KEY POINT Construct Z to have good conditioning!

9 Ways to Compute Z Projection Method (orthogonal and non-orthogonal) Variable Reduction QR Factorization What are they and how do they effect conditioning? How can you use them to compute multipliers?

10 Projection Method Find closest feasible point min c x Ax = 0 x Compute a KKT point: This is optimal projection since problem is convex

11 min c x Ax = 0 x Projection Method Lx (, λ) = c x + ( Ax)' λ ( ) xlx (, λ) = c x + A' λ = 0 KKT are Ax b = 0 Ac Ax AA' λ = 0 ( ') 1 ( ') ( ') 1 AA' λ = Ac λ = AA Ac x = c A' AA Ac 1 x = I A' AA A c ( ) 1 Projection Matrix P = I A' AA' A

12 Project method The matrix ( ) ' 1 Z = I A' AA A is a basis for the null space matrix of A. Check ( ) AZ = A I A' AA' 1 A = A A = 0

13 Get Lagrangian Multipliers for free! The matrix 1 1 A A' AA' = where AA = AA' AA' = I r ( ) ( ) is the right inverse matrix for A. For general problems r min f ( x) st.. Ax λ ' r * = A f( x*) = b

14 Let s try it For min f( x) = 2 x + x + x + x st.. x + x + x + x = 1 Projection matrix Z = I A' AA' A= ( ) [ ] [ ] =

15 Solve FONC for Optimal Point FONC x 1 1 x 1 f( x) A' λ λ x 1 x 1 4 x + x + x + x = 1 2 = =

16 Check Optimality Conditions For x* = [1111]/ 4 f( x*) = [1111]/ 4 Ax* = b Z f(*) x = * = Using Lagrangian ( ) 1 1 Ar = A' AA' = [1111]' 4 λ = A f(*) x = 1/4 r Clearly f(*) x = A' λ

17 For min f( x) = x' Cx 1 2 st.. Ax= b You try it C = A= b = Find projection matrix Confirm optimality conds are Z Cx*=0, Ax* = b Find x* Compute Lagrangian multipliers Check Lagrangian form of the multipliers.

18 Projected Gradient Method Algorithm for min f(x) s.t. Ax=b Let Z=I-A (AA ) -1 A x 0 given g =gradf(x 0 ) While not optimal d=-z g x i+1 = x i+ αd (use linesearch) g =gradf(x i+1 ) i=i+1

19 Projected Gradient Method Equivalent to steepest descent on f(x+zp) where x is some feasible point. So convergence rate depends on condition number of this problem The condition number is less than equal condition of Hessian of f * Compute Z HZ for last example and check it s conditioning:

20 Lagrangian Multipliers Each null space method yields a corresponding calculation of the right inverse. Thus each yields calculation of corresponding dual multipliers.

21 Variable Reduction Method Let A=[B N] A is m by n B is m by m assume m < n Z 1 B N = I is a basis matrix for null space of A 1 1 B B A [ ] r = AAr = BN = I

22 Try on our example Take for example first two columns for B Then A= = [ B N] = Z = A r = Condition number of Z CZ = 158 better but not great

23 QR Factorization Use Gram-Schmidt algorithm to make orthogonal factorize A =QR with Q orthogonal and R upper triangular R1 A' = QR = [ Q1 Q2] 0 where A m n, Q n m, Q n ( n m), R m m Z = Q A = QR T 2 r

24 QR on problem Use matlab command QR [Q R ] = qr(a ) Q2 = Q(:,3:4) Cond(Q2 *C*Q2) = 9.79

25 Next Problem Consider Next Hardest Problem min f ( x) s.. t Ax b How could we adapt gradient projection or other linear equality constrained problem to this problem?