Computational Optimization Constrained Optimization Algorithms
Same basic algorithms Repeat Determine descent direction Determine step size Take a step Until Optimal But now must consider feasibility, e.g. Pick a feasible descent directions Choose step size to maintain feasibility Need dual multipliers to check optimality
Builds on Prior Approaches Unconstrained: min f ( x) Lots of approaches Linear Equality Constrained: min f ( x) st.. Ax= b Convert to unconstrained and solve
Prior Approaches (cont) Linear Inequality Constrained: min f ( x) st.. Ax b Identify active constraints Solve subproblems Nonlinear Inequality Constrained: min f( x) st.. g( x) 0 Linearize constraints Solve subproblems
Feasible Direction Methods Use feasible descent directs. Consider 1 min xqx ' + cx ' 2 st.. Ax= b If we have feasible point ˆx then all x = xˆ + Zp are feasible.
Reduced Problem 1 min p ( x ˆ + Zp) ' Q( x ˆ + Zp) + c'( x ˆ + Zp) 2 1 1 min ' ' ( ˆ ) ' ˆ ˆ ' ˆ p p Z QZp + Qx + c Zp + xqx + c x 2 2 1 min ' ' ( ˆ p pzqzp+ Qx+ c) ' Zp 2 ZQZ ' : = Reduced Hessian Z'( Q( xˆ + Zp) + c) : = Reduced Gradient
Reduced Newton s Equation For general problems Z 2 ' f( x) Z : = Reduced Hessian Z' f( x) : = Reduced Gradient Reduced Newton Equations Z f x Z v Z f x 2 ' ( ) = - ' ( ) v = Z f x Z Z f x 2 1 ( ' ( ) ) ' ( ) in reduced space 2 1 p = Zv = Z( Z' f( x) Z ) Z' f( x) in original space Then apply usual Newton s Method
Reduced Steepest Descent Use reduced steepest descent direction v = Z' f( x) in reduced space d = Zv = ZZ' f( x) in original space Convergence rate depends on condition number of reduced Hessian Cond Z f x Z Cond Z Z Cond f x 2 2 ( ' ( ) ) ( ' ) ( ( )) KEY POINT Construct Z to have good conditioning!
Ways to Compute Z Projection Method (orthogonal and non-orthogonal) Variable Reduction QR Factorization What are they and how do they effect conditioning? How can you use them to compute multipliers?
Projection Method Find closest feasible point 1 2 2 min c x Ax = 0 x Compute a KKT point: This is optimal projection since problem is convex
min c x Ax = 0 x 1 2 1 2 Projection Method Lx (, λ) = c x + ( Ax)' λ ( ) xlx (, λ) = c x + A' λ = 0 KKT are Ax b = 0 Ac Ax AA' λ = 0 ( ') 1 ( ') ( ') 1 AA' λ = Ac λ = AA Ac x = c A' AA Ac 1 x = I A' AA A c ( ) 1 Projection Matrix P = I A' AA' A
Project method The matrix ( ) ' 1 Z = I A' AA A is a basis for the null space matrix of A. Check ( ) AZ = A I A' AA' 1 A = A A = 0
Get Lagrangian Multipliers for free! The matrix 1 1 A A' AA' = where AA = AA' AA' = I r ( ) ( ) is the right inverse matrix for A. For general problems r min f ( x) st.. Ax λ ' r * = A f( x*) = b
Let s try it For 1 2 2 2 2 min f( x) = 2 x + x + x + x 1 2 3 4 st.. x + x + x + x = 1 Projection matrix 1 2 3 4 1 0 0 0 1 1 0 1 0 0 1 1 Z = I A' AA' A= 1 1 1 1 1 1 1 1 0 0 1 0 1 1 0 0 0 1 1 1 1 ( ) [ ] [ ] 3 1 1 1 4 4 4 4 1 3 1 1 4 4 4 4 = 1 1 3 1 4 4 4 4 1 1 1 3 4 4 4 4 1
Solve FONC for Optimal Point FONC x 1 1 x 1 f( x) A' λ λ x 1 x 1 4 x + x + x + x = 1 2 = = 1 2 3 4
Check Optimality Conditions For x* = [1111]/ 4 f( x*) = [1111]/ 4 Ax* = b 3 1 1 1 4 4 4 4 1 1 3 1 1 1 4 4 4 4 1 Z f(*) x = * = 0 1 1 3 1 1 4 4 4 4 4 1 1 1 3 1 4 4 4 4 Using Lagrangian ( ) 1 1 Ar = A' AA' = [1111]' 4 λ = A f(*) x = 1/4 r Clearly f(*) x = A' λ
For min f( x) = x' Cx 1 2 st.. Ax= b You try it 0 13 6 3 13 23 9 3 2 1 2 1 2 C = A= b = 6 9 12 1 1 1 3 1 3 3 3 1 3 Find projection matrix Confirm optimality conds are Z Cx*=0, Ax* = b Find x* Compute Lagrangian multipliers Check Lagrangian form of the multipliers.
Projected Gradient Method Algorithm for min f(x) s.t. Ax=b Let Z=I-A (AA ) -1 A x 0 given g =gradf(x 0 ) While not optimal d=-z g x i+1 = x i+ αd (use linesearch) g =gradf(x i+1 ) i=i+1
Projected Gradient Method Equivalent to steepest descent on f(x+zp) where x is some feasible point. So convergence rate depends on condition number of this problem The condition number is less than equal condition of Hessian of f * Compute Z HZ for last example and check it s conditioning:
Lagrangian Multipliers Each null space method yields a corresponding calculation of the right inverse. Thus each yields calculation of corresponding dual multipliers.
Variable Reduction Method Let A=[B N] A is m by n B is m by m assume m < n Z 1 B N = I is a basis matrix for null space of A 1 1 B B A [ ] r = AAr = BN = I + 0 0 0
Try on our example Take for example first two columns for B Then 2 1 2 1 2 1 2 1 A= = [ B N] = 1 1 3 1 1 1 3 1 1 2 1 1 4 3 1 2 Z = A r = 1 0 0 0 0 1 0 0 Condition number of Z CZ = 158 better but not great
QR Factorization Use Gram-Schmidt algorithm to make orthogonal factorize A =QR with Q orthogonal and R upper triangular R1 A' = QR = [ Q1 Q2] 0 where A m n, Q n m, Q n ( n m), R m m Z = Q A = QR T 2 r 1 1 1 2 1
QR on problem Use matlab command QR [Q R ] = qr(a ) Q2 = Q(:,3:4) Cond(Q2 *C*Q2) = 9.79
Next Problem Consider Next Hardest Problem min f ( x) s.. t Ax b How could we adapt gradient projection or other linear equality constrained problem to this problem?