Optimization III: Constrained Optimization

Optimization III: Constrained Optimization CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Doug James (and Justin Solomon) CS 205A: Mathematical Methods Optimization III: Constrained Optimization 1 / 28

Announcements HW6 due today HW7 out HW8 (last homework) out next Thursday CS 205A: Mathematical Methods Optimization III: Constrained Optimization 2 / 28

Constrained Problems minimize f( x) such that g( x) = 0 h( x) 0 CS 205A: Mathematical Methods Optimization III: Constrained Optimization 3 / 28

Really Difficult! Simultaneously: Minimizing f Finding roots of g Finding feasible points of h CS 205A: Mathematical Methods Optimization III: Constrained Optimization 4 / 28

Implicit Projection Implicit surface: g( x) = 0 CS 205A: Mathematical Methods Optimization III: Constrained Optimization 5 / 28

Implicit Projection Implicit surface: g( x) = 0 Example: Closest point on surface minimize x x x 0 2 such that g( x) = 0 CS 205A: Mathematical Methods Optimization III: Constrained Optimization 5 / 28

Nonnegative Least-Squares minimize x A x b 2 2 such that x 0 CS 205A: Mathematical Methods Optimization III: Constrained Optimization 6 / 28

Manufacturing m materials s i units of material i in stock n products p j profit for product j Product j uses c ij units of material i CS 205A: Mathematical Methods Optimization III: Constrained Optimization 7 / 28

Manufacturing Linear programming problem: maximize x j p jx j such that x j 0 j j c ijx j s i i Maximize profits where you make a positive amount of each product and use limited material. CS 205A: Mathematical Methods Optimization III: Constrained Optimization 8 / 28

Bundle Adjustment min yj,p i ij P i y j x ij 2 2 s.t. P i orthogonal i Applications: Bundler Building Rome in a Day CS 205A: Mathematical Methods Optimization III: Constrained Optimization 9 / 28

Constrained Problems minimize f( x) such that g( x) = 0 h( x) 0 CS 205A: Mathematical Methods Optimization III: Constrained Optimization 10 / 28

Basic Definitions Feasible point and feasible set A feasible point is any point x satisfying g( x) = 0 and h( x) 0. The feasible set is the set of all points x satisfying these constraints. CS 205A: Mathematical Methods Optimization III: Constrained Optimization 11 / 28

Basic Definitions Feasible point and feasible set A feasible point is any point x satisfying g( x) = 0 and h( x) 0. The feasible set is the set of all points x satisfying these constraints. Critical point of constrained optimization A critical point is one satisfying the constraints that also is a local maximum, minimum, or saddle point of f within the feasible set. CS 205A: Mathematical Methods Optimization III: Constrained Optimization 11 / 28

Differential Optimality Without h: Λ( x, λ) f( x) λ g( x) Lagrange Multipliers CS 205A: Mathematical Methods Optimization III: Constrained Optimization 12 / 28

Inequality Constraints at x CS 205A: Mathematical Methods Optimization III: Constrained Optimization 13 / 28

Inequality Constraints at x Two cases: Active: h i ( x ) = 0 Optimum might change if constraint is removed Inactive: h i ( x ) > 0 Removing constraint does not change x locally CS 205A: Mathematical Methods Optimization III: Constrained Optimization 14 / 28

Idea Remove inactive constraints and make active constraints equality constraints. CS 205A: Mathematical Methods Optimization III: Constrained Optimization 15 / 28

Lagrange Multipliers Λ( x, λ, µ) f( x) λ g( x) µ h( x) No longer a critical point! But if we ignore that: 0 = f( x) i λ i g i ( x) j µ j h j ( x) CS 205A: Mathematical Methods Optimization III: Constrained Optimization 16 / 28

Lagrange Multipliers Λ( x, λ, µ) f( x) λ g( x) µ h( x) No longer a critical point! But if we ignore that: 0 = f( x) i λ i g i ( x) j µ j h j ( x) µ j h j ( x) = 0 Zero out inactive constraints! CS 205A: Mathematical Methods Optimization III: Constrained Optimization 16 / 28

Inequality Direction So far: Have not distinguished between h j ( x) 0 and h j ( x) 0 CS 205A: Mathematical Methods Optimization III: Constrained Optimization 17 / 28

Inequality Direction So far: Have not distinguished between h j ( x) 0 and h j ( x) 0 Direction to decrease f: f( x ) Direction to decrease h j : h j ( x ) CS 205A: Mathematical Methods Optimization III: Constrained Optimization 17 / 28

Inequality Direction So far: Have not distinguished between h j ( x) 0 and h j ( x) 0 Direction to decrease f: f( x ) Direction to decrease h j : h j ( x ) f( x ) h j ( x ) 0 CS 205A: Mathematical Methods Optimization III: Constrained Optimization 17 / 28

Dual Feasibility µ j 0 CS 205A: Mathematical Methods Optimization III: Constrained Optimization 18 / 28

KKT Conditions Theorem (Karush-Kuhn-Tucker (KKT) conditions) x R n is a critical point when there exist λ R m and µ R p such that: 0 = f( x ) i λ i g i ( x ) j µ j h j ( x ) ( stationarity ) g( x ) = 0 and h( x) 0 ( primal feasibility ) µ j h j ( x ) = 0 for all j ( complementary slackness ) µ j 0 for all j ( dual feasibility ) CS 205A: Mathematical Methods Optimization III: Constrained Optimization 19 / 28

KKT Example from Book CS 205A: Mathematical Methods Optimization III: Constrained Optimization 20 / 28

KKT Example from Book CS 205A: Mathematical Methods Optimization III: Constrained Optimization 21 / 28

Physical Illustration of KKT Example: Minimal gravitational-potential-energy position x = (x 1, x 2 ) T of a particle attached to inextensible rod (of length l), and above a hard surface. minimize x x 2 (Minimize gravitational potential energy) such that x c 2 l = 0 (rod of length l attached at c) x 2 0 (height 0) Physical interpretation of f, g, h, λ and µ? Physical interpretation of stationarity, primal feasibility, complementary slackness and dual feasibility? CS 205A: Mathematical Methods Optimization III: Constrained Optimization 22 / 28

Sequential Quadratic Programming (SQP) x k+1 x k + arg min d [ ] 1 d 2 H f ( x k ) d + f( x k ) d such that g i ( x k ) + g i ( x k ) d = 0 h i ( x k ) + h i ( x k ) d 0 CS 205A: Mathematical Methods Optimization III: Constrained Optimization 23 / 28

Equality Constraints Only ( Hf ( x k ) [Dg( x k )] Dg( x k ) 0 ) ( d λ ) = ( f( xk ) g( x k ) ) Can approximate H f Can limit distance along d CS 205A: Mathematical Methods Optimization III: Constrained Optimization 24 / 28

Inequality Constraints Active set methods: Keep track of active constraints and enforce as equality, update based on gradient CS 205A: Mathematical Methods Optimization III: Constrained Optimization 25 / 28

Barrier Methods: Equality Case f ρ ( x) f( x) + ρ g( x) 2 2 Unconstrained optimization, crank up ρ until g( x) 0 Caveat: H fρ becomes poorly conditioned CS 205A: Mathematical Methods Optimization III: Constrained Optimization 26 / 28

Barrier Methods: Inequality Case 1 Inverse barrier: h i ( x) Logarithmic barrier: log h i ( x) CS 205A: Mathematical Methods Optimization III: Constrained Optimization 27 / 28

To Read: Convex Programming A ray of hope: Minimizing convex functions with convex constraints Next CS 205A: Mathematical Methods Optimization III: Constrained Optimization 28 / 28