Constrained Optimization. February 29

Transcription

1 Constrined Optimiztion Februry 9

2 Generl Problem min f( ) ( NLP) s.. t g ( ) i E i g ( ) i I i

3 Modeling nd Constrints Adding constrints let s us model fr more richer set of problems. For our purpose we re interested primrily in conve constrints.

4 -d Regression w, y

5 Predict Drug Absorption Humn intestinl cell line CACO- Predict permebility 78 descriptors generted Electronic AE Shpe/Property (PES) rditionl 7 molecules with tested permebility y R i R 78 8

6 Lest Squres Approimtion Wnt g( ) y Define error f(, y) y g( ) ξ Minimize loss i i ( y w' ) ( ) Lgs (, ) LwS (, ) y g( ) i i i i

7 Optiml Solution Wnt: y Xw Mthemticl Model: min L( w, S ) min y Xw min y Xw ' y Xw w w w Optimlity Condition: Solution stisfies: L( w, S) Xy ' + XXw ' w XXw ' Xy ' Solving n n eqution is (n ) ( ) ( )

8 Solution Assume XX ' eists, then ( ) ( ) XXw ' Xy ' w X' X X' y Is this good ssumption?

9 Ridge Regression Inverse typiclly does not eist. Use lest norm solution for fiedλ >. Regulrized problem Optimlity Condition: L λ min L ( w, S) min λ w + y Xw ( w, S) w w ( ) λ w X' y+ X' Xw XX ' + λi w Xy ' n w Requires (n ) opertions

10 Ridge Regression (cont) Inverse lwys eists for ny w X X+λI X' y ( ' ) λ >. Works well but not robust to outliers -norm -norm

11 Robust Approimtion Wnt g( ) y Define error f(, y) y g( ) ξ Minimize loss Lgs (, ) LwS (, ) y w' y i i i w' i i i

12 Optiml Solution Wnt: y Xw Mthemticl Model: min L( w, S) min y Xw min yi i w w w w Optimlity Condition?: Not DIFFENIABLE!!

13 rick Add constrints Unconstrined Model: min L( w, S) min y Xw min yi i w w w w Equivlent constrined model min st.. w z y w z i i i i i... m Liner Progrm: liner objective with liner constrints z i

14 Cn still Add Regulriztion Unconstrined Model: min w yi i w + λ w Equivlent constrined model min st.. w z i,..., m i + λ w z y w z i i i i Qudrtic Progrm: qudrtic objective with liner constrints

15 ry -norm Regulriztion Unconstrined Model: min w yi i w +λ w Equivlent constrined model min w st.. i z y w z i.. m i i i i t w t j.. n j j z + t j Liner Progrm: liner objective with liner constrints

16 Side benefit sprse w -norm is sprse norm drives components to -norm -norm Unit bll

17 Optiml Routing in Networks Route trffic through nodes of network: W set of ordered node pirs w(i,j) r w input trffic of w (dt units/sec) Divide trffic on pths from origin to destintion such tht cost is minimized. P w set if pths from origin to destintion w X p portion of r w ssigned to pth p (flow)

18 4 Simple network Origin destintion trffic rte Assume squred rc cost : construct model

19 Network Constrints Network epressed s constrints p P w r Must route ll trffic for ech w p w flow must be nonnegtive p P, w W p Note flow on rc i, j is F ij ll pths contining rci j w p

20 Finl Model Put it together for finl model: Define D ( F ) s the cost of trffic on rc i, j ij minimize D ( F ) i,j st.. F ij p P ij w ij ij ll pths contining rci j p w p P, w W p r w w p

21 Optimlity Conditions: Esiest Problem Liner equlity constrints min f( ) f R m n s.. t A b A R, b R n m

22 Net Esiest Problem Liner inequlity constrints min f( ) f R m n s.. t A b A R, b R n m

23 Generl Form min f ( ) s.t. A b If some * is fesible (A* b), then ny other fesible point cn be written s * + Zp, where Z is the null spce mtri of A: Z is the null spce mtri of A: p, v Zp Av nd if vav, pv, Zp

24 Equivlent Unconstrined Problem given fesible * min p f ( * + Zp) FONC If * is locl min then FONC stistifed t * Z f(* + Zp) Z f(*) A* b reduced grdient

25 SONC FONC If * is locl min then FONC stistifed t * Z f(* + Zp) Z f(*) A* b reduced grdient SONC nd Z f( ) Z is psd reduced Hessin

26 SOSC If * stisifies FONC Z f(* + Zp) A* b Z f(*) reduced grdient SOSC nd Z f( ) Z is pd reduced Hessin then * is strict locl minimum

27 Null nd Rnge Spces See Appendi, red bout rnge nd null spces. A R mn { p Ap } Null( A) R Rnge( A ) the set of vectors spnned by the columns of A { q q A λ for some λ} R q λiai, where A i is the ith column of A m m

28 Fundmentl heorem of Algebr Null( A) Rnge( A') R where A B { + y n A, y B}

29 Orthogonlity N(A) nd R(A ) re orthogonl subspces p N(A) nd q R(A ) λ λ p'q p A, for some, becuse Ap +, p q for some p N(A) nd q R(A )

30 Equivlent Form of FONC Form Z Bck to f(*) min f ( ) s.t. A b f p+ q ( *), p Null(A), q Rnge(A ) + Zv* A * v* positive definite Z f(*) Z Zv* + Z A * λ λ Z Zv AZ Form f(*) A λ* must be true

31 FONC min p f ( * + Zp) FONC If * is locl min then FONC stistifed t * Z f(* + Zp) Z f(*) A* b reduced grdient or equivlently there eists λ* such tht f(*) A λ *

32 Interprettion - I Alternte FONC f(*) A λ * A* b Note tht the sign of λ* doesn t mtter, becuse -A -b is the sme constrint λ* is clled the Lgrngin multiplier A -A f (*) A b contour set of function unconstrined minimum

33 Interprettion - II f ˆ) ( f descent direction ( ˆ) p < A If FONC violted, set of fesible directions p Ap, nd descent directions f(ˆ) p <, intersect

34 ) ( + + A f λ Solve this for nswer m min - s.t. FONC ( *) * A* b f A λ + + Emple

35 f(*) Z Check AZ 9,,, Z λ

36 f(*) Z AZ spce mtri Null b A* f(*) Should lso stisfy Z b A* A f(*) FONC Stisfies Z λ λ

37 How bout SONC? *) ( *) ( ) ( C Z f Z f f Eercise: Show C is positive definite, nd thus tht * is locl minimum.

38 Hndy wys to compute Null Spce- more lter ZNull(A) in mtlb Orthogonl Projection Mtri (net slide) QR fctoriztion (best numericlly) SVD singulr vlue decomposition Vrible Reduction Method (more lter)

39 Orthogonl Projection Method Use optimiztion. Minimize distnce between given point c nd null spce of A. min p p c st.. Ap f( p*) A' λ Ap* or equivlently ( p* c) A' λ Ap*

40 Orthogonl Projection Method Optimlity conditions give us the solution FONC is ( p* c) A' λ Ap* Ap * Ac AA' λ λ AA' ( ) Ac ( ) ( ) p* A' λ + c A' AA' Ac + c ( I A' AA' A) c

41 Orthogonl Projection Method Finl result is: ( ) ( I A' AA' A) Null Mtrices of A Note null spce mtri is not unique ry it in Mtlb for A [ 5; 4 -] Compre with Null(A) Null( A,r)

42 Liner Equlity Constrined Strt with lmost s esy s unconstrined min f s.t. A b Find fesible point Compute Z Null spce mtri of A New problem min f ( ˆ + Zp p ) Apply your fvorite unconstrined lgorithm bsed on reduced grdient nd reduced Hessin ˆ ( )

43 Net Clss Optimlity conditions for liner inequlity constrints.