Lecture 18. Optimization in n dimensions

Transcription

1 Lecture 8 Optimizatio i dimesios Itroductio We ow cosider the problem of miimizig a sigle scalar fuctio of variables, f x, where x=[ x, x,, x ]T. The D case ca be visualized as fidig the lowest poit of a surface z= f x, y Fig.. Fig. The D miimizatio problem is equivalet to fidig the lowest poit o a surface. A ecessary coditio for a miimum is that f / x i= for all i. The partial derivative f / x i is the ith compoet of the gradiet of f, deoted f, so at a miimum we must have f = I the D case this implies we've bottomed out at the lowest poit of a valley. The gradiet also vaishes at a maximum, so this is a ecessary but ot sufficiet coditio for a miimum. Quadratic fuctios Quadratic fuctios of several variables come up i may applicatios. A quadratic fuctio of variables x i has the form f x, x,, x =c+ bi x i+ i= a x x i= j = ij i j Sice a x x + a x x = a +a ji xi x j ij i j ji j i ij 3 the coefficiets a ij, a ji oly appear as the sum a ij +a ji. Without loss of geerality, therefore, we ca take a ij =a ji. EE Numerical Computig Scott Hudso 5-8-8

2 Lecture 8: Optimizatio i dimesios /9 By differetiatio we have f =c, =aij =b, xi i xi x j 4 which allows us to iterpret the form as a multivariable Taylor series of a arbitrary fuctio. The coditios for a miimum or maximum are for k =,,, =b k + aik x i+ a kj x j=bk + a kj x j = xk i= j= j= 5 I matrix otatio this reads f =b+a x= 6 where bi are the compoets of b, ad a ij are the compoets of the symmetric matrix A. The solutio is x= A b 7 The miimizatio of a quadratic fuctio is equivalet to the solutio of a liear system. However, for a arbitrary fuctio f x we ca't make ay geeral statemets about a miimum. This motivates us to seek a method to systematically search for the miimum of a fuctio of variables. 3 Lie miimizatio We kow how to go about miimizig a fuctio of oe variable. If we start at a poit x ad move oly i the directio of a vector u Fig. the the f x values we ca sample form a fuctio of a sigle variable Fig. x is the startig poit ad u the directio i which we search for a miimum. EE Numerical Computig Scott Hudso 5-8-8

3 Lecture 8: Optimizatio i dimesios 3/9 8 g t = f x +t u Here the variable t is the distace we move i the directio u. We ca use ay of our D miimizatio methods o this fuctio. Of course this is ot likely to fid the miimum of f x. However, suppose we start at x ad move alog the directio u to a miimum. Call this ew poit x =x +t u. The move alog aother directio u to fid a miimum at x =x+t u ad so o. This process should evetually fid a local miimum if oe exists. The process of miimizig the fuctio f x alog a sigle lie defied by some vector u is called lie miimizatio. The algorithm is quite simple Successive lie miimizatio algorithm start with iitial guess x ad search directios ui iterate util coverged for i=,,, fid t that miimizes g t = f x +t ui set x x +t ui A obvious set of directios is the coordiate axes u =, u =,, u = 9 I this case the algorithm simply miimizes f x with respect to x, the with respect to x ad so o. The algorithm will fid a miimum if oe exists, but i may cases it ca be very slow to do so. A example is show i Fig. 3 where we miimize the quadratic fuctio x y f x, y = + x+ y 4 startig at x= y=. The miimum is at x= y=. From the cotour plot we see that the valley of this fuctio is arrow ad orieted 45o to the coordiate axes. Sice we are limited to movig i oly the x or y directio at ay oe time, the algorithm eds up takig may, progressively smaller, zig-zag steps dow the valley. The et movemet is i a diagoal directio alog the valley floor. If that directio was oe of our ui directios the we might be able to take oe big step directly to the miimum. This motivates the developmet of directio set methods which attempt to adapt the ui directios to the geometry of the fuctio beig miimized. Cosider the quadratic fuctio. This ca be writte as f =c+xt b+ x T A x EE Numerical Computig Scott Hudso 5-8-8

4 Lecture 8: Optimizatio i dimesios 4/9 Fig. 3: Successive lie miimizatio alog the coordiate axes. Left: cotours of quadratic fuctio. Right: Progressive results of lie miimizatio. The gradiet is f =b+a x Suppose we have foud the miimum of g t= f x +t u at x. That this poit the gradiet of f must be orthogoal to u, otherwise we could move alog the u to lower values of f. Therefore ut b+ut A x= 3 Now we fid the miimum of g t= f x +t u at x =x+t m u. At this ew x value we wat both u ad u to be orthogoal to the gradiet. This esures that the ew poit remais a miimum alog the u directio as well as the u directio. This requires T T u b+u Ax +t m u = 4 ut A u = 5 Because of 3 this reduces to Two vectors u, u satisfyig this coditio are said to be cojugate. A set of vectors u, u,, u i which all pairs are cojugate is a cojugate set. Oe of the first methods preseted 964 to geerate a set of cojugate directios was Powell's method to which we ow tur. 4 Powell's method Powell showed that a simple additio to the successive lie miimizatio algorithm eables it to fid cojugate directios ad miimize a arbitrary quadratic fuctio of variables i EE Numerical Computig Scott Hudso 5-8-8

5 Lecture 8: Optimizatio i dimesios 5/9 iteratios. After completig the lie miimizatios of the for loop, we form a ew directio v which is the et directio x moved due to the lie miimizatios. We the perform a sigle lie miimizatio alog the directio v. Fially, we discard the first search directio, u, left shift the other directios ui ui+ ad make v the ew u directio. It turs out ay quadratic fuctio will be miimized by iteratios of this procedure. The algorithm is Powell's method start with iitial guess x ad search directios ui iterate util coverged save curret estimate x old x for i=,,, fid t that miimizes f x +t ui set x x +t ui old v [x x old ]/ x x fid t to miimize f x +t v set x x +t v for i=,,, ui ui+ u v Fig. 4: Powell's method. Left: cotours of quadratic fuctio; Right: progressive results of each lie miimizatio. The miimum is foud i = iteratios 6 lie miimizatios total. EE Numerical Computig Scott Hudso 5-8-8

6 Lecture 8: Optimizatio i dimesios 6/9 Fig. 5: The "baaa fuctio. This is actually the egative fuctio so that the valley appears as a hill. Values less tha - have bee chopped off to show greater detail. A Scilab versio of Powell's method is give i the Appedix. Applyig this to fuctio, startig at x= y=, we obtai the results show i Fig. 4. I two iteratios of three lie miimizatios each Powell's method adds oe lie miimizatio after the for loop we arrive at the miimum. Powell's method has figured out the ecessary diagoal directio. A more challegig test is give by the Rosebrock fuctio f x, y = x + y x 6 show i Fig. 5. Because of its shape it is sometimes called the baaa fuctio. The miimum Fig. 6 Powell's method followig the valley of the baaa fuctio. EE Numerical Computig Scott Hudso 5-8-8

7 Lecture 8: Optimizatio i dimesios 7/9 is f,=. Ulike the fuctio of Fig. 4, the valley of this fuctio twists ad the algorithm must followig this chagig directio. Startig at x= y= Powell's method gives the results show i Fig. 6. We ca see how the algorithm tracks the twistig valley ad arrives at the miimum after oly a few iteratios. 5 Newto's method Earlier we saw that the gradiet of the quadratic fuctio f x, x,, x =c+ b i xi + i= a x x i = j= ij i j 7 vaishes at x= A b 8 Newto's method approximates a arbitrary fuctio by a quadratic Taylor series with bi =, a ij =a ji= xi x j xi 9 The vector b is the gradiet of f. The matrix of secod derivatives A is called the Hessia of f. We solve for the miimum of this quadratic Taylor series ad take that to be our ew x. We cotiue util the method has coverged. Newto's method iterate util coverged at x evaluate the gradiet b ad the Hessia A set x x A b Let's apply Newto's method to the baaa fuctio f x, y = x + y x The gradiet is x 4 x y x x b= = y x y The Hessia is f x A= f y x f x y + x 4 y 4 x = 4 x f y Startig at x= y=, we have EE Numerical Computig Scott Hudso 5-8-8

8 Lecture 8: Optimizatio i dimesios b= 8/9, A=, A =, A b= 4 3 ad the ew estimate is x = = y 4 At x=, y= we have b= 4 4 4, A=, A =, A b= ad the ew estimate is x = = y 6 which is the solutio. Newto's method is coceptually simple ad quite powerful. However, it requires us to compute the gradiet ad the Hessia of the fuctio. This may be difficult or impossible i may cases. To overcome this challege, quasi-newto methods have bee developed which attempt to form a approximatio of the Hessia matrix, ad possibly the gradiet also, as the algorithm progresses. Oe such method is the Broyde Fletcher Goldfarb Shao BFGS algorithm, preseted i 97. Aother challege is that for a complicated fuctio, Newto's method eeds to be started sufficietly close to a miimum to be stable. As i the D case this motivates the developmet of hybrid algorithms that attempt to use a quasi-newto method whe it works, but revert to a slowbut-sure backup whe it does ot. The Scilab optim fuctio is of this type. The callig sytax is the same as for the D case [fopt,xopt] = optimlistndcost,f,x; Here fx is the fuctio to be miimized, x is a iitial guess at the miimum, xopt is the computed miimum ad fopt is the miimum fuctio value. By default optim assumes the fuctio f provides both fuctio ad gradiet values. If f returs oly a fuctio value, the listndcost,f statemet takes care of providig umerical derivatives. Here's this fuctio applied to the baaa fuctio. x = [;]; fuctio z = fx z = -x^+*x-x^^; edfuctio [fopt,xopt] = optimlistndcost,f,x; dispxopt; This produces the output.. which is the miimum. EE Numerical Computig Scott Hudso 5-8-8

9 Lecture 8: Optimizatio i dimesios 9/9 6 Appedix Scilab code 6. Powell's method ////////////////////////////////////////////////////////////////////// // optimpowell.sci // 4--3, Scott Hudso, for pedagogic purposes. // Implemets Powell's method for miimizig a fuctio of // variables. ////////////////////////////////////////////////////////////////////// fuctio [xmi, fmi]=optimpowellfu, x, h, tol = legthx; //# of variables searchdir = eye,; //directio for curret search searchdirs = eye,; //set of search directios fuctio s=gfut //local scalar fuctio to pass to D s = fux+t*searchdir //optimizatio routies edfuctio doe = ; while~doe xold = x; //best solutio so far for i=: //miimize alog each of directios searchdir = searchdirs:,i; [a,b,c,fa,fb,fc] = optimbracket-h,h,gfu; [tmi,gmi] = optimgoldea,b,c,fa,fb,fc,gfu,tol/; x = x+tmi*searchdir; //miimum alog this directio ed for i=:- //update search directios searchdirs:,i = searchdirs:,i+; ed v = x-xold; //ew search directio searchdirs:, = v/sqrtv'*v; //add ew search dir uit vector searchdir = searchdirs:,; //miimize alog ew directio [a,b,c,fa,fb,fc] = optimbracket-h,h,gfu; [tmi,gmi] = optimgoldea,b,c,fa,fb,fc,gfu,tol/; x = x+tmi*searchdir; xchage = sqrtsumx-xold.^; if xchage<tol doe = ; ed ed //while xmi = x; fmi = fuxmi; edfuctio EE Numerical Computig Scott Hudso 5-8-8