Multivariate Numerical Optimization

Transcription

1 Jianxin Wei March 1, 2013

2 Outline 1 Graphics for Function of Two Variables 2 Nelder-Mead Simplex Method 3 Steepest Descent Method 4 Newton s Method 5 Quasi-Newton s Method 6 Built-in R Function 7 Linear Programming

3 Graphics for Function of Two Variables Outline 1 Graphics for Function of Two Variables 2 Nelder-Mead Simplex Method 3 Steepest Descent Method 4 Newton s Method 5 Quasi-Newton s Method 6 Built-in R Function 7 Linear Programming

4 Graphics for Function of Two Variables Built-in R Functions persp() Draws perspective plots. You can see the function from different angles by changing the values of theta and phi contour() Create a contour plot or add contour lines to an existing plot.

5 Graphics for Function of Two Variables f <- function(x,y) 0.5*x^2+2.5*y^2 x <- seq(-6,6,length=51) y <- seq(-3,3,length=51) z <- outer(x,y,f) persp(x, y, z, theta = 30, phi =0, expand = 0.5, col = "lightblue") z x f(x)=0.5*x^2+2.5*y^2 y

6 z x y z x z Multivariate Numerical Optimization Graphics for Function of Two Variables theta=30,phi=0 theta=90,phi=0 x y theta=90,phi=45 theta=45,phi= 30 z y y x

7 Graphics for Function of Two Variables The contour plot for f(x)=0.5*x^2+2.5*y^2 f <- function(x,y){ 0.5*x^2+2.5*y^2 } x <- seq(-6,6,length=51) y <- seq(-3,3,length=51) z <- outer(x,y,f) contour(x,y,z)

8 Graphics for Function of Two Variables colored contour plot of 0.5x^2+2.5y^2 f <- function(x,y){ 0.5*x^2+2.5*y^2 } x <- seq(-6,6,length=51) y <- seq(-3,3,length=51) z <- outer(x,y,f) filled.contour(x,y,z, color = terrain.colors)

9 Graphics for Function of Two Variables Plot the function in (interactive)3d Plot the function in (interactive)3d: 1 Library the package rgl 2 open3d() 3 persp3d(x,y,z)

10 Nelder-Mead Simplex Method Outline 1 Graphics for Function of Two Variables 2 Nelder-Mead Simplex Method 3 Steepest Descent Method 4 Newton s Method 5 Quasi-Newton s Method 6 Built-in R Function 7 Linear Programming

11 Nelder-Mead Simplex Method Outline of Nelder-Mead simplex Method 1 For an p dimensional function, it starts with p + 1 points x 1,..., x p+1, arranged so that they enclose a nonzero volume. For example, in two dimensions, it starts with three points, and the three points should form a triangle. 2 The p + 1 points are labeled in order from smallest to largest values of f(x i ), so that f(x 1 ) f(x 2 ) f(x p+1 ) 3 Drop x p+1 and replace it with a point that gives a smaller value. The new point is chosen from the four proposals according the value of f(z 1 ), where z 1 is the reflection point of x p+1 through x mid.

12 Nelder-Mead Simplex Method Nelder-Mead simplex in two dimensions If f(z 1 ) < f(x 1 ), i.e. f(z 1 ) falls in Region A, calculate z 2 by reflection and expansion: If f(z 2 ) < f(z 1 ), replace x 3 with z 2 (Reflection and Expansion). If f(z 2 ) f(z 1 ), replace x 3 with z 1 (Reflection).

13 Nelder-Mead Simplex Method Nelder-Mead simplex in two dimensions If f(z 1 ) < f(x 1 ), i.e. f(z 1 ) falls in Region A, calculate z 2 by reflection and expansion: If f(z 2 ) < f(z 1 ), replace x 3 with z 2 (Reflection and Expansion). If f(z 2 ) f(z 1 ), replace x 3 with z 1 (Reflection). If f(x 1 ) f(z 1 ) < f(x 2 ), i.e. f(z 1 ) falls in Region B, replace x 3 with z 1 (Reflection).

14 Nelder-Mead Simplex Method Nelder-Mead simplex in two dimensions If f(z 1 ) < f(x 1 ), i.e. f(z 1 ) falls in Region A, calculate z 2 by reflection and expansion: If f(z 2 ) < f(z 1 ), replace x 3 with z 2 (Reflection and Expansion). If f(z 2 ) f(z 1 ), replace x 3 with z 1 (Reflection). If f(x 1 ) f(z 1 ) < f(x 2 ), i.e. f(z 1 ) falls in Region B, replace x 3 with z 1 (Reflection). If f(x 2 ) f(z 1 ) < f(x 3 ), i.e. f(z 1 ) falls in Region C, Swap z 1 with x 3.

15 Nelder-Mead Simplex Method Nelder-Mead simplex in two dimensions If f(z 1 ) < f(x 1 ), i.e. f(z 1 ) falls in Region A, calculate z 2 by reflection and expansion: If f(z 2 ) < f(z 1 ), replace x 3 with z 2 (Reflection and Expansion). If f(z 2 ) f(z 1 ), replace x 3 with z 1 (Reflection). If f(x 1 ) f(z 1 ) < f(x 2 ), i.e. f(z 1 ) falls in Region B, replace x 3 with z 1 (Reflection). If f(x 2 ) f(z 1 ) < f(x 3 ), i.e. f(z 1 ) falls in Region C, Swap z 1 with x 3. If f(z 1 ) f(x 3 ), i.e. f(z 1 ) falls in Region D, calculate z 3 by contraction 1: If f(z 3 ) < f(x 3 ), replace x 3 with z 3 (Contraction 1). If f(z 3 ) f(x 3 ), replace x 2, x 3 with x 4, x 5 (Contraction 2).

16 Nelder-Mead Simplex Method Remarks for Nelder-Mead Simplex Method Does not use the derivatives Makes no use of the objective function values other than to compare them (Golden section search share this property) May be rather slow Convergence is not guaranteed Effective when p is small The default method in built-in R function optim()

17 Steepest Descent Method Outline 1 Graphics for Function of Two Variables 2 Nelder-Mead Simplex Method 3 Steepest Descent Method 4 Newton s Method 5 Quasi-Newton s Method 6 Built-in R Function 7 Linear Programming

18 Steepest Descent Method Steepest descent Steepest descent method is one of the oldest and simplest methods for multidimensional optimization. The method can be outlined as: Choose a staring point, search in the direction which the function value goes downhill, f(x). Find the minimum in that direction, calculate the f(x), and search in the new direction. Algorithm: Steepest Descent x 0 =initial guess for k = 0, 1, 2,... s k = f(x k ) {Compute negative Gradient} Choose α k to minimize f(x k + α k s k ){perform line search} x k+1 = x k + α k s k {update solution} end

19 Steepest Descent Method steepest descent method

20 Steepest Descent Method Remarks Very reliable in that it can always make progress provided the gradient is nonzero. Not effective, the convergence rate is only linear.

21 Newton s Method Outline 1 Graphics for Function of Two Variables 2 Nelder-Mead Simplex Method 3 Steepest Descent Method 4 Newton s Method 5 Quasi-Newton s Method 6 Built-in R Function 7 Linear Programming

22 Newton s Method Newton s Method Idea: Approximate the function by a quadratic function. Algorithm: Newton s Method x 0 =initial guess for k = 0, 1, 2,... x k+1 = x k H 1 f (x k) f(x k ) {update solution} end

23 Quasi-Newton s Method Outline 1 Graphics for Function of Two Variables 2 Nelder-Mead Simplex Method 3 Steepest Descent Method 4 Newton s Method 5 Quasi-Newton s Method 6 Built-in R Function 7 Linear Programming

24 Quasi-Newton s Method Quasi-Newton Methods Variants of Newton s method have been developed to reduce its overhead or improve its reliability, or both. Quasi-Newton methods have the general form x k+1 = x k α k B 1 k f(x k), where α k is a line search parameter and B k is some approximation of the Hessian matrix. Many quasi-newton methods are more robust than the pure Newton method and have considerably lower overhead per iteration, yet remain superlinearly convergent.

25 Quasi-Newton s Method BFGS method A quasi-newton method: builds up approximation to Hessian matrix from gradients at start and finish of steps Update term for the Hessian approximation is due to Broyden, Fletcher, Goldfarb and Shanno (proposed separately by all four in 1970) Uses derivative information, calculated either from a user-supplied function or by finite differences If dimension is large, the matrix stored may be very large

26 Quasi-Newton s Method Algorithm: BGFS Method for Unconstrained Optimization x 0 =initial guess B 0 =initial Hessian approximation for k = 0, 1, 2,... x k+1 = x k B 1 f (x k) f(x k ) {update solution} y k = f(x k+1 ) f(x k ) B k+1 = B k + (y k yk T )/(yt k s k) {update approximate Hessian} (B k s k s T k B k)/(s T k B ks k ) end

27 Quasi-Newton s Method CG method A conjugate gradient method: chooses successive search directions that are analogous to axes of an ellipse Does not store a Hessian matrix Suitable for very large problems. Less robust than BFGS method Uses derivative information, calculated either from a user-supplied function or by finite differences

28 Quasi-Newton s Method Algorithm: Conjugate Gradient Method for Unconstrained Optimization x 0 =initial guess g 0 = f(x 0 ) s 0 = g 0 for k = 0, 1, 2,... Choose α k to minimize f(x k + α k s k ) { perform line search} x k+1 = x k + α k s k { update solution } g k+1 = f(x k+1 ) {Compute new gradient} β k+1 = (gk+1 T g k+1)/(gk T g k) s k+1 = g k+1 + β k+1 s k {modify gradient} end

29 Quasi-Newton s Method L-BFGS-B method A limited memory version of BFGS Does not store a Hessian matrix, only a limited number of update steps for it Uses derivative information, calculated either from a user-supplied function or by finite differences Can restrict the solution to lie within a box, the only method of optim() that can do this

30 Built-in R Function Outline 1 Graphics for Function of Two Variables 2 Nelder-Mead Simplex Method 3 Steepest Descent Method 4 Newton s Method 5 Quasi-Newton s Method 6 Built-in R Function 7 Linear Programming

31 Built-in R Function General Optimization R has several different functions: most flexible is optim() which includes several different algorithms. Algorithm of choice depends on how easy it is to calculate derivatives for the function and the dimensions of the objective function. Usually better to supply a function to calculate derivatives, but may be unnecessary extra work. Experiment with different methods...

32 Built-in R Function optim() example We want to minimize f(x 1, x 2 ) = 0.5x x2 2 f <- function (x) 0.5*x[1]^2+2.5*x[2]^2 Gr <- function(x) c(x[1],5*x[2]) optim(c(1,2),f) optim(c(1,2),f,gr=gr, method="bfgs") optim(c(1,2),f,method="bfgs") optim(c(1,2),f,method="cg")... Read the help document of optim().

33 Built-in R Function constroptim() Example Minimize f(x 1, x 2 ) = 0.5x x2 2, subject to x 1 + x 2 > 1 f <- function (x) x[1]^2+x[2]^2 constroptim(c(1,2),f,gr=null,ui=matrix(c(1,1),1,2),ci=1) The feasible region is defined by ui % %theta ci >= 0. The starting value must be in the interior of the feasible region, but the minimum may be on the boundary. ui is the constraint matrix ci is the constraint vector theta is the vector of function variables

34 Linear Programming Outline 1 Graphics for Function of Two Variables 2 Nelder-Mead Simplex Method 3 Steepest Descent Method 4 Newton s Method 5 Quasi-Newton s Method 6 Built-in R Function 7 Linear Programming

35 Linear Programming Standard form for linear programming Minimize subject to the constraints C(x) = c 1 x c k x k a 11 x a 1k b 1 a 21 x a 2k b 2... a m1 x a mk b m and the nonnegativity conditions x 1 0,..., x k 0.

36 Linear Programming Linear programming in R Consider the linear programming problem: Minimize subject to the constraints and x 1, x 2 0. In R, this can be solved as 5x 1 + 8x 2 x 1 + x 2 2 x 2 + 2x 2 3 install.packages("lpsolve") library(lpsolve) lp(objective.in=c(5,8),const.mat=matrix(c(1,1,1,2),2,2), const.rhs=c(2,3),const.dir=c(">=",">="))