Constrained and Unconstrained Optimization

Transcription

1 Constrained and Unconstrained Optimization Carlos Hurtado Department of Economics University of Illinois at Urbana-Champaign Oct 10th, 2017 C. Hurtado (UIUC - Economics) Numerical Methods

2 On the Agenda 1 Numerical Optimization 2 Minimization of Scalar Function 3 Golden Search 4 Newton s Method 5 Polytope Method 6 Newton s Method Reloaded 7 Quasi-Newton Methods 8 Non-linear Least-Square 9 Constrained Optimization C. Hurtado (UIUC - Economics) Numerical Methods

3 Numerical Optimization On the Agenda 1 Numerical Optimization 2 Minimization of Scalar Function 3 Golden Search 4 Newton s Method 5 Polytope Method 6 Newton s Method Reloaded 7 Quasi-Newton Methods 8 Non-linear Least-Square 9 Constrained Optimization C. Hurtado (UIUC - Economics) Numerical Methods

4 Numerical Optimization Numerical Optimization In some economic problems, we would like to find the value that maximizes or minimizes a function. We are going to focus on the minimization problems: or min f (x) x min f (x) s.t. x B x Notice that minimization and maximization are equivalent because we can maximize f (x) by minimizing f (x). C. Hurtado (UIUC - Economics) Numerical Methods 1 / 27

5 Numerical Optimization Numerical Optimization We want to solve this problem in a reasonable time Most often, the CPU time is dominated by the cost of evaluating f (x). We will like to keep the number of evaluations of f (x) as small as possible. There are two types of objectives: - Finding global minimum: The lowest possible value of the function over the range. - Finding a local minimum: Smallest value within a bounded neighborhood. C. Hurtado (UIUC - Economics) Numerical Methods 2 / 27

6 Minimization of Scalar Function On the Agenda 1 Numerical Optimization 2 Minimization of Scalar Function 3 Golden Search 4 Newton s Method 5 Polytope Method 6 Newton s Method Reloaded 7 Quasi-Newton Methods 8 Non-linear Least-Square 9 Constrained Optimization C. Hurtado (UIUC - Economics) Numerical Methods

7 Minimization of Scalar Function Bracketing Method We would like to find the minimum of a scalar funciton f (x), such that f : R R. The Bracketing method is a direct method that does not use curvature or local approximation We start with a bracket: (a, b, c) s.t. a < b < c and f (a) > f (b) and f (c) > f (b) We will search for the minimum by selecting a trial point in one of the intervals. If c b > b a, take d = b+c 2. Else, if c b b a, take d = a+b 2 If f (d) > f (b), there is a new bracket (d, b, c) or (a, b, d). If f (d) < f (b), there is a new bracket (a, d, c). Continue until the distance between the extremes of the bracket is small. C. Hurtado (UIUC - Economics) Numerical Methods 3 / 27

8 Minimization of Scalar Function Bracketing Method We selected the new point using the mid point between the extremes, but what is the best location for the new point d? a b d c One possibility is to minimize the size of the next search interval. The next search interval will be either from a to d or from b to c The proportion of the left interval is w = b a c a The proportion of the new interval is z = d b c a C. Hurtado (UIUC - Economics) Numerical Methods 4 / 27

9 Golden Search On the Agenda 1 Numerical Optimization 2 Minimization of Scalar Function 3 Golden Search 4 Newton s Method 5 Polytope Method 6 Newton s Method Reloaded 7 Quasi-Newton Methods 8 Non-linear Least-Square 9 Constrained Optimization C. Hurtado (UIUC - Economics) Numerical Methods

10 Golden Search Golden Search The proportion of the new segment will be 1 w = c b c a or w + z = d a c a Moreover, if d is the new candidate to minimize the function, Ideally we will have and d b z 1 w = c a c b c a z = 1 2w z 1 w = w = d b c b C. Hurtado (UIUC - Economics) Numerical Methods 5 / 27

11 Golden Search Golden Search The previous equations imply w 2 3w + 1 = 0, or w = In mathematics, the golden ration is φ = This goes back to Pythagoras Notice that 1 1 φ = The Golden Search algorithm uses the golden ratio to set the new point (using a weighed average) This reduces the bracketing by about 40%. The performance is independent of the function that is being minimized. C. Hurtado (UIUC - Economics) Numerical Methods 6 / 27

12 Golden Search Golden Search Sometimes the performance can be improve substantially when a local approximation is used. When we use a combination of local approximation and golden search we get a method called Brent. Let us suppose that we want to minimize y = x(x 2)(x + 2) 2 C. Hurtado (UIUC - Economics) Numerical Methods 7 / 27

13 Golden Search Golden Search Sometimes the performance can be improve substantially when a local approximation is used. When we use a combination of local approximation and golden search we get a method called Brent. Let us suppose that we want to minimize y = x(x 2)(x + 2) y = x(x - 2)(x + 2) x C. Hurtado (UIUC - Economics) Numerical Methods 7 / 27

14 Golden Search Golden Search We can use the minimize scalar function from the scipy.optimize module. 1 >>> def f( x): 2 >>>... return (x - 2) * x * (x + 2) **2 3 >>> from scipy. optimize import minimize_ scalar 4 >>> opt_res = minimize_ scalar ( f) 5 >>> print opt_res. x >>> opt_res = minimize_scalar (f, method = golden ) 8 >>> print opt_res. x >>> opt_res = minimize_scalar (f, bounds =( -3, -1), method = bounded ) 11 >>> print opt_res. x C. Hurtado (UIUC - Economics) Numerical Methods 8 / 27

15 Newton s Method On the Agenda 1 Numerical Optimization 2 Minimization of Scalar Function 3 Golden Search 4 Newton s Method 5 Polytope Method 6 Newton s Method Reloaded 7 Quasi-Newton Methods 8 Non-linear Least-Square 9 Constrained Optimization C. Hurtado (UIUC - Economics) Numerical Methods

16 Newton s Method Newton s Method Let us assume that the function f (x) : R R is infinitely differentiable We would like to find x such that f (x ) f (x) for all x R. Idea: Use a Taylor approximation of the function f (x). The polynomial approximation of order two around a is: p(x) = f (a) + f (a)(x a) f (a)(x a) 2 To find an optimal value for p(x) we use the FOC: p (x) = f (a) + (x a)f (a) = 0 Hence, x = a f (a) f (a) C. Hurtado (UIUC - Economics) Numerical Methods 9 / 27

17 Newton s Method Newton s Method The Newton s method starts with a given x 1. To compute the next candidate to minimize the function we use x n+1 = x n f (x n ) f (x n ) Do this until and x n+1 x n < ε f (x n+1 ) < ɛ Newton s method is very fast (quadratic convergence). Theorem: x n+1 x n < f (x ) 2 f (x ) x n x 2 C. Hurtado (UIUC - Economics) Numerical Methods 10 / 27

18 Newton s Method Newton s Method A Quick Detour: Root Finding Consider the problem of finding zeros for p(x) Assume that you know a point a where p(a) is positive and a point b where p(b) is negative. If p(x) is continuous between a and b, we could approximate as: p(x) p(a) + (x a)p (a) The approximate zero is then: x = a p(a) p (a) The idea is the same as before. Newton s method also works for finding roots. C. Hurtado (UIUC - Economics) Numerical Methods 11 / 27

19 Newton s Method Newton s Method There are several issues with the Newton s method: - Iteration point is stationary - Starting point enter a cycle - Derivative does not exist - Discontinuous derivative Newton s method finds a local optimum, but not a global optimum. C. Hurtado (UIUC - Economics) Numerical Methods 12 / 27

20 Polytope Method On the Agenda 1 Numerical Optimization 2 Minimization of Scalar Function 3 Golden Search 4 Newton s Method 5 Polytope Method 6 Newton s Method Reloaded 7 Quasi-Newton Methods 8 Non-linear Least-Square 9 Constrained Optimization C. Hurtado (UIUC - Economics) Numerical Methods

21 Polytope Method Polytope Method The Polytope (a.k.a. Nelder-Meade) Method is a direct method to find the solution of min f (x) x where f : R n R. We start with the points x 1, x 2 and x 3, such that f (x 1 ) f (x 2 ) f (x 3 ) Using the midpoint between x 2 and x 3, we reflect x 1 to the point y 1 Check if f (y 1 ) < f (x 1 ). If true, you have a new polytope. If not, try x 2. If not, try x 3 If nothing works, shrink the polytope toward x 3. Stop when the size of the polytope is smaller then ε C. Hurtado (UIUC - Economics) Numerical Methods 13 / 27

22 Polytope Method Polytope Method Let us consider the following function: f (x 0, x 1 ) = (1 x 0 ) (x 1 x 2 0 ) 2 The function looks like: C. Hurtado (UIUC - Economics) Numerical Methods 14 / 27

23 Polytope Method Polytope Method Let us consider the following function: The function looks like: f (x 0, x 1 ) = (1 x 0 ) (x 1 x 2 0 ) y 1500 y x x x x C. Hurtado (UIUC - Economics) Numerical Methods 14 / 27

24 Polytope Method Polytope Method Let us consider the following function: The function looks like: f (x 0, x 1 ) = (1 x 0 ) (x 1 x 2 0 ) C. Hurtado (UIUC - Economics) Numerical Methods 14 / 27

25 Polytope Method Polytope Method In python we can do: 1 >>> def f2(x): 2... return (1 -x [0]) ** *( x[1] -x [0]**2) **2 3 >>> from scipy. optimize import fmin 4 >>> opt = fmin ( func =f2,x0 =[0,0]) 5 >>> print ( opt ) 6 [ ] C. Hurtado (UIUC - Economics) Numerical Methods 15 / 27

26 Newton s Method Reloaded On the Agenda 1 Numerical Optimization 2 Minimization of Scalar Function 3 Golden Search 4 Newton s Method 5 Polytope Method 6 Newton s Method Reloaded 7 Quasi-Newton Methods 8 Non-linear Least-Square 9 Constrained Optimization C. Hurtado (UIUC - Economics) Numerical Methods

27 Newton s Method Reloaded Newton s Method What can we do if we want to use Newton s Method for a function f : R n R? We can use a quadratic approximation at a = (a 1,, a n ): p(x) = f (a) + f (a)(x a) (x a) H(a)(x a) where x = (x 1,, x n ). The gradient f (x) is( a multi-variable generalization of the derivative: f (x) = f (x),, x 1 ) f (x) x n C. Hurtado (UIUC - Economics) Numerical Methods 16 / 27

28 Newton s Method Reloaded Newton s Method The hessian matrix H(x) is a square matrix of second-order partial derivatives that describes the local curvature of a function of many variables. H(x) = 2 f (x) x1 2 2 f (x) x 2 x 1. 2 f (x) x n x 1 2 f (x) x 1 x 2 2 f (x) x 1 x n 2 f (x) 2 f (x) x2 2 x 2 x n f (x) x n x 2 2 f (x) xn 2 The FOC is: p = f (a) + H(a)(x a) = 0 We can solve this to get: x = a H(a) 1 f (a) C. Hurtado (UIUC - Economics) Numerical Methods 17 / 27

29 Newton s Method Reloaded Newton s Method Following the same logic as in the one dimensional case: x k+1 = x k H(x k ) 1 f (x k ) How do we compute H(x k ) 1 f (x k )? We can solve: H(x k ) 1 f (x k ) = s f (x k ) = H(x k )s The search direction, s, is the solution of a system of equations (and we know how to solve that!) C. Hurtado (UIUC - Economics) Numerical Methods 18 / 27

30 Quasi-Newton Methods On the Agenda 1 Numerical Optimization 2 Minimization of Scalar Function 3 Golden Search 4 Newton s Method 5 Polytope Method 6 Newton s Method Reloaded 7 Quasi-Newton Methods 8 Non-linear Least-Square 9 Constrained Optimization C. Hurtado (UIUC - Economics) Numerical Methods

31 Quasi-Newton Methods Quasi-Newton Methods For Newton s method we need the Hessian of the function. If the Hessian is unavailable, the full Newton s method cannot be used Any method that replaces the Hessian with an approximation is a quasi-newton method. One advantage of quasi-newton methods is that the Hessian matrix does not need to be inverted. Newton s method, require the Hessian to be inverted, which is typically implemented by solving a system of equations Quasi-Newton methods usually generate an estimate of the inverse directly. C. Hurtado (UIUC - Economics) Numerical Methods 19 / 27

32 Quasi-Newton Methods Quasi-Newton Methods The Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm, the Hessian matrix is approximated using updates specified by gradient evaluations (or approximate gradient evaluations). In python: 1 >>> import numpy as np 2 >>> from scipy. optimize import fmin_ bfgs 3 >>> def f( x): 4... return (1 -x [0]) ** *( x[1] -x [0]**2) **2 5 >>> opt = fmin_ bfgs (f, x0 =[0.5,0.5]) Using the gradient we can improve the approximation 1 >>> d e f g r a d i e n t ( x ) : 2... r e t u r n np. a r r a y (( 2 (1 x [ 0 ] ) x [ 0 ] ( x [ 1 ] x [ 0 ] 2 ), 200 ( x [ 1 ] x [ 0 ] 2 ) ) ) 3 >>> opt2 = fmin bfgs ( f, x0 = [10,10], fprime=gradient ) C. Hurtado (UIUC - Economics) Numerical Methods 20 / 27

33 Quasi-Newton Methods Quasi-Newton Methods One of the methods that requires the fewest function calls (therefore very fast) is the Newton-Conjugate-Gradient (NCG). The method uses a conjugate gradient algorithm to (approximately) invert the local Hessian. If the Hessian is positive definite then the local minimum of this function can be found by setting the gradient of the quadratic form to zero In python 1 >>> from scipy. optimize import fmin_ ncg 2 >>> opt3 = fmin_ncg (f,x0 =[10,10], fprime = gradient ) C. Hurtado (UIUC - Economics) Numerical Methods 21 / 27

34 Non-linear Least-Square On the Agenda 1 Numerical Optimization 2 Minimization of Scalar Function 3 Golden Search 4 Newton s Method 5 Polytope Method 6 Newton s Method Reloaded 7 Quasi-Newton Methods 8 Non-linear Least-Square 9 Constrained Optimization C. Hurtado (UIUC - Economics) Numerical Methods

35 Non-linear Least-Square Non-linear Least-Square suppose it is desired to fit a set of data {x i, y i } to a model, y = f (x; p) where p is a vector of parameters for the model that need to be found. A common method for determining which parameter vector gives the best fit to the data is to minimize the sum of squares errors. (why?) The error is usually defined for each observed data-point as: e i (y i, x i ; p) = y i f (x i ; p) The sum of the square of the errors is: N S (p; x, y) = ei 2 (y i, x i ; p) i=1 C. Hurtado (UIUC - Economics) Numerical Methods 22 / 27

36 Non-linear Least-Square Non-linear Least-Square Suppose that we model some populaton data at several times. y i = f (t i ; (A, b)) = Ae bt The parameters A and b are unknown to the economist. We would like to minimize the square of the error to approximate the data C. Hurtado (UIUC - Economics) Numerical Methods 23 / 27

37 Non-linear Least-Square Non-linear Least-Square Suppose that we model some populaton data at several times. y i = f (t i ; (A, b)) = Ae bt The parameters A and b are unknown to the economist. We would like to minimize the square of the error to approximate the data C. Hurtado (UIUC - Economics) Numerical Methods 23 / 27

38 Constrained Optimization On the Agenda 1 Numerical Optimization 2 Minimization of Scalar Function 3 Golden Search 4 Newton s Method 5 Polytope Method 6 Newton s Method Reloaded 7 Quasi-Newton Methods 8 Non-linear Least-Square 9 Constrained Optimization C. Hurtado (UIUC - Economics) Numerical Methods

39 Constrained Optimization Constrained Optimization Let us find the minimum of a scalar function subject to constrains. min f (x) s.t. g(x) = a and h(x) b x Rn Here we have g : R n R m and h : R n R k. Notice that we can re-write the problem as an unconstrained version: min f (x) + 1 x R n 2 p [ m ] (g i (x) a i ) 2 + i=1 k max {0, h j (x) b j } j=1 For a very large value of p, the constrain needs to be satisfied (penalty method). C. Hurtado (UIUC - Economics) Numerical Methods 24 / 27

40 Constrained Optimization Constrained Optimization If the objective function is quadratic, the optimization problem looks like min x R n q(x) = 1 2 x Gx + x c s.t. g(x) = a and h(x) b The structure of this type of problems can be efficiently exploited. This form the basis for Augmented Lagrangian and Sequential Quadratic Programming problems C. Hurtado (UIUC - Economics) Numerical Methods 25 / 27

41 Constrained Optimization Constrained Optimization The Augmented Lagrangian Methods use a mix of the Lagrangian with penalty method. The Sequential Quadratic Programming Algorithms (SQPA) solve the problem by using Quadratic approximations of the Lagrangean function. The SQPA is the analogous of Newton s method for the case of constraints. How does the algorithm solve the problem? It is possible with extensions of simplex method, which we will not cover. The previous extensions can be solved with the BFGS algorithm C. Hurtado (UIUC - Economics) Numerical Methods 26 / 27

42 Constrained Optimization Constrained Optimization Let us consider the Utility Maximization problem of an agent with constant elasticity of substitution (CES) utility function: U(x, y) = (αx ρ + (1 α) y ρ ) 1 ρ Denote by p x and p y the prices of goods x and y respectively. the constraint optimization problem for the consumer is: max U(x, y; ρ, α) subject to x 0, y 0 and p xx + p y y = M x,y C. Hurtado (UIUC - Economics) Numerical Methods 27 / 27