Unconstrained and Constrained Optimization

Transcription

1 Unconstrained and Constrained Optimization

2 Agenda General Ideas of Optimization Interpreting the First Derivative Interpreting the Second Derivative Unconstrained Optimization Constrained Optimization

3 Optimization There are two ways of eamining optimization. Minimization In this case you are looking for the lowest point on the function. Maimization In this case you are looking for the highest point on the function. 3

4 Needed Terminology Critical Point A point * on a function is said to be a critical point if when you evaluate the derivative of the function at the point *, then the derivative at that point is zero, i.e., f (*) 0. 4

5 What observations can you make about the attributes of a minimum? 5

6 Questions Regarding the Minimum What is the sign of the slope when you are to the left of the minimum point? Another way of saying this is what is f () when < *? Note: * denotes the point where the function is at a minimum. 6

7 Questions Regarding the Minimum Cont. What is the sign of the slope when you are to the right of the minimum point? Another way of saying this is what is f () when > *? What is the sign of the slope when you at the minimum point? Another way of saying this is what is f () when *? 7

8 Graphical Representation of a Minimum y 0 y f()

9 What observations can you make about the attributes of a maimum? 9

10 Questions Regarding the Maimum What is the sign of the slope when you are to the left of the maimum point? Another way of saying this is what is f () when < *? Note: * denotes the point where the function is at a maimum. 0

11 Questions Regarding the Maimum Cont. What is the sign of the slope when you are to the right of the maimum point? Another way of saying this is what is f () when > *? What is the sign of the slope when you at the maimum point? Another way of saying this is what is f () when *?

12 Graphical Representation of a Maimum y 6 y f()

13 Interpreting the First Derivative The first derivative of a function as was shown previously is the slope of the curve evaluated at a particular point. In essence it tells you the instantaneous rate of change of the function at the given particular point. Knowing the slope of the function can tell you where a maimum or a minimum eists on a curve. Why? 3

14 Question Can the derivative tell you whether you are at a maimum or a minimum? The answer is yes if you eamine the slope of the function around the critical point, i.e., the point where the derivative is zero. An easier way of eamining whether you have a maimum or a minimum is to eamine the second derivative of the function. 4

15 The Second Derivative The second derivative of a function f() is the derivative of the function f (), where f () is the derivative of f(). The second derivative can tell you whether the function is concave or conve at the critical point. The second derivative can be denoted by f (). 5

16 Concavity and the Second Derivative The maimum of a function f() occurs when a critical point * is at a concave portion of the function. This is equivalent to saying that f (*) < 0. If f () < 0 for all, then the function is said to be concave. 6

17 Conveity and the Second Derivative The minimum of a function f() occurs when a critical point * is at a conve portion of the function. This is equivalent to saying that f (*) > 0. If f () > 0 for all, then the function is said to be conve. 7

18 Special Case of the Second Derivative Suppose you have a function f() that has a maimum at *. What does it mean when the second derivative is equal to zero, i.e., f (*) 0? This is a point where the second derivative may not be able to tell you whether you have a maimum or a minimum. Usually in this case you will get a saddle point/point of inflection where the point is neither a maimum nor a minimum. 8

19 Eample of Special Case of the Second Derivative Suppose y f() 3, then f () 3 and f () 6, This implies that * 0 and f (*0) 0. y yf() 3 9

20 Unconstrained Optimization An unconstrained optimization problem is one where you only have to be concerned with the objective function you are trying to optimize. An objective function is a function that you are trying to optimize. None of the variables in the objective function are constrained. 0

21 First and Second Order Condition For a Maimum The first order condition for a maimum at a point * on the function f() is when f (*) 0. The second order condition for a maimum at a point * on the function f() is when f (*) < 0.

22 First and Second Order Condition For a Minimum The first order condition for a minimum at a point * on the function f() is when f (*) 0. The second order condition for a minimum at a point * on the function f() is when f (*) > 0.

23 Eample of Using First and Second Order Conditions Suppose you have the following function: f() Then the first order condition to find the critical points is: f () This implies that the critical points are at and 3. 3

24 Eample of Using First and Second Order Conditions Cont. The net step is to determine whether the critical points are maimums or minimums. These can be found by using the second order condition. f () 6 6(-) 4

25 Eample of Using First and Second Order Conditions Cont. Testing implies: f () 6(-) -6 < 0. Hence at, we have a maimum. Testing 3 implies: f (3) 6(3-) 6 > 0. Hence at 3, we have a minimum. Are these the ultimate maimum and minimum of the function f()? 5

26 Relative Vs. Absolute Etremum A relative etremum is a point that is locally greater or lesser than all points around it. A relative etrema can be found by using the first order condition. An absolute etremum is a point that is either absolutely greater than or less than all other points, i.e., f(*) > f() for all not equal to * for a maimum and f(*) < f() for all not equal to * for a minimum. 6

27 Finding the Absolute Etremum To find the absolute etremum, you need to compare all the critical points on the function, as well as, any potential end points of the function like and -. When evaluating a polynomial function at, the value of the function at takes the value of the at the highest ordered variable. 7

28 Finding the Absolute Etremum Cont. Some properties of : + - is undefined c*, where c is any value greater than zero * * (- ) - From the previous eample, the relative etremum points occur at -,, 3, and. The absolute maimum occurs at and the absolute minimum occurs at -. 8

29 9 Unconstrained Optimization: Two Variables Suppose you have a function y f(, ), then to find the critical points, you can use the following first order condition: 0 ), ( 0 ), ( * * * * f f f f

30 Unconstrained Optimization: Two Variables Cont. The second order condition are more comple where you have to eamine the second derivative of each of the variables, as well as, the cross derivative. 30

31 Cross Partial Derivative Suppose you have a function y f(, ), then the cross partial derivative can be represented as: 3 ), ( ), ( ), ( y f f y f f y f f j i j i j i

32 Cross Partial Derivative Eample Suppose you have a function y f(, ) + +3, then: 3 y y

33 Constrained Optimization Constrained Optimization is said to occur when one or more of the variables in the objective function is constrained by some function. Hence a constrained optimization problem will have an objective functions and a set of constraints. 33

34 Constrained Optimization Cont. The Constrained Optimization problem where you are trying to maimize can be set-up as the following: Maimize an objective function f() with respect to given a set of constraints g()c. ma w. r. t. f ( ) subject to g( ) c 34

35 Eample of Constrained Optimization Suppose that you want to maimize f() 5 -, subject to the constraint that. Since is the constraint, the answer to this is trivial where *. 35

36 Constrained Optimization: Two Variables The Constrained Optimization problem where you are trying to maimize can be set-up as the following: Maimize an objective function f(, ) with respect to, given a set of constraints g(, ) c. ma w r. t.,. f (, ) subject to g(, ) c 36

37 Eample of Constrained Optimization: Two Variables Suppose you want to maimize y f(, ) with respect to and given that To solve this problem, we can turn this constrained problem into an unconstrained problem. 37

38 Eample of Constrained Optimization: Two Variables Cont. If we solve the constraint for as a function of we get 600. Plugging into the objective function gives the following new unconstrained maimization problem. Maimize (600 ) w.r.t.. The first order condition is Which implies *50. Which implies *

39 Motivating the Lagrange Method In the previous problem, we made a substitution to turn the constrained optimization problem into an unconstrained problem. While this made solving the problem easier, there may be times when you have multiple constraints or potentially inequality constraints that make changing the constrained into the unconstrained difficult or impossible. 39

40 Motivating the Lagrange Method Cont. Another way of solving the above problem is using Lagrange s method. The Lagrange method uses what is called a Lagrange multiplier λ to transform the problem. The Lagrange multiplier tells us how much will be added to the optimization problem if the constraint was relaed by one unit. 40

41 4 Setting-Up the Lagrange The Constrained Optimization problem where you are trying to maimize can be set-up as the following: Maimize an objective function f(, ) with respect to and given a set of constraints g(, ) c. )), ( ( ), ( ),, ( minimum problem a If ) ), ( ( ), ( ),, ( maimum problem a If g c f L c g f L + + λ λ λ λ

42 Solving the Lagrange Problem To solve the Lagrange problem, you need to optimize L(,, λ) with respect to,, and λ. This is equivalent to using the first and second order conditions. 4

43 43 Eample of Lagrange Suppose you want to maimize y f(, ) with respect to and given that This implies: L 0 L 0 L First Order Conditions 600) ( ),, ( L λ λ λ λ λ