In other words, we want to find the domain points that yield the maximum or minimum values (extrema) of the function.

Transcription

1 1

2 The Lagrange multipliers is a mathematical method for performing constrained optimization of differentiable functions. Recall unconstrained optimization of differentiable functions, in which we want to find the extreme values (max or min values) of a differentiable function. In other words, we want to find the domain points that yield the maximum or minimum values (extrema) of the function. We determine the extrema of by first finding the function s critical domain points, which are points where the gradient (i.e., each partial derivative) is zero. These points may yield (local) maxima, (local) minima or saddle points of the function. We then check the properties of the second derivatives or simply inspect the function values to determine the function s extreme values. 2

3 In constrained optimization of differentiable functions, we still have a differentiable function we want to maximize or minimize. But, we have restrictions on the domain points that we can consider. The set of these points is called the feasible region, and they are given by a constraint function, typically formulated as. Example: consider an inverted paraboloid as the function to maximize, constrained by a set of points defined by a line in the x-y plane. Function s value at each of the domain points chosen from the constraint curve. Consider only the domain points that are on the constraint curve. 3

4 Example: Constrained Maximization Suppose you want to maximize constraint (See ), subject to the 4. Function's maximum constrained value:,. 1. The feasible set consists of points on the unit circle, plotted in the x-y plane. 3. Function s value for each point of the feasible set. 2. Feasible set, dropped for visual clarity. 4

5 We can solve this constrained optimization problem using Calculus and substitution. First, we write the formal expression for the constrained optimization (maximization): Solution by substitution: Now, set 1 st derivative to zero, and find critical points: Substituting the critical point, as determined by inspection, into : 5

6 Lagrange Multiplier Method Outline We now consider an alternative way to perform constrained optimization of differentiable functions, called the Lagrange multiplier method, or just the Lagrangian. We will give the formulation of the basic Lagrange multiplier method and a procedure of how to solve it. This will be followed by an intuitive derivation of the basic Lagrange equations. In addition, an intuitive derivation of the generalized Lagrange multiplier method will be given. Finally, the primal and dual forms of the Lagrange multiplier method will be given. The primal and dual forms offer equivalent methods for performing constrained optimization. 6

7 (Basic) Lagrange Multiplier Method Consider a basic constrained optimization (maximize of minimize) problem:, The formulation of the Basic Lagrange Multiplier method for constrained optimization problem is: We set the partial derivatives of the Lagrangian to zero, and then find the optimal values of the variables that maximize (or minimize) the function. The is called the Lagrange multiplier. 7

8 , Why does the Basic Lagrange Multiplier equation include? We know from basic calculus that setting the function s derivative to zero yields the function s critical points, and then, from the critical points, we can determine the extrema of the function. The constraint function must also be satisfied, and setting the partial derivative gives the constraint, and so, there is some intuition for including in the Basic Lagrange equation. The set of simultaneous equations ensures that only the points satisfying the constraint may be chosen as critical points. 8

9 More Motivation for Including To obtain an intuitive appreciation for why the Lagrange is formulated as such, consider a contour plot of the previous optimization example: Contour plot (-2,2,0) of. Surface plot of. (2,-2,0) These are level curves of. 9

10 Gradient of Note that the gradient of is perpendicular to the level curves of and points in the direction of the maximum rate of change of the function : Note that this direction is given in the plane. Gradient is Parallel with the diagonal in the x-y axis. 10

11 Contour of and Now consider the contour of and the constraint level curve : Contour plot of. Standard Form Constraint level curve:. 11

12 Gradient of The constraint is a level curve of the paraboloid, but plotted in the plane. The gradient of the level curve is perpendicular to the level curve and points in the outward direction.,, Constraint level curve:. 12

13 At Solution Points Informally, notice that, the slope of the tangent line of the contour of (note: the tangent line is the contour line itself) is equal to the slope of the tangent line of the constraint level curve at the critical points, Also, informally, note that at the intersection point of any other contour line of and the level curve, the slope of their tangent lines appear to be different. Contour line. Point on contour line:. Contour line. Their slopes appear to be the same. Constraint level curve:. 13

14 close all; d=linspace(-2,2,1000); [x,y]=meshgrid(d,d); figure; hold all; grid on; contour(x,y,x+y,50); surf(x,y,x+y), shading interp; theta = linspace(0,2*pi); [x1,y1] = pol2cart(theta,1.0); plot3(x1,y1,x1+y1,'linewidth',2,'color','k'); plot3(sqrtm(2)/2.0,sqrtm(2)/2.0,sqrtm(2), 'bo', 'markerfacecolor','b', 'markersize',6); set(gca, 'PlotBoxAspectRatio', [ ]); set(gca,'gridalpha', 0.5, 'GridLineStyle', '--', 'FontSize',22); set(gca,'zlim',[-6 4],'XLim',[-2 2],'YLim',[-2 2]); set(gca,'xtick',[-2:1:2], 'YTick',[-2:1:2], 'ZTick',[-6:2.0:4]); colormap 'jet'; view(49,12) % adjust view so it is easy to see 14

15 Aside: Derivative Interpretation The slope of the tangent line of a curve gives the direction we should travel to stay on the curve. Recall the definition of the derivative (evaluated at : rise run In the limit as at a point, the function must become linear at point ; otherwise the function would not be differentiable. This is because the only way the function would not be linear as at point, would be if the function had a corner at point, and, if so, the function would have an infinite derivative at point, and would not be differentiable at point. Starting from the point at which the derivative is taken,, the slope of the tangent line gives the rise and the run we should take to get to the next infinitesimally closest point of the function, and hence stay on the function. 15

16 Moving along the Tangent Line Imagine you are at one of the points (in this example, or ), and you make an infinitesimally small step along the constraint curve, and along the tangent line of at the point. Your movement will keep you on the constraint curve, since, the slope of the tangent line of a curve gives the direction we should travel to stay on the curve. Constraint level curve:. 16

17 Your infinitesimally small movement will cause one of two possible outcomes: Move parallel with and along a level curve of For example: as shown at point, you will move parallel with and along Cross over a level curve For example: as shown at point, you will move across :, or 17

18 Consider Intersection Point Where Tangents are Different In this case, a movement to the right will cross the level curve, and will touch another level, which represents an increase in, i.,e.,. 18

19 Cannot be an Extrema Point Since your movement touches another level curve, while staying on the constraint curve (i.e., your movement takes you to a valid point in the feasible region on the constraint curve, this means the point under consideration,, cannot be a maximum point, because the function s value at the new point of intersection is. In other words, starting from, and moving along the feasible region, one finds another point in the feasible region that has a greater function value. Therefore, is not an extreme point. 19

20 Tangent Lines are Different at Notice that the tangent line of the level curve is different than the tangent line of the constraint level curve at point. Also, note that if you were to move along the tangent line of the level curve, your movement would take you off the constraint level curve, and hence take you out of the feasible region, i.e.,. Tangent of Tangent of at at 20

21 Conclusion Regarding Intersection Point In general, point cannot be a critical point if the slope of the tangent line of the constraint level curve is different than the slope of the tangent line of the objective level curve at intersection point. If the slopes are different at an intersection point, then that point cannot be an extremum. 21

22 Considering Touching Point Where Tangents are Equal Previously, we considered a point where the tangent of the constraint level curve and the tangent of a level curve of were different. Next, consider where your movement takes you in the case of a touching point, where the tangents are equal. Tangent of Tangent of at 22

23 Considering Touching Point Tangents are Equal Once again, consider moving along the tangent line of the constraint curve, but now in more detail. Say, you took an infinitesimally small step from to along the tangent line of the constraint curve, at the touching point. Tangent of at 23

24 Considering Touching Point Tangents are Equal This infinitesimally small step from to along the tangent line keeps you on the constraint curve, staying in the feasible region. Also, since the tangents are equal, you can move along the objective level curve using the same step. This keeps you on the objective level curve, and does not change the value of the function,. Tangent of Tangent of at 24

25 Touching Point is a Solution Tangents are Equal Therefore, a local extremum occurs at point where the tangent of is equal to a level curve of, because we assume an extremum exists, and the slopes can either be different or same. Finally, we have shown that when the slopes are different at a point #, that point of intersection cannot be an extremum. Therefore, an extremum exists at point where the slopes are the same. In general, if the slopes of the tangents at a touching point of the constraint level function and an objective level curve are equal, then the touching point is a critical point in the constrained extremum problem. In other words; In a constrained maximization or minimization problem, we are constrained to finding an extremum point of considering only those points that satisfy the constraint. The extreme value occurs at a point, where the objective level curve touches, but does not cross, the constraint level curve. At that point, the tangent of the constraint level curve is equal to the tangent of the objective level curve. At this point the slopes touch but do not cross. At this point the extreme value of the function is. 25

26 Next, we use the fact that at a solution point, the slope of the tangent line of the constraint level curve is equal to the slope of the tangent line of the objective level curve, to derive the Lagrange multiplier constrained optimization equation. We will show that the gradient of the objective function can be written as a scalar multiple of the gradient of the constraint function, i.e., 26

27 Known: the slope of the tangent line of the constraint level curve is equal to the slope of the tangent line of the objective level curve at the critical point. Their normal vectors are parallel. Known: the gradient of the objective function is perpendicular to the tangent line of the objective level curve at the critical point. Therefore, the gradient is the normal vector of the objective level curve at the critical point. Known: the gradient of the constraint function is perpendicular to the tangent line of the level curve at the critical point. Therefore, the gradient is the normal vector of the level curve at the critical point. This means the gradient of the objective function is related to the gradient of the constraint function through a scalar multiple. Note: can be either positive or negative. Tangent at Normal vector Therefore, we can write that: for negative for positive 27

28 At a non-solution point, the tangent of the constraint curve is not parallel to the tangent of the objective level curve. At a solution point, the tangent of the constraint curve is parallel to the tangent of the objective level curve. At a solution point, since the tangents are parallel, the normal of the constraint level curve and the normal of the objective level curve are also parallel. This means that at the solution point, the gradient of the objective function is either parallel or anti-parallel to the gradient of the constraint level curve. This means is related to through a scalar multiple. Note: can be either positive or negative. Therefore, we can write that: Tangent at Normal vector for negative for positive 28

29 Thought Experiment: Exhaustive Search of Extrema Do the following for every point on the constraint level curve, i.e., for every point in the feasible region: Imagine you are at a point of the constraint level curve. You take note of the value of the objective level curve at point. You take an infinitesimally small step on the curve to get to point, and so you stay on the curve, i.e., you stay in the feasible region. You take note of the value of the objective level curve at point. If the value of the objective level curve at point is different than the value at point, then point cannot be an extremum. You will note that the slopes of the tangents lines of the two functions and at the point are different. If the value of the objective level curve at point is the same as the value at point, then point is an extremum. You will note that the slopes of the tangents lines of the two functions and at the point are the same. 29

30 Lagrange Optimization Equation The above can be written as: Undoing the differentiation and removing the setting to zero procedure: Since can be either positive or negative, we can now write the Lagrangian: The is called the Lagrange multiplier. 30

31 Lagrange Minimization/Maximization Procedure We set the partial derivatives of the Lagrangian to zero, and then find the optimal values of the variables that maximize (or minimize) the function. Notice that the above three equations comprise the gradient equation derived earlier: And that the last equation extracts the constraint 31

32 (Basic) Lagrange Multiplier Method Consider a basic constrained optimization (maximization or minimization) problem:, The formulation of the basic Lagrange constrained optimization problem is: We set the partial derivatives of the Lagrangian to zero, and then find the optimal values of the variables that maximize (or minimize) the function. The is called the Lagrange multiplier. 32

33 Example Lagrange Computation, The formulation of the basic Lagrange constrained optimization problem is: Now, setting the partial derivatives to zero, and finding critical points: Substituting the critical points into, we find:, 33

34 Graphical Interpretation of a Solution Consider a basic constrained optimization (maximization or minimization) problem:, Assume a constrained solution exists. A solution can exist only in the feasible region, i.e., for points only on the constraint curve. For a point in the feasible region, the tangent of the constraint curve and the tangent of the objective level curve will either intersect or touch (cross or not cross); there is no other alternative. Tangents intersect (cross) Tangents touch (do not cross) On the one hand, at a point where the tangents cross, the point is not a critical point, because if you take an infinitely small step along the constraint curve, you stay on the constraint curve, but you meet an either higher or lower level curve of ; therefore, that point is not an extremum. On the other hand, at a point where the tangents touch, but do not cross, the point is a critical point, because if you take an infinitely small step along the constraint curve, you stay on the constraint curve, and you stay on the level curve of the objective function ; therefore, that point must be an extremum. For example, in 2D, we can visualize a solution by drawing the level curves of and the level curve of, and observing at which point(s) the tangents touch but do not cross (i.e., do not intersect). A solution exists where the tangents touch but do not cross. 34

35 Lagrangian with Multiple Constraints Consider an optimization problem with multiple constraints :, The feasible region is the set of points at which the functions intersect. The Lagrangian may be written as: Note that the following partial derivatives reveal the constraints: The gradient of is the vector sum of scalar multiples of the gradients of the constraint curves. 35

36 Lagrange with Inequality Constraints Consider an optimization problem with an inequality constraint : The Lagrangian with inequality constraint is written as: Notice that the equation appears very similar to the Lagrange multiplier method with equality constraint, except that the is constrained to be. Why should the multiplier with inequality constraints be limited to 36

37 Lagrange with Inequality Constraints To show intuitively why this must be the case, first consider the possibilities: 1. No solution exists and the lines of constant and feasible region do not touch or intersect. 2. A solution exists at the boundary of the feasible region. 3. A solution exists inside the feasible region. In the interesting case where a solution exists, we will show there will be two cases:.. Consider a maximization example with two variables and one inequality constraint. To maximize subject to, let us first look at the boundary of the region allowed by the inequality, i.e.,. 37

38 Lagrange with Inequality Constraints: Solution on Boundary Consider a sketch of the level curve and the level curve We assume a solution exists at point This means that some level curve touches. Then, we assert that and must be parallel (not anti-parallel), and this point will give a maximum (not a minimum) of for the region, because of the following argument: Feasible Region 38

39 Solution on Boundary: Gradient Points Away From Feasible Region At the point, where and touch, the gradient would be pointing away from the feasible region, since: The gradient always points in the direction of maximum increase of a function, and The function decreases as you move towards the inside of the feasible region. Feasible Region 39

40 Solution on Boundary: and Point in Same Direction Now, let s determine the direction of : If and were pointing in opposite directions, then would be pointing inwards towards the feasible region, meaning would have greater values inside the feasible region. Feasible Region If we were to find another point where touches inside the feasible region, we would find an increase in ; since we are still in the feasible region, we must conclude that the point on the boundary cannot be a critical point (not a solution). This contradicts our initial assumption that is a solution. 40

41 Solution on Boundary: and Point in Same Direction Therefore, and must be parallel and be pointing in the same direction (i.e., not anti-parallel). This implies. Feasible Region 41

42 Solution on Boundary: Gives a Maximum of The gradient indicates the function increases away from the feasible region. If a solution exists on the boundary, then this solution must be a maximum, due to the feasible region being at its outer limit. Feasible Region 42

43 Solution on Boundary: Effective and Binding Constraint In the case where a critical point exists on the boundary, the inequality constraint is said to be effective and is called a binding constraint, and Feasible Region 43

44 Summary for Solution on Boundary Consider a sketch of the level curve and the level curve We assume a solution exists at point where touches. Then, and must be parallel (not anti-parallel), and this point will give a maximum (not a minimum) of for the region, because of the following argument: At the point, where, and, touch, the gradient would be pointing away from the feasible region, since, as you move towards the inside of the feasible region. If and were pointing in opposite directions, then would be pointing inwards towards the feasible region, meaning would have greater values inside the feasible region. If we were to find another point where, touches, inside the feasible region, we would find an increase in ; since we are still in the feasible region,, we must conclude that the point on the boundary cannot be a critical point (not a solution). This contradicts our initial assumption. Therefore, and must be parallel and be pointing in the same direction (i.e., not anti-parallel). Therefore, at the critical point on, and must be parallel (not anti-parallel) This implies that 0for. In the case where a critical point exists on the boundary, the inequality constraint is said to be effective and is called a binding constraint, and 44

45 Lagrange with Inequality Constraints: Solution Within Boundary In the case where a critical point exists inside the feasible region, i.e.,, then we can consider any point within the feasible region to determine the extrema of. I.e., the problem is unconstrained, if we assume a solution exists within the feasible region. In other words, the optimizer will not find a solution at the boundary, because we assume a solution exists within the boundary; and within the boundary the optimizer can consider any point, i.e., it is unconstrained within the boundary. This implies to remove the constraint. In this case we say the constraint is not binding, or the constraint is ineffective. The maximum is then found by looking for the unconstrained maximum of, assuming that we look only inside the feasible region. In this case: 45

46 Lagrange with Inequality Constraints: Summary The Lagrangian with inequality constraints can be written as: If the extremum occurs at the boundary of the constraint (the constraint is binding and effective): If the constraint is not binding and ineffective, then the above reduces to: I.e., unconstrained optimization. Argument: We assume that there is a solution, and that the solution does not exist at the boundary. Therefore, any critical point the optimizer finds must be inside the feasible region, due to the constraint. Finally, the optimizer is free to choose any point within the feasible region to find a solution. 46

47 The Lagrange optimization method has a dual form, one called the primal optimization method and the other called the dual optimization method. In some applications it is more suitable to use the dual optimization method, as it leads to a simpler and quicker solution, while in other applications, the primal method is better. In the following we show that under certain conditions, the primal and dual optimization methods are equivalent and lead to the exact same solution to an optimization problem. As an example use of the primal and dual optimization methods being equivalent, we can show that the condition, is also true for a minimization problem. Note that our intuitive verification for the condition, was based on the assumption of a maximization problem. 47

48 Lagrange Optimization: Basic Formulation Consider an optimization problem of the following form: The basic Lagrange formulation (Lagrangian) for this problem is: The are called the Lagrange multipliers for equality constraints. We would then find and set s partial derivatives to zero: Finally, solve for and, and then locate the minima. 48

49 Lagrange Optimization: Generalized Formulation Consider the following, which is called the primal optimization problem: The generalized Lagrangian is given by: The and are called the Lagrange multipliers. 49

50 Deriving an Alternate Expression for the Primal Optimization Problem We will now derive an alternative expression for the primal optimization problem. We call this the min max expression for the primal optimization problem. 50

51 Min Max Expression for Primal Optimization Problem Consider the following quantity: If the choice of violates any of the primal constraints (below), then : For instance, if, then can be chosen as to maximize, and therefore,. In addition, if, then can be chosen as to maximize, and therefore,. 51

52 Min Max Expression for Primal Optimization Problem Now, if the choice of satisfies the primal constraints (below), then : For instance, if, then the value of is irrelevant, since, irrespective of. In addition, if, then will be chosen as to maximize. Taken together,. 52

53 Min Max Expression for Primal Optimization Problem Now, if the choice of satisfies the primal constraints (below), then : Note also that if were allowed to be negative, then would be chosen as to maximize, in which case, and so we would not have a solution, even for good ). This provides further reason for requiring that. 53

54 Min Max Expression for Primal Optimization Problem Therefore: Next, consider the minimization problem: This means that, after performing the maximization of, which we found to be, given that the constraints are satisfied, then minimize the resulting function by finding the optimal value of. I.e., given that the constraints are satisfied: 54

55 Min Max Expression for Primal Optimization Problem In other words, optimization problem. is the same as our original primal Min max representation Original primal optimization problem Finally, define the optimal value of as : We call this the value of the primal optimization problem. 55

56 Deriving a Dual Expression for the Primal Optimization Problem We will now derive the dual expression of the generalized Lagrange optimization formulation. We call the dual expression the Max min expression. We will relate the dual expression to the primal expression, and hence show that the dual expression can also be used to express the generalized Lagrange optimization formulation. Finally, we will show that under certain conditions, the dual expression is equivalent to the primal expression, and thus, either of them can be used to solve an optimization problem posed as a generalized Lagrange optimization formulation. 56

57 The Dual Max Min Expression Consider the quantity: Note that, whereas in the definition of (below) we were optimizing (maximizing) with respect to and, here (above) we are minimizing with respect to. 57

58 The Dual Max Min Expression Now, add a maximization term: This is exactly the same as our primal problem (below), except that the order of the max and the min are now exchanged. Finally, define the optimal value of as : We call this the value of dual optimization problem. 58

59 Primal and Dual Relationship It can be shown that (see next 2 slides): Furthermore, it can be shown that, under certain conditions: This means that under certain conditions, we can solve a given optimization problem by using either the primal or dual methods, and we ll pick the most suitable one. 59

60 2-D Case PROOF: Since we can chose any, we can write on the LHS that. On the RHS, since this implies: implies that we can choose a that minimizes the RHS: 60

61 What does the following mean? For each, we find a that minimizes the function. This will generate answers (i.e., minimums of ), one for each value of. Does the following make sense then? Yes, because for each value of you choose, the term represents the minimum value of the function over all. 61

62 Our Case PROOF: implies that we can choose a combination that maximizes both sides, even with the restriction : implies that we can choose a that minimizes the RHS: 62

63 63

64 Recall: Lagrange Optimization Generalized Formulation Consider the following, which is called the primal optimization problem: The generalized Lagrangian is given by: The and are called the Lagrange multipliers. 64

65 Conditions for Equality : Given without Proof Let,, be the optimal domain points that optimize the objective function: Under certain assumptions, there must exist,, so that:. Once determined,,, will satisfy the Karush-Kuhn-Tucker (KKT) conditions, which are as follows: 65

66 The first two conditions (below) follow from the Lagrange optimization procedure. We set the partial derivatives of to zero, and solve for the variables : Of course, when we solve for the variables and find the optimal values, they should satisfy the following two KKT conditions: 66

67 The last two conditions (below) follow from the initial constraints of the problem. The is the initial constraint: Of course, when we solve for the variables and find the optimal values, should satisfy : 67

68 The third condition (below) follows from the analysis of the primal optimization problem. Recall that in the derivation of the primal optimization problem we wanted to perform the maximization: For the above term to be maximum, and, it is required that, subject to. Furthermore: We note that in the case of strict feasibility (i.e., when is active and a solution exists at ), the condition is satisfied and also is free to vary. When is inactive and, then. 68

69 [1] J. Kitchin, "Matlab in Chemical Engineering at CMU," [Online]. Available: [Accessed 20 February 2015]. [2] C. A. Jones. [Online]. Available: [Accessed 23 February 2015]. [3] D. Klein, "Dan Klein's Homepage," [Online]. Available: [Accessed 23 February 2015]. [4] J. Beggs, "Introduction to the Lagrange Multiplier," [Online]. Available: [Accessed 2 March 2015]. 69