Local and Global Minimum

Transcription

1 Local and Global Minimum Stationary Point. From elementary calculus, a single variable function has a stationary point at if the derivative vanishes at, i.e., 0. Graphically, the slope of the function is zero at the stationary point, which may represent a minimum, a maximum, or a point of inflexion. Local Minimum. A multi variable function,, has a local minimum at if in a small neighborhood around, defined by. Global Minimum. The multi variable function has a global minimum at if for all in a feasible region defined by the problem.

2 Necessary and Sufficient Conditions The necessary conditions must be satisfied at the optimum point. However, other points (maxima, inflexion points) may also satisfy the necessary conditions. If a candidate point satisfies the sufficient condition, then it is indeed the optimum point. However, not being able to satisfy the sufficient conditions does not preclude the existence of an optimum point.

3 Extreme Value Theorem The Extreme Value Theorem (attributed to Karl Weirstrass) provides sufficient conditions for the existence of minimum (or maximum) of a function defined over a complex domain. The theorem states: A continuous function defined over a closed and bounded set attains its maximum and minimum in. According to this theorem, if the feasible region Ω of the problem is closed and bounded, a minimum for the problem exists.

4 Optimality Criteria: Unconstrained Problems Consider a multi variable function,,,, where we wish to investigate the behavior of a candidate point. The point is a local minimum of only if in a small neighborhood of. Let, and use first order Taylor series expansion of to write: 0. Since are arbitrary, the first order necessary condition (FONC) for a local minimum is given as: FONC: If has a local minimum at, then equivalently, 0, 1,,. 0; The points that satisfy FONC are called stationary points of. Besides minima, these include maxima and the points of inflexion.

5 Polynomial Data Fitting Problem: Fit an th degree polynomial: to data points:,, 1,,, such that the mean square error (MSE) is minimized min 1 2 FONC: 0 For 1, 1 Finally, since the problem is convex, FONC are both necessary and sufficient for a minimum.

6 Second Order Conditions Assume now that FONC are satisfied, i.e., 0. Then, we may use second order Taylor series expansion of to write the optimality condition as: 0. The above quadratic form is positive (semi)definite if and only if the Hessian matrix,, is positive (semi)definite. Therefore, the second order necessary condition (SONC) is stated as: SONC: If is a local minimizer of, then 0. A stronger second order sufficient condition is given as: SOSC: If satisfies 0, then is a local minimizer of. In the event that 0, the lowest nonzero derivative must be even ordered for stationary points (necessary condition), and be positive for local minimum (sufficient condition).

7 Optimality Criteria: Equality Constrained Problems Consider an optimization problem with a single equality constraint: min,, subject to: 0 Consider the variation in the objective and constraint functions at 0 0 Or 0 Define a Lagrangian function:,, then FONC: 0; 1,, 0

8 Equality Constrained Problems Consider a problem with multiple equality constrains: min,, subject to: 0, 1,, Define a Lagrangian function:, FONC: 0, 1,, 0; 1,, Or, equivalently,, 0,, 0 A total equations, need to be simultaneously solved for, 1,,and, 1,, Since the equality constraints can be multiplied by 1 without changing the solution, the Lagrange multipliers for the equality constraint can take on both positive and negative values.

9 Example: soda can design:, subject to: 2000 min,

10 Example: soda can design:, subject to: 2000 min, Define,, FONC: FONC are solved as: 6.34, 0.63.

11 Example: soda can design Alternatively, use the equality constraint to solve for as: Define the unconstrained problem: min FONC: 0 FONC are solved to get: SONC: ,, 200

12 Example min,, Subject to:, 10 Define,, 1 FONC: 2 1 0,2 1 0, 10 Solution:,,, with,, Alternatively, substitute 1 Solve unconstrained problem: min 1 1 Solution: ;,,, with,

13 Inequality Constrained Problems Consider the inequality constrained problem: min,, subject to: 0, 1,, Introduce slack variables to convert the constraints to equality: 0, 1,, Define the Lagrangian:,, FONC: 0, 0, 1,, 0; 1,, Note, 0, 1,,define switching conditions of the form: 0or 0. These require a total of 2 cases to be explored for feasibility and optimality.

14 Example: soda can design, subject to: min, Use slack variable to write: Define,, FONC: For 0we obtain No feasible solution for 0 0 ;

15 Example min,, Subject to:, 10 Use slack variable:, 1 0 Define,,, 1 FONC: 2 0, 2 0, 10, 0 For 0, we obtain:, 0,0, 0. For 0, we obtain:,,, with, and,

16 Optimality Criteria: General Optimization Problems Consider the general optimization problem: min, subject to: 0, 1,, ; 0,,, Add slack variables and define the Lagrangian function:,,, FONC (KKT): Gradient: 0; 1,, Feasibility: 0, 1,,; 0, 1,, Switching: 0, 1,, Non negativity: 0, 1,, Regularity : for those 0, are linearly independent Note, 2variables; 2 cases

17 Example: soda can design min,, subject to: 0, 20

18 Example: soda can design min,, subject to: 0, 20 Use slack variable to write: 2 0 Define,,, 2 FONC (KKT): For 0we obtain:, 2 ;, ; No feasible solution for 0

19 Second Order Conditions for Constrained Problems Assume that satisfies the FONC (KKT) conditions, Hessian of the Lagrangian: Define the set of active constraints: : 0, 0 Define active constraint tangent hyperplane as: : 0, 0, SONC: If is a local minimizer of, then 0 SOSC: For : 0, 0, 0, if satisfies 0, then is a local minimizer of SOSC: If 0, then is a local minimizer of

20 Consider min,, Subject to:, : 1 0;, : 0 Lagrangian:,,,, 1 FONC (KKT): 2 0, 2 0, 0, 10, 0 Solution:,,, 1, 0; ;, Active constraint tangent hyperplane: 0 Hessian of the Lagrangian at, 0 1 : 1 0 SOSC: 2 0, indicating an optimum

21 Convex Optimization Problems Example: min,, Subject to:, : 1 0;, : 0 Note: 0, 0; is linear; hence, problem is convex. Define,,,, 1. KKT: 2 1 0, 21 0, 0, 10, 0 For 0:,,, 1, ;, NFS for 0 SOSC: 0

22 Example: design of rectangular beam: min, Subject to: :. 100, :., 20 : 2 0, : 0, : 0 Lagrangian:,,, FONC (KKT): , 0, 0, 0; 15 Note, no feasible solution for 0or 0

23 Drop, to write:,,, FONC (KKT): , 0, 0, 0; 1 3 Solution cases: Results , 0; , 0; , ; NFS NFS , ; NFS

24 The Hessian of the Lagrangian evaluates as: / /. The constraint tangent hyperplane for the active constraint is defined by: / / 0, or 1 /. The SONC evaluate as: 0, indicating there is no isolated minimum for the problem The case 0, 0, 0 above generates a family of optimal solutions, each with These solutions are bounded as: , , and , with associated These constitute the global optimum for the problem

25 Post Optimality Analysis We are interested in studying the change in the objective function value resulting from relaxation of constraints. Consider the perturbed optimization problem: min, subject to, 1,,;,,, Let the optimum solution for the perturbed problem be expressed as:,, with the optimal cost:, ; then, ; The non zero Lagrange multipliers accompanying active constraints determine the cost function sensitivity to constraint relaxation. Non active constraints have zero Lagrange multipliers, and hence, they do not affect the solution.

26 Example: soda can design min,, subject to: 0, 20 Optimum solution:, 2 ; ; Define the perturbed problem min,, subject to:,, 2 Variation in the cost function,, 0,0 6 For 0.1,, 10 3

27 Example: min,, Subject to:, : 1 0;, : 0 A local minimum for the problem exists at:, 0.786,0.618, 0.527, 0.134, Define the perturbed optimization problem: min,, Subject to:, : 1 ;, : The variation in the optimal solution is given as:, 0,0. Then, for 0.1, the new optimum is: Similarly, for 0.1, the new optimum is: 0.35.