Introduction to optimization

Transcription

1 Introduction to optimization G. Ferrari Trecate Dipartimento di Ingegneria Industriale e dell Informazione Università degli Studi di Pavia Industrial Automation Ferrari Trecate (DIS) Optimization Industrial Automation 1 / 17

2 Optimization Optimization is also known as mathematical programming Programming means planning or building an action plan for solving a problem or tacking a decision Optimization falls in the fields of operations research and management science. Ferrari Trecate (DIS) Optimization Industrial Automation 2 / 17

3 Optimization Basic problem Variables: x = [x 1,..., x n ] T min f (x) g i (x) 0 i=1,2,...,m Constraints: g i : R n R, i = 1, 2,..., m. Feasible region X = {x R n : g 1 (x) 0,, g m (x) 0} Feasible solution or feasible point: x X Objective function (or cost): f : X R Ferrari Trecate (DIS) Optimization Industrial Automation 3 / 17

4 Optimization Basic problem min f (x) g i (x) 0 i=1,2,...,m x X is an optimal solution (global minimum point) if f (x ) f (x), x X x X is a local optimal solution (local minimum point) if ɛ > 0 : x X, x x < ɛ f ( x) f (x) f (x) 0 x 1 x 1 ε x 1 + ε x 2 x 2 ε x x 2 + ε Ferrari Trecate (DIS) Optimization Industrial Automation 3 / 17

5 Optimization Basic problem The basic problem can be: infeasible (if X = ) min f (x) g i (x) 0 i=1,2,...,m unbounded (if k < 0 x X : f (x) < k). even if the basic problem is feasible and bounded, optimal solutions could not exist; e.g. min x 0 ex x R e x exist and be not unique (e.g. f costant) 0 x Ferrari Trecate (DIS) Optimization Industrial Automation 3 / 17

6 Optimization Basic problem min f (x) g i (x) 0 i=1,2,...,m No easy way to solve the basic problem in its full generality! Need of numerical algorithms Often, only local optimal solutions can be computed Ferrari Trecate (DIS) Optimization Industrial Automation 3 / 17

7 Optimization Maximum problems max f (x) g i (x) 0 i=1,2,...,m the problem is unbounded if k > 0 x X : f (x) > k. x X is an optimal solution (global maximum point) if f (x ) f (x), x X x X is a local optimal solution (local maximum point) if ɛ > 0 : x X, x x < ɛ f ( x) f (x) Ferrari Trecate (DIS) Optimization Industrial Automation 4 / 17

8 Conversions in the basic problem form Conversions maximum/minumum max f (x) = min f (x) x X x X 0 x Optimal solutions are the same for both problems v -v f(x) -f(x) Ferrari Trecate (DIS) Optimization Industrial Automation 5 / 17

9 Conversions in the basic problem form Conversions maximum/minumum max f (x) = min f (x) x X x X 0 x Optimal solutions are the same for both problems Conversion form to in the constraints {x R n : g(x) 0} = {x R n : g(x) 0}; The feasible region does not change v -v f(x) -f(x) Ferrari Trecate (DIS) Optimization Industrial Automation 5 / 17

10 Conversions in the basic problem form Conversions maximum/minumum max f (x) = min f (x) x X x X 0 x Optimal solutions are the same for both problems Conversion form to in the constraints {x R n : g(x) 0} = {x R n : g(x) 0}; The feasible region does not change Conversion from = to inequalities in the constraints {x R n : g(x) = 0} = {x R n : g(x) 0, g(x) 0}; An equality constraint is replaced by two inequality constraints v -v f(x) -f(x) Ferrari Trecate (DIS) Optimization Industrial Automation 5 / 17

11 Classes of optimization problems Basic problem min f (x) g i (x) 0 i=1,2,...,m f is quadratic if f (x) = x T Qx + c T x (Q matrix, c vector) f is linear if f (x) = c T x f is affine if f (x) = c T x + b (b constant) Ferrari Trecate (DIS) Optimization Industrial Automation 6 / 17

12 Classes of optimization problems Basic problem min f (x) g i (x) 0 i=1,2,...,m f is quadratic if f (x) = x T Qx + c T x (Q matrix, c vector) f is linear if f (x) = c T x f is affine if f (x) = c T x + b (b constant) Notable problems for which efficient algorithms exist f is convex and g i are convex convex programming if f is quadratic and g i are affine quadratic programming if f is linear and g i are affine linear programming If the variables must also verify x Z n we have an integer programming problem (mixed-integer programming problem if only a subset of variables is constrained to integer values) Ferrari Trecate (DIS) Optimization Industrial Automation 6 / 17

13 Convex programming Definition Given two points x, y R n, the set is a segment joining x and y. xy = {λx + (1 λ)y : λ [0, 1]} x 2 y xy x 0 x 1 Definition The set X R n is convex if x, y X one has xy X. Ferrari Trecate (DIS) Optimization Industrial Automation 7 / 17

14 Examples convex not convex A B A B polyhedron not convex (without the boundary) convex R n is convex Ferrari Trecate (DIS) Optimization Industrial Automation 8 / 17

15 Convexity and intersection Proposition (try to prove it at home!) The intersection of two convex sets is a convex set It follows that the empty set is convex Ferrari Trecate (DIS) Optimization Industrial Automation 9 / 17

16 Convexity and intersection Proposition (try to prove it at home!) The intersection of two convex sets is a convex set It follows that the empty set is convex A B A B A B A B convex A B = convex A B not convex A B convex Ferrari Trecate (DIS) Optimization Industrial Automation 9 / 17

17 Convexity and intersection Proposition (try to prove it at home!) The intersection of two convex sets is a convex set It follows that the empty set is convex A B A B A B A B convex A B = convex A B not convex A B convex The union of two convex sets is not convex, in general Ferrari Trecate (DIS) Optimization Industrial Automation 9 / 17

18 Convex functions Definition A function f : X R on a convex set X R n is convex if x, y X and λ [0, 1] one has f (λx + (1 λ)y) λf (x) + (1 λ)f (y) Ferrari Trecate (DIS) Optimization Industrial Automation 10 / 17

19 Convex functions Definition A function f : X R on a convex set X R n is convex if x, y X and λ [0, 1] one has f (λx + (1 λ)y) λf (x) + (1 λ)f (y) f (x) f (x) 0 x f (x) 0 x f (x) 0 x 0 x Ferrari Trecate (DIS) Optimization Industrial Automation 10 / 17

20 Convexity and smoothness Convexity and continuity A convex function f : X R, X R n is continuous in the interior of X. f (x) 0 1 x Ferrari Trecate (DIS) Optimization Industrial Automation 11 / 17

21 Convexity and smoothness Convexity and continuity A convex function f : X R, X R n is continuous in the interior of X. f (x) 0 1 x Theorem - convexity test for smooth functions Let X R n be open and convex and let f : X R be a C 2 function. Then, f is convex only if the Hessian matrix H(x) is positive semidefinite x X. In particular, if X R and f C 2, then f is convex only if d 2 f (x) dx 2 0, x X. Ferrari Trecate (DIS) Optimization Industrial Automation 11 / 17

22 Convex functions and sets Theorem Let g : R n R be a convex function and take c R. Then, the level set X c = {x R n : g(x) c} is convex. g(x) c 0 x X c Ferrari Trecate (DIS) Optimization Industrial Automation 12 / 17

23 Convex functions and sets Theorem Let g : R n R be a convex function and take c R. Then, the level set X c = {x R n : g(x) c} is convex. g(x) c 0 x X c Proof. Pick x, y X c and λ [0, 1] and consider z = λx + (1 λ)y: we have to show that z X c. From the convexity of g one has that g(z) λg(x) + (1 λ)g(y). Since x, y X c one has that implies z X c. g(z) λg(x) + (1 λ)g(y) λc + (1 λ)c = c Ferrari Trecate (DIS) Optimization Industrial Automation 12 / 17

24 Convex functions and sets Key corollary Consider the optimization problem min f (x) g i (x) 0 i=1,2,...,m If functions g i (x), i = 1, 2,..., m are convex, then the feasibile region is convex. Ferrari Trecate (DIS) Optimization Industrial Automation 13 / 17

25 Convex functions and sets Key corollary Consider the optimization problem min f (x) g i (x) 0 i=1,2,...,m If functions g i (x), i = 1, 2,..., m are convex, then the feasibile region is convex. Proof. The proof follows from the previous theorem and the fact that convexity is preserved by intersection. In convex programming, the feasible region is convex Ferrari Trecate (DIS) Optimization Industrial Automation 13 / 17

26 Fundamental theorem of convex programming Theorem If x X is a local optimal solution for the convex programming problem then x is an optimal solution. {min f (x) : g i (x) 0, i = 1, 2,..., m} Remarks Often one tries to transform a programming problem into a convex programming problem by performing suitable changes of variables Ferrari Trecate (DIS) Optimization Industrial Automation 14 / 17

27 Fundamental theorem of convex programming Remarks The optimization problem {max f (x) : g i (x) 0, i = 1, 2,..., m} is not a convex programming problem even if f and g i, i = 1, 2,..., m are convex. Indeed, it is equivalent to { min f (x) : g i (x) 0, i = 1, 2,..., m} where the function f (x) is concave. Notable exception: f (x) linear. Ferrari Trecate (DIS) Optimization Industrial Automation 15 / 17

28 Proof of the theorem The goal is to show f ( x) f (y) y X. Fix y X, y x and let I ɛ ( x) be a neighborhood of x such that z I ɛ ( x) f ( x) f (z). Pick z X such that z xy, z I ɛ ( x) and z x. Such a z exists because z = λ x + (1 λ)y y z x X I ε ( x) and choosing λ sufficiently close to 1 guarantees z I ɛ ( x) choosing λ 1 guarantees z x Ferrari Trecate (DIS) Optimization Industrial Automation 16 / 17

29 Proof of the theorem Then, f ( x) }{{} local optimizer f (z) = f (λ x + (1 λ)y) }{{} λf ( x) + (1 λ)f (y) f convex From the last inequality one has y z x X I ε ( x) (1 λ)f ( x) (1 λ)f (y) }{{} f ( x) f (y) λ 1 Ferrari Trecate (DIS) Optimization Industrial Automation 17 / 17