CMU-Q Lecture 9: Optimization II: Constrained,Unconstrained Optimization Convex optimization. Teacher: Gianni A. Di Caro

Transcription

1 CMU-Q Lecture 9: Optimization II: Constrained,Unconstrained Optimization Convex optimization Teacher: Gianni A. Di Caro

2 GLOBAL FUNCTION OPTIMIZATION Find the global maximum of the function f x (and the point x where it happens) Find the global minimum of the function 2

3 OPTIMIZATION PROBLEMS Optimization problems expressed in mathematical form: min f x x subject to x F f: R + R - is the objective function (for now, m=1) x X + R + is the optimization vector variable F X + is the feasible set (constraints = values the variables can feasibly take) x X + is an optimal solution (global minimum) if x F and f x f(x) for all x F Mathematical programming problem / White box opt 3

4 BASIC PROPERTIES Given an optimization problem: min f x x subject to x F, x X + R + min x f x is equivalent to max x f x If F = the problem has no solution (unfeasible) If F is an open set, only the inf (sup) is guaranteed but not min (max) The problem is unbounded if f 4

5 UNCONSTRAINED OPTIMIZATION min f x x x R n min f x x x a H, b H a K, b K a +, b + R n Sublevel sets (isolines): {x R + : f x = c} 5

6 CONSTRAINED OPTIMIZATION ü Any constrained optimization problem can be formulated as an unconstrained one by including constraint violations as penalty terms in the objective function 6

7 CONSTRAINED EXAMPLE CASES Linear Convex Reals Certainty Non-linear Non-convex Zeals Stochastic Linear case: Find x s.t. Ax b x R l Fall A2016: R S l Lecture, b R13 S Feasibility version à CSP Vector notation 7

8 GLOBAL AND LOCAL OPTIMALITY A point x X + R + is globally optimal (global minimum) if x F and for all y F, f x f(y) A point x R + is locally optimal (local minimum) if x F and there exists ε > 0 small such that for all y F with x y K ε, f x f(y) 8

9 GLOBAL AND LOCAL OPTIMALITY What about discrete spaces? x X + Z + min Z ILP = x 2 s.t. 2x 1 + x 2 > 13 5x 1 + 2x x 1 + x 2 > 5 x 1,x 2 2 Z + 9

10 APPLICATION EXAMPLE: LEAST-SQUARES FITTING Given x Y, y Y for i = 1,, m, find h x = ax + b that optimizes - min c K ax Y + b y Y a,b YdH (a is slope, b is intercept) y Y x Y h x Y y Y 10

11 APPLICATION EXAMPLE: WEBER POINT Given x Y, y Y for i = 1,, m, find the point (x,y ) that minimizes the sum of Euclidean distances: min - c e,f YdH x x Y K + y y Y K (x Y,y Y ) (x, y ) Many modifications, e.g., might want a x b, c y d 11

12 APPLICATION EXAMPLE: MACHINE LEARNING Many machine learning problems can be described as minimizing a loss function - min c L α R i YdH + c α l x l Y, y Y ldh o x Y R + are input features o y Y R (regression), or y Y {0,1} (classification) are outputs o α R + are model parameters 12

13 APPLICATION EXAMPLES FOR LINEAR/INTEGER/BINARY PROGRAMMING Production planning Personnel and factory organization Logistics Project budgeting 13

14 HOW DO WE FIND MINIMA / MAXIMA? Objective function f(x) global maximum shoulder local maximum saddle flat local maximum x state space x X Using rates of change: derivatives / gradients / Jacobians / Hessians / Sampling / Searching in the X + R + domain, the input state space: Figure out where to search / sample next, from the values that are returned from the function, without generating a model of the function Using the sampled data to generate a model of the function and in turn, using it to iteratively direct the search Iteratively constructing a solution, by adding / trying out assignment to solution components x H,x K, x + many, many variants and combinations of these two basic approaches... 14

15 THE OPTIMIZATION UNIVERSE Convex problems Nonconvex problems Semidefinite programming Quadratic programming Integer programming Linear programming 15

16 CONVEX OPTIMIZATION A convex optimization problem is a special case of a general optimization problem where: min f x x subject to x F the target function f: R + R is a convex function the feasible region F is a convex set 16

17 CONVEX SETS A set F R + is convex if for all x, y F and θ 0,1, θx + 1 θ y F A set is convex if, given two points in it, it contains all their possible linear (convex) combinations Convex set Nonconvex set 17

18 CONVEX COMBINATION Given k points P Y R +, i = 1,, k, a point z R + is a convex combination of the points P Y iff: S z = c λ Y YdH P Y, λ Y 0 i, c λ Y S YdH = 1 If k = 2, z = λp H + 1 λ P K, λ H = λ, λ K = 1 λ P H = 2,1, P K = 6,3, λ = 0.75 à z = (3, 1.5) P H = 0,0, P K = 1,0, P y = 0,1, λ Y = {0.5, 0.2, 0.3} à z = (0.2, 0.3) 18

19 CONVEX HULL Given a set P of k points of R +, P = {P H, P K,., P S }, P s convex hull is the smallest convex set, conv P, that includes P conv(p) is the set of all convex combinations of the points in P: S S conv P = z R + z = c λ Y P Y, λ Y, i = 1,, k λ Y 0, c λ Y = 1 YdH YdH 19

20 EXAMPLES OF CONVEX SETS F = {x R + : i = 1,, n, a x Y b} Proof: Let x, y F, and θ [0,1] For all i = 1,, n, a x Y and a y Y, θx Y + 1 θ y Y θa + 1 θ a = a à the convex combination is greater than a Similarly, θx Y + (1 θ)y Y b Therefore θx + 1 θ y F b a a b 20

21 INTERSECTION OF CONVEX SETS a c. b Intersection of convex sets F = - YdH C Y C H,, C - are convex à F is convex Proof (by contradiction): Let s prove it first for two convex sets A and B. Let a and b be two points belonging to C = A B (and, therefore, to both A and B). Let s assume there is a third point c on the line between that a and b, such that c C, meaning that C is not convex. But, for the convexity of A, every point on the line a-b must be in A, and the same holds for B c must be in C! For m intersecting sets the same reasoning can be applied in pairs 21

22 EXAMPLES OF (NON)CONVEX SETS 22

23 EXAMPLES OF CONVEX SETS Which of the following sets are convex? 1. F = - YdH C Y where C H,, C - are convex 2. F = x R + : Ax = b where A R - +, b R - 3. Both 4. Neither Note: The empty set is a convex set! 23

24 LINEAR INEQUALITIES 2x 1 x 2 < 2 Linear inequalities F = x R + : a T x b a R +, b R Half-space ( closed, < open) Convex (obvious by geometrical considerations): Two points x and y in F: ax b, ay b CC(x, y) b a? θx + 1 θ y F? θx + 1 θ y b a θx + 1 θ y θ b a + 1 θ b a = b a 24

25 SYSTEMS OF LINEAR INEQUALITIES Polytopes Joint / Intersection set of Linear inequalities F = x R + : Ax b A R - +, b R - Polyhedron Every half-space inequality defines a convex set Their intersection is convex 25

26 CONVEX FUNCTIONS A function f: R + R is convex if for any x, y R + and λ [0,1] f λx + 1 λ y λf x + 1 λ f(y) y The graph of f is always below (or on) the line segment λf(x) + (1-λ)f(y) connecting (x, f(x)) to (y, f(y)) The line interpolation between any two points in the domain, always over estimates the value of the function For f: R R, this equals to f > 0 26

27 EXAMPLES OF CONVEX FUNCTIONS Exponential: f x = e ae f x = a K e ae 0 for all x R Euclidean (L2) norm: f x = x K = + x K YdH Y θx + 1 θ y K θx K + 1 θ y K = θ x K + (1 θ) y K If f y is convex in y, f(ax b) is convex in x Affine transformation 27

28 EXAMPLES OF CONVEX FUNCTIONS/SETS Sublevel sets (isolines): If f is convex, {x R + : f x c} is a convex set f(λx+(1-λ)y) λf(x)+(1-λ)f(y) λc+(1-λ)c = c 28

29 EXAMPLES OF CONVEX FUNCTIONS Which functions are convex? - 1. f x = YdH a Y f Y x where f Y is convex and a Y 0 for i = 1,, m 2. g x = + YdH x Y for x 0 3. Both 4. Neither 29

30 EXAMPLES OF CONVEX PROBLEMS Weber point in n dimensions: - min c x x (Y) x K YdH where x R + is optimization variable and x H,, x - are problem data A convex optimization problem (why?) Affine transformation over a convex function (Euclidean norm) + Linear combination which is also convex 30

31 EXAMPLES OF CONVEX PROBLEMS Linear programming: min c x x s.t. Ax = a Bx b where x R + is optimization variable, and c R +, A R - +, a R -, B R S +, b R S are problem data A convex optimization problem (why?) Simplex, Interior point methods 31

32 GLOBAL AND LOCAL OPTIMALITY Theorem: For a convex optimization problem, all locally optimal points are globally optimal (one, or infinite global optima) 32

33 PROOF OF THEOREM Suppose x is locally optimal for some R, but not globally optimal There is y such that f y < f(x) x Define z = θx + 1 θ y for θ = 1 K x y y 33

34 PROOF OF THEOREM Then: z is feasible, it s in f s domain (for small enough R) f z = f θx + 1 θ y θf x + 1 θ f(y) < θf x + 1 θ f x = f(x) x z K = K e f x y K = K < R it s inside thee R ball! Therefore, x is not locally optimal, contradicting our assumption 34

35 MAXIMA OF CONVEX FUNCTIONS Max corresponds to points on the frontier of the domain If domain is not a closed set, only a sup is guaranteed 35

36 WHAT ABOUT SOLVING CONVEX PROBLEMS? How could these theorems help us in solving convex optimization problems? 36

37 A GENERAL OPT PROBLEM VS. CONVEX Objective function global maximum shoulder local maximum flat local maximum current state state space 37

38 SOLVING CONVEX PROBLEMS Only need to find a local minimum Convex optimization problems can be solved in polynomial time E.g., Interior point methods for Linear Programming (Simplex is not polynomial) For unconstrained problems, use gradient descent Constrained problems require a gradient projection operator that, given x, returns the closest y F For non differentiable functions, subgradients methods can be used 38

39 SOLVING CONVEX PROBLEMS There are a wide range of tools that can take optimization problems in natural forms and compute a solution Examples include: CVX (MATLAB), GAMS (custom language), cvxpy (Python), Google s OR-Tools, YALMIP (MATLAB), AMPL (custom language) 39

40 SOLVING CONVEX PROBLEMS π Given a (Y) R K for i = 1,, m, - min c x a Y x K YdH s.t. x H + x K = 0 Constrained Weber Point import cvxpy as cp import numpy as np n = 2 m = 10 A = np.random.randn(m,n) x = cp.variable(n) f = sum([cp.norm(x - A[i,:],2) for i in range(m)]) constraints = [sum(x) == 0] result = cp.problem(cp.minimize(f), constraints).solve() print x.value 40

41 AMPL: A SET OF SOLVERS + NICE MODELING LANGUAGE 41

42 SUMMARY (SO FAR) Terminology: Convex optimization problem Convex set Convex function Local and global optimum Big ideas: In convex problems, every locally optimal solution is globally optimal! Polynomial guarantees, very good algorithms in practice 42