CS675: Convex and Combinatorial Optimization Fall 2014 Convex Functions. Instructor: Shaddin Dughmi

Transcription

1 CS675: Convex and Combinatorial Optimization Fall 2014 Convex Functions Instructor: Shaddin Dughmi

2 Outline 1 Convex Functions 2 Examples of Convex and Concave Functions 3 Convexity-Preserving Operations

3 Convex Functions A function f : R n R is convex if the line segment between any points on the graph of f lies above f. i.e. if x, y R n and θ [0, 1], then f(θx + (1 θ)y) θf(x) + (1 θ)f(y) Convex Functions 0/23

4 Convex Functions A function f : R n R is convex if the line segment between any points on the graph of f lies above f. i.e. if x, y R n and θ [0, 1], then f(θx + (1 θ)y) θf(x) + (1 θ)f(y) Inequality called Jensen s inequality (basic form) Convex Functions 0/23

5 Convex Functions A function f : R n R is convex if the line segment between any points on the graph of f lies above f. i.e. if x, y R n and θ [0, 1], then f(θx + (1 θ)y) θf(x) + (1 θ)f(y) Inequality called Jensen s inequality (basic form) f is convex iff its restriction to any line {x + tv : t R} is convex Convex Functions 0/23

6 Convex Functions A function f : R n R is convex if the line segment between any points on the graph of f lies above f. i.e. if x, y R n and θ [0, 1], then f(θx + (1 θ)y) θf(x) + (1 θ)f(y) Inequality called Jensen s inequality (basic form) f is convex iff its restriction to any line {x + tv : t R} is convex f is strictly convex if inequality strict when x y. Convex Functions 0/23

7 Convex Functions A function f : R n R is convex if the line segment between any points on the graph of f lies above f. i.e. if x, y R n and θ [0, 1], then f(θx + (1 θ)y) θf(x) + (1 θ)f(y) Inequality called Jensen s inequality (basic form) f is convex iff its restriction to any line {x + tv : t R} is convex f is strictly convex if inequality strict when x y. Analogous definition when the domain of f is a convex subset D of R n Convex Functions 0/23

8 Convex Functions 1/23 Concave and Affine Functions A function is f : R n R is concave if f is convex. Equivalently: Line segment between any points on the graph of f lies below f. If x, y R n and θ [0, 1], then f(θx + (1 θ)y) θf(x) + (1 θ)f(y)

9 Convex Functions 1/23 Concave and Affine Functions A function is f : R n R is concave if f is convex. Equivalently: Line segment between any points on the graph of f lies below f. If x, y R n and θ [0, 1], then f(θx + (1 θ)y) θf(x) + (1 θ)f(y) f : R n R is affine if it is both concave and convex. Equivalently: Line segment between any points on the graph of f lies on the graph of f. f(x) = a x + b for some a R n and b R.

10 We will now look at some equivalent definitions of convex functions First Order Definition A differentiable f : R n R is convex if and only if the first-order approximation centered at any point x underestimates f everywhere. f(y) f(x) + ( f(x)) (y x) Convex Functions 2/23

11 We will now look at some equivalent definitions of convex functions First Order Definition A differentiable f : R n R is convex if and only if the first-order approximation centered at any point x underestimates f everywhere. f(y) f(x) + ( f(x)) (y x) Local information global information If f(x) = 0 then x is a global minimizer of f Convex Functions 2/23

12 Second Order Definition A twice differentiable f : R n R is convex if and only if its derivative is nondecreasing in all directions. Formally, for all x. 2 f(x) 0 Convex Functions 3/23

13 Second Order Definition A twice differentiable f : R n R is convex if and only if its derivative is nondecreasing in all directions. Formally, for all x. Intepretation 2 f(x) 0 Recall definition of PSD: z 2 f(x)z > 0 for all z R n At x + δ z, infitisimal change in gradient is in roughly in the same direction as z Graph of f curves upwards along any line When n = 1, this is f (x) 0. Convex Functions 3/23

14 Convex Functions 4/23 Epigraph The epigraph of f is the set of points above the graph of f. Formally, epi(f) = {(x, t) : t f(x)}

15 Convex Functions 4/23 Epigraph The epigraph of f is the set of points above the graph of f. Formally, epi(f) = {(x, t) : t f(x)} Epigraph Definition f is a convex function if and only if its epigraph is a convex set.

16 Convex Functions 5/23 Jensen s Inequality (General Form) f : R n R is convex if and only if For every x 1,..., x k in the domain of f, and θ 1,..., θ k 0 such that i θ i = 1, we have f( i θ i x i ) i θ i f(x i ) Given a probability measure D on the domain of f, and x D, f(e[x]) E[f(x)]

17 Convex Functions 5/23 Jensen s Inequality (General Form) f : R n R is convex if and only if For every x 1,..., x k in the domain of f, and θ 1,..., θ k 0 such that i θ i = 1, we have f( i θ i x i ) i θ i f(x i ) Given a probability measure D on the domain of f, and x D, f(e[x]) E[f(x)] Adding noise to x can only increase f(x) in expectation.

18 Local and Global Optimality Local minimum x is a local minimum of f if there is a an open ball B containing x where f(y) f(x) for all y B. Local and Global Optimality When f is convex, x is a local minimum of f if and only if it is a global minimum. Convex Functions 6/23

19 Local and Global Optimality Local minimum x is a local minimum of f if there is a an open ball B containing x where f(y) f(x) for all y B. Local and Global Optimality When f is convex, x is a local minimum of f if and only if it is a global minimum. This fact underlies much of the tractability of convex optimization. Convex Functions 6/23

20 Sub-level sets Sublevel set Level sets of f(x, y) = x 2 + y 2 The α-sublevel set of f is {x domain(f) : f(x) α}. Convex Functions 7/23

21 Sub-level sets Level sets of f(x, y) = x 2 + y 2 Sublevel set The α-sublevel set of f is {x domain(f) : f(x) α}. Fact Every sub-level set of a convex function is a convex set. This fact also underlies tractability of convex optimization Convex Functions 7/23

22 Sub-level sets Sublevel set Level sets of f(x, y) = x 2 + y 2 The α-sublevel set of f is {x domain(f) : f(x) α}. Fact Every sub-level set of a convex function is a convex set. This fact also underlies tractability of convex optimization Note: converse false, but nevertheless useful check. Convex Functions 7/23

23 Convex Functions 8/23 Other Basic Properties Continuity Convex functions are continuous.

24 Other Basic Properties Continuity Convex functions are continuous. Extended-value extension If a function f : D R is convex on its domain, and D is convex, then it can be extended to a convex function on R n. by setting f(x) = whenever x / D. This simplifies notation. Resulting function f : D R is convex with respect to the ordering on R Convex Functions 8/23

25 Outline 1 Convex Functions 2 Examples of Convex and Concave Functions 3 Convexity-Preserving Operations

26 Examples of Convex and Concave Functions 9/23 Functions on the reals Affine: ax + b Exponential: e ax convex for any a R Powers: x a convex on R ++ when a 1 or a 0, and concave for 0 a 1 Logarithm: log x concave on R ++.

27 Examples of Convex and Concave Functions 10/23 Norms Norms are convex. θx + (1 θ)y θx + (1 θ)y = θ x + (1 θ) y Uses both norm axioms: triangle inequality, and homogeneity. Applies to matrix norms, such as the spectral norm (radius of induced ellipsoid)

28 Norms Norms are convex. θx + (1 θ)y θx + (1 θ)y = θ x + (1 θ) y Uses both norm axioms: triangle inequality, and homogeneity. Applies to matrix norms, such as the spectral norm (radius of induced ellipsoid) Max max i x i is convex max i (θx + (1 θ)y) i = max(θx i + (1 θ)y i ) i max i = θ max i θx i + max(1 θ)y i i x i + (1 θ) max i If i m allowed to pick the maximum entry of θx and θy independently, I can do only better. Examples of Convex and Concave Functions 10/23 y i

29 Examples of Convex and Concave Functions 11/23 Log-sum-exp: log(e x 1 + e x e xn ) is convex Geometric mean: ( n i=1 x i) 1 n is concave Log-determinant: log det X is concave Quadratic form: x Ax is convex iff A 0 Other examples in book f(x, y) = log(e x + e y )

30 Examples of Convex and Concave Functions 11/23 Log-sum-exp: log(e x 1 + e x e xn ) is convex Geometric mean: ( n i=1 x i) 1 n is concave Log-determinant: log det X is concave Quadratic form: x Ax is convex iff A 0 Other examples in book f(x, y) = log(e x + e y ) Proving convexity often comes down to case-by-case reasoning, involving: Definition: restrict to line and check Jensen s inequality Write down the Hessian and prove PSD Express as a combination of other convex functions through convexity-preserving operations (Next)

31 Outline 1 Convex Functions 2 Examples of Convex and Concave Functions 3 Convexity-Preserving Operations

32 Convexity-Preserving Operations 12/23 Nonnegative Weighted Combinations If f 1, f 2,..., f k are convex, and w 1, w 2,..., w k 0, then g = w 1 f 1 + w 2 f w k f k is convex.

33 Convexity-Preserving Operations 12/23 Nonnegative Weighted Combinations If f 1, f 2,..., f k are convex, and w 1, w 2,..., w k 0, then g = w 1 f 1 + w 2 f w k f k is convex. proof (k = 2) ( ) x + y g 2 ( ) ( ) x + y x + y = w 1 f 1 + w 2 f f 1 (x) + f 1 (y) w 1 = 2 g(x) + g(y) 2 + w 2 f 2 (x) + f 2 (y) 2

34 Convexity-Preserving Operations 12/23 Nonnegative Weighted Combinations If f 1, f 2,..., f k are convex, and w 1, w 2,..., w k 0, then g = w 1 f 1 + w 2 f w k f k is convex. Extends to integrals g(x) = y w(y)f y(x) with w(y) 0, and therefore expectations E y f y (x).

35 Convexity-Preserving Operations 12/23 Nonnegative Weighted Combinations If f 1, f 2,..., f k are convex, and w 1, w 2,..., w k 0, then g = w 1 f 1 + w 2 f w k f k is convex. Extends to integrals g(x) = y w(y)f y(x) with w(y) 0, and therefore expectations E y f y (x). Worth Noting Minimizing the expectation of a random convex cost function is also a convex optimization problem! A stochastic convex optimization problem is a convex optimization problem.

36 Convexity-Preserving Operations 13/23 Example: Stochastic Facility Location Average Distance k customers located at y 1, y 2,..., y k R n If I place a facility at x R n, average distance to a customer is g(x) = i 1 k x y i

37 Example: Stochastic Facility Location Average Distance k customers located at y 1, y 2,..., y k R n If I place a facility at x R n, average distance to a customer is g(x) = i 1 k x y i Since distance to any one customer is convex in x, so is the average distance. Extends to probability measure over customers Convexity-Preserving Operations 13/23

38 Convexity-Preserving Operations 14/23 Implication Convex functions are a convex cone in the vector space of functions from R n to R. The set of convex functions is the intersection of an infinite set of homogeneous linear inequalities indexed by x, y, θ f(θx + (1 θ)y) θf(x) + (1 θ)f(y) 0

39 Convexity-Preserving Operations 15/23 Composition with Affine Function If f : R n R is convex, and A R n m, b R n, then g(x) = f(ax + b) is a convex function from R m to R.

40 Convexity-Preserving Operations 15/23 Composition with Affine Function If f : R n R is convex, and A R n m, b R n, then g(x) = f(ax + b) is a convex function from R m to R. Proof (x, t) graph(g) t = g(x) = f(ax+b) (Ax+b, t) graph(f)

41 Convexity-Preserving Operations 15/23 Composition with Affine Function If f : R n R is convex, and A R n m, b R n, then g(x) = f(ax + b) is a convex function from R m to R. Proof (x, t) graph(g) t = g(x) = f(ax+b) (Ax+b, t) graph(f) (x, t) epi(g) t g(x) = f(ax + b) (Ax + b, t) epi(f)

42 Composition with Affine Function If f : R n R is convex, and A R n m, b R n, then g(x) = f(ax + b) is a convex function from R m to R. Proof (x, t) graph(g) t = g(x) = f(ax+b) (Ax+b, t) graph(f) (x, t) epi(g) t g(x) = f(ax + b) (Ax + b, t) epi(f) epi(g) is the inverse image of epi(f) under the affine mapping (x, t) (Ax + b, t) Convexity-Preserving Operations 15/23

43 Convexity-Preserving Operations 16/23 Examples Ax + b is convex max(ax + b) is convex log(e a 1 x+b 1 + e a 2 x+b e a nx+b n ) is convex

44 Maximum If f 1, f 2 are convex, then g(x) = max {f 1 (x), f 2 (x)} is also convex. Generalizes to the maximum of any number of functions, max k i=1 f i(x), and also to the supremum of an infinite set of functions sup y f y (x). Convexity-Preserving Operations 17/23

45 Maximum If f 1, f 2 are convex, then g(x) = max {f 1 (x), f 2 (x)} is also convex. Generalizes to the maximum of any number of functions, max k i=1 f i(x), and also to the supremum of an infinite set of functions sup y f y (x). epi g = epi f 1 epi f2 Convexity-Preserving Operations 17/23

46 Convexity-Preserving Operations 18/23 Example: Robust Facility Location Maximum Distance k customers located at y 1, y 2,..., y k R n If I place a facility at x R n, maximum distance to a customer is g(x) = max i x y i

47 Convexity-Preserving Operations 18/23 Example: Robust Facility Location Maximum Distance k customers located at y 1, y 2,..., y k R n If I place a facility at x R n, maximum distance to a customer is g(x) = max i x y i Since distance to any one customer is convex in x, so is the worst-case distance.

48 Example: Robust Facility Location Maximum Distance k customers located at y 1, y 2,..., y k R n If I place a facility at x R n, maximum distance to a customer is g(x) = max i x y i Worth Noting When a convex cost function is uncertain, minimizing the worst-case cost is also a convex optimization problem! A robust (in the worst-case sense) convex optimization problem is a convex optimization problem. Convexity-Preserving Operations 18/23

49 Convexity-Preserving Operations 19/23 Other Examples Maximum eigenvalue of a symmetric matrix A is convex in A max {v Av : v = 1} Sum of k largest components of a vector x is convex in x { } max 1 S x : S = k

50 Minimization If f(x, y) is convex and C is convex and nonempty, then g(x) = inf y C f(x, y) is convex. Convexity-Preserving Operations 20/23

51 Minimization If f(x, y) is convex and C is convex and nonempty, then g(x) = inf y C f(x, y) is convex. Proof (for C = R k ) epi g is the projection of epi f onto hyperplane y = 0. f(x, y) = x 2 + y 2 g(x) = x2 Convexity-Preserving Operations 20/23

52 Convexity-Preserving Operations 21/23 Example Distance from a convex set C f(x, y) = inf x y y C

53 Convexity-Preserving Operations 22/23 Composition Rules If g : R n R k and h : R k R, then f = h g is convex if g i are convex, and h is convex and nondecreasing in each argument. g i are concave, and h is convex and nonincreasing in each argument. Proof (n = k = 1) f (x) = h (g(x))g (x) 2 + h (g(x))g (x)

54 Perspective If f is convex then g(x, t) = tf(x/t) is also convex. Proof epi g is inverse image of epi f under the perspective function. Convexity-Preserving Operations 23/23