1 Convex Sets Affine Sets COM524500 Optimization for Communications Summary: Convex Sets and Convex Functions A set C R n is said to be affine if A point x 1, x 2 C = θx 1 + (1 θ)x 2 C, θ R (1) y = k θ i x i, (2) where θ 1 + θ 2 +... + θ k = 1, is an affine combination of the points x 1,..., x k. An affine set can always be expressed as where x o C, and V is a subspace. C = V + x o (3) The affine hull of a set C (not necessarily affine) is affc = {θ 1 x 1 +... + θ k x k x 1,..., x k C, θ i R, i = 1,..., k, θ 1 +... + θ k = 1} (4) The affine hull is the smallest affine set that contains C. Convex Sets A set C R n is said to be convex if x 1, x 2 C = θx 1 + (1 θ)x 2 C, θ [0, 1] (5) A point k y = θ i x i, (6) where θ 1,..., θ k 0, θ 1 + θ 2 +... + θ k = 1, is a convex combination of the points x 1,..., x k. The convex hull of a set C (not necessarily convex) is convc = {θ 1 x 1 +... + θ k x k x 1,..., x k C, θ i 0, i = 1,..., k, θ 1 +... + θ k = 1} (7) The convex hull is the smallest convex set that contains C. Convex Cones A set C R n is said to be a convex cone if x 1, x 2 C = θ 1 x 1 + θ 2 x 2 C, θ 1, θ 2 0 (8) 1
A point k y = θ i x i, (9) where θ 1,..., θ k 0, is a conic combination of the points x 1,..., x k. The conic hull of a set C (not necessarily convex) is conicc = {θ 1 x 1 +... + θ k x k x 1,..., x k C, θ i 0, i = 1,..., k} (10) Some Examples of Convex Sets Hyperplane: {x a T x = b}. Halfspace: {x a T x b}. Norm ball associated with norm. : B(x c, r) = {x x x c r} (11) where x c is the center and r is the radius. When. is the 2-norm it is known as the Euclidean norm. Ellipsoid: where P 0. E = {x (x x c ) T P 1 (x x c ) 1} (12) Norm cone associated with. : K = {(x, t) x t} (13) When. is the 2-norm K is called the 2nd-order cone or the ice cream cone. A norm cone is not only convex but also a convex cone. Polyhedron: P = {x Ax b, Cx = d} A bounded polyhedron is called a polytope. = {x a T j x b j, j = 1,..., m, c T j x = d j, j = 1,..., p} (14) Simplex: Given a set of vectors v 0,... v k that are affine independent, a simplex is A simplex is a polyhedron. C = conv{v 0,..., v k } = {θ 0 v 0 +... + θ k v k θ 0, 1 T θ = 0} (15) PSD cone: S n + = {X S n X 0} is a convex cone. (recall that S n is the set of all real n n symmetric matrices.) The empty set is convex. A singleton {x o } is convex. 2
Convexity Preserving Operations Intersection: S 1, S 2 convex = S 1 S 2 convex (16) S α convex for every α A = S α convex (17) Affine mapping: If S R n is convex and f : R n R m is affine, then the image of S under f α A f(s) = {f(x) x S} (18) is convex. Similarly, if C R m is convex and f : R n R m is affine, then the inverse image of C under f f 1 (C) = {x f(x) C} (19) is convex. Image under perspective function: The perspective function P : R n+1 R n, with domain domp = R n R ++ is given by P (z, t) = z/t (20) If C domp, then P (C) is convex. Proper Cones and Generalized Inequalities A cone K R n is proper if K is convex K is closed K is solid; i.e., intk K is pointed; i.e., x K, x K = x = 0 Generalized inequality associated with a proper cone K: Properties of generalized inequalities x K y, u K v = x + u K y + v x K y, y K z = x K z x K y, α 0 = αx K αy x K y, y K x = y = x Some examples: K = R n +. Then, x K y x i y i for all i. K = S n +. Then, X K Y means that Y X is PSD. x K y y x K (21) x K y y x intk (22) Minimum and minimal elements: A point x S is the minimum element of S if y S = x K y (23) provided that such an x exists. The minimum element, if it exists, is unique. A point x S is a minimal element of S if y S, y K x = y = x (24) 3
Separating Hyperplane Theorem Suppose that C, D R n are convex and that C D =. Then there exist a 0 and b such that a T x b = x C (25) a T x b = x D (26) The hyperplane {x a T x = b} is called a separating hyperplane for C and D. Supporting Hyperplanes Suppose that x o bdc (a boundary point of C). If there exist a 0 such that a T x a T x o for all x C, then {x a T x = b} is called a supporting hyperplane to C at point x o. For any nonempty convex set C, and for any x o bdc, there exists a supporting hyperplane to C at x o. Dual Cones The dual cone of a cone K is A cone K is called self-dual if K = K. If K is proper then K is also proper. K = {y y T x 0 x K} (27) Some examples: R n + and S n + are self-dual. The dual cone of a norm cone K = {(x, t) x t} is where. is the dual norm of.. K = {(x, t) x t} (28) 4
2 Convex Functions Definition A function f : R n R is convex if domf is convex and for all x, y domf, 0 θ 1, f(θx + (1 θ)y) θf(x) + (1 θ)f(y) (29) A function f : R n R is strictly convex if domf is convex and for all x, y domf, x y, 0 < θ < 1, A function f : R n R is concave if f is convex. Fundamental Properties f(θx + (1 θ)y) < θf(x) + (1 θ)f(y) (30) f is convex if and only if it is convex when restricted to any line that intersects its domain; i.e., for all x domf and ν, g(t) = f(x + tν) (31) is convex over {t x + tν domf}. First order condition: Suppose that f is differentiable. A function f with a convex domain domf is convex if and only if f(y) f(x) + f(x) T (y x) (32) for all x, y domf. Second order condition: Suppose that f is twice differentiable. A function f with a convex domain domf is convex if and only if its Hessian 2 f(x) 0 (33) for all x domf. A function f with a convex domain domf is stricly convex if for all x domf (the converse is not true). Sublevel sets: The sublevel set of f is 2 f(x) 0 (34) C α = {x domf f(x) α} (35) If f is convex, then C α is convex for every α (the converse is not true). Epigraph: The epigraph of f is f is convex if and only if epif is convex. epif = {(x, t) x domf, f(x) t} (36) 5
Examples Examples on R: e ax is convex on R log x is concave on R ++ x log x is convex on R ++ log x e t2 /2 dt is concave on R Examples on R n : A linear function a T x + b is convex and concave. A quadratic function x T P x + 2q T x + r is convex if and only if P 0, and is strictly convex if P 0. Every norm x is convex. max{x 1,..., x n } is convex. The geometric mean ( n x i) 1 n is concave on R n ++. Examples on R n m tr(ax) is linear on R n m, and hence is convex and concave. The negative logarithmetic determinant function log det X is convex on S n ++. tr(x 1 ) is convex on S n ++. Jensen Inequality For a convex f, holds for any x, y domf and 0 θ 1. f(θx + (1 θ)y) θf(x) + (1 θ)f(y) (37) Extension: For a convex f, for any random variable z. f(ez) Ef(z) (38) Jensen inequality can be used to derive certain inequalities; e.g., the arithmetic-geometric mean inequality: a + b ab, a, b 0 (39) 2 and ( n x i ) 1 n 1 n n x i, x i 0, i = 1,..., n (40) 6
Convexity Preserving Operations Nonnegative weighted sums: f 1,..., f m convex, w 1,..., w m 0 = f(x, y) convex in x for each y A, w(y) 0 for each y A = Example: f(x) = x i log x i is convex on R n ++. Composition with an affine mapping: is convex if f is convex. Pointwise maximum and supremum: Examples: m w i f i convex (41) A w(y)f(x, y)dy convex (42) g(x) = f(ax + b) (43) f 1, f 2 convex = g(x) = max{f 1 (x), f 2 (x)} convex (44) f(x, y) convex in x for each y A = g(x) = sup f(x, y) convex (45) y A A piecewise linear function f(x) = max,...,l a T i x + b i is convex. f(x) = sup y C x y is convex for any set C. The largest eigenvalue of X is convex on S n. The 2-norm of X f(x) = λ max (X) = sup y T Xy = sup tr(xyy T ) (46) y 2 =1 y 2 =1 f(x) = X 2 = sup y 2=1 Xy 2 (47) is convex on R n m. Composition: Let f(x) = h(g(x)), where h : R R, and g : R n R. Let h(x) = { h(x), x domh, otherwise (48) Then, f is convex if h is convex and nondecreasing, and g is convex. f is convex if h is convex and nonincreasing, and g is concave. Minimization: f(x, y) convex in (x, y), C convex nonempty = g(x) = inf f(x, y) convex (49) y C provided that g(x) > for some x. Examples: 7
dist(x, S) = inf y S x y is convex for convex S. The Schur complement [ ] A B B T 0 C C 0, A BC B T 0 (50) may be proven by the convex minimization property. Perspective: The perspective of a function f is a function g : R n+1 R g(x, t) = tf(x/t), domg = {(x, t) x/t domf, t > 0} (51) If f is convex then g is convex. Conjugate Function f (y) = f is convex irrespective of the convexity of f. sup (y T x f(x)) (52) x domf domf consists of y for which the supremum is finite; i.e., domf = {y f (y) < }. Examples: The conjugate function of f(x) = 1 2 xt Qx, Q 0, is f (y) = 1 2 yt Q 1 y. The conjugate function of f(x) = log det X, domf = S n ++ is f (Y ) = log det( Y 1 ) n, domf = S n ++. Quasiconvex Functions Definition: A function f : R n R is quasiconvex (or unimodal) if domf is convex and the sublevel set is convex for every α. A function f is quasiconcave if f is quasiconvex. A function f is quasilinear if f is quasiconvex and quasiconcave. Examples: log x is quasilinear on R ++. A linear fractional function is quasilinear. S α = {x domf f(x) α} (53) f(x) = at x + b c T x + d, domf = {x ct x + d > 0} (54) rankx is quasiconcave on S n + (proven using the modified Jensen inequality). Modified Jensen inequality: f is quasiconvex if and only if for any x, y domf, and 0 θ 1, f(θx + (1 θ)y) max{f(x), f(y)} (55) 8
First-order condition: Suppose that f is differentiable. f is quasiconvex if and only if domf is convex and for all x, y domf f(y) f(x) = f(x) T (y x) 0 (56) Second-order condition: Suppose f is differentiable. If f is quasiconvex then for all x domf, y R n, Convexity with respect to Generalized Inequality y T f(x) = 0 = y T 2 f(x)y 0 (57) Let K be a proper cone. A function f : R n R m is K-convex if for all x, y domf and 0 θ 1, f(θx + (1 θ)y) K θf(x) + (1 θ)f(y) (58) For K = R n +, a K-convex function is a function for which each component function f i is convex. Consider K = S n +. f(x) = X T X is K-convex on R n m. f(x) = X 1 is K-convex on S n ++. 9