Convex Optimization 2. Convex Sets Prof. Ying Cui Department of Electrical Engineering Shanghai Jiao Tong University 2018 SJTU Ying Cui 1 / 33
Outline Affine and convex sets Some important examples Operations that preserve convexity Generalized inequalities Separating and supporting hyperplanes Dual cones and generalized inequalities SJTU Ying Cui 2 / 33
Lines and line segments line passing through two distinct points x 1,x 2 R n,x 1 x 2 : points of the form sum of x 1 scaled by θ and x 2 scaled by 1 θ y = θx 1 +(1 θ)x 2, θ R sum of base point x 2 and direction x 1 x 2 scaled by θ y = x 2 +θ(x 1 x 2 ), θ R line segment between two distinct points x 1,x 2 R n,x 1 x 2 : points of the form y = θx 1 +(1 θ)x 2, θ [0,1] θ = 1.2 x1 θ = 1 θ = 0.6 x2 θ = 0 θ = 0.2 Figure 2.1 The line passing through x1 and x2 is described parametrically by θx1 +(1 θ)x2, where θ varies over R. The line segment between x1 and x2, which corresponds to θ between 0 and 1, is shown darker. SJTU Ying Cui 3 / 33
Affine sets affine set: an affine set C R n contains the line through any two distinct points in C x 1,x 2 C and θ R, θx 1 +(1 θ)x 2 C affine combination of points x 1,,x k : a point of the form θ 1 x 1 + +θ k x k, where θ 1,,θ k R and θ 1 + +θ k = 1 an affine set contains every affine combination of its points affine hull of set C R n : the set of all affine combinations of points in C, i.e., affc = {θ 1 x 1 + +θ k x k x 1,,x k C,θ 1 + +θ k = 1} affc is the smallest affine set that contains C: if S is any affine set with C S, then affc S example: empty set, any single point (i.e., singleton) {x 0 }, line, hyperplane, whole space R n, solution set of linear equations C = {x Ax = b} solution set of a system of linear equations is an affine set every affine set can be expressed as the solution set of a system of linear equations SJTU Ying Cui 4 / 33
Convex sets convex set: a convex set C R n contains the line segment between any two distinct points in C x 1,x 2 C and θ [0,1], θx 1 +(1 θ)x 2 C every affine set is also convex convex combination of points x 1,,x k : a point of the form θ 1 x 1 + +θ k x k, where θ 1,,θ k 0 and θ 1 + +θ k = 1 can be generalized to include infinite sums, integrals, and probability distributions a convex set contains every convex combination of its points convex hull of set C R n : the set of all convex combinations of points in C, i.e., convc = {θ 1 x 1 + +θ k x k x i C,θ i 0,i = 1,,k,θ 1 + +θ k = 1} convc is the smallest convex set that contains C: if B is any convex set with C B, then convc B example: line segment, ray Figure 2.2 Some simple convex and nonconvex sets. Left. The hexagon, which includes its boundary (shown darker), is convex. Middle. The kidney shaped set is not convex, since the line segment between the two points in Figure 2.3 The convex hulls of two sets in R 2. Left. The convex hull of a the set shown as dots is not contained in the set. Right. The square contains set of fifteen points (shown as dots) is the pentagon (shown shaded). Right. some boundary points but not others, and is not convex. The convex hull of the kidney shaped set in figure 2.2 is the shaded set. SJTU Ying Cui 5 / 33
Cones cone (or nonnegative homogeneous): a set C R n satisfies x C and θ 0, θx C convex cone: a convex cone C R n is convex and a cone x 1,x 2 C and θ 1,θ 2 0, θ 1 x 1 +θ 2 x 2 C conic combination (or nonnegative linear combination) of points x 1,,x k : a point of the form θ 1 x 1 + +θ k x k, where θ 1,,θ k 0 can be generalized to include infinite sums and integrals a convex cone contains all conic combinations of its elements conic hull of set C R n : set of all conic combinations of points in C, {θ 1 x 1 + +θ k x k x i C,θ i 0,i = 1,,k} the smallest convex cone that contains C example: subspace, line passing through origin, ray with base origin x1 x2 0 Figure 2.4 The pie slice shows all points of the form θ1x1 + θ2x2, where θ1, θ2 0. The apex of the slice (which corresponds to θ1 = θ2 = 0) is at 0; its edges (which correspond to θ1 = 0 or θ2 = 0) pass through the points x1 and x2. 0 Figure 2.5 The conic hulls (shown shaded) of the two sets of figure 2.3. 0 SJTU Ying Cui 6 / 33
Hyperplanes and halfspaces hyperplane: set of form {x a T x = b} (a R n,a 0,b R) analytical interpretation: solution set of a nontrivial linear equation hyperplane is an affine set geometric interpretation: set of points with constant inner product b to given vector a hyperplane with normal vector a and offset from the origin determined by b: an offset x 0 plus all vectors orthogonal to normal vector a {x a T (x x 0) = 0} = x 0 +{v a T v = 0} }{{} a where x 0 is any point in the hyperplane satisfying a T x 0 = b and a denotes the orthogonal complement of a a x0 x a T x = b Figure 2.6 Hyperplane in R 2, with normal vector a and a point x0 in the hyperplane. For any point x in the hyperplane, x x0 (shown as the darker arrow) is orthogonal to a. SJTU Ying Cui 7 / 33
Hyperplanes and halfspaces halfspace: set of form {x a T x b} (a R n,a 0,b R) analytical interpretation: solution set of a nontrivial linear inequality halfspace is not an affine set, but a convex set geometric interpretation: an offset x 0 plus all vectors making an obtuse (or right) angle with outward normal vector a {x a T (x x 0 ) 0} = x 0 +{v a T v 0} where x 0 is any point in the hyperplane satisfying a T x 0 = b a x0 a T x b a T x b Figure 2.7 A hyperplane defined by a T x = b in R 2 determines two halfspaces. The halfspace determined by a T x b (not shaded) is the halfspace extending in the direction a. The halfspace determined by a T x b (which is shown shaded) extends in the direction a. The vector a is the outward normal of this halfspace. a hyperplane divides R n into two halfspaces the boundary of a halfspace is a hyperplane SJTU Ying Cui 8 / 33
Euclidean balls and ellipsoids (Euclidean) ball with center x c and radius r: B(x c,r) = {x x x c 2 r} = {x c +ru u 2 1} u 2 = (u T u) 1/2 denotes the Euclidean norm (l 2 norm) convex set ellipsoid with center x c : E = {x (x x c ) T P 1 (x x c ) 1}, where P S n ++ E = {x c +Au u 2 1}, where A S n ++ P determines how far the ellipsoid extends in every direction from x c, the lengths of the semi-axes of the ellipsoid are given by λ i, where λ i are eigenvalues of P when P = r 2 I, ellipsoid becomes ball when A = P 1/2, two representations are the same convex set xc Figure 2.9 An ellipsoid in R 2, shown shaded. The center xc is shown as a dot, and the two semi-axes are shown as line segments. SJTU Ying Cui 9 / 33
t Norm balls and norm cones norm: a function : R n R (measure of length of vector) nonnegative: x 0 for all x R n definite: x = 0 only if x = 0 homogeneous: tx = t x, for all x R n and t R triangle inequality: x +y x + y, for all x,y R n norm ball with center x c and radius r: {x x x c r} convex set norm cone: {(x,t) x t} R n+1 convex cone second-order (Euclidean norm) cone, i.e., {(x,t) x 2 t} 1 0.5 0 1 0 x2 1 1 x1 0 1 Figure 2.10 Boundary of second-order cone in R 3, {(x1, x2, t) (x 2 1+x 2 2) 1/2 t}. SJTU Ying Cui 10 / 33
Polyhedra polyhedron: solution set of a finite number of linear inequalities and equalities (can be bounded or unbounded) P ={x a T j x b j,j = 1,,m, c T j x = d j,j = 1,,p} ={x Ax b,cx = d} where A = a T 1. a T m, C = c T 1. c T p, denotes vector inequality intersection of a finite number of halfspaces and hyperplanes affine sets (e.g., subspaces, hyperplanes, lines), rays, line segments, and halfspaces are all polyhedra a1 a2 P a5 a3 a4 Figure 2.11 The polyhedron P (shown shaded) is the intersection of five halfspaces, with outward normal vectors a1,...., a5. SJTU Ying Cui 11 / 33
z Positive semidefinite cone S n = {X R n n X = X T }: set of symmetric n n matrices vector space with dimension n(n+1)/2 convex cone S n + = {X Sn X 0}: set of symmetric positive semidefinite n n matrices, X S n + z T Xz 0 for all z R n convex cone, referred [ ] to as positive semidefinite cone x y example: X = S y z 2 + x 0,z 0,xz y 2 1 0.5 0 1 0 0.5 1 y 1 0 x Figure 2.12 Boundary of positive semidefinite cone in S 2. S n ++ = {X S n X 0}: set of symmetric positive definite n n matrices, X S n ++ zt Xz > 0 for all z R n,z 0 not cone (as origin not in it) SJTU Ying Cui 12 / 33
Determine or establish convexity of sets apply definition: a set C is convex if for all x 1,x 2 C and θ [0,1], we have θx 1 +(1 θ)x 2 C use operations that preserve convexity: show that C is obtained from simple convex sets (hyperplanes, halfspaces, norm balls,... ) by operations that preserve convexity intersection affine functions perspective functions linear-fractional functions SJTU Ying Cui 13 / 33
Intersection the intersection of (any number of) convex sets is convex if S a is convex for every a A, then a A S a is convex examples: a polyhedron is the intersection of halfspaces and hyperplanes (which are convex), and so is convex P ={x a T j x b j,j = 1,,m, c T j x = d j,j = 1,,p} =( m j=1{x a T j x b j }) ( p j=1 {x ct j x = d j }) {x a T j x b j }: halfspace; {x c T j x = d j }: hyperplane the positive semidefinite cone is the intersection of an infinite number of halfspaces, and so is convex S n + ={X Sn X 0} = z Rn,z 0{X S n : z T Xz 0} z T Xz, z 0: linear function of X; {X S n : z T Xz 0}: halfspace in S n SJTU Ying Cui 14 / 33
Affine functions affine function f : R n R m : f(x) = Ax +b where A R m n and b R m the image of a convex set under an affine function is convex S R n convex = f(s) = {f(x) x S} convex the inverse image of a convex set under an affine function is convex S R n convex = f 1 (S) = {x f(x) S} convex SJTU Ying Cui 15 / 33
Affine functions Examples: the scaling and translation of a convex set are convex S R n convex, α R, a R n = αs = {αx x S} convex, S +a = {x +a x S} convex the projection of a convex set onto some coordinates is convex S R m R n convex = {x 1 R m (x 1,x 2 ) S,x 2 R n } convex the sum of two convex sets is convex S 1 and S 2 convex = Cartesian produce S 1 S 2 = {(x 1,x 2 ) x 1 S 1,x 2 S 2 } convex, and hence S 1 +S 2 = {x 1 +x 2 x 1 S 1,x 2 S 2 } convex polyhedron {x Ax b,cx = d} is convex the inverse image of R m + {0} under affine function f : R n R m+p given by f = (b Ax,d Cx) the solution set of linear matrix inequality {x A(x) B} is convex, where A(x) = x 1 A 1 + +x n A n, B,A i S m the inverse image of the positive semidefinite cone under affine function f : R n S m given by f(x) = B A(x) SJTU Ying Cui 16 / 33
Perspective functions perspective function P : R n+1 R n : P(z,t) = z/t, domp = R n R ++ scales or normalizes vectors so the last component is one, and then drops the last component the image of a convex set under a perspective function is convex C domp convex = P(C) = {P(x) x C} convex the inverse image of a convex set under a perspective function is convex C R n convex = P 1 (C) = {(x,t) R n+1 x t C,t > 0} convex SJTU Ying Cui 17 / 33
Linear-fractional functions linear-fractional (or projective) function f : R n R m : f(x) = (Ax +b)/(c T x +d), domf = {x c T x +d > 0} where A R m n,b R m,c R n,d R composes the perspective function with an affine function: f = P g perspective function P : R m+1 R m, i.e., P(z,t) = z/t affine function g : R n R m+1, i.e,. g(x) = [ A c T ] x + example: conditional probabilities u {1,,n} and v {1,,m} are r.v.s, and let p ij = Pr(u = i,v = j) and f ij = Pr(u = i v = j) f ij = pij nk=1 p kj is a linear-fractional function the image of a convex set under a perspective function is convex C domf convex = f(c) = {f(x) x C} convex the inverse image of a convex set under a perspective function is convex C R m convex = f 1 (C) = {x f(x) C} convex SJTU Ying Cui 18 / 33 [ b d ]
Proper cones and generalized inequalities a cone K R n is a proper cone if K is convex closed (contains its boundary) solid (has nonempty interior) pointed (contains no line, i.e., x, x K = x = 0) (nonstrict) generalized inequality defined by a proper cone K R n is a partial ordering on R n x K y (y K x) y x K strict generalized inequality defined by a proper cone K R n is a strict partial ordering on R n x K y (y K x) y x intk (nonstrict and strict) generalized inequalities include as a special case ordinary (nonstrict and strict) inequality in R when K = R +, partial ordering K is usual ordering on R, and strict partial ordering K is usual strict ordering < on R SJTU Ying Cui 19 / 33
Proper cones and generalized inequalities Examples nonnegative orthant and componentwise inequality: R n + = {x R n x i 0,i = 1,,n} is a proper cone R n + corresponds to componentwise inequality between vectors: x R n + y x i y i, i = 1,,n drop subscript R n +, understood when appears between vectors positive semidefinite cone and matrix inequality S n + is a proper cone S n + corresponds to matrix inequality: X S n + Y Y X S n + drop subscript S n +, understood when appears between symmetric matrices SJTU Ying Cui 20 / 33
Proper cones and generalized inequalities Properties of generalized inequalities preserved under addition: x K y,u K v = x +u K y +v transitive: x K y,y K z = x K z preserved under nonnegative scaling: x K y,α 0 = αx K αy reflexive: x K x antisymmetric: x K y,y K x = x = y preserved under limits: x i K y i for i = 1,2,...,x i x,y i y as i = x K y SJTU Ying Cui 21 / 33
Minimum and minimal elements on R is a linear ordering: any two points are comparable, i.e., either x y or y x K is not in general a linear ordering: not any two points are comparable, i.e., we can have x K y and y K x concepts like minimum and maximum are more complicated in the context of generalized inequalities when K = R + (partial ordering K is usual ordering on R), the concepts of minimal and minimum are the same, i.e., usual definition of the minimum element of a set SJTU Ying Cui 22 / 33
Minimum and minimal elements x S is the minimum element of S with respect to K if for every y S, we have x K y iff S x +K, where x +K denotes all the points that are comparable to x and greater than or equal to x (w.r.t. K ) if a set has a minimum element, then it is unique x S is a minimal element of S with respect to K if y S, y K x only if y = x iff (x K) S = {x}, where x K denotes all the points that are comparable to x and less than or equal to x (w.r.t. K ) SJTU Ying Cui 23 / 33
Minimum and minimal elements example K = R n +: R n + corresponds to componentwise inequality x S is the minimum element of a set S means all other points of S lie above and to the right of x x S is a minimal element of a set S means that no other point of S lies to the left and below x S2 S1 x2 x1 Figure 2.17 Left. The set S1 has a minimum element x1 with respect to componentwise inequality in R 2. The set x1 + K is shaded lightly; x1 is the minimum element of S1 since S1 x1 + K. Right. The point x2 is a minimal point of S2. The set x2 K is shown lightly shaded. The point x2 is minimal because x2 K and S2 intersect only at x2. SJTU Ying Cui 24 / 33
Separating hyperplane theorem separating hyperplane theorem: Suppose C and D are two nonempty convex sets that do not intersect, i.e., C D =. Then there exist a 0 and b such that a T x b for all x C, a T x b for all x D i.e., affine function f(x) = a T x b is nonpositive on C and nonnegative on D separating hyperplane for sets C and D: {x a T x = b} a T x b a T x b D C a Figure 2.19 The hyperplane {x a T x = b} separates the disjoint convex sets C and D. The affine function a T x b is nonpositive on C and nonnegative on D. SJTU Ying Cui 25 / 33
Supporting hyperplane theorem supporting hyperplane to set C at x 0 bdc = clc \intc: {x a T x = a T x 0 } if a 0 satisfies a T x a T x 0 for all x C geometric interpretation: hyperplane {x a T x = a T x 0 } is tangent to C at x 0, and halfspace {x a T x a T x 0 } contains C a C x0 Figure 2.21 The hyperplane {x a T x = a T x0} supports C at x0. supporting hyperplane theorem: for any nonempty convex set C, and any x 0 bdc, there exists a supporting hyperplane to C at x 0 SJTU Ying Cui 26 / 33
Dual cones dual cone of a cone K: K = {y y T x 0 for all x K} K is a cone and is always convex (even when original cone K is not), i.e., convex cone geometric interpretation: y K iff y is the normal of a hyperplane that supports K at the origin y K z K Figure 2.22 Left. The halfspace with inward normal y contains the cone K, so y K. Right. The halfspace with inward normal z does not contain K, so z K. SJTU Ying Cui 27 / 33
Dual cones examples nonnegative orthant: K = R n +, K = R n + positive semidefinite cone: K = S n +, K = S n + dual of a l 2 norm cone: K = {(x,t) x 2 t}, K = {(x,t) x 2 t} dual of a l 1 norm cone: K = {(x,t) x 1 t}, K = {(x,t) x t} first three examples are self-dual cones SJTU Ying Cui 28 / 33
Dual cones properties K is closed and convex if K is solid (has nonempty interior), then K is pointed (contains no line) if the closure of K is pointed, then K is solid K is the closure of the convex hull of K ( = if K is convex and closed, K = K) K 1 K 2 implies K 2 K 1 Suppose K is a proper cone (convex, closed, solid and pointed). By the first three properties, its dual K is a proper cone, and moreover, by the fourth property, K = K. SJTU Ying Cui 29 / 33
Dual generalized inequalities dual cone K of a proper cone K is proper K is proper, inducing generalized inequality K K is proper, inducing generalized inequality K refer to K as the dual of K properties for proper cone K: x K y iff λ T x λ T y for all λ K 0 x K y iff λ T x < λ T y for all λ K 0,λ 0 hold if K and K are swapped (as K = K ) SJTU Ying Cui 30 / 33
Minimum and minimal elements via dual inequalities Dual characterization of minimum element x is the minimum element of S w.r.t. K iff for all λ K 0, x is the unique minimizer of λ T z over z S geometric interpretation: for all λ K 0, the hyperplane {z λ T (z x) = 0} is a strict supporting hyperplane to S at x (i.e., intersects S only at the point x) S x Figure 2.23 Dual characterization of minimum element. The point x is the minimum element of the set S with respect to R 2 +. This is equivalent to: for every λ 0, the hyperplane {z λ T (z x) = 0} strictly supports S at x, i.e., contains S on one side, and touches it only at x. SJTU Ying Cui 31 / 33
Minimum and minimal elements via dual inequalities Dual characterization of minimal elements if x minimizes λ T z over z S for some λ K 0, then x is minimal λ1 x1 S x2 λ2 Figure 2.24 A set S R 2. Its set of minimal points, with respect to R 2 +, is shown as the darker section of its (lower, left) boundary. The minimizer of λ T 1 z over S is x1, and is minimal since λ1 0. The minimizer of λ T 2 z over S is x2, which is another minimal point of S, since λ2 0. converse is in general false, i.e., x can be minimal in S, but not a minimizer of λ T z over z S for any λ if x is a minimal element of a convex set S, then there exists a nonzero λ K 0 such that x minimizes λ T z over z S convexity plays an important role in the converse SJTU Ying Cui 32 / 33
Optimal production frontier different production methods use different amounts of resources x R n production set P: resource vectors x for all possible production methods efficient (Pareto optimal) methods correspond to resource vectors x that are minimal w.r.t. R n + example (n = 2): x 1, x 2, x 3 are efficient; x 4, x 5 are not fuel P x1 x2 x5 x4 λ x3 labor Figure 2.27 The production set P, for a product that requires labor and fuel to produce, is shown shaded. The two dark curves show the efficient production frontier. The points x1, x2 and x3 are efficient. The points x4 and x5 are not (since in particular, x2 corresponds to a production method that uses no more fuel, and less labor). The point x1 is also the minimum cost production method for the price vector λ (which is positive). The point x2 is efficient, but cannot be found by minimizing the total cost λ T x for any price vector λ 0. SJTU Ying Cui 33 / 33