Convex Optimization - Chapter 1-2. Xiangru Lian August 28, 2015

Transcription

1 Convex Optimization - Chapter 1-2 Xiangru Lian August 28,

2 Mathematical optimization minimize f 0 (x) s.t. f j (x) 0, j=1,,m, (1) x S x. (x 1,,x n ). optimization variable. f 0. R n R. objective function. f j. R n R,i=1,,m. constraint functions. 1 x. optimal or solution of the problem The constraint can be various and is not limited to 0. 2

3 S. basic feasible set. Q. {x S f j (x) 0,j=1,,m}. feasible set. In general optimization problems are unsolvable. 3

4 Classification of optimization problems 1. Constrained problem. 2. Unconstrained problem. 3. Smooth problem. 4. Nonsmooth problem. 5. Linearly constrained problem. a. Linear optimization problem. b. Quadratic optimization problem. 6. Quadratically optimization problem. 7. Feasible. Strictly feasible... They are just constrains on the type of f 0, f j, S and x! 4

5 Classification of solutions 1. Global solution. 2. Local solution. 5

6 Analyze a method M M. numerical method P. a class of problems Σ. Model. A known part of problem P. O. Oracle. A unit answers the successive question of the method. To solve the problem means to find an approximate solution to M with an accuracy ε>0. Thus we need T ε. a stopping criterion Thus our problem class is F (Σ,O,T ε ). 6

7 Performance: The total amount of computational efforts required by M to solve P. 1. analytical complexity 2. arithmetical complexity Optimal. if upper complexity bounds of the method are proportional to the lower complexity bound of the problem class 7

8 Example: Complexity bounds for global optimization Consider a constrained minimization problem without functional constraints, min f(x). (2) x B n The basic feasible set of this problem is B n, which is an n-dimensional box in R n : B n = { x R n 0 x (i) 1,i=1,,n }. (3) The distance is measured using l -norm. 8

9 We further assume that the function is L-Lipschitzian. We can construct an optimal method for this problem and show that it is not solvable by computers. 9

10 Main fields Goals of the methods Classes of functional components General global optimization: Let s wait for quantum computers! Nonlinear optimization: We can find a local minimum under restrictions[1]. Convex optimization: We can find the global minimum under restrictions. Interior-point polynomial-time methods: We can find the global minimum under restrictions for convex sets and functions with explicit structure. 10

11 Description of the oracle Convex optimization also benefits nonconvex optimization. 11

12 Convex sets Line. Points of the form y=θx 1 +(1 θ)x 2, where θ R, form the line passing through x 1 and x 2. Line segment. y=x 2 +θ(x 1 x 2 ) where 0 θ 1. Affine set. A set C R n is affine if the line through any two distinct points in C lies in C. Affine combination. θ 1 x 1 + +θ k x k where θ 1 + +θ k =1 is an affine combination of the points x 1,,x k. Theorem 1. If C is an affine set, x 1,, x k C, then the affine combination of x 1,,x k also belongs to C. 12

13 Theorem 2. If C is an affine set and x 0 C, then the set is a subspace. V =C x 0 ={x x 0 x C} Dimension of an affine set. Defined to be the dimension of the subspace V =C x 0, where x 0 is any element of C. Affine dimension. The affine dimension of a set C is the dimension of its affine hull 2. Affine hull. The set of all affine combinations fo points of some set C. Denoted by aff C. 2. Unit circle in 2-d space has affine dimension 2 but by most definitions of dimension it has dimension 1. 13

14 Relative interior. The relative interior of the set C is the interior relative to aff C: relintc={x C B(x,r) affc C for some r>0}. Relative boundary. The relative boundary of a set C is clc\relintc, where clc is the clojure of C. The affine hull of C is the smallest affine set that contains C. Convex set. A set C is convex if the line segment between any two points in C lies in C. Convex combination. A convex combination of the points x 1,, x k is θ 1 x θ k x k, where θ θ k = 1 and θ i 0, i=1,,k. 14

15 Theorem 3. A set is convex iff it contains every convex combination of its points. Convex hull. A convex hull of a set C, denoted conv C, is the set of all convex combinations of points in C. It is the smallest convex set that contains C. Theorem 4. Suppose C R n is convex and x is a random vector with x C with probability one. Then Ex C. Cone. A set C is called a cone, or nonnegative homogeneous, if for every x C and θ 0 we have θx C. 15

16 Convex cone. A set C is a convex cone if it is convex and a cone, which means for any x 1,x 2 C and θ 1,θ 2 0, we have θ 1 x 1 +θ 2 x 2 C. Conic combination. or nonnegative linear combination. A point of the form θ 1 x 1 + +θ k x k with θ 1,,θ k 0. Theorem 5. A set C is a convex cone iff it contains all conic combinations of its elements. Conic hull. A set of all conic combinations of points in C. This is the smallest convex cone that contains C. 16

17 Examples Hyperplane. A set of the form {x a T x=b}, where a R n,a 0 and b R. (Normal vector is a.) Halfspaces. A hyperplane divides R n into two halfspaces. A (closed) halfspace is a set of the form {x a T x b}, where a 0. The open halfspace uses a strict inequality. Euclidean ball. B(x c,r)={x x x c 2 r}={x c +ru u 2 1}. 17

18 Ellipsoid. E={x (x x c ) T P 1 (x x c ) 1}={x c +Au u 2 1}, where P =P T 0 and A is square and nonsingular. Norm ball. {x x x c r}. Theorem 6. Norm balls are convex. Norm cone. {(x,t) x t} R n For Euclidean norm: {( {(x,t) R n+1 x x 2 t}= t ) ( x t ) T ( I I )( x t ) } 0,t 0. 18

19 Figure 1. An example of norm cone. This is also called Lorentz cone. 19

20 Polyhedron. The solution set of a finite number of linear equalities and inequalities and thus the intersection of a finite number of half spaces and hyperplanes. P = {x a j T x b j, j = 1,, m, c j T x = d j, j = 1,, p}={x Ax b,cx=d}. Theorem 7. The intersection of convex sets is a convex set. Theorem 8. Polyhedra are convex sets. Nonnegative orthant. The set of points with nonnegative components, i.e., R + n ={x R n x i 0,i=1,,n}. 20

21 Affinely independent. k+1 points v 0,,v k are affinely independent if v 1 v 0,,v k v 0 are linearly independent. Simplexes. The simplex determined by k + 1 affinely independent points v 0,,v k R n is C= conv{v 0,,v k }={θ 0 v 0 + +θ k v k θ 0,1 T θ=1}. The affine dimension of this simplex is k. Probability simplex. x 0,1 T x=1. Unit simplex. x 0,1 T x 1. The above simplex can be discribed using polyhedron. 21

22 Theorem 9. A generalization of convex hull is {θ 1 v 1 + +θ k v k θ 1 + +θ m =1,θ i 0,i=1,,k}, where m k 4. This defines a polyhedron, and conversely, every polyhedron can be represented in this form. Positive semidefinite cone. S + n = {X S n X 0}, where S n denotes the set of symmetric n n matrices. 4. We can interpret it as the convex hull of the points v 1,,v m, plus the conic hull of the points v m+1,,v k. 22

23 Operations that preserve convexity Theorem 10. Convexity is preserved under intersection. Theorem 11. Every closed convex set S is a (usually infinite) intersection of halfspaces. In fact, a closed convex set S is the intersection of all halfspaces that contain it: S= {H H halfspace,s H}. Affine function. A function f: R n R m is affine if it is a sum of a linear function and a constant, i.e., f(x)=ax+b. Theorem 12. Convexity is preserved under affine function, i.e., if S is convex and f is an affine function, then f(s) is convex. 23

24 Note that the inverse function of an affine function is an affine function. Examples of affine function include translation and projection. Corollary 13. The partial sum of two convex sets S 1,S 2 R n R m defined as S={(x,y 1 +y 2 ) (x,y 1 ) S 1,(x,y 2 ) S 2 }, where x R n and y i R m is convex. Perspective function. P: R n+1 R n, with domain dom P = R n R ++, as P(z,t)=z/t. The perspective function scales or normalizes vectors so the last component is one, and then drops the last component. 24

25 Theorem 14. If C domp is convex, then its image P(C)={P(x) x C} is convex, where P is perspective function. Proof. For any two points (x,z 1 ),(y,z 2 ) in C we have θx+(1 θ)y θz 1 +(1 θ)z 2 P(C). From this we know the perspective function maps line segments to line segments and thus it preserves the convexity of sets. 25

26 Theorem 15. The inverse image of a convex set under the perspective function is also convex. If C R n is convex, then is convex. P 1 (C)={(x,t) R n+1 x/t C,t>0} Linear-fractional function. Compose the perspective function with an( affine function. Suppose g: R n R m+1 is affine: ) g(x)=, then A c T ) x+ ( b d f =P g=(ax+b)/(c T x+d), domf ={x c T x+d>0} is a linear-fractional function. Affine function is a special case of linear-fractional function. 26

27 Corollary 16. Linear-fractional functions preserve convexity. Example 17. (Conditional probabilities) Let p ij denote prob(u = 1, v = j). Then the conditional probability f,j = prob(u = v = j) = p,j n is obtained by a linear-fractional k=1 p kj mapping from p,j. It follows that if C is a convex set of joint probabilities for (u, v), then the associated set of conditional probabilities of u given v is also convex. 27

28 Generalized inequalities Proper cone. A cone R n that 1. convex 2. closed 3. solid, which means it has nonempty interior 4. pointed, which means that it is x K, x K oriented, x=0. proper cone can be used to define a partial ordering on R n. Generalized inequality. x K y y x K, x K y y x intk. 28

29 Linear ordering. Any two points are comparable. on R is linear ordering but generalized inequality generally does not have this property 5. Example 18. When K= R +, K is the usual ordering on R. Properties of generalized inequalities 1. Preserved under addition. 2. Transitive. 3. Preserved under nonnegative scaling. 4. Reflexive. 5. Antisymmetric. 5. This makes concepts like minimun and maximum more complicated. 29

30 6. Preserved under limits. Minimum and minimal elements Minimum. x S is the minimum element of S with respect to the generalized inequality K if for every y S we have x K y. Minimal. x S is a minimal element of S with respect to the generalized inequality K if y S,y K x only if y=x. Maximum/Maximal. Defined in a similar way. Theorem 19. If a set has a minimum element, then it is unique. A set can have many different minimal elements. Theorem 20. A point x S is the minimum element of S iff S x+k. 30

31 Theorem 21. A point x S is a minimal element iff (x K) S= {x}. Figure 2. Left: minimum. Right: minimal. 31

32 Example 22. (Minimum and minimal elements of a set of symmetric matrices) We associate with eacha S n ++ an ellipsoid centered at the origin, given by E A ={x x T A 1 x 1}. We have A B iff E A E B. Let v 1,,v k R n be given and define S={P S n ++ v T i P 1 v i 1,i=1,,k}, which corresponds to the set of ellipsoids that contain the points v 1,, v k. The set does not have a minimum element but have minimal elements. 32

33 Figure 3. E 2 is a minimal element. 33

34 Separating and supporting hyperplanes Theorem 23. (Separating hyperplane theorem) Suppose C and D are two convex sets that do not intersect, i.e., C D=. Then there exists a0and b such that a T x b for all x C and a T x b for all x D. The hyperplane {x a T x=b} is called a separating hyperplane for the sets C and D. Proof. Construct a plane orthogonal to the line formed by two points achieving the dist(c, D). Strict separation. If the inequalities in separation become strict, it is called a strict separation. Strict separation is not always 34

35 possible even for closed convex sets. Theorem 24. Any two convex sets C and D, at least one of which open, are disjoint iff there exists a separating hyperplane. Supporting hyperplane. Suppose C R n, and x 0 is a point in its boundary bdc= clc\intc. If a 0 satisfies a T x a T x 0 for all x C, then the hyperplane {x a T x=a T x 0 } is called a supporting hyperplane to C at the point x This is equivalent to saying that the point x 0 and the set C are separated by the hyperplane. 35

36 Figure 4. The supporting hyperplane. Theorem 25. (Supporting hyperplane theorem) For any nonempty convex set C, and any x 0 bdc, there exists a supporting hyperplane to C at x 0. 36

37 Theorem 26. If a set is closed, has nonempty interior, and has a supporting hyperplane at every point in its boundary, then it is convex. 37

38 Dual cones and generalized inequalities Dual cone. Let K be a cone. The set K ={y x T y 0for allx K} is called the dual cone of K. Figure 5. y is in K and z is not. Geometrically, y K iff y is the normal of a hyperplane that supports K at the origin. 38

39 Theorem 27. K is a cone, and is always convex, even when the original cone is not. Example 28. (Positive semidefinite cone) The positive semidefinite cone S n + is self-dual under the standard inner product tr(xy)= n i,j=1 X ij Y ji = n i,j=1 X ij Y ij. Dual norm. u = sup{u T x x 1}. Example 29. (Dual of a norm cone) The dual of the norm cone K={(x,t) R n+1 x t} is the cone defined by the dual norm, 39

40 i.e., K ={(u,v) R n+1 u v}. The properties of dual cone 1. K is closed and convex. 2. K 1 K 2 K 2 K If K has nonempty interior, then K is pointed. 4. If the closure of K is pointed then K has nonempty interior. 5. K is the closure of the convex hull of K. (Hence if K is convex and closed, K =K.) 40

41 Theorem 30. If K is a proper cone, then so is its dual K. Dual generalized inequalities Dual generalized inequality. K. Note that if the convex cone K is proper, then K is proper. Since for a proper cone K = K, the dual generalized inequality associated with K is K. Theorem x K y iff λ T x λ T y for all λ K x K y iff λ T x<λ T y for all λ K 0. 0,λ The geometric interpretation is easy to get. 41

42 Example 32. (Theorem of alternatives for linear strict generalized inequalities) Suppose K R m is a proper cone. Consider the strict generalized inequality where x R n. Ax K b, An alternative is there exists a λ such that λ T A=0, λ T b 0, λ K 0, λ 0. Theorem 33. x is the minimum element of S, with respect to the generalized inequality K, iff for all λ K 0, x is the unique minimizer of λ T z over z S. 42

43 Figure 6. Dual characterization of minimum element. The point x is the minimum element of the set S with respect to R + 2. This is equivalent to: for every λ 0, the hyperplane {z λ T (z x) = 0} strictly supports S at x, i.e., contains S on one side, and touches it only at x. 43

44 Theorem 34. If λ K 0 and x minimizes λ T z over z S, then x is minimal. Note that the converse is generally false. Theorem 35. Provided that the set S is convex, we can say that for any minimal element x there exists a nonzero λ K 0 such that x minimizes λ T z over z S. 44

45 45

46 Figure 8. Why the converse is not true and why instead of. K=R + 2 for example. Left. The point x 1 S 1 is minimal, but is not a minimizer of λ T z over S 1 for any λ 0. Right. The point x 2 S 2 is not minimal, but it does minimize λ T z over z S 2 for λ=(0,1) 0. 46

47 Bibliography [1] Xiangru Lian, Yijun Huang, Yuncheng Li and Ji Liu. Asynchronous parallel stochastic gradient for nonconvex optimization. ArXiv preprint arxiv: ,,