Numerical Optimization

Transcription

1 Convex Sets Computer Science and Automation Indian Institute of Science Bangalore , India. NPTEL Course on

2 Let x 1, x 2 R n, x 1 x 2. Line and line segment Line passing through x 1 and x 2 : {y R n : y = λx 1 + (1 λ)x 2, λ R} Line Segment, LS[x 1, x 2 ]: {y R n : y = λx 1 + (1 λ)x 2, λ [0, 1]}

3 Affine sets Definition A set X R n is affine if for any x 1, x 2 X, and λ R, λx 1 + (1 λ)x 2 X. If X R n is an affine set, x 1, x 2,..., x k X and i λ i = 1, then the point λ 1 x 1 + λ 2 x λ k x k X. Examples Solution set of linear equations: {x R n : Ax = b} where A R m n, b R m. A subspace or a translated subspace

4

5 Result If X is an affine set and x 0 X, then {x x 0 : x X} forms a subspace. Proof. Let Y = {x x 0 : x X}. To show that Y is a subspace, we need to show, αy 1 + βy 2 Y for any y 1, y 2 Y and α, β R. Let y 1, y 2 Y and α, β R. Therefore, y 1 + x 0 X and y 2 + x 0 X. αy 1 + βy 2 = α(y 1 + x 0 ) + β(y 2 + x 0 ) (α + β)x 0. αy 1 + βy 2 + x 0 = α(y 1 + x 0 ) + β(y 2 + x 0 ) + (1 α β)x 0. Since X is an affine set, αy 1 + βy 2 + x 0 X. Thus, αy 1 + βy 2 Y Y is a subspace.

6 Definition Let X = {x 1, x 2,..., x k }. A point x is said to be an affine combination of points in X if x = k λ i x i, i=1 k λ i = 1. i=1 Definition Let X R n. The set of all affine combinations of points in X is called the affine hull of X. aff(x) = { k λ i x i : x 1,..., x k X, i i=1 λ i = 1}

7 Convex Sets Definition A set C R n is convex if for any x 1, x 2 C and any scalar λ with 0 λ 1, we have Examples Some simple convex sets λx 1 + (1 λ)x 2 C.

8 Some examples of nonconvex sets

9 Convex sets Examples: The empty set φ, any singleton set {x 0 } and R n are convex subsets of R n. A closed ball in R n, B[x 0, r] and an open ball in R n, B(x 0, r), are convex sets. Any affine set is a convex set. A line segment is a convex set, but not an affine set.

10 Possible to construct new convex sets using given convex sets Definition Let P R n, Q R n, α R. The scalar multiple αp of the set P is defined as αp = {x : x = αp, p P} The sum of two sets P and Q is the set, P + Q = {x : x = p + q, p P, q Q}.

11 Theorem If {C i }, i A, is any collection of convex sets, then i A C i is a convex set. Proof. Let C = i A C i. Case 1. If C is empty or singleton, then clearly C is a convex set. Case 2. To show that, for any x 1, x 2 C and λ [0, 1], λx 1 + (1 λ)x 2 C. Let x 1, x 2 C. Clearly, x 1, x 2 C i for every i A. Since every C i is convex, λx 1 + (1 λ)x 2 C i, i A, λ [0, 1] λx 1 + (1 λ)x 2 i A C i, λ [0, 1] λx 1 + (1 λ)x 2 C, λ [0, 1].

12 Theorem If C 1 and C 2 are convex sets, then C 1 + C 2 is a convex set. Proof. Let x 1, y 1 C 1 and x 2, y 2 C 2. So, x 1 + x 2, y 1 + y 2 C 1 + C 2. We need to show, λ(x 1 + x 2 ) + (1 λ)(y 1 + y 2 ) C 1 + C 2 λ [0, 1]. Since C 1 and C 2 are convex, z 1 = λx 1 + (1 λ)y 1 C 1 λ [0, 1], and z 2 = λx 2 + (1 λ)y 2 C 2 λ [0, 1]. Thus, z 1 + z 2 C 1 + C (1) Now, x 1 + x 2 C 1 + C 2 and y 1 + y 2 C 1 + C 2. Therefore, from (1), z 1 + z 2 = λ(x 1 + x 2 ) + (1 λ)(y 1 + y 2 ) C 1 + C 2 λ [0, 1]. Thus, C 1 + C 2 is a convex set.

13 Theorem If C is a convex set and α R, then αc is a convex set. Proof. Let x 1, x 2 C. Since C is convex, λx 1 + (1 λ)x 2 C λ [0, 1]. Also, αx 1, αx 2 αc. Therefore, λ(αx 1 ) + (1 λ)(αx 2 ) = α(λx 1 + (1 λ)x 2 ) αc for any λ [0, 1]. Therefore, αc is a convex set.

14 Hyperplane Definition Let b R and a R n, a 0. Then, the set is said to be a hyperplane in R n. H = {x : a T x = b} a denotes the normal to the hyperplane H. If a = 1, then b is the distance of H from the origin. In R 2, hyperplane is a line In R 3, hyperplane is a plane

15

16

17

18 Half-spaces The sets H + = {x : a T x b} and H = {x : a T x b} are called closed positive and negative half-spaces generated by H.

19

20 Some more examples of convex sets H = {x : a T x = b} is a convex set Let a 1, a 2,..., a m R n and b 1, b 2,..., b m R a T 1 b 1 a T 2 Define, A =. and b = b 2. a T m Then, {x : Ax = b} is a convex set. H + = {x : a T x b} and H = {x : a T x b} are convex sets. b m {x : Ax b} and {x : Ax b} are convex sets.

21

22

23

24

25 Convex Hull Definition The convex hull of a set S is the intersection of all the convex sets which contain S and is denoted by conv(s). Note: Convex Hull of a set S is the convex set. The smallest convex set that contains S is called the convex hull of the set S. Examples: conv({x, y}) = LS[x, y] where x and y are two points Let S = {(x, 0) : x R} {(0, y) : y R}. Then conv(s) = R 2.

26

27

28

29 S = {(x 1, x 2 ) : x 2 = x 2 1} conv(s) = {(x 1, x 2 ) : x 2 x 2 1}

30 Theorem CS1 Let S R n be a nonempty, closed convex set and y / S. Then there exists a unique point x 0 S with minimum distance from y. Further, x 0 is the minimizing point iff (y x 0 ) T (x x 0 ) 0 for all x S.

31 Proof Sketch Given S R n, a closed convex set and y / S. First show that there exists a minimizing point x 0 in S that is closest to y. Let δ = inf x S x y. Note that S is not bounded.

32 Proof Sketch (continued) Consider the set S B[y, 2δ]. By Weierstrass Theorem, there exists a minimizing point x 0 in S that is closest to y. x 0 = arg min x S B[y,2δ] x y

33 Proof Sketch (continued) Show uniqueness using triangle inequality Assume that there exists ˆx S such that y x 0 = y ˆx = δ. Since S is convex, 1 2 (x 0 + ˆx) S. Using triangle inequality, 2 y (x 0 + ˆx) y x 0 + y ˆx = 2δ 2 We have a contradiction if strict inequality holds.

34 Proof Sketch (continued) To show that x 0 is the unique minimizing point iff (y x 0 ) T (x x 0 ) 0 for all x S. Let x S. Assume (y x 0 ) T (x x 0 ) 0. y x 2 = y x 0 + x 0 x 2 = y x x 0 x 2 + 2(y x 0 ) T (x 0 x) Using the assumption, (y x 0 ) T (x 0 x) 0, we get y x 2 y x 0 2 This implies that x 0 is the minimizing point.

35 Proof Sketch (continued) Assume that x 0 is the minimizing point, that is, y x 0 2 y z 2 z S. Consider any x S. Since S is convex, λx + (1 λ)x 0 S λ [0, 1]. Therefore, y x 0 2 y x 0 λ(x x 0 ) 2 That is, y x 0 2 y x 0 2 +λ 2 x x 0 ) 2 2λ(y x 0 ) T (x x 0 ) 2(y x 0 ) T (x x 0 ) λ x x 0 ) 2 Letting λ 0 +, the result follows.

36 Theorem CS1 Let S R n be a nonempty, closed convex set and y / S. Then there exists a unique point x 0 S with minimum distance from y. Further, x 0 is the minimizing point iff (y x 0 ) T (x x 0 ) 0 for all x S.

37 Separating hyperplanes Definition Let S 1 and S 2 be nonempty subsets in R n and H = {x : a T x = b} be a hyperplane. 1 The hyperplane H is said to separate S 1 and S 2, if a T x b x S 1 and a T x b x S 2. 2 If a T x > b x S 1 and a T x < b x S 2, then the hyperplane H is said to strictly separate S 1 and S 2. 3 The hyperplane H is said to strongly separate S 1 and S 2 if a T x b + ɛ x S 1 and a T x b x S 2 where ɛ is a positive scalar.

38 H: Separating hyperplane The hyperplane H = {x : a T x = b} is said to separate S 1 and S 2, if a T x b x S 1 and a T x b x S 2.

39 H: Strictly separating hyperplane If a T x > b x S 1 and a T x < b x S 2, then the hyperplane H = {x : a T x = b} is said to strictly separate S 1 and S 2.

40 H: Strongly separating hyperplane The hyperplane H = {x : a T x = b} is said to strongly separate S 1 and S 2 if a T x b + ɛ x S 1 and a T x b x S 2 where ɛ is a positive scalar.

41 Separation of a closed convex set and a point Result Let S be a nonempty closed convex set in R n and y / S. Then there exists a nonzero vector a and a scalar b such that a T y > b and a T x b x S. Proof. By Theorem CS1, there exists a unique minimizing point x 0 S such that (x x 0 ) T (y x 0 ) 0 for each x S. Letting a = (y x 0 ) and b = a T x 0, we get a T x b for each x S and a T y b = (y x 0 ) T (y x 0 ) > 0 (since y x 0 ).

42 Cone Definition A set K R n is called a cone if for every x K and λ 0, we have λx K. K is a convex cone if it is convex and a cone.

43 Some examples of cones

44 Farkas Lemma Farkas Lemma Let A R m n and c R n. Then, exactly one of the following two systems has a solution: (I) Ax 0, c T x > 0 for some x R n (II) A T y = c, y 0 for some y R m.

45 Let a 1, a 2,..., a m R n. a T 1 a T 2 Define, A =.. a T m For example,

46

47

48 Farkas Lemma (Geometrical Interpretation) Exactly one of the following two systems has a solution: (I) Ax 0, c T x > 0 for some x R n (II) A T y = c, y 0 for some y R m

49 Farkas Lemma Let A R m n and c R n. Then, exactly one of the following two systems has a solution: (I) Ax 0, c T x > 0 (II) A T y = c, y 0 for some x R n for some y R m Proof of Farkas Lemma (a) Suppose, system (II) has a solution. Therefore, y 0 such that c = A T y. Consider x R n such that Ax 0. So, c T x = y T Ax 0. That is, System I has no solution.

50 Farkas Lemma Let A R m n and c R n. Then, exactly one of the following two systems has a solution: (I) Ax 0, c T x > 0 (II) A T y = c, y 0 for some x R n for some y R m Proof of Farkas Lemma (continued) (b) Suppose, system (II) has no solution. Let S = {x : x = A T y, y 0}, a closed convex set and c / S. Therefore, p R n and α R such that p T x α x S and c T p > α. This means, α 0 (Since 0 S) c T p > 0. Also, α p T A T y = y T Ap y 0. Since y 0, Ap 0 (as y can be made arbitrarily large). Thus, p R n such that Ap 0, c T p > 0 System (I) has a solution.

51 Farkas Lemma Let A R m n and c R n. Then, exactly one of the following two systems has a solution: (I) Ax 0, c T x > 0 for some x R n (II) A T y = c, y 0 for some y R m. Corollary Let A R m n. Then exactly one of the following systems has a solution: (I) Ax < 0 for some x R n (II) A T y = 0, y 0 for some nonzero y R m.

52 Corollary Let A R m n. Then exactly one of the following systems has a solution: (I) Ax < 0 for some x R n (II) A T y = 0, y 0 for some nonzero y R m. Proof. We can write system I as Ax + ze 0 for some x R n, z > 0 where e is a m-dimensional vector containing all 1 s. That is, ( ) ( ) x x (A e) 0, (0,..., 0, 1) > 0 z z for some (x z) T R n+1.

53 Proof. (continued) Recall Farkas Lemma. Therefore, System II is ( ) A T y = (0,..., 0, 1) T, y 0 for some y R m. e T That is A T y = 0, e T y = 1 and y 0 for some y R m. Using Farkas Lemma, the result follows.

54 Supporting Hyperplanes of sets at boundary points Definition Let S R n, S φ. Let x 0 be a boundary point of S. A hyperplane H = {x : a T (x x 0 ) = 0} is called a supporting hyperplane of S at x 0, if either S H + (that is, a T (x x 0 ) 0 x S), or S H (that is, a T (x x 0 ) 0 x S).

55 Theorem Let S be a nonempty convex set in R n and x 0 be a boundary point of S. Then, there exists a hyperplane that supports S at x 0.

56 Theorem Let S 1 and S 2 be two nonempty disjoint convex sets in R n. Then, there exists a hyperplane that separates S 1 and S 2. Proof. Consider the set S = S 1 S 2 = {x 1 x 2 : x 1 S 1, x 2 S 2 }. Note that S is a convex set and 0 / S. Thus, we have a convex set and a point not in that set. So, there exists a vector a such that a T x 0 x S. In other words, a T x 1 a T x 2 x 1 S 1, x 2 S 2.