Submodular Functions, Optimization, and Applications to Machine Learning

Transcription

1 Submodular Functions, Optimization, and Applications to Machine Learning Spring Quarter, Lecture 10 Prof. Je Bilmes University of Washington, Seattle Department of Electrical Engineering May 2nd, 2016 Clockwise from top left:v Lásló Lovász Jack Edmonds Satoru Fujishige George Nemhauser Laurence Wolsey András Frank Lloyd Shapley H. Narayanan Robert Bixby William Cunningham William Tutte Richard Rado Alexander Schrijver Garrett Birkhoff Hassler Whitney Richard Dedekind Prof. Je Bilmes + f (A) + f (B) f (A [ B) = f (Ar ) + 2f (C ) + f (Br ) = f (Ar ) +f (C ) + f (Br ) f (A \ B) = f (A \ B) EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F1/64 (pg.1/203)

2 Logistics Cumulative Outstanding Reading Review Read chapters 2 and 3 from Fujishige s book. Read chapter 1 from Fujishige s book. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F2/64 (pg.2/203)

3 Logistics Announcements, Assignments, and Reminders Review Homework 3, available at our assignment dropbox ( due (electronically) Monday (5/2) at 11:55pm. Homework 2, available at our assignment dropbox ( due (electronically) Monday (4/18) at 11:55pm. Homework 1, available at our assignment dropbox ( due (electronically) Friday (4/8) at 11:55pm. Weekly O ce Hours: Mondays, 3:30-4:30, or by skype or google hangout (set up meeting via our our discussion board (https: //canvas.uw.edu/courses/ /discussion_topics)). Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F3/64 (pg.3/203)

4 Logistics Class Road Map - IT-I Review L1(3/28): Motivation, Applications, & Basic Definitions L2(3/30): Machine Learning Apps (diversity, complexity, parameter, learning target, surrogate). L3(4/4): Info theory exs, more apps, definitions, graph/combinatorial examples, matrix rank example, visualization L4(4/6): Graph and Combinatorial Examples, matrix rank, Venn diagrams, examples of proofs of submodularity, some useful properties L5(4/11): Examples & Properties, Other Defs., Independence L6(4/13): Independence, Matroids, Matroid Examples, matroid rank is submodular L7(4/18): Matroid Rank, More on Partition Matroid, System of Distinct Reps, Transversals, Transversal Matroid, L8(4/20): Transversals, Matroid and representation, Dual Matroids, L9(4/25): Dual Matroids, Properties, Combinatorial Geometries, Matroid and Greedy L10(4/27): Matroid and Greedy, Polyhedra, Matroid Polytopes, Polymatroid Finals Week: June 6th-10th, L11(5/2): L12(5/4): L13(5/9): L14(5/11): L15(5/16): L16(5/18): L17(5/23): L18(5/25): L19(6/1): L20(6/6): Final Presentations maximization. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F4/64 (pg.4/203)

5 Logistics The greedy algorithm Review In combinatorial optimization, the greedy algorithm is often useful as a heuristic that can work quite well in practice. The goal is to choose a good subset of items, and the fundamental tenet of the greedy algorithm is to choose next whatever currently looks best, without the possibility of later recall or backtracking. Sometimes, this gives the optimal solution (we saw three greedy algorithms that can find the maximum weight spanning tree). Greedy is good since it can be made to run very fast O(n log n). Often, however, greedy is heuristic (it might work well in practice, but worst-case performance can be unboundedly poor). We will next see that the greedy algorithm working optimally is a defining property of a matroid, and is also a defining property of a polymatroid function. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F5/64 (pg.5/203)

6 Logistics Matroid and the greedy algorithm Review Let (E,I) be an independence system, and we are given a non-negative modular weight function w : E! R +. Algorithm 1: The Matroid Greedy Algorithm 1 Set X ;; 2 while 9v 2 E \ X s.t. X [{v} 2I do 3 v 2 argmax {w(v) :v 2 E \ X, X [{v} 2I}; 4 X X [{v} ; Same as sorting items by decreasing weight w, and then choosing items in that order that retain independence. Theorem D: Let (E,I) be an independence system. Then the pair (E,I) is a matroid if and only if for each weight function w 2R E +, Algorithm?? leads to a set I 2I of maximum weight w(i). I Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F6/64 (pg.6/203)

7 Review from Lecture 6 The next slide is from Lecture 6. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F7/64 (pg.7/203)

8 Matroids by bases In general, besides independent sets and rank functions, there are other equivalent ways to characterize matroids. Theorem (Matroid (by bases)) Let E be a set and B be a nonempty collection of subsets of E. Thenthe following are equivalent. 1 B is the collection of bases of a matroid; 2 if B,B 0 2B,andx 2 B 0 \ B, thenb 0 x + y 2Bfor some y 2 B \ B 0. 3 If B,B 0 2B,andx 2 B 0 \ B, thenb y + x 2Bfor some y 2 B \ B 0. Properties 2 and 3 are called exchange properties. Proof here is omitted but think about this for a moment in terms of linear spaces and matrices, and (alternatively) spanning trees. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F8/64 (pg.8/203)

9 Matroid and the greedy algorithm proof of Theorem Assume (E,I) is a matroid and w : E!R + is given.... Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F9/64 (pg.9/203)

10 Matroid and the greedy algorithm proof of Theorem Assume (E,I) is a matroid and w : E!R + is given. Let A =(a 1,a 2,...,a r ) be the solution returned by greedy, where r = r(m) the rank of the matroid, and we order the elements as they were chosen (so w(a 1 ) w(a 2 ) w(a r )).... Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F9/64 (pg.10/203)

11 Matroid and the greedy algorithm proof of Theorem Assume (E,I) is a matroid and w : E!R + is given. Let A =(a 1,a 2,...,a r ) be the solution returned by greedy, where r = r(m) the rank of the matroid, and we order the elements as they were chosen (so w(a 1 ) w(a 2 ) w(a r )). A is a base of M, andletb =(b 1,...,b r ) be any another base of M with elements also ordered decreasing by weight, so w(b 1 ) w(b 2 ) w(b r ).... Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F9/64 (pg.11/203)

12 Matroid and the greedy algorithm proof of Theorem Assume (E,I) is a matroid and w : E!R + is given. Let A =(a 1,a 2,...,a r ) be the solution returned by greedy, where r = r(m) the rank of the matroid, and we order the elements as they were chosen (so w(a 1 ) w(a 2 ) w(a r )). A is a base of M, andletb =(b 1,...,b r ) be any another base of M with elements also ordered decreasing by weight, so w(b 1 ) w(b 2 ) w(b r ). We next show that not only is w(a) w(b) but that w(a i ) w(b i ) for all i.... Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F9/64 (pg.12/203)

13 =.. z Matroid and Greedy Polyhedra Matroid Polytopes Polymatroid Matroid and the greedy algorithm Air proof of Theorem Assume otherwise, and let k be the first (smallest) integer such that w(a k ) <w(b k ).Hencew(a j ) w(b j ) for j<k.. a, ) zw (a) 2 w( an., ) z WC an ) z wlar. ) wihizwiidz wide win tw... >-,, z Bh... Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F9/64 (pg.13/203)

14 Matroid and the greedy algorithm proof of Theorem Assume otherwise, and let k be the first (smallest) integer such that w(a k ) <w(b k ).Hencew(a j ) w(b j ) for j<k. Define independent sets A k 1 = {a 1,...,a k 1 } and B k = {b 1,...,b k }.... Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F9/64 (pg.14/203)

15 Matroid and the greedy algorithm proof of Theorem Assume otherwise, and let k be the first (smallest) integer such that w(a k ) <w(b k ).Hencew(a j ) w(b j ) for j<k. Define independent sets A k 1 = {a 1,...,a k 1 } and B k = {b 1,...,b k }. Since A k 1 < B k,thereexistsab i 2 B k \ A k 1 where A k 1 [{b i }2I for some 1 apple i apple k.... Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F9/64 (pg.15/203)

16 Matroid and the greedy algorithm proof of Theorem Assume otherwise, and let k be the first (smallest) integer such that w(a k ) <w(b k ).Hencew(a j ) w(b j ) for j<k. Define independent sets A k 1 = {a 1,...,a k 1 } and B k = {b 1,...,b k }. Since A k 1 < B k,thereexistsab i 2 B k \ A k 1 where A k 1 [{b i }2I for some 1 apple i apple k. But w(b i ) w(b k ) >w(a k ), and so the greedy algorithm would have chosen b i rather than a k, 4contradicting what greedy does. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F9/64 (pg.16/203)

17 Matroid and the greedy algorithm converse proof of Theorem Given an independence system (E,I), suppose the greedy algorithm leads to an independent set of max weight for every non-negative weight function. We ll show (E,I) is a matroid. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F10/64 (pg.17/203)...

18 Matroid and the greedy algorithm converse proof of Theorem Given an independence system (E,I), suppose the greedy algorithm leads to an independent set of max weight for every non-negative weight function. We ll show (E,I) is a matroid. Emptyset containing and down monotonicity already holds (since we ve started with an independence system). Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F10/64 (pg.18/203)...

19 Matroid and the greedy algorithm converse proof of Theorem Given an independence system (E,I), suppose the greedy algorithm leads to an independent set of max weight for every non-negative weight function. We ll show (E,I) is a matroid. Emptyset containing and down monotonicity already holds (since we ve started with an independence system). Let I,J 2I with I < J. Suppose to the contrary, that I [{z} /2 I for all z 2 J \ I. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F10/64 (pg.19/203)...

20 Matroid and the greedy algorithm converse proof of Theorem Given an independence system (E,I), suppose the greedy algorithm leads to an independent set of max weight for every non-negative weight function. We ll show (E,I) is a matroid. Emptyset containing and down monotonicity already holds (since we ve started with an independence system). Let I,J 2I with I < J. Suppose to the contrary, that I [{z} /2 I for all z 2 J \ I. Define the following modular weight function w on E, and define k = I. 8 >< k +2 if v 2 I, w(v) = k +1 if v 2 J \ I, >: 0 if v 2 E \ (I [ J). (10.1) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F10/64 (pg.20/203)...

21 Matroid and the greedy algorithm converse proof of Theorem Now greedy will, after k iterations, recover I, but it cannot choose any element in J \ I by assumption. Thus, greedy chooses a set of weight k(k + 2). = WCI ) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F10/64 (pg.21/203)...

22 Matroid and the greedy algorithm converse proof of Theorem Now greedy will, after k iterations, recover I, but it cannot choose any element in J \ I by assumption. Thus, greedy chooses a set of weight k(k + 2). On the other hand, J has weight w(j) J (k + 1) (k + 1)(k + 1) >k(k + 2) (10.2) so J has strictly larger weight but is still independent, contradicting greedy s optimality. = wet Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F10/64 (pg.22/203)...

23 Matroid and the greedy algorithm converse proof of Theorem Now greedy will, after k iterations, recover I, but it cannot choose any element in J \ I by assumption. Thus, greedy chooses a set of weight k(k + 2). On the other hand, J has weight w(j) J (k + 1) (k + 1)(k + 1) >k(k + 2) (10.2) so J has strictly larger weight but is still independent, contradicting greedy s optimality. Therefore, there must be a z 2 J \ I such that I [{z} 2I,andsince I and J are arbitrary, (E,I) must be a matroid. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F10/64 (pg.23/203)

24 Matroid and greedy As given, the theorem asked for a modular function w 2 R E +. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F11/64 (pg.24/203)

25 Matroid and greedy As given, the theorem asked for a modular function w 2 R E +. This will not only return an independent set, but it will return a base if we keep going even if the weights are 0. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F11/64 (pg.25/203)

26 Matroid and greedy As given, the theorem asked for a modular function w 2 R E +. This will not only return an independent set, but it will return a base if we keep going even if the weights are 0. If we don t want elements with weight 0, we can stop once (and if) the weight hits zero, thus giving us a maximum weight independent set. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F11/64 (pg.26/203)

27 Matroid and greedy As given, the theorem asked for a modular function w 2 R E +. This will not only return an independent set, but it will return a base if we keep going even if the weights are 0. If we don t want elements with weight 0, we can stop once (and if) the weight hits zero, thus giving us a maximum weight independent set. We don t need non-negativity, we can use any w 2 R E and keep going until we have a base. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F11/64 (pg.27/203)

28 Matroid and greedy As given, the theorem asked for a modular function w 2 R E +. This will not only return an independent set, but it will return a base if we keep going even if the weights are 0. If we don t want elements with weight 0, we can stop once (and if) the weight hits zero, thus giving us a maximum weight independent set. We don t need non-negativity, we can use any w 2 R E and keep going until we have a base. If we stop at a negative value, we ll once again get a maximum weight independent set. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F11/64 (pg.28/203)

29 Matroid and greedy As given, the theorem asked for a modular function w 2 R E +. This will not only return an independent set, but it will return a base if we keep going even if the weights are 0. If we don t want elements with weight 0, we can stop once (and if) the weight hits zero, thus giving us a maximum weight independent set. We don t need non-negativity, we can use any w 2 R E and keep going until we have a base. If we stop at a negative value, we ll once again get a maximum weight independent set. Exercise: what if we keep going until a base even if we encounter negative values? Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F11/64 (pg.29/203)

30 Matroid and greedy As given, the theorem asked for a modular function w 2 R E +. This will not only return an independent set, but it will return a base if we keep going even if the weights are 0. If we don t want elements with weight 0, we can stop once (and if) the weight hits zero, thus giving us a maximum weight independent set. We don t need non-negativity, we can use any w 2 R E and keep going until we have a base. If we stop at a negative value, we ll once again get a maximum weight independent set. Exercise: what if we keep going until a base even if we encounter negative values? We can instead do as small as possible thus giving us a minimum weight independent set/base. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F11/64 (pg.30/203)

31 Summary of Important (for us) Matroid Definitions Given an independence system, matroids are defined equivalently by any of the following: All maximally independent sets have the same size. A monotone non-decreasing submodular integral rank function with unit increments. The greedy algorithm achieves the maximum weight independent set for all weight functions. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F12/64 (pg.31/203)

32 Convex Polyhedra Convex polyhedra a rich topic, we will only draw what we need. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F13/64 (pg.32/203)

33 Convex Polyhedra Convex polyhedra a rich topic, we will only draw what we need. Definition AsubsetP R E is a polyhedron if there exists an m n matrix A and vector b 2 R m (for some m 0) suchthat IEKM P = x 2 R E : Ax apple b (10.3) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F13/64 (pg.33/203)

34 Convex Polyhedra * Convex polyhedra a rich topic, we will only draw what we need. Definition AsubsetP R E is a polyhedron if there exists an m n matrix A and vector b 2 R m (for some m 0) suchthat P = x 2 R E : Ax apple b (10.3) Thus, P is intersection of finitely many a ne halfspaces, which are of the form a i x apple b i where a i is a row vector and b i arealscalar.. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F13/64 (pg.34/203)

35 Convex Polytope A polytope is defined as follows Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F14/64 (pg.35/203)

36 Convex Polytope A polytope is defined as follows Definition AsubsetP R E is a polytope if it is the convex hull of finitely many vectors in R E.Thatis,if9, x 1,x 2,...,x k 2R E such that for all x 2 P, there exits { i } with P i i =1and i 0 8i with x = P i ix i. ear Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F14/64 (pg.36/203)

37 Convex Polytope A polytope is defined as follows Definition AsubsetP R E is a polytope if it is the convex hull of finitely many vectors in R E.Thatis,if9, x 1,x 2,...,x k 2R E such that for all x 2 P, there exits { i } with P i i =1and i 0 8i with x = P i ix i. We define the convex hull operator as follows: ( kx conv(x 1,x 2,...,x k ) def = ix i : 8i, i 0, and X i=1 i i =1 ) (10.4) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F14/64 (pg.37/203)

38 Convex Polytope - key representation theorem A polytope can be defined in a number of ways, two of which include Theorem AsubsetP R E is a polytope iff it can be described in either of the following (equivalent) ways: P is the convex hull of a finite set of points. If it is a bounded intersection of halfspaces, that is there exits matrix A and vector b such that P = {x : Ax apple b} (10.5) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F15/64 (pg.38/203)

39 Convex Polytope - key representation theorem A polytope can be defined in a number of ways, two of which include Theorem AsubsetP R E is a polytope iff it can be described in either of the following (equivalent) ways: P is the convex hull of a finite set of points. If it is a bounded intersection of halfspaces, that is there exits matrix A and vector b such that P = {x : Ax apple b} (10.5) This result follows directly from results proven by Fourier, Motzkin, Farkas, and Carátheodory. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F15/64 (pg.39/203)

40 Linear Programming Theorem (weak duality) Let A be a matrix and b and c vectors, then max {c x Ax apple b} applemin {y b : y 0,y A = c } (10.6) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F16/64 (pg.40/203)

41 Linear Programming Theorem (weak duality) Let A be a matrix and b and c vectors, then max {c x Ax apple b} applemin {y b : y 0,y A = c } (10.6) Theorem (strong duality) Let A be a matrix and b and c vectors, then max {c x Ax apple b} =min{y b : y 0,y A = c } (10.7) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F16/64 (pg.41/203)

42 Linear Programming duality forms There are many ways to construct the dual. For example, max {c x x 0, Ax apple b} =min{y b y 0,y A c } (10.8) max {c x x 0, Ax = b} =min{y b y A c } (10.9) min {c x x 0, Ax b} = max {y b y 0,y A apple c } (10.10) min {c x Ax b} = max {y b y 0,y A = c } (10.11) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F17/64 (pg.42/203)

43 Linear Programming duality forms How to form the dual in general? We quote V. Vazirani (2001) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F18/64 (pg.43/203)

44 Linear Programming duality forms How to form the dual in general? We quote V. Vazirani (2001) Intuitively, why is [one set of equations] the dual of [another quite di erent set of equations]? In our experience, this is not the right question to be asked. As stated in Section 12.1, there is a purely mechanical procedure for obtaining the dual of a linear program. Once the dual is obtained, one can devise intuitive, and possibly physical meaningful, ways of thinking about it. Using this mechanical procedure, one can obtain the dual of a complex linear program in a fairly straightforward manner. Indeed, the LP-duality-based approach derives its wide applicability from this fact. Also see the text Convex Optimization by Boyd and Vandenberghe, chapter 5, for a great discussion on duality and easy mechanical ways to construct it. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F18/64 (pg.44/203)

45 Vector, modular, incidence Recall, any vector x 2 R E can be seen as a normalized modular function, as for any A E, wehave x(a) = X a2a x a (10.12) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F19/64 (pg.45/203)

46 Vector, modular, incidence Recall, any vector x 2 R E can be seen as a normalized modular function, as for any A E, wehave x(a) = X a2a x a (10.12) Given an A E, definethetheincidencevector1 A 2{0, 1} E on the unit hypercube as follows: n Thant eristic or o def 1 A = x 2{0, 1} E : x i =1iff i 2 A (10.13) equivalently, 1 A (j) def = ( 1 if j 2 A 0 if j/2 A (10.14) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F19/64 (pg.46/203)

47 Review from Lecture 6 The next slide is review from lecture 6. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F20/64 (pg.47/203)

48 Matroid Slight modification (non unit increment) that is equivalent. Definition (Matroid-II) Asetsystem(E,I) is a Matroid if (I1 ) ;2I (I2 ) 8I 2 I,J I ) J 2 I (down-closed or subclusive) (I3 ) 8I,J 2I,with I > J, thenthereexistsx 2 I \ J such that J [{x} 2I Note (I1)=(I1 ), (I2)=(I2 ), and we get (I3) (I3 ) using induction. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F21/64 (pg.48/203)

49 Independence Polyhedra For each I 2I of a matroid M =(E,I), wecanformtheincidence vector 1 I. #,.±± V Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F22/64 (pg.49/203)

50 Independence Polyhedra For each I 2I of a matroid M =(E,I), wecanformtheincidence vector 1 I. Taking the convex hull, we get the independent set polytope, thatis ( ) [ E P ind. set = conv {1 I } (10.15) I2I EIQD Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F22/64 (pg.50/203)

51 Independence Polyhedra For each I 2I of a matroid M =(E,I), wecanformtheincidence vector 1 I. Taking the convex hull, we get the independent set polytope, thatis ( ) [ P ind. set = conv {1 I } (10.15) I2I Since {1 I : I 2I} P ind. set,wehave max {w(i) :I 2I}applemax {w x : x 2 P ind. set }. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F22/64 (pg.51/203)

52 Independence Polyhedra For each I 2I of a matroid M =(E,I), wecanformtheincidence vector 1 I. Taking the convex hull, we get the independent set polytope, thatis ( ) [ P ind. set = conv {1 I } (10.15) I2I Since {1 I : I 2I} P ind. set,wehave max {w(i) :I 2I}applemax {w x : x 2 P ind. set }. Now take the rank function r of M, and define the following polyhedron: P + r = x 2 R E : x 0,x(A) apple r(a), 8A E (10.16) HAI = atfaxakra ) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F22/64 (pg.52/203)

53 Independence Polyhedra For each I 2I of a matroid M =(E,I), wecanformtheincidence vector 1 I. Taking the convex hull, we get the independent set polytope, thatis ( ) ice. [ P ind. set = conv {1 I } (10.15) I2I Since / c- Pt {1 I : I 2I} P ind. set,wehave max {w(i) :I 2I}applemax {w x : x 2 P ind. set }. qmaxtwtaxept } Now take the rank function r of M, and define the following polyhedron: P r + D = x 2 R E : x 0,x(A) apple r(a), 8A E (10.16) Now, take any x 2 P ind. set,thenwehavethatx2p r + P ind. set P r + ). We show this next. (or Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F22/64 (pg.53/203)

54 P ind. set P + r If x 2 P ind. set,then x = X i i1 Ii (10.17) for some appropriate vector =( 1, 2,..., n). Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F23/64 (pg.54/203)

55 P ind. set P + r If x 2 P ind. set,then x = X i i1 Ii (10.17) for some appropriate vector =( 1, 2,..., n). Clearly, for such x, x 0. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F23/64 (pg.55/203)

56 P ind. set P + r If x 2 P ind. set,then x = X i i1 Ii (10.17) for some appropriate vector =( 1, 2,..., n). Clearly, for such x, x 0. Now, for any A E, x(a) =x 1 A = X i i1 Ii 1 A (10.18) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F23/64 (pg.56/203)

57 P ind. set P + r If x 2 P ind. set,then x = X i i1 Ii (10.17) for some appropriate vector =( 1, 2,..., n). Clearly, for such x, x 0. Now, for any A E, x(a) =x 1 A = X i i1 Ii 1 A (10.18) apple X i i max j:i j A 1 I j (E) (10.19) It,.#IE 4 = II ; ) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F23/64 (pg.57/203)

58 P ind. set P + r If x 2 P ind. set,then x = X i i1 Ii (10.17) for some appropriate vector =( 1, 2,..., n). Clearly, for such x, x 0. Now, for any A E, x(a) =x 1 A = X i i1 Ii 1 A (10.18) apple X i i max j:i j A 1 I j (E) (10.19) = max 1 I j (E) = max A \ I (10.20) j:i j A I2I Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F23/64 (pg.58/203)

59 P ind. set P + r If x 2 P ind. set,then x = X i i1 Ii (10.17) for some appropriate vector =( 1, 2,..., n). Clearly, for such x, x 0. Now, for any A E, x(a) =x 1 A = X i i1 Ii 1 A (10.18) apple X i i max j:i j A 1 I j (E) (10.19) = max I j (E) = max A \ I j:i j A I2I (10.20) = r(a) (10.21) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F23/64 (pg.59/203)

60 P ind. set P + r If x 2 P ind. set,then x = X i i1 Ii (10.17) for some appropriate vector =( 1, 2,..., n). Clearly, for such x, x 0. Now, for any A E, x(a) =x 1 A = X i i1 Ii 1 A (10.18) apple X i i max j:i j A 1 I j (E) (10.19) = max I j (E) = max A \ I j:i j A I2I (10.20) = r(a) (10.21) Thus, x 2 P r + and hence P ind. set P r +. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F23/64 (pg.60/203)

61 Matroid Polyhedron in 2D P + r = x 2 R E : x 0,x(A) apple r(a), 8A E (10.22) Consider this in two dimensions. We have equations of the form: x 1 0 and x 2 0 (10.23) x 1 apple r({v 1 }) E { 0113 (10.24) the x 2 apple r({v 2 }) (10.25) x 1 + x 2 apple r({v 1,v 2 }) - E 942 } (10.26) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F24/64 (pg.61/203)

62 Matroid Polyhedron in 2D P + r = x 2 R E : x 0,x(A) apple r(a), 8A E (10.22) Consider this in two dimensions. We have equations of the form: Because r is submodular, we have x 1 0 and x 2 0 (10.23) x 1 apple r({v 1 }) (10.24) x 2 apple r({v 2 }) (10.25) x 1 + x 2 apple r({v 1,v 2 }) (10.26) r({v 1 })+r({v 2 }) r({v 1,v 2 })+r(;) (10.27) so since r({v 1,v 2 }) apple r({v 1 })+r({v 2 }), thelastinequalityiseither touching (so inactive) or active. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F24/64 (pg.62/203)

63 Matroid Polyhedron in 2D x 2 apple r({v 2 }) x1 + x 2 apple r({v 1, v 2 }) PF x 2 0 " 9 x 1 0 x 1 apple r({v 1 }) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F25/64 (pg.63/203)

64 Matroid Polyhedron in 2D x 2 r(v2)=1 x 1 + x 2 = r({v 1, v 2 }) = 1 - r(v1)=1 x 1 Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F25/64 (pg.64/203)

65 Matroid Polyhedron in 2D x 2 r({v 1, v 2 }) = 0 x 1 Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F25/64 (pg.65/203)

66 Matroid Polyhedron in 2D x 2 r(v2)=1 x 1 + x 2 a = r({v 1, v 2 }) = 2 r(v1)=1 x 1 Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F25/64 (pg.66/203)

67 Matroid Polyhedron in 2D And, if v2 is a loop... x 2 r({v 1, v 2 }) = 1 r(v2)=0 r(v1)=1 x 1 Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F25/64 (pg.67/203)

68 Matroid Polyhedron in 2D x 2 x 2 r(v2)=1 x 1 + x 2 = r({v 1, v 2}) = 1 x 2 r(v2)=1 x 1 + x 2 = r({v 1, v 2}) = 2 r({v 1, v 2}) = 0 x 1 x r(v1)=1 1 x r(v1)=1 1 And, if v2 is a loop... x 2 r({v 1, v 2}) = 1 x 2 apple r({v 2 }) x1 + x 2 apple r({v 1, v 2 }) r(v2)=0 x r(v1)=1 1 x 2 0 x 1 0 x 1 apple r({v 1 }) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F25/64 (pg.68/203)

69 Matroid Polyhedron in 2D x 2 apple r({v 2 }) x 2 0 Not Possible x 1 0 x 1 apple r({v 1 }) Possible x 1 + x 2 apple r({v 1, v 2 }) Not Possible Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F26/64 (pg.69/203)

70 Matroid Polyhedron in 3D P + r = x 2 R E : x 0,x(A) apple r(a), 8A E (10.28) Consider this in three dimensions. We have equations of the form: x 1 0 and x 2 0 and x 3 0 (10.29) x 1 apple r({v 1 }) (10.30) x 2 apple r({v 2 }) (10.31) x 3 apple r({v 3 }) (10.32) x 1 + x 2 apple r({v 1,v 2 }) (10.33) x 2 + x 3 apple r({v 2,v 3 }) (10.34) x 1 + x 3 apple r({v 1,v 3 }) (10.35) x 1 + x 2 + x 3 apple r({v 1,v 2,v 3 }) (10.36) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F27/64 (pg.70/203)

71 Matroid Polyhedron in 3D Consider the simple cycle matroid on a graph consisting of a 3-cycle, G =(V,E) with matroid M =(E,I) where I 2I is a forest. Eh Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F28/64 (pg.71/203)

72 Matroid Polyhedron in 3D Consider the simple cycle matroid on a graph consisting of a 3-cycle, G =(V,E) with matroid M =(E,I) where I 2I is a forest. So any set of either one or two edges is independent, and has rank equal to cardinality. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F28/64 (pg.72/203)

73 Matroid Polyhedron in 3D Consider the simple cycle matroid on a graph consisting of a 3-cycle, G =(V,E) with matroid M =(E,I) where I 2I is a forest. So any set of either one or two edges is independent, and has rank equal to cardinality. The set of three edges is dependent, and has rank 2. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F28/64 (pg.73/203)

74 Matroid Polyhedron in 3D Two view of P r + associated with a matroid ({e 1,e 2,e 3 }, {;, {e 1 }, {e 2 }, {e 3 }, {e 1,e 2 }, {e 1,e 3 }, {e 2,e 3 }}). Exe, x x " 1 diet x3 "#t x x1 x2 0 Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F29/64 (pg.74/203)

75 Matroid Polyhedron in 3D P + r associated with the free matroid in 3D. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F30/64 (pg.75/203)

76 Matroid Polyhedron in 3D P + r associated with the free matroid in 3D Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F30/64 (pg.76/203)

77 Another Polytope in 3D Thought question: what kind of polytope might this be? Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F31/64 (pg.77/203)

78 Another Polytope in 3D Thought question: what kind of polytope might this be? Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F31/64 (pg.78/203)

79 Matroid Independence Polyhedron So recall from a moment ago, that we have that P ind. set = conv {[ I2I {1 I }} P + r = x 2 R E : x 0,x(A) apple r(a), 8A E (10.37) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F32/64 (pg.79/203)

80 Matroid Independence Polyhedron So recall from a moment ago, that we have that P ind. set = conv {[ I2I {1 I }} P r + = x 2 R E : x 0,x(A) apple r(a), 8A E (10.37) In fact, the two polyhedra are identical (and thus both are polytopes). Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F32/64 (pg.80/203)

81 Matroid Independence Polyhedron So recall from a moment ago, that we have that P ind. set = conv {[ I2I {1 I }} P r + = x 2 R E : x 0,x(A) apple r(a), 8A E (10.37) In fact, the two polyhedra are identical (and thus both are polytopes). We ll show this in the next few theorems. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F32/64 (pg.81/203)

82 Maximum weight independent set via greedy weighted rank Theorem Let M =(V,I) be a matroid, with rank function r, then for any weight function w 2 R V +, there exists a chain of sets U 1 U 2 U n V such that where i 0 satisfy nx max {w(i) I 2I}= ir(u i ) (10.38) nx w = i=1 i=1 i1 Ui (10.39) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F33/64 (pg.82/203)

83 Maximum weight independent set via weighted rank Proof. Firstly, note that for any such w 2 R E,wehave 0 B B w 1 w 2. w n 1 C C C A = w 1 w 2 0 B B C C C A + w 2 w 3 0 B B B B C C C C C A + + w n 1 w n 0 B B B B C C C C C A + w n 0 B B B B C C C C C A (10.40)... Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F34/64 (pg.83/203) in :p

84 Maximum weight independent set via weighted rank Proof. Firstly, note that for any such w 2 R E,wehave 0 B B w 1 w 2. w n 1 C C C A = w 1 w 2 0 B B C C C A + w 2 w 3 0 B B B B C C C C C A + + w n 1 w n 0 B B B B C C C C C A + w n 0 B B B B C C C C C A (10.40) If we can take w in decreasing order (w 1 w 2 w n ), then each coe cient of the vectors is non-negative (except possibly the last one, w n ).... Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F34/64 (pg.84/203)

85 Maximum weight independent set via weighted rank Proof. Now, again assuming w 2 R E +, order the elements of V as (v 1,v 2,...,v n ) such that w(v 1 ) w(v 2 ) w(v n )... Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F34/64 (pg.85/203)

86 Maximum weight independent set via weighted rank Proof. Now, again assuming w 2 R E +, order the elements of V as (v 1,v 2,...,v n ) such that w(v 1 ) w(v 2 ) w(v n ) Define the sets U i based on this order as follows, for i =0,...,n Note that Vo U0 = C A, 1 U 1 = U i def = {v 1,v 2,...,v i } (10.41) >= 1 1 `. >; 1,...,1 C U` = 0 9, etc. A > = 0 (n `) B >; A 0 Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F34/64 (pg.86/203)

87 Maximum weight independent set via weighted rank Proof. Now, again assuming w 2 R E +, order the elements of V as (v 1,v 2,...,v n ) such that w(v 1 ) w(v 2 ) w(v n ) Define the sets U i based on this order as follows, for i =0,...,n U i def = {v 1,v 2,...,v i } (10.41) Define the set I as those elements where the rank increases, i.e.: I def = {v i r(u i ) >r(u i 1 )}. (10.42) Hence, given an i with v i /2 I, r(u i )=r(u i 1 ). rain Fetus... Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F34/64 (pg.87/203)

88 Maximum weight independent set via weighted rank Proof. Now, again assuming w 2 R E +, order the elements of V as (v 1,v 2,...,v n ) such that w(v 1 ) w(v 2 ) w(v n ) Define the sets U i based on this order as follows, for i =0,...,n U i def = {v 1,v 2,...,v i } (10.41) Define the set I as those elements where the rank increases, i.e.: I def = {v i r(u i ) >r(u i 1 )}. (10.42) Hence, given an i with v i /2 I, r(u i )=r(u i 1 ). Therefore, I is the output of the greedy algorithm for max {w(i) I 2I}. since items v i are ordered decreasing by w(v i), andweonly choose the ones that increase the rank, which means they don t violate independence. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F34/64 (pg.88/203)

89 Maximum weight independent set via weighted rank Proof. Now, again assuming w 2 R E +, order the elements of V as (v 1,v 2,...,v n ) such that w(v 1 ) w(v 2 ) w(v n ) Define the sets U i based on this order as follows, for i =0,...,n U i def = {v 1,v 2,...,v i } (10.41) Define the set I as those elements where the rank increases, i.e.: I def = {v i r(u i ) >r(u i 1 )}. (10.42) Hence, given an i with v i /2 I, r(u i )=r(u i 1 ). Therefore, I is the output of the greedy algorithm for max {w(i) I 2I}. And therefore, I is a maximum weight independent set (can even be a base, actually).... Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F34/64 (pg.89/203)

90 Maximum weight independent set via weighted rank Proof. Now, we define 0 i as follows : n def ± = w(v n ) it WEIR } i def = w(v i ) w(v i+1 ) for i =1,...,n 1 (10.43) (10.44)... Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F34/64 (pg.90/203)

91 Maximum weight independent set via weighted rank Proof. Now, we define i as follows i def = w(v i ) w(v i+1 ) for i =1,...,n 1 (10.43) n def = w(v n ) And the weight of the independent set w(i) is given by (10.44) w(i) = X v2i w(v) = (10.46)... Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F34/64 (pg.91/203)

92 Maximum weight independent set via weighted rank Proof. Now, we define i as follows i def = w(v i ) w(v i+1 ) for i =1,...,n 1 (10.43) n def = w(v n ) And the weight of the independent set w(i) is given by (10.44) w(i) = X v2i w(v) = nx w(v i ) r(u i ) r(u i 1 ) (10.45) i=1. I { 4.,t%EI} (10.46)... Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F34/64 (pg.92/203)

93 Maximum weight independent set via weighted rank Proof. Now, we define i as follows i def = w(v i ) w(v i+1 ) for i =1,...,n 1 (10.43) n def = w(v n ) And the weight of the independent set w(i) is given by w(i) = X v2i w(v) = (10.44) nx w(v i ) r(u i ) r(u i 1 ) (10.45) i=1 nx 1 = w(v n )r(u n )+ i=1 w(v i ) w(v i+1 ) r(u i ) (10.46)... Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F34/64 (pg.93/203)

94 Maximum weight independent set via weighted rank Proof. Now, we define i as follows i def = w(v i ) w(v i+1 ) for i =1,...,n 1 (10.43) n def = w(v n ) And the weight of the independent set w(i) is given by w(i) = X v2i w(v) = (10.44) nx w(v i ) r(u i ) r(u i 1 ) (10.45) i=1 nx 1 = w(v n )r(u n )+ i=1 w(v i ) w(v i+1 ) r(u i ) = nx i=1 ir(u i ) (10.46)... Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F34/64 (pg.94/203)

95 Maximum weight independent set via weighted rank Proof. Now, we define i as follows i def = w(v i ) w(v i+1 ) for i =1,...,n 1 (10.43) n def = w(v n ) And the weight of the independent set w(i) is given by w(i) = X v2i w(v) = (10.44) nx w(v i ) r(u i ) r(u i 1 ) (10.45) i=1 nx 1 = w(v n )r(u n )+ i=1 w(v i ) w(v i+1 ) r(u i ) = nx i=1 ir(u i ) (10.46) Since we took v 1,v 2,... in decreasing order, for all i, andsince w 2 R E +,wehave i 0 Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F34/64 (pg.95/203)

96 Linear Program LP Consider the linear programming primal problem maximize w x subject to x v 0 (v 2 V ) x(u) apple r(u) (8U V ) (10.47) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F35/64 (pg.96/203)

97 Linear Program LP Consider the linear programming primal problem maximize w x subject to x v 0 (v 2 V ) x(u) apple r(u) (8U V ) (10.47) And its convex dual (note y 2 R 2n +, y U is a scalar element within this exponentially big vector): P minimize U V y Ur(U), subject to y U 0 (8U V ) P U V y U1 U w (10.48) f Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F35/64 (pg.97/203)

98 Linear Program LP Consider the linear programming primal problem maximize w x subject to x v 0 (v 2 V ) x(u) apple r(u) (8U V ) (10.47) And its convex dual (note y 2 R 2n +, y U is a scalar element within this exponentially big vector): P minimize U V y Ur(U), subject to y U 0 (8U V ) P U V y U1 U w (10.48) Thanks to strong duality, the solutions to these are equal to each other. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F35/64 (pg.98/203)

99 Linear Program LP Consider the linear programming primal problem maximize w x s.t. x v 0 (v 2 V ) x(u) apple r(u) (8U V ) (10.49) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F36/64 (pg.99/203)

100 Linear Program LP Consider the linear programming primal problem maximize w x s.t. x v 0 (v 2 V ) x(u) apple r(u) (8U V ) (10.49) This is identical to the problem max w x such that x 2 P + r (10.50) where, again, P + r = x 2 R E : x 0,x(A) apple r(a), 8A E. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F36/64 (pg.100/203)

101 Linear Program LP Consider the linear programming primal problem maximize w x s.t. x v 0 (v 2 V ) x(u) apple r(u) (8U V ) (10.49) This is identical to the problem max w x such that x 2 P + r (10.50) where, again, P + r = x 2 R E : x 0,x(A) apple r(a), 8A E. Therefore, since P ind. set P + r, the above problem can only have a larger solution. I.e., max w x s.t. x 2 P ind. set apple max w x s.t. x 2 P + r. (10.51) - - Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F36/64 (pg.101/203)

102 Polytope equivalence Hence, we have the following relations: max {w(i) :I 2I}applemax {w x : x 2 P ind. set } (10.52) apple max w x : x 2 P r + (10.53) 8 9 < X def = min =min y : U r(u) :8U, y U 0; X = y U 1 U w ; U V U V (10.54) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F37/64 (pg.102/203)

103 Polytope equivalence Hence, we have the following relations: max {w(i) :I 2I}applemax {w x : x 2 P ind. set } (10.52) apple max w x : x 2 P r + (10.53) 8 9 < X def = min =min y : U r(u) :8U, y U 0; X = y U 1 U w ; U V U V Theorem states that (10.54) nx max {w(i) :I 2I}= ir(u i ) (10.55) i=1 +10 for the chain of U i s and i 0 that satisfies w = P n i=1 i1 Ui (i.e., the r.h.s. of Eq is feasible w.r.t. the dual LP). Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F37/64 (pg.103/203)

104 Polytope equivalence Hence, we have the following relations: max {w(i) :I 2I}applemax {w x : x 2 P ind. set } (10.52) apple max w x : x 2 P r + (10.53) 8 9 < X def = min =min y : U r(u) :8U, y U 0; X = y U 1 U w ; U V U V Theorem states that (10.54) nx max {w(i) :I 2I}= ir(u i ) (10.55) i=1 for the chain of U i s and i 0 that satisfies w = P n i=1 i1 Ui (i.e., the r.h.s. of Eq is feasible w.r.t. the dual LP). Therefore, we also have max {w(i) :I 2I}= i=1 Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F37/64 (pg.104/203) nx ir(u i ) min (10.56)

105 Polytope equivalence Hence, we have the following relations: max {w(i) :I 2I}applemax {w x : x 2 P ind. set } (10.52) apple max w x : x 2 P r + (10.53) 8 9 < X def = min =min y : U r(u) :8U, y U 0; X = y U 1 U w ; U V U V (10.54) Therefore, all the inequalities above are equalities. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F37/64 (pg.105/203)

106 Polytope equivalence Hence, we have the following relations: max {w(i) :I 2I}= max {w x : x 2 P ind. set } (10.52) = max w x : x 2 P r + (10.53) 8 9 < X def = min =min y : U r(u) :8U, y U 0; X = y U 1 U w ; U V U V (10.54) Therefore, all the inequalities above are equalities. : Fl And since w 2 R E + is an arbitrary direction into the positive orthant, we see that P r + = P ind. set Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F37/64 (pg.106/203)

107 Polytope equivalence Hence, we have the following relations: max {w(i) :I 2I}= max {w x : x 2 P ind. set } (10.52) = max w x : x 2 P r + (10.53) 8 9 < X def = min =min y : U r(u) :8U, y U 0; X = y U 1 U w ; U V U V (10.54) Therefore, all the inequalities above are equalities. And since w 2 R E + is an arbitrary direction into the positive orthant, we see that P r + = P ind. set That is, we have just proven: Theorem P + r = P ind. set (10.57) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F37/64 (pg.107/203)

108 Polytope Equivalence (Summarizing the above) For each I 2I of a matroid M =(E,I), wecanformtheincidence vector 1 I. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F38/64 (pg.108/203)

109 Polytope Equivalence (Summarizing the above) For each I 2I of a matroid M =(E,I), wecanformtheincidence vector 1 I. Taking the convex hull, we get the independent set polytope, that is P ind. set = conv {[ I2I {1 I }} (10.58) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F38/64 (pg.109/203)

110 Polytope Equivalence (Summarizing the above) For each I 2I of a matroid M =(E,I), wecanformtheincidence vector 1 I. Taking the convex hull, we get the independent set polytope, that is P ind. set = conv {[ I2I {1 I }} (10.58) Now take the rank function r of M, and define the following polyhedron: P + r = x 2 R E : x 0,x(A) apple r(a), 8A E (10.59) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F38/64 (pg.110/203)

111 Polytope Equivalence (Summarizing the above) For each I 2I of a matroid M =(E,I), wecanformtheincidence vector 1 I. Taking the convex hull, we get the independent set polytope, that is P ind. set = conv {[ I2I {1 I }} (10.58) Now take the rank function r of M, and define the following polyhedron: - polyto P + r = x 2 R E : x 0,x(A) apple r(a), 8A E (10.59) Theorem P + r = P ind. set (10.60) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F38/64 (pg.111/203)

112 Greedy solves a linear programming problem So we can describe the independence polytope of a matroid using the set of inequalities (an exponential number of them). Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F39/64 (pg.112/203)

113 Greedy solves a linear programming problem So we can describe the independence polytope of a matroid using the set of inequalities (an exponential number of them). In fact, considering equations starting at Eq 10.52, the LP problem with exponential number of constraints max {w x : x 2 P + r } is identical to the maximum weight independent set problem in a matroid, and since greedy solves the latter problem exactly, we have also proven: Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F39/64 (pg.113/203)

114 Greedy solves a linear programming problem So we can describe the independence polytope of a matroid using the set of inequalities (an exponential number of them). In fact, considering equations starting at Eq 10.52, the LP problem with exponential number of constraints max {w x : x 2 P + r } is identical to the maximum weight independent set problem in a matroid, and since greedy solves the latter problem exactly, we have also proven: Theorem The LP problem max {w x : x 2 P + r } can be solved exactly using the greedy algorithm. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F39/64 (pg.114/203)

115 Greedy solves a linear programming problem So we can describe the independence polytope of a matroid using the set of inequalities (an exponential number of them). In fact, considering equations starting at Eq 10.52, the LP problem with exponential number of constraints max {w x : x 2 P + r } is identical to the maximum weight independent set problem in a matroid, and since greedy solves the latter problem exactly, we have also proven: Theorem The LP problem max {w x : x 2 P + r } can be solved exactly using the greedy algorithm. Note that this LP problem has an exponential number of constraints (since P r + is described as the intersection of an exponential number of half spaces). Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F39/64 (pg.115/203)

116 Greedy solves a linear programming problem So we can describe the independence polytope of a matroid using the set of inequalities (an exponential number of them). In fact, considering equations starting at Eq 10.52, the LP problem with exponential number of constraints max {w x : x 2 P + r } is identical to the maximum weight independent set problem in a matroid, and since greedy solves the latter problem exactly, we have also proven: Theorem The LP problem max {w x : x 2 P + r } can be solved exactly using the greedy algorithm. Note that this LP problem has an exponential number of constraints (since P r + is described as the intersection of an exponential number of half spaces). This means that if LP problems have certain structure, they can be solved much easier than immediately implied by the equations. Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F39/64 (pg.116/203)

117 ... Matroid and Greedy Polyhedra Matroid Polytopes Polymatroid Base Polytope Equivalence Consider convex hull of indicator vectors just of the bases of a matroid, rather than all of the independent sets. Be opium " ( IB,, Ira,, Is,, :) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F40/64 (pg.117/203)

118 Base Polytope Equivalence Consider convex hull of indicator vectors just of the bases of a matroid, rather than all of the independent sets. Consider a polytope defined by the following constraints: x 0 (10.61) x(a) apple r(a) 8A V (10.62) x(v )=r(v ) (10.63) Prof. Je Bilmes EE596b/Spring 2016/Submodularity - Lecture 10 - May 2nd, 2016 F40/64 (pg.118/203)