Submodularity Reading Group. Matroid Polytopes, Polymatroid. M. Pawan Kumar

Transcription

1 Submodularity Reading Group Matroid Polytopes, Polymatroid M. Pawan Kumar

2 Outline Linear Programming Matroid Polytopes Polymatroid

3 Polyhedron Ax b A : m x n matrix b: m x 1 vector

4 Bounded Polyhedron = Polytope Ax b A : m x n matrix b: m x 1 vector

5 Vertex z is a vertex of P = {x, Ax b} z is not a convex combination of two points in P There does not exist x, y P and 0 < λ < 1 x z and y z such that z = λ x + (1-λ) y

6 Vertex z is a vertex of P = {x, Ax b} Recall A is an m x n matrix A z is a submatrix of A Contains all rows of A such that a it z = b i

7 Vertex z is a vertex of P Proof? Rank of A z = n See hidden slides

8 Proof Sketch: Necessity Let z be a vertex of P Suppose rank(a z ) < n A z c = 0 for some c 0 Then there exists a d > 0 such that z - dc P z + dc P Contradiction

9 Proof Sketch: Sufficiency Suppose rank(a z ) = n but z is not a vertex of P z = (x+y)/2 for some x, y P, x y z For each a in A z a T x b = a T z a T y b = a T z Implies A z (x-y) = 0 Contradiction

10 Linear Program Maximize a linear function max x c T x s.t. A x b Objective function Constraints Over a polyhedral feasible region A: m x n matrix b: m x 1 vector c: n x 1 vector x: n x 1 vector

11 Example max x x 1 + x 2 s.t. x 1 0 x 2 0 4x 1 x 2 8 2x 1 + x x 1-2x 2-2 What is c? A? b?

12 Example x 1 0 x 2 0

13 Example x 1 0 x 2 0 4x 1 x 2 = 8

14 Example x 1 0 x 2 0 4x 1 x 2 8

15 Example x 1 0 x 2 0 4x 1 x 2 8 2x 1 + x 2 10

16 Example x 1 0 x 2 0 4x 1 x 2 8 2x 1 + x x 1-2x 2-2

17 Example x 1 0 x 2 0 4x 1 x 2 8 2x 1 + x x 1-2x 2-2 x 1 + x 2 = 0 max x x 1 + x 2

18 Example x 1 + x 2 = 8 Optimal solution x 1 0 x 2 0 4x 1 x 2 8 2x 1 + x x 1-2x 2-2 max x x 1 + x 2

19 Outline Linear Programming Duality LP Solutions Matroid Polytopes Polymatroid

20 Example max x 3x 1 + x 2 + 2x 3 2 x 7 x s.t. -x 1 0, -x 2 0, -x x x 1 + x 2 + 3x x 1 + 2x 2 + 5x x 1 + x 2 + 2x 3 36 Scale the constraints, add them up 3x 1 + x 2 + 2x 3 90 Upper bound on solution

21 Example max x 3x 1 + x 2 + 2x 3 1 x s.t. -x 1 0, -x 2 0, -x 3 0 x 1 + x 2 + 3x x 1 + 2x 2 + 5x x 4x 1 + x 2 + 2x 3 36 Scale the constraints, add them up 3x 1 + x 2 + 2x 3 36 Upper bound on solution

22 Example max x 3x 1 + x 2 + 2x 3 1 x s.t. -x 1 0, -x 2 0, -x 3 0 x 1 + x 2 + 3x x 1 + 2x 2 + 5x x 4x 1 + x 2 + 2x 3 36 Scale the constraints, add them up 3x 1 + x 2 + 2x 3 36 Tightest upper bound?

23 Example max x 3x 1 + x 2 + 2x 3 s.t. y 4 y 5 y 6 y 1 y 2 y 3 -x 1 0, -x 2 0, -x 3 0 x 1 + x 2 + 3x x 1 + 2x 2 + 5x x 1 + x 2 + 2x 3 36 We should be able to add up the inequalities y 1, y 2, y 3, y 4, y 5, y 6 0

24 Example max x 3x 1 + x 2 + 2x 3 s.t. y 4 y 5 y 6 y 1 y 2 y 3 -x 1 0, -x 2 0, -x 3 0 x 1 + x 2 + 3x x 1 + 2x 2 + 5x x 1 + x 2 + 2x 3 36 Coefficient of x 1 should be 3 -y 1 + y 4 + 2y 5 + 4y 6 = 3

25 Example max x 3x 1 + x 2 + 2x 3 s.t. y 4 y 5 y 6 y 1 y 2 y 3 -x 1 0, -x 2 0, -x 3 0 x 1 + x 2 + 3x x 1 + 2x 2 + 5x x 1 + x 2 + 2x 3 36 Coefficient of x 2 should be 1 -y 2 + y 4 + 2y 5 + y 6 = 1

26 Example max x 3x 1 + x 2 + 2x 3 s.t. y 4 y 5 y 6 y 1 y 2 y 3 -x 1 0, -x 2 0, -x 3 0 x 1 + x 2 + 3x x 1 + 2x 2 + 5x x 1 + x 2 + 2x 3 36 Coefficient of x 3 should be 2 -y 3 + 3y 4 + 5y 5 + 2y 6 = 2

27 Example max x 3x 1 + x 2 + 2x 3 s.t. y 4 y 5 y 6 y 1 y 2 y 3 -x 1 0, -x 2 0, -x 3 0 x 1 + x 2 + 3x x 1 + 2x 2 + 5x x 1 + x 2 + 2x 3 36 Upper bound should be tightest min y 30y y y 6

28 Dual min y 30y y y 6 s.t. y 1, y 2, y 3, y 4, y 5, y 6 0 -y 1 + y 4 + 2y 5 + 4y 6 = 3 -y 2 + y 4 + 2y 5 + y 6 = 1 -y 3 + 3y 4 + 5y 5 + 2y 6 = 2 Original problem is called primal Dual of dual is primal

29 Dual max x c T x s.t. A x b

30 Dual min y 0 max x c T x - y T (A x b) KKT Condition? A T y = c min y 0 b T y s.t. A T y = c

31 max x c T x s.t. A x b Primal min y 0 b T y s.t. A T y = c Dual

32 Strong Duality p = max x c T x s.t. A x b Primal If p or d, then p = d. Think back to the intuition of dual d = min y 0 b T y s.t. A T y = c Dual Skipping the proof

33 Question max x c T x s.t. A 1 x b 1 A 2 x b 2 A 3 x = b 3 Dual?

34 Outline Linear Programming Duality LP Solutions Matroid Polytopes Polymatroid

35 Graphical Solution x 1 + x 2 = 8 x 1 0 x 2 0 4x 1 x 2 8 2x 1 + x x 1-2x 2-2 Optimal solution at a vertex max x x 1 + x 2

36 Graphical Solution x 1 = 3 x 1 0 x 2 0 4x 1 x 2 8 2x 1 + x x 1-2x 2-2 Optimal solution at a vertex max x x 1

37 Graphical Solution x 2 = 6 x 1 0 x 2 0 4x 1 x 2 8 2x 1 + x x 1-2x 2-2 Optimal solution at a vertex max x x 2

38 Graphical Solution x 1 0 x 2 0 4x 1 x 2 8 2x 1 + x x 1-2x 2-2 An optimal solution always at a vertex Proof? max x c T x

39 Solving the LP Dantzig (1951): Simplex Method Search over vertices of the polyhedra Worst-case complexity is exponential Smoothed complexity is polynomial Khachiyan (1979, 1980): Ellipsoid Method Polynomial time complexity LP is a P optimization problem Karmarkar (1984): Interior-point Method Polynomial time complexity Competitive with Simplex Method

40 Solving the LP Plenty of standard software available Mosek ( C++ API Matlab API Python API Free academic license

41 Outline Linear Programming Matroid Polytopes Polymatroid

42 Incidence Vector of Set Matroid M = (S, I) Set X S Incidence vector v X {0,1} S 1, if s X v X (s) = 0, if s X

43 Example (Uniform Matroid) S = {1,2,,9} s w(s) k = X = {1, 3, 5} v X?

44 Example (Graphic Matroid) S = E v e 0 1 e 2 X S e 3 v 1 v 2 e 4 e 6 v 4 e 5 v 5 e 7 e 9 v 3 e 8 v 6

45 Example (Graphic Matroid) S = E X S v X? e 3 v 1 v 2 v 3 v e 0 1 e 2 e 4 e 6 v 4 v 6 e 5 e 7 e 9 e 8 v 5

46 Incidence Vectors of Independent Sets Matroid M = (S, I) v X {0,1} S, X I A? Convex Hull Ax b b?

47 Outline Linear Programming Matroid Polytopes Independent Set Polytope Base Polytope Polymatroid

48 Independent Set Polytope Matroid M = (S, I) v X {0,1} S, X I x Real S Convex Hull Ax b

49 Independent Set Polytope Matroid M = (S, I) v X {0,1} S, X I x Real S Necessary conditions Sufficient for integral x Proof? x s 0, for all s S s U x s r M (U), for all U S Why? Why?

50 Independent Set Polytope Matroid M = (S, I) v X {0,1} S, X I x Real S Necessary conditions Sufficient for all x Lectures slides for proof x s 0, for all s S s U x s r M (U), for all U S Why? Why?

51 Outline Linear Programming Matroid Polytopes Independent Set Polytope Base Polytope Polymatroid

52 Base Polytope Matroid M = (S, I) v X {0,1} S, X B x Real S Convex Hull Ax b

53 Matroid M = (S, I) v X {0,1} S, X B Base Polytope x Real S s S x s = r M (S) x s 0, for all s S s U x s r M (U), for all U S

54 Outline Linear Programming Matroid Polytopes Polymatroid

55 Polymatroid Set S Submodular function f Real vector x of size S x1 P f = {x 0, x(u) f(u) for all U S} Polymatroid EP f = {x(u) f(u) for all U S} Extended Polymatroid

56 Outline Linear Programming Matroid Polytopes Polymatroid Tight Sets Optimization

57 Tight Sets Set S Submodular function f EP f = {x(u) f(u) for all U S} U is tight with respect to x EP f if x(u) = f(u) Tight sets are closed under union Proof? Tight sets are closed under intersection

58 Proof Sketch Let T and U be tight wrt x EP f f(t) + f(u) f(t U) + f(t U) x(t U) + x(t U) = x(t) + x(u) = f(t) + f(u) All inequalities must be equalities

59 Outline Linear Programming Matroid Polytopes Polymatroid Tight Sets Optimization

60 Decreasing f(null set) maintains submodularity Primal Problem max w T x x EP f Assume f(null set) 0 Otherwise EP f is empty f(null set) can be set to 0 Why?

61 Primal Problem max w T x x EP f Assume w 0 Otherwise the optimal solution is infinity Why?

62 Greedy Algorithm max w T x x EP f Order s 1,s 2,,s n S such that w(s i ) w(s i+1 ) Define U i = {s 1,s 2,..,s i } x G i = f(u i ) f(u i-1 ) x G EP f Proof? Hidden slides

63 Proof Sketch We have to show that x G (T) f(t) for all T S Trivial when T = null set Mathematical induction on T

64 Proof Sketch Let k be the largest index such that s k T Clearly T U k x G (T) = x G (T\{s k }) + x G k f(t\{s k }) + x G k Induction Why? = f(t\{s k }) + f(u k ) - f(u k-1 ) Definition f(t) Submodularity Why? Why?

65 Greedy Algorithm max w T x x EP f Order s 1,s 2,,s n S such that w(s i ) w(s i+1 ) Define U i = {s 1,s 2,..,s i } x G i = f(u i ) f(u i-1 ) x G is optimal Proof? Hidden slides

66 Dual Problem max w T x x EP f min T y T f(t) y T 0, for all T S T y T v T = w Let us first try to find a feasible dual solution

67 Greedy Algorithm Order s 1,s 2,,s n S such that w(s i ) w(s i+1 ) Define U i = {s 1,s 2,..,s i } y G U i = w(s i ) - w(s i+1 ) y G S = w(s n ) y G is feasible y G T = 0, for all other T Proof? Hidden slides

68 Proof Sketch Trivially, y G 0 Consider s i S T si y G T = j i yg U j = w(s i ) T y T v T = w

69 Optimality Primal feasible solution x G Dual feasible solution y G Primal value at x G = Dual value at y G Proof? Hidden slides

70 Proof Sketch w T x G = s S w(s)x G s = i {1,2,,n} w(s i )(f(u i ) - f(u i-1 )) = i {1,2,,n-1} f(u i )(w(s i ) - w(s i+1 )) + f(s)w(s n ) = T y G T f(t)

71 Optimality Primal feasible solution x G Dual feasible solution y G Primal value at x G = Dual value at y G Therefore, x G is an optimal primal solution And, y G is an optimal dual solution