CS 473: Algorithms. Ruta Mehta. Spring University of Illinois, Urbana-Champaign. Ruta (UIUC) CS473 1 Spring / 29

Transcription

1 CS 473: Algorithms Ruta Mehta University of Illinois, Urbana-Champaign Spring 2018 Ruta (UIUC) CS473 1 Spring / 29

2 CS 473: Algorithms, Spring 2018 Simplex and LP Duality Lecture 19 March 29, 2018 Some of the slides are courtesy Prof. Chekuri Ruta (UIUC) CS473 2 Spring / 29

3 Outline Simplex: Intuition and Implementation Details Computing starting vertex: equivalent to solving an LP! Infeasibility, Unboundedness, and Degeneracy. Duality: Bounding the objective value through weak-duality Strong Duality, Cone view. Ruta (UIUC) CS473 3 Spring / 29

4 Part I Recall Ruta (UIUC) CS473 4 Spring / 29

5 Feasible Region and Convexity Canonical Form Given A R n d, b R n 1 and c R 1 d, find x R d 1 max : s.t. c x Ax b Ruta (UIUC) CS473 5 Spring / 29

6 Feasible Region and Convexity Canonical Form Given A R n d, b R n 1 and c R 1 d, find x R d 1 max : s.t. c x Ax b 1 Each linear constraint defines a halfspace, a convex set. 2 Feasible region, which is an intersection of halfspaces, is a convex polyhedron. 3 Optimal value attained at a vertex of the polyhedron. Ruta (UIUC) CS473 5 Spring / 29

7 Ruta (UIUC) CS473 6 Spring / 29

8 Part II Simplex Ruta (UIUC) CS473 7 Spring / 29

9 Simplex Algorithm Simplex: Vertex hoping algorithm Moves from a vertex to its neighboring vertex Ruta (UIUC) CS473 8 Spring / 29

10 Simplex Algorithm Simplex: Vertex hoping algorithm Moves from a vertex to its neighboring vertex Questions Which neighbor to move to? When to stop? How much time does it take? Ruta (UIUC) CS473 8 Spring / 29

11 Observations For Simplex Suppose we are at a non-optimal vertex ˆx and optimal is x, then c x > c ˆx. Ruta (UIUC) CS473 9 Spring / 29

12 Observations For Simplex Suppose we are at a non-optimal vertex ˆx and optimal is x, then c x > c ˆx. How does (c x) change as we move from ˆx to x on the line joining the two? Ruta (UIUC) CS473 9 Spring / 29

13 Observations For Simplex Suppose we are at a non-optimal vertex ˆx and optimal is x, then c x > c ˆx. How does (c x) change as we move from ˆx to x on the line joining the two? Strictly increases! Ruta (UIUC) CS473 9 Spring / 29

14 Observations For Simplex Suppose we are at a non-optimal vertex ˆx and optimal is x, then c x > c ˆx. How does (c x) change as we move from ˆx to x on the line joining the two? Strictly increases! d = x ˆx is the direction from ˆx to x. (c d) = (c x ) (c ˆx) > 0. In x = ˆx + δd, as δ goes from 0 to 1, we move from ˆx to x. c x = c ˆx + δ(c d). Strictly increasing with δ! Due to convexity, all of these are feasible points. Ruta (UIUC) CS473 9 Spring / 29

15 Cone Definition Given a set of vectors D = {d 1,..., d k }, the cone spanned by them is just their positive linear combinations, i.e., cone(d) = {d d = k λ i d i, where λ i 0, i} i =1 Ruta (UIUC) CS Spring / 29

16 Cone at a Vertex Let z 1,..., z k be the neighboring vertices of ˆx. And let d i = z i ˆx be the direction from ˆx to z i. Lemma Any feasible direction of movement d from ˆx is in the cone({d 1,..., d k }). Ruta (UIUC) CS Spring / 29

17 Improving Direction Implies Improving Neighbor Lemma If d cone({d 1,..., d k }) and (c d) > 0, then there exists d i such that (c d i ) > 0. Ruta (UIUC) CS Spring / 29

18 Improving Direction Implies Improving Neighbor Lemma If d cone({d 1,..., d k }) and (c d) > 0, then there exists d i such that (c d i ) > 0. Proof. To the contrary suppose (c d i ) 0, i k. Since d is a positive linear combination of d i s, (c d) = (c k i =1 λ i d i ) = k i =1 λ i (c d i ) 0 A contradiction! Ruta (UIUC) CS Spring / 29

19 Improving Direction Implies Improving Neighbor Lemma If d cone({d 1,..., d k }) and (c d) > 0, then there exists d i such that (c d i ) > 0. Proof. To the contrary suppose (c d i ) 0, i k. Since d is a positive linear combination of d i s, (c d) = (c k i =1 λ i d i ) = k i =1 λ i (c d i ) 0 A contradiction! Theorem If vertex ˆx is not optimal then it has a neighbor where cost improves. Ruta (UIUC) CS Spring / 29

20 How Many Neighbors a Vertex Has? Geometric view... A R n d (n > d), b R n, the constraints are: Ax b Geometry of faces r linearly independent hyperplanes forms (d r) dimensional face. Ruta (UIUC) CS Spring / 29

21 How Many Neighbors a Vertex Has? Geometric view... A R n d (n > d), b R n, the constraints are: Ax b Geometry of faces r linearly independent hyperplanes forms (d r) dimensional face. Vertex: 0-D face. formed by d L.I. hyperplanes. Ruta (UIUC) CS Spring / 29

22 How Many Neighbors a Vertex Has? Geometric view... A R n d (n > d), b R n, the constraints are: Ax b Geometry of faces r linearly independent hyperplanes forms (d r) dimensional face. Vertex: 0-D face. formed by d L.I. hyperplanes. Edge: 1-D face. formed by (d 1) L.I. hyperlanes. Ruta (UIUC) CS Spring / 29

23 How Many Neighbors a Vertex Has? Geometric view... A R n d (n > d), b R n, the constraints are: Ax b Geometry of faces r linearly independent hyperplanes forms (d r) dimensional face. Vertex: 0-D face. formed by d L.I. hyperplanes. Edge: 1-D face. formed by (d 1) L.I. hyperlanes. In 2-dimension (d = 2) x x 1 Ruta (UIUC) CS Spring / 29

24 How Many Neighbors a Vertex Has? Geometric view... A R n d (n > d), b R n, the constraints are: Ax b In 3-dimension (d = 3) Geometry of faces r linearly independent hyperplanes forms (d r) dimensional face. Vertex: 0-dimensional face. formed by d L.I. hyperplanes Edge: 1-D face. formed by (d 1) L.I. hyperlanes. image source: webpage of Prof. Forbes W. Lewis Ruta (UIUC) CS Spring / 29

25 How Many Neighbors a Vertex Has? Geometry view... One neighbor per tight hyperplane. Therefore typically d. Suppose x is a neighbor of ˆx, then on the edge joining the two d 1 constraints are tight. These d 1 are also tight at both ˆx and x. One more constraints, say i, is tight at ˆx. Relaxing i at ˆx leads to x. 3 x Ruta (UIUC) CS Spring / 29

26 Simplex Algorithm Simplex: Vertex hoping algorithm Moves from a vertex to its neighboring vertex Questions + Answers Which neighbor to move to? One where objective value increases. Ruta (UIUC) CS Spring / 29

27 Simplex Algorithm Simplex: Vertex hoping algorithm Moves from a vertex to its neighboring vertex Questions + Answers Which neighbor to move to? One where objective value increases. When to stop? When no neighbor with better objective value. Ruta (UIUC) CS Spring / 29

28 Simplex Algorithm Simplex: Vertex hoping algorithm Moves from a vertex to its neighboring vertex Questions + Answers Which neighbor to move to? One where objective value increases. When to stop? When no neighbor with better objective value. How much time does it take? At most d neighbors to consider in each step. Ruta (UIUC) CS Spring / 29

29 Simplex in Higher Dimensions Simplex Algorithm 1 Start at a vertex of the polytope. 2 Compare value of objective function at each of the d neighbors. 3 Move to neighbor that improves objective function, and repeat step 2. 4 If no improving neighbor, then stop. Simplex is a greedy local-improvement algorithm! Works because a local optimum is also a global optimum convexity of polyhedra. Ruta (UIUC) CS Spring / 29

30 Part III Implementation of the Pivoting Step (Moving to an improving neighbor) Ruta (UIUC) CS Spring / 29

31 Moving to a Neighbor Fix a vertex ˆx. Let the d hyperplanes/constraints tight at ˆx be, d a ij x j = b i, 1 i d Equivalently, Âx = ˆb j=1 A neighbor vertex x is connected to ˆx by an edge. d 1 hyperplanes tight on this edge. To reach x, one hyperplane has to be relaxed, while maintaining other d 1 tight. 3 x Ruta (UIUC) CS Spring / 29

32 Moving to a Neighbor (Contd.)  1 =.. d 1... d d.. Lemma Moving in direction d i from ˆx, all except constraint i remain tight. Proof. For a small ɛ > 0, let y = ˆx + ɛ(d i ), then Ây = Â(ˆx + ɛd i ) = ˆx + ɛâ(  1 ) (.,i ) Ruta (UIUC) CS Spring / 29

33 Moving to a Neighbor (Contd.)  1 =.. d 1... d d.. Lemma Moving in direction d i from ˆx, all except constraint i remain tight. Proof. For a small ɛ > 0, let y = ˆx + ɛ(d i ), then Ây = Â(ˆx + ɛd i ) = ˆx + ɛâ(  1 ) (.,i ) = ˆb + ɛ[0,..., 1,..., 0] T Clearly, j a kjy j = b k, k i, and j a ijy j = b i ɛ < b i. Ruta (UIUC) CS Spring / 29

34 Computing the Neighbor Move in d i direction from ˆx, i.e., ˆx + ɛd i, and STOP when hit a new hyperplane! Need to ensure feasibility. Above lemma implies inequalities 1 through d will be satisfied. For any k > d, where A k is k th row of A, A k (ˆx + ɛd i ) b k (A k ˆx) + ɛ(a k d i ) b k ɛ(a k d i ) b k (A k ˆx) Ruta (UIUC) CS Spring / 29

35 Computing the Neighbor Move in d i direction from ˆx, i.e., ˆx + ɛd i, and STOP when hit a new hyperplane! Need to ensure feasibility. Above lemma implies inequalities 1 through d will be satisfied. For any k > d, where A k is k th row of A, A k (ˆx + ɛd i ) b k (A k ˆx) + ɛ(a k d i ) b k ɛ(a k d i ) b k (A k ˆx) (If (A k d i ) > 0) ɛ b k (A k ˆx ) A k d i Ruta (UIUC) CS Spring / 29

36 Computing the Neighbor Move in d i direction from ˆx, i.e., ˆx + ɛd i, and STOP when hit a new hyperplane! Need to ensure feasibility. Above lemma implies inequalities 1 through d will be satisfied. For any k > d, where A k is k th row of A, A k (ˆx + ɛd i ) b k (A k ˆx) + ɛ(a k d i ) b k ɛ(a k d i ) b k (A k ˆx) (If (A k d i ) > 0) ɛ b k (A k ˆx ) A k d i If moving towards hyperplane k (positive) Ruta (UIUC) CS Spring / 29

37 Computing the Neighbor Move in d i direction from ˆx, i.e., ˆx + ɛd i, and STOP when hit a new hyperplane! Need to ensure feasibility. Above lemma implies inequalities 1 through d will be satisfied. For any k > d, where A k is k th row of A, A k (ˆx + ɛd i ) b k (A k ˆx) + ɛ(a k d i ) b k ɛ(a k d i ) b k (A k ˆx) (If (A k d i ) > 0) ɛ b k (A k ˆx ) A k d i If moving towards hyperplane k (If (A k d i ) < 0) ɛ b k (A k ˆx ) A k d i If moving away from hyperplane k. No upper bound, and -ve lower bound! (positive) (negative) Ruta (UIUC) CS Spring / 29

38 Computing the Neighbor Algorithm NextVertex(ˆx, d i ) Set ɛ. For k = d n ɛ b k (A k ˆx ) A k d i If ɛ > 0 and ɛ < ɛ then set ɛ ɛ If ɛ < then return ˆx + ɛd i. Else return null. If (c d i ) > 0 then the algorithm returns an improving neighbor. Ruta (UIUC) CS Spring / 29

39 Factory Example max : x 1 + 6x 2 s.t. 0 x x x 1 + x ˆx = (0, 0) Ruta (UIUC) CS Spring / 29

40 Factory Example max : x 1 + 6x 2 s.t. 0 x x x 1 + x ˆx = (0, 0) [ ] 1 0 Â = 0 1 [ ] Â = = [d d 2 ] Ruta (UIUC) CS Spring / 29

41 Factory Example max : x 1 + 6x 2 s.t. 0 x x x 1 + x ˆx = (0, 0) [ ] 1 0 Â = 0 1 [ ] Â = = [d d 2 ] Moving in direction d 1 gives (200, 0) Ruta (UIUC) CS Spring / 29

42 Factory Example max : x 1 + 6x 2 s.t. 0 x x x 1 + x ˆx = (0, 0) [ ] 1 0 Â = 0 1 [ ] Â = = [d d 2 ] Moving in direction d 1 gives (200, 0) Moving in direction d 2 gives (0, 300). Ruta (UIUC) CS Spring / 29

43 Solving Linear Programming in Practice 1 Naïve implementation of Simplex algorithm can be very inefficient Exponential number of steps! Ruta (UIUC) CS Spring / 29

44 Solving Linear Programming in Practice 1 Naïve implementation of Simplex algorithm can be very inefficient 1 Choosing which neighbor to move to can significantly affect running time 2 Very efficient Simplex-based algorithms exist 3 Simplex algorithm takes exponential time in the worst case but works extremely well in practice with many improvements over the years 2 Non Simplex based methods like interior point methods work well for large problems. Ruta (UIUC) CS Spring / 29

45 Issues 1 Starting vertex 2 The linear program could be infeasible: No point satisfy the constraints. 3 The linear program could be unbounded: Polygon unbounded in the direction of the objective function. 4 More than d hyperplanes could be tight at a vertex, forming more than d neighbors. Ruta (UIUC) CS Spring / 29

46 Computing the Starting Vertex Equivalent to solving another LP! Find an x such that Ax b. If b 0 then trivial! Ruta (UIUC) CS Spring / 29

47 Computing the Starting Vertex Equivalent to solving another LP! Find an x such that Ax b. If b 0 then trivial! x = 0. Otherwise. Ruta (UIUC) CS Spring / 29

48 Computing the Starting Vertex Equivalent to solving another LP! Find an x such that Ax b. If b 0 then trivial! x = 0. Otherwise. min : s.t. Trivial feasible solution: s j a ijx j s b i, s 0 i Ruta (UIUC) CS Spring / 29

49 Computing the Starting Vertex Equivalent to solving another LP! Find an x such that Ax b. If b 0 then trivial! x = 0. Otherwise. min : s.t. s j a ijx j s b i, s 0 i Trivial feasible solution: x = 0, s = min i b i. Ruta (UIUC) CS Spring / 29

50 Computing the Starting Vertex Equivalent to solving another LP! Find an x such that Ax b. If b 0 then trivial! x = 0. Otherwise. min : s.t. s j a ijx j s b i, s 0 i Trivial feasible solution: x = 0, s = min i b i. If Ax b feasible then optimal value of the above LP is s = 0. Ruta (UIUC) CS Spring / 29

51 Computing the Starting Vertex Equivalent to solving another LP! Find an x such that Ax b. If b 0 then trivial! x = 0. Otherwise. min : s.t. s j a ijx j s b i, s 0 i Trivial feasible solution: x = 0, s = min i b i. If Ax b feasible then optimal value of the above LP is s = 0. Checks Feasibility! Ruta (UIUC) CS Spring / 29

52 Unboundedness: Example maximize x 2 x 1 + x 2 2 x 1, x 2 0 Unboundedness depends on both constraints and the objective function. Ruta (UIUC) CS Spring / 29

53 Unboundedness: Example maximize x 2 x 1 + x 2 2 x 1, x 2 0 Unboundedness depends on both constraints and the objective function. If unbounded in the direction of objective function, then the pivoting step in the simplex will detect it. Ruta (UIUC) CS Spring / 29

54 Degeneracy and Cycling More than d constraints are tight at vertex ˆx. Say d + 1. Suppose, we pick first d to form  such that ˆx = ˆb, and compute directions d 1,..., d d. Ruta (UIUC) CS Spring / 29

55 Degeneracy and Cycling More than d constraints are tight at vertex ˆx. Say d + 1. Suppose, we pick first d to form  such that ˆx = ˆb, and compute directions d 1,..., d d. Then NextVertex(ˆx, d i ) will encounter (d + 1) th constraint tight at ˆx and return the same vertex. Hence we are back to ˆx! Ruta (UIUC) CS Spring / 29

56 Degeneracy and Cycling More than d constraints are tight at vertex ˆx. Say d + 1. Suppose, we pick first d to form  such that ˆx = ˆb, and compute directions d 1,..., d d. Then NextVertex(ˆx, d i ) will encounter (d + 1) th constraint tight at ˆx and return the same vertex. Hence we are back to ˆx! Same phenomenon will repeat! Ruta (UIUC) CS Spring / 29

57 Degeneracy and Cycling More than d constraints are tight at vertex ˆx. Say d + 1. Suppose, we pick first d to form  such that ˆx = ˆb, and compute directions d 1,..., d d. Then NextVertex(ˆx, d i ) will encounter (d + 1) th constraint tight at ˆx and return the same vertex. Hence we are back to ˆx! Same phenomenon will repeat! This can be avoided by adding small random perturbation to b i s. Ruta (UIUC) CS Spring / 29