CS 473: Algorithms Ruta Mehta University of Illinois, Urbana-Champaign Spring 2018 Ruta (UIUC) CS473 1 Spring 2018 1 / 29
CS 473: Algorithms, Spring 2018 Simplex and LP Duality Lecture 19 March 29, 2018 Some of the slides are courtesy Prof. Chekuri Ruta (UIUC) CS473 2 Spring 2018 2 / 29
Outline Simplex: Intuition and Implementation Details Computing starting vertex: equivalent to solving an LP! Infeasibility, Unboundedness, and Degeneracy. Duality: Bounding the objective value through weak-duality Strong Duality, Cone view. Ruta (UIUC) CS473 3 Spring 2018 3 / 29
Part I Recall Ruta (UIUC) CS473 4 Spring 2018 4 / 29
Feasible Region and Convexity Canonical Form Given A R n d, b R n 1 and c R 1 d, find x R d 1 max : s.t. c x Ax b Ruta (UIUC) CS473 5 Spring 2018 5 / 29
Feasible Region and Convexity Canonical Form Given A R n d, b R n 1 and c R 1 d, find x R d 1 max : s.t. c x Ax b 1 Each linear constraint defines a halfspace, a convex set. 2 Feasible region, which is an intersection of halfspaces, is a convex polyhedron. 3 Optimal value attained at a vertex of the polyhedron. Ruta (UIUC) CS473 5 Spring 2018 5 / 29
Ruta (UIUC) CS473 6 Spring 2018 6 / 29
Part II Simplex Ruta (UIUC) CS473 7 Spring 2018 7 / 29
Simplex Algorithm Simplex: Vertex hoping algorithm Moves from a vertex to its neighboring vertex Ruta (UIUC) CS473 8 Spring 2018 8 / 29
Simplex Algorithm Simplex: Vertex hoping algorithm Moves from a vertex to its neighboring vertex Questions Which neighbor to move to? When to stop? How much time does it take? Ruta (UIUC) CS473 8 Spring 2018 8 / 29
Observations For Simplex Suppose we are at a non-optimal vertex ˆx and optimal is x, then c x > c ˆx. Ruta (UIUC) CS473 9 Spring 2018 9 / 29
Observations For Simplex Suppose we are at a non-optimal vertex ˆx and optimal is x, then c x > c ˆx. How does (c x) change as we move from ˆx to x on the line joining the two? Ruta (UIUC) CS473 9 Spring 2018 9 / 29
Observations For Simplex Suppose we are at a non-optimal vertex ˆx and optimal is x, then c x > c ˆx. How does (c x) change as we move from ˆx to x on the line joining the two? Strictly increases! Ruta (UIUC) CS473 9 Spring 2018 9 / 29
Observations For Simplex Suppose we are at a non-optimal vertex ˆx and optimal is x, then c x > c ˆx. How does (c x) change as we move from ˆx to x on the line joining the two? Strictly increases! d = x ˆx is the direction from ˆx to x. (c d) = (c x ) (c ˆx) > 0. In x = ˆx + δd, as δ goes from 0 to 1, we move from ˆx to x. c x = c ˆx + δ(c d). Strictly increasing with δ! Due to convexity, all of these are feasible points. Ruta (UIUC) CS473 9 Spring 2018 9 / 29
Cone Definition Given a set of vectors D = {d 1,..., d k }, the cone spanned by them is just their positive linear combinations, i.e., cone(d) = {d d = k λ i d i, where λ i 0, i} i =1 Ruta (UIUC) CS473 10 Spring 2018 10 / 29
Cone at a Vertex Let z 1,..., z k be the neighboring vertices of ˆx. And let d i = z i ˆx be the direction from ˆx to z i. Lemma Any feasible direction of movement d from ˆx is in the cone({d 1,..., d k }). Ruta (UIUC) CS473 11 Spring 2018 11 / 29
Improving Direction Implies Improving Neighbor Lemma If d cone({d 1,..., d k }) and (c d) > 0, then there exists d i such that (c d i ) > 0. Ruta (UIUC) CS473 12 Spring 2018 12 / 29
Improving Direction Implies Improving Neighbor Lemma If d cone({d 1,..., d k }) and (c d) > 0, then there exists d i such that (c d i ) > 0. Proof. To the contrary suppose (c d i ) 0, i k. Since d is a positive linear combination of d i s, (c d) = (c k i =1 λ i d i ) = k i =1 λ i (c d i ) 0 A contradiction! Ruta (UIUC) CS473 12 Spring 2018 12 / 29
Improving Direction Implies Improving Neighbor Lemma If d cone({d 1,..., d k }) and (c d) > 0, then there exists d i such that (c d i ) > 0. Proof. To the contrary suppose (c d i ) 0, i k. Since d is a positive linear combination of d i s, (c d) = (c k i =1 λ i d i ) = k i =1 λ i (c d i ) 0 A contradiction! Theorem If vertex ˆx is not optimal then it has a neighbor where cost improves. Ruta (UIUC) CS473 12 Spring 2018 12 / 29
How Many Neighbors a Vertex Has? Geometric view... A R n d (n > d), b R n, the constraints are: Ax b Geometry of faces r linearly independent hyperplanes forms (d r) dimensional face. Ruta (UIUC) CS473 13 Spring 2018 13 / 29
How Many Neighbors a Vertex Has? Geometric view... A R n d (n > d), b R n, the constraints are: Ax b Geometry of faces r linearly independent hyperplanes forms (d r) dimensional face. Vertex: 0-D face. formed by d L.I. hyperplanes. Ruta (UIUC) CS473 13 Spring 2018 13 / 29
How Many Neighbors a Vertex Has? Geometric view... A R n d (n > d), b R n, the constraints are: Ax b Geometry of faces r linearly independent hyperplanes forms (d r) dimensional face. Vertex: 0-D face. formed by d L.I. hyperplanes. Edge: 1-D face. formed by (d 1) L.I. hyperlanes. Ruta (UIUC) CS473 13 Spring 2018 13 / 29
How Many Neighbors a Vertex Has? Geometric view... A R n d (n > d), b R n, the constraints are: Ax b Geometry of faces r linearly independent hyperplanes forms (d r) dimensional face. Vertex: 0-D face. formed by d L.I. hyperplanes. Edge: 1-D face. formed by (d 1) L.I. hyperlanes. In 2-dimension (d = 2) x 2 300 200 x 1 Ruta (UIUC) CS473 13 Spring 2018 13 / 29
How Many Neighbors a Vertex Has? Geometric view... A R n d (n > d), b R n, the constraints are: Ax b In 3-dimension (d = 3) Geometry of faces r linearly independent hyperplanes forms (d r) dimensional face. Vertex: 0-dimensional face. formed by d L.I. hyperplanes. 3 2 1 Edge: 1-D face. formed by (d 1) L.I. hyperlanes. image source: webpage of Prof. Forbes W. Lewis Ruta (UIUC) CS473 14 Spring 2018 14 / 29
How Many Neighbors a Vertex Has? Geometry view... One neighbor per tight hyperplane. Therefore typically d. Suppose x is a neighbor of ˆx, then on the edge joining the two d 1 constraints are tight. These d 1 are also tight at both ˆx and x. One more constraints, say i, is tight at ˆx. Relaxing i at ˆx leads to x. 3 x 2 1 4 Ruta (UIUC) CS473 15 Spring 2018 15 / 29
Simplex Algorithm Simplex: Vertex hoping algorithm Moves from a vertex to its neighboring vertex Questions + Answers Which neighbor to move to? One where objective value increases. Ruta (UIUC) CS473 16 Spring 2018 16 / 29
Simplex Algorithm Simplex: Vertex hoping algorithm Moves from a vertex to its neighboring vertex Questions + Answers Which neighbor to move to? One where objective value increases. When to stop? When no neighbor with better objective value. Ruta (UIUC) CS473 16 Spring 2018 16 / 29
Simplex Algorithm Simplex: Vertex hoping algorithm Moves from a vertex to its neighboring vertex Questions + Answers Which neighbor to move to? One where objective value increases. When to stop? When no neighbor with better objective value. How much time does it take? At most d neighbors to consider in each step. Ruta (UIUC) CS473 16 Spring 2018 16 / 29
Simplex in Higher Dimensions Simplex Algorithm 1 Start at a vertex of the polytope. 2 Compare value of objective function at each of the d neighbors. 3 Move to neighbor that improves objective function, and repeat step 2. 4 If no improving neighbor, then stop. Simplex is a greedy local-improvement algorithm! Works because a local optimum is also a global optimum convexity of polyhedra. Ruta (UIUC) CS473 17 Spring 2018 17 / 29
Part III Implementation of the Pivoting Step (Moving to an improving neighbor) Ruta (UIUC) CS473 18 Spring 2018 18 / 29
Moving to a Neighbor Fix a vertex ˆx. Let the d hyperplanes/constraints tight at ˆx be, d a ij x j = b i, 1 i d Equivalently, Âx = ˆb j=1 A neighbor vertex x is connected to ˆx by an edge. d 1 hyperplanes tight on this edge. To reach x, one hyperplane has to be relaxed, while maintaining other d 1 tight. 3 x 2 1 4 Ruta (UIUC) CS473 19 Spring 2018 19 / 29
Moving to a Neighbor (Contd.)  1 =.. d 1... d d.. Lemma Moving in direction d i from ˆx, all except constraint i remain tight. Proof. For a small ɛ > 0, let y = ˆx + ɛ(d i ), then Ây = Â(ˆx + ɛd i ) = ˆx + ɛâ(  1 ) (.,i ) Ruta (UIUC) CS473 20 Spring 2018 20 / 29
Moving to a Neighbor (Contd.)  1 =.. d 1... d d.. Lemma Moving in direction d i from ˆx, all except constraint i remain tight. Proof. For a small ɛ > 0, let y = ˆx + ɛ(d i ), then Ây = Â(ˆx + ɛd i ) = ˆx + ɛâ(  1 ) (.,i ) = ˆb + ɛ[0,..., 1,..., 0] T Clearly, j a kjy j = b k, k i, and j a ijy j = b i ɛ < b i. Ruta (UIUC) CS473 20 Spring 2018 20 / 29
Computing the Neighbor Move in d i direction from ˆx, i.e., ˆx + ɛd i, and STOP when hit a new hyperplane! Need to ensure feasibility. Above lemma implies inequalities 1 through d will be satisfied. For any k > d, where A k is k th row of A, A k (ˆx + ɛd i ) b k (A k ˆx) + ɛ(a k d i ) b k ɛ(a k d i ) b k (A k ˆx) Ruta (UIUC) CS473 21 Spring 2018 21 / 29
Computing the Neighbor Move in d i direction from ˆx, i.e., ˆx + ɛd i, and STOP when hit a new hyperplane! Need to ensure feasibility. Above lemma implies inequalities 1 through d will be satisfied. For any k > d, where A k is k th row of A, A k (ˆx + ɛd i ) b k (A k ˆx) + ɛ(a k d i ) b k ɛ(a k d i ) b k (A k ˆx) (If (A k d i ) > 0) ɛ b k (A k ˆx ) A k d i Ruta (UIUC) CS473 21 Spring 2018 21 / 29
Computing the Neighbor Move in d i direction from ˆx, i.e., ˆx + ɛd i, and STOP when hit a new hyperplane! Need to ensure feasibility. Above lemma implies inequalities 1 through d will be satisfied. For any k > d, where A k is k th row of A, A k (ˆx + ɛd i ) b k (A k ˆx) + ɛ(a k d i ) b k ɛ(a k d i ) b k (A k ˆx) (If (A k d i ) > 0) ɛ b k (A k ˆx ) A k d i If moving towards hyperplane k (positive) Ruta (UIUC) CS473 21 Spring 2018 21 / 29
Computing the Neighbor Move in d i direction from ˆx, i.e., ˆx + ɛd i, and STOP when hit a new hyperplane! Need to ensure feasibility. Above lemma implies inequalities 1 through d will be satisfied. For any k > d, where A k is k th row of A, A k (ˆx + ɛd i ) b k (A k ˆx) + ɛ(a k d i ) b k ɛ(a k d i ) b k (A k ˆx) (If (A k d i ) > 0) ɛ b k (A k ˆx ) A k d i If moving towards hyperplane k (If (A k d i ) < 0) ɛ b k (A k ˆx ) A k d i If moving away from hyperplane k. No upper bound, and -ve lower bound! (positive) (negative) Ruta (UIUC) CS473 21 Spring 2018 21 / 29
Computing the Neighbor Algorithm NextVertex(ˆx, d i ) Set ɛ. For k = d + 1... n ɛ b k (A k ˆx ) A k d i If ɛ > 0 and ɛ < ɛ then set ɛ ɛ If ɛ < then return ˆx + ɛd i. Else return null. If (c d i ) > 0 then the algorithm returns an improving neighbor. Ruta (UIUC) CS473 22 Spring 2018 22 / 29
Factory Example max : x 1 + 6x 2 s.t. 0 x 1 200 0 x 2 300 x 1 + x 2 400 ˆx = (0, 0) Ruta (UIUC) CS473 23 Spring 2018 23 / 29
Factory Example max : x 1 + 6x 2 s.t. 0 x 1 200 0 x 2 300 x 1 + x 2 400 ˆx = (0, 0) [ ] 1 0 Â = 0 1 [ ] Â 1 1 0 = = [d 0 1 1 d 2 ] Ruta (UIUC) CS473 23 Spring 2018 23 / 29
Factory Example max : x 1 + 6x 2 s.t. 0 x 1 200 0 x 2 300 x 1 + x 2 400 ˆx = (0, 0) [ ] 1 0 Â = 0 1 [ ] Â 1 1 0 = = [d 0 1 1 d 2 ] Moving in direction d 1 gives (200, 0) Ruta (UIUC) CS473 23 Spring 2018 23 / 29
Factory Example max : x 1 + 6x 2 s.t. 0 x 1 200 0 x 2 300 x 1 + x 2 400 ˆx = (0, 0) [ ] 1 0 Â = 0 1 [ ] Â 1 1 0 = = [d 0 1 1 d 2 ] Moving in direction d 1 gives (200, 0) Moving in direction d 2 gives (0, 300). Ruta (UIUC) CS473 23 Spring 2018 23 / 29
Solving Linear Programming in Practice 1 Naïve implementation of Simplex algorithm can be very inefficient Exponential number of steps! Ruta (UIUC) CS473 24 Spring 2018 24 / 29
Solving Linear Programming in Practice 1 Naïve implementation of Simplex algorithm can be very inefficient 1 Choosing which neighbor to move to can significantly affect running time 2 Very efficient Simplex-based algorithms exist 3 Simplex algorithm takes exponential time in the worst case but works extremely well in practice with many improvements over the years 2 Non Simplex based methods like interior point methods work well for large problems. Ruta (UIUC) CS473 25 Spring 2018 25 / 29
Issues 1 Starting vertex 2 The linear program could be infeasible: No point satisfy the constraints. 3 The linear program could be unbounded: Polygon unbounded in the direction of the objective function. 4 More than d hyperplanes could be tight at a vertex, forming more than d neighbors. Ruta (UIUC) CS473 26 Spring 2018 26 / 29
Computing the Starting Vertex Equivalent to solving another LP! Find an x such that Ax b. If b 0 then trivial! Ruta (UIUC) CS473 27 Spring 2018 27 / 29
Computing the Starting Vertex Equivalent to solving another LP! Find an x such that Ax b. If b 0 then trivial! x = 0. Otherwise. Ruta (UIUC) CS473 27 Spring 2018 27 / 29
Computing the Starting Vertex Equivalent to solving another LP! Find an x such that Ax b. If b 0 then trivial! x = 0. Otherwise. min : s.t. Trivial feasible solution: s j a ijx j s b i, s 0 i Ruta (UIUC) CS473 27 Spring 2018 27 / 29
Computing the Starting Vertex Equivalent to solving another LP! Find an x such that Ax b. If b 0 then trivial! x = 0. Otherwise. min : s.t. s j a ijx j s b i, s 0 i Trivial feasible solution: x = 0, s = min i b i. Ruta (UIUC) CS473 27 Spring 2018 27 / 29
Computing the Starting Vertex Equivalent to solving another LP! Find an x such that Ax b. If b 0 then trivial! x = 0. Otherwise. min : s.t. s j a ijx j s b i, s 0 i Trivial feasible solution: x = 0, s = min i b i. If Ax b feasible then optimal value of the above LP is s = 0. Ruta (UIUC) CS473 27 Spring 2018 27 / 29
Computing the Starting Vertex Equivalent to solving another LP! Find an x such that Ax b. If b 0 then trivial! x = 0. Otherwise. min : s.t. s j a ijx j s b i, s 0 i Trivial feasible solution: x = 0, s = min i b i. If Ax b feasible then optimal value of the above LP is s = 0. Checks Feasibility! Ruta (UIUC) CS473 27 Spring 2018 27 / 29
Unboundedness: Example maximize x 2 x 1 + x 2 2 x 1, x 2 0 Unboundedness depends on both constraints and the objective function. Ruta (UIUC) CS473 28 Spring 2018 28 / 29
Unboundedness: Example maximize x 2 x 1 + x 2 2 x 1, x 2 0 Unboundedness depends on both constraints and the objective function. If unbounded in the direction of objective function, then the pivoting step in the simplex will detect it. Ruta (UIUC) CS473 28 Spring 2018 28 / 29
Degeneracy and Cycling More than d constraints are tight at vertex ˆx. Say d + 1. Suppose, we pick first d to form  such that ˆx = ˆb, and compute directions d 1,..., d d. Ruta (UIUC) CS473 29 Spring 2018 29 / 29
Degeneracy and Cycling More than d constraints are tight at vertex ˆx. Say d + 1. Suppose, we pick first d to form  such that ˆx = ˆb, and compute directions d 1,..., d d. Then NextVertex(ˆx, d i ) will encounter (d + 1) th constraint tight at ˆx and return the same vertex. Hence we are back to ˆx! Ruta (UIUC) CS473 29 Spring 2018 29 / 29
Degeneracy and Cycling More than d constraints are tight at vertex ˆx. Say d + 1. Suppose, we pick first d to form  such that ˆx = ˆb, and compute directions d 1,..., d d. Then NextVertex(ˆx, d i ) will encounter (d + 1) th constraint tight at ˆx and return the same vertex. Hence we are back to ˆx! Same phenomenon will repeat! Ruta (UIUC) CS473 29 Spring 2018 29 / 29
Degeneracy and Cycling More than d constraints are tight at vertex ˆx. Say d + 1. Suppose, we pick first d to form  such that ˆx = ˆb, and compute directions d 1,..., d d. Then NextVertex(ˆx, d i ) will encounter (d + 1) th constraint tight at ˆx and return the same vertex. Hence we are back to ˆx! Same phenomenon will repeat! This can be avoided by adding small random perturbation to b i s. Ruta (UIUC) CS473 29 Spring 2018 29 / 29