Outline. CS38 Introduction to Algorithms. Linear programming 5/21/2014. Linear programming. Lecture 15 May 20, 2014

Transcription

1 5/2/24 Outline CS38 Introduction to Algorithms Lecture 5 May 2, 24 Linear programming simplex algorithm LP duality ellipsoid algorithm * slides from Kevin Wayne May 2, 24 CS38 Lecture 5 May 2, 24 CS38 Lecture 5 2 Standard Form LP Linear programming "Standard form" LP. Input: real numbers a ij, c j, b i. Output: real numbers x j. n = # decision variables, m = # constraints. Maximize linear objective function subject to linear inequalities. (P) max c j x j n j n s. t. a ij x j j b i i m x j j n (P) max c T x Linear. No x 2, x y, arccos(x), etc. Programming. Planning (term predates computer programming). May 2, 24 CS38 Lecture Brewery Problem: Converting to Standard Form Equivalent Forms Original input. Standard form. Add slack variable for each inequality. Now a 5-dimensional problem. max 3A 23B s. t. 5A 5B 48 4A 4B 6 35A 2B 9 A, B max 3A 23B s. t. 5A 5B S C 48 4A 4B S H 6 35A 2B S M 9 Easy to convert variants to standard form. (P) max c T x Less than to equality: x + 2y 3z 7 ) x + 2y 3z + s = 7, s Greater than to equality: x + 2y 3z 7 ) x + 2y 3z s = 7, s Min to max: min x + 2y 3z ) max x 2y + 3z Unrestricted to nonnegative: x unrestricted ) x = x + x, x +, x 5 6

2 5/2/24 Brewery Problem: Feasible Region Linear programming geometric perspective Hops 4A + 4B 6 (, 32) Malt 35A + 2B 9 (2, 28) (26, 4) Corn 5A + 5B 48 Beer May 2, 24 CS38 Lecture 5 7 (, ) Ale (34, ) 8 Brewery Problem: Objective Function Brewery Problem: Geometry Brewery problem observation. Regardless of objective function coefficients, an optimal solution occurs at a vertex. (, 32) (, 32) (2, 28) (2, 28) 3A + 23B = $6 vertex (26, 4) (26, 4) Beer 3A + 23B = $8 Beer (, ) Ale (34, ) 3A + 23B = $442 (, ) Ale (34, ) 9 Convexity Geometric perspective Convex set. If two points x and y are in the set, then so is x + (- ) y for. convex combination Vertex. A point x in the set that can't be written as a strict convex combination of two distinct points in the set. Theorem. If there exists an optimal solution to (P), then there exists one that is a vertex. (P) max c T x vertex Intuition. If x is not a vertex, move in a non-decreasing direction until you reach a boundary. Repeat. x y x + d x' = x + * d convex Observation. LP feasible region is a convex set. not convex x - d x 2 2

3 5/2/24 Geometric perspective Geometric perspective Theorem. If there exists an optimal solution to (P), then there exists one that is a vertex. Theorem. If there exists an optimal solution to (P), then there exists one that is a vertex. Pf. Suppose x is an optimal solution that is not a vertex. There exist direction d such that x ± d 2 P. A d = because A(x ± d) = b. Assume c T d (by taking either d or d). Consider x + d, > : Pf. Suppose x is an optimal solution that is not a vertex. There exist direction d such that x ± d 2 P. A d = because A(x ± d) = b. Assume c T d (by taking either d or d). Consider x + d, > : Case. [ there exists j such that d j < ] Increase to * until first new component of x + d hits. x + *d is feasible since A(x + *d) = Ax = b and x + *y. x + *d has one more zero component than x. c T x' = c T (x + *d) = c T x + * c T d c T x. d k = whenever x k = because x ± d 2 P Case 2. [d j for all j ] x + d is feasible for all since A(x + d) = b and x + d x. As!, c T (x + d)! because c T d >. if c T d =, choose d so that case applies 3 4 Intuition Intuition. A vertex in R m is uniquely specified by m linearly independent equations. Linear programming linear algebraic perspective 4A + 4B 6 35A + 2B 9 4A + 4B = 6 35A + 2B = 9 (26, 4) May 2, 24 CS38 Lecture Basic Feasible Solution Basic Feasible Solution Theorem. Let P = { x : Ax = b, x }. For x 2 P, define B = { j : x j > }. Then x is a vertex iff A B has linearly independent columns. Theorem. Let P = { x : Ax = b, x }. For x 2 P, define B = { j : x j > }. Then x is a vertex iff A B has linearly independent columns. Notation. Let B = set of column indices. Define A B to be the subset of columns of A indexed by B. Ex. x A , b 5 2, B {, 3}, A B Pf. ( Assume x is not a vertex. There exist direction d not equal to such that x ± d 2 P. A d = because A(x ± d) = b. Define B' = { j : d j }. A B' has linearly dependent columns since d not equal to. Moreover, d j = whenever x j = because x ± d. Thus B µ B, so A B' is a submatrix of A B. Therefore, A B has linearly dependent columns

4 5/2/24 Basic Feasible Solution Basic Feasible Solution Theorem. Let P = { x : Ax = b, x }. For x 2 P, define B = { j : x j > }. Then x is a vertex iff A B has linearly independent columns. Theorem. Given P = { x : Ax = b, x }, x is a vertex iff there exists B µ {,, n } such B = m and: A B is nonsingular. Pf. ) Assume A B has linearly dependent columns. There exist d not equal to such that A B d =. x B = A B - b. x N =. basic feasible solution Extend d to R n by adding components. Pf. Augment A B with linearly independent columns (if needed). Now, A d = and d j = whenever x j =. For sufficiently small, x ± d 2 P ) x is not a vertex. A , b x 2, B {, 3, 4 }, A B Assumption. A 2 R m n has full row rank. 9 2 Basic Feasible Solution: Example Basic feasible solutions. max 3A 23B s. t. 5A 5B S C 48 4A 4B S H 6 35A 2B S M 9 Simplex algorithm {B, S H, S M } (, 32) Basis {A, B, S M } (2, 28) Infeasible {A, B, S H } (9.4, 25.53) Beer {A, B, S C } (26, 4) {S H, S M, S C } (, ) Ale {A, S H, S C } (34, ) May 2, 24 CS38 Lecture Simplex Algorithm: Intuition Simplex Algorithm: Initialization Simplex algorithm. [George Dantzig 947] Move from BFS to adjacent BFS, without decreasing objective function. replace one basic variable with another 3A 23B Z 5A + 5B S C 48 4A 4B S H 6 35A 2B S M 9 Greedy property. BFS optimal iff no adjacent BFS is better. Challenge. Number of BFS can be exponential! edge Basis = {S C, S H, S M } A = B = Z = S C = 48 S H = 6 S M =

5 5/2/24 Simplex Algorithm: Pivot Simplex Algorithm: Pivot 3A 23B Z 5A + 5B S C 48 4A 4B S H 6 35A 2B S M 9 Basis = {S C, S H, S M} A = B = Z = S C = 48 S H = 6 S M = 9 3A 23B Z 5A + 5B S C 48 4A 4B S H 6 35A 2B S M 9 Basis = {S C, S H, S M} A = B = Z = S C = 48 S H = 6 S M = 9 Substitute: B = /5 (48 5A S C ) 6 3 A 23 5 S C Z A B 5 S C A 4 5 S C S H A 4 3 S C S M 55 Basis = {B, S H, S M} A = S C = Z = 736 B = 32 S H = 32 S M = 55 Q. Why pivot on column 2 (or )? A. Each unit increase in B increases objective value by $23. Q. Why pivot on row 2? A. Preserves feasibility by ensuring RHS. min ratio rule: min { 48/5, 6/4, 9/2 } Simplex Algorithm: Pivot 2 Simplex Algorithm: Optimality 6 3 A 23 5 S C Z A B 5 S C A 4 5 S C S H A 4 3 S C S M 55 Substitute: A = 3/8 (32 + 4/5 S C S H ) Basis = {B, S H, S M} A = S C = Z = 736 B = 32 S H = 32 S M = 55 Q. When to stop pivoting? A. When all coefficients in top row are nonpositive. Q. Why is resulting solution optimal? A. Any feasible solution satisfies system of equations in tableaux. In particular: Z = 8 S C 2 S H, S C, S H. Thus, optimal objective value Z* 8. Current BFS has value 8 ) optimal. S C 2 S H Z 8 B S C 8 S H 28 A S C 3 8 S H S C 85 8 S H S M Basis = {A, B, S M} S C = S H = Z = 8 B = 28 A = 2 S M = S C 2 S H Z 8 B S C 8 S H 28 A S C 3 8 S H S C 85 8 S H S M Basis = {A, B, S M} S C = S H = Z = 8 B = 28 A = 2 S M = Simplex Tableaux: Matrix Form Simplex Algorithm: Corner Cases Initial simplex tableaux. c T B x B c T N x N Z A B x B A N x N b x B, x N Simplex algorithm. Missing details for corner cases. Q. What if min ratio test fails? Q. How to find initial basis? Q. How to guarantee termination? Simplex tableaux corresponding to basis B. T (c N c T B A B A N ) x N Z c T B A B b I x B A B A N x N A B b x B, x N subtract c B T A B - times constraints multiply by A B - x B = A B - b x N = basic feasible solution c N T c B T A B - A N optimal basis

6 5/2/24 Unboundedness Phase I Simplex Q. What happens if min ratio test fails? 2x 4 2x 5 Z 2 x 4x 4 8x 5 3 x 2 5x 4 2x 5 4 x 3 5 x, x 2, x 3, x 4, x 5 all coefficients in entering column are nonpositive Q. How to find initial basis? A. Solve (P'), starting from basis consisting of all the z i variables. (P) max c T x m ( P ) min z i å i= s. t. A x + I z = b x, z ³ Case : min > ) (P) is infeasible. A. Unbounded objective function. x 3 8x 5 Case 2: min =, basis has no z i variables ) OK to start Phase II. Z 2 2x 5 x 2 x 3 x 4 x 5 4 2x 5 5 Case 3a: min =, basis has z i variables. Pivot z i variables out of basis. If successful, start Phase II; else remove linear dependent rows Simplex Algorithm: Degeneracy Simplex Algorithm: Degeneracy Degeneracy. New basis, same vertex. Degeneracy. New basis, same vertex. Degenerate pivot. Min ratio = x 5 2 x 6 6x 7 Z x 4 x 4 8x 5 x 6 9x 7 x 2 2 x 4 2x 5 2 x 6 3x 7 x 3 x 6 x, x 2, x 3, x 4, x 5, x 6, x 7 Cycling. Infinite loop by cycling through different bases that all correspond to same vertex. Anti-cycling rules. Bland's rule: choose eligible variable with smallest index. Random rule: choose eligible variable uniformly at random. Lexicographic rule: perturb constraints so nondegenerate Lexicographic Rule Lexicographic Rule Intuition. No degeneracy ) no cycling. Intuition. No degeneracy ) no cycling. Perturbed problem. much much greater, say ² i = ± i for small ± Perturbed problem. much much greater, say ² i = ± i for small ± (P ) max c T x (P ) max c T x Lexicographic rule. Apply perturbation virtually by manipulating ² symbolically: Claim. In perturbed problem, x B = A B - (b + ²) is always nonzero. Pf. The j th component of x B is a (nonzero) linear combination of the components of b + ² ) contains at least one of the ² i terms Corollary. No cycling. which can't cancel

7 5/2/24 Simplex Algorithm: Practice Remarkable property. In practice, simplex algorithm typically terminates after at most 2(m + n) pivots. but no polynomial pivot rule known Issues. Choose the pivot. Maintain sparsity. Ensure numerical stability. Preprocess to eliminate variables and constraints. LP duality Commercial solvers can solve LPs with millions of variables and tens of thousands of constraints. May 2, 24 CS38 Lecture (P) max 3A 23B s. t. 5A 5B 48 4A 4B 6 35A 2B 9 A, B (P) max 3A 23B s. t. 5A 5B 48 4A 4B 6 35A 2B 9 A, B Goal. Find a lower bound on optimal value. Goal. Find an upper bound on optimal value. Easy. Any feasible solution provides one. Ex. Multiply 2 nd inequality by 6: 24 A + 24 B 96. Ex. (A, B) = (34, ) ) z* 442 Ex 2. (A, B) = (, 32) ) z* 736 Ex 3. (A, B) = (7.5, 29.5) ) z* 776 Ex 4. (A, B) = (2, 28) ) z* 8 ) z* = 3 A + 23 B 24 A + 24 B 96. objective function 39 4 (P) max 3A 23B s. t. 5A 5B 48 4A 4B 6 35A 2B 9 A, B (P) max 3A 23B s. t. 5A 5B 48 4A 4B 6 35A 2B 9 A, B Goal. Find an upper bound on optimal value. Goal. Find an upper bound on optimal value. Ex 2. Add 2 times st inequality to 2 nd inequality: Ex 2. Add times st inequality to 2 times 2 nd inequality: ) z* = 3 A + 23 B 4 A + 34 B 2. ) z* = 3 A + 23 B 3 A + 23 B 8. Recall lower bound. (A, B) = (2, 28) ) z* 8 Combine upper and lower bounds: z* =

8 5/2/24 : Economic Interpretation (P) max 3A 23B s. t. 5A 5B 48 4A 4B 6 35A 2B 9 A, B Idea. Add nonnegative combination (C, H, M) of the constraints s.t. Brewer: find optimal mix of beer and ale to maximize profits. (P) max 3A 23B s. t. 5A 5B 48 4A 4B 6 35A 2B 9 A, B 3A 23B (5C 4H 35M ) A (5C 4H 2M ) B 48C 6H 9M Dual problem. Find best such upper bound. Entrepreneur: buy individual resources from brewer at min cost. C, H, M = unit price for corn, hops, malt. Brewer won't agree to sell resources if 5C + 4H + 35M < 3. (D) min 48C 6H 9M s. t. 5C 4H 35M 3 5C 4H 2M 23 C, H, M (D) min 48C 6H 9M s. t. 5C 4H 35M 3 5C 4H 2M 23 C, H, M LP Duals Double Dual Canonical form. Canonical form. (P) max c T x (D) min y T b s. t. A T y c y (P) max c T x (D) min y T b s. t. A T y c y Property. The dual of the dual is the primal. Pf. Rewrite (D) as a maximization problem in canonical form; take dual. (D' ) max y T b s. t. A T y c y (DD) min c T z s. t. (A T ) T z b z Taking Duals LP dual recipe. Primal (P) constraints maximize a x = b i a x b a x b i minimize y i unrestricted y i y i Dual (D) variables Strong duality variables x j x j unrestricted a T y c j a T y c j a T y = c j constraints Pf. Rewrite LP in standard form and take dual. May 2, 24 CS38 Lecture

9 5/2/24 LP Strong Duality Theorem. [Gale-Kuhn-Tucker 95, Dantzig-von Neumann 947] For A 2 R m x n, b 2 R m, c 2 R n, if (P) and (D) are nonempty, then max = min. (P) max c T x (D) min y T b s. t. A T y c y Generalizes: Dilworth's theorem. König-Egervary theorem. Max-flow min-cut theorem. von Neumann's minimax theorem. Pf. [ahead] 49 9