Lecture notes on the simplex method September We will present an algorithm to solve linear programs of the form. maximize.

Transcription

1 Cornell University, Fall 2017 CS 6820: Algorithms Lecture notes on the simplex method September The Simplex Method We will present an algorithm to solve linear programs of the form maximize subject to c x Ax b x 0 (1) assuming that b 0, so that x = 0 is guaranteed to be a feasible solution. Let n denote the number of variables and let m denote the number of constraints. A simple transformation modifies any such linear program into a form such that each variable is constrained to be non-negative, and all other linear constraints are expressed as equations rather than inequalities. The key is to introduce additioinal variables, called slack variables which account for the difference between and left and right sides of each inequality in the original linear program. In other words, linear program (1) is equivalent to maximize subject to c x Ax + y = b x, y 0 (2) where x R n and y R m. The solution set of {Ax + y = b, x 0, y 0} is a polytope in the (n + m)-dimensional vector space of ordered pairs (x, y) R n R m. The simplex algorithm is an iterative algorithm to solve linear programs of the form (2) by walking from vertex to vertex, along the edges of this polytope, until arriving at a vertex which maximizes the objective function c x. To illustrate the simplex method, for concreteness we will consider the following linear program. maximize 2x 1 + 3x 2 subject to x 1 + x 2 8 2x 1 + x 2 12 x 1 + 2x 2 14 x 1, x 2 0 This LP has so few variables, and so few constraints, it is easy to solve it by brute-force enumeration of the vertices of the polytope, which in this case is a 2-dimensional polygon. 1

2 The vertices of the polygon are [ 0 7 ], [ 2 6 ], [ 4 4 ], [ 6 0 ], [ 0 0 ]. The objective function 2x 1 + 3x 2 is maximized at the vertex [ 2 6 ], where it attains the value 22. It is also easy to certify that this is the optimal value, given that the value is attained at [ 2 6 ]: simply add together the inequalities to obtain x 1 + x 2 8 x 1 + 2x x 1 + 3x 2 22, which ensures that no point in the feasible set attains an objective value greater than 22. To solve the linear program using the simplex method, we first apply the generic transformation described earlier, to rewrite it in equational form as maximize 2x 1 + 3x 2 subject to x 1 + x 2 + y 1 = 8 2x 1 + x 2 + y 2 = 12 x 1 + 2x 2 + y 3 = 14 x 1, x 2, y 1, y 2, y 3 0 From now on, we will be choosing a subset of two of the five variables (called the basis), setting them equal to zero, and using the linear equations to express the remaining three variables, as well as the objective function, as a function of the two variables in the basis. Initially the basis is {x 1, x 2 } and the linear program can be written in the form maximize 2x 1 + 3x 2 subject to y 1 = 8 x 1 x 2 y 2 = 12 2x 1 x 2 y 3 = 14 x 1 2x 2 x 1, x 2, y 1, y 2, y 3 0 which emphasizes that each of y 1, y 2, y 3 is determined as a function of x 1, x 2. Now, as long as the basis contains a variable which has a positive coefficient in the objective function, we select one such variable and greedily increasing its value until one of the non-negativity constraints becomes tight. At that point, one of the other variables attains the value zero: it enters the basis, and the variable whose value we increased leaves the basis. For example, we could choose to increase x 1 from 0 to 6, at which point y 2 = 0. Then the new basis becomes {y 2, x 2 }. Rewriting the equation y 2 = 12 2x 1 x 2 as x 1 = y x 2, (3) 2

3 we may substitute the right side of (3) in place of x 1 everywhere in the above linear program, arriving at the equivalent form maximize 12 y 2 + 2x 2 subject to y 1 = 2 + 1y 2 2 1x 2 2 x 1 = 6 1y 2 2 x 2 y 3 = 8 + 1y 2 2 3x 2 2 x 1, x 2, y 1, y 2, y 3 0 At this point, x 2 still has a positive coefficient in the objective function, so we increase x 2 from 0 to 4, at which point y 1 = 0. Now x 2 leaves the basis, and the new basis is {y 1, y 2 }. We use the equation x 2 = 4 + y 2 2y 1 to substitute a function of the basis variables in place of x 2 everywhere it appears, arriving at the new linear program maximize 20 4y 1 + y 2 subject to x 2 = 4 + y 2 2y 1 x 1 = 4 y 2 + y 1 y 3 = 2 y 2 + 3y 1 x 1, x 2, y 1, y 2, y 3 0 Now we increase y 2 from 0 to 2, at which point y 3 = 0 and the new basis is {y 1, y 3 }. Substituting y 2 = 2 y 3 + 3y 1 allows us to rewrite the linear program as maximize 22 y 1 y 3 subject to x 2 = 6 + y 1 y 3 x 1 = 2 2y 1 + y 3 (4) y 2 = 2 + 3y 1 y 3 x 1, x 2, y 1, y 2, y 3 0 At this point, there is no variable with a positive coefficient in the objective function, and we stop. It is trivial to verify that the solution defined by the current iteration namely, x 1 = 2, x 2 = 6, y 1 = 0, y 2 = 2, y 3 = 0 is optimal. The reason is that we have managed to write the objective function in the form 22 y 1 y 3. Since the coefficient on each of the variables y 1, y 3 is negative, and y 1 and y 3 are constrained to take non-negative values, the largest possible value of the objective function is achieved by setting both y 1 and y 3 to zero, as our solution does. More generally, if the simplex method terminates, it means that we have found an equivalent representation of the original linear program (2) in a form where the objective function 3

4 attaches a non-positive coefficient to each of the basis variables. Since the basis variables are required to be non-negative, the objective function is maximized by setting all the basis variables to zero, which certifies that the solution at the end of the final iteration is optimal. There is another way that the simplex method can terminate, which is not illustrated by the example above. It may happen, in one iteration, that when we choose a variable with a positive coefficient in the objective function, and we can increase this variable to an arbitrarily large value without violating any of the non-negativity constraints. (This situation happens when the increasing variable appears with a non-negative coefficient in each of the equations that defines the non-basic variables as a linear function of the variables in the basis.) In that case, we have verified that the optimum of the linear program is equal to +, i.e. the objective function c x takes unboundedly large values as x ranges over the set of vectors satisfying the constraints. Note that, in our running example, the final objective function assigned coefficient 1 to both y 1 and y 3. This is closely related to the fact that the simple certificate of optimality described above (before we started running the simplex algorithm) we obtained by summing the first and third inequalities of the original linear program, each with a coefficient of 1. We will see in the following section that this is not a coincidence. Before leaving this discussion of the simplex method, we must touch upon a subtle issue regarding the question of whether the algorithm always terminates. A basis is an n-element subset of n + m variables, so there are at most ( ) n+m n bases; if we can ensure that the algorithm never returns to the same basis as in a previous iteration, then it must terminate. Note that each basis determines a unique point (x, y) R n+m defined by setting the basis variables to zero and assigning to the remaining variables the unique values that satisfy the equation Ax + y = b and as the algorithm proceeds from basis to basis, the objective function value at the corresponding points never decreases. If the objective function strictly increases when moving from basis B to basis B, then the algorithm is guaranteed never to return to basis B, since the objective function value is now strictly greater than its value at B, and it will never decrease. On the other hand, it is possible for the simplex algorithm to shift from one basis to a different basis with the same objective function value; this is called a degenerate pivot, and it happens when the set of variables whose value is 0 at the current solution is a strict superset of the basis. There exist pivot rules (i.e., rules for selecting the next basis in the simplex algorithm) that are designed to avoid infinite loops of degenerate pivots. Perhaps the simplest such rule is Bland s rule, which always chooses to remove from the basis the lowest-numbered variable that has a positive coefficient in the objective function. (And, in case there is more than one variable that may move into the objective function to replace it, the rule also chooses the lowest-numbered such variable.) Although the rule is simple to define, proving that it avoids infinite loops is not easy, and we will omit the proof from these notes. Instead, the following section is devoted to presenting a pivot rule that is much more algorithmically costly, but leads to an easier and more conceptual proof of termination. 4

5 1.1 A non-cycling pivot rule based on infinitesimals In order for a degenerate pivot to be possible when solving a given linear program using the simplex method, the equation Ax + y = b must have a solution in which n + 1 or more of the variables take the value 0. Generically, a system of m linear equations in m + n unknown does not have solutions with strictly more than n of the variables equal to 0. If we modify the linear system Ax+y = b by perturbing it slightly, we should expect that such a modification will, generically, eliminate the possibility of encountering degenerate pivots when running the simplex algorithm. These algebraic considerations can also be visualized in geometric terms. The linear system Ax + y = b consists of m linearly independent equations in m + n unknown, so its solution set is an n-dimensional subspace of the m + n-dimensional vector space of all (x, y) pairs. In this n-dimensional space, the inequalities x 0, y 0 define m + n halfspaces, each bounded by a hyperplane of the form {x j = 0} or {y i = 0}. In n-dimensional space, n hyperplanes in general position will intersect at a single point, whereas n + 1 or more hyperplanes in general position will have an empty intersection. For example, in three dimensions, any three planes in general position have one intersection point, whereas four planes in general position have an empty intersection. However, it is perfectly possible for four planes (not in general position) to have a non-empty intersection, as when the four faces of a square pyramid meet at its apex. If one were to run the simplex algorithm to optimize a linear function on a three-dimensional polyhedron such as a square pyramid, it is possible that one or more iterations of the algorithm would start at the apex of the pyramid, with a basis consisting of any three of the four slack variables corresponding to the four sides of the pyramid. A degenerate pivot would substitute a new basis by replacing one of these three slack variables with the slack variable that was omitted from the previous basis. After performing this operation, and setting the variables in the new basis equal to zero, we remain situated at the same vertex as before, namely the apex of the pyramid. Now consider modifying the polyhedron by perturbing the height of each face of the pyramid by a very small amount potentially a different small quantity for each of the faces. This operation modifies the shape of the top of the polyhedron slightly; the apex point is replaced with a very short line segment joining two vertices, as illustrated in Figure 1. Figure 1: A pyramid and its generic perturbation The modified polyhedron has no degenerate vertices; every vertex is situated at the intersection of three (and only three) faces. The degenerate pivot at the apex of the pyramid has been replaced by a non-degenerate pivot that makes progress by moving along the short line segment joining the two topmost vertices of the modified polyhedron. The foregoing discussion about perturbations could potentially by implemented by adding a short random vector ɛ to the vector b occurring on the right side of the equation Ax+y = b, 5

6 resulting in the new equation Ax + y = b + ɛ. To justify that this works, one first needs to reason about the probability that the modified polyhedron has no degenerate vertices. This probability turns out to be equal to 1, assuming that the distribution of the vector ɛ is absolutely continuous with respect to Lebesgue measure, but that fact requires proof. Next one would need to reason about whether every vertex of the new polyhedron is situated near a vertex of the original polyhedron (i.e., at a distance tending to zero as the length of the vector ɛ tends to zero). This also turns out to be true, but again requires proof. Instead of using probability, we will adopt a more algebraic formalism that gracefully sidesteps some of these issues. The idea will be to treat the coefficients of the linear program as belonging to an extension field of the real numbers that is still totally ordered, but contains infinitesimal numbers representing the perturbations. Definition 1. An ordered field is a field F with a distinguished subset F >0, called the set of positive elements, that is closed under addition and multiplication, does not contain 0, and satisfies the property that for every non-zero x F, exactly one of x and x is positive. One can define a total ordering on F by specifying that x < x if and only if x x is positive. Inequalities defined in this way obey the usual algebraic rules for manipulating inequalities, for example the rule which asserts that the validity of an inequality is preserved under scaling both sides by the same positive scalar. Definition 2. If F is an ordered field that contains R as a subfield, an element x F is called finite if there is some r R such that r < x < r, and otherwise we refer to x as infinite. If x 0 and x 1 is infinite we say that x is infinitesimal. The finite elements constitute a subring F < and the infinitesimals are an ideal in this subring. There is a surjective homomorphism F < R whose kernel is the ideal of infinitesimal elements. This homomorphism is defined by mapping each finite x F to the real number [x] = inf{r R r > x}. (5) Lemma 1. There exists an ordered field F that contains R as a subfield, and whose ideal of infinitesimal elements is an infinite-dimensional vector space over R. Proof. Let ɛ be a formal variable (i.e., a symbol with no numerical meaning) and let F be the field of rational functions R(ɛ). In other words, an element of F is an equivalence class of fractions P/Q where P and Q are both polynomials in the formal variable ɛ, with real coefficients, and Q 0. The equivalence relation is defined by specifying that P 1 /Q 1 = 3 3ɛ P 2 /Q 2 if and only if P 1 Q 2 = P 2 Q 1. For example, 2 3+ɛ and are two expressions 1 ɛ 3 1+ɛ+ɛ 2 representing the same element of R(ɛ). The ordering of F is defined as follows. A non-zero polynomial P = a 0 + a 1 ɛ + + a n ɛ n is positive if and only if the first non-zero element of the coefficient sequence a 0, a 1,..., a n is positive. A quotient P/Q is positive if and only if P and Q are either both positive or both negative. Under this ordering, the monomials ɛ, ɛ 2, ɛ 3,... constitute an infinite set of infinitesimal elements that are linearly independent over R. 6

7 We defined linear programs and the simplex algorithm using the field of real numbers, but the problem definition and the algorithm both make sense over any ordered field. In particular, we can let K be an ordered extension field of R that contains m linearly independent infinitesimal elements ɛ 1, ɛ 2,..., ɛ m, and we can modify the linear program (2) by replacing the equation Ax + y = b with Ax + y = b + ɛ, where ɛ denotes the column vector (ɛ 1, ɛ 2,..., ɛ m ). Claim 2. Fix a matrix A and vector b over the real numbers, and fix an ordered extension field F R that contains m linearly independent infinitesimal elements ɛ 1, ɛ 2,..., ɛ m. Let ɛ = (ɛ 1, ɛ 2,..., ɛ m ). For any x F n and y F m satisfying Ax + y = b + ɛ, at most n of the elements x 1,..., x n, y 1,..., y m are equal to 0. Proof. For i = 1,..., m the equation ɛ i = n a ij x j + y i b i j=1 shows that ɛ i belongs to the R-linear subspace of K generated by {1, x 1,..., x n, y 1,..., y m }. Of course, the element 1 F also belongs to this subspace, which we will denote by L. Since 1, ɛ 1,..., ɛ m are linearly independent over R, the dimension of L over R is at least m + 1. In particular, since the set {1, x 1,..., x n, y 1,..., y m } contains a basis for L, it must contain at least m + 1 non-zero elements, and hence at most n elements are equal to 0. Claim 3. Fix A, b, F, ɛ as in Claim 2. When one solves the linear program maximize subject to c x Ax + y = b + ɛ x, y 0 (6) using the simplex method, there are no degnerate pivots and the algorithm terminates after performing at most ( ) m+n n pivots. Proof. A degenerate pivot, by definition, occurs when n + 1 or more coordinates of the vectors x, y are non-zero. Claim 2 shows that this can never happen when (x, y) belongs to the solution set of Ax + y = b + ɛ. Hence, there are no degenerate pivots. This means that the value of the objective function c x (interpreted as an element of the ordered field F) strictly increases with each iteration of the algorithm. Consequently, no two iterations of the algorithm can use the same basis, and the number of iterations is bounded above by the number of bases, which is ( ) m+n n. A pivot rule that guarantees termination. To define a non-cycling pivot rule for the simplex method solving an ordinary linear program (i.e., with coefficients in R) our strategy will be to run the simplex method in tandem on two linear programs: one defined over R with constraint set {Ax + y = b, x, y 0}, and another defined over the extension field F with constraint set {Ax + y = b + ɛ, x, y 0}. Let us distinguish the two linear programs 7

8 by calling the first one LP R and the second one LP F. We run the simplex method to solve LP F, and we let x (t), y (t) denote the values of the vectors x, y at the start of iteration t, for t = 0, 1,..., T, where T either denotes the final iteration of the algorithm, or the last iteration in which all components of the vector x (T ) are finite. Recall the homomorphism x [x] from F < to R defined in equation (5). We will abuse notation and apply this homomorphism to vector spaces as well: if x is a vector with components in F < we let [x] denote the vector over R obtained by applying the operation x i [x i ] to each component of x. Now take the sequence of vector pairs {x (t), y (t) } T t=0 representing the execution of the simplex method solving LP F and translate it to the sequence of vector pairs {[x (t) ], [y (t) ]} T t=0. This sequence represents an initial segment of a valid execution of the simplex method solving LP R. In every iteration, n of the variables in the set {[x (t) j ] j = 1,..., n} {[y(t) i ] i = 1,..., m} are equal to zero, because the n variables in the basis of the corresponding iteration of the LP F are equal to 0, and [0] = 0. The remaining m variables in the set are non-negative, because their counterparts in the LP F execution are non-negative, and the homomorphism x [x] preserves non-negativity. To conclude our analysis we will show that {[x (t) ], [y (t) ]} T t=0 represents a terminating execution of the simplex method solving LP R. By the definition of T, we know that in iteration T the simplex method solving LP F either terminates (because when the objective function is written in terms of the current basis, every coefficient is non-positive) or it chooses a variable in the basis and increases its value from 0 to an infinite element of F. In the former case the simplex method solving LP R also terminates at iteration T because its objective function coefficients are non-positive. (The homomorphism x [x] preserves non-positivity.) In the latter case, the simplex method solving LP R terminates in iteration T because it discovers that the optimum is equal to The simplex method takes exponential time in the worst case An example due to Klee and Minty illustrates that the simplex method can take exponential time in the worst case. Consider the linear program maximize x n subject to 0 x 1 1 (7) δx i x i+1 1 δx i for 1 i < n Here, δ is a positive number less than 1/2. The polyhedron defined by the constraints of this linear program is shaped like an n- dimensional hypercube with tilted sides, as depicted in Figure 2. It is called a Klee-Minty cube. There is an execution of the simplex method that visits each of the 2 n vertices of the Klee-Minty cube, starting from (0, 0,..., 0) and ending at (0, 0,..., 1). The sequence of vertices can be defined recursively as follows. The back face of the Klee-Minty cube, where the equation x n = δx n 1 is satisfied, is a copy of the (n 1)-dimensional Klee-Minty cube. We run an execution of the simplex method on this (n 1)-dimensional Klee-Minty cube, with the modified objective function x n 1. The equation x n = δx n 1 guarantees that 8

9 Figure 2: A Klee-Minty cube in three dimensions the true objective function increases as the modified objective function increases, which means that this remains a valid execution of the simplex method on the back face of the n-dimensional Klee-Minty cube. After reaching the vertex (0, 0,..., 1, δ), we move from the back face to the front face, where the equation x n = 1 δx n 1 is satisfied, arriving at the vertex (0, 0,..., 1, 1 δ). Then we run through the vertices of the front face by reversing the order in which we visited the corresponding vertices on the back face. As we do this, x n 1 strictly decreases with each pivot; due to the equation x n = 1 δx n 1 this means that x n strictly increases with each pivot, confirming that this is a valid execution of the simplex algorithm. 2 The Simplex Method and Strong Duality An important consequence of the correctness and termination of the simplex algorithm is linear programming duality, which asserts that for every linear program with a maximization objective, there is a related linear program with a minimization objective whose optimum matches the optimum of the first LP. Theorem 4. Consider any pair of linear programs of the form maximize c x minimize b η subject to Ax b and subject to A η c (8) x 0 η 0 If the optimum of the first linear program is finite, then both linear programs have the same optimum value. Proof. Before delving into the formal proof, the following intuition is useful. If a i denotes the i th row of the matrix A, then the relation Ax b can equivalently be expressed by stating that a i x b i for j = 1,..., m. For any m-tuple of non-negative coefficients η 1,..., η m, we 9

10 can form a weighted sum of these inequalities, m m η i a i x η i b i, (9) j=1 obtaining an inequality implied by Ax b. Depending on the choice of weights η 1,..., η m, the inequality (9) may or may not imply an upper bound on the quantity c x, for all x 0. The case in which (9) implies an upper bound on c x is when, for each variable x j (j = 1,..., n), the coefficient of x j on the left side of (9) is greater than or equal to the coefficient of x j in the expression c x. In other words, the case in which (9) implies an upper bound on c x for all x 0 is when m j {1,..., n} η i a ij c j. (10) We can express (9) and (10) more succinctly by packaging the coefficients of the weighted sum into a vector, η. Then, inequality (9) can be rewritten as j=1 and the criterion expressed by (10) can be rewritten as i=1 η Ax η b, (11) η A c. (12) The reasoning surrounding inequalities (9) and (10) can now be summarized by saying that for any vector η R m satisfying η 0 and η A c, we have c x η Ax η b (13) for all x 0 satisfying Ax b. (In hindsight, proving inequality (13) is trivial using the properties of the vector ordering and our assumptions about x and η.) Applying (13), we may immediately conclude that the minimum of η b over all η 0 satisfying η A c, is greater than or equal to the maximum of c x over all x 0 satisfying Ax b. That is, the optimum of the first LP in (8) is less than or equal to the optimum of the second LP in (8), a relation known as weak duality. To prove that the optima of the two linear programs are equal, as asserted by the theorem, we need to furnish vectors x, η satisfying the constraints of the first and second linear programs in (8), respectively, such that c x = b η. To do so, we will make use of the simplex algorithm and its termination condition. At the moment of termination, the objective function has been rewritten in a form that has no positive coefficient on any variable. In other words, the objective function is written in the form v ξ x η y for some coefficient vectors ξ R n and η R m such that ξ, η 0. An invariant of the simplex algorithm is that whenever it rewrites the objective function, it preserves the property that the objective function value matches c x for all pairs (x, y) R n R m such that Ax + y = b. In other words, we have x R n v ξ x η (b Ax) = c x. (14) 10

11 Equating the constant terms on the left and right sides, we find that v = η b. Equating the cofficient of x j on the left and right sides for all j, we find that η A = ξ + c c. Thus, the vector η satisfies the constraints of the second LP in (8). Now consider the vector (x, y) which the simplex algorithm outputs at termination. All the variables having a non-zero coefficient in the expression ξ x η y belong to the algorithm s basis, and hence are set to zero in the solution (x, y). This means that v = v ξ x η y = c x and hence, using the relation v = η b derived earlier, we have c x = b η as desired. 11