AMATH 383 Lecture Notes Linear Programming

Transcription

1 AMATH 8 Lecture Notes Linear Programming Jakob Kotas (jkotas@uw.edu) University of Washington February 4, 014 Based on lecture notes for IND E 51 by Zelda Zabinsky, available from These notes are not copyrighted and are free for distribution for academic purposes. Corrections should be directed to the author. A linear program is a technique for optimization of a linear objective function linear equality and inequality constraints. 1 1D Case The easiest nontrivial example of an LP would be to minimize a linear function in one variable. As an example: min x x 1,xapple 4 We call the function that we are minimizing (or maximizing) the objective function and the limits on x the constraints. Here we have two inequality constraints, but we can also have equality constraints. If the objective function and constraints are linear in the variables we are solving over, this is called a linear program (LP). Without knowing anything about linear programming, it is clear that the solution (or objective function value ) of this problem is -, which occurs at the optimal value of x = 1. Adding a constant value to the objective function will change the objective function value, but not the optimal value of x. Thus if our real objective function from whatever application we are looking at contains a constant, we usually leave it o the optimization problem and just add it back on later. It should also be obvious that an endpoint will always be an optimal value for a 1D linear problem. What if the objective function had slope zero? 1

2 min x 1,xapple 4 In this case, there are infinitely many x values that are optimal (all x in [-1,4]), resulting in an objective function value of -. Note that the infinite solutions of x also contain both endpoints, thus, it does not violate the rule that an (at least one) endpoint is a solution to the LP. It is also possible that the domain, which is defined by the constraints, could be unbounded. For example: min x x 1 This is the same as the first example except we have removed the constraint x apple 4. In this case, the final answer is still the same: the optimal value of x is still 1, and the optimal objective function value is still. This is because we weren t using the constraint x apple 4 anyway. We say that the constraint x apple 4 was inactive. What if we removed the constraint x 1 instead? This is an active constraint, because it is what limits our x from dropping any lower. The problem would become min x x apple 4 In this case, the optimal value of x and the optimal objective function value are both 1 and we say the problem is unbounded. If the problem is bounded, there exists a finite optimal objective function value. If the problem is unbounded, there may or may not exist such a value. We can see that removing a constraint in a minimization problem can only ever lower our objective function value, or keep it the same, but never make it rise. Similarly adding a constraint can only ever raise our objective function value or keep it the same. The opposite is true in a maximization problem. Finally let s discuss what is known as a redundant constraint. This is a constraint that doesn t matter because it can never become active. An example would be min x x 1,xapple 4,x The third constraint, x is redundant because any x that satisfies the constraint x 1 automatically satisfies the constraint x. We call a value of x that satisfies all the constraints a feasible x, and the domain defined by the constraints the feasible region.

3 D Case Now let s look at a more advanced problem, a linear problem with two variables. In D, the feasible region will be a convex polygon. We say a polygon is convex if there does not exist a set of two points within the polygon such that the whole line segment between those two points is contained within the polygon. An equivalent definition is that the polygon s interior angles are all apple 180. The notion of endpoints in the 1D case is translated to vertices in the D case. A vertex is a point where at least two constraints are active. (Note: how could there be more than two constraints active anywhere? If there is a degenerate constraint: three or more constraints intersecting at a single vertex.) Thus, we can always solve an LP by checking the objective function value at every vertex. However, as the number of variables (dimensions) grows, the number of vertices grows exponentially in the number of dimensions, making checking every vertex prohibitively time-costly. We will soon explain a smarter way of finding the optimum, but first, an example. An example of a degenerate point (C). We also see an example of a redundant constraint, the nearly-vertical line on the left side. Image from Wikimedia Commons.

4 Diet problem Here we present the first example of a linear problem. We seek to decide the number of units of di erent foods to consume every day so we meet the minimum daily requirement (MDR) of di erent nutrients at minimum cost. wheat rye MDR carbs/unit 5 8 protein/unit 4 15 vitamins/unit 1 cost/unit Let x be the number of units of wheat and y be the number of units of rye to consume every day. The LP becomes min 0.6x +0.5y 5x +y 8 4x +y 15 x + y x 0 y 0 Notice the feasible region is unbounded, as x and y can be arbitrarily large. First we plot the feasible region alone, defined as the region above the three lines in the plot: 4

5 Note that only the second constraint is non-redundant. Then we can visualize the problem by plotting the objective function value as the z axis on a surface plot. The notch in the lower corner of the plane is due to the constraints. Here we have only plotted x and y up to 10, but due to linearity, the objective value will not be improved by considering any larger values than these. It appears that the optimal values of x and y are roughly 4 and 0, respectively, and that the objective function value is around 1.5. Ok, but how would we solve this problem more precisely and methodically? Furthermore, this visual approach is not going to work for higher dimensions. 5

6 4 Alloy mix problem Now we present a more complex problem in 4 dimensions. Say we have four metals A,B,C,D. metal density carbon % phosphorus % price $/kg A B C D We want to make an alloy at minimum cost having the following requirements: range density carbon % phosphorus % min max We will decide the amount of x A,x B,x C,x D per 1 kg of alloy. The LP is: min x A +.5x B +1.5x C +x D 6500x A x B + 600x C x D x A x B + 600x C x D apple x A +0.5x B +0.15x C +0.11x D x A +0.5x B +0.15x C +0.11x D apple x A x B x C +0.1x D x A x B x C +0.1x D apple x A + x B + x C + x D =1 x A 0 x B 0 x C 0 x D 0 (Why don t we need the constraints x A,x B,x C,x D apple 1? Because it would be redundant.) Now it s suddenly impossible to see graphically where the vertices are, much less which one would minimize the objective function value. Next we introduce an algorithm to methodically choose the optimal vertex that solves the LP. 6

7 5 Simplex method Image from Wikimedia Commons George Dantzig published the simplex method in 194. It is an algorithm that jumps from one vertex to an adjacent vertex always in a direction that improves the objective function value. We will assume for the moment that every vertex is nondegenerate, and that we already know the location of one vertex, which will be our starting point for the algorithm. Before we get to the algorithm, we need to put our problem in a specific form that the algorithm will operate on. It can be shown that all convex polyhedra can be transformed to standard form: A = {x R n Ax = b, x 0} Where A is an m n matrix and b R m. This means there are m equality constraints in n nonnegative variables with m < n. Equality constraints and nonnegativity constraints easily fit into this framework. Inequality constraints can also be expressed in this way with the addition of a new variable that will bring one equality and one nonnegativity constraint. For example, say we have the inequality constraint This is equivalent to x +4y +5z apple 10 x +4y +5z + w = 10 w 0 here, w is called the slack variable. In the reverse case, we introduce a surplus variable: becomes x +8y 9z 5 x +8y 9z w = 5 w 0

8 A theorem states that a vector x R n is a vertex of the polyhedron i Ax = b, at least n m of the components of x are zero, and the columns of A corresponding to the indices of the nonzero entries of x are linearly independent. If Ax = b and exactly n m of the components of x are zero, then x is a nondegenerate vertex. This would be a good time for a few examples. Example 1 We look at the polygon: x + y apple 1 x, y 0 We introduce the slack variable w and convert the feasible polygon to standard form: x + y + w =1 x, y, w 0 This can be written in matrix form: x y w 5 = [1] x, y, w 0 apple apple apple The original vertices were,, and. Now that our decision vector includes the slack variable w, we must write the vector includingthe w for each. Itis easy to determine x 0 0 w since we know x + y + w = 1. So our vertices in 4 y 5 form are 4 0 5, 4 1 5, and w We see that our matrix is 1, so m = 1, and n =. n m =soweknowthere must be at least zeros in a point for it to be a vertex, and exactly if it is nondegenerate. This checks out! 8

9 Example We take the polygon: x + y apple 1 x apple 1 x, y 0 We introduce the slack variables w and z and convert the feasible polygon to standard form: x + y + w =1 or, in matrix form, apple x + z =1 x, y, w, z x y w z 5 = apple 1 1 After solving for w and z for each of the known vertices, we arrive at , , Now we have n = 4, m =, and n m =, so these three are all vertices, the last of which degenerate since it has zeros. This is confirmed in the above plot of the feasible region. 9

10 Example We take the polygon: x 1 + x +x apple 8 x +6x apple 1 x 1 apple 4 x apple 6 x 1,x,x 0 We add the nonnegative slack variables x 4,x 5,x 6,x to eliminate inequality constraints: x 1 + x +x + x 4 =8 x +6x + x 5 = 1 x 1 + x 6 =4 x + x =6 x 1,x,x,x 4,x 5,x 6,x 0 We assemble the vector x = x 1 x x 0 and write the above system in matrix form: x = Let s look at the point x = Is it a vertex? (We also sometimes call vertices basic feasible solutions, or BFS.) Matrix multiplication confirms that Ax = b. Since n =, and m = 4, we know x will be a vertex if at least n m = components of x are zero. Here, we have 4 components of x zero, so x is a degenerate vertex. Look at y = Again Ay = b, but now we have zero components, so y is a nondegenerate vertex. So now we look at the general case. We seek to solve standard form LPs : min c 0 x Ax = b, x 0. Let x be a nondegenerate BFS of the feasible polyhedron. Let B(1), B(), B(m) be the indices of the basic variables: these are the nonzero elements of x. Nondegeneracy of x ensures that there will indeed be exactly m such indices. For example, the nondegenerate BFS has B(1) = 1,B() =,B() = 5, and B(4) = 6. 10

11 Then, let B be the basis matrix : B = A B(1) A B() A B(m) We can see that x B, the vector of only the nonzero elements of x, is x B = B 1 b. We then seek to move from x to x + d, where is a vector along the direction of an edge towards another vertex. We will do this by selecting a nonbasic variable x j which is currently at zero level, and increasing it to a positive value, while keeping the remaining nonbasic variables at zero. Thus d j = 1, and d i = 0 for every nonbasic i other than j. The vector of basic variables x B changes to x B + d B,whered B = d B(1) d B() d B(m) 0. But how do we find d B? We know that Ax = b and we want A(x + d) =b for > 0. Therefore A(x + d) =b + Ad = b ) Ad =0. 0=Ad = nx mx A i d i = A B(i) d B(i) + A j = Bd B + A j ) d B = i=1 i=1 B 1 A j We know that B 1 exists since the columns of B are LI. The direction d is called the jth basic direction. We ensured that equality constraints hold. How about nonnegativity? If x is nondegenerate, x B > 0, and x B + d B 0 for su ciently small. So as we move along in the direction of d, we expect that the value of the objective function is improving. At what rate? It would be c 0 (x + d) c 0 x = c 0 d (per unit travelled in d). nx mx c 0 d = c i d i = c B(i) d B(i) + c j = c 0 Bd B + c j = c j c 0 BB 1 A j i=1 i=1 where c 0 B = c B(1) c B() c B(m). We call this final quantity the reduced cost c j : c j = c j c 0 BB 1 A j A theorem states that if c 0, then x is optimal. But we can t keep going along d forever, or we will exit the feasible region. We must stop at the next vertex. This means we should go to the maximum possible,,whilestaying in the polyhedron. = max{ 0 x + d P } where P is the standard form polyhedron for our problem. x + d always satisfies Ax = b by design. Thus it can become infeasible only if one of its components < 0. If d 0, then x + d 0 for all 0. We let!1and the problem is 11

12 unbounded. Otherwise, we want x i + d i to be satisfied for every i with d i < 0. Solving for gives: = min i d(i)<0 x i d i (remembering that d i < 0 which leads to an additional flip in the sign of the inequality.) We move to x + d and repeat the algorithm. 6 Example: Simplex Method We solve the following standard form LP by the simplex method: min x 1 x 1 + x + x + x 4 = x 1 + x +4x 4 = x 1,x,x,x 4 0 Converting the constraint to matrix form: apple apple x = 0 4, x 0 Take the point x = as our initial vertex. Matrix multiplication confirms x satisfies Ax = b, and n = 4, m =, so n m = and x has two zero components, so x is nondegenerate. Thus this is a valid initial vertex to begin the simplex algorithm. Since the indices of our basic (nonzero) variables are 1 and, B will be composed of the 1st and nd columns of A: B = apple ) B 1 = apple 0 1/ 1 1/ Similarly, since c =[, 0, 0, 0] 0, and the basic variables have index 1 and, we get c B = [, 0] 0. Let s calculate the reduced costs to see if we are optimal at this point. We calculate all the reduced costs at once with a single calculation: c 0 = c 0 c 0 BB 1 A apple 0 1/ = [ 0 0 0] [ 0] 1 1/ apple =[0 0 4] Note that c will always be 0 in the indices of the basic variables. nonpositive elements, this x is not optimal. Here, since c has 1

13 We could choose either the rd or 4th variable to enter the basis as we move to the next vertex, since both have negative reduced cost. Let s let the rd variable enter. We need to find d, the direction of motion. We have an equation for d B : d B = apple d1 d = B 1 A = apple / 1/ d is 1, since is the index of the entering variable. All other nonbasic variables have d = 0. In this case that just leaves d 4 = 0. So we have d =[ / 1/ 1 0] 0. Then we need to determine how far we can travel in the d direction: = min i d(i)<0 x i d i = x 1 d 1 = 1 / =/ The new vertex is y = x + d = / 1/ = / / 0 5 This vertex should have improved our objective function value. Did it? Before it was c 0 x =x 1 =, now it is c 0 y =y 1 = 0. We see that the third variable has entered the basis, replacing the first variable. Now the indices of our basic variables are,. (Note that this is not in order! It s because has replaced 1.) This y is nondegenerate because it has only zeros. We now revert to calling y x again, as we will restart the algorithm. So: and c B =[c c ] = [0 0]. B = apple c 0 = c 0 ) B 1 = c 0 BB 1 A = [ 0 0 0] apple 0 1/ 1 1/ Now we see that c 0, so the algorithm terminates. An optimal value of x =[x 1 x x x 4 ] 0 = [0 4/ / 0] 0 and the optimal function value is 0. Note that there may be other optimal values for x if there is a tie. 1

14 Solution of alloy mix problem Let s go back to the alloy mix problem and now solve it using the simplex method. We switch to matrix notation: min x x = where x 0 = x A x B x C x D s 1 s s s 4 s 5 s 6, x 0 5 In many problems, it is not obvious where to even start! We need a BFS to begin. Fortunately there is an additional part of the simplex method that can be done initially to find a BFS to begin with. We will not go through the details here, but it involves solving another LP before the one we actually seek to solve. In our case, the first LP gives us a starting BFS of: x 0 = We can check that n = 10, m =, thus there must be at least n m = zeros for a BFS, and exactly n m = if it is a nondegenerate BFS. This checks out. Furthermore, by matrix multiplication we see that it satisfies Ax = b. The basis indices for this BFS are 1,,4,6,,8,9. The matrix B corresponds to the square matrix made of these columns of A: B = and the basic cost vector c B corresponds to the vector made of these elements of c: c B = as before, the reduced cost vector is c 0 = c 0 c 0 B B 1 A 14

15 which is not nonnegative. However, after one iteration of the simplex method, we end up at the BFS x 0 = which has c 0. Thus, the optimal alloy mix is 1.4% metal A, 54.5% metal B, 8.1% metal D, and none of metal C. The optimal function value at this point is E ciency of the algorithm In practice, the simplex method is remarkably e cient. Most LPs are solvable in polynomial time, meaning that solving time scales with the problem size as a polynomial function. However in 19 a worst-case scenario was formulated that takes exponential time, which is equally bad as finding and checking all vertices. This is because in such a scenario the simplex method visits every vertex. In fact it has been shown that even variations on the method seem to always have a worst-case family of problems for which the algorithm takes exponential time. Nevertheless the simplex method and its variations continue to be in common use because of their relative simplicity and relatively fast computation speeds for most problems. 9 Employee scheduling & Integer programming LPs are very useful in the area of resource allocation. Let s look at the following example and formulate it as an LP. This example is from jensen/ormm/instruction/powerpoint/or models 09/0 lp.ppt A large company has a toll free hotline that is open 4- to answer questions regarding a new product. The following table summarizes the number of full-time equivalent (FTE) employees that must be on duty in each time block. Interval T ime F T Es

16 A large company may hire both full-time and part-time employees. The former work 8- hour shifts and the latter work 4-hour shifts; their respective hourly wages are 15.0and1.95. Employees may start work only at the beginning of 1 of 6 intervals. Part-time employees can only answer 5 calls in the time a full-time employee can answer 6 calls. (i.e., a part-time employee is only 5/6 of a full-time employee.) At least two-thirds of the employees working at any one time must be full-time employees. We let x t be the number of FTEs that begin the day at the start of interval t and work for 8 hours. Let y t be the number of part time employees that begin the day at interval t. We are trying to minimize the total wages given to all employees while fulfilling the constraints. min (x x 6 ) (y y 6 ) x 6 + x y 1 15 x 1 + x y 10 x + x y 40 x + x y 4 0 x 4 + x y 5 40 x 5 + x y 6 5 y 1 apple 1 (x 1 + x 6 ). y 6 apple 1 (x 5 + x 6 ) x t 0, y t 0 8t The output of the simplex method is x 1,...x 6 = , 0, 40, , , and y 1,...y 6 =0, , 0, , 0, 0. But how do we hire a fraction of a person? This problem is actually an integer program, because the output vector should be integervalued. However, the simplex method cannot solve integer programs. Integer programming is not as simple as it sounds i.e., you cannot just round the value from the simplex method to the nearest integer. This is a whole di erent class of problems with its own algorithms. However we will not cover them in this class. 16