An Intro to LP and the Simplex Algorithm. Primal Simplex

Transcription

1 An Intro to LP and the Simplex Algorithm Primal Simplex Linear programming i contrained minimization of a linear objective over a olution pace defined by linear contraint: min cx Ax b l x u A i an m n matrix, c i a n vector, and x, b, l, and u are n vector. ( 5 -x + x2 2x - < x2 2 -x +2x2 < 4 It often helpful to illutrate the linear algebra with a concrete example. To the left i the running example that ll be ued in thee note. -3x - < x x2 2 x3 x23 The olution pace the primal polyhedron i the triangle bounded by the three contraint 3x 3, 2x 2, and x + 4. The objective function x + i hown a the line above and to the left of the triangle. For the poition hown, the value of the objective z cx i. - - x2 2 x min x + 3x 3 (C 2x 2 (C2 x (C3 3 x (C4 4 5 For thi example, we want to minimie the objective function. Specifically, we want to find a point x in the feaible region of the primal polyhedron for which z cx i minimied. To minimie the value of cx, we want to move in the direction c. You can think of it a liding the objective in the direction c until we ve reached the deired point. In thi example, the point x i the one labelled x2 (,, and the optimal value i z. Notice that thi example i conveniently contructed o that the primal polyhedron i bounded by the explicit contraint C, C2, and C3 the imple bound contraint C4 are not really needed. In general, however, the imple bound contraint l x u are neceary. For purpoe of thee note, it will be convenient to have them available for a few of the explanation which follow. The contraint C, C2, and C3 are often referred to a the explicit contraint; they make up the ytem Ax b ued by the implex algorithm. The bound contraint C4 are often referred to a implicit bound contraint, becaue they are enforced by the implex algorithm and are not included in the ytem Ax b Theory allow u to guarantee that if a LP problem ha a bounded optimal olution, it occur at an extreme point. It poible for the objective function to be aligned in uch a way that it include everal extreme point; in thi cae they will all have the ame optimal objective value.

2 In n-pace, a point i defined by n contraint atified at equality. A contraint that atified at equality at a particular point i aid to be tight; at the point labelled x3, the tight contraint are C and C3. For each extreme point of the primal polyhedron, we can identify a et of tight contraint. Contraint that are not tight at a particular point are looe. It poible to have more than the neceary number of tight contraint at a particular point. At x2, there are three tight contraint: C, C2, and x (from C4. Thi i called degeneracy; we ll come back to it later. Baic Solution For now, aume that a point ha jut enough tight contraint. The et of tight contraint correpond to what known a a baic olution. If it happen that the point in quetion i in the feaible region of the polyhedron, then it a feaible baic olution. To ee how one obtain the value of the primal variable for a baic olution, we need ome linear algebra. The original inequalitie are converted to equalitie by the addition of a lack variable for each contraint. The augmented minimization problem can be written min ( c x x A I b (2 x It mut be true that i to avoid violating the original inequality contraint. Applied to the running example, we have min x x + 3 2x x Written in matrix notation, the contraint ytem look like x To get the value of the variable at a baic feaible olution, we need to olve (2 in uch a way that the contraint that hould be tight, are tight. One way to pecify thi i to require that the lack variable for thoe contraint be. We can create thi ituation by partitioning the variable into two et, called the baic et and the nonbaic et. Since each column of the coefificient matrix ( A I correpond to a variable, we ve alo partitioned the column. One common notation for thi i that B repreent the baic column, N repreent the nonbaic 2 2

3 column, x B repreent the baic variable, and x N repreent the nonbaic variable. Both x B and x N are, in general, a mix of the original variable x j and the lack variable i. For our example, to pecify the point x3 we want and to be, o we ll et 3 B 2 x B x 2 2 N x N 3 In matrix notation, we ve rewritten the contraint ytem a x A I Bx B + Nx N b To olve thi for x B, we multiply through by B and rearrange to get x B B b B Nx N (3 Since we ve decreed that and (i.e., x N are, we have that the value of the variable x B B b. An overbar i one common notation to indicate that a vector ha been multiplied by the bai invere, o we ll ue b B b. Applying (3 to the running example give x 2/7 /7 /7 3/7 3 2/7 /7 2 /7 3/7 2 3/7 5/7 4 3/7 5/7 x 2/7 2/7 /7 5/7 /7 3/7 ( 2 25/7 3/7 5/7 3 ( Thi particular form of the contraint ytem i often referred to a a dictionary. For a much more comprehenive introduction to implex uing thi dictionary form, have a look at the text by Chvátal []. Optimality Now, how do we know if thi point i the bet poible point? We need to be able to tell if it poible to improve the objective function value. To accomplih thi, we need to rewrite the objective function in a way that matche the current baic olution. The original objective function for the running example i x +. We ve added lack variable in the coefificient matrix, and if we do the ame in the objective, we d have x In the LP literature, the coefificient of the objective are commonly denoted with the ymbol c, and we have c (. To rewrite the objective function o that it matche the current baic olution, we partition c into a baic part c B and a nonbaic part c N to match the partition of the contraint ytem. Now we can rewrite the objective a z c B x B + c N x N and ubtitute for x B from (3 to get z c B B b + (c N c B B Nx N (4 3

4 Notice that the firt term i a contant it the current value of the objective. The econd term allow u to evaluate the effect of making one of the nonbaic variable nonzero. The expreion (c N c B B N can be boiled down into a vector of reduced cot c j, one for each nonbaic variable x j. Let expand (4 for the running example: z ( 2/7 /7 /7 3/7 3 2/7 /7 2 + /7 3/7 3/7 5/7 4 3/7 5/7 ( /7 4/ /7 4/ /7 + (( ( /7 4/7 3/7 + ( /7 4/7 To determine the effect of increaing x j from, we examine the correponding c j. Since we re minimiing, we need c j < in order to make the objective maller. If c j for all j, we re done. If ome c j <, then we can increae the correponding variable and decreae the objective function. For our example, increaing look like the bet bet the value of z will decreae by 4 7 for every unit increae in. ( Pivoting To ummarize the ituation o far: We re at ome feaible baic olution. Examining the value of the reduced cot c j, we ve identified a nonbaic variable x j which we would like to increae in order to improve the objective. If we change the value of x j, we will certainly move away from the current extreme point. But to where? Ideally, we d like to move to another feaible baic olution. So we need to ee how that can be arranged. Recall (3: x B B b B Nx N. Jut a with the reduced cot, we can teae apart B N to produce tranformed column of N, denoted a j, which will tell u the effect on x B a we change ome nonbaic x j. Recall that (3 expanded to x 2/7 2/7 /7 5/7 /7 3/7 25/7 3/7 5/7 2 ( Provided that we conider changing only one x j at a time, it will be eay to determine the effect. Let look firt at. It current value i, and we intend to increae it. ha no upper bound, o it can be increaed indefinitely without loing feaibility. What of the effect on other variable? Other nonbaic variable (e.g., are unaffected. The effect on the value of the baic variable i determined by the correponding column a j. For, the relevant column i ( /7 3/7 5/7. A 3 increae, x will increae and and 2 will decreae. The latter two preent a problem: If they fall below, we ve moved out of the feaible region for the problem. So the maximum value for i 5, at which point the value of and 2 will have been driven to zero. (5 4

5 Let look at that again, from the viewpoint of how we moved in the polyhedron. The nonbaic variable are lack; if we increae one, it correpond to moving away from a contraint. We re increaing, o we re moving away from C3. Since remain, contraint C remain tight we re moving along the line defined by C. We can keep moving along the periphery of the polytope until we bang into another contraint put another way, until we drive a baic lack variable to. In thi cae, we drive 2 to and bang into contraint C2 at the point x2. Notice the direction of motion: For every 7 unit that increae, x will increae by unit and will decreae by 3 unit. Thi i exactly the required ratio to follow along the contraint 3x 3. Thi i, of coure, no accident: Our contraint ytem, combined with the rule that only one nonbaic variable can change, guarantee that we ll move along an edge of the polyhedron. Remember: we re at a point defined by n tight contraint. In the jargon of n- dimenional geometry, we ll move away from the hyperplane defined by one of thoe contraint, along the edge defined by the interection of all the other hyperplane that were tight at the point, until we bang into the hyperplane at the other end of the edge. We ve reached a new extreme point. Of coure, at thi point we re violating the convention of a baic olution: we have a nonbaic variable,, that i non-zero. But fortunately, we alo have a baic variable, 2, that we ve jut reduced to. So we do the obviou thing we wap them. 2 move from x N to x B, it cofificient column move from N to B, and it objective coefificient move from c N to c B. and all it aociated coefificient move in the other direction, from the nonbaic partition to the baic partition. The entire proce of moving from one extreme point to another i a pivot. Following the matrix math, we now have 3 B 2 x B x 2 N x N 2 Expanding, we get x /5 /5 2/5 3/5 3 /5 /5 2 2/5 3/5 3/5 7/5 4 3/5 7/5 x /5 /5 2/5 3/5 ( 5 3/5 7/5 2 ( 2 and (paring the gory detail z + ( /5 4/5 2 From thi, we can ee that there no room for further improvement, becaue the reduced cot are all poitive. Increaing a nonbaic variable will increae (woren the objective. 5

6 5 -x +2x2 < 4 2x - < x2 2-3x - < x x2 2 x3 x* x23 -x + x x The figure to the left illutrate the ituation after the pivot. If we evaluate the objective function at the point x2 x, we have z. Viually, it eay to ee that x i the bet we can do if the objective move any further in the direction (, it will leave the feaible region of the polytope. - Degeneracy Let go back a few paragraph to the entence But fortunately, we alo have a baic variable, 2, that we ve jut reduced to. We could jut a well have aid But fortunately, we alo have a baic variable,, that we ve jut reduced to. What will happen if we chooe to move to the nonbaic partition? We have B 3 2 x B x 2 N x N x2 2 Working through the math yield x 2 /3 /3 5/3 2/3 ( x2 5 7/3 /3 z + ( 4/3 /3 Apparently, we need to increae. Looking at the value of the baic variable, we can ee that the limiting variable i 2 ; in fact, it limit the increae of to. Another way of looking at the ituation i that we re dicovering that contraint 3x 3 (C and (C4 are not ufificient to keep u from reducing the objective by moving along the x axi in the direction (. For that, we need the contraint 2x 2 (C2. 6

7 Plowing ahead and performing the pivot, we have B 3 2 x B x N x N x2 2 2 and x /2 /2 5/2 3/2 ( x2 5 3/2 /2 2 z + ( /2 /2 2 Examining the reduced cot, we can ee that the olution i optimal all reduced cot are poitive. To ummarie: We have one optimal point, x (. Thi point i degenerate, however, with three tight contraint where only two are required. Thi mean that there are ( poible bae, x B x x B x x B x x N 2 x N x2 2 The bai on the left i optimal, with C and C2 pecified a the tight contraint. The bai in the middle i alo optimal it ue C2 and, a combination which prevent further motion in the direction of the objective. The bai on the right, while it correctly pecifie the optimal point (, doe not atify the criteria for optimality. However, at the point where we needed to chooe between the bai on the left and the bai on the right, there wa no eay way to ee that chooing the bai on the right would require another pivot to reach optimality. I m going to potpone further explanation until we ve conidered the dual problem and dual implex. x N 2 ( x2 Finite Lower and Upper Bound The explanation to thi point ha only dealt with variable with bound of the form x. There a good reaon for thi: it eay to explain. But once you ve graped the idea for the imple cae, the extenion to arbitrary lower and upper bound in t hard. It remain true that any nonbaic variable i at a bound, but now that bound can be either a lower or an upper bound, not necearily zero. It eaiet to ee how thi will work by augmenting the matrix expreion for the variable 7

8 and objective value. ( x B x N B B N l/u I z ( c B c N( B l/u N ( c B (6 c N( B N N I The expreion l/u i intended to indicate the ue of the lower or upper bound, a appropriate for the tatu of the nonbaic variable. In place of the vector of nonbaic variable x N, we have N, to more accurately reflect that a nonbaic variable can increae from a lower bound (poitive N j or decreae from an upper bound (negative N j. The interpretation of reduced cot change appropriately: If we re minimiing, and a nonbaic variable can increae from a lower bound, then the correponding reduced cot mut be poitive at optimality. On the other hand, for a nonbaic variable which can decreae from an upper bound, the correponding reduced cot mut be negative at optimality. At the point in a pivot when we ve choen a nonbaic variable to increae or decreae, we have to do a bit more work to determine the limit on the change. A before, we need to conider the affect of the change on the baic variable. One part of the tet i conceptually the ame. We need to find the firt baic variable that i driven to bound. Now, however, we need to conider that the nonbaic variable could be increaing or decreaing, and the baic variable could be driven to a lower or an upper bound. Finally, it could be that the limiting bound i in fact the other bound of the nonbaic variable. In thi cae, there i no change in the baic/non-baic partition. We imply update the value of the variable to reflect the new value of the nonbaic variable. Fair Pricing We ve been uing imple reduced cot c N c N c B B N to elect a nonbaic variable that can be ued to improve the value of the objective function. That a little naive. Multiplying a column by a calar can dramatically change the value of the reduced cot c j aociated with a column, without changing the actual problem in any way. What we d like to do i put all the reduced cot on an equal footing. One way to accomplih thi i to ue the value of a unit vector in the direction of motion. Look back to the ytem of equation (6. If a ingle nonbaic variable i changed, all k N except for the N j aociated with the choen variable x j. The imple reduced cot aociated with thi variable can be written a c j c B a j e j where ej i a unit vector with a in the poition correponding to x j. The direction of motion iη j. B a j e j To calculate a reduced cot which repreent the change in objective over a unit length in the direction of motion, we imply normalie by the length ofη j, a c j cη j / η j. Dual Simplex For every LP problem, we can contruct omething called the dual problem. Typically, you ll ee omething like thi: max cx Ax b x minyb ya c y (7 8

9 The primal problem on the left i tated a a maximiation problem, to match the tandard preentation of duality. Recognie that max cx min( cx and you hould have no trouble with converion. For our running example, the dual problem will be min 3y + 2y 2 + 4y 3 3y + 2y 2 y 3 (D y y 2 + 2y 3 (D2 y (D3 Here a picture of the dual polytope. You re looking down into the interior of the feaible region; the optimal point i the one on the left of the black triangle, y ( /5 4/ y3 -.5 y y There are many correpondence between the dual and primal problem. By laying on a large amount of linear algebra, it poible to prove them. We ll tart with the following obervation: cx yax yb For a primal maximiation problem and a dual minimiation problem, the primal objective cx tart low and increae, while the dual objective yb tart high and decreae. Thi give an upper and lower bound on the optimum value of the olution. It alo poible to how that the dual variable y c B B. In word, the dual variable are the product of the primal baic cot coefificient and the primal bai invere. To prove thi require 9

10 ome fairly intricate linear algebra with carefully partitioned matrice. See [2] for the gory detail of thi and additional primal/dual correpondence. It can be hown that each baic olution of the primal polyhedron ha a correponding baic olution in the dual polyhedron. Notice I didn t ay feaible baic olution. The primal approache optimality by a equence of primal feaible, but dual infeaible, baic olution. The dual approache optimality by a equence of dual feaible, but primal infeaible, baic olution. Further, it can be hown that the objective value at optimum i equal for the primal and the dual. For the general cae of upper and lower bounded primal variable, ee [2]. We can get a quick reult by auming that all primal variable have a lower bound of, and, further, that thi bound i explicitly incorporated into the contraint ytem. Adding lack variable to the primal contraint, and noting that thi mean we can remove the contraint y in the dual ytem, we have the following correpondence: max ( c x E I A I x ( I I E b I min b y E y I A I ye y I ( c I I What can we ee? For one, the preence of primal lack variable tranlate directly into the contraint y. Now, conider primal reduced cot. Becaue all primal variable are free (i.e., they have no implicit bound contraint, they mut remain baic. (To be nonbaic, a variable mut be at bound. Hence only lack will be nonbaic. The reduced cot are c N c N c B B N I But c N and y c B B and N, o c N y. Given a maximiation primal, the condition I for optimality i c N. If we re not yet at an optimal point, then it mut be that ome c k >, hence the correponding y k < and i infeaible. Coming from the other direction, if you were to olve the example problem uing dual implex, the equence of dual baic feaible olution and the correponding primal baic infeaible olution i (Remember, min cx max cx. Dual Primal Objective ( ( 2 2 ( /2 ( /5 4/5 Reoptimiing after Modifying the Contraint Sytem Thee correpondence are ueful to know becaue it mean that we can witch back and forth between primal and dual implex algorithm while working on the ame problem. Thi i very ueful becaue we will be able to add contraint without affecting dual feaibility, and add

11 variable without affecting primal feaibility. Thi mean that it poible to augment a contraint ytem and reoptimize very efificiently. We ll make heavy ue of dual implex to reoptimie after adding cutting plane, o let quickly ee how thi work. The original bai matrix i B, and we want to add a new contraint a B x B + a N x N + k b k. Becaue we re adding a new row to the bai, we ll need to add a new column. ( Suppoe that we elect to make k the new baic variable. The new bai matrix will be B B a B. With a bit of linear algebra, we can invert thi while retaining the partition, and we B have (B a B B. Augmenting c B to c B ( c B and calculating the new dual variable give y c B (B ( c B B a B B ( y The new dual i introduced with the value, which retain dual feaibility. Now, what really happened? When we decided to make k baic, we aid, in eence, contraint k i not tight (i.e., not atified at equality, and k will be non-zero. If thi new contraint i an effective cutting plane, thi will be true. But... it will be true becaue the cutting plane cut off the current optimal point in other word, the current optimal point i infeaible with repect to thi new contraint, and k <. We ve lot primal feaibility, o we can t imply continue with primal implex. But we retain dual feaibility, and we can ue dual implex to reoptimie. The relation Strong Duality cx yax yb i known a weak duality. We re now prepared to demontrate a tronger tatement, called (reaonably enough trong duality. For the pair of primal and dual implex problem (7, there are exactly four poibilitie. The objective function cx and yb are finite and equal. 2. cx and the dual problem i infeaible. 3. yb and the primal problem i infeaible. 4. Both the primal and dual are infeaible. c a x b a a 2 cx a 2 x b 2 One way to undertand thi reult i to look at it from a geometric point of view. The figure to the left how two contraint, a x b and a 2 x b 2, and an objective cx. The normal to the contraint and objective are alo hown. Auming we re maximiing cx, thi i the geometry at an optimal point.

12 One way of looking at the dual contraint ytem ya c i that it mut be poible to build the normal for the objective function, c, from the normal of the tight contraint in thi cae, a and a 2 uing a poitive linear combination of the normal. Geometrically, the normal of the objective mut lie inide the cone formed by the normal of the tight contraint which form the optimal point. c a a 2 cx a x b a2 x b 2 A the extreme point become more acute, the cone of normal broaden. c cx a a 2 a x b a 2 x b 2 It poible to go too far, however. At ome point, the contraint will diverge in the direction of the objective. Thi ay that we can increae cx without limit the problem i unbounded. Further, if you examine the figure cloely, you ll ee that the cone formed by a and a 2 now open downward. No linear combination y a + y 2 a 2 with y can form the normal c. The primal i unbounded, and the dual i infeaible. 2

13 One common tatement of duality goe by the name of Farka Lemma, and it come in many variation. Let look at one of them: Farka Lemma: Either {y R m + ya c} or (excluively there exit x Rn + uch that Ax and cx >. cx -5x + 2 (-5,2 x - 2 (,3 (,-2 (-5,2 (,3 (,-2 What i thi aying? A homogenou ytem (i.e., with a contraint ytem with a righthand-ide of form a cone with a point at the origin. Here, we re intereted in the cone Ax, and x will be retricted to the poitive orthant R n +. If we alo have cx >, then the primal problem i unbounded, a can be een in the figure to the left. Examining the normal for the two contraint, we can ee that it will not be poible to contruct the normal for the objective function a a poitive linear combination of the normal of the contraint. (The mall figure in the lower right bring the normal together and make thi obviou. Reaoning in the other direction, any c we can contruct a a poitive linear combination ya c mut reach it maximum value at the origin. Hence the dual and primal are feaible and bounded, with z. An alternative line of reaoning i to note that for any x inide the cone defined by Ax, and any c ya, it mut be that yax c x. Implementation It a long way from the baic mathematic of implex to an actual implementation. The iue involved are efificiency and numerical tability. It impoible to do jutice to them in thee note, o I ll imply point to a pair of reference. The technical report [3] decribe in detail the actual algorithm ued in the DYLP linear programming code. The text by Maro [4] i an excellent general reference for implementing implex algorithm. Reference [] V. Chvátal. Linear Programming. W. H. Freeman and Company, New York, N.Y., 983. [2] L. Hafer. A Note on the Relationhip of Primal and Dual Simplex. Technical Report SFU-CMPT TR 998-2, School of Computing Science, Simon Fraer Univerity, Burnaby, B.C., V5A S6, December 998. [3] L. Hafer. DYLP: a dynamic lp code. Technical Report SFU-CMPT TR , School of Computing Science, Simon Fraer Univerity, Burnaby, B.C., V5A S6, December 998. [4] I. Maro. Computational Technique of the Simplex Method. Kluwer Academic Publiher, Norwell, Maachuett,