What is linear programming (LP)? NATCOR Convex Optimization Linear Programming 1. Solving LP problems: The standard simplex method

Transcription

1 NATCOR Convex Optimization Linear Programming 1 Julian Hall School of Mathematics University of Edinburgh jajhall@ed.ac.uk 14 June 2016 What is linear programming (LP)? The most important model used in optimal decision-making Grew out of US Army Air Force logistics problems in WW2 First practical LP problem was formulated by George Dantzig in 1946 Dantzig invented the principal solution technique the simplex algorithm in 1947 In the Top 10 algorithms of the 20th century The algorithm that runs the world New Scientist (2012) LP and the simplex method have revolutionised organised human decision-making G. B. Dantzig ( ). J. A. J. Hall NATCOR Convex Optimization: Linear Programming 1 2 / 18 What is linear programming (LP)? A linear programming model The diet problem Given Costs of foodstuffs Nutrient information and daily requirements What is the cheapest diet? Let x j be the amount of food j purchased (x j 0), for j = 1,..., n Cost c j of food j gives total cost f = c j x j to be minimized Let a ij be the amount of nutrient i in food j Matching the daily requirement b i gives a ij x j = b i, for i = 1,..., m Solving LP problems: The standard simplex method General LP problem in standard form is with m equations and n variables (m < n) The standard (tableau) simplex method Choose a column q with positive entry in ĉ B N â q â pq ĉ q ĉ T Choose a row p with least ratio between components in b and âq Update the tableau Need a better idea of what s going on and how to solve LP problems efficiently RHS b bp J. A. J. Hall NATCOR Convex Optimization: Linear Programming 1 3 / 18 J. A. J. Hall NATCOR Convex Optimization: Linear Programming 1 4 / 18

2 Solving LP problems: The feasible region The simplex algorithm: Choosing where to move General LP problem is with m equations and n variables (m < n) The feasible region Solution of Ax = b is a n m dimensional hyperplane in R n Intersection with x 0 is the feasible region is a polyhedron A vertex has n m zero components m components given by Ax = b A solution of the LP occurs at a vertex of n = 3; m = 1 ĉ 3 x ĉ 1 General LP problem is At a vertex x, consider moving along an edge of to improve the value of f Corresponds to increasing a variable x j from zero Rate of change ĉ j of f with x j is known If no positive ĉ j then x is optimal so stop Increase variable x q with greatest ĉ j J. A. J. Hall NATCOR Convex Optimization: Linear Programming 1 5 / 18 J. A. J. Hall NATCOR Convex Optimization: Linear Programming 1 6 / 18 The simplex algorithm: Choosing how far to go The simplex algorithm: Basic and nonbasic variables x; f αd x + αd; f + αĉ 1 Increase variable x q with greatest ĉ j > 0 Edge direction d corresponding x q is known Move along x + αd Stop when first component of x + αd is zeroed x q increases to α > 0 Objective increases by α ĉ q > 0 Repeat! At a vertex x There are n m nonbasic variables x N with value zero Their indices form a set N The columns of A corresponding to N form the matrix N The components of c corresponding to N form the vector c N There are m basic variables x B with values uniquely defined by the m equations Their indices form a set B The columns of A corresponding to B form the nonsingular basis matrix B The components of c corresponding to B form the vector c B The partitioned LP in standard form is then maximize f = c T B x B + c T N x N subject to Bx B + Nx N = b x B 0, x N 0 J. A. J. Hall NATCOR Convex Optimization: Linear Programming 1 7 / 18 J. A. J. Hall NATCOR Convex Optimization: Linear Programming 1 8 / 18

3 The simplex algorithm: The reduced objective and optimality The simplex algorithm: Data requirements Using the partitioned equations Bx B + Nx N = b x B = B 1 (b Nx N ) = b B 1 Nx N where b = B 1 b Each simplex iteration requires Reduced costs ĉ = c N N T B T c B Reduced RHS b = B 1 b B â q â pq N RHS b bp Hence, substituting for x B, the objective function is Edge direction d corresponding to increasing x q ĉ q ĉ T now x B = b B 1 N x N f = c T B x B + c T N x N = c T B ( b B 1 Nx N ) + c T N x N = f + ĉ T x N where f = c T B b is the objective value when x N = 0 ĉ = c N N T B T c B is the vector of reduced costs Behaviour of f as x N increases from zero yields the optimality condition ĉ 0 This is the key to solving LP problems x N = x q eq when increasing just x q Hence x B = b x q B 1 N eq Edge direction is thus [ ] eq d = where âq âq = B 1 (Neq) = B 1 aq J. A. J. Hall NATCOR Convex Optimization: Linear Programming 1 9 / 18 J. A. J. Hall NATCOR Convex Optimization: Linear Programming 1 10 / 18 The simplex algorithm: Computational requirements Find reduced costs ĉ = c N N T (B T c B ) thus: Solve B T π = c B Form ĉ = c N N T π Find basic edge direction âq = B 1 aq thus: Solve B âq = aq Reduced RHS b is maintained by updating The revised simplex method Data required by the algorithm can be found by solving square systems of equations B T π = c B and B âq = aq Efficiency depends on How B 1 is represented How the following relation between successive matrices B is exploited B := B + (ap aq)e T p J. A. J. Hall NATCOR Convex Optimization: Linear Programming 1 11 / 18 The simplex algorithm: Efficient implementation Standard simplex method Maintains B 1 N, b and ĉ in a rectangular tableau Requires O(mn) storage and O(mn) computation per iteration Inefficient and prohibitively expensive for large problems Revised simplex method Computes âq = B 1 aq as required, forms ĉ and updates b = B 1 b Requires up to O(m 2 ) storage and O(m 2 ) computation per iteration but vastly less for sparse LP problems Efficient for large (sparse) problems Important to exploit matrix sparsity J. A. J. Hall NATCOR Convex Optimization: Linear Programming 1 12 / 18

4 The simplex algorithm: Recap The simplex algorithm: Does it terminate? In how many iterations? x; f αd x + αd; f + αĉ 1 Increase variable x q with greatest ĉ j > 0 Edge direction d corresponding x q is known Move along x + αd Stop when first component of x + αd is zeroed x q increases to α > 0 Objective increases by α ĉ q > 0 Repeat! Proof of termination In each iteration the objective increases by α ĉ q > 0 So, cannot re-visit vertices Number of vertices is finite So, the algorithm terminates In practice the algorithm visits n! O(m + n) m!(n m)! vertices Could it visit all the vertices? x 0 x J. A. J. Hall NATCOR Convex Optimization: Linear Programming 1 13 / 18 J. A. J. Hall NATCOR Convex Optimization: Linear Programming 1 14 / 18 The simplex algorihtm: The worst case can happen! lee-minty problems maximize 10 j 1 x j subject to x i j i x j 100 n i for i = 1,..., n j=i+1 0,..., x n 0 Problems have n variables and n constraints Feasible region has 2 n vertices Simplex algorithm visits all of them! It is exponential in time Led to better simplex variants which weight ĉ j by d j Goldfarb and Reid (1977) J. A. J. Hall NATCOR Convex Optimization: Linear Programming 1 15 / 18 Bigger LP problems Suppose N people are choosing diets with limits on the available food Person k has an individual LP min c T k x k s.t. A k x k = bk and x k 0 N Person k consumes x kj = [x k] j of food j so x kj b j k=1 This availability constraint links the individual LPs Dantzig-Wolfe structure min f = c T c T N x N s.t. A A 0N x N b0 A 1 = b1.... A N x N = bn 0... x N 0 Solution methods can exploit problem structure J. A. J. Hall NATCOR Convex Optimization: Linear Programming 1 16 / 18

5 Simplex method: Why use it? Linear Programming 1: Summary Interior point methods (IPM) are the modern alternative to the simplex method For single LP problems IPM are generally faster For some classes of single LP problems the simplex method is faster When solving sequences of related LP problems the simplex method is preferable Branch-and-bound for discrete optimization Sequential linear programming for nonlinear optimization Why? Simplex method yields a basic feasible solution Simplex method can be re-started easily from an optimal solution of one LP to solve a related LP quickly 69 years old and going strong! Described the simplex algorithm Identified that efficient implementation depends on techniques for solving related systems of equations Remains to consider how to exploit LP problem structure and matrix sparsity: Wednesday 14:30 15:30; 16:00 17:00 J. A. J. Hall NATCOR Convex Optimization: Linear Programming 1 17 / 18 J. A. J. Hall NATCOR Convex Optimization: Linear Programming 1 18 / 18