R n a T i x = b i} is a Hyperplane.

Geometry of LPs Consider the following LP : min {c T x a T i x b i The feasible region is i =1,...,m}. X := {x R n a T i x b i i =1,...,m} = m i=1 {x Rn a T i x b i} }{{} X i The set X i is a Half-space. The set H i = {x R n a T i x = b i} is a Hyperplane. The feasible region X is given by the intersection of m half-spaces and is known as a Polyhedron. A Polyhedron is a closed convexset (verify?). If it is bounded it is called a Polytope. LP: Minimize a linear function over a polyhedral set. Move in the direction c as far as possible while staying within the feasible region X. Verify that any LP can be written in this form 1

Example min c 1 x 1 + c 2 x 2 s.t. x 1 + x 2 1, x 1 0, x 2 0. X2 X2 X1 + X2 = 1 c =(1, 1) T,x =(0, 0) T X1 X1 + X2 = 1 c =(1, 0) T,x =(0,x 2 ) T for 0 x 2 1 X1 X2 X2 X1 + X2 = 1 c =(0, 1) T,x =(x 1, 0) T for 0 x 1 X1 X1 + X2 = 1 c =( 1, 1) T Unbounded. X1 2

Extreme Point Optimality An optimal solution (if it exists) always lies on a boundary of the feasible region X If there is an optimal solution, then there is one at a corner or a vertex or an extreme point of the polyhedron X. A point x X is an extreme point of X if it cannot be expressed as a convex combination of some other points x 1,...,x m X. A polyhedron has a finite number of extreme points. Representation theorem: When X is a polytope, any point x X can be expressed as a convex combination of the extreme points of X, i.e. x = Ii=1 λ i x i where x i vert(x) for all i =1,...,I, Ii=1 λ i =1,andλ i 0. 3

Proof of Extreme Point Optimality Consider the LP min{c T x x X}, where X is a polytope. Let x be an optimal solution, and suppose that there are no extreme points of X that are optimal. Then c T x <c T x i for all x i vert(x). By the representation thm: x = I i=1 λ i x i,then c T x = > I i=1 I i=1 = c T x and we have a contradiction. λ i c T x i λ i c T x 4

Unbounded Polyhedra A feasible direction of an unbounded polyhedra X R n is a (non-zero) vector d R n, such that if x 0 X then (x 0 + λd) X for all λ 0. An extreme direction of an unbounded polyhedra X R n is a direction d R n that cannot be expressed as a convexcombination of other directions of X. A polyhedron has a finite number of extreme directions. Representation theorem for general polyhedra: Any point x in a non-empty polyhedron can be expressed as a convexcombination of the extreme points of X, and a linear combination of the extreme directions of X, i.e. x = I i=1 λ i x i + J j=1 µ j d j where x i vert(x) foralli =1,...,I, I i=1 λ i =1, λ i 0, d j are the extreme directions of X for all j =1,J,andµ j 0. Extreme point optimality for general polyhedra can be proven using the above result. 5

Characterizing Extreme Points Consider an LP in standard form: min{c T x Ax = b, x 0}, where c R n, A R m n, and b R m. that rank(a) =m n. Assume Let A B =[a i 1,...,a i m ] be a matrix formed by m l.i. columns of A (called a Basis). Let A N be the matrixof the remaining columns, i.e. A = [A B A N ]. Partition the vector x = [ xb x N ] where x B =(x i1,..., x im ) T (Basic variables), and x N is the vector of remaining components of x (Non-basic variables). Set x B = A 1 B b and x N =0. Notex = [ A 1 B b 0 ( n m a solution to Ax = b. Known as a Basic solution. How many basic solutions? Ans: At most ] ) is. If x B 0, i.e. x 0, then the solution is a Basic Feasible Solution (BFS). 6

Example Consider the following polyhedral set: X := {x R 2 x 1 + x 2 6, x 2 3, x 1 0 x 2 0}. In standard form: x 1 + x 2 + x 3 = 6 x 2 + x 4 = 3 x 1, x 2, x 3, x 4, 0 [ ] 1 1 1 0 Note A =[a 1,a 2,a 3,a 4 ]=. 0 1 0 1 Choose A B =[a 1,a 2 ]. Then x B =(x 1,x 2 ) T =(3, 3) T. The solution x =(3, 3, 0, 0) T is a basic solution (also BFS). Choose A B =[a 2,a 4 ]. Then x B =(x 2,x 4 ) T =(6, 3) T. The solution x =(0, 6, 0, 3) T is a basic solution (not BFS). Note that there are 5 = a 1 and a 3 are not l.i.). ( 4 2 ) 1 basic solutions (since 7

Example (contd) A x BFS [a 1,a 2 ] (3, 3, 0, 0) T Yes [a 1,a 4 ] (6, 0, 0, 3) T Yes [a 2,a 3 ] (0, 3, 3, 0) T Yes [a 2,a 4 ] (0, 6, 0, 3) T No [a 3,a 4 ] (0, 0, 6, 3) T Yes x1+x2 = 6 (0,6) (0,3) (3,3) x2 = 3 (0,0) (6,0) Each BFS correspond to an extreme point of the feasible region. Algorithm? Check all BFS (extreme points) and choose best. Exponential impractical for large problems. 8

Optimization Strategy Idea: Move from one extreme point to a better (adjacent) extreme point. Need to characterize adjacent extreme points. Need a strategy to move to a better adjacent extreme point. Need a stopping condition. Need to detect unboundedness and infeasibility. 9

Adjacent BFS Two extreme points are adjacent if they share an edge of the polyhedron. Algebraically, two BFS x 1 and x 2 are adjacent if (only) one of the basic variables in BFS x 1 is non-basic in BFS x 2, and vice versa. Moving from a BFS to an adjacent BFS one of the non-basic variables takes on a positive value (enters the basis), while one of the basic variables drops down to zero (leaves the basis). Values of other basic variables change too. (0, 3, 3, 0) (3, 3, 0, 0) (0, 0, 6, 3) (6, 0, 0, 4) 10

Improving Direction Which non-basic variable should enter the basis? Ans: One that improves the objective function value (z). Let N := {j x j is nonbasic, i.e. a j is a column of A N }. Also partition c = [ cb c N ]. Now, A B x B + A N x N = b x B = A 1 B b A 1 B A Nx N. The objective function value z = c T B x B + c T N x N = c T B A 1 B b + ( c T N ct B A 1 B A N ) xn = c T B A 1 B b + j N = c T B A 1 B b + j N (c j c T } B{{ A 1 B a j) x } j r j x j. Note for the current bfs the objective value is z = c T B x B = c T B A 1 B b. If a nonbasic variable x j becomes positive, the objective value increases by r j. Therefore choose j s.t. r j < 0 to reduce the obj. func. The quantity r j is known as the reduced cost for nonbasic variable x j. If r j 0 for all j N then the current solution is optimal. r j 11

Step Length How much should the value of the (chosen) nonbasic variable be increased? Ans: As long as all of the basic variables 0. Suppose we chose the nonbasic variable j to increase (enter into the basis). Then x B = A 1 B b A 1 B a jx j. Let B = {i x i is basic}, andlety ij =(A 1 B a j) i where i B and j N. Then, x i =(A 1 B b) i y ij x j i B. Suppose y ij > 0, then increasing x j would reduce x i. For feasibility, we must have x i 0foralli B,i.e. if y ij > 0 (A 1 B b) i y ij x j 0 i B, i.e. x j (A 1 B b) i y ij i B. Thus we can increase x j to: x j = min {i B y ij >0} (A 1 B b) i y ij. 12

Step Length (contd.) Suppose the minimization in the above expression is achieved for î B. Then we increase x j =(A 1 B b) î /y îj, and consequently the basic variable xî drops to zero (leaves the basis). Suppose y ij 0foralli B, then increasing x j would not cause any of the basic variables to decrease in value. We can then increase x j as much as we want without losing feasibility and keep on improving the objective. In this case the problem is unbounded. What if min {i B yij >0} { } (A 1 B b) i y ij = 0? This implies that one of the basic variables was at a value zero. Such a BFS is known as a degenerate BFS and may cause problems in the algorithm. However there are ways of recovering from degeneracy problems. 13

Finding an Initial BFS Consider an LP min{c T x Ax = b, x 0}. If A has an m m identity submatrix I, thenusei as the initial basis. Otherwise, suitably multiply the constraints by ±1 such that b 0. Introduce auxiliary variables y = (y 1,...,y m ) T, and construct the following LP: min e T y s.t. Ax+ Iy = b, x, y 0. Use I as the starting basis, and solve the above problem to optimality. If the optimal objective value is zero y =0,then we have a basis consisting of only the columns of A, which we can use as an initial BFS for the original problem. If the optimal objective value > 0, then the original problem is infeasible. 14

The Simplex Method - G. B. Dantzig (1947) 1. Find an initial feasible extreme point solution (BFS). If non exists, STOP, the LP is infeasible. 2. Find a direction (along an edge) in which the objective function improves (i.e. find j N such that r j < 0). If none exists, STOP, the current solution is optimal. 3. Move along the direction until you reach an adjacent feasible extreme point (BFS). If you can move along the edge of the feasible region without reaching an adjacent feasible extreme point (i.e. y ij 0), then STOP, the LP is unbounded. 4. Go to step 2 with the current feasible extreme point. 15

Remarks In the absence of degeneracy, the Simplexmethod terminates finitely with an optimal solution to the LP. In the worst case, it might require terminate. ( n m ) steps to In most practical problems the performance is much better. The performance is sensitive to the number of rows. Degeneracy might cause the method to cycle. However this can be avoided by some anti-cycling rules in choosing the variables leaving and entering the basis. The optimal solution produced by the simplexalgorithm is a BFS, therefore, (at most) m of the n variables are positive. 16