IE 5531: Engineering Optimization I Lecture 3: Linear Programming, Continued Prof. John Gunnar Carlsson September 15, 2010 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 1 / 29
Administrivia Lecture slides 1, 2, 3 posted http://www.tc.umn.edu/~jcarlsso/syllabus.html PS 1 posted this evening Xi Chen's oce hours: Tuesdays 10:00-12:00, ME 1124, Table B Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 2 / 29
Today: Linear Programming, continued Linearization Mathematical preliminaries Simplex method Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 3 / 29
Recap A linear program (LP) is a mathematical optimization problem in which the objective function and all constraint functions are linear: minimize 2x 1 x 2 +4x 3 s.t. x 1 + x 2 + x 4 2 3x 2 x 3 = 5 x 3 + x 4 3 x 1 0 x 3 0 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 4 / 29
2-dimensional LPs If x R 2, it is easy to solve a linear program Consider the problem minimize x 1 x 2 s.t. x 1 + 2x 2 3 2x 1 + x 2 3 x 1, x 2 0 How do we solve this? Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 5 / 29
The graphical method Draw half-spaces corresponding to the constraints: Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 6 / 29
The graphical method Draw half-spaces corresponding to the constraints: Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 6 / 29
The graphical method Draw half-spaces corresponding to the constraints: Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 6 / 29
The graphical method Draw half-spaces corresponding to the constraints: Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 6 / 29
The graphical method Draw level sets of the objective function (they're lines orthogonal to c) Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 7 / 29
Active constraints We say a constraint a T x b is active at a point x if we have a T x = b In the previous example we had two active constraints: x 1 + 2x 2 = 3 and 2x 1 + x 2 = 3, while x 1, x 2 > 0 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 8 / 29
Facts about LP All LP problems fall into one of three classes: Problem is infeasible: the feasible region is empty Problem is unbounded: the feasible region is unbounded in the objective function direction Problem is feasible and bounded: There exists an optimal solution x There may be a unique optimal solution or multiple optimal solutions All optimal solutions are on a face of the feasible region There is always at least one corner optimizer if the face has a corner If a corner point is not worse than its neighboring corners, then it is optimal Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 9 / 29
Linearizing a problem LP can also be used to model certain non-linear problems A convex function is a function f ( ) : R n R satisfying f (λx + (1 λ) y) λf (x) + (1 λ) f (y) for all x, y R n and λ [0, 1] (bowl-shaped) A concave function is a function f ( ) : R n R satisfying f (λx + (1 λ) y) λf (x) + (1 λ) f (y) for all x, y R n and λ [0, 1] (hill-shaped) We claim that any piecewise linear convex function can be minimized by solving an LP Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 10 / 29
Linearizing a problem Consider the function f ( ) dened by { f (x) = max i=1,...,m c T i x + d i } It is easy to prove that this function is convex We can solve the problem minimize f (x) s.t. Ax b by solving an LP Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 11 / 29
Linearizing a problem The LP is minimize z s.t. z c T i Ax b x + d i i {1,..., m} Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 12 / 29
Absolute values Problems involving absolute values can be handled as well; consider minimize n i=1 c i x i s.t. Ax b Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 13 / 29
Absolute values The LP is minimize n i=1 c i z i s.t. z i x i i {1,..., n} z i x i i {1,..., n} Ax b Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 14 / 29
Data tting Consider the unconstrained problem of minimizing the largest residual minimize max b i a T i x i i where a i and b i are given, for i {1,..., m} Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 15 / 29
Data tting Consider the unconstrained problem of minimizing the largest residual minimize max b i a T i x i i where a i and b i are given, for i {1,..., m} The LP is minimize z s.t. z b i a T i x i ) z (b i a Ti x i Note that we can impose additional linear constraints on x, say C x d We could even impose something like n i=1 x i q! Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 15 / 29
Polyhedra Denition A polyhedron is a set that can be described in the form {x R n : Ax b} where A is an m n matrix and b R n. By the equivalence of linear programs, we know that a set of the form is also a polyhedron {x R n : Ax = b, x 0} Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 16 / 29
Boundedness Denition A set S R n is bounded if there exists a constant K such that S is contained in a ball of radius K. Note: a linear program can be bounded, but have an unbounded feasible set! However, if a linear program has a bounded feasible set, it must be bounded Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 17 / 29
Hyperplanes and half spaces Let a R n be nonzero and let b be a scalar. Denition The set { x : a T x = b} is called a hyperplane. Denition The set { x : a T x b} is called a half-space. Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 18 / 29
Convex sets Denition A set S R n is convex if, for any x, y S and any λ [0, 1], we have λx + (1 λ) y S. Intuitively, this means that the line segment between two points in the set must also lie in the set Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 19 / 29
Facts about convex sets The intersection of convex sets is convex Every polyhedron is convex The sub-level set of a convex function is convex (converse?) Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 20 / 29
Linear independence We say a set of vectors x 1,... x k is linearly dependent if there exist real numbers a 1,..., a k, not all of which are zero, such that a 1 x 1 + + a k x k = 0 If no such real numbers exist, we say that x 1,..., x k is linearly independent If x 1,..., xn R n are linearly independent, then the matrix (x 1,..., xn) is invertible Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 21 / 29
Gradient The gradient of a function f (x) : R n R, is the vector f dened by f / x 1 f =. f / x n The gradient vector always points in the direction that the function is increasing the fastest The gradient of a linear function c T x is c Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 22 / 29
Extreme points Denition Let P be a polyhedron. A point x P is said to be an extreme point of P if we cannot nd two vectors y, z P, both dierent from x, and a scalar λ [0, 1], such that x = λy + (1 λ) z In other words, x does not lie on the line segment between two other points in P Denition Let P be a polyhedron. A point x P is said to be a vertex of P if there exists some c such that c T x < c T y for all y P not equal to x. The vector c is said to dene a supporting hyperplane to P at x. Vertices and extreme points are the same thing! Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 23 / 29
Algorithmic interpretation of extreme points We gave two geometric interpretations of vertices/extreme points However, this does not suggest how a computer might nd an extreme point How can a computer recognize a vertex? How can we make a computer tell that two corners are neighboring? How can we make a computer terminate and declare optimality? How can we recognize vertices/extreme points directly from the polyhedron {x : Ax = b, x 0}? Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 24 / 29
Basic feasible solution Consider a polyhedron dened by {x : Ax = b, x 0} where A is an m n matrix and b R n. What describes the extreme points? Select m linearly independent columns, denoted by the indices B, from A, and solve A B x B = b Then, set all other variables x N to 0 If all entries x B 0, then x is called a basic feasible solution (BFS) A basic feasible solution is the same thing as a corner or extreme point this is an algebraic description, rather than a geometric description Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 25 / 29
Example Consider the polyhedron {x : Ax = b, x 0}, where ( ) ( ) 5 6 7 5 17 A = ; b = 4 6 9 8 16 If we take B = {1, 2}, then we solve ( ) 5 6 4 6 x B = ( 17 16 and nd that x B = (1; 2). Thus the BFS is x = (1; 2; 0; 0) However, if we take B = {2, 3}, then we solve ( ) ( ) 6 7 17 6 9 x B = 16 and nd that x B = (3.42, 0.5), which is not a BFS ) Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 26 / 29
Example We can enumerate all of the vertices of the polyhedron {x : Ax = b, x 0}, where A = by choosing all subsets B ( 5 6 7 5 4 6 9 8 ) ( 17 ; b = 16 ) B {1, 2} {1, 3} {1, 4} {2, 3} {2, 4} {3, 4} xb (1; 2) (2.41; 0.70) (2.8; 0.6) (3.42; 0.5) (3.11; 0.33) (5.09; 3.73) BFS? Y Y Y N N N Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 27 / 29
The simplex method One way to solve a linear program is clearly to write out all of the BFS's, although that would clearly be slow A better strategy is to start at a BFS, and move to a better neighboring BFS if one is available If no neighboring BFS exists, we're done! How to identify a neighboring BFS? Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 28 / 29
Neighboring basic solutions Two basic solutions are neighboring or adjacent if they dier by exactly one basic (or nonbasic) variable A basic feasible solution is optimal if no better neighboring feasible solution exists How to check if this is true? Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 15, 2010 29 / 29