CS 473: Algorithms. Ruta Mehta. Spring University of Illinois, Urbana-Champaign. Ruta (UIUC) CS473 1 Spring / 50

Transcription

1 CS 473: Algorithms Ruta Mehta University of Illinois, Urbana-Champaign Spring 2018 Ruta (UIUC) CS473 1 Spring / 50

2 CS 473: Algorithms, Spring 2018 Introduction to Linear Programming Lecture 18 March 26, 2018 Some of the slides are courtesy Prof. Chekuri Ruta (UIUC) CS473 2 Spring / 50

3 Part I Introduction to Linear Programming Ruta (UIUC) CS473 3 Spring / 50

4 Today... Linear Programming and Standard Formulation Geometry Vertex Solution Simplex Method Ruta (UIUC) CS473 4 Spring / 50

5 A Factory Example Problem Your factory can produce Laptop and iphone using Copper. 1 One ton of Copper one Laptop 2 One ton of Copper one iphone 3 We have 200 tons of Copper. 4 Laptop can be sold for $1 and iphone for $6. Ruta (UIUC) CS473 5 Spring / 50

6 A Factory Example Problem Your factory can produce Laptop and iphone using Copper. 1 One ton of Copper one Laptop 2 One ton of Copper one iphone 3 We have 200 tons of Copper. 4 Laptop can be sold for $1 and iphone for $6. How many units of Laptop and iphone should your factory manufacture to maximize profit? Ruta (UIUC) CS473 5 Spring / 50

7 A Factory Example Problem Your factory can produce Laptop and iphone using Copper. 1 One ton of Copper one Laptop 2 One ton of Copper one iphone 3 We have 200 tons of Copper. 4 Laptop can be sold for $1 and iphone for $6. How many units of Laptop and iphone should your factory manufacture to maximize profit? Solution: Ruta (UIUC) CS473 5 Spring / 50

8 A Factory Example Problem Your factory can produce Laptop and iphone using Copper. 1 One ton of Copper one Laptop 2 One ton of Copper one iphone 3 We have 200 tons of Copper. 4 Laptop can be sold for $1 and iphone for $6. How many units of Laptop and iphone should your factory manufacture to maximize profit? Solution: manufacture only iphone Ruta (UIUC) CS473 5 Spring / 50

9 A Factory Example Problem Your factory can produce Laptop and iphone using resources C, B, A. 1 One unit of A and C each One Laptop 2 One unit of B and C each One iphone Ruta (UIUC) CS473 6 Spring / 50

10 A Factory Example Problem Your factory can produce Laptop and iphone using resources C, B, A. 1 One unit of A and C each One Laptop 2 One unit of B and C each One iphone 3 We have 200 units of A, 300 units of B, and 400 units of C. 4 Product Laptop can be sold for $1 and product iphone for $6. Ruta (UIUC) CS473 6 Spring / 50

11 A Factory Example Problem Your factory can produce Laptop and iphone using resources C, B, A. 1 One unit of A and C each One Laptop 2 One unit of B and C each One iphone 3 We have 200 units of A, 300 units of B, and 400 units of C. 4 Product Laptop can be sold for $1 and product iphone for $6. How many units of Laptop and iphone should your factory manufacture to maximize profit? Solution: Ruta (UIUC) CS473 6 Spring / 50

12 A Factory Example Problem Your factory can produce Laptop and iphone using resources C, B, A. 1 One unit of A and C each One Laptop 2 One unit of B and C each One iphone 3 We have 200 units of A, 300 units of B, and 400 units of C. 4 Product Laptop can be sold for $1 and product iphone for $6. How many units of Laptop and iphone should your factory manufacture to maximize profit? Solution: Formulate as a linear program. Ruta (UIUC) CS473 6 Spring / 50

13 A Factory Example Problem Can produce Laptop and iphone, using resources A, B, C. 1 A, C Laptop 2 B, C iphone 3 Have A: 200, B: 300, and C: Price of L: $1, and ip: $6. How many units to manufacture to max profit? Ruta (UIUC) CS473 7 Spring / 50

14 A Factory Example Problem Can produce Laptop and iphone, using resources A, B, C. 1 A, C Laptop Suppose x 1 units of Laptop and x 2 units of iphone. 2 B, C iphone 3 Have A: 200, B: 300, and C: Price of L: $1, and ip: $6. How many units to manufacture to max profit? Ruta (UIUC) CS473 7 Spring / 50

15 A Factory Example Problem Can produce Laptop and iphone, using resources A, B, C. 1 A, C Laptop 2 B, C iphone 3 Have A: 200, B: 300, and C: Price of L: $1, and ip: $6. How many units to manufacture to max profit? Suppose x 1 units of Laptop and x 2 units of iphone. max x 1 + 6x 2 s.t. x (A) x x 1 + x x 1 0 x 2 0 (B) (C) Ruta (UIUC) CS473 7 Spring / 50

16 Linear Programming Formulation Let us produce x 1 units of Laptop and x 2 units of iphone. Our profit can be computed by solving maximize x 1 + 6x 2 subject to x x x 1 + x x 1, x 2 0 Ruta (UIUC) CS473 8 Spring / 50

17 Linear Programming Formulation Let us produce x 1 units of Laptop and x 2 units of iphone. Our profit can be computed by solving maximize x 1 + 6x 2 subject to x x x 1 + x x 1, x 2 0 What is the solution? Ruta (UIUC) CS473 8 Spring / 50

18 Maximum Flow in Network s t Ruta (UIUC) CS473 9 Spring / 50

19 Maximum Flow in Network Need to compute values f s1, f s2,... f 25,... f 5t, f 6t such that s t Ruta (UIUC) CS473 9 Spring / 50

20 Maximum Flow in Network s Need to compute values f s1, f s2,... f 25,... f 5t, f 6t such that t f s1 15 f s2 5 f s3 10 f f 21 4 f 25 8 f 32 4 f f 36 9 f 42 6 f 4t 10 f f 5t 10 f f 6t 10 Ruta (UIUC) CS473 9 Spring / 50

21 Maximum Flow in Network s Need to compute values f s1, f s2,... f 25,... f 5t, f 6t such that t and f s1 15 f s2 5 f s3 10 f f 21 4 f 25 8 f 32 4 f f 36 9 f 42 6 f 4t 10 f f 5t 10 f f 6t 10 f s1 + f 21 = f 14 f s2 + f 32 = f 21 + f 25 f s3 = f 32 + f 35 + f 36 f 14 + f 54 = f 42 + f 4t f 25 + f 35 + f 65 = f 54 + f 5t f 36 = f 65 + f 6t Ruta (UIUC) CS473 9 Spring / 50

22 Maximum Flow in Network s Need to compute values f s1, f s2,... f 25,... f 5t, f 6t such that t and f s1 15 f s2 5 f s3 10 f f 21 4 f 25 8 f 32 4 f f 36 9 f 42 6 f 4t 10 f f 5t 10 f f 6t 10 f s1 + f 21 = f 14 f s2 + f 32 = f 21 + f 25 f s3 = f 32 + f 35 + f 36 f 14 + f 54 = f 42 + f 4t f 25 + f 35 + f 65 = f 54 + f 5t f 36 = f 65 + f 6t f s1 0 f s2 0 f s3 0 f 4t 0 f 5t 0 f 6t 0 Ruta (UIUC) CS473 9 Spring / 50

23 Maximum Flow in Network s Need to compute values f s1, f s2,... f 25,... f 5t, f 6t such that t and f s1 15 f s2 5 f s3 10 f f 21 4 f 25 8 f 32 4 f f 36 9 f 42 6 f 4t 10 f f 5t 10 f f 6t 10 f s1 + f 21 = f 14 f s2 + f 32 = f 21 + f 25 f s3 = f 32 + f 35 + f 36 f 14 + f 54 = f 42 + f 4t f 25 + f 35 + f 65 = f 54 + f 5t f 36 = f 65 + f 6t f s1 0 f s2 0 f s3 0 f 4t 0 f 5t 0 f 6t 0 maximize: f s1 + f s2 + f s3. Ruta (UIUC) CS473 9 Spring / 50

24 Maximum Flow as a Linear Program For a general flow network G = (V, E) with capacities c e on edge e E, we have variables f e indicating flow on edge e Maximize e out of s f e subject to f e c e for each e E f e f e = 0 v V \ {s, t} e out of v e into v f e 0 for each e E. Ruta (UIUC) CS Spring / 50

25 Maximum Flow as a Linear Program For a general flow network G = (V, E) with capacities c e on edge e E, we have variables f e indicating flow on edge e Maximize e out of s f e subject to f e c e for each e E f e f e = 0 v V \ {s, t} e out of v e into v f e 0 for each e E. Number of variables: m, one for each edge. Number of constraints: m + n 2 + m. Ruta (UIUC) CS Spring / 50

26 Minimum Cost Flow with Lower Bounds... as a Linear Program For a general flow network G = (V, E) with capacities c e, lower bounds l e, and costs w e, we have variables f e indicating flow on edge e. Suppose we want a min-cost flow of value at least F. Minimize w e f e subject to e E e out of s f e F f e c e f e l e for each e E f e f e = 0 for each v V {s, t} e out of v e into v f e 0 for each e E. Ruta (UIUC) CS Spring / 50

27 Minimum Cost Flow with Lower Bounds... as a Linear Program For a general flow network G = (V, E) with capacities c e, lower bounds l e, and costs w e, we have variables f e indicating flow on edge e. Suppose we want a min-cost flow of value at least F. Minimize w e f e subject to e E e out of s f e F f e c e f e l e for each e E f e f e = 0 for each v V {s, t} e out of v e into v f e 0 for each e E. Number of variables: m, one for each edge Number of constraints: 1 + m + m + n 2 + m = 3m + n 1. Ruta (UIUC) CS Spring / 50

28 Linear Programs Problem Find a vector x R d that maximize/minimize subject to d j=1 c jx j d j=1 a ijx j b i d j=1 a ijx j = b i d j=1 a ijx j b i for i = 1... p for i = p q for i = q n Ruta (UIUC) CS Spring / 50

29 Linear Programs Problem Find a vector x R d that maximize/minimize subject to d j=1 c jx j d j=1 a ijx j b i d j=1 a ijx j = b i d j=1 a ijx j b i for i = 1... p for i = p q for i = q n Input is matrix A = (a ij ) R n d, column vector b = (b i ) R n, and row vector c = (c j ) R d Ruta (UIUC) CS Spring / 50

30 Canonical Form of Linear Programs Canonical Form A linear program is in canonical form if it has the following structure maximize subject to d j=1 c jx j d j=1 a ijx j b i for i = 1... n Ruta (UIUC) CS Spring / 50

31 Canonical Form of Linear Programs Canonical Form A linear program is in canonical form if it has the following structure maximize subject to d j=1 c jx j d j=1 a ijx j b i for i = 1... n Conversion to Canonical Form 1 Replace j a ijx j = b i by a ij x j b i and j j a ij x j b i 2 Replace j a ijx j b i by j a ijx j b i Ruta (UIUC) CS Spring / 50

32 Matrix Representation of Linear Programs A linear program in canonical form can be written as maximize subject to c x Ax b where A = (a ij ) R n d, column vector b = (b i ) R n, row vector c = (c j ) R d, and column vector x = (x j ) R d 1 Number of variable is d 2 Number of constraints is n Ruta (UIUC) CS Spring / 50

33 Other Standard Forms for Linear Programs maximize subject to c x Ax b x 0 minimize subject to c x Ax b x 0 minimize subject to c x Ax = b x 0 Ruta (UIUC) CS Spring / 50

34 Linear Programming: A History 1 First formal application to problems in economics by Leonid Kantorovich in the 1930s 1 However, work was ignored behind the Iron Curtain and unknown in the West Ruta (UIUC) CS Spring / 50

35 Linear Programming: A History 1 First formal application to problems in economics by Leonid Kantorovich in the 1930s 1 However, work was ignored behind the Iron Curtain and unknown in the West 2 Rediscovered by Tjalling Koopmans in the 1940s, along with applications to economics Ruta (UIUC) CS Spring / 50

36 Linear Programming: A History 1 First formal application to problems in economics by Leonid Kantorovich in the 1930s 1 However, work was ignored behind the Iron Curtain and unknown in the West 2 Rediscovered by Tjalling Koopmans in the 1940s, along with applications to economics 3 First algorithm (Simplex) to solve linear programs by George Dantzig in 1947 Ruta (UIUC) CS Spring / 50

37 Linear Programming: A History 1 First formal application to problems in economics by Leonid Kantorovich in the 1930s 1 However, work was ignored behind the Iron Curtain and unknown in the West 2 Rediscovered by Tjalling Koopmans in the 1940s, along with applications to economics 3 First algorithm (Simplex) to solve linear programs by George Dantzig in Kantorovich and Koopmans receive Nobel Prize for economics in 1975 ; Dantzig, however, was ignored 1 Koopmans contemplated refusing the Nobel Prize to protest Dantzig s exclusion, but Kantorovich saw it as a vindication for using mathematics in economics, which had been written off as a means for apologists of capitalism Ruta (UIUC) CS Spring / 50

38 Back to the Factory example Produce x 1 units of product 1 and x 2 units of product 2. Our profit can be computed by solving maximize x 1 + 6x 2 subject to x x x 1 + x x 1, x 2 0 Ruta (UIUC) CS Spring / 50

39 Back to the Factory example Produce x 1 units of product 1 and x 2 units of product 2. Our profit can be computed by solving maximize x 1 + 6x 2 subject to x x x 1 + x x 1, x 2 0 What is the solution? Ruta (UIUC) CS Spring / 50

40 Solving the Factory Example x Feasible values of x 1 and x 2 are shaded region. 200 x 1 maximize x 1 + 6x 2 subject to x x x 1 + x x 1, x 2 0 Ruta (UIUC) CS Spring / 50

41 Solving the Factory Example x Feasible values of x 1 and x 2 are shaded region. 2 Objective (Cost) function is a direction the line represents all points with same value of the function 200 x 1 maximize x 1 + 6x 2 subject to x x x 1 + x x 1, x 2 0 Ruta (UIUC) CS Spring / 50

42 Solving the Factory Example x x 1 1 Feasible values of x 1 and x 2 are shaded region. 2 Objective (Cost) function is a direction the line represents all points with same value of the function; moving the line until it just leaves the feasible region, gives optimal values. maximize x 1 + 6x 2 subject to x x x 1 + x x 1, x 2 0 Ruta (UIUC) CS Spring / 50

43 Linear Programming in 2-d 1 Each constraint a half plane 2 Feasible region is intersection of finitely many half planes it forms a polygon. 3 For a fixed value of objective function, we get a line. Parallel lines correspond to different values for objective function. 4 Optimum achieved when objective function line just leaves the feasible region Ruta (UIUC) CS Spring / 50

44 An S. Dasgupta, Example C.H. Papadimitriou, in 3-d and U.V. Vazirani 227 Figure 7.12 A polyhedron defined by seven inequalities. x2 5 B 4 2 A 3 1 C 6 7 x1 max x 1 + 6x x 3 x x x 1 + x 2 + x x 2 + 3x x 1 0 x 2 0 x x3 7.6 The simplex algorithmpolytope The extraordinary power and expressiveness of linear programs would be little consolation if we did not have a way to solve them efficiently. This Figure is thefrom role ofdasgupta the simplex algorithm. etal book. At a high level, the simplex algorithm takes a set of linear inequalities and a linear objec- Ruta (UIUC) CS Spring / 50

45 Part II Simple Algorithm Ruta (UIUC) CS Spring / 50

46 Factory Example: Alternate View Original Problem Recall we have, maximize x 1 + 6x 2 subject to x x x 1 + x x 1, x 2 0 Ruta (UIUC) CS Spring / 50

47 Factory Example: Alternate View Original Problem Recall we have, maximize x 1 + 6x 2 subject to x x x 1 + x x 1, x 2 0 Transformation Consider new variable z 1 and z 2, such that z 1 = x 1 + 6x 2 and z 2 = x 2. Then x 1 = z 1 6z 2. In terms of the new variables we have maximize z 1 subject to z 1 6z z z 1 5z z 1 6z 2 0 z 2 0 Ruta (UIUC) CS Spring / 50

48 Transformed Picture Feasible region rotated, and optimal value at the right-most point on polygon Ruta (UIUC) CS Spring / 50

49 Observations about the Transformation Observations 1 Linear program can always be transformed to get a linear program where the optimal value is achieved at the point in the feasible region with highest x-coordinate 2 Optimum value attained at a vertex of the polygon 3 Since feasible region is convex, and objective function linear, every local optimum is a global optimum Ruta (UIUC) CS Spring / 50

50 A Simple Algorithm in 2-d 1 optimum solution is at a vertex of the feasible region 2 a vertex is defined by the intersection of two lines (constraints) Ruta (UIUC) CS Spring / 50

51 A Simple Algorithm in 2-d 1 optimum solution is at a vertex of the feasible region 2 a vertex is defined by the intersection of two lines (constraints) Algorithm: 1 find all intersections between the n lines at most n 2 points 2 for each intersection point p = (p 1, p 2 ) 1 check if p is in feasible region (how?) 2 if p is feasible evaluate objective function at p: val(p) = c 1 p 1 + c 2 p 2 3 Output the feasible point with the largest value Ruta (UIUC) CS Spring / 50

52 A Simple Algorithm in 2-d 1 optimum solution is at a vertex of the feasible region 2 a vertex is defined by the intersection of two lines (constraints) Algorithm: 1 find all intersections between the n lines at most n 2 points 2 for each intersection point p = (p 1, p 2 ) 1 check if p is in feasible region (how?) 2 if p is feasible evaluate objective function at p: val(p) = c 1 p 1 + c 2 p 2 3 Output the feasible point with the largest value Running time: O(n 3 ). Ruta (UIUC) CS Spring / 50

53 Geometry in d-dimentsion maximize subject to d j=1 c jx j d j=1 a ijx j b i for i = 1... n Q: The set of points defined by a linear constraint {x R d 1 convex 2 non-convex d a ij x j b i } is, j=1 Ruta (UIUC) CS Spring / 50

54 Geometry in d-dimentsion maximize subject to d j=1 c jx j d j=1 a ijx j b i for i = 1... n Q: The set of points defined by a linear constraint {x R d 1 convex 2 non-convex This is also called a halfspace. d a ij x j b i } is, j=1 Ruta (UIUC) CS Spring / 50

55 Geometry in d-dimentsion maximize subject to d j=1 c jx j d j=1 a ijx j b i for i = 1... n Q: Intersection of a finitely many convex sets is, 1 convex 2 non-convex Ruta (UIUC) CS Spring / 50

56 Geometry in d-dimentsion maximize subject to d j=1 c jx j d j=1 a ijx j b i for i = 1... n Q: Intersection of a finitely many convex sets is, 1 convex 2 non-convex Thus feasible set, {x d j=1 a ijx j b i for i = 1... n}, is convex. Defines a polytope. Ruta (UIUC) CS Spring / 50

57 Geometry in d-dimentsion maximize subject to d j=1 c jx j d j=1 a ijx j b i for i = 1... n Caratheodory Theorem. Every point x in a d-dimensional polytope can be written as a convex combination of (d + 1) vertices Ruta (UIUC) CS Spring / 50

58 Geometry in d-dimentsion maximize subject to d j=1 c jx j d j=1 a ijx j b i for i = 1... n Caratheodory Theorem. Every point x in a d-dimensional polytope can be written as a convex combination of (d + 1) vertices. Q: If x is a convex combination of vertices v 1,..., v k, then for a constant vector c which of the following holds 1 (c x) max k i =1 (c v i ) 2 (c x) max k i =1 (c v i ) Ruta (UIUC) CS Spring / 50

59 Geometry in d-dimentsion maximize subject to d j=1 c jx j d j=1 a ijx j b i for i = 1... n Caratheodory Theorem. Every point x in a d-dimensional polytope can be written as a convex combination of (d + 1) vertices. Q: If x is a convex combination of vertices v 1,..., v k, then for a constant vector c which of the following holds 1 (c x) max k i =1 (c v i ) 2 (c x) max k i =1 (c v i ) There exists a vertex solution. Ruta (UIUC) CS Spring / 50

60 Geometry in d-dimentsion maximize subject to d j=1 c jx j d j=1 a ijx j b i for i = 1... n Caratheodory Theorem. Every point x in a d-dimensional polytope can be written as a convex combination of (d + 1) vertices. If x is a convex combination of vertices v 1,..., v k, then 1 min k i =1 (c v i ) (c x) max k i =1 (c v i ) There exists a vertex solution. Ruta (UIUC) CS Spring / 50

61 Linear Programming in d-dimensions (Summary) maximize subject to d j=1 c jx j d j=1 a ijx j b i for i = 1... n 1 Each linear constraint defines a halfspace Convex set Ruta (UIUC) CS Spring / 50

62 Linear Programming in d-dimensions (Summary) maximize subject to d j=1 c jx j d j=1 a ijx j b i for i = 1... n 1 Each linear constraint defines a halfspace Convex set. 2 Feasible region is an intersection of halfspaces Convex polytope Ruta (UIUC) CS Spring / 50

63 Linear Programming in d-dimensions (Summary) maximize subject to d j=1 c jx j d j=1 a ijx j b i for i = 1... n 1 Each linear constraint defines a halfspace Convex set. 2 Feasible region is an intersection of halfspaces Convex polytope. 3 Optimal value attained at a vertex of the polyhedron. Using the Caratheodory Theorem. (Or the transformation) Ruta (UIUC) CS Spring / 50

64 Linear Programming in d-dimensions (Summary) maximize subject to d j=1 c jx j d j=1 a ijx j b i for i = 1... n 1 Each linear constraint defines a halfspace Convex set. 2 Feasible region is an intersection of halfspaces Convex polytope. 3 Optimal value attained at a vertex of the polyhedron. Using the Caratheodory Theorem. (Or the transformation) 4 Tight inequality d j=1 a ij x j = b i defines hyperplane of (d 1) dim. 5 A vertex is defined by intersection of d hyperplanes. Solution of Âx = ˆb, where  is d d.  has non-zero determinant linear independence Ruta (UIUC) CS Spring / 50

65 Simple Algorithm in d Dimensions Real problem: d-dimensions, n-constraints Ruta (UIUC) CS Spring / 50

66 Simple Algorithm in d Dimensions Real problem: d-dimensions, n-constraints 1 optimum solution is at a vertex of the feasible region 2 a vertex is defined by the intersection of d hyperplanes 3 number of vertices can be Ruta (UIUC) CS Spring / 50

67 Simple Algorithm in d Dimensions Real problem: d-dimensions, n-constraints 1 optimum solution is at a vertex of the feasible region 2 a vertex is defined by the intersection of d hyperplanes 3 number of vertices can be Ω(n d ) Running time: O(dn d+1 ) which is not polynomial since problem size is at least nd. Also not practical. How do we find the intersection point of d hyperplanes in R d? Ruta (UIUC) CS Spring / 50

68 Simple Algorithm in d Dimensions Real problem: d-dimensions, n-constraints 1 optimum solution is at a vertex of the feasible region 2 a vertex is defined by the intersection of d hyperplanes 3 number of vertices can be Ω(n d ) Running time: O(dn d+1 ) which is not polynomial since problem size is at least nd. Also not practical. How do we find the intersection point of d hyperplanes in R d? Using Gaussian elimination to solve Âx = ˆb where  is a d d matrix and ˆb is a d 1 matrix. Ruta (UIUC) CS Spring / 50

69 Simplex Algorithm Simplex: Vertex hoping algorithm Ruta (UIUC) CS Spring / 50

70 Simplex Algorithm Simplex: Vertex hoping algorithm Moves from a vertex to its neighboring vertex Ruta (UIUC) CS Spring / 50

71 Simplex Algorithm Simplex: Vertex hoping algorithm Moves from a vertex to its neighboring vertex Questions Which neighbor to move to? When to stop? How much time does it take? Ruta (UIUC) CS Spring / 50

72 Observations For Simplex Suppose we are at a non-optimal vertex ˆx = (ˆx 1,..., ˆx d ) and optimal is x = (x 1,..., x d ), then c x > c ˆx. Ruta (UIUC) CS Spring / 50

73 Observations For Simplex Suppose we are at a non-optimal vertex ˆx = (ˆx 1,..., ˆx d ) and optimal is x = (x 1,..., x d ), then c x > c ˆx. How does (c x) change as we move from ˆx to x on the line joining the two? Ruta (UIUC) CS Spring / 50

74 Observations For Simplex Suppose we are at a non-optimal vertex ˆx = (ˆx 1,..., ˆx d ) and optimal is x = (x 1,..., x d ), then c x > c ˆx. How does (c x) change as we move from ˆx to x on the line joining the two? Strictly increases! Ruta (UIUC) CS Spring / 50

75 Observations For Simplex Suppose we are at a non-optimal vertex ˆx = (ˆx 1,..., ˆx d ) and optimal is x = (x 1,..., x d ), then c x > c ˆx. How does (c x) change as we move from ˆx to x on the line joining the two? Strictly increases! d = x ˆx is the direction from ˆx to x. x = ˆx + δd. As δ goes from 0 to 1, x moves from ˆx to x. Ruta (UIUC) CS Spring / 50

76 Observations For Simplex Suppose we are at a non-optimal vertex ˆx = (ˆx 1,..., ˆx d ) and optimal is x = (x 1,..., x d ), then c x > c ˆx. How does (c x) change as we move from ˆx to x on the line joining the two? Strictly increases! d = x ˆx is the direction from ˆx to x. x = ˆx + δd. As δ goes from 0 to 1, x moves from ˆx to x. (c d) = (c x ) (c ˆx) > 0. Ruta (UIUC) CS Spring / 50

77 Observations For Simplex Suppose we are at a non-optimal vertex ˆx = (ˆx 1,..., ˆx d ) and optimal is x = (x 1,..., x d ), then c x > c ˆx. How does (c x) change as we move from ˆx to x on the line joining the two? Strictly increases! d = x ˆx is the direction from ˆx to x. x = ˆx + δd. As δ goes from 0 to 1, x moves from ˆx to x. (c d) = (c x ) (c ˆx) > 0. c x = c ˆx + δ(c d). Strictly increasing with δ! Due to convexity, all of these are feasible points. Ruta (UIUC) CS Spring / 50

78 Cone Definition Given a set of vectors D = {d 1,..., d k }, the cone spanned by them is just their positive linear combinations, i.e., cone(d) = {d d = k λ i d i, where λ i 0, i} i =1 Ruta (UIUC) CS Spring / 50

79 Cone (Contd.) Lemma If d cone(d) and (c d) > 0, then there exists d i such that (c d i ) > 0. Proof. To the contrary suppose (c d i ) 0, i k. Since d is a positive linear combination of d i s, A contradiction! (c d) = (c k i =1 λ i d i ) = k i =1 λ i (c d i ) 0 Ruta (UIUC) CS Spring / 50

80 Improving Direction Implies Improving Neighbor Let z 1,..., z k be the neighboring vertices of ˆx. And let d i = z i ˆx be the direction from ˆx to z i. Lemma Any feasible direction of movement d from ˆx is in the cone({d 1,..., d k }). Ruta (UIUC) CS Spring / 50

81 Observations For Simplex Suppose we are at a non-optimal vertex ˆx = (ˆx 1,..., ˆx d ) and optimal is x = (x 1,..., x d ), then c x > c ˆx. d = x ˆx is the direction from ˆx to x. (c d) = (c x ) (c ˆx) > 0. Ruta (UIUC) CS Spring / 50

82 Observations For Simplex Suppose we are at a non-optimal vertex ˆx = (ˆx 1,..., ˆx d ) and optimal is x = (x 1,..., x d ), then c x > c ˆx. d = x ˆx is the direction from ˆx to x. (c d) = (c x ) (c ˆx) > 0. Let d i be the direction towards neighbor z i. d Cone({d 1,..., d k }) d i, (c d i ) > 0. Ruta (UIUC) CS Spring / 50

83 Observations For Simplex Suppose we are at a non-optimal vertex ˆx = (ˆx 1,..., ˆx d ) and optimal is x = (x 1,..., x d ), then c x > c ˆx. d = x ˆx is the direction from ˆx to x. (c d) = (c x ) (c ˆx) > 0. Let d i be the direction towards neighbor z i. d Cone({d 1,..., d k }) d i, (c d i ) > 0. Theorem If vertex ˆx is not optimal then it has a neighbor where the objective value (c x) improves. Ruta (UIUC) CS Spring / 50

84 How Many Neighbors a Vertex Has? Geometric view... A R n d (n > d), b R n, the constraints are: Ax b Faces n constraints/inequalities. Each defines a hyperplane. Vertex: 0-dimensional face. Edge: 1D face.... Hyperplane: (d 1)D face. Ruta (UIUC) CS Spring / 50

85 How Many Neighbors a Vertex Has? Geometric view... A R n d (n > d), b R n, the constraints are: Ax b Faces n constraints/inequalities. Each defines a hyperplane. Vertex: 0-dimensional face. Edge: 1D face.... Hyperplane: (d 1)D face. r linearly independent hyperplanes forms d r dimensional face. Ruta (UIUC) CS Spring / 50

86 How Many Neighbors a Vertex Has? Geometric view... A R n d (n > d), b R n, the constraints are: Ax b Faces n constraints/inequalities. Each defines a hyperplane. Vertex: 0-dimensional face. Edge: 1D face.... Hyperplane: (d 1)D face. r linearly independent hyperplanes forms d r dimensional face. Vertices being of 0D, d L.I. hyperplanes form a vertex. Ruta (UIUC) CS Spring / 50

87 How Many Neighbors a Vertex Has? Geometric view... A R n d (n > d), b R n, the constraints are: Ax b Faces n constraints/inequalities. Each defines a hyperplane. Vertex: 0-dimensional face. Edge: 1D face.... Hyperplane: (d 1)D face. r linearly independent hyperplanes forms d r dimensional face. Vertices being of 0D, d L.I. hyperplanes form a vertex. In 2-dimension (d = 2) x x 1 Ruta (UIUC) CS Spring / 50

88 How Many Neighbors a Vertex Has? Geometric view... A R n d (n > d), b R n, the constraints are: Ax b In 3-dimension (d = 3) Faces n constraints/inequalities. Each defines a hyperplane. Vertex: 0-dimensional face. Edge: 1D face.... Hyperplane: (d 1)D face. r linearly independent hyperplanes forms d r dimensional face. Vertices being of 0D, d L.I. hyperplanes form a vertex image source: webpage of Prof. Forbes W. Lewis Ruta (UIUC) CS Spring / 50

89 How Many Neighbors a Vertex Has? Geometry view... One neighbor per tight hyperplane. Therefore typically d. Suppose x is a neighbor of ˆx, then on the edge joining is defined by (d 1) hyperplanes. hx and x also shares these d 1 hyperplanes In addition one more hyperplane, say (Ax) i = b i, is tight at ˆx. Relaxing this at ˆx leads to x. Ruta (UIUC) CS Spring / 50

90 Simplex Algorithm Simplex: Vertex hoping algorithm Moves from a vertex to its neighboring vertex Questions + Answers Which neighbor to move to? One where objective value increases. Ruta (UIUC) CS Spring / 50

91 Simplex Algorithm Simplex: Vertex hoping algorithm Moves from a vertex to its neighboring vertex Questions + Answers Which neighbor to move to? One where objective value increases. When to stop? When no neighbor with better objective value. Ruta (UIUC) CS Spring / 50

92 Simplex Algorithm Simplex: Vertex hoping algorithm Moves from a vertex to its neighboring vertex Questions + Answers Which neighbor to move to? One where objective value increases. When to stop? When no neighbor with better objective value. How much time does it take? At most d neighbors to consider in each step. Ruta (UIUC) CS Spring / 50

93 Simplex in 2-d Simplex Algorithm 1 Start from some vertex of the feasible polygon. 2 Compare value of objective function at current vertex with the value at 2 neighboring vertices of polygon. 3 If neighboring vertex improves objective function, move to this vertex, and repeat step 2. 4 If no improving neighbor (local optimum), then stop. Ruta (UIUC) CS Spring / 50

94 Simplex in Higher Dimensions Simplex Algorithm 1 Start at a vertex of the polytope. 2 Compare value of objective function at each of the d neighbors. 3 Move to neighbor that improves objective function, and repeat step 2. 4 If no improving neighbor, then stop. Ruta (UIUC) CS Spring / 50

95 Simplex in Higher Dimensions Simplex Algorithm 1 Start at a vertex of the polytope. 2 Compare value of objective function at each of the d neighbors. 3 Move to neighbor that improves objective function, and repeat step 2. 4 If no improving neighbor, then stop. Simplex is a greedy local-improvement algorithm! Works because a local optimum is also a global optimum convexity of polyhedra. Ruta (UIUC) CS Spring / 50

96 Solving Linear Programming in Practice 1 Naïve implementation of Simplex algorithm can be very inefficient Ruta (UIUC) CS Spring / 50

97 Solving Linear Programming in Practice 1 Naïve implementation of Simplex algorithm can be very inefficient Exponential number of steps! Ruta (UIUC) CS Spring / 50

98 Solving Linear Programming in Practice 1 Naïve implementation of Simplex algorithm can be very inefficient 1 Choosing which neighbor to move to can significantly affect running time 2 Very efficient Simplex-based algorithms exist 3 Simplex algorithm takes exponential time in the worst case but works extremely well in practice with many improvements over the years 2 Non Simplex based methods like interior point methods work well for large problems. Ruta (UIUC) CS Spring / 50

99 Polynomial time Algorithm for Linear Programming Major open problem for many years: is there a polynomial time algorithm for linear programming? Ruta (UIUC) CS Spring / 50

100 Polynomial time Algorithm for Linear Programming Major open problem for many years: is there a polynomial time algorithm for linear programming? Leonid Khachiyan in 1979 gave the first polynomial time algorithm using the Ellipsoid method. 1 major theoretical advance 2 highly impractical algorithm, not used at all in practice 3 routinely used in theoretical proofs. Ruta (UIUC) CS Spring / 50

101 Polynomial time Algorithm for Linear Programming Major open problem for many years: is there a polynomial time algorithm for linear programming? Leonid Khachiyan in 1979 gave the first polynomial time algorithm using the Ellipsoid method. 1 major theoretical advance 2 highly impractical algorithm, not used at all in practice 3 routinely used in theoretical proofs. Narendra Karmarkar in 1984 developed another polynomial time algorithm, the interior point method. 1 very practical for some large problems and beats simplex 2 also revolutionized theory of interior point methods Ruta (UIUC) CS Spring / 50

102 Polynomial time Algorithm for Linear Programming Major open problem for many years: is there a polynomial time algorithm for linear programming? Leonid Khachiyan in 1979 gave the first polynomial time algorithm using the Ellipsoid method. 1 major theoretical advance 2 highly impractical algorithm, not used at all in practice 3 routinely used in theoretical proofs. Narendra Karmarkar in 1984 developed another polynomial time algorithm, the interior point method. 1 very practical for some large problems and beats simplex 2 also revolutionized theory of interior point methods Following interior point method success, Simplex has been improved enormously and is the method of choice. Ruta (UIUC) CS Spring / 50

103 Degeneracy 1 The linear program could be infeasible: No points satisfy the constraints. 2 The linear program could be unbounded: Polygon unbounded in the direction of the objective function. 3 More than d hyperplanes could be tight at a vertex, forming more than d neighbors. Ruta (UIUC) CS Spring / 50

104 Infeasibility: Example maximize x 1 + 6x 2 subject to x 1 2 x 2 1 x 1 + x 2 4 x 1, x 2 0 Infeasibility has to do only with constraints. Ruta (UIUC) CS Spring / 50

105 Infeasibility: Example maximize x 1 + 6x 2 subject to x 1 2 x 2 1 x 1 + x 2 4 x 1, x 2 0 Infeasibility has to do only with constraints. No starting vertex for Simplex. Ruta (UIUC) CS Spring / 50

106 Infeasibility: Example maximize x 1 + 6x 2 subject to x 1 2 x 2 1 x 1 + x 2 4 x 1, x 2 0 Infeasibility has to do only with constraints. No starting vertex for Simplex. How to detect this? Ruta (UIUC) CS Spring / 50

107 Unboundedness: Example maximize x 2 x 1 + x 2 2 x 1, x 2 0 Unboundedness depends on both constraints and the objective function. Ruta (UIUC) CS Spring / 50

108 Unboundedness: Example maximize x 2 x 1 + x 2 2 x 1, x 2 0 Unboundedness depends on both constraints and the objective function. If unbounded in the direction of objective function, then Simplex detects it. Ruta (UIUC) CS Spring / 50

109 Degeneracy and Cycling S. Dasgupta, C.H. Papadimitriou, and U.V. Vazirani 227 More than d inequalities tight at a vertex. Figure 7.12 A polyhedron defined by seven inequalities. x2 5 B 4 2 A 3 1 C 6 7 x1 max x 1 + 6x x 3 x x x 1 + x 2 + x x 2 + 3x x 1 0 x 2 0 x x3 7.6 The simplex algorithm The extraordinary power and expressiveness of linear programs would be little consolation if we did not have a way to solve them efficiently. This is the role of the simplex algorithm. At a high level, the simplex algorithm takes a set of linear inequalities and a linear objec- Ruta (UIUC) CS Spring / 50

110 Degeneracy and Cycling S. Dasgupta, C.H. Papadimitriou, and U.V. Vazirani 227 More than d inequalities tight at a vertex. Figure 7.12 A polyhedron defined by seven inequalities. x2 5 B 4 2 A 3 1 C 6 7 x1 max x 1 + 6x x 3 x x x 1 + x 2 + x x 2 + 3x x 1 0 x 2 0 x x3 Depending 7.6 The simplex on how Simplex algorithm is implemented, it may cycle at this vertex. The extraordinary power and expressiveness of linear programs would be little consolation if we did not have a way to solve them efficiently. This is the role of the simplex algorithm. At a high level, the simplex algorithm takes a set of linear inequalities and a linear objec- Ruta (UIUC) CS Spring / 50 We will see how in the next lecture.