# PRIMAL-DUAL INTERIOR POINT METHOD FOR LINEAR PROGRAMMING. 1. Introduction

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 PRIMAL-DUAL INTERIOR POINT METHOD FOR LINEAR PROGRAMMING KELLER VANDEBOGERT AND CHARLES LANNING 1. Introduction Interior point methods are, put simply, a technique of optimization where, given a problem with both equality and inequality constraints, reduces the problem to a sequence of equality constrained problems. One of these methods include barrier methods, in which a barrier function is considered. Intuitively, barrier methods convert a constrained problem to an unconstrained problem by selecting a specially constructed function B(x) which has the property that B vanishes at the constrained optimal value, and B(x) as x tends to the boundary of the feasible region. A sequence of equalities is then considered, in which the limiting value converges to the optimal value of the constrained value. In this paper we consider an approach known as the Primal-Dual Method for Linear Programming problems. There is a Primal-Dual method for nonlinear problems, but we shall only cover the case for linear problems here. 2. Lagrangians and Dual Problems The heart of optimization lies in problems of the following form: Date: September 3, Key words and phrases. Linear Programming, Interior Point Methods. 1

2 2 KELLER VANDEBOGERT AND CHARLES LANNING min f(x) (2.1) s.t g i (x) 0 h j (x) = 0 We refer to each g i as an inequality constraint, and each h j as an equality constraint. We then form the Lagrangian Function for the problem (2.1). m l (2.2) L(x, µ, λ) = f(x) µ i g i (x) λ j h j (x) i=1 j=1 The constants µ i and λ j are referred to as our Lagrange multipliers. In general, the Lagrangian is extremely important to the theory of optimization. Another question can be asked, in which we consider minimizing the Lagrangian with respect to our variable x. This curiosity will lead us in the direction of formulating the Dual Problem. Define: (2.3) q(µ, λ) = inf x L(x, µ, λ) We require that the infimum exists and is finite. We can refer to q as the dual function, and it is easy to see that this provides a lower bound on the solution of (2.1). Now, in order for the direction of our inequality constraints to remain the same, it is essential that all µ i 0. This is essentially a constraint, and with this, we can introduce the Dual Problem: (2.4) max q(µ, λ) s.t µ i 0

3 PRIMAL-DUAL INTERIOR POINT METHOD FOR LINEAR PROGRAMMING 3 It is natural to consider the difference f(x) q(µ, λ), which is referred to as the duality gap. As x x, where x denotes the solution of (2.1), it can be shown that the duality gap will tend to 0. We are now in a position to derive the Primal-Dual method. 3. Derivation of Primal-Dual Interior Point Method for Linear case Consider a linear programming problem of the following form: (3.1) min s.t c T x Ax = b x 0 Where b R m, c R n, and A R m n. We intend to construct the dual problem for this problem, known as the primal problem. We thus form the Lagrangian function: (3.2) L(x, y, s) = c T x y T (Ax b) + s T x Where y is the vector of our Lagrange multipliers corresponding to the equality constraints, and s 0 is the vector of inequality constraint multipliers. We now intend to rewrite (3.2) in order to deduce some properties of our dual problem. We have: L(x, y, s) = c T x y T (Ax b) s T x = c T x y T Ax + y T b + s T x = y T b + ( A T y + c s) T x

4 4 KELLER VANDEBOGERT AND CHARLES LANNING Where the last step uses the fact that y T A = (A T y) T. Here is where we use properties of linear functions and the assumption of our dual function in the previous section. If we examine inf x L(x, y, s), we see that we are looking for inf x ( A T y + c s) T x. However, we required that the dual function be bounded below. It is obvious that a linear function will be bounded below if and only if its slope is identically 0. Thus, we see that A T y + c s = 0. We have thus derived the corresponding dual problem: (3.3) max s.t b T y A T y + s = c s 0 We can now consider a barrier reformulation of our linear problem, which uses the fact that the solution set for (3.1) is precisely the same as the following: (3.4) n min c T x µ ln(x i ) s.t Ax = b i=1 x 0 This is easy to see purely by the properties of barrier reformulations. We see that as x tends to the boundary of our feasibly region, our barrier function will tend to infinity, effectively forcing our problem to remain constrained. If we consider the Lagrangian function for the above barrier reformulation, we easily deduce the following system:

5 PRIMAL-DUAL INTERIOR POINT METHOD FOR LINEAR PROGRAMMING 5 x L(x, y) = c µx 1 e A T y = 0 (3.5) y L(x, y) = b Ax = 0 x > 0 Where we let X denote the square matrix whose ith diagonal entry is just the ith entry of x. This is where we can introduce a slack vector s = µx 1 e, where e is an nx1 vector of 1 s, and reformulate (3.5) in the following form: A T y + s = c (3.6) Ax = b, x > 0 Xs = µe We can then immediately formulate the dual barrier problem as so: (3.7) n max b T y µ ln(s i ) s.t A T y = s i=1 s 0 Which yields the following system almost immediately after considering the Lagrangian: A T y + s = c, s > 0 (3.8) Ax = b, Xs = µe But then we immediately combine the conditions above from (3.4) and (3.8) and derive the following conditions, which are essential to constructing the Primal-Dual interior point method algorithm.

6 6 KELLER VANDEBOGERT AND CHARLES LANNING A T y + s = c, s > 0 (3.9) Ax = b, x > 0 Xs = µe We now consider the system (3.9), and intend to apply Newton s method for multidimensional optimization. We assume the reader is familiar with this method, and consider the following vector function F γ (x, y, s) which we have perturbed by a small arbitrary parameter γ: (3.10) F γ (x, y, s) = Ax b A T y + s c Xs γµe We now consider the gradient of our function. Each column represents the matrix derivative with respect to vectors x, y, and s, respectively. We have: (3.11) F γ (x, y, s) = A A T I S 0 X Of course the search directions for Newton s method are determined by solving the system F γ (x, y, s)d = F γ (x, y, s), where D is our direction method. We then have: (3.12) A A T I d x d y = Ax b A T y + s c S 0 X Xs γµe d s Now assume we are given initial iterates (x 0, y 0, s 0 ) which satisfy x 0, s 0 > 0. Then we can define the initial dual residuals as so:

7 PRIMAL-DUAL INTERIOR POINT METHOD FOR LINEAR PROGRAMMING 7 (3.13) r 0 P = b Ax 0 r 0 D = c A T y 0 s 0 This then quickly generalizes to the following iteration scheme, where (x k, y k, s k ) is the kth iterate: (3.14) A A T I d x rp k d y = r k S k 0 X k D X k s + γµ k e d s Of course we then require finding a step size for our next iterate. There is no perfect step size, but the general method is to follow a central path that converges to a solution. This is the biggest advantage of interior point methods, which do not require explicit calculation of the central path. Instead, we form a horn neighborhood, which is named because of the shape of the neighborhood as it follows the central path. See [1] for an explicit illustration. The horn neighborhood improves calculation efficiency and speed because our iterates need only stay within some distance of the central path, which tends to 0 as the method converges. Intuitively, this will of course converge to the optimal point, but with minimal computing power compared to explicit calculations of the central path. See [1] for additional theoretical considerations on this derivation. 4. The Primal-Dual Interior Point Algorithm: Simplified Version for Linear Programming There is a simplified algorithm for this method [1] which has the following explicit form. This can be found by means of backward substitution in the system (3.12).

8 8 KELLER VANDEBOGERT AND CHARLES LANNING (1) Initialization step: Choose β, γ (0, 1) and let ɛ > 0 be your tolerance. Choose x 0 = s 0 = e and y 0 = 0 (2) Set k=0 (3) We now begin the iteration scheme: Set: rp k = b Ax k r k D = c A T y k s k µ k = (xk ) T s k n (4) Check our termination: If r k P < ɛ, rk D < ɛ, and (xk ) T s k < ɛ, then we stop. (5) Compute our directions using the following: (4.1) Md y = r where M = A(S k ) 1 X k A T r = b + A(S k ) 1 (X k r k D γµ k e) We then find our other directions with the following: (4.2) d s = r k D A T d y (4.3) d x = x k + (S k ) 1 (γµ k e X k d s ) (6) We then calculate step size using the following: { (4.4) αp max = min x } i : (d x ) i < 0, i I (d x ) i { (4.5) αd max min s } i : (d s ) i < 0, i I (d s ) i

9 PRIMAL-DUAL INTERIOR POINT METHOD FOR LINEAR PROGRAMMING 9 We then choose our value for α max to be the minimum of (4.4) and (4.5), and the value α k is then chosen as: α k = min{1, θα max Where θ (0, 1). In practice we will often choose θ to be close to 1. In our code we chose θ =.95. (7) Update our iterates: x k+1 = x k + α k d x y k+1 = y k + α k d y s k+1 = s k + α k d s (8) Repeat the algorithm, with k = k + 1. In practice, the above algorithm is computationally efficient and fairly open to adjustments. Indeed, changes to our starting points may yield different paths to convergence, and changes to the constants may affect the amount of iterations required for desired precision. 5. Numerical Tests and Implementation We tested our code on various problems and obtained excellent results. On most problems, between 10 and 20 iterations are needed when our ɛ is 10 8, which will be our standard tolerance, as explained later. In these tests, we will encounter both minimum and maximum problems. Out code is designed to minimize a given objective function, but if we would maximize, then the algorithm simply multiplies all coefficients in the objective function by -1. Let s look at the following example.

10 10 KELLER VANDEBOGERT AND CHARLES LANNING max 2x 1 + 2x 2 (5.1) s.t. x 1 + x 2 3 x 1, x 2 0 This problem was solved just 8 iterations, with the optimal value found to be 6 at (x 1, x 2 ) = (1.5, 1.5). In fact, there are infinitely many optimal solutions lying on the line x 1 + x 2 = 3, however our method will always return the analytic center in these cases. Our solution can be seen in Figure 1. Figure 1. Problem 5.1. For a more interesting example, let s look at the following.

11 PRIMAL-DUAL INTERIOR POINT METHOD FOR LINEAR PROGRAMMING11 max 3x 1 + 5x 2 s.t.x 1 3 (5.2) 2x x 1 + 2x 2 18 x 1, x 2 0 Our algorithm solved this one in 11 iterations, with optimal value 36 found at (x 1, x 2 ) = (2, 6). If we change our initial point from (1, 1, 1, 1, 1) to others, the algorithm still reaches the solution in about 11 iterations (occasionally 10 or 12). These can be seen in Figure 2. Figure 2. Problem 5.2 using various initial points. If we let ɛ be 10 32, then we obtain a solution that is within of the solution of obtained when ɛ is the

12 12 KELLER VANDEBOGERT AND CHARLES LANNING usual 10 8, and our algorithm takes 42 iterations. Thus, an increase in ɛ is probably not worth the machine time unless accuracy is imperative. Our algorithm does not just handle constraints, though. When the constraint is =, our algorithm will not add a slack, and when it is, our algorithm will add a slack and set its coefficient in that constraint to -1, instead of 1. For example, consider the following. min 2x 1 + 3x 2 s.t. 0.5x x 2 4 (5.3) x 1 + 3x 2 20 x 1 + x 2 = 10 x 1, x 2 0 In this problem, we have all three types of constraints. Our code finds an optimal solution in just 15 iterations. See Figure 3. Note that in this example, our set of feasible points is not a region, and is instead a line segment, denoted by the thicker constraint line.

13 PRIMAL-DUAL INTERIOR POINT METHOD FOR LINEAR PROGRAMMING13 Figure 3. Problem More Numerical Tests We have chosen additional problems with various numbers of variables, and with different constraint types. Our tolerance in each is Problems 5.1, 5.2, and 5.3 will be included as well. (1) max 2x 1 + 2x 2 s.t. x 1 + x 2 3 x 1, x 2 0 Iterations: 8 Optimal Solution: x1 = x2 =

14 14 KELLER VANDEBOGERT AND CHARLES LANNING Optimal Value: (2) max 3x 1 + 5x 2 s.t. x 1 3 2x x 1 + 2x 2 18 (3) x 1, x 2 0 Iterations: 11 Optimal Solution: x1 = x2 = Optimal Value: min 2x 1 + 3x 2 s.t. 0.5x x 2 4 x 1 + 3x 2 20 x 1 + x 2 = 10 x 1, x 2 0 Iterations: 15 Optimal Solution:

15 PRIMAL-DUAL INTERIOR POINT METHOD FOR LINEAR PROGRAMMING15 Optimal Value (4) max 2x 1 + 7x 2 + 6x 3 + 4x 4 s.t. x 1 + x x x x 1 + x 2 + x x x x x x 4 80 (5) x 1, x 2, x 3, x 4 0 Iterations: 15 Optimal Solution: x1 = e-10. x2 = x3 = x4 = Optimal Value: max 2x 1 x 2 + 2x 3 s.t. 2x 1 + x 2 10 x 1 + 2x 2 2x 3 20 x 2 + 2x 3 5 x 1, x 2, x 3 0 Iterations: 15 Optimal Solution:

16 16 KELLER VANDEBOGERT AND CHARLES LANNING x1 = x2 = e-10. x3 = Optimal Value: (6) min 2x 1 3x 2 4x 3 s.t. 3x 1 + 2x 2 + x 3 = 10 2x 1 + 5x 2 + 3x 3 = 15 x 1, x 2, x 3 0 Iterations: 10 Optimal Solution: x1 = x2 = e-10. x3 = Optimal Value: Note that the number of iterations was never any more that 15, and our accuracy was always within 10 9 of the actual solution. The iterations seem to be higher when working with more variables (in example 4, we use four variables and three slacks) and when there are constraints. 7. Conclusion In conclusion, we see that interior point methods are computationally efficient even with dealing with relatively large constraints. The theory

17 PRIMAL-DUAL INTERIOR POINT METHOD FOR LINEAR PROGRAMMING17 behind them is based on the concept of Primal-Dual barrier reformulations and uses a modified Newton s method to stay sufficiently close to the central path of convergence, as opposed to explicit calculation of this central path, which is the reason this method is so successful. Our numerical tests all converged successfully, and the code used was extremely flexible in dealing with equality-inequality constraints. References [1] Lesaja, Goran Introducing Interior-Point Methods for Introductory Operations Research Courses and/or Linear Programming Courses. Open Operational Research Journal, 3: doi: /

### LECTURE 13: SOLUTION METHODS FOR CONSTRAINED OPTIMIZATION. 1. Primal approach 2. Penalty and barrier methods 3. Dual approach 4. Primal-dual approach

LECTURE 13: SOLUTION METHODS FOR CONSTRAINED OPTIMIZATION 1. Primal approach 2. Penalty and barrier methods 3. Dual approach 4. Primal-dual approach Basic approaches I. Primal Approach - Feasible Direction

### Applied Lagrange Duality for Constrained Optimization

Applied Lagrange Duality for Constrained Optimization Robert M. Freund February 10, 2004 c 2004 Massachusetts Institute of Technology. 1 1 Overview The Practical Importance of Duality Review of Convexity

### Computational Methods. Constrained Optimization

Computational Methods Constrained Optimization Manfred Huber 2010 1 Constrained Optimization Unconstrained Optimization finds a minimum of a function under the assumption that the parameters can take on

### COMS 4771 Support Vector Machines. Nakul Verma

COMS 4771 Support Vector Machines Nakul Verma Last time Decision boundaries for classification Linear decision boundary (linear classification) The Perceptron algorithm Mistake bound for the perceptron

### Lecture 7: Support Vector Machine

Lecture 7: Support Vector Machine Hien Van Nguyen University of Houston 9/28/2017 Separating hyperplane Red and green dots can be separated by a separating hyperplane Two classes are separable, i.e., each

### Introduction to Constrained Optimization

Introduction to Constrained Optimization Duality and KKT Conditions Pratik Shah {pratik.shah [at] lnmiit.ac.in} The LNM Institute of Information Technology www.lnmiit.ac.in February 13, 2013 LNMIIT MLPR

### Mathematical Programming and Research Methods (Part II)

Mathematical Programming and Research Methods (Part II) 4. Convexity and Optimization Massimiliano Pontil (based on previous lecture by Andreas Argyriou) 1 Today s Plan Convex sets and functions Types

### 16.410/413 Principles of Autonomy and Decision Making

16.410/413 Principles of Autonomy and Decision Making Lecture 17: The Simplex Method Emilio Frazzoli Aeronautics and Astronautics Massachusetts Institute of Technology November 10, 2010 Frazzoli (MIT)

### Introduction to Optimization

Introduction to Optimization Second Order Optimization Methods Marc Toussaint U Stuttgart Planned Outline Gradient-based optimization (1st order methods) plain grad., steepest descent, conjugate grad.,

### Sparse Optimization Lecture: Proximal Operator/Algorithm and Lagrange Dual

Sparse Optimization Lecture: Proximal Operator/Algorithm and Lagrange Dual Instructor: Wotao Yin July 2013 online discussions on piazza.com Those who complete this lecture will know learn the proximal

### Support Vector Machines. James McInerney Adapted from slides by Nakul Verma

Support Vector Machines James McInerney Adapted from slides by Nakul Verma Last time Decision boundaries for classification Linear decision boundary (linear classification) The Perceptron algorithm Mistake

### Machine Learning for Signal Processing Lecture 4: Optimization

Machine Learning for Signal Processing Lecture 4: Optimization 13 Sep 2015 Instructor: Bhiksha Raj (slides largely by Najim Dehak, JHU) 11-755/18-797 1 Index 1. The problem of optimization 2. Direct optimization

### All lecture slides will be available at CSC2515_Winter15.html

CSC2515 Fall 2015 Introduc3on to Machine Learning Lecture 9: Support Vector Machines All lecture slides will be available at http://www.cs.toronto.edu/~urtasun/courses/csc2515/ CSC2515_Winter15.html Many

### A Short SVM (Support Vector Machine) Tutorial

A Short SVM (Support Vector Machine) Tutorial j.p.lewis CGIT Lab / IMSC U. Southern California version 0.zz dec 004 This tutorial assumes you are familiar with linear algebra and equality-constrained optimization/lagrange

### 10. Network dimensioning

Partly based on slide material by Samuli Aalto and Jorma Virtamo ELEC-C7210 Modeling and analysis of communication networks 1 Contents Introduction Parameters: topology, routing and traffic Dimensioning

### Projection-Based Methods in Optimization

Projection-Based Methods in Optimization Charles Byrne (Charles Byrne@uml.edu) http://faculty.uml.edu/cbyrne/cbyrne.html Department of Mathematical Sciences University of Massachusetts Lowell Lowell, MA

### Algorithms for convex optimization

Algorithms for convex optimization Michal Kočvara Institute of Information Theory and Automation Academy of Sciences of the Czech Republic and Czech Technical University kocvara@utia.cas.cz http://www.utia.cas.cz/kocvara

### Discrete Optimization. Lecture Notes 2

Discrete Optimization. Lecture Notes 2 Disjunctive Constraints Defining variables and formulating linear constraints can be straightforward or more sophisticated, depending on the problem structure. The

### Theoretical Concepts of Machine Learning

Theoretical Concepts of Machine Learning Part 2 Institute of Bioinformatics Johannes Kepler University, Linz, Austria Outline 1 Introduction 2 Generalization Error 3 Maximum Likelihood 4 Noise Models 5

### On the Dual Approach to Recursive Optimization

On the Dual Approach to Recursive Optimization Messner - Pavoni - Sleet Discussion by: Ctirad Slavík, Goethe Uni Frankfurt Conference on Private Information, Interdependent Preferences and Robustness:

### 5. DUAL LP, SOLUTION INTERPRETATION, AND POST-OPTIMALITY

5. DUAL LP, SOLUTION INTERPRETATION, AND POST-OPTIMALITY 5.1 DUALITY Associated with every linear programming problem (the primal) is another linear programming problem called its dual. If the primal involves

### Section Notes 5. Review of Linear Programming. Applied Math / Engineering Sciences 121. Week of October 15, 2017

Section Notes 5 Review of Linear Programming Applied Math / Engineering Sciences 121 Week of October 15, 2017 The following list of topics is an overview of the material that was covered in the lectures

### New Methods for Solving Large Scale Linear Programming Problems in the Windows and Linux computer operating systems

arxiv:1209.4308v1 [math.oc] 19 Sep 2012 New Methods for Solving Large Scale Linear Programming Problems in the Windows and Linux computer operating systems Saeed Ketabchi, Hossein Moosaei, Hossein Sahleh

### Compact Sets. James K. Peterson. September 15, Department of Biological Sciences and Department of Mathematical Sciences Clemson University

Compact Sets James K. Peterson Department of Biological Sciences and Department of Mathematical Sciences Clemson University September 15, 2017 Outline 1 Closed Sets 2 Compactness 3 Homework Closed Sets

### Alternating Direction Method of Multipliers

Alternating Direction Method of Multipliers CS 584: Big Data Analytics Material adapted from Stephen Boyd (https://web.stanford.edu/~boyd/papers/pdf/admm_slides.pdf) & Ryan Tibshirani (http://stat.cmu.edu/~ryantibs/convexopt/lectures/21-dual-meth.pdf)

### Least-Squares Minimization Under Constraints EPFL Technical Report #

Least-Squares Minimization Under Constraints EPFL Technical Report # 150790 P. Fua A. Varol R. Urtasun M. Salzmann IC-CVLab, EPFL IC-CVLab, EPFL TTI Chicago TTI Chicago Unconstrained Least-Squares minimization

### A Numerical Study of Active-Set and Interior-Point Methods for Bound Constrained Optimization

A Numerical Study of Active-Set and Interior-Point Methods for Bound Constrained Optimization Long Hei 1, Jorge Nocedal 2, Richard A. Waltz 2 1 Department of Industrial Engineering and Management Sciences,

### 60 2 Convex sets. {x a T x b} {x ã T x b}

60 2 Convex sets Exercises Definition of convexity 21 Let C R n be a convex set, with x 1,, x k C, and let θ 1,, θ k R satisfy θ i 0, θ 1 + + θ k = 1 Show that θ 1x 1 + + θ k x k C (The definition of convexity

### Modified Augmented Lagrangian Coordination and Alternating Direction Method of Multipliers with

Modified Augmented Lagrangian Coordination and Alternating Direction Method of Multipliers with Parallelization in Non-hierarchical Analytical Target Cascading Yongsu Jung Department of Mechanical Engineering,

### Problem Set #3 (With Corrections)

IC-32 Optimal Control Winter 2006 Benoît Chachuat ME C2 401, Ph: 33844, benoit.chachuat@epfl.ch Problem Set #3 (With Corrections) 1. Consider the following NLP problem: min x IR 3 f(x) := x2 1 + x 1 x

### Linear programming and duality theory

Linear programming and duality theory Complements of Operations Research Giovanni Righini Linear Programming (LP) A linear program is defined by linear constraints, a linear objective function. Its variables

### 5 The Theory of the Simplex Method

5 The Theory of the Simplex Method Chapter 4 introduced the basic mechanics of the simplex method. Now we shall delve a little more deeply into this algorithm by examining some of its underlying theory.

### Open and Closed Sets

Open and Closed Sets Definition: A subset S of a metric space (X, d) is open if it contains an open ball about each of its points i.e., if x S : ɛ > 0 : B(x, ɛ) S. (1) Theorem: (O1) and X are open sets.

### Generalized Network Flow Programming

Appendix C Page Generalized Network Flow Programming This chapter adapts the bounded variable primal simplex method to the generalized minimum cost flow problem. Generalized networks are far more useful

### Optimality certificates for convex minimization and Helly numbers

Optimality certificates for convex minimization and Helly numbers Amitabh Basu Michele Conforti Gérard Cornuéjols Robert Weismantel Stefan Weltge October 20, 2016 Abstract We consider the problem of minimizing

### Delay-Constrained Optimized Packet Aggregation in High-Speed Wireless Networks

Teymoori P, Yazdani N. Delay-constrained optimized packet aggregation in high-speed wireless networks. JOURNAL OF COMPUTER SCIENCE AND TECHNOLOGY 28(3): 525 539 May 2013. DOI 10.1007/s11390-013-1353-1

### Topology and Topological Spaces

Topology and Topological Spaces Mathematical spaces such as vector spaces, normed vector spaces (Banach spaces), and metric spaces are generalizations of ideas that are familiar in R or in R n. For example,

### Subject of Investigation

Subject of Investigation A 3D Analysis of the Node to Node and Node to Surface Contact Phenomenon using Penalty Function and Lagrange Multipliers Constraints in Matlab Academic Advisor Professor José M.

### 14.5 Directional Derivatives and the Gradient Vector

14.5 Directional Derivatives and the Gradient Vector 1. Directional Derivatives. Recall z = f (x, y) and the partial derivatives f x and f y are defined as f (x 0 + h, y 0 ) f (x 0, y 0 ) f x (x 0, y 0

### MATLAB Solution of Linear Programming Problems

MATLAB Solution of Linear Programming Problems The simplex method is included in MATLAB using linprog function. All is needed is to have the problem expressed in the terms of MATLAB definitions. Appendix

### A New Unblocking Technique to Warmstart Interior Point Methods based on Sensitivity Analysis. December 19th, 2006 revised June 1, 2007 MS

A New Unblocking Technique to Warmstart Interior Point Methods based on Sensitivity Analysis Jacek Gondzio Andreas Grothey December 19th, 2006 revised June 1, 2007 MS-06-005 1 A New Unblocking Technique

### An Overview of Cutting Plane Methods for Semidefinite Programming

SF2827: Topics in Optimization An Overview of Cutting Plane Methods for Semidefinite Programming Supervisor: Anders Forsgren Department of Mathematics Royal Institute of Technology (KTH) andersf@kth.se

### Convexity: an introduction

Convexity: an introduction Geir Dahl CMA, Dept. of Mathematics and Dept. of Informatics University of Oslo 1 / 74 1. Introduction 1. Introduction what is convexity where does it arise main concepts and

### SYSTEMS OF NONLINEAR EQUATIONS

SYSTEMS OF NONLINEAR EQUATIONS Widely used in the mathematical modeling of real world phenomena. We introduce some numerical methods for their solution. For better intuition, we examine systems of two

### A Comparative study on Algorithms for Shortest-Route Problem and Some Extensions

International Journal of Basic & Applied Sciences IJBAS-IJENS Vol: No: 0 A Comparative study on Algorithms for Shortest-Route Problem and Some Extensions Sohana Jahan, Md. Sazib Hasan Abstract-- The shortest-route

### Chapter 3. Bootstrap. 3.1 Introduction. 3.2 The general idea

Chapter 3 Bootstrap 3.1 Introduction The estimation of parameters in probability distributions is a basic problem in statistics that one tends to encounter already during the very first course on the subject.

### Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization. Author: Martin Jaggi Presenter: Zhongxing Peng

Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization Author: Martin Jaggi Presenter: Zhongxing Peng Outline 1. Theoretical Results 2. Applications Outline 1. Theoretical Results 2. Applications

### 21-256: Lagrange multipliers

21-256: Lagrange multipliers Clive Newstead, Thursday 12th June 2014 Lagrange multipliers give us a means of optimizing multivariate functions subject to a number of constraints on their variables. Problems

### Constrained Optimization with Calculus. Background Three Big Problems Setup and Vocabulary

Constrained Optimization with Calculus Background Three Big Problems Setup and Vocabulary Background Information In unit 3, you learned about linear programming, in which all constraints and the objective

### Algorithmic Game Theory and Applications. Lecture 6: The Simplex Algorithm

Algorithmic Game Theory and Applications Lecture 6: The Simplex Algorithm Kousha Etessami Recall our example 1 x + y

### A Simplex Based Parametric Programming Method for the Large Linear Programming Problem

A Simplex Based Parametric Programming Method for the Large Linear Programming Problem Huang, Rujun, Lou, Xinyuan Abstract We present a methodology of parametric objective function coefficient programming

### SUBSTITUTING GOMORY CUTTING PLANE METHOD TOWARDS BALAS ALGORITHM FOR SOLVING BINARY LINEAR PROGRAMMING

Bulletin of Mathematics Vol. 06, No. 0 (20), pp.. SUBSTITUTING GOMORY CUTTING PLANE METHOD TOWARDS BALAS ALGORITHM FOR SOLVING BINARY LINEAR PROGRAMMING Eddy Roflin, Sisca Octarina, Putra B. J Bangun,

### VARIANTS OF THE SIMPLEX METHOD

C H A P T E R 6 VARIANTS OF THE SIMPLEX METHOD By a variant of the Simplex Method (in this chapter) we mean an algorithm consisting of a sequence of pivot steps in the primal system using alternative rules

### WE consider the gate-sizing problem, that is, the problem

2760 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS I: REGULAR PAPERS, VOL 55, NO 9, OCTOBER 2008 An Efficient Method for Large-Scale Gate Sizing Siddharth Joshi and Stephen Boyd, Fellow, IEEE Abstract We consider

### Delay-minimal Transmission for Energy Constrained Wireless Communications

Delay-minimal Transmission for Energy Constrained Wireless Communications Jing Yang Sennur Ulukus Department of Electrical and Computer Engineering University of Maryland, College Park, M0742 yangjing@umd.edu

### A Lagrange method based L-curve for image restoration

Journal of Physics: Conference Series OPEN ACCESS A Lagrange method based L-curve for image restoration To cite this article: G Landi 2013 J. Phys.: Conf. Ser. 464 012011 View the article online for updates

### Alternating Projections

Alternating Projections Stephen Boyd and Jon Dattorro EE392o, Stanford University Autumn, 2003 1 Alternating projection algorithm Alternating projections is a very simple algorithm for computing a point

### Chapter 7. Linear Programming Models: Graphical and Computer Methods

Chapter 7 Linear Programming Models: Graphical and Computer Methods To accompany Quantitative Analysis for Management, Eleventh Edition, by Render, Stair, and Hanna Power Point slides created by Brian

### The exam is closed book, closed notes except your one-page (two-sided) cheat sheet.

CS 189 Spring 2015 Introduction to Machine Learning Final You have 2 hours 50 minutes for the exam. The exam is closed book, closed notes except your one-page (two-sided) cheat sheet. No calculators or

### Optimal Control Techniques for Dynamic Walking

Optimal Control Techniques for Dynamic Walking Optimization in Robotics & Biomechanics IWR, University of Heidelberg Presentation partly based on slides by Sebastian Sager, Moritz Diehl and Peter Riede

### Data Analysis 3. Support Vector Machines. Jan Platoš October 30, 2017

Data Analysis 3 Support Vector Machines Jan Platoš October 30, 2017 Department of Computer Science Faculty of Electrical Engineering and Computer Science VŠB - Technical University of Ostrava Table of

### Sparsity Based Regularization

9.520: Statistical Learning Theory and Applications March 8th, 200 Sparsity Based Regularization Lecturer: Lorenzo Rosasco Scribe: Ioannis Gkioulekas Introduction In previous lectures, we saw how regularization

### Approximation Algorithms: The Primal-Dual Method. My T. Thai

Approximation Algorithms: The Primal-Dual Method My T. Thai 1 Overview of the Primal-Dual Method Consider the following primal program, called P: min st n c j x j j=1 n a ij x j b i j=1 x j 0 Then the

### Math Introduction to Operations Research

Math 300 Introduction to Operations Research Examination (50 points total) Solutions. (6 pt total) Consider the following linear programming problem: Maximize subject to and x, x, x 3 0. 3x + x + 5x 3

### Problem set 2. Problem 1. Problem 2. Problem 3. CS261, Winter Instructor: Ashish Goel.

CS261, Winter 2017. Instructor: Ashish Goel. Problem set 2 Electronic submission to Gradescope due 11:59pm Thursday 2/16. Form a group of 2-3 students that is, submit one homework with all of your names.

### Appendix A MATLAB s Optimization Toolbox Algorithms

Appendix A MATLAB s Optimization Toolbox Algorithms Abstract MATLAB s Optimization Toolbox (version 7:2) includes a family of algorithms for solving optimization problems. The toolbox provides functions

### A Nonlinear Presolve Algorithm in AIMMS

A Nonlinear Presolve Algorithm in AIMMS By Marcel Hunting marcel.hunting@aimms.com November 2011 This paper describes the AIMMS presolve algorithm for nonlinear problems. This presolve algorithm uses standard

### A Deterministic Global Optimization Method for Variational Inference

A Deterministic Global Optimization Method for Variational Inference Hachem Saddiki Mathematics and Statistics University of Massachusetts, Amherst saddiki@math.umass.edu Andrew C. Trapp Operations and

### Today. Gradient descent for minimization of functions of real variables. Multi-dimensional scaling. Self-organizing maps

Today Gradient descent for minimization of functions of real variables. Multi-dimensional scaling Self-organizing maps Gradient Descent Derivatives Consider function f(x) : R R. The derivative w.r.t. x

### FRACTAL CURVES CHELLE RITZENTHALER

FRACTAL CURVES CHELLE RITZENTHALER Abstract. Fractal curves are employed in many different disciplines to describe anything from the growth of a tree to measuring the length of a coastline. We define a

### Bilinear Programming

Bilinear Programming Artyom G. Nahapetyan Center for Applied Optimization Industrial and Systems Engineering Department University of Florida Gainesville, Florida 32611-6595 Email address: artyom@ufl.edu

### Decomposition in Integer Linear Programming

Decomposition in Integer Linear Programming M.V. Galati Advanced Analytics - Operations R & D, SAS Institute, Chesterbrook, PA 9 Ted K. Ralphs Department of Industrial and System Engineering, Lehigh University,

### Convex Optimization - Chapter 1-2. Xiangru Lian August 28, 2015

Convex Optimization - Chapter 1-2 Xiangru Lian August 28, 2015 1 Mathematical optimization minimize f 0 (x) s.t. f j (x) 0, j=1,,m, (1) x S x. (x 1,,x n ). optimization variable. f 0. R n R. objective

### BCN Decision and Risk Analysis. Syed M. Ahmed, Ph.D.

Linear Programming Module Outline Introduction The Linear Programming Model Examples of Linear Programming Problems Developing Linear Programming Models Graphical Solution to LP Problems The Simplex Method

### MLR Institute of Technology

Course Name : Engineering Optimization Course Code : 56021 Class : III Year Branch : Aeronautical Engineering Year : 2014-15 Course Faculty : Mr Vamsi Krishna Chowduru, Assistant Professor Course Objective

### Support Vector Machines for Face Recognition

Chapter 8 Support Vector Machines for Face Recognition 8.1 Introduction In chapter 7 we have investigated the credibility of different parameters introduced in the present work, viz., SSPD and ALR Feature

### Lecture 4: Convexity

10-725: Convex Optimization Fall 2013 Lecture 4: Convexity Lecturer: Barnabás Póczos Scribes: Jessica Chemali, David Fouhey, Yuxiong Wang Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:

### Discrete Optimization 2010 Lecture 5 Min-Cost Flows & Total Unimodularity

Discrete Optimization 2010 Lecture 5 Min-Cost Flows & Total Unimodularity Marc Uetz University of Twente m.uetz@utwente.nl Lecture 5: sheet 1 / 26 Marc Uetz Discrete Optimization Outline 1 Min-Cost Flows

### Review for Mastery Using Graphs and Tables to Solve Linear Systems

3-1 Using Graphs and Tables to Solve Linear Systems A linear system of equations is a set of two or more linear equations. To solve a linear system, find all the ordered pairs (x, y) that make both equations

### Optimal Proxy-Limited Lines for Representing Voltage Constraints in a DC Optimal Powerflow

Optimal Proxy-Limited Lines for Representing Voltage Constraints in a DC Optimal Powerflow by Michael Schlindwein A thesis submitted in fulfillment of the requirements for the degree of Master of Science

### Repetition: Primal Dual for Set Cover

Repetition: Primal Dual for Set Cover Primal Relaxation: k min i=1 w ix i s.t. u U i:u S i x i 1 i {1,..., k} x i 0 Dual Formulation: max u U y u s.t. i {1,..., k} u:u S i y u w i y u 0 Harald Räcke 428

### OPERATIONS RESEARCH. Linear Programming Problem

OPERATIONS RESEARCH Chapter 1 Linear Programming Problem Prof. Bibhas C. Giri Department of Mathematics Jadavpur University Kolkata, India Email: bcgiri.jumath@gmail.com 1.0 Introduction Linear programming

### 4.1 Graphical solution of a linear program and standard form

4.1 Graphical solution of a linear program and standard form Consider the problem min c T x Ax b x where x = ( x1 x ) ( 16, c = 5 ), b = 4 5 9, A = 1 7 1 5 1. Solve the problem graphically and determine

### MS&E 213 / CS 269O : Chapter 1 Introduction to Introduction to Optimization Theory

MS&E 213 / CS 269O : Chapter 1 Introduction to Introduction to Optimization Theory By Aaron Sidford (sidford@stanford.edu) April 29, 2017 1 What is this course about? The central problem we consider throughout

### Technische Universität München Zentrum Mathematik

Question 1. Incidence matrix with gaps Technische Universität München Zentrum Mathematik Prof. Dr. Dr. Jürgen Richter-Gebert, Bernhard Werner Projective Geometry SS 2016 www-m10.ma.tum.de/projektivegeometriess16

### A New Unblocking Technique to Warmstart Interior Point Methods based on Sensitivity Analysis

A New Unblocking Technique to Warmstart Interior Point Methods based on Sensitivity Analysis Jacek Gondzio Andreas Grothey December 19th, 26 revised June 7th, 27 revised January 25th, 28 MS-6-5 1 A New

### Tilburg University. Document version: Publisher final version (usually the publisher pdf) Publication date: Link to publication

Tilburg University Multi-stage Adjustable Robust Mixed-Integer Optimization via Iterative Splitting of the Uncertainty set (Revision of CentER Discussion Paper 2014-056) Postek, Krzysztof; den Hertog,

### Physics 235 Chapter 6. Chapter 6 Some Methods in the Calculus of Variations

Chapter 6 Some Methods in the Calculus of Variations In this Chapter we focus on an important method of solving certain problems in Classical Mechanics. In many problems we need to determine how a system

### Trajectory Optimization

C H A P T E R 12 Trajectory Optimization So far, we have discussed a number of ways to solve optimal control problems via state space search (e.g., Dijkstra s and Dynamic Programming/Value Iteration).

### SUBSTITUTING GOMORY CUTTING PLANE METHOD TOWARDS BALAS ALGORITHM FOR SOLVING BINARY LINEAR PROGRAMMING

ASIAN JOURNAL OF MATHEMATICS AND APPLICATIONS Volume 2014, Article ID ama0156, 11 pages ISSN 2307-7743 http://scienceasia.asia SUBSTITUTING GOMORY CUTTING PLANE METHOD TOWARDS BALAS ALGORITHM FOR SOLVING

### A CLASS OF INDIRECT UTILITY FUNCTIONS PREDICTING GIFFEN BEHAVIOUR. P.G. MOFFATT, School of Economics, University of East Anglia, UK.

A CLASS OF INDIRECT UTILITY FUNCTIONS PREDICTING GIFFEN BEHAVIOUR P.G. MOFFATT, School of Economics, University of East Anglia, UK September 2010 Abstract The problem of recognising Giffen behaviour is

### Lecture 2. Topology of Sets in R n. August 27, 2008

Lecture 2 Topology of Sets in R n August 27, 2008 Outline Vectors, Matrices, Norms, Convergence Open and Closed Sets Special Sets: Subspace, Affine Set, Cone, Convex Set Special Convex Sets: Hyperplane,

### A PARAMETRIC SIMPLEX METHOD FOR OPTIMIZING A LINEAR FUNCTION OVER THE EFFICIENT SET OF A BICRITERIA LINEAR PROBLEM. 1.

ACTA MATHEMATICA VIETNAMICA Volume 21, Number 1, 1996, pp. 59 67 59 A PARAMETRIC SIMPLEX METHOD FOR OPTIMIZING A LINEAR FUNCTION OVER THE EFFICIENT SET OF A BICRITERIA LINEAR PROBLEM NGUYEN DINH DAN AND

### Linear Programming has been used to:

Linear Programming Linear programming became important during World War II: used to solve logistics problems for the military. Linear Programming (LP) was the first widely used form of optimization in

### Optimization for Machine Learning

Optimization for Machine Learning (Problems; Algorithms - C) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html

### Algorithms for two-stage stochastic linear programmming

Algorithms for two-stage stochastic linear programmming Basic Course on Stochastic Programming, IMPA 2016 Description Consider the following two-stage stochastic linear program c x + N min x s.t. Ax =

### 3 INTEGER LINEAR PROGRAMMING

3 INTEGER LINEAR PROGRAMMING PROBLEM DEFINITION Integer linear programming problem (ILP) of the decision variables x 1,..,x n : (ILP) subject to minimize c x j j n j= 1 a ij x j x j 0 x j integer n j=

### 9. Support Vector Machines. The linearly separable case: hard-margin SVMs. The linearly separable case: hard-margin SVMs. Learning objectives

Foundations of Machine Learning École Centrale Paris Fall 25 9. Support Vector Machines Chloé-Agathe Azencot Centre for Computational Biology, Mines ParisTech Learning objectives chloe agathe.azencott@mines