INTRODUCTION TO LINEAR AND NONLINEAR PROGRAMMING DAVID G. LUENBERGER Stanford University TT ADDISON-WESLEY PUBLISHING COMPANY Reading, Massachusetts Menlo Park, California London Don Mills, Ontario
CONTENTS Chapter 1 Introduction 1.1 Optimization 1 1.2 Types of problems 2 1.3 Size of problems 5 1.4 Iterative algorithms and convergence 6 PART I Linear Programming Chapter 2 Basic Properties of Linear Programs 2.1 Introduction 11 2.2 Examples of linear programming problems 14 2.3 Basic Solutions 16 2.4 The fundamental theorem of linear programming 18 2.5 Relations to convexity 20 2.6 Exercises 25 Chapter 3 The Simplex Method 3.1 Pivots 27 3.2 Adjacent extreme points 33 3.3 Determining a minimum feasible Solution 36 3.4 Computational procedure simplex method 40 3.5 Artiflcial variables 43 *3.6 Variables with upper bounds 48 3.7 Matrix form of the simplex method 53 3.8 The revised simplex method 55 *3.9 The simplex method and LU decomposition 60 3.10 Summary 63 3.11 Exercises 63 ix
x Chapter 4 Duality 4.1 Dual linear programs 68 4.2 The duality theorem 71 4.3 Relations to the simplex procedura 73 4.4 Sensitivity and complementary slackness 76 *4.5 The dual simplex method 78 *4.6 The primal-dual algorithm 81 4.7 Exercises 86 Chapter 5 Reduction of Linear Inequalities 5.1 Introduction 90 5.2 Redundant equations 91 5.3 Null variables 91 5.4 Nonextremal variables 94 5.5 Reduced problems 96 5.6 Applications of reduction 104 5.7 Exercises 105 PART II Unconstrained Problems Chapter 6 Basic Properties of Solutions and Algorithms 6.1 Necessary conditions for local minima 110 6.2 Sufficient conditions for a relative minimum 113 6.3 Convex and concave functions 114 6.4 Minimization and maximization of convex functions... 118 6.5 Global convergence of descent algorithms 120 6.6 Speed of convergence 127 6.7 Summary 131 6.8 Exercises 131 Chapter 7 Basic Descent Methods 7.1 Fibonacci and golden section search 134 7.2 Line search by curve fitting 137 7.3 Global convergence of curve fitting 143 7.4 Closedness of line search algorithms 146 7.5 Inaecurate line search 147 7.6 The method of steepesj descent.. 148 7.7 Newton's method 155 7.8 Coordinate descent methods 158 7.9 Spacer steps 161 7.10 Summary 162 7.11 Exercises 163
xi Chapter 8 Conjugate Direction Methods 8.1 Conjugate directions 168 8.2 Descent properties of the conjugate direction method... 171 8.3 The conjugate gradient method 173 8.4 The conjugate gradient method as an optimal process... 176 8.5 The partial conjugate gradient method 178 8.6 Extension to nonquadratic problems 181 8.7 Parallel tangents 184 8.8 Exercises 186 Chapter 9 Quasi-Newton Methods 9.1 Modified Newton method 189 9.2 Construction of the inverse 192 9.3 Davidon-Fletcher-Powell method 194 9.4 Convergence properties 197 *9.5 Scaling 201 9.6 Combination of steepest descent and Newton's method.. 205 9.7 Updating procedures 210 9.8 Summary 211 9.9 Exercises 213 PART III Constrained Minimization Chapter 10 Constrained Minimization Conditions 10.1 Constraints 219 10.2 Tangent plane 221 10.3 Necessary and sufficient conditions (equality constraints).. 224 10.4 Eigenvalues in tangent subspace 227 10.5 Sensitivity 230 10.6 Inequality constraints 232 10.7 Summary 236 10.8 Exercises 237 Chapter 11 Primat Methods 11.1 Advantage of primal methods 240 11.2 Feasible direction methods 241 11.3 Global convergence 243 11.4 The gradient projection method 247 11.5 Convergence rate of the gradient projection method... 254 11.6 The reduced gradient method 262 11.7 Convergence rate of the reduced gradient method.... 267 11.8 Variations 270 11.9 Summary 272 11.10 Exercises 273
xii Chapter 12 Penalty and Barrier Methods 12.1 Penalty methods 278 12.2 Barrier methods 281 12.3 Properties of penalty and barrier functions 283 *12.4 Extrapolation 289 12.5 Newton's method and penalty functions 290 12.6 Conjugate gradients and penalty methods 292 12.7 Normalization of penalty functions 294 12.8 Penalty functions and gradient projection 296 12.9 Penalty functions and the reduced gradient method.... 299 12.10 Summary 301 12.11 Exercises 302 Chapter 13 Cutting Plane and Dual Methods 13.1 Cutting plane methods 306 13.2 Kelley's convex cutting plane algorithm 308 13.3 Modifications 310 13.4 Local duality 312 13.5 Dual canonical convergence rate 317 13.6 Separable Problems 318 13.7 Duality and penalty functions 320 13.8 Exercises 322 Appendix A Mathematical Review A.l Sets 324 A.2 Matrix notation 325 A.3 Spaces 326 A.4 Eigenvalues and quadratic forms 327 A.5 Topological concepts 328 A.6 Functions 329 Appendix B Convex Sets B.l Basic definitions 332 B.2 Hyperplanes and polytopes 334 B.3 Separating and supporting hyperplanes 337 B.4 Extreme points 338 Appendix C Gaussian Elimination Bibliography 344 Index 353