Contents. I Basics 1. Copyright by SIAM. Unauthorized reproduction of this article is prohibited.

Transcription

1 page v Preface xiii I Basics 1 1 Optimization Models Introduction Optimization: An Informal Introduction Linear Equations Linear Optimization Exercises Least-Squares Data Fitting Exercises Nonlinear Optimization Optimization Applications Crew Scheduling and Fleet Scheduling Exercises Support Vector Machines Exercises Portfolio Optimization Exercises Intensity Modulated Radiation Treatment Planning Exercises Positron Emission Tomography Image Reconstruction.. 32 Exercises Shape Optimization Notes Fundamentals of Optimization Introduction Feasibility and Optimality Exercises Convexity Derivatives and Convexity v

2 page vi vi Exercises The General Optimization Algorithm Exercises Rates of Convergence Exercises Taylor Series Exercises Newton s Method for Nonlinear Equations Systems of Nonlinear Equations Exercises Notes Representation of Linear Constraints Basic Concepts Exercises Null and Range Spaces Exercises Generating Null-Space Matrices Variable Reduction Method Orthogonal Projection Matrix Other Projections The QR Factorization Exercises Notes II Linear Programming 95 4 Geometry of Linear Programming Introduction Exercises Standard Form Exercises Basic Solutions and Extreme Points Exercises Representation of Solutions; Optimality Exercises Notes The Simplex Method Introduction The Simplex Method General Formulas Unbounded Problems Notation for the Simplex Method (Tableaus) Deficiencies of the Tableau...139

3 page vii vii Exercises The Simplex Method (Details) Multiple Solutions Feasible Directions and Edge Directions Exercises Getting Started Artificial Variables The Two-Phase Method The Big-M Method Exercises Degeneracy and Termination Resolving Degeneracy Using Perturbation Exercises Notes Duality and Sensitivity The Dual Problem Exercises Duality Theory Complementary Slackness Interpretation of the Dual Exercises The Dual Simplex Method Exercises Sensitivity Exercises Parametric Linear Programming Exercises Notes Enhancements of the Simplex Method Introduction Problems with Upper Bounds Exercises Column Generation Exercises The Decomposition Principle Exercises Representation of the Basis The Product Form of the Inverse Representation of the Basis The LU Factorization Exercises Numerical Stability and Computational Efficiency Pricing The Initial Basis Tolerances; Degeneracy Scaling...266

4 page viii viii Preprocessing Model Formats Exercises Notes Network Problems Introduction Basic Concepts and Examples Exercises Representation of the Basis Exercises The Network Simplex Method Exercises Resolving Degeneracy Exercises Notes Computational Complexity of Linear Programming Introduction Computational Complexity Exercises Worst-Case Behavior of the Simplex Method Exercises The Ellipsoid Method Exercises The Average-Case Behavior of the Simplex Method Notes Interior-Point Methods for Linear Programming Introduction The Primal-Dual Interior-Point Method Computational Aspects of Interior-Point Methods The Predictor-Corrector Algorithm Exercises Feasibility and Self-Dual Formulations Exercises Some Concepts from Nonlinear Optimization Affine-Scaling Methods Exercises Path-Following Methods Exercises Notes...353

5 page ix ix III Unconstrained Optimization Basics of Unconstrained Optimization Introduction Optimality Conditions Exercises Newton s Method for Minimization Exercises Guaranteeing Descent Exercises Guaranteeing Convergence: Line Search Methods Other Line Searches Exercises Guaranteeing Convergence: Trust-Region Methods Exercises Notes Methods for Unconstrained Optimization Introduction Steepest-Descent Method Exercises Quasi-Newton Methods Exercises Automating Derivative Calculations Finite-Difference Derivative Estimates Automatic Differentiation Exercises Methods That Do Not Require Derivatives Simulation-Based Optimization Compass Search: A Derivative-Free Method Convergence of Compass Search Exercises Termination Rules Exercises Historical Background Notes Low-Storage Methods for Unconstrained Problems Introduction The Conjugate-Gradient Method for Solving Linear Equations Exercises Truncated-Newton Methods Exercises Nonlinear Conjugate-Gradient Methods Exercises Limited-Memory Quasi-Newton Methods...470

6 page x x Exercises Preconditioning Exercises Notes IV Nonlinear Optimization Optimality Conditions for Constrained Problems Introduction Optimality Conditions for Linear Equality Constraints Exercises The Lagrange Multipliers and the Lagrangian Function Exercises Optimality Conditions for Linear Inequality Constraints Exercises Optimality Conditions for Nonlinear Constraints Statement of Optimality Conditions Exercises Preview of Methods Exercises Derivation of Optimality Conditions for Nonlinear Constraints Exercises Duality Games and Min-Max Duality Lagrangian Duality Wolfe Duality More on the Dual Function Duality in Support Vector Machines Exercises Historical Background Notes Feasible-Point Methods Introduction Linear Equality Constraints Exercises Computing the Lagrange Multipliers Exercises Linear Inequality Constraints Linear Programming Exercises Sequential Quadratic Programming Exercises Reduced-Gradient Methods Exercises...588

7 page xi xi 15.7 Filter Methods Exercises Notes Penalty and Barrier Methods Introduction Classical Penalty and Barrier Methods Barrier Methods Penalty Methods Convergence Exercises Ill-Conditioning Stabilized Penalty and Barrier Methods Exercises Exact Penalty Methods Exercises Multiplier-Based Methods Dual Interpretation Exercises Nonlinear Primal-Dual Methods Primal-Dual Interior-Point Methods Convergence of the Primal-Dual Interior-Point Method. 645 Exercises Semidefinite Programming Exercises Notes V Appendices 659 A Topics from Linear Algebra 661 A.1 Introduction A.2 Eigenvalues A.3 Vector and Matrix Norms A.4 Systems of Linear Equations A.5 Solving Systems of Linear Equations by Elimination A.6 Gaussian Elimination as a Matrix Factorization A.6.1 Sparse Matrix Storage A.7 Other Matrix Factorizations A.7.1 Positive-Definite Matrices A.7.2 The LDL T and Cholesky Factorizations A.7.3 An Orthogonal Matrix Factorization A.8 Sensitivity (Conditioning) A.9 The Sherman Morrison Formula A.10 Notes...688

8 page xii xii B Other Fundamentals 691 B.1 Introduction B.2 Computer Arithmetic B.3 Big-O Notation, O( ) B.4 The Gradient, Hessian, and Jacobian B.5 Gradient and Hessian of a Quadratic Function B.6 Derivatives of a Product B.7 The Chain Rule B.8 Continuous Functions; Closed and Bounded Sets B.9 The Implicit Function Theorem C Software 703 C.1 Software Bibliography 707 Index 727