An augmented Lagrangian method for equality constrained optimization with fast infeasibility detection

Transcription

1 An augmented Lagrangian method for equality constrained optimization with fast infeasibility detection Paul Armand 1 Ngoc Nguyen Tran 2 Institut de Recherche XLIM Université de Limoges Journées annuelles 2017 des GdR MOA et MIA Institut de Mathématiques de Bordeaux Octobre, Phd Supervisor 2 Phd Student Paul Armand & Ngoc Nguyen Tran An augmented Lagrangian method with fast ID Octobre, / 22

2 1 Introduction 2 Algorithm 3 Global convergence analysis 4 Asymptotic analysis 5 Numerical experiments 6 Future work Paul Armand & Ngoc Nguyen Tran An augmented Lagrangian method with fast ID Octobre, / 22

3 Motivation Introduction We consider the equality constrained problem of the form min f (x) s.t. c(x) = 0, (P) where f : R n R, c : R n R m are smooth functions. Optimization algorithms try to find local solution of (P). In case the algorithm do not find a feasible solution, it should return a stationary point of the feasibility problem to avoid long sequences of iterations. min x R n c(x), Rapid infeasibility detection plays a central role in branch-and-bound methods for mixed-integer nonlinear programming, in parametric studies of optimization models, in searching global minimizers... Paul Armand & Ngoc Nguyen Tran An augmented Lagrangian method with fast ID Octobre, / 22

4 Introduction Introduction Instead of solving min f (x) s.t. c(x) = 0, (P) we consider the problem (P ρ ) min ρf (x) s.t. c(x) = 0, (P ρ ) where ρ > 0 is called the feasibility parameter. Any feasible solution is optimal for (P 0 ). Paul Armand & Ngoc Nguyen Tran An augmented Lagrangian method with fast ID Octobre, / 22

5 Optimality conditions Introduction Augmented Lagrangian associated to (P ρ ): L ρ,σ (x, λ) := ρf (x) + λ c(x) + 1 2σ c(x) 2, where σ > 0 is a quadratic penalty parameter, λ R m is the vector of Lagrange multipliers. (x, λ ) is a solution of (P ρ ) x is a strict local minimum of L ρ,σ (, λ ) for σ small enough. 1st order optimality conditions for minimizing L ρ,σ (, λ) : ( ) ρg(x) + A(x)y Φ(w, λ, ρ, σ) := = 0 c(x) + σ(λ y) where g := f, A := c and w := (x, y). Paul Armand & Ngoc Nguyen Tran An augmented Lagrangian method with fast ID Octobre, / 22

6 State of the art Introduction Augmented Lagrangian (AL) method was proposed independently by Hestenes (1969) and Powell (1969). Main idea: solve a sequence of unconstrained problems with an update strategy of the parameters. Implementations: LANCELOT (1992), ALGENCAN (2007; 2008) and SPDOPT (2017a; 2017b). Some AL algorithms with infeasibility detection capabilities: Martínez and Prudente (2012), Birgin et al. (2015), Birgin et al. (2014), Gonçalves et al. (2015), Armand and Omheni (2017a)). Fast local convergence in the infeasible case has not been established for these AL algorithms. Our goal is to propose a new algorithm rapidly converging in the infeasible case. Paul Armand & Ngoc Nguyen Tran An augmented Lagrangian method with fast ID Octobre, / 22

7 Introduction Main properties of our new algorithm Newton-type method: Solve the linear system J ρk,σ k,θ k (w k )(w + k w k) = Φ(w k, λ k+1, ρ k+1, σ k+1 ), (1) where ( ) 2 J ρ,σ,θ (w) = xx L ρ,σ(w) + θi A(x) A(x) σi and Φ(w, λ, ρ, σ) := ( ) ρg(x) + A(x)y. c(x) + σ(λ y) If the progress towards the feasibility is not sufficient, the algorithm progressively switches to the solution of by driving ρ and λ to zero. 1 min x R n 2 c(x) 2, Dynamic update of ρ to get a fast convergence to an infeasible stationary point. Paul Armand & Ngoc Nguyen Tran An augmented Lagrangian method with fast ID Octobre, / 22

8 Algorithm Algorithm: Outer iterations Choose ɛ > 0, a (0, 1), τ (0, 1). Set k = 0, i 0 = 0, F = 1. 1 If c(x k ) ɛ, set F = 0. 2 Choose ζ k > 0 such that {ζ k } 0. If c(x k ) a c(x ik ) + ζ k, go to Step 4. 3 If F = 1, choose 0 < ρ k+1 τρ k and set σ k+1 = σ k, else choose 0 < σ k+1 τσ k and set ρ k+1 = ρ k. Set λ k+1 = ρ k+1 ρ k λ k, i k+1 = i k and go to Step 5. 4 Choose 0 < σ k+1 σ k. Set λ k+1 = y k, ρ k+1 = ρ k, i k+1 = k. 5 Choose θ k > 0 such that Inertia(J ρk,σ k,θ k (w k )) = (n, m, 0) and compute w + k by solving the linear system (1). 6 Choose ε k > 0 such that {ε k } 0. If Φ(w + k, λ k+1, ρ k+1, σ k+1 ) ε k, set w k+1 = w + k. Otherwise, apply a sequence of inner iterations to find w k+1 such that Φ(w k+1, λ k+1, ρ k+1, σ k+1 ) ε k. Paul Armand & Ngoc Nguyen Tran An augmented Lagrangian method with fast ID Octobre, / 22

9 Algorithm Algorithm: Inner iterations The aim of the inner iterations is to minimize the merit function: ϕ(w) = ρf (x) + λ c(x) + 1 2σ c(x) 2 + ν 2σ c(x) σ(λ y) 2, where σ = σ k+1, ρ = ρ k+1, λ = λ k+1 are fixed and ν > 0. 1st order optimality conditions of min ϕ(w) : Φ(w, λ, ρ, σ) = 0. Similar to (Armand and Omheni, 2017a, Algorithm 2) Paul Armand & Ngoc Nguyen Tran An augmented Lagrangian method with fast ID Octobre, / 22

10 Global convergence analysis Stationary points Definition x R n is a Fritz-John (FJ) point of problem (P) if there exists (z, y) R + R m and (z, y) (0, 0) such that zg(x) + A(x)y = 0 and c(x) = 0. If z > 0, x is a Karush-Kuhn-Tucker point. If z = 0, x is a singular stationary point (LICQ does not hold). x R n is an infeasible stationary point of problem (P) if c(x) 0 and A(x)c(x) = 0. Paul Armand & Ngoc Nguyen Tran An augmented Lagrangian method with fast ID Octobre, / 22

11 Global convergence analysis Global convergence analysis: outer iterations Lemma Assume that Algorithm generates an infinite sequence {w k }. Let K N be the set of iteration indices for which the condition checking the feasibility in Step 2 is satisfied. (i) If K =, then the subsequence {c k } k K 0 and {ρ k } is eventually constant. (ii) If K <, then lim inf c k > 0, {σ k ρ k } 0 and {σ k λ k } 0. Theorem 1 Assume that Algorithm generates an infinite sequence {w k } such that the sequence {x k } lies in a compact set. (i) Any feasible limit point of {x k } is a FJ point of (P). (ii) If the sequence {x k } has no feasible limit point, then any limit point is an infeasible stationary point of problem (P). 1 Related results: Armand and Omheni (2017a), Armand and Omheni (2017b). Paul Armand & Ngoc Nguyen Tran An augmented Lagrangian method with fast ID Octobre, / 22

12 Global convergence analysis Example of infeasible stationary point min x 1 s.t. x 2 1 x = 0, x x = 0, x 0 = (5, 2), y ( 0 = (1, 1), 2x1 (2x A(x)c(x) = ) ), 2x 2 x = (0, 0), y = (2035.9, ). Example of infeasible stationary point L Ac c σ ρ Iterations Paul Armand & Ngoc Nguyen Tran An augmented Lagrangian method with fast ID Octobre, / 22

13 Asymptotic analysis Asymptotic result near an infeasible stationary point Assumptions 1 {w k } w = (x, y ), where x is an infeasible stationary point of problem (P). 2 2 f and 2 c are Lipschitz continuous over an open neighborhood of x. 3 The second order sufficient optimality conditions hold at x for the feasibility problem min x R n c(x) 2. 1 = k 0, k k 0, σ k = σ k0 and {ρ k } 0. For w := (x, y) R n+m, we define F (w) = (A(x)y, c(x) σ k0 y). Theorem (Rate of convergence in the infeasible case) Assume that all above assumptions hold. Let t (0, 2]. If the feasibility parameter is chosen so that ρ k+1 = O( F (w k ) t ), then w k+1 w = O( w k w t ). (2) In addition, if ρ k+1 = Θ( F k t ) and if ε k = Ω(ρ t k ) for 0 < t < t, then for k large enough there is no inner iterations, i.e., w k+1 = w + k. Paul Armand & Ngoc Nguyen Tran An augmented Lagrangian method with fast ID Octobre, / 22

14 Asymptotic analysis Rate of convergence in the infeasible case min x 1 s.t. x 2 1 x = 0, x x = 0, x 0 = (5, 2), y ( 0 = (1, 1), 2x1 (2x A(x)c(x) = ) ), 2x 2 x = (0, 0), y = (2035.9, ) F 10 6 t=2 t=1.6 t=1.2 t= Iterations Paul Armand & Ngoc Nguyen Tran An augmented Lagrangian method with fast ID Octobre, / 22

15 Numerical experiments Implementation of SPDOPT-ID The stopping criterion: Optimal solution: (g k + A k y k /ρ k, c k ) 10 8 Infeasible stationary point: (A k y k, c k σ k y k ) 10 8 and ρ k The update condition: SPDOPT-ID: Set η k = c(x k ) and check c(x k ) d max{η ij : (k l 1) + j k} + 10σ k ρ k. SPDOPT-IDOld: Set η k = c(x k ) + 10σ k ρ k and check c(x k ) d max{η ij : (k l 1) + j k}. Paul Armand & Ngoc Nguyen Tran An augmented Lagrangian method with fast ID Octobre, / 22

16 Numerical experiments Results on standard problems Results of 128 problems with only equality constraints from CUTEr 3 with 2 n 20192, 1 m Performance profiles comparing number of function evaluations and number of gradient evaluations for SPDOPT-AL (Armand and Omheni (2017a)), SPDOPT-ID and SPDOPT-IDOld. 1 Function evaluations 1 Gradient evaluations ρs(τ) 0.9 ρs(τ) τ SPDOPT AL SPDOPT ID SPDOPT IDOld τ SPDOPT AL SPDOPT ID SPDOPT IDOld 3 Gould, Orban and Toint Paul Armand & Ngoc Nguyen Tran An augmented Lagrangian method with fast ID Octobre, / 22

17 Numerical experiments Results on infeasible problems Results of 127 problems with only equality constraints from CUTEr with the additional infeasible constraint c 1(x) = 0. Performance profiles comparing number of function evaluations and number of gradient evaluations for SPDOPT-AL, SPDOPT-ID and SPDOPT-IDOld. 1 Function evaluations 1 Gradient evaluations ρs(τ) 0.5 ρs(τ) τ SPDOPT AL SPDOPT ID SPDOPT IDOld τ SPDOPT AL SPDOPT ID SPDOPT IDOld Paul Armand & Ngoc Nguyen Tran An augmented Lagrangian method with fast ID Octobre, / 22

18 Numerical experiments Results on both sets Results of 255 problems from the set of standard problems and infeasible problems. Performance profiles comparing number of function evaluations and number of gradient evaluations for SPDOPT-AL, SPDOPT-ID and SPDOPT-IDOld. 1 Function evaluations 1 Gradient evaluations ρs(τ) 0.5 ρs(τ) τ SPDOPT AL SPDOPT ID SPDOPT IDOld τ SPDOPT AL SPDOPT ID SPDOPT IDOld Paul Armand & Ngoc Nguyen Tran An augmented Lagrangian method with fast ID Octobre, / 22

19 Future work Future work We develop this work to the general nonlinear optimization problem min f (x) s.t. c(x) = 0, x 0. Specifically, we consider the unconstrained problem below min x R n x>0 ϕ λ,ρ,σ,µ (x) := ρf (x) + λ c(x) + 1 2σ c(x) 2 ρµ n ln[x] i, i=1 where λ R m is an estimate of the vector of Lagrange multipliers associated with the equality constraints, ρ > 0 is the feasibility parameter, σ > 0 is the quadratic penalty parameter and µ > 0 is the barrier parameter. With a suitable update of parameters, the sequence of iterates converges to an infeasible stationary point superlinearly. Paul Armand & Ngoc Nguyen Tran An augmented Lagrangian method with fast ID Octobre, / 22

20 References I Future work R. Andreani, E. G. Birgin, J. M. Martínez, and M. L. Schuverdt. On Augmented Lagrangian Methods with General Lower-Level Constraints. SIAM Journal on Optimization, 18(4): , R. Andreani, E. G. Birgin, J. M. Martínez, and M. L. Schuverdt. Augmented Lagrangian methods under the constant positive linear dependence constraint qualification. Mathematical Programming, 111(1):5 32, Paul Armand and Riadh Omheni. A globally and quadratically convergent primal-dual augmented lagrangian algorithm for equality constrained optimization. Optimization Methods and Software, 32(1):1 21, 2017a. Paul Armand and Riadh Omheni. A mixed logarithmic barrier-augmented lagrangian method for nonlinear optimization. Journal of Optimization Theory and Applications, 2017b. doi: /s x. E. G. Birgin, J. M. Martínez, and L. F. Prudente. Augmented lagrangians with possible infeasibility and finite termination for global nonlinear programming. Journal of Global Optimization, 58(2): , Paul Armand & Ngoc Nguyen Tran An augmented Lagrangian method with fast ID Octobre, / 22

21 References II Future work E. G. Birgin, J. M. Martínez, and L. F. Prudente. Optimality properties of an augmented lagrangian method on infeasible problems. Computational Optimization and Applications, 60(3): , A. R. Conn, N. I. M. Gould, and P. L. Toint. LANCELOT: A Fortran Package for Large-Scale Nonlinear Optimization (Release A). Number 17 in Springer Series in Computational Mathematics. Springer-Verlag, New York, M. L. N. Gonçalves, J. G. Melo, and L. F. Prudente. Augmented lagrangian methods for nonlinear programming with possible infeasibility. Journal of Global Optimization, 63(2): , M. R. Hestenes. Multiplier and gradient methods. Journal of Optimization Theory and Applications, 4(5): , Jose Mario Martínez and Leandro da Fonseca Prudente. Handling infeasibility in a large-scale nonlinear optimization algorithm. Numerical Algorithms, 60(2): , M. J. D. Powell. A method for nonlinear constraints in minimization problems. In Optimization (Sympos., Univ. Keele, Keele, 1968), pages Academic Press, London, Paul Armand & Ngoc Nguyen Tran An augmented Lagrangian method with fast ID Octobre, / 22

22 Future work Thank you for your attention! Paul Armand & Ngoc Nguyen Tran An augmented Lagrangian method with fast ID Octobre, / 22