Two-phase matrix splitting methods for asymmetric and symmetric LCP

Size: px

Start display at page:

Download "Two-phase matrix splitting methods for asymmetric and symmetric LCP"

Linda Newman
6 years ago
Views:

1 Two-phase matrix splitting methods for asymmetric and symmetric LCP Daniel P. Robinson Department of Applied Mathematics and Statistics Johns Hopkins University Joint work with Feng, Nocedal, and Pang National University of Singapore Complementarity and Its Extensions December 19, 2012

4 Outline 1 Motivation 2 Two-phase matrix splitting method for BQP 3 Two-phase matrix splitting method for LCP 4 Summary

5 Outline 1 Motivation 2 Two-phase matrix splitting method for BQP 3 Two-phase matrix splitting method for LCP 4 Summary

6 BQP (symmetric LCP) Given M R n n and q R n, solve LCP minimize x f(x) = 1 2 xt Mx + q T x subject to x 0 Given M R n n and q R n, find x such that x 0, Mx + q 0, x (Mx + q) = 0 symmetric positive definite M : x is minimizer of BQP if and only if x is a solution to LCP symmetric M : x is a first-order solution to BQP if and only if x is solution to LCP general M : no convenient relationship between BQP and LCP

7 BQP (symmetric LCP) Given M R n n and q R n, solve LCP minimize x f(x) = 1 2 xt Mx + q T x subject to x 0 Given M R n n and q R n, find x such that x 0, Mx + q 0, x (Mx + q) = 0 symmetric positive definite M : x is minimizer of BQP if and only if x is a solution to LCP symmetric M : x is a first-order solution to BQP if and only if x is solution to LCP general M : no convenient relationship between BQP and LCP

8 Bound-constrained quadratic program (BQP) The problem: minimize x f(x) def = 1 2 xt Mx + q T x subject to x 0 Applications: subproblems (e.g. Lancelot), contact problems without friction, sparse optimization ( minimize f(x) + x 1 ) Previous work: Moré and Toraldo (1989) (two-phase, gradient based, convex/nonconvex), Kočvara and Zowe (1994) (two-phase, matrix splitting based, strictly convex), Dostál and Schöberl (2005) (linear CG, projections, adaptive precision control) Linear complementarity problem (LCP) The problem: find x satisfying x 0, Mx + q 0, x (Mx + q) = 0 Applications: contact problems without friction, options pricing Previous work: Feng, Linetsky, Morales, Nocedal (2010) (two-phase, matrix splitting based, no convergence proof)

9 Bound-constrained quadratic program (BQP) The problem: minimize x f(x) def = 1 2 xt Mx + q T x subject to x 0 Applications: subproblems (e.g. Lancelot), contact problems without friction, sparse optimization ( minimize f(x) + x 1 ) Previous work: Moré and Toraldo (1989) (two-phase, gradient based, convex/nonconvex), Kočvara and Zowe (1994) (two-phase, matrix splitting based, strictly convex), Dostál and Schöberl (2005) (linear CG, projections, adaptive precision control) Linear complementarity problem (LCP) The problem: find x satisfying x 0, Mx + q 0, x (Mx + q) = 0 Applications: contact problems without friction, options pricing Previous work: Feng, Linetsky, Morales, Nocedal (2010) (two-phase, matrix splitting based, no convergence proof)

10 Bound-constrained quadratic program (BQP) The problem: minimize x f(x) def = 1 2 xt Mx + q T x subject to x 0 Applications: subproblems (e.g. Lancelot), contact problems without friction, sparse optimization ( minimize f(x) + x 1 ) Previous work: Moré and Toraldo (1989) (two-phase, gradient based, convex/nonconvex), Kočvara and Zowe (1994) (two-phase, matrix splitting based, strictly convex), Dostál and Schöberl (2005) (linear CG, projections, adaptive precision control) Linear complementarity problem (LCP) The problem: find x satisfying x 0, Mx + q 0, x (Mx + q) = 0 Applications: contact problems without friction, options pricing Previous work: Feng, Linetsky, Morales, Nocedal (2010) (two-phase, matrix splitting based, no convergence proof)

11 Outline 1 Motivation 2 Two-phase matrix splitting method for BQP 3 Two-phase matrix splitting method for LCP 4 Summary

12 BQP Given M R n n and q R n, solve minimize x Basic approach f(x) = 1 2 xt Mx + q T x subject to x 0 1 Predict active variables at a solution (cheap) - Projected SOR on system Mx = q - More generally, use splitting M = B + C with B 0 - Notation: y = FPI(x, p, B, C) means that y is the result of performing p steps of projected SOR with initial guess x and splitting M = B + C 2 Subspace phase - Accelerate convergence and pick up additional activities - For A := {i : x i = 0} and F = [1 : n]/a solve M FF x F = q F

13 BQP Given M R n n and q R n, solve minimize x Basic approach f(x) = 1 2 xt Mx + q T x subject to x 0 1 Predict active variables at a solution (cheap) - Projected SOR on system Mx = q - More generally, use splitting M = B + C with B 0 - Notation: y = FPI(x, p, B, C) means that y is the result of performing p steps of projected SOR with initial guess x and splitting M = B + C 2 Subspace phase - Accelerate convergence and pick up additional activities - For A := {i : x i = 0} and F = [1 : n]/a solve M FF x F = q F

14 BQP Given M R n n and q R n, solve minimize x Basic approach f(x) = 1 2 xt Mx + q T x subject to x 0 1 Predict active variables at a solution (cheap) - Projected SOR on system Mx = q - More generally, use splitting M = B + C with B 0 - Notation: y = FPI(x, p, B, C) means that y is the result of performing p steps of projected SOR with initial guess x and splitting M = B + C 2 Subspace phase - Accelerate convergence and pick up additional activities - For A := {i : x i = 0} and F = [1 : n]/a solve M FF x F = q F

15 Algorithm for BQP (recall: f(x) = 0.5x T Mx + q T x) 1 Compute single projected matrix splitting iteration (Mx = q) 2 Calculate Cauchy point x k,c 3 Additional matrix splitting iterations x k,f 4 Projected search in direction defined by step 3 x k,pf 5 Subspace step x k,s is computed as solution to minimize x f(x) subject to x A = 0 6 Projected search in direction defined by subspace step x k+1 z x k d k,c FPI(x k, 1, B, C) y x k,f = x k,pf x k,c x k,s x k+1 x

16 Algorithm for BQP (recall: f(x) = 0.5x T Mx + q T x) 1 Compute single projected matrix splitting iteration (Mx = q) 2 Calculate Cauchy point x k,c 3 Additional matrix splitting iterations x k,f 4 Projected search in direction defined by step 3 x k,pf 5 Subspace step x k,s is computed as solution to minimize x f(x) subject to x A = 0 6 Projected search in direction defined by subspace step x k+1 x k,s x k+1 x z x k,f = x k,pf x k,c x k d k,c FPI(x k, 1, B, C) y Other work Moré and Toraldo (more basic step 1 and modified step 2) Kočvara, Zowe (essentially use steps 3 and 5)

17 Lemma (Cauchy step is a descent direction) If M = B + C is a splitting of the symmetric matrix M R n n such that B 0, then d k,c, Mx k + q d k,c, Bd k,c 0, where d k,c is the Cauchy step. Moreover, if B is either symmetric or positive definite, and d k,c, Mx k + q = 0, then x k solves the LCP.

18 Convergence result for BQP If M = B + C is a symmetric matrix such that B 0, then every limit point of the iterates generated by our algorithm is a first-order solution to BQP. The single matrix splitting iteration in step 1 supplies a sufficient descent direction used to compute the Cauchy step in step 2 Steps 1 and 2 are needed to prove convergence Steps 3 6 generally improve performance Limit points are guaranteed if f has bounded level sets on x 0 Summary of our BQP algorithm Our method is a two-phase method based on matrix splitting iterations is provably convergent on both convex and nonconvex BQPs and the numerical results?

19 Convergence result for BQP If M = B + C is a symmetric matrix such that B 0, then every limit point of the iterates generated by our algorithm is a first-order solution to BQP. The single matrix splitting iteration in step 1 supplies a sufficient descent direction used to compute the Cauchy step in step 2 Steps 1 and 2 are needed to prove convergence Steps 3 6 generally improve performance Limit points are guaranteed if f has bounded level sets on x 0 Summary of our BQP algorithm Our method is a two-phase method based on matrix splitting iterations is provably convergent on both convex and nonconvex BQPs and the numerical results?

20 Convergence result for BQP If M = B + C is a symmetric matrix such that B 0, then every limit point of the iterates generated by our algorithm is a first-order solution to BQP. The single matrix splitting iteration in step 1 supplies a sufficient descent direction used to compute the Cauchy step in step 2 Steps 1 and 2 are needed to prove convergence Steps 3 6 generally improve performance Limit points are guaranteed if f has bounded level sets on x 0 Summary of our BQP algorithm Our method is a two-phase method based on matrix splitting iterations is provably convergent on both convex and nonconvex BQPs and the numerical results?

21 Randomly generated strictly convex problems: n = 10, 000 Projected Gradient Ours Dostál and Schöberl Cond iter nss Ax iter nsplit nss Ax iter Ax

22 Current work Dostál and Schöberl becomes superior for large condition number Uses an adaptive precision control during the subspace phase Our heuristic was to set a fixed number of CG iterations (5) and then dramatically increasing it when the active-set settles down fine if problems are well-conditioned fine if have a good pre-condioner for CG Currently combining our matrix splitting iterations with an adaptive recursive subspace phase Goal: be superior on convex problems for all condition numbers!

23 Outline 1 Motivation 2 Two-phase matrix splitting method for BQP 3 Two-phase matrix splitting method for LCP 4 Summary

24 LCP Given square matrix M R n n and vector q R n, find x such that Basic approach x 0, Mx + q 0, x (Mx + q) = 0 1 Predict active variables at a solution (cheap) - Assume splitting M = B + C is such that the fixed-point iterations are contractions, i.e., z 2 z 1 ρ z 1 z 0 for some ρ (0, 1) 2 Subspace phase - Accelerate convergence and pick up additional activities Convergence is guaranteed if the subspace step is not used (Cottle, Pang, Stone)

25 LCP Given square matrix M R n n and vector q R n, find x such that Basic approach x 0, Mx + q 0, x (Mx + q) = 0 1 Predict active variables at a solution (cheap) - Assume splitting M = B + C is such that the fixed-point iterations are contractions, i.e., z 2 z 1 ρ z 1 z 0 for some ρ (0, 1) 2 Subspace phase - Accelerate convergence and pick up additional activities Convergence is guaranteed if the subspace step is not used (Cottle, Pang, Stone)

26 LCP algorithm 1 Compute n f 2 matrix splitting iterations x k,f,n f 2 Subspace step x k,s is computed as before 3 Compute 2 additional fixed-point iterations x k,s,2 4 Update x k+1 as follows: If contraction is maintained, then x k+1 x k,s,2 If min(x, Mx + q) 2 is sufficiently small, then x k+1 x k,s,2 Otherwise, x k+1 x k z x k x k = x k,f,0 y x k,f,1 x k,s = x k,s,0 p s xk,s,1 p 1 p 2 x k,s,2 pd x k,s = x k,s,0 x k+1 = x k,s,ns x k,f,nf = FPI(x k, nf, B, C) ˆp 1 x k,f,nf 1 ˆp 2 x k,f,nf k,f,nf +1 x x

27 Convergence result for LCP Let M = B + C be a splitting of the matrix M such that the resulting matrix splitting iterations are contractions, then either x K is a solution to the LCP for some integer K 0 and the algorithm terminates, or lim inf min(x k, Mx k + q) = 0, k 0 and if the iterates are bounded, then there exists a limit point of the iterates that is a solution to LCP. contraction ensured if M is diagonally dominant more generally, contraction holds if M is an H-matrix with positive diagonals limit points are guaranteed if min(x, Mx + q) has bounded level curves on x 0 global convergence is based on contraction property of matrix splitting iteration in step 1 subspace steps accelerate convergence

28 Summary of our LCP algorithm Our method is a two-phase method based on matrix splitting iterations is provably convergent provided the matrix splitting iteration is convergent and the numerical results? American options pricing (Black-Scholes-Merton model) BSM Parameters Algorithm FLMN Our Algorithm σ T x l x u iter nsplit nss atm iter nsplit nss atm $ $ $ $ $ $ $ $24.44 Identical performance. This is good!

29 Summary of our LCP algorithm Our method is a two-phase method based on matrix splitting iterations is provably convergent provided the matrix splitting iteration is convergent and the numerical results? American options pricing (Black-Scholes-Merton model) BSM Parameters Algorithm FLMN Our Algorithm σ T x l x u iter nsplit nss atm iter nsplit nss atm $ $ $ $ $ $ $ $24.44 Identical performance. This is good!

30 Summary of our LCP algorithm Our method is a two-phase method based on matrix splitting iterations is provably convergent provided the matrix splitting iteration is convergent and the numerical results? American options pricing (Black-Scholes-Merton model) BSM Parameters Algorithm FLMN Our Algorithm σ T x l x u iter nsplit nss atm iter nsplit nss atm $ $ $ $ $ $ $ $24.44 Identical performance. This is good!

31 Current work Developing adaptive subspace phase ideas for LCP that are analogous to the ideas introduced by Dostál and Schöberl for the BQP case Can we formulate a two-phase method with subspace acceleration that has convergence guarantees under weaker assumptions (no contraction assumption)

32 Outline 1 Motivation 2 Two-phase matrix splitting method for BQP 3 Two-phase matrix splitting method for LCP 4 Summary

33 Summary Matrix splitting iterations may be used (in different ways) to efficiently solve BQPs and LCPs Our algorithm for BQP is a two-phase method based on matrix splitting iterations and is provably convergent on both convex and nonconvex problems Matrix splitting iterations for BQP are generally superior to simple gradient directions for identifying an optimal active set Our algorithm for LCP is a two-phase method based on matrix splitting iterations and is provably convergent (provided the matrix splitting iteration is convergent) The more sophisticated matrix splitting iterations, e.g. Gauss-Seidel, can be substantially more expensive than gradient directions Generally, the subspace phase greatly reduces the number of iterations required, but comes with additional cost Combining our ideas with those by Dostál and Schöberl seems promising for both BQP and LCP

34 My can we list Can we extend ideas presented here to the case of minimizing general quadratic programs? Can we weaken the assumptions needed to solve LCPs by considering semismooth Newton methods? Can we improve the subspace phase? Can we develop a rapidly adapting quadratic programming active-set method based on work by Hintermuler, Ito, and Kunisch for solving quadratic programs? Can we develop conditions/subproblems that allow very early termination of sequential quadratic programming methods? Can we solve nonsmooth optimization problems by combining sampling with bundle methods? Can we contribute to the new area of differential variational inqualities? Can we steer augmented Lagrangian methods? Can we mitigate degeneracy for general NLPs.

35 References Moré and Toraldo Algorithms for bound constrained quadratic programming problems Kočvara and Zowe An iterative two-step algorithm for linear complementarity problems Feng, Linetsky, Morales, and Nocedal On the solution of complementarity problems arising in American options pricing

Sparse Optimization Lecture: Proximal Operator/Algorithm and Lagrange Dual

Sparse Optimization Lecture: Proximal Operator/Algorithm and Lagrange Dual Instructor: Wotao Yin July 2013 online discussions on piazza.com Those who complete this lecture will know learn the proximal