Augmented Lagrangian Methods

Transcription

1 Augmented Lagrangian Methods Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. IMA, August 2016 Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

2 Minimization with Linear Constraints: Basics Consider the linearly constrained problem, min f () s.t. A = b, where f : R n R is smooth. How do we recognize that a point is a solution of this problem? (Such optimality conditions can provide the foundation for algorithms.) Karush-Kuhn-Tucker (KKT) condition is a first-order necessary condition. If is a local solution, there eists a vector of Lagrange multipliers λ R m such that f ( ) = A T λ, A = b. When f is smooth and conve, these conditions are also sufficient. (In fact, it s enough for f to be conve on the null space of A.) Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

3 Minimization with Linear Constraints Define the Lagrangian function: L(, λ) := f () + λ T (A b). Can write the KKT conditions in terms of L as follows: [ L(, λ L( ) =, λ ] ) λ L(, λ = 0. ) Suppose now that f is conve but not smooth. First-order optimality conditions (necessary and sufficient) are that there eists λ R m such that A T λ f ( ), A = b, where f is the subdifferential. In terms of the Lagrangian, we have 0 L(, λ ), λ L(, λ ) = 0. Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

4 Augmented Lagrangian Methods With f proper, lower semi-continuous, and conve, consider: min f () s.t. A = b. The augmented Lagrangian is (with ρ > 0) L(, λ; ρ) := f () + λ T (A b) + ρ }{{} 2 A b 2 2. }{{} Lagrangian augmentation Basic augmented Lagrangian (a.k.a. method of multipliers) is (Hestenes, 1969; Powell, 1969) k = arg min L(, λ k 1 ; ρ); λ k = λ k 1 + ρ(a k b); Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

5 A Favorite Derivation...more or less rigorous for conve f. Write the problem as min ma λ f () + λ T (A b). Obviously, the ma w.r.t. λ will be +, unless A = b, so this is equivalent to the original problem. This equivalence is not very useful, computationally: the ma λ function is highly nonsmooth w.r.t.. Smooth it by adding a proimal point term, penalizing deviations from a prior estimate λ: min { ma f () + λ T (A b) 1 2 λ λ λ 2ρ }. Maimization w.r.t. λ is now trivial (a concave quadratic), yielding λ = λ + ρ(a b). Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

6 A Favorite Derivation (Cont.) Inserting λ = λ + ρ(a b) leads to min f () + λ T (A b) + ρ 2 A b 2 = L(, λ; ρ). Hence can view the augmented Lagrangian process as: min L(, λ; ρ) to get new ; Shift the prior on λ by updating to the latest ma: λ + ρ(a b). repeat until convergence. Add subscripts, and we recover the augmented Lagrangian algorithm of the first slide! Can also increase ρ (to sharpen the effect of the pro term), if needed. Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

7 Inequality Constraints, Nonlinear Constraints The same derivation can be used for inequality constraints: min f () s.t. A b. Apply the same reasoning to the constrained min-ma formulation: min ma f () λ 0 λt (A b). After the pro-term is added, can find the minimizing λ in closed form (as for pro-operators). Leads to update formula: ma ( λ + ρ(a b), 0 ). This derivation etends immediately to nonlinear constraints c() = 0 or c() 0. Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

8 Eplicit Constraints, Inequality Constraints There may be other constraints on (such as Ω) that we prefer to handle eplicitly in the subproblem. For the formulation min f (), s.t. A = b, Ω, the min step can enforce Ω eplicitly: k = arg min Ω L(, λ k 1; ρ); λ k = λ k 1 + ρ(a k b); This gives an alternative way to handle inequality constraints: introduce slacks s, and enforce them eplicitly. That is, replace by min f () s.t. c() 0, min f () s.t. c() = s, s 0. Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

9 Eplicit Constraints, Inequality Constraints (Cont.) The augmented Lagrangian is now L(, s, λ; ρ) := f () + λ T (c() s) + ρ 2 c() s 2 2. Enforce s 0 eplicitly in the subproblem: ( k, s k ) = arg min,s L(, s, λ k 1; ρ), s.t. s 0; λ k = λ k 1 + ρ(c( k ) s k ) There are good algorithmic options for dealing with bound constraints s 0 (gradient projection and its enhancements). This is used in the Lancelot code (Conn et al., 1992). Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

10 Quick History of Augmented Lagrangian Dates from at least 1969: Hestenes, Powell. Developments in 1970s, early 1980s by Rockafellar, Bertsekas, and others. Lancelot code for nonlinear programming: Conn, Gould, Toint, around 1992 (Conn et al., 1992). Lost favor somewhat as an approach for general nonlinear programming during the net 15 years. Recent revival in the contet of sparse optimization and its many applications, in conjunction with splitting / coordinate descent. Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

11 Alternating Direction Method of Multipliers (ADMM) Consider now problems with a separable objective of the form min f () + h(z) s.t. A + Bz = c, (,z) for which the augmented Lagrangian is L(, z, λ; ρ) := f () + h(z) + λ T (A + Bz c) + ρ 2 A Bz c 2 2. Standard AL would minimize L(, z, λ; ρ) w.r.t. (, z) jointly. However, since coupled in the quadratic term, separability is lost. In ADMM, minimize over and z separately and sequentially: k = arg min L(, z k 1, λ k 1 ; ρ); z k = arg min L( k, z, λ k 1 ; ρ); z λ k = λ k 1 + ρ(a k + Bz k c). Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

12 ADMM Main features of ADMM: Does one cycle of block-coordinate descent in (, z). The minimizations over and z add only a quadratic term to f and h, respectively. Usually does not alter the cost much. Can perform the (, z) minimizations ineactly. Can add eplicit (separated) constraints: Ω, z Ω z. Many (many!) recent applications to compressed sensing, image processing, matri completion, sparse principal components analysis... ADMM has a rich collection of antecendents, dating even to the 1950s (operator splitting). For an comprehensive recent survey, including a diverse collection of machine learning applications, see Boyd et al. (2011). Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

13 ADMM for Consensus Optimization Given the unconstrained (but separable) problem m min f i (), form m copies of the, with the original as a master variable: m f i ( i ) subject to i = 0, i = 1, 2,..., m. min, 1, 2,..., m i=1 i=1 Apply ADMM, with z = ( 1, 2,..., m ). Get m L(, 1, 2,..., m, λ 1,..., λ m ; ρ) = f i ( i )+(λ i ) T ( i )+ ρ 2 i 2 2. i=1 The minimization w.r.t. z = ( 1, 2,..., m ) is separable! k i = arg min f i ( i )+(λ i i k 1 )T ( i k 1 )+ ρ k 2 i k 1 2 2, i = 1, 2,..., m. Can be implemented in parallel. Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

14 Consensus, continued The minimization w.r.t. can be done eplicitly averaging: k = 1 m m i=1 ( k i + 1 ) λ i k 1. ρ k Update to λ i can also be done in parallel, once the new k is known (and communicated): λ i k = λi k 1 + ρ k( i k k), i = 1, 2,..., m. If the initial λ i 0 have m i=1 λi 0 =, can see that m iterations k. Can simplify the update for k : i=1 λi k = 0 at all k = 1 m m k i. i=1 Gather-Scatter implementation. Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

15 ADMM for Awkward Intersections The feasible set is sometimes an intersection of two or more conve sets that are easy to handle separately (e.g. projections are easily computable), but whose intersection is more difficult to work with. Eample: Optimization over the cone of doubly nonnegative matrices: General form: min X f (X ) s.t. X 0, X 0. min f () s.t. Ω i, i = 1, 2,..., m Again, use a different copy i for each set, and constrain them all to be the same: min f () s.t. i Ω i, i = 0, i = 1, 2,..., m., 1, 2,..., m Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

16 ADMM for Awkward Intersections Separable minimizations over Ω i, i = 1, 2,..., m: i k = arg min i Ω i (λ i k 1 )T ( i k 1 ) + ρ k 2 k i 2 2, i = 1, 2,..., m. Optimize over the master variable (unconstrained, with quadratic added to f ): k = arg min f () + Update multipliers: m i=1 (λ i k 1 )T ( i k 1 ) + ρ k 2 i k 1 2 2, λ i k = λi k 1 + ρ k( k k i ), i = 1, 2,..., m. Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

17 ADMM: A Simpler Form Often, a simpler version is enough: min f () + h(z) s.t. A = z, (,z) equivalent to min f () + h(a), often the one of interest. In this case, the ADMM can be written as k = arg min f () + ρ 2 A z k 1 d k z k = arg min z h(z) + ρ 2 A k 1 z d k d k = d k 1 (A k z k ) the so-called scaled version (Boyd et al., 2011). Updating z k is a proimity computation: z k = pro h/ρ ( A k 1 d k 1 ) Updating k may be hard: if f is quadratic, involves matri inversion; if f is not quadratic, may be as hard as the original problem. Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

18 ADMM: Convergence Consider the problem min f () + h(a), where f and h are lower semi-continuous, proper, conve functions and A has full column rank. The ADMM algorithm presented in the previous slide converges (for any ρ > 0) to a solution, if one eists, otherwise it diverges. This is a cornerstone result by Eckstein and Bertsekas (1992). As in IST/FBS/PGA, convergence is still guaranteed with ineactly solved subproblems, as long as the errors are absolutely summable. The recent eplosion of interest in ADMM is clear in the citation records of the review paper of Boyd et al. (2011) (2800 and counting) and of the paper by Eckstein and Bertsekas (1992): Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

19 ADMM for a More General Problem Consider the problem min R n J g j (H (j) ), where H (j) R p j n, i=1 and g 1,..., g J are l.s.c proper conve fuctions. Map it into min f () + h(a) as follows (with p = p p J ): f () = 0 A = H (1). H (J) h : R p 1+ +p J R p n, z (1) R, h. = z (J) J g j (z (j) ) j=1 This leads to a convenient version of ADMM. Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

20 ADMM for a More General Problem (Cont.) Resulting instance of k = arg min z (1) k z (J) k ( J ) Az z k d k 2 1 ( J ) 2 = (H (j) ) T H (j) (H (j) ) T (z (j) k 1 + d (j) k 1 ) j=1 = arg min g 1 + ρ u 2 u H(1) k 1 + d (1) ( k = pro g1 /ρ H (1) k 1 d (1) ) k = arg min g J + ρ u 2 u H(J) k 1 + d (J) ( k = pro gj /ρ H (J) k 1 d (J) ) k 1 d k = d k 1 A k + z k Key features: matrices are handled separately from the pro operators; the pro operators are decoupled (can be computed in parallel); requires a matri inversion (can be a curse or a blessing). (Afonso et al., 2010; Setzer et al., 2010; Combettes and Pesquet, 2011) j=1 Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

21 Special Case: l 2 -l 1 Standard problem: min 1 2 A b In this case, the ADMM becomes Subproblems are k = arg min 1 + ρ 2 A z k 1 d k z k = arg min z h(z) + ρ 2 A k 1 z d k d k = d k 1 (A k z k ) k := (A T A + ρ k I ) 1 (A T b + ρ k z k 1 λ k ), z k := min z τ z 1 + (λ k ) T ( k z) + ρ k 2 z k 2 2 = pro τ/ρk ( k + λ k /ρ k ) λ k+1 := λ k + ρ k ( k z k ). Solving for k is the most complicated part of the calculation. If the least-squares part is underdetermined (A is m n with n > m), can make Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

22 Moreover, in some compressed sensing applications, we have AA T = I. In this case, k can be recovered at the cost of two matri-vector multiplications involving A. Otherwise, can solve for k ineactly, using a few steps of an iterative method. The YALL1 code solves this problem, and other problems with more general regularizers (e.g. groups). Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

23 ADMM and Pro-Linear Algorithms The subproblems are not too different from those obtained in pro-linear algorithms (e.g. SpaRSA): λ k is asymptotically similar to the gradient term in pro-linear, that is, λ k f ( k ); Thus, the minimization over z is quite similar to the pro-linear step. Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

24 ADMM for Sparse Inverse Covariance Reformulate as ma log det(x ) X, S τ X 1, X 0 Subproblems are: ma log det(x ) X, S τ Z 1 s.t. X Z = 0. X 0 X k := arg ma X log det(x ) X, S U k 1, X Z k 1 ρ k 2 X Z k 1 2 F := arg ma log det(x ) X, S ρ k X 2 X Z k 1 + U k /ρ k 2 F Z k := pro τ/ρk (X k + U k ); U k+1 := U k + ρ k (X k Z k ). Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

25 Solving for X Get optimality condition for the X subproblem by using X log det(x ) = X 1, when X is s.p.d. Thus, which is equivalent to Form eigendecomposition X 1 S ρ k (X Z k 1 + U k /ρ k ) = 0, X 1 ρ k X (S ρ k Z k 1 + U k ) = 0. (S ρ k Z k 1 + U k ) = QΛQ T, where Q is n n orthogonal and Λ is diagonal with elements λ i. Seek X with the form Q ΛQ T, where Λ has diagonals λ i. Must have 1 ρ k λ i λ i = 0, i = 1, 2,..., n. λ i Take positive roots: λ i = [λ i + λ 2 i + 4ρ k ]/(2ρ k ), i = 1, 2,..., n. Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

26 Further Reading S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, Distributed optimization and statistical learning via the alternating direction methods of multipliers, Foundations and Trends in Machine Learning, 3, pp , S. Boyd, Alternating Direction Method of Multipliers, Talk at NIPS Workshop on Optimization and Machine Learning, December 2011: videolectures.net/nipsworkshops2011 boyd multipliers/ J. Eckstein and D. P. Bertsekas, On the Douglas-Rachford splitting method and the proimal point algorithm for maimal monotone operators, Mathematical Programming, 55, pp , W. Deng, W. Yin, and Y. Zhang, Group sparse optimization by alternating direction method, CAAM Department, Rice University, See also yall1.blogs.rice.edu Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27

27 References I Afonso, M., Bioucas-Dias, J., and Figueiredo, M. (2010). Fast image recovery using variable splitting and constrained optimization. IEEE Transactions on Image Processing, 19: Boyd, S., Parikh, N., Chu, E., Peleato, B., and Eckstein, J. (2011). Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning, 3(1): Combettes, P. and Pesquet, J.-C. (2011). Signal recovery by proimal forward-backward splitting. In Fied-Point Algorithms for Inverse Problems in Science and Engineering, pages Springer. Conn, A., Gould, N., and Toint, P. (1992). LANCELOT: a Fortran package for large-scale nonlinear optimization (Release A). Springer Verlag, Heidelberg. Eckstein, J. and Bertsekas, D. (1992). On the Douglas-Rachford splitting method and the proimal point algorithm for maimal monotone operators. Mathematical Programming, 5: Hestenes, M. (1969). Multiplier and gradient methods. Journal of Optimization Theory and Applications, 4: Powell, M. (1969). A method for nonlinear constraints in minimization problems. In Fletcher, R., editor, Optimization, pages Academic Press, New York. Setzer, S., Steidl, G., and Teuber, T. (2010). Deblurring poissonian images by split bregman techniques. Journal of Visual Communication and Image Representation, 21: Stephen Wright (UW-Madison) Augmented Lagrangian IMA, August / 27