Conic Optimization via Operator Splitting and Homogeneous Self-Dual Embedding

Transcription

1 Conic Optimization via Operator Splitting and Homogeneous Self-Dual Embedding B. O Donoghue E. Chu N. Parikh S. Boyd Convex Optimization and Beyond, Edinburgh, 11/6/2104 1

2 Outline Cone programming Homogeneous embedding Operator splitting Numerical results Conclusions Cone programming 2

3 Cone programming minimize c T x subject to Ax + s = b, s K variables x R n and (slack) s R m K is a proper convex cone K nonnegative orthant LP K Lorentz cone SOCP K positive semidefinite matrices SDP the modern canonical form for convex optimization popularized by Nesterov, Nemirovsky, others, in 1990s Cone programming 3

4 Cone programming parser/solvers like CVX, CVXPY, YALMIP translate or canonicalize to cone problems focus has been on symmetric self-dual cones for medium scale problems with enough sparsity, interior-point methods reliably attain high accuracy but they scale superlinearly in problem size open source software (SDPT3, SeDuMi,... ) widely used Cone programming 4

5 This talk a new first order method that solves general cone programs finds primal and dual solutions, or certificate of primal/dual infeasibility obtains modest accuracy quickly scales to large problems and is easy parallelized is matrix-free: only requires z Az, w A T w Cone programming 5

6 Some previous work projected subgradient type methods (Polyak 1980s) primal-dual subgradient methods (Chambelle-Pock 2011) matrix-free interior-point methods (Gondzio 2012) can use iterative linear solver (CG) in any interior-point method Cone programming 6

7 Outline Cone programming Homogeneous embedding Operator splitting Numerical results Conclusions Homogeneous embedding 7

8 Primal-dual cone problem pair primal and dual cone problems: minimize c T x subject to Ax + s = b (x, s) R n K maximize b T y subject to A T y + r = c (r, y) {0} n K primal variables x R n, s R m ; dual variables r R n, y R m K is dual of closed convex proper cone K note that R n K and {0} n K are dual cones Homogeneous embedding 8

9 Example cones K is typically a Cartesian product of smaller cones, e.g., R, {0}, R + second-order cone Q = {(x, t) R k+1 x 2 t} positive semidefinite cone {X S k X 0} exponential cone cl{(x, y, z) R 3 y > 0, e x/y z/y} these cones would handle almost all convex problems that arise in applications Homogeneous embedding 9

10 Optimality conditions KKT conditions (necessary and sufficient, assuming strong duality): primal feasibility: Ax + s = b, s K dual feasibility: A T y + c = r, r = 0, y K complementary slackness: y T s = 0 equivalent to zero duality gap: c T x + b T y = 0 Homogeneous embedding 10

11 Primal-dual embedding KKT conditions as feasibility problem: find that satisfy (x, s, r, y) R n K {0} n K r 0 A T [ ] c s = A 0 x + b 0 c T b T y 0 reduces solving cone program to finding point in intersection of cone and affine set no solution if primal or dual problem infeasible/unbounded Homogeneous embedding 11

12 Homogeneous self-dual (HSD) embedding (Ye, Todd, Mizuno, 1994) find nonzero that satisfy (x, s, r, y) R n K {0} n K, τ 0, κ 0 r 0 A T c x s = A 0 b y κ c T b T 0 τ this feasibility problem is homogeneous and self-dual τ = 1, κ = 0 reduces to primal-dual embedding due to skew symmetry, any solution satisfies (x, y,τ) (r, s,κ), τκ = 0 Homogeneous embedding 12

13 Recovering solution or certificates any HSD solution (x, s, r, y,τ,κ) falls into one of three cases: 1. τ > 0, κ = 0: (ˆx, ŷ, ŝ) = (x/τ, y/τ, s/τ) is a solution 2. τ = 0, κ > 0: in this case c T x + b T y < 0 if b T y < 0, then ŷ = y/( b T y) certifies primal infeasibility if c T x < 0, then ˆx = x/( c T x) certifies dual infeasibility 3. τ = κ = 0: nothing can be said about original problem (a pathology) Homogeneous embedding 13

14 Homogeneous primal-dual embedding HSD embedding obviates need for phase I / phase II solves to handle infeasibility/unboundedness is used in all interior-point cone solvers is a particularly nice form to solve (for reasons not completely understood) Homogeneous embedding 14

15 Notation define x r 0 A T c u = y, v = s, Q = A 0 b τ κ c T b T 0 HSD embedding is: find (u, v) that satisfy v = Qu, (u, v) C C with C = R n K R + Homogeneous embedding 15

16 Outline Cone programming Homogeneous embedding Operator splitting Numerical results Conclusions Operator splitting 16

17 Consensus problem consensus problem: minimize f(x)+g(z) subject to x = z f, g convex, not necessarily smooth, can take infinite values p is optimal objective value Operator splitting 17

18 Alternating direction method of multipliers ADMM is: for k = 0,..., ( x k+1 = argmin x ρ > 0 step-size z k+1 = argmin z λ k+1 = λ k x k+1 + z k+1 f(x)+(ρ/2) x z k λ k 2 2 ( g(z)+(ρ/2) x k+1 z λ k 2 2 λ (scaled) dual variable for x = z constraint same as many other operator splitting methods for consensus problem, e.g., Douglas-Rachford method ) ) Operator splitting 18

19 Convergence of ADMM under benign conditions ADMM guarantees: f(x k )+g(z k ) p λ k λ, an optimal dual variable x k z k 0 Operator splitting 19

20 ADMM applied to HSD embedding HSD in consensus form minimize I C C (u, v)+i Qũ=ṽ (ũ, ṽ) subject to (u, v) = (ũ, ṽ) I S is indicator function of set S ADMM is: (ũ k+1, ṽ k+1 ) = Π Qu=v (u k +λ k, v k +µ k ) u k+1 = Π C (ũ k+1 λ k ) v k+1 = Π C (ṽ k+1 µ k ) λ k+1 = λ k ũ k+1 + u k+1 µ k+1 = µ k ṽ k+1 + v k+1 Π S (x) is Euclidean projection of x onto S Operator splitting 20

21 Simplifications (straightforward, but not immediate) if λ 0 = v 0 and µ 0 = u 0, then λ k = v k and µ k = u k for all k simplify projection onto Qu = v using Q T = Q nothing depends on ṽ k, so can be eliminated Operator splitting 21

22 Final algorithm for k = 0,..., ũ k+1 = (I + Q) 1 (u k + v k ) u k+1 = Π C (ũk+1 v k) v k+1 = v k ũ k+1 + u k+1 parameter free homogeneous same complexity as ADMM applied to primal or dual alone Operator splitting 22

23 Variation: Approximate projection replace exact projection with any ũ k+1 that satisfies ũ k+1 (I + Q) 1 (u k + v k ) 2 µ k, where µ k > 0 satisfy k µ k < useful when an iterative method is used to compute ũ k+1 implied by the (more easily verified) inequality (Q + I)ũ k+1 (u k + v k ) 2 µ k by skew-symmetry of Q Operator splitting 23

24 Convergence can show the following (even with approximate projection): for all iterations k > 0 we have u k C, v k C, (u k ) T v k = 0 as k, Qu k v k 0 with τ 0 = 1, κ 0 = 1, (u k, v k ) bounded away from zero Operator splitting 24

25 Solving the linear system in first step need to solve equations I A T c ũ x A I b ũ y = c T b T 1 ũ τ w x w y w τ let so [ I A T M = A I I + Q = ], h = [ M ] h h T 1 [ ] c b it follows that ] ([ ] ) [ũx = (M + hh T ) 1 wx w ũ y w τ h, y Operator splitting 25

26 Solving the linear system, contd. applying matrix inversion lemma to (M + hh T ) 1 yields and ] [ũx = (M 1 M 1 hh T M 1 )([ ] ) wx (1+h T M 1 w τ h h) ũ y first compute and cache M 1 h ũ τ = w τ + c T ũ x + b T ũ y so each iteration requires that we compute [ ] M 1 wx w y and perform vector operations with cached quantities Operator splitting 26 w y

27 Direct method to solve [ I A T A I ] = z y ][ zx [ wx compute sparse permuted LDL factorization of matrix re-use cached factorization for subsequent solves factorization guaranteed to exist for all permutations, since matrix is symmetric quasi-definite w y ] Operator splitting 27

28 Indirect method by elimination z x = (I + A T A) 1 (w x A T w y ) z y = w y + Az x can apply conjugate gradient (CG) to first equation CG requires only multiplies by A and A T terminate CG iterations when residual smaller than µ k easily parallelized; can exploit warm-starting Operator splitting 28

29 Scaling / preconditioning convergence greatly improved by scaling / preconditioning: replace original data A, b, c with  = DAE, ˆb = Db, ĉ = Ec D and E are diagonal positive; D respects cone boundaries D and E chosen by equilibrating A (details in paper) stopping condition retains unscaled (original) data Operator splitting 29

30 Outline Cone programming Homogeneous embedding Operator splitting Numerical results Conclusions Numerical results 30

31 SCS software package available from: written in C with matlab and python hooks can be called from CVX and CVXPY solves LPs, SOCPs, ECPs, and SDPs includes sparse direct and indirect linear system solvers can use single or double precision, ints or longs for indices Numerical results 31

32 Portfolio optimization z R p gives weights of (long-only) portfolio with p assets maximize risk-adjusted portfolio return: maximize µ T z γ(z T Σz) subject to 1 T z = 1, z 0 µ, Σ are return mean, covariance γ > 0 is risk aversion parameter Σ given as factor model Σ = FF T + D F R q p is factor loading matrix can be transformed to SOCP Numerical results 32

33 Portfolio optimization results assets p factors q SOCP variables n SOCP constraints m nonzeros in A SDPT3: solve time 1.14 sec sec OOM SCS direct: solve time 0.17 sec 4.7 sec 37.1 sec iterations SCS indirect: solve time 0.23 sec 12.2 sec 101 sec average CG iterations iterations Numerical results 33

34 l 1 -regularized logistic regression fit logistic model, with l 1 regularization data z i R p, i = 1,...,q with labels y i { 1, 1} solve minimize q i=1 log(1+exp(y iw T z i ))+µ w 1 over variable w R p ; µ > 0 regularization parameter can be transformed to exponential cone program (ECP) Numerical results 34

35 l 1 -regularized logistic regression results small medium large features p samples q ECP variables n ECP constraints m nonzeros in A SCS direct: solve time 22.1 sec 165 sec 1020 sec iterations SCS indirect: solve time 24.0 sec 199 sec 1290 sec average CG iterations iterations Numerical results 35

36 Large random SOCP randomly generated SOCP with known optimal value n = variables, m = constraints nonzeros in A, 22.5Gb memory to store indirect solver, tolerance 10 3, parallelized over 32 threads results: 740 SCS iterations, about 5000 matrix multiplies 10 hours wall-clock time c T x c T x / c T x = b T y b T y / b T y = Numerical results 36

37 Outline Cone programming Homogeneous embedding Operator splitting Numerical results Conclusions Conclusions 37

38 Conclusions HSD embedding is great for first-order methods diagonal preconditioning critical matrix-free algorithm: only z Az, w A T w SCS is now standard large scale solver in CVXPY Conclusions 38