Solving basis pursuit: infeasible-point subgradient algorithm, computational comparison, and improvements

Transcription

1 Solving basis pursuit: infeasible-point subgradient algorithm, computational comparison, and improvements Andreas Tillmann ( joint work with D. Lorenz and M. Pfetsch ) Technische Universität Braunschweig SIAM Conference on Applied Linear Algebra 2012 A. Tillmann 1 / 22

2 Outline 1 Motivation 2 Infeasible-Point Subgradient Algorithm ISAL1 3 Comparison of l 1 -Solvers Testset Construction and Computational Results Improvements with Heuristic Optimality Check 4 Possible Future Research A. Tillmann 2 / 22

3 Outline Motivation 1 Motivation 2 Infeasible-Point Subgradient Algorithm ISAL1 3 Comparison of l 1 -Solvers Testset Construction and Computational Results Improvements with Heuristic Optimality Check 4 Possible Future Research A. Tillmann 3 / 22

4 Motivation Sparse Recovery via l 1 -Minimization Seek sparsest solution to underdetermined linear system: min x 0 s. t. Ax = b (A R m n, m < n) Finding minimum-support solution is N P-hard. Convex relaxation: l 1 -minimization / Basis Pursuit: min x 1 s. t. Ax = b (L1) Several conditions (RIP, Nullspace Property, etc) ensure l 0 -l 1 -equivalence A. Tillmann 4 / 22

5 Motivation Solving the Basis Pursuit Problem (L1) can be recast as a linear program Broad variety of specialized algorithms for (L1) A classic algorithm from nonsmooth optimization: (projected) subgradient method competitive? Which algorithm is the best? A. Tillmann 5 / 22

6 Outline Infeasible-Point Subgradient Algorithm 1 Motivation 2 Infeasible-Point Subgradient Algorithm ISAL1 3 Comparison of l 1 -Solvers Testset Construction and Computational Results Improvements with Heuristic Optimality Check 4 Possible Future Research A. Tillmann 6 / 22

7 Infeasible-Point Subgradient Algorithm Projected Subgradient Methods Problem: min f (x) s.t. x F (f, F convex) standard projected subgradient iteration x k+1 = P F ( x k α k h k ), α k > 0, h k f (x k ) applicability: only reasonable if projection is easy A. Tillmann 7 / 22

8 Infeasible-Point Subgradient Algorithm Projected Subgradient Methods Problem: min f (x) s.t. x F (f, F convex) standard projected subgradient iteration x k+1 = P F ( x k α k h k ), α k > 0, h k f (x k ) applicability: only reasonable if projection is easy idea: replace exact projection by approximation infeasible subgradient iteration x k+1 = P ε k F (x k α k h k ), P ε k F P F 2 ε k A. Tillmann 7 / 22

9 Infeasible-Point Subgradient Algorithm ISAL1 ISA = Infeasible-Point Subgradient Algorithm... works for arbitrary convex objectives and constraint sets... incorporates adaptive approximate projections P ε F such that P ε F (y) P F(y) 2 ε for every ε 0... converges to optimality (under reasonable assumptions) whenever projection accuracies (ε k ) suciently small, for stepsizes α k > 0 with k=0 α k =, k=0 α2 k < for dynamic stepsizes α k = λ k (f (x k ) ϕ ) / h k 2 2 with ϕ ϕ... converges to ϕ with dynamic stepsizes using ϕ ϕ A. Tillmann 8 / 22

10 Infeasible-Point Subgradient Algorithm ISAL1 = Spezialization of ISA to l 1 -Minimization ISAL1 f (x) = x 1, F = { x Ax = b }, sign(x) x 1 exact projected subgradient step for (L1): ( ) x k+1 = P F x k α k h k ) = (x k α k h k ) A T (AA T ) (A(x 1 k α k h k ) b A. Tillmann 9 / 22

11 Infeasible-Point Subgradient Algorithm ISAL1 = Spezialization of ISA to l 1 -Minimization ISAL1 f (x) = x 1, F = { x Ax = b }, sign(x) x 1 exact projected subgradient step for (L1): y k z k x k α k h k Solution of AA T z = Ay k b x k+1 y k A T z k = P F (y k ) AA T is s.p.d. may employ CG to solve equation system A. Tillmann 9 / 22

12 Infeasible-Point Subgradient Algorithm ISAL1 = Spezialization of ISA to l 1 -Minimization ISAL1 f (x) = x 1, F = { x Ax = b }, sign(x) x 1 inexact projected subgradient step for (L1): y k z k x k+1 x k α k h k Solution of AA T z Ay k b y k A T z k = P ε k F (y k ) AA T is s.p.d. may employ CG to solve equation system Approximation: Stop after a few CG iterations (CG residual norm σ min (A) ε k P ε k F ts ISA framework) A. Tillmann 9 / 22

13 Outline Comparison of l 1 -Solvers 1 Motivation 2 Infeasible-Point Subgradient Algorithm ISAL1 3 Comparison of l 1 -Solvers Testset Construction and Computational Results Improvements with Heuristic Optimality Check 4 Possible Future Research A. Tillmann 10 / 22

14 Our Testset Comparison of l 1 -Solvers Testset Construction and Computational Results 100 matrices A (74 dense, 26 sparse) dense: 512 {1024, 1536, 2048, 4096} 1024 {2048, 3072, 4096, 8192} sparse: 2048 {4096, 6144, 8192, 12288} 8192 {16384, 24576, 32768, 49152} random (e.g., partial Hadamard, random signs,...) concatenations of dictionaries (e.g., [Haar, ID, RST],...) columns normalized 2 or 3 vectors x per matrix such that each resulting (L1) instance (with b := Ax ) has unique optimum x A. Tillmann 11 / 22

15 Comparison of l 1 -Solvers Constructing Unique Solutions Testset Construction and Computational Results 274 instances with known, unique solution vectors x : For each matrix A, choose support S which obeys 1 pick S at random, and ERC(A, S) := max A S a j 1 < 1. j / S 2 try increasing some S by repeatedly adding the resp. arg max 3 For dense A's, use L1TestPack to construct another unique solution support (via optimality condition for (L1)) Entries of x S random with high dynamic range A. Tillmann 12 / 22

16 Comparison Setup Comparison of l 1 -Solvers Testset Construction and Computational Results Only exact solvers for (L1): min x 1 s. t. Ax = b Tested algorithms: ISAL1, SPGL1, YALL1, l 1 -Magic, SolveBP (SparseLab), l 1 -Homotopy, CPLEX (Dual Simplex) Use default settings (black box usage) Solution x optimal, if x x 2 Solution x acceptable, if x x A. Tillmann 13 / 22

17 Comparison of l 1 -Solvers Testset Construction and Computational Results Running Time vs. Distance from Unique Optimum ISAL1 (w/o HOC) SPGL1 YALL1 CPLEX l1 MAGIC SparseLab / PDCO Homotopy x x Running Times [sec] A. Tillmann 14 / 22

18 Comparison of l 1 -Solvers Testset Construction and Computational Results Results: Average Performances CPLEX (Dual Simplex): most reliable solver SPGL1: apparently very fast with usually acceptable solutions ISAL1: mostly very accurate, but rather slow SolveBP: often produces acceptable solutions, but slow l 1 -Magic: fast for sparse A, but results often inacceptable YALL1: very fast, but results largely inacceptable l 1 -Homotopy: usually pretty accurate (not always acceptable), but on the slow side Can we achieve better performances without changing default (tolerance) settings? A. Tillmann 15 / 22

19 Comparison of l 1 -Solvers Improvements with Heuristic Optimality Check New Optimality Check for l 1 -Minimization Optimality criterion for (L1): x arg min x 1 x 1 Im(A T ) x: Ax=b Exact evalution too expensive A. Tillmann 16 / 22

20 Comparison of l 1 -Solvers Improvements with Heuristic Optimality Check New Optimality Check for l 1 -Minimization Optimality criterion for (L1): x arg min x 1 x 1 Im(A T ) x: Ax=b Exact evalution too expensive Heuristic Optimality Check (HOC): Estimate support S of given x and (approximately) solve A T S w = sign(x S). If w is dual-feasible ( A T w 1), compute x from A S x S = b. If x is primal-feasible, it is optimal if b T w = x 1. Allows safe jumping to optimum (from infeasible points) A. Tillmann 16 / 22

21 Comparison of l 1 -Solvers Improvements with Heuristic Optimality Check Impact of Heuristic Optimality Check (Example) HOC in ISAL1 HOC in SPGL1 HOC in l 1 -Magic Iteration k Iteration k Iteration k (red curves: distance to known optimum, blue curves: feasibility violation) A. Tillmann 17 / 22

22 Comparison of l 1 -Solvers Improvements with HOC (numbers) Improvements with Heuristic Optimality Check ( instances) ISAL1 SPGL1 YALL1 l 1 -Mag. l 1 -Hom. solved faster w/ HOC 184/212 /0 /0 /0 161/181 solved only w/ HOC improved (ERC-based) 99.5% 88.0% 6.5% 78.0% 91.0% improved (other part) 36.5% 20.3% 0.0% 18.9% 74.3% A. Tillmann 18 / 22

23 Comparison of l 1 -Solvers Improvements with HOC (numbers) Improvements with Heuristic Optimality Check ( instances) ISAL1 SPGL1 YALL1 l 1 -Mag. l 1 -Hom. solved faster w/ HOC 184/212 /0 /0 /0 161/181 solved only w/ HOC improved (ERC-based) 99.5% 88.0% 6.5% 78.0% 91.0% improved (other part) 36.5% 20.3% 0.0% 18.9% 74.3% Explanation for higher HOC success rate on ERC-based testset: ERC = w = (A T S ) sign(x S ) satises A T w x 1. A. Tillmann 18 / 22

24 Comparison of l 1 -Solvers Improvements with HOC (pictures) Improvements with Heuristic Optimality Check without HOC: ISAL1 SPGL1 YALL1 l 1 -Magic l 1 -Hom. x x 2 x x 2 x x 2 x x 2 x x Running Times [sec] Running Times [sec] Running Times [sec] Running Times [sec] Running Times [sec] with HOC: ISAL1 SPGL1 YALL1 l 1 -Magic l 1 -Hom. x x 2 x x 2 x x 2 x x 2 x x Running Times [sec] Running Times [sec] Running Times [sec] Running Times [sec] Running Times [sec] A. Tillmann 19 / 22

25 Comparison of l 1 -Solvers Rehabilitation of l 1 -Homotopy Improvements with Heuristic Optimality Check The Homotopy method provably solves (L1) via a sequence of problems min 1 2 Ax b 2 + λ x 1 for parameters λ 0 decreasing to zero. Also: Provably fast for su. sparse solutions. not fast, inaccurate?! x x Running Times [sec] A. Tillmann 20 / 22

26 Comparison of l 1 -Solvers Rehabilitation of l 1 -Homotopy Improvements with Heuristic Optimality Check The Homotopy method provably solves (L1) via a sequence of problems min 1 2 Ax b 2 + λ x 1 for parameters λ 0 decreasing to zero. Also: Provably fast for su. sparse solutions. nal λ = 0 nal λ = 10 9 x x 2 x x Running Times [sec] Numerical issues? Running Times [sec] Winner! A. Tillmann 20 / 22

27 Possible Future Research Possible Future Research ISAL1 with variable target values? HOC helpful in Approximate Homotopy Path algorithm? Extensions to Denoising problems? PF ε for, e.g., F = { x Ax b 2 δ }? HOC schemes? Testsets, solver comparisons (also for implicit matrices),...? Really hard instances?... e.g., construction of Mairal & Yu for which the (exact) Homotopy path has O(3 n ) kinks? A. Tillmann 21 / 22

28 SPEAR Project ISA theory: Lorenz, Pfetsch & T.: An Infeasible-Point Subgradient Method Using Adaptive Approximate Projections, 2011 ISAL1 theory, HOC & numerical results: Lorenz, Pfetsch & T.: Solving Basis Pursuit: Subgradient Algorithm, Heuristic Optimality Check, and Solver Comparison, 2011/2012 Papers, Matlab Codes (ISAL1, HOC, L1TestPack), Testset, Slides, Posters etc. available at: A. Tillmann 22 / 22