Convex or non-convex: which is better?

Size: px

Start display at page:

Download "Convex or non-convex: which is better?"

Vivien Nash
5 years ago
Views:

1 Sparsity Amplified Ivan Selesnick Electrical and Computer Engineering Tandon School of Engineering New York University Brooklyn, New York March 7 / 4

2 Convex or non-convex: which is better? (for sparse-regularized linear inverse problems) Benefits of convex optimization:. Absence of suboptimal local minima. Continuity of solution as a function of input data 3. Algorithms guaranteed to converge to a global optimum 4. Regularization parameters easier to set But convex regularization tends to under-estimate signal values. Non-convex regularization often performs better! Can we design non-convex sparsity-inducing penalties that maintain the convexity of the cost function to be minimized? / 4

3 An alternative Conventional sparse-regularized least squares: J(x) = y Ax + λ x Alternatively, use non-convex penalty ψ that maintains convexity of cost function F F (x) = y Ax + λψ(x) I. Selesnick. Total Variation Denoising via the Moreau Envelope. IEEE Signal Processing Letters, 4():6, February 7. I. W. Selesnick and I. Bayram. Enhanced Sparsity by Non-Separable Regularization. IEEE Trans. Signal Process., 64(9):98 33, May 6. A. Parekh and I. W. Selesnick. Enhanced low-rank matrix approximation. IEEE Signal Processing Letters, 3(4): , April 6. Y. Ding and I. W. Selesnick. Artifact-Free Wavelet Denoising: Non-convex Sparse Regularization, Convex Optimization. IEEE Signal Processing Letters, (9): , September 5. I. W. Selesnick, A. Parekh, and I. Bayram. Convex -D Total Variation Denoising with Non-convex Regularization. IEEE Signal Processing Letters, ():4 44, February 5. 3 / 4

4 MC penalty The minimax-concave (MC) penalty φ: R R is defined as x φ(x) := x, x, x 3 x φ(x) 3 3 x C.-H. Zhang. Nearly unbiased variable selection under minimax concave penalty. Statistics:894 94,. The Annals of 4 / 4

5 Huber function The MC penalty can be expressed as φ(x) = x s(x) where s is the Huber function x The Huber function. 5 / 4

6 Huber function The Huber function s : R R is defined as s(x) := x, x x, x. The Huber function can be written as s(x) = min v R { v + (x v) }, equivalently s = ( ) where denotes infimal convolution. H. H. Bauschke and P. L. Combettes. Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer,. 6 / 4

7 Huber function 3.5 x x +.5 x x The Huber function as the pointwise minimum of three functions. 7 / 4

8 Scaled functions Let b R. The scaled Huber function s b : R R is defined as s b (x) := s(b x)/b, b. For b =, the function is defined as s (x) :=. The scaled MC penalty function φ b : R R is defined as φ b (x) := x s b (x) where s b is the scaled Huber function. 8 / 4

9 Scaled functions Scaled Huber function and MC penalty for several values of the scaling parameter. 3 Scaled Huber function s b (x) b =.4 b =. b = x 3 Scaled MC penalty φ b (x) b =. b =.7 b =. b = x 9 / 4

10 Convexity condition Let λ > and a R. Define f : R R, f (x) = (y ax) + λ φ b (x) where φ b is the scaled MC penalty. If b a /λ, then. f is convex. the minimizer is given by firm thresholding, x opt = firm(y/a; λ/a, /b ). / 4

11 Firm threshold function firm(y; λ, µ) 4 Firm threshold function 3 λ µ y / 4

12 Aims The minimizer of the scalar function f is easily obtained via firm thresholding. However, the situation in the multivariate case is more complicated. To generalize this process to the multivariate case, we aim to:. define a multivariate MC penalty (non-convex),. define a sparse-regularized least squares cost function (convex), 3. generalize the convexity condition, 4. provide a method to calculate a minimizer. / 4

13 Generalized Huber function Let B R M N. We define the generalized Huber function S B : R N R as { S B (x) := min v + v R N B(x } v). In the notation of infimal convolution, we have S B = B. The generalized Huber function satisfies S B (x) x, x R N. 3 / 4

14 Generalized Huber function Generalized Huber function S B (x) 3 B = x x Contours of S B 4 / 4

15 Generalized Huber function Generalized Huber function S B (x) B = [.5] x x Contours of S B 5 / 4

16 Generalized MC penalty Let B R M N. We define the generalized MC (GMC) penalty function ψ B : R N R as ψ B (x) := x S B (x) where S B is the generalized Huber function. The generalized MC penalty satisfies ψ B (x) x for all x R N. 6 / 4

17 Generalized MC penalty x S B (x) ψ B (x) = x S B (x) x x x x Contours Contours Contours 7 / 4

18 Generalized MC penalty x S B (x) ψ B (x) = x S B (x) x x x x Contours Contours Contours 8 / 4

19 Convexity condition Let y R M, A R M N, and λ >. Define F : R N R as F (x) = y Ax + λ ψ B(x) where ψ B is the generalized MC penalty. If B T B λ AT A, then F is a convex function. Hence, for convexity of F, we may simply set B = γ/λ A, γ. () 9 / 4

20 Proximal algorithm Using forward-backward splitting, we have: Let λ > and γ <. Let y R N and A R M N. Then a saddle-point (x opt, v opt ) of F can be obtained by the iterative algorithm: Set ρ = max{, γ/( γ)} A T A Set µ : < µ < /ρ For i =,,,... w (i) = x (i) µa T( A(x (i) + γ(v (i) x (i) )) y ) u (i) = v (i) µγa T A(v (i) x (i) ) x (i+) = soft(w (i), µλ) v (i+) = soft(u (i), µλ) end where i is the iteration counter. / 4

21 Example: frequency-domain sparsity Noise free signal Noisy signal, σ = Denoising via frequency domain sparsity Denoising [L norm] 4 4 RMSE =.43 5 Denoising [GMC penalty] 4 4 RMSE =.49 5 Average RMSE L norm L + debias IPS MUSR GMC 6 4 FFT of noisy signal 5 Optimized Fourier coefficients o GMC x L norm λ Frequency Frequency / 4

22 Example: time-frequency domain sparsity.3 True signal Time frequency representation (db) Frequency (khz) Time (ms) Noisy signal Frequency (khz) 6 4 Time (ms) Time frequency representation (db).3.3 RMSE =.6 Time (ms) Denoising using L norm Frequency (khz) 6 4 Time (ms) Time frequency representation (db).3.3 RMSE =.6 Time (ms) Denoising using GMC penalty Frequency (khz) 6 4 Time (ms) Time frequency representation (db).3 Time (ms) Time (ms) / 4

23 Example: sparsity-assisted signal smoothing (SASS) Biosensor data SASS using L norm (a) SASS using GMC penalty 5 5 (b) Magnified view data L norm GMC (c) (d) 3 / 4

24 Conclusion Convex and the non-convex methods for sparse-regularized least squares are usually mutually exclusive and incompatible. To bridge these two approaches, we introduce a non-convex alternative to the L norm that preserves the convexity of the cost function to be minimized. The proposed penalty leads to optimization problems with no extraneous suboptimal local minima and allows the use of globally convergent, computationally efficient, scalable convex optimization algorithms. The advantages compared to L norm regularization are (i) more accurate estimation of high-amplitude components of sparse solutions, (ii) a higher level of sparsity in a sparse approximation problem. 4 / 4

A fast algorithm for sparse reconstruction based on shrinkage, subspace optimization and continuation [Wen,Yin,Goldfarb,Zhang 2009]

A fast algorithm for sparse reconstruction based on shrinkage, subspace optimization and continuation [Wen,Yin,Goldfarb,Zhang 2009] Yongjia Song University of Wisconsin-Madison April 22, 2010 Yongjia Song