Sparse Solutions to Linear Inverse Problems. Yuzhe Jin

Transcription

1 Sparse Solutions to Linear Inverse Problems Yuzhe Jin

2 Outline Intro/Background Two types of algorithms Forward Sequential Selection Methods Diversity Minimization Methods Experimental results Potential application to Speech

3 Background: Sparseness A large vector with only a very small # of non-zero entries

4 Why sparse? In what scenarios? Bio-magnetic inverse problem Band-limited extrapolation, especially for Speech signals Direction-of-arrival estimation Channel equalization, Echo cancellation Image restoration Represent a signal of interest using the minimum number of vectors from an overcompleted dictionary.

5 Problem Description x1 x 2 x 3. b = a a a... a a a. xn 2 a dictionary with each x n 1 column as a codeword xn n 2 n 1 n a measurement sparse source, only a few of the entries are non-

6 Problem Description x1 x 2 x 3. b = a a a... a a a. xn 2 a dictionary with each x n 1 column as a codeword xn n 2 n 1 n a measurement b = x a i i sparse source, only a few of the entries are non-

7 With multiple measurements x11... x1, m x21... x2, m x31... x 3, m b1... b m a1 a2 a3... an 2 an 1 a = n xn 2,1... xn 2, m xn 1,1... xn 1, m Multiple x... n,1 x measurements nm, B=AX Multiple sources with same sparsity profile

8 With multiple measurements x11... x1, m x21... x2, m x31... x 3, m b1... b m a1 a2 a3... an 2 an 1 a = n xn 2,1... xn 2, m xn 1,1... xn 1, m Multiple x... n,1 x measurements nm, B=AX Multiple sources with same sparsity profile

9 Then, add noise to the observations New Model: B=AX+N Modeling error Noise present Tradeoff between fit and sparsity of solution

10 Then, add noise to the observations New Model: B=AX+N Tradeoff between fit and sparsity of solution AX - B Modeling error Noise present

11 Type I: Forward Sequential Selection x1 x 2 x 3. b = a a a... a a a. xn 2 Known x n 1 xn n 2 n 1 n Known Unknown

12 Type I: Forward Sequential Selection x1 x 2 x 3. b = a a a... a a a. xn 2 x n 1 xn n 2 n 1 n

13 Type I: Forward Sequential Selection x1 x 2 x 3. b = a a a... a a a. xn x n 1 xn n 2 n 1 n

14 How to compute residual? Basic Matching Pursuit Remove contribution from the selected vector Project onto one direction P a p 1 Orthogonal Matching Pursuit k p b Remove contribution from the selected subspace Project onto a sub-space P S p b S = S a p 1 p p 1 k p [, ]

15 Another variation: Order Recursive Matching Pursuit Most like Orthogonal Matching Pursuit Main Difference: k p The vector is selected by = arg max normalization term Correction: k < ba, > a k ( p 1) k 2 a = P ( p 1) k S k Remove the contribution from vectors found previously Normalized inner product (since the codewords are not orthonormal) p a

16 Type II: Diversity Minimization Metric of Sparseness E p = 2: Like 2-norm ( p) Commonly used in engineering solutions Nothing about sparsity ( x) = xi [ ] p = : a count on the non-zero entries in x Direct measurement on sparseness Very hard to solve, exhaustive search, NP-hard, combinatorial i= 1 What if p takes some value in between? n In practice, setting p to some value between.8 ~ 1. gives good tradeoff between computational complexity and quality of sparse solution p

17 Now, we have a cost function. What s Next? Gradient Descent FOCUSS-class Algorithms FOCUSS Regularized FOCUSS Add in a regularization term to improve matrix condition for computing its inverse Introduce bias, tradeoff between bias and quality of convergence

18 Experiments Setup Different number of measurements Different SNR Compare performances across 4 algorithms M-OMP, M-ORMP, M-FOCUSS, R-M-FOCUSS Test: If an algorithm correctly identifies the non-zero positions, Percentage Success The MSE between the recovered vector and the ground truth, MSE

19 Experimental results MSE SNR=2dB M-OMP M-ORMP M-FOCUSS R-M-FOCUSS MSE SNR=3dB L L.2 SNR=4dB.2 SNR=5dB MSE.1 MSE L L

20 Experimental results MSE L= SNR(dB).4 L=3 M-OMP M-ORMP M-FOCUSS R-M-FOCUSS Percentage Success L= SNR(dB) L=3 1 MSE SNR(dB) L=5.2 Percentage Success SNR(dB) L=5 1 MSE SNR(dB) Percentage Success SNR(dB)

21 Applications to Speech Speech: Sparse in frequency domain Extrapolation Compression

22 Conclusion These algorithms are able to explore sparse linear inverse problems efficiently. They are robust to signal contaminated by noise. Speech signal is sparse in certain transform domain. We can apply this technique to speech.

23 References Sparse solutions to linear inverse problems with multiple measurements vectors, S. F. Cotter, B. D. Rao, K. Engan, K.K-Delgado, IEEE Trans. Sig. Proc., July 25 Sparse signal reconstruction from limited data using FOCUSS: a re-weighted minimum norm algorithm, I. F. Gorodnitsky, B. D. Rao, IEEE Trans. Sig. Proc., March 1997 Extrapolation and spectral estimation with iterative weighted norm modification, S. D. Cabrera, T. W. Parks, IEEE Trans. Sig. Proc., April 1991 Forward sequential algorithms for best basis selection, S. F. Cotter, J. Adler, B. D. Rao, K. K-Delgado, IEE Proc. Image Sig. Proc., Oct 1999