Blind Compressed Sensing Using Sparsifying Transforms

Transcription

1 Blind Compressed Sensing Using Sparsifying Transforms Saiprasad Ravishankar and Yoram Bresler Department of Electrical and Computer Engineering and Coordinated Science Laboratory University of Illinois at Urbana-Champaign May 29,

2 Key Topics of Talk Non-adaptive Compressed Sensing (CS) Synthesis dictionary learning-based blind compressed sensing Transform learning vs. Dictionary learning Transform learning-based blind compressed sensing Application to magnetic resonance imaging (MRI) Transform learning based MRI (TLMRI) Conclusions 2

3 Compressed Sensing (CS) CS enables accurate recovery of images from far fewer measurements than the number of unknowns Sparsity of image in transform domain or dictionary measurement procedure incoherent with transform Reconstruction non-linear and expensive Reconstruction problem (NP-hard) - min x Data Fidelity Regularizer {}}{{}}{ Ax y 2 2 +λ x C P : image as vector, y C m : measurements. Ψx 0 (1) A C m P : sensing matrix (m < P), Ψ : transform (Wavelets, Contourlets, Total Variation). l 0 norm counts non-zeros. Iterative algorithms for CS reconstruction are usually expensive. 3

4 Application: Compressed Sensing MRI (CSMRI) Data - samples in k-space of spatial Fourier transform of object, acquired sequentially. Acquisition rate limited by MR physics, physiological constraints on RF energy deposition. CS accelerates the data acquisition in MRI. CSMRI with non-adaptive transforms or dictionaries limited to fold undersampling [Ma et al. 08]. Two directions to improve CSMRI - better or adaptive sparse modeling better choice of sampling pattern (F u) [EMBC, 2011] Fig. from Lustig et al. 07 4

5 Synthesis Model for Sparse Representation Given a signal y R n, and dictionary D R n K, we assume y = Dx with x 0 K a union of subspaces model. Real world signals modeled as y = Dx +e, e is deviation term. Given D, sparsity level s, the synthesis sparse coding problem is This problem is NP-hard. ˆx = argmin x y Dx 2 2 s.t. x 0 s 5

6 Synthesis Dictionary Learning The DL problem (NP-hard) - min D,B N R j x Db j 2 2 s.t. d k 2 = 1 k, b j 0 s j (2) R j x C n - n n patch indexed by location in image. R j C n P extracts patch. D C n K - patch based dictionary. b j C K - sparse, x j Db j. s - sparsity, B = [b 1 b 2... b N ]. DL minimizes fit error of all patches using sparse representations w.r.t. D. 6

7 Synthesis-based Blind Compressed Sensing (BCS) (P0) min x,d,b Sparse Fitting Regularizer {}}{ N R j x Db j 2 2 +ν Data Fidelity {}}{ Ax y 2 2 s.t. d k 2 = 1 k, b j 0 s j. B C n N : matrix that has the sparse codes b j as its columns. (P0) learns D C n K, and reconstructs x, from only undersampled y dictionary adaptive to underlying image. (P0) is NP-hard, non-convex even if l 0 norm relaxed to l 1. DLMRI 1 solves (P0) for MRI and works better than non-adaptive CS. Synthesis BCS algorithms have no guarantees and are expensive. 1 [Ravishankar & Bresler 11] 7

8 2D Random Sampling - 6 fold undersampling LDP 2 reconstruction (22 db) LDP error magnitude DLMRI reconstruction (32 db) DLMRI error magnitude 0 MRI data from Miki Lustig. 2 [Lustig et al. 07] 8

9 Alternative: Sparsifying Transform Model Given a signal y R n, and transform W R m n, we model Wy = x +η with x 0 m and η - error term. Natural signals are approximately sparse in Wavelets, DCT. Given W, and sparsity s, transform sparse coding is ˆx = argmin x Wy x 2 2 s.t. x 0 s ˆx = H s(wy) computed by thresholding Wy to the s largest magnitude elements. Sparse coding is cheap. Signal recovered as W ˆx. Sparsifying transforms exploited for compression (JPEG2000), etc. 9

10 Alternative: Sparsifying Transform Learning Square Transform Models Unstructured transform learning [IEEE TSP, 2013 & 2015] Doubly sparse transform learning [IEEE TIP, 2013] Online learning for Big Data [IEEE JSTSP, 2015] Convex formulations for transform learning [ICASSP, 2014] Overcomplete Transform Models Unstructured overcomplete transform learning [ICASSP, 2013] Learning structured overcomplete transforms with block cosparsity (OCTOBOS) [IJCV, 2014] Applications: Sparse representation, Image & Video denoising, Classification, Blind compressed sensing (BCS) for imaging. 10

11 Square Transform Learning Formulation (P1) min W,B Sparsification Error {}}{ N WR j x b j 2 2 +λ s.t. b j 0 s j Regularizer {}}{ (0.5 W 2F log detw ) Sparsification error - measures deviation of data in transform domain from perfect sparsity. Regularizer enables complete control over conditioning & scaling of W. If (Ŵ, ˆB) such that the condition number κ(ŵ) = 1, ŴR j x = ˆb j, ˆb j 0 s j globally identifiable by solving (P1). (P1) favors both a low sparsification error and good conditioning. The solution to (P1) is unitary as λ. 11

12 Transform-based Blind Compressed Sensing (BCS) (P2) min x,w,b s.t. Sparsification Error {}}{ N Data Fidelity Regularizer {}}{{}}{ WR j x b j 2 2 +ν Ax y 2 2 +λ v(w) N b j 0 s, x 2 C. (P2) learns W C n n, and reconstructs x, from only undersampled y transform adaptive to underlying image. v(w) log detw +0.5 W 2 F controls scaling and κ of W. x 2 C is an energy/range constraint. C > 0. 12

13 Transform BCS: Identifiability & Uniqueness Proposition 1 Let x C p, and let y = Ax with A C m p. Suppose x 2 C W C n n is a unitary transform N WR jx 0 s Further, let B denote the matrix that has WR j x as its columns. Then, (x,w,b) is a global minimizer of Problem (P2), i.e., it is identifiable by solving (P2). Given minimizer (x,w,b) of (P2), (x,θw,θb) is another equivalent minimizer Θ s.t. Θ H Θ = I, j Θb j 0 s. The optimal x is invariant to such transformations of (W,B). When W is constrained to be doubly sparse and unitary, uniqueness can be guaranteed under additional (e.g., spark) conditions. 13

14 Alternative Transform BCS Formulations (P3) min x,w,b N WR j x b j 2 2 +ν Ax y 2 2 s.t. W H W = I, N b j 0 s, x 2 C. (P3) is also a unitary synthesis dictionary-based BCS problem, with W H the synthesis dictionary. (P4) min x,w,b N WR j x b j 2 2 +ν Ax y 2 2 +λv(w)+ N η2 b j 0 s.t. x 2 C. 14

15 Block Coordinate Descent (BCD) Algorithm for (P2) (P2) solved by alternating between updating W, B, and x. Alternate a few times between the W and B updates, before performing an image update. Sparse Coding Step solves (P2) for B with fixed x, W. min B N WR j x b j 2 2 s.t. N b j 0 s. (3) Cheap Solution: Let Z C n N be the matrix with WR jx as its columns. Solution ˆB = H s(z) computed exactly by zeroing out all but the s largest magnitude coefficients in Z. 15

16 BCD Algorithm for (P2) Transform Update Step solves (P2) for W with fixed x, B. min W N WR j x b j λ W 2 F λlog detw (4) Let X C n N be the matrix with R jx as its columns. Closed-form solution: Ŵ = 0.5R ( ( ) Σ+ Σ 2 )1 2 +2λI V H L 1 (5) where XX H +0.5λI = LL H, and L 1 XB H has a full SVD of VΣR H. Solution is unique if and only if XB H is non-singular. 16

17 BCD Algorithm for (P2) Image Update Step solves (P2) for x with fixed W, B. min x N WR j x b j 2 2 +ν Ax y 2 2 s.t. x 2 C. (6) Least squares problem with l 2 norm constraint. Solution is unique as long as the set of overlapping patches cover all image pixels. Solve Least squares Lagrangian formulation: min x N WR jx b j 2 2 +ν Ax y 2 2 +µ ( x 2 2 C ) The optimal multiplier ˆµ R + is the smallest real such that ˆx 2 C. ˆµ and ˆx can be found cheaply. (7) 17

18 BCS Convergence Guarantees - Notations Define the barrier function ψ s (B) as { N 0, ψ s (B) = b j 0 s +, else χ C (x) is the barrier function corresponding to x 2 C. (P2) is equivalent to the problem of minimizing the objective g(w,b,x) = N WR jx b j 2 2 +ν Ax y 2 2 +λv(w)+ψ(b)+χ(x) For H C p q, ρ j (H) is the magnitude of the j th largest element (magnitude-wise) of H. X C n N denotes a matrix with R j x, 1 j N, as its columns. 18

19 Transform BCS Convergence Guarantees Theorem 1 For the sequence {W t,b t,x t } generated by the BCD Algorithm with initial (W 0,B 0,x 0 ), we have {g (W t,b t,x t )} g = g (W 0,B 0,x 0 ). {W t,b t,x t } is bounded, and all its accumulation points are equivalent, i.e., they achieve the same value g of the objective. x t x t as t. Every accumulation point (W,B,x) is a critical point of g satisfying the following partial global optimality conditions W argmin W x argmin g (W,B, x) (8) x ( ) ( ) g W,B,x, B argmin g W, B,x (9) B 19

20 Transform BCS Convergence Guarantees Theorem 2 Each accumulation point (W,B,x) of {W t,b t,x t } also satisfies the following partial local optimality conditions g(w + W,B + B,x) g(w,b,x) = g (10) g(w,b + B,x + x) g(w,b,x) = g (11) The conditions each hold for all x C p, and all W C n n satisfying W F ǫ for some ǫ = ǫ(w) > 0, and all B C n N in R1 R2 R1. The half-space Re ( tr { (WX B) B H}) 0. R2. The local region defined by B < ρ s (WX). Furthermore, if WX 0 s, then B can be arbitrary. 20

21 Global Convergence Guarantees Proposition 2 For each initialization, the iterate sequence in the BCD algorithm converges to an equivalence class (same objective values) of critical points of the objective, that are also partial global/local minimizers. Proposition 3 The BCD algorithm is globally convergent to a subset of the set of critical points of the objective. The subset includes all (W,B,x) that are at least partial global and partial local minimizers. 21

22 Computational Advantages of Transform BCS Cost per iteration of transform BCS: O(p 4 NL) N overlapping patches of size p p, W C n n, n p 2. L : # inner alternations between transform update & sparse coding. Cost per iteration of Synthesis BCS method DLMRI 3 : O(p 6 NJ). D C n K, n p 2, K n, sparsity s n. J : # of inner iterations of dictionary learning using K-SVD 4. In practice, transform BCS converges quickly and is much cheaper for large p. In 3D or 4D imaging, n = p 3 or p 4, and the gain in computations is about a factor n in order. 3 [Ravishankar & Bresler 11] 4 [Aharon et al. 06], 22

23 TLMRI Convergence - 4x Undersampling (s = 3.4%) 9.7 x 105 Reference Sampling mask 10 1 Objective Function x t x t Iteration Number Objective Iteration Number (t) x t x t 1 2 vs. t 23

24 Convergence & Learning - 4x Undersampling (s = 3.4%) Zero-filling (28.94 db) Zero-filling Error 0 TLMRI (32.66 db) real (top), imaginary (bottom) parts of learnt W 24

25 Comparison (PSNR & Runtime) to Recent Methods Sampling Scheme Undersampling Zero-filling LDP 5 PBDWS 6 DLMRI 7 PANO 8 TLMRI 2D Random 4x x Cartesian 4x x Avg. Runtime (s) TLMRI is up to 5.5 db better than LDP, that uses Wavelets + TV. TLMRI provides up to 1 db improvement in PSNR over the PBDWS method that uses redundant Wavelets and trained patch-based geometric directions, and is up to 1.6 db better than the non-local PANO method. It is up to 0.35 db better than DLMRI, that learns 4x overcomplete dictionary. TLMRI is 10x faster than DLMRI, and 4x faster than the PBDWS method. TLMRI provides the best reconstructions, and is the fastest. 5 [Lustig et al. 07] 6 [Ning et al. 13] 7 [Ravishankar & Bresler 11] 8 [Qu et al. 14] 25

26 Example - 2D random 5x Undersampling Reference DLMRI (28.54 db) TLMRI (30.47 db) Sampling Mask DLMRI Error TLMRI Error

27 Conclusions We introduced a transform-based BCS framework Proposed BCS algorithms have a low computational cost. We provided novel convergence guarantees for the algorithms, that do not require any restrictive assumptions. For CSMRI, the proposed approach is better than leading image reconstruction methods, while being much faster. Future work: convergence of algorithm to global minimizer & convergence rate. 27

28 Thank you! Questions?? 28

29 Convergence Guarantees - Definitions Definition 1 Let φ : R q (,+ ] be a proper function and let z domφ. The Fréchet sub-differential of the function φ at z is the following set: ˆ φ(z) { h R q : liminf b z,b z 1 b z (φ(b) φ(z) b z,h ) 0} (12) If z / domφ, then ˆ φ(z) =. The sub-differential of φ at z is defined as φ(z) { h R q : z k z,φ(z k ) φ(z),h k ˆ φ(z k ) h }. (13) Lemma 1 A necessary condition for z R q to be a minimizer of the function φ : R q (,+ ] is that z is a critical point of φ, i.e., 0 φ(z). If φ is a convex function, this condition is also sufficient. 29