When Sparsity Meets Low-Rankness: Transform Learning With Non-Local Low-Rank Constraint For Image Restoration

Transcription

1 When Sparsity Meets Low-Rankness: Transform Learning With Non-Local Low-Rank Constraint For Image Restoration Bihan Wen, Yanjun Li and Yoram Bresler Department of Electrical and Computer Engineering Coordinated Science Laborarory University of Illinois at Urbana-Champaign March 9, 2017 B. Wen et al. STROLLR Learning and Restoration

2 Outline of Talk Local sparsity v.s Non-local low-rankness STROLLR - Sparsifying TRansfOrm Learning and Low-Rank model STROLLR Image Restoration: formulation & algorihms Applications in image denoising and inpainting. B. Wen et al. STROLLR Learning and Restoration

3 Image Properties: to differentiate Signal from corruptions. Image Properties 1. Local properties - Sparsity - Natural patches are sparsifiable. - Synthesis model - Analysis model => Transform model 2. Non-local properties Low-rankness - Image contains similar patches. - Group & process - Low-rank approximation

4 Sparse Coding Transform Learning Non-local Low-Rankness Image Restoration

5 Sparse Coding Transform Learning Non-local Low-Rankness Image Restoration

6 Example: Barbara

7 Example: Barbara

8 Example: Barbara

9 Example: Barbara

10 Sparse Model: Synthesis Model Synthesis Model (SM): Given synthesis dictionary D R n K, a signal y R n satisfies y = Dx, with sparse x, i.e., x 0 n. General SM: y = Dx +e, where e is a small deviation term. D R n K : Sparsifying transform x R K is sparse Dictionary Learning: popular sparse signal modeling approach. Sparse coding in SM is NP-hard! Approximate methods: Greedy algorithms / l 1 norm relaxation. Not efficient enough. B. Wen et al. STROLLR Learning and Restoration

11 Sparse Model: Transform Model Transform Model - generalization of analysis model. Given transform W R m n, signal y R n satisfies Wy = x +η, with x 0 m, and a small deviation η in the transform domain. W R m n : Sparsifying transform Transform sparse coding: x R m is sparse ˆx = argmin x Wy x 2 2 s.t. x 0 s. Exact and cheap solution: ˆx = H s(wy) computed by thresholding Wy to the s largest magnitude elements (projection onto l 0 ball). A least squares signal estimate: ŷ = W ˆx. B. Wen et al. STROLLR Learning and Restoration

12 STROLLR Modeling (P1) min {W,X,{D i }} W U X 2 F +γ2 s X 0 + γ l N i=1 { } U V i D i 2 F +θ2 rank(d i) s.t. W T W = I n U = [u 1 u 2... u N ] R n N : matrix of image patch vectors. X = [x 1 x 2... x N ] R n N : matrix of sparse codes of u i s. UV i R n M : matrix of patch-vectors via block matching (BM) with reference patch u i. D i R n M : the low-rank approximation of UV i. STROLLR Modeling: Local patch sparsity = Sparse coding for U with adaptive W. Non-local low-rankness = Low-rank approximation of UV i. B. Wen et al. STROLLR Learning and Restoration

13 STROLLR Restoration (P2) min {W,X,{D i },U} +γ l N i=1 W U X 2 F +γ2 s X 0 +γ f N { } UV i D i 2 F +θ2 rank(d i) i=1 { } A i u i y i 2 2 s.t. W T W = I n Corrupted measurement y i = A i u i +h i h i R n : additive noise, and A i R n n : corruption operator. U = [u 1 u 2... u N ] R n N, where u i is i-th overlapping image patch. UV i R n M : block matching within Q Q search window, centered at u i. Simple algorithm via Block Coordinate Descent: Exact and closed-form solution within each step. B. Wen et al. STROLLR Learning and Restoration

14 Four Major Steps: 1. Sparse Coding 2. Transform Update 3. Low-rank Approximation 4. Patch Restoration

15 STROLLR Algorithm ˆX = argmin X Step 1: Sparse Coding: update X with fixed W. Standard transform-domain sparse coding: W U X 2 F +γ2 s X 0 (1) Cheap hard thresholding: ˆX = H γs (W U). Ŵ = argmin W Step 2: Transform Update: update W with fixed X. Singular Value Decomposition: SΣG T = UX T W U X 2 F s.t. W T W = I n (2) Exact transform update: Ŵ = GS T. B. Wen et al. STROLLR Learning and Restoration

16 STROLLR Algorithm ˆD i = argmin UV i D i 2 F +θ2 rank(d i) (3) D i Step 3: Low-Rank Approximation: solve for D i i: Block matching to form UV i. Apply SVD: ΦΩΨ T = UV i : û i = argmin u i Low-rank Approximation: ˆD i = ΦH θ (Ω)Ψ T. W u i x i 2 2 +γf Ai ui yi 2 2 +γl Step 4: Patch Reconstruction: solve for u i, with fixed W, X, and {D i }. Different restoration problems apply different A i. j C i u i D j,i 2 2 (4) B. Wen et al. STROLLR Learning and Restoration

17 Image Inpainting and Denoising û i = {(1+ C i γ l)i n +γ fa i} 1 (W T x i +γ l D j,i +γ fa iy i) (5) j C i Inpainting: A i is diagonal binary matrix i. Least squares solution to (4), with cheap inversion of diagonal matrix. û i = argmin u i Denoising: special case when A i = I i. W u i x i 2 2 +γf ui yi 2 2 +γl Simple solution as weighted average: û i = (W T x i +γ f y i +γ l j C i D j,i )/(1 +γ f + C i γ l ). j C i u i D j,i 2 2 (6) B. Wen et al. STROLLR Learning and Restoration

18 Example: Image Face Corrupted measurement with 90% pixels missing Inpainted image using STORLLR PSNR = 28.1 db Testing Images (size)

19 Inpainting results: Random pixel removal Additive Gaussian noise PSNR (db) Testing Images (size)

20 Testing Images (size)

21 Take-home message Prior works on local and non-local image structures Sparsifying TRansfOrm Learning and Low-Rank (STROLLR) combines both local sparsity and non-local structure in a single variational formulation. STROLLR modeling and restoration with efficient algorithms. Applications: Image Denoisng, inpainting, etc. B. Wen et al. STROLLR Learning and Restoration

22 Thank you! Questions?? B. Wen et al. STROLLR Learning and Restoration