Image Super-Resolution via Sparse Representation

Transcription

1 Image Super-Resolution via Sparse Representation Jianchao Yang, John Wright, Thomas Huang and Yi Ma accepted by IEEE Trans. on Image Processing 2010 Presented by known 2010/4/20 1

2 Super-Resolution Techniques that enhance the resolution of an imaging system Low-cost imaging sensor High-definition display 2

3 Previous Approaches Register multiple low-resolution images insufficient # of input images ill-conditioned registration unknown blurring operators Interpolation based Simple interpolation over-smooth Using prior limited in modeling textures 3

4 Previous Approaches(2) Machine learning based Learn the co-occurrence of low and high resolution image patches Need enormous patch pairs Fix k neighbors Markov random field [Freeman et. al. IJCV 00] Primal sketch prior [Sun et. al. CVPR 03] Neighbor embedding [Chang et. al. CVPR 04] Soft edge prior [Dai et. al. ICCV 07] Trained patches Input Output Original 4

5 Key Idea High-resolution patches have a sparse linear representation with respect to an compact learned overcomplete dictionary of patches randomly sampled from similar images Trained patches Input Output Original * Patches are overlapped 5

6 * Patches are overlapped Key Idea low-resolution patches patch pairs LR dictionary high-resolution patches [Training] HR dictionary Input LR image cut into patches For each patches [Testing] = L * α = H * α : 求 α, α is sparse : 求 6

7 Notation the high resolution image the low resolution image the high resolution image patch the low resolution image patch dictionaries for high resolution dictionaries for low resolution bold lower case letter: vector unbold upper case letter: matrix unbold lower case letter: scalar 7

8 Problem and Constraint Problem Formulation Given low resolution image Trained dictionaries, To recover high resolution image Assumption & Constraints Reconstruction constraint Sparse prior S: down-sampling filter H: blurring filter 8

9 Why Sparse? Natural images may generally be describe in terms of a small number of structure primitives(edges, lines) [Field,1994] Filter images with a set of log-gabor filters and collecting histograms of the result high kurtosis [Field 1993] Image source: 9

10 Proposed Approaches Generic image local sparse prior(recover local detail) global constraint (remove artifact) Face image (Face Hallucination) global constraint from face prior local sparse prior 10

11 Local Sparse Prior find a sparse representation correspond to for each low-resolution image patch F : feature extraction operator recover by l 1 -norm (assume the coefficients are sufficiently sparse) [Donoho 2006] Reformulate by Lagrange Multiplier 11

12 Sparse Prior (neighbor) Consider neighbor patches P: extract overlap region w: previous reconstruct on the overlap Reformulate: 12

13 Global Constraint S: down-sampling filter H: blurring filter to ensure the reconstructed image X 0 satisfy global constraint Solve by back-projection method s: up-sample scale factor p: back-projection filter ( 類似 deblur) Gradient descent 2 version (?!) 13

14 Generic image SR(algo 1) 1. local sparse prior (recover local detail) 2. global constraint (remove artifact) 14

15 Global Optimization D: down-sampling filter H: blurring filter Simultaneously solve the coefficients of all the patches 從 X 取出 patch (i,j) Penalty function encoded prior knowledge 15

16 Proposed Approaches Generic image local sparse prior(recover local detail) global constraint (remove artifact) Face image (Face Hallucination) global constraint from face prior local sparse prior 16

17 Face Prior Using Non-negative matrix factorization(nmf) A(nxm): training face images (column) U(nxr): trained face basis V(rxm): coef correspond to each train face 17

18 Using Face Prior to find p(x Y) find Maximum a posteriori (MAP) find coef c* that satisfy: prior S: down-sampling filter H: blurring filter U: trained face basis high-pass filter willing smooth medium resolution image 18

19 Face image SR (algo 2) 1. global constraint from face prior 2. local sparse prior 19

20 Learning Dictionary Pairs Goal to learn compact dictionary pairs Single Dictionary Training t 個 patches Joint Dictionary Training 20

21 Joint Dictionary Training N: # dimension of high res patches M: # dimension of low res patches 21

22 Dictionary Training To Solve: 1. initialize D with Gaussian random matrix 2.Fix D, update Z 3.Fix Z, update D 4. iterate 2,3 until converge 22

23 Features for low res patches Goal to ensure the computed coefficients fit the most relevant part of the low-resolution images Idea using some kind of high pass filter Method apply 1 st and 2 nd order derivatives on upsampled(2 倍 ) whole low-resolution image separate into patches 23

24 Experiment Single image super-resolution Generic super-resolution Face super-resolution Effect of Dictionary Size Robustness to Noise 24

25 Experiment setup Generic image Low resolution patch 3x3(up to 6x6) overlap 1 pixel High resolution patch 9x9 overlap 3 pixel Face image 5x5 for both low and high resolution Dictionary Train from 100,000 patches Size

26 Input bicubic neighbor-embedding proposed origin Input bicubic back-projection neighbor-embedding soft edge prior proposed 26

27 Face Super-resolution Input two-step generic Input bicubic back-proj proposed origin NMF+biliter filter 27

28 Effect of dictionary size RMS error 2 version (?!) 28

29 29

30 Robustness to noise The equation of sparse prior can be formulated as a probabilistic model Laplacian Gaussian 2 version (?!) 30

31 Conclusion A novel approach toward single image super-resolution based on sparse representations are presented jointly trained coupled dictionaries from high- and low-resolution image patch pairs. enforced the compatibilities among adjacent patches both locally and globally Experimental results demonstrate the effectiveness of the sparsity as a prior for patch-based super-resolution both for generic and face images. 31

32 Future Works Determine the optimal dictionary size for natural image patches Tighter connections to the theory of compressed sensing conditions on the appropriate patch size conditions on the appropriate features approaches for training the coupled dictionaries 32

33 THANK YOU ~ ~ 33

34 Lagrange Multiplier From wiki 圖及數學知識網站 平常要求 f(x,y) 的極值 求解 若 x,y 的值被 g(x,y)=0 所限制 改求 f(x,y)+λg(x,y) 的極值求解 34

35 Solution to l 0, l 1, l 2 sparse x 2 1-l 0 ball x 2 1-l 0 ball x 1 x 1 Equivalence breakdown point(ebp) 1-l 1 ball x 2 1-l 0 ball x 2 1-l 0 ball x 1 x 1 1-l 1 ball 1-l 2 ball 35

36 How to estimate EBP? 放大 l 1 ball C, 則 P 也會跟著放大, 當 P 的表面碰到 y 的時候的 x, 就是 x 1 也就是 x 0 1-l 1 ball Theorem: 如上圖,C 是一個 1-l 1 ball, 若 P(=AC) 是 k-neighborly, 且 y=ax0 中 x0 是 k-sparse 則以 l 1 -minimization 求出來的解即為 x0, 且是唯一解 Definition: If P is k-neighborly C 上 (k-1)d 表面都可以對到 P 上 Properties If x0 is k-sparse 與 1-l 1 ball 的 (k-1)d 表面有交點 這些表面都是 simplex (n-d simplex, n+1 vertices) P=AC 36