Adaptive Multiple-Frame Image Super- Resolution Based on U-Curve IEEE Transaction on Image Processing, Vol. 19, No. 12, 2010 Qiangqiang Yuan, Liangpei Zhang, Huanfeng Shen, and Pingxiang Li Presented by In-Yong Song School of Electrical Engineering and Computer Science Kyungpook National Univ.
Abstract Super-resolution reconstruction Recovery of high-resolution image from low-resolution images Noisy, blurred and down-sampled Use of maximum a posteriori model (MAP) Generation of noise and blurry depending on parameter Adaptively selecting optimal regularization parameter Proposed method Adaptive MAP reconstruction method based upon U- curve 2/ 34
Previous method Introduction Frequency domain method Tsai and Hung Representation of multiple-frame SR problem Kim et al Improving Tsai and Huang method» Considering observation noise and spatial blurring Bose et al. Recursive total least squares method Rhee and Kang Discrete cosine transform (DCT) based method Wavelet transform-based SR methods 3/ 34
Spatial domain method Non-uniform interpolation approach Iterative back projection (IBP) approach Projection onto convex sets (POCS) approach Deterministic regularized approach Maximum likelihood (ML) approach Maximum a posteriori (MAP)approach Joint MAP approach Hybrid approach 4/ 34
Proposed method Based on MAP reconstruction model Regularization parameter Important role Controlling tradeoff between fidelity and prior item Disadvantage Time-consuming and subjective Avoiding disadvantage Use of adaptive selection method 5/ 34
Adaptive selection methods Using classical method Bose et al» Use of L-curve method Nguyen et al.» Use of cross validation method (GCV) Using general Bayesian framework Kang and Katsaggelos, He and Kondi, Molina et al. and Zibetti et al. 6/ 34
Advantage and Disadvantage of each method L-curve and GCV approach» Providing good solution» High cost Bayesian framework method» Lower computational load» Attaching parameter and parameter distribution function U-curve method Proposed by Krawczy-Stando and rudnicki Selecting regularization parameter in inverse problems 7/ 34
Observation Model Description of observation model Degradation process form HR image to LR image Assuming HR image to be shifted, blurred, downsampled, and additive noise Fig. 1. Degradation model of the HR image. yk DkBkMkx nk (1) l l 2 where 1 and be the down-sampled factors for rows and columns M k stands for the warp matrix, Bk is the blurring matrix (PSF) D is the down-sampling matrix, n is the noise vector. k k 8/ 34
Assuming down-sample factors and blurring function Same between LR images ( Dk, Bk D, B ) Representation of whole observation model y1 DBM1x n1 y2 DBM2xn 2 y DBMx n (2) yp DBM pxn p T T where y [ y, y y ], M [ M, M M ] and [, ] T. 1 2 p 1 2 p 1 2 n n n n p 9/ 34
MAP Reconstruction Model Formulation of MAP Reconstruction model Adding some prior information about HR image to regularize SR problem Formulation of reconstruction function using MAP model and how to solve Presentation of estimated HR image Using Bayes rule ^ x arg max px ( y) (3) ^ py ( xpx ) ( ) x arg max py ( ) (4) 10 / 34
Independent of py ( y x) py ( ) Minus log of the functional in (8) ^ 1 p Zero-mean Gaussian noise and independent LR frame 1 ^ x arg max py ( y xpx ) ( ) (5) where 1 p is the likelihood distribution of the LR images, and px ( ) is the prior distribution of the HR images. p x arg min (log py ( y x) log px ( )) (6) p 1 p yk k k 1 p 2 2 k 1 2 k k 2 DB M x py ( y x) exp (7) 11 / 34
Use of Prior model in this paper 1 1 px ( ) exp Qx C 2 (8) where is a parameter that controls the variance of the prior distribution and Q represents a linear high-pass operation that penalizes the estimation that is not smooth. Substituting (7) and (8) in (6) Cost function ^ 2 2 x arg min J( x) arg min ydbmx Qx (9) 12 / 34
Optimization Differentiating (9) with respect to x J( x) 2 M T B T D T ( ydbmx) 2 Q T Qx 0 (10) Successive approximation iteration n 1 n n n x x r (11) n 1 T T T T r J( x) M B D ( ydbmx) Q Qx (12) 2 n Termination condition of iteration x DBMr n 1 x x n 2 ( r ) n n n T n 2 r 2 n 2 Qr (13) d (14) 13 / 34
U-Curve Method Description of novel approach Estimating regularization parameter based on U- curve First, inducing principle of U-curve and its properties Second, Use of u-curve method in SR problem U-Curve and Its Properties Expression similar to traditional Tikhonov regularization model Writing traditional model of Tikhonov regularization Cost function in (9) ^ 2 2 x arg min y Ax Qx (15) 14 / 34
Using singular value decomposition (SVD) least squares method 0 T A U V g U y 0 0 gv x ^ a r i i i 2 2 i1 ( i qi ) (16) From functions (15) and (16) 2 qg R( ) y Ax (17) P( ) Qx 2 r 2 4 2 i i i1 2 2 2 ( i qi ) qg (18) r 2 2 2 i i i1 2 2 2 ( i qi ) 15 / 34
From Previous discussion 1 1 U ( ) R( ) (19) P( ) Fig. 2. Typical U-curve. Three characteristic parts (a) it is monotonically decreasing on the left side (b) it is monotonically increasing on the right side (c) in the middle it is almost horizontal with monotonous change 16 / 34
U-curve properties (4/3) a) for (0,( r / q1) ),the function U ( ) is strictly decreasing (4/3) b) for (( 1 / q r ), ), the function U ( ) is strictly increasing c) lim U ( ) 0 d) lim U ( ) Remark: The optimal regularization parameter can be selected in the (4/3) (4/3) interval (( r / q1),( 1/ qr) ), and objective of the U-curve criterion for selecting the regularization parameter is to choose a parameter for which the curvature of the U-curve attains a local maximum close to the left part of the U-curve 17 / 34
Selection Steps Determining optimal regularization parameter Singular value decomposition Incorporate the result of Step 1 Plot the u-curve Select the maximum curvature point 18 / 34
Experimental Results Evaluation of reconstruction results Use of mean square error (MSE) and structural similarity (SSIM) 1 MSE x x ln ln 1 1 2 2 2 (20) SSIM (2 uu C)(2 C) ( u u C )( C ) x x 1 xx 2 2 2 2 2 x x 1 x x 2 (21) where x represents the original HR image, and x represents the reconstructed hr image. Comparing previous and proposed method 19 / 34
Simulated Data Image of cameraman, boat, and castle Classification of two cases Degradation parameters known case Degradation parameters unknown case Termination condition of iteration x n1 2 n x 10 2 x n 6 20 / 34
1) Degradation parameters known case Case 1 PSF of 3x3 window size» 0.5 variance Down-sampled factor 2 Zero-mean Gaussian-noise» 0.01 variance Case 2 PSF of 5x5 window size» 1.0 variance Down-sampled factor 2 Zero-mean Gaussian-noise» 0.02 variane 21 / 34
Parameter of image Table 1. Displacement Parameters of the Four LR Images. Table 2. PSF and Noise Parameters of Case 1 and Case 2. Fig. 3. L-curve, U-curve, and the selected regularization parameter of the cameraman image and boat image in Case 1 and 2. 22 / 34
Result of Case 1 Fig. 4. Reconstruction results of the cameraman image in Case 1. (a) Original HR image. (b) LR image. (c) BI. (d) Adaptive iteration. (e) L-curve (f) U-curve. Fig. 7. Reconstruction results of the boat image in Case 1. (a) Original HR image. (b) LR image. (c) BI. (d) Adaptive iteration. (e) L-curve. (f) U-curve. 23 / 34
Fig. 5. Detailed regions cropped from Fig. 4. (a) Original HR image. (b) LR image. (c) BI. (d) Adaptive iteration. (e) L-curve (f) U-curve. Fig. 8. Detailed regions cropped from Fig. 7. (a) Original HR image. (b) LR image. (c) BI. (d) Adaptive iteration. (e) L-curve. (f) U-curve. 24 / 34
Fig. 6. Difference between the reconstruction results and the original image in Case 1. (a) BI. (b) Adaptive iteration. (c) L-curve. (d) U-curve. Fig. 9. Difference between the reconstruction results and the original image in Case 1. (a) BI. (b) Adaptive iteration. (c) L-curve. (d) U-curve. 25 / 34
Table 3. MSE and SSIM value of different reconstruction methods in Case 1 (cameraman). Table 4. MSE and SSIM value of different reconstruction methods in Case 1 (boat). 26 / 34
Showing robustness of the proposed method Fig. 14. Reconstruction MSE value under different SNR noise. (a) Cameraman image. (b) Boat image. 27 / 34
Result of Case 2 Fig. 10. Reconstruction results of the cameraman image in Case 2. (a) Original HR image. (b) LR image. (c) BI. (d) Adaptive iteration. (e) L-curve. (f) U-curve. Fig. 12. Reconstruction results of the boat image in Case 2. (a) Original HR image. (b) LR image. (c) BI. (d) Adaptive iteration. (e) L-curve. (f) U-curve. 28 / 34
Fig. 11. Detailed regions cropped from Fig. 10. (a) Original HR image. (b) LR image. (c) BI. (d) Adaptive iteration. (e) L-curve. (f) U-curve. Fig. 13. Detailed regions cropped from Fig. 12. (a) Original HR image. (b) LR image. (c) BI. (d) Adaptive iteration. (e) L-curve. (f) U-curve. 29 / 34
Table 5. MSE and SSIM value of different reconstruction methods in Case 2 (cameraman) Table 6. MSE and SSIM value of different reconstruction methods in Case 2 (boat) 30 / 34
2) Degradation parameters unknown case Fig. 15. Reconstruction results of the castle image. (a) LR image. (b) BI. (c) BCI. (d) Adaptive iteration. (e) L-curve. (f) U-curve. Fig. 16. Detailed regions cropped from Fig. 15. (a) LR image. (b) BI. (c) BCI. (d) Adaptive iteration. (e) L-curve. (f) U-curve. 31 / 34
Real data Fig. 17. Reconstruction results of the text image. (a) LR image. (b) BI. (c) BCI. (d) Adaptive iteration. (e) L-curve. (f) U-curve. Fig. 18. Detailed regions cropped from Fig. 17. (a) LR image. (b) BI. (c) BCI. (d) Adaptive iteration. (e) L-curve. (f) U-curve. 32 / 34
Optimal analysis Showing efficacy of u-curve Use of cameraman image in Case 1 Fig. 19. Change of the MSE value versus the regularization parameter. 33 / 34
Proposed method U-cure method Conclusion Use of Selecting regularization parameter in MAP SR reconstruction model First, Using data fidelity and prior model to construct function for regularization parameter» Plotting U-curve Lastly, selecting optimal regularization parameter Advantage of method First, computational efficiency Second, Selecting more optimal regularization parameter than L-curve 34 / 34