EFFICIENT PERCEPTUAL, SELECTIVE,

Transcription

1 EFFICIENT PERCEPTUAL, SELECTIVE, AND ATTENTIVE SUPER-RESOLUTION RESOLUTION Image, Video & Usability (IVU) Lab School of Electrical, Computer, & Energy Engineering Arizona State University

2 Outline Problem statement Existing super-resolution (SR) approaches Proposed Perceptual Attentive (PA) SR Framework Simulation results Conclusion

3 Problem Statement Super-Resolution (SR) image re-construction is the process of combining the information from multiple Low-Resolution (LR) aliased and noisy frames of the same scene to estimate a High-Resolution (HR) unaliased and sharp/de-blurred image. - High computational complexity requirements. - No perceptual model incorporated in existing approaches.

4 SR Observation Model The values of the pixels in the k-th low- resolution frame Y k of the sequence can be expressed in matrix notation as: k = D k. H k. Fk. z + n k, k = 1,..., K y Warping g( (F k ) where y k and z are, respectively, the lexicographic form of Y k and the undegraded HR image Z, n k is the additive i noise and D k, H k, F k are the downsampling, blurring, & sub-pixel warping matrices. Y k : N 1 N 2 pixels; Z: rn 1 x rn 2, where r is the magnification factor. Inverting (D k.h k.f k ) to obtain z is not a trivial task especially that the system is ill-conditioned Blurring (H k ) Downsampling (D k )

5 Existing Super-Resolution Approaches Iterative SR methods - Projection onto convex sets (POCS) [Stark et al., 1989] - Least-square error minimization [Schultz et al., 1996] - Regularized maximum a Posteriori i (MAP) methods [Hardie et al., 1997] Fusion-Restoration (FR) SR methods (also known as twostep methods) [Elad et al., 2001; Farsiu et al., 2004; Hardie et al., 2007] - Non-iterative fusion step followed by a restoration step. - More computationally efficient

6 MAP-Based SR Algorithm [Hardie et al. 97] The HR image estimate can be computed by maximizing the a posteriori probability, Pr(X {Y k }), or by maximizing the log-likelihood function: z ( log[pr( z )]), k 1 K = arg max y k =,..., This results in the following cost function to be minimized assuming a Gaussian distribution for the noise and Pr(z {y k }): 1 ( T 1 T 1 f ( z) = y Wz )( y Wz) + z C z 2 2 σ 2 2 n where y is a vector concatenating all the LR observations y k, k=1,..,k, σ n is the noise standard deviation and C is the covariance matrix of z, W is the degradation matrix. An iterative gradient descent minimization procedure is used to update the HR estimate as follows: z l + 1 = z l ε l f ( z ) z= = z where ε l is the step size l

7 Fast Two-Step (FTS) SR Algorithm [Farsiu04] Fast Two-Step SR (FTS) algorithm consists of a non-iterative data fusion step followed by an iterative gradient-descent deblurring step. Data Fusion Step: Estimated by registration followed by a median operator resulting in a blurred version of the HR frame, Xˆ = H. Zˆ, where Xˆ is the deblurred HR frame. Deblurring Step: Reduces to the problem of estimating from and can be formulated as Zˆ = arg min A ( HZ Xˆ ) + λ. Γ ( Z ) λ is the regularization parameter A is a diagonal matrix with values equal to the square root of the number of pixels that contribute to generate the data fusion matrix Ẑ Γ is a bilateral total variation regularization term given by: Γ BTV where the operators S shift X by l and m pixels in the horizontal and vertical directions x y generating several scales of derivatives. Is a spatially decaying factor 0 < α < 1 BTV P P l + m l m ΓBTV ( Z) = α Z S x. S y. Z l + m 0 1 l= Pm= 0 l m S, S α 1 Xˆ Ẑ

8 Fast Two-Step (FTS) SR Algorithm [Farsiu04] (Cont.) The solution of the deblurring step can be obtained using an iterative gradient- descent scheme as follows: Zˆ λ { = ˆ T T + ( ˆ ˆ 1 Zn β H A sign AHZ n X ) + n P P l + m ( l m ) ˆ l m. α I S. (.. ˆ x S y sign Z n S x S y Z n ) l= Pm= 0 β where l + m 0 and is the step size in the direction of the gradient. At each iteration, every pixel on the HR grid is processed through the deblurring stage in original SR method Results in a relatively high computational complexity.

9 Proposed Perceptual Attentive (PA) SR Framework Strategy Reduce the computational complexity by finding a perceptually significant constraint set of pixels to process while maintaining the desired HR visual quality. Perceptual contrast sensitivity threshold model used to determine perceptually significant pixels (active pixels). Visual Attention information of salient regions used to further reduce the set of processed pixels and to processed active pixels in attentive regions with a higher accuracy.

10 Proposed Perceptual Attentive (PA) SR Framework A computationally efficient Perceptual Attentive (PA) Super-Resolution framework that utilizes available visual attention information of perceived edges is presented to significantly reduce SR computations. The proposed method divides the image into: Active regions consisting of pixels on perceptual edges determined by contrast sensitivity threshold model. Attentive active regions determined by active pixels lying in the visually attended regions. At each iteration of the SR algorithm, the active pixels on visible ibl edges determined d by the contrast sensitivity threshold model and lying inside the attentive region are treated with higher accuracy than the pixels in the background region. Bilinearly interpolated LR image (q =4) Perceptual Active pixels (Active Background) Visual Attention regions VA and Perceptual Active pixels (Active Forground)

11 Proposed Perceptual Attentive (PA) SR Framework Let z 0 = initial HR estimate Determine active pixels using perceptual JND model Update HR estimate for active pixels No Yes Determine active pixels in attentive regions using perceptual JND model and Saliency Map Update HR estimate for active pixels in attentive regions Yes No No Max Iterations Reached? Yes Yes Max Iterations Reached? No STOP

12 Perceptual Contrast Sensitivity Threshold Model The contrast sensitivity threshold is used to detect the set of perceptually significant pixels (active pixels). The contrast sensitivity threshold is the measure of the smallest contrast, or Just Noticeable Difference (JND), that yields a visible signal over a uniform background. JND is computed per image block based on block mean luminance (8 by 8 blocks used) and using initial SR estimate (obtained by interpolating one of the LR images or by applying a median shift and add of the LR images) JND thresholds are precomputed for all possible discrete mean luminances and stored in a look-up tbl table.

13 Perceptual Contrast Sensitivity Threshold Model For each 8x8 image block, the mean of the block is computed and the corresponding JND threshold t JND is retrieved (Look up Table). Once the t JND is obtained, the center pixel of a sliding 3 3 window is compared to its 4 cardinal neighbors. If any absolute difference is greater than the t JND, then the corresponding pixel mask is flagged as 1 and the pixel is labeled as active pixel. The window will scan all the pixels of the HR image estimate.

14 Perceptual Contrast Sensitivity Threshold Model The luminance-adjusted contrast sensitivity JND thresholds are approximated using a power function [Watson93] as follows: t JND = t N blk 1 N blk 1 n 1 = 0 n = 0 N 2 blk 2 I (128 n 1, n ) 2 a T 128 where is set to The contrast sensitivity threshold t 128 is evaluated by mapping a parabolic approximation of a perceptual model proposed by [Ahumada92] to the spatial domain: Lmax: maximum luma t 128 = L max TM g L min a T Lmin: minimum luma Mg: # of grayscale levels T based on the model proposed by [Ahumada92]

15 Visual Attention (VA) HVS perceives a large field of view by a number of fixation points, attended to with high visual acuity, connected by fast eye movements referred to as saccades. VA models provide a relative quantitative measure of the attended regions. It is stated that artifacts in salient regions are likely to be more annoying to the observer than artifacts in less salient regions.

16 Visual Attention VA Models produce saliency maps The saliency map (SM) is simply a likelihood map in which regions with large values have higher probability of being selected by the HVS as a fixation region as compared to regions with lower values. The resulting attended regions are used in the proposed Perceptual Attentive (PA) Super-Resolution method to promote the detected subset of attended edge pixels for further enhancement. The visual attention model purposed by Itti et al. [Itti98, Koch06] is used in our implementation Any other good VA model can be used with the proposed p framework.

17 Simulation results Fusion-Restoration SR high-resolution images are used to produce, each, a simulated sequence of sixteen low-resolution images: Blurred with a 4x4 Gaussian filter with std=1. Shifted by multiples of 0.25 pixels in all different directions. Subsampled by a factor of 4 in each direction. An additive Gaussian noise of variance 10 is added to the resulting LR sequence. Clock-256x256 HR Frame LR Frames Carrier-256x256 HR Frame LR Frames Cameraman-256x256 HR LR Frames

18 Simulation results FTS [Farsiu 04] SELP-FTS [Proposed] PA-FTS [Proposed] (a) Original image (b) Bilinearly interp. LR image with 4 VA regions (c) FTS method (d) SELP-FTS method, ε = (e) Proposed PA-FTS, ε=0.0001, s=10, VA regions=4 Visual performance results for the 256x256 images with a resolution enhancement factor 4, number LR images = 16, additive Gaussian noise variance = 10, and Parameters λ = 0.08, β = 1, α = 0.6, P = 2.

19 Simulation results (cont.) (a) Original image (b) Bilinearly interp. LR image with 5 VA regions. (c) FTS SR method (d) SELP-FTS SR method, ε = (e) Proposed PASR-FTS, ε=0.0001, s=10, VA regions =4 Visual performance results for the 256x256 images with a resolution enhancement factor 4, number LR images = 16, additive Gaussian noise variance = 10, and Parameters λ = 0.08, β = 1, α = 0.6, P = 2.

20 Simulation results (cont.) CLOCK image Number of processed pixels at each iteration

21 Simulation results (cont.) CAMERAMAN image Number of processed pixels at each iteration

22 Simulation results (cont.) CARRIER image Number of processed pixels at each iteration

23 Simulation results (cont.) PSNR and Percentage Pixel Savings Results CARRIER CLOCK PLANE CAMERAMAN PSNR (db) Pixel Savings PSNR (db) Pixel Savings PSNR (db) Pixel Savings PSNR (db) Pixel Savings Bicubic % % % % FTS % % % % SELP-FTS % % % % PA-FTS % % % %

24 Conducted Subjective Test Set of images used are Carrier, Clock, Fighterplane, and Cameraman from USC database. From the sequence of the low resolution images estimate the HR image for a set of four images (magnification factor r = 4) using FTS [Farsiu04], proposed SELP-FTS, and the proposed PA-FTS method. The SR images are displayed side by side for comparison. Each case is randomly repeated 4 times with the left and right images swapped. SR images produced by the perceptual selective methods (SELP or PA-FTS) compared to the non- selective FTS method and rated from 1-5 corresponding to worse, slightly worse, same, slightly better, and better. Left Image Quality Worse Slightly Worse Same Slightly Better Better Right Image Quality (1) (2) (3) (4) (5)

25 Conducted Subjective Test The Mean Opinion Score (MOS) is calculated by averaging the responses of all the subjects for each different pair of images. If MOS > 3 means that selective SELP-FTS and PA-FTS are Better. If MOS < 3 means that nonselective FTS is Better. 10 Subjects took the test. MOS Results: Methods Carrier CLock Fighter Cameraman Average SELP FTS vs. FTS PA FTS vs. FTS

26 Simulation results MAP based SR high-resolution images are used to produce, each, a simulated sequence of four low-resolution images: Blurred with a 4x4 average filter. Shifted by 0.25 pixels in 4 different directions. Subsampled by a factor of 4 in each direction. An additive Gaussian noise of variance 4 is added to the resulting LR sequence. Clock-256x256 HR Frame LR Frames Carrier-256x256 HR Frame LR Frames Parrots-256x256 HR Frame LR Frames

27 Simulation results MAP SR [Hardie 97] SELP SR [Proposed] PASR [Proposed] (a) Original image (b) Bilinearly interp. LR image with 3 VA regions. (c) MAP SR method (d) SELP SR method, ε = (e) Proposed PASR, ε=0.0001, s=15, VA regions =3 Visual performance results for the 256x256 CLOCK image with a resolution enhancement factor 4, number LR images = 4, and additive Gaussian noise variance = 4, regularization operator = 150. Visual performance results for the 256x256 CARRIER image with a resolution enhancement factor 4, number LR images = 4, and additive Gaussian noise variance = 4, regularization operator = 150.

28 Simulation results (cont.) (a) Original image (b) Bilinearly interp. LR image with 4 VA regions. (c) MAP SR method (d) SELP SR method, ε = (e) Proposed PASR, ε=0.0001, s=10, VA regions =4 Visual performance results for the 256x256 PARROTS image with a resolution enhancement factor 4, number LR images = 4, and additive Gaussian noise variance = 4, regularization operator = 150.

29 Simulation results (cont.) CLOCK image (a) SNR gain (b) Number of active pixels at each iteration

30 Simulation results (cont.) CARRIER image (a) SNR gain (b) Number of active pixels at each iteration

31 Simulation results (cont.) PARROTS image (a) SNR gain (b) number of active pixels at each iteration

32 Simulation results (cont.) PSNR and Percentage Pixel Savings Results PSNR (db) CLOCK CARRIER PARROTS Pixel Savings PSNR(dB) Pixel Savings PSNR(dB) Pixel Savings Bilinear % % % MAP SR % % % SELP SR % % % PASR % % %

33 Conducted Subjective Test Set of images used are Carrier, Clock, Fighter-plane from USC database and Parrots from LIVE database. From the sequence of the low resolution images estimate the HR image for a set of four images (magnification factor q = 4) using MAP [Hardie97], the proposed SELP, and the proposed p PASR method. The SR images are displayed side by side for comparison. Each case is randomly repeated 4 times with the left and right images swapped. Right image is compared to the left image and rated from 1-5 corresponding to worse, slightly worse, same, slightly better, and better. Left Image Quality Worse Slightly Worse Same Slightly Better Better Right Image Quality (1) (2) (3) (4) (5)

34 Conducted Subjective Test The Mean Opinion Score (MOS) is calculated by averaging the responses of all the subjects for each different pair of images. If MOS > 3 means that Proposed PASR is Better. If MOS < 3 means that SELP or MAP SR are Better. Six Subjects took the test. MOS Results: Methods Carrier CLock Fighter Parrots Average MAP vs. PASR SELP vs. PASR

35 Conclusion A Perceptually Attentive Super-Resolution (PASR) method is presented to reduce the computational complexity while maintaining the visual quality for SR. During each iteration, only a subset of active pixels is selected for SR processing based on a locally computed JND thresholds. The active pixels are further reduced by classifying them into non-attentive and attentive areas using visual attention. The attended regions are further iterated upon in order to achieve a higher accuracy in these regions by setting a lower stopping threshold as compared to the nonattended region. Significantly reduces the number of pixels processed at each iteration without causing a visually perceived loss in quality. Simulation results showed 65-74% increase over the non-selective FTS scheme and a 30-40% increase in pixel savings over a highly efficient selective perceptual but not attentive SR scheme (SELP-FTS). More significant savings when applied to MAP SR.

36 References [Itti98] L. Itti, C. Koch, E. Niebur, A model of saliency-based visual attention for rapid scene analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 20, Issue 11, pp , 1259 Nov [Koch06] D. Walther, C. Koch, Modeling attention to salient proto-objects, Neural Networks, Vol. 19, n.9, pp , Nov [Ahumada92] A. J. Ahumada, Jr. and H. A. Peterson, "Luminance-model-based model DCT quantization for color image compression," SPIE Human Vision, Visual Processing, and Digital Display III, vol. 1666, pp , [Watson 93] A. B. Watson, "DCT quantization matrices visually optimized for individual images," SPIE Human Vision, Visual Processing, and Digital Display IV, vol. 1913,,pp pp , [Farsiu04] S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, Fast and robust multiframe super-resolution, IEEE Transactions on Image Processing, vol. 13, pp , [Hardie 97] R. C. Hardie, K. J. Barnard and E. E. Armstrong, Joint MAP registration and high-resolution image estimation using a sequence of undersampled images, IEEE Transactions on Image Processing, vol. 6, no. 12, pp , Dec [Ferzli 08] R. Ferzli, Z. A. Ivanovski and L. J. Karam, "An efficient, selective, perceptual-based superresolution estimator," IEEE International Conference on Image Processing, Oct [Sadaka09] N. Sadaka and L. J. Karam, Efficient, Perceptual, Attentive, Superresolution, IEEE International Conference on Image Processing, 2009.