Adaptive Kernel Regression for Image Processing and Reconstruction Peyman Milanfar* EE Department University of California, Santa Cruz *Joint work with Sina Farsiu, Hiro Takeda AFOSR Sensing Program Review, June 21, 2007
Summary Motivation: Existing methods for processing, reconstruction, and interpolation of spatial data are too strongly dependent upon underlying assumptions about signal and noise models. Develop universal and robust methods based on adaptive nonparametric statistics for processing and reconstruction of image and video data. Goal: Demonstrate the applicability of the Kernel Regression framework to a wide class of problems, producing algorithms competitive with state of the art.
Outline Introduction and Motivation Classic Kernel Regression Data-Adaptive Kernel Regression Experiments Conclusion
Data Sampling Scenarios Possibly random positions Full samples Incomplete samples Irregular samples Denoising Denoising + interpolation Denoising + reconstruction Want a unified method for treating all these scenarios H. Takeda, S. Farsiu, and P. Milanfar, "Kernel Regression for Image Processing and Reconstruction" IEEE Trans. on Image Processing, vol. 16, no. 2, Feb. 2007
Outline Introduction and Motivation Classic Kernel Regression Data-Adaptive Kernel Regression Experiments Conclusion
Kernel Regression Framework The data model Zero-mean, i.i.d noise (No other assump.) A sample The sampling position The number of samples The regression function The specific form of may remain unspecified.
Local Approximation in KR The data model Local representation (N-terms Taylor expansion) Note Unknowns We really only need to estimate the first unknown,. Other localized representations are also possible.
KR Optimization Problem The local representation is valid when x i and x are close. We should give a higher weight to nearby samples The regression order N+1 terms This term gives the estimated pixel value at x The choice of the kernel function is open. e.g. Gaussian Smoothing parameter Weighted Least Squares
Classic Kernel Regression as a Locally Linear Filter The Net Effect of Classic L 2 KR Equivalent kernel
Moving Beyond Classical Four factors to be considered Kernel function ( ): The Regression order ( ): Bias/Variance tradeoff Basis Function: Different polynomial bases, wavelets etc., over-complete dictionaries Smoothing matrix ( ): Data-adaptive choice will produce locally nonlinear filters with supeiror performance
Outline Introduction and Motivation Classic Kernel Regression Data-Adaptive Kernel Regression Experiments Conclusion
Data-Adapted Kernels Classic kernel Classic Kernels Data-adapted kernel Data Adapted Kernels
A Special Case: The Separable Kernel Consider the denoising problem with regularly sampled data Spatial kernel Radiometric kernel Special case: N = 0, m=2 Radiometric smoothness With Gaussian Kernels, this is just the Bilateral filter! (Tomasi, et al. 98) N>0 can generalize the bilateral filter!
Bilateral Kernel is Unstable Essentially noiseless case Fairly Noisy case
Steering Kernel Method Local dominant orientation estimate based on local gradient covariance 5 10 15 20 25 30 35 40 45 50 5 10 15 20 25 30 35 40 45 50
Better Approach: Parameterization Scaling parameter Elongation matrix Rotation matrix.
Steering Kernel is Stable! Essentially noiseless case Fairly Noisy case
The Net Effect of Kernel Regressors in L 2 Classic Kernel: Locally Linear Filtering: Depends only on x-x i Steering Kernel: Locally Non-Linear Filtering: Depends on both x-x i and y-y i
Iterated Application of Steering Steering kernel: Direction Strength 5 10 Given data 15 20 25 30 35 40 45 50 5 10 15 20 25 30 35 40 45 50 Orientation estimation Iteration! Steering kernel regression Takeda, H., S. Farsiu, P. Milanfar, Kernel Regression for Image Processing and Reconstruction, IEEE Transactions on Image Processing, Vol. 16, No. 2, pp. 349-366, February 2007.
Iterative Algorithm Initialize: Noisy data Initial Gradient Est. Smoothing Matrix Est. Kernel Reg. Iterate: Smoothing Matrix Est. Kernel Reg.
Outline Introduction and Motivation Classic Kernel Regression Data-Adaptive Kernel Regression Experiments Conclusion
Gaussian Noise Removal (STD=25) Original image Noisy image, RMSE=24.87 (SNR=5.69dB)
Gaussian Noise Removal (STD=25) Portilla and Simonceli, RMSE=6.64 Bayes Least Square-Gaussian Scale Mixture (2003) Iterative steering kernel regression, N=2, RMSE=6.63 Kervrann, et al. RMSE=6.61 Optimal spatial adaptation (2006) Buades, et al. Non-Local Means, RMSE=8.45
Gaussian Noise Removal (STD=25) Portilla and Simonceli, RMSE=6.64 Bayes Least Square-Gaussian Scale Mixture (2003) Iterative steering kernel regression, N=2, RMSE=6.63 Kervrann, RMSE=6.61 Optimal spatial adaptation (2005) Non-Local Means, RMSE=8.45
Compression Artifact Removal (Deblocking) Original pepper image JPEG with quality of 10, RMSE=9.76
Compression Artifact Removal (Deblocking) Portilla and Simonceli, RMSE=8.80 Bayes Least Square-Gaussian Scale Mixture (2003) Iterative steering kernel regression, N=2, RMSE=8.48 Kervrann, RMSE=8.85 Optimal spatial adaptation (2005) Non-Local Mean, RMSE=8.64
Film Grain Reduction (Real Noise) Real noisy image
Film Grain Reduction (Real Noise) Portilla and Simonceli, Bayes Least Square-Gaussian Scale Mixture, Denoising on YCrCb channels Iterative steering kernel regression, N=2, Denoising on YCrCb channels Kervrann, Optimal spatial adaptation, Denoising on YCrCb channels Non-Local Mean, Denoising on YCrCb channels
Comparison in Absolute Residual Image in the Y channel Portilla and Simonceli, Bayes Least Square-Gaussian Scale Mixture, Denoising on YCrCb channels Iterative steering kernel regression, Denoising on YCrCb channels Kervrann, Optimal spatial adaptation, Denoising on YCrCb channels Non-Local Mean, Denoising on YCrCb channels
Real Noise Reduction Real noisy image
Real Noise Reduction Portilla and Simonceli, Bayes Least Square-Gaussian Scale Mixture, Denoising on YCrCb channels Iterative steering kernel regression, N=2, Denoising on YCrCb channels Kervrann, Optimal spatial adaptation, Denoising on YCrCb channels Non-Local Mean, Denoising on YCrCb channels
What about SKR for Interpolation? The Problem: At the location x where we wish to interpolate, there is no pixel value (yet) Denoising Interpolation y i Exist Pixel Exists Pixel Doesn t? The Solution: Produce a pilot, low-complexity, estimate of the pixel? then apply the more sophisticated adaptive kernel techniques described earlier.
Illustration: 30% of Pixels Retained Local Constant Kernel Estimate Steering Kernel Estimate 5 10 15 20 25 30 35 40 45 50 5 10 15 20 25 30 35 40 45 50
Interpolation of Irregular Samples Irregularly downsampled image (Randomly delete 85% of pixels) Iterative steering kernel regression, N=2, RMSE=8.48 Delaunayspline smoother, RMSE=9.05 Classic kernel regression, N=2 RMSE=9.69
Deblurring Taking the PSF into account: Point spread function The model in matrix form, we have Unknown function Shift The Taylor expansion at The Taylor expansion at
Deblurring Example PSF:19x19 uniform, BSNR:40dB The original image The degraded image RMSE = 29.67
Deblurring Example PSF:19x19 uniform, BSNR:40dB ForWaRD RMSE = 14.37 LPA-ICI RMSE =13.36 Data-adaptive kernel reg. RMSE = 14.21
Super-resolution Example Motion Estimation Image Adaptive Reconstruction Kernel (Interpolation, Regression Denoising, and Deblurring) Resolution enhancement from video frames captured by a commercial webcam (3COM Model No.3719)
Conclusion Adaptive kernel regression is a powerful tool for a wide class of problems. It provides results competitive with state of the art in denoising, interpolation, deblurring, resolution enhancement. Don t try this at home. Other examples and software are available at http://www.soe.ucsc.edu/~htakeda