Locally Adaptive Regression Kernels with (many) Applications Peyman Milanfar EE Department University of California, Santa Cruz Joint work with Hiro Takeda, Hae Jong Seo, Xiang Zhu
Outline Introduction/Motivation Locally Adaptive Regression Kernels Applications Adaptive Kernel Regression
Pixels A Modern Paradigm: Measuring Patches Similarity Between Images
At the smallest scale. Similarity between pixels H. Takeda, S. Farsiu, and P. Milanfar, Kernel Regression for Image Processing and Reconstruction, IEEE Trans. on Image Processing, vol. 16, no. 2, pp. 349 366, February 2007
A little bigger scale Similarity between patches Non Local Means (NLM) denoising Block Matching (BM3d) denoising BM3d A. Buades, B. Coll, J.M Morel, "A review of image denoising algorithms, with a new one", SIAM Multiscale Modeling and Simulation, Vol 4 (2), pp: 490 530, 2005 (NLM) K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, Image denoising by sparse 3D transform domain collaborative filtering, IEEE Trans. Image Proc., vol. 16, no. 8, pp. 2080 2095, August 2007 (BM3d)
A little bigger scale Similarity between patches (e.g. Visual Saliency) H.J. Seo and P. Milanfar, Static and Space time Visual Saliency Detection by Self Resemblance, The Journal of Vision 9 (12):15, 1 27, http://journalofvision.org/9/12/15/, doi:10.1167/9.12.15
At the everyday scale.. Similarity between images H.J. Seo and P. Milanfar, Training free, Generic Object Detection using Locally Adaptive Regression Kernels, Accepted for publication in IEEE Trans. on Pattern Analysis and Machine Intelligence H.J. Seo and P. Milanfar, Action Recognition from One Example, Submitted to IEEE Trans. on Pattern Analysis and Machine Intelligence
Defining a point wise measure To measure the similarity of two pixels, consider Spatialdistance Gray level distance y Gray-level Δ Spatial Δ x
Simplest measures Basic ways to incorporate the two Δs: BilateralKernel [Tomasi, Manduchi, 98] (pointwise) Non Local Means Kernel [Buades, et al. 05] (patchwise) y Euclidean distance Gray-level Δ Spatial Δ x
Bilateral Kernels (BL) [Tomasi et al. 98] Pixels Pixel similarity Spatial similarity = Sensitive to 1) noise 2) variation in illumination
Non local Means (NLM) [Buades et al. 05] Patches Patchsimilarity Spatial similarity = Smoothing effect, more robust, but still quite sensitive
Defining a measure More effective ways to combine the two Δs: LARKKernel [Takeda, et al. 07] BeltramiKernel [Sochen, et al. 98] y Signal-induced distance Riemannian Metric Gray-level Δ Spatial Δ x
A digression to Non parametric Kernel Regression The data fitting problem Zero-mean, i.i.d noise (No other assump.) Given samples The sampling position The regression function The number of samples The particular form of may remain unspecified for now.
The data model Locality in Kernel Regression Local representation (N term Taylor expansion) Unknowns Note With a polynomial basis, we only need to estimate the first unknown, Other localized representations are also possible, and may be advantageous.
Optimization Problem We have a local representation with respect to each sample: Optimization N+1 terms The regression order This term give the estimated pixel value z(x). The choice of the kernel function is open, e.g. Gaussian.
What do these weights look like?
Locally Linear/Nonlinear Estimators The optimization yields a point wise linear (but perhaps space varying) estimator: Kernel function The smoothing parameter The weighted linear combinations of the given data Normalized version of K The regression order By choosing a data adaptive kernel (BL,NLM,LARK), the filters become nonlinear.
To summarize Classic Kernel: Locally Linear Filter: Uses distance x-x i Data Adaptive Kernel: Locally Non Linear Filter: Uses x-x i and y-y i
Recall More effective ways to combine the two Δs: LARKKernel [Takeda, et al. 07] BeltramiKernel [Sochen, et al. 98] y Signal-induced distance Riemannian Metric Gray-level Δ Spatial Δ x
LARK Kernels Compute
LARK Kernels Gradient vector field Local covariance matrices Locally Adaptive Regression Kernel: LARK Structure tensor
Gradient matrix over a local patch: Gradient Covariance Matrix and Local Geometry
Image as a Surface Embedded in the Euclidean 3 space Arclength on the surface Chain rule Regularization term Riemannian metric
(Dense) LARK Kernels as Visual Descriptors Measure the similarity of pixels using the metric implied by the local structure of the image H. J. Seo, and P. Milanfar, "Training free, Generic Object Detection using Locally Adaptive Regression Kernels", Accepted in Trans. on Pattern Analysis and Machine Intelligence
Comparisons Image Bilateral Non local Means LARK
Effect of C (covariance matrix) How to estimate matters VS. Naïve estimation of Accurate estimation of Using regularized (SVD)
Robustness of LARK Descriptors 3 3 3 3 1 1 1 1 2 2 2 2 Original Brightness Contrast WGN image change change sigma = 10 1 2 3
A Variant Better suited for Restoration = LARK Measures edginess LSK [*] [*] H. Takeda, S. Farsiu, P. Milanfar, Kernel Regression for Image Processing and Reconstruction, IEEE Transactions on Image Processing, Vol. 16, No. 2, pp. 349 366, February 2007
Some 2 and 3 D Signal Processing Applications Denoising Interpolation Super resolution Deblurring
Gaussian Noise Removal Noisy image, RMSE=24.87 LARK RMSE=6.63 KSVD Elad, et al. (2007) RMSE= 6.89 BM3D Foi, et al. (2007) RMSE=6.35 State of Art for Gaussian Noise
Film Grain Reduction (Real Noise) Noisy image
Film Grain Reduction (Real Noise) LARK
Film Grain Reduction (Real Noise) LARK KSVD BM3D
Adaptive Kernels for Interpolation What about missing pixels? Local metric (covariance) is undefined! Using a pilot estimate, fill the missing pixels: Kernel weights Iteration! Adaptive kernel regression????????????????
Reconstruction from Sparse and Irregular Samples Randomly delete 85% of pixels Reconstruction
Effect of Regression Order The order regression N trades off bias/variance (blur/noise). N+1 terms So far, we have kept N fixed. Ideally, when treating edge areas: want N when treating flat areas: want N
Implicitly changing N Sharpeningthe LARK Kernel Sharpness parameter Laplacian operator
Automatically setting the local sharpness Local Measure of Sharpness: X. Zhu and P. Milanfar, Automatic Parameter Selection for Denoising Algorithms Using a No Reference Measure of Image Content, to appear in IEEE Trans. Image Processing
LARK based Simultaneous Sharpening/Deblurring/Denoising Net effect: aggressive denoising in flat areas Selective denoising and sharpening in edgy areas LARK based filter Locally adaptive denoise/deblur filters
Examples original image original image proposed method State of the art Methods
Examples
Experimental Results 2 Proposed Method
Experimental Results 2 Shan s Method
Application to Enhancing SEM Images 16 low voltage SEM images of the same sample
Application to SEM data Input frame Average frame
Application to SEM data Input frame Average frame, denoised
Application to SEM data Input frame Nonlinear registered merged output
Application to SEM data Input frame Average frame with nonlinear regis. Enhanced
Application to All Focus Imaging Most imaging systems have limited depth of field How to get around this? Make aperture smaller Acquire multiple images, varying the focal length (focus stacking)
Application to All Focus Imaging How to merge these images together? 12 frames
Fusing images using the local sharpness measure The combines image will have the best part of each of the frames as given by The local measure of sharpness given by the singular values of: Input sequence Motion Compensation Sharp Region Detection Locally Adaptive Image Fusion All focus Output
Input sequence Focus Stacking Examples
Output Focus Stacking Examples
Focus Stacking Examples Output (Photoshop CS4)
More Focus Stacking Examples Output (Helicon Focus) Input images (37 frames) Output (Ours) From: http://www.heliconsoft.com/focus_samples.html
Focus stacking Example Input images (24 frames) Output (ours)
Focus stacking Example Output
Super resolution Motion Estimation Image Adaptive Reconstruction Kernel (Interpolation, Regression Denoising, (2D) and Deblurring)
Space Time Descriptors Setup is similar to 2 D, but.. Samples from nearby frames Covariance matrix is now 3x3 Contains implicit motion information Space time processing 1 1 Spatial gradients Temporal gradients 2 3 4 2 3 5 4 5
Super resolution without motion estimation Adaptive Kernel Regression (3D) Original (QCIF, 144 x 176, 12 frames) x3 in space, x2 in time Upscaled video (432 x 528, 24 frames)
3 D Motion Deblurring The input frame at time t = 2 The deblurred frame at t = 2
Street Car Sequence The input frame at time t = 3 The deblurred frame at t = 3
Street Car Sequence The input frame at time t = 4 The deblurred frame at t = 4
Conclusions & Future Work LARK Kernels are very effective descriptors The approach is simple. Performance competitive or exceeding state of the art in Denoising/Deblurring/Super resolution (without explicit priors)