Locally Adaptive Regression Kernels with (many) Applications

Transcription

1 Locally Adaptive Regression Kernels with (many) Applications Peyman Milanfar EE Department University of California, Santa Cruz Joint work with Hiro Takeda, Hae Jong Seo, Xiang Zhu

2 Outline Introduction/Motivation Locally Adaptive Regression Kernels Applications Adaptive Kernel Regression

3 Pixels A Modern Paradigm: Measuring Patches Similarity Between Images

4 At the smallest scale. Similarity between pixels H. Takeda, S. Farsiu, and P. Milanfar, Kernel Regression for Image Processing and Reconstruction, IEEE Trans. on Image Processing, vol. 16, no. 2, pp , February 2007

5 A little bigger scale Similarity between patches Non Local Means (NLM) denoising Block Matching (BM3d) denoising BM3d A. Buades, B. Coll, J.M Morel, "A review of image denoising algorithms, with a new one", SIAM Multiscale Modeling and Simulation, Vol 4 (2), pp: , 2005 (NLM) K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, Image denoising by sparse 3D transform domain collaborative filtering, IEEE Trans. Image Proc., vol. 16, no. 8, pp , August 2007 (BM3d)

6 A little bigger scale Similarity between patches (e.g. Visual Saliency) H.J. Seo and P. Milanfar, Static and Space time Visual Saliency Detection by Self Resemblance, The Journal of Vision 9 (12):15, 1 27, doi: /

7 At the everyday scale.. Similarity between images H.J. Seo and P. Milanfar, Training free, Generic Object Detection using Locally Adaptive Regression Kernels, Accepted for publication in IEEE Trans. on Pattern Analysis and Machine Intelligence H.J. Seo and P. Milanfar, Action Recognition from One Example, Submitted to IEEE Trans. on Pattern Analysis and Machine Intelligence

8 Defining a point wise measure To measure the similarity of two pixels, consider Spatialdistance Gray level distance y Gray-level Δ Spatial Δ x

9 Simplest measures Basic ways to incorporate the two Δs: BilateralKernel [Tomasi, Manduchi, 98] (pointwise) Non Local Means Kernel [Buades, et al. 05] (patchwise) y Euclidean distance Gray-level Δ Spatial Δ x

10 Bilateral Kernels (BL) [Tomasi et al. 98] Pixels Pixel similarity Spatial similarity = Sensitive to 1) noise 2) variation in illumination

11 Non local Means (NLM) [Buades et al. 05] Patches Patchsimilarity Spatial similarity = Smoothing effect, more robust, but still quite sensitive

12 Defining a measure More effective ways to combine the two Δs: LARKKernel [Takeda, et al. 07] BeltramiKernel [Sochen, et al. 98] y Signal-induced distance Riemannian Metric Gray-level Δ Spatial Δ x

13 A digression to Non parametric Kernel Regression The data fitting problem Zero-mean, i.i.d noise (No other assump.) Given samples The sampling position The regression function The number of samples The particular form of may remain unspecified for now.

14 The data model Locality in Kernel Regression Local representation (N term Taylor expansion) Unknowns Note With a polynomial basis, we only need to estimate the first unknown, Other localized representations are also possible, and may be advantageous.

15 Optimization Problem We have a local representation with respect to each sample: Optimization N+1 terms The regression order This term give the estimated pixel value z(x). The choice of the kernel function is open, e.g. Gaussian.

16 What do these weights look like?

17 Locally Linear/Nonlinear Estimators The optimization yields a point wise linear (but perhaps space varying) estimator: Kernel function The smoothing parameter The weighted linear combinations of the given data Normalized version of K The regression order By choosing a data adaptive kernel (BL,NLM,LARK), the filters become nonlinear.

18 To summarize Classic Kernel: Locally Linear Filter: Uses distance x-x i Data Adaptive Kernel: Locally Non Linear Filter: Uses x-x i and y-y i

19 Recall More effective ways to combine the two Δs: LARKKernel [Takeda, et al. 07] BeltramiKernel [Sochen, et al. 98] y Signal-induced distance Riemannian Metric Gray-level Δ Spatial Δ x

20 LARK Kernels Compute

21 LARK Kernels Gradient vector field Local covariance matrices Locally Adaptive Regression Kernel: LARK Structure tensor

22 Gradient matrix over a local patch: Gradient Covariance Matrix and Local Geometry

23 Image as a Surface Embedded in the Euclidean 3 space Arclength on the surface Chain rule Regularization term Riemannian metric

24 (Dense) LARK Kernels as Visual Descriptors Measure the similarity of pixels using the metric implied by the local structure of the image H. J. Seo, and P. Milanfar, "Training free, Generic Object Detection using Locally Adaptive Regression Kernels", Accepted in Trans. on Pattern Analysis and Machine Intelligence

25 Comparisons Image Bilateral Non local Means LARK

26 Effect of C (covariance matrix) How to estimate matters VS. Naïve estimation of Accurate estimation of Using regularized (SVD)

27 Robustness of LARK Descriptors Original Brightness Contrast WGN image change change sigma =

28 A Variant Better suited for Restoration = LARK Measures edginess LSK [*] [*] H. Takeda, S. Farsiu, P. Milanfar, Kernel Regression for Image Processing and Reconstruction, IEEE Transactions on Image Processing, Vol. 16, No. 2, pp , February 2007

29 Some 2 and 3 D Signal Processing Applications Denoising Interpolation Super resolution Deblurring

30 Gaussian Noise Removal Noisy image, RMSE=24.87 LARK RMSE=6.63 KSVD Elad, et al. (2007) RMSE= 6.89 BM3D Foi, et al. (2007) RMSE=6.35 State of Art for Gaussian Noise

31 Film Grain Reduction (Real Noise) Noisy image

32 Film Grain Reduction (Real Noise) LARK

33 Film Grain Reduction (Real Noise) LARK KSVD BM3D

34 Adaptive Kernels for Interpolation What about missing pixels? Local metric (covariance) is undefined! Using a pilot estimate, fill the missing pixels: Kernel weights Iteration! Adaptive kernel regression????????????????

35 Reconstruction from Sparse and Irregular Samples Randomly delete 85% of pixels Reconstruction

36 Effect of Regression Order The order regression N trades off bias/variance (blur/noise). N+1 terms So far, we have kept N fixed. Ideally, when treating edge areas: want N when treating flat areas: want N

37 Implicitly changing N Sharpeningthe LARK Kernel Sharpness parameter Laplacian operator

38 Automatically setting the local sharpness Local Measure of Sharpness: X. Zhu and P. Milanfar, Automatic Parameter Selection for Denoising Algorithms Using a No Reference Measure of Image Content, to appear in IEEE Trans. Image Processing

39 LARK based Simultaneous Sharpening/Deblurring/Denoising Net effect: aggressive denoising in flat areas Selective denoising and sharpening in edgy areas LARK based filter Locally adaptive denoise/deblur filters

40 Examples original image original image proposed method State of the art Methods

41 Examples

42 Experimental Results 2 Proposed Method

43 Experimental Results 2 Shan s Method

44 Application to Enhancing SEM Images 16 low voltage SEM images of the same sample

45 Application to SEM data Input frame Average frame

46 Application to SEM data Input frame Average frame, denoised

47 Application to SEM data Input frame Nonlinear registered merged output

48 Application to SEM data Input frame Average frame with nonlinear regis. Enhanced

49 Application to All Focus Imaging Most imaging systems have limited depth of field How to get around this? Make aperture smaller Acquire multiple images, varying the focal length (focus stacking)

50 Application to All Focus Imaging How to merge these images together? 12 frames

51 Fusing images using the local sharpness measure The combines image will have the best part of each of the frames as given by The local measure of sharpness given by the singular values of: Input sequence Motion Compensation Sharp Region Detection Locally Adaptive Image Fusion All focus Output

52 Input sequence Focus Stacking Examples

53 Output Focus Stacking Examples

54 Focus Stacking Examples Output (Photoshop CS4)

55 More Focus Stacking Examples Output (Helicon Focus) Input images (37 frames) Output (Ours) From:

56 Focus stacking Example Input images (24 frames) Output (ours)

57 Focus stacking Example Output

58 Super resolution Motion Estimation Image Adaptive Reconstruction Kernel (Interpolation, Regression Denoising, (2D) and Deblurring)

59 Space Time Descriptors Setup is similar to 2 D, but.. Samples from nearby frames Covariance matrix is now 3x3 Contains implicit motion information Space time processing 1 1 Spatial gradients Temporal gradients

60 Super resolution without motion estimation Adaptive Kernel Regression (3D) Original (QCIF, 144 x 176, 12 frames) x3 in space, x2 in time Upscaled video (432 x 528, 24 frames)

61 3 D Motion Deblurring The input frame at time t = 2 The deblurred frame at t = 2

62 Street Car Sequence The input frame at time t = 3 The deblurred frame at t = 3

63 Street Car Sequence The input frame at time t = 4 The deblurred frame at t = 4

64 Conclusions & Future Work LARK Kernels are very effective descriptors The approach is simple. Performance competitive or exceeding state of the art in Denoising/Deblurring/Super resolution (without explicit priors)