Locally Adaptive Regression Kernels with (many) Applications

Transcription

1 Locally Adaptive Regression Kernels with (many) Applications Peyman Milanfar EE Department University of California, Santa Cruz Joint work with Hiro Takeda, Hae Jong Seo, Xiang Zhu

2 Outline Introduction/Motivation Locally Adaptive Regression Kernels Applications Adaptive Kernel Regression

3 Pixels A Modern Paradigm: Measuring Patches Similarity Between Images

4 At the everyday scale.. Similarity between images H.J. Seo and P. Milanfar, Training free, Generic Object Detection using Locally Adaptive Regression Kernels, Accepted for publication in IEEE Trans. on Pattern Analysis and Machine Intelligence H.J. Seo and P. Milanfar, Action Recognition from One Example, Submitted to IEEE Trans. on Pattern Analysis and Machine Intelligence

5 Defining a point wise measure To measure the similarity of two pixels, consider Spatialdistance Gray level distance y Gray-level Δ Spatial Δ x

6 Euclidean measures Natural ways to incorporate the two Δs: BilateralKernel [Tomasi, Manduchi, 98] (pointwise) Non Local Means Kernel [Buades, et al. 05] (patchwise) y Euclidean distance Gray-level Δ Spatial Δ x

7 Bilateral Kernel (BL) [Tomasi et al. 98] Pixels Pixel similarity Spatial similarity =

8 Non local Means (NLM) [Buades et al. 05] Patches Patchsimilarity Spatial similarity = Smoothing effect

9 Defining a better measure More effective ways to combine the two Δs: LARKKernel [Takeda, et al. 07] BeltramiKernel [Sochen, et al. 98] y Signal-induced distance Riemannian Metric Gray-level Δ Spatial Δ x

10 A digression to Non parametric Kernel Regression The data fitting problem Zero-mean, i.i.d noise (No other assump.) Given samples The sampling position The regression function The number of samples The particular form of may remain unspecified for now.

11 The data model Locality in Kernel Regression Local representation (N term Taylor expansion) Unknowns Note With a polynomial basis, we only need to estimate the first unknown, Other localized representations are also possible, and may be advantageous.

12 Optimization Problem We have a local representation with respect to each sample: Optimization N+1 terms The regression order This term give the estimated pixel value z(x). The choice of the kernel function is open, e.g. Gaussian.

13 To summarize Classic Kernel: Locally Linear Filter: Uses distance x-x i Data Adaptive Kernel: Locally Non Linear Filter: Uses x-x i and y-y i

14 Recall More effective ways to combine the two Δs: LARKKernel [Takeda, et al. 07] BeltramiKernel [Sochen, et al. 98] y Signal-induced distance Riemannian Metric Gray-level Δ Spatial Δ x

15 LARK Kernels Compute

16 LARK Kernels Gradient vector field Local covariance matrices Locally Adaptive Regression Kernel: LARK Structure tensor

17 Gradient matrix over a local patch: Gradient Covariance Matrix and Local Geometry C l G T G Capturing locally dominant orientations

18 Image as a Surface Embedded in the Euclidean 3 space Arclength on the surface Chain rule Regularization term Riemannian metric

19 (Dense) LARK Kernels as Visual Descriptors Measure the similarity of pixels using the metric implied by the local structure of the image H. J. Seo, and P. Milanfar, "Training free, Generic Object Detection using Locally Adaptive Regression Kernels", Accepted in Trans. on Pattern Analysis and Machine Intelligence

20 Robustness of LARK Descriptors Original Brightness Contrast WGN image change change sigma =

21 A Variant Better suited for Restoration = LARK Measures edginess LSK [*] [*] H. Takeda, S. Farsiu, P. Milanfar, Kernel Regression for Image Processing and Reconstruction, IEEE Transactions on Image Processing, Vol. 16, No. 2, pp , February 2007

22 Some 2 and 3 D Signal Processing Applications Denoising Interpolation Super resolution Deblurring

23 Film Grain Reduction (Real Noise) Noisy image

24 Film Grain Reduction (Real Noise) LARK

25 Film Grain Reduction (Real Noise) LARK KSVD BM3D

26 Adaptive Sharpening/Denoising Sharpeningthe LARK Kernel Sharpness parameter Laplacian operator

27 Automatically setting the local sharpness Local Measure of Sharpness: X. Zhu and P. Milanfar, Automatic Parameter Selection for Denoising Algorithms Using a No Reference Measure of Image Content, to appear in IEEE Trans. Image Processing

28 LARK based Simultaneous Sharpening/Deblurring/Denoising Net effect: aggressive denoising in flat areas Selective denoising and sharpening in edgy areas LARK based filter Locally adaptive denoise/deblur filters

29 Examples original image original image proposed method State of the art Methods

30 Examples

31 Experimental Results 2 Proposed Method

32 Application to All Focus Imaging Most imaging systems have limited depth of field How to get around this? Make aperture smaller Acquire multiple images, varying the focal length (focus stacking)

33 Application to All Focus Imaging How to merge these images together? 12 frames

34 Fusing images using the local sharpness measure The combines image will have the best part of each of the frames as given by The local measure of sharpness given by the eigenvalues of: Input sequence Motion Compensation Sharp Region Detection Locally Adaptive Image Fusion All focus Output

35 Input sequence Focus Stacking Examples

36 Output Focus Stacking Examples

37 Focus Stacking Examples Output (Photoshop CS4)

38 More Focus Stacking Examples Output (Helicon Focus) Input images (37 frames) Output (Ours) From:

39 Focus stacking Example Input images (24 frames) Output

40 Space Time Descriptors Setup is similar to 2 D, but.. Samples from nearby frames Covariance matrix is now 3x3 Contains implicit motion information Space time processing 1 1 Spatial gradients Temporal gradients

41 Super resolution without Motion Estimation Adaptive Kernel Regression (3D) Original (QCIF, 144 x 176, 12 frames) x3 in space, x2 in time Upscaled video (432 x 528, 24 frames)

42 Computer Vision Applications: Object (2 D) and Action (3 D) Detection and Recognition.. from one example

43 Take a look at this:

44 See it here?

45 How about here?

46 Or here?

47 Single Example, No Training! (Most) people can find the Dragon Fruit from one look Even if they ve never seen it before.

48 Holy Grail of Computer Vision: Visual Seach Visual Search : Robustly detect objects/actions of interest within images/videos from a single query query target image query target video 1) Whether objects (actions) are present or not, 2) How many objects (actions)? 3) Where are they located?

49 Summary: Object Detection with Local Regression Kernels LARK LSK Query Target Descriptors NLM BL SSIM Compare Visual Features Target with bounding box SIFT HOG

50 Object Detection System Overview Query stage 1 stage 2 PCA Target Dense LARK Descriptors compute feature images 2) Significance Tests 3) Non-maxima Suppression Final result stage 3 1) Resemblance Map (RM) using Matrix Cosine Similarity [*] H. J. Seo, and P. Milanfar, "Training free, Generic Object Detection using Locally Adaptive Regression Kernels", To appear in TPAMI 2010

51 Stage 2: Feature Extraction from Descriptors Densely collected Vectorization Descriptors Apply PCA to for dimensionality reduction Retain the d largest principal components Project and onto Energy d Eigenvalue rank

52 System Overview : Stage 3 Query stage 1 stage 2 PCA Target Compute local steering kernels compute feature images stage 3 2) Significance Tests 3) Non-maxima Suppression Final result 1) Resemblance Map (RM) using Matrix Cosine Similarity

53 Stage 3: Finding similarity between features Target image is scanned into overlapping patches Query Target Query Target

54 Non standard correlation based Metric The vector cosine similarity Query Target patch Inner product between two normalized vectors Measures angle while discarding the magnitude

55 Stage 3: Matrix Cosine Similarity What about a set of vectors? Matrix Cosine Similarity Frobenius Inner product between normalized matrices Query Target patch A weighted sum of the column wise vector cosine similarities We can prove optimality of this approach in a naïve Bayes sense.

56 Thresholding: : Control False Discovery Rate 0.4 Empirical PDF % confidence level Significance level Y. Benjamini, Y. Hochberg (1995), "Controlling the false discovery rate: a practical and powerful approach to multiple testing". Journal of the Royal Statistical Society, Series B, 57 (1):

57 Some sample results: Query

58 Some Examples Query query Target target

59 Experimental Results query Target Target Target

60 Some Examples query target target

61 Some Examples Hand-drawn Query Targets

62 Quantitative comparison Recall LARK LSK SSIM NLM BL SIFT HOG Precision Recall = TP/nP Precision = TP/(TP+FP) TP: true positives FP: false positives np: a total number of positives (Weizmann general object database)

63 Action Detection/Recognition System Query stage 1 stage 2 PCA Target Compute space-time local steering kernels compute feature volumes No Motion Estimation No Segmentation No Learning No Prior Information 2) Significance Tests 3) Non-maxima Suppression Final result stage 3 1) Resemblance Map (RM) using Matrix Cosine Similarity H. Seo and P. Milanfar, Action Recognition from One Example, Submitted to Trans. on Pattern Analysis and Machine Intelligence

64 Action Detection Example Query No Motion Estimation No Segmentation No Learning No Prior Information

65 Experimental Results (Multiple Actions)

66 Action Recognition Automatic cropping of a short action clip (25 frames) Action Category 1 5 Query Action Detection 2 6 Scoring Rank least similar most similar

67 Action Classification Performance Average confusion matrices Classification rate: 96 % Classification rate: % Bend Jack Jump Pjump Run Side Skip Walk Wave1 Wave Bend Jack Jump Pjump Run Side Skip Wave1 Walk box hclp hwav jog run walk box hclp hwav jog run walk (Weizmann dataset) Wave2 (KTH dataset) 90 video sequences 600 video sequences

68 Conclusions & Future Work LARK Kernels are very effective descriptors The approach is simple. Performance competitive or exceeding state of the art in Denoising/Deblurring/Super resolution (without explicit priors) Image and Video Detection/Recognition (without training)