Computer Vision I - Filtering and Feature detection

Transcription

1 Computer Vision I - Filtering and Feature detection Carsten Rother 30/10/2015 Computer Vision I: Basics of Image Processing

2 Roadmap: Basics of Digital Image Processing Computer Vision I: Basics of Image Processing 30/10/ What is an Image? Point operators (ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) Linear filtering Non-linear filtering Edges detection (ch. 4.2) Interest Point detection (ch )

3 Reminder: Convolution Computer Vision I: Basics of Image Processing 30/10/ Replace each pixel by a linear combination of its neighbours and itself 2D convolution (discrete) g = f h Centred at 0,0 f x, y h x, y g x, y g x, y = k,l f x k, y l h k, l the image f is implicitly mirrored

4 Reminder: Application: Noise removal Computer Vision I: Basics of Image Processing 30/10/ Noise is what we are not interested in: Sensor noise (Gaussian, shot noise), quantisation artefacts, light fluctuation, etc. Typical assumption is that the noise is not correlated between pixels Basic Idea: neighbouring pixel contain information about intensity

5 Reminder: Gaussian (Smoothing) Filters Computer Vision I: Basics of Image Processing 30/10/ Nearby pixels are weighted more than distant pixels Isotropic Gaussian (rotational symmetric)

6 The box filter does noise removal Computer Vision I: Basics of Image Processing 30/10/ Box filter takes the mean in a neighbourhood Image Noise Pixel-independent Gaussian noise added Filtered Image

7 Derivation of the Box Filter Computer Vision I: Basics of Image Processing 30/10/ y r is true gray value (color) x r observed gray value (color) Noise model: Gaussian noise p x r y r ) = N x r ; y r, σ ~ exp[ x r y r 2 2σ 2 ] y r x r

8 Computer Vision I: Basics of Image Processing 30/10/ Derivation of Box Filter Further assumption: independent noise p x y) ~ exp[ x r y r 2 2σ 2 ] r Find the most likely solution for the true signal y Maximum-Likelihood principle (probability maximization): posterior likelihood y = argmax y p y x) = argmax y p x y p y p(x) p(x) is a constant (drop it), assume (for now) uniform prior p(y). We get: p y x) = p x y exp[ x r y r 2 2σ 2 ] r the solution is trivial: y r = x r for all r Comments: 1) ~ means equal up to scale (i.e. 5x ~ x) 2) a 2 = a 2 2 = a 2 (i.e. squared L2 norm) 3) a is norm of a (i.e. L1 norm) prior additional assumptions about the signal y are necessary!!!

9 Derivation of Box Filter Computer Vision I: Basics of Image Processing 30/10/ Assumption: not uniform prior p y but in a small vicinity the true signal is nearly constant Maximum-a-posteriori: For one pixel r : p y x) exp[ x r y r 2 2σ 2 ] r y r = argmax yr exp[ x r y r 2 2σ 2 ] Only one y r in a window W(r) take neg. logarithm: y r = argmin yr x r y r 2 2σ 2 Note, Log-function is monotonically increasing hence minimum does not change, i.e. x 1 < x 2 then log x 1 < log x 2

10 Derivation of Box Filter Computer Vision I: Basics of Image Processing 30/10/ Let us minimize: y r = argmin yr x r y r 2 factor 1/2σ 2 is irrelevant Take derivative and set to 0: F y r = x r y r 2 W means number of pixel in window W F y r = 2 r x r y r = 2 (! x r W y r ) = 0 r y r (the average) Box filter is optimal under pixel-independent, Gaussian Noise assumption and assuming that the signal is constant in a window.

11 Roadmap: Basics of Digital Image Processing Computer Vision I: Basics of Image Processing 30/10/ What is an Image? Point operators (ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) Linear filtering Non-linear filtering Edges detection (ch. 4.2) Interest Point detection (ch )

12 Non-linear filters Computer Vision I: Basics of Image Processing 30/10/ There are many different non-linear filters. (see Image Processing in SS16) We only look at the Median filter, Bilateral filter and Joint Bilateral Filter

13 Shot noise (Salt and Pepper Noise) Computer Vision I: Basics of Image Processing 30/10/ Gaussian filtered Original + shot noise (a random number of independent pixels have a random value) Median filtered

14 Another example Computer Vision I: Basics of Image Processing 30/10/ Original Noisy Mean Median

15 Median Filter Computer Vision I: Basics of Image Processing 30/10/ Replace each pixel with the median in a neighbourhood: median Median filter: order the values and take the middle one No strong smoothing effect since values are not averaged Very good to remove outliers (shot noise) Used a lot for post processing of outputs (e.g. optical flow)

16 Median Filter: Derivation Computer Vision I: Basics of Image Processing 30/10/ Reminder, for Gaussian noise we did solve the following Maximum Likelihood Estimation Problem: y r = argmax yr exp[ x r y r 2 p y x) 2σ 2 ] = argmin yr x r y r 2 = 1/ W x r median mean Mean and median are quite Different For Median we solve the following problem: y r = argmax yr exp[ x (see next slide) r y r 2σ 2 ] = argmin yr x r y r = Median (W r ) This is a different noise distribution than Gaussian.

17 Median Filter Derivation Computer Vision I: Basics of Image Processing 30/10/ function: minimize the following: F y r = x r y r Problem: not differentiable, good news: it is convex Optimal solution is the median of all values

18 Bilateral Filter Computer Vision I: Basics of Image Processing 30/10/ More sharp Original + Gaussian noise Gaussian filtered Bilateral filtered Edge over-smoothed Edge not over-smoothed

19 Bilateral Filter in Pictures Computer Vision I: Basics of Image Processing 30/10/ Gaussian Filter weights Output (sketched) Centre pixel Noisy input Bilateral Filter weights Output

20 Bilateral Filter in equations Computer Vision I: Basics of Image Processing 30/10/ Filters looks at: a) distance to surrounding pixels (as Gaussian) b) Intensity of surrounding pixels Linear combination Same as Gaussian filter Consider intensity Problem: computation is slow O Nw ; approximations can be done in O(N) Comment: Guided filter is similar and can be computed exactly in O(N) See a tutorial on:

21 Application: Bilateral Filter Computer Vision I: Basics of Image Processing 30/10/ Cartoonization HDR compression (Tone mapping)

22 Joint Bilateral Filter Computer Vision I: Basics of Image Processing 30/10/ ~ ~ Same as Gaussian Consider intensity f is the input image which is processed ~ f is a guidance image where we look for pixel similarity

23 Application: Combine Flash and No-Flash Computer Vision I: Basics of Image Processing 30/10/ input image f ~ Guidance image f Joint Bilateral Filter ~ ~ We don t care about absolute colors [Petschnigg et al. Siggraph 04]

24 Reminder: Model versus Algorithm Computer Vision I: Introduction 30/10/ Example: Interactive Image Segmentation Goal z = R, G, B n x = 0,1 n Given z; derive binary x: Model: Energy function E x = i θ i x i + i,j θ ij (x i, x j ) Algorithm to minimization: x = argmin x E(x)

25 Filtering for Binary Segmentation Computer Vision I: Introduction 30/10/ New query image z i θ i (x i = 0) θ i (x i = 1) Dark means likely background Dark means likely foreground Optimum with unary terms only

26 Filtering for Binary Segmentation Computer Vision I: Basics of Image Processing 30/10/ Ratio Image θ i x i =1 θ i (x i =0) is the Input Image f Filtered Ratio Image ~ Guidance Input Image f (also shown user brush strokes) Results are very similar: This is an alternative to energy minimization This can also be done for stereo, etc. Cost Volume filtering (Winner takes all Result) Energy minimization [C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz, Fast Cost- Volume Filtering for Visual Correspondence and Beyond, CVPR 11]

27 Roadmap: Basics of Digital Image Processing Computer Vision I: Basics of Image Processing 30/10/ What is an Image? Point operators (ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) Linear filtering Non-linear filtering Edges detection (ch. 4.2) Interest Point detection (ch )

28 Idealized edge types Computer Vision I: Basics of Image Processing 30/10/ We focus on this

29 What are edges? Computer Vision I: Basics of Image Processing 30/10/ Corresponds to fast changes in the image The magnitude of the derivative is large Image of 2 step edges Image of 2 ramp edges Slice through the image Slice through the image

30 What are fast changes in the image? Computer Vision I: Basics of Image Processing 30/10/ Texture or many edges? Scanline 250 Image Edges defined after smoothing Scanline 250 smoothed with Gausian

31 Edges and Derivatives Computer Vision I: Basics of Image Processing 30/10/ We will look at this mainly

32 Edge filters in 1D Computer Vision I: Basics of Image Processing 30/10/ We can implement this as a linear filter: Forward differences: 1/2-1 1 Central differences: 1/

33 Reminder: Seperable Filters Computer Vision I: Basics of Image Processing 30/10/ This is the centralized difference-operator

34 Edge Filter in 1D: Example Computer Vision I: Basics of Image Processing 30/10/ Based on 1st derivative Smooth with Gaussian to filter out noise Calculate derivative Find its optima

35 Edge Filtering in 1D Computer Vision I: Basics of Image Processing 30/10/ Simplification: (saves one operation) Derivative of Gaussian

36 Edge Filtering in 2D Computer Vision I: Basics of Image Processing 30/10/

37 Edge Filter in 2D: Example Computer Vision I: Basics of Image Processing 30/10/ y x

38 Edge Filter in 2D Computer Vision I: Basics of Image Processing 30/10/ x-derivatives with different Gaussian smoothing

39 What is a gradient Computer Vision I: Basics of Image Processing 30/10/

40 What is a gradient Computer Vision I: Basics of Image Processing 30/10/

41 What is a gradient Computer Vision I: Basics of Image Processing 30/10/

42 What is a Gradient Computer Vision I: Basics of Image Processing 30/10/

43 Example Gradient magnitude image Computer Vision I: Basics of Image Processing 30/10/ First smoothed with Gaussian First smoothed with broad Gaussian In Image Processing Lecture we look at how to get edge chains?

44 Half-way Slide Computer Vision I: Basics of Image Processing 30/10/ minutes break

45 Roadmap: Basics of Digital Image Processing Computer Vision I: Basics of Image Processing 30/10/ What is an Image? Point operators (ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) Linear filtering Non-linear filtering Edges detection (ch. 4.2) Interest Point detection (ch )

46 What region should we try to match? Computer Vision I: Basics of Image Processing 30/10/ We want to find a few regions where this image pair matches (applications later) Look for a region that is unique, i.e. not ambiguous

47 Goal: Interest Point Detection Computer Vision I: Basics of Image Processing 30/10/ Goal: predict a few interest points in order to remove redundant data efficiently Should be invariant against: a. Geometric transformation scaling, rotation, translation, affine transformation, projective transformation etc. b. Color transformation additive (lightning change), multiplicative (contrast), linear (both), monotone etc.; c. Discretization (e.g. spatial resolution, focus);

48 Points versus Lines Computer Vision I: Basics of Image Processing 30/10/ Apeture problem Lines are not as good as points?

49 Harris Corner Detector Basic Idea Computer Vision I: Basics of Image Processing 30/10/ Local measure of feature uniqueness: Shifting the window in any direction: How does it change? [Szeliski and Seitz]

50 Harris Corner Detector Computer Vision I: Basics of Image Processing 30/10/ How similar is a local window with its neighbor? Auto-correlation function: Δy W (x, y) Δx is a small window around is a convolution kernel and used to decrease the influence of pixels far from, e.g. with Gaussian For simplicity we use for now = 1.

51 Harris Corner Detector Computer Vision I: Basics of Image Processing 30/10/ One is interested in properties of at each position We could evaluate for all discrete shifts = +/-1. But we would like to do smaller shifts and have a fast method. Let us look at a linear approximation of Taylor expansion around, i.e. the + ε(δx, Δy) Gradient at (u, v)

52 Reminder: Taylor Expansion Computer Vision I: Basics of Image Processing 30/10/ A function f(x) is approximated by: f x ~ n f n a n! x a n The approximation is most accurate at point a

53 Harris Corner Detector Computer Vision I: Basics of Image Processing 30/10/ Put it together: with Q is call the Structure Tensor We compute this at any image location (x, y)

54 Harris Corner Detector Computer Vision I: Basics of Image Processing 30/10/ The auto-correlation function Function c is (after approximation) a quadratic function in Isolines are ellipses ( is symmetric and positive definite) Eigenvector x 1 with (larger) Eigenvalue λ 1 is the direction of fastest change in function c Eigenvector x 2 with (smaller) Eigenvalue λ 2 is direction of slowest change in function c and x 2 Δx Δy x 1 Function c Note c = 0 for Δx = Δy = 0

55 Computer Vision I: Basics of Image Processing 30/10/ Harris Corner Detector Some examples isolines for : (a) Flat (b) Edges (c) Corners a. Homogenous regions: both -s are small b. Edges: one is small the other one is large c. Corners: both -s are large (this is what we are looking for!)

56 Harris Corner Detector Computer Vision I: Basics of Image Processing 30/10/ Image λ 1 (larger eigenvalue) λ 2 (smaller eigenvalue) Zoom in

57 Computer Vision I: Basics of Image Processing 30/10/ Harris Corner Detector Cornerness is a characteristic of Proposition by Harris: Downweights edges where λ 1 λ 2 Harris Value Smallest eigenvalue

58 Harris Corners - Example Computer Vision I: Basics of Image Processing 30/10/

59 H-score (red- high, blue - low) Computer Vision I: Basics of Image Processing 30/10/

60 Threshold (H-score > value) Computer Vision I: Basics of Image Processing 30/10/

61 Non-maximum suppression Computer Vision I: Basics of Image Processing 30/10/ (all points are set to 0 for which a higher H-score in a window-neighborhood exists)

62 Harris Corners in Red Computer Vision: Algorithms and Applications -- 30/10/

63 Other examples Computer Vision I: Basics of Image Processing 30/10/

64 Maximally stable extremal regions Computer Vision I: Basics of Image Processing 30/10/ Invariant to affine transformation of gray-values Both small and large structures are detected

65 Literature Computer Vision I: Basics of Image Processing 30/10/ There is a large body of literature on detectors and descriptors (later lecture) A comparison paper (e.g. what is the most robust corner detectors): K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman, J. Matas, F. Schaffalitzky, T. Kadir: A Comparison of Affine Region Detectors (IJCV 2006)