Computer Vision I - Basics of Image Processing Part 2

Transcription

1 Computer Vision I - Basics of Image Processing Part 2 Carsten Rother 07/11/2014 Computer Vision I: Basics of Image Processing

2 Roadmap: Basics of Digital Image Processing Computer Vision I: Basics of Image Processing 07/11/ What is an Image? Point operators (ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) main focus Linear filtering Non-linear filtering Multi-scale image representation (ch. 3.5) (will be done in SS15 as part of Image Processing) Edges detection and linking (ch. 4.2) Line detection and vanishing point detection (ch. 4.3) (will be done in SS15 as part of Image Processing) Interest Point detection (ch )

3 Reminder: Convolution (linear filter) Computer Vision I: Basics of Image Processing 07/11/ Replace each pixel by a linear combination of its neighbours and itself. 2D convolution (discrete) g = f h Centred at 0,0 f x, y h x, y g x, y g x, y = k,l f x k, y l h k, l the image f is implicitly mirrored

4 Reminder: Median Filter Computer Vision I: Basics of Image Processing 07/11/ Replace each pixel with the median in a neighbourhood: median Median filter: order the values and take the middle one No strong smoothing effect since values are not averaged Very good to remove outliers (shot noise) Used a lot for post processing of outputs (e.g. optical flow)

5 Reminder: Motivation Bilateral Filter Computer Vision I: Basics of Image Processing 07/11/ Original + Gaussian noise Gaussian filtered Bilateral filtered Edge over-smoothed Edge not over-smoothed

6 Reminder: Bilateral Filter in pictures Computer Vision I: Basics of Image Processing 07/11/ Gaussian Filter weights Output (sketched) Centre pixel Noisy input Bilateral Filter weights Output

7 Reminder: Bilateral Filter in equations Computer Vision I: Basics of Image Processing 07/11/ Filters looks at: a) distance to surrounding pixels (as Gaussian) b) Intensity of surrounding pixels Linear combination Same as Gaussian filter Consider intensity Problem: computation is slow O Nw ; approximations can be done in O(N) See a tutorial on:

8 Bilateral filter is a non-linear filter Computer Vision I: Basics of Image Processing 07/11/ Linear Filter satisfy Additivity: T[X + Y] = T[X] + T[Y] Image: ) Operator T is (linear) boxfilter: compute filter output for central element of the image: T X + Y = 2 3 T X + T Y = ) Operator T is (non-linear) Bilteral filter with w(i, j, k, l) = exp( f i, j f k, l Compute filter output for central element of the image: 2 ) T X + Y = ( ) / ( ) = 1.92 T X + T Y = 2 * [ ( ) / ( ) ] = 2*(1/1.72) = 1.16 (Note: exp(-1) = 0.36; exp(-4) = 0.04)

9 Application: Bilateral Filter Computer Vision I: Basics of Image Processing 07/11/ Cartoonization HDR compression (Tone mapping)

10 Joint Bilateral Filter Computer Vision I: Basics of Image Processing 07/11/ ~ ~ Same as Gaussian Consider intensity f is the input image which is processed ~ f is a guidance image where we look for pixel similarity

11 Application: combine Flash and No-Flash input image f ~ Guidance image f Joint Bilateral Filter ~ ~ We don t care about absolute colors [Petschnigg et al. Siggraph 04] Computer Vision I: Basics of Image Processing 07/11/

12 Reminder: Application: Cost Volume Filtering Computer Vision I: Basics of Image Processing 07/11/ Reminder from first Lecture: Interactive Segmentation Goal z = R, G, B n x = 0,1 n Given z; derive binary x: Model: Energy function E x = i θ i x i + i,j θ ij (x i, x j ) Unary terms Pairwise terms Algorithm to minimization: x = argmin x E(x)

13 Reminder: Unary term New query image z i θ i (x i = 0) θ i (x i = 1) Dark means likely background Dark means likely foreground Optimum with unary terms only Computer Vision I: Introduction 07/11/

14 Cost Volumne for Binary Segmenation Label Space (here 2) Computer Vision I: Basics of Image Processing 07/11/ θ i (x i = 1) θ i (x i = 0) Image (x,y) For two labels, we can also look at the ratio Image f: f i = θ i (x i = 1) / θ i (x i = 0)

15 Application: Cost Volume Filtering Computer Vision I: Basics of Image Processing 07/11/ Ratio Cost-volume is the Input Image f Filtered cost volume ~ Guidance Input Image f (also shown user brush strokes) Results are very similar: This is an alternative to energy minimization! Cost Volume filtering (Winner takes all Result) Energy minimization [C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz, Fast Cost- Volume Filtering for Visual Correspondence and Beyond, CVPR 11]

16 Morphological operations Computer Vision I: Basics of Image Processing 07/11/ Two steps: 1. Perform convolution with a structural element : binary mask (e.g. circle or square) black is 1 white is 0 2. Threshold the continuous output f to recover a binary image:

17 Other non-linear operations on binary images Computer Vision I: Basics of Image Processing 07/11/ Binary Image Distance transform Skeleton Binary Input Image Connected components

18 Roadmap: Basics of Digital Image Processing Computer Vision I: Basics of Image Processing 07/11/ What is an Image? Point operators (ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) main focus Linear filtering Non-linear filtering Multi-scale image representation (ch. 3.5) (will be done in SS15 as part of Image Processing) Edges detection and linking (ch. 4.2) Line detection and vanishing point detection (ch. 4.3) (will be done in SS15 as part of Image Processing) Interest Point detection (ch )

19 Goal: Find long edge chains Computer Vision I: Basics of Image Processing 07/11/ What do we want: Good detection: we want to find edges not noise Good localization: find true edge Single response: one per edge (independent of edge sharpness) Long edge-chains

20 Idealized edge types Computer Vision I: Basics of Image Processing 07/11/ We focus on this

21 What are edges? Computer Vision I: Basics of Image Processing 07/11/ correspond to fast changes in the image The magnitude of the derivative is large Image of 2 step edges Image of 2 ramp edges Slizce through the image Slizce through the image

22 What are fast changes in the image? Computer Vision I: Basics of Image Processing 07/11/ Texture or many edges? Scanline 250 Image Edges defined after smoothing Scanline 250 smoothed with Gausian

23 Edges and Derivatives Computer Vision I: Basics of Image Processing 07/11/ We will look at this first

24 Edge filters in 1D Computer Vision I: Basics of Image Processing 07/11/ We can implement this as a linear filter: Forward differences: 1/2-1 1 Central differences: 1/

25 Reminder: Seperable Filters Computer Vision I: Basics of Image Processing 07/11/ This is the centralized difference-operator

26 Edge Filter in 1D: Example Computer Vision I: Basics of Image Processing 07/11/ Based on 1st derivative Smooth with Gaussian to filter out noise Calculate derivative Find its optima

27 Edge Filtering in 1D Computer Vision I: Basics of Image Processing 07/11/ Simplification: (saves one operation) Derivative of Gaussian

28 Edge Filtering in 2D Computer Vision I: Basics of Image Processing 07/11/

29 Edge Filter in 2D: Example Computer Vision I: Basics of Image Processing 07/11/ y x

30 Edge Filter in 2D Computer Vision I: Basics of Image Processing 07/11/ x-derivatives with different Gaussian smoothing

31 What is a gradient Computer Vision I: Basics of Image Processing 07/11/

32 What is a gradient Computer Vision I: Basics of Image Processing 07/11/

33 What is a gradient Computer Vision I: Basics of Image Processing 07/11/

34 What is a Gradient Computer Vision I: Basics of Image Processing 07/11/

35 Example Gradient magnitude image Computer Vision I: Basics of Image Processing 07/11/ First smoothed with Gaussian First smoothed with broad Gaussian Our goal was to get thin edge chains?

36 How to get edge chains Computer Vision I: Basics of Image Processing 07/11/ Rough Outline of a good edge detector (such as Canny) 1. Compute robust Gradient Image: ( (D x G) I, (D y G) I) 2. Find edge-points ( edgels ): non-maximum suppression 3. Link-up edge-points to get chains 4. Do hysteresis to clean-up chains edge-points or edgel

37 Non-maximum surpression Not Edge point Edge point Magnitude image Zoom on pixel-grid Wich pixel is an edge point? (non-max surpression) 1. Check if pixel is local maximum at gradient direction (+/- 180deg) (for this interpolate values for p,r) 2. Accept edge-point if above a threshold Computer Vision I: Basics of Image Processing 07/11/

38 Link up edge-points to get chains Computer Vision I: Basics of Image Processing 07/11/ Not Edge point Edge point 1. Link-up neighbouring pixels if both are edge-points.

39 Clean up chains with Hysteresis Computer Vision I: Basics of Image Processing 07/11/ Keep a chain: High start threshold Low threshold along the chain

40 Final Result Image Not much smoothing (fine scale) much smoothing (coarse scale) small threshold much smoothing (coarse scale) large threshold Computer Vision I: Basics of Image Processing 07/11/

41 Future Lecture: Segmentation Computer Vision I: Basics of Image Processing 07/11/ So far we looked at jumps in gray-scale images? Humans perceive edges very differently (edges depend on semantic) average human drawing Hard for computer vision methods without semantic reasoning [from Martin, Fowlkes and Mail 2004]

42 Try to learn semantically meaningful image edges Computer Vision I: Basics of Image Processing 07/11/ Features: brightness gradient; color gradient; texture gradient; weighted combination, etc.

43 Half-way break Computer Vision I: Basics of Image Processing 07/11/ minutes break

44 Roadmap: Basics of Digital Image Processing Computer Vision I: Basics of Image Processing 07/11/ What is an Image? Point operators (ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) main focus Linear filtering Non-linear filtering Multi-scale image representation (ch. 3.5) (will be done in SS15 as part of Image Processing) Edges detection and linking (ch. 4.2) Line detection and vanishing point detection (ch. 4.3) (will be done in SS15 as part of Image Processing) Interest Point detection (ch )

45 What region should we try to match? Computer Vision I: Basics of Image Processing 07/11/ We want to find a few regions where this image pair matches: Applications later Look for a region that is unique, i.e. not ambiguous

46 Goal: Interest Point Detection Computer Vision I: Basics of Image Processing 07/11/ Goal: predict a few interest points in order to remove redundant data efficiently Should be invariant against: a. Geometric transformation scaling, rotation, translation, affine transformation, projective transformation etc. b. Color transformation additive (lightning change), multiplicative (contrast), linear (both), monotone etc.; c. Discretization (e.g. spatial resolution, focus);

47 Points versus Lines Computer Vision I: Basics of Image Processing 07/11/ Apeture problem Lines are not as good as points?

48 Harris Detector Computer Vision I: Basics of Image Processing 07/11/ Local measure of feature uniqueness: Shifting the window in any direction: how does it change [Szeliski and Seitz]

49 Harris Detector Computer Vision I: Basics of Image Processing 07/11/ How similar is the image Autocorrelation function: to itself? Δy (x, y) Δx is a small window around is a convolution kernel and used to decrease the influence of pixels far from, e.g. the Gaussian For simplicity we use w(u, v) = 1

50 Harris Detector Computer Vision I: Basics of Image Processing 07/11/ One is interested in properties of at each position Let us look at a linear approximation of Taylor expansion around + ε(δx, Δy) Gradient at (u, v)

51 Harris Detector Computer Vision I: Basics of Image Processing 07/11/ Put it together: with Q is call the Structure Tensor We compute this at any image location (x, y)

52 Harris Detector Computer Vision I: Basics of Image Processing 07/11/ The autocorrelation function Function c is (after approximation) a quadratic function in Isolines are ellipses ( is symmetric and positive definite); Eigenvector x 1 with (larger) Eigenvalue λ 1 is the direction of fastest change in function c Eigenvector x 2 with (smaller) Eigenvalue λ 2 is direction of slowest change in function c and x 2 Δx Δy x 1 Function c Note c = 0 for Δx = Δy = 0

53 Computer Vision I: Basics of Image Processing 07/11/ Harris Detector Some examples isolines for : (a) Flat (b) Edges (c) Corners a. Homogenous regions: both -s are small b. Edges: one is small the other one is large c. Corners: both -s are large (this is what we are looking for!)

54 Harris Detector Computer Vision I: Basics of Image Processing 07/11/ Image λ 1 (larger eigenvalue) λ 2 (smaller eigenvalue) Zoom in

55 Computer Vision I: Basics of Image Processing 07/11/ Harris detector Cornerness is a characteristic of Proposition by Harris: Downweights edges where λ 1 λ 2 Harris Value Smallest eigenvalue

56 Harris Corners - example Computer Vision I: Basics of Image Processing 07/11/

57 H-score (red- high, blue - low) Computer Vision I: Basics of Image Processing 07/11/

58 Threshold (H-score > value) Computer Vision I: Basics of Image Processing 07/11/

59 Non-maximum suppression Computer Vision I: Basics of Image Processing 07/11/

60 Harris corners in red Computer Vision: Algorithms and Applications -- 07/11/

61 Other examples Computer Vision I: Basics of Image Processing 07/11/

62 Maximally stable extremal regions Computer Vision I: Basics of Image Processing 07/11/ Invariant to affine transformation of gray-values Both small and large structures are detected

63 Literature Computer Vision I: Basics of Image Processing 07/11/ There is a large body of literature on detectors and descriptors (later lecture) A comparison paper (e.g. what is the most robust corner detectors): K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman, J. Matas, F. Schaffalitzky, T. Kadir: A Comparison of Affine Region Detectors (IJCV 2006)