Computer Vision I - Algorithms and Applications: Basics of Image Processing

Transcription

1 Computer Vision I - Algorithms and Applications: Basics of Image Processing Carsten Rother 28/10/2013 Computer Vision I: Basics of Image Processing

2 Link to lectures Computer Vision I: Basics of Image Processing 28/10/ Slides of Lectures and Exercises will be online: (on our webpage > teaching > Computer Vision)

3 Roadmap: Basics Digital Image Processing Computer Vision I: Basics of Image Processing 28/10/ Images Point operators (ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) main focus Linear filtering Non-linear filtering Fourier Transformation (ch. 3.4) Multi-scale image representation (ch. 3.5) Edges (ch. 4.2) Edge detection and linking Lines (ch. 4.3) Line detection and vanishing point detection

4 Roadmap: Basics Digital Image Processing Computer Vision I: Basics of Image Processing 28/10/ Images Point operators (ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) main focus Linear filtering Non-linear filtering Fourier Transformation (ch. 3.4) Multi-scale image representation (ch. 3.5) Edges (ch. 4.2) Edge detection and linking Lines (ch. 4.3) Line detection and vanishing point detection

5 What is an Image Computer Vision I: Image Formation Process 28/10/ We can think of the image as a function: I x, y, I: For every 2D point (pixel) it tells us the amount of light it receives The size and range of the sensor is limited: I x, y, I: a, b c, d [0, m] Colour image is then a vector-valued function: I x, y = I R x, y I G x, y I B x, y, I: a, b c, d 0, m 3 Comment, in most lectures we deal with grey-valued images and extension to colour is obvious

6 Images as functions Computer Vision I: Basics of Image Processing 28/10/ [from Steve Seitz]

7 Digital Images Computer Vision I: Basics of Image Processing 28/10/ We usually do not work with spatially continuous functions, since our cameras do not sense in this way. Instead we use (spatially) discrete images Sample the 2D domain on a regular grid (1D version) Intensity/color values usually also discrete. Quantize the values per channel (e.g. 8 bit per channel)

8 Computer Vision I: Basics of Image Processing 28/10/ Comment on Continuous Domain / Range There is a branch of computer vision research ( variational methods ), which operates on continuous domain for input images and output results Continuous domain methods are typically used for physics-based vision: segmentation, optical flow, etc. (we may consider this briefly in later lectures) Continues domain methods then use different optimization techniques, but still discretize in the end. In this lecture and other lectures we mainly operate in discrete domain and discrete or continuous range for output results

9 Roadmap: Basics Digital Image Processing Computer Vision I: Basics of Image Processing 28/10/ Images Point operators (ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) main focus Linear filtering Non-linear filtering Fourier Transformation (ch. 3.4) Multi-scale image representation (ch. 3.5) Edges (ch. 4.2) Edge detection and linking Lines (ch. 4.3) Line detection and vanishing point detection

10 Point operators Computer Vision I: Basics of Image Processing 28/10/ Point operators work on every pixel independently: J x, y = h I x, y Examples for h: Control contrast and brightness; h(z) = az + b original Contrast enhanced

11 Example for Point operators: Gamma correction Computer Vision I: Basics of Image Processing 28/10/ Intensity range: [0,1] Inside cameras: h z = z 1/γ where often γ = 2.2 (called gamma correction) In (old) CRT monitors An intensity z was perceived as: h z = z γ (γ = 2.2 typically) Today: even with linear mapping monitors, it is good to keep the gamma corrected image. Since human vision is more sensitive in dark areas. Important: for many tasks in vision, e.g. estimation of a normal, it is good to run h z = z γ to get to a linear function

12 Example for Point Operators: Alpha Matting Computer Vision I: Basics of Image Processing 28/10/ Foreground F Background B Matte α (amount of transparency) Composite C C x, y = α x, y F x, y + 1 α x, y B(x, y)

13 Roadmap: Basics Digital Image Processing Computer Vision I: Basics of Image Processing 28/10/ Images Point operators (ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) main focus Linear filtering Non-linear filtering Fourier Transformation (ch. 3.4) Multi-scale image representation (ch. 3.5) Edges (ch. 4.2) Edge detection and linking Lines (ch 4.3) Line detection and vanishing point detection

14 Linear Filters / Operators Computer Vision I: Basics of Image Processing 28/10/ Properties: Homogeneity: T[aX] = at[x] Additivity: T[X + Y] = T[X] + T[Y] Superposition: T[aX + by] = at[x] + bt[y] Example: Convolution Matrix-Vector operations

15 Convolution Computer Vision I: Basics of Image Processing 28/10/ Replace each pixel by a linear combination of its neighbours and itself. 2D convolution (discrete) g = f h Centred at 0,0 f x, y h x, y g x, y g x, y = k,l f x k, y l h k, l = k,l f k, l h x k, y l

16 Convolution Computer Vision I: Basics of Image Processing 28/10/ Linear h f 0 + f 1 = h f 0 + h f 1 Associative f g h = f g h Commutative f h = h f Shift-Invariant g x, y = f x + k, y + l h g x, y = (h f)(x + k, y + l) (behaves everywhere the same) Can be written in Matrix form: g = H f Correlation (not mirrored filter): g x, y = f x + k, y + l h k, l k,l

17 Examples Computer Vision I: Basics of Image Processing 28/10/ Impulse function: f = f δ δ y x Box Filter:

18 Application: Noise removal Noise is what we are not interested in: sensor noise (Gaussian, shot noise), quantisation artefacts, light fluctuation, etc. Typical assumption is that the noise is not correlated between pixels Basic Idea: neighbouring pixel contain information about intensity Computer Vision: Algorithms and Applications -- 28/10/

19 Noise removal Computer Vision I: Basics of Image Processing 28/10/

20 The box filter does noise removal Computer Vision I: Basics of Image Processing 28/10/ Box filter takes the mean in a neighbourhood Image Noise Pixel-independent Gaussian noise added Filtered Image

21 Derivation of the Box Filter Computer Vision I: Basics of Image Processing 28/10/ y r is true gray value (color) x r observed gray value (color) Noise model: Gaussian noise: p x r y r ) = N x r ; y r, σ ~ exp[ x r y r 2 2σ 2 ] y r x r

22 Derivation of Box Filter Computer Vision I: Basics of Image Processing 28/10/ Further assumption: independent noise p x y) ~ exp[ x r y r 2 2σ 2 ] r Find the most likely solution the true signal y Maximum-Likelihood principle (probability maximization): y p y p x y = argmax y p y x) = argmax y p(x) posterior p(x) is a constant (drop it out), assume (for now) uniform prior p(y). So we get: p y x) = p x y exp[ x r y r 2 2σ 2 ] the solution is trivial: y r = x r for all r r prior likelihood additional assumptions about the signal y are necessary!!!

23 Derivation of Box Filter Computer Vision I: Basics of Image Processing 28/10/ Assumption: not uniform prior p y but in a small vicinity the true signal is nearly constant Maximum-a-posteriori: For one pixel r : take neg. logarithm: p y x) exp[ x r y r 2 y r = argmax yr exp[ x r y r 2 y r = argmin yr x r y r 2 r 2σ 2 ] 2σ 2 ] Only one y r in a window W(r)

24 Derivation of Box Filter Computer Vision I: Basics of Image Processing 28/10/ How to do the minimization: y r = argmin yr x r y r 2 Take derivative and set to 0: F y r = x r y r 2 y r (the average) Box filter optimal under pixel-independent Gaussian Noise and constant signal in window

25 Gaussian (Smoothing) Filters Computer Vision I: Basics of Image Processing 28/10/ Nearby pixels are weighted more than distant pixels Isotropic Gaussian (rotational symmetric)

26 Gaussian Filter Computer Vision I: Basics of Image Processing 28/10/ Input: constant grey-value image More noise needs larger sigma

27 Handling the Boundary (Padding) Computer Vision I: Basics of Image Processing 28/10/

28 Gaussian for Sharpening Computer Vision I: Basics of Image Processing 28/10/ Sharpen an image by amplifying what is smoothing removes: g = f + γ (f h blur f)

29 How to compute convolution efficiently? Computer Vision I: Basics of Image Processing 28/10/ Separable filters (next) Fourier transformation (see later) Integral Image trick (see exercise) Important for later (integral Image trick): The Box filter (mean filter) can be computed in O(N). Naive implemettaioin would be O(Nw) where w is the number of elements in box filter

30 Separable filters Computer Vision I: Basics of Image Processing 28/10/ For some filters we have: f h = f (h x h y ) Where h x, h y are 1D filters. Example Box filter: h x h h y y h x Now we can do two 1D convolutions: f h = f h x h y = (f h x ) h y Naïve implementation for 3x3 filter: 9N operations versus 3N+3N operations

31 Can any filter be made separable? Computer Vision I: Basics of Image Processing 28/10/ Note: h x h y h y h x h x h y Apply SVD to the kernel matrix: If all σ i are 0 (apart from σ 0 ) then it is separable.

32 Example of separable filters Computer Vision I: Basics of Image Processing 28/10/

33 Roadmap: Basics Digital Image Processing Computer Vision I: Basics of Image Processing 28/10/ Images Point operators (ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) main focus Linear filtering Non-linear filtering Fourier Transformation (ch, 3.4) Multi-scale image representation (ch. 3.5) Edges (ch. 4.2) Edge detection and linking Lines (ch. 4.3) Line detection and vanishing point detection

34 Non-linear filters Computer Vision I: Basics of Image Processing 28/10/ There are many different non-linear filters. We look at a selection: Median filter Bilateral filter (Guided Filter) Morphological operations

35 Shot noise (Salt and Pepper Noise) - motivation Computer Vision I: Basics of Image Processing 28/10/ Gaussian filtered Original + shot noise Median filtered

36 Another example Computer Vision I: Basics of Image Processing 28/10/ Original Noised Mean Median

37 Median Filter Computer Vision I: Basics of Image Processing 28/10/ Replace each pixel with the median in a neighbourhood: median Median filter: order the values and take the middle one No strong smoothing effect since values are not averaged Very good to remove outliers (shot noise) Used a lot for post processing of outputs (e.g. optical flow)

38 Median Filter: Derivation Computer Vision I: Basics of Image Processing 28/10/ Reminder: for Gaussian noise we did solve the following ML problem y r = argmax yr exp[ x r y r 2 2σ 2 ] = argmin yr x r y r = 1/ W x r p y x) 2 median mean Does not look like a Gaussian distribution For Median we solve the following problem: y r = argmax yr exp[ x r y r 2σ 2 ] = argmin yr x r y r = Median (W r ) Due to absolute norm it is more robust

39 Median Filter Derivation Computer Vision I: Basics of Image Processing 28/10/ function: minimize the following: F y r = x r y r Problem: not differentiable, good news: it is convex Optimal solution is the mean of all values

40 Motivation Bilateral Filter Computer Vision I: Basics of Image Processing 28/10/ Original + Gaussian noise Gaussian filtered Bilateral filtered

41 Bilateral Filter in pictures Computer Vision I: Basics of Image Processing 28/10/ Gaussian Filter weights Output (sketched) Centre pixel Noisy input Bilateral Filter weights Output

42 Bilateral Filter in equations Computer Vision I: Basics of Image Processing 28/10/ Filters looks at: a) distance of surrounding pixels (as Gaussian) b) Intensity of surrounding pixels Linear combination Similar to Gaussian filter Consider intensity Problem: computation is slow O Nw ; approximations can be done in O(N) Comment: Guided filter (see later) is similar and can be computed exactly in O(N) See a tutorial on:

43 Application: Bilteral Filter Computer Vision I: Basics of Image Processing 28/10/ Cartoonization HDR compression (Tone mapping)

44 Joint Bilteral Filter Computer Vision I: Basics of Image Processing 28/10/ ~ ~ Similar to Gaussian Consider intensity f is the input image which is processed ~ f is a guidance image where we look for pixel similarity

45 Application: combine Flash and No-Flash Computer Vision I: Basics of Image Processing 28/10/ input image f ~ guidance image f Joint Bilateral Filter ~ ~ We don t care about absolute colors [Petschnigg et al. Siggraph 04]

46 Application: Cost Volume Filtering Computer Vision I: Basics of Image Processing 28/10/ Reminder from first Lecture: Interactive Segmentation Goal z = R, G, B n x = 0,1 n Given z; derive binary x: Model: Energy function E x = i θ i x i + i,j θ ij (x i, x j ) Unary terms Pairwise terms Algorithm to minimization: x = argmin x E(x)

47 Reminder: Unary term New query image z i θ i (x i = 0) θ i (x i = 1) Dark means likely background Dark means likely foreground Optimum with unary terms only Computer Vision I: Introduction 28/10/

48 Cost Volumne for Binary Segmenation Label Space (here 2) Computer Vision I: Basics of Image Processing 28/10/ θ i (x i = 1) θ i (x i = 0) Image (x,y) For 2 Labels, we can also look at the ratio Image: I i = θ i (x i = 1) / θ i (x i = 0)

49 Application: Cost Volumne Filtering Computer Vision I: Basics of Image Processing 28/10/ ~ Guidance Input Image f (user brush strokes) Ratio Cost-volume is the Input Image f Filtered cost volume An alternative to energy minimization Winner takes all Result Energy minimization [C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz, Fast Cost- Volume Filtering for Visual Correspondence and Beyond, CVPR 11]

50 Application: Cost volume filtering for dense Stereo Computer Vision I: Basics of Image Processing 28/10/ Stereo Image pair ~ Guidance Image f 20-label cost volume f Box filter Stereo result (winner takes all) Bilateral filter Guided filter True solution [C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz, Fast Cost- Volume Filtering for Visual Correspondence and Beyond, CVPR 11]

51 Application: Cost volume filtering for dense Stereo Computer Vision I: Basics of Image Processing 28/10/ Very competative in terms of results for a fast methods (Middleburry Ranking) [C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz, Fast Cost- Volume Filtering for Visual Correspondence and Beyond, CVPR 11]

52 Recent Trend: Guided Filter Computer Vision I: Basics of Image Processing 28/10/ i Diferent pixel coordinates i, j linear combination of image f Size of window ω k is fixed, e.g. 7x7. Sum over all windows ω k which contain pixels: i and j Different to biltarel filter since a sum over small windows ω k 7x7 pixels [He, Sun ECCV 10]

53 Recent Trend: Guided Filter Computer Vision I: Basics of Image Processing 28/10/ Diferent pixel cooridinates i, j linear combination of image f Different to biltarel filter since a sum over small windows mean in window ω k variance in window ω k Size of window ω k is fixed, e.g. 7x7. Sum over all windows ω k which contain pixels: i and j f i ~ fj ~ ~ f i A window ω k which is centred exactly on the edge ~ Case 1: I i, I j on the same side: f i μ k (f ~ j μ k ) have the same sign. Then W ij large Case 2: I i, I j on the same side: f ~ ~ i μ k (f j μ k ) have different sign. Then W ij small

54 Bilteral Filter and Guided Filter behave very similiarly Computer Vision I: Basics of Image Processing 28/10/

55 Bilteral Filter and Guided Filter behave very similiarly Computer Vision I: Basics of Image Processing 28/10/ Guided Filter Bilteral Filter

56 Guided Filter: Can be computed in O(N) Computer Vision I: Basics of Image Processing 28/10/ Can also be written as: (see paper for detail) Integral Image trick [He, Sun ECCV 10]

57 Applications: Matting Computer Vision I: Basics of Image Processing 28/10/ Guideanace Image f ~ Input Image f Output Image g [He, Sun ECCV 10]

58 Morphological operations Computer Vision I: Basics of Image Processing 28/10/ Perform convolution with a structural element : binary mask (e.g. circle or square) black is 1 white is 1 Then perform thresholding to recover a binary image

59 Opening and Closing Operations Computer Vision I: Basics of Image Processing 28/10/ Opening operation: dilate erode f, s, s Closing opertiaon: erode dialte f, s, s Input image opening closing erode and dilate are not commutative

60 Application: Denoise Binary Segmentation Computer Vision I: Basics of Image Processing 28/10/ Input Segmentation Opening than closing Closing than opening Note: nothing is commutative

61 Application: Binary Segmentation Computer Vision I: Basics of Image Processing 28/10/ Extend morphological operations to deal with cost volume and make it edge preserving (same idea as in joint bilateral filter) Ratio Cost-volume is the Input Image f Energy minimization Result: Edge preserving Opening and closing Energy minimization ours Again: An alternative to energy minization [Criminisi, Sharp, Blake, GeoS: Geodesic Image Segmentation, ECCV 08]

62 Related nonlinear operations on binary images Computer Vision I: Basics of Image Processing 28/10/ Binary Image Distance transform Skeleton Binary Input Image Connected components

63 Roadmap: Basics Digital Image Processing Computer Vision I: Basics of Image Processing 28/10/ Images Point operators (Ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) main focus Linear filtering Non-linear filtering Fourier Transformation (ch. 3.4) Multi-scale image representation (ch. 3.5) Edges (Ch. 4.2) Edge detection and linking Lines (Ch 4.3) Line detection and vanishing point detection

64 Fourier Transformation to analyse Filters Computer Vision I: Basics of Image Processing 28/10/ How does a sinusoid influences a given filter/image h(x)? Complex valued, continuous sinusoid for different frequency ω Filter/Image o x = h x s x = A e j(wx+φ) = A [ cos(ω x + φ) + j sin(ω x + φ) ] Amplitude Output signal phase The output is also a sinusoid Simply try all possible ω and record A, φ. The Fourier transformation of h(x) is then: H ω = h x = A e jφ = A (cos φ + j sin φ )

65 Fourier Transform Computer Vision I: Basics of Image Processing 28/10/ Low-pass filter: Band-pass filter:

66 Fourier Pair: Computation discrete continuous Computer Vision I: Basics of Image Processing 28/10/ h x H(ω) H ω = h x e jωx dx h x = H ω e jωx dω N 1 H k = 1 N x=0 h x e j2πkx N N 1 h(x) = 1 N k=0 H k e j2πkx N Discrete Fourier transformation N is the range of signal (image region) Inverse Discrete Fourier transformation

67 Discrete Inverse Fourier Transform: Visualization Computer Vision I: Basics of Image Processing 28/10/ N 1 h(x) = 1 N k=0 H k e j2πkx N For this signal a reconstruction with sinus function only is sufficient

68 Discrete Inverse Fourier Transform: Visualization Computer Vision I: Basics of Image Processing 28/10/ N 1 h(x) = 1 N k=0 H k e j2πkx N [from wikipedia]

69 Example: Discrete 2D Computer Vision I: Basics of Image Processing 28/10/ Original Amplitude Phase

70 Example: Discrete 2D Computer Vision I: Basics of Image Processing 28/10/ Original Amplitude Phase

71 Example: Discrete 2D Computer Vision I: Basics of Image Processing 28/10/

72 Fast Fourier Transformation Computer Vision I: Basics of Image Processing 28/10/ Important property: Fast computation: f x, h(x) (g x h x ) = G(ω) H(ω) O(Nw) convolution f x h(x) O(N logn) fourier transform O(N logn) inverse fourier transform F ω, H(ω) O(N) multiplication F ω H(ω) O(N log N)

73 Roadmap: Basics Digital Image Processing Computer Vision I: Basics of Image Processing 28/10/ Images Point operators (Ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) main focus Linear filtering Non-linear filtering Fourier Transformation (ch. 3.4) Multi-scale image representation (ch. 3.5) Edges (Ch. 4.2) Edge detection and linking Lines (Ch 4.3) Line detection and vanishing point detection

74 Reading for next class Computer Vision I: Introduction 28/10/ This lecture: Chapter 3 (in particular: 3.2, 3.3) - Basics of Digital Image Processing Next lecture: Chapter 3.5: multi-scale representation Chapter 4.2 and Edge and Line detection Chapter 2 (in particular: 2.1, 2.2) Image formation process And a bit of Hartley and Zisserman chapter 2