Computer Vision I - Algorithms and Applications: Basics of Image Processing Carsten Rother 28/10/2013 Computer Vision I: Basics of Image Processing
Link to lectures Computer Vision I: Basics of Image Processing 28/10/2013 2 Slides of Lectures and Exercises will be online: http://www.inf.tu-dresden/index.php?node_id=2091&ln=en (on our webpage > teaching > Computer Vision)
Roadmap: Basics Digital Image Processing Computer Vision I: Basics of Image Processing 28/10/2013 3 Images Point operators (ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) main focus Linear filtering Non-linear filtering Fourier Transformation (ch. 3.4) Multi-scale image representation (ch. 3.5) Edges (ch. 4.2) Edge detection and linking Lines (ch. 4.3) Line detection and vanishing point detection
Roadmap: Basics Digital Image Processing Computer Vision I: Basics of Image Processing 28/10/2013 4 Images Point operators (ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) main focus Linear filtering Non-linear filtering Fourier Transformation (ch. 3.4) Multi-scale image representation (ch. 3.5) Edges (ch. 4.2) Edge detection and linking Lines (ch. 4.3) Line detection and vanishing point detection
What is an Image Computer Vision I: Image Formation Process 28/10/2013 5 We can think of the image as a function: I x, y, I: For every 2D point (pixel) it tells us the amount of light it receives The size and range of the sensor is limited: I x, y, I: a, b c, d [0, m] Colour image is then a vector-valued function: I x, y = I R x, y I G x, y I B x, y, I: a, b c, d 0, m 3 Comment, in most lectures we deal with grey-valued images and extension to colour is obvious
Images as functions Computer Vision I: Basics of Image Processing 28/10/2013 6 [from Steve Seitz]
Digital Images Computer Vision I: Basics of Image Processing 28/10/2013 7 We usually do not work with spatially continuous functions, since our cameras do not sense in this way. Instead we use (spatially) discrete images Sample the 2D domain on a regular grid (1D version) Intensity/color values usually also discrete. Quantize the values per channel (e.g. 8 bit per channel)
Computer Vision I: Basics of Image Processing 28/10/2013 8 Comment on Continuous Domain / Range There is a branch of computer vision research ( variational methods ), which operates on continuous domain for input images and output results Continuous domain methods are typically used for physics-based vision: segmentation, optical flow, etc. (we may consider this briefly in later lectures) Continues domain methods then use different optimization techniques, but still discretize in the end. In this lecture and other lectures we mainly operate in discrete domain and discrete or continuous range for output results
Roadmap: Basics Digital Image Processing Computer Vision I: Basics of Image Processing 28/10/2013 9 Images Point operators (ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) main focus Linear filtering Non-linear filtering Fourier Transformation (ch. 3.4) Multi-scale image representation (ch. 3.5) Edges (ch. 4.2) Edge detection and linking Lines (ch. 4.3) Line detection and vanishing point detection
Point operators Computer Vision I: Basics of Image Processing 28/10/2013 10 Point operators work on every pixel independently: J x, y = h I x, y Examples for h: Control contrast and brightness; h(z) = az + b original Contrast enhanced
Example for Point operators: Gamma correction Computer Vision I: Basics of Image Processing 28/10/2013 11 Intensity range: [0,1] Inside cameras: h z = z 1/γ where often γ = 2.2 (called gamma correction) In (old) CRT monitors An intensity z was perceived as: h z = z γ (γ = 2.2 typically) Today: even with linear mapping monitors, it is good to keep the gamma corrected image. Since human vision is more sensitive in dark areas. Important: for many tasks in vision, e.g. estimation of a normal, it is good to run h z = z γ to get to a linear function
Example for Point Operators: Alpha Matting Computer Vision I: Basics of Image Processing 28/10/2013 12 Foreground F Background B Matte α (amount of transparency) Composite C C x, y = α x, y F x, y + 1 α x, y B(x, y)
Roadmap: Basics Digital Image Processing Computer Vision I: Basics of Image Processing 28/10/2013 13 Images Point operators (ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) main focus Linear filtering Non-linear filtering Fourier Transformation (ch. 3.4) Multi-scale image representation (ch. 3.5) Edges (ch. 4.2) Edge detection and linking Lines (ch 4.3) Line detection and vanishing point detection
Linear Filters / Operators Computer Vision I: Basics of Image Processing 28/10/2013 14 Properties: Homogeneity: T[aX] = at[x] Additivity: T[X + Y] = T[X] + T[Y] Superposition: T[aX + by] = at[x] + bt[y] Example: Convolution Matrix-Vector operations
Convolution Computer Vision I: Basics of Image Processing 28/10/2013 15 Replace each pixel by a linear combination of its neighbours and itself. 2D convolution (discrete) g = f h Centred at 0,0 f x, y h x, y g x, y g x, y = k,l f x k, y l h k, l = k,l f k, l h x k, y l
Convolution Computer Vision I: Basics of Image Processing 28/10/2013 16 Linear h f 0 + f 1 = h f 0 + h f 1 Associative f g h = f g h Commutative f h = h f Shift-Invariant g x, y = f x + k, y + l h g x, y = (h f)(x + k, y + l) (behaves everywhere the same) Can be written in Matrix form: g = H f Correlation (not mirrored filter): g x, y = f x + k, y + l h k, l k,l
Examples Computer Vision I: Basics of Image Processing 28/10/2013 17 Impulse function: f = f δ δ y x Box Filter:
Application: Noise removal Noise is what we are not interested in: sensor noise (Gaussian, shot noise), quantisation artefacts, light fluctuation, etc. Typical assumption is that the noise is not correlated between pixels Basic Idea: neighbouring pixel contain information about intensity Computer Vision: Algorithms and Applications -- 28/10/2013 18
Noise removal Computer Vision I: Basics of Image Processing 28/10/2013 19
The box filter does noise removal Computer Vision I: Basics of Image Processing 28/10/2013 20 Box filter takes the mean in a neighbourhood Image Noise Pixel-independent Gaussian noise added Filtered Image
Derivation of the Box Filter Computer Vision I: Basics of Image Processing 28/10/2013 21 y r is true gray value (color) x r observed gray value (color) Noise model: Gaussian noise: p x r y r ) = N x r ; y r, σ ~ exp[ x r y r 2 2σ 2 ] y r x r
Derivation of Box Filter Computer Vision I: Basics of Image Processing 28/10/2013 22 Further assumption: independent noise p x y) ~ exp[ x r y r 2 2σ 2 ] r Find the most likely solution the true signal y Maximum-Likelihood principle (probability maximization): y p y p x y = argmax y p y x) = argmax y p(x) posterior p(x) is a constant (drop it out), assume (for now) uniform prior p(y). So we get: p y x) = p x y exp[ x r y r 2 2σ 2 ] the solution is trivial: y r = x r for all r r prior likelihood additional assumptions about the signal y are necessary!!!
Derivation of Box Filter Computer Vision I: Basics of Image Processing 28/10/2013 23 Assumption: not uniform prior p y but in a small vicinity the true signal is nearly constant Maximum-a-posteriori: For one pixel r : take neg. logarithm: p y x) exp[ x r y r 2 y r = argmax yr exp[ x r y r 2 y r = argmin yr x r y r 2 r 2σ 2 ] 2σ 2 ] Only one y r in a window W(r)
Derivation of Box Filter Computer Vision I: Basics of Image Processing 28/10/2013 24 How to do the minimization: y r = argmin yr x r y r 2 Take derivative and set to 0: F y r = x r y r 2 y r (the average) Box filter optimal under pixel-independent Gaussian Noise and constant signal in window
Gaussian (Smoothing) Filters Computer Vision I: Basics of Image Processing 28/10/2013 25 Nearby pixels are weighted more than distant pixels Isotropic Gaussian (rotational symmetric)
Gaussian Filter Computer Vision I: Basics of Image Processing 28/10/2013 26 Input: constant grey-value image More noise needs larger sigma
Handling the Boundary (Padding) Computer Vision I: Basics of Image Processing 28/10/2013 27
Gaussian for Sharpening Computer Vision I: Basics of Image Processing 28/10/2013 28 Sharpen an image by amplifying what is smoothing removes: g = f + γ (f h blur f)
How to compute convolution efficiently? Computer Vision I: Basics of Image Processing 28/10/2013 29 Separable filters (next) Fourier transformation (see later) Integral Image trick (see exercise) Important for later (integral Image trick): The Box filter (mean filter) can be computed in O(N). Naive implemettaioin would be O(Nw) where w is the number of elements in box filter
Separable filters Computer Vision I: Basics of Image Processing 28/10/2013 30 For some filters we have: f h = f (h x h y ) Where h x, h y are 1D filters. Example Box filter: h x h h y y h x Now we can do two 1D convolutions: f h = f h x h y = (f h x ) h y Naïve implementation for 3x3 filter: 9N operations versus 3N+3N operations
Can any filter be made separable? Computer Vision I: Basics of Image Processing 28/10/2013 31 Note: h x h y h y h x h x h y Apply SVD to the kernel matrix: If all σ i are 0 (apart from σ 0 ) then it is separable.
Example of separable filters Computer Vision I: Basics of Image Processing 28/10/2013 32
Roadmap: Basics Digital Image Processing Computer Vision I: Basics of Image Processing 28/10/2013 33 Images Point operators (ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) main focus Linear filtering Non-linear filtering Fourier Transformation (ch, 3.4) Multi-scale image representation (ch. 3.5) Edges (ch. 4.2) Edge detection and linking Lines (ch. 4.3) Line detection and vanishing point detection
Non-linear filters Computer Vision I: Basics of Image Processing 28/10/2013 34 There are many different non-linear filters. We look at a selection: Median filter Bilateral filter (Guided Filter) Morphological operations
Shot noise (Salt and Pepper Noise) - motivation Computer Vision I: Basics of Image Processing 28/10/2013 35 Gaussian filtered Original + shot noise Median filtered
Another example Computer Vision I: Basics of Image Processing 28/10/2013 36 Original Noised Mean Median
Median Filter Computer Vision I: Basics of Image Processing 28/10/2013 37 Replace each pixel with the median in a neighbourhood: 5 6 5 5 6 5 4 20 5 4 5 5 4 6 5 4 6 5 median Median filter: order the values and take the middle one No strong smoothing effect since values are not averaged Very good to remove outliers (shot noise) Used a lot for post processing of outputs (e.g. optical flow)
Median Filter: Derivation Computer Vision I: Basics of Image Processing 28/10/2013 38 Reminder: for Gaussian noise we did solve the following ML problem y r = argmax yr exp[ x r y r 2 2σ 2 ] = argmin yr x r y r = 1/ W x r p y x) 2 median mean Does not look like a Gaussian distribution For Median we solve the following problem: y r = argmax yr exp[ x r y r 2σ 2 ] = argmin yr x r y r = Median (W r ) Due to absolute norm it is more robust
Median Filter Derivation Computer Vision I: Basics of Image Processing 28/10/2013 39 function: minimize the following: F y r = x r y r Problem: not differentiable, good news: it is convex Optimal solution is the mean of all values
Motivation Bilateral Filter Computer Vision I: Basics of Image Processing 28/10/2013 40 Original + Gaussian noise Gaussian filtered Bilateral filtered
Bilateral Filter in pictures Computer Vision I: Basics of Image Processing 28/10/2013 41 Gaussian Filter weights Output (sketched) Centre pixel Noisy input Bilateral Filter weights Output
Bilateral Filter in equations Computer Vision I: Basics of Image Processing 28/10/2013 42 Filters looks at: a) distance of surrounding pixels (as Gaussian) b) Intensity of surrounding pixels Linear combination Similar to Gaussian filter Consider intensity Problem: computation is slow O Nw ; approximations can be done in O(N) Comment: Guided filter (see later) is similar and can be computed exactly in O(N) See a tutorial on: http://people.csail.mit.edu/sparis/bf_course/
Application: Bilteral Filter Computer Vision I: Basics of Image Processing 28/10/2013 43 Cartoonization HDR compression (Tone mapping)
Joint Bilteral Filter Computer Vision I: Basics of Image Processing 28/10/2013 44 ~ ~ Similar to Gaussian Consider intensity f is the input image which is processed ~ f is a guidance image where we look for pixel similarity
Application: combine Flash and No-Flash Computer Vision I: Basics of Image Processing 28/10/2013 45 input image f ~ guidance image f Joint Bilateral Filter ~ ~ We don t care about absolute colors [Petschnigg et al. Siggraph 04]
Application: Cost Volume Filtering Computer Vision I: Basics of Image Processing 28/10/2013 46 Reminder from first Lecture: Interactive Segmentation Goal z = R, G, B n x = 0,1 n Given z; derive binary x: Model: Energy function E x = i θ i x i + i,j θ ij (x i, x j ) Unary terms Pairwise terms Algorithm to minimization: x = argmin x E(x)
Reminder: Unary term New query image z i θ i (x i = 0) θ i (x i = 1) Dark means likely background Dark means likely foreground Optimum with unary terms only Computer Vision I: Introduction 28/10/2013 47
Cost Volumne for Binary Segmenation Label Space (here 2) Computer Vision I: Basics of Image Processing 28/10/2013 48 θ i (x i = 1) θ i (x i = 0) Image (x,y) For 2 Labels, we can also look at the ratio Image: I i = θ i (x i = 1) / θ i (x i = 0)
Application: Cost Volumne Filtering Computer Vision I: Basics of Image Processing 28/10/2013 49 ~ Guidance Input Image f (user brush strokes) Ratio Cost-volume is the Input Image f Filtered cost volume An alternative to energy minimization Winner takes all Result Energy minimization [C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz, Fast Cost- Volume Filtering for Visual Correspondence and Beyond, CVPR 11]
Application: Cost volume filtering for dense Stereo Computer Vision I: Basics of Image Processing 28/10/2013 50 Stereo Image pair ~ Guidance Image f 20-label cost volume f Box filter Stereo result (winner takes all) Bilateral filter Guided filter True solution [C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz, Fast Cost- Volume Filtering for Visual Correspondence and Beyond, CVPR 11]
Application: Cost volume filtering for dense Stereo Computer Vision I: Basics of Image Processing 28/10/2013 51 Very competative in terms of results for a fast methods (Middleburry Ranking) [C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz, Fast Cost- Volume Filtering for Visual Correspondence and Beyond, CVPR 11]
Recent Trend: Guided Filter Computer Vision I: Basics of Image Processing 28/10/2013 52 i Diferent pixel coordinates i, j linear combination of image f Size of window ω k is fixed, e.g. 7x7. Sum over all windows ω k which contain pixels: i and j Different to biltarel filter since a sum over small windows ω k 7x7 pixels [He, Sun ECCV 10]
Recent Trend: Guided Filter Computer Vision I: Basics of Image Processing 28/10/2013 53 Diferent pixel cooridinates i, j linear combination of image f Different to biltarel filter since a sum over small windows mean in window ω k variance in window ω k Size of window ω k is fixed, e.g. 7x7. Sum over all windows ω k which contain pixels: i and j f i ~ fj ~ ~ f i A window ω k which is centred exactly on the edge ~ Case 1: I i, I j on the same side: f i μ k (f ~ j μ k ) have the same sign. Then W ij large Case 2: I i, I j on the same side: f ~ ~ i μ k (f j μ k ) have different sign. Then W ij small
Bilteral Filter and Guided Filter behave very similiarly Computer Vision I: Basics of Image Processing 28/10/2013 54
Bilteral Filter and Guided Filter behave very similiarly Computer Vision I: Basics of Image Processing 28/10/2013 55 Guided Filter Bilteral Filter
Guided Filter: Can be computed in O(N) Computer Vision I: Basics of Image Processing 28/10/2013 56 Can also be written as: (see paper for detail) Integral Image trick [He, Sun ECCV 10]
Applications: Matting Computer Vision I: Basics of Image Processing 28/10/2013 57 Guideanace Image f ~ Input Image f Output Image g [He, Sun ECCV 10]
Morphological operations Computer Vision I: Basics of Image Processing 28/10/2013 58 Perform convolution with a structural element : binary mask (e.g. circle or square) black is 1 white is 1 Then perform thresholding to recover a binary image
Opening and Closing Operations Computer Vision I: Basics of Image Processing 28/10/2013 59 Opening operation: dilate erode f, s, s Closing opertiaon: erode dialte f, s, s Input image opening closing erode and dilate are not commutative
Application: Denoise Binary Segmentation Computer Vision I: Basics of Image Processing 28/10/2013 60 Input Segmentation Opening than closing Closing than opening Note: nothing is commutative
Application: Binary Segmentation Computer Vision I: Basics of Image Processing 28/10/2013 61 Extend morphological operations to deal with cost volume and make it edge preserving (same idea as in joint bilateral filter) Ratio Cost-volume is the Input Image f Energy minimization Result: Edge preserving Opening and closing Energy minimization ours Again: An alternative to energy minization [Criminisi, Sharp, Blake, GeoS: Geodesic Image Segmentation, ECCV 08]
Related nonlinear operations on binary images Computer Vision I: Basics of Image Processing 28/10/2013 62 Binary Image Distance transform Skeleton Binary Input Image Connected components
Roadmap: Basics Digital Image Processing Computer Vision I: Basics of Image Processing 28/10/2013 63 Images Point operators (Ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) main focus Linear filtering Non-linear filtering Fourier Transformation (ch. 3.4) Multi-scale image representation (ch. 3.5) Edges (Ch. 4.2) Edge detection and linking Lines (Ch 4.3) Line detection and vanishing point detection
Fourier Transformation to analyse Filters Computer Vision I: Basics of Image Processing 28/10/2013 64 How does a sinusoid influences a given filter/image h(x)? Complex valued, continuous sinusoid for different frequency ω Filter/Image o x = h x s x = A e j(wx+φ) = A [ cos(ω x + φ) + j sin(ω x + φ) ] Amplitude Output signal phase The output is also a sinusoid Simply try all possible ω and record A, φ. The Fourier transformation of h(x) is then: H ω = h x = A e jφ = A (cos φ + j sin φ )
Fourier Transform Computer Vision I: Basics of Image Processing 28/10/2013 65 Low-pass filter: Band-pass filter:
Fourier Pair: Computation discrete continuous Computer Vision I: Basics of Image Processing 28/10/2013 66 h x H(ω) H ω = h x e jωx dx h x = H ω e jωx dω N 1 H k = 1 N x=0 h x e j2πkx N N 1 h(x) = 1 N k=0 H k e j2πkx N Discrete Fourier transformation N is the range of signal (image region) Inverse Discrete Fourier transformation
Discrete Inverse Fourier Transform: Visualization Computer Vision I: Basics of Image Processing 28/10/2013 67 N 1 h(x) = 1 N k=0 H k e j2πkx N For this signal a reconstruction with sinus function only is sufficient
Discrete Inverse Fourier Transform: Visualization Computer Vision I: Basics of Image Processing 28/10/2013 68 N 1 h(x) = 1 N k=0 H k e j2πkx N [from wikipedia]
Example: Discrete 2D Computer Vision I: Basics of Image Processing 28/10/2013 69 Original Amplitude Phase
Example: Discrete 2D Computer Vision I: Basics of Image Processing 28/10/2013 70 Original Amplitude Phase
Example: Discrete 2D Computer Vision I: Basics of Image Processing 28/10/2013 71
Fast Fourier Transformation Computer Vision I: Basics of Image Processing 28/10/2013 72 Important property: Fast computation: f x, h(x) (g x h x ) = G(ω) H(ω) O(Nw) convolution f x h(x) O(N logn) fourier transform O(N logn) inverse fourier transform F ω, H(ω) O(N) multiplication F ω H(ω) O(N log N)
Roadmap: Basics Digital Image Processing Computer Vision I: Basics of Image Processing 28/10/2013 73 Images Point operators (Ch. 3.1) Filtering: (ch. 3.2, ch 3.3, ch. 3.4) main focus Linear filtering Non-linear filtering Fourier Transformation (ch. 3.4) Multi-scale image representation (ch. 3.5) Edges (Ch. 4.2) Edge detection and linking Lines (Ch 4.3) Line detection and vanishing point detection
Reading for next class Computer Vision I: Introduction 28/10/2013 74 This lecture: Chapter 3 (in particular: 3.2, 3.3) - Basics of Digital Image Processing Next lecture: Chapter 3.5: multi-scale representation Chapter 4.2 and 4.3 - Edge and Line detection Chapter 2 (in particular: 2.1, 2.2) Image formation process And a bit of Hartley and Zisserman chapter 2