Computer Vision 2 SS 18 Dr. Benjamin Guthier Professur für Bildverarbeitung Computer Vision 2 Dr. Benjamin Guthier
1. IMAGE PROCESSING Computer Vision 2 Dr. Benjamin Guthier
Content of this Chapter Non-linear Filtering Morphological Operations Optical Flow Fourier Transformation Computer Vision 2 Dr. Benjamin Guthier 3 1. Image Processing
Learning Goals After this chapter, you will be able to Implement an efficient median filter for large window sizes Implement a bilateral filter Implement the morphological operations erosion / dilation Explain the steps to calculate motion vector of one pixel Explain the idea of Fourier Transformation Explain, but not memorize the Fourier transform formulas Describe applications where it is useful Computer Vision 2 Dr. Benjamin Guthier 4 1. Image Processing
Recommended Reading R. Szeliski: Computer Vision Algorithms and Applications, Chapter 3 (image processing) and 8.4 (optical flow) Computer Vision 2 Dr. Benjamin Guthier 5 1. Image Processing
NON-LINEAR FILTERING Computer Vision 2 Dr. Benjamin Guthier 6 1. Image Processing
Motivation So far, only linear filters: Gaussian, Sobel Output is a linear combination of the input Sometimes non-linear operations are preferable To remove salt and pepper noise (random high / low pixels) To preserve edges when smoothing an image Computer Vision 2 Dr. Benjamin Guthier 7 1. Image Processing
Motivation Salt and Pepper Noise Gaussian filtering only blurs the noise. It is still visible. The energy of the noise remains in the image. Gaussian Filter Source: wikipedia.org Computer Vision 2 Dr. Benjamin Guthier 8 1. Image Processing
Motivation Strong Blurring Large Gaussian filter masks also blur the edges Sometimes we want to preserve edges Gaussian Filter Source: freeimages.com Computer Vision 2 Dr. Benjamin Guthier 9 1. Image Processing
Median Filter Uses a filter mask (e.g., 3 3) Calculates median of all pixels in the mask New value of the center pixel is the median 0 2 3 2 2 2 3 4 2 3 2 1 3 0 1 2 1 2 2 3 2 3 2 3 4 Pixels in the mask (sorted): 0, 1, 2, 2, 2, 3, 3, 3, 4 Median is the 5 th value New value of center pixel: 2 Computer Vision 2 Dr. Benjamin Guthier 10 1. Image Processing
Median Filter (2) Noisy pixels are outliers No effect on median 3 3 median filter removes noise without blurring Median Filter Source: wikipedia.org Computer Vision 2 Dr. Benjamin Guthier 11 1. Image Processing
Median Filter Implementation Straightforward algorithm (window size n n): Sort pixels in window by value Pick the middle one For large n, sorting is expensive Complexity of sorting n 2 pixels: O n 2 log n 2 = O(n 2 log n) Better algorithm (for large n): Create histogram of pixels in window Find median from the histogram Update the histogram after moving window Computer Vision 2 Dr. Benjamin Guthier 12 1. Image Processing
Median Filter Implementation (2) Finding median from a histogram Sum of all histogram bins equals window size n n Pixels in histogram are sorted Sum up bins until the sum exceeds half the window size The bin index is the median Window 2 3 2 3 4 2 1 3 0 Color / Index # Pixels Sum of bins Histogram 0 1 2 3 4 1 1 3 3 1 1 2 5 8 9 median 5 > 3 3 / 2 Computer Vision 2 Dr. Benjamin Guthier 13 1. Image Processing
Median Filter Implementation (3) Updating histogram after moving window to the right Subtract pixels left of the new window from histogram for each pixel I(x, y) in left column decrease hist[i(x, y)] by 1 Add new pixels on the right side to histogram Remove from histogram 0 2 3 2 2 2 3 4 2 3 2 1 3 0 1 2 1 2 2 3 Add to histogram Computer Vision 2 Dr. Benjamin Guthier 14 1. Image Processing
Median Filter Implementation (4) Finding median from histogram takes 128 steps on average Large constant, but independent of n Updating the histogram takes 2n steps Overall runtime: O(n) Better than O(n 2 log n) for large n Computer Vision 2 Dr. Benjamin Guthier 15 1. Image Processing
Bilateral Filter Motivation: Filter with a large mask without blurring edges Blur textures while maintaining sharp boundaries Application (among others): beautify filters in digital cameras to smoothen the skin Idea: Limit filtering to pixels within a mask with similar intensity value to center pixel Computer Vision 2 Dr. Benjamin Guthier 16 1. Image Processing
Bilateral Filter (2) Bilateral filtering is a weighted sum of pixels (i, j) around a center pixel (x, y): I x, y = σ i,j I i, j w(x, y, i, j) σ i,j w(x, y, i, j) Weighting function w is the product of a domain kernel d and a range kernel r: w x, y, i, j = d x, y, i, j r(x, y, i, j) Domain kernel is the usual Gaussian function d x, y, i, j = exp x i 2 + y j 2 2σ d 2 Computer Vision 2 Dr. Benjamin Guthier 17 1. Image Processing
Bilateral Filter (3) Range kernel measures similarity of intensity r x, y, i, j = exp I x, y I i, j 2 2σ r 2 σ d and σ r define the size of the domain and the range kernel, respectively is the Euclidean norm of the RGB color difference Computer Vision 2 Dr. Benjamin Guthier 18 1. Image Processing
Bilateral Filter Example Original image Filtered image Noise is filtered out Edge is preserved Domain kernel Range kernel Product of the two kernels Source: Durand and Dorsey: Fast bilateral filtering for the display of high-dynamic range images. SIGGRAPH 2002. Computer Vision 2 Dr. Benjamin Guthier 19 1. Image Processing
Bilateral Filter Example (2) Bilateral Filter Source: freeimages.com Computer Vision 2 Dr. Benjamin Guthier 20 1. Image Processing
Morphology Computer Vision 2 Dr. Benjamin Guthier 21 1. Image Processing
Morphology Morphological operations are used mainly on binary images Can also be applied to grayscale or color images Working with binary images is very fast E.g., in optical inspection tasks They change the shape of binary objects Make them bigger (dilation) Smaller (erosion) Fill in holes (closing) Remove dots and open gaps (opening) (names assume white objects on black background) Computer Vision 2 Dr. Benjamin Guthier 22 1. Image Processing
Morphology Examples Original Dilation Erosion Closing Opening Computer Vision 2 Dr. Benjamin Guthier 23 1. Image Processing
Morphological Operations Uses structuring element B Defines a square or circular window around center pixel Dilation: I x, y = max I(i, j) i,j B Replace center pixel with maximum pixel within B Erosion: I x, y = min I(i, j) i,j B Replace center pixel with minimum pixel within B Closing: dilation, followed by erosion Opening: erosion, followed by dilation Computer Vision 2 Dr. Benjamin Guthier 24 1. Image Processing
Dilation Example Structuring element B is a circle Dilation: replace center pixel with maximum in B Center pixel becomes white Center pixel remains black Computer Vision 2 Dr. Benjamin Guthier 25 1. Image Processing
OPTICAL FLOW Computer Vision 2 Dr. Benjamin Guthier 26 1. Image Processing
Optical Flow Global image registration only accounts for camera motion E.g., one 3 3 Matrix for motion between two frames Object motion requires fine-grained motion compensation E.g., block-based as in video compression, or Motion of every pixel: Optical flow Input: Two frames of a video I t and I t+1 Output: One 2D motion vector (u, v) for every pixel (x, y) Computer Vision 2 Dr. Benjamin Guthier 27 1. Image Processing
Optical Flow (2) Pixel motion is described by: I t x, y = I t+1 (x + u, y + v) The intensity at (x, y) moves to position (x + u, y + v) from frame t to frame t + 1 Intensity remains constant (assumption of the approach) Approximate intensity at t + 1 by Taylor expansion I t+1 x + u, y + v I t x, y + I t x, y u v + I t x, y t Temporal derivative is approximated by forward difference I t (x, y) t I t+1 x, y I t (x, y) Computer Vision 2 Dr. Benjamin Guthier 28 1. Image Processing
Taylor Expansion Intuition (1D) Cut through one image row The object moves from left to right I t+1 x + u I t x + I t x u + I t+1 x I t x I I t+1 x I t x I t x = I t+1 x + u I t x u x x + u x Computer Vision 2 Dr. Benjamin Guthier 29 1. Image Processing
Gradient Constraint Equation Combining constancy assumption with Taylor expansion gradient constraint equation: u I t x, y v + I t+1 x, y I t x, y = 0 One equation with two unknowns u and v Gradient I t (x, y) and pixel values I t+1 (x, y) and I t (x, y) are known To solve, consider pixels (x, y ) in window W around (x, y). Find (u, v) that minimizes the weighted sum: arg min (u,v) x,y W g x, y I t x, y g(x, y ): Gaussian weighting function u v + I t+1 x, y I t x, y Set partial derivatives w.r.t. u and v to zero and solve equation system 2 Computer Vision 2 Dr. Benjamin Guthier 30 1. Image Processing
Example Direction of motion is encoded as hue, intensity as value Source: docs.opencv.org Computer Vision 2 Dr. Benjamin Guthier 31 1. Image Processing
FOURIER TRANSFORMATION Computer Vision 2 Dr. Benjamin Guthier 32 1. Image Processing
Acknowledgements Some of the slides on Fourier transformation were taken from a lecture by Prof. S. Narasimhan at the Carnegie Mellon University, Pittsburgh, PA. Computer Vision 2 Dr. Benjamin Guthier 33 1. Image Processing
Fourier Transformation Idea Represent a signal as a weighted sum of sines and cosines Signal contains a frequency of 2 Hz with an amplitude of 4, and a frequency of 3 Hz with an amplitude of 0.5 Works similarly in 1D (time varying signals) and 2D (images) 1D: temporal frequencies (change per unit of time) 2D: spatial frequencies (change per unit of distance) Transformation to frequency domain has many advantages! Computer Vision 2 Dr. Benjamin Guthier 34 1. Image Processing
Sum of Sinusoids Target f 1 f 2 f 3 1 0 0 0 0 f 0 f 0 + f 1 f 0 + f 1 + f 2 f 0 + + f 3 1 1 1 1 0 0 0 0 Use sinus functions as building block: f i (x) = A i sin(ω i x + φ i ) Adding them up can create any target function 1 0 f 0 + + f 6 Computer Vision 2 Dr. Benjamin Guthier 35 1. Image Processing
Fourier Transform Goal: For a given signal f(x), find amplitude and phase of all frequencies that are contained in it f(x) (Inverse) Fourier Transform F(ω) f(x): value of the signal at position x F(ω): amplitude A and phase φ at frequency ω F(ω) are complex numbers: F ω = Re ω + i Im(ω) A = ± Re ω 2 + Im ω 2 1 Im(ω) and φ = tan Re(ω) Computer Vision 2 Dr. Benjamin Guthier 36 1. Image Processing
Frequency Spectrum Example f x = sin 2πkx + 1 sin 2π 3k x 3 = + 1 F ω 1 3 k 2k 3k ω Computer Vision 2 Dr. Benjamin Guthier 37 1. Image Processing
Frequency Spectrum Example (2) As we have seen: + + + f(x) + + + = F ω f x A σ k=1 1 k sin(2πkx) 1 2 3 10 ω Computer Vision 2 Dr. Benjamin Guthier 38 1. Image Processing
Discrete Fourier Transform (DFT) Fourier transform for one-dimensional discrete signals with x [0, W 1] and ω [0, W 1] is defined as: W 1 F ω = x=0 f x e 2πi W ωx with e 2πi W ωx = cos 2π W ωx i sin 2π W ωx And its inverse: W 1 f x = 1 W ω=0 F ω e 2πi W ωx Computer Vision 2 Dr. Benjamin Guthier 39 1. Image Processing
DFT Properties For each frequency ωτw, DFT can be viewed as filtering signal f(x) with a complex sinusoid at frequency ωτw F(ω) is the amount of frequency Τ ω W contained in f(x) F(ω) is W-periodic, so ω may also be in W 2, W 2 1 Left and right side will be swapped If all f(x) are real numbers (which they are in images), then F ω = F (W ω) Where F denotes complex conjugation F(ω) is symmetric and only W 2 complex values must be stored Computer Vision 2 Dr. Benjamin Guthier 40 1. Image Processing
DFT Symmetry f(x) F ω F ω : amplitude ω scaled to [ 0.5,0.5] F 0 = σ x=0 W 1 f x, so scales in frequency spectrum vary Computer Vision 2 Dr. Benjamin Guthier 41 1. Image Processing
2D Fourier Transform Discrete Fourier transform for 2D images f(x, y) W 1 F(ω 1, ω 2 ) = x=0 H 1 e 2πi W ω1x y=0 Building blocks are now plane waves ω 1 : horizontal frequency W ω 2 : vertical frequency H e 2πi ω 1 W x+ω2 H y e 2πi H ω 2y f(x, y) Computer Vision 2 Dr. Benjamin Guthier 42 1. Image Processing
Plane Waves Illustration ω 1 = 0 1 2 7 ω 2 = 0 1 7 64 example plane waves (spatial domain) 64 different frequency combinations Image is a linear combination of plane waves Source: wikipedia.org Computer Vision 2 Dr. Benjamin Guthier 43 1. Image Processing
Fourier Domain Examples Images and their Fourier spectrum Spatial lines show up as orthogonal lines in the frequency spectrum Computer Vision 2 Dr. Benjamin Guthier 44 1. Image Processing
Fourier Transform and Convolution Convolution in spatial domain is identical to multiplication in frequency domain: f = f g F = F G f : filtered image, f: original image, g: filter mask F, F, G: Fourier transform of the above : spatial convolution (filtering), : element-wise multiplication Filter images with large filter masks by: f = f g IFT FT FT F = F G Computer Vision 2 Dr. Benjamin Guthier 45 1. Image Processing
Fourier Transform and Convolution (2) f(x, y) F x, y Gaussian filter maintains shape after FT g(x, y) G x, y f (x, y) F x, y Computer Vision 2 Dr. Benjamin Guthier 46 1. Image Processing
Low-Pass Filtering Let low frequencies pass Eliminate high frequencies Creates blurred image, removes detail Computer Vision 2 Dr. Benjamin Guthier 47 1. Image Processing
High-Pass Filtering Let high frequencies pass Eliminate low frequencies Only keeps detail (edges) Computer Vision 2 Dr. Benjamin Guthier 48 1. Image Processing
Application: JPEG Compression Idea: Image distortions are more visible in the low frequency spectrum. Use less bits to store higher frequencies Uses the discrete cosine transform (DCT) Similar to Fourier transform, but using only real values and cosines Better properties for compression Splits image into blocks of 8 8 pixels Processes each block individually Computer Vision 2 Dr. Benjamin Guthier 49 1. Image Processing
Application: JPEG Compression (2) JPEG compression algorithm (simplified) Convert into color space that separates brightness from color Subsample color channels Convert each 8 8 block using the DCT Quantize DCT coefficients, such that high frequencies are stored with less bits (lossy compression) Compress sequence of DCT coefficients using lossless compression Computer Vision 2 Dr. Benjamin Guthier 50 1. Image Processing
Application: JPEG Compression (3) JPEG quality increases from left to right Increased quality = more bits used to encode high frequencies Source: wikipedia.org At low quality, only ω 1, ω 2 = 0 is encoded Only average color of block Computer Vision 2 Dr. Benjamin Guthier 51 1. Image Processing
Application: Eulerian Video Magnification Idea: Use Fourier transform to visualize pulse signal from a video In a video, each pixel is a 1D signal over time If video is perfectly still, signal should be constant Contains only DC frequency (F ω = 0, for ω > 0) However, it is not constant H.-Y. Wu, M. Rubinstein, et al. Eulerian Video Magnification for Revealing Subtle Changes in the World, Siggraph 2012. Computer Vision 2 Dr. Benjamin Guthier 52 1. Image Processing
Eulerian Video Magnification (2) Luminance variation of one pixel over time Averaged over multiple pixels pulses Video frame Computer Vision 2 Dr. Benjamin Guthier 53 1. Image Processing
Eulerian Video Magnification (3) Pulse s periodic variation over time shows up as a peak in the frequency spectrum between 0.4 and 4 Hz (24-240 bpm) Video: 425 frames @ 25 fps, peak at index 18 Frequency: 18 425 beats frame 25 frames s 60 s min = 64 beats min Increase amplitude of peak and transform back Pulse signal has been amplified Pulse Harmonics Temporal domain Frequency domain Computer Vision 2 Dr. Benjamin Guthier 54 1. Image Processing