Computer Vision and Graphics (ee2031) Digital Image Processing I

Transcription

1 Computer Vision and Graphics (ee203) Digital Image Processing I Dr John Collomosse J.Collomosse@surrey.ac.uk Centre for Vision, Speech and Signal Processing University of Surrey

2 Learning Outcomes After attending this lecture, and doing the reading and labwork, you should be able to: Describe the basic framework for performing linear filtering on a digital image (convolution) Implement image blurring and sharpening operations. Compare and contrast several low-pass filters and describe their operation in the context of image processing. Define the Fourier transform in both continuous and discrete terms for the D and 2D cases. Describe the convolution theorem, its links to the Fourier transform, and its implications for digital image processing. Credit: Some images in these slides from Noah Snavly (Cornell). David Lowe (Columbia). Steve Seitz (Washington). Various creative commons sources.

3 Further reading:

4 What is an Image? An image is a rectangular grid (raster) of picture cels (= pixels) Greyscale (Y) = Typically byte (8 bits) per pixel. 0=black, 255=white.

5 Colour Images An image is a rectangular grid (raster) of picture cels (= pixels) Colour images use 3 rasters (Red, Green, Blue) Y= 0.30 (R) (G) + 0. (B) For this introduction we process a colour image simply by processing R, G and B rasters independently, as you would a greyscale image.

6 Image as a function We can think of a greyscale image as a function R 2 R f(x,y) = the intensity of pixel (x,y) = A digital image f(x,y) only has compact support

7 Image as a function When we do image processing we can think of transforming the function f(x,y) to form a new function g(x,y). g (x,y) = f (x,y) + 20 g (x,y) = f (-x,y) These lectures will focus on a class of transform called Linear transforms Because they: ) are useful (e.g. Noise reduction, finding edges, sharpen detail) 2) can be performed efficiently via convolution

8 Noise reduction Suppose we take a photo of a stationary scene. f(x,y) = I(x,y) + N(0,σ) image = signal + noise If noise obeys central limit theorem, then we can take many photos and average them to obtain a less noisy result.

9 Gaussian In that example we used a Gaussian distribution to model noise N(µ,σ) A Gaussian distribution is a generalised normal distribution with any mean (here µ=0) and standard deviation (here σ=0). N(0,) is the Normal distribution

10 Noise reduction What if we only have one photo (i.e. typical image processing)? A pixel is very similar to its neighbours (spatial coherence) Average groups of neighbouring pixels together Input f(x,y) Output g(x,y) The window centres on each pixel and computes mean value of pixels beneath it. The result is written to a new image.

11 Another way of saying the same... Consider a window containing a set of values. For each pixel in the input: /9 x. Window values are multiplied with image beneath 2. The sum of these products is written to output image Convolution Input f(x,y) Output g(x,y) e.g. (0 x ) + (4 x ) + (0 x ) +... = 4/9

12 Terminology Image f(x,y) was transformed into g(x,y) via convolution. Each pixel was replaced by a linear combination of its neighbours. This is called linear filtering. The weightings for each pixel were defined by the window Input f(x,y) * = Convolution operator, not multiplication! Output g(x,y) i.e. the prescription for a linear filter is the values in the window Window = Template = Kernel = Filter = Mask =...

13 Closer look at the box filter /9 /9 /9 /9 /9 /9 /9 /9 /9 Box filter / Box blur Mean filter Any filter is itself a signal h(x,y) Can be padded with zeros to match image size

14 Example of Box Filter Blocky / square artifacts More on this later...

15 Closer look at convolution process Convolu,on expressed using * operator /9 /9 /9 /9 /9 /9 /9 /9 /9 side 2k+ i.e. k=

16 Other choices of filter We can put different values in the window to create different effects (i.e. produce different linear filters): * = Original Identical image

17 Other choices of filter: We can put different values in the window to create different effects (i.e. produce different linear filters): * = Original Shifted left By pixel

18 Other choices of filter: We can put different values in the window to create different effects (i.e. produce different linear filters): * = Original Sharpening filter (accentuates edges)

19 Example of Image Sharpening

20 More on blurring We get better results blurring with a Gaussian filter vs. the box filter. Original x3 box filter x3 Gaussian Box Gaussian

21 2D Gaussian In this case x and y are measured as offsets from the centre of the template. The standard deviation is fixed. The window truncates the Gaussian function beyond a certain distance

22 Convolution - Topics Convolution is a versatile filtering mechanism, but:- ) As described, it is slow O(nm) and will take ages to process modern digital images e.g. multi-megapixel 2) We don t yet understand why particular sets of values in the filters have the result they do... To answer both we need to understand Fourier s theorem.

23 Fourier s Theorem Any periodic signal can be synthesised by summing (possibly infinitely) many sine and cosine waves of various amplitudes and frequencies (or equivalently: many cosine waves with various phase, ampltiude and frequency)

24 Fourier Synthesis Example Adding sine waves of increasing frequency (with decreasing amplitude) to make a square wave: y = sin(t) + sin(3*t)/3; y = sin(t) + sin(3*t)/3 + sin (5*t)/5 + sin(7*t)/7 + sin (9*t)/9;

25 Fourier Transform (Terminology) The Fourier Transform (FT) is a piece of mathematics that decomposes a real signal into its individual frequency components. FT ( Analysis) Spatial domain IFT ( Synthesis) Frequency domain You can convert a signal in the spatial domain to the frequency domain, and back again, with no loss of information.

26 Fourier Transform (Continuous) D Fourier transform: D Inverse Fourier Transform f(x) is the signal. F(u) is the response at frequency u. Complex numbers The response comprises a magnitude (r) and a phase (φ).

27 Fourier Transform (Continuous) D Fourier transform: D Inverse Fourier Transform f(x) is the signal. F(u) is the response at frequency u. Complex numbers Normalisation required because u is angular frequency (u=2πv)

28 Discrete Fourier Transform Because a digital signal has compact support, the DFT is used on digital signals. It is near-identical in form to continuous FT. D Discrete Fourier Transform: D Inverse DFT: The Fast Fourier Transform (FFT) is a fast way of computing DFT, that works only when N is a power of 2. (Cooley et al. 965) Demo

29 2D DFT FT and DFT also work over 2D (or any-d) signals. The 2D case is very important, because images are 2D signals - recall f(x,y) 2D DFT: 2D IDFT: Recall that converting to/from frequency domain is loss-less. 2D DFT / IDFT allows us to manipulate image in frequency domain. Image FT Frequency domain Manipulate frequencies IFT Result image

30 2D DFT Implementation The 2D DFT is a separable transform.... is computable by running D DFT over each image row, and then running each column of the result through its own D DFT. Separability makes 2D DFT fast to compute. If image has side length in powers of 2, can use FFT instead of DFT to speed up even further (equivalently you can pad the image with a border of zeros until it has sides of power 2).

31 2D DFT What does an image look like in frequency domain? Visualising F(u,v) (i.e. amplitude of frequencies) Higher frequencies f(x,y) Lower frequencies Origin (i.e. dc component) F(u,v) Demo

32 2D DFT Simpler examples f(x,y) F(u,v) What will F (u,v) look like?

33 2D DFT Simpler examples What will F (u,v) look like? Although the result is predominantly what you would expect, there are additional high frequencies introduced. This is because the signal isn t periodic (most images aren t)

34 2D DFT Image processing by manipulating frequency domain F(u,v):- Ideal Low-pass filter Ideal High-pass filter

35 Recall: Convolution Convolu,on expressed using * operator /9 /9 /9 /9 /9 /9 Slow O(nm) /9 /9 /9 side 2k+ i.e. k=

36 Convolution Theorem Convolution can be performed faster by converting both the image and filter into the frequency domain (2D DFT), multiplying them together, and converting the result back (2D IDFT). F[.] indicates FT of function. f(x,y) image FT IFT h(x,y) filter FT By considering convolution in this way, we can also understand why filters behave the way they do.

37 Recall: Why is Gaussian better? We get better results blurring with a Gaussian filter vs. the box filter. Original x3 box filter x3 Gaussian Box Gaussian

38 Fourier Analysis of Common Filters Visualisations of D box and Gaussian filters. FT FT

39 Fourier Analysis of Common Filters Visualisations of 2D box and Gaussian filters. Box Sinc FT Gaussian Gaussian FT

40 Observations The Ideal low pass filter is sinc : sinc (x) = sin(x)/x The FT of a box is a sinc scaled according to the size of the box. The FT of a Gaussian of σ is a Gaussian of /σ The opposite holds too (i.e. FT of a sinc is a box, etc.).

41 Recall: Box vs. Gaussian blur Can you explain the artifacts in the box filtered image? Original Box Gaussian Box Gaussian

42 Question How do we produce this ideal low-pass filtering scenario: f(x,y) image FT IFT h(x,y) filter? FT Answer: Use 2D sinc filter ( ideal low-pass filter ) But sinc is an infinite series and thus cannot be represented in digital images, because they have compact support

43 Ideal low-pass filter Sinc is an oscillating, infinite series and unsuitable for digital images, because they have compact support A truncated sinc signal in spatial filter creates artifacts in the frequency domain and thus ringing artifacts in the image. FT IFT FT

44 Gaussian low-pass filter A Gaussian does not have this problem. Although it is an infinite series it does not oscillate, and is well behaved (FT of a Gaussian σ is a Gaussian /σ). So, /σ determines the bandwidth of the frequencies passed. FT IFT FT

45 Back to sharpening: * = Original Sharpening filter (accentuates edges) unfiltered filtered

46 Back to sharpening What does the blurring take out? = original smoothed (5x5) detail Boost detail and add it back : + α = original detail sharpened Source: S. Lazebnik

47 Sharpening image blurred image scaled impulse Gaussian Laplacian of Gaussian

48 Fourier analysis - LoG FT Laplacian of Gaussian (LoG) Similar to Gaussiam, the FT of LoG is a LoG. What will this do to the high frequencies?

49 Example of Image Sharpening

50 Summary After attending this lecture, and doing the reading and labwork, you should be able to: Describe the basic framework for performing linear filtering on a digital image (convolution) Implement image blurring and sharpening operations. Compare and contrast several low-pass filters and describe their operation in the context of image processing. Define the Fourier transform in both continuous and discrete terms for the D and 2D cases. Describe the convolution theorem, its links to the Fourier transform, and its implications for digital image processing.