EEM 561 Machine Vision. Week 3: Fourier Transform and Image Pyramids

Transcription

1 EEM 561 Machine Vision Week 3: Fourier Transform and Image Pyramids Spring 2015 Instructor: Hatice Çınar Akakın, Ph.D. Anadolu University

2 Linear Image Transformations In analyzing images, it s often useful to make a change of basis. Transformed image Vectorized image Fourier transform, or Wavelet transform, or Steerable pyramid transform = Source: A.Torralba

3 Self-inverting transforms Same basis functions are used for the inverse transform U transpose and complex conjugate Source: A.Torralba

4 Background Fourier Series Any periodic fuction can be expressed as the sum of sines and /or cosines of different frequencies, each multiplied by different coefficients Fourier Transform Other functions that are not periodic (but whose area under curve is finite) can be expressed as the integral of sines and/or cosines multiplied by a weigthing function

5 Preliminary Concepts Remember that, In 1-D; The period of cos(t) is2π seconds, i.e. The signal repeats itself after2π seconds. The period of cos(2π t) is one second. Frequency is measured in Hz (Hertz) and is the number of periods (or cycles) per second, denoted f. That is, f = 1 T The function cos(2π f t) has a frequency of f and a period of 1/f Angular frequency is measured in rad/s, denoted by ω, so ω=2π/t.

6 Sum of cosines and sines to Express a periodic function

7 Frequency Spectra example : g(t) = sin(2πf t) + (1/3)sin(2π(3f) t) = + Slides: Efros

8 Fourier Transform Fourier transform stores the magnitude and phase at each frequency Magnitude encodes how much signal there is at a particular frequency Phase encodes spatial information (indirectly) For mathematical convenience, this is often notated in terms of real and complex numbers Amplitude: A R I 2 2 ( ) ( ) Phase: tan 1 I( ) R( )

9 Example:

10 Fourier transform visualization imaginary j 1 real color key F[m,n] = M -1N-1 åå k= 0 l = 0 Fourier transform matrix f [k,l]e -pi æ ç è km M + ln N ö ø input signal Source: A.Torralba

11 The Discrete Fourier Transform (DFT) of One Variable Suppose we want to obtain M equally spaced samples of F(μ) taken over the period μ=0 to μ=1/ T (only one needed). Take samples at μ = m M T M 1 F m = f n e j2πmn/m n 0 M 1 f n = 1 F M m e j2πmn/m m 0 m = 0, 1, 2,, M-1 DFT n = 0, 1, 2,, M-1 IDFT F u = M 1 n 0 f n e j2πux/m M 1 f x = 1 M m 0 F m e j2πux/m In image (spatial) processing, the more natural and widely used expression

12 The 2D impulse and Its Sifting Property (Continuous) Impulse of two continuous variables t and z : if t z 0 The impulse ( t, z), ( t, z) 0 otherwise and ( t, z) dtdz 1 The sifting property and f ( t, z) ( t, z) dtdz f (0,0) f ( t, z) ( t t, z z ) dtdz f ( t, z )

13 The 2D impulse and Its Sifting Property (Discrete) 1 if x y 0 The impulse ( x, y), ( x, y) 0 otherwise The sifting property and x y x y f ( x, y) ( x, y) f (0,0) f ( x, y) ( x x, y y ) f ( x, y )

14 The 2-D Discrete Fourier Transform and its Inverse DFT: M 1N 1 F(, ) f ( x, y) e x 0 y 0 j2 ( x/ M y/ N ) 0,1, 2,..., M 1; 0,1, 2,..., N 1; f ( x, y) is a digital image of size M N. IDFT: M 1N 1 1 f ( x, y) F(, ) e MN x 0 y 0 j 2 ( x/ M y/ N )

15 To get some sense of what basis elements look like, we plot a basis element --- or rather, its real part --- as a function of x,y for some fixed u, v. We get a function that is constant when (ux+vy) is constant. The magnitude of the vector (u, v) gives a frequency, and its direction gives an orientation. The function is a sinusoid with this frequency along the direction, and constant perpendicular to the direction. v e i e i ux vy ux vy u Source: A.Torralba

16 Here u and v are larger than in the previous slide. e v i ux vy e i u ux vy Source: A.Torralba

17 And larger still... e i ux vy v e u i ux vy Source: A.Torralba

18 Convolution The convolution of two functions f(t) and h(t) is defined as: Fourier transform pairs f ( t) h( t) H( ) F( ) f ( t) h( t) H( ) F( )

19 Bracewell s pictorial dictionary of Fourier transform pairs Bracewell, The Fourier Transform and its Applications, McGraw Hill 1978

20 Szeliski, Computer Vision, 2010

21 Magnitude FT Image Some important Fourier Transforms Source: A.Torralba

22 Magnitude FT Image Some important Fourier Transforms Source: A.Torralba

23

24 Fourier Amplitude Spectrum A B C fx(cycles/image pixel size) fx(cycles/image pixel size) fx(cycles/image pixel size) Source: A.Torralba

25 Phase and Magnitude Curious fact all natural images have about the same magnitude transform hence, phase seems to matter, but magnitude largely doesn t Demonstration Take two pictures, swap the phase transforms, compute the inverse - what does the result look like? Source: A.Torralba

26 Spectrum is insensitive to translation But rotates by the same angle as the rotated image

27 Computer Vision - A Modern Approach - Set: Pyramids and Texture - Slides by D.A. Forsyth

28 This is the magnitude transform of the cheetah pic Computer Vision - A Modern Approach - Set: Pyramids and Texture - Slides by D.A. Forsyth

29 This is the phase transform of the cheetah pic Computer Vision - A Modern Approach - Set: Pyramids and Texture - Slides by D.A. Forsyth

30 Computer Vision - A Modern Approach - Set: Pyramids and Texture - Slides by D.A. Forsyth

31 This is the magnitude transform of the zebra pic Computer Vision - A Modern Approach - Set: Pyramids and Texture - Slides by D.A. Forsyth

32 Reconstruction with zebra phase, cheetah magnitude Computer Vision - A Modern Approach - Set: Pyramids and Texture - Slides by D.A. Forsyth

33 Reconstruction with cheetah phase, zebra magnitude Computer Vision - A Modern Approach - Set: Pyramids and Texture - Slides by D.A. Forsyth

34 Summary of Steps for Filtering in the Frequency Domain 1. Given an input image f(x,y) of size MxN, obtain the padding parameters P and Q. Typically, P = 2M and Q = 2N. 2. Form a padded image, f p (x,y) of size PxQ by appending the necessary number of zeros to f(x,y). 3. Multiply f p (x,y) by (-1) x+y to center its transform. 4. Compute the DFT, F(u,v) of the image from step Generate a real, symmetric filter function, H(u,v), of size PxQ with center at coordinates (P/2, Q/2). Form the product G(u,v) = H(u,v)F(u,v) using array multiplication Obtain the processed image: (, ) (, ) ( 1) x g y p x y real G u v 7. Obtain the final processed result, g(x,y), by extracting the MxN region from the top, left quadrant of g p (x,y)

35 Filtering in spatial domain * =

36 Filtering in frequency domain FFT FFT = Inverse FFT Slide: Hoiem

37 FFT in Matlab Filtering with fft im = double(imread( '))/255; im = rgb2gray(im); % im should be a gray-scale floating point image [imh, imw] = size(im); hs = 50; % filter half-size fil = fspecial('gaussian', hs*2+1, 10); fftsize = 1024; % should be order of 2 (for speed) and include padding im_fft = fft2(im, fftsize, fftsize); % 1) fft im with padding fil_fft = fft2(fil, fftsize, fftsize); % 2) fft fil, pad to same size as image im_fil_fft = im_fft.* fil_fft; % 3) multiply fft images im_fil = ifft2(im_fil_fft); % 4) inverse fft2 im_fil = im_fil(1+hs:size(im,1)+hs, 1+hs:size(im, 2)+hs); % 5) remove padding Displaying with fft figure(1), imagesc(log(abs(fftshift(im_fft)))), axis image, colormap jet Slide: Hoiem

38 Image Sharpening Using Frequency Domain Filters Edges and sharp transitions in gray-values in an image contribute significantly to high-frequency content of its Fourier transform. Regions of relatively uniform gray-values in an image contribute to low-frequency content of its Fourier transform. Hence, image sharpening in the Frequency domain can be done by attenuating the low-frequency content of its Fourier transform. This would be a highpass filter!

39 Contrast Sensitivity Function From:

40 Contrast Sensitivity Function A demo of human contrast sensitivity as a function of spatial frequency. Frequency rises from left to right at a constant rate. Contrast drops from bottom to top at a constant rate. The bars are visible further up for middle frequencies, showing these are more salient to the human visual system. 40

41 Contrast sensitivity Contrast Sensitivity Function Blackmore & Campbell (1969) Maximum sensitivity ~ 6 cycles / degree of visual angle Invisible visible Spatial frequency (cycles/degree) High Low

42 Let Hu ( ) denote the difference of Gaussian filter /2 1 u /2 2 H ( u) Ae Be -u - with A B and 1 2 The corresponding filter in the spatial domain x 2 x h( x) 2 Ae 2 Ae

43 Image Smoothing using Frequency Domain Filters Ideal Lowpass filter (ILPF) Mathematically Cutoff Frequency Where D(u,v) is the distance from (u,v) to origin. an ILPF cannot be realized with real electronic components

44 Gaussian Lowpass Filters Gaussian Lowpass Filters (GLPF) in two dimensions is given By letting D 0 = cutoff frequency) H ( u, v) e D 0 H ( u, v) e D 2 2 (, )/2 0 D u v D 2 2 ( u, v)/2 σ = measure of spread about the centre

45 A highpass (HP) filter is obtained from a given lowpass (LP) filter using H ( u, v) 1 H ( u, v) HP An 2D ideal highpass filter (IHPF) with cutoff frequency D 0 : H u, v = 0 if D(u, v) D 0 1 if D u, v > D 0 LP

46

47 Ideal Butterworth Gaussian Spatial representations of HP filters

48 Sampling Pixels Continuous world Source: A.Torralba

49 Sampling Source: A.Torralba

50 Subsampling by a factor of 2 Throw away every other row and column to create a 1/2 size image Source: J. Hays

51 Aliasing problem Sub-sampling may be dangerous. Characteristic errors may appear: Wagon wheels rolling the wrong way in movies Checkerboards disintegrate in ray tracing Striped shirts look funny on color television Source: D. Forsyth

52

53

54 Nyquist-Shannon Sampling Theorem When sampling a signal at discrete intervals, the sampling frequency must be 2 f max f max = max frequency of the input signal This will allows to reconstruct the original perfectly from the sampled version v v v good bad Source: J. Hays

55 Anti-aliasing Solutions: Sample more often Get rid of all frequencies that are greater than half the new sampling frequency Will lose information But it s better than aliasing Apply a smoothing filter Source: J. Hays

56 Algorithm for downsampling by factor of 2 1. Start with image(h, w) 2. Apply low-pass filter im_blur = imfilter(image, fspecial( gaussian, 7, 1)) 3. Sample every other pixel im_small = im_blur(1:2:end, 1:2:end);

57 Subsampling without pre-filtering 1/2 1/4 (2x zoom) 1/8 (4x zoom) Slide by Steve Seitz

58 Subsampling with Gaussian pre-filtering Gaussian 1/2 G 1/4 G 1/8 Slide by Steve Seitz

59 Pyramids and Scale Space Basic idea: different scales are appropriate for describing different objects in the image, and we may not know the correct scale/size ahead of time.

60 Gaussian pre-filtering Solution: filter the image, then subsample blur F 0 subsample blur subsample * F 0 H F 1 F 1 * H F 2 Source:N. Snavely

61 Gaussian pyramid blur F 0 subsample blur subsample * F 0 H F 1 F 1 * H F 2 Source:N. Snavely

62 Gaussian pyramids [Burt and Adelson, 1983] In computer graphics, a mip map [Williams, 1983] A precursor to wavelet transform Gaussian Pyramids have all sorts of applications in computer vision Source: S. Seitz

63 Image representation Pixels: great for spatial resolution, poor access to frequency Fourier transform: great for frequency, not for spatial info Pyramids/filter banks: balance between spatial and frequency information Source: J. Hays

64 Major uses of image pyramids Compression Object detection Scale search Features Detecting stable interest points Registration Course-to-fine Source: J. Hays

65 The Laplacian Pyramid Synthesis Compute the difference between upsampled Gaussian pyramid level and Gaussian pyramid level. band pass filter - each level represents spatial frequencies (largely) unrepresented at other level.

66 Laplacian pyramid Source: Forsyth

67 gg Source: A. Torralba

68

69

70 Application: Representing Texture Source: Forsyth

71 Texture and Material

72 Texture and Orientation

73 Texture and Scale

74 How can we represent texture? Compute responses of blobs and edges at various orientations and scales

75 Overcomplete representation: filter banks LM Filter Bank Gaussian first and second derivative filters at three scales and 6 orientations 8 LoG filters at different scales, and 4 Gaussians at different scales Code for filter banks:

76

77

78

79

80 Upsampling This image is too small for this screen: How can we make it 10 times as big? Simplest approach: repeat each row and column 10 times ( Nearest neighbor interpolation ) Source:N. Snavely

81 Image interpolation d = 1 in this example Recall how a digital image is formed It is a discrete point-sampling of a continuous function If we could somehow reconstruct the original function, any new image could be generated, at any resolution and scale Adapted from: S. Seitz

82 Image interpolation d = 1 in this example Recall how a digital image is formed It is a discrete point-sampling of a continuous function If we could somehow reconstruct the original function, any new image could be generated, at any resolution and scale Adapted from: S. Seitz

83 Image interpolation 1 d = 1 in this example What if we don t know? Guess an approximation: Can be done in a principled way: filtering Convert to a continuous function: Reconstruct by convolution with a reconstruction filter, h Adapted from: S. Seitz

84 Image interpolation Ideal reconstruction Nearest-neighbor interpolation Linear interpolation Gaussian reconstruction Source: B. Curless

85 Image interpolation Original image: x 10 Nearest-neighbor interpolation Bilinear interpolation Bicubic interpolation Source:N. Snavely