Thinking in Frequency

Transcription

1 Thinking in Frequency Computer Vision Jia-Bin Huang, Virginia Tech Dali: Gala Contemplating the Mediterranean Sea (1976)

2 Administrative stuffs Course website: Office hours - Jia-Bin (440 Whittemore Hall ) Friday at 11:00 AM 12:00 PM; 1:30 PM 2:00 PM Office hours Yuliang (344 Whittemore) Wednesday 2:30 PM - 3:30 PM Monday (on HW due date) HW 1 will be posted next Tue (Sept 12). Due date: Sept 25.

3 Using ARC Cluster: NewRiver g/newriver/ Using MATLAB on NewRiver /matlab/

4 Previous class: Image Filtering Linear filtering is sum of dot product at each position Can smooth, sharpen, translate (among many other uses) Gaussian filters Low pass filters, separability, variance Attend to details: filter size, extrapolation, cropping Applications: Noise models and nonlinear image filters Texture representation

5 Today s class Review of image filtering in spatial domain Fourier transform and frequency domain Frequency view of filtering Image downsizing and interpolation Goals: Understand 2D Fourier transform Understand how to implement filtering in Fourier domain Understand aliasing and how to prevent aliasing

6 Demo

7 Questions Write as filtering operations, plus some pointwise operations: +, -,.*,> out( m, n) 1 out( m, n) k, l { 1,1} k, l { 1,0,1} out( m, n) 0 if in( m, n) in( m k, n l) out = 1 + in*h1; in( m k, n l) 2 out = (in.^2)*h2; out( m, n) 1 if in( m, n) ( in( m, n 1) in( m, n 1)) / 2 ( in( m, n 1) in( m, n 1)) / out = in > in*h3;

8 Review: questions Fill in the blanks: a) _ = D * B b) A = _ * _ c) F = D * _ d) _ = D * D Filtering Operator A E B F G C H I D Slide: Hoiem

9 Hybrid Images Slide credit: Derek Hoiem A. Oliva, A. Torralba, P.G. Schyns, Hybrid Images, SIGGRAPH 2006

10 Why do we get different, distance-dependent interpretations of hybrid images?? Slide credit: Derek Hoiem

11 Why does a lower resolution image still make sense to us? What do we lose? Slide credit: Derek Hoiem Image:

12 Thinking in terms of frequency

13 Jean Baptiste Joseph Fourier ( ) had crazy idea (1807): Any univariate function can be rewritten as a weighted sum of sines and cosines of different frequencies. Don t believe it? Neither did Lagrange, Laplace, Poisson and other big wigs Not translated into English until 1878! But it s (mostly) true! called Fourier Series there are some subtle restrictions...the manner in which the author arrives at these equations is not exempt of difficulties and...his analysis to integrate them still leaves something to be desired on the score of generality and even rigour. Laplace Lagrange Legendre Slides: Efros

14 How would math have changed if the Slanket or Snuggie had been invented? Slide credit: James Hays

15 A sum of sines Our building block: Asin( x Add enough of them to get any signal f(x) you want!

16 Frequency Spectra example : g(t) = sin(2πf t) + (1/3)sin(2π(3f) t) = + Slides: Efros

17 Frequency Spectra

18 Frequency Spectra = + =

19 Frequency Spectra = + =

20 Frequency Spectra = + =

21 Frequency Spectra = + =

22 Frequency Spectra = + =

23 Frequency Spectra = A k 1 1 sin(2 kt ) k

24 Example: Music We think of music in terms of frequencies at different magnitudes

25 Other signals We can also think of all kinds of other signals the same way Cats(?) xkcd.com

26 Fourier analysis in images Intensity Image Fourier Image

27 Signals can be composed + = More:

28 Fourier Transform Teases away fast vs. slow changes in the image. Slide credit: A Efros Image as a sum of basis images

29 Extension to 2D in Matlab, check out: imagesc(log(abs(fftshift(fft2(im)))));

30 Fourier Transform Fourier transform stores the magnitude and phase at each frequency Magnitude encodes how much signal there is at a particular frequency Phase encodes spatial information (indirectly) For mathematical convenience, this is often notated in terms of real and complex numbers Amplitude: A R I 2 2 ( ) ( ) Phase: tan 1 I( ) R( ) Euler s formula:

31 Salvador Dali invented Hybrid Images? Salvador Dali Gala Contemplating the Mediterranean Sea, which at 30 meters becomes the portrait of Abraham Lincoln, 1976

32

33

34 Strong Vertical Frequency (Sharp Horizontal Edge) Diagonal Frequencies Strong Horz. Frequency (Sharp Vert. Edge) Log Magnitude Low Frequencies

35 Man-made Scene

36 Can change spectrum, then reconstruct

37 Low and High Pass filtering

38 Computing the Fourier Transform Continuous Discrete k = -N/2..N/2 Fast Fourier Transform (FFT): NlogN

39 The Convolution Theorem The Fourier transform of the convolution of two functions is the product of their Fourier transforms F[ g h] F[ g]f[ h] The inverse Fourier transform of the product of two Fourier transforms is the convolution of the two inverse Fourier transforms F 1 [ gh] F 1 [ g] [ h] Convolution in spatial domain is equivalent to multiplication in frequency domain! F 1

40 Properties of Fourier Transforms Linearity Fourier transform of a real signal is symmetric about the origin The energy of the signal is the same as the energy of its Fourier transform See Szeliski Book (3.4)

41 Filtering in spatial domain * =

42 Spatial domain * = FFT FFT Inverse FFT = Frequency domain

43 FFT in Matlab Filtering with fft im = % im: gray-scale floating point image [imh, imw] = size(im); fftsize = 1024; % fftsize: should be order of 2 (for speed) and include padding hs = 50; % fil: Gaussian filter fil = fspecial('gaussian', hs*2+1, 10); % im_fft = fft2(im, fftsize, fftsize); % 1) fft im with padding fil_fft = fft2(fil, fftsize, fftsize); % 2) fft fil, pad to same size as image im_fil_fft = im_fft.* fil_fft; % 3) multiply fft images im_fil = ifft2(im_fil_fft); % 4) inverse fft2 im_fil = im_fil(1+hs:size(im,1)+hs, 1+hs:size(im, 2)+hs); % 5) remove padding Displaying with fft figure(1), imagesc(log(abs(fftshift(im_fft)))), axis image, colormap jet

44 Questions Which has more information, the phase or the magnitude? What happens if you take the phase from one image and combine it with the magnitude from another image?

45 Phase vs. Magnitude Intensity image FFT Use random magnitude Inverse FFT Use random phase Inverse FFT Magnitude Phase

46 Filtering Why does the Gaussian give a nice smooth image, but the square filter give edgy artifacts? Gaussian Box filter

47 Gaussian

48 Box Filter

49 Question Match the spatial domain image to the Fourier magnitude image B A C D E

50 Image half-sizing This image is too big to fit on the screen. How can we reduce it? How to generate a halfsized version?

51 Image sub-sampling 1/8 1/4 Throw away every other row and column to create a 1/2 size image - called image sub-sampling Slide by Steve Seitz

52 Image sub-sampling 1/2 Why does this look so crufty? Aliasing! What do we do? 1/4 (2x zoom) 1/8 (4x zoom) Slide by Steve Seitz

53 Image sub-sampling Source: F. Durand

54 Even worse for synthetic images Source: L. Zhang

55 Aliasing problem 1D example (sinewave): Source: S. Marschner

56 Aliasing problem 1D example (sinewave): Source: S. Marschner

57 Aliasing problem Sub-sampling may be dangerous. Characteristic errors may appear: Wagon wheels rolling the wrong way in movies Checkerboards disintegrate in ray tracing Striped shirts look funny on color television Source: D. Forsyth

58 Aliasing Occurs when your sampling rate is not high enough to capture the amount of detail in your image Can give you the wrong signal/image an alias To do sampling right, need to understand the structure of your signal/image To avoid aliasing: sampling rate 2 * max frequency in the image said another way: two samples per cycle This minimum sampling rate is called the Nyquist rate Source: L. Zhang

59 Wagon-wheel effect (See Source: L. Zhang

60 Wagon-wheel effect

61 Sampling an image Examples of GOOD sampling

62 Undersampling Examples of BAD sampling -> Aliasing

63 Anti-aliasing Forsyth and Ponce 2002

64 Gaussian (low-pass) pre-filtering G 1/4 G 1/8 Gaussian 1/2 Solution: filter the image, then subsample Source: S. Seitz

65 Subsampling with Gaussian pre-filtering Gaussian 1/2 G 1/4 G 1/8 Solution: filter the image, then subsample Source: S. Seitz

66 Compare with... 1/2 1/4 (2x zoom) 1/8 (4x zoom) Source: S. Seitz

67 Why does a lower resolution image still make sense to us? What do we lose? Image:

68 Why do we get different, distance-dependent interpretations of hybrid images??

69 Clues from Human Perception Early processing in humans filters for various orientations and scales of frequency Perceptual cues in the mid-high frequencies dominate perception When we see an image from far away, we are effectively subsampling it Early Visual Processing: Multi-scale edge and blob filters

70 Hybrid Image in FFT Hybrid Image Low-passed Image High-passed Image

71 Upsampling This image is too small for this screen: How can we make it 10 times as big? Simplest approach: repeat each row and column 10 times ( Nearest neighbor interpolation )

72 Image interpolation d = 1 in this example Recall how a digital image is formed It is a discrete point-sampling of a continuous function If we could somehow reconstruct the original function, any new image could be generated, at any resolution and scale Adapted from: S. Seitz

73 Image interpolation d = 1 in this example Recall how a digital image is formed It is a discrete point-sampling of a continuous function If we could somehow reconstruct the original function, any new image could be generated, at any resolution and scale Adapted from: S. Seitz

74 Image interpolation 1 d = 1 in this example What if we don t know? Guess an approximation: Can be done in a principled way: filtering Convert to a continuous function: Reconstruct by convolution with a reconstruction filter, h Adapted from: S. Seitz

75 Image interpolation Ideal reconstruction Nearest-neighbor interpolation Linear interpolation Gaussian reconstruction Source: B. Curless

76 Reconstruction filters What does the 2D version of this hat function look like? performs linear interpolation (tent function) performs bilinear interpolation Often implemented without cross-correlation E.g., Better filters give better resampled images Bicubic is common choice Cubic reconstruction filter

77 Image interpolation Original image: x 10 Nearest-neighbor interpolation Bilinear interpolation Bicubic interpolation

78 Image interpolation Also used for resampling

79 Things to Remember Sometimes it makes sense to think of images and filtering in the frequency domain Fourier analysis Can be faster to filter using FFT for large images (N logn vs. N 2 for auto-correlation) Images are mostly smooth Basis for compression Remember to low-pass before sampling

80 Thank you Next class: Pyramid, template matching