Frequency analysis, pyramids, texture analysis, applications (face detection, category recognition)

Transcription

1 Frequency analysis, pyramids, texture analysis, applications (face detection, category recognition)

2 Outline Measuring frequencies in images: Definitions, properties Sampling issues Relation with Gaussian smoothing Image pyramids Gabor filters and wavelets Examples: Texture classification and category recognition

3 A representation for image changes We need a change of basis to move from pixel intensities (space domain) to frequency domain. dot = A measure of image content at this frequency and orientation

4 Fourier basis etc.

5 More formally, content at frequency u is obtained by taking the magnitude of the dot product with the function cos u Same with sin u

6 Combining cos and sin: Fourier Transform 1-D case: F ( u) f ( x) e i ux dx f ( x) F ( u) e i ux du -D case: F ( u, v) f ( x, y) e i ( ux vy) dxdy f ( x, y) F( u, v) e i ( ux vy) dudv

7 Signal domain Examples (1D) 0 u Frequency domain u 0

8 Example from Ponce&Forsyth Magnitude spectrum

9 Example from Ponce&Forsyth Magnitude spectrum

10 Questions How do discrete images differ from continuous images? How do we avoid aliasing while sampling?

11 Sampling an image Examples of GOOD sampling

12 Undersampling Examples of BAD sampling -> Aliasing

13 Constructing a pyramid by taking every second pixel leads to layers that badly misrepresent the top layer

14 Low-pass filtering before sampling The minimum frequency at which we must sample a signal in order to be able to fully reconstruct it called the Nyquist frequency. Nyquist frequency = times the maximum frequency contained in the waveform. The message of the FT is that high frequencies lead to trouble with sampling. Solution: suppress high frequencies before sampling multiply the FT of the signal with something that suppresses high frequencies or convolve with a low-pass filter Common solution: use a Gaussian multiplying FT by Gaussian is equivalent to convolving image with Gaussian.

15 Questions How can we represent images at multiple scales?

16 Sampling without smoothing. Top row shows the images, sampled at every second pixel to get the next; bottom row shows the magnitude spectrum of these images.

17 Sampling with smoothing. Top row shows the images. We get the next image by smoothing the image with a Gaussian with sigma 1 pixel, then sampling at every second pixel to get the next; bottom row shows the magnitude spectrum of these images.

18 Representation of scale

19 Representation of scale The Gaussian pyramid Smooth with gaussians, because a gaussian*gaussian=another gaussian Synthesis smooth and subsample Analysis Start with the top image (coarse) and move to lower (fine) image layers Applications Search for correspondence look at coarse scales, then refine with finer scales Lowers computational cost Edge tracking a good edge at a fine scale has parents at a coarser scale

20

21 Given input I Laplacian Pyramids Construct Gaussian pyramid I G 1,..,I G n Take the difference between consecutive levels: I L k = I G k I G k-1 Image I L k is an approximation of the Laplacian at scale number k Laplacian is a band-pass filter: Both high frequencies (edges and noise) and low frequencies (slow variations of intensity across the image)

22 Laplacian Pyramids

23 Frequency/Scale Tradeoff: Gabor filters and wavelets

24 Power spectrum from Fourier transform Local spectrum (after smoothing) Local spectrum (after smoothing)

25 Low frequency content in all directions High frequency at 45 o orientation Fourier Transform = content of entire image at all frequencies and orientations

26 Frequency content in local neighborhood at every point in image I = I * STFT Fourier Basis = Box filter Short Time Fourier Transform (STFT)

27 Frequency content in local neighborhood at every point in image I = I * Gabor Fourier Basis = Gaussian filter Gabor Filter

28 Gabor Filter Example Odd (antisymmetic) Even (Symmetric)

29 High frequency along axis Lower frequency along diagonal Even lower frequency

30

31 Odd Gabor filter First Derivative

32 Even Gabor filter Laplacian

33 If scale small compared to inverse frequency, the Gabor filters become derivative operators = f = 1/6 G x G xx

34 Can we select the frequency and the scale arbitrarily?

35 Low frequency Frequency/scale mismatch: 1/frequency >> scale Small scale Gabor filter

36 High frequency Frequency/scale mismatch: 1/frequency << scale Large scale Gabor Gabor filter filter

37 Wavelets: Scale and frequency are made consistent by scaling one basis filter. The scale of the filter is changed and, at the same time, the frequency is also changed. Arbitrary Scale+frequency Consistently scaled Scale+frequency

38 Example

39 Example: Face Detection Input Wavelet decomposition From Henry Schneiderman

40 Example: Texture Classification Profound observation: Cows and buildings don t look the same! Basic idea: Model the distribution of texture over the image (or over a region) and classify in different classes based on the texture models learned from training examples. Cow Building

41 The Concept of Texton Clustering Texton Dictionary Multiple training images of the same texture Filter responses over a bank of filters

42 Example of Filter Banks Isotropic Gabor S Gaussian derivatives at different scales and orientations LM MR8

43 Example Textons (LM)

44 Example: Visual words in photographs Images Word maps Visual dictionary

45 Modeling Texton Distributions Training image Filter Responses Texton Map Model = Histogram of textons in the image

46 Bus Election Government Gigabyte Gigahertz Memory Observers Political Frequency of occurrence Analogy with Text Analysis Political observers say that the government of Zorgia does not control the political situation. The government will not hold elections Analogy: Text fragment Image region Word Texton Word from vocabulary

47 Bus Election overnment Gigabyte Gigahertz Memory Observers Political Bus Election Government Gigabyte Gigahertz Memory Observers Political Frequency of occurrence Frequency of occurrence Bus Election Government Gigabyte Gigahertz Memory Observers Political Frequency of occurrence Analogy with Text Analysis The ZH-0 unit is a 00Gigahertz processor with Gigabyte memory. Its strength is its bus and high-speed memory Histogram from training political fragments Histogram from input fragment Compare Histogram from training computer fragments Word from vocabulary

48 Classification Compare with Stored Models from Training Images Models of Plastic Input Image (or Region of an Input Image) Model Models of Grass

49 Example Classification Input Region Textons

50 Examples

51 True label Example Performance (Confusion Matrix) Inferred label Grass Tree Cow Sky Building Aeroplane Face Building 1 1 Grass 6 3 Tree 8 Cow 1 Sky 44 Aeroplane 1 14 Face 15

52 Sources: J. Winn, A. Criminisi and T. Minka. Object Categorization by Learned Universal Visual Dictionary. Proc. IEEE Intern. Conf. Comp. Vision. 005 M. Varma and A. Zisserman. A statistical approach to texture classification from single images. IJCV, 6(1 ):61 81, April 005. Current research topics: How many textons/words? What filters? How to construct clusters? How to compare histogram distributions? How to exploit the spatial distribution of textons (these examples completely ignore the relative positions of textons in the image)?

53 G ( x, y) 1 e x y Gaussian Derivatives of Gaussian Separable, low-pass filter G G G F( g)( u, v) F( f cos ( x, y) x ( x, y) * g) G x G x xe x G y G y ( x, y) x g( x, y) e F( f ). F( g) sin t G y G F( ( x, y) y i( ux g G vy) dudv ) x ye ( x, y) y x y uf( g) Directional Derivatives Laplacian Fourier Transform Separable, output of convolution is gradient at scale : I I * G Not-separable, approximated by A difference of Gaussians. Output of convolution is Laplacian of image: Zero-crossings correspond to edges F(g)(u,v) = component of image at frequency sqrt(u +v ) in direction (u,v) Transform of Gaussian is Gaussian 1/ Output of convolution is magnitude of derivative in direction Filter is linear combination of derivatives in x and y

54 e a 1 x b 1 1 i y a i a x b i y Directional smoothing Steerability Gabor Filters Smooth with different scales in orthogonal directions Generalization of property of derivatives: is steerable if the rotated filter can be expressed as a linear combination of basis filters. even : cos( odd : sin ( I k L k s k 1 t ( t) I S k 1 S s k G ( k ( k ( x * I G t s k x x ) x k ( t) * I k k 1 k y y y) e y) e x x y y Wavelets Gaussian Pyramids Compute the local contribution of frequency f k x k y in the direction (k x,k y ) at scale. Even filters approximate nd derivative, odd filters approximate 1 st derivative. Decompose the image locally using Replication of a basis function over scale Gaussian smooth image and subsample at each stage Laplacian Pyramids Compute Laplacian by difference of Gaussian at each stage