Multiresolution Image Processing 2 Processing and Analysis of Images at Multiple Scales What is Multiscale Decompostion? Why use Multiscale Processing? How to use Multiscale Processing? Related Concepts: Subbands, Wavelets
3 Motivation Images can not be adequately modeled with a single description, at a single scale. 4 http://www-bcs.mit.edu/people/adelson/pub_pdfs/pyramid83.pdf 2
5 6 Multiscale Decomposition: Pyramids We can model images as being composed from a combination of simpler images at increasing scales. Gaussian Pyramid: Laplacian Pyramid: Good for compression 3
7 The Laplacian Pyramid Synthesis preserve difference between upsampled Gaussian pyramid level and Gaussian pyramid level band pass filter - each level represents spatial frequencies largely unrepresented at other levels Analysis reconstruct Gaussian pyramid, take fine-scale layer 8 http://www-bcs.mit.edu/people/adelson/pub_pdfs/pyramid83.pdf 4
9 Pyramids Construction: An Overcomplete Redundant Representation Original Invention by: Burt, Adelson, 983 5
Image Processing with Pyramids: 2 Image Blending 6
3 Feathering + Encoding transparency = Ix,y = ar, ag, ab, a I blend = I left + I right 4 Effect of Window Sie left right 7
5 Effect of Window Sie 6 Good Window Sie Optimal Window: smooth but not ghosted 8
7 What is the Optimal Window? To avoid seams window >= sie of largest prominent feature To avoid ghosting window <= 2*sie of smallest prominent feature Natural to cast this in the Fourier domain largest frequency <= 2*sie of smallest frequency image frequency content should occupy one octave power of two FFT 8 Pyramid Blending Left pyramid blend Right pyramid 9
9 Pyramid Blending 2 laplacian level 4 laplacian level 2 laplacian level left pyramid right pyramid blended pyramid
2 Simplification: Two-band Blending Brown & Lowe, 23 Only use two bands: high freq. and low freq. Blends low freq. smoothly Blend high freq. with no smoothing: use binary mask 22 2-band Blending Low frequency l > 2 pixels High frequency l < 2 pixels
23 Linear Blending 24 2-band Blending 2
25 Direct Merge: Multiscale Merge: 26 Very early computational approach to creating large depth-of-field http://web.mit.edu/persci/people/adelson/pub_pdfs/rca84.pdf 3
27 Image Analysis with Pyramids: Detection Recognition Segmentation Etc. 28 Related Notion: Subband Coding Decomposition of a Signal/Image into a set of complementary bandlimited components Analysis: Filter + Downsample Synthesis: Upsample + Filter 4
5 29 Subband Coding: Perfect Reconstruction What relationship between the filters guarantees perfect reconstruction? Key Tool: The -transform n n n x X Key Idea: Aliasing Cancellation 3 Perfect reconstruction conditions: Key relationships: Downsampling: Upsampling: 2 n x n x d otherwise for even 2 / n n x n x u 2 2 / 2 / X X X d 2 X X u 2 2 ˆ X X H X H G X H X H G X Perfect Reconstruction: 2 G H G H G H G H
3 Example: 2-Channel with Perfect Reconstruciton 32 Family of Solutions: The whole approach can be extended to M subbands The same arguments can be applied in a separable fashion to image decomposition along rows and columns. 6
33 Now, in 2 dimensions Horiontal high pass Frequency domain Horiontal low pass 34 Apply the wavelet transform separable in both dimensions Horiontal high pass, vertical high pass Horiontal high pass, vertical low-pass Horiontal low pass, vertical high-pass Horiontal low pass, Vertical low-pass 7
35 Simoncelli and Adelson, in Subband coding, Kluwer, 99. To create 2-d filters, apply the -d filters separably in the two spatial dimensions 36 Wavelet/QMF representation 8
37 Good and bad features of wavelet/qmf filters Bad: Aliased subbands Non-oriented diagonal subband Good: Not overcomplete so same number of coefficients as image pixels. Good for image compression JPEG 2 38 Example: 4 subband image decomposition 9
39 Steerable filters Analye image with oriented filters Avoid preferred orientation Said differently: We want to be able to compute the response to an arbitrary orientation from the response to a few basis filters By linear combination Notion of steerability 4 Reprinted from Shiftable MultiScale Transforms, by Simoncelli et al., IEEE Transactions on Information Theory, 992, copyright 992, IEEE 2
4 42 Fourier construction Slice Fourier domain Concentric rings for different scales Slices for orientation Feather cutoff to make steerable Tradeoff steerable/orthogonal 2
43 But we need to get rid of the corner regions before starting the recursive circular filtering http://www.cns.nyu.edu/ftp/eero/simoncelli95b.pdf Simoncelli and Freeman, ICIP 995 44 Non-oriented steerable pyramid http://www.merl.com/reports/docs/tr95-5.pdf 22
45 3-orientation steerable pyramid http://www.merl.com/reports/docs/tr95-5.pdf 46 Steerable pyramids Good: Oriented subbands Non-aliased subbands Steerable filters Bad: Overcomplete Have one high frequency residual subband, required in order to form a circular region of analysis in frequency from a square region of support in frequency. 23
47 Gaussian Image pyramids Progressively blurred and subsampled versions of the image. Adds scale invariance to fixed-sie algorithms. Laplacian Shows the information added in Gaussian pyramid at each spatial scale. Useful for noise reduction & coding. Wavelet/QMF Steerable pyramid Bandpassed representation, complete, but with aliasing and some non-oriented subbands. Shows components at each scale and orientation separately. Non-aliased subbands. Good for texture and feature analysis. 48 Related Notion: Wavelet Transform Simplest case: Discrete Haar Wavelet Transform in -D y y2 y 3 y4 4 2 2 2 x x2 x 3 2x4 Transform of signal H 4 Given signal y Hx 24
49 Related Notion: Wavelet Transform Important points: Note the action of each row of H y gives information about the signal at different scales of resolution Rows of H are the coefficients of the corresponding QMF system Orthogonal Transformation Basis vectors are finite support H HH Can be applied in separable way in 2-D Non-redundant square transformation H 4 4 2 H T 2 2 2 T I 5 Discrete Haar Wavelet Example 25
5 Continuous Wavelet Series Expansion f Arbitrary starting coarse scale x c j k j k x d j k, j, k k j j k x Scaling functions Scaling coeffs. Detail coeffs. Wavelet functions Coarse Scale Approximation Fine-scale details 52 The Scaling Functions x V j, k j j 2 j x 2 2 x k h n 2 2 n Span x k j, k x n Haar Example: 26
53 The Wavelet Functions x W j, k j j 2 j x 2 2 x k h n 2 n Span x k j, k 2x n Haar Example: 54 Their Relationships Haar Example: 27
55 The Discrete Case: The Fast WT If fx is composed of discrete samples k is discrete, the transform is similar. f x W j, k j, k x W j, k j, k k j j k And the coefficients can be obtained as: Finer scale coefficients x Coarser scale detail coeffs. HPF Coarser scale approximation coeffs. LPF 56 The Discrete Case: The Fast WT Resemblance to the QMF setup is not coincidental! 28
57 2-D Wavelet Analysis Scaling Functions: x, y x y Wavelet Functions: Horiontal Vertical Diagonal x, y x y H x, y y x V x, y x y D 58 2-D Wavelet Analysis: Example 29
59 Multiscale Motion Estimation Construct a Gaussian Pyramid and estimate motion from coarse-to-fine levels. Compute motion estimates at each scale. Coarse Significantly better than nonmultiscale Fine 6 Multiscale Methods Details. Estimate motion at coarsest scale 2. Undo motion in the sequence at the next level. 3. Estimate residual motion 4. Update motion estimate. 5. Repeat in a coarse-tofine fashion. Offers much better performance than non-multiscale 3