Beyond Wavelets: Multiscale Geometric Representations

Beyond Wavelets: Multiscale Geometric Representations Minh N. Do Department of Electrical and Computer Engineering University of Illinois at Urbana-Champaign www.ifp.uiuc.edu/ minhdo minhdo@uiuc.edu

Acknowledgments Collaborators: Prof. Martin Vetterli (EPFL and UC Berkeley) Dr. Arthur L. A. da Cunha (JP Morgan) Dr. Yue Lu (EPFL) Mr. Duncan Po (MathWorks) Dr. Jianping Zhou (Intel) Funding source: National Science Foundation CAREER Award 1

What Do Image Processors Do for Living? Compression: At 158:1 compression ratio... 1. Motivation 2

What Do Image Processors Do for Living? Denoising (restoration/filtering) Noisy image Clean image 1. Motivation 3

What Do Image Processors Do for Living? Feature extraction (e.g. for content-based image retrieval) 1. Motivation 4

Fundamental Question: Parsimonious Representation of Visual Information A randomly generated image A natural image Natural images live in a very tiny bit of the huge image space (e.g. R 512 512 3 ) 1. Motivation 5

Mathematical Foundation: Sparse Representations Fourier, Wavelets... = construction of bases for signal expansions: f = n c n ψ n, where c n = f, ψ n. Non-linear approximation: ˆf M = c n ψ n, where I M : indexes of biggest M coefficients. n I M Sparse representation: How fast f ˆf M 0 as M (e.g. f ˆf M 2 2 CM α ). 1. Motivation 6

The Success of Wavelets Wavelets provide a sparse representation for piecewise smooth signals. Multiresolution, tree structures, fast transforms and algorithms, etc. Unifying theory fruitful interaction between different fields. 1. Motivation 7

Fourier vs. Wavelets Non-linear approximation: N = 1024 data samples; keep M = 128 coefficients 40 20 0 Original 20 0 100 200 300 400 500 600 700 800 900 1000 40 20 0 20 0 100 200 300 400 500 600 700 800 900 1000 40 20 0 Using Fourier (of size 1024): SNR = 22.03 db Using wavelets (Daubechies 4): SNR = 37.67 db 20 0 100 200 300 400 500 600 700 800 900 1000 1. Motivation 8

Wavelets and Filter Banks analysis synthesis Mallat x H 1 2 y 1 2 G 1 x 1 + xˆ H 0 2 G 0 2 y 0 x 0 2 G 1 + 2 2 G 1 G 0 + Daubechies 2 G 0 1. Motivation 9

Is This the End of the Story? Photo: Ilya Pollak 1. Motivation 10

Wavelets in 2-D (and Higher Dimensions) In 1-D: Wavelets are well adapted to abrupt changes or singularities. In 2-D: Separable wavelets are well adapted to point-singularities (only). But, there are (mostly) line- and curved-singularities... 1. Motivation 11

Geometrical Structures of Multidimensional Signals Image (2-D): objects with smooth boundaries Video (3-D): moving objects carve out the spatio-temporal space Most information is contained in lower dimensional structures! 1. Motivation 12

The Failure of Wavelets Wavelets (with nonlinear approximations) cannot see the difference between these two images. Wavelets fail to capture the geometrical regularity (e.g. smooth contours) in images and multidimensional signals. 1. Motivation 13

Goal: Sparse Representation for Typical Images with Smooth Contours smooth regions 00000000000 11111111111 00000000000 11111111111 000000000000 111111111111 00000000000 11111111111 00000000000 11111111111 00000000000 11111111111 00000000000 11111111111 00000000000 11111111111 00000000000 11111111111 0000000000 1111111111 000000000 111111111 0000000 1111111 smooth boundary Goal: Exploring the intrinsic geometrical structure in natural images. Action is at the edges! 1. Motivation 14

Wavelet vs. New Scheme 00000000 11111111 00000000 11111111 000000000 111111111 000000000 111111111 00000000 11111111 0000000 1111111 000000 111111 000000 111111 00000 11111 00 00 000 11 11 111 Wavelet 00000000 11111111 00000000 11111111 000000000 111111111 000000000 111111111 00000000 11111111 0000000 1111111 000000 111111 000000 111111 00000 11111 000 111 00 11 01 New scheme For images: Wavelet scheme... see edges but not smooth contours. New scheme... requires challenging non-separable constructions. 1. Motivation 15

And What The Nature Tells Us... Human visual system: Extremely efficient: 10 7 bits 20-40 bits (per second). Receptive fields are characterized as localized, multiscale and oriented. Sparse components of natural images (Olshausen and Field, 1996): Search for Sparse Code 16 x 16 patches from natural images 1. Motivation 16

[Candès and Donoho (1999, 2004)] Curvelets Optimal representation for functions in R 2 with curved (C 2 ) singularities. c2 j/2 basis functions 2 j c2 j/2 2 j wavelet curvelet Key idea: parabolic scaling relation for C 2 curves: width length 2 1. Motivation 17

Wish List for New Image Representations Multiresolution... successive refinement Localization... both space and frequency Critical sampling... correct joint sampling Directionality... more directions Anisotropy... more shapes Our emphasis is on discrete framework that leads to algorithmic implementations. 1. Motivation 18

Outline 1. Motivation 2. Discrete-domain construction using filter banks: contourlets and surfacelets 3. Directional multiresolution analysis 4. Approximation power 5. Applications 6. Summary and outlook 2. Discrete-domain construction using filter banks 19

Challenge: Being Digital! Pixelization: Digital directions: 2. Discrete-domain construction using filter banks 20

Contourlets [Do and Vetterli, 2003] [Do and Vetterli (2003)] In a nutshell: contourlet transform is an efficient directional multiresolution expansion that is digital friendly! contourlets = multiscale, local and directional contour segments Starts with a discrete-domain construction that is amenable to efficient algorithms, and then investigates its convergence to a continuous-domain expansion. The expansion is defined on rectangular grids seamless translation between the continuous and discrete worlds. 2. Discrete-domain construction using filter banks 21

Discrete-Domain Construction using Filter Banks Idea: Multiscale and Directional Decomposition Multiscale step: capture point discontinuities, followed by... Directional step: link point discontinuities into linear structures. 2. Discrete-domain construction using filter banks 22

Analogy: Hough Transform in Computer Vision Input image Edge image Hough image = = Challenges: Perfect reconstruction. Fixed transform with low redundancy. Sparse representation for images with smooth contours. 2. Discrete-domain construction using filter banks 23

Multiscale Decomposition using Laplacian Pyramids decomposition reconstruction Reason: avoid frequency scrambling due to ( ) of the HP channel. Laplacian pyramid as a frame operator tight frame exists. New reconstruction: efficient filter bank for dual frame (pseudo-inverse). 2. Discrete-domain construction using filter banks 24

Directional Filter Banks (DFB) Feature: division of 2-D spectrum into fine slices using tree-structured filter banks. ω 2 (π, π) 4 0 1 2 3 7 5 6 6 5 ω 1 7 4 3 2 1 0 ( π, π) Background: Bamberger and Smith ( 92) cleverly used quincunx FB s, modulation and shearing. We propose: use DFB to construct directional bases, plus orthogonality. 2. Discrete-domain construction using filter banks 25

Our Simplified DFB: Two Building Blocks Frequency splitting by the quincunx filter banks (Vetterli 84). x Q Q y 0 y 1 Q Q + ˆx Shearing by resampling 2. Discrete-domain construction using filter banks 26

Sampling in Multidimensional Filter Banks Sampling is represented by integer matrices: x[n] M x d [n] x d [n] = x[mn] For example, quincunx sampling lattice: n 2 n 2 n 1 Q = ( ) 1 1. 1 1 n 1 2. Discrete-domain construction using filter banks 27

Examples of Multidimensional Sampling R Q 2. Discrete-domain construction using filter banks 28

How Frequency is Divided into Finer Direction? H (l) k M (l) Q Q 0000 1111 0000 1111 0000 1111 0000 1111 000 111 0 1 0000 1111 0000 1111 0000 1111 0000 1111 000 111 H (l+1) 2k H (l+1) 2k+1 M (l+1) M (l+1) Overall sampling: = [2 Dl 2 i ] R s l(k) separable sampling, then shearing M (l) k 2. Discrete-domain construction using filter banks 29

Multichannel View of the Directional Filter Bank H 0 S 0 y 0 S 0 G 0 x H 1 S 1 y 1 S 1 G 1 + ˆx H 2 l 1 S 2 l 1 y 2 l 1 S 2 l 1 G 2 l 1 Use two separable sampling matrices: [ ] 2 l 1 0 0 k < 2 l 1 ( near horizontal direction) 0 2 S k = [ ] 2 0 0 2 l 1 2 l 1 k < 2 l ( near vertical direction) 2. Discrete-domain construction using filter banks 30

General Bases from the DFB An l-levels DFB creates a local directional basis of l 2 (Z 2 ): { g (l) k [ S(l) k }0 k<2 n] l, n Z 2 G (l) k are directional filters: Sampling lattices (spatial tiling): 2 2 l 1 2 l 1 2 2. Discrete-domain construction using filter banks 31

Example of DFB Impulse Responses 32 equivalent filters for the first half channels (basically horizontal directions) of a 6-levels DFB that use the Haar filters 2. Discrete-domain construction using filter banks 32

Pyramidal Directional (or Contourlet) Filter Banks Motivation: + add multiscale into the directional filter bank + improve its non-linear approximation power. (2,2) ω 2 (π,π) ω 1 multiscale dec. ( π, π) directional dec. Properties: + Flexible multiscale and directional representation for images (can have different number of directions at each scale!) + Tight frame with small redundancy (< 33%) + Computational complexity: O(N) for N pixels. 2. Discrete-domain construction using filter banks 33

Wavelets vs. Contourlets 50 50 100 100 150 150 200 200 250 50 100 150 200 250 250 50 100 150 200 250 Comparing a few actual 2-D wavelets (5 on the left) and contourlets (4 on the right). 2. Discrete-domain construction using filter banks 34

Examples of Discrete Contourlet Transform 2. Discrete-domain construction using filter banks 35

Generalize to Multidimensions: Surfacelets [Lu and Do (2005)] Multiscale and multidirection transforms for N-D (N 2) signals. Basis: oriented surface patches. 150 30 25 100 20 W_Z 15 Z 50 10 5 0 0 0 0 0 0 5 5 50 50 10 10 W_Y 100 100 W_X 15 Y 20 20 15 X 25 25 150 150 30 30 A surfacelet in the frequency domain. A surfacelet in the spatial domain. Efficient tree-structured transform using filter banks. Refinable angular resolution. 2. Discrete-domain construction using filter banks 36

Multiresolution Analysis: Laplacian Pyramid w 2 W j w 1 V j 1 = V j W j, L 2 (R 2 ) = j Z W j. V j V j 1 V j has an orthogonal basis {φ j,n } n Z 2, where φ j,n (t) = 2 j φ(2 j t n). W j has a tight frame {µ j 1,n } n Z 2 where µ j 1,2n+ki = ψ (i) j,n, i = 0,...,3. 3. Directional multiresolution analysis 38

Directional Multiresolution Analysis: LP + DFB 0000 1111 000000 111111 0000000 1111111 000000 111111 00000 11111 0000 1111 000 111 00 11 ω 2 V j 000 111 0000 1111 00000 11111 000000 111111 000000 111111 00000 11111 000 111 01 W j W (l j) j,k ω 1 V j 1 2 j 2 j+l j 2 ρ (l j) j,k,n W j = 2 lj 1 k=0 W (l j) j,k { } W (l j) j,k has a tight frame ρ (l) j,k,n where n Z 2 j,k,n (t) = g (l) k [m S(l) }{{ k n] } m Z 2 ρ (l) DFB basis µ j 1,m (t) }{{} LP frame = ρ (l) j,k (t 2j 1 S (l) k n). 3. Directional multiresolution analysis 39

Contourlet Frames Theorem { } (Contourlet Frames) [DoV:03]. ρ (l j) j,k,n is a tight frame of j Z, 0 k<2 lj L2 (R 2 ) for finite l j., n Z 2 Theorem (Connection with Filter Banks) [DoV:05] Suppose x[n] = f, φ L,n, n Z 2, for some function f L 2 (R 2 ). Furthermore, suppose x contourlet transform { a J, d (l j) j,k } j=1,...,j; k=0,...,2 lj 1 where a J is the lowpass subband, and d (l j) j,k subbands. Then are bandpass directional a J [n] = f, φ L+J,n d (l j) j,k [n] = f, ρ(l j) L+j,k,n for j = 1,...,J; k = 0,...,2 l j 1, n Z 2. 3. Directional multiresolution analysis 40

Sampling Grids of Contourlets w l l w w/4 (a) l/2 (b) l/2 w/4 (c) (d) 3. Directional multiresolution analysis 41

Example Contourlet Basis Images Basis functions, Level 2 Basis functions, Level 3 Basis functions, Level 4 3. Directional multiresolution analysis 42

Contourlet Features Defined via iterated filter banks fast algorithms, tree structures, etc. Defined on rectangular grids seamless translation between continuous and discrete worlds. Different contourlet kernel functions (ρ j,k ) for different directions. These functions are defined iteratively via filter banks. With FIR filters compactly supported contourlet functions. 3. Directional multiresolution analysis 43

[Do and Vetterli (2005)] Contourlet Approximation Π L Π L see a wavelet Desire: Fast decay as contourlets turn away from the discontinuity direction Key: Directional vanishing moments 4. Approximation power 45

Contourlets with Parabolic Scaling Support size of the contourlet function ρ l j j,k : width 2j and length 2 l j+j To satisfy the parabolic scaling (for C 2 curved singularities): width length 2, simply set: the number of directions in is doubled at every other finer scale. u = u(v) u w v ω 2 (π, π) ω 1 l ( π, π) 4. Approximation power 46

Supports of Contourlet Functions LP DFB Contourlet * = * = Key point: Each generation doubles spatial resolution as well as angular resolution. 4. Approximation power 47

Geometrical Intuition At scale 2 j (j 0): d 2 j,k,n 2 j/2 width 2 j 01 d j,k,n length 2 j/2 #directions 2 j/2 00 11 0000 1111 θ j,k,n A v S 2 j f,ρ j,k,n 2 3j/4 d 3 j,k,n d j,k,n 2 j / sinθ j,k,n = f,ρ j, k,n 2 3j/4 k 3 θ j, k,n k2 j/2 for k = 1,...,2 j/2 4. Approximation power 48

Nonlinear Approximation Rates C 2 regions 00 11 000000000000 111111111111 000000000000 111111111111 0000000000000 1111111111111 0000000000000 1111111111111 000000000000 111111111111 000000000000 111111111111 000000000000 111111111111 000000000000 111111111111 000000000000 111111111111 0000000000 1111111111 0000000000 1111111111 00000000 11111111 C 2 boundary With sufficient DVMs: f, ρ j, k,n 2 3j/4 k 3 and number of coefficients N j, k 2 j/2 k This leads to f ˆf (contourlet) M 2 2 (log M) 3 M -2 While f ˆf (Fourier) M 2 2 M -1/2 and f ˆf (wavelet) M 2 2 M -1 4. Approximation power 49

Non-linear Approximation Experiments Image size = 512 512. Keep M = 4096 coefficients. Original image Wavelets: Contourlets: PSNR = 24.34 db PSNR = 25.70 db 4. Approximation power 50

Detailed Non-linear Approximations Wavelets M = 4 M = 16 M = 64 M = 256 Contourlets M = 4 M = 16 M = 64 M = 256 4. Approximation power 51

Embedded Structure for Compression and Modeling So far, best-m term approximation: ˆf M = λ I M c λ ρ λ, where I M is the set of indexes of the M-largest c λ. For compression, additional cost required to specify I M Naive approach: M log 2 N bits With embedded tree (wavelets): M bits Embedded trees for wavelets are crucial in state-of-the-art image compression (EZW,...), rate-distortion analysis (Cohen et al.), and multiscale statistical modeling (Baraniuk et al.) 4. Approximation power 52

Contourlet Embedded Tree Structure Embedded tree data structure for contourlet coefficients: successively locate the position and direction of image contours. Since significant contourlet coefficients are organized in trees, best M-tree approximation (using M-node tree): f (contourlet) ˆf M tree 2 (log M) 3 M 2 D(R) (log R) 3 R -2 The embedded tree data structure of contourlets is also used effectively in image modeling [Po and Do, 2006]. 4. Approximation power 53

Image Enhancement using Contourlets [Cunha, Zhou, and Do (2006)] (a) Enhanced by non-subsampled wavelet transform (b) Enhanced by non-subsampled contourlet transform 6. Applications 55

Image Interpolation using Contourlets [Mueller, Lu, and Do (2007)] Downsample and then upsample 4 times. (a) Original (b) Bilinear (26.19 db) (c) Wavelet linear (28.23 db) (d) Proposed (29.53 db) 6. Applications 56

Restore Damaged Images using Contourlets [Mueller, Lu, and Do (2007)] Damaged image 6. Applications Adaptive interpolation Our method 57

Video Denoising using Surfacelets [Lu and Do (2007)] Denoised by 3D wavelets Denoised by surfacelets 6. Applications 58

Speculation: Another Wavelet Story? Wavelets (harmonics analysis) Pyramids (computer vision) Filter banks (signal processing) Multiresolution Analysis Theory Algorithms Applications 1-D Story: scale and shift 6. Summary and outlook 60

Speculation: Another Wavelet Story? Curvelets (harmonics analysis) Hough transform (computer vision) Directional filter bank (signal processing) Directional Multiresolution Analysis Theory Algorithms Applications 2-D Story: scale, space, and direction 6. Summary and outlook 61

Beyond 2-D and Single Image... New Audio-Visual Paradigm Existing audio-visual recording and playback Single camera and microphone Little processing Viewers: passive Future Sensors (cameras, microphones) will be cheap Massive computing and storage capabilities will be available Viewers: interactive, immersive, remote Require new signal processing theory and algorithms 6. Summary and outlook 62

Emerging Application: Free-Viewpoint 3D Video or Remove Reality The system uses multiple cameras to record a dynamic scene. Multiple receivers that can independently and freely choose viewpoint for 3D viewing. 6. Summary and outlook 63

In Higher Dimensions Many multidimensional data. For example: 3-D (video, MRI), 4-D (temporal volume sequences, light-field, seismic data), 6-D (bidirectional texture functions), 7-D (plenoptic functions),.. Geometric information hidden in the data. w t epipolar image v Model of interest: Singularities live on smooth surfaces. 6. Summary and outlook 64

Summary Image processing relies on prior information about images. Geometrical structure is the key! Strong motivation for more powerful image representations: scale, space, and direction. New desideratum beyond wavelets: localized direction New two (and higher) dimensional discrete framework and algorithms: Flexible directional and multiresolution image representation. Effective for images with smooth contours contourlets and surfacelets. Dream: Another fruitful interaction between harmonic analysis, computer vision, and signal processing. 65

References M. N. Do and M. Vetterli, Contourlets, Beyond Wavelets, G. V. Welland ed., Academic Press, 2003. M. N. Do and M. Vetterli, The contourlet transform: an efficient directional multiresolution image representation, IEEE Trans. on Image Proc., Dec. 2005. D. D.-Y. Po and M. N. Do, Directional multiscale modeling of images using the contourlet transform, IEEE Trans. on Image Proc., June 2006. A. Cunha, J. Zhou, and M. N. Do, The nonsubsampled contourlet transform: Theory, design, and applications, IEEE Trans. on Image Proc., Oct. 2006. A. Cunha and M. N. Do, On two-channel filter banks with directional vanishing moments, IEEE Trans. on Image Proc., 2007. Y. Lu and M. N. Do, Multidimensional directional filter banks and surfacelets, IEEE Trans. on Image Proc., 2007. Downloadable papers and software: www.ifp.uiuc.edu/ minhdo 66