Beyond Wavelets: Multiscale Geometric Representations

Size: px

Start display at page:

Download "Beyond Wavelets: Multiscale Geometric Representations"

Shawn Stanley
5 years ago
Views:

1 Beyond Wavelets: Multiscale Geometric Representations Minh N. Do Department of Electrical and Computer Engineering University of Illinois at Urbana-Champaign minhdo

2 Acknowledgments Collaborators: Prof. Martin Vetterli (EPFL and UC Berkeley) Dr. Arthur L. A. da Cunha (JP Morgan) Dr. Yue Lu (EPFL) Mr. Duncan Po (MathWorks) Dr. Jianping Zhou (Intel) Funding source: National Science Foundation CAREER Award 1

3 What Do Image Processors Do for Living? Compression: At 158:1 compression ratio Motivation 2

4 What Do Image Processors Do for Living? Denoising (restoration/filtering) Noisy image Clean image 1. Motivation 3

5 What Do Image Processors Do for Living? Feature extraction (e.g. for content-based image retrieval) 1. Motivation 4

6 Fundamental Question: Parsimonious Representation of Visual Information A randomly generated image A natural image Natural images live in a very tiny bit of the huge image space (e.g. R ) 1. Motivation 5

7 Mathematical Foundation: Sparse Representations Fourier, Wavelets... = construction of bases for signal expansions: f = n c n ψ n, where c n = f, ψ n. Non-linear approximation: ˆf M = c n ψ n, where I M : indexes of biggest M coefficients. n I M Sparse representation: How fast f ˆf M 0 as M (e.g. f ˆf M 2 2 CM α ). 1. Motivation 6

8 The Success of Wavelets Wavelets provide a sparse representation for piecewise smooth signals. Multiresolution, tree structures, fast transforms and algorithms, etc. Unifying theory fruitful interaction between different fields. 1. Motivation 7

9 Fourier vs. Wavelets Non-linear approximation: N = 1024 data samples; keep M = 128 coefficients Original Using Fourier (of size 1024): SNR = db Using wavelets (Daubechies 4): SNR = db Motivation 8

10 Wavelets and Filter Banks analysis synthesis Mallat x H 1 2 y 1 2 G 1 x 1 + xˆ H 0 2 G 0 2 y 0 x 0 2 G G 1 G 0 + Daubechies 2 G 0 1. Motivation 9

11 Is This the End of the Story? Photo: Ilya Pollak 1. Motivation 10

12 Wavelets in 2-D (and Higher Dimensions) In 1-D: Wavelets are well adapted to abrupt changes or singularities. In 2-D: Separable wavelets are well adapted to point-singularities (only). But, there are (mostly) line- and curved-singularities Motivation 11

objects carve out the spatio-temporal space Most

13 Geometrical Structures of Multidimensional Signals Image (2-D): objects with smooth boundaries Video (3-D): moving objects carve out the spatio-temporal space Most information is contained in lower dimensional structures! 1. Motivation 12

14 The Failure of Wavelets Wavelets (with nonlinear approximations) cannot see the difference between these two images. Wavelets fail to capture the geometrical regularity (e.g. smooth contours) in images and multidimensional signals. 1. Motivation 13

15 Goal: Sparse Representation for Typical Images with Smooth Contours smooth regions smooth boundary Goal: Exploring the intrinsic geometrical structure in natural images. Action is at the edges! 1. Motivation 14

16 Wavelet vs. New Scheme Wavelet New scheme For images: Wavelet scheme... see edges but not smooth contours. New scheme... requires challenging non-separable constructions. 1. Motivation 15

And What The Nature Tells Us... Human visual system: Extremely efficient: 10 7 bits 20-40 bits (per second).

17 And What The Nature Tells Us... Human visual system: Extremely efficient: 10 7 bits bits (per second). Receptive fields are characterized as localized, multiscale and oriented. Sparse components of natural images (Olshausen and Field, 1996): Search for Sparse Code 16 x 16 patches from natural images 1. Motivation 16

18 [Candès and Donoho (1999, 2004)] Curvelets Optimal representation for functions in R 2 with curved (C 2 ) singularities. c2 j/2 basis functions 2 j c2 j/2 2 j wavelet curvelet Key idea: parabolic scaling relation for C 2 curves: width length 2 1. Motivation 17

19 Wish List for New Image Representations Multiresolution... successive refinement Localization... both space and frequency Critical sampling... correct joint sampling Directionality... more directions Anisotropy... more shapes Our emphasis is on discrete framework that leads to algorithmic implementations. 1. Motivation 18

20 Outline 1. Motivation 2. Discrete-domain construction using filter banks: contourlets and surfacelets 3. Directional multiresolution analysis 4. Approximation power 5. Applications 6. Summary and outlook 2. Discrete-domain construction using filter banks 19

21 Challenge: Being Digital! Pixelization: Digital directions: 2. Discrete-domain construction using filter banks 20

22 Contourlets [Do and Vetterli, 2003] [Do and Vetterli (2003)] In a nutshell: contourlet transform is an efficient directional multiresolution expansion that is digital friendly! contourlets = multiscale, local and directional contour segments Starts with a discrete-domain construction that is amenable to efficient algorithms, and then investigates its convergence to a continuous-domain expansion. The expansion is defined on rectangular grids seamless translation between the continuous and discrete worlds. 2. Discrete-domain construction using filter banks 21

23 Discrete-Domain Construction using Filter Banks Idea: Multiscale and Directional Decomposition Multiscale step: capture point discontinuities, followed by... Directional step: link point discontinuities into linear structures. 2. Discrete-domain construction using filter banks 22

24 Analogy: Hough Transform in Computer Vision Input image Edge image Hough image = = Challenges: Perfect reconstruction. Fixed transform with low redundancy. Sparse representation for images with smooth contours. 2. Discrete-domain construction using filter banks 23

Laplacian pyramid as a frame operator tight frame exists.

25 Multiscale Decomposition using Laplacian Pyramids decomposition reconstruction Reason: avoid frequency scrambling due to ( ) of the HP channel. Laplacian pyramid as a frame operator tight frame exists. New reconstruction: efficient filter bank for dual frame (pseudo-inverse). 2. Discrete-domain construction using filter banks 24

ω 2 (π, π) 4 0 1 2 3 7 5 6 6 5 ω 1 7 4 3 2 1 0 ( π, π) Background: Bamberger and Smith ( 92)

26 Directional Filter Banks (DFB) Feature: division of 2-D spectrum into fine slices using tree-structured filter banks. ω 2 (π, π) ω ( π, π) Background: Bamberger and Smith ( 92) cleverly used quincunx FB s, modulation and shearing. We propose: use DFB to construct directional bases, plus orthogonality. 2. Discrete-domain construction using filter banks 25

27 Our Simplified DFB: Two Building Blocks Frequency splitting by the quincunx filter banks (Vetterli 84). x Q Q y 0 y 1 Q Q + ˆx Shearing by resampling 2. Discrete-domain construction using filter banks 26

28 Sampling in Multidimensional Filter Banks Sampling is represented by integer matrices: x[n] M x d [n] x d [n] = x[mn] For example, quincunx sampling lattice: n 2 n 2 n 1 Q = ( ) n 1 2. Discrete-domain construction using filter banks 27

29 Examples of Multidimensional Sampling R Q 2. Discrete-domain construction using filter banks 28

30 How Frequency is Divided into Finer Direction? H (l) k M (l) Q Q H (l+1) 2k H (l+1) 2k+1 M (l+1) M (l+1) Overall sampling: = [2 Dl 2 i ] R s l(k) separable sampling, then shearing M (l) k 2. Discrete-domain construction using filter banks 29

31 Multichannel View of the Directional Filter Bank H 0 S 0 y 0 S 0 G 0 x H 1 S 1 y 1 S 1 G 1 + ˆx H 2 l 1 S 2 l 1 y 2 l 1 S 2 l 1 G 2 l 1 Use two separable sampling matrices: [ ] 2 l k < 2 l 1 ( near horizontal direction) 0 2 S k = [ ] l 1 2 l 1 k < 2 l ( near vertical direction) 2. Discrete-domain construction using filter banks 30

32 General Bases from the DFB An l-levels DFB creates a local directional basis of l 2 (Z 2 ): { g (l) k [ S(l) k }0 k<2 n] l, n Z 2 G (l) k are directional filters: Sampling lattices (spatial tiling): 2 2 l 1 2 l Discrete-domain construction using filter banks 31

33 Example of DFB Impulse Responses 32 equivalent filters for the first half channels (basically horizontal directions) of a 6-levels DFB that use the Haar filters 2. Discrete-domain construction using filter banks 32

34 Pyramidal Directional (or Contourlet) Filter Banks Motivation: + add multiscale into the directional filter bank + improve its non-linear approximation power. (2,2) ω 2 (π,π) ω 1 multiscale dec. ( π, π) directional dec. Properties: + Flexible multiscale and directional representation for images (can have different number of directions at each scale!) + Tight frame with small redundancy (< 33%) + Computational complexity: O(N) for N pixels. 2. Discrete-domain construction using filter banks 33

35 Wavelets vs. Contourlets Comparing a few actual 2-D wavelets (5 on the left) and contourlets (4 on the right). 2. Discrete-domain construction using filter banks 34

36 Examples of Discrete Contourlet Transform 2. Discrete-domain construction using filter banks 35

37 Generalize to Multidimensions: Surfacelets [Lu and Do (2005)] Multiscale and multidirection transforms for N-D (N 2) signals. Basis: oriented surface patches W_Z 15 Z W_Y W_X 15 Y X A surfacelet in the frequency domain. A surfacelet in the spatial domain. Efficient tree-structured transform using filter banks. Refinable angular resolution. 2. Discrete-domain construction using filter banks 36

38 Outline 1. Motivation 2. Discrete-domain construction using filter banks: contourlets and surfacelets 3. Directional multiresolution analysis 4. Approximation power 5. Applications 6. Summary and outlook 3. Directional multiresolution analysis 37

39 Multiresolution Analysis: Laplacian Pyramid w 2 W j w 1 V j 1 = V j W j, L 2 (R 2 ) = j Z W j. V j V j 1 V j has an orthogonal basis {φ j,n } n Z 2, where φ j,n (t) = 2 j φ(2 j t n). W j has a tight frame {µ j 1,n } n Z 2 where µ j 1,2n+ki = ψ (i) j,n, i = 0,...,3. 3. Directional multiresolution analysis 38

40 Directional Multiresolution Analysis: LP + DFB ω 2 V j W j W (l j) j,k ω 1 V j 1 2 j 2 j+l j 2 ρ (l j) j,k,n W j = 2 lj 1 k=0 W (l j) j,k { } W (l j) j,k has a tight frame ρ (l) j,k,n where n Z 2 j,k,n (t) = g (l) k [m S(l) }{{ k n] } m Z 2 ρ (l) DFB basis µ j 1,m (t) }{{} LP frame = ρ (l) j,k (t 2j 1 S (l) k n). 3. Directional multiresolution analysis 39

41 Contourlet Frames Theorem { } (Contourlet Frames) [DoV:03]. ρ (l j) j,k,n is a tight frame of j Z, 0 k<2 lj L2 (R 2 ) for finite l j., n Z 2 Theorem (Connection with Filter Banks) [DoV:05] Suppose x[n] = f, φ L,n, n Z 2, for some function f L 2 (R 2 ). Furthermore, suppose x contourlet transform { a J, d (l j) j,k } j=1,...,j; k=0,...,2 lj 1 where a J is the lowpass subband, and d (l j) j,k subbands. Then are bandpass directional a J [n] = f, φ L+J,n d (l j) j,k [n] = f, ρ(l j) L+j,k,n for j = 1,...,J; k = 0,...,2 l j 1, n Z Directional multiresolution analysis 40

42 Sampling Grids of Contourlets w l l w w/4 (a) l/2 (b) l/2 w/4 (c) (d) 3. Directional multiresolution analysis 41

43 Example Contourlet Basis Images Basis functions, Level 2 Basis functions, Level 3 Basis functions, Level 4 3. Directional multiresolution analysis 42

44 Contourlet Features Defined via iterated filter banks fast algorithms, tree structures, etc. Defined on rectangular grids seamless translation between continuous and discrete worlds. Different contourlet kernel functions (ρ j,k ) for different directions. These functions are defined iteratively via filter banks. With FIR filters compactly supported contourlet functions. 3. Directional multiresolution analysis 43

45 Outline 1. Motivation 2. Discrete-domain construction using filter banks: contourlets and surfacelets 3. Directional multiresolution analysis 4. Approximation power 5. Applications 6. Summary and outlook 4. Approximation power 44

46 [Do and Vetterli (2005)] Contourlet Approximation Π L Π L see a wavelet Desire: Fast decay as contourlets turn away from the discontinuity direction Key: Directional vanishing moments 4. Approximation power 45

47 Contourlets with Parabolic Scaling Support size of the contourlet function ρ l j j,k : width 2j and length 2 l j+j To satisfy the parabolic scaling (for C 2 curved singularities): width length 2, simply set: the number of directions in is doubled at every other finer scale. u = u(v) u w v ω 2 (π, π) ω 1 l ( π, π) 4. Approximation power 46

48 Supports of Contourlet Functions LP DFB Contourlet * = * = Key point: Each generation doubles spatial resolution as well as angular resolution. 4. Approximation power 47

49 Geometrical Intuition At scale 2 j (j 0): d 2 j,k,n 2 j/2 width 2 j 01 d j,k,n length 2 j/2 #directions 2 j/ θ j,k,n A v S 2 j f,ρ j,k,n 2 3j/4 d 3 j,k,n d j,k,n 2 j / sinθ j,k,n = f,ρ j, k,n 2 3j/4 k 3 θ j, k,n k2 j/2 for k = 1,...,2 j/2 4. Approximation power 48

50 Nonlinear Approximation Rates C 2 regions C 2 boundary With sufficient DVMs: f, ρ j, k,n 2 3j/4 k 3 and number of coefficients N j, k 2 j/2 k This leads to f ˆf (contourlet) M 2 2 (log M) 3 M -2 While f ˆf (Fourier) M 2 2 M -1/2 and f ˆf (wavelet) M 2 2 M Approximation power 49

Non-linear Approximation Experiments Image size = 512 512. Keep M = 4096 coefficients.

51 Non-linear Approximation Experiments Image size = Keep M = 4096 coefficients. Original image Wavelets: Contourlets: PSNR = db PSNR = db 4. Approximation power 50

52 Detailed Non-linear Approximations Wavelets M = 4 M = 16 M = 64 M = 256 Contourlets M = 4 M = 16 M = 64 M = Approximation power 51

53 Embedded Structure for Compression and Modeling So far, best-m term approximation: ˆf M = λ I M c λ ρ λ, where I M is the set of indexes of the M-largest c λ. For compression, additional cost required to specify I M Naive approach: M log 2 N bits With embedded tree (wavelets): M bits Embedded trees for wavelets are crucial in state-of-the-art image compression (EZW,...), rate-distortion analysis (Cohen et al.), and multiscale statistical modeling (Baraniuk et al.) 4. Approximation power 52

54 Contourlet Embedded Tree Structure Embedded tree data structure for contourlet coefficients: successively locate the position and direction of image contours. Since significant contourlet coefficients are organized in trees, best M-tree approximation (using M-node tree): f (contourlet) ˆf M tree 2 (log M) 3 M 2 D(R) (log R) 3 R -2 The embedded tree data structure of contourlets is also used effectively in image modeling [Po and Do, 2006]. 4. Approximation power 53

55 Outline 1. Motivation 2. Discrete-domain construction using filter banks: contourlets and surfacelets 3. Directional multiresolution analysis 4. Approximation power 5. Applications 6. Summary and outlook 6. Applications 54

56 Image Enhancement using Contourlets [Cunha, Zhou, and Do (2006)] (a) Enhanced by non-subsampled wavelet transform (b) Enhanced by non-subsampled contourlet transform 6. Applications 55

57 Image Interpolation using Contourlets [Mueller, Lu, and Do (2007)] Downsample and then upsample 4 times. (a) Original (b) Bilinear (26.19 db) (c) Wavelet linear (28.23 db) (d) Proposed (29.53 db) 6. Applications 56

58 Restore Damaged Images using Contourlets [Mueller, Lu, and Do (2007)] Damaged image 6. Applications Adaptive interpolation Our method 57

59 Video Denoising using Surfacelets [Lu and Do (2007)] Denoised by 3D wavelets Denoised by surfacelets 6. Applications 58

60 Outline 1. Motivation 2. Discrete-domain construction using filter banks: contourlets and surfacelets 3. Directional multiresolution analysis 4. Approximation power 5. Applications 6. Summary and outlook 6. Summary and outlook 59

61 Speculation: Another Wavelet Story? Wavelets (harmonics analysis) Pyramids (computer vision) Filter banks (signal processing) Multiresolution Analysis Theory Algorithms Applications 1-D Story: scale and shift 6. Summary and outlook 60

62 Speculation: Another Wavelet Story? Curvelets (harmonics analysis) Hough transform (computer vision) Directional filter bank (signal processing) Directional Multiresolution Analysis Theory Algorithms Applications 2-D Story: scale, space, and direction 6. Summary and outlook 61

63 Beyond 2-D and Single Image... New Audio-Visual Paradigm Existing audio-visual recording and playback Single camera and microphone Little processing Viewers: passive Future Sensors (cameras, microphones) will be cheap Massive computing and storage capabilities will be available Viewers: interactive, immersive, remote Require new signal processing theory and algorithms 6. Summary and outlook 62

64 Emerging Application: Free-Viewpoint 3D Video or Remove Reality The system uses multiple cameras to record a dynamic scene. Multiple receivers that can independently and freely choose viewpoint for 3D viewing. 6. Summary and outlook 63

65 In Higher Dimensions Many multidimensional data. For example: 3-D (video, MRI), 4-D (temporal volume sequences, light-field, seismic data), 6-D (bidirectional texture functions), 7-D (plenoptic functions),.. Geometric information hidden in the data. w t epipolar image v Model of interest: Singularities live on smooth surfaces. 6. Summary and outlook 64

66 Summary Image processing relies on prior information about images. Geometrical structure is the key! Strong motivation for more powerful image representations: scale, space, and direction. New desideratum beyond wavelets: localized direction New two (and higher) dimensional discrete framework and algorithms: Flexible directional and multiresolution image representation. Effective for images with smooth contours contourlets and surfacelets. Dream: Another fruitful interaction between harmonic analysis, computer vision, and signal processing. 65

67 References M. N. Do and M. Vetterli, Contourlets, Beyond Wavelets, G. V. Welland ed., Academic Press, M. N. Do and M. Vetterli, The contourlet transform: an efficient directional multiresolution image representation, IEEE Trans. on Image Proc., Dec D. D.-Y. Po and M. N. Do, Directional multiscale modeling of images using the contourlet transform, IEEE Trans. on Image Proc., June A. Cunha, J. Zhou, and M. N. Do, The nonsubsampled contourlet transform: Theory, design, and applications, IEEE Trans. on Image Proc., Oct A. Cunha and M. N. Do, On two-channel filter banks with directional vanishing moments, IEEE Trans. on Image Proc., Y. Lu and M. N. Do, Multidimensional directional filter banks and surfacelets, IEEE Trans. on Image Proc., Downloadable papers and software: minhdo 66

Contourlets: Construction and Properties

Contourlets: Construction and Properties Minh N. Do Department of Electrical and Computer Engineering University of Illinois at Urbana-Champaign www.ifp.uiuc.edu/ minhdo minhdo@uiuc.edu Joint work with