Applications and Outlook

Size: px

Start display at page:

Download "Applications and Outlook"

Estella Hood
5 years ago
Views:

pplications and Outlook Outline. Image modeling using the contourlet transform.

o epartment of Electrical and Computer Engineering University of Illinois at Urbana-Champaign

Outlook www.ifp.uiuc.edu/ minhdo minhdo@uiuc.

1 pplications and Outlook Outline. Image modeling using the contourlet transform. Critically sampled (CRISP) contourlet transform Minh N. o epartment of Electrical and Computer Engineering University of Illinois at Urbana-Champaign 3. Image denoising and enhancement using the nonsubsampled contourlet transform 4. Outlook minhdo minhdo@uiuc.edu Joint work with rthur Cunha, Yue Lu, Jianping Zhou (UIUC), and uncan Po (MathWorks). Image modeling using the contourlet transform. Image Modeling using the Contourlet Transform Contourlet Coefficient Relationships Coarser Scale Finer Scale Reference Coefficient Neighbors Cousins Parent. Image modeling using the contourlet transform. Image modeling using the contourlet transform 3

2 Marginal Statistics of Contourlet Coefficients Joint Statistics Histogram 3 Histogram P N C Coefficient mplitude Coefficient mplitude Marginal statistics of two finest subbands of the image Peppers. The kurtoses of the two distributions are measured at 4.5 and 9.4, showing that the coefficients are highly non-gaussian P N 5 4 C Top: Joint histograms of next neighbors ottom: Joint histograms of distance (three coefficients away) neighbors. Image modeling using the contourlet transform 4. Image modeling using the contourlet transform 5 Conditional istribution ependence Characterization using Mutual Information Histogram Coefficient mplitude Histogram Coefficient mplitudes Histogram Coefficient mplitude Mutual information estimates between a contourlet coefficient () and its parent (P ), its spatial neighbors (N ), and its directional cousins (C). Lena arbara Peppers I(; P )..4. I(; N ) I(; C) The kurtoses of the distributions are measured at 3.9,.9, and.99. Contourlet coefficients are non-gaussian but conditionally Gaussian. Mutual information estimates with a single parent, neighbor, and cousin. Lena arbara Peppers I(; P )..4.8 I(;N ) I(;N ) I(;C ).8..6 I(;C ) I(;C 3 ) Image modeling using the contourlet transform 6. Image modeling using the contourlet transform 7

Wavelet-domain Hidden Markov Models [CrouseN:98] Contourlet-domain Hidden Markov Tree Models [Po:4]

Contourlet HMT models all inter-scale, inter-direction, and inter-location independencies. 4.

Image modeling using the contourlet transform 9 enoising Results: Zelda Texture Retrieval Results: rodatz

9% Left to right, top to bottom: (a) Zelda image, (b) noisy image (4.6d), (c) wiener (5.

3 Wavelet-domain Hidden Markov Models [CrouseN:98] Contourlet-domain Hidden Markov Tree Models [Po:4] irection Scale 3 Wavelet HMT models coefficients on each direction independently. Contourlet HMT models all inter-scale, inter-direction, and inter-location independencies. 4. Image modeling using the contourlet transform 8. Image modeling using the contourlet transform 9 enoising Results: Zelda Texture Retrieval Results: rodatz atabase verage retrieval rates wavelet HMT contourlet HMT 9.87% 93.9% Left to right, top to bottom: (a) Zelda image, (b) noisy image (4.6d), (c) wiener (5.78d), (d) wavelet thresholding (6.5d), (e) wavelet HMT (7.63d), and (f) contourlet HMT (7.7d). Top: Wavelets do better (> 5%). ottom: Contourlets do better (> 5%).. Image modeling using the contourlet transform. Image modeling using the contourlet transform

4 Outline Frequency Partition of CRISP-Contourlets [Lu:3]. Image modeling using the contourlet transform. Critically sampled (CRISP) contourlet transform ω (π, π) Image denoising and enhancement using the nonsubsampled contourlet transform 3 ω 4. Outlook ( π, π) directional highpass: 3 n (n =,,...) directions at each level. lowpass bands: directional lowpass bands.. Critically sampled (CRISP) contourlet transform. Critically sampled (CRISP) contourlet transform 3 Intuition ehind the Frequency Partitioning Sampling Matrices for CRISP-Contourlets Proposition. For a spectrum support F to be critically sampled by a matrix M, the area of the support must be 4π det(m). ω (π, π) ω (π, π) Each directional bandpass pair can be shifted and combined to form a parallelogram support. ( π, π) ω For each directional bandpass region area = 4π /(4 m n ) ( π, π) ω The parallelogram support can be maximally decimated. If the shifts satisfy certain conditions, the split and shifted support can also be maximally decimated. Proposition. With 3 l directions, the spectrum support Rk l of bandpass directional subbands of the CRISP-contourlet transform is critically sampled by S l k = diag( l,4) (or diag(4, l )): m Z R l k (ω π(s l k) T m) =. The decimation matrices (for n > ) in CRISP-contourlets are diagonal: ( M v (m,n) m+n+ = m+ ) ( and M (m,n) m+ h = m+n+ ). Critically sampled (CRISP) contourlet transform 4. Critically sampled (CRISP) contourlet transform 5

5 CRISP-Contourlet Transform lock iagram Iteration for Finer irectionality ( π, π) ω (π, π) ω P PR L SC PR SPR L R SPR SPR PR Iteration on lowpass bands PR SC SPR Iteration on directional decomposition R The number of directions can be 3 n, for all n. The refinement of directionality is achieved via an iteration of two filter banks. M rbitrary multiscale and multidirectional decomposition through an iterated combination of 3 filter banks: Type L outputs are followed by SC filters; Type outputs are followed by PR filters. Type outputs are followed by SPR filters.. Critically sampled (CRISP) contourlet transform 6. Critically sampled (CRISP) contourlet transform 7 CRISP-Contourlet Transform using Nonuniform s Outline ω (π, π). Image modeling using the contourlet transform. Critically sampled (CRISP) contourlet transform ω 3. Image denoising and enhancement using the nonsubsampled contourlet transform 4. Outlook 3 ( π, π) Critically sampled (CRISP) contourlet transform 8 3. Image denoising and enhancement using the nonsubsampled contourlet transform 9

6 Motivation Critically-Sampled s For image-analysis applications (denoising, enhancement,...), we want: NLYSIS SYNTHESIS irectional multiresolution shift-invariant image representation H G Structured transforms with fast algorithms + ˆ Critical sampling is not critical H G nswer: Nonsubsampled Contourlet Transform (NSCT) Problem: Shift-variant 3. Image denoising and enhancement using the nonsubsampled contourlet transform 3. Image denoising and enhancement using the nonsubsampled contourlet transform Nonsubsampled s Iterated Nonsubsampled Contourlet Transform: Illustration NLYSIS SYNTHESIS H (z^) H H G G + ˆ H H H (z^) H (z^) H (z^) dvantages: Key: Filtering with holes using the equivalent sampling matrices Shift-invariant Fast transforms via the à trous algorithm 3. Image denoising and enhancement using the nonsubsampled contourlet transform 3. Image denoising and enhancement using the nonsubsampled contourlet transform 3

Example of Nonsubsampled Contourlet Transform

(b) (a) Original image. (b) Lowpass image.

Image denoising and enhancement using the

nonsubsampled contourlet transform 5 Filter

(NSCT) We have to design for two cases: pyramid

(z) G (z) H (z) G (z) How to void Spectral

McClellan transformation) We first design a set

f(e jw ) and substitute H (z) G (z) H (z) G (z)

x=f(z) In both cases perfect reconstruction

This is easier than biorthogonal filter bank

7 Example of Nonsubsampled Contourlet Transform (/) Example of Nonsubsampled Contourlet Transform (/) Finest bandpass directional images (a) (b) (a) Original image. (b) Lowpass image. (c) andpass directional images. (c) 3. Image denoising and enhancement using the nonsubsampled contourlet transform 4 3. Image denoising and enhancement using the nonsubsampled contourlet transform 5 Filter esign for the Nonsubsampled Contourlet Transform (NSCT) We have to design for two cases: pyramid filters for the LP and fan filters for the F H (z) G (z) H (z) G (z) How to void Spectral Factorization? nswer: Use - to - mapping (a.k.a. McClellan transformation) We first design a set of - polynomials that solve P (x)q (x) + P (x)q (x) =. Then choose a mapping function f(z) such that f(e jw ) and substitute H (z) G (z) H (z) G (z) H i (z) = P i (x) x=f(z), and G i (z) = Q i (x) x=f(z) In both cases perfect reconstruction means H (z)g (z) + H (z)g (z) = for i =,. This is easier than biorthogonal filter bank condition. 3. Image denoising and enhancement using the nonsubsampled contourlet transform 6 3. Image denoising and enhancement using the nonsubsampled contourlet transform 7

8 Good Things about Mapping [Cunha] How about Vanishing Moments? It preserves perfect reconstruction If mapping is symmetric, then resulting - filter also symmetric easy linear phase FIR design Fast implementation through lifting factorization Easy control of the frequency response through the mapping filter: If Q (x) = Q ( x) then H (z) will have the complementary response of H (z). Filters can be symmetric vertically and horizontally allows for simple symmetric extension. Proposition 3. [Cunha:4] Suppose P(x) and Q(x) are such that P(x)Q(x) + P( x)q( x) = and further that P(x) = (x + x) N PR P (x), Q(x) = (x + x) N QR Q (x). Then if we set f(x,y) = x + (x + ) N x (y + ) N y f(x, y) the - function P(f(x, y)) will have zeroes of multiplicity N x N P and N y N P at x = and y = respectively. Similarly Q(f(x,y)) will have zeroes of multiplicity N x N Q and N y N Q at x = and y = respectively. 3. Image denoising and enhancement using the nonsubsampled contourlet transform 8 3. Image denoising and enhancement using the nonsubsampled contourlet transform 9 esign Mapping Function f(x,y) for Pyramid Filters esign Example for Pyramid Filters simple form of f(x, y) is by using maximally flat filters: L N P N,L (x) := ( + x) N l= ( ) N + l N l ( x) l l Then set f(x, y) = + P N,L (x)p N,L (y) so that f(,) = Top: nalysis; ottom: Synthesis 3. Image denoising and enhancement using the nonsubsampled contourlet transform 3 3. Image denoising and enhancement using the nonsubsampled contourlet transform 3

esign Mapping Function f(x,y) for Fan Filters enoising Result: Hat Zoom We can use diamond maximally flat filters and then modulate to get fan filters: N N i N f N (x, y)

N j ( x) i ( + x) N i ( y) j ( + y) N j N i ( x) i ( + x) N i ( y) i ( + y) N j Comparison against SI-Wavelet (nonsubsampled wavelet transform) methods using ayes shrink

Image denoising and enhancement using the nonsubsampled contourlet transform 3 3.

9 esign Mapping Function f(x,y) for Fan Filters enoising Result: Hat Zoom We can use diamond maximally flat filters and then modulate to get fan filters: N N i N f N (x, y) = i= j= N N N + i i= N i Then set f(x,y) = + f N (x, y) as before. N j ( x) i ( + x) N i ( y) j ( + y) N j N i ( x) i ( + x) N i ( y) i ( + y) N j Comparison against SI-Wavelet (nonsubsampled wavelet transform) methods using ayes shrink with adaptive soft thresholding [Cunha] Noisy Lena SI-Wavelet NSCT en PSNR.3db PSNR 3.8d PSNR 3.4d f N : N= N=3 N=6 3. Image denoising and enhancement using the nonsubsampled contourlet transform 3 3. Image denoising and enhancement using the nonsubsampled contourlet transform 33 enoising Result: Peppers enoising Result: arbara Noisy Lena SI-Wavelet NSCT en PSNR.4db PSNR 3.38d PSNR 3.53d Noisy arbara SI-Wavelet NSCT en PSNR.5db PSNR 9.34d PSNR 9.95d 3. Image denoising and enhancement using the nonsubsampled contourlet transform Image denoising and enhancement using the nonsubsampled contourlet transform 35

Comparison gainst SI-Wavelet Method nother pplication: Image Enhancement [Zhou] Lena PSNR (d) Peppers PSNR (d) Std Noisy SI-Wavelet NSCT Noisy SI-Wavelet NSCT σ = 8.3 35. 35.7 8.7 34.3 34.3 σ =.3 3.83 3.

10 Comparison gainst SI-Wavelet Method nother pplication: Image Enhancement [Zhou] Lena PSNR (d) Peppers PSNR (d) Std Noisy SI-Wavelet NSCT Noisy SI-Wavelet NSCT σ = σ = σ = σ = PSNR results of the tested denoising schemes for different noise levels. Image Enhancement lgorithm... Nonsubsampled contourlet decomposition Nonlinear mapping on the coefficients Zero-out noises Keep strong edges or features Enhance weak edges or features Nonsubsampled contourlet reconstruction 3. Image denoising and enhancement using the nonsubsampled contourlet transform Image denoising and enhancement using the nonsubsampled contourlet transform 37 Image Enhancement Result: arbara Image Enhancement Result: OCT Image (a) (a) (b) (a) Original image. (b) Enhanced by WT. (c) Enhanced by NSCT. (c) (b) (a) Original OCT image. (b) Enhanced by WT. (c) Enhanced by NSCT. (c) 3. Image denoising and enhancement using the nonsubsampled contourlet transform Image denoising and enhancement using the nonsubsampled contourlet transform 39

Outline Speculation: nother Wavelet Story?

(signal processing). Critically sampled (CRISP) contourlet transform 3.

Outlook Multiresolution nalysis Theory lgorithms pplications - Story: scale and shift 4. Outlook 4 4.

11 Outline Speculation: nother Wavelet Story?. Image modeling using the contourlet transform Wavelets (harmonics analysis) Pyramids (computer vision) Filter banks (signal processing). Critically sampled (CRISP) contourlet transform 3. Image denoising and enhancement using the nonsubsampled contourlet transform 4. Outlook Multiresolution nalysis Theory lgorithms pplications - Story: scale and shift 4. Outlook 4 4. Outlook 4 Speculation: nother Wavelet Story? Where is The Complexity in Images? Curvelets (harmonics analysis) Hough transform (computer vision) irectional filter bank (signal processing) irectional Multiresolution nalysis Theory lgorithms pplications - Story: scale, space, and direction 4. Outlook 4 4. Outlook 43

12 65 6 Operational Rate-istortion by JPEG- Lenna 5x5 istortion/rate using JPEG arbara 5x5 istortion/rate using JPEG 65 f f u u v 6 v Opt. u+v Opt. u+v eyond - and Single Image... New udio-visual Paradigm PSNR log(55 /MSE) PSNR log(55 /MSE) Existing audio-visual recording and playback Single camera and microphone Little processing Viewers: passive Compression Rate (bits/pixel) Compression Rate (bits/pixel) Future Percent of bits used by u in u+v Percent of bits used by u in u+v Sensors (cameras, microphones) are cheap Massive computing and storage capabilities are available Viewers: interactive, immersive, remote 6 6 Percent (%) 5 4 Percent (%) 5 4 Require new signal processing theory and algorithms Compression Rate (bits/pixel) Compression Rate (bits/pixel) Note: Video can be treated as a special case (frames in one shoot multiple images of a scene) 4. Outlook Outlook 45 Image-based Rendering Light-field Parameterization Recording camera plane image plane object surface Processing Playback t v L(u,v,s,t) rchive u s L(u,v,s,t ) Goal: Synthesize arbitrary viewpoint from a set of fixed sensors. Input data at a huge rate, for example: M/frame frame/s 5 views =.8 Gbps. Challenge: New representations for IR data that incorporate geometric constraints. Each light ray is addressed by a 4- coordinate (u,v, s,t). ssuming the object surface is Lambertian, then L(u, v, s, t) = L(u, v, s, t ). The 4- light field data L(u,v, s, t) lie in lower dimensional manifolds. 4. Outlook Outlook 47

o, CRISP-contourlets: a critically sampled directional multiresolution image representation, Proc. of SPIE conf. on Wavelet pplications in Signal and Image Proc.

13 t Epipolar Constraint from Multiple Views u epipolar image References..-Y. Po and M. N. o, irectional multiscale modeling of images using the contourlet transform, IEEE Trans. on Image Proc., to appear. Y. Lu and M. N. o, CRISP-contourlets: a critically sampled directional multiresolution image representation, Proc. of SPIE conf. on Wavelet pplications in Signal and Image Proc., San iego, US, ugust 3. Linear singularities in epipolar planes... ridgelets? s Software and downloadable papers: minhdo Curved singularities in image planes... contourlets? Ultimate: High dimensional representations that can deal effectively with lower dimensional singularities

Beyond Wavelets: Directional Multiresolution Image Representation

Beyond Wavelets: Directional Multiresolution Image Representation Minh N. Do Department of Electrical and Computer Engineering University of Illinois at Urbana-Champaign www.ifp.uiuc.edu/ minhdo minhdo@uiuc.edu