Time Domain Lapped Transform and its Applications

Transcription

1 Time Domain Lapped Transform and its Applications Jie Liang School of Engineering Science Vancouver, BC, Canada BIRS Workshop on Multimedia and Mathematics Acknowledgements rof. Trac D. Tran, The Johns Hopkins University Dr. Chengjie Tu, Microsoft Corporation Dr. Lu Gan, The University of Newcastle, Australia Banff, AB, Canada, July 23-28, 2005

2 Introduction Outline From filter bank to time-domain lapped transform (TDLT) Fast TDLT re/post-filtering for 2D and 3D Wavelet transform Error Resilient TDLT Current Works Summary 2

3 3 M M M ] [ ˆ n x ) ( F 0 z ) ( F z ) ( F M z Filter Bank Fundamental M M M x[n] ) ( H 0 z ) ( H z ) ( H M z rocessing ( ) ) ( ) ( ) ( z z ) ( -K - 0 z z z z K X X Y = = + K = M M O O O L L L O O M M m m m K K K m m m x x x y y y = K m K m m m x x x y K

4 Filter Bank Fundamental x[n] H 0 ( z) M M F 0 ( z) H ( z) M M F ( z) H M ( z) M M F M ( z) xˆ [ n] x ( z ) (z) erfect Reconstruction: R (z) (z) = I, or R (z) = - (z). Fast Implementation: Factorization of (z) x 0 M- y ( z ) R (z) (z) G 0 ( z ) G ( ) z G K ( z) m xˆ ( z ) y m 4

5 Linear hase Filter Banks Linear phase: Desired property for image/video coding General structure [Vaidyanathan93, Gao0, Gan0] U 0 V 0 z z V G ( ) G ( z) K 2 z G G ( z ) 0 U i, Vi : Invertible matrices. Can be optimized for different applications. 5

6 Rate-Distortion Optimization Objective: Design the filter bank to minimize the MSE for a given bit rate. - x (z) (z) xˆ = x+ e δ 2 = trace { R } ee Q Design Criterion: Coding Gain MSE reduction of transform coding w.r.t. CM 2 σ CM γ = 0 log0 2 σ 6

7 A Special Case: Block Transform x m Q xˆ m (z) x m+ Q ˆ + x m Karehunen-Loeve Transform (KLT): Optimal block transform Signal dependent, No fast algorithm Discrete Cosine Transform (DCT): Fast approx. of the KLT for AR() signals. Drawbacks of block transform: Blocking artifact Limited compression capability 7

8 Lapped Transform [Malvar et al. 985] Apply post-processing of the DCT to Improve compression efficiency and reduce blocking artifact A special case of linear phase filter banks DCT DCT DCT V V Q Q V V IDCT IDCT IDCT post-processing pre-processing 8

9 Time-Domain Lapped Transform [Tran-200] DCT Q IDCT V DCT Q IDCT V V V DCT Q IDCT More compatible to DCT-based schemes Also a special case of linear phase filter banks Adopted by MS WMV-9, SMTE VC-, HD-DVD. 9

10 Effect of refiltering A flattened image 0

11 DCT vs LT: Basis Functions 8-point DCT 8 x 6 TDLT

12 Frequency Responses 8-point DCT 8 x 6 TDLT 8.83 db 9.6 db DC Att , Mirr Att , Stopband , Coding Gain = db 5 Magnitude Response (db) Normalized Frequency DC Att , Mirr Att , Stopband.9696, Coding Gain = 9.65 db 5 Magnitude Response (db) Normalized Frequency 2

13 Applications & Generalizations Fast TDLT [Liang et al. 200] re/ost-filtering for Wavelet [Liang et al. 2003] Error Resilient TDLT [Tu et al. 2002, Liang et al. 2005] Generalized Lapped Transform [Liang et al. 2002] Adaptive Entropy Coding for TDLT [Tu et al. 200] Oversampled TDLT [Gan-Ma-2002] Undersampled TDLT [Tu et al. 2004] Regularity Constrained TDLT [Dai et al. 200] Adaptive TDLT [Dai et al. 2005] 3

14 Introduction Fast TDLT Outline re/post-filtering for 2D and 3D Wavelet transform Error Resilient TDLT Current Works Summary 4

15 M x M refilter: Fast Orthogonal TDLT 0 V = V General structure of V: T V M/2 M- θ 4 θ 3 θ 2 θ θ 0 cosθ sin sin θ θ θ 5 cosθ 5

16 Fast Orthogonal TDLT Quasi-optimal coding gain [Liang-Tran-Tu-0]: 9.26 db for M = 8 Close to optimal filter bank Can be generalized to large block size (e.g., 28) -0.7π -0.2π -0.05π 6

17 V Fast Biorthogonal TDLT V T : More freedoms, better performance Fast approximation (lifting steps, LU factorization): s0 s s2 s3 0 U0 U 2 U2 Integer Solutions: > 0.3 db higher than orthogonal TDLT S0 S S2 S3 0 U0 U 2 U2 Gain 4/3 8/7 8/7 8/7 -/6 / 4 -/4 / 2-3/8 3 / /2 9/8 9/8 9/8 -/6 / 4 -/4 / 2-3/8 3 / / 4 -/4 / 2 - / 2 3 /

18 Image Coding erformance TDLT vs Wavelet Both coded by SIHT [Said, earlman, 996] (Improved entropy coding in [Tu, Tran, 200]) WT TDLT Lena 34 SNR Barb >.5dB Comp. Ratio 8

19 Image Coding erformance WT 32:: db TDLT 32:: db 9

20 Introduction Fast TDLT Outline re/post-filtering for 2D and 3D Wavelet transform Error Resilient TDLT Current Works Summary 20

21 JEG 2000 & Tiling Artifact No blocking artifact if WT is applied to the entire image Used by JEG 2000 roblem: Memory requirement Tradeoff: Tiling approach Tiling Artifact Tile size: 64 x 64, 0.2bpp 2

22 Average MSE of All Rows & Columns MSE is more than doubled at tile boundaries MSE X ix el MSE Y ix el 22

23 re/post-filtering for WT Apply small pre/post filters at tile boundaries W T Q I W T - W T Q I W T - W T Q I W T 23

24 roblem Formulation Effect of pre/post-filters: In LT: affect all subbands In WT: affect some subbands x7 x6 x5 x4 x3 x2 x x0 y6 y5 y4 y3 y2 y y0 γ (+ ) γ (+ z ) z 2 5/3 WT - k k 2 ˆx ˆx ˆx ˆx ˆx0 L H 24

25 { { Jie Liang roblem Formulation Boundary Filter Bank Optimization can be performed similar to LT { x 0 { y 0.. u 0 u 0 rocessing F 0.. G 0 -. y 0 { x y {... rocessing. F G u u. y 25

26 Fast Structure Optimal pre/post-filters for WT and DCT are similar /2 /2 S0 S SN-2 U N-2 /2 /2 /2 /2 /2 SN- U 0 /2 Examples: S = [5/3, 4/3, 6/5, 9/8], U = [/8, /4, 5/8], S = [2,,, ], U = [/8, /4, /2], (lossless) 26

27 Examples 9/7 WT, 8 x 8 re/ost Filters: 0.2bits/pixel JEG 2000 (Kakadu) JEG 2000 & re/ost db db 27

28 Average MSE of All Rows and Columns JEG 2000 (Kakadu) JEG 2000 & re/ost MSE 200 MSE Xixel X ixel MSE 00 MSE Y ixel Y ixel 28

29 Applications in 3D WT Video Coding Divide video sequence into groups of frames Apply 2D WT within each frame Apply another D WT in temporal direction WT WT WT Advantages: Lower complexity (no motion estimation) Full scalabilities: SNR, resolution, and frame rate. 29

30 Jittering Artifact erformance degradation at group boundaries 44 frames 6 frames per GO 20 : SNR (db) Solution : Global WT [Xu et al. 2002] roblems: Memory, random access, error resilience Avg. SNR: db, STD: 0.7 db Frame 30

31 re/ost-filtering for 3-D WT Apply pre-filter before WT, and post-filter after IWT revious pre/post design can be directly applied. re re WT WT WT 3

32 Comparison with GO and Global WT 37 Claire.qcif at 20 : Group: 6 frames 6-tap pre/post filter 9/7 WT Up to 2.5 db gain at boundaries SNR (db) Global WT (35.67dB) Filtered GO WT (35.57dB) GO WT (35.32dB) Frame 32

33 Introduction Fast TDLT Outline re/post-filtering for 2D and 3D Wavelet transform Error Resilient TDLT Current Works Summary 33

34 Error Concealment and Error Resilience Compressed bit stream is sensitive to transmission error/loss Traditional solutions: Channel coding, Retransmission Not always acceptable Human visual system can tolerate some errors: Error concealment at the decoder is preferred. Error resilient encoder: Encoder introduces some redundancies to facilitate concealment at the decoder. Lapped transform is a good candidate 34

35 Motivation for Error Resilient LT Encoder: DCT DCT DCT V V Can spread the information of each block into two blocks Decoder: rediction of the lost block is easier Conflicting requirements: Compression Error resilience Trade-off required 35

36 Error Resilient Lapped Transform [Hemami996] First error resilient LT design. Encoder: Trading compression for error resilience. Decoder:. Estimate lost blocks by mean reconstruction method: 2. Apply inverse lapped transform. yˆ yˆ + yˆ /2 ( ) n n n+ [Chung-Wang999, 2002] Multiple description coding Maximal smoothness reconstruction Improved visual quality 36

37 Error Resilient TDLT [Tu-Tran-Liang-2003] Only need to design the pre/post-filters Old approach: M x 2M unknowns re/post-filter: M/2 x M/2 More flexibilities: Biorthogonal filter Non-perfect-construction design Limitation: Mean reconstruction is still used I D C T I D C T sˆn sˆn+ /2 /2 - T T - 37

38 General Wiener Filter Solution Existing Method General Framework [Liang et al. 05] yˆ n yˆ n yˆ n+ I D C T I D C T sˆn sˆn+ /2 /2 - T T - xˆ n xˆ n+ yˆ n xˆ n xˆ n+ sˆn ˆn+ General framework: Estimate and from and min MSE=tr yˆ n+ I D C T I D C T sˆ n sˆ n+ - H - {( T )( ˆ ) } H R R R = E Hs - x Hs - x ee { R } ee 2M 2M = x sˆ sˆ sˆ xˆ n xˆ n+ Expression can be found if Rxx is known (e.g., AR()). s 38

39 Special Cases Divide H into two stages: yˆ n yˆ n+ s s ˆn ˆn+ I D C T I D C T H sˆn - T T - xˆ n xˆ n+ H : M x 2M Wiener filter (Near optimal performance) Link to previous approach: H = [I I] / 2 Mean reconstruction method. 39

40 Design Criteria for Optimization Use Matlab to find optimal pre-filter and post-filter Trade coding performance for error concealment. Objective function for optimization: maximize ε = (CG) - α (CR) + β (RG) Coding Gain (CG): MSE when there is no transmission error Concealment Residual (CR): The MSE after transmission error and error concealment Reconstruction Gain (RG) [Hemami96] Control the distribution of error to improve visual quality 40

41 Design Example MSE MSE at Each ixel (CG: 8.42 db) TDLT: 0.8 Old: 0.5 New: 0.06 Cfg : - Coding gain optimized filters - M x 2M Wiener filter Cfg 2: - Joint optimized filters - Mean reconstruction - [Tu et al 03] ixel Cfg 3: - Joint optimized filters, - M x 2M Wiener: 60% less than the mean reconstruction method. 4

42 Simulation Results (52 x 52 Lena, bpp, 50% loss) Loss attern Cfg : 24.3 / 40. db Cfg 2: 26.0 / 38.3 db Cfg 3: 30.5 / db 42

43 Introduction Fast TDLT Outline re/post-filtering for 2D and 3D Wavelet transform Error Resilient TDLT Current Works Summary 43

44 Multiple Description Coding An alternative to improve the robustness to transmission error: Create multiple (equally important) output bit streams. Each stream alone can give a coarse reconstruction. Quality can be improved if more descriptions are received. One block Group Entropy coding MDC Image TDLT Transformed Image Group 2 Group 3 Entropy coding Entropy coding MDC2 MDC3 Group 4 Entropy coding MDC4 Error scenarios (5 cases): 44

45 2-D Error Concealment Current method: -D prediction Average of row and column results Ideal method: Joint prediction from 2-D neighbors How to predict? How to design the pre/postfilters? 45

46 Difficulties Related work: Edge directed prediction [Li-Orchard-2002] But geometrical structure is disturbed by prefiltering ixel-by-pixel approach may not work refiltered image After IDCT (with loss) 46

47 Summary DCT Q IDCT V DCT Q IDCT V V DCT Q IDCT V Time domain lapped transform Fast algorithms Applications in 2D and 3D WT Error resilient design Comparison to JEG 2000: Lower complexity Competitive performance romising for handheld devices 47

48 References T. D. Tran, J. Liang, and C. Tu, "Lapped transform via timedomain pre- and post-filtering," IEEE Trans. on Signal rocessing, vol. 5, No. 6, pp , Jun J. Liang, C. Tu and T. D. Tran, "Optimal pre/post-filtering for wavelet-based image and video compression," IEEE Trans. on Image rocessing, to appear. J. Liang, C. Tu, T. D. Tran and L.Gan, Wiener filtering for generalized error resilient time domain lapped transform," 2005 IEEE Int. Conf. on Acoustics, Speech, and Signal rocessing, hiladelphia, A, Mar C. Tu, T. D. Tran and J. Liang, "Error resilient pre-/post-filtering for DCT-based block coding systems," IEEE Trans. on Image rocessing, to appear. C. Tu and T. D. Tran, "Context based entropy coding of block transform coefficients for image compression," IEEE Trans. on Image rocessing, vol., pp , Nov