Jordi Cenzano Ferret. UPC Barcelona (June 2008)

Transcription

1 MULTIVIEW DEPTH VIDEO CODING USING 3D WAVELET ABSTRACT Jordi Cenzano Ferret UPC Barcelona (June 28) In this work we will propose a multiview depth video coding (MDVC) using a 3D wavelet coding scheme. The 3D wavelet coding scheme could code a video sequence using the redundancies in 3 dimensions: horizontal space(x), vertical space (Y), and view (V) without block matching, inherits the space scalability of discrete wavelet transform (DWT), and depending on coding order it can be time scalability as well. First we will introduce the 3D DWT and some of its properties applied to video coding, after that we propose a MDVC based on 3D DWT. We will do some experiments compressing and decompressing multiview depth video sequences with and without losses, to do so we ll use the Haar wavelet function, different wavelet levels, and different quantization values. To measure the coding quality of the presented scheme we will use the Peak Signal to Noise Ratio (PSNR). The experiments have shown that we can achieve compression ratios around 6 with reasonable image quality. This means that we need to improve this method. One idea for future work is to use motion estimation between views or time or both. Coding the bitstream properly we can maintain the scalability on 3 of 4 th signal dimensions and we ll increase the compression ratio. a[ d[ (a) Analysis k k k 1 (b) Synthesis h [2n h [2 h [2n h [2 1 k h [ n 2 a[ + h [ n 2 d[ Note: The h and h1 are the wavelet functions converted into filter impulsional responses Figure 1: DWT computed as filter banks The filter banks scheme should be iterated over a[ (low band signal approximation). In the Figure 2 we see a typical DWT scheme of 2 levels for 1 dimension signal coding using Haar as wavelet function INTRODUCCIÓN In this document we will propose a multiview depth video coding (MDVC) using a 3D wavelet coding scheme. The core of this coding scheme is the Discrete Wavelet Transform (DWT)[1][2]. In order to understand properly the proposed scheme, first we will introduce the DWT and some of its properties applied to image (or video) coding. The DWT is an invertible and non-expansive transformation that has the separability property, a valuable property for image coding. There are different ways to compute the DWT, we will use the Mallat algorithm in order to calculate DWT as filter banks (see Figure 1).

2 and then we will see clearly that the properties of DWT are very interesting for image coding (Figure 3) a tmp [ (Original signal) d 11 [ Figure 3: DWT of 1 levels for 2D signals (Images) The resultant image is the composition of the outputs of 2D DWT coder. The outputs of this coder are four images of quarter resolution of original one. The I is the original one downsampled and low pass filtered, and the other three are the horizontal details, vertical details, and diagonal details. This means that I can transmit the complete 2D DWT result for high resolutions displays, but if my display has a low resolution screen only need to decode a part of the bitstream. This is space scalability. And remember that we can implement more transformation levels iterating over I. And using again the separability property we can extend the DWT from 2D to 3D (DWT 3D), see Figure a [ d 1 [ Figure 2: DWT analysis of 2 levels for 1D signals We see that the signal a [ is very similar to the original signal but is quarter sampled and low pass filtered. The other resultant signals d 1 [ and d 11 [ are the details that a [ needs to be converted, again, into the original one through Inverse Discrete Wavelet Transform (IDWT). We have to note that we can concatenate a [, d 1 [, and d 11 [ and we have the same number of samples than, this demonstrate the non-expansive property of this transformation. If we want, we can continue iterating over a [, the only limit is length of the a [ signal, we need more samples than the filter impulsional response to calc the DWT. We can use the separability property to apply, in a very simple way, the same scheme into a 2D signal (image),

3 Finally we have used a common arithmetic coder/decoder [3] to encode/decode the quantized coefficients to/from the bitstream. 2. EXPERIMENTS Figure 4: DWT of 1 levels for 3D signals (video) Once we have introduced our 3D DWT block, now we can present our complete codec scheme. Compose To test our coder we have used the depth images of the sequences ballet and breakdancers [3] from Microsoft research web page. These sequences are recorded using 8 cameras, and after that they have calculated the depth images for each one. The original sequences have 99 frames per camera and the frame resolution is 124x768 at 8bpp. To check our algorithm, due to computational reasons, we have used only the first frame of 4 cameras (,2,4,6) and we have downsampled the original images to 512x384 8bpp (factor of 2 in H and V dimensions, without filtering). (a) Ballet (b) Breakdancers Figure 5: 3D DWT complete codec scheme We can see that the result signal of 3D DWT, the DWT coefficients, are quantized, the quantizer is the only block that has loses, if we eliminate the quantizer (Pr ) of our scheme, the scheme become in a lossless codifier. The quantizer block follows the simple algorithm described in Figure 6. If (Pr > ) Iwq round(iw * Pr)) Else Iwq Iw If (Pr > ) Iw Iwq / Pr Else Iw Iwq (a) Quantizer algorithm Figure 7: Input signal To measure the similarities between original images and reconstructed images we have used the PSNR [4], see the definition in Figure 8. PSNR 1Log1 M M N i 1N 1 j MAX Figure 8: PSNR Definition 2 I I( i, j) K( i, j) 2 (b) de-quantizer algorithm Figure 6: Quantizer algorithm

4 To have a baseline (lossless) to compare our proposed coding method we have compressed, using an arithmetic coder (winrar[5]), the four original downsampled images obtaining the following results: - Original size: Bytes (512x384, 8bpp, 4cams) Ballet sequence: - Rar file size: Bytes - Compression ratio (CR) 768k/111K 6.9 Breakdancers sequence: - Rar file size: Bytes - Compression ratio 768k/124K 6.2 Now we know that we can achieve, using a typical arithmetic compressor, a CR about 6. In the experiments we have used a Haar wavelet function. The Haar wavelet has a really short impulsional response, ant it can be used over short signals (in View dimension we only have 4 samples). We have used two levels of wavelet coding: 1 and 2. Finally we have changed the quantification step (Pr): - : No losses. - 1: Step of : Step of : Step of 1. Breakdancers sequence (512x384 8Bpp) Cams:,2,4,6 Original size: 768,432 Bytes Wavelet function: Haar (2 coefficients) Wavelet levels: 1 Pr PSNR (db) CR Wavelet levels: 2 Pr PSNR (db) CR Table 2: Ballet sequence data In the Figure 9a we can see a part (zoom) of the reconstructed image of the ballet sequence using the Haar wavelet, 1 levels, and without the quantifier block (Pr ), the signal is exactly the original (PSNR db). In the Figure 9b we can see the same part of the image codified using Haar wavelet with 2 levels and Pr.1 (PSNR 44.8dB). And finally in the Figure 9c there is the same reconstructed part of the image with Haar wavelet, 2 levels and Pr.1 (PSNR 28.9dB). We see that the reconstructed image with a PSNR44.8dB has a subjective quality quite good, and it seems that will be enough to use it as 3D depth image. We can see that the image with a PSNR 28.9dB has a lot of artifacs and it would create a lot of problems to use it as 3D depth image. 3. RESULTS In the following tables we present the results calculated using the matlab environment: Ballet sequence (512x384 8Bpp) Cams:,2,4,6 Original size: 768,432 Bytes Wavelet function: Haar (2 coefficients) Wavelet levels: 1 Pr PSNR (db) CR Wavelet levels: 2 Pr PSNR (db) CR Table 1: Ballet sequence data

5 5. FUTURE WORK (a) PSNR db First of all we have to check this scheme with other wavelet functions (ex: Daubechies), and more wavelet levels. And obviously check all these parameters against as much test sequences as possible. To do so we would need a more powerful computation environment. To increase the CR of this scheme we can could implement a motion estimation block in time or /and view dimension. Other improvement can be use, instead of the arithmetic coder, a coder adapted to wavelet compression, like SPITH coder. 6. REFERENCES [1] Philippe Salembier, Wavelets: Theory and applications, Apuntes, ETSETB / TSC / UPC, Barcelona, Primavera 26. [2] Martin Vetterli y Jelena Kovacevic, Wavelets and subband coding, Prentice hall, Pages: New Jersey, (b) PSNR 44.8dB [3] p/3dvideodownload, Microsoft research, [4] Wikipedia, [5] Winrar, (c) PSNR 28.9dB Figure 9: Zoom of different reconstructed images (ballet sequence) 4. CONCLUSIONS We can archive CR around 6 with reasonable image quality. Too poor to use an image based scheme. The depth images need a high PSNR in order to avoid 3D artifacts. The proposed scheme needs more work to improve the CR and the image quality. The proposed method has scalability in all dimensions (X,Y,t,V). A valuable property for nowadays media technology, where a media creator needs to reach as much displays as possible with very different features (mobile telephone, PDA, plasma TV, car LCD, etc ).