A Comparative Study of Depth-Map Coding Schemes for 3D Video

Transcription

1 A Comparative Study of Depth-Map Coding Schemes for 3D Video Harsh Nayyar Nirabh Regmi Audrey Wei Abstract This work presents a comparative study of depthmap coding schemes for 3D Video. We first investigate two blockbased transform coding approaches: the DCT, and a trained KLT scheme. We then present a novel approach to depth map compression. By applying Block Truncation Coding (BTC) with fixed block sizes within a frame, we are able to outperform both the DCT and the KLT with respect to rate and distortion. Our key contribution is the design of an adaptive Block Truncation Coding scheme (A-BTC) that utilizes the Lagrangian optimization framework to adaptively select the BTC block-size. Our results demonstrate that the A-BTC approach generally outperforms all of the other techniques examined by our work. Keywords 3D video compression; depth-map compression; adaptive block truncation coding (A-BTC); BTC; DCT; KLT I. INTRODUCTION 3D Televisions have recently begun to gain traction in the marketplace, with a variety of stereo displays now available from many manufacturers. Although such displays are an important advance, they are encumbered by the need for special eye-wear and a limited true look-around effect. The vision for true recreation of a 3D scene is likely to be achieved by future autostereoscopic displays. By emitting a large number of views, such displays are not only able to overcome the dependence on special eye-wear but they also provide the look-around effect that viewers expect from a realistic 3D scene. Given that autostereoscopic displays require a large number of views (potentially greater than 50), efficient compression techniques are critical for practical systems. In [1] Müller et al. show that encoding these views using a scheme such as the multi-view coding (MVC) profile of H.264/AVC results in a bitrate that is linearly increasing with the number of views. This can rapidly become infeasible as the required number of views increases. One proposed solution [1] to this issue is to encode two or three views along with the corresponding depth maps. Such a pipeline is envisioned to allow for the synthesis of an arbitrary number of intermediate views. The critical component in the success of such an approach is the manner in which the depth maps are encoded. This work assesses the relative performance of the Discrete Cosine Transform (DCT), the Karhunen-Loève Transform (KLT), and Block Truncation Coding (BTC) as applied to depth-map compression. This paper further presents a novel Adaptive Block Truncation Coding (A-BTC) technique that incorporates the Lagrangian optimization framework. This approach adaptively selects the block-size with which the BTC is applied to the depth-maps. In order to focus our work, we restrict the scope of our analysis to the spatial dimension, and ignore any potential gains that may arise from exploiting temporal redundancy. In the next section, we formally present the problem this work addresses. Section III describes some related work in this area. Section IV outlines our evaluation framework. Sections V and VI describe the application of the DCT and the KLT to depth-map compression respectively. Section VII presents BTC as a better approach to compressing depth-maps, while section VIII describes and evaluates our Adaptive Block Truncation Coding scheme. Section IX compares and contrasts the key findings of our study, and the paper concludes in Section X. II. PROBLEM DESCRIPTION The intermediate view-synthesis stage is highly dependent on the quality of the input depth-maps. In particular, the edges present in the depth-maps highly influence the quality of the synthesized intermediate view. As a result, it is important to design a coding scheme that preserves these edges, while achieving the required compression. III. RELEVANT LITERATURE Several different approaches exist in the literature for efficient coding of depth-maps with some giving particular attention to the preservation of edges. In [2] the authors present edge-adaptive transforms (EATs) as an alternative to the DCT to encode depth maps. The EATs avoid filtering across edges in each image block and so do not create high coefficient values. This new method when selectively chosen along with the DCT helps in decreasing the bitrate. In [3], the authors use a shape-adaptive wavelet transform and an explicit encoding of the locations of major edges to encode depth maps. The paper presents a novel approach to generate small wavelet coefficients along edges to reduce the bits required to code the depth-map. Platelet coding as described in [4] is an edge-aware coding scheme that uses a segmentation procedure based on a quadtree decomposition to model the depth map with piecewise linear functions. This scheme is efficient at causing significantly less degradation along edges.

2 IV. EVALUATION FRAMEWORK In order to fairly compare the different schemes that are investigated, we evaluate each scheme by applying our test sequences to the system described below. A. System Overview Figure 1 depicts the top level view of the system that [1] describes. We use this framework in order to evaluate the performance of our compressed depth-maps from each scheme. B. Test Sequences Fig. 1. System level overview. We use the Kendo and Balloons multi-view sequences available from the Tanimoto Laboratory [7]. Specifically, we use the sequences from Cameras 1 and 3 and their associated depth-maps to synthesize an intermediate view corresponding to Camera 2. These sequences are 1024 x 768. For reference, Figure 2 shows the luminance component of the first frame of each Camera 2 sequence and the associated depth-maps: In the equations above, it is assumed that the dimension of the test sequence frames is n x m (width x height). It is important to note that we synthesize a reference intermediate view from uncompressed depth-maps. It is with respect to this reference synthesized view that we measure our distortion. The prime denotes the luminance component resulting from synthesis performed from the compressed depth-maps. For a sequence of frames, the PSNR that we report is the average of each frame. Finally, we perform entropy coding on each scheme in order to ensure a consistent and fair comparison of rates amongst the schemes we compare. V. DISCRETE COSINE TRANSFORM The Discrete Cosine Transform (DCT) is widely used in image compression as it is easily implementable, computationally efficient and achieves a compression factor that is comparable to an optimum transform, the KLT [5]. Another advantage is that the DCT decorrelates the input signal in a data-independent manner. For these reasons, we find the DCT being worthy of comparison in our comparative study. In our implementation, we first apply the DCT-II with blocksizes (M) of 8 and 16. We then apply a uniform mid-tread quantizer with step-sizes varying from 2 1 to 2 8. We construct the pmf for each quantized DCT coefficient, and use it to determine the entropy code for the sequence. The results for our test sequences are shown in Figure 3: Fig. 2. Luminance (top) and example depth-map (bottom) of Balloons (left) and Kendo (right) from Camera 1. C. Performance Metrics We calculate PSNR as follows: MSE = 1 nm n m i=1 j=1 ( ) P SNR = 10 log MSE ( Y ij Y ij ) 2 Fig. 3. Rate-Distortion performance for DCT based compression with M=8,16 for the first 50 frames in Balloons and Kendo. The 16x16 DCT seems to be marginally better than the 8x8 DCT for coarse quantization and worse for finer quantization. However, a general claim for different block-sized DCTs can not be advanced as we compare the distortion in the synthesized images, rather than the depth-maps. The relative rate-distortion performances between block-sizes is not clear and the basic assumption that higher block sizes yield lower rate and PSNR is not valid for transform coding in depth map compression. In Figure 4, the difference of the reconstructed and the original intermediate views is depicted.

3 interesting cross-over points in the curves. Specifically, the results show that the larger block-size of 16x16 outperforms the 8x8 block for coarse quantization. Meanwhile, for finer quantization, the smaller block-size yields better performance. We believe that it is possible to explain this behaviour by the uniform mid-tread quantizer that we apply. Because we are applying uniform quantization to all of the coefficients, at higher step sizes, we stand to lose relatively more information in 8x8 blocks as compared to 16x16 blocks. Conversely, at lower quantization step-sizes, the benefits of a smaller block dominate and allow the 8x8 block to outperform the larger block size. The somewhat unpredictable nature of the performance suggests that the KLT is highly dependent on the training set. Fig. 4. Error visualization for Balloons (frames 1) synthesized using depth maps compressed with DCT M=8, Q=128. VI. KARHUNEN-LOÈVE TRANSFORM The Karhunen-Loève Transform (KLT) exploits the statistical properties of the depth-maps, causing it to be signal dependent and inseparable for image blocks. Conversely, it is optimal in the sense that the KLT achieves maximal decorrelation in the depth-map values, and also maximally compacts its information [5]. For these reasons, we find the KLT as another central comparison with the A-BTC described in Section VIII. Similar to how the DCT is performed, we perform our KLT scheme by using either M = 8 or M = 16 blocks followed by a uniform mid-tread quantizer with step sizes varying from 2 1 to 2 8. The training set is constructed by selecting each row of every MxM block for all frames in both views, and forming a matrix of samples as depicted in Figure 5. In this figure, m and n denote the width and height of the test sequences respectively while p denotes the number of frames. This is the input for the autocorrelation matrix from which we obtain the transform [6]. Finally, we construct the pmf (at each quantization step size) for each quantized DCT coefficient, and use it to determine the entropy code for the sequence. Fig. 6. Rate-Distortion performance for KLT based compression with M=8,16 for the first 50 frames in Balloons and Kendo. Finally, in Figure 7, the difference between the reconstructed and the original intermediate views is shown. Fig. 5. Construction of training set for KLT. Figure 6 illustrates the relationship between the 8x8 KLT and 16x16 KLT for the first 50 frames of both the Balloons and Kendo sequences. There are several important aspects of these results that merit further discussion. First, we notice that the Kendo sequence appears to achieve strictly better performance than the Balloons sequence. Next, we note that there are Fig. 7. Error visualization for Balloons (frames 1) synthesized using depthmaps compressed with KLT M=8, Q=128.

4 VII. FIXED BLOCK-SIZE BLOCK TRUNCATION CODING In this section, we first describe the basic Block Truncation Coding (BTC) algorithm as outlined in [8]. We investigate this scheme as its authors claim that it is good at preserving edges. We then describe how we calculate the rate for this scheme as we apply it to our depth-map application. Finally, we include our results for this scheme. A. BTC Algorithm The BTC is a simple two level non-parametric quantizer that adapts over local regions of the image by preserving local sample moments. It works on a block level where a block of size M is coded to two different quantized values. Several variations of mapping input values to these two levels using the BTC framework have been designed. In this paper we use a quantizer that preserves the first and second sample moments of the image. Let n be defined as the number of elements in a block of size M (n = M 2 ). The first and second moments and the variance of the block are defined as follows: size. We use this pmf to assign bits to the quantized values and achieve entropy. 2) Encoding the positions of a and b in the block: We interpret the quantized output of the BTC as a bitmap with a and b corresponding to 1 and 0 respectively. In order to exploit the statistical dependency of the bitmap ordering, we scan columns of the block to form a sequence, perform run-length coding, and assign bits according to the entropy of the run length vector. C. Results In this section, we perform the calculations on the first frame of the Balloons and Kendo sequences as we realize that the first frame results are representative of the sequences results. So, in order to save execution time, we perform our analysis on both of the first frames. X = 1 n X 2 = 1 n n i=1 n i=1 X i X 2 i σ 2 = X 2 X 2 where X i is the i th coefficient in the block. The threshold X th is set to X and the two quantized values (a and b) are selected as in the following pseudo-code: IF X i < X th, Xout i = a ELSE, Xout i = b Let q be defined as the number of X i in the block that are greater than X th. Then preserving the first two moments of the elements in the block, the two quantized values (a and b) can be defined as: B. Rate Calculation ( ) a = X q σ n q (n ) b = X q + σ q The rate calculation consists of two main components so that we are able to estimate the entropy in order to compare with our other schemes: 1) Entropy coding for a and b: We do this by calculating the pmf for all unique values of a and b for each block Fig. 8. Rate-Distortion performance for Fixed BTC scheme for Frame 1 in Balloons and Kendo with the fixed M varied between 2 1 and 2 6. Figure 8 depicts the performance we achieve as we vary the fixed block-size parameter between 2 1 and 2 6. We observe that as the block size increases, both the rate and PSNR decrease. Fig. 9. Error visualization for the balloons sequence Frame 1. M=64, M=16, M=2 (l-r). Finally, Figure 9 depicts the error in Balloons for M = 64, M = 16, and M = 2 respectively. There are two important trends here. First, we notice that as we decrease the fixed block-size, the error we observe decreases. Second, and perhaps more critically, we note that the error appears to be concentrated along the edges in the synthesized images.

5 VIII. ADAPTIVE BLOCK TRUNCATION CODING (A-BTC) In this section we first describe the motivation behind modifying the fixed block-size BTC to our proposed Adaptive Block Truncation Coding scheme (A-BTC). We further discuss how we incorporate the Lagrangian optimization framework into this scheme. We finally present and discuss the results that the A-BTC achieves. A. Motivation The R-D curves of Figure 8 reveal that the fixed block-size BTC performs significantly better with lower block-sizes but at a rate penalty. However, in Figure 9 we see that increased distortion for larger block sizes is actually concentrated around the edges. This suggests that more bits need to be spent to encode the depth-maps around the edges, while fewer bits could suffice to efficiently code other regions of the depthmaps. The A-BTC adaptively (according to the Lagrangian cost function) selects different block sizes for encoding different components of a depth-map: smaller block sizes for regions with edges and larger block-sizes for the remaining regions. B. Algorithm Overview This section presents a summary of the key steps in our proposed A-BTC scheme. 1) Perform the pre-processing stage to obtain necessary inputs to perform optimization. 2) For each block in a depth-map pair, recursively break down the block until the specified minimum block-size. 3) Evaluate the Lagrangian cost at each block-size level and return this to the parent block. 4) Compare the Lagrangian cost of using the parent block size and the 4 children blocks. Select the minimizing option. The specific details of the algorithm are left until the next section. In short, we recursively evaluate Lagrangian costs for the candidate block-sizes, and select the block-sizes that minimize the Lagrangian cost. C. Algorithmic Details The A-BTC can be considered as an addendum to our Fixed BTC scheme as described in Section VII. In other words, as the first step of this scheme, we perform the fixed block-size BTC for each block-size from M = 2 1 to M = 2 6, for each depth map pair (left, right). We then proceed to synthesize the intermediate views resulting from each compressed depth map pair. The above pre-processing stage provides us with the necessary inputs for performing the Lagrangian optimization. Our Lagrangian optimization seeks to minimize the standard Lagrangian cost function: J = D + λr with λ defined as: λ = 0.2Q 2 We can manipulate Q as we see fit in order to control the rate-distortion tradeoff. The distortion that we use in this formulation is the straightforward MSE distortion. We use the MSE calculation as defined in Section IV-C but on a per block basis rather than a per frame basis. We now address the rate calculation. This is performed as we describe in VII-B with two additional considerations. Because we are now adapting across varying block sizes (from M = 2 1 to M = 2 6 ), we must additionally encode this information. We make a conservative estimate and use a fixed length 3-bit code to encode this information. The second consideration has to do with the fact that we consider both depth-maps in a pair simultaneously. Consequently, we evaluate the rate for both depth-maps, and we use the mean rate to finally evaluate J. In order to normalize our cost calculation, we multiply the MSE by the number of pixels in the candidate block and compute the total number of bits required to encode a block. This allows us to directly compare the Lagrangian cost for a parent block with the sum of the Lagrangian costs for its four children blocks. D. Results We first implement the A-BTC scheme described above with the minimum block-size set to two, and the maximum block-size set to 64. Moreover, we vary Q between 2 0 and 2 8 to investigate different settings of lambda. Figure 10 shows the results we obtain for the first frame of the Balloons and Kendo sequences respectively. The results show that Kendo increasingly outperforms Balloons at higher rates, and our lambda range can effectively control the rate-distortion tradeoff. Fig. 10. Rate-Distortion performance for A-BTC scheme for Frame 1 in Balloons and Kendo with the maximum block-size set to M = 64 and minimum block-size set to M = 2. Q sweeps between 2 0 and 2 8 with increasing powers of 2. Next, we decide to investigate the effect of changing the maximum block-size available to the A-BTC scheme. We

6 again set the minimum block-size to two and vary Q between 2 0 and 2 8. We investigate setting the maximum block-size to 8, 16, 32, and 64. Figure 11 shows the results for these settings. We derive a very useful observation from these results: increasing the maximum block-size has the effect of shifting the ratedistortion curve to the left. Intuitively, by offering larger blocksizes to encode low-frequency background blocks, we attain a reduction in rate with no observable distortion penalty. At the same time, we must acknowledge that this effect appears to have a diminishing gain. Fig. 12. for M=8 Rate-Distortion comparison of BTC and Block Transform schemes rates, we observe a cross over in the performance. For these rates, our block transform schemes appear to outperform the Fixed BTC. This obviously requires further analysis. An initial hypothesis might be that this could be resolved by allowing more flexibility in the block sizes. We address this further in Section XI. Fig. 11. Rate-Distortion performance for A-BTC scheme for Frame 1 in Balloons as M max is varied between 8, 16, 32 and 64. IX. EVALUATION We now systematically evaluate the different schemes that we have described above. We first compare our block transform schemes against the fixed BTC scheme. We then evaluate the gain of the A-BTC over the Fixed BTC. A. Fixed BTC vs. Block Transforms In Figure 12, we compare the performance of the Fixed BTC with the performance of the DCT and KLT with blocksize M=8. Meanwhile, Figure 13 compares the performance of the Fixed BTC with the performance of the DCT and KLT with block-size M=16. First, we notice that the DCT outperforms the KLT for both M = 8 and M = 16. This suggests that the performance of the KLT scheme is likely rather sensitive to the training set. Constructing a better training set would ideally give us a KLT that outperforms the DCT. However, there is no obvious way to achieving this optimal training set. In the Figure 12 results, it is clear that the Fixed BTC scheme outperforms both of the block transform based schemes. At low rates (about 0.05bpp), we observe a gain of approximately 1.1dB for both test sequences. The results of Figure 13 are less conclusive. In particular, a key point to notice is that for the Kendo sequence, at low Fig. 13. Rate-Distortion comparison of BTC and Block Transform schemes for M = 16 B. A-BTC vs. Fixed BTC It is clear from the results of Figure 14 that the adaptive scheme that we propose holds promise. Specifically, we observe that as expected, we are able to trade off rate and distortion and achieve a performance that is strictly superior to the fixed block-size curves. X. CONCLUSION In this work, we conduct an evaluation of three main schemes for compressing depth-maps: DCT, KLT, and BTC.

7 Third, we might also investigate preserving higher order moments in the BTC. In addition, our results for the A- BTC suggest that increasing the maximum block-size has the effect of shifting the R-D curve to the left, improving R-D performance. We believe that this result merits further study. A study of the characteristics of different block sizes in different depth-map images would be useful in selecting the optimal set of block sizes. Finally, we would like to perform a comparison with another promising scheme for depth map compression: the Platelets method. This would involve implementing the method as described in [4]. Fig. 14. Rate-Distortion comparison of BTC and Block Transform schemes for M = 16 Our analysis confirms that depth maps cannot be treated as ordinary images; it is important to pay special attention to edges. Our work begins with an investigation of the DCT and KLT based block transform schemes. Our analysis reveals that the DCT achieves slightly better performance than the KLT. Moreover, we observe that there is a cross-over when comparing blocks of M = 8 and M = 16. The main focus of the work is a novel application of the BTC to our depth-map compression problem. Our first attempt, the Fixed BTC scheme, is able to generally outperform the block transform approaches. By observing that distortion is concentrated along edges, we leverage small blocks to encode this information and larger blocks to encode other regions. This observation leads to our proposed A-BTC method. We conclude this work with the observation that the A-BTC is a promising new approach to depth map compression. XI. FUTURE WORK In this work, we have proposed a promising scheme which we denote as the A-BTC. We believe that there exist several interesting future directions that can be investigated to further develop this work. First, we would like to improve our optimization framework. In this work, as described above, we calculate the Lagrangian cost function jointly based on both the left and right depth maps. This is sub-optimal. An optimal scheme would iterate through dependent depth maps until the block size decisions converge. Second, we also note that the Lagrangian optimization framework we propose is computationally expensive. Moreover, it will become even more so as we proceed in the direction described directly above. The key reason for this complexity is that the algorithm operates in a bottom-up fashion. We have conducted some initial work on developing a heuristic approach that works from the top-down and would like to explore this further. ACKNOWLEDGEMENTS The authors would like to sincerely thank Professor Bernd Girod and Mina Makar for their invaluable guidance in the development of this work. Moreover, we would like to extend our gratitude to the Tanimoto Laboratory of Nagoya University for their view synthesis software and their Balloons and Kendo multi-view test sequences. REFERENCES [1] K. Müller, P. Merkle, and T. Wiegand, 3-D video representation using depth maps, Proceedings of the IEEE, vol. PP, no. 99, pp. 1-14, [2] G. Shen, W.-S. Kim, S. K. Narang, A. Ortega, J. Lee, and H. Wey, Edge-adaptive transforms for efficient depth-map coding, in Proc. of 28th Picture Coding Symposium (PCS 10), Nagoya, Japan, Dec [3] M. Maitre and M. N. Do, Depth and Depth-Color Coding Using Shape-Adaptive Wavelets, Journal of Visual Communication and Image Representation., vol. 21, issue 5-6, July [4] Y Morvana, D. Farina and P.H.N.de With, Novel Coding Technique for Depth Images using Quadtree Decomposition and Plane Approximation, Visual Communications and Image Processing 2005 Proc of SPIE., vol. 5960, , doi: / [5] N. Ahmed, T. Natarajan, and K. R. Rao, Discrete cosine transform, IEEE Trans. Compiti., vol. C-23, pp , [6] Z. Li and M. Drew, Karhunen-Loeve Transform, in Fundamentals of Multimedia. Upper Saddle River. Pearson Education, 2004, ch. 8, sec pp [7] Tanimoto Laborotory, Nagoya University. [8] E. Delp and O. Mitchell, Image Compression Using Block Truncation Coding, Communications, IEEE Transactions on., vol. 27, no. 9, pp , Sep