Key-Words: - Free viewpoint video, view generation, block based disparity map, disparity refinement, rayspace.

Transcription

1 New View Generation Method for Free-Viewpoint Video System GANGYI JIANG*, LIANGZHONG FAN, MEI YU AND FENG SHAO Faculty of Information Science and Engineering Ningbo University Ningbo CHINA Abstract: - View navigation is an attracted function of free viewpoint video (FVV) system. Disparity map which describes correspondence between adjacent views plays an important role in view generation. However, disparity estimation is complicated and time consuming for user side with constrained resource environment. In view of disparity compensation prediction algorithm is usually utilized at the encoder to lift up the coding efficiency, block based can be easily obtained without more extra efforts. Therefore, a framework of view generation oriented FVV system is presented in this paper, in which block based is generated and encoded with CABAC losslessly at the server side, while at the user side, the received block based is refined for view generation, and spatial correlation in block based is utilized to accelerate the generation of pixel-wise. Experimental results show that the proposed framework makes a tradeoff between the transmission cost and effort of view generation. Key-Words: - Free viewpoint video, view generation, block based, disparity refinement, rayspace. 1 Introduction It has been recognized that multi-view video signal processing is a key technology for a wide variety of future applications, including free viewpoint video (FVV) system, three-dimensional television, immersive teleconference and surveillance etc [1,2]. FVV can offer a 3D depth impression of the observed scenery and allow interactive selection of viewpoint and direction within a certain operating range, which is regarded as the most challenging audio-video application scenario[3,4]. A classical FVV system normally includes the components of multi-view capture, pre-processing, multi-view video coding (MVC) encoder, transmission, MVC decoder, view generation and display[5]. At the server side, the most difficult problem is how to improve the coding efficiency of multi-view video. While at user side, the most timeconsuming parts are view generation and MVC decoding. Therefore, how to reduce the complexity of these two parts will alleviate the burden of a receiver. From view-point of the user, view generation should be fast so that real-time navigation can be achieved. Additionally, the quality of synthesized view should also be high enough to support the realistic impression. In the JVT 23th meeting, multi-view video plus depth (MVD) as basic format for advanced future 3DTV[4] and FVV[6] systems which fulfills the requirement of rendering a wide range continuum of views at the decoder was proposed by Smolic and Mueller[7]. And then, Merkle, Smolic studied on the multi-view video plus depth representation and found out that coding artifacts on depth data strongly influences the reconstruction quality of rendered arbitrary views in a FVV scenario[8]. As a result, high bit rates are needed for depth data compression or doing post-processing on the decoded depth maps to instead. Image-based rendering refers to a collection of techniques and representations that allow 3D scenes and objects to be visualized in a realistic way without full 3D model reconstruction[9,10]. For interactive photo-realistic applications, accurate pixel-wise disparity information is generally used to synthesize high quality novel views[11]. Light field, ray-space and lumigraph are similar ideas which can model radiances of a scene by a 4D plenoptic function[12-16]. A novel view from an arbitrary position and direction can be generated by appropriately combing image pixels from the existing views[4]. However, these benefits have to be paid by dense sampling of the real world with plenty of original view images. In real situation, only sparse multi-view images are compressed by MVC scheme and transmitted to the user side, and dense intermediate view images are generated by some interpolating means[5]. To interpolate dense intermediate view images, accurate, which determines the quality of synthesized image, will be calculated beforehand. ISSN: Issue 6, Volume 7, June 2008

2 This paper is focused on arbitrary view rendering using block-based generated at the server side. Transmission cost of block based is discussed. Quality of reconstructed views is compared between two schemes, that is, block based scheme and pixel-wise scheme. This paper is organized as follows: Section 2 gives the framework of view generation oriented FVV system. Sections 3 to section 5 describe three important components including generation and compression of block based, refinement and arbitrary view generation in the proposed framework. Section 6 reports some experimental results on different test images and Section 7 gives the conclusion. 2 Framework Of View Generation Oriented Free Viewpoint Video System In many applications, user side is a terminal with constrained resource environment, such as TV set, mobile phone or PDA etc. Its capability of memory and processing is not powerful enough to process complicated operations. Therefore, to realize free viewpoint navigation in these devices will confront with many technical problems. Compared with the resource limitation at the user side, the complexity of server side can be left out of account. Thus, moving part of time-consuming task related with view generation to the server side is an alternative scheme. Fig.1 gives the proposed framework of view generation oriented FVV system, in which block based is generated and encoded with CABAC losslessly at the server side. At the user side, the received block based disparity map is refined for view generation, and spatial correlation in block based is utilized to accelerate the generation of pixel-wise disparity map. By using the refined pixel-wise, dense ray-space volume is obtained so that novel views can be rendered easily. The new framework is based on the following considerations: 1) To improve coding efficiency, correlations among multi-view videos are usually exploited. Disparity compensation prediction algorithm is used to calculate the correspondences between adjacent views. Thus the block based can be easily obtained at MVC encoding step without more extra computational efforts. 2) For view generation at the user side, pixel-wise is needed. Thus, if pixel-wise is generated and transmitted to the user side, disparity estimation can be avoided for the user side. However, in this case, the total bit rate will increase obviously because of the extra pixelwise. Taking into account the tradeoff between the total bit rate and the resolution of, block based is adopted in our scheme. Since the size of block based is smaller than that of the original image, and the values of is restricted in the range of maximum disparity, the increased bit rate is small compared with original multi-view coding. While at the user side, even though the refinement of block based has to be done to obtain pixel-wise, the effort is not so hard since the disparity estimation can be restricted within a small range indicated by the coarse disparity provided by the block based. multi-view video block based generation Information share multi-view video encoding encoding with CABAC Network decoding multi-view video decoding pixel-wise refinement View generation monoscopic display stereoscopic display multi-view display Fig.1: Framework of view generation oriented FVV system ISSN: Issue 6, Volume 7, June 2008

3 (a) original view 1 of Akko&Rayo (b) 4 4 block based (c) 8 8 block based (d) block based Fig.2: Disparity maps with different resolutions 3 Generation and Compression of Block Based Disparity Map 3.1 Discussion on resolution of block based The resolution of not only influences the bit rate of the coded bit-stream but also the quality of rendered images. High resolution favors view generation but results low coding efficiency, while low resolution lowers the quality of rendered image if extra effort on refinement is left out. Fig.2 gives block based s with different resolutions, that is, 4 4, 8 8 and block based s. In the figure, 4 4 block based is with the size of , while the size is and for 8 8 and s, respectively. From the figure, it is seen that as the size of block increases, the accuracy of is reduced greatly. The is encoded with CABAC losslessly because error of disparity resulted from lossy coding degrades quality of rendered image significantly. Tables 1-3 give comparison of the above three s on bit rate where bit rates of corresponding two views resulted from Simulcast and HBP coding schemes are used as benchmark. In Table 2 and Table 3, the proportion η is defined by bit rate of η = (1) bit rate of and two views It is seen that bit rate of 4 4 block based disparity map brings about 6%~8% extra bit rate compared with traditional multi-view coding schemes when QP=22. The data are 1%~2% and 0.5% for 8 8 and block based s, respectively. If QP is lifted up to 32, the proportion of bitstream of is increased in comparison with bitstream of multi-view video. The proportion is around 20% for 4 4 block based, which indicates that the transmission cost for 4 4 block based is too high. Even though block based only brings no more than 2% extra bitstream, its usability is limited due to the lack of precision. 8 8 block based seems to make a tradeoff between transmission cost and effort of view generation. 3.2 Generation of block based To obtain more accurate and smooth, a disparity estimation method considering ISSN: Issue 6, Volume 7, June 2008

4 neighborhood constraints is used at the server side. Let I L and I R represent the left and right images respectively, (x,y) denote the current block to be estimated. An energy function including data item E data and smoothness item E smooth is defined as E(x, y, dv)=e data (x, y, dv)+λe smooth (x, y, dv) (2) Edata ( w w L i= w j = w x, y, dv) = I ( x + j, y + i) I ( x + dv + j, y + i) R (3) 1 3 E smooth = dv dv 4 i (4) i= 0 The smoothness item utilizes neighboring disparity vectors to enhance smoothness of disparity map. The relationship between dv and dv 0 ~dv 3 is illustrated in Fig.3, where dv is disparity of current block. Smooth factor λ is a constant that controls the influence of the smoothness item on the energy function. Fig.4 shows the influence of smooth factor λ on 8 8 block based generation. It is seen that the higher the λ is, the smoother the is. dv 0 dv 1 dv 2 dv 3 dv Fig.3: Smoothness constraints between disparity vectors Table 1. Bit rates of block based s in comparison with bit rates of two views unit: kbps Views View_01 View_12 View_23 View_34 QP= Simulcast Bit rate of QP= two views QP= HBP-MVC QP= block based Bit rate of 8 8 block based block based Table 2. Proportion of bit rate of block based in total bit rate of two views and (QP=22) Disparity map Coding schemes Proportion η 4 4 block based Simulcast 7.97% 7.54% 7.68% 7.84% HBP-MVC 6.89% 6.46% 6.16% 6.45% 8 8 block based Simulcast 2.01% 1.82% 1.90% 1.91% HBP-MVC 1.72% 1.55% 1.51% 1.55% block based Simulcast 0.46% 0.42% 0.44% 0.43% HBP-MVC 0.40% 0.36% 0.35% 0.34% Table 3. Proportion of bit rate of block based in total bit rate of two views and (QP=32) Disparity map Coding schemes Proportion η 4 4 block based Simulcast 21.37% 20.64% 20.65% 21.05% HBP-MVC 21.24% 20.08% 19.94% 20.71% 8 8 block based Simulcast 6.05% 5.59% 5.71% 5.75% HBP-MVC 6.01% 5.41% 5.48% 5.64% block based Simulcast 1.43% 1.33% 1.36% 1.32% HBP-MVC 1.42% 1.29% 1.30% 1.29% Table 4. Influence of λ on compression efficiency of 8 8 block based and reconstruction of right view Smooth factor λ Reconstructed right view (db) Bit rate of (kbps) ISSN: Issue 6, Volume 7, June 2008

5 (a) λ=0 (b) λ=0.2 (c) λ=0.5 (d) λ=0.8 (e) λ=1 Fig.4: Influence of smooth factor λ on 8 8 block based generation Smooth factor λ not only influences disparity map itself, but also compression efficiency of and quality of reconstructed image. Table 4 shows such influences. It is seen that the bit rate of compressed 8 8 block based with respect to different λ varies greatly due to different smoothness of. When λ=1, the bit rate is almost half of the bit rate related with λ=0. But for quality of reconstructed right view, even though higher λ results lower quality, the difference is small. Considering the 8 8 block based should be refined at the user side, high compression efficiency is more important compared with quality of reconstructed view. Therefore, λ=1 is selected in the proposed framework. 4 Refinement of 8 8 Block Based Disparity Map At the user side, pixel-wise is necessary for view generation. Based on the block based received from the server side, fast disparity refinement scheme can be designed. Similar to normal 2D image, also has strong spatial correlation. If disparity estimation is accurate, objects in a scene with same depth will have same disparity. To reduce complexity of disparity estimation, some blocks with accurate disparities are extracted directly from by using this smoothness assumption so that the reestimation of these blocks can be avoided. The disparity refinement algorithm is carried out according to the following steps: Step1. In 8 8 block based, let dv be disparity of current block, and dv i (i [0, 7]) be disparity of one of the eight neighbors of current block. If f 8 8 (x, y) of current block (x, y) equals to 1 according to Eq(5), the current disparity dv is regarded as an accurate one, and it will be copied to each pixel within the corresponding block of pixelwise. Otherwise, the current 8 8 block will be split into four smaller blocks with the size of 4 4, and each of them is marked as inaccurate sub-blocks. 1, if ( dv dvi ) < 2, i [0,7] f8 8 ( x, y) = (5) 0, otherwise Fig.5(a) gives after 8 8 block based consistency test. The white areas indicate the inaccurate blocks, while the gray regions are copied directly from the 8 8 block based. As we can see, the accurate blocks are either background of a scene or at middle part of foreground objects. Step2. In obtained by Step1, for those inaccurate 4 4 sub-blocks, the disparity consistency of 4 4 sub-blocks is done as follows. For the top left 4 4 sub-block as shown in Fig.5(b), if its disparity is consistent with disparities of its ISSN: Issue 6, Volume 7, June 2008

6 three neighbors, that is, dv1, dv2 and dv3, the dv of the top left 4 4 sub-block is thought to be accurate so that it will be copied to the corresponding pixels within the pixel-wise. The procedure of consistent checking of the other three 4 4 subblocks in Fig.5(b) is similar with the top left subblock. Then the disparities of inaccurate 4 4 subblocks will be searched again and compared with its four neighbors, similar as Eq.(5) except that the eight neighbors are replaced with four neighbors. Fig.5(c) gives the after step2, in which the white pixels are those inaccurate blocks even after the 4 4 sub-block verification. Step3. In the pixel-wise, the inaccurate 4 4 disparity is applied to accelerate the refinement of inaccurate regions. Let dv denote current disparity of an inaccurate pixel and it is used as initial position of searching process. A small range for searching the optimal disparity is defined as [dv-4, dv+4]. Therefore, for pixel that has initial block disparity, the computation complexity of disparity estimation is decreased greatly. Fig.5(d) shows the final refined pixel-wise. (a) verified 8 8 (b) neighborhood for 4 4 disparity consistency test (c) verified 4 4 (d) final pixel-wise Fig.5: Disparity consistency test 5 View Generation Intermediate view interpolation is vital for IBR(Image Based Rendering) based FVV system, since it determines the quality of synthesized virtual view. When the correspondences between view n and view n+1 are obtained, the dense intermediate at α position can be generated through the following steps. (1) For the pixel at position x in the view n with disparity d, the position of the corresponding pixel in the intermediate view can be obtained by simply scaling the disparity according to α which indicates the location of the intermediate view between view n and view n+1. Disparity αd is assigned to the pixel at position x+αd in intermediate view. In this case, the values of the pixels between view n and view n+1 are interpolated in term of α. This is illustrated in Fig.6 by the pair of arrows which are opposite with each other. (2) The intermediate obtained by the above step will have some holes. For the holes in corresponding intermediate view, the pixels within the holes are regarded as occluded pixels. These pixels can only be seen in one view, and a simple strategy is introduced to estimate their disparities. As we known, the disparities of background objects are normally smaller than that of the foreground objects, and in general, ISSN: Issue 6, Volume 7, June 2008

7 background objects may be occluded by foreground objects. Therefore, for the occluded pixels, the nearest matched pixels at the left and right direction are compared. Let d L and d R denote the corresponding disparities with respect to the left and right directions, then the disparity d of current occluded pixel is determined by dr,and left occluded, if dl dr d = (6) dl,and right occluded, otherwise In this case, the pixels are interpolated according to occlusion type. If current pixel is labeled as left occluded pixel, that is, d L d R, the pixel at the view n indicated by the disparity d R is directly copied to the interpolating pixel. The similar operation can be done to the right occluded pixel. If the of intermediate view is obtained, the interpolation of intermediate view is easy to be done. 1.0 right occluded Ray-space representation describes the rays in a scene as a 4D function f(x,y,θ,ϕ), where (θ,ϕ) denotes the direction of the ray, (x,y) denotes the intersection of the ray and the reference plane, and f(x,y,θ,ϕ) represents the intensity of the specific ray. An important feature in ray-space is that an image with respect to a certain viewpoint is given as a subspace of ray-space. Fig.7 shows the relationship between real space and ray-space. Intermediate views with respect to virtual cameras, denoted as the gray ones in the figure, are required to be rendered. The relationship between the virtual view and the ray-space can be explained as X = x + ztanθ (7) Therefore, when dense sampling of a scene is acquired, it is comparatively easy to synthesize a novel view. view n+1 α 0.0 matched left occluded intermediate view view n Fig.6: Generation of of intermediate view (a) real space Fig.7: Relationship between real space and ray-space (b) ray-space 5 Experimental Results and Analysis 5.1 Performance analysis of pixel-wise disparity map and 8 8 block based We have compared the performance of pixel-wise and 8 8 block based in transmission cost and quality of reconstructed views. The pixel-wise is also generated by the disparity estimation method considering neighborhood constraints mentioned in section III except that the searched disparity of 7 7 current block is allocated to the center pixel of the block instead of all pixels of the 7 7 block. The pixel-wise is then encoded with JMVM 4.0[17]. AKKO&RAYO is used as the test sequence. Table 5 gives bit rates with respect to pixel-wise disparity map encoded by JMVM with different QP and 8 8 ISSN: Issue 6, Volume 7, June 2008

8 block based encoded by CABAC. Table 6 shows the corresponding PSNRs of reconstructed views. It is seen that the scheme of 8 8 block based reduced about 81.00%, 59.37% and 11.00% bit rate compared with the scheme of pixel-wise when QP is set to be 22, 27 and 32 respectively. When QP is 37, the bit rate of pixel-wise becomes lower than that of 8 8 block based. However, the quality of reconstructed views is not satisfied compared with the scheme of 8 8 block based. It should be noted that as the QP grows, the bit rate of encoded pixel-wise is reduced, but the quality of reconstructed views which reflects the accuracy of is also reduced due to the lossy encoding. By contrast, the bit rate of 8 8 block based is invariable by taking the advantage of lossless encoding. The refinement of 8 8 block based can further lift up the quality of reconstructed views about 1dB. Table 5. Comparison of pixel-wise and 8 8 block based in bit rate 8 8 block based Pixel-wise disparity reduced coded map coded by JMVM bit rate by CABAC (kbps) (%) (kbps) QP Table 6. Comparison of pixel-wise and 8 8 block based in PSNR of reconstructed views unit: db Views view1 view2 view3 view Pixel-wise QP (JMVM) block based (CABAC) Refined 8 8 block based (CABAC) 5.2 Performance of the proposed method in view generation Experiments are also performed to evaluate the efficiency of the proposed method in view generation. The test image Xmas with 101 parallel views is used to evaluate intermediate view interpolation without camera parameter. The 3DTV test data Akko&Kayo is used to render arbitrary view image. Akko&Kayo is captured by 3 5 camera array, here, 5 images in the middle row are selected, the maximum disparity between two anchor images is 20 pixels. Fig.8 shows PSNR and running time of interpolating intermediate view with different camera intervals. In the figures, the proposed method outperforms BMI(Block-based Matching Interpolation) method[18], because it can obtain more accurate disparities. Moreover, the proposed method runs much faster than BMI because it has initial disparity for searching. For rendering virtual viewpoint image, as illustrated in Fig.9, dense intermediate viewpoint images are firstly interpolated by the proposed method. Here, 19 intermediate viewpoint images are interpolated between two anchor view images according to the sampling theory. Then, these dense multi-view images are projected into ray-space. When given a virtual viewpoint position, the corresponding data can be extracted from the interpolated dense ray-space. Fig.9(a) shows a virtual view at the baseline, while Fig.9(b) and Fig.9(c) show the virtual views at forward and backward positions from the baseline respectively. As we move forward, details of a scene can be seen, meanwhile, the field of view becomes smaller. On the contrary, when we move backward, the field of view becomes larger. The emergence of black regions at top and bottom in Fig.9(c) is resulted from the fact that dense intermediate views are interpolated in horizontal direction but not vertical direction. From the rendered virtual view images, it is seen that the synthesized images have photoreality without ghosting artifacts. 6 Conclusions In this paper, a framework of view generation oriented free viewpoint video system is presented, which focus on the requirement and capability of user side. The calculation of, which is required at the user side, is moved to the server side, based on the fact that the user side is a terminal with constrained resource environment and block based can be easily obtained at the server side due to the usage of disparity compensation prediction at the MVC encoder. This change will alleviate the burden of view synthesis and it is more suitable for real time applications. ISSN: Issue 6, Volume 7, June 2008

9 (a) PSNR of interpolated intermediate view (b) run time of interpolating intermediate view (c) original intermediate image (d) BMI (e) Proposed method (PSNR=28.36dB,TIME=32.8s) (PSNR=36.17dB,TIME=1.1s) Fig.8: Result of intermediate view generation of Xmas (maximum disparity=50) (a) virtual view at baseline (b) virtual view forward 0.5m (c) virtual view backward 0.5m Fig.9: Results of arbitrary view rendering Acknowledgment This work was supported by the Natural Science Foundation of China (grant , ), the Program for New Century Excellent Talents in University (NCET ), Natural Science Foundation of Ningbo (grant 2007A610037). Gangyi Jiang is correspondence author. References: [1] Ishfaq Ahmad, Multiview Video: Get Ready for Next-Generation Television, IEEE Distributed Systems Online, vol.8, no.3, 2007, art. no o3006. [2] A. Smolic, D. McCutchen, 3DAV Exploration of Video-Based Rendering Technology in MPEG, IEEE Transactions on Circuits and Systems for Video Technology, vol.14, no.3, 2004, pp [3] H. Kimata, M. Kitahara, K. Kamikura, et al., Low-Delay Multiview Video Coding for Free- Viewpoint Video Communication, Systems and Computers in Japan, vol.38, no.5, 2007, pp ISSN: Issue 6, Volume 7, June 2008

10 [4] A. Smolic, K. Mueller, P. Merkle, et al., 3D Video and Free Viewpoint Video - Technologies, Applications and MPEG Standards, IEEE Int. Conf on Multimedia and Expo, 2006, pp [5] M. Tanimoto, T. Fujii, H. Kimata, S. Sakazawa, Proposal on Requirements for FTV, Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG, JVT-W127, 23rd Meeting: San Jose, California, Apr [6] A. Smolic, and P. Kauff, Interactive 3D Video Representation and Coding Technologies, Proc. of the IEEE, Special Issue on Advances in Video Coding and Delivery, vol.93, no.1, Jan [7] Aljoscha Smolic, Karsten Mueller, Philipp Merkle, Nicole Atzpadin, Christoph Fehn, Markus Mueller, Oliver Schreer, Ralf Tanger, Peter Kauff, Thomas Wiegand, Multi-view video plus depth (MVD) format for advanced 3D video systems, ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q.6, JVT-W100, San Jose, USA, April, [8] Philipp Merkle, Aljoscha Smolic, Karsten Mueller, Thomas Wiegand, MVC: Experiments on Coding of Multi-view Video plus Depth, ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q.6, JVT-X064, Geneva, CH, 29 June - 5 July, [9] S.C. Chan, H.-Y. Shum, K.-T. Ng, Image-Based Rendering and Synthesis, IEEE Signal Processing Magazine, vol.24, no.6, 2007, pp [10] C. Zhang T. Chen, A Survey on Image- Based Rendering-Representation, Sampling and Compression, EURASIP Signal Processing: Image Commun., vol.19, no.1, 2004, pp [11] E. Kurutepe, M. R. Civanlar, A. M. Tekalp, Interactive Transport of Multi-view Videos for 3DTV Applications, Journal of Zhejiang University Science, vol.7, no.5, 2006, pp [12] M. Levoy, P. Hanrahan, Light Field Rendering, Proc. of Siggraph, 1996, pp [13] X. Tong, R. M. Gray, Interactive Rendering from Compressed Light Fields, IEEE Trans. Circuits Syst. Video Techn., vol.13, no.11, 2003, pp [14] N. Fukushima, T. Yendo and T. Fujii, Real- Time Arbitrary View Interpolation and Rendering System Using Ray-Space, Proceedings of SPIE Three-Dimensional TV, Video and Display, 2005, pp [15] L. Zhong, M. Yu, Y. Zhou, G. Jiang, Quality and Performance Evaluation of Ray- Space Interpolation for Free Viewpoint Video Systems, Lecture Notes in Computer Science, LNCS3992, 2006, pp [16] S. J. Gortler, R. Grzeszczuk, R. Szeliski, M. F. Cohen.: The Lumigraph, Proc. of SIGGRAPH-96, 1996, pp [17] JMVM 4.0 Software, ISO/IEC JTC1/SC29/WG11, W208: San Jose, California, USA, April, 2007 [18] T. Yendo, T. Fujii, M. Tanimoto, Ray- Space Acquisition and Reconstruction within Cylindrical Objective Space, Proceedings of SPIE Stereoscopic Displays and Virtual Reality System, vol.6055, 2006, pp ISSN: Issue 6, Volume 7, June 2008