An Independent Motion and Disparity Vector Prediction Method for Multiview Video Coding

Transcription

1 Preprint Version (2011) An Independent Motion and Disparity Vector Prediction Method for Multiview Video Coding Seungchul Ryu a, Jungdong Seo a, Dong Hyun Kim a, Jin Young Lee b, Ho-Cheon Wey b, and Kwanghoon Sohn a a Department of Electrical and Electronic Engineering, Yonsei University, 50 Yonsei-ro, Seodaemun-gu, Seoul, , South Korea; b Samsung Advanced Institute of echnology, Samsung Electronics Co., Ltd., San 14-1, Nongseo-dong, Kiheung-gu, Yongin-si, Gyeonggi-do, , South Korea. ABSRAC he inter-view prediction is used as well as the temporal prediction in order to exploit both the temporal and inter-view redundancies in multiview video coding. Accordingly, the multiview video coding has two types of motion vectors that are the temporal motion vector and the disparity vector, respectively. he disparity vector is generally uncorrelated with the temporal motion vector. However, they are used together to predict the motion vector regardless of their types, therefore an efficiency of the conventional predictive coding of multiview video coding is decreased. In order to increase the accuracy of the predicted motion vector, a new motion vector prediction method including virtual temporal motion vector and virtual disparity vector is proposed for both the multiview video and multiview video plus depth formats. he experimental results show that the proposed method can reduce the coding bitrates by 6.5% in average and 14.6% at maximum in terms of Bjontegaard metric compared to the conventional method. Keywords: motion vector prediction, multiview video coding, virtual vector, temporal motion vector, disparity vector 1. INRODUCION he multiview video (MVV) format is widely used in challenging applications, such as 3DV, free viewpoint video, and entertainment applications. However, the amount of data needed for the multiview video is extremely increased in proportion to the number of viewpoints, and consequently an effective compression is very important for a successful multiview video system. Accordingly, a widely used 2D video coding standard H.264/AVC has been extended into the multiview video, which is known as multiview video coding (MVC) 1. he key technology of MVC is to reduce redundancies among the sequences of different views. For this purpose, the interview and temporal combined prediction structure (Fig. 1) is employed in MVC. his structure also adopts hierarchical B-picture structure 2 due to its effectiveness. In this structure, the type of view is classified into three groups (I-, P-, and B-views) according to the types of the employed inter-view prediction. In the I-view, temporal prediction (intra-view prediction) is only performed. In addition, anchor pictures of I-view have employed only intra prediction without any reference picture. On the other hand, P- and B-views employ both temporal and inter-view predictions. A bidirectional inter-view prediction is performed in B-view while a unidirectional inter-view prediction is adopted for P- view. In cases of inter-view and temporal combined structures (P-view and B-view), MVC has two different motion vectors such as the temporal motion vector (MV) and the disparity vector (DV). he characteristics of these motion vectors are absolutely different from each other, e.g. the magnitude of DV is relatively larger than that of MV. However, they are used together to predict the motion vector regardless of their types. Specifically, the motion vector is predicted using motion information of neighboring macroblocks (MB) based on the assumption that the adjacent MBs have motion consistency. However, this assumption is obviously invalid in MVC due to adaptation of inter-view and temporal combined prediction technique as mentioned above. herefore, the efficiency of the conventional motion vector prediction method employed in MVC is decreased.

2 Figure 1. Hierarchical B-picture structure in MVC In recent decades, several methods have been proposed in order to improve the performance of the motion vector prediction for MVC 3-8. An efficient representation method of motion data, which is called inter-view direct mode, was proposed 3-4. Even though the method is effective as an additional mode, it cannot solve the inherent problem caused by inter-view and temporal combined prediction structure. Competition-based motion vector coding methods were proposed in order to increase an accuracy of the predicted motion vector 5-6. he conventional skip mode is extended into an interview direction, but the method did not consider other modes 7. Lee et al. 8 proposed the virtual inter-view motion vector to increase the probability of inter-view motion vector selection. However, gain obtained by Lee s method is insufficient due to the rough derivation of virtual disparity. In this paper, a new motion vector prediction is proposed to resolve the inconsistency problem of MV and DV. his paper is organized as follows. Section 2 describes the conventional and the proposed motion vector prediction methods. In Section 3, the performance evaluation of the proposed method is presented. Section 4 concludes the paper. 2. PROPOSED MEHODS 2.1 Proposed motion vector prediction method In MVC, a motion vector is predicted using motion vectors of neighboring MBs and its residual is encoded. A residue of the motion vector mv r is computed as follows: mv = mv, (1) r mv p where mv is an actual motion vector of the current MB, and mv p is a predicted motion vector that is median for each component (horizontal and vertical) of the neighboring motion vectors based on the assumption of a motion consistency. he assumption is, however, invalid for most cases in MVC due to two different types of motion vectors. In order to solve this problem, a new motion vector prediction method is proposed in this paper. In the proposed method, only same types of motion vectors of neighboring MBs are used to predict mv p unlike the conventional prediction scheme in which motion vectors of neighboring MBs are all used regardless of their types. For example, only MVs of neighboring MBs are used for the temporal prediction, and only DVs are used for the inter-view prediction.

3 Figure 2. Overall algorithm of the proposed motion vector prediction However, for some cases, all the types of neighboring motion vectors are same. In these cases, virtual MV or virtual DV is required since there is no available motion vector used to predict mv p. For instance, when all the neighboring motion vectors are MV during the inter-view prediction, there is no available DV for predicting the corresponding DV. Accordingly, a virtual DV and virtual MV should be generated to solve the problem. A virtual DV is generated using disparity information of anchor pictures or the derived disparity from depth information. Meanwhile, zero-vector is used as a virtual MV due to its simplicity in the proposed method. he overall algorithm of the proposed motion vector prediction is described in Fig Virtual disparity vector Recently, MPEG has started an activity to develop the next generation of 3D video standard (3DV) 9. A more efficient format, multiview color video plus depth sequences (MVD), is used in this standard activity. In MVD format, depth information and color video sequences are transmitted together, while the color video sequences are only transmitted in MVV format. In this paper, two scenarios are assumed for MVV (MVC) and MVD (3DV) formats, respectively. In the first scenario, the color videos are only transmitted and stored. On the other hand, multiview color video sequences and depth information are both transmitted in the second scenario. Accordingly, two virtual DV generation methods are introduced for the two scenarios, respectively. he former is the method based on layered global disparity vectors (GDV) while the latter is the method based on depth information Layered GDV based method (Scenario 1: MVV format) In the first scenario, where only color video sequences are transmitted, there is no directly available disparity information. herefore, disparity information should be derived from other information to obtain virtual disparity vectors. In MVC, the anchor pictures have only DVs because only inter-view predictions are performed in the P-view and B- view. hese DVs are classified into n-layers according to their magnitudes and the average of DVs in each layer is defined as a layered global disparity vector LGDV i given as follows: LGDV i = 1 N DVi, k, (2) i k i th layer where LGDV i is the i th layered global disparity vector, DV i,k is k th DV and N i is the number of DVs in the i th layer.

4 Figure 3. Description of Layered GDV (black circles: DVs of anchor picture) In the proposed method, LGDV i is used as the virtual disparity vector. For this purpose, one of LGDVs is selected among them based on the coarse assumption that an object with large disparity has lager motion than an object with small disparity 10. he global motion vector GMV and the local motion vector LMV are defined as (3) and (4), respectively. GMV LMV 1 = mv N i, (3) i 1 = mv M j, (4) j where mv i, mv j are motion vectors in the previous coded frame and that of the neighbor MBs, respectively. N and M are the number of MBs in a frame and neighbor MBs, respectively. he GMV and the LMV are used to select one of LGDVs. If LMV is larger than GMV, larger LGDV is selected as the virtual disparity vector and vice versa. An example of the procedure (n= 4) is illustrated in Fig. 3 and Fig. 4. Note that the value of 0.7 is empirically used as α. he flow of Selection among LGDVs: If (LMV > (1+ α) * GMV) Virtual DV = LGDV 4 else if (LMV > GMV) Virtual DV = LGDV 3 else if (LMV > (1- α) * GMV) Virtual DV = LGDV 2 else Virtual DV = LGDV 1 Figure 4. Pseudo code for selecting the LGDV

5 Figure 5. Corresponding points between different views Depth based method (Scenario 2: MVD format) In the second scenario, where color videos and depth information are transmitted together, depth information can be converted into disparity information using multiview geometry with camera parameters. Accessibility to both camera parameters and the depth map of the current view is assumed in this scenario. he derivation procedure of a disparity is describes as follows (Fig. 5): 1) Project the pixel location (x c,y c ) of the coded MB from current view cc into world coordinates (u,v,z) using transformation: 1 [ uvz,, ] = Rc ( r) A ( cc)[ xc, yc, 1] Dx ( c, yc, cc) + c ( c) (5) where A(c) is an intrinsic camera parameter matrix of camera c, R(c) is a rotation matrix of camera c, (c) is a translation vector of camera c and D(x,y,c) is a depth value related to camera c, at pixel (x,y). 2) Map the location of world coordinates (u,v,z) into local coordinates of the reference view c r using: 1 [ xrzr, yrzr, zr ] = A( cr ) R ( cr ){[ u, v, w] ( cr )} (6) where (x r,y r ) and z r are a corresponding pixel and a depth value at point (x r,y r ) in coordinates of the reference view, respectively. 3) Calculate a disparity (d x,d y ) between the pixel points in current view and reference view: [ d, d ] = [ x, y ] [ x, y ] (7) x y c c r r he number of disparities obtained by this procedure is as many as the number of pixels in a MB. he disparity of the central pixel is chosen as the representative disparity in order to prevent confusion. he selected representative disparity is used as the virtual DV

6 3. EXPERIMENAL RESULS We implemented the proposed method with the MVC reference software JMVC est sequences 12 named as Kendo, Balloons, Pantomime, and Champagne_ower are used for experiments. he sequences were encoded according to the common conditions 12 with GOP size of 15. he experiments have been performed with quantization parameters QP = 27, 32, 37, and 42. In our experiments, the motion/disparity estimation is performed with search range of In order to evaluate the performance of the proposed motion vector prediction method, comparisons with respect to the original JMVC 6.0 have been done in terms of BontegaarD BitRate (BDBR) 13. A negative sign of BDBR denotes average reduction of bitrate compared to the original JMVC 6.0. he results of the scenario 1 and the scenario 2 are shown in able 1 and able 2, respectively. he coding efficiencies of the Layered GDV based method (scenario 1) and the depth based method (scenario 2) are compared with the original JMVC 6.0, respectively. As seen in the ables, the proposed method always improves the coding efficiency of MVC. he average bitrate savings are about 6.2% for the scenario 1 and 6.7% for the scenario 2 with up to 14.6% (scenario 2, Kendo, B-View) for the view-temporal prediction structure. he overall coding efficiency for the scenario 2 is better than that for the scenario 1 due to relatively more accurate disparity information. In some cases, however, the coding efficiency for the scenario 2 is worse than that of the scenario 1 caused by a crudeness of depth information ( Pantomime and Champagne_ower ). he better quality of depth map can lead to increase of coding efficiency for the scenario 2. On the other hand, the scenario 1 has an advantage that additional disparity information is not required to be transmitted unlike the scenario 2. he proposed method also has gained a bitrate saving up to 1.4% for temporal only prediction structure as well as the view-temporal prediction structure. sequence able 1 Results of the proposed method for scenario 1: MVV format is assumed BDBR (%) for scenario 1 (MVV) emporal-only prediction View-temporal prediction I-View P-View B-View Kendo Balloons Pantomime Champagne_ower Average able 2 Results of the proposed method for scenario 2: MVD format is assumed BDBR (%) for scenario 2 (MVD) sequence emporal-only prediction View-temporal prediction I-View P-View B-View Kendo Balloons Pantomime Champagne_ower Average

7 4. CONCLUSION In this paper, a new motion vector prediction method is proposed to overcome the inconsistency problem caused by two different types of motion vectors. In the proposed method, only same types of motion vectors of neighboring MBs are used to predict the corresponding motion vector. In order to manage the absence of available motion vectors, two methods of virtual DV generation are introduced for two scenarios. Experiments show that the proposed method reduced bitrates about 6.2% and 6.7% respectively for scenarios 1 and 2 in average, and 14.6% at maximum according to the Bjontegaard measure 13 for the view-temporal prediction structure. Future work includes a study on the modified DIREC mode which is more suitable for the proposed method. Another direction of future work includes the generation method of virtual MV. REFERENCES [1] A. Vetro,. Wiegand, and G. Sullivan, Overview of the stereo and multiview video coding extensions of the H.264/MPEG-4 AVC standard, Proc. IEEE, 99(4), (2011). [2] H. Schwarz, D. Marpe, and. Wiegand, "Hierarchical B pictures," ISO/IEC JC1/SC29/WG11 and IU- Q6/SG16, Doc. JV-P014, Poznan, Poland, (2005). [3] J. Konieczny, and M. Domanski, Depth-based Inter-view Prediction of Motion vectors for Improved Multiview Video Coding, Proc. IEEE 3DV-conference: he rue Vision Capture, ransmission and Display of 3D Video, ampere, Finland, (2010). [4] X. Guo, Y. Lu, F. Wu, and W. Gao, Inter-view Direct Mode for Multiview Video Coding, IEEE ransactions on Circuits and Systems for Video echnology, 16(12), (2006). [5] S. Ryu, J. Seo, X. Liu, J. Y. Lee, H. Wey, and K. Sohn, Analysis of Motion Vector Predictor in Multiview Video coding System, Proc. IEEE International Symposium on Parallel and Distributed Processing with Applications Workshops, Busan, Korea, (2011). [6] S. Ryu, J. Seo, D. H. Kim, J. Y. Lee, H. Wey, and K. Sohn, Adaptive competition for motion vector prediction in multi-view video coding, Proc. IEEE 3DV-conference: he rue Vision Capture, ransmission and Display of 3D Video, Antalya, urkey, (2011). [7] J. Y. Lee, H. Wey, D.-S. Park, and C.-Y, Kim, emporal and inter-view skip modes for multi-view video coding, Proc. IEEE 3DV-conference: he rue Vision Capture, ransmission and Display of 3D Video, Antalya, urkey, (2011). [8] S. H. Lee, S. H. Lee, J. H. Yang, and N. I. Cho, A Motion vector Prediction Method for Multi-view Video Coding, Journal of Visual Communication and Image Representation, 21(7), (2010). [9] A. Vetro, S. Yea, and A. Smolic, owards a 3D Video Format for Auto-Stereoscopic Displays, Proc. SPIE 7073, 70730F (2008). [10] D. Kim, D. Min, and K. Sohn, A Stereoscopic Video Generation Method using Stereoscopic Display Characterization and Motion Analysis, IEEE ransactions on Broadcasting, 54(2), (2008). [11] Joint Video eam of IU- VCEG and ISO/IEC MPEG WD1, Reference software for MVC (JMVC) 6.0, Doc. JV-AF14, Geneva, Switzerland, (2009). [12] H. Schwarz, D. Marpe, and. Wiegand, Description of Exploration Experiments in 3D Video Coding, ISO/IEC JC1/SC29/WG11 MPEG2010/N11274, Dresden, Germany, (2010). [13] G. Bjontegaard, Calculation of Average PSNR Differences between RD-curves, IU- Video Coding Experts Group document VCEG-M33, (2001).