Affine SKIP and MERGE Modes for Video Coding

Transcription

1 Affine SKIP and MERGE Modes for Video Coding Huanbang Chen #1, Fan Liang #2, Sixin Lin 3 # School of Information Science and Technology, Sun Yat-sen University Guangzhou , PRC 1 chhuanb@mail2.sysu.edu.cn 2 isslf@mail.sysu.edu.cn Huawei Technologies Co., Ltd. Huawei Base, Bantian, Longgang, Shenzhen , PRC 3 linsixin@huawei.com Abstract The latest video coding standard HEVC has already improved 40% coding efficiency than H.264/AVC, but it still follows the hybrid coding structure with the translational motion compensation scheme. In this paper, an affine motion representation is described first. The proposed algorithms to improve the motion vector coding for Affine SKIP/MERGE modes are also presented. The experimental results show that these algorithms can achieve on average 3.5%, 2.2%, and 1.4% BD-rate gains for low delay P, low delay B, and random access configurations, respectively, compared to HM The BD-rate saving is up to 17.1% for sequences with complex motion. I. INTRODUCTION For the past decade, ITU-T and ISO/IEC have set up a series of digital video coding standards, including H.261, MPEG-1, H.262/MPEG-2 and H.264/AVC. This standards have influenced our life through digital TVs, video conference and digital cinema, etc. The latest video coding standard, H.265/HEVC was published by the Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T and ISO/IEC in January, 2013 [1]. Many new techniques were integrated into this standard, including variable coding block range from to 8 8, more accurate intra prediction modes, larger DCT transform sizes, advanced motion vector prediction (AMVP), and sample adaptive offset (SAO), which result in about 40% less bit rate than H.264/AVC with comparable quality [2]. In modern video coding standards, motion compensation prediction plays an important role in reducing the temporal redundancy between consecutive frames, which is strictly influenced by the motion vector representation. There are three difference modes for MV (motion vector) coding scheme in HEVC, including SKIP, MERGE and AMVP. For SKIP/ MERGE modes, the motion information including motion vector, inter direction, reference index of current coding block is completely inherited from its spatial adjacent or temporal co-located coded blocks; for AMVP mode, only the motion vector predictor (MVP) is chosen and the difference between real motion vector and MVP must be transmitted. All of these modes established a given candidate set which includes spatial and temporal MVPs and the motion vector competition (MVC) based scheme is applied to choose the best one from the set. MMSP 15, Oct Oct. 21, 2015, Xiamen, China /15/$31.00 c 2015 IEEE. However, the motion vector representation in HEVC is based on a translational motion model, which assumes all pixels in a block have same MV. In real world, it exists more complicated movement, such as rotation, zoom, shearing and so on. Using translational motion model to describe complex motion, the encoder tends to split a larger object into numerous small ones to approximate the motion, which increases the side information as well as prediction errors [3]. Since that, many researchers have investigated to use complex motion model, like affine, quadratic motion model [4]. In [5] and [6], the authors studied the use of affine motion model for H.263. In [3], the authors extended HEVC by an affine motion model with 6 estimated parameters. In [7], a parametric SKIP mode for HEVC based on global motion estimation (GME) technique was proposed. Eight picture level high order motion parameters were transmitted, which were uncompressed in floating point precision. The encoder decided in terms of rate-distortion optimization whether to use the new skip mode or not. In [8], the authors proposed a block based video coder that supported multiple motion models, including translation and affine. A cubic spline framework was utilized to obtain fractional-accuracy samples for motion compensation. All above mentioned methods have to estimate motion parameters in block or picture level, which increase the coding complexity and are rarely practical for real-time applications. In [9], Huang et al. integrated novel Affine SKIP and DIRECT modes into HEVC test model HM 1.0. The authors used spatial motion vectors of neighbor blocks to generate affine motion parameters for current block, without extra information and motion parameters estimation. Inspired by this novel affine representation, this paper extends it to support non-square block partitions and multiple reference pictures to increase coding efficiency. The remainder of this paper is structured as follows. Section II describes the affine SKIP/MERGE modes proposed in [9]. Section III introduces our proposed techniques. Experimental results and conclusions are given in Section IV and Section V, respectively. II. AFFINE MOTION REPRESENTATION A. Affine Based Motion Compensation Let (x, y) be the position of the pixel in current coding frame, (x, y ) is the corresponded position of same pixel in

2 the reference picture. So the affine transform can be described as: { x = ax + by + e y (1) = cx + dy + f A C B T LT T RT w w 0 1 D E where a, b, c, d, e, and f are the affine parameters. Through equation (1), we get { v x = (1 a)x by e (2) v y = (1 d)y cx f here v x = x x and v y = y y is the horizontal and vertical motion values of the pixel. F T LB G w 2 v 0 (0,0) v 1 (w,0) Fig. 2. Affine motion candidates. v 2 (0,h) Fig. 1. Affine Motion Model. (w,h) Let V = (ω 0, ω 1, ω 2 ) be an affine motion predictor, the best V is chosen from Ω = {U 0 U 1 U 2 }. The candidate set is arranged in the ascending order by D(V ) = v 1 v 0 + v 2 v 0. Here D(V ) is a measure of the deformation of the current block. The maximum number of candidate set Ω is 12, and only the top five ones will be selected. These modes were only utilized in 2N 2N square partition, and the different refpic and interdir between neighbor blocks haven t been taken into account. For further information cf. [9]. As shows in Fig. 1, once we obtain 3 motion vectors at the top-left, top-right, and bottom-left corners of a w h block, the corresponded block in the reference picture can be projected. Let the top-left corner be the origin of coordinates, substituting the three corner coordinates (0, 0), (w, 0), (0, h) and their motion vector (v x0, v y0 ), (v x1, v y1 ), (v x2, v y2 ) into equation (2), and solving it, we get { v x = vx1 vx0 w v y = vy1 vy0 w x + vx2 vx0 h y + v x0 x + vy2 vy0 h y + v y0. (3) Through equation (3), every pixel s motion vector in the block can be calculated and motion compensated prediction process can be operated one by one pixel. If the motion is beyond 1/4 pixel accuracy, bilinear interpolation will be performed after normal 1/4 pixel interpolation described in HEVC standard. B. Affine SKIP and MERGE Modes Similar to SKIP and MERGE modes in HEVC, the motion information, donated as ω, including v (motion vectors), refpic (reference pictures), interdir (prediction direction) for Affine SKIP/MERGE modes proposed in [9] can be inherited from the spatial neighbor blocks. As showed in Fig. 2, the top-left motion information ω 0 is derived from the motion information in neighbor block A, B, and C, donated as a set U 0 = {ω A, ω B, ω C }. Similarly, ω 1 and ω 2 is derived from the set U 1 = {ω D, ω E } and U 2 = {ω F, ω G }, respectively. III. PROPOSED ALGORITHMS The key to above representation is derived three corner motion vectors. This section describes three proposed algorithms for improving the coding efficiency of affine motion vector coding for Affine SKIP and MERGE modes. A. Extend Affine MERGE Mode to PU Level In the HEVC standard, the basic unit for compression, termed coding unit (CU), is a 2N 2N square block. Each CU contains one or multiple prediction units (PUs). Translational MERGE modes can be utilized in 2N 2N, 2N N, N 2N, N N, and other four asymmetrical partitions [10]. The previous method only supports 2N 2N partition Affine MERGE mode and this proposal extends it to support non-square blocks. B. Add Temporal Motion Candidates For previous method, the motion informations only derived from spatial adjacent blocks. In order to utilize the motion informations in temporal picture, we use 3 co-located blocks of the co-located picture to supplement the affine candidate sets. Due to duplicated MV or unavailable of spatial neighbor block, the number of candidate set U 0, U 1, U 2 may be less than 3, 2, 2, respectively. As showed in Fig. 2, if the candidate number in the set U 0 is less than 3, the top-left co-located motion information ω LT in co-located picture will be added into the set. As well, the top-right, and left-bottom co-located ω RT,

3 and ω LB will be added into the set U 1, and U 2 respectively, when the number of the set is less than 2. C. Modify MVs through a scale scheme Since HEVC supports multiple reference pictures and bidirection prediction, the MVPs in the three corners may project to different reference frames. Furthermore, depending on the prediction direction, the block where a derived MVP located may have one or two MVs. The relation between interdir and motion information ω is: ω = v 0, refpic 0 if interdir = 0 v 1, refpic 1 if interdir = 1 v 0, refpic 0, v 1, refpic 1 if interdir = 2. (4) It can be solved by modifying the MVs when 3 corner s candidate block have different refpic or interdir. The modified MVs are derived as the following rules: 1) Select a combination V = {ω 0, ω 1, ω 2 } from each set U 0, U 1, U 2 ; 2) Modify depending on the prediction directions: If ω 0, ω 1, and ω 2 have same interdir, go to step 3; If all 3 directions are different, the combination will be abandoned; If only two prediction directions are same, donated as interdir S, and the other one is interdir X, then the combination direction is set to interdir S. If interdir X is not equal to 2, the MV for the opposite direction of this position will be generated used a scaled MV. Otherwise, if interdir X is equal to 2, the MV in extra direction will be discarded. After this operation, we update the combination V, then go to step 3; 3) Modify depending on the reference picture: If all 3 refpic are same, the combination is available; If all 3 refpic are different, the combination will be abandoned; If only two reference pictures are same, donated as refpic S, and the other one is refpic X, then the combination reference picture is set to refpic S. The MV project to refpic X will be scaled to refpic S. After this operation, we update the combination V. Two examples for this modified process are shown in Fig. 3. In Fig. 3 (a), LT, LB reference blocks directions are 0, and RT block s direction is 2. So we discard the MV in reference picture list 1. After that, since block LB s reference picture is a, different to b for RT and LT blocks, the MV for LB will be scaled to the reference picture b. The modified candidate will be: V = {LT v 0, RT v 0, LBv 0, refpic = b, interdir = 0}. (5) As shown in Fig. 3 (b), LT and LB blocks directions are 2, and RT block direction is 0. So we generated the MV of reference picture list 1, using the MV in list 0. After that, the reference picture scaled process will be product for each list same as example 1. The modified candidate will be: Fig. 3. LBv 0 LBv 0' RTv 0 LTv 0 LB RT LT current X RTv 1 a b c d e f g LBv 0 LBv 0' RTv 0 LTv 0 (a) LB RT LT current LBv 1 RTv 1'' LTv 1' LTv 1 a b c d e f g (b) time time Examples for modified candidate motion vector. LT v 0, RT v 0, LBv 0, refpic = b V = LT v 1, RT v 1, LBv 1, refpic = f. (6) interdir = 2 IV. EXPERIMENTAL RESULTS In order to prove the coding efficiency, the proposed algorithms are integrated into HEVC reference software HM 12.0 [11]. Three configurations, low delay P (LDP), low delay B (LDB), and random access (RA), defined in [12], are tested for the experiment. All test sequences are listed in Table I. In addition to four classes in HEVC common test conditions, 4 other sequences are examined, donated as Class Affine, which have complicated movement, including rotation, zoom, shearing, etc. Experimental results are evaluated in terms of Y BD-rate, which is calculated by 4 rate and PSNR points generated with 4 QP values {22, 27, 32, 37} [13]. Two anchors are generated for comparison. Anchor 1 is the original HM 12.0 software, and the results of an implementation for proposal [9] in HM 12.0 are compared as anchor 2. As shown in Table I, the coding efficiency of Affine SKIP/ MERGE modes almost outperform anchor 1 for all sequences, except rarely lost in ParkScene and BQTerrace. The bitrate

4 TABLE I TEST SEQUENCES AND EXPERIMENTAL RESULTS IN TERMS OF BD-RATE FOR THE LDP, LDB AND RA CODING CONFIGURATIONS. NEGATIVE BD-RATE VALUES INDICATE A GAIN IN CODING EFFICIENCY. Type Sequence Name Resolution FPS Length VS. Anchor 1(%) VS. Anchor 2(%) LDP LDB RA LDP LDB RA Kimono ParkScene Class B Cactus BasketballDrive BQTerrace BasketballDrill Class C BQMall PartyScene RaceHorses BasketballPass Class D BQSquare BlowingBubbles RaceHorses FourPeople Class E Johnny KristenAndSara Affine Jets SpinCalendar Tractor BlueSky Average savings range from 0.2% lost (BQTerrace, LDB) to 17.1% gain (Tractor, LDP). The average coding efficiency compared to HM 12.0 are 3.5%, 2.2%, and 1.4% in configuration LDP, LDB and RA, respectively. When compared to [9], the results also prove that the proposed algorithms are better than previous scheme. The maximum gain is 4.3% in SpinCalendar, LDB configuration. It can be explained by the enhancement of available candidate set for affine modes, which increase the Affine SKIP/MERGE modes coverage rate for test sequences. It can be seen from the percentage coverage of Affine SKIP/MERGE modes shown in Table II. For all configurations, the results of sequences in Class Affine perform obvious better than others, especially in Tractor and SpinCalendar. For Tractor, a sequence with a camera zoom motion, and SpinCalendar, a sequence with a global rotation motion, the proposed coder achieves a bit rate reduction of 17.1% and 11.8%, respectively, in LDP configuration. The RD curve for coding the sequence Tractor is presented in Fig. 4. The RD curve shows that the proposed algorithms not only save coding bit rates, but also increase the video quality. All results prove that our proposed scheme is suitable for situation with complex motion. Consider about the complexity of the proposed algorithm, it increase about 20% encoding time and 30% decoding time compared to HM The computational complexity added by the affine SKIP and Merge modes mainly comes from the extra interpolations required by affine motion compensation, which will be further optimized in the future. And for the encoder side, additional mode decision process also raise the encoding time, using early termination strategy can well deal with this situation. Fig. 4. TABLE II COMPARISON OF AFFINE MODES USAGE COVERAGE BETWEEN PROPOSAL AND ANCHOR 2 Sequence Name Proposal Anchor 2 LDP LDB RA LDP LDB RA Jets SpinCalendar Tractor BlueSky Y PSNR(dB) Y PSNR vs Bitrate 34 HM Prop Bitrate(kbps) RD Curve of coding the sequence Tractor at LDP Configuration. V. CONCLUSIONS Affine SKIP/DIRECT modes based on affine motion model are proposed in previous work [9]. To further improve the coding efficiency, we extend these modes into PU level and utilize motion information from temporal picture for constructing affine motion prediction candidate set. Furthermore, to solve the problem that candidates may have different reference pictures and prediction directions, a scaled motion vector scheme is proposed. In comparison with the anchor in HM

5 12.0, the experimental results show on average 3.5%, 2.2%, and 1.4% bit rate reduction in configuration LDP, LDB and RA, respectively, can be achieved by the proposed methods. ACKNOWLEDGEMENT The authors acknowledge the support by Huawei Technologies Co., Ltd. REFERENCES [1] G. Sullivan, J. Ohm, W.-J. Han, and T. Wiegand, Overview of the high efficiency video coding (hevc) standard, Circuits and Systems for Video Technology, IEEE Transactions on, vol. 22, no. 12, pp , [2] J. Ohm, G. Sullivan, H. Schwarz, T. Tan, and T. Wiegand, Comparison of the coding efficiency of video coding standardsincluding high efficiency video coding (hevc), Circuits and Systems for Video Technology, IEEE Transactions on, vol. 22, no. 12, pp , [3] M. Narroschke and R. Swoboda, Extending hevc by an affine motion model, in PCS, 2013, pp [4] M. Karczewicz, J. Niewegłowski, and P. Haavisto, Video coding using motion compensation with polynomial motion vector fields, Signal Processing: Image Communication, vol. 10, no. 1-3, pp , [5] H. Jozawa, K. Kamikura, K. Yanaka, and H. Watanabe, Video coding using adaptive global mc and local affine mc, In EUSIPCO-96, Trieste, Italy, vol. 96, [6] K. Zhang, M. Bober, and J. Kittler, Video coding using affine motion compensated prediction, in Acoustics, Speech, and Signal Processing, IEEE International Conference on, 1996, pp [7] A. Glantz, M. Tok, A. Krutz, and T. Sikora, A block-adaptive skip mode for inter prediction based on parametric motion models, in Image Processing (ICIP), th IEEE International Conference on. IEEE, 2011, pp [8] H. Lakshman, H. Schwarz, and T. Wiegand, Adaptive motion model selection using a cubic spline based estimation framework, in Image Processing (ICIP), th IEEE International Conference on. IEEE, 2010, pp [9] H. Huang, J. W. Woods, Y. Zhao, and H. Bai, Affine skip and direct modes for efficient video coding, in VCIP, 2012, pp [10] I.-K. Kim, J. Min, T. Lee, W.-J. Han, and J. Park, Block partitioning structure in the hevc standard, Circuits and Systems for Video Technology, IEEE Transactions on, vol. 22, no. 12, pp , [11] I.-K. Kim, K. McCann, K. Sugimoto, B. Bross, and W.-J. Han, High efficiency video coding (hevc) test model 12 (hm 12) encoder description, in Proc. 14th JCT-VC Meeting, Vienna, Austria, Jul. JCTVC-N1002, 2013, pp [12] F. Bossen, Common test conditions and software reference conguration, in Proc. 12th JCT-VC Meeting, Genera, Switzerland, Jan. JCTVC- L1100, 2013, pp [13] G. Bjontegaard, Calculation of average psnr differences between rdcurves, in ITU-T VCEG document VCEG-M33. Austin, Texas, USA, Apr. 2001, 2001.