CNN ORIENTED FAST PU MODE DECISION FOR HEVC HARDWIRED INTRA ENCODER

Transcription

1 CNN ORIENTED FAST PU MODE DECISION FOR HEVC HARDWIRED INTRA ENCODER Nan Song, Zhenyu Liu, Xiangyang Ji, Dongsheng Wang RIIT&TNList/ Department of Automation/ IMETU, Tsinghua University, Beijing , China. ABSTRACT The number of intra prediction modes in High Efficiency Video Coding (HEVC) has been increased up to 35. To the end of alleviating the complexity of intra coding, we bring in the convolution neural network (CNN) to obtain the candidate modes of current PU and adopt the corner detection algorithm to further reduce the candidate modes. The virtues of proposed algorithm include: Firstly, our algorithm skip the rough PU mode decision (RMD) process and get the candidate list from CNN directly. In other words, the computations are relaxed in our algorithm. Secondly, the inputs of CN- N in proposed algorithm merely contain the source image pixels and quantization parameter (QP), this feature makes it friendly to high parallel hardwired encoder. As compared with HM-15.0, experiments show that our algorithm decreases the intra coding time by 27.92% while the corresponding BDBR augment is 1.15%. At last but not least, our algorithm possesses a stable coding performance. In specific, for the most sensitive sequence (Class F), our algorithm could save 27.10% intra coding time with 2.01% BDBR increase. Index Terms HEVC, intra coding mode, fast PU mode decision, CNN, corner detection 1. INTRODUCTION High Efficiency Video Coding (HEVC)[1], the latest video coding standard, was jointly introduced by International T- elecommunications Union (ITU-T) and International Standardization Organization (ISO/IEC) in As compared with H.264/AVC (the previous generation standard), HEVC reduced 50% bit-rate with the same subjective picture quality, especially on the video of high-definition specification. To improve the accuracy of signal prediction, HEVC introduced more complex prediction algorithms than H.264/AVC in intra coding. Firstly, HEVC introduces a new set of units for the splitting of pictures, which including coding unit(cu), prediction unit(pu) and transform unit(tu). Those new units makes the picture partition of HEVC is more flexible and precise than H.264/AVC. Secondly, the number of prediction modes employed by HEVC has been increase up to 35, including DC mode, Planar mode and 33 kinds of angular This work is funded by National Science and Technology Major Project (2016YFB ), Projects of International Cooperation and Exchanges NSFC ( ). modes. In contrast, H.264/AVC is just equipped with 9 kinds of modes for 4 4 and 8 8 blocks, and 4 modes for blocks[2]. experiments revealed that the encoding complexity of HEVC is about 5 times of H.264/AVC s. Namely, the high compression efficiency of HEVC is achieved at the cost of the high computational complexity[3]. In intra coding mode, PU mode decision costs about 60% 70% time. And in PU mode, the computations come from rough PU mode decision (RMD) process and full rate distortion optimization (RDO) process. For RMD stage, there are mainly two approaches to reduce the complexity. The first one is adopting the edge strength extractor to get PU candidate list. For example, In Chen et al. work, they calculated edge strength to obtain the kernel density estimation alike histogram and voted the candidate modes[4]. Another method applied the rough to fine search strategy to reduce the overall complexity in RMD stage. For instance, Fini et al. first calculated rough costs of 19 prediction modes, including even number angular predictions and DC/Planar predictions, and then refined the angular prediction search around the most promising direction [5]. To decrease candidate list for RDO, algorithms in literature [5] [6] proposed to adopt texture information to reduce the candidate list of current PU. While Gwon et al. brought in Bayesian network as the technique to decrease the candidate modes[7]. In fact, the source texture extractor defined experimentally can hardly get the accurate PU directional modes, which are determined by the edge curvature, texture topology, quantization scale, beside the edge direction. For example, Chen et al. work, which is based on the edge direction statistics, brought in 3.47% BDBR increase in class F coding. In addition, a VLSI friendly algorithm should maintain the high parallelism. For example, the data dependency between neighboring PUs of algorithms in literatures [5-7] impedes the parallel processing. In this paper, we introduce CNN [8] as the classification to get the candidate modes and further reduce the candidate list by corner detection algorithm. When coding class F sequences, the worst BDBR augment is lowered to 2.01%. Since we use the source image pixels as the CN- N inputs, the proposed algorithm is friendly to high parallel hardwired encoder design. The rest of paper is organized as follows. In Section 2, we introduce the fast PU mode decision algorithm using convolution neural network. The CNN architecture and its training /17/$ IEEE 239 GlobalSIP 2017

2 Fig. 1. Flowchart of intra PU mode decision(the procedures in blue blocks represent our works. ) method are described in Section 3. The experimental results are illustrated in Section 4. Finally, we give the conclusions in Section CNN BASED FAST PU MODE DECISION METHOD Function FastPUmode(P, QP) ifs p == 8 then I DownSamp(P) I = P forj = 0 4 do fori = 0 4 do x i,j = I i,j +I i+1,j I i,j+1 I i+1,j+1 y i,j = I i,j I i+1,j +I i,j+1 I i+1,j+1 iedgepwr+ = ( x 2 i,j + y2 i,j ) end for end for ifiedgepwr < TH 1 then C n DC and Planar if iedgepwr > TH 2 then ifs p == 4 then C m CNN4 strong(i,qp) C m CNN8 strong(i,qp) C n CornerDetection(I,C m) ifs p == 4 then C m CNN4 weak(i,qp) C m CNN8 weak(i,qp) C n CornerDetection(I,C m) return C n End Function In original HM software, the intra PU mode decision mainly contains two processes, i.e., rough PU mode decision (RMD) process and full rate distortion optimization (RDO) process. The flowchart of intra PU mode decision task in original H- M is described as Fig.1.(a). In RMD process, PU traverses 35 modes to get N modes which map the minimum SATD costs as the candidate list. Obviously, when the PU size is 4 4/8 8, its candidate list possesses 8 10 modes. In RDO stage, PU gets the best mode by searching the candidate list. In this paper, we employ CNN to process the 8 8 and 4 4 PU mode decision task. The flowchart of our method is shown in Fig.1.(b). In the flowchart, we first get pixels matrix P and quantization parameter(qp) when PU size is 4 4/8 8. The matrix P comprises current PU and its top row and left column neighboring pixels. Then we feed QP and matrix P to FastPUMode function. The function FastPUMode is the essential procedure, which applies the convolution neural network to determine the PU candidate list. It is worth mentioning that, in our method, about 63.84% PUs whose candidate list are no more than 5 modes. Thus, the computational complexity of RDO stage is relieved. The pseudo code of FastPUMode function is depicted by Fig. 2. For the current PU with size 8 8, we first apply the local-averaging and sub-sampling function DownSamp to derive the 5 5 matrix I. Specifically, we adopt 2 2 filter to sub-sample the current PU pixels of 8 8 PU by calculating the means of the window. For 4 4 PU, its matrixiis the same to P. We next carry out the coarse edge strength analysis to divide the PU into three types, i.e., flat PU, weak edge PU and strong edge PU. In this procedure, two auxiliary thresholds TH 1 and TH 2 are defined. In our experiments, TH 1 = 4 Fig. 2. Pseudo code of FastPUMode function (I i,j : The entry in matrix I, x, y: The horizontal and vertical components of the edge,iedgepwr: The coarse edge strength of PU. ) QP 2,TH 2 = 25 QP 2. If PU whoseiedgepwr is smaller than TH 1, we think it is flat PU. While iedgepwr of PU is larger thanth 2, the PU is strong edge. Otherwise, it is weak edge PU. In FastPUMode function, if the PU is flat, DC and Planar modes are appropriate to be chosen as its candidate list C n. If the PU is a strong one, we send it to CNN strong. Otherwise, it will be given to CNN weak. It is noted that, we choose 8 modes which map the maximum outputs of CNN as the PU candidate list C m. In CornerDetection function, the candidate list C m would be further reduced. We first adopt sobel operator to get the horizontal and vertical components of the gradient as, Gx i,j = I i+1,j 1 +2 I i+1,j +I i+1,j+1 I i 1,j 1 2 I i 1,j I i 1,j+1, (1) Gy i,j = I i 1,j+1 +2 I i,j+1 +I i+1,j+1 I i 1,j 1 2 I i,j 1 I i+1,j 1 in where, I i,j is the entry in matrix I with i,j [1,3]. Thus we will get 9 groups ofgx i,j,gy i,j in different(i,j). Then the matrix M which is borrowed from Harris[9] will be calculated as, M = [ A C C B ], (2) in which, A = i,j Gx2 i,j, B = i,j Gy2 i,j and C = i,j Gx i,j Gy i,j. 240

3 Fig. 3. The Proposed Architecture Of CNN Finally, we obtain the corner response R by Tr(M)2 Det(M). From Harris work, we know the R in inverse proportion to the probability that the block owning corner. In our experiments, if R < 10, we think there is a corner in the block. Thus, if current PU has no corner, we will choose the first 5 modes of C m as the new list C n. Otherwise, the C n is the same asc m. 3. PROPOSED ARCHITECTURE OF CNN To alleviate the computational complexity of hardwired, the CNN units in our algorithm are shared the same architecture, as depicted in Fig.3. The proposed architecture consists of input layer, two lower alternating convolution layers, and the upper full-connected Multilayer Perceptron (MLP). Counting the output layer, our CNN comprises five layers, which are explained as follows: The input is a 5 5 pixels block. In FastPUMode function, the input layer of CNN is matrixi. The first convolution layer has 8 feature maps. Each neuron is connected to a 3 3 field in the input layer receptively. The size of those feature maps is 3 3.The kernels in this layer are regarded as feature extractors. The second convolution holds seventeen 1 1 neurons including 16 outputs of the convolution kernels and one for QP. In our experiments, the PU mode list may be different with variable QP, thus we should consider QP as a parameter. The kernel size is 3 3 as well. The last two hidden layers are fully connected MLP. They possess 71 and 35 neurons respectively. It should be noted that the QP is also one input of first MLP. In our algorithm, the training method of CNN is based on Gradient Descent Algorithm[10]. The measures we taking to train CNN will be further introduced in the following two sub-section Training Data Extraction In the training sample selection stage, we adopt following s- trategies to it: Pick out flat PUs. From previous work[11], we studied that the flat PUs were burdens to the CNN because their simplex textures would dull the CNN training process. Fortunately, those PUs could be well predicted by DC and Planar as well. So it is suitable to pick them out from training data set. To take full advantage of the edge strength information, we do not normalize the input pixels. Thus vanishing gradient problem can be avoided by using the modified sigmoidal activation function. Finally, the training data set in our experiments come from nine video sequences which are including PeopleOnStreet, BasketballDrive, Kimono, PartyScene, RaceHorsesC, BasketballPass, Vidyo3, Johnny and ChinaSpeed with the QP ranging from 22 to Target Value Extraction About the selection of target values, instead of the traditional methods which choose modes as their classification, we take the target values based on rate distortion cost (RDC). The reason lies that the sample integer can hardly get the meaning of the PU modes, but the RDCs are the determining criterion of PU modes. We do not directly choose RDCs as the target values either, because the RDCs are so large for CNN that maybe cause the saturation of networks. The target values we select from follow steps: Get the 35 RDCs of current PU. In this step, we traverse all modes by RDO process instead of RMD process. We denote RDC as RDC m, m stands for the relevant mode. Calculate the average value C of RDCs, C = m=0 Derive the target valuet m as follows: - if therdc m < C,T m is, - otherwise, RDC m, (3) T m = ln( RDC m + C + e), (4) T m = ln(rdc m C + e), (5) in which, the function ln( ) is natural logarithm and e represents the natural base. In these equation, we can acknowledge that therdc m is in inverse proportion to thet m. Thus, we can choose 8 modes which map the maximum outputs of CNN as PU candidate list. 241

4 Table 1. Computational Complexity of CNN Architecture Tanh Add/Sub Mult Covn Covn MLP MLP Total Table 2. Coding Performance of Proposed Algorithm Class Sequence BDPSNR BDBR T [db] [%] [%] A PeopleOnStreet Traffic BasketballDrive BQTerrace B Cactus Kimono ParkScene Tennis BasketballDrill BasketballDrillText C BQMall PartyScene RaceHorsesC BasketballPass BlowingBubbles D BQSquare RaceHorses Keiba Vidyo Vidyo E Vidyo Johnny KristenAndSara F SlideEditing ChinaSpeed Average CNN with our scheme has a good performance on classification, the probability of the best PU mode in the candidate list C m is about Besides, the computational complexity of proposed CNN, as showed in Table 1, is also optimistic. It is obvious that, our CNN just costs 5475 add/sub operations and 5475 multiplications. Table 3. Performance Comparison Between Our Proposed Algorithm and Previous works Algorithm BDBR[%] T [%] Class F BDBR[%] T [%] Proposed [4] [5] [6] T = Tr Tp T r 100% with T r and T p denoting the intra encoding time of original HM-15.0 and the counterpart of our CNN based PU mode decision. It is observed that our algorithm decreases the intra coding time by 27.92% with the BDBR augment is only 1.15%. Besides, for the most sensitive sequences (class F), our algorithm reduces about 27.10% intra coding time on average while the corresponding BDBR increases to 2.01%. The table 3 shows the comparison between Our algorithm and precious algorithms. Algorithm [4] adopted empirical texture extractor to obtain candidate modes. In Algorithm [4], Chen et al. first calculated the edge strengths by 2 2 filter, then they got the kernel density estimation alike histogram and voted the candidate modes. As expected, the BDBR of proposed method is 0.31% lower than their work. And for Class F, the worst BDBR augment in our algorithm is reduced 1.46% as well. Algorithm [5] and [6] decreased the candidate modes by texture analysis. Compared with Algorithm [5] and [6], proposed algorithm saves about 14.11% and 9.03% intra coding time respectively. The reason lies that we introduce the corner detection algorithm to delete the modes of candidate list. In our statistic, with the addition of flat PUs, there will be about 63.84% PUs whose candidate list are no more than 5 modes. However, because algorithm [5] got the candidate list by the same method to original HM in RMD stage, the BDBR augment of our proposed algorithm is 0.81% higher than algorithm [5]. 4. EXPERIMENTS The proposed algorithm is compared with the HEVC test model reference software HM-15.0 to verify its performance in coding quality and coding speed. Since the main test platform is Huawei RH-5885 sever, which combines Intel R Xeon TM E v2 2.2GHz processors and 128GB memory. Totally, 25 video sequences including A to F class with four QP values of 22, 27, 32, 37 are tested and the configuration file is encoder intra main.cfg. Table 2 illustrates the coding performance of our proposed algorithm. In the evaluations of coding quality, BDPSNR and BDBR are used to represent the average PSNR and Bitrate differences[12]. T is regarded as the reference to the computational complexity reduction, which is represented as 5. CONCLUSION In this paper, we propose the algorithm of fast PU mode decision to the intra coding mode of HEVC. We apply CNN to select the candidate PU modes that undergo the refined RDO process. In addition, we propose the corner detection algorithm to further reduce the candidate modes list. The inputs of our CNN are merely composed of the source image pixels and QP. This feature prompts the proposed method efficient to the high parallel hardwired encoder design. As compared with the previous work, our algorithm provided the advanced performance in coding Class F sequences (27.10% time saving with BDBR = +2.01%). Experiments illustrated that our method could reduce the averaged 27.92% coding time at the cost of 1.15% BDBR augment. 242

5 6. REFERENCES [1] G. J. Sullivan, J. Ohm, Woo Jin Han, and T. Wiegand, Overview of the high efficiency video coding (hevc) standard, IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, no. 12, pp , [2] G. J. Sullivan and T. Wiegand, Video compression - from concepts to the h.264/avc standard, Proceedings of the IEEE, vol. 93, no. 1, pp , [3] Gary J. Sullivan and Jens Rainer Ohm, Recent developments in standardization of high efficiency video coding (hevc), Proc Spie, vol. 7798, no. 1, pp , [4] Guang Chen, Zhenyu Liu, T Ikenaga, and Dongsheng Wang, Fast hevc intra mode decision using matching edge detector and kernel density estimation alike histogram generation, in IEEE International Symposium on Circuits and Systems, 2013, pp [5] Mohammadreza Ramezanpour Fini and Farzad Zargari, Two stage fast mode decision algorithm for intra prediction in hevc, Multimedia Tools and Applications, vol. 75, no. 13, pp , [6] Thałsa L. Da Silva, Luciano V. Agostini, and Luis A. Da Silva Cruz, Fast hevc intra prediction mode decision based on edge direction information, in Signal Processing Conference, 2012, pp [7] Daehyeok Gwon, Haechul Choi, and Jonghee M. Youn, Hevc fast intra mode decision based on edge and satd cost, in Multimedia and Broadcasting, 2015, pp [8] Wu Liu, Tao Mei, Yongdong Zhang, Cherry Che, and Jiebo Luo, Multi-task deep visual-semantic embedding for video thumbnail selection, in Computer Vision and Pattern Recognition, 2015, pp [9] C Harris, A combined corner and edge detector, Proc Alvey Vision Conf, vol. 1988, no. 3, pp , [10] Yann Lcun, Leon Bottou, Yoshua Bengio, and Patrick Haffner, Gradient-based learning applied to document recognition, Proceedings of the IEEE, vol. 86, no. 11, pp , [11] Z. Liu, X. Yu, Y. Gao, S. Chen, X. Ji, and D. Wang, Cu partition mode decision for hevc hardwired intra encoder using convolution neural network., IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society, vol. 25, no. 11, pp , [12] Gisle Bjontegaard, Calculation of average psnr differences between rd-curves,