H.264/AVC Baseline Profile to MPEG-4 Visual Simple Profile Transcoding to Reduce the Spatial Resolution Jae-Ho Hur, Hyouk-Kyun Kwon, Yung-Lyul Lee Department of Internet Engineering, Sejong University, Kunja-Dong, Kwangjin-Gu, Seoul, Korea Received 9 May 2006; accepted 16 May 2006 ABSTRACT: In this study, we propose a heterogeneous transcoding method of converting an H.264/Advanced video coding (AVC) Baseline profile (BP) video bitstream into an MPEG-4 Visual simple profile (VSP) video bitstream. The proposed method reduces the spatial resolution for mobile terminals, which support only low resolution video bitstreams. When the H.264/AVC BP video bitstream is transformed into the MPEG-4 VSP video bitstream, the conversions between the H.264/AVC BP block types and the MPEG-4 VSP block types are performed by analyzing the macroblocks (MBs) conversion probability and calculating the difference values of motion vector. The proposed transcoding method runs on average 5.5 times faster than the cascaded transcoding methods, for a degradation of the PSNR (peaksignal-to ratio) of less than 0.5 db. VC 2006 Wiley Periodicals, Inc. Int J Imaging Syst Technol, 16, 24 33, 2006; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ima.20064 Key words: transcoding; H.264/AVC; MPEG-4; motion estimation; motion compensation I. INTRODUCTION Multimedia communication has become one of the fastest growing branches of the communications industry. Multimedia services for network environments, such as video communications, digital libraries, video adaptations, and video on demand, are becoming increasingly prevalent. In multimedia applications, it is often necessary to adapt the bitrates of the video bitstream to the bandwidths of the different kinds of communication channels. Therefore, various video transcoding techniques (Youn et al., 1999; Vetro et al., 2003) have been developed to convert one compressed bitstream format into another, while guaranteeing the QoS (quality of service). These different transcoding technologies are commonly classified according to the techniques that are used for performing transcoding in the pixel-domain (Yin et al., 2000, 2002) and in the DCT (discrete cosine transform)-domain (Assuncao and Ghanbari, 1998; Lin and Lee, 2001). So far, most transcoding methods have been developed to perform conversions between homogeneous video bitstreams, such as MPEG-2 to MPEG-2 (Assuncao and Ghanbari, 1998), H.261 to H.261 (Morrison, 1994), and H.263 to H.263 (Sun et al., 1996). Correspondence to: Yung-Lyul Lee; e-mail: yllee@sejong.ac.kr In this study, a pixel-domain transcoding method is proposed for conversion of H.264/Advanced video coding (AVC) baseline profile (BP) (Wiegand, 2003) and MPEG-4 visual simple profile (VSP) (Li et al., 2000), because DCT-domain transcoding cannot be employed in this case, due to the nonlinear loop filtering (List et al., 2003) included in the H.264/AVC standard. In terms of the video quality, the best transcoding method is a cascaded pixel-domain transcoding technique that first decodes the encoded bitstream completely and then reencodes the decoded video. However, using this method increases the computational complexity, because reencoding the decoded video requires all of the video coding tools to be utilized. Therefore, a transcoding method is developed, which reuses the incoming information contained in each macroblock (MB) in the H.264/AVC decoder, in order to reduce the computational complexity. However, in order to take into account the case of small MPEG-4 compatible mobile devices, which cannot decode H.264/ AVC bitstreams compressed with high resolution because of their low processing power and limited transmission bandwidth, a transcoder is required to change the compressed bitstream format and reduce the spatial resolution before the compressed bitstream is transmitted to the small mobile device. The main difference between the H.264/AVC BP and the MPEG-4 VSP formats is described in Section II. The new transcoding method, which operates by block type conversion from the H.264/AVC BP blocks to the MPEG-4 VSP blocks, is described in Section III, along with its motion vector (MV) mapping technology. The experimental results and analyses of the proposed algorithm are provided in Section IV, and our concluding remarks are given in Section V. II. H.264 AND MPEG-4 As shown in Table I, in comparison to MPEG-4 VSP, the H.264/ AVC BP has some superior features, such as the 4 4 integer transform, multiple reference frames, universal variable length coding (UVLC) or context-adaptive variable length coding (CAVLC), and the various block types used for motion estimation (ME) and motion compensation (MC). The H.264/AVC BP performs quarterpixel ME/MC for the seven variable-sized blocks, namely the 16 16, 16 8, 8 16 and 8 8 MC units for each 16 16 MB and 8 4, 4 8 and 4 4 MC units for each 8 8 block, as shown in ' 2006 Wiley Periodicals, Inc.
Table I. Coding tools of the H.264/AVC BP and the MPEG-4 VSP. Coding Tools H.264/AVC BP MPEG-4 VSP DCT 4 4 Integer Transform 8 8 DCT MC Unit 16 16, 16 8, 8 16, 8 8, 16 16, 8 8 8 4, 4 8, 4 4 MC Accuracy 1/4 pel 1/2 pel VLC Table Universal VLC, CAVLC Separable table Intra prediction Spatial prediction AC/DC Prediction Inner loop filter Deblocking filter None Figure 1a. Therefore, there are many different block modes in each MB, including the seven Inter modes, and the Intra16 16, Intra4 4 and SKIP modes, as shown in Figure 1a. On the other hand, the MPEG-4 VSP performs half-pixel ME/MC for two blocks, namely the 16 16 and 8 8 blocks of each MB, with the result that each MB contains the Inter16 16, Inter8 8, Intra and SKIP modes, as shown in Figure 1b. III. PROPOSED TRANSCODING METHOD As shown in Figure 2, in cascaded pixel-domain transcoding, the H.264/AVC BP bitstream is first decoded by the H.264/AVC decoder and the decoded video is then reencoded by the MPEG-4 VSP encoder, without reusing the information contained in each MB. The cascaded transcoder requires a high computational complexity, because it must carry out the decisions for the ME/MC and MB modes again. A. Proposed Spatial Resolution Reduction Transcoding. In this Section, the block-type conversion method and motion vector mapping process of the proposed heterogeneous transcoder are introduced, in which the heterogeneous transcoder converts one format of video stream (e.g., H.264/MPEG-4 AVC BP) into another format of video stream (e.g., MPEG-4 VSP). Performing brute-force ME (motion estimation) and mode decision for each MB causes a transcoder to have high computational complexity. To reduce this computational complexity, the incoming motion vectors are used for motion vector mapping. In the proposed transcoder, which is depicted in Figure 3, the MPEG-4 VSP encoder utilizes the motion vectors and MB information contained in each MB in the H.264/AVC BP bitstream, as shown in the dotted lines in this figure. The H.264/AVC standard adopts 1/4 pixel ME/MC, seven Inter-blocks, and Intra16 16, Intra4 4 and SKIP blocks, as shown in Figure 1a. Therefore, the block-type conversions between the H.264/AVC BP bitstream and the MPEG-4 VSP bitstream need to be performed by transforming the MB modes listed in Figure 1a to those listed in Figure 1b. As a reference, in the case where there is no residual data, the SKIP mode in the H.264/AVC standard can have the (0,0) motion vector or the median predicted motion vector (PMV), whereas the SKIP mode in the MPEG-4 Visual standard can only have the (0,0) motion vector. The proposed spatial resolution reduction transcoding operation is performed by adding an average filter that down-samples 2 2 pixels to one pixel between the H.264/AVC decoder and the MPEG-4 VSP encoder, as shown in Figure 3. The spatial resolution reduction of 2 2 MBs (a total of four MBs) in the H.264/AVC standard to one MB in the MPEG-4 Visual standard, due to the spatial resolution being reduced to one half of its original size, is performed both horizontally and vertically. Before the transcoding operation is performed, every SKIP mode MB in the H.264/AVC BP bitstream is changed into an Inter16 16 mode MB as a preprocessing step for using 2 2 MBs, as shown in Figure 4. The SKIP MB mode in the H.264/AVC standard can have the (0,0) motion vector or the PMV when there is no residual data. Before developing the proposed transcoder using block mode conversion and motion vector mapping to reduce the spatial resolution, we investigated the block mode decision results in the case of an H.264/AVC BP to MPEG-4 VSP cascaded transcoder, which down-samples the spatial resolution, and the results are shown in Table II. To obtain the results, the simulation was performed by using H.264/AVC BP bitstreams encoded by using the QP value of 20 for the various test sequences, such as \Foreman," \Paris," \News," and \Coast." And MPEG-4 VSP compliant output bitstream was reencoded by using QP value of 12 during the cascaded Figure 1. MB modes in video coding standards: (a) MB modes in the (a) H.264/AVC BP, (b) MPEG-4 VSP. Vol. 16, 24 33 (2006) 25
Figure 2. Cascaded pixel-domain transcoding. transcoding. When a group of 2 2 MBs in the H.264/AVC BP bitstream includes at least two MB types smaller than the Inter16 8 or Inter8 16 block mode, these 2 2 MBs are mostly converted to the SKIP (35.99% ¼ 13.53% + 13.91% + 8.55%), Inter16 16 (6.5% ¼ 3.48% + 2.16% + 0.86%), and Inter8 8 (8.34% ¼ 1.88% + 3.21% + 3.25%) modes of the MPEG-4 VSP block modes, as shown in the first three rows of Table II. This combination of 2 2 MBs amounts to *51.37% of the data in the H.264/AVC BP bitstream. When a group of 2 2 MBs is composed of four Inter16 16 or three Inter16 16 + one MB block type all of which are smaller than the Inter16 8 or Inter8 16 block mode in the H.264/AVC BP bitstream, these 2 2 MBs are mostly converted to the Inter16 16 (31.69% ¼ 25.94% + 5.75%) or SKIP (11.49% ¼ 3.48% + 8.01%) mode of the MPEG-4 VSP block modes, as shown in the 4th and 5th rows of Table II. This combination of 2 2 MBs amounts to *44.12% of the data in the H.264/AVC BP bitstream. When a group of 2 2 MBs is composed of one Intra (Intra16 16 or Intra4 4) + three Inter, or two Intra + two Inter modes in the H.264/AVC BP bitstream, these 2 2 MBs are mostly converted into the Inter8 8, SKIP, or Inter16 16 mode of the MPEG-4 VSP block modes, as shown in the 6th and 7th rows of Table II. Finally, when a group of 2 2 MBs is composed of three Intra (Intra16 16 or Intra4 4) + one Inter or four Intra modes in the H.264/AVC BP bitstream, these 2 2 MBs are intentionally converted into the Intra mode of the MPEG-4 VSP block modes, without considering the data referred to in the 8th and 9th rows of Table II, because the probability of this combination of 2 2 MBs occurring is too small to consider. The proposed transcoding method of reducing the spatial resolution was developed on the basis of the data listed in Table II, which correspond to the results obtained when using the cascaded transcoding method to reduce the spatial resolution. The proposed method can be described as follows: 1. The 2 2 MBs include at least two MB block types smaller than the Inter16 8 or Inter8 16 block mode (Fig. 5a). Each MB in the H.264/AVC BP bitstream is converted into one 8 8 block in the MPEG-4 VSP encoder, in which the quarterpixel motion vectors, a_mv i,j, where i,j ¼ 0,1, of each 8 8 block in an MB are calculated by averaging the quarter-pixel motion vectors in the sixteen 4 4 subblocks, mv m,n,k,l, where k,l ¼ 0,1,2,3, of the H.264/AVC standard and dividing them by 16. The 4 4 subblock is a basic motion vector unit of the H.264/AVC standard, in terms of the motion vector memory implementation, so that the motion vectors are calculated as follows: 0 a m i;j ¼ B @ P 3 P 3 k 0 j 0 1 m m;n;k;l þ 8, þ 1C 2 i; j ¼ 0; 1; ð1þ 16 A Figure 3. Proposed transcoding method. 26 Vol. 16, 24 33 (2006)
Figure 4. Preprocessing step before block-type conversion and motion vector mapping. where the subscripts i and j denote the horizontal and vertical motion vector indices of the 8 8 subblocks in the intermediate MB, respectively, as shown in Figure 5a, and the subscripts k and l denote the horizontal and vertical motion vector indices of the 4 4 subblocks in each (m,n)-th MB of the H.264/AVC standard, respectively. The divisor of 2 in Eq. (1) is needed, because the horizontal and vertical resolutions are reduced to one half of their original size. In this case, the SKIP mode is the same as the Inter16 16 mode, except that the a_mv i,j vector is a zero vector and the CBP coefficients are zero. The block conversion process is applied such that if the direction of each a_mv i,j is the same and the difference values between the four motion vectors, a_mv i,j, are smaller than 9, then the Inter16 16 mode is selected, otherwise the Inter8 8 mode is selected. The threshold value of 9 was chosen based on the results of two experiments. The first experiment was performed in three cases. Case I is the conversion of the intermediate MB mode of Figure 5a into the Inter16 16 mode of the MPEG-4 block mode, Case II is the conversion of the intermediate MB mode of Figure 5a into the Inter8 8 mode, and Case III is an alternative method in which, if the direction of each a_mv i,j is the same and the difference values between a_mv i,j values are all smaller than 9, then the intermediate MB mode is converted into the Inter16 16 of the MPEG-4 block mode, otherwise the intermediate MB mode is converted into the Inter8 8 mode of the MPEG-4 block mode. The PSNR-bitrate curves of these three cases are shown in Figure 6. Case I shows a better PSNR result than both Case II and Case III at lower bitrates, whereas Case III (the alternative method) using the threshold T ¼ 9 shows a better result than both Case I and Case II at higher bitrates and at our target bitrate range of 130 200 kbps for a mobile terminal. Therefore, we selected the alternative method. However, Case I can be applied if the target bitrates of the application in question are in the low range. After applying the alternative method using various thresholds as the second experiments, as shown in Figure 7, we found that the best result is obtained when T is 9, even if the PSNR values are quite similar in the higher threshold values. Therefore, a threshold value of 9 was selected based on the two experimental results shown in Figures 6 and 7. 2. The 2 2 MBs are composed of four Inter16 16 or three Inter16 16 + one MB block modes, which is smaller than the Inter16 8 or Inter8 16 block mode (Fig. 5b). The direct conversion to the Inter16 16 mode is performed in the MPEG-4 VSP encoder of the transcoder. The SKIP mode is one of the Inter16 16 modes. 3. In Figure 5c, the 2 2 MBs are composed of one Intra + three any Inter or two Intra + two Inter modes. Table II. The block mode conversion results for cascaded transcoding from the H.264/AVC 2 2 MB modes to the MPEG-4 MB mode for various CIF test sequences. MB Modes of the MPEG-4 VSP 2 2 MB Modes of the H.264/AVC BP Intra SKIP Inter 16 16 Inter 8 8 Total Two Inter16 16, two blocks smaller than 0.20 13.53 3.48 1.88 19.09 Inter16 8 or Inter8 16 (Fig. 5a) One Inter16 16, three blocks smaller than 0.22 13.91 2.16 3.21 19.49 Inter16 8 or Inter8 16 (Fig. 5a) Four blocks smaller than Inter16 8or 0.14 8.55 0.86 3.25 12.79 Inter8 16 (Fig. 5a) Four Inter16 16 (Fig. 5b) 0.26 3.48 25.94 0.09 29.77 Three Inter16 16, one block smaller than 0.13 8.01 5.75 0.47 14.35 Inter16 8 or Inter8 16 (Fig. 5b) One Intra, three Inter (Fig. 5c) 0.03 1.15 0.32 1.26 2.76 Two Intra, two Inter (Fig. 5c) 0.02 0.37 0.10 0.65 1.14 Three Intra, one Inter (Fig. 5d) 0.00 0.10 0.04 0.16 0.31 Four Intra (Fig. 5d) 0.05 0.09 0.01 0.14 0.29 Total number 1.04 49.19 38.65 11.11 100.00 All values given are in percentages. Vol. 16, 24 33 (2006) 27
28 Vol. 16, 24 33 (2006) Figure 5. Block-type conversion and motion vector mapping.
Figure 6. PSNR comparisons when Case I (all 16 16), Case II (all 8 8), and Case III (alternative method) are applied for various bitrates. The Intra block, which can be either an Intra16 16 or Intra4 4 block, is converted into an Inter16 16 block by using the average motion vector of the Inter blocks in the 2 2 MBs. Then, the same process as that shown in Figure 5a is applied. In this case, any Inter mode means all kinds of Inter block mode regardless of blockpartitioning. 4. The 2 2 MBs are composed of three Intra + one Inter or four Intra modes (Fig. 5d). These 2 2 MBs are intentionally converted into the Intra mode in the MPEG-4 VSP encoder, without considering the data in Table II, because the probability of this combination of 2 2 MBs occurring is too small to consider. Figure 8 shows the flow chart of the proposed transcoder designed to reduce the spatial resolution. The motion vector refinement process for this transcoder is explained in Section III.B. Figure 8 gives a summary of the transcoder described in this Section. The SKIP mode in the MPEG-4 Visual standard is the same as the Inter16 16 block mode, except that the motion vector is zero and the CBP coefficients are zero. B. Motion Vector Refinement. To improve the coding efficiency, which can be degraded by the motion vector mapping calculations corresponding to Eq. (1), motion vector refinement is performed. Before the motion vector refinement process, the Figure 7. PSNR comparisons when various thresholds are applied to the difference values between a_mv i,j vectors in Case III. Vol. 16, 24 33 (2006) 29
Figure 8. Flow chart of block-type conversion including motion vector mapping. The divided MB is a macroblock that has a block type, which is smaller than the Inter16 8 or Inter8 16 block mode. 30 Vol. 16, 24 33 (2006)
Figure 9. Average PSNR-bitrate curves in the proposed transcoding method that reduces the spatial resolution according to the motion vector refinement window. quarter-pixel motion vector is 1/4 scaled to the integer-pixel motion vector. In the proposed H.264/AVC BP to MPEG-4 VSP transcoding operation, a search for the 1 integer-pixel motion vector is performed, and then a half-pixel motion search involving the eight nearest neighbor half-pixels immediately surrounding the integerpixel motion vector is performed to find the best half-pixel motion vector. To determine the optimum search window of the integer motion vector used for the motion vector refinement, various search ranges in integer units were experimented with. Figure 9 shows the rate-distortion curves when the search windows in the proposed transcoder were 0, 1, 2, 3, and 4, respectively. When a motion vector refinement search window of 1 was applied, the PSNR values of this search window were almost saturated in comparison with those of a higher motion vector refinement search window. IV. SIMULATION RESULT The proposed transcoding method was implemented using the H.264/AVC JM 7.3 decoder (http://bs.hhi.de/~suehring/tml/download/old_jm/jm73.zip) and the MPEG-4 Visual MoMuSys-FDIS- Figure 10. Transcoding time comparisons between the cascaded and proposed transcoding methods, which reduce the spatial resolution from CIF size to QCIF size, in which a relative time comparison is performed. Vol. 16, 24 33 (2006) 31
Figure 11. Comparison of the PSNR between the cascaded transcoding method and the proposed transcoding method. V1.0 encoder (MoMuSys-FDIS-V1.0-990812). To evaluate the performance of the proposed transcoding method, the PSNR values, bitrates, and computation times were analyzed for four video sequences, namely the \Foreman" sequence of the CIF (Common Intermediate Format) formats, as well as the \Coast" sequence, \News" sequence, and \Paris" sequence of the same format, all of which have 300 frames. For the evaluation of the proposed H.264/AVC BP to MPEG-4 VSP transcoding method that reduces the frame size, the H.264/ AVC and MPEG-4 frame rates were set to 30 fps and the CIF sequences were used as the input sequences. For these two sets of experiments, each sequence was compressed with the \I, P, P, P,..." scheme, i.e., only the first frame is an Intraframe. The computation times of the proposed transcoding methods were compared with those of the cascaded transcoding method, as shown in Figure 10. In Figure 10, the average speed of the proposed resolution reduction transcoder is 5.5 times higher than that of the cascaded resolution reduction transcoder. Quantization values of 20 and 12 were used in the H.264/AVC BP and MPEG-4 VSP bitstreams, respectively, for our target bitrates. Figure 11 shows the plots of the PSNR-bitrate for the cascaded transcoding method versus that of the proposed transcoding method. In Figure 11, the PSNR of the proposed transcoding method is almost same as that of the cascaded transcoding method for the \News," \Paris," and \Coast" sequences, but is about 0.1 0.5 db lower than that of the cascaded transcoding method in the case of the \Foreman" sequence. First, the proposed block mode conversion from 2 2 input MBs in H.264/MPEG-4 AVC into one MB in MPEG-4 VSP selects the candidate MB modes according to the MB statistics in Table II. Second, in the case of Figure 5a 5c, the differences of the quarterpixel unit motion vector in the candidate Inter MB (Inter16 16, SKIP, or Inter8 8) are calculated to decide the final Inter MB mode in MPEG-4 VSP as shown in Figures 5 and 8. Third, in the case of Figure 5d, the candidate MB is decided to Intra MB. Therefore, the proposed method performs an efficient block mode decision by utilizing the candidate MB statistics and final MB mode selection. V. CONCLUSION We developed a low-complexity heterogeneous H.264/AVC BP to MPEG-4 VSP video transcoder, by reusing the 4 4 block based- H.264/AVC motion vectors and utilizing the H.264/AVC MB information. Through the results of the simulation, we showed that the proposed transcoding method is able to reduce the computational complexity without degrading the video quality. Therefore, the proposed transcoding method can be used in practical multimedia applications, such as digital libraries and video adaptations, in which the users only have an MPEG-4 compatible terminal. ACKNOWLEDGMENTS This research is partially supported by the Ubiquitous Computing and Network (UCN) Project and the Ministry of Information and 32 Vol. 16, 24 33 (2006)
Communication (MIC) 21 st Century Frontier R & D Program in Korea. REFERENCES P. Assuncao and M. Ghanbari, A frequency domain video transcoder for dynamic bit-rate reduction of MPEG-2 bitstreams, IEEE Trans Circ Syst Video Technol 8 (1998), 953 967. J. Jeongnam Youn, M.T. Sun, and C.W. Lin, Motion vector refinement for high performance transcoding, IEEE Trans Multimed 1 (1999), 30 40. C.W. Lin and Y.R. Lee, Fast algorithms for DCT-domain video transcoding, in Proceedings of the IEEE International Conference on Image Processing, Thessaloniki, Greece, September, 2001, Vol. I, pp. 421 424. D.G. Morrison, M.E. Nilsson, and M. Ghanbari, Reduction of bit-rate of compressed video while in its coded form, in Proceedings of the 6th International Workshop on Packet Video, Portland, OR, September, 1994, pp. 392 406. P. List, A. Joch, J. Lainema, G. Bjøntegaard, and M. Karczewicz, Adaptive deblocking filter, IEEE Trans Circ Syst Video Technol 13 (2003), 614 619. H. Sun, W. Kwok, and J.W. Zdepski, Architectures for MPEG compressed bitstream scaling, IEEE Trans Circ Syst Video Technol 6 (1996), 191 199. T. Wiegand, Joint Final Committee Draft(JFCD) of joint Video specification (ITU-T Rec. H. 264 ISO/ICE 1449-10 AVC), JVT-G050, March 2003. A. Vetro, C. Christopoulos, and H. Sun, Video transcoding architectures and techniques: An overview, IEEE Signal Process Mag (2003), 18 29. W. Li, J.-R. Ohm, M. van der Schaar, H. Jiang, S. Li, MPEG-4 Video Verification version 17.0, ISO IEC JTC1 SC29 WG11 N3515, Beijing, July, 2000. P. Yin, M. Wu, and B. Lui, Video transcoding by reducing spatial resolution, in Proceedings of the IEEE International Conference on Image Processing, Vancouver, BC, Canada, October, 2000,Vol. I, pp. 972 975. P. Yin, M. Wu, B. Lui, and H. Sun, Drift compensation for reduced spatial resolution transcoding, IEEE Trans Circ Syst Video Technol 12 (2002), 1009 1020. Vol. 16, 24 33 (2006) 33