COMPRESSED DOMAIN SPATIAL SCALING OF MPEG VIDEO BIT- STREAMS WITH COMPARATIVE EVALUATION

COMPRESSED DOMAIN SPATIAL SCALING OF MPEG VIDEO BIT- STREAMS WITH COMPARATIVE EVALUATION M. Mahdi Ghandi, M. Emad Modirzadeh Tehrani, M. R. Hashemi, and Omid Fatemi ESE Department, University of Essex Department of Electrical and Computer Engineering, University of Tehran Wivenhoe park, Colchester, CO4 35Q, UK No. 68, Hedayat st., Darroos, 19497, Tehran, Iran P.O.BOX 14395/515, Tehran, Iran mahdi@essex.ac.uk emad@modirzadeh.com hashemi@comnete.com omid@fatemi.net ABSTRACT As more and more muldia content is delivered and stored in the MPEG-2 format, there is an increasing interest in developing processing techniques that can be applied directly to the compressed domain. In applications such as video browsing, picture in picture and in the forthcoming broadcast of high definition digital television (HDTV) to the home there is a need for spatial downscaling of a video stream before transmission or storage. In this paper we introduce an efficient to spatially scale MPEG-2 video in the compressed domain. We also present a comparative performance evaluation of compressed domain processing techniques. Simulation results indicate that the proposed scheme is faster than the current compressed domain s while maintaining a comparable subjective quality. Keywords- Spatial Scaling, Compressed Domain ing, Bit-rate Reduction, Video ing 1 INTRODUCTION Pre-encoded video is widely used in muldia applications for the efficient storage and transmission of information. MPEG-2, now established as a standard, is generally used for digital video broadcasting (DVB), digital video disk (DVD) and HDTV. In applications such as video browsing and picture in picture, and also due to the heterogeneity of the present transmission channels there is a need to scale down video bitstreams. With the forthcoming broadcast of HDTV programs to the home there will be an increasing demand for downscaled signals for existing television sets. The conventional approach to downscaling an MPEG-2 compressed video consists of three main steps: decoding the bitstream, performing the downscaling in the pixel domain, and the re-encoding of the resulting sequence. Figure 1 illustrates the block diagram of such a system. This approach is computationally intensive, and places a heavy burden on the server it is running on, both in terms of computing power and storage capacity. The burden increases when the server has to support quality of service to heterogeneous clients. Decoding Units Encoding Units IDCT MC ME DCT Quant VLC Figure 1- Video downscaling in the spatial domain We note that the spatial scalability that is already supported in MPEG-2 has major drawbacks such as an increase in the overall bit-rate and a limit of two on the number of layers that can be supported [1]. Hence, it can not be used in general applications. A more efficient approach for spatial scaling is to perform as much processing as possible in the compressed domain. Figure 2 represents the block diagram of a typical compressed domain system. IDCT MC DCT Quant VLC Figure 2- Video downscaling in the compressed domain Motion estimation (ME) is the most computationally intensive block in a spatial-domain. ME irrespective of the scene complexity, comprises about 66-68% of the processing power required to encode a P- Picture using fast discrete cosine transform (DCT). This

value for B-Pictures, on average, for ME plus macroblock decisions is about 70% of all processing power [2]. Bhaskaran et al. [3] have proposed a compressed domain where they process the extracted motion vectors to generate the required scaled down values. The spatially down-sampled inter-coded frames are then re-encoded with the new motion vectors. One can further improve the compressed domain downsampling performance, by performing the motion compensation (MC) in the compressed domain as well. Several computationally efficient s have been proposed in the literature [4, 5, 6, 7]. The down-sampling operation can also be represented in matrix notation as suggested in [4, 9, 10]. DCT being a linear unitary transform is distributive over matrix multiplication. This fact has been used for down sampling of video frames in the DCT domain. However, this involves matrix multiplication with the DCT of the downsampling matrix. This multiplication can be costly enough to trade off any gains obtained by operating directly in the compressed domain. Ahuja et al. [11] have proposed a for down-sampling in the compressed domain which is computationally much faster, produces visually sharper images, and gives significant improvements in peak signal to noise ratio () (typically 4- improvement compared to bilinear interpolation). Inverse MC-DCT MC-DCT Quant VLC 2.2 Motion Vector sub-sampling Figure 3- Proposed for downscaling in the compressed domain In this paper we propose an improved compressed domain for spatial scaling of MPEG-2 bitstreams. A simplified block diagram of this system is illustrated in Figure 3. We note that in the suggested, MC, DCT, IDCT, and ME are performed in the compressed domain. The outline of this paper is as follows. In Section 2, we review the problem of compressed domain downscaling in more details. In Section 3, the proposed compressed domain downscaling is described. Simulation results are presented in Section 4 Followed by a conclusion and the references. 2 COMPRESSED DOMAIN DOWNSCALING Down-scaling an MPEG video in the compressed domain can be grouped into four main problems. 2.1 DCT coefficients sub- sampling To downscale the intra-coded (I) frames one has to find the DCT domain representation of the downscaled video, using the DCT coefficients of the original bitstream. Several computationally efficient solutions for this problem have been proposed in the literature. In [8] each of the four input blocks is linearly manipulated to create the corresponding transformed scaled block. The manipulated blocks are then combined, producing the composite block. To downscale inter-coded (P and B) frames, we have to estimate the motion vectors (MV) of the down-sampled video from the motion vectors of the original bit-stream. Several compressed domain motion vector downscaling s have been introduced in the literature. The straightforward of estimating the downscaled MV is to average the four corresponding original. This will yield poor results when the four original are not well aligned, which is typically the case. Ghanbari et al. [2] have proposed an improved motion vector sub-sampling approach. In their suggested Median, the distance between each vector and the rest of the original adjacent motion vectors is calculated as the sum of their Euclidean distances as follows: 4 d = MV MV (1) i k= 1 k i where i represents the index of the four macroblocks. i The median vector is defined as the vector that has the least distance from its adjacent vectors: med( V ) = vk V such that min d i = d k This extracts the motion vector situated in the middle of the rest of the. The magnitude of the selected MV is then scaled to reflect the reduction in the spatial resolution. Behaskaran et al. [3] have introduced an Adaptive Motion Vector Re-sampling (AMVR) where the spatial activity is taken into account. In their suggested scheme the new motion vector of the downscaled macroblock is k

the weighted average of the four motion vectors, where the weights correspond to the block activity. The scaled-down motion vector can also be selected randomly from the four original motion vectors [12]. Normally adjacent motion vectors have similar values and hence, randomly selecting one of them doesn't have a significant effect on the final picture quality. In fact, simulation results indicate that this outperforms the previously mentioned s in some cases. We have implemented and compared the simple Average Method, Median Method, Adaptive Motion Vector Resampling, and Random Selection. Table 1 represents simulation results where each input bitstream is down scaled by a factor of two. As shown in Table 1, the Median has generated the best results. Note that as the suggested in [13] was slower than the above-mentioned algorithms, its results are not included here. Sequence 1.5 Mb/s 352x240 Marry Flower 6 Mb/s Football 6 Mb/s Mobile 18 Mb/s Scaling by 2 () () () () () () Average Median AMVR Random 16 sec. 16 sec. 16 sec. 15 sec. 37.42 37.69 37.26 37.24 28 sec. 28 sec. 28 sec. 26 sec. 39.45 39.90 39.43 39.04 30 sec. 30 sec. 30 sec. 29 sec. 33.39 34.52 34.51 33.59 36 sec. 35 sec. 35 sec. 35 sec. 30.07 30.73 29.70 30.74 18 sec. 18 sec. 18 sec. 18 sec. 31.71 32.05 31.40 31.45 42 sec. 40 sec. 40 sec. 41 sec. 27.63 28.74 27.68 27.76 Table 1- Simulation results comparing various motion vector resampling s 2.3 Macroblock Type Selection The second issue in downscaling the inter-coded (P and B) frames is to select the macroblock type for the subsampled macroblocks. Typically a macroblock in an inter-coded P frame can be intra-coded, forward predicted, or skipped. A B-frame macroblock on the other hand can be intra-coded, skipped, forward predicted (from the previous P or I frame), backward predicted (from the next P or I frame), or bi-directionally predicted (from both the previous and next P or I frames). When all original macroblock types are not identical, one has to estimate the macroblock type from the corresponding originals. Several algorithms have been introduced in the literature. Bhaskaran et al. [3] have proposed a where the new macroblock type is decided in the same manner as in the rate control module of a typical MPEG encoder except that in order to reduce the required calculation, the block activity is calculated using DCT coefficients. If the rate control module indicates that a previously intra-coded macroblock should be predicted in the down-sampled version, the motion vectors are estimated using the AMVR described in the previous section. They view intra-coded or skipped macroblocks as predicted macroblocks with zero valued motion vectors. Panchanathan et al. [14] have proposed a more efficient macroblock coding type selection scheme. In their a downscaled macroblock is intra-coded if and only if more than 50% of the original macroblocks are intracoded. An inter-coded macroblock is forward/backward predicted if more than 75% of the original macroblocks are forward/backward predicted. Otherwise the majority of the original block types will specify the downscaled prediction type. They have shown that this simplified type selection can increase the down-sampling quality by up to 2.2 [14]. 2.4 DCT Motion Compensation Once the initial IDCT is performed in the compressed domain, one has to perform the Motion Compensation in the compressed domain as well. Several compressed domain motion compensation (MC-DCT) algorithms have been introduced in the literature [5, 6, 7, 15]. The basic principle of MC-DCT is to perform the horizontal and vertical shifts of four reference blocks by matrix multiplication and adding the four shifted blocks to generate a new one[5]. We note that matrix multiplications and additions are easily mapped into the DCT domain. The brute-force computation of MC-DCT in the case where the MC block is not aligned in any direction with the block structure of the reference frame requires real floating point arithmetic; eight multiplication and four additions of 8x8 matrices. In [16] a fast MC-DCT

scheme has been introduced to reduce the number and the complexity of the operations. In this, complex floating point operations are simplified into a set of binary calculations. Another fast MC-DCT algorithm is introduced in [17]. In general, a target block is predicted from (up to) four blocks. In many cases reference blocks can be shared across multiple target blocks. Therefore, careful rearrangement of computation steps across correlated target blocks can speed up the overall process. They have shown that the sharing of macroblok information can result in 44% improvement over the brute-force. 3 PROPOSED METHOD FOR COMPRESSED DOMAIN DOWNSCALING In this paper we propose a more efficient compressed domain downscaling. The block diagram of the proposed is illustrated in Figure 4. Note that here on, for simplicity, we concentrate on the case of scaling down by a factor of two, but our can be extended to the general case of downscaling by any integer value. In the majority of motion vector re-sampling algorithms a scaled down motion vector (MV) is estimated from the 4 corresponding original. In this scheme the resulted may not generate the best results, as there is no direct relation between them and the corresponding down scaled error residuals. I-TYPE Pictures P-TYPE Pictures B-TYPE Pictures B. : Buffer * (large picture) * (large picture) DCT Err. and DCT Err. and Inverse MC-DCT Inverse MC-DCT Quant VLC (small picture) Resampled Quant Figure 4- Block diagram of the proposed efficient compressed domain downscaling (small picture) Resampled VLC - MC-DCT VLC Quant - MC-DCT In order to achieve a higher picture quality, for the intercoded frames we have reconstructed the residuals using the information currently available in the compressed domain. Note that the motion vector sub-sampling was performed using the algorithm in [2]. For each intercoded P and B frame motion compensation has been implemented using the algorithm in [16]. The motion compensated frames that are reconstructed in the DCT domain are then scaled down using the algorithm in [11]. The difference between this and the reconstructed scaled down frame using the new motion vectors represents the new residual values as determined by the following equation: s _ final _ err = S ( err MC _ DCT( ref, MV )) MC _ DCT( s _ ref, s _ MV ) (2) Where S(b) is the fast compressed domain scaling scheme, err represents the original residual s DCT coefficients, MV represents the original set of, s-mv is the re-sampled MV, ref is the original reference frame and s-ref is the scaled reference frame. Finally, for macroblock type selection, we have proposed the following criteria: If all four original MBs are skipped, the scaled MB is also skipped. If one of the four original MBs is intracoded, then the scaled MB is also intra-coded. Otherwise, the scaled down macroblock will be inter-coded as explained before. Simulation results indicate that the above criteria will result in higher quality of the downscaled sequence. 4 SIMULATION RESULTS We have tested our improved compressed domain spatial scaling with several MPEG-2 bitstreams. Each sequence is downscaled by a factor of 2. It is worth mentioning that the output quality results have been normalized based on their output bit-rates. We found that not modifying the error residuals based on the generated sub-sampled will reduce the resulted by up to 11.6 on average compared to our enhanced. Furthermore except for the intra-coded frames, the drops drastically if error residual modification is not applied. These results are very obvious and we do not show the tables.

We have also tested our macroblock type selection scheme on several test sequences. For each sequence, 16 type selection conditions were considered as below: If n (1, 2, 3, or 4) of the four original MBs are skipped, the scaled MB is also skipped. If m (1, 2, 3, or 4) of the four original MBs are intra-coded, then the scaled MB is also intra-coded. Otherwise, the scaled down macroblock will be inter-coded as explained before. The results for two test sequences are displayed in Table 2 and 3. As shown our proposed criteria explained in part 3, generate the best results in terms of picture quality () compare to other type selection conditions. Table 4 compares the suggested with the compressed domain downscaling proposed by Ghanbari et al. [2]. As the simulation results indicate, our proposed is 17% faster than Ghanbari s, and 70% faster than the spatial domain where we decode/down-sample/re-encode. In terms of picture quality, our generates sub-sampled images that are slightly better than Ghanbari s, by 0.1 in average. The suggested is slightly worse than the conventional spatial domain, by 0.1 on average. Based on these results, we can conclude that the proposed is significantly more efficient than the spatial domain approach and results in comparable subjective quality. Type of original MBs 1 Intra-coded 2 Intra-coded 3 Intra-coded 4 Intra-coded 1 skipped 29.72 29.50 29.29 29.29 2 skipped 30.14 29.92 29.69 29.69 3 skipped 30.30 30.07 29.84 29.84 4 skipped 30.33 30.08 29.85 29.85 Table 2: Average output quality in various Macro-Block Type Selection schemes, the Flower sequence (6 Mb/s,, GOP: 15, 3). Type of original MBs 1 Intra-coded 2 Intra-coded 3 Intra-coded 4 Intra-coded 1 skipped 33.69 32.91 31.89 31.18 2 skipped 33.83 33.07 32.30 31.29 3 skipped 33.86 33.09 32.31 31.30 4 skipped 33.89 33.09 32.31 31.30 Table 3- Average output quality in various Macro-Block Type Selection schemes, the Football sequence (6 Mb/s,, GOP: 12, 3) Sequence name 1.5 Mb/s 352x240 Marry Flower 6 Mb/s Football 6 Mb/s Mobile 18 Mb/s Scaling by 2 Suggested Method Method in [2] Re- Encode 16 sec. 20 sec. 86 sec. 37.69 37.71 37.9 28 sec. 34 sec. 86 sec. 39.90 39.77 41.12 30 sec. 37 sec. 102 sec. 34.52 34.43 34.42 35 sec. 41 sec. 102 sec. 30.73 30.8 30.06 18 sec. 22 sec. 72 sec. 32.05 32.01 32.00 40 sec. 48 sec. 105 sec. 28.74 28.85 28.9 Table 4- Simulation results comparing the proposed scaling, the in [2] and the re-encode Figure 5 displays the results of the same test for each frame in two sequences. As illustrated, our generates comparable quality throughout the entire sequence.

() () 45 40 35 30 25 20 (1.5 Mb/s) 352x240 1 28 55 82 109 136 163 190 217 244 271 298 325 352 379 406 433 Frames Proposed Re-Encode 35 30 25 20 15 Flow er(6 Mb/s) 740x480 1 13 25 37 49 61 73 85 97 109 121 133 145 157 169 181 193 Frames Proposed Re-Encode Figure 5- Performance comparison of the suggested with the conventional spatial domain scaling 5 CONCLUSION In this paper we have proposed a more efficient compressed domain downscaling algorithm. The proposed has been implemented and tested on several standard MPEG2 bitstreams. As discussed in [18], the simulation results indicate that the proposed algorithm is faster than the current processes while maintaining a comparable subjective quality. Our compressed domain is 70% faster than the spatial domain scaling. ACKNOLEDGMENT The authors would like to thank Mabna Pardaz Co. for providing their full support and access to their facilities and Mahmoud Ghandi for his valuable contributions. REFERENCES [1] International standard, Information Technology - Generic Coding of Moving Pictures and Associated Audio Information: Video, ISO/IEC 13818-2, 1996. [2] Tamer Shanableh, Mohammad Ghanbari, Heterogeneous Video Transcoding to Lower Spatio- Temporal Resolutions and Different Encoding Formats, IEEE Transactions on Muldia, Vol.2, No.2, June 2000. [3] Bo Shen, Ishvar K. Sethi and Vasudev Bhaskaran, Adaptive Motion Vector Re-sampling for Compressed Video Down-Scaling, IEEE Transactions on Circuits and Systems for Video Technology, Vol.9, September 1999. [4] Neri Merhav and Vasudev Bhaskaran, Fast Algorithms for DCT-Domain Image Down-Sampling and for Inverse Motion Compensation, IEEE Transaction on Circuits and Systems For Video Technology, Vol. 7, No. 3, June 1997. [5] S. F. Chand and D. G. Messerschmitt, "Manipulation and compositing of MC-DCT Compressed Video", IEEE Journal of Selected Areas in Communicatin, Special Issue on Intelligent Signal ing, Vol 13, Jan. 1995. [6] P A. A. Assuncao and M. Ghanbari, Transcoding of MPEG2 Video in the Frequency Domain, ICAASP, 1997. [7] Soam Acharya and Brian Smith, Compressed Domain Transcoding of MPEG, Proceedings of the IEEE International Conference on Muldia, 1998. [8] Balas K. Natarajan and Bhaskaran Vasudev, A Fast Approximate Algorithm for Scaling Down Digital Images in the DCT Domain, Proceedings of the 1995 International Conference on Image ing, Vol.2. [9] S. F. Chang and D. G. Messerschmitt, "Manipulation and Compositing of MC-DCT Compressed Video", IEEE J. Select. Areas Commun. Vol. 13, Jan 1995. [10] Q. Hu and S. Panchanathan, Image/video spatial scalability in compressed domain, IEEE Trans. Industrial Electronics, Vol. 45, Feb. 1998. [11] Rakesh Dugad, Narendra Ahuja, A Fast Scheme for Image Size Change in the Compressed Domain, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 11, No.4, April 2001. [12] N. Bjork and C. Christopoulos, Transcoder Architecture For Video Coding, IEEE Transaction on Consumer Electronics, Vol. 44, February 1998 [13] M.R. Hashemi, L. Winger, S. Panchanathan, Compressed Domain Motion Vector Re-sampling for Downscaling of MPEG Video, 1999 IEEE International Conference on Image ing, Oct. 1999, Kobe Japan.

[14] M. R. Hashemi, L. Winger, S. Panchanathan, "Macroblock Type Selection for Compressed Domain Down-Sampling of MPEG Video", IEEE Canadian Conference on Electrical and Computer Engineering, May, 1999. [15] Neri Merhav and Vasudef Bhaskaran, "A Fast Algorithm for DCT-Domain Inverse Motion Compensation", 1997 International Conference on Image ing, October 1997. [16] P. A. Assuncao, M. Ghanbari Fast Computing of MC-DCT for Video Transcoding, IEE Electronics letters, Jan. 1997. [17] Junehwa song and Boon lock Yeo, "A fast DCT Domain Inverse Motion Compensation Algorithm based on shared information in a macroblock", International Conference on Image ing, October 1997. [18] M. Mahdi Ghandi, M. Emad Modirzadeh, M. R. Hashemi, O. Fatemi "Compressed Domain Spatial Scaling of MPEG Video Sequences", IEEE International Conference on Consumer Electronics, 2002.