Department of Electrical Engineering Multimedia Processing Spring 2011 IMPLEMENTATION OF H.264/AVC, AVS China Part 7 and Dirac VIDEO CODING STANDARDS INSTRUCTOR Dr. K R. Rao Term Project Sharan K Chandrashekar (1000676767) sharan.chandrashekar@mavs.uta.edu
ACKNOWLEDGEMENT I would like to acknowledge the continuous guidance and support of Dr. K.R. Rao (Electrical Engineering Department at the University of Texas at Arlington), throughout the various stages of this project. His sound technical knowledge of the subject and professional acumen has been an outstanding source of inspiration in pursuing this course. I sincerely appreciate the help and advice given by Dr.Rao for the successful completion of this project. I would also like to thank Mr. Touseef Khan, senior graduate student at the University, for helping me understand the key concepts underlying the project. Finally I would like to thank my fellow students for their valuable inputs which helped greatly in improvising the project at various stages.
Introduction Effective digital representation of image and video has been a subject of overwhelming research over the past 20 years [1]. The need for an international image and video coding standards arose with the increased commercial interest in video communications. New applications in the field of communication, multimedia and broadcasting became possible based on digital video coding technology. Introduction to the video coding and decoding process Figure 1: General video coding and decoding process [7] Firstly, the sender might choose to pre-process the video using format conversion or enhancement techniques. The encoder encodes the video representing it as a bit stream. The video is decoded following which, an optional post-processing step such as format conversion, error concealment or video enhancement may be performed before the receiver receives it. - Video coding standards H.264/AVC, Dirac and AVS China are the latest standards adopted by ITU-T/ISO-IEC, BBC and China standards organization respectively [2].
H.264/AVC [2]: H.264/AVC, an open licensed standard, was developed as a result of the collaboration between the ISO/IEC Moving Picture Experts Group and the ITU-T Video Coding Experts Group. It is the most efficient video compression technique available today. It provides an increase in the coding efficiency up to 50% over a wide range of bit rates and resolutions compared to its predecessors. Some of its major applications include video broadcasting, Video on demand, MMS over various platforms like DSL, Ethernet, LAN, wireless and mobile networks etc. H.264 Encoder:
H.264 Decoder: H.264 Standard Profiles: Figure 4: H.264 decoder block diagram [7] -
Three profiles were defined for the first version of H.264/MPEG4-AVC namely Baseline, Extended and Main profiles. The Baseline profile was targeted at applications in which a minimum of computational complexity and a maximum of error robustness [8]. The Main profile was aimed at applications that require a maximum of coding efficiency, with somewhat less emphasis on error robustness [8]. The Extended profile was designed to provide a compromise between the Baseline and Main profile capabilities with an additional focus on the specific needs of video streaming applications, and further added robustness to errors and packet losses [8]. Dirac [10]: Dirac is a video compression system developed by the British Broadcasting Corporation (BBC) utilizing motion compensation and wavelet transforms. It is a motion-compensated video codec like the Moving Picture Experts Group (MPEG) standard [10], i.e. the image motion is tracked and the motion information is used to track a later frame. Dirac video codec applications span from mobile and internet to ultra HDTV and film and video production. Wavelet Transforms: The 2D discreet wavelet transforms provide Dirac with the flexibility to operate at a range of resolutions. Applied to two-dimensional images, wavelet filters are normally applied in both vertical and horizontal directions to each image component to produce four so called sub bands termed Low-Low (LL), Low-High (LH), High-Low (HL) and High-High (HH) [24]. In the case of two dimensions, only the LL band is iteratively decomposed to obtain the required data. The number of samples in each resulting sub band is as implied by figure 7. The critical sampling ensures that after each decomposition the resulting bands all have one quarter of the samples of the input signal.
Dirac encoder: - Figure 9: Dirac encoder block diagram [10]
Dirac Decoder: AVS China [6]: Figure 10: Dirac decoder block diagram [2] The AVS video codec was developed by the Audio Video Coding Standard Working Group of China. The ten parts of the AVS China family is as shown in table 1. Table 1: Ten Parts of the AVS China Standard Family [2]
AVS China comprises of four different profiles namely Jizhun, Jiben, Shenzan and Jiaqiang of which the Jiben profile (basic profile) is defined in AVS Part 7 for mobile applications. Key applications Jizhun profile Jiben profile Shenzhan profile Jiaqiang profile Television broadcasting, HDTV, etc. Mobility applications, etc. Video surveillance, etc. Multimedia entertainment, etc. Table 2: Applications of various profiles of AVS China [5] AVS China Encoder:
AVS China Decoder: CIF and QCIF Formats: Figure 12: AVS China decoder block diagram [17] Common Intermediate Format (CIF) and Quadrature Common Intermediate Format (QCIF) determine the resolution of the frame. The resolution of CIF is 352x288 and the resolution of QCIF is 1/4 of CIF, which is 176x144 [26]. Consider the YCbCr family of color spaces where Y represents the luminance, Cb represents the blue-difference chroma component and Cr represents the red-difference chroma component [25]. For QCIF and CIF, the luminance Y is equal to the resolution. If sampling resolution 4:2:0 is used, the for CIF, the Cb and Cr are 176 x 144 lines and for QCIF, the Cb and Cr are 88 x 72 lines.
Some Important Parameters: Structural Similarity Metric (SSIM) [22]: This index is a method for measuring the similarity between two frames. It is a full reference metric, or in other words, the measuring of image quality is done using an initial uncompressed or distortion-free frame as reference. Mean Squared Error (MSE) [22]: The MSE is computed by averaging the squared intensity differences of the distorted and reference image/frame pixels. Two distorted images with the same MSE may have very different types of errors, some of which are much more visible than others. Peak Signal-to-Noise Ratio (PSNR) [22]: The PSNR is most commonly used as a measure of quality of reconstruction of compression codecs. The signal in this case is the original data, and the noise is the error introduced by compression.
Implementation of the JM 17.2 [14] Software for the H.264/AVC video codec YUV File 1: coastguard_cif.yuv Specifications: CIF sequence: coastguard_cif.yuv Total number of frames: 300 Height: 352; Width: 288 Total number of frames used: 100 Original file size: 14850 kb Frame rate: 30 frames/second
QP = 0 QP = 30 QP = 50 YUV File 1: coastguard_cif.yuv JM 17.2 Software
MSE PSNR (db) EE 5359 SPRING 2011 YUV File 1: coastguard_cif.yuv 70 PSNR (db) vs. 65 60 55 50 45 40 35 30 25 20 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 x 10 4 350 MSE vs. 300 250 200 150 100 50 0 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 x 10 4 JM 17.2 Software
YUV File 2: coastguard_qcif.yuv Specifications: QCIF sequence: coastguard_qcif.yuv Total number of frames: 300 Height: 176; Width: 144 Total number of frames used: 100 Original file size: 3713 kb Frame rate: 30 frames/second
QP = 0 QP = 30 QP = 50 YUV File 2: coastguard_qcif.yuv JM 17.2 Software
MSE PSNR (db) EE 5359 SPRING 2011 YUV File 2: coastguard_qcif.yuv 80 PSNR (db) vs. 70 60 50 40 30 20 0 500 1000 1500 2000 2500 3000 3500 4000 4500 450 MSE vs. 400 350 300 250 200 150 100 50 0 0 500 1000 1500 2000 2500 3000 3500 4000 4500 JM 17.2 Software
YUV File 3: container_cif.yuv Specifications: CIF sequence: container_cif.yuv Total number of frames: 300 Height: 352; Width: 288 Total number of frames used: 100 Original file size: 14850 kb Frame rate: 30 frames/second
QP = 0 QP = 30 QP = 50 YUV File 3: container_cif.yuv JM 17.2 Software
MSE PSNR (db) EE 5359 SPRING 2011 YUV File 3: container_cif.yuv 70 PSNR (db) vs. 65 60 55 50 45 40 35 30 25 20 0 5000 10000 15000 250 MSE vs. 200 150 100 50 0 0 5000 10000 15000 JM 17.2 Software
YUV File 4: container_qcif.yuv Specifications: QCIF sequence: container_qcif.yuv Total number of frames: 300 Height: 176; Width: 144 Total number of frames used: 100 Original file size: 3713 kb Frame rate: 30 frames/second
QP = 0 QP = 30 QP = 50 YUV File 4: container_qcif.yuv JM 17.2 Software
MSE PSNR (db) EE 5359 SPRING 2011 YUV File 4: container_qcif.yuv 70 PSNR (db) vs. 65 60 55 50 45 40 35 30 25 20 0 500 1000 1500 2000 2500 3000 3500 400 MSE vs. 350 300 250 200 150 100 50 0 0 500 1000 1500 2000 2500 3000 3500 JM 17.2 Software
Implementation of the Dirac 1.0.2 Software [15] for the Dirac video codec YUV File 1: coastguard_cif.yuv Specifications: CIF sequence: coastguard_cif.yuv Total number of frames: 300 Height: 352; Width: 288 Total number of frames used: 100 Original file size: 14850 kb Frame rate: 12.5 frames/second
QF = 0 QF = 8 QF = 12 YUV File 1: coastguard_cif.yuv Dirac 1.0.2 Software
MSE PSNR (db) EE 5359 SPRING 2011 YUV File 1: coastguard_cif.yuv 50 PSNR (db) vs. 45 40 35 30 25 20 0 500 1000 1500 2000 2500 3000 3500 4000 350 MSE vs. 300 250 200 150 100 50 0 0 500 1000 1500 2000 2500 3000 3500 4000 Dirac 1.0.2 Software
YUV File 2: coastguard_qcif.yuv Specifications: QCIF sequence: coastguard_qcif.yuv Total number of frames: 300 Height: 176; Width: 144 Total number of frames used: 100 Original file size: 3713 kb Frame rate: 12.5 frames/second
QF = 0 QF = 8 QF = 12 YUV File 2: coastguard_qcif.yuv Dirac 1.0.2 Software
MSE PSNR (db) EE 5359 SPRING 2011 YUV File 2: coastguard_qcif.yuv 50 PSNR (db) vs. 45 40 35 30 25 20 0 100 200 300 400 500 600 700 800 900 300 MSE vs. 250 200 150 100 50 0 0 100 200 300 400 500 600 700 800 900 Dirac 1.0.2 Software
YUV File 3: container_cif.yuv Specifications: CIF sequence: container_cif.yuv Total number of frames: 300 Height: 352; Width: 288 Total number of frames used: 100 Original file size: 14850 kb Frame rate: 12.5 frames/second
QF = 0 QF = 8 QF = 12 YUV File 3: container_cif.yuv Dirac 1.0.2 Software
MSE PSNR (db) EE 5359 SPRING 2011 YUV File 3: container_cif.yuv 50 PSNR (db) vs. 45 40 35 30 25 20 0 500 1000 1500 2000 2500 250 MSE vs. 200 150 100 50 0 0 500 1000 1500 2000 2500 Dirac 1.0.2 Software
YUV File 4: container_qcif.yuv Specifications: QCIF sequence: container_qcif.yuv Total number of frames: 300 Height: 176; Width: 144 Total number of frames used: 100 Original file size: 3713 kb Frame rate: 12.5 frames/second
QF = 0 QF = 8 QF = 12 YUV File 4: container_qcif.yuv Dirac 1.0.2 Software
MSE PSNR (db) EE 5359 SPRING 2011 YUV File 4: container_qcif.yuv 50 PSNR (db) vs. 45 40 35 30 25 0 50 100 150 200 250 300 350 400 450 500 250 MSE vs. 200 150 100 50 0 0 50 100 150 200 250 300 350 400 450 500 Dirac 1.0.2 Software
Implementation of the AVS-M Software [27] for the AVS China video codec YUV File 1: coastguard_cif.yuv Specifications: CIF sequence: coastguard_cif.yuv Total number of frames: 300 Height: 352; Width: 288 Total number of frames used: 100 Original file size: 14850 kb Frame rate: 30 frames/second
QP = 0 QP = 30 QP = 50 YUV File 1: coastguard_cif.yuv AVS-M Software
MSE PSNR (db) EE 5359 SPRING 2011 YUV File 1: coastguard_cif.yuv 60 PSNR (db) vs. 55 50 45 40 35 30 25 0 0.5 1 1.5 2 2.5 x 10 4 250 MSE vs. 200 150 100 50 0 0 0.5 1 1.5 2 2.5 x 10 4 Dirac 1.0.2 Software
YUV File 2: coastguard_qcif.yuv Specifications: QCIF sequence: coastguard_qcif.yuv Total number of frames: 300 Height: 176; Width: 144 Total number of frames used: 100 Original file size: 3713 kb Frame rate: 30 frames/second
QP = 0 QP = 30 QP = 50 YUV File 2: coastguard_qcif.yuv Dirac 1.0.2 Software
MSE PSNR (db) EE 5359 SPRING 2011 YUV File 2: coastguard_qcif.yuv 60 PSNR (db) vs. 55 50 45 40 35 30 25 0 1000 2000 3000 4000 5000 6000 7000 180 MSE vs. 160 140 120 100 80 60 40 20 0 0 1000 2000 3000 4000 5000 6000 7000 Dirac 1.0.2 Software
YUV File 3: container_cif.yuv Specifications: CIF sequence: container_cif.yuv Total number of frames: 300 Height: 352; Width: 288 Total number of frames used: 100 Original file size: 14850 kb Frame rate: 30 frames/second
QP = 0 QP = 30 QP = 50 YUV File 3: container_cif.yuv Dirac 1.0.2 Software
MSE PSNR (db) EE 5359 SPRING 2011 YUV File 3: container_cif.yuv 60 PSNR (db) vs. 55 50 45 40 35 30 25 20 0 0.5 1 1.5 2 2.5 3 x 10 4 250 MSE vs. 200 150 100 50 0 0 0.5 1 1.5 2 2.5 3 x 10 4 Dirac 1.0.2 Software
YUV File 4: container_qcif.yuv Specifications: QCIF sequence: container_qcif.yuv Total number of frames: 300 Height: 176; Width: 144 Total number of frames used: 100 Original file size: 3713 kb Frame rate: 30 frames/second
QP = 0 QP = 30 QP = 50 YUV File 4: container_qcif.yuv Dirac 1.0.2 Software
MSE PSNR (db) EE 5359 SPRING 2011 YUV File 4: container_qcif.yuv 60 PSNR (db) vs. 55 50 45 40 35 30 25 20 0 1000 2000 3000 4000 5000 6000 7000 8000 200 MSE vs. 180 160 140 120 100 80 60 40 20 0 0 1000 2000 3000 4000 5000 6000 7000 8000 Dirac 1.0.2 Software
MSE PSNR(dB) EE 5359 SPRING 2011 Comparison of the three codecs YUV File 1: coastguard_cif.yuv 70 PSNR (db) vs. 65 60 55 50 45 40 35 30 H.264 25 Dirac AVS China 20 0 0.5 1 1.5 2 2.5 x 10 4 350 300 MSE vs. H.264 Dirac AVS China 250 200 150 100 50 0 0 0.5 1 1.5 2 2.5 x 10 4
PSNR (db) SSIM EE 5359 SPRING 2011 1 SSIM vs. 0.95 0.9 0.85 0.8 0.75 0.7 0.65 H.264 Dirac AVS China 0 0.5 1 1.5 2 2.5 x 10 4 YUV File 2: container_qcif.yuv 70 PSNR (db) vs. 65 60 55 50 45 40 35 30 H.264 25 Dirac AVS China 20 0 1000 2000 3000 4000 5000 6000 7000 8000
SSIM MSE EE 5359 SPRING 2011 400 350 300 MSE vs. H.264 Dirac AVS China 250 200 150 100 50 0 0 1000 2000 3000 4000 5000 6000 7000 8000 1 SSIM vs. 0.95 0.9 0.85 0.8 0.75 0.7 H.264 Dirac AVS China 0.65 0 1000 2000 3000 4000 5000 6000 7000 8000
MSU Video Quality Measurement Tool [21]
CONCLUSIONS The H.264/AVC has an edge over the AVS China and Dirac video codecs in terms of performance providing higher compression ratios and quality. The plots suggest that the PSNR and SSIM increase with increasing bit rates and the error decreases. Faster processors can be used to tackle competitive video file sizes. FUTURE WORK So far, all the work done has been for the baseline profiles of the three codes. There is plenty of scope to explore the mainline and high profiles. Other video formats such as SD and HD can also be used for simulation, provided ample time and faster processors. Also, it could be interesting to compare other video codecs (successor versions of the H.264) and analyze their performance.
Abbreviations and Acronyms: AVC: Advanced Video Coding AVS: Audio Video Standard CIF: Common Intermediate Format HDTV: High-Definition Television IEC: International Electrotechnical Commission ISO: International Organization for Standardization ITU-T: International Telecommunication Union - Telecommunication Standardization sector MSE: Mean Square Error PSNR: Peak Signal to Noise ratio QCIF: Quarter Common Intermediate Format SMPTE: Society of Motion Picture and Television Engineers SSIM: Structural Similarity Metric References [1] T. Sikora, Digital Video Coding Standards and Their Role in Video Communications, Signal Processing for Multimedia, J.S. Byrnes (Ed.), IOS press, pp. 225-251, 1999. [2] K. R. Rao, and D. N. Kim, Current Video Coding Standards: H.264/AVC, Dirac, AVS China and VC-1, IEEE 42nd Southeastern symposium on system theory (SSST), March 7-9 2010, pp. 1-8, March 2010. [3] K. Onthriar, K. K. Loo and Z. Xue, Performance comparison of emerging Dirac video codec with H.264/AVC, IEEE International Conf. on Digital Telecommunications, ICDT 2006, vol. 6, Page: 22, Issue: 29-31, Aug 2006. [4] X-F Wang, and D-B Zhao, Performance comparison of AVS and H.264/AVC video coding standards, J. Comput. Sci. & Technol., vol. 21, No. 3, pp.310-314, May 2006. [5] L. Yu, S. Chen, and J. Wang, Overview of AVS video coding standards, Signal Processing: Image Communication, vol. 24, pp. 263-276, April 2009. [6] L. Fan et al, Overview of AVS Video Standard, IEEE International conference on multimedia and expo (ICME), vol.1, pp. 423-426, June 2004. [7] J. Ostermann et al, Video coding with H.264/AVC: Tools, Performance, and Complexity, IEEE Circuits and Systems magazine, vol. 4, Issue 1, pp. 7 28, Aug 2004. [8] D. Marpe, T. Wiegand, and G.J. Sullivan, The H.264/MPEG4 Advanced Video Coding Standard and its Applications, IEEE Communications magazine, vol. 44, Issue: 8, pp. 134 143, August 2006. [9] I. E. Richardson, The H.264 advanced video compression standard, Wiley, 2 nd edition, 2010. [10] T. Borer and T. Davies, Dirac video compression using open technology, BBC EBU Technical Review, July 2005.
[11] T. Borer, Dirac coding: Tutorial & Implementation, EBU Networked Media Exchange seminar, EBU, Geneva, 22-23 June 2009. [12] Dirac Specification, Version 2.2.3, Available: http://diracvideo.org/download/specification/diracspec-latest.pdf [13] BBC Research on Dirac: http://www.bbc.co.uk/rd/projects/dirac/technology.shtml [14] H.264/AVC Software source reference: http://iphome.hhi.de/suehring/tml/download/ [15] Dirac video download source reference: http://diracvideo.org/download/ [16] YUV video sequences source: http://trace.eas.asu.edu/yuv/ [17] L. Fan, Mobile Multimedia Broadcasting Standards, Springer, 2009. [18] H. Malvar et al, Low-complexity transform and quantization in H.264/AVC, IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, pp. 598 603, July 2003. [19] J. Ribas-Corbera et al, A generalized hypothetical reference decoder for H.264/AVC, IEEE Transactions on Circuits and Systems, vol. 13, no. 7, pp. 674 687, July 2003. [20]Compression for High-Quality, High Bandwidth Video, Intel Software Network: http://software.intel.com/en-us/articles/compression-for-high-quality-high-bandwidth-video/ [21] MSU video quality measurement tool: http://compression.ru/video/quality_measure/video_measurement_tool_en.html [22] Z. Wang, et al, Image quality assessment: From error visibility to structural similarity, IEEE Transactions on Image Processing, vol.13, no.4, pp. 600-612, April 2004. [23] A. Ravi, and K.R. Rao, Performance analysis and comparison of the Dirac video codec with H.264/MPEG-4 part 10 AVC, International Journal of Wavelets, Multiresolution and Information Processing (accepted), January 2010. Available: http://www-ee.uta.edu/dip/courses/ee5359/index.html [24] Dirac developer support documentation: http://dirac.sourceforge.net/documentation/algorithm/algorithm/wlt_transform.xht [25] YCbCr Article: http://en.wikipedia.org/wiki/ycbcr [26] S.Kwon, A. Tamhankar and K.R. Rao, Overview of H.264 / MPEG-4 Part 10, Journal of Visual Communication and Image Representation, vol. 17, pp.186-216, April 2006. [27] AVS China Software Download Tool (FTP Client): http://filezilla-project.org/