White paper: Video Coding A Timeline Abharana Bhat and Iain Richardson June 2014 Iain Richardson / Vcodex.com 2007-2014
About Vcodex Vcodex are world experts in video compression. We provide essential analysis and advice on technology, strategy and intellectual property. Our input will help you get the most out of your video compression technology. Video compression is the technology behind moving digital images. It is essential to video on phones, cameras, laptops and TV. In fact, anything you can watch on a screen uses video compression. Vcodex.com Historical timeline of video coding standards and formats 1. Who should read this? The majority of video codecs in use today conform to one of the international standards for video coding. Therefore, if you are learning about video codecs or into codec development, it is important to know what a video coding standard is and how the existing standards have evolved. Even the latest video codecs such as H.264/AVC and HEVC have a basic structure similar to older codecs such as MPEG-2 part-2. This white paper begins by describing a video coding standard and the need for standardisation. The process of standardisation is briefly explained. The remainder of the paper provides a timeline of development of popular video coding standards and video coding formats. 2. What is a video coding standard? A video coding standard is a document describing a bitstream structure and a decoding method for video compression. The standard does not define the encoder; rather it defines the output structure that an encoder should produce. Video coding standards usually define a toolkit or a set of tools for compression. Not all pieces of the toolkit need to be implemented to create a standard compliant bitstream. However, decoders conforming to the standard must implement some subset of the toolkit. 3. Why standardize? Standardisation enables encoders and decoders from different manufacturers to work together across a range of applications. In order to provide the maximum flexibility and encourage innovation, the standards do not include all practical issues and design techniques of the decoding process. Standardisation allows for decoding of bit streams from previous and emerging standards. There is also a greater freedom for consumers to choose between manufacturers. For developers of standard-compliant video coding systems, the published standard is an essential point of reference as it defines the capabilities that a video codec must conform to, in order to successfully interwork with other systems. Iain Richardson/Vcodex.com 2007-2014 2 of 9
4. The standardisation process The main steps towards the finalisation of a standard are shown in Figure 1. Figure 1. The standardisation process (Copyright: Vcodex Limited) In the first phase, the requirements for a specific application or a field of applications are identified. The next phase involves the development of different algorithms by various laboratories, companies or contributors. The developed algorithms are compared and a set of basic techniques, which forms the core of the standard, is selected and refined in a joint effort during the collaboration phase. At the end of this phase a draft standard is issued, which has to be validated by compliance testing based on computer simulations or hardware tests and field trials. After successful validation and refinements, the final standard is published. Iain Richardson/Vcodex.com 2007-2014 3 of 9
5. The timeline Two standardisation bodies, the international standards organisation (ISO) and the international telecommunications union (ITU), have developed a series of standards that have shaped the development of the media industry. Popular ISO coding standards include MPEG- 1, MPEG-2 and MPEG-4. ITU-T has published the H.26x line of coding standards including H.261, H.262, H.263 and H.263+. Other video coding formats such as Theora, VC-1, VPx and Daala have been developed over the years. The timeline of major standards and video formats over the last 25 years is shown in figure 2. Figure 2. Timeline for various video coding standards and formats (Copyright: Vcodex Limited) Many of the basic concepts of video coding such as transform coding, motion estimation and compensation and entropy coding were developed in the 1970s and 1980s. MPEG-1 was standardised on the early 1990s. MPEG-4 was developed in the late 1990s and soon after H.263 was standardised. H.264/AVC was published in 2003 and Google released the VP8 video coding format in 2010. 2013 saw the publication of the High Efficiency Video Coding (HEVC) standard and Google s VP9 format. The following sections cover these popular video coding standards and formats in detail. Iain Richardson/Vcodex.com 2007-2014 4 of 9
6. The MPEG family of video coding standards The ISO/IEC Motion Picture Experts Group (MPEG) publishes standards for media coding, storage and transport. These standards consist of different Parts. Each part covers a certain aspect of the whole specification. In this article, we discuss the parts related to video. Table 1 lists the key features of popular MPEG video coding standards. Table 1: MPEG family Standard Key Features MPEG-1 part-2 [1] Standardised in 1993 Developed for video and audio storage on CD-ROMs Has support for progressive video Supports YUV 4:2:0 at common intermediate format (352x288) resolution Group of pictures include I-picture, P-picture and B-picture Motion vectors are coded losslessly MPEG-2 part-2 [2] Standardised in 1995 Supports video on DVDs, standard definition TVs and high definition TVs Can handle YUV 4:2:0 and YUV 4:2:2 formats Supports interlaced and progressively scanned pictures Introduction of profiles and levels to define the various capabilities of the standard as sub-sets. For example, MPEG-2 Main profile / high level is suitable for high definition television (HDTV) Scalable extensions which permit the division of a continuous video signal into two or more coded bit streams representing the video at different resolutions, picture quality (i.e. SNR), or picture rates Other features include data partitioning, non-linear quantization, VLC tables and improved mismatch control MPEG-4 part-2 (Visual) [3] Standardised in 1999 Supports video on low-bit rate multimedia applications on mobile platforms and the Internet. Supports object-based or content-based coding where a video scene is coded as a set of foreground and background objects. Supports coding of synthetic video and audio including animation Shares subset with H.263 ( short header mode ) MPEG-4 Part 10 [4] (Advanced video coding) Standardised in 2003 Co-published as H.264 AVC, see Table 2 for details Iain Richardson/Vcodex.com 2007-2014 5 of 9
7. The H26x family of video coding standards The video coding experts group (VCEG), a working group of the ITU-T, published early standards in the H.26x family such as H.261, H.262 and H.263. The H.264/AVC or MPEG-4 Part-10 video coding standard was co-developed by the Joint Video Team (JVT), a collaboration between VCEG and MPEG. The High Efficiency Video Coding (HEVC) standard is the latest member of the H.26x family and is published by the Joint Collaborative Team on Video Coding (JCT-VC). Table 2 gives a high level overview of the key features of the H.26x standards. Table 2: H.26x line of standards Standard Features H.261 [5] Standardised in 1988 [6] Developed for video conferencing over ISDN Support for CIF and QCIF resolutions in YUV 4:2:0 format Uses block-based hybrid coding with integer pixel motion compensation. H.262 [2] Standardised as MPEG-2 part-2 (Video) in 1995 See Table 1 for details H.263 / H.263+ [7] H.263 was standardised in 1996. H.263+ was standardised in 1998 Improved quality compared to H.261 at lower bit rate to enable video conferencing/telephony Sub-pixel motion vectors up to 1/8th pixel accuracy (H.263+) for improved compression Shares subset with MPEG-4 part 2 H.264 Advanced Video Coding [4] Standardised in 2003 Supports video on the Internet, computers, mobile and HDTVs Significantly improved picture quality compared to H.263, at low bit rates but at the cost of increased computational complexity Improved motion compensation with variable block-size, multiple reference frames and weighted prediction In-loop deblocking filter to reduce block discontinuities H.265 / HEVC [8] Standardised in 2013, HEVC s basic structure is similar to H.264/AVC with improvements such as: Support for ultra HD video up to 8K resolutions with frame rates up to 120 fps Greater flexibility in prediction modes and transform block sizes More sophisticated interpolation and deblocking filters Features to support parallel processing More efficient compared to H.264 in terms of bitrate savings for the same picture quality [9,10] Iain Richardson/Vcodex.com 2007-2014 6 of 9
8. VC-1 standard VC-1 is a video codec specification standardised by the Society of Motion Picture and Television Engineers (SMPTE) in 2006. It was implemented by Microsoft as Microsoft Windows Media Video (WMV) 9. Table 3 lists some key features of the VC-1 standard. Table 3: VC-1 coding standard Standard Features VC-1 or wmv- 9 [11] Adaptive block transform where an 8 8 block can be coded using either an 8 8 transform, two 8 4 transforms, two 4 8 transforms, or four 4 4 transforms. 16-bit transforms for easier implementation on hardware devices compared to the 8-bit transforms used by MPEG and H.26x codecs. Fading compensation as a preprocessing step to detect fades and use alternate methods to adjust luminance. In-loop deblocking filter to reduce block discontinuities 9. Other popular video coding formats This section highlights some of the key features of other popular video coding formats such as VP8, VP9, Theora and Daala. In May 2010, Google announced the start of a new open media project called WebM with the goal of providing the web with a high-quality, open, royalty-free video codec that anyone can use, and that can inspire future innovators [12]. At the core of the project are VP8 and the new VP9 compression standards. Theora, published in September 2004, was developed by Xiph.org to support Internet video delivery. Xiph.org s latest video coding format, Daala, is still under development. Table 4 highlights some of the features of these video coding formats. Table 4: Other popular video coding formats Standard Features Theora [13] Based on On2 s VP3.2 format Support for video on the Internet Bitstream format was frozen in 2004 Support for more pixel formats - YUV444, YUV422 and YUV420 No B-pictures No global motion compensation Only half-pixel precision motion compensation In-loop deblocking filter to reduce block discontinuities VP8 [14] Designed to operate in low bandwidth environment such as web video Support for Web video format - YUV420, 8-bit colour depth, progressive scan and resolution up to 4K Flexible reference frames with buffer size limited to three frames Efficient intra and inter prediction with motion vector accuracy of 1/4th pixel for luma Iain Richardson/Vcodex.com 2007-2014 7 of 9
and 1/8th pixel for chroma Adaptive in-loop deblocking filter to reduce block discontinuities Features such as data partitioning enable parallel processing VP9 [15] Has basic structure similar to VP8 with improvements such as: Support for resolution up to 8k and frame rate up to 120fps Use of superblocks (32x32 and 64x64) to exploit high correlation over larger areas in HD video Enhanced interpolation for motion compensation with 8-tap filters to achieve 1/8th pixel accuracy in motion vectors. Reference frame scaling for improved bitrate adjustment, where the reference frame can be scaled up or down during inter prediction. Daala [16] Daala is being designed to differ from common video compression techniques. Some distinguishing features include: Lapped transforms instead of block-based DCT Lifting pre- and post-filtering instead of deblocking filter Frequency domain intra prediction instead of pixel domain prediction Time/Frequency Resolution Switching where image blocks can be split apart or merged together in the frequency domain. Further reading [1] ISO/IEC 11172-2, Information technology coding of moving pictures and associated audio for digital storage media at up to about 1. Mbit/s - part 2: Video, 1993 [MPEG-1 Video] [2] ISO/IEC 13818-2, Information technology: generic coding of moving pictures and associated audio information: Video, 1995 [MPEG-2 Video] [3] ISO/IEC 14996-2, Information technology - coding of audio-visual objects - part-2: Visual, 1998 [MPEG- 4 Visual] [4] ISO/IEC 14996-10 and ITU-T Rec. H.264, Advanced Video Coding, 2003 [MPEG-4 part-10/h.264] [5] ITU-T Recommendation H.261, Video CODEC for audiovisual services at px64 kbits/s, 1988 [H.261] [6] http://www.itu.int/rec/t-rec-h.261-198811-s/en [H.261 document] [7] ITU-T Recommendation H.263, Video coding for low-bit rate communications, Version-2, 1998 [H.263] [8] h.265/hevc: ITU-T Recommendation H.265, Infrastructure of audiovisual services Coding of moving video: High efficiency video coding, April 2013 [H.265] [9] Grois et al, Performance Comparison of H.265/MPEG-HEVC, VP9, and H.264/MPEG-AVC Encoders, PICTURE CODING SYMPOSIUM 2013 (PCS 2013), San José, CA, USA, Dec 8-11, 2013. [10] Ohm et al, Comparison of the Coding Efficiency of Video Coding Standards Including High Efficiency Video Coding (HEVC), IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 22, NO. 12, DECEMBER 2012. pp1669-1684. [11] SMPTE 421M-2006, "VC-1 Compressed Video Bitstream Format and Decoding Process", 2006 [VC-1] [12] WebM project blog: http://blog.webmproject.org/ [13] http://www.theora.org/ [Theora] [14] RFC-6386: VP8 Data Format and Decoding Guide, 2011. [VP8] [15] A VP9 Bitstream Overview: draft-grange-vp9-bitstream-00, 2013. http://tools.ietf.org/id/draft-grange-vp9- bitstream-00.txt [VP9] [16] https://www.xiph.org/daala/ [Daala] To find out more about any of the video coding standards mentioned in this article, start by reading the materials available on the Vcodex website: http://vcodex.com/introduction.asp. Iain Richardson/Vcodex.com 2007-2014 8 of 9
About the authors Vcodex is led by Professor Iain Richardson, an internationally known expert on the MPEG and H.264 video compression standards. Based in Aberdeen, Scotland, he frequently travels to the US and Europe. Professor Richardson is the author of The H.264 Advanced Video Compression Standard, a widely cited work in the research literature. He has written three further books and over 50 journal and conference papers on image and video compression. He regularly advises companies on video codec technology, video coding patents and mergers/acquisitions in the video coding industry. Professor Richardson leads an internationally renowned image and video coding research team, contributes to the MPEG industry standards group and is sought after as an expert witness and litigation consultant. Abharana Bhat specializes in video compression, media streaming and video quality assessment. Her PhD thesis on quality assessment of compressed video included the development of a novel, fully automated video quality metric. Iain Richardson iain@vcodex.com Vcodex.com Iain Richardson/Vcodex.com 2007-2014 9 of 9