Video Compression Dr. Y. H. Chan Contents Fundamentals of video Picture formats Frame rates Temporal redundancy spatial redundancy compensation prediction estimation compensation vector Full search algorithm -step search algorithm Complexity vs performance Codec Types of frames Transmission order Coding of different frames Video stream syntax Sequence GOP Picture Macroblock Block Learning Objectives Examine both the theoretic and practical aspects of the video compression process. Describe the temporal redundancy that may be exploited by video compression algorithm. Address motion-compensated prediction coding in the video compression. Describe MPEG compression standards. References: Multimedia Communications: Applications, Networks, Protocols and Standards, Fred Halsall, Addison-Wesley, Chapter 4 Contents (Cont.) Standards : Background & History MPEG-/-/-4/H-4/AVS MPEG- : Profiles and Levels Scalability - various schemes and their use MPEG-4 : Goals coding of video objects H.4 : New features Inter-mode: Sub-macroblocks of various sizes Multiple reference frames B-frame prediction weighting (/8)-pixel motion vector precision Intra-mode: Spatial prediction Integer transform 4
Fundamentals of Video Picture Format (High Resolution) CCIR-0 4:: format Y: 70 480 UV: 0 480 Application: Original digitization format used in CCIR-0 (Television Studios) CCIR-0 4::0 format Y: 70 480 UV: 0 40 (Absence of chrominance samples in alterative lines) Application: Digital video broadcast applications 7 Introduction The bit rate requirements resulting from all digital video are substantially larger than the bit rates of the transmission channels. Compression is necessary NOT just a single standard associated with video H., H., MPEG-, MPEG- and MPEG-4 Picture Format (Low Resolution) CIF (Common Intermediate Format) 4::0 format Y: 0 88 UV: 80 44 Applications: A picture quality comparable with that obtained with VCRs quality or video conferencing QCIF 4::0 format Y: 80 44 UV: 90 7 Applications: Mobile multimedia 8
Digital Video Video (moving pictures) : a sequence of digitized picture. Video Compression Principles Video (moving pictures) : a sequence of digitized picture. Moving JPEG (0: to 4:): spatial redundancy Consider: a person or vehicle moving across the screen in a movie Last for at least s Assume: frame rate 0 frame per second 80 frames By sending only information relating to those segments of each frame that have movement associated with them Considerable savings in bandwidth Temporal redundancy Digital Video (Frame rates) frame rates: full-motion video: 4-0 frames/s or even 0 frames/s for HDTV animation: -9 frames/s video telephony: -0 frames/s video conferencing & interactive media applications: -0 frames/s Video - Temporal Redundancy Similarities exist between successive frames frame t- frame t Processing for reducing temporal redundancies Frame Difference Estimation and Compensation Coding Processing for reducing spatial EIE4 Multimedia redundancies Technology 0 9
-Compensation Prediction Hybrid Coding Scheme with MC Prediction I(x,y,t-) I(x,y,t) Estimation Compensation motion vector (u,v) e(x,y,t) D-DCT Coding estimation - the process of finding corresponding pixels among frames, i.e. the motion vector (u,v). vector - the relative displacement of a object from one frame to another Prediction error: I(x,y,t) Encoder e(x,y,t) I(x+u,y+v,t-) D-DCT Decoding e ( x y ) = I I ( ( x x y y ) ) The prediction error is encoded by the DCT coding method ˆ( I I ( x x y u ) y v ) Frame memory Decoder Compensation compensation is performed in both the encoder and the decoder, but motion estimation is needed only in encoder asymmetric property -Compensation Moving Object Stationary background Previous frame Current frame compensation (u,v) Displaced Object time t the process of compensating for the displacement of moving objects from one frame to another Prediction for luminance signal I(x,y,t) with the moving object: I ˆ ( x, y, t ) = I ( x u, y v, t ) 4 Block Matching Estimation The block that gives the best match for a given matching criterion is chosen as the motion vector. search window (-D, D) N D Full searching most accurate heavy computation Fast searching reduced computation complexity deterioration in picture quality N N D N D frame t- D Matching Criterion: Sum Absolute Difference SAD ( k ) = N N I ( ) I ( + k + ) Block I frame t I ( ( ( ( k u v ( u v ) = arg min SAD ( k (k,l) ) :block in the current frame ) :block in the referenceframe ):position of the candidate motion vector ): the final motion vector ),, t,,, t, t,, t,, t t t,, l,,, l i = 0 j = 0 t i,, j t i, j l, l
(-,0) Searching window (-,0) Searching window Reference frame Current frame Candidate macroblock Full Search (0,) Macroblock of interest (0,0) (,0) (0,-) The local coordinate system used in the searching window are used to determine the motion vector of the macro block. 7 Reference frame Current frame Full Search (0,) (0,0) (,0) Macroblock of interest (0,-) MAE y = 9 7 8 8 4 MAE plane in the window - - - -4 - - x = - - -4 - - - 0 4 9 (-,0) Searching window (-,0) Searching window Reference frame (0,) (0,0) (0,-) Reference frame (0,) (0,0) (0,-) Current frame (,0) Macroblock of interest MAE y = - - - -4 - - x = - - -4 - - - 0 4 Current frame (,0) Macroblock of interest MAE Macroblock of interest y = x = - - - -4 - - MAE 9 7 8 8 4 8 7 4 7 8 4 8 8 4 7 9 4 4 9 7 8 8 4 9 4 7 8 4 7 7 7 7 8 7 4 8 7 4 4 9 7 7 8 0 0 8 8 4 7 8 9 4 7 7 9 7 9 7 4 9 7 4 4 8 7 8 7 4 8 7 - - -4 - - - 0 4 Full Search MAE plane in the window 8 Full Search MAE plane in the window 0 4 0 4 0 4 0
Reference frame Current frame (0,) (-,0) (0,0) (,0) (0,-) Searching window min MAE=0 happens at (-,-) the motion vector of the block is (-,-). Realization complexity is high. y = x = - - - -4 - - 9 7 8 8 4 8 7 4 7 8 4 8 8 4 7 9 4 4 9 7 8 8 4 9 4 7 8 4 7 7 7 7 8 7 4 8 7 4 4 9 7 7 8 0 0 8 8 4 7 8 9 4 7 7 9 7 9 7 4 9 7 4 4 8 7 8 7 4 8 7 - - -4 - - - 0 4 Reference frame Current frame (0,) (-,0) (0,0) (,0) Macroblock of interest (0,-) Searching window y = x = - - - -4 - - 9 7 8 8 4 8 7 4 7 8 4 8 8 4 7 9 4 4 9 7 8 8 4 9 4 7 8 4 7 7 7 7 8 7 4 8 7 4 4 9 7 7 8 0 0 8 8 4 7 8 9 4 7 7 9 7 9 7 4 9 7 4 4 8 7 8 7 4 8 7 - - -4 - - - 0 4 Full Search Macroblock of interest MAE plane in the window -step search Step MAE plane in the window n-step Hierachical Search (n-shs) (Tutorial) Reference frame Current frame -step search Step (0,) (-,0) (0,0) (,0) Macroblock of interest (0,-) Searching window y = - - - -4 - - 9 7 8 8 4 8 7 4 7 8 4 8 8 4 7 9 4 4 9 7 8 8 4 9 4 7 8 4 7 7 7 7 8 7 4 8 7 4 4 9 7 7 8 0 0 8 8 4 7 8 9 4 7 7 9 7 9 7 4 9 7 4 4 8 7 8 7 4 8 7 x = - - -4 - - - 0 4 MAE plane in the window 4 4 0 4 0 7 4 0 7 9
Reference frame Current frame -step search Step (0,) (-,0) (0,0) (,0) Macroblock of interest (0,-) Searching window y = - - - -4 - - 9 7 8 8 4 8 7 4 7 8 4 8 8 4 7 9 4 4 9 7 8 8 4 9 4 7 8 4 7 7 7 7 8 7 4 8 7 4 4 9 7 7 8 0 0 8 8 4 7 8 9 4 7 7 9 7 9 7 4 9 7 4 4 8 7 8 7 4 8 7 x = - - -4 - - - 0 4 MAE plane in the window Global and Local Minima -step search can be trapped in the local minima Local minima of the distortion function giving prediction images with higher error than full search So, many research works have been carried out to find alternative approaches for block matching motion estimation. 7 (-,0) Searching window Reference frame (0,) (0,0) (0,-) Current frame -step search Step (,0) Macroblock of interest Two in all checked candidate blocks provides the min MAE. the motion vector for the concerned block is (-,0) or (-,-). Low complexity but may not provide the best motion vector. y = - - - -4 - - 9 7 8 8 4 8 7 4 7 8 4 8 8 4 7 9 4 4 9 7 8 8 4 9 4 7 8 4 7 7 7 7 8 7 4 8 7 4 4 9 7 7 8 0 0 8 8 4 7 8 9 4 7 7 9 7 9 7 4 9 7 4 4 8 7 8 7 4 8 7 x = - - -4 - - - 0 4 MAE plane in the window Codec 4 0 7 9 7 8 4 0 7 9 7 8
Video Encoder Source video Frame Memory + - Regulator D-DCT Quantizer VLC coder Buffer Compressed bit stream 000... Dequantizer Inverse D-DCT Compensation Frame Memory Estimation 9 Predictive frame vectors + + Picture Types Forward prediction I P P P P P P P I Intra (I) picture without reference to any other pictures provide access points to the coded sequence where decoding begin coded with only moderate compression (0:) Predicted (P) picture reference to the I-pictures or P-pictures generally used as a reference for further prediction Compression ratio is about 0: Video Decoder Compressed bit stream 00.. Qunatizer step size VLC Buffer Dequantizer Decoder Inverse D-DCT + + Recontructed video vectors Compensation Frame Memory Forward prediction With B-picture I B B B P B B B I Bidirectional prediction Bidirectional (B) picture using motion compensated prediction from a past and a future I or P picture never used as a reference or further prediction highest compression ratio (0:) 0
Transmission and Display Order Display Order I B B B P B B B I B B B P B B B I 4 7 8 9 0 4 7 Transmission Order I P B B B I B B B P B B B I B B B 4 9 7 8 0 7 4 Rule: Before you send a B frame, make sure all its reference frames have been sent. Send the reference frame first if not. Solution A suitable reordered frame sequence that meets the defined requirements is: I,P,B,B,P,B,B,P, B,B, I, B, B,...,4,,, 7,,,0,8,9,,, Example An MPEG- system uses the frame sequence shown as below. Derive a suitable reordered sequence that ensures firstly, only two frames must be stored in the decoder, and secondly, the required I and/or P-frames are available to decode each P- and B-frame as they are received. Coding of I-Picture I-Frame -D FDCT Q VLC Coded I-Frame Intraframe coding 8 8 DCT Any weighting matrix for DCT coefficients possible Differential coding of DC coefficient Uniform qunatization Run length coding of zeros with zig-zag scan Entropy coding 4
Coding of P-Pictures compensated prediction from an I- or P-pictures One motion vector per macroblock Coding of prediction error with 8 8 DCT, uniform qunatization, zig-zag scan (like I-picture) Coding of B-Pictures Forward Vectors ref frame current frame ref frame Estimator Estimator Interpolation Backward Vectors Encode the motion vectors Coded Vectors for B-frame + + Prediction Error -D FDCT Q VLC Coded residual signal 7 9 Coding of Vectors Highly correlated among neighboring macroblocks In many standards, macroblock is the unit for motion estimation. It will be fine tuned to the block level if necessary Code the difference using variable length code (VLC) compensated prediction from two consecutive P- or I-pictures either only forward prediction ( vector/macroblock) or only backward prediction ( vector/macroblock) or average of forward and backward prediction = interpolation ( vectors/macroblock) Coding of prediction error with 8 8 DCT, uniform qunatization, zig-zag scan (like I-picture) 8 40
Video Stream Syntax Video Stream Syntax IBBPBBIBBPBBIB... Group of Picture Macroblock Y Y Y Y Group of Picture Picture Block Cr Cb 4 Video Stream Syntax Six Layers Sequence Layer GOP Layer Picture Layer Layer Macroblock Layer Block Layer Supplementary notes on Video Stream Syntax 8 8 Block Address Quant. value vector Estimation Unit. Macroblock Y Y Y Y4 Cr Macrolock layer MB Header Y Y Y Y4 Cr Cb Block layer Y Y Y Y4 Cr Cb DC AC.. EOB DCT Unit 4 Cb 44
Macrolock layer Picture layer MB Header Y Y Y Y4 Cr Cb Picture Header 4 MB 4 layer Header MB MB MB MB 4.. MB N Start code position Quant. value GOP start code: indicates the start of a GOP Time stamp: used for synchronization purposes Parameters: defines the particular sequence of frame types that are used in each GOP (e.g. IPPBPP) 4 47 Sequence start code Video parameters Quantization value IBBPBBIBBPBBIB... Group of Picture Group of Picture Sequence layer Sequence Header GOP GOP GOP GOP4 GOP GOP GOP layer GOP Header P P P P4 P P GOP Header Picture layer Picture Header 4 Picture start code Picture type. 4 Sequence Sequence layer GOP GOP GOP GOP4 GOP GOP Header GOP layer GOP Header P P P P4 P P Picture layer Picture Header 4 layer Header MB MB MB MB 4.. MB N Macroblock layer MB Header Y Y Y Y4 Cr Cb Block layer Y Y Y Y4 Cr Cb 48
Layer Each slice is a contiguous sequence of macroblocks in raster scan order, starting at specific address or position in the picture specified in the slice order Each element in the slice is a macroblock Reasons for concepts : (i) to allow flexibility in signal changes in some of the coding parameters (ii) to optimize quality for a given bit rate (iii) to control the bit rate 49 What is MPEG? MPEG means Moving Picture Experts Group a working group of ISO/IEC Development of international standards for compression, decompression, processing, and coded representation of moving pictures, audio and their combination. Standards: Background & History Chronological Table of Video Coding Standards ITU-T VCEG H. (990) ISO/IEC MPEG MPEG- (99) H. (99/9) H.+ (997/98) MPEG- (H.) (994/9) H.++ (000) H.4 ( MPEG-4 Part 0 ) (00) MPEG-4 v (998/99) MPEG-4 v (999/00) MPEG-4 v (00) 990 99 994 99 998 000 00 00
MPEG Achievements Built MPEG-, MPEG- Widely adopted in audiovisual industry Digital TV; VCD, DVD, Video-On-Demand, archiving, Music on the Internet MPEG-4 is still on-going First version ready at end of 998 Second version was approved at the end of 999 MPEG-7 Multimedia Content Description Interface Working draft was appeared at the end of year 999 IS had been approved at July 00 History: MPEG- Audio/Video on Digital Storage Media (. Mbit/s, CIF) Start Late 988 Committee Draft 9/90 (Bitstream Syntax and Decoding) ISO/IEC 7- - (Systems, Video, Audio, Compliance, Software) MPEG- MPEG- Progressive scanning with a refresh rate of 0Hz (for NTSC) and Hz (for PAL). NTSC: Y = 40 C b = C r = 7 0 PAL: Y = 88 C b = C r = 7 44 Support I-, P- and B- frames I-frames must be used for the various random-access functions associated with VCRs. A new layer called slice is added in the structure of the stream such that the decoder can resynchronize more quickly in case of error.
Example A digitized video is to be compressed using the MPEG- standard. Assuming a frame sequence of: IBBPBBPBBPBBI... and average compression ratios of 0: (I), 0: (P) and 0: (B), derive the average bit rate that is generated by the encoder for both the NTSC and PAL digitization formats. Solution (Cont.) NTSC frame size: Without compression = 40 8 + (7 0 8) =.070 Mbits per frame With compression =.07 /9.4 = 4.70 kbits per frame Hence bit rate generated at 0 fps =.040Mbps 7 9 Solution Frame sequence = IBBPBBPBBPBBI... Hence: I-frame: / P-frame: / B-frame: 8/ Average CR = ( 0. + 0.0 + 8 0.0)/ = 0.04 or 9.4: Solution (Cont.) PAL frame size: Without compression = 88 8 + (7 44 8) =. Mbits per frame With compression =. /9.4 = 4.04 kbits per frame Hence bit rate generated at fps =.040Mbps Normally, allowing for packetization and multiplexing overheads, a bandwidth of. Mbps is allocated for the video. Hence, assuming a maximum bit rate of. Mbps, this leaves 00 kbps for the compressed audio stream. 8 0
MPEG- MPEG- standard is intented to be generic and to serve a wide range of applications. The concepts of profile and level have been introduced to make the practical implementations of the specification easier. What is a profile? A profile is a "defined subset of the syntax of the specification". Imposes some bounds to the full syntax Defines which tools or functionalities may be used to produce a bitstream and how. What is a level? A level is a "defined set of constraints on the values which may be taken by the parameters of the specification within a particular profile". History: MPEG- Audio/Video for broadcast (Digital TV, HDTV, Cable, Satellite) 4-0Mbit/s for Digital TV Committee Draft /9 (Bitstream Syntax and Decoding) Profiles@Levels, supporting interlacing ISO/IEC 88- - (Systems, Video, Audio, Compliance, Software, DSM-CC) Levels A level specifies the range of the parameters that are supported by the implementation image size frame rate bit rate Four levels are defined in MPEG-: each targeted at a particular application domain. Level Samples/lines Lines/frame Frames/sec High (H) 90 0 High-440 (H-4) 440 0 Main (ML) 70 7 0 Low (L) 88 0 4
There are profiles associated with each level: Simple Main Spatial resolution Quantization accuracy High. The different combinations of levels and profiles form a framework for all standards activities associated with MPEG-. One of the most popular setting is the MP@ML standard which is for digital television broadcasting. advanced television (ATV) in North America, digital video broadcast (DVB) in Europe, and multiple sub-nyquist sampling encoding (MUSE) in Japan. MPEG- Scalable Coding Concepts A new feature of MPEG- is bit stream scalability, which allows for a layered representation of the coded bit stream Reasons: to provide interoperability between different services to give flexible support to receivers with different display capabilities to provide a layered video bitstream that is amenable for prioritized transmission. to made it possible for transmission in the presence of channel errors, such as in ATM 7 Common MPEG- Profile and Levels in Simplified Form Level Spatial resolution layer High Enhancement Samples/line Lines/frame Frames/sec High- 440 Lower Samples/line Lines/frame Frames/sec Enhancement Samples/line Lines/frame Frames/sec Lower Samples/line Lines/frame Frames/sec Main Enhancement Samples/line Lines/frame Frames/sec Lower Samples/line Lines/frame Frames/sec Low Enhancement Samples/line Lines/frame Frames/sec Lower Samples/line Lines/frame Frames/sec Simple Pictures: I,P Chroma: 4::0 70 7 0 Main Pictures: I,P,B Chroma: 4::0 90 0 Profile SNR Pictures: I,P,B Chroma: 4::0 Spatial Pictures: I,P,B Chroma: 4:: High Pictures: I,P,B Chroma: 4:: 90 0 90 7 440 0 440 0 70 7 70 7 0 70 7 0 0 0 440 0 70 7 0 70 7 88 88 0 88 0 0 0 Example of uses Scaling between HDTV and SDTV (Standard Definition Television) important for HDTV to be compatible with SDTV transmission using HDTV quality but using scalable coding: low resolution - base layer high resolution - base layer + enhancement layer 8
Scalable Encoder and Decoder two layers provided : Base Layer : lower-resolution video with reduced bit rate (downsampling spatially or temporally) upscaled reconstructed base-layer video (upsampled spatially or temporally) to be used as a prediction for the recovering of the original input video signal Enhancement Layer to code the prediction errors Reconstruction : Low quality Reconstruction : to makes use of Base Layer only High quality Reconstruction : to makes use of both Base Layer and Enhancement Layer 9 Data partitioning Single-layer encoder Data partitioner 7 Scalable Coding Schemes Data Partitioning when two channels are available, critical bitstreams for better channel and less critical for poorer channel SNR (Quality) Scalability multiple-quality video services Spatial Scalability multiple-dimension displays Temporal Scalability Multiple frame rates 70 Multiplexer Video Enh. Layer Bitstream Base Layer Bitstream Output Bitstream Priority breakpoint base layer: include all critical information (such as header, motion vectors, enh layer: the remaining bitstream when two channels are available, critical bitstreams for better channel and less critical for poorer channel SNR Scalability possible way for degradation of the video quality in prioritized transmission media reduced quality video : using base layer, protected from transmission error it is based on a frequency (DCT-domain) scalability technique to obtain higher quality video signal : using both lower and higher layer bit streams 7
encodes and transmits the difference between the nonquantized DCT coefficients and the quantized coefficients from the base layer with finer quantization step size + - Enhancement Encoder Base Layer Decoder Enh. Layer Bitstream Enhancement Decoder + + High quality video Video Base Layer Encoder Base Layer Bitstream At base Layer : DCT coefficients are coarsely quantized and transmitted ( reduce bit rate) Base Layer Decoder Low quality video 7 to code the difference signal the lower layer decoder spatially interpolates the signal to full spatial resolution for prediction in the enhancement layer Video Enhancement Encoder Enh. Layer Bitstream Enhancement Decoder High resolution video Downsampling Upsampling Upsampling Spatial Spatial Spatial the input video is spatially decimated to lower spatial resolution signal Base Layer Encoder Base Layer Bitstream Lower resolution video with reduced bit rate Base Layer Decoder Low resolution video 7 Spatial Scalability It supports displays with different spatial resolutions at the receiver. Base Layer: the input video is spatially decimated to lower spatial resolution signal and encoded (to send more important data, error free if possible). lower spatial resolution video: to be constructed from the base layer. Enhancement Layer: the lower layer decoder spatially interpolates the signal to full spatial resolution for prediction in the enhancement layer. for high resolution video: bit streams from both layers are decoded. Spatial Scalability - Decoder A single-layer decoder decodes only the base layer to produce a reduced-resolution output sequence. A two-layer decoder can reconstruct a fullresolution sequence as follows:. Decode the base layer and up-sample to the original resolution.. Decode the enhancement layer.. Add the decoded residual from the enhancement layer to the decoded base layer to formthe output frame. 74 7
4 Temporal Scalability It shares similar concept with the spatial scalability Base Layer: to provide the basic temporal rate (to send more important data, error free if possible). for low quality display - only the lower layer is decoded. Enhancement Layer: to be coded with temporal prediction with respect to the lower layer. for high quality display - bit streams from both layers are decoded and recombined for display. Advantages Layered coding facilitates Unequal error protection Efficient use of resources Customer satisfaction Multiple services possible to combine different scalability tools into a hybrid coding scheme. i.e. interoperability between services with different spatial resolutions and frame rates can be supported by means of combining the spatial scalability and the temporal scalability tools into a hybrid layered coding scheme. 77 79 to code the difference signal the lower layer decoder temporally interpolates the signal to full temporal resolution for prediction in the enhancement layer Video Enhancement Encoder Enh. Layer Bitstream Enhancement Decoder Full frame-rate video Downsampling Upsampling Upsampling Temporal Temporal Temporal the input video is temporally decimated to lower temporal resolution signal temporal Base Layer Encoder Base Layer Bitstream Base Layer Decoder Lower frame-rate video with reduced bit rate Low frame-rate video 78 Comparison Between MPEG- and MPEG- Coding Parameter MPEG- MPEG- Standard 99 994 Main application Digital Video Digital TV (and HDTV) on CD-ROM Spatial resolution ~88x0 pixels 7x70 pixels (x440 pixels) Temporal resolution -0 frames/s 0-0 fields/s (00-0 fields/s) Bit rate. Mb/s ~4 Mb/s (~0 Mb/s) Quality Comparable to Comparable to VHS NTSC/PAL for TV Compression ratio ~0-0 ~0-40 over PCM (~0-40) 80
History: MPEG- MPEG- existed once upon a time, but its goal, enabling HDTV, could be accomplished using the tools of MPEG-, and hence the work item was abandoned MPEG-4: the Developments Original Target: Advanced Coding at Rates Lower than 4 kbit/s More information is in digital form, More information is on-line, More information is audio/visual, Applications & services become multimedia,. Applications & services become interactive, Audiovisual information now covers all bitrates and all networks In short: multimedia is happening 8 8 MPEG-4 Issues that upcoming MPEG-4 standards address In July 994, the focus was shifted from compression new functionalities + compression compression no longer the only reason for doing coding low bitrate, although still important, is not MPEG-4 s only focus! Having AV information available wherever you are Reusing the audiovisual content, combining elements of content in new ways More freedom to interact with what is within the content 84
MPEG-4 (Coding of Audio-Visual Objects) Generic Standard for Coding of Multimedia Contents 99-998 First Call to International Standard ISO/IEC Project # 449 8 MPEG-4: Coding of AV Objects Audiovisual scene is composed of Objects. Compositor puts objects in scene. Objects can be different in nature. natural or synthetic AV, text & graphics, animated faces, arbitrary shape or rectangular Coding scheme can differ for individual objects. Principle is independent of bitrate. from low bitrates to (virtually) lossless quality. 87 MPEG-4: New Direction (Goals). Interactive computer applications. Wireless communications. Integration of AV data into a number of applications Video database Interactive home shopping Video games Content-Based Interactivity Wireless video Internet video Universal Access Improved Coding Efficiency Video e-mail Multimedia authoring Synthetic natural AV 8 MPEG-4 Audiovisual Scene scene D person furniture background AV presentation voice video globe desk 88
More about the goals of MPEG-4 Interactivity Content based manipulation and bitstream editing Improved temporal random access Original Decoded Decoded and Manipulated Overview of MPEG-4 System 89 9 More about the goals of MPEG-4 Integration of natural and synthetic material Mixing synthetic and natural objects together in the same scene Virtual environments 90 encoder multiplexer demultiplexer decoder (Conventional System) VLBC Core and the Generic MPEG-4 Coder (MV) Texture (DCT) MPEG-4 VLBV Core Coder (similar to H./MPEG-) Shape (MV) Texture (DCT) Generic MPEG-4 Coder 9
Basic Block Diagram of MPEG-4 Video Coder Arbitrary Shape Shape Coder + Texture Coder Processor and Buffer + Compensator Previous/Next Reconstructed VOPs Store Estimator MV Predictor and Coder 9 System Multiplexer - Video Object partial data MOTION CODER Overview of MPEG-4 System Scene segmentation and depth layering O contour motion texture contour motion texture contour motion texture O O Layered encoding bitstream layer bitstream layer bitstream layer Separate decoding 94 multiplexer demultiplexer compositor Processing Steps in MPEG-4 Compositor Demux Syntactic decoding Decompression Elementary streams Syntactic decoded streams BIFS(Binary Format for Scenes) Set of nodes to represent the primitive scene objects to be composed, the scene graph constructs, the behavior and interactivity Update stream: modify the scene in time Anim Stream: animate the scene in time Primitive AV Object Scene description (script or classes) MPEG-4 terminal 9 AV-objects Network Layer Composition information Upstream data (user events, class request etc.) H.4
The H.4 Video Coding Standard Multi-mode, multi-reference MC vector can point out of image border /4-, /8-pixel motion vector precision B-frame prediction weighting Integer transform Multi-mode intra-prediction In-loop de-blocking filter UVLC (Uniform Variable Length Coding) NAL (Network Abstraction Layer) SP-slices Position of H.4 97 99 A Note on Terminology of H.4 The following terms are used interchangeably: H.L The Work of the JVT or JVT CODEC JM.x, JM.x, JM4.x The Thing Beyond H.L The AVC or Advanced Video CODE Proper Terminology going forward: MPEG-4 Part 0 (Official MPEG Term) ISO/IEC 449-0 AVC H.4 (Official ITU Term) Basic Marcoblock Coding Structure Input Video Signal Split into Macroblocks of x pixels each - Decoder Coder Control Transform/ Scal./Quant. Scaling & Inv. Transform Control Data Quant. Transf. coeffs Intra/Inter Intra-frame Prediction - Compensation De-blocking Filter Output Video Signal Estimation Data Entropy Coding 98 00
Compensation Input Video Signal Split into Macroblocks of x pixels each - Decoder Coder Control Transform/ Scal./Quant. Scaling & Inv. Transform Control Data Quant. Transf. coeffs Entropy Coding Intra/Inter Intra-frame Prediction - Compensation Estimation De-blocking Filter MB Types 8x8 Types x 0 8x8 Output Video Signal x8 0 8x4 0 8x 0 4x8 0 0 Data Various block sizes and shapes B-frame Prediction Weighting I 0 B B B P 4 B B Time Playback order: I 0 B B B P 4 B B... Bitstream order: I 0 P 4 B B B P 8 B... 8x8 0 4x4 0 0 0 Multiple Reference Frames Input Video Signal Split into Macroblocks of x pixels each - Decoder Coder Control Transform/ Scal./Quant. Scaling & Inv. Transform Control Data Quant. Transf. coeffs Entropy Coding Intra/Inter Intra-frame Prediction - Compensation De-blocking Filter Output Video Signal Estimation Data Multiple Reference Frames for Compensation Integer Transform Input Video Signal Decoder Split into Macroblocks 8x8 DCT is replaced by 4x4 x and pixels x Integer Transforms. e.g. Y=HXH T H = Advantages of integer transform: Intra/Inter - Coder Control Transform/ Scal./Quant. Intra-frame Prediction - Compensation No multiplcation (only shift-add) No rounding error Estimation Scaling & Inv. Transform De-blocking Filter Output Video Signal Control Data Quant. Transf. coeffs Data Entropy Coding 0 04
Example: Coding a x residual macroblock All 4x4 luma and chroma blocks are ordered as follows and transformed with a 4x4 integer transform. DC coefficients of blocks 0, are extracted to form block -, which is further transformed with another 4x4 integer transform DC coefficients of blocks 8- (-) are extracted to form block (7), which is further transformed with a x integer transform Blocks are encoded and sent according to the sequential order. Note all DC coefficients in blocks 0-, 8- and - are removed as they are encoded in block -, & 7. If a block is encoded in intra mode, a prediction is made based on previously encoded blocks and the prediction result is subtracted from the current block prior to encoding. Modes of prediction: 9 optional prediction modes to predict a 4x4 luma block 4 optional modes to predict a x luma block mode to each predict a 4x4 chroma block. 0 07 Intra-prediction Modes Input Video Signal Split into Macroblocks x pixels - Decoder Intra/Inter Coder Control Transform/ Scal./Quant. Intra-frame Prediction - Compensation Estimation Scaling & Inv. Transform In older standards, blocks are encoded independently Control in intra mode without making Data use of the interblock correlation inefficient Quant. H.4 supports Transf. coeffs Directional spatial prediction in intra mode (9 types for 4x4 luma, for 4x4 chroma, 4 types for x luma) 4x4 luma case: De-blocking Q A B C D E F G H Filter I a b c d J e f g h K i j Output k l L m n o Video p Signal e.g., Mode (horizontal): a=b=c=d=i, Data e=f=g=h=j i=j=k=l=k, m=n=o=p=l 4 Entropy Coding 7 0 8 - DC 0 08
Comparison Summary (H.4) Video coding is based on hybrid video coding and similar in spirit to other standards but with important differences New key features are: Enhanced motion compensation Small blocks for transform coding Enhanced entropy coding Substantial bit-rate savings (up to 0%) relative to other standards for the same quality Enhancement on perceptive quality seems better than that on PSNR The complexity of the encoder triples that of the prior ones The complexity of the decoder doubles that of the prior ones 09 0
AVS Overview of AVS.0 Video High-efficiency coding Focused application initially HDTV Lower complexity, lower cost Complete solution: Video, audio, systems, DRM Simple, comprehensive patent licensing AVS (Audio Video coding Standard) AVS activity in China Make a simple and low-cost industry standard Use as more as we can the parts from MPEG, avoid/around disagreement for IPR policy Co-design on both technology and IP One-stop-shopping license Integration of blocking technology from partners Open standard Impact to China Organization of standard in China SAC(Standardization administration of China) Information standardization committees AVS Working group 4
AVS-Video Tools Major Tools Transform bit-implemented 8x8 transform (different kernel as compared with H.4) Quantization and scaling scaling only in encoder Intra prediction modes compensation x/x8/8x/8x8 modes Quarter-pel interpolation 4-taps interpolation filter Deblocking Entropy coding END 7 What you have learnt? Video coding exploits the temporal redundancy between successive frames in video. -compensated prediction: the process of compensating for the displacement of moving objects from one frame to another. Video compression standards: H, H, MPEG,,4 and H.4. -compensated prediction, DCT, differential coding, runlength coding and huffman coding Scalability 8