COMP 9 Advanced Distributed Systems Multimedia Networking The Video Data Type Coding & Compression Basics Kevin Jeffay Department of Computer Science University of North Carolina at Chapel Hill jeffay@cs.unc.edu September 7, 999 http://www.cs.unc.edu/~jeffay/courses/comp9f99 The Video Data Type Outline! What is video?» Video components» Representations of video signals» Color spaces! Digital Video» Coding! Compression basics» Simple compression» Interpolation-based techniques» Predictive techniques» Transforms» Statistical techniques
Video Basics The components of video R-value G-value B-value! Video deals with absorbed and projected light» Cameras absorb light and monitors project light! The primary colors in this domain are:» red, green, and blue Video Basics The components of video transmission! Video is a multi-dimensional signal y x R-component G-component B-component time
Video Basics Video as a -dimensional signal! Representation of a -dimensional image (R, G, B), (R, G, B), (R, G, B),, (R, G, B) row, col! Representation of motion (-dimensional images) ms NTSC ( fps) ms PAL (5 fps) Frame i Frame i+ 5 Video Basics Resolution! Television broadcast standards» NTSC 55 lines» PAL 65 lines! Computer graphics standards» VGA 6x8» SVGA x768! Multimedia standards» CIF 5x88» QCIF 76x! Digital video standards» CCIR 6 7x8» HDTV x5 Videophone (8x) QCIF (76x) DVI (56x) H.6 CIF (5x88) NTSC (x8) Image sizes (in picture elements) 6
Video Basics Color spaces! RGB is not widely used for transmitting a signal between capture and display devices» It s difficult to manage separate inputs & outputs (and requires too much bandwidth)! Composite formats are used instead» Luminance ( Y ) the brightness of the monochrome signal» Chrominance the coloring information» Chrominance is typically represented by two color difference signals: U and V ( hue and tint ) or I and Q ( saturation and color ) 7 Video Basics Color spaces R R U = Plane Y = R=G=B (gray values) White G G V = Plane B B! NTSC video» Y =.R +.59G +.B» I =.6R.8G.B» Q =.R.5G +.B! PAL video/digital recorders» Y =.R +.6G +.B» U = (B Y) x.9» V = (R Y) x.877 8
Video Basics Digital video! Sample an analog representation of video (RGB or YUV) & quantize» Two dimensions of video are already discretized» Sample in the horizontal direction according to the resolution of the media! 8-bits per component per sample is common» bits per picture element (pixel)! Storage/transmission requirements» NTSC x 8 x x = 5x 6 bits/sec (9 MB/s or bits/pixel (bpp)) 9 The Video Data Type Outline! What is video?» Video components» Representations of video signals» Color spaces! Digital Video» Coding! Compression basics» Simple compression» Interpolation-based techniques» Predictive techniques» Transforms» Statistical techniques
Digital Video Compression Techniques! Do we really need every bit of a video stream?» Not if redundancy exists» Not if we can t perceive the effect of eliminating the bit! Eliminating redundancy» Spatial redundancy» Temporal redundancy! Eliminating imperceptible detail» Coding» Domain transformation Digital Video Compression Techniques Simple Interpolative Predictive Transform Statistical Truncation CLUT Run-length Sub-sampling DPCM Motion Compensation Discrete Cosine Transform Huffman & Arithmetic coding Fixed Adaptive Video Input Color Components (PCM Signals) Video Compression Algorithm Bit Bit Assignment Compressed Bit-Stream Adapted from Buford p.7
Video Compression Issues! Bandwidth requirements of resulting stream» Bits per pixel (bpp)! Image quality! Compression/decompression speed» Latency» Cost» Symmetry! Robustness» Tolerance of errors and loss! Application requirements» Live video» Stored video Simple Image Compression Truncation! Reducing the number of bits per pixel» Throw away the least significant bits of each sample value! Example» Go from RGB at 8 bits/component sample (8:8:8) to 5 bits (5:5:5) Go from bpp to 5 bpp This gives acceptable results» Go from YUV at 8 bits/component sample 6:5:5 (6 bpp)! Advantage simple!
Simple Compression Schemes Color-table lookup (CLUT)! Quantize coarser in the color domain» Pixel values represent indices into a color table» Tables can be optimized for individual images! Entries in color table stored at full resolution (e.g. bits) 5 6 7 5 6 7! Example:» 8-bit indices (56 colors) gives ( x 8) x 8 + ( x 56) =.7x 6 bits/sec 5 Simple Compression Schemes Run-length encoding 7 5 5 5 5 5 5 5 RLE 7 (5, 7)! Replace sequences of pixel components with identical values with a pair (value, count)! Works well for computer-generated images, cartoons. works less well for natural video! Also works well with CLUT encoded images (i.e., multiple techniques may be effectively combined) 6
Interpolative Compression Schemes Color sub-sampling! Do not acquire chrominance component values at all sampling points» Humans have poor acuity for color changes» UV and IQ components were defined with this in mind! Example: Color representation in digital tape recorders» Subsampling by a factor of horizontally is performed Y component U component V component 7 Interpolative Compression Schemes Color sub-sampling! Subsampling by a factor of horizontally & vertically Y component U component V component! Interpolating between samples provides excellent results» Chrominance still sampled at 8 bpp 8
Interpolative Compression Schemes Color sub-sampling! Intermediate pixels either take on the value of nearest sampling point or their value is computed by interpolation! Bi-linear interpolation: (,) (,) U(, ) ) = U(,)x.75 + U(,)x.5 + U(,)x.75 + U(,)x.5 (,) (,) Sub-sampled U or V component 9 Interpolative Compression Schemes Color sub-sampling Y component U component V component! Storage/transmission requirements reduction:» Within a x pixel block: (8 bpp luminance)x6 samples + (8 bpp chrominance)x bpp = 6 = 9» A 6.5% reduction overall
Predictive Compression Schemes Exploiting spatial & temporal redundancy! Adjacent pixels are frequently similar» Do pixel-by-pixel DPCM compression Leads to smearing of high-contrast edges» ADPCM a little better, a little worse Introduces edge quantization noise! Motion Estimation If the future is the similar to the past, encode only the difference between frames» This assumes we can store a previous frame to compare with a future one Transform-Based Compression Exploiting redundancy in other domains! A simple linear transformation x array of pixels -D array of differences A B A B A C A D A C D» Encode differences with less precision! Storage savings» Original array: pixels x 8 bpp = bits» Transformed array: 8 bits + ( pixels x bpp) = bits
Transform-Based Compression The Discrete Cosine Transform (DCT)! A transformation into the frequency domain! Example: 8 adjacent pixel values (e.g., luminance) 55 8 5 6 7 Sample values 7-8 5 6 7 Level-shifted values! What is the most compact way to represent this signal? Transform-Based Compression The Discrete Cosine Transform (DCT)! Represent the signal in terms of a set of cosine basis functions.... -. -. -. -..... -. -. -. -.
Transform-Based Compression The Discrete Cosine Transform (DCT)..5. -.5 -.. -. Sampled.5 Hz cosine wave! The basis functions derive from sampling cosine functions of increasing frequency» From -.5 Hz» Basis functions sampled at 8 discrete points 5 The Discrete Cosine Transform Represent input as a sum of scaled basis functions Level-shifted values DCT coefficients 7 = 7 Σi = 5 X i -8 5 6 7-5 5 6 7 i.... -.. -.. -.. 5 6 7 -.. -. -. -. -. 6
Transform-Based Compression The Discrete Cosine Transform (DCT)! The -dimensional transform: F(µ) = C(µ) 7 Σ x= f(x) cos (x+)µπ 6» F(µ) is the DCT coefficient for µ =..7» f(x) is the x th input sample fot x =..7» C (µ) is a constant (equal to -.5 if µ = and otherwise)! The -dimensional (spatial) transform: F(µ,ν) = C(µ) C(ν) 7 Σ y= 7 Σ f(x,y) x= cos (x+)µπ 6 cos (y+)νπ 6 7 Transform-Based Compression The Discrete Cosine Transform (DCT)! DCT coefficients encode the spatial frequency of the input signal» DC coefficient zero spatial frequency (the average sample value)» AC coefficients higher spatial frequencies 5-5 DC coefficient AC coefficients! Claim: Higher frequency coefficients will be zero and can be ignored 8
Transform-Based Compression The two-dimensional DCT! Apply the DCT in x and y dimensions simultaneously to 8x8 pixel blocks» Code coefficients individually Video Frame with fewer bits 7-9 - -8-8 - -8 6 - - - -8 5 - -5 - - - - -8-8 6 5 6 5 - -5 - - - 6-9 - 9 7-6 - DCT Coefficients 9 Statistical Compression Huffman coding! Exploit the fact that not all sample values are equally likely» Samples values are non-uniformly distributed» Encode common values with fewer bits and less common values with more bits! Process each image to determine the statistical distribution of sample values» Generate a codebook a table used by the decoder to interpret variable length codes» Codebook becomes part of the compressed image
Statistical Compression Huffman coding Symbol Probability Code A.75 B.5 C.65 D.65 P(ACBD) = P(BCD) =.5 P(CD) =.5 P(A) =.75 P(B) =.5 P(C) =.6 P(D) =.6! Order all possible sample values in a binary tree by combining the least likely samples into a sub-tree! Label the branches of the tree with s and s» Huffman code is the sequence of s and s on the path from the root to the leaf node for the symbol
COMP 9 Advanced Distributed Systems Multimedia Networking Video Compression Standards Kevin Jeffay Department of Computer Science University of North Carolina at Chapel Hill jeffay@cs.unc.edu September, 999 http://www.cs.unc.edu/~jeffay/courses/comp9f99 The Video Data Type Compression Standards u Basic compression techniques» Truncation, CLUT, run-length coding» sub-sampling & interpolation» DPCM» DCT» Huffman coding u Common algorithms» JPEG/MJPEG» H.6/H.6» MPEG-,-
Compression Algorithms JPEG u A still image ( continuous tone ) compression standard» DCT-based u Modes of compression» sequential image components coded in order scanned v Baseline default compression» progressive image coded in multiple passes so partial images can be displayed during decoding» lossless guaranteed no loss» hierarchical image encoded at multiple resolutions u Typical results» : compression ( bpp) JPEG Compression architecture sequential mode u Inputs are 8 or -bit samples» baseline = 8-bit samples u Image components are compressed separately» DCT operates on 8x8 pixel blocks Digitized still image (or video frame) RGB Color-Space Convertor YUV JPEG JPEG Compressed component bit-stream Bit-Stream Formatting To file or network Level Shifting DCT Quantizer Entropy
JPEG Compression Quantization u DCT coefficient quantization is the key to compression» Quantize according to the visual important of each coefficient u The application specifies a quantization table» A table of step-sizes from -55» Default tables specified for the baseline coder Image Store DCT DCT Quantizer DCT coefficients 7-9 - -8-8 - -8 6 - - - -8 5 - -5 - - - - -8-8 6 5 6 5 - -5 - - - 6-9 - 9 7-6 - 7 6 9 5 7 6 9 5 8 6 9 5 8 6 9 5 8 6 9 5 8 7 9 5 8 7 5 8 7 5 8 7 6 Step-size table 5 JPEG Compression Quantization example DCT coefficients 7-9 - -8-8 - -8 6 - - - -8 5 - -5 - - - - -8-8 6 5 6 5 - -5 - - - 6-9 - 9 7-6 - 7 6 9 5 7 6 9 5 8 6 9 5 8 6 9 5 8 6 9 5 8 7 9 5 8 7 5 8 7 5 8 7 6 Step-size table Image Store DCT DCT Quantizer u Step-size table can be scaled to control degree of compression» Scaling factor called the q-factor 6
7 JPEG Compression Coding coefficients u DC coefficients difference-coded» DC coefficients from adjacent 8x8 blocks strongly correlated u AC coefficients run-length and Huffman coded DCT DCT Quantizer Quantizer Entropy Entropy Image Store Quantized DCT coefficients 8 7 JPEG Compression Coding DC coefficients u DC coefficients DPCM coded and recoded using a variable length entropy (Huffman) code DCT DCT Quantizer Quantizer Entropy Entropy Image Store 9 (-9), ac, ac, ac,, (9-7), ac, ac, ac,
JPEG Compression Coding AC coefficients u Zig-zag order the AC coefficients to increase effectiveness of run-length coding» DC, AC, AC, AC, AC, AC, AC, AC, AC, AC, u Run-length code the stream» Each non-zero coefficient encoded as a pair: 5 6 7 DC AC AC 5 6 7 (run-length of preceding zero coefficients, amplitude of non-zero coefficient) 9 JPEG Compression Coding AC coefficients u Each coefficient encoded as a variable length pair» ([run length, size], amplitude) u First element coded using a variable-length Huffman code» A coding table ( code book ) must be provided v Can be generated on-the-fly with an additional pass over the coefficients v Up to four code books per image may be specified v The codebook becomes part of the coded bit-stream u Second element coded as a variable length integer» whose length is specified in the previous symbol
Sequential JPEG Compression Summary Complete compression pipeline Image Store Step-size Table 7 6 9 5 7 6 9 5 8 6 9 5 8 6 9 5 8 6 9 5 8 7 9 5 8 7 5 8 7 5 8 7 6 Huffman Code-book Color-Space Convertor Level Shifting DCT DCT Quantizer Entropy Compressed Image bit-stream ( component) u Compression comes from:» Chrominance subsampling» DCT coefficient quantization» Difference coding DC coefficients» Statistical & run-length coding of AC coefficients u Qualitative results:».5 -.5 bpp ok for some applications».5 -.75 bpp ok for many».75 -.5 bpp excellent».5 -. indistinguishable JPEG Compression Examples of quality v. bpp. bpp. bpp.9 bpp.7 bpp.5 bpp. bpp
JPEG Compression Examples of quality v. bpp. bpp.7 bpp JPEG Compression Examples of quality v. bpp.7 bpp.5 bpp
JPEG Compression Other modes of operation u Lossy compression modes» sequential image components coded in order scanned v Default mode» progressive image coded in multiple passes so partial images can be displayed during decoding v Useful for transmission of images over slow communications links» hierarchical image encoded at multiple resolutions v Useful for images that will be displayed on heterogeneous displays u Lossless mode» Guaranteed lossless» Uses DPCM encoding rather than DCT 5 JPEG Compression Modes Loseless mode operation A B C? Predictor C B A C + B A C + (B A)/ B + (C A)/ (C + B)/ Prediction Code 5 6 7 u Uses prediction instead of the DCT» Each pixel s value is expressed as a function of neighboring pixels» A code word identifies the predictor being used 6
JPEG Compression Modes Loseless mode operation Image Store Predictor Prediction Codewords Huffman Code-book DPCM Entropy Compressed Image bit-stream u Predicted samples are DPCM encoded u Differences are entropy coded as before u Achieves approximately : compression 7 JPEG Compression Modes Progressive mode operation u Encode the image in scans to enable the display of a series of progressively refined images» Requires an image-sized coefficient buffer between quantizer & entropy coder» Scans of image components are also interleaved in bit-stream Image Store Coefficient Buffer Color-Space Convertor Level Shifting DCT DCT Quantizer Entropy Compressed Image bit-stream (n components) 8
JPEG Compression Modes Progressive mode operation u Scan the coefficient buffer in multiple passes» Transmit portions of each coefficient DCT coefficients 8x8 pixel blocks : : 6 6 Sequential Encoding 7 6 Coefficient value (bits) Transmission order : : 6 6 7 6 5 First scan (Most significant bits) JPEG Compression Modes Progressive mode operation Successive approximation Second scan u Scan the coefficient buffer in multiple passes» Transmit portions of each coefficient Third scan 9 DCT coefficients 8x8 pixel blocks : : 6 6 Sequential Encoding 7 6 Coefficient value (bits) Transmission order 7 6 First scan 5 Spectral selection Second scan
JPEG Compression Mode Hierarchical mode operation u Encode the image at multiple resolutions» Each image differs from the previous by a factor of in either the vertical or horizontal dimension» Images created by filtering and subsampling u Each resolution encoded by either the sequential or progressive algorithm JPEG Compression Mode Hierarchical mode operation u Start with the lowest desired resolution & iteratively encode until the full image resolution has been coded» Each iteration encodes an image with a factor of higher resolution in one dimension Filter Filter & Up Up Sample Sample DCT DCT Decoder Inverse Quantizer Image Store DPCM Filter Filter & Down Down Sample Sample DCT DCT Lowest resolution image Quantizer Entropy
Motion JPEG Applying JPEG to moving images u Video can be (trivially) encoded as a sequence of stills» This practice is routine in the digital video editing world u The issue is how to encode and transmit side information» Quantization tables, Huffman code-book may/may not change between frames The Video Data Type Compression Standards u Basic compression techniques» Truncation, CLUT, run-length coding» sub-sampling & interpolation» DPCM» DCT» Huffman coding u Common algorithms» JPEG/MJPEG» H.6/H.6» MPEG-,-
Compression Algorithms H.6 (px6) u A telecommunications (ITU) standard for audio & video transmission over digital phone lines (ISDN) u H.6 primarily intended for interactive video applications» Design of the standard driven by a 5 ms maximum encoding/decoding delay goal u A scalable coding architecture capable of generating bit streams from 6 kbps ( x6 ) to,9 kbps ( x6 ) in 6 kbps increments» p =, produces a low res videophone (Common use is for ISDN BRI kbps video, 6 kbps audio)» p 6 produces an acceptable videoconference and allows multipoint communication 5 H.6 Video formats u Inputs» 55 or 65 line composite video» 8 bits/sample» frames/second QCIF (76x) u Color space» Y, Cr, Cb u Outputs» CIF or QCIF CIF (5x88) NTSC (x8)», 5,, or 7.5 frames/second 6
H.6 Video frame representation u Chrominance components are subsampled : horizontally & vertically u Each video frame is subdivided into 6x6 macroblocks Y component: 8x8 blocks Cr component: 8x8 block (same for Cb) 7 H.6 Video compression pipeline u Two compression modes, selectable on a frame-by-frame basis» INTRA-frame mode DCT-based compression á la JPEG v video is treated as a sequence of stills» INTER-frame mode Incorporates motion estimation & DPCM prediction v temporal redundancy is eliminated to further improve compression Compression Mode INTER-frame Pipeline INTRA-frame Pipeline Color-Space Convertor? Motion Predictor DPCM DCT Quantizer Entropy Coder Frame Store 8
H.6 Video Compression INTRA-frame mode Luminance Macroblock (6x6 pixel block) 8x8 pixel Macroblock blocks DCT DCT Buffer Buffer Quantizer Entropy Entropy Coder Coder Xmission Control Control Chrominance Macroblocks (8x8 pixel block) Step Step Size Size u Compression is similar to JPEG» DCT encoding» linear quantization» entropy coding u Quantization is uniform across all AC coefficients» But is adaptive and driven by the space remaining in a transmission buffer 9 H.6 INTER-Frame Mode Motion estimation & prediction Frame n Frame n u Motion estimation is performed only on luminance macroblocks» Compare a luminance macroblock with its neighbors in the previous frame» If the difference is small, do not compress the block, only record location of matching block» If the difference is large send the difference between this macroblock and a previous neighboring macroblock into the DCT compression pipeline
H.6 INTER-Frame Mode Motion estimation & prediction Frame n (6b x +w x, 6b y +w y ) (w x, w y ) (6b x, 6b y ) Frame n u Finding a predictor is the process of finding the minimally different adjacent 6x6 block in the previous frame» Construct a motion vector a relative displacement w for block b that minimizes the mean absolute distortion (MAD): 56 5 5 Σ Σ frame n [6b x +i, 6b y +j] frame n [(6b x +w x )+i, (6b y +w y )+j] j= i= H.6 INTER-Frame Mode Motion compensated prediction Previous Frame + Frame From DCT decode pipeline Luminance Macroblock Motion Vector Search 6x6 Predicted Block DPCM Motion Estimation 6x6 Difference Block Motion Motion Accuracy Computation no yes Motion Vectors To DCT encode pipeline Threshold
H.6 INTER-frame Mode Complete pipeline Loop Filter Loop Filter Motion Estimator Motion Previous Frame Frame yes no DCT + DCT Decoder Inverse Inverse Quantizer = video frame buffer = 6x6 pixel block = 8x8 pixel block = control data Motion Vectors Luminance Macroblock Macroblock Buffer Buffer 8x8 pixel block DCT DCT Quantizer Step Step Size Size Step Step Size Size Entropy Entropy Coder Coder Step-Size Adjustment Xmission Control Control Chrominance Macroblocks (8x8 pixel block) DCT DCT Quantizer Entropy Entropy Coder Coder H.6 Video frame representation u Macroblocks combined into groups of blocks (GOBs)» An by array of macroblocks 8 pixels 76 pixels 5 6 7 8 9 5 6 7 8 9 5 6 7 8 9 88 pixels 5 pixels 5 6 7 9 8 pixels 76 pixels 5 CIF Image: GOBs QCIF Image: GOBs
H.6 Data Transmission Bit-stream format = fixed-length = variable-length u Picture data is hierarchically transmitted Picture Layer picture header GOB data GOB data Group of Blocks Layer GOB header MB data MB data Macroblock Layer MB header block data block 6 data Block Layer DCT coeff DCT coeff EOB 5 ITU H. Teleconferencing Standards Teleconferencing over ISDN u H.6 Video communications at px6 kbps u H. Syntax for multiplexing audio and video packets u H. Protocol for call setup and negotiation of end-system ( terminal ) capabilities u H. Conference control protocol u G.7 ISDN audio coding standard at 6 kbps u G.7 High-quality audio at 6 kbps u G.78 Reduced quality speech at 6 kbps 6
H.6 Video Compression Low-bitrate video compression for data networks u Based on H.6 (& MPEG-, -) u Includes new image formats: Format Image Size Maximum Number of coded bits/picture sub-qcif QCIF CIF CIF 6CIF 8 x 96 76 x 5 x 88 7 x 576,8 x,5 6 6 56 5 u Added coding efficiency from:» Unrestricted motion vectors» Bi-directional motion estimation/prediction» Arithmetic coding of AC coefficients 7 H.6 Video Compression Companion standards u H.6 Low bit-rate video coding u H. Terminal systems u H.5 Conference control u H. Audio/video multiplexing u G.7 Audio coding 5. and 6. kbps u For Internet conferencing there is also the related T. Document Conferencing standards family 8
The Video Data Type Compression Standards u Basic compression techniques» Truncation, CLUT, run-length coding» sub-sampling & interpolation» DPCM» DCT» Huffman coding u Common algorithms» JPEG/MJPEG» H.6/H.6» MPEG-,- 9 Compression Algorithms MPEG u A family of audio/video coding schemes» MPEG- A video coding standard for digital storage/ retrieval devices v VHS quality video coded at approximately.5 Mbps» MPEG- Video coding for digital television v SIF/CIF to HDTV resolutions at data rates up to Mbps» MPEG- Coding of audio/visual objects for multimedia applications v Coding of natural & synthetic images v Object-based encoding for content access & manipulation» MPEG-7 A content/meta-data representation standard for content search and retrieval
MPEG- Video Compression Requirements u MPEG intended primarily for stored video applications» A generic standard» But a basic assumption is that video will be coded once and played multiple times u Support for VCR-like operations» Fast forward/forward scan» Rewind/reverse scan» Direct random access» MPEG- Video Compression Relation to H.6 u Similar to H.6» INTER and INTRA picture types, entropy encoded, motion compensated, DCT-based compression 6x6 Macroblock Motion Estimator Motion Previous DCT Frame + DCT Frame Decoder 8x8 pixel block Macroblock Buffer Buffer DCT DCT Inverse Inverse Quantizer Quantizer u but with more aggressive motion compensation:» prediction approximately the same as in H.6» interpolation (bi-directional prediction)» DPCM encoding of motion vectors Entropy Entropy Coder Coder Motion Vectors
MPEG- Video Compression Relation to JPEG u Non-uniform quantization for intra-coded pictures» Uniform quantization for inter-coded pictures 6x6 Macroblock Motion Estimator Motion Previous DCT Frame + DCT Frame Decoder 8x8 pixel block Macroblock Buffer Buffer Motion Vectors Step-size table DCT DCT 8 6 9 6 6 7 6 9 6 7 7 9 Inverse Inverse Quantizer Quantizer 9 6 6 7 9 9 5 7 7 9 8 6 7 9 9 5 8 6 7 9 5 6 56 Entropy Entropy Coder Coder 9 7 8 56 69 7 8 8 58 69 8 MPEG Video Compression Motion compensated prediction Frame n k Frame n u The predictor search space is not specified in the standard» Implementations can perform as exhaustive a search as they desire u Find the motion vector w that minimizes some cost function f 5 5 Σ Σ f(frame n [6b x +i, 6b y +j] frame n [6b x +w x +i, 6b x +w y +j]) w j= i=
MPEG Motion Compensated Prediction Bi-directional prediction Frame n k Frame n Frame n+k u Besides simple prediction, interpolation (bi-directional prediction) is used to achieve further compression u A future frame and a past frame are used to predict the current frame» Deals effectively with scene changes and new object appearances» Produces predictors (pairs) with better statistical properties 5 MPEG Motion Compensated Prediction Bi-directional prediction I B B B P B B B I u MPEG defines three picture (frame) types:» I intracoded pictures coded as a still image» P predicted pictures predicted from the previous I or P picture» B interpolated pictures predicted from the previous I or P picture and the next I or P picture 6
MPEG Motion Compensated Prediction Bi-directional prediction 5 6 7 8 9 I B B B P B B B I u Directional prediction implies that frames cannot be encoded or transmitted in the order they are scanned u Encoding & transmission order:» I P 5 B B B I 9 B 6 B 7 B 8 P B B B I 7 B B 5 B 6 7 MPEG Motion Compensated Prediction Bi-directional prediction I or P picture (frame n y) B picture (frame n) I or P picture (frame n+x) u Bi-directional prediction modes are selectable on a macroblock by macroblock basis within a B picture u Macroblocks can be predicted by:» themselves: frame' n [i, j] = frame n [i, j]» a previous frame: frame' n [i, j] = frame n y [i+w x, j+w y ]» a future frame: frame' n [i, j] = frame n+x [i+w x, j+w y ]» a previous & future frame: frame' n [i, j] = (frame n y [i+w x, j+w y ] + frame n+x [i+w' x, j+w' y ])/ 8
Bi-Directional Motion Compensation Compression rates u Some prototypical results for movies encoded at x, fps and constant quality Crocodile Dundee ET 9 MPEG Video Compression Decoder architecture = video frame = 6x6 pixel block = 8x8 pixel block = control data 8x8 pixel block data Entropy Entropy Decoder Stream Stream Demultiplexor Inverse Inverse Quantizer Step Step Size Size Motion Vectors DCT DCT Decoder DPCM DPCM Decoder 6x6 Macroblocks Macroblock Buffer + Buffer Prediction Control Control Decoded Picture Picture Buffer Buffer Macroblock Type Past Past Reference Frame Frame Future Future Reference Frame Frame 5
MPEG Video Compression Coded bit-stream u MPEG has a layered bit-stream similar to H.6 u There are seven layers:» Sequence Layer v decoding parameters (bit-rate, buffer size, picture resolution, frame rate, )» Group of Pictures Layer v a random access point» Picture Layer v picture type and reference picture information» Slice Layer v position and state information for decoder resynchronization» Macroblock Layer v coded motion vectors» Block Layer v coded DCT coefficients, quantizer step size, etc. 5 MPEG- New & Improved MPEG- u A coding standard for the broadcast industry» Coding for video that originates from cameras» Offers little benefit for material originally recorded on film u But included is support for:» Higher (chrominance) sampling rates» Resilience to transmission errors» u More mature and powerful coding/compression technology is used» Unrestricted motion search with / pel resolution for motion vectors 5