Multimedia Communication

Transcription

1 Multimedia Communication Video Perception and Coding Dr. Andreas Kassler Universität Ulm Slide 1

2 Perception and Coding of Video Image Coding See Lecture Datenkompression (Dr. Jochen Messner) or See Lecture Mediale Informatik (Prof. Dr. Weber) Perception of Video See Lecture Mediale Informatik (Prof. Dr. Weber) Intra-Frame Coding Inter-Frame Coding ITU-T Standards H.261 H.263 ISO-Standards MPEG-I MPEG-II MPEG-IV Slide 2

3 Literature V. Bhaskaran, K. Konstantinidis: Image and Video Compression Standards, Kluwer, 1999 W. Effelsberg, R. Steinmetz: Video Compression Techniqes, dpunkt.verlag, 1998 F. Kuo, W. Effelsberg, J. Garcia-Lunes-Aceves (Hrsg.): Multimedia Communications, Prentice Hall, 1998 H.26X: R. Shaphorst: Videoconferencing and Videotelephony, Artech House, 1999 MPEG: MPEG: Slide 3

4 3 dimensions 2 spatial + 1 temporal Video Perception and Capturing Main characteristics Frame size (width x height) Frame rate (frame rate [fps]) Color accuracy (bits/per pixel, color space) r s Higher compression ratio possible due to temporal redundancy t Slide 4

5 Video Capturing Video Perception and Capturing Sampling & Quantisation CCIR 601 HDTV (HD-I, HD-P) EDTV (EQTV) HD-1440 Analog television signal PAL NTSC conventional Hz Hz Sub-Sampling interlaced Hz Hz progressive Hz Hz SQCIF, QCIF, CIF 4CIF, 16CIF HDTV Slide 5

6 Standard Intermediate Format, 352x240, 30 Hz Video Perception and Capturing pixel/row pixel/col ratio fps Mb/s Slide 6 Distributed Systems Department MPEG-1 MPEG-2 H.261 H.263 Quarter Common Intermediate Format, 25 Hz Characteristics SQCIF : y QCIF ~4: ,84-3,8 - - y y CIF (SIF) ~4: ,1-30,4 y y o o 4CIF ~4: o CCIR ~4: ,9 - y - - EDTV : ,9 - y CIF : o HD : ,5 - y - - HDTV : ,0 - y - - Future Gbit/s PAL, non-interlaced y : supported, o : optional, - : not supported

7 Video Perception and Capturing t Image 2 Image 1 Image 0 Image 6 Image 5 Image 4 Image 3 Image 2 Image 1 Image 0 T t Slide 7 t r 1. half picture 2. half picture T s T s s Interlaced Scanning Progressive Scanning

8 Video Perception and Capturing Interlaced Scanning - saves bandwidth - Reduces flicker - Odd and 50 Hz (europe) 60 Hz (US) - Originally developed for CRT (Cathode Ray Tube) displays - The odd and even fields merge together due to HVS persistence and the phosphor persistence Slide 8

9 Video Perception and Coding Slide 9

10 Video Perception and Coding Slide 10

11 Video Perception and Capturing Slide 11

12 Video Perception and Capturing Slide 12

13 Video Perception and Capturing Interlaced: sawtooth artifacts around details in motion, (mice teeth or combing). Slide 13

14 Video Perception and Capturing Interlaced: Line flicker significant and sudden change in brightness between neighbored horizontal scan lines HVS is most sensitive to brightness changes in an image fast flickering can cause fine detail or boarders of objects to appear to be jumping up and down. Slide 14

15 Video Perception and Capturing 4:3 16:9 Slide 15

16 Video Perception and Capturing Slide 16

17 Overview of Compression Methods Motion JPEG (M-JPEG) Apply JPEG coding to single frames Compression ratio ~ 10:1 H.261(1990)/H.263(1996) Symmetric DCT-based compression optimized for real-time conferencing Compression ratio ~ 50:1 Motion Pictures Expert Group (MPEG) MPEG-I(1991), MPEG-2 (1994), MPEG-4 (1998) Asymmetric (fast decompression, slow compression) Suitable for ondemand Scenarios Compression ratio ~ 50:1 On new machines compression in real-time possible Slide 17

18 Overview of Compression Methods 10k 100k 1M 10M 100M 1G datarate Video communication 1:1 10:1 compresson ratio MPEG 4 (1998) MPEG 1 (8/93) MPEG 2 (11/94) 100:1 Bit/s TV HDTV Low Low bit bit rate rate appl. appl Mobile Mobile A/V A/V Comm Comm H.263 (03/96) Videoconference Videoconference Videophone Videophone H.261 (3/93) Storage&Retrieval Storage &Retrieval Videoconference Videoconference TV TV distribution distribution Communication&Storage Communication Storage MPEG 2 (11/94) High High Definition- Definition applications applications source: Multimedia World 1000:1 Slide 18

19 Intraframe vs. Interframe Coding Slide 19

20 Intraframe/Interframe Interframe Coding Exploit redundancies Spatial correlation Intraframe Coding Remove redundancies within a single frame Two complementary block based schemes Transform based coding JPEG Subsampling on Luminance/Chrominance planes Human eye more sensitive to luminance subsample colour components Fewer samples per line, fewer lines per frame Macroblock: N Luminance, M chrominance blocks E.g. 4:2:0: reduction in bitstream Temporal correlation Interframe Coding Successive frames look similar Only differences between frames is encoded Motion compensation Y Cb Cr Slide 20

21 Temporal correlation Intraframe/Interframe Interframe Coding from P-frame P I-frame B-frame from I-frame I P-frame Slide 21

22 Frame types Intraframe/Interframe Interframe Coding I-frames: Intra Coded for error resiliency and joining movie P-Frames: Predicted from previous I- or P-frame B-Frames: Bi-directional frames Interpolation between two frames I B B B P B B B P B B B I Display order Group of Pictures (GOP): frame pattern not standardized May be adapted during runtime (scene changes) Slide 22

23 Frame types Intraframe/Interframe Interframe Coding Compression Order!= Transmission Order Decoder must receive reference P-frame to decompress B-frame Adds substantial delay (~160 ms), need additional buffer (>=3 frames) Display I B B B P B B B P B B B I B B B P Bitstream I P B B B P B B B I B B B P Presentation I B B B P B B B P B B B I I P B B B P B B B Transmission order Slide 23

24 Encoder Intraframe Coding Transform Coding Image block Coefficients Bitstream TT Q Entropy Coding Q Transform Quantizer Decoder Bitstream Entropy Decoding Q -1 Q -1 T -1 T -1 Reconstructed Image block Postprocessor needed? Quantization table in encoder and decoder are the same Inverse Transform Slide 24

25 Intraframe Coding Transform Coding Transform samples to frequency domain Can better exploit redundancy, eyes are more insensible to high freq. Used in JPEG, H.26X, MPEG standards Discrete Cosine Transform (DCT) NxN pixel block, high N leads to complex implementation, tradeoff:n=8 Loss-less operation, implementation lossy due to rounding 2D Matrix transform: C = DCT T *B*DCT with DCT T *DCT = I c b (2 1) ( ) ( ) u (2 i + 1) v j + u, v = 4 e u e v b i, j cos[ 16 π ]cos[ 16 π i= 0 j= u (2 i + 1) v (2 j + 1) i, j = 4 e ( u ) e ( v ) c u, v cos[ 16 π ]cos[ 16 π u= 0 v= 0 e ( u ) = 1 ( u = 0 ), 1 ( otherwise 2 ] ] ) Slide 25

26 Intraframe Coding DC coefficient DCT AC coefficients 8x8 Image block DCT coefficients Slide 26

27 Intraframe Coding Original image transformed image Slide 27

28 DCT Coding Intraframe Coding Exploit redundancy in frequency domain Threshold coding and quantization using quantization matrix Lossy step z kl = round ( y q kl kl ) = y kl q ± kl qkl 2 Afterwards zig-zag scanning Convert 2D to 1D coefficients Generate long run of 0 Loss-less entropy coding Differential coding of DC coefficients Predict DC, code difference Using modified RLE for AC coefficients Run: length of zero run Level: amplitude of non-zero symbol Variable length codes (e.g. Huffman) Short codes for frequent symbols Slide 28

29 Intraframe Coding - Example DCT Quantizer Quantizer Matrix Y Quantizer Matrix Cb/Cr 79; EOB Slide 29

30 Intraframe Coding VLC Example EOB Slide 30

31 Interframe Coding Two stage process Stage 1: Processing to reduce temporal redundancy Segment into macroblocks Interframe coder: motion estimation Stage 2: Processing to reduce spatial redundancy in difference Process difference frame from stage 1 Use DCT coding Intraframe coding of difefrence frame Stage1 f n f n+1 Stage2 residual Slide 31

32 Interframe Coding Simple: Conditional Replenishment Exploit temporal redundancy Transmit only changing blocks Only exact repetition of blocks or complete replacement by new blocks Need to look at frame differences and estimate these to further improve performance MPEG, H.26X Only temporal redundancy reduction in ITU-T Rec. H.120 previous current Slide 32

33 Interframe Coding Advanced Motion Estimation and compensation Exploit temporal redundancy, predict current frame Block based motion estimation Simple, hardware based implementation possible Object based motion compensation Pixel based motion compensation Large computation overhead previous current Slide 33

34 Interframe Coding Block based motion estimation and compensation Block Matching Algorithm (BMA) for Encoder Motion Estimation (ME) Find block that has maximum correlation value in reference frame to actual processed block. Spatial displacement vector is called Motion Vector Differential coding in x and y Motion compensation (MC): decoding process represents motion by applying transmitted MVs to prior decoded picture current Previous=reference Search Range Slide 34

35 Block matching algorithm Algorithm 2D full search Interframe Coding Fast Search (reduce complexity but also match accuracy) Two-Dimensional Logarithm Parallel Hierarchical One-Dimensional Search Hierarchical Motion Estimation Match criterion Distortion measures MSE mostly used in software Mean Absolute Error mostly used in hardware Search Range Determines performance Slide 35

36 Interframe Coding Two-Dimensional Logarithmic Three step hierarchical Slide 36

37 Hybrid Motion Compensated DCT Coder Structure Interframe Coding DCT Coder Structure Video in MB + - Motion compensated Prediction INTRA/INTER DCT Decoder Coding Coding Control Control Q Q -1 Q -1 VL-Entropy- Coder Coder Mode Qparam Encoded residual IDCT Motion Motion Compensation Compensation input frame approx. + Frame Frame Buffer Buffer Prediction- Coder Coder Motion- Motion- Estimation delay Motion Vector Slide 37

38 MPEG Slide 38

39 Motivation for Standards Specification available to all Everybody can implement Interoperability Standards Agreement by consensus, not by interest of one partner Relatively open committee Industry Research institutes competitors Slide 39

40 Standards Scope of Video Coding Standards Specify Syntax Decoder Enables interoperability Vendors differentiate themselves Optimizations Complexity Pre-Processing Coder Pre-Processing Coder Post-Processing Decoder Standard Slide 40

41 MPEG Standards MPEG Moving Pictures Expert Group Standardisation via ISO/IEC International Standardization Organization and International Electrotechnical Commission MPEG-1, MPEG-2, MPEG-4 Video and audio coding Defines data stream syntax and decompressor, no compressor! Several applications possible MPEG-1: Video CD, MP3 MPEG-2: DVD, Set Top Box MPEG-4: Fixed and Mobile Web MPEG-7 Multimedia Description Language and Schemes MPEG-21 Defines open framework for multimedia delivery and consumption To enable transparent and augmented use of multimedia resources across a University wide of Ulm, Distributed range Systems, of networks Dr. A. Kassler and devices Slide 41

42 MPEG Tools Slide 42

43 MPEG-I Slide 43

44 Goals MPEG I ISO/IEC ,, ISO/IEC Compress audio& video at T1 speed ~ 1.5 mbps Progressive, not interlaced Quality 4:2:0 subsampling SIF (352x240)@30 fps, CIF@25fps CD-quality audio: 2 channel, 16 bit/sample, 48 khz = bps compression!! Due to limitations on macroblocks/s standard definition TV not possible Random access <0.5s Fast forward/backward width Height Nr. Macroblocks Macroblocks/s Frame rate VBV buffer size Bitrate <= 768 pixel <= 576 lines <= 396 <= 9900 <= 25 fps >= bits <= bps introduce D frames that consist only of DC-Coefficients of single blocks Easy editing Slide 44

45 Stream Structure Sequence MPEG-I GOP: IBBPBB, IBBBPBBB, IBBPBBPBB, IPPPPPP, Frame: I, P, B Slice Any number of sequential Macroblocks Encoded without reference to any other slice If corrupted, decoding can start at next slice Macroblock 16 Block 16 Cb Cr Y Macroblock Slide 45

46 Coding of Macroblocks Intra coding MPEG-I Similar to JPEG, however: Default quantization table (QT), QT may be changed by specifying new table in sequence header Scaling the quantization by Mquant (1,..31) Set explicitly for each slice May be changed per macroblock Higher Mquant: higher compression ratio Non-Intra coding Flat QT for residual blocks (all values are 16) Use only luminance samples for motion estimation Slide 46

47 Non-Intracoding Steps 1. Intracode MPEG-I If failed to find good motion estimation use Intracoding 2. Find best match to macroblock in reference frame MBA 3. Subtract each block in macroblock from best match 4. Process residual block Array of error values will be input in DCT, Q, with flat QT Residual DC coefficient not treated separately! Should be quantized to 0 anyway (why?) Skip block, if estimation was so good that quantized to 0 4. Process motion vector Send full motion vector for first macro block in slice Use predictive coding for the remaining motion vectors in the slice Slide 47

48 P-Frames MPEG-I IF entire macroblock was quantized to 0 Write skip -code into macroblock header proceed with next macroblock ELSE Calculate_bits_inter_forward = bits_motionvector + bits_residual IF Nr_bits(intracoding) <= Calculate_bits_inter_forward Code as intra //then next macroblock in slice assumes a value of 0 for preceeding motion vector ELSE Code as inter Slide 48

49 B-Frames MPEG-I Calculate_bits_inter_forward, Calculate_bits_intra Find backward prediction, Calculate_bits_inter_backward Do Interpolated Prediction F: Forward predicted macroblock B: Backward predicted Macroblock C: Current Macroblock R: Residual Macroblock Calculate bits for interpolated prediction: motion vectors + residual R i F + i, j B i, j, j = C i, j, i, j = 2 Use the best wrt. minimum bits 0,..15 Slide 49

50 Rate Control MPEG-I Typically, bits(i_frame) > bits(p_frame) > bits (B_frame) Using constant rate transmission (e.g. ISDN) time to transmit I-frames is longest Also amount of bits depend on complexity of scene Must control amount of bits per frame Adjust quantizer via Mquant Delay between encoder-input and decoder-output must be constant Buffer needed at encoder Variable rate input in buffer, emptied at constant rate Buffer needed at decoder Constant rate input in buffer, removed with variable sized blocks VBV: size of decoder buffer signalled in sequence header Overflow avoid! Underflow avoid! Slide 50

51 MPEG-I Rate Control Rate Control Test Model (TM) 5 Rate Control Based on constant nr. Bits per GOP Start with initial distribution 140:52:36 for I:P:B frames per GOP After frame has been processed calculate remaining bits in GOP And Recompute target bits/frame based on new situation Within each frame Estimate which Mquant is necessary to achieve the bits/frame After coding each macro block re-estimate nominal scale factor Use remaining bits Use nr. Bits for current macroblock Before coding of macro block, examine pixel values Measure spatial activity of macro block Modify nominal scale up or down by up to 2:1 Quiet macroblock: conserve bits Active macroblock: spend more bits for better quality Slide 51

52 Video in MB + - Motion compensated Prediction DCT MPEG- I INTRA/INTER Coding Coding Control Control Q -1 Q -1 Mode Qparam Fillgrade VL-Entropy- Q Buffer Coder Buffer Coder Out IDCT Motion Motion Compensation Compensation input frame approx. + Frame Frame Buffer Buffer Prediction- Coder Coder Motion- Motion- Estimation delay Motion Vector Slide 52

53 MPEG- I Buffer Buffer VLC VLC Decode Decode Q -1-1 IDCT + Prediction- Coder DC Coefficients Motion- Motion- Predictor Motion Vector Simplified Decoder Buffer/ Buffer/ Reorder Reorder Out Reference Reference Frame Frame memory memory Slide 53

54 Multiplexing MPEG-I Combine >= 1 data streams into single stream Video and audio Timing information Afterwards suited for transmission/storage Decoder: Slide 54

55 Bitstream syntax MPEG-I Hierarchical structure of headers and data Sequence header contains control information, like frame size and rate Group header contains time information Picture header contains information about next frame (e.g. type) Slice header contains position of slide and quantization information Macro-block header contains type of macro-block etc. <sequence> ::= <sequence header> <group of pictures> {<sequence header> <group of pictures>} <sequence end code> <group of pictures> ::= <group header> <picture> {<picture>} <picture> ::= <picture header> <slice> {<slice>} <slice> ::= <slice header> <macroblock> {<macroblock>} <macroblock> ::= <macroblock header> <block> {<block>} Slide 55

56 MPEG-II Slide 56

57 Goals MPEG-II ISO/IEC 13818, 9 parts, extends MPEG-I Better quality than MPEG-I Digital broadcasting Support interlaced video CCIR601 HDTV Support 4:2:0, 4:2:2 and 4:4:4 chroma subsampling Permit 9- and 10-bit precision DC coefficients Concealment motion vector blocks (CMV) together with intracoded MBs If MB is lost CMV in MB above points to similar macroblock Higher data rate (4 100 Mbps) Restriction of slice structure to max. a single row Pan and Scan information associated with frames for e.g. 16:9 to 4:3 conversion Scalable Streams: Receivers with different capabilities Slide 57

58 MPEG-II Scalability Images can be coded in different layers Base layer contains low quality, vital information Enhancement layers increase quality Scalability modes Data Partitioning, e.g. Base layer lower order coefficients Enhancement layer higher order coefficients Signal-to-Noise (SNR): Each frame partitioned into additive layers, e.g. Base layer course quantization Enhancement layer fine quantization Spatial: Each frame coded at different resolutions, e.g. Base layer standard format Enhancement layer HDTV add-on Temporal Base layer e.g. 30 fps, enhancement layer e.g. 60 fps Slide 58

59 MPEG-II Data partitioning Data partitioning Enables scalability wrt. error resilience Important information in base layer Less important in enhancement layer Add more redundancy to base layer during transmission Better protection for vital information Example: Base layer: Motion Vectors, lower bits of DC coeff Enhancement layer: higher bits of DC, AC coefficients Slide 59

60 MPEG-II Additional Features Additional Features Prediction Modes and Motion Compensation Field and frame prediction //due to interlaced support 16 x 8 pixel motion compensation //MPEG-1 has 16 x 16 Each 16 x 16 MB split into two 16 x 8 blocks Apply field/frame prediction independently P-frame: Assign 2 MVs for each MB B-frame: Assign 2 or 4 MVs for each MB Dual-prime prediction (16 x 16 blocks) for P-frames Main MV (of same parity field) 2 nd MV (opposite parity field) derived from Main MV and differential MV Predictions are average of two reference fields Non-linear quantization tables New VLC tables for DCT coefficients Slide 60

61 MPEG-II Half Pixel Resolution MC Using motion vectors with half-pixel resolution allows for increased efficiency in motion compensation Image has thus virtually higher resolution Linear interpolation of in-between pixel values Half-pel right transition A C B D h 1 A + B = Half-pel down transition v 1 A + C = h v 2 2 = = C B D D Half-pel down-right transition hv A + B + C + D = Slide 61

62 DCT Coding MPEG-II Frame DCT Coding Field DCT Coding Slide 62

63 Scan Order Alternatives MPEG-II Zig-Zag Scan Alternate Scan interlaced Slide 63

64 MPEG-II Profiles and levels MPEG-II Profile: defined subset of entire bit stream syntax specified by MPEG-2 MPEG-II Level: a set of constraints for each profile Maximum bitrate Base layer: 960x576@30 Base layer: 720x576@30 Base layer: 720x576@30 Base layer: 352x288@30 Slide 64

65 System Layer MPEG-II Multiplexing and transmission of audio and video Output of audio and video coder elementary stream (ES) Video Video Coder Coder Audio Audio Coder Coder ES ES Packetizer Packetizer Video PES Audio PES PS PS MUX MUX Program Stream MPEG-II Systems Spec TS TS MUX MUX Transport Stream Slide 65

66 MPEG-II Picture width Picture Height Aspect Ratio Bit Rate Picture Rate Slice Sequence Header GOP Header Sequence Sequence Header Frame Header Frame 1 Sequence... Frame Header Frame N Macroblock Block Temporal References Frame Type VBV Delay... Extension Startcode Frame Structure Slice Header Macro Blocks 1..N Slice Header Macro... Blocks 1..N Slice Header Macro Blocks 1..N Address Type Quantizer Scale Motion Vectors Coded Blk pattern Block 1 Block 2... System Layer Elementary Stream Slide 66

67 MPEG-II System Layer Packetized Elementary Stream (PES) ES divided into packets of variable length E.g. each compressed video frame in one PES packet Each packet has header to identify payload, timing, Program Stream Low error environment Consists of packs, each containing one or more PES Pack header contains sync information Accommodates <= 16 video and <= 32 audio PES PES-header Every pack stamp from single system clock reference (SCR) PES-packets video video audio video pack Slide 67

68 MPEG-II compress to elementary stream data (ES) I-frame data B B P B B P B B I Packetization using variable size PES packets I-frame data B B P PES Packet PES Header TS Packet TS Header (4 Bytes) TS Payload PES Payload also possible: fixed size packets System Layer Transport Stream Fixed size packets (188 bytes) Slide 68

69 MPEG-II Sync-Byte= 0x PID Continuity Adaptation Counter 4 Field A.F.Length Flags 5 Optional Fields Stuffing Bytes Splice Private Private PCR OPCR Countdown Data Len 8 8 Data... Transport Stream Payload (PES Packets) Adaptation Field Control Scrambling Control Transport Priority Payload Unit Start Indicator Transport Error Indicator PES Priority Random Access Indicator Discontinuity Indicator 8 3 Optional Fields Flags Adaptation Field Extension Length Slide 69

70 Error Sources Transport Stream MPEG-II Corrupt ES header lose frames Corrupt PES header lose whole ES TS carries many PES and its own header Header contains info on navigation through PES via PSI and PID If corrupted many PES are lost PCR provides timing information Errors in PCR timing is lost Slide 70

71 MPEG-II Packing Efficiency 89.00% 84.00% 79.00% 74.00% 69.00% with TS Header without TS header TS packets/aal5 frame Slide 71

72 MPEG-II delay (ms) Slide Packing Delay (ms) number of MPEG-2 TS per AAL5 frame transport rate (Mbps)

73 MPEG-II average loss ratio 100% 10% 1% 0% 30dB TS packets 38dB TS packets 44dB TS packets 30dB frames 38dB frames 44dB frames nr. TS packets per AAL5 PDU Slide 73

74 MPEG-II Frame Nr. 84, 1 TS scheme 30 db 38 db 44 db Slide 74

75 MPEG-II Frame Nr. 84, 2 TS scheme 30 db 38 db 44 db Slide 75

76 MPEG-II Frame Nr. 84, 8 TS scheme 30 db 38 db 44 db Slide 76

77 MPEG-II 38 db 40 1E E+12 1E+10 1E PSNR [db] 15 1 TS 2 TS 8 TS 38 db 1E E E-08 1E E frame nr. Slide 77

78 MPEG-4 Slide 78

79 MPEG-4 ISO/IEC MPEG-4 Rob Koenen (Ed.): MPEG-4 Overview, V.21, März /mpeg-4.htm Low bitrate video, operating in erroneous environment For UMTS, videophones, Data rates: 5, 64 kbit/s fps Slide 79

80 MPEG-4 MPEG-4 High quality video, 64 kbit/s.. 4 mbit/s Resolution: small. digital TV Support for progressive and interlaced Standardisation Representation of audio/visual objects - AVO Each AVO may be coded in a different way independent from other AVO Natural or synthetic objects Composition of Objects Audiovisual scene = combined AVOs Multiplexing and Synchronisation of AVOs Slide 80

81 MPEG-4 4 defines System Decoder Model Specifies reference decoder Description language MPEG-4 Binary syntax of AVO bitstream-representation Scene information Concepts, Tools, Algorithms for Content based compression of AVOs and combined AVOs Object Manipulation Object Transmission Random access to Objects Animation Scaling Error resilience Slide 81

82 MPEG-4 Scene Composition Scene Composition Description contains Hierarchical AVOs as tree Position of AVO in space and time Convert to global coordinates Attributes Texture, parameters for animation, Description based on VRML concepts Interactions with scene Change view point Drag object Start/stop stream Select language Slide 82

83 MPEG-4 Scene Composition Enables sprite coding Transmit background once Buffer background at decoder Transmit foreground and camera movements separately Slide 83

84 Scene Composition MPEG-4 Slide 84

85 Scene Graph Example MPEG-4 Slide 85

86 Videocoder MPEG-4 Each video object in a scene is coded separately Compresses rectangular still images and video Similar to MPEG-1/2, interlaced support Motion compensation Video coding of arbitrarily shaped video objects Form-adaptive DCT, 8x8 DCT Transparency information through alpha-plane 12 bit video Compressor Generates Timing Information Inserts time-stamps estimated time to decompress and decompress time Lower bound on buffer for decoder Slide 86

87 Video Object Plane MPEG-4 VOP 1 VOP 2 VOP 3 Slide 87

88 Structure of Encoder MPEG-4 Slide 88

89 Structure of Decoder MPEG-4 Slide 89

90 Videocoder Very Low Bitrate Video (VLBV) Additionally High bitrate tools interlaced Additionally content based tools Shape coding scalability MPEG-4 Slide 90

91 Videocoder For natural scenes MPEG-4 Motion Motion Compensation VOP Buffer VOP Buffer Shape Shape Coding Coding Motion Motion Coding Coding Texture Texture Coding Coding MUX MUX Slide 91

92 MPEG-4 MPEG-4 4 Visual Bitstream Hierarchy Visual Object Sequence (VS) complete MPEG-4 scene which may contain any 2-D or 3-D natural or synthetic objects Video Object (VO) rectangular frame, arbitrarily shaped object or scene background Video Object Layer (VOL) single layer of scalable, or the only layer of non-scalable form Visual object sequence header, video object header, and video object layer header may be repeated in a single bitstream to enable random access Group of Video Object Planes (GOV) optional grouping of VOPs in MPEG-4 bitstream Slide 92

93 MPEG-4 MPEG-4 4 Visual Bitstream Hierarchy Video Object Plane (VOP) time sample of a video object conventional video frame is represented by a rectangular-shaped VOP Video Packet (VP) provides for error resiliency of the MPEG-4 bitstream activation of this mode is optional, but highly recommended for transmission in error-prone environments contains an integer number of macroblocks, and approximately constant number of bits Macroblock (MB) 4 blocks of luminance samples and 2 blocks of chrominance samples Block 8x8 matrix of video samples or DC coefficients Slide 93

94 MPEG-4 Visual Object Sequence VOS header VO 1 VO 2... VO n VO header Video Object VOL 1 VOL 2... VOL n MPEG-4 decoder configuration information Video Object Layer VOL header GOV 1 GOV 2... GOV n Group of Video Object Planes GOV header VOP header VOP 1 VOP 2... VOP n Video Object Plane VP 1 VP 2... VP n MPEG-4 elementary streams (ES) Video Packet VP header MB 1 header Block 1 Block 2... Block n MB 2 header... Slide 94

95 VOP Encoding VOP encoded macroblockwise MPEG-4 Extend bounding box of VOP on right-bottom side to match 16x16 raster VOP Shape block (binary alpha block) Bounding box Slide 95

96 VOP Encoding Macroblock coding Entirely Inside VOP: Conventional DCT scheme MPEG-4 Partially outside VOP: Coded by DCT after padding Outside VOP: not coded Slide 96

97 MPEG-4 VOP Encoding Shape Adaptive DCT Coding of arbitrary shaped objects 1-D column DCT 1-D row DCT Slide 97

98 MPEG-4 Quantization Visually weighted quantization Intra Blocks Inter Blocks QF [][] v u = ( F [][ v u ] 16 // W [ v ][ u ] K quantiser _ scale )// ( 2 quantiser _ scale ) Slide 98

99 Motion Compensation VOP Coding Modes: I, P, B MPEG-4 Slide 99

100 Scalability MPEG-4 Complexity scalability at encoder encoders of different complexity can generate valid bitstreams Complexity scalability at decoder less powerful decoders may decode only a part of the bitstream Spatial scalability Allows decoder to decode subset of total bitstream to reconstruct and display textures, images and video objects, Max. 11 levels Temporal Scalability reconstruct and display video at reduced temporal resolution, max 3 levels Quality Scalability Bitstream is composed of number of layers of different bitrate combination of a subset of the layers can still be decoded bitstream parsing either during transmission or in the decoder Fine Grained Scalability (FGS) Combines above scalability aspects in fine grain steps up to 11 Slide 100

101 Spatial Scalability Distributed Systems Department 2 nd enh. layer 1 st enh. layer Base layer MPEG-4 Temporal Scalability Slide 101

102 MPEG-4 I B B B P B B B P B B B I 24 fps = base, first, second and third enhancement layer Group of Pictures (GOP): started by I-frame independent series of frames GOP pattern determined by encoder B-frames are not referenced and can be dropped for temporal scalability Slide 102

103 MPEG-4 I B B P B B P B B I 18 fps = base, first, and second enhancement layer Slide 103

104 MPEG-4 I B P B P B I 12 fps = base, and first enhancement layer Slide 104

105 MPEG-4 I P P I 6 fps = base layer Slide 105

106 DCT DCT bitplane bitplane Entropy Entropy coder coder MPEG-4 Enhancement layer Video in MB Motion compensated Prediction DCT DCT Base layer Q Q Q -1 Q -1 IDCT Motion Motion Compensation Compensation input frame approx. + Frame Frame Buffer Buffer Motion- Motion- Estimation delay Slide 106

107 MPEG-4 At low bitrates: FGS is not efficient Girod: Image Communication II Slide 107

108 MPEG-4 Layer Organization Simulcast Each layer independent Hierarchical Higher layer depends on lower layers 1 Base + N enhancement Layers Typically linear granularity (K bits/layer Simulcast Layering Hierarchical Layering Slide 108

109 SNR Scalability Example MPEG-4 32 kbps 128 kbps 256 kbps Slide 109

110 Spatial Scalability Example MPEG-4 14 kbps 34 kbps 47 kbps Slide 110

111 Synthetical Objects Synthetical visual objects Virtual parts of a scene, e.g. background Can insert animations Synthetical audio objects Text-to-speech Generates speech from given text and prosodic parameters pitch contour, phoneme duration, 200 bit/s to 1.2 Kbit/s Control of face animation Score driven synthesis MPEG-4 Generates music from given scene could generate the sound of a piano, that of falling water, Slide 111

112 Multiplexing Synchronization Layer Generates SL-Packets MPEG-4 Generates header with synchronization/timing information Sequence Number Bitrate Timestamp for decoding/rendering Flexible Multiplex Layer Optional Grouping of streams with similar QoS requirements Transport Multiplex Layer Adaptation to transport-system UDP/IP ATM/AAL5 Slide 112

113 MPEG-4 MPEG-4 elementary streams (e.g. natural/synthetic audio video streams) SL-streams Sync Sync Layer Layer (SL) (SL) Sync Sync Layer Layer (SL) (SL) Sync Sync Layer Layer (SL) (SL) Sync Sync Layer Layer (SL) Sync Sync Layer Layer (SL) (SL) (SL) FlexMux FlexMux Layer Layer (FML) (FML) FlexMux FlexMux Layer Layer (FML) (FML) FlexMux FlexMux Layer Layer (FML) (FML) FlexMux FlexMux Layer Layer (FML) (FML) FML-streams TML-streams TransMux Layer Layer (TML) (TML) TransMux Layer Layer (TML) (TML) TransMux Layer Layer (TML) (TML) Slide 113

114 Transport of MPEG-4 4 streams MPEG-4 is transport agnostic MPEG-4 on MPEG-2 MPEG-4 Amendment MPEG-2 Systems standard, , defines mapping to MPEG-2 TS MPEG-4 over IP See later Slide 114

115 MPEG-4 Delivery Multimedia Integration Framework (DMIF) Delivery Multimedia Integration Framework (DMIF) Framework and Session protocol, similar to ftp ftp returns data DMIF returns pointers to multimedia data Management of multimedia streaming over generic delivery technologies User Commands with ACK Interaction Remote partner, Broadcast systems, Storage systems Remote interactive peers Establishment of channels with different QoS req. and datarates QoS monitoring Controls FlexMux and TransMux layer Manages MPEG-4 sync layer information Slide 115

116 MPEG-4 Error Resilience Tools Error Resilience Tools Built into MPEG-4 as target environment is mobile communication Resync Markers Extended Header Code Data partitioning Reversible VLCs Slide 116

117 Resync Marker MPEG-4 Periodic Resync marker in bitstream determines start of new Video Packet Distinguishable from all possible VLC codewords Video Packet Header contains MB Addr of first MB contained in this packet spatial resynchronization Quant. Param. necessary to decode that first macroblock differential decoding process can be resynchronized Header Extension Code (HEC) additional resynchronization information Slide 117

118 Resync Marker Data recovery MPEG-4 at decoder: data is encoded using Reversible VLC Forward Decoding Backward Decoding Error concealment techniques at receiver still apply e.g. estimate motion vectors, copy blocks from previous frame Slide 118

119 Decoder MPEG-4 Slide 119

120 ITU Standards H.120 H.261 H.263 Slide 120

121 ITU Standards ITU-T Rec. H.120 First video coding standard Not in use today Technical features Conditional replenishment DPCM Scalar quantization VLC Version 2, 1988 Motion compensation Background prediction kbps kbps Slide 121

122 ITU Standards H.261/H.263 H.261 Goal: Real-time Videoconferencing via ISDN (px64 kbps) p= 1,..30. Videophone quality for p=1, p=2 (1 or two ISDN channels). For p=6 medium quality (384 kbps) Total (De-)Compression delay < 150 ms Constant bitrate Not very flexible scheme, only video, no audio ITU Standard 1990 H.263 Goal: Support for low bitrates (e.g. V34 modems) Several enhancements to H.261 More formats (resolution) ITU Standard 1998 Slide 122

123 ITU Standards Format SQCIF Resolution 128x96 Max. framerate (fps) <29.97 Required for H.261 Optional Reguired for H.263 YES QCIF 176x144 <29.97 YES YES CIF 352x288 <29.97 Optional Optional 4CIF 16CIF 704x x1152 <29.97 <29.97 Not defined Not defined Optional Optional Data rate (before comp.) 1.3 Mbps 9 Mbps Avg. data rate (after comp) 26 kbps 64 kbps 36 Mbps 438 Mbps 2900 Mbps 384 kbps 3-6 Mbps Mbps Slide 123

124 Frame Structure Hierarchical Frame CIF or QCIF GOB (Group of Blocks) 1/3 QCIF 3 x 11 Macro Blocks MB (Macro Block) H.261 CIF 4:2:0 subsampling (CCIR 601) 4 Luminance (Y) + 1 Cb + 1 Cr block Luminance block size = 16x16 Chrominance block size = 8x8 Block 8x8 pixel QCIF MB GOB Y 11 3 Cb Cr Slide 124

125 H.261 Frame Types Intra frames (I-Frames) and residual coding 8 x 8 DCT Quantization Entropy coding (RLE, Huffman) At least one I-Frame for every 132 P-frames to avoid error propagation Inter-coded frames (P-Frames) Predictive coding Only transmit blocks that are different DPCM from previous blocks Motion compensation optional +15,.. 15 pixel distance Differential encoding of MV No B-frames! P I Slide 125

126 H.261 External control Coding control Video Signal Source coder Source decoder Video multiplex coder Video multiplex decoder a) Video coder Transmission buffer Receiving buffer Coded bitstream b) Video decoder Slide 126

127 H.261 Coding control Coding control Mode Frame + Memory - MUX DCT Q INTER Entropy Encoding Smooth prediction block filter out high frequency coefficients due to prediction error Motion Compensation Q -1 Q-1 IDCT + Frame Memory Buffer H.261 Encoder Structure Slide 127 Distributed Systems Department Predicted Picture NTSC to to CIF Input Can be switched on MB basis INTRA Loop Filter 0 MUX Motion Estimation MV Error Correction Output

128 H.261 step size Error Correction Buffer VLC Decoder Q-1 Q -1 Input INTER/INTRA Mode IDCT 0 MUX + Decoded Output Loop Filter Motion Compensation Frame Memory H.261 Decoder Structure Slide 128

129 H.261 Hierarchical Stream Structure Slide 129

130 Summary H.261 H.261 International standard for video conferencing (1990) CIF or QCIF Framerate between 7.5 and 30 fps Bit-rate: n x 64 kbps, multiple of ISDN rate, including audio Constant data rate Variable video quality Acceptable quality at 128 kbps Using buffering and adaptive quantization Slide 130

131 H.263 Slide 131

132 Overview H.263 ITU-T Rec. H.263 in February 1998 Video coding for low bit rate communication Extension of H.261 Purpose: video conference at data rates <= 64 kbps ~ 20kbps for video to allow also audio, signalling and data over V.34 Size of GOB restricted to single MB-row 5 different image sizes (sub-qcif, QCIF, CIF, 4CIF, 16CIF) Framerate typically < 10fps Visual quality similar to H.261 at 50% bitrate reduction Same compression core as MPEG-4 Slide 132

133 H.263 New features compared to H.261 Motion Compensation Precision Half-pixel accurate motion estimation as in MPEG-2 Use bi-linear interpolation to calculate half-pixel value Has lowpass filtering as side effect No loop filter compared increases motion prediction and perceived quality: 2 db PSNR improvement, 64 kbps H.261 uses integer precision No Filter in the loop Optimized VLC tables Optimized macroblock adressing H.261 uses MB address to indicate how many MBs skipped H.263 transmits a single bit for each skipped MB Slide 133

134 H.263 H.263 Options for better performance Need to be negotiated with decoder Use external signaling (e.g. ITU-T H.246) Options are: Unrestricted Motion Vector Mode (UMV-mode) Advanced Prediction Mode (AP-mode) PB-frames mode Syntax-based arithmetic coding mode (SAC-mode) Slide 134

135 H.263 Advanced Coding Options Unrestricted Motion Vectors (UMV) Motion Vectors over pictures boundaries helps if motion is near edge When referenced pixel lies outside the edge Use closest edge pixel Extended motion vector range Slide 135

136 H.263 Advanced Coding Options Advanced Prediction Mode (AP-mode) Automatical activation of UMV Mode if AP-mode switched on 4 motion vectors per macro block 1 MV per block But only 1 MV per default 3 MVs used to calculate median for predictor Mvpredicted = median(mv1, MV2, MV3) MV = MVdelta + Mvpredicted only Mvdelta is transmitted MV2 MV3 MV2 MV3 MV1 MV MV1 MV MV2 MV3 MV2 MV3 MV1 MV MV1 MV Slide 136

137 H.263 Advanced Coding Options Overlapped Block Motion Compensation Mode each pixel in an luminance prediction block (8x8 pixel) is a weighted sum of three prediction values, using three MVs (current MV + 2 remote MVs) Reduces artefacts and improves perceived quality due to inherent filtering B. Girod, Image Communication-II Slide 137

138 Advanced Coding Options OBMC Weights H.263 Weighting values for prediction with MV of current MB Weighting values for prediction with MV of top and bottom neighbor MB Weighting values for prediction with MV of left and right neighbor MB Luminance pixel amplitude multiplied by proper weight, Sum of weighted values divided by 8 Slide 138

139 Advanced Coding Options Syntax based arithmetic coding (SAC) Instead of Huffman VLC 0.2 db PSNR improvement, 64 kbps PB-Frames P- and B- frame interleaved Only parts of B-picture are bi-directionally predicted 12 blocks/mb: 6 for P, 6 for B 1 db PSNR improvement, 64 kbps H.263 Forward prediction PB frame P B P Bidirectional prediction Play sequence Transmission sequence Forward only prediction area B-Block Bidirectional prediction area P-MacroBlock Slide 139

140 H % reduction Performance Comparison Slide 140 B.Girod, E.Steinbach, N.Färber, Comparison of the H.263 and H.261 Video Compression Standards, SPIE Proceedings Vol.CR60, Standards and Common Interfaces for Video Information Systems, Oct 1995, Philadelphia

141 H.263 Performance Comparison Slide 141 B.Girod, E.Steinbach, N.Färber, Comparison of the H.263 and H.261 Video Compressio SPIE Proceedings Vol.CR60, Standards and Common Interfaces for Video Information Systems, O n Standards, ct 1995, Philadelphia

142 Performance of PB-Frames mode H.263 Performance of AP-PB-SAC mode Performance Comparison Slide 142 B.Girod, E.Steinbach, N.Färber, Comparison of the H.263 and H.261 Video Compression Standards, SPIE Proceedings Vol.CR60, Standards and Common Interfaces for Video Information Systems, Oct 1995, Philadelphia

143 H.263 Summary Video coding for low bitrate Like H.261, used in tele-conferencing equipment Better performance compared to H.261 Motion estimation at half-pixel resolution Improved motion vectors Advanced coding options But must be negotiated with the decoder via external signaling, e.g. according to ITU-T H.246 A lot more complex than H.261 But can use ASICs, VLSI, Slide 143

144 post H.263 developments post H.263 developments H.26L (long term) or H.264 H.26L (long term) or H.264 Slide 144

145 Post H.263 Developments Extensions to H.263 H.263+ H H.26L renamed to H.264 H.264 = ISO/IEC /MPEG-4 part 10 Applications conversational services Internet 3GPP streaming services Internet 3GPP Other services Digital cinema 3GPP Multimedia messaging Slide 145

146 Post H.263 Common Coder features Hybrid Motion Compensation DPCM with residual and intra DCT Macroblock structure is 16 x 16: 8 x 8 for Y, 16 x 16 for chroma Subsampling 4:2:0 Block based motion estimation Motion vectors over boundaries Variable block size motion Block transform DCT Scalar quantuzation I, P, B frames Slide 146

147 Video in MB + - Motion compensated Prediction DCT Post H.263 INTRA/INTER Coding Coding Control Control Q Q VL-Entropy- Coder Q -1 Q -1 IDCT Coder Mode Qparam Fillgrade Buffer Buffer Out Intra Intra Frame Frame Prediction Prediction Motion Motion Compensation Compensation input frame approx. + Frame Frame Buffer Buffer Prediction- Coder Coder Motion- Motion- Estimation delay Motion Vector Slide 147

148 Video in MB + - Motion compensated Prediction DCT Post H.263 INTRA/INTER Coding Coding Control Control Q Q VL-Entropy- Coder Q -1 Q -1 IDCT Coder Mode Qparam Fillgrade Buffer Buffer Out Intra Intra Frame Frame Prediction Prediction Motion Motion Compensation Compensation Prediction- Coder Coder Motion- Motion- Estimation MB Types input frame approx. 8x8 Types + 16x16 0 Frame Frame 8x8 Buffer Buffer delay 8x4 4x Motion 1 Data Motion vector accuracy 1/4 pel Motion Vector 16x x x x Slide 148

149 Video in MB + - Motion compensated Prediction DCT Post H.263 INTRA/INTER Coding Coding Control Control Q Q VL-Entropy- Coder Q -1 Q -1 IDCT Coder Mode Qparam Fillgrade Buffer Buffer Out Intra Intra Frame Frame Prediction Prediction Motion Motion Compensation Compensation input frame approx. + Frame Frame Buffer Buffer Prediction- Coder Coder Motion- Motion- Estimation delay Multiple Reference Frames Generalized B-frames Motion Vector Slide 149

150 Post H.263 D t =4 D t =3 D t =1 M = 5 Long-Term Memory Prediction: Additional time component d t to block-wise motion vector (d x,d y ) Additional time component d t to block-wise motion vector (d x,d y ) In the example, 5 prior decoded frames as reference In the example, 5 prior decoded frames as reference Reference frames must be available at decoder Slide 150

151 Video in MB + - Motion compensated Prediction INTRA/INTER Post H.263 DCT Q Q -1-1 Intra Intra Frame Frame Prediction Prediction Motion Motion Compensation Compensation Motion- Motion- Estimation input frame approx. Directional Coding Coding spatial Mode Control Qparam (9 Control types for spatial luma, Control prediction (9 types for luma, Data 1 chroma) Fillgrade Out Q A B C D Quant. E VL-Entropy- F G H I a b c Buffer Buffer Transf. d coeffs Coder Coder J e f g h K i j k l Prediction- 7 L m n o p Coder Coder Entropy 2 M Coding 8 N IDCT O P + Output Video Signal e.g., Mode 3: diagonal down/right prediction Frame Frame a, f, k, p are predicted by Buffer Buffer Motion (A + 2Q + I + Data 2) >> 2 mode delay selected at encoding time based on pixel availability and efficiency Motion Vector Slide 151

152 Video in 4x4 4x4 Block Block Integer Integer Transform Motion reduces noise noise at at compensated boundary Prediction eliminates rounding error error Fast Fast implementation + - DCT Post H input frame approx. 2 1 Q -1 Q -1 IDCT + Frame Frame Buffer Buffer delay Motion Vector Mode Qparam Buffer Buffer Slide 152 Distributed Systems Department Y X MB x 00 x 01 x 02 x x 10 x 11 x 12 x 13 = x 20 x 21 x 22 x x 30 x 31 x 32 x 33 Inverse Inverse Transform y 00 y 01 y 02 y y 10 y 11 y 12 y 13 = y 20 y 21 y 22 y 23 University 1 of Ulm, Distributed Systems, Dr. A. 1 Kassler y 30 y 31 y 32 y 33 2 INTRA/INTER Coding Coding Control Control Q Prediction- Coder Coder Intra 1 Intra 1 Frame Frame 1 Prediction Prediction Motion Motion 2 1 Compensation Compensation Motion- Motion Estimation Q VL-Entropy- Coder Coder Fillgrade Out

153 1 Video in + DCT Hadamard transform of of DC DC coeffs for 8x8 Motion coeffs for 8x8 chroma and and 16x16 16x16 Intra Intra luma luma blocks blocks compensated Prediction Used Used for for uniform image image regions, where where DC DC coefficients are are highly highly correlated = D D D D D D D D D D D D D D D D Intra Intra 1 Frame Frame 1 Prediction Prediction Motion Motion 1 1 Compensation Compensation Prediction- Coder Coder Motion- Motion- Estimation Post H // 2 input frame approx. 1 Q -1 Q -1 IDCT + Frame Frame Buffer Buffer delay Motion Vector Mode Qparam Buffer Buffer Slide 153 Distributed Systems Department Y D x x x x MB x x x x x x x x x x x x INTRA/INTER Coding Coding Control Control Q Q VL-Entropy- Coder Coder Fillgrade Out

154 Video in MB Prediction- Coder Coder + DCT - Motion Logarithmic compensated step size control Prediction Smaller step size for for chroma (per H.263 Annex T) T) Extended range of of step Intra Intra sizes Frame Frame Can change to to any step Prediction Prediction size at at macroblock level Motion Motion Quantization reconstruction Compensation Compensation is is one multiply, one add, one Motion- Motionshift Estimation Post H.263 INTRA/INTER Coding Coding Control Control input frame approx. Q Q -1 Q -1 IDCT + Frame Frame Buffer Buffer delay Q VL-Entropy- Coder Motion Vector Coder Mode Qparam Fillgrade Buffer Buffer Slide 154 Out

155 + DCT Additionally, deblocking - filter after DCT Video in MB INTRA/INTER improves visual and objective Motion quality compensated Prediction Smoothes 4 x 4 block edges No No unnecessary blurring due to to content adaptive filtering Intra Intra Frame Frame For Inter-frames, prediction is is based on Prediction Prediction on filtered data! Prediction- Coder Coder Motion Motion- Motion- Compensation Compensation Estimation 3 level filter Post H.263 input frame approx. P2 P1 P0 Q0 Q1 Q2 P2 P1 P0 Q0 Q1 Q2 2 level filter Coding Coding Control Control Q Q -1 Q -1 IDCT + Frame Frame Buffer Buffer delay Q VL-Entropy- Coder Motion Vector Coder Mode Qparam Fillgrade Buffer Buffer Slide 155 Out Distributed Systems Department

156 Post H.263 Deblocking filter Subjective results First decoded intra frame at 0.28 bits/sample No filter Deblocking filter Courtesy H. Schäfer, HHI Slide 156

157 Deblocking filter Subjective results Decoded Inter frame No filter Post H.263 Courtesy H. Schäfer, HHI Deblocking filter Slide 157

158 Post H.263 Video in MB INTRA/INTER Coding Coding Control Control + - Context Adaptive VLC (CAVLC) Motion compensated Prediction DCT Q Q -1-1 Number of of non-zero elements Their Their levels and and signs Total Total number of of 0 before last last non-zero Run Run before each each non-zero Intra Intra Frame Frame Prediction Prediction Context based Adaptive Binary Arithmetic input frame approx. Coding (CABAC) Motion Motion Compensation Compensation IDCT VL-Entropy- Coder Coder Prediction- Coder Coder symbols delay Motion- Motion- Exploits symbol correlations using contexts Estimation Motion Vector + Frame Frame Buffer Buffer Uses Uses adaptive probability models for for most most symbols Mode Qparam Fillgrade Buffer Buffer Out Slide 158

159 H.264 Video Coding Layer Encoder Video Coding Layer Decoder VCL-NAL Interface Network Abstraction Layer Encoder Network Abstraction Layer Decoder NAL Encoder Interface NAL Decoder Interface mapping to H.320 Transport Layer mapping to MPEG-2 Systems mapping to H.324 (M) mapping to RTP/IP mapping to ISO file format Slide 159

160 Post H.263 H.263 combined with Long-Term Memory Prediction reference pictures % PSNR [db] Foreman 10 Hz, QCIF 100 frames encoded TMN Bit-Rate [kbps] Slide 160

161 40 30 Average Bit-Rate Savings in 34 db Post H % Average Tempete Container Silent Foreman Mobile & Calendar Mother & Daughter Stefan Number of Reference Frames Slide 161

162 Video Coding - Standards Video Coding Standards Comparison Slide 162

163 MPEG-1 Summary ISO/IEC (1993) developed by ISO/IEC JTC1 SC29 WG11 (MPEG) Widespread usage, overtaken by MPEG-2 (backward comp.) Superior quality to H.261 at high bit rates > 1mbps for CIF Approx. VHS quality 1,, 2mbps SIF (352 x 240) resolution Additional technical features to H.261 Bi-directional motion prediction Half-pixel motion compensation Slide 163

164 MPEG-2 Summary ISO/IEC , ITU-T H.262 developed (1994) jointly by ITU-T and ISO/IEC SC29 WG11 Widespread use in DVD and HDTV Useful between 4, 30 mbps Additional technical features to H.261 Interlaced picture support scalability The same as MPEG-1 for progressive scan, compatibility required Slide 164

165 Summary H.263 ITU-T Rec. H.263 (version 1 in 1995) The current best standard for video telecommunication Superior to H.261 in all conditions At low bitrates factor 2 better! Version 2: 1997/1998 H.263+ Version 3: 2000 H Slide 165

166 MPEG-4 Summary ISO/IEC Based on H.263 design Adds several features Shape coding Coding of synthetic content Version 2: 2000 Version 3: 2001 Studio Profile: 4:2:2, 4:4:4 subsampling, more MPEG-2 elements Fine Grained Scalability (FGS) Scalable enhancement layer Stop temporal prediction in enhancement layer to prevent temporal error propagation Code enhancement layer by bit-planes progressive transmission Slide 166

167 >H.263 Summary Compression beyond H.263 Real-time low cost Enhanced error and packet loss resilience Bit-rate adaptivity Gain goal over 1999: 50%saving in bits for same quality More work on prediction Will lead to most performance improvements Slide 167

168 PSNR [db] 38 H Video Coding Efficiency MPEG Variable block size motion compensation (H ) frame difference coding, H Integer-pel motion Compensation, H Intraframe DCT coding (DCT 1974, JPEG 1992) Factor 4.5 Foreman 10 Hz, QCIF 133 frames encoded Bit-Rate [kbps] Slide 168

169 Comparison of MPEG-2 2 and H.264/AVC CIF, 30Hz : 512 kbit/s CIF, 30Hz : 340 & 1024 kbit/s Mit freundlicher Genehmigung von H. Schäfer, HHI Slide 169

170 Comparison of MPEG-2 2 and H.264/AVC 512 kbit/s 512 kbit/s CIF, 30Hz : 512 kbit/s CIF, 30Hz : 340 & 1024 kbit/s Mit freundlicher Genehmigung von H. Schäfer, HHI Slide 170

171 Comparison of MPEG-2 2 and H.264/AVC 512 kbit/s 512 kbit/s CIF, 30Hz : 512 kbit/s CIF, 30Hz : 340 & 1024 kbit/s Mit freundlicher Genehmigung von H. Schäfer, HHI Slide 171

172 Comparison of MPEG-2 2 and H.264/AVC 512 kbit/s 512 kbit/s CIF, 30Hz : 512 kbit/s CIF, 30Hz : 340 & 1024 kbit/s Mit freundlicher Genehmigung von H. Schäfer, HHI Slide 172

173 Comparison of MPEG-2 2 and H.264/AVC 512 kbit/s 512 kbit/s CIF, 30Hz : 512 kbit/s CIF, 30Hz : 340 & 1024 kbit/s Mit freundlicher Genehmigung von H. Schäfer, HHI Slide 173

174 Comparison of MPEG-2 2 and H.264/AVC 512 kbit/s 512 kbit/s CIF, 30Hz : 512 kbit/s CIF, 30Hz : 340 & 1024 kbit/s Mit freundlicher Genehmigung von H. Schäfer, HHI Slide 174

175 Comparison of MPEG-2 2 and H.264/AVC 512 kbit/s 512 kbit/s CIF, 30Hz : 512 kbit/s CIF, 30Hz : 340 & 1024 kbit/s Mit freundlicher Genehmigung von H. Schäfer, HHI Slide 175

176 Comparison of MPEG-2 2 and H.264/AVC 340 kbit/s 1024 kbit/s CIF, 30Hz : 512 kbit/s CIF, 30Hz : 340 & 1024 kbit/s Mit freundlicher Genehmigung von H. Schäfer, HHI Slide 176

177 Comparison of MPEG-2 2 and H.264/AVC 340 kbit/s 1024 kbit/s CIF, 30Hz : 512 kbit/s CIF, 30Hz : 340 & 1024 kbit/s Mit freundlicher Genehmigung von H. Schäfer, HHI Slide 177

178 Comparison of MPEG-2 2 and H.264/AVC 340 kbit/s 1024 kbit/s CIF, 30Hz : 512 kbit/s CIF, 30Hz : 340 & 1024 kbit/s Mit freundlicher Genehmigung von H. Schäfer, HHI Slide 178

179 Comparison of MPEG-2 2 and H.264/AVC 340 kbit/s 1024 kbit/s CIF, 30Hz : 512 kbit/s CIF, 30Hz : 340 & 1024 kbit/s Mit freundlicher Genehmigung von H. Schäfer, HHI Slide 179

180 Comparison of MPEG-2 2 and H.264/AVC 340 kbit/s 1024 kbit/s CIF, 30Hz : 512 kbit/s CIF, 30Hz : 340 & 1024 kbit/s Mit freundlicher Genehmigung von H. Schäfer, HHI Slide 180

181 Comparison of MPEG-2 2 and H.264/AVC 340 kbit/s 1024 kbit/s CIF, 30Hz : 512 kbit/s CIF, 30Hz : 340 & 1024 kbit/s Mit freundlicher Genehmigung von H. Schäfer, HHI Slide 181

182 Video Traffic Models Slide 182

183 Traffic Models for Video Video Traffic Models Necessary to calculate bandwidth requirements Constant Bitrate CBR Variable Bitrate VBR Multiplexing gain: factor 2,..6 Different time scales Buffer occupancy Quality, size, f/s Codec Buffer Short: Packets within slice or frame Medium: Per GOP (IBP pattern) Long: Per scene (few seconds) higher variability Look similar at all time scales, long-term correlation, heavy-tailed highly variable stochastic process For short queues, long term correlation does not matter Slide 183

184 Video Traffic Models MPEG-4 frame size traces MPEG-4 and H.263 Video Traces for Network Performance Evaluation Frank Fitzek, Martin Reisslein, Slide 184

185 Video Traffic Models MPEG-4 frame size traces MPEG-4 and H.263 Video Traces for Network Performance Evaluation Frank Fitzek, Martin Reisslein, Slide 185

186 Video Traffic Models MPEG-4 frame size traces MPEG-4 and H.263 Video Traces for Network Performance Evaluation Frank Fitzek, Martin Reisslein, Slide 186

187 Video Traffic Models MPEG-4 frame size histograms MPEG-4 and H.263 Video Traces for Network Performance Evaluation Frank Fitzek, Martin Reisslein, Slide 187

188 Video Traffic Models MPEG-4 frame size histograms MPEG-4 and H.263 Video Traces for Network Performance Evaluation Frank Fitzek, Martin Reisslein, Slide 188

189 Video Traffic Traces MPEG-4 frame size histograms MPEG-4 and H.263 Video Traces for Network Performance Evaluation Frank Fitzek, Martin Reisslein, Slide 189

190 Video Traffic Traces H.263 frame size traces MPEG-4 and H.263 Video Traces for Network Performance Evaluation Frank Fitzek, Martin Reisslein, Slide 190

191 Video Traffic Traces H.263 frame size traces MPEG-4 and H.263 Video Traces for Network Performance Evaluation Frank Fitzek, Martin Reisslein, Slide 191

192 Video Traffic Traces H.263 frame size traces MPEG-4 and H.263 Video Traces for Network Performance Evaluation Frank Fitzek, Martin Reisslein, Slide 192