New Techniques for Improved Video Coding Thomas Wiegand Fraunhofer Institute for Telecommunications Heinrich Hertz Institute Berlin, Germany wiegand@hhi.de
Outline Inter-frame Encoder Optimization Texture Analysis and Synthesis Model-Aided Coding T. Wiegand: New Techniques for Improved Video Coding 1
Scope of Video Coding Standardization Only Restrictions on the Bitstream, Syntax, and Decoder are standardized: Permits optimization beyond the obvious Permits complexity reduction for implementability Provides no guarantees of quality Source Pre-Processing Encoding Sink Post-Processing & Error Recovery Decoding Scope of Standard T. Wiegand: New Techniques for Improved Video Coding 2
Hybrid Video Encoder Input Video Signal Split into Macroblocks 16x16 pixels - Decoder Coder Control Transform/ Scal./Quant. Scaling & Inv. Transform Control Data Quant. Transf. coeffs Entropy Coding Intra/Inter Intra-frame Prediction Motion- Compensation De-blocking Filter Output Video Signal Motion Estimation Motion Data T. Wiegand: New Techniques for Improved Video Coding 3
Inter-frame Optimization of Video Encoding Most video encoders select locally optimal transform coefficients as they ignore temporal dependencies Time Motion Compensation New approach that takes inter-frame dependencies into account T. Wiegand: New Techniques for Improved Video Coding 4
Linear Signal Model for Decoder motion data transform coefficients s = Ms+ Tc+ p reconstructed sample values inverse transform & scaling static predictor Very large size of matrices Numbers when optimizing 4 QCIF pictures jointly: M, T: 100,000 x 100,000 entries T. Wiegand: New Techniques for Improved Video Coding 5
Quadratic Program Use Mean Squared Error distortion Select transform coefficient levels by solving: original video signal minimize subject to: i (s v) T (s v) + λ abs(c i ) s = Ms+ Tc+ p Equivalent to a quadratic program Can be solved by standard solution methods! T. Wiegand: New Techniques for Improved Video Coding 6
Results Used H.264/MPEG4-AVC Compare looking at 2 or 3 frames at a time Compare to H.264/MPEG4-AVC Test Model Constant QP Motion estimation on original frames Only optimized inter-frames T. Wiegand: New Techniques for Improved Video Coding 7
FlowerGarden 30Hz IPPP.. 39 38 37 36 35 34 33 32 31 30 29 Average Y PSNR [db] TM2 2 Frames Jointly 3 Frames Jointly 0 100 200 300 400 500 Bit-Rate [kbits/sec] 16% Bit-rate improvement T. Wiegand: New Techniques for Improved Video Coding 8
FlowerGarden 30Hz IbPbPbP.. 37 Average Y PSNR [db] 36 35 34 33 32 31 TM2 3 Frames Jointly 30 0 50 100 150 200 250 300 Bit-Rate [kbits/sec] 15% Bit-rate improvement T. Wiegand: New Techniques for Improved Video Coding 9
Summarizing Inter-frame Optimization Very complex: has only recently become possible Problems with H.264/MPEG4-AVC: rounding Adjust design to support efficient encoding methods Heavy incorporation of encoding methods has been used in H.264/MPEG4-AVC development with Lagrangian encoder optimization Do not standardize the encoder! T. Wiegand: New Techniques for Improved Video Coding 10
Texture Analysis and Synthesis Textures with large amount of visible details are difficult to code e.g., grass, sand, clouds, water... Viewer often does not perceive large differences between different versions of grass, sand, clouds, water... Where is the original texture? T. Wiegand: New Techniques for Improved Video Coding 11
Video Analysis and Synthesis Approach: Exact reproduction of details is irrelevant if Textures are shown with limited spatial accuracy Viewer does not know the original video Consequence: Mean Squared Error (MSE) distortion criterion is not suitable for efficient coding of detail-irrelevant textures Issue: Could we use similarity criteria (color histograms, etc) instead of MSE as coding distortion? Scheme: If information needed for approximate reproduction of detail-irrelevant textures requires less bit-rate than using MSE Ł bit-rate savings T. Wiegand: New Techniques for Improved Video Coding 12
Video Codec Architecture Video In Encoder Bits Decoder Video Out TA Side Info TS Video coding using a texture analyzer (TA) and a texture synthesizer (TS) T. Wiegand: New Techniques for Improved Video Coding 13
Texture Analyzer Principle Ref. mask Ref. frame Split & Merge Segmentation Motion Compensation Texture Analyzer Current mask Current frame T. Wiegand: New Techniques for Improved Video Coding 14
Texture Synthesizer Stuffing procedure T. Wiegand: New Techniques for Improved Video Coding 15
Texture Synthesizer Match surrounding samples between current picture and warped pictures to find synthesis sample for the current sample Decoder processing required! Volume matching for water and temporal consistence T. Wiegand: New Techniques for Improved Video Coding 16
Flowergarden, CIF, 30 Hz, QP=16, 41 frames Bit-rate savings: 19.4% T. Wiegand: New Techniques for Improved Video Coding 17
Canoe, CIF, 30 Hz, QP=16, 73 frames Bit-rate savings: 8.8 % T. Wiegand: New Techniques for Improved Video Coding 18
Summarizing Texture Analysis and Synthesis Textures with large amount of visible details are difficult to code e.g., grass, sand, clouds, water... Errors in these textures are barely visible Texture analysis and synthesis can help to represent these textures efficiently Method requires processing at the decoder side Other methods require processing at the decoder side to achieve coding efficiency improvements through backward adaptation (CABAC) T. Wiegand: New Techniques for Improved Video Coding 19
Model-Aided Coding Model-based coding (MBC): Extreme low bit-rates possible Missing generality Hybrid video coding (H.263/4, MPEG-2/4,...) Robust against scene changes Higher bit-rates Combination of MBC with hybrid video coding Model-aided codec High coding efficiency No restriction to scene content T. Wiegand: New Techniques for Improved Video Coding 20
Architecture of the Model-Aided Codec Coder- Control control data video signal S [ x,y,t ] - e Intraframe DCT Coder Intraframe Decoder DCTcoefficients Modelbased Coder Motion Compensation Reconst. Frame motion data FAPs Model-based Decoder Model Frame Decoder T. Wiegand: New Techniques for Improved Video Coding 21
Coding Results Sequenz Clapper Board: 8.33 Hz, CIF resolution, 260 frames Same average bit-rate ( 12 kbit/s) H.263 annexes D, F, I, J, T H.263 (TMN-10) Model-Aided Codec (MAC) T. Wiegand: New Techniques for Improved Video Coding 22
Reconstruction Quality vs. Bit-rate 45 % 3.5 db Gain of 3.5 db PSNR compared to H.263 Coder (TMN-10) Bit-rate reduction 45 % T. Wiegand: New Techniques for Improved Video Coding 23
Model-Aided Coding with Approximate Geometry H.264 (TML-8) MAC T. Wiegand: New Techniques for Improved Video Coding 24
Summarizing Model-Aided Coding Waveform-coding and 3-D model-based coding can be combined to efficiently exploit statistical dependencies in the video signal The trick is to generate a model/representation of the video signal and to animate the central representation to predict the video signal: 3D computer graphic models Mosaics MCTF T. Wiegand: New Techniques for Improved Video Coding 25
Concluding Remarks Three Techniques are presented that improve against existing H.264/MPEG4-AVC coding 1. Inter-frame encoder optimization Exploit dependencies over multiple pictures Incorporate/anticipate efficient encoding into decoder standardization process (do not standardize encoder) 2. Video analysis and synthesis (VAS) Consider subjective methods in video compression (investigate alternative distortion measures) Allow more processing at the decoder side for backward adaptive coding 3. Model-aided coding Further enhance exploitation of statistical depenedencies Incorporate more semantic ideas combination with VAS T. Wiegand: New Techniques for Improved Video Coding 26
Peter Eisert Bernd Girod Acknowledgements Patrick Ndjiki-Nya Brad Schumitsch Heiko Schwarz Aljoscha Smolic T. Wiegand: New Techniques for Improved Video Coding 27