Spring 2017: Multimedia Communication Lec 10 Video Coding Standard and System - HEVC Zhu Li Course Web: http://l.web.umkc.edu/lizhu/ Z. Li Multimedia Communciation, Spring 2017 p.1
Outline Lecture 09 Video Signal Processing ReCap Course Project Topic: Light Field Compression Video Coding Standard and Systems HEVC History HEVC Performance HEVC Key Coding Features Z. Li Multimedia Communciation, Spring 2017 p.2
Coding Structure Video Signal Processing I P P P P P P GoP Fast Block Motion Estimation: 1 2 3 4 5 6 7 I B B P B B P GoP Z. Li Multimedia Communciation, Spring 2017 p.3
MV coding modes in HEVC 3 MV coding modes key ideas, predicting from current block s spatio-temporal neighbors new merge mode: saving MV coding bits by combine small blocks into larger blocks. Z. Li Multimedia Communciation, Spring 2017 p.4
Intra Prediction in HEVC Much more modes DC mode: copy DC values from neighbor Planar mode: top row or left col average Angular: pixels on certain line Ref: Jani Lainema, Frank Bossen, Woojin Han, Junghye Min, Kemal Ugur, Intra Like a sparse transform basis! Coding of the HEVC Standard. IEEE Trans. Circuits Syst. Video Tech. 22(12): 1792-1801 (2012) Z. Li Multimedia Communciation, Spring 2017 p.5
Deblocking Filter Reduce blocking artifact in the reconstructed frames Can improve both subjective and objective quality Filter in H.261: [1/4, 1/2, 1/4]: Applied to non-block-boundary pixels in each block. A low-pass smoothing filter. In H.264 (and H.263v2), this is used in the prediction loop to improve motion estimation accuracy. Decoder needs to do the same. Also called loop filter. H.264: 4x4 block level Before. and After H.265: 8x4 block level Z. Li Multimedia Communciation, Spring 2017 p.6
Sample Adaptive Offset (SAO) Filter Classify pixels on block edge as one of the four categories Offset its pixel value accordingly Z. Li Multimedia Communciation, Spring 2017 p.7
Scalable Video Coding Scalable video coding is desired in heterogeneous networks: Types of scalability: Z. Li Multimedia Communciation, Spring 2017 p.8
HW-3: Motion Estimation Fractional Pel Motion Compensation Use bi-linear interpolation to scale up the frames 2x2 Do ½ pel motion estimation and compensation Compare block size and pel accuracy effects on the residual MSE B-Frames Implement a special version of B-frames, have forward and backward motion vectors Compare the results with P frame motion estimation Fast Motion Estimation Implement Diamond search pattern, compare with the exhaustive search Data set: foreman sequence frames 150, 151, and 152. Z. Li Multimedia Communciation, Spring 2017 p.9
Outline Lecture 09 Video Signal Processing ReCap Course Project Topic: Light Field Compression Video Coding Standard and Systems HEVC History HEVC Performance HEVC Key Coding Features Z. Li Multimedia Communciation, Spring 2017 p.10
What is Light Field? Capturing all lights in an observing volume Z. Li Multimedia Communciation, Spring 2017 p.11
Light Field Acquisition and Format Lytro System Images were acquired using a Lytro Illum camera LFR (light field raw) file format as defined by Lytro contains the raw image data: Output of camera sensor, overlaid with GRBG Bayer filter; 7728 5368 samples - 10-bit little endian integer values; A demosaicing process is required to obtain the color information from the sensor data; Demosaicing: raw data RGB image; From Prof. Fernando Pereira s San Diego JPEG Presentation Z. Li Multimedia Communciation, Spring 2017 p.12
View Extraction Processes Compression Code Point: Light fields were extracted from the LFR files using the Light Field Toolbox v0.4 made available by Donald Dansereau, QUT, AU; http://lightfield-forum.com/tag/lightfield-toolbox/ The process of obtaining a specific view from the full light field involves two main stages: Coding Multi-perspectives Multi-focus From Prof. Fernando Pereira s San Diego JPEG Presentation Z. Li Multimedia Communciation, Spring 2017 p.13
Lenslet (sensor) vs Sub-aperture (synthesis) Images Lenslet sensor capture Sub-aperture image Z. Li Multimedia Communciation, Spring 2017 p.14
Light Field Data Set EPFL data set (6.6GB) : http://mmspg.epfl.ch/epfl-light-fieldimage-dataset a) Bikes b) Desktop c) Flowers d) Friends 1 e) Magnets 1 f) Stone Pillars Outside Z. Li Multimedia Communciation, Spring 2017 p.15
Light Field Compression Test View Point Compression Pre-selected 9 perspective view point in the light field Z. Li Multimedia Communciation, Spring 2017 p.16
Naïve Solution Direct Coding of Lenslet R-D curves: Frenando Pereira/IST results: JPEG JPEG 2000 H.264/AVC Intra HEVC Intra Z. Li Multimedia Communciation, Spring 2017 p.17
Naïve Solution Codec Performances JPEG, JPEG2K, AVC Intra, HEVC Intra Fernando Pereira, at JPEG meeting in San Diego, 2016.02 Bikes Desktop Flowers Z. Li Multimedia Communciation, Spring 2017 p.18
Re-Focusing Point Performance Re-Focusing Operating Points Z. Li Multimedia Communciation, Spring 2017 p.19
Re-Focusing Point R-D curves Also across different codec, naïve solution (Fernando Pereira, IST) JPEG JPEG 2000 H.264/AVC Intra HEVC Intra Z. Li Multimedia Communciation, Spring 2017 p.20
LF Coding Discussions Redundancy in the lenslet images Across lenslet prediction? o Optical correction? o Affine block matching? R-D optimization for certain view perspective? Across focal lens R-D optimization? Z. Li Multimedia Communciation, Spring 2017 p.21
Outline Lecture 09 Video Signal Processing ReCap Course Project Topic: Light Field Compression Video Coding Standard and Systems HEVC History HEVC Performance HEVC Key Coding Features Z. Li Multimedia Communciation, Spring 2017 p.22
HEVC/H.265 History HEVC Objective: Credit: Vivienne Sze & Madhukar Budagavi, ISCAS 2014 Tutorial Z. Li Multimedia Communciation, Spring 2017 p.23
New Design Features HEVC (H.265) vs AVC (H.264) Credit: Vivienne Sze & Madhukar Budagavi, ISCAS 2014 Tutorial Z. Li Multimedia Communciation, Spring 2017 p.24
HEVC Development History Long hard fight, many casualties Z. Li Multimedia Communciation, Spring 2017 p.25
HEVC Performance PSNR HEVC vs AVC Z. Li Multimedia Communciation, Spring 2017 p.26
HEVC Performance Subjective Score HEVC vs AVC Z. Li Multimedia Communciation, Spring 2017 p.27
HEVC Coding Pipeline HEVC Diagram Credit: Prof. Oscar Au, Tutorial on HEVC Z. Li Multimedia Communciation, Spring 2017 p.28
HEVC Coding Structure Quad Tree Decomposition: Slide Credit: Vivienne Sze & Madhukar Budagavi, ISCAS 2014 Tutorial Ref: G. Schuster, PhD Thesis, 1996: Optimal Allocation of Bits Among Motion, Segmentation and Residual Z. Li Multimedia Communciation, Spring 2017 p.29
HEVC Coding Structure CTU- Coding Tree Unit Upto 64x64 pel block Macro Block equivalent Aka LCU: Largest Coding Unit CU Coding Unit Square NxN pel Like Block PU Prediction Unit Inter Prediction Intra Prediction Z. Li Multimedia Communciation, Spring 2017 p.30
Prediction Unit PU: Basic Unit for Transform & Quantization Z. Li Multimedia Communciation, Spring 2017 p.31
Transform Unit (TU) TU Square blocks for transform & quantization/coding, size: 4x4, 8x8, 16x16, 32x32 MaxTU size: 32 for luma, 16 for chroma Min TU size: 4x4 for both luma and chroma TU size can be larger than PU for inter PU case (MV merge) Z. Li Multimedia Communciation, Spring 2017 p.32
HEVC Transforms Transform + Quant: Z. Li Multimedia Communciation, Spring 2017 p.33
HEVC Intra-Prediction Intra-Prediction Modes Z. Li Multimedia Communciation, Spring 2017 p.34
Intra-Predicted Basis As if it is a 1-non zero coefficient transform Ref: J. Laniema and W.-J. Han, Intra Picture Prediction in HEVC, Chapter in, Springer-Velag Book on High Efficiency Video Coding (HEVC): Algorithms and Architectures, Springer, 2014. Ed. V. Sze et. Al. Z. Li Multimedia Communciation, Spring 2017 p.35
Pre-Processing for Intra Prediction Smoothing filtering before intra prediction Z. Li Multimedia Communciation, Spring 2017 p.36
Post-Processing for Intra Prediction Remove discontinuity: Z. Li Multimedia Communciation, Spring 2017 p.37
HEVC Intra as Still Image Coding Standard Advancing the state or art by quite a bit Z. Li Multimedia Communciation, Spring 2017 p.38
HEVC Inter Prediction Supports 4x4 block ¼ pel accuracy Motion Compensation Z. Li Multimedia Communciation, Spring 2017 p.39
Interpolation Filters Quite a bit of memory and computing cost: Z. Li Multimedia Communciation, Spring 2017 p.40
Fast Motion Estimation in HEVC HM Diamond Search Z. Li Multimedia Communciation, Spring 2017 p.41
HEVC Entropy Coding Binary Arithmetic Coding: Z. Li Multimedia Communciation, Spring 2017 p.42
HEVC CABAC Throughput Improvement Context Bypassing Mode: Z. Li Multimedia Communciation, Spring 2017 p.43
Parallel Processing Tools: Slice/Tile Credit: Vivienne Sze & Madhukar Budagavi, ISCAS 2014 Tutorial Z. Li Multimedia Communciation, Spring 2017 p.44
Profiles, Levels and Tiers Operating HEVC in Profiles/Levels/Tiers Z. Li Multimedia Communciation, Spring 2017 p.45
HEVC Resources Main Spec: http://www.itu.int/itu-t/recommendabons/rec.aspx?rec=11885 T-CSVT Special Issue: 2012: Combined Issue on HEVC Standard and Research: http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6403920 2016: Special Issue on HEVC Extensions and Efficient HEVC Implementations: http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=7372356 Springer Book: V. Sze, M. Budagavi, G. J. Sullivan (Editors), High Efficiency Video Coding (HEVC): Algorithms and Architectures, Springer, 2014, http://www.springer.com/engineering/signals/book/ 978-3-319-06894-7 HM (open source software): https://hevc.hhi.fraunhofer.de/svn/svn_hevcsovware/ HARP: A python HM access tool: http://www.lms.lnt.de/harp/ Z. Li Multimedia Communciation, Spring 2017 p.46
Summary Light Field Compression Sensor data at lenslet Support new perspective and re-focusing Interesting R-D optimization issue at new perspective and focusing point in the light field Calls for new coding tools, based on HEVC intra, e.g, HEVC Crown Jewel of 20+ years of research and engineering Significant performance gains over the previous state of art Worthwhile to master this tool for a variety of research and biz opportunities. Z. Li Multimedia Communciation, Spring 2017 p.47