High Efficiency Video Coding (HEVC)

High Efficiency Video Coding (HEVC) 1

The MPEG Vision 2 Three years ago in 2009, it was expected -- Ultra-HD (e.g., 4kx2k) video will emerge -- Mobile HD applications will become popular -- Video bitrate using current technology will go up faster than the network infrastructure Now, we see

London 2012 Olympic Games 3 Immersive experience in Super Hi-Vision 16 times the resolution of Full HD (1080p) Pioneered by NHK & BBC 7680x4320

Go beyond Full HD 4 DVD (720x480) SDTV (1280x720) HDTV (1920x1080) Digital Cinema 2K (2048x1080) VIDEO RESOLUTION COMPARISON FROM 480P TO 4320P Digital Cinema 4K (4096x2160) Super Hi-Vision / Ultra HD (7680x4320)

High Efficiency Video Coding (HEVC) 5 The latest draft video coding standard developed by a joint team of experts from ISO/IEC MPEG and ITU-T VCEG Goal: substantially better performance than the H.264/AVC standard, especially in coding HD and Ultra- HD video International Standard (IS) in April 2013

Timeline 6 2005-: 2010/01: 2010/04: 2010/10: 2013/01: 2013/04: MPEG/VCEG exploration activities Joint Collaborative Team (JCT-VC) Final Call-for-Proposals (CfP) 1 st JCT-VC Meeting (Dresden, Germany) 27 proposals received Working Draft 1.0 (WD1.0) and HM-1.0 Final Draft International Standard (FDIS) International Standard (IS)

HEVC Version 1 7 Press letter of 103 rd Geneva Meeting (N13253) The next major milestone in MPEG video history is achieved ISO/IEC JTC1/SC29/WG11 MPEG is proud to announce the completion of the new High Efficiency Video Coding (HEVC) standard which has been promoted to Final Draft International Standard (FDIS) status at the 103rd MPEG meeting.

Subjective Assessment 8 HM 5.0 JM 18.2 Over -50% MOS Class B (HD): -67% Class C (SD): -49% Bit Rate [Kbps] J. R. Ohm, G. J. Sullivan, F. Bossen, T. Wiegand, V. Baroncini, M. Wien, and J. Xu, JCT-VC AHG report: HM subjective quality investigation (AHG22), JCTVC-H0022, San José, CA, Feb., 2012.

New milestone since year 2003 9 42 Park Scene, 1920x1080, 24Hz H.264/AVC YUV-PSNR (db) 41 40 39 38 37 36 35 34 33 32 HEVC 35.4% H.263 MPEG-2 MPEG-4 H.264/MPEG-2 (MP) MPEG-4 (ASP) H.263 (HLP) H.264/MPEG-4 AVC (HP) HEVC (MP) 31 0 2 4 6 8 10 12 14 Bitrate (Mbps) J.-R. Ohm, G. J. Sullivan, H. Schwarz, T. K. Tan, and T. Wiegand, Comparison of the Coding Efficiency of Video Coding Standards Including High Efficiency Video Coding (HEVC), IEEE Trans. CSVT, Dec., 2012

HEVC vs. AVC (1/2) 10 HM10.0 (Main) vs. JM18.4 (High) Test Condition ALL INTRA 23% RANDOM ACCESS LOW DELAY 33.5% 36.4% 0 10 20 30 40 50 BD-rate Saving (%) B. Li, G. J. Sullivan, and J. Xu, Comparison of Compression Performance of HEVC Draft 10 with AVC High Profile, JCTVC-M0329, Incheon, April, 2013.

HEVC vs. AVC (2/2) 11 Class Resolution Y BD-Rate (%) All Intra Radom Access Low Delay A 2500x1600-23.6-36.6 B 1080p -22.7-39.8-42.1 C 480p -19.7-30.3-32.7 D 240p -16.4-28.0-29.9 E 720p -28.8-44.1 F 480p, 720p -28.6-31.2-33.8 (Minus sign means coding gain)

Encoding Time 12 HM10.0 (Main) vs. JM18.4 (High) Test Condition ALL INTRA RANDOM ACCESS LOW DELAY 71% 97% 109% 20% 40% 60% 80% 100% 120% Encoding Time Ratio (%) B. Li, G. J. Sullivan, and J. Xu, Comparison of Compression Performance of HEVC Draft 10 with AVC High Profile, JCTVC-M0329, Incheon, April, 2013.

Decoding Time 13 HM10.0 (Main) vs. JM18.4 (High) Test Condition ALL INTRA 42% RANDOM ACCESS LOW DELAY 56% 107% 20% 40% 60% 80% 100% 120% Decoding Time Ratio (%) B. Li, G. J. Sullivan, and J. Xu, Comparison of Compression Performance of HEVC Draft 10 with AVC High Profile, JCTVC-M0329, Incheon, April, 2013.

Multi-thread Decoding 14 Frame Rate (fps) 90 80 70 60 50 40 30 20 10 0 HM-9.0 (RA-Main) 3840x2160(12Mbps) 1080p 720p 60fps 30fps 0 0.5K 1K 1.5K 2K 2.5K Vertical Resolution T. Tan, Y. Suzuki, and F. Bossen, On software complexity: decoding 4K60p content on a laptop, JCTVC- L0098, Geneva, CH, Jan., 2013.

Few weeks later 15 Display 4.99 inch, 1920x1080 Camera 13 Mega Pixels Video Codec 1080p@30fps MPEG4, H.264, H.263, DivX, VC-1, VP8, WMV7/8, Sorenson Spark, HEVC Samsung UNPACKED 2013 - http://www.youtube.com/watch?v=yaw6csapnfk

HEVC TOOL FEATURES 16

HEVC Tool Features CABAC Tiles Wavefront 17 Current Frame Asymmetric Motion Partitioning Merged Skip / Motion Merging Advanced MV Prediction DCT-based Interpolation Filter Frame Buffer Inter Prediction + Inter - DCT Q Entropy Coding Bitstream Residual Quad-tree Trans. Transform Skipping Adaptive Coeff. Scanning Deblocking Filter Sample Adaptive Offset Intra Prediction Intra In-loop filter + More Directions Pre-/Post-filtering Direct Chroma + IQ IDCT

Coding Unit (CU) 18 Basic unit for coding, conceptually similar to macroblock but now can be of variable size H.264/AVC 16 64 HEVC 16 MB MB 64 CU CU CU CU CU CU CU CU CTU 0 CTU 1

Prediction Unit (PU) 19 16x16 MB PU PU PU PU PU PU PU PU PU PU PU PU PU PU PU PU PU CU (Only for SCU) Asymmetric Sub-MB Partition

Transform Unit (TU) Residual Quad-tree Transform (RQT) Transform can cross PU boundaries 16x16 MB TU TU 8 TU TU 8 4 TU TU TU TU 4 TU TU TU TU TU TU TU TU TU TU TU TU 20 PU Aligned May be skipped if 4x4 CU RQT TU TU TU TU TU TU TU TU TU TU TU TU

Intra Prediction 21 More directions (up to 33) Adaptive pre-filtering of reference pixels Boundary smoothing for DC/Ver./Hor. modes Direct mode for Chroma Reference Sample Mean 0: DC LB LB 1: Planar TR TR 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 Horizontal Vertical

DCT-based Interpolation Filter 22 P -3 P -2 P -1 P 0 P a P 1 P 2 P 3 P 4 M Integer-pels Spatial Domain Samples {P -3, P -2,, P 4 } M Integer-pels Fraction-pel Forward DCT Inverse DCT DCT Domain Coefficients {C -3, C -2,, C 4 } p a 2M 1 C π k( 2M 1+ 2a) 0 = + Ck cos 2 k = 1 4M DCT Coefficients

Motion Merging (1/2) 23 Optional 5 Candidates at most

Motion Merging (2/2) 24 Before Merging After Merging mv0 mv1 mv0 mv1 mv4 mv5 mv4 mv5 mv2 mv3 mv2 mv3 Current PU 0 mv5 mv6 mv7 mv6 mv7 Current PU1 mv7 Merge w/ Nearby Motion Irregular Motion Partitions

Temporal Merge Candidate Derivation 25 2 3 Co-located Current 1 L0 L0 Ref0 Current L1 Ref0 L1

Advanced Motion Vector Prediction 2 Find First Available 26 B2 B1 B0 1 T1 Optional 3 2 Candidates at most Find First Available A1 A0 Current PU T0 If LCU boundary, exclude A0 & T0

Transform Skipping 27 Operable in 4x4 TUs Switchable with DST (intra) / DCT (inter) TS Disabled TS Enabled (QP37, 608.4Kbps, 34.8dB) (QP36, 600.8Kbps, 36.1dB)

Significance Map Scanning 28 TU Size Prediction Type Scanning Order 4x4, 8x8 Intra (Vertical, Horizontal) Vertical, Horizontal All Intra, Inter 4x4 Sub-diagonal Vertical Horizontal 4x4 Sub-diagonal 4 4

Sample Adaptive Offset (1/3) 29 Band Offset (BO): intensity-based sample classification Edge Offset (EO): edge-based sample classification [BO] 0 4 Bands MAX [EO] EO Off Off EO C C BO BO BO EO BO BO BO EO C C Off EO EO EO LCUs in a Frame

Sample Adaptive Offset (2/3) 30 I. Positive Edge Offset Pixel Level x-1 x x+1 x-1 x x+1 x-1 x x+1 Pixel Index II. Negative Edge Offset Pixel Level x-1 x x+1 x-1 x x+1 Pixel Index x-1 x x+1

Sample Adaptive Offset (3/3) 31 SAO Disabled SAO Enabled (Frame15, 32.6dB, 3.0Mbps) (Frame15, 32.7dB, 3.1Mbps)

Parallel-friendly Design 32 Transcoding to WPP Sequential Bitstream Chunks re-ordering Chunk encoding using 16 cores Live Content Chunk re-ordering 2 entry points G. Clare, F. Henry, and S. Pateux, Wavefront and Cabac Flush: Different Degrees of Parallelism Without Transcoding, JCTVC-F275, Torino, IT, July, 2011.

Wavefront Parallel Processing (WPP) 33 Pass CABAC states to next Wave Parsing Syntax Elements Decompressing Decoded Pixels [n Waves in a Slice] Dependent Parsing Dependent Syntax Elements Dependent Decompressing Dependent Decoded Pixels Flush CABAC States Parallel Parallel

Tiles 34 Syntax Elements Decoded Pixels Parsing Decompressing Parsing Syntax Elements Decompressing Decoded Pixels [4 Tiles in a Slice/Frame] Parallel Parallel Prediction can t cross Tiles

Parallel Merge Group 35 Group m-1,n-1 Group m,n-1 Group m+1,n-1 Above Inferable Group m-1,n CUs/PUs within Group m,n Left Inferable PUs/CUs within a group run in parallel

Profiles & Tool Summary 36 Tool \ Profile Main Still Picture Main Main 10 Bit Depth 8 8 8, 9, 10 CU Size 16x16~64x64 PU Partition Symmetric Symmetric, Asymmetric TU Partition Residual Quad-tree Transform MV Prediction - AMVP, MRG, MRG-Skip Interpolation Filter - DCT-IF Intra Prediction DC, Planar, 33 Directions, DM Transform DCT 4x4~32x32, Skip 4x4, DST 4x4 (Intra) In-loop Filter De-blocking, SAO Entropy Coding CABAC (Tiles, Wavefront) * Support picture resolution ranging from 128x96 to 8192x4320.

Information 37 http://phenix.int-evry.fr/jct/ (Website) http://mailman.rwth-aachen.de/mailman/listinfo/jct-vc (Subscribe) https://hevc.hhi.fraunhofer.de/svn/svn_hevcsoftware/ (SVN, Software Manual JCTVC-J0470) Text Specification Draft 9 (JCTVC-K1003)

RANGE EXTENSIONS TO HEVC 38

Range Extensions 39 Goal: minimum changes to HEVC V1 to support non- 4:2:0 chroma formats and bit depths beyond 8 bits and to improve lossless coding Finalizes in July 2014

RANGE EXTENSIONS TOOL FEATURES 40

RExt Tool Features One separate CABAC for significant map coding for TS block 41 Current Frame + - DCT Q Entropy Coding Bitstream Frame Buffer Inter Prediction Inter Residual DPCM Large TS block Residual rotation Cross component prediction Intra Prediction Intra In-loop + filter + IQ IDCT

Residual DPCM 42 Second-order residual prediction Vertical/horizontal prediction Implicit for Intra, explicit for Inter Short-distance residual prediction Processed at parsing stage sub. sub. sub. add add add p a-p b-p c-p d-p Parallel Encoding p a-p b-a c-b d-c Parallel Decoding Residual Block Reference Samples Residual DPCM

Intra Residual Rotation 43 Reverse scan order for intra residual blocks when transform skip is in use Much similar to the pattern of coefficients tend to be zero Rotate 180 degree tend to be non-zero tend to be non-zero Residual Block tend to be zero Rotated Block

Cross Component Prediction 44 Predict chroma residuals from luma residuals Apply to intra DM and inter (if luma residual exists) Adapt α {-8,-4,,8} at TU level for U/V separately Chroma αα 88 Luma 2 nd -order Chroma Recon. Chroma Residual Recon. Luma Residual Decoded 2 nd -order Chroma Residual

Common Test Conditions (AHG5) 45 Coding Structure All Intra Random access Low delay B QP Range (Lossy) MT (22, 27, 32, 37) HT (17, 22, 27, 32) SHT (12,17, 22, 27) Test Sequence Color Format Bit Depth Resolution (# of Seqs) RGB 4:4:4 8, 10,12 2560x1600 (1), 1920x1080 (7) YUV 4:4:4 10 2560x1600 (1), 1920x1080 (6) YUV 4:2:2 10 2560x1600 (1), 1920x1080 (6)

HM-14.0+RExt-7.0 vs. JM-18.6 46 Format All Intra Y BD-Rate (%) Radom Access Low Delay B MT HT SHT MT HT MT HT RGB 4:4:4-32.6-25.1-19.6-36.0-25.1-36.2-25.0 YUV 4:4:4-22.4-19.0-14.8-35.1-29.8-39.8-32.9 YUV 4:2:2-19.7-15.8-11.7-30.2-27.8-35.9-31.0 B. Li, J. Xu, and Gary. J. Sullivan, Comparison of Compression Performance of HEVC 4:4:4 Range Extensions Test Model 7 and HEVC Screen Content Coding Extensions Test Model 1 with AVC High 4:4:4 Predictive profile, JCTVC-R0101, Sapporo, June, 2014.

Information 47 http://phenix.int-evry.fr/jct/ (Website) http://mailman.rwth-aachen.de/mailman/listinfo/jct-vc (Subscribe) https://hevc.hhi.fraunhofer.de/svn/svn_shvcsoftware/ (SVN) Edition 2 Draft Text of High Efficiency Video Coding (HEVC), Including Format Range (RExt), Scalability (SHVC), and Multi-View (MV-HEVC) Extensions (JCTVC-R1013)

SCREEN CONTENT CODING (SCC) 48

Screen Content Coding (SCC) 49 Screen content video, usually a mixture of text, graphics and nature scene images, exhibits very different characteristics from camera-captured video

Screen Sharing 50 Share & Play Together by TM

Cloud Gaming 51

Camera-captured vs. Screen Contents 52 Camera-capture video Continuous-tone with camera noise Smooth edges, complicated texture, thick lines with rich colors Screen content video Discontinuous-tone with less/no noise Sharp edges, simple shapes, thin lines (e.g. 1-pixel wide) with few colors

Why SCC is Challenging? 53 Missing details and annoying artifacts Original SCM-1.0 (All Intra, QP27, 2.9Mbps, 42.5dB)

Requirements 54 From the discussions on the reflector, SCC should address 4:4:4 chroma sampling format (RGB, YUV) Up to 10-bit for each color component Low latency/complexity on encoder & decoder Temporal stability Subjectively lossless Mathematically lossless for some application Low bitrate for discontinuous-tone contents

Screen Content Coding 55 Future HEVC extensions in coding screen content targeting at coding of 4:4:4 8-bit sequences 2014/01: Final Call-for-Proposals 2014/03: Evaluation of Proposals 2014/04: SCC Test Model 1.0 2014/07: SCC Test Model 2.0 2015/02: Proposed Draft Amendment (PDAM) 2015/10: Final Draft Amendment (FDAM)

SCC Tool Features 56 Adaptive color transform Current Frame + - DCT Q Entropy Coding Bitstream IBC mode Palette mode Frame Buffer Inter Prediction Inter Intra Prediction Intra In-loop + filter + IQ IDCT

Intra Block Copy (IBC) 57 Similar to inter motion compensation, except using the current frame as reference referring to un-deblocked samples

Palette Mode 58 Adaptive color quantization for PCM pixels Represent a block by major colors & an index map Specify the major color a pixel mapped to by index [Block Samples] Analysis Probability p(x) Index 0 1 Quantization Major Color 0 0 0 0 [Index Map] 0 0 0 0 1 1 1 1 1 1 1 1 0 1 2 3 4 5 6 7 8 9 Intensity Value x

Other tools 59 Palette mode Major color coding (e.g. stuffing, merging, ) Index map coding (e.g. transition mode, dictionary, ) Intra string matching PCM sample coding Index coding Intra line copy

Major Color Coding (1/2) 60 Major color table (MCT) propagation Infer major colors from left or above CU (or previously decode ones when unavailable) Major color merge (left or above CU) Long-term palette prediction Signal a long-term palette at the slice header Triplet palette coding Three component-wise palette MCT propagation MCT propagation Left Reference CU Above Reference CU Current CU

Major Color Coding (2/2) 61 Palette stuffing (predictor propagation)

Index Map Coding (1/2) 62 Find a match inside current CU Run mode (copy-left, copy-above) Transition copy

Index Map Coding (2/2) 63 Search a match outside current CU 1-D/2-D String matching Re-quantization required on reconstructed pixels Search Range Search Range

Dictionary-based String Matching 64 Patterns repeat frequently within text and graphics regions (e.g. text, icons, lines etc.)

Reconstruction-based String Matching 65 Preserve block structure Hybrid 1-D/2-D string matching for index coding More flexible than IBC, higher data dependency

Intra Line Copy 66 Split a PU equally into 1xN/Nx1 lines Lower data dependency than string matching; more flexible than PU-based IBC BV0, PU1 Search Range BV0, PU0 BV1, PU0 Line (size=1)...... 2N PU0 PU1 Search Range

Common Test Conditions (SCC) 67 Coding Structure All Intra (AI) Random access (RA) Low delay B (LB) Category QP Range Lossy: 22, 27, 32, 37 Lossless Test Sequence Resolution (# of Seqs) Text and graphics with motion 1920x1080 (6), 1280x720 (8) Mixed content 2560x1440 (4), 1920x1080 (2) Animation 1280x720 (2) Camera-captured content 1920x1080 (4)

SCM-1.0 vs. JM-18.6 68 Coding Structure Y BD-Rate (%) Text, Graphics Mixed Content Anima. Camera 1080p 720p 1440p 1080p 720p 1080p RGB Sequences AI -85.8-71.6-69.5-74.9-36.8-45.1 RA -78.8-65.3-60.5-70.1-39.6-49.8 LB -78.4-61.3-55.3-62.0-42.9-46.6 YUV Sequences AI -77.2-56.6-54.0-63.8-23.6-26.9 RA -69.4-54.0-48.3-62.0-32.6-40.1 LB -68.8-52.8-45.9-56.1-39.1-40.0 B. Li, J. Xu, and Gary. J. Sullivan, Comparison of Compression Performance of HEVC 4:4:4 Range Extensions Test Model 7 and HEVC Screen Content Coding Extensions Test Model 1 with AVC High 4:4:4 Predictive profile, JCTVC-R0101, Sapporo, June, 2014.

Information 69 http://phenix.int-evry.fr/jct/ (Website) http://mailman.rwth-aachen.de/mailman/listinfo/jct-vc (Subscribe - AHG7: Screen Content Coding) https://hevc.hhi.fraunhofer.de/svn/svn_shvcsoftware/ (SVN) HEVC Screen Content Coding Draft Text 1(JCTVC-R1005)

TCSVT Special Issue on HEVC 70 IEEE Transaction on Circuits and Systems for Video Technology (TCSVT), vol. 22, no. 12, Dec., 2012 Special Section: HEVC Standard G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, Overview of the High Efficiency Video Coding (HEVC) Standard J.-R. Ohm, G. J. Sullivan, H.Schwarz, T. K. Tan, and T. Wiegand, Comparison of the Coding Efficiency of Video Coding Standards Including High Efficiency Video Coding (HEVC) Special Issue: Emerging Research and Standards in Next Generation Video Coding (HEVC)

71 Thank You ~