High Efficiency Video Coding (HEVC)

Transcription

1 High Efficiency Video Coding (HEVC) 1

2 The MPEG Vision 2 Three years ago in 2009, it was expected -- Ultra-HD (e.g., 4kx2k) video will emerge -- Mobile HD applications will become popular -- Video bitrate using current technology will go up faster than the network infrastructure Now, we see

3 London 2012 Olympic Games 3 Immersive experience in Super Hi-Vision 16 times the resolution of Full HD (1080p) Pioneered by NHK & BBC 7680x4320

4 Go beyond Full HD 4 DVD (720x480) SDTV (1280x720) HDTV (1920x1080) Digital Cinema 2K (2048x1080) VIDEO RESOLUTION COMPARISON FROM 480P TO 4320P Digital Cinema 4K (4096x2160) Super Hi-Vision / Ultra HD (7680x4320)

5 High Efficiency Video Coding (HEVC) 5 The latest draft video coding standard developed by a joint team of experts from ISO/IEC MPEG and ITU-T VCEG Goal: substantially better performance than the H.264/AVC standard, especially in coding HD and Ultra- HD video International Standard (IS) in April 2013

6 Timeline : 2010/01: 2010/04: 2010/10: 2013/01: 2013/04: MPEG/VCEG exploration activities Joint Collaborative Team (JCT-VC) Final Call-for-Proposals (CfP) 1 st JCT-VC Meeting (Dresden, Germany) 27 proposals received Working Draft 1.0 (WD1.0) and HM-1.0 Final Draft International Standard (FDIS) International Standard (IS)

7 HEVC Version 1 7 Press letter of 103 rd Geneva Meeting (N13253) The next major milestone in MPEG video history is achieved ISO/IEC JTC1/SC29/WG11 MPEG is proud to announce the completion of the new High Efficiency Video Coding (HEVC) standard which has been promoted to Final Draft International Standard (FDIS) status at the 103rd MPEG meeting.

8 Subjective Assessment 8 HM 5.0 JM 18.2 Over -50% MOS Class B (HD): -67% Class C (SD): -49% Bit Rate [Kbps] J. R. Ohm, G. J. Sullivan, F. Bossen, T. Wiegand, V. Baroncini, M. Wien, and J. Xu, JCT-VC AHG report: HM subjective quality investigation (AHG22), JCTVC-H0022, San José, CA, Feb., 2012.

9 New milestone since year Park Scene, 1920x1080, 24Hz H.264/AVC YUV-PSNR (db) HEVC 35.4% H.263 MPEG-2 MPEG-4 H.264/MPEG-2 (MP) MPEG-4 (ASP) H.263 (HLP) H.264/MPEG-4 AVC (HP) HEVC (MP) Bitrate (Mbps) J.-R. Ohm, G. J. Sullivan, H. Schwarz, T. K. Tan, and T. Wiegand, Comparison of the Coding Efficiency of Video Coding Standards Including High Efficiency Video Coding (HEVC), IEEE Trans. CSVT, Dec., 2012

10 HEVC vs. AVC (1/2) 10 HM10.0 (Main) vs. JM18.4 (High) Test Condition ALL INTRA 23% RANDOM ACCESS LOW DELAY 33.5% 36.4% BD-rate Saving (%) B. Li, G. J. Sullivan, and J. Xu, Comparison of Compression Performance of HEVC Draft 10 with AVC High Profile, JCTVC-M0329, Incheon, April, 2013.

11 HEVC vs. AVC (2/2) 11 Class Resolution Y BD-Rate (%) All Intra Radom Access Low Delay A 2500x B 1080p C 480p D 240p E 720p F 480p, 720p (Minus sign means coding gain)

12 Encoding Time 12 HM10.0 (Main) vs. JM18.4 (High) Test Condition ALL INTRA RANDOM ACCESS LOW DELAY 71% 97% 109% 20% 40% 60% 80% 100% 120% Encoding Time Ratio (%) B. Li, G. J. Sullivan, and J. Xu, Comparison of Compression Performance of HEVC Draft 10 with AVC High Profile, JCTVC-M0329, Incheon, April, 2013.

13 Decoding Time 13 HM10.0 (Main) vs. JM18.4 (High) Test Condition ALL INTRA 42% RANDOM ACCESS LOW DELAY 56% 107% 20% 40% 60% 80% 100% 120% Decoding Time Ratio (%) B. Li, G. J. Sullivan, and J. Xu, Comparison of Compression Performance of HEVC Draft 10 with AVC High Profile, JCTVC-M0329, Incheon, April, 2013.

14 Multi-thread Decoding 14 Frame Rate (fps) HM-9.0 (RA-Main) 3840x2160(12Mbps) 1080p 720p 60fps 30fps 0 0.5K 1K 1.5K 2K 2.5K Vertical Resolution T. Tan, Y. Suzuki, and F. Bossen, On software complexity: decoding 4K60p content on a laptop, JCTVC- L0098, Geneva, CH, Jan., 2013.

15 Few weeks later 15 Display 4.99 inch, 1920x1080 Camera 13 Mega Pixels Video Codec MPEG4, H.264, H.263, DivX, VC-1, VP8, WMV7/8, Sorenson Spark, HEVC Samsung UNPACKED

16 HEVC TOOL FEATURES 16

17 HEVC Tool Features CABAC Tiles Wavefront 17 Current Frame Asymmetric Motion Partitioning Merged Skip / Motion Merging Advanced MV Prediction DCT-based Interpolation Filter Frame Buffer Inter Prediction + Inter - DCT Q Entropy Coding Bitstream Residual Quad-tree Trans. Transform Skipping Adaptive Coeff. Scanning Deblocking Filter Sample Adaptive Offset Intra Prediction Intra In-loop filter + More Directions Pre-/Post-filtering Direct Chroma + IQ IDCT

18 Coding Unit (CU) 18 Basic unit for coding, conceptually similar to macroblock but now can be of variable size H.264/AVC HEVC 16 MB MB 64 CU CU CU CU CU CU CU CU CTU 0 CTU 1

19 Prediction Unit (PU) 19 16x16 MB PU PU PU PU PU PU PU PU PU PU PU PU PU PU PU PU PU CU (Only for SCU) Asymmetric Sub-MB Partition

20 Transform Unit (TU) Residual Quad-tree Transform (RQT) Transform can cross PU boundaries 16x16 MB TU TU 8 TU TU 8 4 TU TU TU TU 4 TU TU TU TU TU TU TU TU TU TU TU TU 20 PU Aligned May be skipped if 4x4 CU RQT TU TU TU TU TU TU TU TU TU TU TU TU

21 Intra Prediction 21 More directions (up to 33) Adaptive pre-filtering of reference pixels Boundary smoothing for DC/Ver./Hor. modes Direct mode for Chroma Reference Sample Mean 0: DC LB LB 1: Planar TR TR Horizontal Vertical

22 DCT-based Interpolation Filter 22 P -3 P -2 P -1 P 0 P a P 1 P 2 P 3 P 4 M Integer-pels Spatial Domain Samples {P -3, P -2,, P 4 } M Integer-pels Fraction-pel Forward DCT Inverse DCT DCT Domain Coefficients {C -3, C -2,, C 4 } p a 2M 1 C π k( 2M 1+ 2a) 0 = + Ck cos 2 k = 1 4M DCT Coefficients

23 Motion Merging (1/2) 23 Optional 5 Candidates at most

24 Motion Merging (2/2) 24 Before Merging After Merging mv0 mv1 mv0 mv1 mv4 mv5 mv4 mv5 mv2 mv3 mv2 mv3 Current PU 0 mv5 mv6 mv7 mv6 mv7 Current PU1 mv7 Merge w/ Nearby Motion Irregular Motion Partitions

25 Temporal Merge Candidate Derivation Co-located Current 1 L0 L0 Ref0 Current L1 Ref0 L1

26 Advanced Motion Vector Prediction 2 Find First Available 26 B2 B1 B0 1 T1 Optional 3 2 Candidates at most Find First Available A1 A0 Current PU T0 If LCU boundary, exclude A0 & T0

27 Transform Skipping 27 Operable in 4x4 TUs Switchable with DST (intra) / DCT (inter) TS Disabled TS Enabled (QP37, 608.4Kbps, 34.8dB) (QP36, 600.8Kbps, 36.1dB)

28 Significance Map Scanning 28 TU Size Prediction Type Scanning Order 4x4, 8x8 Intra (Vertical, Horizontal) Vertical, Horizontal All Intra, Inter 4x4 Sub-diagonal Vertical Horizontal 4x4 Sub-diagonal 4 4

29 Sample Adaptive Offset (1/3) 29 Band Offset (BO): intensity-based sample classification Edge Offset (EO): edge-based sample classification [BO] 0 4 Bands MAX [EO] EO Off Off EO C C BO BO BO EO BO BO BO EO C C Off EO EO EO LCUs in a Frame

30 Sample Adaptive Offset (2/3) 30 I. Positive Edge Offset Pixel Level x-1 x x+1 x-1 x x+1 x-1 x x+1 Pixel Index II. Negative Edge Offset Pixel Level x-1 x x+1 x-1 x x+1 Pixel Index x-1 x x+1

31 Sample Adaptive Offset (3/3) 31 SAO Disabled SAO Enabled (Frame15, 32.6dB, 3.0Mbps) (Frame15, 32.7dB, 3.1Mbps)

32 Parallel-friendly Design 32 Transcoding to WPP Sequential Bitstream Chunks re-ordering Chunk encoding using 16 cores Live Content Chunk re-ordering 2 entry points G. Clare, F. Henry, and S. Pateux, Wavefront and Cabac Flush: Different Degrees of Parallelism Without Transcoding, JCTVC-F275, Torino, IT, July, 2011.

33 Wavefront Parallel Processing (WPP) 33 Pass CABAC states to next Wave Parsing Syntax Elements Decompressing Decoded Pixels [n Waves in a Slice] Dependent Parsing Dependent Syntax Elements Dependent Decompressing Dependent Decoded Pixels Flush CABAC States Parallel Parallel

34 Tiles 34 Syntax Elements Decoded Pixels Parsing Decompressing Parsing Syntax Elements Decompressing Decoded Pixels [4 Tiles in a Slice/Frame] Parallel Parallel Prediction can t cross Tiles

35 Parallel Merge Group 35 Group m-1,n-1 Group m,n-1 Group m+1,n-1 Above Inferable Group m-1,n CUs/PUs within Group m,n Left Inferable PUs/CUs within a group run in parallel

36 Profiles & Tool Summary 36 Tool \ Profile Main Still Picture Main Main 10 Bit Depth 8 8 8, 9, 10 CU Size 16x16~64x64 PU Partition Symmetric Symmetric, Asymmetric TU Partition Residual Quad-tree Transform MV Prediction - AMVP, MRG, MRG-Skip Interpolation Filter - DCT-IF Intra Prediction DC, Planar, 33 Directions, DM Transform DCT 4x4~32x32, Skip 4x4, DST 4x4 (Intra) In-loop Filter De-blocking, SAO Entropy Coding CABAC (Tiles, Wavefront) * Support picture resolution ranging from 128x96 to 8192x4320.

37 Information 37 (Website) (Subscribe) (SVN, Software Manual JCTVC-J0470) Text Specification Draft 9 (JCTVC-K1003)

38 RANGE EXTENSIONS TO HEVC 38

39 Range Extensions 39 Goal: minimum changes to HEVC V1 to support non- 4:2:0 chroma formats and bit depths beyond 8 bits and to improve lossless coding Finalizes in July 2014

40 RANGE EXTENSIONS TOOL FEATURES 40

41 RExt Tool Features One separate CABAC for significant map coding for TS block 41 Current Frame + - DCT Q Entropy Coding Bitstream Frame Buffer Inter Prediction Inter Residual DPCM Large TS block Residual rotation Cross component prediction Intra Prediction Intra In-loop + filter + IQ IDCT

42 Residual DPCM 42 Second-order residual prediction Vertical/horizontal prediction Implicit for Intra, explicit for Inter Short-distance residual prediction Processed at parsing stage sub. sub. sub. add add add p a-p b-p c-p d-p Parallel Encoding p a-p b-a c-b d-c Parallel Decoding Residual Block Reference Samples Residual DPCM

43 Intra Residual Rotation 43 Reverse scan order for intra residual blocks when transform skip is in use Much similar to the pattern of coefficients tend to be zero Rotate 180 degree tend to be non-zero tend to be non-zero Residual Block tend to be zero Rotated Block

44 Cross Component Prediction 44 Predict chroma residuals from luma residuals Apply to intra DM and inter (if luma residual exists) Adapt α {-8,-4,,8} at TU level for U/V separately Chroma αα 88 Luma 2 nd -order Chroma Recon. Chroma Residual Recon. Luma Residual Decoded 2 nd -order Chroma Residual

45 Common Test Conditions (AHG5) 45 Coding Structure All Intra Random access Low delay B QP Range (Lossy) MT (22, 27, 32, 37) HT (17, 22, 27, 32) SHT (12,17, 22, 27) Test Sequence Color Format Bit Depth Resolution (# of Seqs) RGB 4:4:4 8, 10, x1600 (1), 1920x1080 (7) YUV 4:4: x1600 (1), 1920x1080 (6) YUV 4:2: x1600 (1), 1920x1080 (6)

46 HM-14.0+RExt-7.0 vs. JM Format All Intra Y BD-Rate (%) Radom Access Low Delay B MT HT SHT MT HT MT HT RGB 4:4: YUV 4:4: YUV 4:2: B. Li, J. Xu, and Gary. J. Sullivan, Comparison of Compression Performance of HEVC 4:4:4 Range Extensions Test Model 7 and HEVC Screen Content Coding Extensions Test Model 1 with AVC High 4:4:4 Predictive profile, JCTVC-R0101, Sapporo, June, 2014.

47 Information 47 (Website) (Subscribe) (SVN) Edition 2 Draft Text of High Efficiency Video Coding (HEVC), Including Format Range (RExt), Scalability (SHVC), and Multi-View (MV-HEVC) Extensions (JCTVC-R1013)

48 SCREEN CONTENT CODING (SCC) 48

49 Screen Content Coding (SCC) 49 Screen content video, usually a mixture of text, graphics and nature scene images, exhibits very different characteristics from camera-captured video

50 Screen Sharing 50 Share & Play Together by TM

51 Cloud Gaming 51

52 Camera-captured vs. Screen Contents 52 Camera-capture video Continuous-tone with camera noise Smooth edges, complicated texture, thick lines with rich colors Screen content video Discontinuous-tone with less/no noise Sharp edges, simple shapes, thin lines (e.g. 1-pixel wide) with few colors

53 Why SCC is Challenging? 53 Missing details and annoying artifacts Original SCM-1.0 (All Intra, QP27, 2.9Mbps, 42.5dB)

54 Requirements 54 From the discussions on the reflector, SCC should address 4:4:4 chroma sampling format (RGB, YUV) Up to 10-bit for each color component Low latency/complexity on encoder & decoder Temporal stability Subjectively lossless Mathematically lossless for some application Low bitrate for discontinuous-tone contents

55 Screen Content Coding 55 Future HEVC extensions in coding screen content targeting at coding of 4:4:4 8-bit sequences 2014/01: Final Call-for-Proposals 2014/03: Evaluation of Proposals 2014/04: SCC Test Model /07: SCC Test Model /02: Proposed Draft Amendment (PDAM) 2015/10: Final Draft Amendment (FDAM)

56 SCC Tool Features 56 Adaptive color transform Current Frame + - DCT Q Entropy Coding Bitstream IBC mode Palette mode Frame Buffer Inter Prediction Inter Intra Prediction Intra In-loop + filter + IQ IDCT

57 Intra Block Copy (IBC) 57 Similar to inter motion compensation, except using the current frame as reference referring to un-deblocked samples

58 Palette Mode 58 Adaptive color quantization for PCM pixels Represent a block by major colors & an index map Specify the major color a pixel mapped to by index [Block Samples] Analysis Probability p(x) Index 0 1 Quantization Major Color [Index Map] Intensity Value x

59 Other tools 59 Palette mode Major color coding (e.g. stuffing, merging, ) Index map coding (e.g. transition mode, dictionary, ) Intra string matching PCM sample coding Index coding Intra line copy

60 Major Color Coding (1/2) 60 Major color table (MCT) propagation Infer major colors from left or above CU (or previously decode ones when unavailable) Major color merge (left or above CU) Long-term palette prediction Signal a long-term palette at the slice header Triplet palette coding Three component-wise palette MCT propagation MCT propagation Left Reference CU Above Reference CU Current CU

61 Major Color Coding (2/2) 61 Palette stuffing (predictor propagation)

62 Index Map Coding (1/2) 62 Find a match inside current CU Run mode (copy-left, copy-above) Transition copy

63 Index Map Coding (2/2) 63 Search a match outside current CU 1-D/2-D String matching Re-quantization required on reconstructed pixels Search Range Search Range

64 Dictionary-based String Matching 64 Patterns repeat frequently within text and graphics regions (e.g. text, icons, lines etc.)

65 Reconstruction-based String Matching 65 Preserve block structure Hybrid 1-D/2-D string matching for index coding More flexible than IBC, higher data dependency

66 Intra Line Copy 66 Split a PU equally into 1xN/Nx1 lines Lower data dependency than string matching; more flexible than PU-based IBC BV0, PU1 Search Range BV0, PU0 BV1, PU0 Line (size=1) N PU0 PU1 Search Range

67 Common Test Conditions (SCC) 67 Coding Structure All Intra (AI) Random access (RA) Low delay B (LB) Category QP Range Lossy: 22, 27, 32, 37 Lossless Test Sequence Resolution (# of Seqs) Text and graphics with motion 1920x1080 (6), 1280x720 (8) Mixed content 2560x1440 (4), 1920x1080 (2) Animation 1280x720 (2) Camera-captured content 1920x1080 (4)

68 SCM-1.0 vs. JM Coding Structure Y BD-Rate (%) Text, Graphics Mixed Content Anima. Camera 1080p 720p 1440p 1080p 720p 1080p RGB Sequences AI RA LB YUV Sequences AI RA LB B. Li, J. Xu, and Gary. J. Sullivan, Comparison of Compression Performance of HEVC 4:4:4 Range Extensions Test Model 7 and HEVC Screen Content Coding Extensions Test Model 1 with AVC High 4:4:4 Predictive profile, JCTVC-R0101, Sapporo, June, 2014.

69 Information 69 (Website) (Subscribe - AHG7: Screen Content Coding) (SVN) HEVC Screen Content Coding Draft Text 1(JCTVC-R1005)

70 TCSVT Special Issue on HEVC 70 IEEE Transaction on Circuits and Systems for Video Technology (TCSVT), vol. 22, no. 12, Dec., 2012 Special Section: HEVC Standard G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, Overview of the High Efficiency Video Coding (HEVC) Standard J.-R. Ohm, G. J. Sullivan, H.Schwarz, T. K. Tan, and T. Wiegand, Comparison of the Coding Efficiency of Video Coding Standards Including High Efficiency Video Coding (HEVC) Special Issue: Emerging Research and Standards in Next Generation Video Coding (HEVC)

71 71 Thank You ~