Development of mixed screen content coding technology: SCC-HEVC extensions framework

Similar documents
Next-Generation 3D Formats with Depth Map Support

High Efficiency Video Coding: The Next Gen Codec. Matthew Goldman Senior Vice President TV Compression Technology Ericsson

High Efficiency Video Coding (HEVC)

White paper: Video Coding A Timeline

High Efficiency Video Coding (HEVC) test model HM vs. HM- 16.6: objective and subjective performance analysis

HEVC The Next Generation Video Coding. 1 ELEG5502 Video Coding Technology

A COMPARISON OF CABAC THROUGHPUT FOR HEVC/H.265 VS. AVC/H.264. Massachusetts Institute of Technology Texas Instruments

RECOMMENDATION ITU-R BT

Testing HEVC model HM on objective and subjective way

Comparative and performance analysis of HEVC and H.264 Intra frame coding and JPEG2000

Upcoming Video Standards. Madhukar Budagavi, Ph.D. DSPS R&D Center, Dallas Texas Instruments Inc.

NEW CAVLC ENCODING ALGORITHM FOR LOSSLESS INTRA CODING IN H.264/AVC. Jin Heo, Seung-Hwan Kim, and Yo-Sung Ho

CONTENT ADAPTIVE COMPLEXITY REDUCTION SCHEME FOR QUALITY/FIDELITY SCALABLE HEVC

An Multi-Mini-Partition Intra Block Copying for Screen Content Coding

A comparison of CABAC throughput for HEVC/H.265 VS. AVC/H.264

Reducing/eliminating visual artifacts in HEVC by the deblocking filter.

The Scope of Picture and Video Coding Standardization

Homogeneous Transcoding of HEVC for bit rate reduction

Updates in MPEG-4 AVC (H.264) Standard to Improve Picture Quality and Usability

Scalable Extension of HEVC 한종기

EE Low Complexity H.264 encoder for mobile applications

Advanced Video Coding: The new H.264 video compression standard

COMPARISON OF HIGH EFFICIENCY VIDEO CODING (HEVC) PERFORMANCE WITH H.264 ADVANCED VIDEO CODING (AVC)

Transcoding from H.264/AVC to High Efficiency Video Coding (HEVC)

Intel Stress Bitstreams and Encoder (Intel SBE) HEVC Getting Started

VIDEO COMPRESSION STANDARDS

Bi-directional optical flow for future video codec

Fast Decision of Block size, Prediction Mode and Intra Block for H.264 Intra Prediction EE Gaurav Hansda

ISO/IEC JTC1/SC29/WG11 N4065 March 2001, Singapore. Deadline for formal registration Deadline for submission of coded test material.

A NOVEL SCANNING SCHEME FOR DIRECTIONAL SPATIAL PREDICTION OF AVS INTRA CODING

Module 7 VIDEO CODING AND MOTION ESTIMATION

EFFICIENT PU MODE DECISION AND MOTION ESTIMATION FOR H.264/AVC TO HEVC TRANSCODER

Multi-View Image Coding in 3-D Space Based on 3-D Reconstruction

Video Quality Analysis for H.264 Based on Human Visual System

International Journal of Emerging Technology and Advanced Engineering Website: (ISSN , Volume 2, Issue 4, April 2012)

A COST-EFFICIENT RESIDUAL PREDICTION VLSI ARCHITECTURE FOR H.264/AVC SCALABLE EXTENSION

Mark Kogan CTO Video Delivery Technologies Bluebird TV

CROSS-PLANE CHROMA ENHANCEMENT FOR SHVC INTER-LAYER PREDICTION

ARCHITECTURES OF INCORPORATING MPEG-4 AVC INTO THREE-DIMENSIONAL WAVELET VIDEO CODING

MediaTek High Efficiency Video Coding

Implementation and analysis of Directional DCT in H.264

Professor, CSE Department, Nirma University, Ahmedabad, India

EE 5359 MULTIMEDIA PROCESSING SPRING Final Report IMPLEMENTATION AND ANALYSIS OF DIRECTIONAL DISCRETE COSINE TRANSFORM IN H.

DIGITAL TELEVISION 1. DIGITAL VIDEO FUNDAMENTALS

Recent Developments in Video Compression Standardization CVPR CLIC Workshop, Salt Lake City,

Analysis of 3D and Multiview Extensions of the Emerging HEVC Standard

Efficient Parallel Architecture for a Real-time UHD Scalable HEVC Encoder

Prediction Mode Based Reference Line Synthesis for Intra Prediction of Video Coding

Chapter 11.3 MPEG-2. MPEG-2: For higher quality video at a bit-rate of more than 4 Mbps Defined seven profiles aimed at different applications:

Interframe coding A video scene captured as a sequence of frames can be efficiently coded by estimating and compensating for motion between frames pri

ENCODER COMPLEXITY REDUCTION WITH SELECTIVE MOTION MERGE IN HEVC ABHISHEK HASSAN THUNGARAJ. Presented to the Faculty of the Graduate School of

THE H.264 ADVANCED VIDEO COMPRESSION STANDARD

FAST HEVC TO SCC TRANSCODING BASED ON DECISION TREES. Wei Kuang, Yui-Lam Chan, Sik-Ho Tsang, and Wan-Chi Siu

OVERVIEW OF IEEE 1857 VIDEO CODING STANDARD

Performance Evaluation of Kvazaar HEVC Intra Encoder on Xeon Phi Many-core Processor

PREPRINT OF A PAPER TO BE PUBLISHED at 32nd PICTURE CODING SYMPOSIUM (PCS 2016), Nuremberg, Germany, Dec 4-7, 2016.

Video Compression MPEG-4. Market s requirements for Video compression standard

Performance Comparison between DWT-based and DCT-based Encoders

Selected coding methods in H.265/HEVC

Video Coding Using Spatially Varying Transform

Sample Adaptive Offset Optimization in HEVC

INTERNATIONAL ORGANISATION FOR STANDARDISATION ORGANISATION INTERNATIONALE DE NORMALISATION ISO/IEC JTC1/SC29/WG11 CODING OF MOVING PICTURES AND AUDIO

FAST MOTION ESTIMATION DISCARDING LOW-IMPACT FRACTIONAL BLOCKS. Saverio G. Blasi, Ivan Zupancic and Ebroul Izquierdo

STACK ROBUST FINE GRANULARITY SCALABLE VIDEO CODING

STUDY AND IMPLEMENTATION OF VIDEO COMPRESSION STANDARDS (H.264/AVC, DIRAC)

High Efficiency Video Coding. Li Li 2016/10/18

IBM Research Report. Inter Mode Selection for H.264/AVC Using Time-Efficient Learning-Theoretic Algorithms

ECE 417 Guest Lecture Video Compression in MPEG-1/2/4. Min-Hsuan Tsai Apr 02, 2013

JPEG 2000 vs. JPEG in MPEG Encoding

H.264/AVC BASED NEAR LOSSLESS INTRA CODEC USING LINE-BASED PREDICTION AND MODIFIED CABAC. Jung-Ah Choi, Jin Heo, and Yo-Sung Ho

EFFICIENT INTRA PREDICTION SCHEME FOR LIGHT FIELD IMAGE COMPRESSION

Analysis of Motion Estimation Algorithm in HEVC

Video compression Beyond H.264, HEVC

An Improved H.26L Coder Using Lagrangian Coder Control. Summary

Digital Video Processing

A LOW-COMPLEXITY AND LOSSLESS REFERENCE FRAME ENCODER ALGORITHM FOR VIDEO CODING

Comparative and performance analysis of HEVC and H.264 Intra frame coding and JPEG2000

Image/video compression: howto? Aline ROUMY INRIA Rennes

On the Adoption of Multiview Video Coding in Wireless Multimedia Sensor Networks

VHDL Implementation of H.264 Video Coding Standard

Investigation of the GoP Structure for H.26L Video Streams

ISO/IEC INTERNATIONAL STANDARD. Information technology JPEG 2000 image coding system: Motion JPEG 2000

New Techniques for Next Generation Video Coding

Fast Transcoding From H.264/AVC To High Efficiency Video Coding

Context-Adaptive Binary Arithmetic Coding with Precise Probability Estimation and Complexity Scalability for High- Efficiency Video Coding*

Video Codecs. National Chiao Tung University Chun-Jen Tsai 1/5/2015

Performance Comparison of AV1, JEM, VP9, and HEVC Encoders

A Quantized Transform-Domain Motion Estimation Technique for H.264 Secondary SP-frames

Compression of 3-Dimensional Medical Image Data Using Part 2 of JPEG 2000

ABSTRACT

NOT FOR DISTRIBUTION OR REPRODUCTION

Video Coding Standards: H.261, H.263 and H.26L

FAST MOTION ESTIMATION WITH DUAL SEARCH WINDOW FOR STEREO 3D VIDEO ENCODING

Transcoding from H.264/AVC to High Efficiency Video Coding (HEVC)

Laboratoire d'informatique, de Robotique et de Microélectronique de Montpellier Montpellier Cedex 5 France

Scalable Multiresolution Video Coding using Subband Decomposition

Fast Intra Mode Decision in High Efficiency Video Coding

Rotate Intra Block Copy for Still Image Coding

Homogeneous Transcoding of HEVC

High Efficiency Video Decoding on Multicore Processor

Transcription:

Development of mixed screen content coding technology: SCC-HEVC extensions framework DRAGORAD MILOVANOVIC, ZORAN BOJKOVIC University of Belgrade Studentski Trg 1, 11000 Belgrade SERBIA dragoam@gmail.com http://www.zoranbojkovic.com Abstract: The status of high efficiency video coding development is reviewed based on public available documents in HEVC version 4 standardization process. Here, the requirements and results of call for proposals on screen content coding (SCC) tools for HEVC are presented. The most significant additional coding tools are highlighted. To achieve high coding efficiency in SCC-HEVC, new coding tools that explore the correlation in screen content are developed: IBC block vector search, inter block search, palette mode, and adaptive colour transform. The performances of the coding tools are approved in verifications tests through cooperation phase of standardization process. Key-Words: image and video technologies, mixed content coding, standardization process 1 Introduction HEVC (High Efficiency Video Coding) is the joint video coding standardization project of the ITU-T Video Coding Experts Group (VCEG) and ISO/IEC Moving Picture Experts Group (MPEG). The technical development has been performed by the Joint Collaborative Team on Video Coding (JCT-VC) since 2010. The scope of this group was extended to continue working on Format Range Extensions (RExt) and Screen Content Coding (SCC) as extensions of HEVC, expected to be finalized in early 2016. The design of HEVC incorporates the latest state-ofthe-art technologies and algorithmic advances to address the persistent demand for broader usage of video content, video migration to broadband networks, diversification of mobile devices, everhigher resolutions for cameras and displays, and increasingly high video quality [1]. HEVC achieved first edition 1E of the specification in January 2013. Another milestone has been achieved for HEVC by finalizing the Range Extensions FRExt amendment, with technology allowing efficient compression of video content for colour sampling formats beyond 4:2:0 and up to 16 bits of processing precision. In particular, the lossless and near lossless range of visual quality is more efficiently compressed than is possible with the first version 1E technology. The amendment is integrated into a new edition of HEVC to be published as the second 2E version in October 2014. Further Screen Content Coding Extensions remain in development for completion in early 2016 for video containing rendered graphics, text, or animation as well as (or instead of) camera-captured video scenes. Rec. ITU-T H.265 ISO/IEC 23008-2 fourth version 4E (High efficiency video coding with extensions for screen content coding 06/2016) refers to the integrated text additionally containing screen content coding extensions, additional supplement enhancement information, and corrections to various minor defects in the prior content of the specification. The third edition published by ISO/IEC as ISO/IEC ISO/IEC 23008-2:2016 corresponds to the fourth version (4E). Screen content coding is defined as the coding of video containing a significant proportion of rendered (moving or static) graphics, text, or animation rather than, or in addition to, cameracaptured video scenes. The different characteristics between screen content and camera-captured content make the encoding of screen content different from camera-captured content. To achieve high coding efficiency for screen content, new coding tools should be considered to explore the correlation in screen content. In the first part of the paper, the SSC-HEVC standardization framework is reviewed based on the JCT-VC working documents. In the final part, the most significant developed coding tools are highlighted. ISBN: 978-1-61804-355-9 44

2 HEVC FRExt extensions High Efficiency Video Coding (HEVC) is the current joint video coding standardization project of the ITU-T Video Coding Experts Group (ITU-T Q.6/SG 16) and ISO/IEC Moving Picture Experts Group (ISO/IEC JTC 1/SC 29/WG 11). The Joint Collaborative Team on Video Coding (JCT- VC) was established to work on this project. The scope of this group was extended to continue working on Format Range Extensions (RExt), Scalable HEVC (SHVC) and Screen Content Coding (SCC) as extensions of HEVC. The main steps of HEVC technical developments are organized in four phases (Table 1): 1E. The HEVC first base specification finalized in 2013 2E. Fidelity range extensions (FRExt), Scalable video coding (SHVC) and Multi-view video coding (MV- HEVC) extensions finalized in 2014 3E. 3D video coding (3D-HEVC) extension finalized in 2015 4E. Screen Content Coding (SCC) extensions will be included in the fourth version of HEVC, which is expected to be finalized in 2016. Table 1. Four editions of ITU-T Rec. H.265. ITU-T Rec. H.265.1 High efficiency video coding Annex A Profiles, tiers and levels Annex B Byte stream format 1E 13.04.2013 Annex C Hypothetical reference decoder Annex D Supplemental enhancement information Annex E Video usability information Annex F Common syntax, semantics and decoding process for multi-layer video coding extensions 2E 29.10.2014 Annex G Multiview coding Annex H Scalable high efficiency video coding 3E Annex I 3D High efficiency video coding 29.04.2015 ITU-T Rec. H.265.1 High efficiency video coding with extensions for screen content coding 4E Extensions profiles: Screen-Extended Main, 06/2016 Screen-Extended Main 10, Screen-Extended Main 4:4:4, Screen-Extended Main 4:4:4 10 The Fidelity Range Extensions (FRExt) expands the range of bit depths and color sampling formats supported by the standard, and include an increased emphasis on high-quality coding, lossless coding, and screen-content coding. The drafted range extensions for HEVC include support for the 4:2:2 and 4:4:4 enhanced chroma sampling structures and sample bit depths beyond 10 bits per sample. Additional areas of work include coding of screen content (graphics and other noncamera-view or mixed content), very high bit-rate and lossless coding, coding of auxiliary pictures (e.g., alpha transparency planes), and direct coding of RGB source content [2,3]. With HM 16.0 reference software the RExt branch was merged into the mainline HM software. The several relatively small changes to the decoding process have been developed for the 4:4:4 chroma sampling, near-lossless or lossless encoding, and screen content, that improve coding efficiency [2]. Intra-picture block prediction (IBC). Prediction operate by copying blocks of previously decoded regions within the same picture, in a similar manner to how motion compensation operates when referencing other decoded picture. Smoothing disabling for intra-picture prediction. Encoder disable a smoothing pre-filtering that is otherwise applied to intra-picture spatial prediction signal. Transform skip mode modification. Applies both to lossy and lossless mode, and include enabling horizontal and vertical DPCM coding modes for residual signals, supports of transform skipping for any block size, and rotation of 4x4 residual signals for more efficient entropy coding. Cross-component decorrelation methods. Correlation between different color components are typically larger in RGB color representation, compared to YCbCr (where the chroma components are already substantially decorrelated differences relative to the luma). Also, the penalty (in terms of bit rate increase) of not exploiting such correlations is naturally larger in color formats without subsampling of components. Therefore, methods for inter-component prediction are being investigated as possible additional elements to be applied within the encoding/decoding processes. Improved compression in lossless and near lossless coding. The HEVC first edition already enables lossless compression by skipping the transform, quantization and loop filtering, whereas prediction (motion-compensated or intra-picture) and entropy coding are used mostly as is. Substantially different techniques have been proposed that may lead to additional improvements when coding at very high fidelities. Whereas the HEVC first edition and its FRExt extensions enable improved compression of screen content by the simple options of the block copying and transform bypass mode, other, more sophisticated methods particularly suitable for SCC coding synthetic image structures (which have characteristics such as sharp edges and repetitive patterns) are under development. 2.1 Improvement of SCC efficiency Screen content has several new features not previously available in camera-captured content, including [4]: Sharp content. Compared with camera-captured content, screen content usually has with sharp edges. The sharp edges make the transform, which works well to reduce redundancy for camera-captured content, not suitable for screen content. To help the encode sharp content, transform skip has been designed for screen content. ISBN: 978-1-61804-355-9 45

Large motion. Screen content may contain large motions. For example, when browsing a web page, a large motion exists when scrolling the page. Conventional motion estimation algorithm may not be able to handle large motion well. Thus, new motion estimation algorithms to handle the large motions for screen content may be required. Unnatural motion. Screen content may contain unnatural motions, such as fading in and fading out. The conventional motion model, which usually only takes translational motions into consideration, may not be able to handle screen content well. Repeating patterns. Repeating patterns are also very common in screen content. For example, the screen content may contain the same letter many times. The same letters belong to a repeating pattern and using previous letters to predict the current letter may be quite efficient. To utilize the correlation among repeating patterns, Intra Block Copy (IBC) has been developed. With the help of IBC, similar blocks can be predicted and encoded efficiently. The different characteristics between screen content and camera-captured content make the encoding of screen content different from camera-captured content. To achieve high coding efficiency for screen content, new coding tools should be considered to explore the correlation in screen content. 2.2 SCC competition phase The Screen Content Coding (SCC) extensions improve compression capability for video containing a significant portion of rendered (moving or static) graphics, text, or animation rather than (or in addition to) camera-captured video scenes. Example applications include wireless displays, remote computer desktop access, and real-time screen sharing for videoconferencing. SCC will be included in the fourth version of HEVC, which is expected to be finalized in February 2016. The intent of document Requirements for an extension of HEVC for coding of screen content is to provide necessary information that can be used to help select new coding tools and develop new extensions of HEVC targeted at these specific video applications. When compressing these new types of content, e.g. screen content, any new tools in future HEVC extensions should significantly outperform the current HEVC standard or they should provide capabilities not available in those standards. Furthermore, the document includes quality requirements for the given applications so that tests performed in MPEG can be designed to determine whether coding tools developed in MPEG meet those quality requirements. Document [5] summarizes requests in the following three groups: Mixed content. In general, this type of video signal contains a mix of camera-captured natural videos, images, and computer-generated graphics and text. The camera-captured natural videos and images constitute a portion or portions of a source video frame and may have been compressed with visible compression artefacts. The camera-captured natural videos and images usually have been chroma-subsampled to YUV 4:2:0 and then chroma-upsampled to YUV 4:4:4, and thus may have significant chroma redundancy. Non-camera-captured medical content with overlaid and/or supplemental computer-generated graphics and text also falls into this category. Requirements for mixed content are as follow. Quality: up to visually lossless for all or part of the picture; Bitrate: comparable to that supported by HEVC Main 10 4:4:4; Configurations: including a low-delay configuration; Codec complexity: based on the framework of HEVC Main 10 4:4:4, with low extra complexity. Text and graphics with motion. In general, this type of video signal represents the computer-generated graphics and text commonly seen on computer displays. Typical motion observed in this type of content includes window switching and moving, text scrolling, etc. This type of content usually exhibits clear textures and sharp edges with distinct colours. In addition, the text and graphics material in this group is generally not subject to prior lossy compression or processing. Requirements for text and graphics with motion are as follow. Quality: up to visually lossless, mathematically lossless; Bitrate: comparable to that supported by HEVC Main 10 4:4:4; Configurations: including a low-delay configuration; Codec complexity: based on the framework of HEVC Main 10 4:4:4, with low extra complexity. Animation content. In general, this type of video signal represents the computer-generated animation content commonly seen in gaming and data visualization. Typical characteristics of this type content include fast motion, rotation, smooth shade, 3D effect, highly saturated colours with full resolution, clear textures and sharp edges with distinct colours. In addition, the animation content material in this group is generally not subject to prior lossy compression or processing. Requirements for animation content are as follow. Colour sampling format: RGB or YUV 4:4:4; Resolution: same as those supported by HEVC Main 10 4:4:4; Frame rate: same as those supported by HEVC Main 10 4:4:4; Bit-depth: 8 or 10 bits per colour component. ISBN: 978-1-61804-355-9 46

The Joint Call for Proposals for coding of screen content represented the start of standardization process [6]. Because there is evidence that significant improvements in coding efficiency can be obtained by exploiting the characteristics of screen content with novel dedicated coding tools, a CfP is being issued with the target of possibly developing future extensions of the HEVC standard including specific tools for screen content coding. The proposed technologies are evaluated based upon objective metrics and through subjective testing. Results of these tests are made public, taking into account that no direct identification of any of the proponents will be made (unless it is specifically requested or authorized by a proponent to be explicitly identified). Test material. Below is a list of the 4:4:4 screen content sequences to be used. Both the provided RGB and YCbCr formats of each sequence shall be processed. Subjective testing will be performed on only the 1920x1080 and 1280x720 resolutions and using only the RGB format. Parameters and conditions. The two test conditions are lossy (the decoded compressed content is not necessarily numerically identical to the uncompressed content) and mathematically lossless (the decoded compressed content is numerically identical to the uncompressed content). Coding constraint conditions. AI (All Intra - all pictures are coded as Intra pictures), LD (Low delay - the first picture is an Intra picture, and there are no backward references for inter prediction /bi-prediction may be applied, but only without picture reordering/) and RA (Random Access - Intra picture every 16, 32, and 64 pictures for 20 fps, 30 fps, and 60 fps sequences, respectively). Subjective test procedure. Subjective testing will be performed only on the bitstreams generated using the AI and LD coding constraints, and only for the 1280x720 and 1920x1080 RGB sequences. Four QP values are associated with each anchor sequence. For subjective testing, submitted bitstreams associated with only the three larger of these four QP values will be used. Thus, subjective testing will be performed on the three sequences comprising the lower-rate subset. Anchors have been generated by encoding the source sequences using the software and configuration files described below from the HM-12.1 RExt-5.1 software. The tests were performed in four Laboratories located in Europe. The report Results of CfP on Screen Content Coding Tools for HEVC summarizes the result of the Joint Call for Proposals for Coding of Screen Content, for possibly extending HEVC by associated tools [7]. Seven responses (Qualcom, NCTU&ITRI, MediaTek, Huawei, Microsoft, Mitsubishi, InterDigital) to the Joint Call for Proposals were received during the April 2014 meetings of ITU-T SG 16/Q.6 (VCEG) and ISO/IEC JTC 1/SC 29/WG 11 (MPEG). As a result, it was concluded that the responses provided justification for launching a new standardization project with the goal of extending HEVC, tentatively to be finalized in late 2015. The technical development are performed by JCT-VC. The study of tools included in the proposals showed many commonalities in terms of the basic concepts, however with various differences in detailed implementation as well as encoder and decoder complexity. The following categories of tools were identified from the proposals: Block based intra block copy with various aspects of partitioning, search range, mode switching and displacement vector coding; Line based intra copy; Palette mode with various aspects of coding the colour palette itself as well as the indices; String matching based on dictionary coding; Cross component prediction and adaptive colour transform; Miscellaneous small modifications of existing tools (such as de-blocking, RDPCM, etc.); Encoder-only modifications. From the results of objective (PSNR vs. bit rate) testing of the submitted proposals, substantial bit rate savings were observed to indicate a potential benefit of dedicated tools for screen content coding. The result of the formal subjective assessment also showed a clear improvement for several proposals in comparison to the anchor. Furthermore, various cases showed a visual quality very close to full transparency even for lossy coding. Therefore, after the submissions to the CfP provided evidence that significant improvements in coding efficiency can be obtained by exploiting the characteristics of screen content with novel dedicated coding tools, it was planned to develop an extension of the HEVC standard that provides the following: Compression tools with a benefit in compression performance for video with screen content or mixed content characteristics: A minimum set of profiles for maximum interoperability between devices in the foreseen application domains is targeted. This extension are developed by JCT-VC as a twin text specification, extending ISO/IEC 23008-2, with the following tentative timeline: First Test Model 14/04, PDAM 10/2014, DAM 02/2015, FDAM 10/2015. ISBN: 978-1-61804-355-9 47

2.3 SCC collaboration phase The development of SCC extensions for HEVC are based on set of core experiments (CE) specifies tools under investigation and time line of simulation and cross-check reports. Common test conditions (CTC) for screen content experimentation specifies test scenarios under consideration, test sequences, basic encoder configuration, and objective / subjective evaluation of visual quality (Table 2). The document Common test conditions for screen content coding defines common test conditions and software reference configurations to be used in the development of HEVC screen content coding (SCC) extension [8]. Common test conditions (CTC) are defined for conducting experiments in an environment where outcome of the experiments can be compared. This document defines two coding modes, three configurations, two encoding formats and four quantization parameter (QP) values. Table 2. Description of core experiments in SCC-HEVC. CE1 CE2 CE3 CE4 CE5 Color gamut scalability Line-based Intra copy Palette mode String matching for sample coding Inter-component prediction, adaptive color transforms HEVC Screen content coding test model (SCM) provides an encoder-side description, which serves as a tutorial for the encoding model implemented in the HM reference software [9-14]. The purpose of the document is to share a common understanding of the reference encoding methods supported in the software, in order to facilitate the assessment of the technical impact of the proposed new technologies during the standardization process. The common test conditions and software reference configurations that should be used for experimental work are described in CTC. The only changes in SCM1 with respect to the HEVC Range extensions test model TM7 are in block vector search for intra block copy (IBC) and hash-based search method for the inter mode [9-10]. The main changes from SCM1 to SCM2 are the addition of palette mode, residual adaptive colour transform, modifications to the block vector predictor, and block vector difference coding for IBC [10,11]. The main new coding tool from SCM2 to SCM3 is adaptive motion vector resolution. Other aspects of the design, such as the hash-based inter search and IBC search, palette coding, and adaptive colour transform, have also been improved [11,12]. The main changes from SCM3 to SCM4 are unification of intra block copy (IBC) and inter signalling, grouping of bypass bins for palette indices and escape values, and enabling the palette mode for non-4:4:4 chroma formats [12,13]. 3 SCC coding tools The coding tools described in SCM4 test model are implemented in the four point release of the 16 th HEVC reference software. For an understanding of the encoder upon which SCM4 is built, it is necessary refer to the HEVC test model 16 update 2 of encoder description and the reference software HEVC-16.4+SCM-4.0 [14]. Reference software is being made available to provide a reference implementation of the HEVC standard being developed by the JCT-VC. One of the main goals of the reference software is to provide a basis upon which to conduct experiments in order to determine which coding tools provide desired coding performance. It is not meant to be a particularly efficient implementation of anything, and one may notice its apparent unsuitability for a particular use. It should not be construed to be a reflection of how complex a production-quality implementation of a future HEVC standard would be. An outline of developed SCC coding tools is follow. IBC block vector search. In order to evaluate the ratedistortion (RD) cost of using the Intra BC mode, for each CU, block matching (BM) is performed at the encoder to find the optimal block vector. In SCM, first a local area search is performed. This is followed by a search over the entire picture for certain CU sizes. Fig. 1 Spatial candidates for IBC block vector prediction [13]. Inter block search. Compared to the HEVC Range extensions test model 7, SCM modifies the inter block search in two ways. The inter search is modified to adapt to the characteristics commonly found in screen content sequences. Furthermore, the inter block search is extended to the whole picture using hash-based techniques. Hash-based search is applied only to 2N 2N blocks. For one block, two hash values are calculated in a similar way but with different CRC truncated polynomials. The first hash value is used for retrieval and the second hash value is used to exclude some of the hash conflicts. Fig. 2 Search candidates for the modified method [13]. ISBN: 978-1-61804-355-9 48

Palette mode. The basic idea behind a palette mode is that the samples in the CU are represented by a small set of representative colour values. This set is referred to as the palette. It is also possible to indicate a sample that is outside the palette by signalling an escape symbol followed by (possibly quantized) component values. This is illustrated in Figure 3. In this example, the palette size is 4. The first 3 samples use palette entries 2, 0, and 3, respectively, for reconstruction. The blue sample represents an escape symbol. For decoding a palettecoded block, the decoder needs to have the following information: the palette entries and palette indices or escape symbol for each sample (in case of escape symbol, additional component values - possibly quantized). In addition, on the encoder side the appropriate palette to be used with that CU needs to be derived. Fig. 3 Example of a block coded in palette mode [13]. Adaptive Colour Transform (ACT). ACT performs inloop colour space conversion in the prediction residual domain using colour transform matrices based on the YCoCg and YCoCg-R colour spaces. ACT is turned on or off adaptively at the CU level using the flag cu_residual_act_flag. ACT can be combined with Cross Component Prediction (CCP), which is another inter component de-correlation method already supported in HEVC RExt. When both are enabled, ACT is performed after CCP at the decoder, as shown in Figure 4. Care is taken on the encoder side when performing ACT, in order to avoid doubling the encoder complexity by searching all the possible modes twice - in both the original colour space and the converted colour space. Fig. 4 SCC decoder flow of in-loop ACT [13]. Adaptive motion vector resolution. Adaptive MV resolution allows the MVs of an entire picture to be signalled in either quarter-pel precision (same as HEVC version 1) or integer-pel precision. Hash based motion statistics are kept and checked in order to properly decide the appropriate MV resolution for the current picture without relying on multi-pass encoding. 4 Conclusion Mixed screen content has several new features not previously available in camera-captured content, including sharp content, large motion, unnatural motion and repeating patterns. The different characteristics between screen content and cameracaptured content make the encoding of screen content different from camera-captured content. To achieve high coding efficiency for SCC-HEVC, new coding tools are considered to explore the correlation in screen content: block based intra block copy (with various aspects of partitioning, search range, mode switching and displacement vector coding), line based intra copy, palette mode (with various aspects of coding the colour palette itself as well as the indices), string matching based on dictionary coding, cross component prediction and adaptive colour transform, miscellaneous small modifications of existing tools (such as de-blocking, RDPCM), encoder-only modifications. Here, the SCC-HEVC extensions profiles under consideration in fourth edition (4E) are Screen- Extended Main, Screen-Extended Main 10, Screen- Extended Main 4:4:4, Screen-Extended Main 4:4:4 10 [15, 16]. The most significant SCC-HEVC developed additional coding tools are IBC block vector search, inter block search, palette mode, and adaptive colour transform. The performances of the new tools are approved in verifications tests through cooperation phase of standardization process (Table 3, Table 4). References: [1] K.R.Rao, Z.S.Bojkovic, D.A.Milovanovic, Introduction to multimedia communications: applications,middleware,networking, Wiley, 2005. [2] G.J. Sullivan, J. Boyce, Y. Chen, J.-R. Ohm, Standardized extensions of High Efficiency Video Coding (HEVC), IEEE Journal of Selected topics in signal processing, vol.7, pp.1001-1016, Dec. 2013. [3] B. Li, J. Xu, An HEVC-based Screen Content Coding scheme, TechReport, MSR- TR-2015-3, Jan. 2015. [4] D. Marpe, T. Wiegand, S. Gordon, H.264/MPEG4-AVC Fidelity Range Extensions: tools, profiles, performance, and application areas, IEEE ICIP 2005, pp.i-593-6. ISBN: 978-1-61804-355-9 49

[5] Doc. N14174 ISO/IEC JTC1/SC29/WG11, Requirements for an extension of HEVC for coding of screen content, 107th Meeting: San Jose, USA, Jan. 2014. [6] Doc. N14175 ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q6, Joint Call for Proposals for coding of screen content, 107th Meeting: San Jose, USA, Jan. 2014. [7] Doc. N14399 ISO/IEC JTC1/SC29/WG11, Results of CfP on Screen Content Coding Tools for HEVC, 108th Meeting: Valencia, ES, April 2014. [8] JCT-VC Q1015 ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, Common test conditions for screen content coding, 17th Meeting: Valencia, ES, April 2014. [9] JCT-VC Q1013 ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, HEVC Range extensions test model 7 encoder description, 17th Meeting: Valencia, ES, April 2014. [10] JCT-VC Q1014 ITU-T SG16 WP3 and coding test model 1 (SCM 1), 17th Meeting: Valencia, ES, March 2014. [11] JCT-VC R1014 ITU-T SG16 WP3 and coding test model 2 (SCM 2), 18th Meeting: Sapporo, JP, June 2014. [12] JCT-VC T1014 ITU-T SG16 WP3 and coding test model 3 (SCM 3), 19th Meeting: Strasbourg, Fr, Oct. 2014. [13] JCT-VC T1014 ITU-T SG16 WP3 and coding test model 4 (SCM 4), 20th Meeting: Geneva, CH, Feb. 2015. [14] JCT-VC T1002 ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, High Efficiency Video Coding (HEVC) Test Model 16 (HM 16) Improved encoder description Update 2, 20th Meeting: Geneva, CH, Feb. 2015. HM16.5 Software Manual (available online https://hevc.hhi.fraunhofer.de/svn/svn_he VCSoftware/tags/HM-16.4+SCM-4.0/) [15] JCT-VC T1005 ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, High Efficiency Video Coding (HEVC) Screen Content Coding: Draft 3, 20th Meeting: Geneva, CH, Feb. 2015. [16] Doc. N15138 ISO/IEC JTC1/SC29/WG11, Text of ISO/IEC 23008-2:2013/PDAM5 Screen Content Coding Extensions, 108th Meeting: Geneva, CH, Feb. 2015. Table 3. FRExt-HEVC standardization track. 10/2012 01/2013 14/2013 08/2013 10/2013 01/2014 04/2014 07/2014 HEVC 2E Test Model TM1 TM2 TM3 TM4 TM5 TM6 TM7 Draft text WD1 WD2 WD3 WD4 WD5 WD6 WD7 Reference software HM12 HM-12.1 +RExt5 HM-13.0 +RExt6 HM-14.0 +RExt7 Conformance test CD1 CD2 Table 4. SCC-HEVC standardization track. 01/2014 04/2014 07/2014 10/2014 02/2015 06/2015 10/2015 02/2016 06/2016 CfP PDAM5 DAM FDAM HEVC 4E Test Model SCM1 SCM2 SCM3 SCM4 Draft text WD1 WD2 WD3 WD4 Reference software HM-14.0 HM-15.0 HM-16.2 HM-16.4 +RExt-7.0 +RExt-8.0 +SCM-3.0 +SCM-4.0 +SCM-1.0 +SCM-2.0 ISBN: 978-1-61804-355-9 50