EE 5359 MULTIMEDIA PROCESSING HEVC TRANSFORM

Transcription

1 EE 5359 MULTIMEDIA PROCESSING HEVC TRANSFORM SPRING 2016 By ASHRITA MANDALAPU ( ) MANU RAJENDRA SHEELVANT ( ) MOIZ MUSTAFA ZAVERI ( )

2 ACRONYMS AVC CTU CU DCT DFT DST FPS HD Advanced Video Coding Coding Tree Unit Coding Unit Discrete Cosine Transform Discrete Fourier Transform Discrete Sine Transform Frames Per Second High Definition HDTV High Definition Television HEVC High Efficiency Video Coding HM ICIP ICME IDCT ISO ITU-T HEVC Test Model IEEE International Conference in Image Processing Institute for Computational & Mathematical Engineering Inverse Discrete Cosine Transform International Organization for Standardization International Telecommunication Union (Telecommunication Standardization Sector) MDDT Mode-Dependent Directional Transform PU ROT Prediction Unit Rotational Transform

3 RQT TB TU SIMD Residual Quadtree Transform Block Transform Unit Single Instruction Multiple Data

4 ABSTRACT This paper describes the need, design and implementation of transforms in the High Efficiency Video Coding (HEVC) standard. Core transform matrices range from 4x4 to 32x32 and are finite precision approximations of the discrete cosine transform (DCT). Also, an approximation of the Discrete Sine Transform (DST) is used for intra 4x4 luma block. Furthermore, various properties of DCT, basis vectors, scaling, and flags that are involved in the transform process are discussed. In the end, the arithmetic and hardware complexity analysis is analyzed. INTRODUCTION HEVC is a successor to the H.264/AVC video coding standard [1]. One of its primary objectives is to provide approximately two times the compression efficiency of its predecessor without any detectable loss in visual quality. HEVC thus achieves 2x higher compression compared to H.264/AVC. [1] HEVC adheres to the hybrid video coding structure; it uses spatial and temporal prediction, transform of the prediction residual, and entropy coding of the transform and prediction information. HEVC provides high throughput (Ultra-HD 120fps) and low power. HEVC features friendly implementations such as built-in parallelism. The HEVC standard specifies core transform matrices of size 4 4, 8 8, 16 16, and to be used for two-dimensional transforms in the context of block-based motioncompensated video compression. HEVC specifies two-dimensional transforms resembling the integer discrete cosine transform (IDCT) for all transform sizes. Multiple transform sizes improve compression performance, but also increase the implementation complexity. Hence a careful design of the core transforms is needed. HEVC will provide a flexible, reliable and robust solution to support the next decade of video. HEVC benefits include Reduce the burden on global networks Easier streaming of HD video to mobile devices Account for advancing screen resolutions (e.g. Ultra-HD)

5 HEVC CORE TRANSFORM DESIGN During the development of HEVC, several different approximations of the IDCT were studied for the core transform. The first version of the HEVC Test Model HM1 used the H.264/AVC transforms for 4 4 and 8 8 blocks and integer approximation of Chen s fast IDCT [14] for and blocks. The HEVC core transform matrices were designed to have the following properties [7]: Closeness to the IDCT Almost orthogonal basis vectors Almost equal norms of all basis vectors Same symmetry properties as the IDCT basis vectors Smaller transform matrices are embedded in larger transform matrices as shown in Figure 1. 8-bit representation of transform matrix elements 16-bit transpose buffer Multipliers can be represented using 16 bits or less with no cascaded multiplications or intermediate rounding Accumulators can be implemented using less than 32 bits In the block-based hybrid video coding approach, transforms are applied to the residual signal resulting from inter or intra frame prediction as shown in Figure 2. At the encoder, 1. The residual signal of a frame from input block and Intra/Inter prediction is divided into square blocks of size N N where N=2 M and M is an integer. 2. Each residual block (U) is then input to a two-dimensional N N forward transform. 3. The two-dimensional transform can be implemented as a separable transform by applying an N-point one-dimensional transform to each row and each column separately. C shown in Figure 2 is the transform matrix. 4. The resulting transform coefficients are then subjected to quantization. It is a process in which they are divided by quantization step size Qstep) to obtain quantized transform coefficients level.

6 are designed to achieve near lossless reconstruction of the input residual block when joined without the intermediate quantization and de-quantization steps. In video coding standards such as HEVC, the de-quantization process and inverse transforms are specified. Discrete Cosine Transform Figure 1: Block-based hybrid video coding (a) Encoder (b) Decoder [1] At the decoder, 1. The quantized transform coefficients are then de-quantized (which is equivalent to multiplication by Qstep). 2. Finally, a two-dimensional N N separable inverse transform is applied to the de-quantized transform coefficients (coeff Q). 3. Thus, resulting in a residual block of quantized samples which are then added to the intra- or inter-prediction samples to obtain the reconstructed block. The N transform coefficients of an N-point 1D DCT applied to the input samples can be expressed as [1] ω i = N 1 j=0 u i c ij (1) Where ω i = N transform coefficients u i = input samples i = 0,...,N-1 Elements c ij of the DCT matrix C are defined as [1] c ij = A cos[ π (j + 1 ) i] (2) N N 2 where i,j = 0,,N-1 and A is equal to 1 and for i=0 and i>0 respectively. Furthermore, the basis vectors c i of the DCT are defined as c i = [c i0,, c i(n 1) ] T where i = 0,..., N-1. [1] The forward and inverse transform matrices are transposes of each other. Typically, they

7 Properties of DCT: i. Real and orthogonal, i.e., C = C C 1 = C T, where C is the cosine transform matrix. ii. Fast transform. iii. Basis vectors provide good energy compaction. iv. Basis vectors have equal norm, i.e., c T i c i = 1. v. The elements of a DCT matrix of size 2 M 2 M are a subset of the elements of a DCT matrix of size 2 M+1 2 M+1. vi. The even basis vectors of DCT are symmetric, while the odd basis vectors are anti-symmetric. vii. DCT is a separable transform, which means it can be represented as a product of two or more 1-D DCTs. Advantages: i. Reconstruction errors are less severe using DCT. Thereby reducing the appearance of blocking artifacts. ii. Fast implementation which helps reduce number of operations as well as computational complexity. iii. It is lossless. The core transforms matrices of HEVC are finite precision approximations of the DCT matrix. The benefit of using finite precision in a video coding standard is that the approximation to the real-valued DCT matrix is specified in the standard rather than being implementation dependent. This avoids encoder-decoder mismatch and drift caused by manufacturers implementing the IDCT with slightly different floating point representations. HEVC Core Transform Design Principles To measure the degree of approximation, the following measures are defined for an integer N-point DCT approximation with scaled matrix elements equal to d ij and basis vectors equal to d i = [d i0,, d i(n 1) ] T where i = 0,..., N-1. [1] 1) Orthogonality measure: (3) 2) Closeness to DCT measure: (4) 3) Norm measure: (5) where i,j = 0,,N-1, c ij are the DCT matrix elements of equation (4), and α in equation (4) is the scale factor which is defined as

8 d 00 N 1 2. The ratio of the two corresponding lengths in two similar geometric figures is called as Scale factor. operation is used to get the final answer positive. In this case, the measurements cannot be taken negative so it is used to get a positive value of the measurement. Division The division of a luma/chroma CB into luma/chroma TBs is performed recursively based on the quadtree approach. This is called the Residual Quadtree (RQT). Here, three important parameters come into play: i. Maximum Bit Depth (d max ) ii. Maximum allowed transform size (n max ) iii. Minimum allowed transform size (n min ) SCANNING There are different types of scans done. Here are some of them: Scan Patterns: In HEVC, the scan in a 4 4 TB is diagonal. The scan in a larger TB is divided into 4 4 sub-blocks and the scan pattern consists of a diagonal scan of the 4 4 sub-blocks and a diagonal scan within each of the 4 4 subblocks. Horizontal and vertical scans may also be applied in the intra case for 4 4 and 8 8 TBs. The horizontal and vertical scans are defined by row-by-row and column-by-column scans, respectively, within the 4 4 subblocks. Figure 3 shows the different types of scan patterns. Figure 3: Diagonal, Horizontal & Vertical Scan Patterns [2] Figure 2: Division of a luma/chroma CB into luma/chroma TBs to get the RQT [2] Scan Passes: Given the scan patterns in HEVC, a coefficient group (CG) corresponds to a 4 4 sub-block. A 4 4 TB consists of exactly one

9 CG. TBs of size 8 8, 16 16, are partitioned into non-overlapping 4 4 CGs. Each scan pass codes a syntax element for the coefficients within a CG, as follows: i. split_transform_flag: When 0, indicates no internal splitting in the TB. When 1, splitting has taken place. ii. rqt_root_cbf (code block flag): Indicates if atleast one non-zero transform coefficient is available to be transmitted. iii. cu_skip_flag: Indicates whether or not the transformation of that CU must be performed. iv. significant_coeff_flag: indicates the significance of each coefficient (zero/nonzero). In each scan pass, a syntax is coded only when necessary as determined by the previous scan passes. For example, if a coefficient is not significant, the remaining scan passes are not necessary for that coefficient. Data processing is localized within a CG and once a CG is fully processed, its coefficient levels can be reconstructed before proceeding to the next one. With this syntax-plane coding approach, syntax elements are separated into different scan passes, thus helping speculative coding algorithms, since the next syntax element to be processed within a scan pass is known. TRANSFORM COEFFICIENT CODING DESIGN v. coeff_abs_level_greater1_flag: indicates whether the absolute value of a coefficient level is greater than 1. vi. vii. coeff_abs_level_greater2_flag: indicates whether the absolute value of a coefficient level is greater than 2. coeff_sign_flag: indicates the sign of a significant coefficient (0: positive, 1: negative). Figure 4: Block diagram of block-based hybrid video coding [3]

10 HEVC introduces several new features and tools for the transform coefficient coding to help improve upon H.264/AVC [1]. Few of them are larger transform block (TB) sizes mode dependent coefficient scanning last significant coefficient coding multilevel significance maps improved significance flag context modeling Sign data hiding. Transform coefficient coding in HEVC strives to achieve a balance between coding efficiency and practicality. So, only if features and tools addressing the practical issues improve coding efficiency or at worst, resulted in a slight degradation, they were adopted. With this in mind, the key principles in the design of transform coefficient coding in HEVC can be summarized as follows. 1) Improve coding efficiency, as this is one of the primary goals of HEVC. 2) Reduce the number of coded bins on average and in the worst-case guarantee a minimum throughput. 3) Increase the percentage of bypass bins and group bypass bins together for higher throughput. 4) Reduce the dependency in context derivation of the current bin on previously coded bins. 5) Avoid interleaving syntax elements; a serial dependency exists if a syntax element depends on the value of the previous syntax element. 6) Reduce the number of contexts: in H.264/AVC, the number of contexts used for coefficient coding is a high percentage of the total number of contexts. 7) Simplify scans for the hardware and SIMD implementation. 8) Simplify and modularize the coding of large TBs. Basis Vectors in DCT Using the basis matrix each of the remaining sub-matrices can be derived. (subset property). The right half of the basis matrix can be derived from its left half. (Symmetry property). The inverse transform matrix of HEVC is simply the transpose of the forward transform matrix. The coefficients of the smaller basis matrices (16 16, 8 8 and 4 4) can be derived from the coefficients of the basis matrix using the formula:[1]

11 d N 32 ij = d 32 i( N ),j, i, j = 0,., N 1 (6) Figure 5: The left half of the basis matrix approximation of DCT used in HEVC [1] Figure 7: (a) The 4 4 basis matrix approximation of DCT used in HEVC [5] (b) Norm of the 4 4 basis matrix approximation of DCT used in HEVC The 8x8, 16x16 and 32x32 basis vectors can be found in appendix A, while the norms of the 8x8 and 16x16 basis vectors can be found in appendix B. Larger Coding Tree Block and Larger Transform Unit Figure 6: The 4 4 basis matrix approximation of DCT used in HEVC [1] Norms of Basis Vectors The basis vectors of DCT have equal norm, i.e.,c T i C i = 1, where C is the cosine transform matrix In HEVC, up to 32x32 transform is supported. However, for videos of HD resolution or higher, larger transform matrices can help improve the transform coding efficiency. Here, a 64x64 DCT matrix is discussed which is an extension of the transform matrices used in HEVC. If, computational complexity needs to be reduced, the top-left 32x32 coefficients can be kept, and the rest can be zeroed out. The integer transform matrix Ti,j where ii, jj (0,

12 63) is derived by scaling the DCT-II matrix by S, followed by rounding.[9] T i,j = [S. w 0 2 π.i.(2j+1) cos + 0.5] (7) N 128 Where w 0 = { 0.5, i = 0 1, otherwise And S is the scaling factor whose value is to better maintain the orthogonality of the transform matrix. Also, in order to keep the intermediate values of the transformed coefficients within the 16-bit range, all coefficients are right shifted by 2 bits after horizontal and after vertical transform, in contrast to the right shift used presently in HEVC transforms.[9] The 64x64 DCT transform matrix has been derived using the formula in (7) and included in appendix C. HEVC Transform Operation Before and after DCT/IDCT transform the data must be limited to a maximum of 116 bits (including the sign bit). Truncation of log 2 N 1 and log 2 N + 6 bits after first and second forward transforms respectively.[3] Similarly, resulting coefficients after the inverse transforms are also scaled down by the fixed scaling factor of 7 and 12. [3] Figure 9: Detailed implementation of HEVC integer transform and quantization [1] Figure 8: Encoding and decoding chain involving DCT in HEVC [3]

13 additions. Similarly, for 2-D transformation, number of multiplications required is 2N 3 and number of additions required are 2N 2 (N 1). However, by using the antisymmetry properties that every basis vector inherits from DCT, we can greatly reduce the number of arithmetic operations for each of these transformations. Example: Figure 10: Example of forward and inverse transform and scaling in HEVC [1] Complexity Analysis Arithmetic Analysis With simple matrix multiplication, number of operations required to achieve 1-D inverse transformation is N 2 multiplications and In HEVC, the direct 1-D 4-point transformation would require 16 multiplications and 12 additions. The 2-D transformation will require 128 multiplications and 96 additions. However, using even-odd decomposition, 1-D transformation requires 6 multiplications and 8 additions while the 2-D transformation requires 48 multiplications and 64 additions. This translates to 62.5% savings in the number of multiplications and 33.3% savings in the number of additions. Next, the direct 1- D 8-point transformation would require 64 multiplications and 56 additions. The 2-D transformation would require 1024 multiplications and 896 additions. However, using even-odd decomposition, 1-D transformation requires 22 multiplications and 28 additions while the 2-D transformation requires 352 multiplications and 448 additions.

14 Hardware Analysis In modern day devices it has become necessary to support both video capture and playback. This requires both forward and inverse transforms to be implemented on the same device. One such technique is called unified forward-inverse transform. This technique uses the symmetry between the forward and inverse transforms to share hardware while performing transformation operations. On implementing this technique in RTL for a throughput of one 32-point 1D transform per cycle on a 45-nm library, it is seen that unified implementation requires about 44% less area than a separate implementation of forward and inverse transforms. Figure 12: Equations for 4-point DCT [1] Figure 11: 4-point DCT implementation [1] Figure 13: 32-point inverse transform unified architecture [1] Here, we see the architecture of the 4-point inverse transform. The even matrix multiplication is denoted as Even4 and the odd matrix multiplication is denoted as Odd4. The outputs of the Even4 and Odd4 blocks are added and subtracted to get the 4-point inverse transform output. The

15 addition/subtraction network is denoted as AddSub4. The even part of the transform is exactly the 4-point inverse transform. The odd part of the transform is denoted by Odd8. Similar to the 4-point inverse transform, the output of the even and odd parts of the transform are added and subtracted to get the 32-point inverse transform architecture. CONCLUSIONS In this project, a very important and integral section of HEVC encoding and decoding has been explained i.e., HEVC transform. The various processes involved in transforming the residual block of data were discussed. Furthermore, the design of HEVC transform matrices, the basis vectors and the importance of DCT in HEVC transform were explained in detail. In the end, we examine the complexity of implementation of HEVC transform and discuss how it can be simplified and made more feasible using unified architecture. The implementations of unified architecture were synthesized in a 45-nm library at 250MHz. It is seen that the unified implementation requires around 44% less area than a separate implementation. Hardware area savings at other frequencies are in the range of 43 45%. [1] REFERENCES [1] M. Budagavi et al, Core transform design in the High Efficiency Video Coding (HEVC) standard, IEEE J. Sel. Topics in Signal Processing, vol. 7, no. 6, pp , Dec [2] Fraunhofer, Transform Coding Using the Residual Quadtree (RQT), [Online]. Available: deo-coding-analytics/researchgroups/image-video-coding/researchtopics/transform-coding-using-the-residualquadtree-rqt.html [3] P.K. Meher et al, Efficient Integer DCT Architectures for HEVC, IEEE Transactions on Circuits and Systems for Video Technology, vol. 24, no. 1, pp , Jan [4] J. Sole et al, Transform coefficient coding in HEVC, IEEE Trans. Circuits Syst. Video Technol., vol. 22, no. 12, pp , Dec

16 [5] A. K. Jain, Fundamentals of Digital Image Processing, Englewood Cliffs, NJ: Prentice Hall, pp , [6] Y. Wang, "DCT and Transform Coding," Polytechnic University, [Online]. Available: oding_dct.pdf. [7] HEVC Test Model HM-16.3, Feb [Online]. Available: hevc.hhi.fraunhofer.de/svn/ svn_hevcsoftware/tags/hm-16.3/ [8] V. Sze, M. Budagavi, and G. J. Sullivan, High Efficiency Video Coding (HEVC): Algorithms and Architectures, Springer-Verlag, New York, USA, pp , [9] J. Chen et al, Coding tools investigation for next generation video coding based on HEVC, [ ], SPIE. Optics + photonics, San Diego, California, USA, 9 13, Aug [10] T. Nguyen et al, Improved context modeling for coding quantized transform coefficients in video compression, in Proc. Picture Coding Symp., pp , Dec [11] H. S. Malvar et al, Low complexity transform and quantization in H.264/AVC, IEEE Trans. Circuits Syst. for Video Technol., vol. 13, no. 7, pp , July [12] W. Hwangbo and C. M. Kyung, A multitransform architecture for H.264/AVC high-profile coders, IEEE Trans. Multimedia, vol. 12, no. 3, pp , Apr [13] M. Winken et al, Transform coding in the HEVC test model, in Proc. IEEE ICIP, pp , Sept [14] M. Budagavi and V. Sze, Unified forward + inverse architecture for HEVC, in Proc. IEEE Int. Conf. Image Process., pp , Sept [15] J. Dong et al, 2-D order-16 integer transforms for HD video coding, IEEE Trans. Circuits Syst. Video Technol., vol. 19, no. 10, pp , Oct [16] J. S. Park et al, 2-D large inverse transform (16x16, 32x32) for HEVC (high efficiency video coding), J. Semicond. Technol. Sci., vol. 12, no. 2, pp , June [17] W. Han et al, Improved video compression efficiency through flexible unit

17 representation and corresponding extension of coding tools, IEEE Trans. Circuit Syst. Video Technol., vol. 20, no. 12, pp , Dec [23] K. R. Rao and P. Yip, Discrete Cosine Transform: Algorithms, Advantages, Applications, San Diego, CA: Academic, [18] F. Bossen et al, HEVC complexity and implementation analysis, IEEE Trans. Circuits Syst. for Video Technol., vol. 22, no. 12, pp , Dec [19] N. Ling, High efficiency video coding and its 3D extension: A research perspective, Keynote Speech, ICIEA, Singapore, July [20] M. Wien, HEVC coding tools and specifications, Tutorial, IEEE ICME, San Jose, CA, July [21] D. Grois, B. Bross and D. Marpe, HEVC/H.265 Video Coding Standard (Version 2) including the Range Extensions, Scalable Extensions, and Multiview Extensions, (Tutorial), IEEE ICIP, Quebec City, Canada, Sept Access online: d/public.php?service=files&t=8edc97d26d4 6d4458a9c1a17964bf881. Password: a2fazmgnk. [22] M. Wien, High Efficiency Video Coding: Coding Tools and Specification, Springer, 2014.

18 Accessing JCT-VC Document Database In an internet browser, visit Create an account by clicking the link shown in the web page. Enter your details, and put THE UNIVERSITY OF TEXAS AT ARLINGTON in the organization field and also put in the appropriate motivation like STUDIES/ COURSEWWORK, etc. Check for suggesting the registration was successful. You are good to access the content like ALL MEETINGS and NEXT MEETING.

19 Appendix A The 8 8 basis matrix approximation of DCT used in HEVC [5] {64, 64, 64, 64, 64, 64, 64, 64} {89, 75, 50, 18,-18,-50,-75,-89} {83, 36,-36,-83,-83,-36, 36, 83} {75,-18,-89,-50, 50, 89, 18,-75} {64,-64,-64, 64, 64,-64,-64, 64} {50,-89, 18, 75,-75,-18, 89,-50} {36,-83, 83,-36,-36, 83,-83, 36} {18,-50, 75,-89, 89,-75, 50,-18} The basis matrix approximation of DCT used in HEVC [5] { } { } { } { } { } { } { } { } { } { } { } { } { } { } { } { } The basis matrix approximation of DCT used in HEVC [5] { } { } { } { } { } { } { } { } { } { } { } { } { } { } { } { } { } { } { } { } { } { } { } { } { } { } { } { } { } { } { } { }

20 Appendix B Figure 14: Norm of the 8 8 basis matrix approximation of DCT used in HEVC Figure 15: Norm of the basis matrix approximation of DCT used in HEVC

21 Appendix C C++ program code to derive 64x64 DCT approximated basis vector matrix #include <stdio.h> #include <conio.h> #include <math.h> main () { float mat[64][64]; int i,j,s=2048; float c=pow( ,0.5); float k; } } for(i=0;i<64;i++) { for(j=0;j<64;j++) { k=(((2*j)+1)*3.14*i)/128; if (i=0) float w=pow(0.5,0.5); else float w=1; mat[i][j]=s*w*c*cos(k)+0.5; printf("%f",mat[i][j]); printf("\t"); } printf("\n"); getch();

22 The 64x64 DCT approximated basis vector matrix Row Row Row 3

23 Row Row

24 Row Row

25 Row Row

26 Row Row

27 Row Row Row

28 Row Row

29 Row Row

30 Row Row

31 Row Row Row

32 Row Row

33 Row Row

34 Row Row

35 Row Row Row

36 Row Row