Fast transforms for Intra-prediction-based image and video coding

Transcription

1 03 Data Compression Conference Fast transforms for Intrapredictionbased image and video coding Ankur Saxena, Felix C. Fernandes Samsung Telecommunications America 30 E. Lookout Dr, Richardson, TX 7508, USA Yuriy A. Reznik InterDigital Communications 970 Scranton Rd, San Diego, CA 9, USA Abstract In this paper, we provide an overview of the DCT/DST transform scheme for intra coding in the HEVC standard. A unique feature of this scheme is the use of DSTVII transforms in addition to DCTII. We further derive factorizations for fast joint computation of DCTII and DSTVII transforms of several sizes. Simulation results for the DCT/DST scheme in the HM reference software for HEVC are also provided together with a discussion on computational complexity. I. INTRODUCTION Most image and videocoding standards such as JPEG, MPEG, MPEG4 AVC/H.64, employ blockbased transform coding as a framework for efficient image and video compression. The pixel domain data is transformed to frequency domain using a transform process on a blockbyblock basis. For typical images, most of the energy is concentrated in lowfrequency transform coefficients. Following the transform, a relatively large stepsize quantizer can be used for higherfrequency transform coefficients in order to compact energy more efficiently and to attain better compression. Hence it is required to devise the optimal transform for each image block to fully decorrelate the transform coefficients. The KarhunenLoeve Transform (KLT is known to be optimal under certain conditions for transform coefficients. However, practical use of KLT is limited due to its high computational complexity. It is well known that Discrete Cosine Transform of Type II (DCTII provides an attractive alternative to KLT in terms of decorrelation and energy compaction [3] and performance close to KLT. But with the advent of intra prediction, this is no longer the case and the optimal transform should adapt to the intra prediction mode. Recently, Han, Saxena & Rose [7] analytically derived Discrete Sine Transform of TypeVII (DSTVII with frequency and phase components different from the conventional DCT to be the actual KLT along the prediction direction for intra modes in H.64/AVC. They also showed that if prediction is not performed along a particular direction, then DCT performs close to KLT. The idea was applied to the vertical and horizontal modes in intraprediction in H.64/AVC and a combination of the DST and conventional DCT was used for the vertical and horizontal modes. Saxena & Fernandes in [3], [5] showed that a combination of DCTII and DSTVII is indeed the optimal transform for all the oblique modes in unified intra prediction [9] in the HEVC standardization. The DCT/DST transform scheme described in this paper was also adopted in the HEVC standardization for block size 4x4 for Intra Luma blocks [7]. Recently, a simplification of the DCT/DST technology [8] was adopted in HEVC which removes the modedependency of whether to select between DCT or DST as the horizontal (and respectively vertical transform, and DST was always used for the 4x4 Intra Luma blocks. In the literature, until recently there were no wellknown algorithms for finding fast implementations of the DSTVII similar to various fast implementations for DCT and DST Types I to IV. Recently, Chivukula and Reznik have presented a technique for finding fast implementations for DSTVII and DSTVI (transpose of DSTVII in [5], where they show that the 4x4 DST transform matrix can be In the remainder of the paper, unless otherwise specified, DCT and DST are referred to as DCTII and DSTVII respectively /3 $ IEEE DOI 0.09/DCC

2 Fig.. Examples of Category oblique modes are shown in (a and (b where prediction is performed from one direction only: either the top row or left colu. (c shows example of Category oblique mode where prediction is performed from both directions top row and left colu. (a (b Fig.. (a Vertical Intra prediction mode, and (b DCT and DST basis vectors. For DCT, the first basis vector is flat. For DST, the first basis vector is zero at one boundary, and maximum at the other boundary. This mimics the prediction error residue behavior for vertical mode more efficiently as compared to the DCT basis vector. implemented using 5 multiplications. Note that, by contrast the 4x4 DCT in HM, the Test Model for HEVC standardization is implemented via 6 multiplications. In this paper, we extend study of factorization techniques started in [5], and show that for certain sizes DCTII and DSTVII transforms allow joint factorizations. Such embedded joint factorizations can be of interest for efficient hardware implementations. The rest of the paper is organized as follows: Sec. II outlines the details of unified intra prediction, and the discussion about the DCT/DST transform scheme in the HEVC standardization along with experimental results. Sec. III discusses fast factorization of DCTII and DSTVII transforms, followed by Conclusions in Sec. IV. II. INTRAPREDICTION CODING MODES AND TRANSFORMS IN HEVC STANDARD A. Review of Unified Intra Prediction The ongoing HEVC standardization uses unified intra prediction in which up to 35 different intraprediction modes for the Luma component of the video at a particular block size can be defined. In general, these 35 intraprediction modes can be divided into 3 categories as follows: Category oblique modes (Fig. : Here prediction is performed from the decoded pixels from either the top row or the left colu. The vertical mode and the horizontal mode in [9] are special cases of this oblique mode when prediction direction is vertical or horizontal respectively. Category oblique modes (Fig. : Here prediction is performed from both the top row and the left colu pixels. 3 DC and Planar modes: These are nondirectional modes, and can be used for fitting flat regions in case of DC mode, or surface fitting in case of Planar mode. 4

3 T int DSTVII T int DCTII [ 8 8 ( ] π(i +(j + sin [ N λ(icos where : λ(i ( ] πi(j + N [, if i 0 0, otherwise i,j0,...,n N4. i,j0,...,n N4 Fig. 3. Integer transform matrices defined in HEVC standard and their relationship to 4point DSTVII and DCTII transforms. B. Transforms in HEVC for intra prediction modes In the HEVC, standard, the DCT/DST transform scheme described was adopted in [7] for block size 4x4 for Intra Luma blocks. As an example, we briefly describe why DST is the optimal transform rather than DCT for some prediction modes. Specifically, consider the Vertical prediction mode in Fig. (a, which is a special case of Category Oblique modes. Here, after prediction from the top row pixel x 0j (or its reconstruction, the energy in the prediction residues x ij (for j {,N}, and for i {,N} increases as we go down the colu. Now consider, the basis vectors of DCTII and DSTVII in Fig. (b. For DCT, the first basis vector is flat, while for DST, the first basis vector increases as we go from left to right, with zero at the left boundary, and maxima at the right boundary. Hence, the first basis vector of DST mimics the prediction error residue behavior more efficiently as compared to the DCT basis vector, and DST would be a better transform to compress such a kind of residue. For a detailed mathematical derivation of this, we refer the reader to [7] and [3]. We also show the exact integer based approximations of DCTII and DSTVII transform matrices being used in the HEVC standardization in Fig. 3. C. Performance of DCT/DST Intracoding scheme Here, we present the results when the DCT/DST scheme is applied at size 4x4 in the HEVC Test Model. Note that, additional gains of applying the DCT/DST scheme on 8x8 to 3x3 blocks are minimal given the high implementation complexity of 8x8 and 3x3 DST in hardware, and can be found in [6]. In fact, a different category of transforms: secondary transforms can provide further gains for higher order blocks, and we refer the reader to [4] for more details about the secondary transforms. The DCT/DST scheme is only applied to Intra Transform Units and Luma component. For Chroma and Inter Transform Units, the conventional DCT is always used. For our simulations, we encoded full length sequences (which had 50 to 600 frames at various resolutions varying from 46x40 to 560x600. These video sequences constitute the designated test set for the HEVC standardization. Full details about the GOP size, Intra period, coding structure of these video sequences etc. are available in []. Note that we present the results for only the evaluation of the DCT/DST as an alternate transform here instead of DCT and retain all the other testing settings as in []. We present the results in HM 7.0, the seventh version of the Test Model for HEVC standardization for the following 4 settings: Intra High Efficiency0 bits, IntraMain, Random Access High Efficiency0 bits, and Random Access Main as stipulated in the Common Test Conditions []. Table I show the average BjøntegaardDelta rate (BDrate [] gains in Luma for the DCT/DST scheme in HM 7.0. Classes A to E include the various test sequences that are the designated test set for the HEVC standardization. More details about these sequences is in []. For Class E sequences, simulations were not performed for Random Access conditions as stipulated in the common test conditions for the 5

4 TABLE I BDRATE [] GAINS WHEN THE DCT/DST SCHEME IS APPLIED TO 4X4 INTRA LUMA BLOCKS FOR VARIOUS VIDEO SEQUENCES FOR THE INTRA HIGH EFFICIENCY (HE0 BITS, INTRA MAIN, RANDOM ACCESS HE0 BITS AND RANDOM ACCESS MAIN SETTINGS IN HM 7.0. NOTE THAT NEGATIVE BDRATE MEANS COMPRESSION GAIN. Sequence Name Intra Main Intra HE 0 RA HE 0 RA Main Class A Traffic x600 PeopleOnStreet Nebuta StreamLocomotive Class B Kimono x080 ParkScene Cactus BasketballDrive BQTerrace Class C BasketballDrill x480 BQMall PartyScene RaceHorsesC Class D BasketBallPass x40 BQSquare BlowingBubbles RaceHorsesD Class E FourPeople.4.5 N/A N/A 80x70 Johnny.0.3 N/A N/A KristenAndSara.7. N/A N/A Summary Average Enc. Time (% Dec. Time (% HEVC standardization. In these simulations, the comparison was made within (a HM 7.0 with the modedependent DCT/DST scheme disabled, and (b HM 7.0 with the modedependent DCT/DST scheme enabled. From these set of test results, the average gain for the Luma component when the DCT/DST scheme for the Intra High Efficiency0 bits and Intra Main settings is 0.8%, and 0.9% respectively. For the Random Access High Efficiency0 bits, and Random Access Main settings, the average gains are around 0.3 %. Note that unlike H.64/AVC days, most of the tools adopted in the HEVC standard provide gains only in the range of 0.5% to.5%, and not in the range of 50%. The average encoding and decoding times for the HM codec remain almost the same by using DST as an alternate transform than DCT, since there is only one condition to be checked (intra prediction mode and decide whether to apply the DCT or DST. Finally, note that in the HEVC standardization, recently a simplification of whether to apply DCT or DST as the horizontal (respectively vertical transform for DC and Category Oblique modes was adopted [9]. More specifically, it was proposed to always use DST as the horizontal and vertical transform. It was shown that such a simplification brings around 0. % loss in average BDRates, but does not require a switching logic to choose between DCT and DST in the codec depending on the intra prediction mode. This may possibly improve the throughput in a hardware codec. III. FAST FACTORIZATIONS OF DCT/DST TRANSFORMS The purpose of this section is twofold. On one hand, we would like to show that Intraprediction transforms adopted in HEVC standard allow fast factorizations, and show example factorizations for 6

5 transforms of length N 4. On the other hand, we also would like to discuss relationships between DCTII and DSTVII transforms of different sizes and show several possible joint factorizations of such transforms. Such factorizations may be of interest for efficient hardware implementations and future uses in video coding. For background information and prior results regarding factorizations of DCTII and DSTVII transforms the reader is referred to [3], [5], [8], [0], [0], []. New results presented in this section include: a decomposition of an point DCTII into an N +point DCTVI and Npoint DSTVII, b fast algorithms for computing DCTVI and DCTVII transforms, c computation of DCTII of composite sizes k N( utilizing Npoint DCTII and DSTVII as building blocks. A. Notation Let N be the length of data sequence. The matrices of Discrete Fourier Transform (DFT and Discrete Cosine and Sine transforms of types II, III, VI, and VII will be denoted as follows: DFT: [F N ] e DCTII: C II N cos DCTIII: C III N cos DCTVI: C VI N+ cos DCTVII: C VII N+ cos DSTVI: S VI N sin DSTVII: S VII N sin π j ( m(n+π N ( (m+nπ N ( m(n+π N+ ( (m+nπ N+ ( (m+(n+π N+ ( (m+(n+π N+ N, m,n 0,...,N, m,n 0,...,N ;, m,n 0,...,N ;, m,n 0,...,N;, m,n 0,...,N;, m,n 0,...,N ;, m,n 0,...,N ; In the above definitions, we have intentionally omitted normalization constants (such as /N and λ i (i 0?/ :conventionally used in DCTII as they don t affect factorization structures of the transforms. Subindices N or N + indicate lengths of transforms. We follow Z.Wang and B.R.Hunt s convention coupling Npoint DSTVI/VII with N +point DCTVI/VII [0]. As easily noticed, transforms of types II and III, as well as VI and VII are closely related: ( C II N T C III N ; ( C VI N+ T C VII N+; ( S VI T N S VII N. B. Relationship between point DCTII, N +point DCTVI, and Npoint DSTVII transforms In this section we prove the following statement. Theorem. The following holds: CN+ II P N+ CVI N+ I N J N ( J N I N where P N+ is a matrix, such that when applied to a vector x, it produces the following sign alterations and reordering: x i x i i 0,...,N, x i+ ( i ( x N++i i 0,...,N, and I N and J N are N N identity and orderreversal matrices respectively. S VII N 7

6 x0 x y0 y Y0 Y X0 X x0 x Npoint DCTII X0 X x y Y X4 x N+point DCTII X N+ point DCTVI Npoint DCTII Npoint DSTVII xn xn xn xn+ xn+ yn yn yn y0 y YN YN YN Y0 Y Y XN XN XN X X3 X5 Input index mapping Npoint DCTII N+point DCTII Npoint DSTVII Postadditions and output index mapping xn xn xn yn3 yn yn N point DSTVII YN3 YN YN XN5 XN3 XN xn(n+3 xn(n+ xn(n+ Npoint DCTII Npoint DCTII N+point DCTII Npoint DSTVII xn(n+3 xn(n+ xn(n+ Fig. 4. Computing DCTII of composite sizes: (a split of point DCTII into an N +point DCTVI and Npoint DSTVI; (b computation of DCTII of length N(. Proof: Let us consider a long input sequence x x 0,...,x N, and apply DCTII over it: k We first look at even output values (k i, i 0,...,N: i We can split this sum as follows: N i N N N π(n +k (, k 0,...,N. π(n +i ( π(n +i π(n +i π(n +i π(n +i nn+ nn+ nn+ N + x N n cos π(n +i π(n +i. π(( (n +i π((n n+i π(n +i, 8

7 which implies that i X C VI i + X C VI i where X C VI is a DCTVI transform over the first N + elements of input sequence x, and X C VI is a DCTVI transform over the following input: [ x xn n, if n 0,...,N, n 0, if n N. We now turn out attention to the odd output values (k i +, i 0,...,N : i+ We can split this sum as follows: [ N i+ ( i+ x N n sin which implies that where X S VII and ˆX S VII ( i x n sin ( i x n sin [ N ( i+ x N++n sin π(n + (i + ( ( i+ x N n sin π(n n(i + π(n + (i + π(n ++N + (i + ( π(n + N + (i + π(n n(i + ] π(n n(i + + x N n sin nn+ ] N π(n + (i + x N n sin, i+ ( i+ VII XS i +( i i VII ˆXS i is an Npoint DSTVII transform over a sequence x n x N++n, n 0,...,N, is an Npoint DSTVII transform over a sequence ˆx n x N n, n 0,...,N. By combining all these mappings we arrive at expression (. We present a flowgraph of the resulting mapping between DCTII, DCTVI, and DSTVII transforms in Figure 4.a. Only permutations and sign changes are needed to convert output of DCTVI, and DSTVII into DCTII. 9

8 ( π ( π ( π ( π ( π ( π ( π ( π ( π ( π ( π ( π ( π ( π ( π ( π Fig. 5. Flowgraph of Winograd s factorization of DFT of length 9, and flowgraphs of 9point DCTII, 5point DCTVI, and 4point DSTVII implied by mappings (,3. C. Fast algorithms for computing DCTII, DCTVI, and DSTVII From ( we already know that fast computation of DCTVI and DSTVII can be reduced to computing subsets of DCTII. In turn, thanks to Heideman[8] it is also known that computation of DCTII of odd numbers is equivalent to computing samelength DFT. Considering length transforms, we can summarize Heideman s result as follows: ( CN+ II [R(FN+ ] H rows 0,...,N H [I(F N+ ], (3 rows N+,...,N where R (F N+ and I (F N+ denote real and imaginary parts of the DFT transform matrix of size, and H and H are permutation and signinversion matrices [8]. In combination with ( this formula shows that an N +point DCTVI, an Npoint DSTVII and an point DCTII can be computed by mapping to a point DFT. Many algorithms for fast computing of DFT are readily available [4]. For example, in case of N 4, we can use Winograd DFT module of length 9 shown in Figure 5.a. By using mappings (3 and ( we adopt it to produce 9point DCTII, 5point DCTVI, and 4point DSTVII. This is shown in Figure 5.b. We note that all the resulting algorithms are very efficient in terms of multiplicative complexity. Thus, obtained 9point DCTII requires only 8 nontrivial multiplications. In contrast, the least complex algorithms for computing DCTII of size 8 (nearest dyadicsize requires multiplications[]. The obtained 4point DSTVII is also very efficient: it uses only 5 multiplications. This DCTVII factorization is immediately suitable for implementing transform in HEVC video codec. We show 4point integer transforms adopted in HEVC standard in Figure 6. In same picture, we also show flowgraphs of the the corresponding DCTII and DSTVII transforms, and modified versions of these flowgraphs allowing computation of integer transforms defined in HEVC standard. D. Fast computation of transforms of lengths k N( It is known that a transform of a composite length N pq, where p and q are coprime, can be decomposed into a cascade of pqpoint transforms and qppoint transforms followed by pq p q additions. This class of techniques is called Prime Factor Algorithms (PFA [], []. In Figure 4.b, we show how to compute DCTII of length N(. This factorization includes Npoint DCTII subtransforms, and additionally N point DCTII transforms, which, in 0

9 X x0 64 X x x X x 3 X x 36 ( 8 x cos π X x X x3 3 ( sin π sin ( π 8 8 ~ 8 C II x 3 X x cos ( π 8 (a (b (c X x 0 0 X x X x X x3 VII ~ 8 S4 x 0 x x 3 x x ( 9 3 (d (e (f Fig. 6. Factorizations of 4point HEVC transforms: (a and (d show integer transform matrices defined by the standard, (c and (f show factorizations of DSTVII and DCTII transforms, (b and (e show how same factorizations can be applied to compute integer transforms. 64 X 0 X X 3 X 0 X x 0 x x 3 x x 0 sin 4 ( π 9 9 sin ( π 3 9 sin ( π 9 9 sin ( π 9 9 X 0 X X X 3 X X 3 X 0 X turn include Npoint DSTVII as part of their flowgraph. Hence, a system that implements and uses N point DCTII and DSTVII, can easily compute an N( transform by reusing them. Same principle more generally applies to computing transforms of lengths k N(. We note, that in addition to ability to reuse Npoint DCTII and DSTVII, transforms of lengths k N( may be considerably faster than transforms of nearest dyadic sizes. We illustrate this in Table. Here, we show complexities of factorizations of DCTII of lengths 36 and 7 and show that they are less complex (in terms of multiplications that fastest known (FeigWinograd [6] algorithms for computing DCTII of lengths 3 and 64 respectively. TABLE II COMPARISON OF SEVERAL EXISTING AND PROPOSED FACTORIZATIONS OF DCT AND DST TRANSFORMS. Transform N Algorithm Features Complexity Matrixvector product DCTII 4 Loeffler, et al. [] 4 muls, 9 adds 6 muls, adds DSTVII 4 this paper, [5] 5 muls, adds 6 muls, adds DCTVI 5 this paper 3 muls, 5 adds 5 muls, 0 adds DCTII 8 Loeffler, et al. [] muls, 9 adds 64 muls, 56 adds DCTII 9 this paper, Heideman [8] uses 4point DSTVII 8 muls, 34 adds 8 muls, 7 adds DCTII 3 Feig,Winograd [6] 76 muls, 09 adds 04 muls, 99 adds DCTII 36 this paper uses 4point DSTVII 68 muls, 39 adds 96 muls, 60 adds DCTII 64 Feig,Winograd [6] 84 muls, 53 adds 4096 muls, 403 adds DCTII 7 this paper uses 4point DSTVII 63 muls, 587 adds 584 muls, 5 adds IV. CONCLUSIONS In this paper, we have provided an overview of the DCT/DST transform scheme for intra coding in the HEVC standard. We have shown that such transforms allow efficient factorizations by utilizing mappings between DSTVII, DCTII, and DFT. We have also produced several results on joint computation of DCTII and DSTVII transforms of several sizes, which might be of interest for implementation of future video coding algorithms. REFERENCES [] G. Bjøntegaard, Calculation of Average PSNR Differences between RD curves, ITUT SG6/Q6, VCEGM33, April 00.

10 [] F. Bossen, Common HM test conditions and software reference configurations, ITUT and ISO/IEC JCTVCI00, May 0. [3] V. Britanak, K. R. Rao, and P. Yip, Discrete Cosine and Sine Transforms: General Properties, Fast Algorithms and Integer Approximations. Oxford, UK: Academic Press Elsevier, 007. [4] C. Burrus and T. Parks, DFT/FFT and Convolution Algorithms Theory and Implementation. New York: Wiley, 985. [5] R. K. Chivukula and Y. A. Reznik, Fast computing of discrete cosine and sine transforms of types vi and vii, in Applications of Digital Image Processing XXXIV, ser. Proc. SPIE, A. G. Tescher, Ed., vol. 835, 0, pp [6] E. Feig and S. Winograd, On the multiplicative complexity of discrete cosine transforms (corresp., IEEE Trans. Info. Theory, vol. IT38, pp , 99. [7] J. Han, A. Saxena, and K. Rose, Towards jointly optimal spatial prediction and adaptive transform in video/image coding, in IEEE ICASSP, March 00, pp [8] M. T. Heideman, Computation of an oddlength DCT from a realvalued DFT of the same length, IEEE Trans. Signal Processing, vol. 40, no., pp. 54 6, 99. [9] J. H. Min, S. Lee, IlKoo Kim, WooJin Han, J. Lainema and K. Ugur, Unification of the directional intra prediction methods in TMuC, ITUT and ISO/IEC JCTVCB00, July 00. [0] A. K. Jain, A sinusoidal family of unitary transforms, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. PAMI, no. 4, pp , Oct [] B. G. Lee, Input and output index mapping for a primefactor decomposed computation of discrete cosine transform, IEEE Trans. Acoust., Speech, Signal Processing, vol. 37, no., pp , Feb [] C. Loeffler, A. Ligtenberg, and G. Moschytz, Practical fast D DCT algorithms with multiplications, in IEEE Int. Conf. Acoust., Speech, Signal Processing (ICASSP, vol., May 989, pp [3] A. Saxena and F. Fernandes, Mode Dependent DCT/DST for intra prediction in blockbased image/video coding, in IEEE ICIP, Sept 0, pp [4], On secondary transforms for intra prediction residual, in IEEE ICASSP, March 0, pp [5], Jointly optimal intra prediction and adaptive primary transform, ITUT and ISO/IEC JCTVCC08, Oct 00. [6], Modedependent DCT/DST for intraprediction in video coding, ITUT and ISO/IEC JCTVCD033, Jan 0. [7], Modedependent DCT/DST without 4*4 full matrix multiplication for intra prediction, ITUT and ISO/IEC JCTVCE5, March 0. [8] K. Ugur and O. Bici, Performance evaluation of DST in intra prediction, ITUT & ISO/IEC JCTVCI058, May 0. [9] K. Ugur and A. Saxena, Summary report of Core Experiment on intra transform mode dependency simplifications, ITUT & ISO/IEC JCTVCJ00, July 0. [0] Z. Wang and B. R. Hunt, The discrete W transform, Applied Mathematics and Computation, vol. 6, no., pp. 9 48, 985. [] S. Winograd, On computing the Discrete Fourier Transform, Proc. National Academy of Sciences, USA, vol. 73, pp , 976. [] P. Yang and M. Narasimha, Prime factor decomposition of the discrete cosine transform, in IEEE Int. Conf. Acoust., Speech, Signal Processing (ICASSP, March 985, pp. pp