Performance analysis of Integer DCT of different block sizes used in H.264, AVS China and WMV9.

Transcription

1 Performance analysis of Integer DCT of different block sizes used in H.264, AVS China and WMV9. Aim: To investigate performance analysis of integer DCT of block sizes 8X8, 16X16 and 32X32 used in H.264, AVS China and WMV9. Abstract: Discrete cosine transform (DCT) has been serving as the main component of video coding systems. The integer discrete cosine transform (Int DCT) is an integer approximation of the discrete cosine transform. It can be implemented exclusively with integer arithmetic. It proves to be highly advantageous in cost and speed for hardware implementations. In particular, transforms of sizes larger than 4x4 or 8x8, especially 16x16 and 32x32 are proposed because of their increased applicability to the de-correlation of high resolution video signals. For example, order-16 integer transform is simple, low computational complexity transform but has high coding efficiency. This project discusses how the use of larger transforms, especially in high resolution videos, can provide higher coding gain. The DCT-based systems have huge advantage to image applications because they provide a high compression ratio. However, their coding systems are limited to operating in only lossy coding because distortion of decoded image is unavoidable with these lossy algorithms. On the other hand, the integer transform, is becoming popular as a key technique to lossless and lossy unified waveform coding. Especially the integer DCT is attractive as the unified coding comparable to the conventional DCT-based algorithms. Submitted by: Suvinda Mudigere Srikantaiah UTA ID: Suvinda.mudigeresrikantaiah@mavs.uta.edu

2 Introduction: In digital image processing, data compression is necessary to improve efficiency in storage and transmission. Transformation is one popular technique for data compression. By first transforming correlated pixels into weakly correlated ones, and after a ranking in their energy contents, for example, and retaining only the most significant components, high compression ratio is possible. Since inverse transformation is needed to reproduce the original image from the compressed data, it is important that the transform process be simple and fast. The family of orthogonal transforms is well suited for this application because the inverse of an orthogonal matrix is its transpose. The discrete cosine transform (DCT) is widely accepted as having a high efficiency [1]. The DCT matrix elements are real numbers and for a 16-order DCT, 8 bits are needed to represent these numbers in order to ensure perfectly negligible image reconstruction errors due to finite-length number representation. If the transform matrix elements are integers, then it may be possible to have a smaller number of bit representation and at the same time zero truncation errors. Moreover, the resultant cosine values are difficult to approximate in fixed precision integers, thus producing rounding errors in practical applications. Rounding errors can introduce enough error into the computations and alter the orthogonality property of the transform. Using the principle of dyadic symmetry [2] order-8 integer cosine transform (ICT) which has zero truncation errors was introduced. This requires a small number, as little as 2 bit representation and comparable efficiency to the DCT [3]. Briefly, an ICT matrix is in the form I = KJ where I is the orthogonal ICT matrix, and K is a diagonal matrix whose elements take on values that serve to scale the rows of the matrix J so that the relative magnitudes of elements of the ICT matrix I are similar to those in the DCT matrix. The matrix J is orthogonal with elements that are all integers. Integer cosine transforms (ICT) can be generated from DCT-II by replacing the real numbered elements of the DCT-II matrix with integers keeping the relative magnitudes and orthogonal relationship among the matrix elements [6]. The integer transform coefficients result in a computationally less intense procedure that implements similar energy concentration like DCT-II. It can be implemented

3 using integer arithmetic without mismatch between encoder and decoder. The orthogonality of ICT deps on the elements of the transform matrix for orders greater than four. Due to this constraint, the magnitudes of elements t to be quite large for large ICTs [8]. This led to the development of ICTs that are mutually orthogonal by using the principle of dyadic symmetry [5]. Thus the elements in transform matrices can be selected without orthogonality constraint. Transforms used in some standards [4]: Standard Transform 1. MPEG-4 part 10/H X 8, 4 X 4 integer DCT 2. WMV-9 8 X 8, 8 X 4, 4 X 8, 4 X 4 integer DCT 3. AVS China Asymmetric 8 X 8 integer DCT Table no.1: Transforms used in standards H.264 [4], WMV-9 [17] and AVS China [16]. Insight into the project: As part of an image-compression system, the role of the ICT is to de-correlate the picture elements of image blocks for subsequent quantization and entropy encoding. The order-8 ICT was derived using the principle of dyadic symmetry. This concept gives a different development that leads to the order-16 ICT [5]. Equations relating the elements of the ICT matrix so as to satisfy the orthogonality conditions among the columns of the ICT matrix are first written. Then a search method is proposed to find integer solutions for these elements. It is noticed that the coding performance of the Int-DCT is similar to that of the conventional lossy DCT in a low bitrate but it is slightly worse than that of the conventional lossy DCT in a high bit-rate because of rounding errors. The aim of this project is to analyze performance of order-8, order-16 and order-32 integer transforms in H.264, AVS China and WMV9. It is essential to get a good understanding of the three technologies before moving ahead with the analysis. It is also necessary for one to get a good grasp of discrete cosine transform on which integer transform is based. So, in the first section, a brief description of the same is presented. The next section an overview of the coding standards is given. The third section deals with the extension of order 8 transform matrix to order 16 and order 32 transform matrices. The last section discusses performance analysis of order 8, order 16 and order 32 integer transforms in H.264, AVS China and WMV9.

4 I. DCT In images/videos the DCT-II (which represents a signal/image as the sum of cosine functions with different frequencies) is primarily used for transform coding. DCT II is basically a tool that strives to achieve maximum compression efficiency without compromising the quality of video. Definition of DCT and IDCT [11]: The forward discrete cosine transform (DCT) of N samples is formulated by for u = 0, 1,..., N - 1, where The function f(x) represents the value of the x th sample of the input signal. F(u) represents a discrete cosine transformed coefficient for u = 0, 1,, N 1 First of all this transform is applied to the rows, then to the columns of image data matrix. The inverse discrete cosine transform (IDCT) of N samples is formulated by: for x = 0, 1,..., N 1, where

5 This is used for image decompression. The DCT-II is probably the most commonly used form, and is often simply referred to as "the DCT" [6]. Given an input function f(i,j) over two integer variables i and j (a piece of an image), the 2D DCT transforms it into a new function F(u,v), with integer u and v running over the same range as i and j. The general definition of the transform is: where i,u = 0,1,,M 1; j,v = 0,1,, N 1; the constants C(u) (or C(v)) are defined as where l = u,v II. Overview of coding standards: 1. H.264: H.264 video coding standard has the same basic functional elements as previous standards (MPEG-1, MPEG-2, MPEG-4 part 2, H.261, and H.263) [16], i.e., transform for reduction of spatial correlation, quantization for bit rate control, motion compensated prediction for reduction of temporal correlation, and entropy encoding for reduction of statistical correlation. However, to fulfill better coding performance, the important changes in H.264 occur in the details of each functional element by including intra-picture prediction, a new 4 X 4 integer transform, multiple reference pictures, variable block sizes and a quarter pel precision for motion compensation, a deblocking filter, and improved entropy coding. The block diagram of H.264 encoder and decoder is as shown below:

6 Block diagram of H.264 Encoder and Decoder. The H.264/AVC encoder can be understood from its typical structural diagram as illustrated in Figure 2.3. The encoding operation starts by splitting the first frame of a sequence or random access point into macroblocks (MB). This frame is usually intracoded with no use of reference frames. The samples in a block are predicted from previously encoded neighboring blocks. The encoding process selects the best neighboring block and determines how the samples from these blocks are combined for inter prediction. The decoder is notified of this selection process.

7 The decoder performs an inverse of the operations done by the encoder. It inverts the entropy coding process and then using the motion data and the type of prediction performs the prediction process. Inverse scaling and transforming of the residual is also done and the deblocking filter is applied to the result to get the video output. Improved coding efficiency comes at the expense of added complexity to the coder/decoder. H.264 utilizes some methods to reduce the implementation complexity. Multiplier-free integer transform is introduced. Multiplication operation for the exact transform is combined with the multiplication of quantization. 2. AVS China The AVS video coding standard is based on the classic hybrid DPCM-DCT coder, which was first introduced by Jain and Jain in 1979[1]. Temporal redundancy is removed by motioncompensated DPCM coding. Residual spatial redundancy is removed first by spatial prediction, and finally by transform coding. Statistical redundancy is removed by entropy coding. These basic coding tools are enhanced by a set of minor coding tools that remove any remaining redundancy, code side information efficiently and provide syntax for the coded bitstream. The algorithm is highly adaptive, since video data statistics are not stationary and because perceptual coding is also used to maximize perceived quality. The adaptivity is applied at both the Picture layer and the Macroblock layer. The block diagram of AVS China encoder and decoder is as shown in the figure [2]. The encoder shown accepts input video and stores multiple frames in a set of frame buffers. These buffers provide the storage and delay required by multi-frame motion estimation. The motion estimation unit can accept original frames from the input buffers or reconstructed coded frames from the forward and backward reference frame stores in the encoder. Motion estimation produces motion vectors used by the motion compensation unit to produce a forward prediction or interpolated prediction for the current frame. Motion vectors are coded for transmission first by a predictive encoder, and then by entropy encoding.

8 Block diagram of AVS China Encoder The prediction produced by the motion compensation unit is subtracted from the current frame and the difference signal, i.e., the prediction error, is coded by the DCT and quantization units. In the case of intra-coded macroblocks, the data passes through the intra prediction process to the DCT. The signal is then VLC encoded, formatted with the motion vectors and other side information and stored temporarily in the rate buffer. The signal is also decoded by the inverse quantizer and inverse DCT, and stored in the forward or backward frame buffers for subsequent use in motion compensation. The rate buffer smooths the variable data rate produced by coding into a constant rate for storage or transmission. A feedback path from the rate buffer controls the

9 quantizer to prevent buffer overflow. A mode decision unit selects the motion compensation mode for pictures and macroblocks. Block diagram of AVS China Decoder The decoder shown in Figure 2b accepts the constant rate signal from the storage or transmission and stores it temporarily in a rate buffer. The data is read out at a rate demanded by the decoding of each macroblock and picture. The signal is parsed to separate the quantization parameter, motion vectors and other side information from the coded data signal. The signal is then decoded by the inverse quantizer and inverse DCT to reconstruct the prediction error or intra coded data. The quantizer is controlled by the extracted parameter. The motion vectors are decoded, reconstructed and used by the motion compensation unit to produce a prediction for the current picture. This is added to the reconstructed prediction error to produce the output signal. In the case of intra-coded macroblocks, the data passes from the DCT through the intra prediction process.

10 3. WMV 9 Windows Media 9 Series includes a variety of audio and video codecs, which are key components for authoring and playback of digital media. The Windows Media Video 9 (WMV- 9) codec is the latest video codec in this suite and is based on technology that can achieve stateof-the-art compressed video quality from very low bit rates (such as 160_120 at10 Kbps for modem applications) through very high bit rates (1280_720/1920_1080 at4 8 Mbps for highdefinition video, and even higher bit rates for mastering). This section describes in detail the overall structure of the WMV-9 codec, and covers the key innovations critical for its good performance. Block diagram of WMV 9 encoder and decoder

11 The codec uses a block-based motion compensation and spatial transform scheme which, at a high level, is similar to all popular video compression standards since MPEG-1 and H.261. Broadly speaking, these standards as well as WMV-9 perform block-by-block motion compensation from the previous reconstructed frame using a two-dimensional quantity called the motion vector (MV) to signal spatial displacement. A prediction of the current block is formed by looking up a same-sized block in the previous reconstructed frame that is displaced from the current position by the motion vector. Subsequently, the displaced frame difference, or residual error, is computed as the difference between the actual block and its motion-compensated prediction. This residual error is transformed using a linear energy-compacting transform then quantized and entropy coded. On the decoder side, quantized transform coefficients are entropy decoded, dequantized and inverse transformed to produce an approximation of the residual error, which is then added to the motion-compensated prediction to generate the reconstruction. The high level description of the codec is shown in Fig. 3. Needless to say, the above description only provides a high level overview and does not discuss implementation details. III. Extension of order 8 transform matrix to order 16 and order 32 transform matrices Orthogonality property satisfied by transform matrices: Consider H to be a transform matrix. It can be defined as : where k, n= 0,1,..,N-1

12 The elements of the matrices H and are irrational numbers. Thus, the finite bit precision in a computer will not reconstruct the same data if forward and inverse transforms are done in cascade. Moreover, if the forward and the inverse transforms are implemented in different machines with different floating-point representations, the error can be large. Scaling the matrices H and and then rounding it to the nearest integer can negate the errors. But, if the scaling factors are large, the norms of the rows (basis vectors) turn out to be very high and the computational complexity will increase. Thus, an orthogonal (4X4) matrix H with small integer elements is highly desired. This has led to the development of ICTs by the principle of dyadic symmetry. The ICTs maintain the structure such as relative magnitudes, signs, dyadic symmetries, and orthogonality among the elements of the transform matrix. For example, The order-16 transform considered is an exted version of the order-8 ICT adopted in AVS. As shown in (1), T8 is the order-8 transform matrix. Without significant increase in complexity, T8 can be exted to order-16 transform T16 as shown in (2). The normalized basis vectors of T16 have the waveforms similar to that of DCT. (1) (1) T8: Order 8 transform matrix [5].

13 Denoting even symmetry with E and odd symmetry with O about the solid line represents mirror image and negative mirror image. The resulting matrix which is 16X16 is shown below. Same technique is used to ext it to 32X32. (2) (2) T16: Order 16 transform matrix [5].

14 IV. Performance Evaluation: In finding efficiency of integer DCT, standard images are applied as an input signal. Transforms considered will be DCT, Integer DCT of block sizes 8, 16 and 32. The following operations are performed in this project for the purpose of performance analysis: a) Variance distribution for I order Markov process, ρ = 0.9 (Plot and Tabulate) b) Normalized basis restriction error vs. # of basis function (Plot and Tabulate) c) Plot fractional correlation (0<ρ<1) Comparison of performances of 8X8 ICT Code: N=8; rh=0.9; n=1:1:8; for i=1:n for j=1:n R(i,j)= rh^(abs(i-j)); %calculating variances of DCT coefficients dct8= dctmtx(n); dctinv8= transpose(dct8); dctcov= dct8*r*dctinv8; %calculating variances of H.264 integer DCT coefficients h264_mtrx= [ ; ; ; ; ; ; ; ]; h264inv8= inv(h264_mtrx); h264cov= h264_mtrx*r*h264inv8; h264cov=abs(h264cov); %calculating variances of WMV9 integer DCT coefficients wmv9_mtrx= [ ; ; ; ; ; ; ; ]; wmv9inv8= inv(wmv9_mtrx); wmv9cov= wmv9_mtrx*r*wmv9inv8; wmv9cov=abs(wmv9cov);

15 %calculating variances of AVS China integer DCT coefficients avs_mtrx= [ ; ; ; ; ; ; ; ]; avsinv8= inv(avs_mtrx); avscov= avs_mtrx*r*avsinv8; avscov=abs(avscov); % for i=1:n C(i)=dctcov(i,i); H(i)=h264cov(i,i); W(i)=wmv9cov(i,i); A(i)=avscov(i,i); figure(1) g=semilogy(n,c,n,h,n,w,n,a); set(g,{'color'},{'r';'g';'b';'k'}) leg(g,'dct','h.264','wmv9','avschina'); title('variances of transform coefficients'); xlabel('index k'); ylabel('variances'); sumc=sum(c); sumh=sum(h); sumw=sum(w); suma=sum(a); psumc(n)=c(n); psumh(n)=h(n); psumw(n)=w(n); psuma(n)=a(n); for i=n-1:-1:1 psumc(i)=psumc(i+1)+c(i); psumh(i)=psumh(i+1)+h(i); psumw(i)=psumw(i+1)+w(i); psuma(i)=psuma(i+1)+a(i); figure(2) f=plot(n,psumc/sumc*100,n,psumh/sumh*100,n,psumw/sumw*100,n,psuma/suma*100); set(f,{'color'},{'r';'g';'b';'k'}) leg(f,'dct','h.264','wmv9','avschina'); title('normalized basis restriction error versus the number of basis'); xlabel('samples retained m'); ylabel('jm - MSE %');

16 %calculating fractional correction for various values of rho for rh1=1:9 for i=1:n for j=1:n R1(i,j)= (rh1/10)^(abs(i-j)); dctcov1= dct8*r1*dctinv8; h264cov1= h264_mtrx*r1*h264inv8; wmv9cov1= wmv9_mtrx*r1*wmv9inv8; avscov1= avs_mtrx*r1*avsinv8; ctcov1= diag(diag((dctcov1))); htcov1= diag(diag(h264cov1)); wtcov1= diag(diag(wmv9cov1)); atcov1= diag(diag(avscov1)); Cmod1= dctinv8*ctcov1*dct8; Hmod1= h264inv8*htcov1*h264_mtrx; Wmod1= wmv9inv8*wtcov1*wmv9_mtrx; Amod1= avsinv8*atcov1*avs_mtrx; cc(rh1)= ((det(r1-cmod1))^2)/(det((r1- eye(n)))^2); hc(rh1)= ((det(r1-hmod1))^2)/(det((r1- eye(n)))^2); wc(rh1)= ((det(r1-wmod1))^2)/(det((r1- eye(n)))^2); ac(rh1)= ((det(r1-amod1))^2)/(det((r1- eye(n)))^2); figure(3) n1=0.1:0.1:0.9; h=plot(n1,cc,n1,hc,n1,wc,n1,ac); set(h,{'color'},{'r';'g';'b';'k'}) leg(h,'dct','h.264','wmv9','avschina'); title('fractional correlation vs rho for N=8'); xlabel('rho'); ylabel('fractional correlation');

17 Results: Graphs and tabulation for Order 8 ICT a) Variances of transform coefficients N DCT H.264 WMV9 AVSChina b) Normalized basis restriction error versus the number of basis N DCT H.264 WMV9 AVSChina

18 Jm - MSE % Variances 10 1 Variances of transform coefficients for N=8 DCT H.264 WMV9 AVSchina Index k Normalized basis restriction error versus the number of basis for N=8 DCT H.264 WMV9 AVSchina Samples retained m

19 fractional correlation 7 x Fractional correlation vs rho for N=8 DCT H.264 WMV9 AVSchina rho Comparison of performances of 16 X 16 ICT Code: N=16; rh=0.9; n=1:1:16; for i=1:n for j=1:n R(i,j)= rh^(abs(i-j)); %calculating variances of DCT coefficients dct16= dctmtx(n); dctinv16= transpose(dct16); dctcov= dct16*r*dctinv16; %calculating variances of H.264 integer DCT coefficients h264_mtrx= xlsread('h264_16.xlsx','a3:p18') h264inv16= inv(h264_mtrx);

20 h264cov= h264_mtrx*r*h264inv16; h264cov=abs(h264cov); %calculating variances of WMV9 integer DCT coefficients wmv9_mtrx= xlsread('wmv9_16.xlsx','a3:p18') wmv9inv16= inv(wmv9_mtrx); wmv9cov= wmv9_mtrx*r*wmv9inv16; wmv9cov=abs(wmv9cov); %calculating variances of AVS China integer DCT coefficients avs_mtrx= xlsread('avs_16.xls','a3:p18') avsinv16= inv(avs_mtrx); avscov= avs_mtrx*r*avsinv16; avscov=abs(avscov); % for i=1:n C(i)=dctcov(i,i); H(i)=h264cov(i,i); W(i)=wmv9cov(i,i); A(i)=avscov(i,i); figure(1) g=semilogy(n,c,n,h,n,w,n,a); set(g,{'color'},{'r';'g';'b';'k'}) leg(g,'dct','h.264','wmv9','avschina'); title('variances of transform coefficients for N=16'); xlabel('index k'); ylabel('variances'); sumc=sum(c); sumh=sum(h); sumw=sum(w); suma=sum(a); psumc(n)=c(n); psumh(n)=h(n); psumw(n)=w(n); psuma(n)=a(n); for i=n-1:-1:1 psumc(i)=psumc(i+1)+c(i); psumh(i)=psumh(i+1)+h(i); psumw(i)=psumw(i+1)+w(i); psuma(i)=psuma(i+1)+a(i); figure(2) f=plot(n,psumc/sumc*100,n,psumh/sumh*100,n,psumw/sumw*100,n,psuma/suma*100); set(f,{'color'},{'r';'g';'b';'k'}) leg(f,'dct','h.264','wmv9','avschina');

21 title('normalized basis restriction error versus the number of basis for N=16'); xlabel('samples retained m'); ylabel('jm - MSE %'); %calculating fractional correction for various values of rho for rh1=1:9 for i=1:n for j=1:n R1(i,j)= (rh1/10)^(abs(i-j)); dctcov1= dct16*r1*dctinv16; h264cov1= h264_mtrx*r1*h264inv16; wmv9cov1= wmv9_mtrx*r1*wmv9inv16; avscov1= avs_mtrx*r1*avsinv16; ctcov1= diag(diag((dctcov1))); htcov1= diag(diag(h264cov1)); wtcov1= diag(diag(wmv9cov1)); atcov1= diag(diag(avscov1)); Cmod1= dctinv16*ctcov1*dct16; Hmod1= h264inv16*htcov1*h264_mtrx; Wmod1= wmv9inv16*wtcov1*wmv9_mtrx; Amod1= avsinv16*atcov1*avs_mtrx; cc(rh1)= ((det(r1-cmod1))^2)/(det((r1- eye(n)))^2); hc(rh1)= ((det(r1-hmod1))^2)/(det((r1- eye(n)))^2); wc(rh1)= ((det(r1-wmod1))^2)/(det((r1- eye(n)))^2); ac(rh1)= ((det(r1-amod1))^2)/(det((r1- eye(n)))^2); figure(3) n1=0.1:0.1:0.9; h=plot(n1,cc,n1,hc,n1,wc,n1,ac); set(h,{'color'},{'r';'g';'b';'k'}) leg(h,'dct','h.264','wmv9','avschina'); title('fractional correlation vs rho for N=16'); xlabel('rho'); ylabel('fractional correlation');

22 Results: Graphs and tabulation for Order 16 ICT a) Variances of transform coefficients N DCT H.264 WMV9 AVSChina

23 b) Normalized basis restriction error versus the number of basis N DCT H.264 WMV9 AVSChina

24 Jm - MSE % Variances 10 1 Variances of transform coefficients for N=16 DCT H.264 WMV9 AVSchina Index k Normalized basis restriction error versus the number of basis for N=16 DCT H.264 WMV9 AVSchina Samples retained m

25 fractional correlation 1.4 x Fractional correlation vs rho for N=16 DCT H.264 WMV9 AVSchina rho Comparison of performances of 32 X 32 ICT Code: N=16; rh=0.9; n=1:1:16; for i=1:n for j=1:n R(i,j)= rh^(abs(i-j)); %calculating variances of DCT coefficients dct16= dctmtx(n); dctinv16= transpose(dct16); dctcov= dct16*r*dctinv16; %calculating variances of H.264 integer DCT coefficients h264_mtrx= xlsread('h264_16.xlsx','a3:p18')

26 h264inv16= inv(h264_mtrx); h264cov= h264_mtrx*r*h264inv16; h264cov=abs(h264cov); %calculating variances of WMV9 integer DCT coefficients wmv9_mtrx= xlsread('wmv9_16.xlsx','a3:p18') wmv9inv16= inv(wmv9_mtrx); wmv9cov= wmv9_mtrx*r*wmv9inv16; wmv9cov=abs(wmv9cov); %calculating variances of AVS China integer DCT coefficients avs_mtrx= xlsread('avs_16.xls','a3:p18') avsinv16= inv(avs_mtrx); avscov= avs_mtrx*r*avsinv16; avscov=abs(avscov); % for i=1:n C(i)=dctcov(i,i); H(i)=h264cov(i,i); W(i)=wmv9cov(i,i); A(i)=avscov(i,i); figure(1) g=semilogy(n,c,n,h,n,w,n,a); set(g,{'color'},{'r';'g';'b';'k'}) leg(g,'dct','h.264','wmv9','avschina'); title('variances of transform coefficients for N=16'); xlabel('index k'); ylabel('variances'); sumc=sum(c); sumh=sum(h); sumw=sum(w); suma=sum(a); psumc(n)=c(n); psumh(n)=h(n); psumw(n)=w(n); psuma(n)=a(n); for i=n-1:-1:1 psumc(i)=psumc(i+1)+c(i); psumh(i)=psumh(i+1)+h(i); psumw(i)=psumw(i+1)+w(i); psuma(i)=psuma(i+1)+a(i); figure(2) f=plot(n,psumc/sumc*100,n,psumh/sumh*100,n,psumw/sumw*100,n,psuma/suma*100); set(f,{'color'},{'r';'g';'b';'k'}) leg(f,'dct','h.264','wmv9','avschina');

27 title('normalized basis restriction error versus the number of basis for N=16'); xlabel('samples retained m'); ylabel('jm - MSE %'); %calculating fractional correction for various values of rho for rh1=1:9 for i=1:n for j=1:n R1(i,j)= (rh1/10)^(abs(i-j)); dctcov1= dct16*r1*dctinv16; h264cov1= h264_mtrx*r1*h264inv16; wmv9cov1= wmv9_mtrx*r1*wmv9inv16; avscov1= avs_mtrx*r1*avsinv16; ctcov1= diag(diag((dctcov1))); htcov1= diag(diag(h264cov1)); wtcov1= diag(diag(wmv9cov1)); atcov1= diag(diag(avscov1)); Cmod1= dctinv16*ctcov1*dct16; Hmod1= h264inv16*htcov1*h264_mtrx; Wmod1= wmv9inv16*wtcov1*wmv9_mtrx; Amod1= avsinv16*atcov1*avs_mtrx; cc(rh1)= ((det(r1-cmod1))^2)/(det((r1- eye(n)))^2); hc(rh1)= ((det(r1-hmod1))^2)/(det((r1- eye(n)))^2); wc(rh1)= ((det(r1-wmod1))^2)/(det((r1- eye(n)))^2); ac(rh1)= ((det(r1-amod1))^2)/(det((r1- eye(n)))^2); figure(3) n1=0.1:0.1:0.9; h=plot(n1,cc,n1,hc,n1,wc,n1,ac); set(h,{'color'},{'r';'g';'b';'k'}) leg(h,'dct','h.264','wmv9','avschina'); title('fractional correlation vs rho for N=16'); xlabel('rho'); ylabel('fractional correlation');

28 Results: Graphs and tabulation for Order 32 ICT a) Variances of transform coefficients N DCT H.264 WMV9 AVS China

29 b) Normalized basis restriction error versus the number of basis N DCT H.264 WMV9 AVSChina

30 Jm - MSE % Variances Variances of transform coefficients for N=32 DCT H.264 WMV9 AVSchina Index k Normalized basis restriction error versus the number of basis for N=32 DCT H.264 WMV9 AVSchina Samples retained m

31 fractional correlation 6 x Fractional correlation vs rho for N=32 DCT H.264 WMV9 AVSchina rho References: 1. N. Ahmed, T. Natarajan, and K. R. Rao, "Discrete Cosine Transform", IEEE Trans. Computers, vol. C-32, pp , Jan W. K. Cham and Y. T. Chan An Order-16 Integer Cosine Transform, IEEE Trans. Signal proc. vol. 39, issue no. 5, pp , May W. K. Cham, Development of integer cosine transforms by the principle of dyadic symmetry, in Proc. Inst. Electr. Eng. I: Commun. Speech Vis., vol no. 4, pp , Aug S. Kwon, A. Tamhankar, K.R. Rao, Overview of H.264/MPEG-4 part 10, Special issue on Emerging H.264/AVC video coding standard, J. Visual Communication and Image Representation, vol. 17, pp , Apr W. Cham and C. Fong Simple order-16 integer transform for video coding ICIP, Hong Kong, Sept.2010.

32 6. R. Joshi, Y.A. Reznik and M. Karczewicz, Efficient large size transforms for highperformance video coding, SPIE 0ptics + Photonics, vol. 7798, paper , San Diego, CA, Aug M. Costa and K. Tong, A simplified integer cosine transform and its application in image compression, Communications Systems Research Section, TDA Progress Report pp , Nov A.T. Hinds, Design of high-performance fixed-point transforms using the common factor method, SPIE 0ptics + Photonics, vol. 7798, paper , San Diego, CA, Aug S. Chokchaitam, M. Iwahashi and N. Kambayashi, Optimum word length allocation of integer DCT and its error analysis, Elsevier, Signal Processing: Image Communication vol. 19, pp , July C Wei, P. Hao Q. Shi, Integer DCT-based Image Coding, National Lab on Machine Perception, Peking University, Beijing, , China. 11. P.C. Yip and K.R. Rao, The transform and data compression handbook, Boca Raton, FL: CRC Press, Y. Zeng, et al Integer DCTs and fast algorithms, IEEE Trans. Signal proc. vol. 49, pp , Nov P. Chen, Y. Ye and M. Karczewicz, Video coding using exted block sizes, ITU-T Q.6/SG16, T09-SG16- C-0123, Geneva, Jan B. Lee, et al A transform kernel with quantization for (ultra) high definition video coding, ITU-T Q.6/SG16 VCEG, VCEG-AK13, Yokohoma, Japan, April G. Mandyam, N. Ahmed, and N. Magotra, Lossless image compression using the discrete cosine transform, Journal of Visual Communication and Image Representation, Vol.8, No.1, pp , March, W.Gao, et al AVS - The Chinese next-generation video coding standard, Joint development lab., Institute of computing science, Chinese academy of sciences, Beijing, China. 17. S. Srinivasan, et al Windows Media Video 9: Overview and Applications, Signal Processing: Image Communication, vol. 9, pp , Oct