IEEE Int. Conf. on Imag. Processing, Oct. 99, Kobe, Japan Analysis of Error Propagation in Hybrid Video Coding with Application to Error Resilience Niko Farber, Klaus Stuhlmuller, and Bernd Girod Telecommunications Laboratory University of Erlangen-Nuremberg Cauerstrasse 7/NT, D-91058 Erlangen, Germany ffaerber stuhl girodg@nt.e-technik.uni-erlangen.de Abstract A theoretical analysis of the overall mean squared error (MSE) in hybrid video coding is presented for the case of error prone transmission. The derived model for interframe error propagation includes the eects of INTRA coding and spatial loop ltering and corresponds to simulation results very accurately. For a given target bit rate, only four parameters are necessary to describe the overall distortion behavior of the decoder. Using the model, the optimal trade-o between INTRA and INTER coding can be determined for a given packet loss probability by minimizing the expected MSE at the decoder. 1 Introduction All common video coding schemes, including standards like H.263 and MPEG, use motion compensated prediction to exploit the redundancy between successive frames of a video sequence. While motion compensated prediction yields signicant gains in coding eciency, it also introduces interframe error propagation in case of transmission errors. Since these errors decay slowly they are very annoying and inuence the overall performance signicantly. To optimize video transmission systems in noisy environments, it is therefore important to consider the eect of error propagation. While several heuristic approaches have been investigated in the literature to reduce the inuence of error propagation (e.g. [1] [2][3]), up to now no theoretical framework has been proposed to model the inuence of transmission errors on the decoded picture quality. The proposed model includes the effects of INTRA coding and spatial loop ltering and corresponds to simulation results very accurately. It is shown that the model can be used, e.g., to determine the optimal percentage of INTRA coded macroblocks for a given loss probability. 2 Analysis of Interframe Error Propagation In this section we describe how the overall distortion at the video decoder is inuenced by transmission errors. Note that two dierent types of errors contribute to the overall distortion at the decoder: Errors that are caused by signal compression at the encoder D e, and errors that are caused by transmission errors D v. To model the overall distortion D = D e + D v, both components need to be analyzed. We now focus on errors caused by transmission errors and treat D e in section 3. We describe errors that are introduced by residual transmission errors using a stationary random process U which generates the zero{mean error signal u[x; y]. In other words, we assume that, on average, the same error variance u 2 is introduced in each frame. Obviously, the parameter u 2 is directly related to the loss probability, since an increased number of lost packets will also increase the amount ofintroduced errors. However, it also depends on several implementation issues, like packetization, resynchronization and error concealment aswell as on the encoded video sequence. In the following we assume transmission over a packet network and hence use the packet error rate (PER) to characterize the quality of the channel. For a given sequence, xed packet size, and given decoder implementation, the introduced error variance can be linearly related to the packet error rate, yielding 2 u = 2 u 0 PER ; (1) where 2 u 0 is a constant parameter describing the sensitivity of the video decoder to an increase in error rate. If the decoder can cope well with residual errors, the value is low. For example, 2 u 0 is typically high for complex motion but can also be reduced by advanced error concealment techniques. Errors that are introduced at a given point in time
propagate due to the recursive DPCM structure of the decoder. This temporal error propagation is typical for hybrid video coding that relies on motion compensated prediction in the interframe mode. It is very important to consider this eect for the design of the overall system since it has a signicant inuence on the sensitivity of the video decoder to packet loss. For example, even small values of u 2 may result in unacceptable picture quality if errors are accumulated in the decoder loop without being attenuated in some way. In the following we therefore derive an analytical model for the overall accumulated distortion that is caused by transmission errors. In particular, we investigate how the energy of introduced errors propagates due to the recursive DPCM structure in the decoder. More formally, we are interested in the signal v[x; y; t], which is the dierence between the reconstructed frames at encoder and decoder. Assume that the residual error u[x; y] isintroduced at t =0 (after resynchronization and error concealment) such that v[x; y; 0] = u[x; y]. We are then mainly interested in the variance 2 v [t] of the propagated error signal and in its average over time. The following analysis is an extension of previous work [4] [5]. 2.1 Decay over Time We assume that a separable loop lter is applied to the reconstructed frames after each time step. This loop lter shall describe the overall eect of various spatial lter operations that are performed during encoding. Spatial ltering can either be introduced by an explicit loop lter, as e.g. in H.261, or implicitly as a side eect of half{pel motion compensation with linear interpolation, as in H.263 or MPEG-2. Other prediction techniques like overlapped block motion compensation (OBMC) or deblocking lters inside the DPCM loop may also contribute to the overall loop lter. Though the exact eect is dicult to derive theoretically for the individual prediction techniques, we found that the overall eect can be described by a separable average loop lter F (!) with impulse response f[k]. We will rst analyze the eect of this loop lter on the propagated error energy and then add the eect of INTRA coding to the derived model. Regarding the decoder as a linear system H t (! x ;! y ) with parameter t, the variance of v[x; y; t] can be obtained as 2 v [t] = 1 4 2 Z Z + +,, jh t (! x ;! y )j 2 uu (! x ;! y )d! x d! y ; (2) where uu is the power spectral density (PSD) of the signal u[x; y]. As mentioned above, the spatial loop lter F (!) is applied to the reconstructed frame in horizontal and vertical direction in each time step. The resulting two-dimensional lter shall be denoted F (! x ;! y ) with impulse response f[x; y]. Then, the impulse response of the decoder h t [x; y] can be dened recursively as h t [x; y] =h t,1 [x; y] f[x; y], where `' denotes discrete two-dimensional convolution. Based on the central limit theorem we expect h t [x; y] tobe Gaussian for large t. Therefore, the squared magnitude of the transfer function of the decoder can be approximated in the base band j! x j; j! y j <,by where 2 f j ^H t (! x ;! y )j 2 = exp,(! 2 x +! 2 y)t 2 f ; (3) is dened as 2 f = X k k 2 f[k], X k kf[k]! 2 : (4) For example, in the case of bilinear interpolation with f[k] =[1=21=2], we obtain 2 f =1=4. In addition to the Gaussian approximation for H t (! x ;! y )we also approximate the PSD of the introduced error signal u[x; y] by ^ uu (! x ;! y )= 2 u4 2 g exp,(! 2 x +! 2 y) 2 g ; (5) i.e., a separable Gaussian PSD with the energy u. 2 The parameter g 2 determines the shape of the PSD and can be used to match (5) with the true PSD. Note that the same shape parameter is assumed in horizontal and vertical direction. Though this is not necessarily a very accurate assumption, it greatly simplies the following analysis and provides sucient accuracy. With the approximations for jh t (! x ;! y )j and uu (! x ;! y )we can solve (2) analytically (after extending the integration to [,1; +1]), yielding ^ 2 v[t] = 2 u = 1+t 2 u[t]; (6) where = f 2 =2 g is a parameter describing the eciency of the loop lter to remove the introduced error, and [t] is the power transfer factor after t time steps. The value of depends on the strength of the loop lter as well as the shape of the power spectral density of the introduced error u[x; y]. If no spatial ltering is applied in the predictor, = 0 and the decay in error energy is only inuenced by INTRA coding as described below. The value of usually increases when more spatial ltering is applied in the predictor
or when the introduced error includes high spatial frequencies that can easily be removed by the loop lter. So far we did not consider INTRA coded macroblocks, which cause a faster decay in error energy. If the INTRA mode is selected once every T frames for each macroblock, and the update time for a specic macroblock is selected randomly in this interval, the eect on the variance can be modeled as a linear decay. With = 1=T being the percentage of INTRA coded macroblocks, the nal equation for the power transfer factor becomes [t] = 1, t 1+t ; (7) for 0 t<t.for t T the error energy is removed completely and thus, [t] = 0. 2.2 Time Average For the evaluation of video transmission systems it is necessary to average the distortion over the whole sequence in order to provide a single gure of merit. Even though the time averaged squared error is somewhat questionable as a measure of subjective quality, this approach is still very useful to, e.g., provide an overview for a large set of simulation parameters, such as error rates, test sequences, bit rates, etc. In the following we are therefore interested in the time averaged distortion D v that is introduced by transmission errors. Since each individual error propagates over at most T successive frames and the decoder is linear, we can derive the average distortion D v as the superposition of T error signals that are shifted in time. If we further assume that the superimposed error signals are uncorrelated from frame to frame, we can calculate D v directly from (7), yielding D v = 2 u TX t=0 1, t 1+t : (8) Finally, we can approximate the sum by anintegral using trapezoid based numerical integration. The solution of this integral yields the closed form expression + D v = 2 u 2 ln 1+, 1 + 1 2 ; (9) which can be used instead of (8) to avoid the summation. In practice, the above assumptions are less restrictive than they may seem. For example, the assumption of uncorrelatedness is automatically met when individual error signals are spatially and/or temporally separated in the decoded video sequence. This is very common for low residual error rates but may become a problem otherwise. Therefore we expect that the accuracy of the model will decrease at high error rates. The assumption of stationarity, on the other hand, means that the eect of lost packets is approximately constant for each transmitted packet. However, we found that (8) can also be used for typical variations of errors introduced by packet loss. Only for extreme variations, e.g., for an almost static video scene with a short sequence of heavy motion, the dominant eect of a few packets can cause problems for the proposed model. 3 Application to Error Resilience In the following we will verify the derived model for interframe error propagation by a comparison with simulation results for a practical problem. The simulation results are obtained with an H.263 video codec and the problem at hand is to minimize the mean squared error (MSE) at the decoder for a lossy channel by adjusting the percentage of INTRA coded macroblocks. The INTRA update technique used in our simulations corresponds to the assumption in (7) and is identical to the update rule required in H.263 to avoid IDCT mismatch (however with a variable update interval T instead of the xed value T = 132). Obviously there is a trade-o to be considered for the selection of the INTRA percentage. On the one hand, an increased percentage of INTRA coded macroblocks helps to reduce interframe error propagation, and therefore reduces D v as described by (8). On the other hand, a high INTRA percentage increases the distortion D e that is caused by compression at a given target bit rate. From simulations with several test sequences we observed that the increase in MSE when increasing the number of INTRA coded macroblocks is approximately linear with the INTRA percentage for a xed bit rate. Hence, the distortion of the encoded (error-free) sequence can be described by D e = D P + (D I, D P ) ; (10) with the distortion D P for mere INTER coding and the distortion D I for mere INTRA coding. These distortions have to be measured from the sequence at the target bit rate and depend very much on the spatial detail and the amount of motion in the sequence. E.g., for a sequence with high motion and little spatial detail (like Foreman) (D I, D P )islow, whereas for a sequence with moderate motion and high spatial detail (like Salesman) (D I, D P ) is high. Our goal is to minimize the total distortion at the decoder which receives the encoded sequence over an
erroneous channel. Assuming that the distortion introduced by the encoder D e and the distortion introduced by transmission errors D v are statistically independent, the total distortion is given by D = D e + D v : (11) For PER = 0 (error-free) the only relevant term is D e and hence, the optimum percentage of INTRA coded macroblocks is zero. For other loss rates, the optimum performance also depends on D v and the optimum selection of INTRA percentage becomes less obvious. Note that for a given packet error rate the optimization depends on the four parameters D P, D I, 2 u 0, and. These parameters need to be adjusted to t the model to the given video sequence and video codec implementation (rate distortion performance, loop lter, packetization, resynchronization, error concealment). 3.1 Comparison with Simulation Results The simulation environment is described as follows. The QCIF test sequences Silent Voice (SI), Foreman (FM), Mother & Daughter (MD), and Salesman (SM) are encoded with an H.263 encoder at 12.5 fps (frames 0-0) at a constant bit rate of 65 kbps. No options are used, however, each Group Of Block (GOB) is encoded with a header to improve resynchronization. We simulate the transmission over a wireless channel at various noise levels, characterized by the ratio of bit-energy to noise-spectral-density (E b =N 0 ). However, the underlying details of the simulated transmission system (channel model, modulation, forward error correction) are not considered here since the only relevant parameter for the video codec is the resulting packet error rate. For the investigated values of E b =N 0 we obtain PER [%] = f0:00; 0:32; 1:06; 2:82; 6:16g. We assume that packets are either delivered error-free or dropped completely. In the latter case, the decoder receives an error indication and envokes error concealment for any GOB that overlaps with the lost packet. No special packetization is used, i.e., new GOBs are not necessarily aligned with the beginning of a packet. For error concealment the previous-frame GOB is simply copied to the current frame buer. The payload of the packet is set to the average GOB size, i.e. 65000/12.5/9 = 577 bit. For each PER and codec constellation random realizations are simulated to obtain average results. To assure that the distortion at the decoder is measured in a steady state, the rst 50 encoded frames are not used for evaluation. Fig. 1 compares the simulation results with the proposed model. The total distortion D is plotted vs. the INTRA percentage () for the investigated packet error rates. As commonly done in video coding, the distortion is expressed as PSNR, which is dened as 10 log 10 (5 2 =D). It can be seen that the analytical solution corresponds remarkably well with the measurements. Note that there are only four sequence specic parameters that need to be adjusted, namely u 2 0,, D P, and D I. For the test sequences used, these parameters have been extracted by matching the model to the four measurement points that are indicated in Fig. 1. The resulting model parameters are summarized in Tab. 1, which also includes an evaluation of the accuracy of the model. For this evaluation we measured the model error [db] = PSNR,PSNR 0 for each simulated data point and calculated the average absolute and maximum absolute model error as presented in the last two lines of Tab. 1. As can be seen, the average absolute error is less than 0. db. PSNR [db] 33 32 31 29 28 27 26 24 23 22 21 0.32 1.06 0.00 (PER [%]) 2.62 FM 6.16 Matched Simulation Model 20 1 3 6 11 22 PSNR [db] 36 35 34 33 32 31 29 28 27 26 24 SI 23 1 3 6 11 22 INTRA [%] INTRA [%] Figure 1: PSNR at the decoder vs. percentage of INTRA coded macroblocks for dierent packet error rates (PER). Left: Foreman. Right: Silent Voice. Table 1: Model parameters and accuracy. MD SI FM SM 2 48.47 2.10 509.93 63.02 u0 0.03 0.12 0.12 0.08 10 log 10 (5 2 =D I) 29.91.22 24.62.19 10 log 10 (5 2 =D P ) 38.72 35.34 31.90 37.55 Efjjg [db] 0.22 0.12 0.20 0.12 MAXfjjg [db] 0.89 0.53 0.76 0.62 3.2 Optimum Selection of INTRA Percentage In the previous section we showed that the selection of INTRA percentage can have a signicant inuence
on the decoded video quality. Corresponding simulation results are also presented in [3] and [6]. However, no consistent algorithm for the optimum selection of INTRA percentage has been proposed so far. Though [1] and [2] show the advantage of contentadaptive intra-refresh, the INTRA percentage is used as a free parameter or set to heuristic values. Only recently, rate-distortion optimized mode selection methods have been proposed that also consider transmission error eects [7] [8] [9]. Since the proposed model describes the overall distortion as a function of with sucient accuracy, it can also be used for the optimum selection of INTRA percentage. In Fig. 2 the optimal percentage is plotted for the four test sequences over the packet error rate by minimizing (11) numerically. To perform this optimization at the encoder, the model parameters need to be available. While the measurement ofd I and D P is not problematic, the estimation of 2 u 0 and is based on decoded video sequences in this paper. Because this would require several decoding steps at the encoder, we are currently investigating simpler approaches to estimate these parameters. Note that a variation of around the optimum has only little inuence on the performance (see Fig. 1). Therefore, 2 u 0 and do not have to be estimated very accurately. Furthermore, the optimum selection of is only one possible application for the derived model which has been selected because of its practical importance. The main contribution of this paper, however, is the derivation of D v in (8) which is veried by experimental data. optimal INTRA [%] 35 20 15 10 5 FM MD 0 0 1 2 3 4 5 6 7 8 9 10 PER [%] Figure 2: Optimal INTRA percentage vs. packet error rate (PER) for the four test sequences. 4 Conclusions We derived a theoretical model for the decoded picture quality (expressed as MSE) after video transmission over unreliable channels. The proposed model includes the eects of INTRA coding and spatial loop SI SM ltering. It has been shown that it corresponds to simulation results very accurately. For the investigated test sequences and simulation parameters the average absolute error is less than 0. db. Only four model parameters are necessary to describe the overall distortion behavior at the decoder. These parameters depend on the statistics of the video sequence and several implementation issues of the video codec, such as the loop lter and error concealment. For known parameters, the model can be used to, e.g., determine the optimal percentage of INTRA coded macroblocks for a given packet error rate. References [1] P. Haskell and D. Messerschmitt, \Resynchronization of motion compensated video aected by ATM cell loss," in Proc. IEEE Int. Conference on Acoustics, Speech, and Signal Processing, ICASSP'92, San Francisco, vol. 3, pp. 545-548, March 1992. [2] J. Liao and J. Villasenor, \Adaptive Intra Update for Video Coding over Noisy Channels," in Proc. IEEE Int. Conference on Image Processing, ICIP'96, Lausanne, Switzerland, vol. 3, pp. 763-766, Sept. 1996. [3] Q. F. Zhu and L. Kerofsky, \Joint source coding, transport processing, and error concealment for H.323-based packet video," in Proc. Visual Communications and Image Processing, VCIP'99, San Jose, SPIE vol. 3653, pp. 52-62, January 1999. [4] B. Girod, N. Farber, \Error-Resilient Standard- Compliant Video Coding," in: A. Katsaggelos, N. Galatsanos (eds), Recovery Techniques for Image and Video Compression and Transmission, Kluwer Academic Publishers, Boston, Oktober 1998. [5] B. Girod, N. Farber, \Feedback-Based Error Control for Mobile Video Transmission," Proceedings of the IEEE, Special issue on video for mobile multimedia, accepted for publication (Oct. 1999). [6] G. C^ote, S. Wenger, and M. Gallant, \Eects of standard-compliant macroblock intra refresh on ratedistortion performance," ITU-T SG15 contribution Q13E37, July 1998. [7] S. Wenger, G. C^ote, \Using RFC2429 and H.263+ at low to medium bit-rates for low-latency applications," in Proc. Packet Video Workshop, New York, April 1999. [8] G. C^ote and F. Kossentini, \Optimal Intra Coding of Blocks for Robust Video Communication over the Internet," EURASIP Image Communication, Special Issue on Real-time Video over the Internet, to appear. [9] T.Wiegand, N. Farber, and B. Girod, \Error- Resilient Video Transmission Using Long-Term Memory Motion-Compensated Prediction," submitted to IEEE Journal on Selected Areas in Communications.