Linköping University Post Print. A Minimum Mean Square Error Estimation and Mixture-Based Approach to Packet Video Error Concealment

Transcription

1 Linköping University Post Print A Minimum Mean Square Error Estimation and Mixture-Based Approach to Packet Video Error Concealment Daniel Persson and Thomas Eriksson N.B.: When citing this work, cite the original article IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE. Daniel Persson and Thomas Eriksson, A Minimum Mean Square Error Estimation and Mixture-Based Approach to Packet Video Error Concealment, 2007, Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, 2007, Postprint available at: Linköping University Electronic Press

2 A MINIMUM MEAN SQUARE ERROR ESTIMATION AND MIXTURE-BASED APPROACH TO PACKET VIDEO ERROR CONCEALMENT Daniel Persson and Thomas Eriksson Chalmers University of Technology Department of Signals and Systems Goteborg Sweden ABSTRACT In this paper, a minimum mean square error-optimized mixture-based estimator is used for packet video error concealment. At the same time as on-line computational complexity is reduced, performance in peak signal-to-noise ratio (PSNR) is increased in comparison with a Gaussian mixture-based estimator that obtains its parameters through probability density estimation by means of the expectation maximization algorithm. Moreover, our method increases performance in PSNR compared to several other previous error concealment algorithms. Index Terms- Packet video, error concealment, estimation. 1. INTRODUCTION Today's frequently used video compression methods utilize blockbased motion-compensated inter-frame prediction, discrete cosine transform-based quantization, and variable length coding, to reduce the bit stream. High compression ratios are achieved in this way, but the video stream sensitivity to communication channel and storage media errors also increases. Among the various error resilience techniques used for combating this problem, methods that work at the decoder side without extra redundancy from the encoder are referred to as error concealment schemes. Error concealment methods are usually categorized into spatial approaches, that use only spatially surrounding pixels for estimation of lost blocks, and temporal approaches, that use motion information and pixels from previous frames I 12. In tyis paper, we combine spatial and temporal error concealment. The MVs are considered to be available through protection in a high priority layer, or estimated by the median of the MVs of the surrounding blocks [2], and we propose a mean square error (MSE) risk- and mixture-based estimator that classifies the situation of local image correlation in order to choose spatial or temporal error concealment. In the case of lost MVs, the proposed method may be combined with any method for finding the MVs. Performance in peak signal-to-noise ratio (PSNR) is increased, while on-line computational complexity is reduced, compared to our own GMM-based estimator with parameters retrieved by means of the expectation maximization (EM) algorithm that was reported on in [3], [4], and [5]. The new method also increases performance in PSNR compared with other previous methods that combine spatial and temporal error concealment, e.g. Shirani et al's method [6], Zhu et al's method [7], and motion compensated copying [2]. The rest of the paper is organized as follows. In Section 2, the minimum MSE (MMSE) estimation framework is defined, and an argumentation for a mixture-based estimator is given. Section 3 describes the experimental setup, and presents simulation results. The paper is concluded in Section /07/$ IEEE 2. MMSE ESTIMATION WITH MIXTURE-BASED ESTIMATOR In this section, mixture-based packet video error concealment is formulated as a MMSE-problem. The proposed technique is a further development of the approach reported on in [3], [4], and [5], that has previously shown to increase PSNR compared to the schemes [6], [7], and [2] Motivation of the proposed method We start by summarizing [3], [4], and [5], and further discussing the advantages and the shortcomings of the method. Thereafter, the new approach is explained. In [3], [4], and [5], a group of neighboring pixel values, that are lost at the decoder side, are represented by elements of the stochastic vector variable X. An estimate X g(y) (1) is formed, where Y is a stochastic vector whose elements represent pixels in a neighborhood to X. Further, the MMSE problem g* (Y) argmine[ X+ i g(y) 11 ] g(y) (2) is posed, that has the solution g*(y) E(X Y) xfxiy (x IY) dx. (3) The density fx,y (X, y) is modeled with a GMM M (4) ml where Z' [X', Y'] and f (m) (z) are Gaussian distributions with means and covariances o (m) z _ F AX L 0(m) - I c(m) ZZ 1--x c,r2) c(m) C(-)j - (5) Cxx YX and where the a priori weights 0(m) are all positive and sum to one. The estimator (3) may then be written g*(y) m~~~~ Eo(m)(Y (c,7)(c( 2)* (Y+6y ) (6) ml I ICASSP 2007

3 where 0 (m)(y) 0(m)f(m) (Y) have (7) (k) (y) are a posteriori probabilities that sum to 1. A qualitative interpretation of the form of the mixture (6) is that the a posteriori weights 0(m) (Y) are used for classifying the local image correlation, and choosing appropriate linear estimators c m) (C(2) * 1(YA (m)) + 0 () for the situation at hand. As seen in [3], [4], and [5], the estimator (6) with M > 1 increases performance in PSNR compared to the linear estimator obtained when M 1. Though the above scheme has shown to increase PSNR compared to several previous methods [6], [7], [2], a few remarks about its disadvantages can be made. 1 (k) i The formulation in (1) and (2) leaves us with a density estimation problem in order to achieve fz (z). In [3], [4], and [5], the density was achieved by means of the EM algorithm that is a maximum likelihood (ML) method, and therefore does not necessarily minimize the MSE. A consistent treatment, where all parameters of the final estimator are achieved by minimizing the MSE, would be more eligible. However, a MMSE formulation for finding the 0(m), 0 (m), and C (-) of the estimator (6) would be an extremely difficult problem. i The estimator (6) has a high computational complexity online, when X is estimated from Y. The quadratic forms in the exponents of f (m) (Y) contribute significantly to the computational complexity. i In [3], [4], and [5], a vector Z with 64 dimensions was employed. It would be desirable to increase the number of dimensions of Y substantially, but with the estimator (6), this would be expensive due to matrix multiplications, both for estimator optimization, and for on-line computations. To summarize, the benefit of a simplification of the estimator (6) is threefold: the parameters may be obtained in the MMSE-sense, online complexity is reduced, and the estimate may be based on more information. In what follows, we rephrase the problem stated in (1) and (2) with this in mind. We form the new estimate X g(y, 0) (8) where g(y, 0) is the estimator function with parameters 0. Optimal parameters 0* are found by minimizing the MSE risk (9) 0* argmine(x g(y, 0)112). 0 In contrast to the parameters of the estimator (6), that depend on ML estimation, the parameters of (8) are achieved in the MMSE sense. As PSNR is a standard measure ofvideo quality [7], [6], [3], [4], [5], and a function of the MSE, the MSE risk was chosen. By minimizing the MSE, the PSNR is maximized. For the proposed estimator, we choose a form that is heavily inspired by the solution in (6). The means 0 () and 0 (m) are removed, the matrices C ) (Cr )) * are replaced by matrices A(m), the functions 0 (m) are replaced by simpler functions 0 (m), and two subsets of Y are employed in the estimator (8): Yc for classification in the a posteriori weights (C stands for classification), and Yp for prediction (P stands for prediction). By introducing Yc and Yp, more information may be used in the new simple a posteriori weights 0 (m), than for the prediction. We now g(y; 0) f E o(m) (y )A(m)y (10) ml 1(k)h(k)(YC) o (m) (YC) h(m) (Yc) (1 1) Ek 0 (k) h(k)(c h((yc exp e( y(m),y(m)2 2) C D(m)''I (12) where Ym)1 and Y,(m)2 are vectors containing elements of Yc, c(m) > 0 is a scalar, and D(m) is the dimension of the vectors and <m) 2. In the discussion of the GMM-based estimator, we concluded that different a posteriori weights focus on different situations of video correlation. In accordance with this, the vectors YC(m)'l and Y(m)2 should be chosen so that a specific situation of video correlation is given priority. For example, in order to generate a mixture component that focuses on spatial correlation, y(i), 1 and y(m),2 should be chosen so that the exponent of (12) incorporates the difference of the values of many spatially neighboring pixels. The exponent of (12) may be represented as ()y(m),l + y(m), C ic( m) (m) D(m)TW(m)yc (13) where the matrices W(m) not are positive definite. This means that the new mixture (10) not is GMM-based though it has the essential functionality of the estimator (6). Also, (10) is easily optimized in the MMSE way, it reduces on-line complexity compared to (6), and it may take a Yc with high number of dimensions without severely increasing the computational complexity. The parameters 0 are {0(m), c(m), A(m) }. Though the estimation of X from Y is performed on-line, the optimal estimator parameters 0* are found off-line Algorithm for solving the MMSE estimation problem Since the solving of the MMSE problem (9), with the proposed estimator (10), does not have a closed form solution, an algorithm for iterative solving of (9) with (10) is now proposed. We solve iteratively for the parameters A(m), 0(m) and c(m). The algorithm increases PSNR in every iteration. The parameters A(m). The matrices A(m) are first considered. It is easy to show that the problem that consists in finding A(')* X g(y, 0) 112] argmine[ X A('r) (14) is convex. The proof will be given in a longer journal paper. By setting the derivative with respect to A(r) equal to zero, and thereafter solving for A(r), we achieve RI(R2)* 1 A(r)* (15) E[0(r)(Yc)XY> i R, 0(Yc)( )0(Yc)(m)A( m 1,m R2 E[(0( /r )(YcL))2Yp ]. )YPYPp] (16) (17) Theparameters 0(m) and c(m). Since the MSE is not convex in 0(m), these parameters are updated by searches in the space of possi- I- 750

4 ble vectors 0 [0(1),..., 0(M) ]T. In each iteration, the MSE is compared at the point describing the parameter set 0 obtained in the previous iteration, and inthe two points 0+0 [0(1)a(1),..., 0(M)a(M)]T, where 0 is some scalar, and a is a normalized random vector with elements a(l) to a(m). The point that yields the minimum value of the MSE is chosen as the new 0. This algorithm is faster than gradient descent with backtracking line search, since many evaluations of the MSE are avoided. The parameters c(m) are updated in the same way. For optimization in practice, the expectations are replaced by arithmetic means. 3. EXPERIMENTS In this section, the proposed method is evaluated and compared to methods suggested by other authors. Details of the simulations, which are chosen to fit state-of-the-art block-based video coders, are given in Section 3.1. These conditions are impartial to all the compared schemes. Results of the experiments are presented in Section Simulation prerequisites Coding and packetization. We focus on predictively coded frames (P-frames) (An application of the proposed method to restoration of intra-coded frames (I-frames) is completely analogous). MVs are calculated for blocks with a search range of 8 pixels for each component. The coder works in the limit of perfect quantization. Each row of blocks of pixels is divided into blocks of pixels, that are interleaved into two packets, so that if a packet is lost, there is a high probability that surrounding pixels are available. Concerning the MVs, two situations are investigated. In the first scenario, the MVs are coded with their respective pixel information in the same packet, and when a packet is lost, lost MVs are estimated by the median of MVs of the neighboring blocks [2]. In the second scenario, MVs are protected in a high priority layer, and considered available. These assumptions are similar to the assumptions in [7]. Errors. The packets are randomly assigned as lost. Simulations are run for loss probabilities ranging from 0.05 to 0.3. In the case of MVs protected in a high priority layer, only packets containing pixel information are lost. Data. We use the luminance component of 124 MPEG-1 movies from [8]. The movies are divided into two independent sets, one for off-line optimization of the parameters 0, and another for evaluation. In order to show the robustness of our scheme, we use more movies for the evaluation than for the training. The sets used for parameter optimization and evaluation contain 35 and 89 randomly selected movies respectively. Benchmarking. The proposed estimator is compared to other schemes that mix spatial and temporal information given the MVs: Shirani et al's method [6], Zhu et al's method [7], and GMM-based error concealment [3], [4], [5], as well as to motion compensated copying [2]. All methods use the same MV information. Proposed estimator. Each lost block is repaired by splitting it into four blocks whose pixels are represented by X. The surrounding pixels represented by Yc and Yp will vary in the experiments, and will be described for each experiment. We choose to work with few mixture components since we strive for low on-line complexity. A mixture with M 2 components is investigated. Mixture component 1 focuses on spatial correlation by employing Y<(1)'1 and Yj(1),2 such that all possible differences between closest spatial neighbors in Yc are included in the exponent. In the same way, mixture component 2 focuses on temporal correlation by em- ploying <2)1 and <2)2 such that all possible differences between closest temporal neighbors in Yc are included in the exponent. Varying available information. Spatially surrounding pixels may not be available, because the block in question is a border block, or because several consecutive blocks are lost. Different models are obtained and stored for each of these cases. In the case when no spatial surrounding information is available, we reduce our estimator to the special linear MMSE solution to (9) and (10) A(1) E(X Yp)(E(YpTYp))* 1 that is obtained when M 1, and now Yp only represents pixels in the previous frame. By assuming mirror invariance of the model, only four model cases need to be stored. Off-Line parameter estimation. The parameters of the estimator are found off-line. A choice of M 2 mixture components was previously made. The parameters 0 [(1),0(2)]Tand c [cc1), c(2)]t are initialized by I [1,1]T, and the parameters A(m) are initialized by the linear MMSE solution to (9) and (10) obtained when M 1, i.e. A(m) (XYp) (E(YpTYp))* 1. For the update of O(m) and (m) we choose In each iteration, realizations of ZT [XT, yt] are used. In each of the 10 first iterations, two iterations are performed for 0(m) and c(m) respectively, as well as one iteration for the A(m). In the ten final iterations, only the A(m) are updated Results Estimator comparison. The purpose of this first experiment is to see that our estimator strategy yields higher performance in PSNR and lower computational complexity than the GMM-based estimator [3], [4], [5], when both methods use the same number of mixture components and the same information. In this experiment, we set Yc Yp, as illustrated in Figure 1, and pixels surrounding a lost block are guaranteed to be available. The results presented in Table 1 show that the proposed method gives better performance in PSNR, with around a third of the computational complexity. Error concealment comparison. The proposed estimator (10) is compared to different error concealment schemes. In preliminary simulations, the approach with Yc and Yp chosen as in Figure 2, was compared to the approach with Yc and Yp chosen as in Figure 1. Since Yc and Yp chosen as in Figure 2 yielded better results while maintaining low complexity, this approach was chosen for comparison to other previous error concealment methods. Note that by choosing Yc as in Figure 2, significantly more information is used for the classification, than if Yc would have been chosen as in Figure 1. In the experiments, pixels surrounding a lost block were not guaranteed to be available. The errors propagate in a few tens of frames in each movie. Figure 3 presents the results for the case when the MVs are not available, and replaced by the median of the MVs of the surrounding blocks. In Figure 4, we see the results in the case when the MVs are available. The proposed method gives best performance in PSNR. The GMM-based method with M 20 gives a comparable result, but this comes at a cost of around 26 times higher on-line computational complexity than the proposed method. A GMM-based estimator with M 2 has 2.6 times higher computational complexity than the proposed scheme. Examples in a longer journal paper will show that our method also improves subjective visual performance. 4. CONCLUSION In this paper, MMSE- and mixture-based error concealment, that may be run in real-time, is presented. For a motivation of the new technique, we use as our starting point an estimator methodology based on GMMs and probability density estimation by means of the I- 751

5 Frame t-i - X YC Yp 1I 12 pixels Frame t 4 pixels 36r G Proposed method, M 2 D GMM, M " GMM, M 2 n:l / A! A x A.r Zhu et al's method 4 C Shirani et al's method n300 -:<- Motion-compensated copying V, Fig. 1. Illustration of variables to be used with the proposed estimator in the estimator comparison. Blocks of size are divided into four blocks that are estimated separately. The vector X is lost at the decoder side, and is estimated by using a vector of surrounding pixels Yc Yp. -28t 1 24II Loss rate percentage p Fig. 3. Comparisons of different error concealment schemes, in the case when the MVs are not available, and replaced by the median of the MVs of the surrounding blocks. - x m YP Fig. 2. Illustration of variables to be used with the proposed estimator in the error concealment comparison. Blocks of size are divided into four blocks that are estimated separately. The vector X is lost at the decoder side, and is estimated by using vectors of surrounding pixels Yc and Yp. MV Type of Proposed GMM, GMM, test method, M 2 M 1 M 2 Not available Closed Not available Open Available Closed Available Open Table 1. Comparison of results in PSNR for the proposed method, and the GMM-based method. The number M is the number of mixture components. EM algorithm. All parameters of the new estimator are easily obtained off-line in the MMSE sense. The proposed estimator gives better performance in PSNR than the GMM approach when using the same data and number of mixture components. At the same time, the proposed estimator has lower computational complexity. Moreover, the new technique may, without substantial increase in computational complexity, incorporate much bigger surrounding to the lost block as input to the estimator. We finally show that the proposed method gives an important increase in performance compared to a range of other well-known previous error concealment methods. 5. REFERENCES [1] Y. Wang and Q.-F. Zhu, "Error control and concealment for video communication: a review," Proc. IEEE, vol. 86, pp , May [2] P. Haskell and D. Messerschmitt, "Resynchronization of motion 7 I.XIR P- C 32S l - G Proposed method, M 2 D GMM, M 20 ' GMM, M 2 E GMM, M 1 Zhu et al's method x Shirani et al's method Motion-compensated copying Loss rate percentage Fig. 4. Comparisons of different error concealment schemes, in the case when the MVs are available. compensated video affected by atm cell loss," in Proc. ICASSP, Mar. 1992, pp [3] D. Persson and P. Hedelin, "A statistical approach to packet loss concealment for video," in Proc. ICASSP, Mar. 2005, pp [4] D. Persson, T. Eriksson, and P. Hedelin, "Qualitative analysis of video packet loss concealment with gaussian mixtures," in Proc. ICASSP, May 2006, pp. II-961 -II-964. [5] D. Persson, T. Eriksson, and P. Hedelin, "Statistical packet video error concealment," IEEE Trans. Image Processing, in review. [6] S. Shirani, F. Kossentini, and R. Ward, "A concealment method for video communications in an error-prone environment," IEEE J Select. Areas Commun., vol. 18, pp , June [7] Q.-F. Zhu, Y. Wang, and L. Shaw, "Coding and cell-loss recovery in DCT-based packet video," IEEE Trans. Circuits Syst. Video Technol., vol. 3, pp , June [8] "Prelinger archives," Online resource. 30 I- 752