IMAGE ERROR CONCEALMENT BASED ON JOINT SPARSE REPRESENTATION AND NON-LOCAL SIMILARITY

Transcription

1 IMAGE ERROR CONCEALMENT BASED ON JOINT SPARSE REPRESENTATION AND NON-LOCAL SIMILARITY Ali Akbari 1 Maria Trocan 2 Bertrand Granado 3 Institut Supérieur d Electronique de Paris (ISEP), Paris, France Laboratoire d Informatique de Paris 6, Pierre et Marie Curie University, Paris, France 1 ali.akbari@isep.fr, 2 maria.trocan@isep.fr, and 3 Bertrand.Granado@lip6.fr ABSTRACT In this paper, an image error concealment method based on joint local sparse representation and non-local similarity is proposed. The proposed method obtains an optimal sparse representation of an image patch, including missing pixels and known neighboring pixels for recovery purpose. At first, a pair of dictionary and a mapping function are simultaneously learned offline from a training data set. The dictionary pair transfers the original image patch and corrupted image patch into a domain that the intrinsic relationship between patches can be derived more accurate than the spatial domain. Then, in the error recovery process, the sparse representation of corrupted patch is obtained and mapped using the mapping function. The obtained sparse representation is further refined by exploiting the self-similarities in the natural images to obtain a good estimate of the sparse representation on the original and clean image. Finally, the concealed image is obtained by back projection of this sparse presentation into the spatial domain. Compared with the state-of-the-arts error concealment algorithms, experimental results show that the proposed method has better reconstruction performance in terms of objective and subjective evaluations in different loss scenarios and validate its effectiveness for the error concealment applications. Index Terms Error Concealment, sparse representation, dictionary learning, mapping learning, robust transmission 1. INTRODUCTION In the existing image and video transmission systems, the frames to be transmitted are divided into B B nonoverlapped blocks which are further coded separately by an entropy encoder, prior to be packed. When transmitted over an error-prone channel, the undesired packet erasures lead to occurrence of unpleasant missing areas in the received image. Fig. 1 shows different error patterns occurred during the transmission over an error-prone channel. To make the reconstructed image acceptable, the lost packets need to be retransmitted or recovered by other means. (a) (b) (c) Fig. 1. Typical block loss pattern [8]: (a) Isolated loss. (b) Consecutive loss. (c) Random loss. Recently, a robust transmitter-based image transmission scheme has been proposed in [1], in which a high quality image can be recovered at high loss rates at the expense of adding a simple random linear encoder at the transmitter side and thus sending the additional information. In contrast to the transmitter-based methods, Error Concealment (EC) techniques, as a receiver-based method, represent an alternative solution for mitigating the negative effects of the packet loss. The EC techniques recover the missing information without modifying the encoder or sending any additional information, leading to better bandwidth use. In fact, these methods exploit the high spatial and/or temporal correlation existing among the lost areas and the correctly received neighboring pixels. Many EC techniques have been proposed in the literature, exploiting the image properties to recover the structure and/or texture of the lost areas. Different interpolation-based EC techniques are proposed in [2 8]. In a different way, statistical tools for the image error concealment are considered in [9 11]. Recently, example learning based EC techniques method have been proposed in [12, 13]. In this paper, a novel EC method based on local joint sparse representation and non-local similarity is proposed. The proposed method obtains an optimal sparse representation of a patch (including missing pixels and their correctly received neighbors), using a dictionary pair and a mapping function that are trained offline using a set of training image patches. In addition, the sparse representation is further improved by exploiting the self-similarities present in natural images. This approach provides a solution to the problem of the LSR algorithm [12], which can not assure an accurate sparse representation for the /17/$ IEEE 6 GlobalSIP 2017

2 Fig. 2. Block diagram of the proposed joint local sparsity and non-local self-similarities based image EC algorithm. corrupted patch. In our approach, the sparse representation of the patch, including known and unknown pixels, is mapped to a more accurate sparse representation. Consequently, the proposed method performs the recovery of lost blocks more successfully than the state-of-the arts schemes. 2. IMAGE EC USING JOINT LOCAL SPARSITY AND NON-LOCAL SELF-SIMILARITIES (JLSNS) Generally, the image EC problem states: given a corrupted received image Y, the image X is recovered using just the correctly received information: ˆX = arg min Y HX 2 2, (1) X where H is a diagonal matrix, whose diagonal entries are either 0 or 1. This minimization problem is extremely ill-posed, since for a given corrupted image Y, many images satisfy this minimization problem. In this paper, in order to solve this ill-posed problem, two constraints are considered to regularize the solution space: 1) local sparsity prior which assumes that each N N image patch in the image X, represented as a column vector x R N, can be linearly represented using a few number of elements (atoms) chosen from an overcomplete dictionary D R N K (K N) and 2) non-local self-similarities prior which assumes that natural images often have many repetitive structures. Based on these local and non-local models, the proposed Joint Local Sparsity and Non-Local Self-Similarities based EC algorithm (JLSNS) can be described in three steps. Firstly, the sparse representation of each corrupted patch over a trained dictionary is obtained. Secondly, this sparse representation vector is mapped to a space that is as close as possible to the sparse representation of the original and clean patch. Finally, using the results from this mapped sparse representation, we further regularize and refine the solution space using the non-local self-similarities Joint local sparsity-based EC (JLS) Let L be one B B lost block in the corrupted image Y and S be the set of available pixels, or support area. In our Fig. 3. Lost block and structure of the corrupted patch y. Each square stands for one pixel. L denotes the set of missing pixels and S denotes the support area. implementation, the lost block is initially estimated by spatial interpolation from the nearest undamaged pixels. Consider the corrupted patch y of size N N, represented as a column vector y = [u, v] T, where v R P is a group of P unknown pixels in L and u R N P contains a set of adjacent and available pixels in S, within the N N block (see Fig. 3.) In this paper, we consider N = 25. For the joint local sparsity model, two dictionaries D x and D y are trained using the patches sampled from the training images. In the learning sparse representation-based EC algorithm (LSR), proposed in [12], it is assumed that the corrupted vector y and the concealed vector x have the same sparse representation, i.e. α x = α y. For each corrupted patch y, the LSR method tries to find the sparse representation vector with respect to D y. Then this sparse representation vector is used to recover the corrupted patch via ˆx = D x α y. However, this approach does not guarantee the compatibility between adjacent blocks and leads to weak performance for the patches that are located in high frequency areas of the image. In this paper, we assume that the sparse representation of each corrupted patch y with respect to the dictionary D y and the sparse representation of the corresponding concealed patch x with respect to the dictionary D x are related via a mapping matrix M, i.e., α x = Mα y, learned from a set of training image patches. Given the dictionary pair D x, D y and the mapping matrix M, the corrupted image y can be concealed by solving the following optimization problem: arg min α x,α y R K y D y α y x D x α x γ α x Mα y λ x α x p + λ y α y p, (2) where γ, λ x and λ y are the regularization parameters. Inspired by results in [14] and our practical implementation for the EC application, we have found that, given a trained dictionary learned with l 1 -norm, the concealed image by (2) has higher quality by using the l 0 norm. In addition, the reconstruction time using l 0 -norm is lower than the one obtained using the l 1 -norm, leading thus to a less complex recovery. 7

3 2.2. Joint local sparsity and non-local similarity based EC (JLSNS) As a complementary regularization term to the joint local sparse model, a non-local similarity regularization term is added to the EC optimization. Different from the non-local models proposed in the spatial domain [14, 15], we propose to exploit the non-local similarities in the joint representation domain. Further, the difference between the sparse representation of the patch x in the concealed image, i.e. α x (and obtained by solving the minimization problem (2)), and the true sparse representation of the corresponding patch in the original image, i.e. α t, should be as close as possible to zero. By incorporating this constraint into the JLS model in Eq. (2), we have the following optimization problem: arg min α x,α y R K y D y α y x D x α x γ α x Mα y δ α x α t λ x α x 0 + λ y α y 0, (3) where δ is the regularization parameter. Since the true sparse representation α t is unknown, we try to provide a good prediction of it by exploiting the non-local similarities in the image that is reconstructed via the JNS-based model (2). α t is obtained by searching similar patches in the whole reconstructed image and predicted as: L α p = w i α i, (4) i=1 where α i is the sparse representation of the i-th similar patch to the patch x and L represents the number of similar patches. w i is the weight assigned to the i-th similar patch as defined in [15], via calculation of the Euclidean distance between the patches. In this paper, we select L = 10. To summarize the proposed EC algorithm, given the dictionary pair D x, D y and a mapping matrix M, Eq. (3) can be iteratively solved by alternatively updating α y and α x. At first, the corrupted patch y is represented with respect to the dictionary D y to obtain the sparse representation vector ˆα y. This l 0 -optimization problem can be efficiently solved by greedy algorithms as OPM [16]. Then, the reconstructed patch ˆx is obtained as ˆx = DMˆα y. Finally, the unknown pixels v are replaced by ˆv obtained from ˆx = [û, ˆv] T (see Fig. 3.) Since the size of lost blocks in the corrupted image are usually large (usually 8 8 or pixel blocks), they are split into subblocks of size P P and further these subblocks are sequentially recovered. Note that the first subblocks to be concealed are located in the corners of a lost block, since they provide more reliable information. This procedure also ensures that the pixels in the context area, i.e. u, are correctly received or already recovered Dictionary and mapping learning In this section, an offline learning procedure for training the dictionary pair D y and D x and learning the mapping matrix M are discussed. For generating the training data, at first, the original images are corrupted using an isolated loss pattern (see Fig. 1a). Then, the lost blocks are recovered by simple interpolation to generate an initial estimation of each corrupted image. We consider the vector representations of the image patches of size N N -containing both known and corrupted pixels- in the initial reconstructed images as the set of corrupted training patches Y = {Y 1, Y 2,, Y n }. The corresponding image patches in the original images are collected as the set of original training patches X = {X 1, X 2,, X n }. Given the training dataset pairs S = {X, Y}, the desired dictionary pair as well as the appropriate mapping are learned by the following optimization [17]: arg min Y D y Λ y X D x Λ x γ Λ x MΛ y 2 2 D x,d y,m + λ x Λ x 1 + λ y Λ y 1 + λ m M 2 2, (5) where the γ, λ x, λ m, and λ y are the regularization parameters. The first and second terms are fidelity terms, the third term is the mapping fidelity term to represent the mapping errors between sparse representation vectors α x and α y. The l 1 -norms enforce sparsity to regularize the solution spaces. As discussed before, we consider l 1 -norm for the dictionary learning algorithm. The sparse representations of the training sets X and Y over D x and D y are linked by the mapping matrix M. The optimization problem (5) is not convex with respect to D x, D y, and M, but is convex with respect to one term when the remaining two terms are fixed. The mapping matrix and the dictionary pair are jointly optimized with the approach proposed in [17]. 3. EXPERIMENTAL RESULTS For our experimental framework, we use a set of 8-bit grayscale standard images of pixels. For estimating the efficiency of the proposed JLSNS error concealment algorithm, the Peak Signal-to-Noise Ratio (PSNR) is chosen as objective quality measure, along with the Structural SIMilarity (SSIM) in order to obtain a perceptual quality measure. The performance of the JLSNS EC algorithm is compared with state-of-the-arts EC techniques, including nonnormative spatial EC for H.264 (AVC) [18], content adaptive technique (CAD) [5], edge recovery technique based on visual clearness (VC) [4], Markov Random Fields approach (MRF) [10], orientation adaptive interpolation (OAI) [6], sparse linear prediction (SLP) [7], adaptive linear prediction (ALP) [8], and learning sparse representation-based EC (LSR) [12] 1. The three loss patterns, including isolated loss, consecutive loss and random loss, as shown in Fig. 1, are considered for our evaluation. For the random loss scenario, 30% block loss is randomly generated. The average PSNR 1 The implementation of some techniques are available online at http: //dtstc.ugr.es/ jkoloda/research.html 8

4 Table 1. Average PSNR and SSIM using Several EC Techniques for Three Types of Loss (Isolated Loss (), Consecutive Loss () and Random Loss ()) EC Technique Loss CAD MRF OAI SLP LSR ALP JLSNS Lena PSNR SSIM SSIM PSNR SSIM Peppers PSNR SSIM PSNR SSIM PSNR SSIM Goldhill SSIM PSNR SSIM PSNR SSIM Table 2. Reconstruction Time (in seconds) Obtained for Lena Image using Several EC Techniques for Isolated Loss CAD EC Technique MRF OAI SLP LSR ALP JLSNS CAD SLP LSR JNSLS Fig. 4. Subjective comparison for Lena by different EC techniques for 30% random block loss. The upper-right figure shows the loss pattern. the l 0 -regularization in the recovery process. The regularization parameters λ x, λ y, λ m, γ, and δ are empirically set to 0.01, 0,01, 0.1, 0.1, and 0.1 respectively. For the EC, the lost blocks are partitioned into 2 2 subblocks with 1-pixel-width overlap depth between adjacent subblocks. The PSNR and SSIM results obtained by different EC techniques can be found in Table 1 for three loss patterns. The run time of the proposed JLSNS-based EC is compared with the ones obtained for the state-of-the arts EC methods in Table 1 for the random loss pattern (30% loss) on a typical computer (3.2 GHz Intel Xeon Core and 8 GB Memory) using a non-optimized MATLAB implementation. It can be seen that the proposed algorithm improves the performance of the concealment for all three types of loss, both in terms of quality and complexity, especially for the random loss pattern which models the real-world scenarios. To verify the effectiveness of the proposed method, some visual results are presented in Fig. 4 for the random loss pattern (30% loss). The common reconstruction artifacts associated with the existing algorithms are the blocking and blurring effects. From Fig. 4 and Table 1, we can see that the proposed method provides very high quality image recovery. Further, it is able to reconstruct accurately structures as edges. OAI 4. CONCLUSION and SSIM values over 10 trials are reported for the random loss pattern. In our experiments, a number of training image patches of size 5 5 pixels, which are highly textured, are collected from a set of natural images that are randomly selected from the CVG-Granada dataset 2. A dictionary pair of size and a mapping matrix of size are learned using the algorithm proposed in [17]. The Lasso algorithm [19] is used for the l 1 -regularization in the dictionary learning algorithm, while the OMP algorithm [16] is used for 2 1http://decsai.ugr.es/cvg/dbimagenes An image error concealment technique based on joint local sparse representation and non-local self-similarity was proposed. In order to reveal the intrinsic relation between the lost areas and the correctly received neighboring areas, the proposed method performs sparse representation of the target patch, including missing and known neighboring pixels. Using a trained dictionary and a mapping function learned offline, the proposed method finds an optimal sparse representation for the EC purpose, leading to high quality image recovery. The objective and subjective performance of the proposed scheme validate its effectiveness for the error concealment applications. 9

5 5. REFERENCES [1] A. Akbari, M. Trocan, and B. Granado, Sparse recoverybased error concealment, IEEE Transactions on Multimedia, vol. pp, no. 99, [2] H. Asheri, H. R. Rabiee, N. Pourdamghani, and M. Ghanbari, Multi-directional spatial error concealment using adaptive edge thresholding, IEEE J CE, vol. 58, no. 3, pp , Aug [3] M. Kim, H. Lee, and S. Sull, Spatial error concealment for H.264 using sequential directional interpolation, IEEE J CE, vol. 54, no. 4, pp , Nov [4] J. Koloda, V. Sánchez, and A. M. Peinado, Spatial error concealment based on edge visual clearness for image/video communication, J. Circuits, Syst., Signal Process., vol. 32, no. 2, pp , Apr [5] Z. Rongfu, Z. Yuanhua, and H. Xiaodongl, Content-adaptive spatial error concealment for video communication, IEEE J CE, vol. 50, no. 1, pp , Feb [6] X. Li and M. T. Orchard, Novel sequential error-concealment techniques using orientation adaptive interpolation, IEEE J CASVT, vol. 12, no. 10, pp , Oct [7] J. Koloda, J. Ostergaard, S. H. Jensen, V. Sanchez, and A. M. Peinado, Sequential error concealment for video/images by sparse linear prediction, IEEE J MM, vol. 15, no. 4, pp , Jun [8] J. Liu, G. Zhai, X. Yang, B. Yang, and L. Chen, Spatial error concealment with an adaptive linear predictor, IEEE J CASVT, vol. 25, no. 3, pp , Mar [9] G. Zhai, X. Yang, W. Lin, and W. Zhang, Bayesian error concealment with dct pyramid for images, IEEE J CASVT, vol. 20, no. 9, pp , Sep [10] S. Shirani, F. Kossentini, and R. Ward, An adaptive markov random field based error concealment method for video communication in an error prone environment, in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Phoenix, AZ, Mar. 1999, pp [11] G. Zhai, J. Cai, W. Lin, X. Yang, and W. Zhang, Image errorconcealment via block-based bilateral filtering, in Proc. IEEE Int. Conf. Multimedia Expo (ICME), Hannover, Germany, Apr. 2008, pp [12] A. Akbari, M. Trocan, and B. Granado, Image error concealment using sparse representations over a trained dictionary, in Proc. IEEE Picture Coding Symposium (PCS), Nuremberg, Germany, Dec [13], Joint-domain dictionary learning based error concealment using common space mapping, in Proceeding of IEEE Conference on Digital Signal Processing (DSP), London, UK, Aug [14] J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman, Non-local sparse models for image restoration, in 2009 IEEE 12th International Conference on Computer Vision, Sept 2009, pp [15] W. Dong, L. Zhang, G. Shi, and X. Li, Nonlocally centralized sparse representation for image restoration, IEEE Transactions on Image Processing, vol. 22, no. 4, pp , April [16] Y. C. Pati, R. Rezaiifar, and P. S. Krishnaprasad, Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition, in Signals, Systems and Computers, Conference Record of The Twenty- Seventh Asilomar Conference on, Nov 1993, pp [17] S. Wang, L. Zhang, Y. Liang, and Q. Pan, Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis, in 2012 IEEE Conference on Computer Vision and Pattern Recognition, June 2012, pp [18] V. Varsa, M. M. Hannuksela, and Y.-K. Wang, Non-normative error concealment algorithms, ITU-T SG16, VCEG-N62, vol. 50, Sep [19] R. Tibshirani, Regression shrinkage and selection via the lasso, Journal of the Royal Statistical Society, Series B, vol. 58, no. 1, pp ,