Iterated Denoising for Recovery Onur G. Guleryuz Epson Palo Alto Laboratory 3145 Proter Drive, Palo Alto, CA 94304 oguleryuz@erd.epson.com July 3, 2002 Abstract In this paper we propose an algorithm for image recovery where completely lost blocks in an image/video-frame are recovered using spatial information surrounding these blocks. Our primary application is on lost regions of pixels containing textures, edges and other image features that are not readily handled by prevalent recovery and error concealment algorithms. The proposed algorithm is based on the iterative application of a generic denoising algorithm and it does not necessitate any complex preconditioning, segmentation, or edge detection steps. Utilizing locally sparse linear transforms and overcomplete denoising, we obtain good PSNR performance in the recovery of such regions. In addition to results on image recovery, the paper provides further insights into the usefulness of popular transforms like wavelets, wavelet packets, Discrete Cosine Transform (DCT) and complex wavelets in providing sparse image representations. 1 Introduction In many image and video compression applications the decoder has to contend with channel corrupted data and has to resort to image recovery and error concealment algorithms. Throughout the years many techniques have been proposed that enable decoders to avoid the severe consequences of channel errors (the reader is referred to [1, 2] for extensive reviews). In the case of images the lost or erroneous data has to be predicted spatially while for video both temporal and spatial prediction can be attempted. In this paper we will primarily be concerned with the recovery of lost data using spatial prediction alone. As such, for video, the presented techniques are directly applicable in cases where temporal prediction is not possible or prudent, for example, in cases involving severely corrupted motion vectors and/or intra marked macroblocks in the popular MPEG algorithms [3, 4]. We will primarily concern ourselves with the
recovery of completely lost image blocks though our algorithms can be adapted to situations where partial information is available and the lost data corresponds to nonrectangular or irregularly shaped regions. Of particular interest to us is the robust recovery of image blocks that contain textures, edges, and other features that are not readily handled by prevalent algorithms [1, 2]. While visual appearance and uniformity will be important, we will mainly be after significant PSNR improvements in recovered regions. Figure 1 illustrates some examples of recovery over edge and texture regions by the algorithms proposed in this paper. ORIGINAL LOST 16x16 BLOCK FILLED WITH LOCAL MEAN RECOVERED PSNR= 5.11 db PSNR= 16.55 db PSNR= 25.92 db (complex wavelets) PSNR= 7.43 db PSNR= 20.00 db PSNR= 28.02 db (complex wavelets) PSNR= 3.71 db PSNR= 17.93 db PSNR= 29.03 db (DCT 9x9) Locations of blocks used in most of the examples. PSNR= 8.91 db PSNR= 22.59 db PSNR= 26.24 db (DCT 16x16) Figure 1: Some examples of the proposed recovery algorithm over 16 16 lost blocks from various regions of the standard image Barbara. The main recovery algorithms utilized in this paper are based on denoising using overcomplete transforms. As will be seen, this is a slightly different application of denoising which is conventionally used to combat additive noise [6, 7, 8]. When we say denoising using overcomplete transforms, we mean it in its simplest form where several complete transformations are evaluated over a target area of interest, the transform coefficients hard-thresholded and inverse transformed to obtain several partially denoised regions, which are then averaged to obtain the final result. Similar to established denoising techniques, the role of the utilized transformations will be very important. In fact, the success of our algorithms is dependent on sparse image representations with the utilized linear transforms. One of the contributions of this paper will therefore be the identification of pros and cons of various linear
transformations in sparse image representation. The paper is organized as follows: Section 2 gives details of the main algorithm in order to facilitate later discussion. The rationale for the use of denoising for image recovery is examined in Section 3. Section 4 considers the role of the transformation in the recovery of various types of image regions. The spatial predictability of the lost block and its effects on the performance of the proposed algorithm is discussed in Section 5. Section 6 provides simulation parameters followed by Section 7 of concluding remarks. 2 Main Algorithm Before proceeding into a detailed discussion it is convenient to discuss the base algorithm utilized in this work in order to facilitate later results. The algorithm starts by grouping the pixels in the lost block into layers as shown in Figure 2. Layers are Lost Block = Layer 0 Layer 1 Layer 2 Layer 3. Figure 2: Layers of pixels in recovery. Each layer is recovered using the preceding layers surrounding it. recovered in stages with each layer recovered by mainly using the information from the preceding layers, that is, layer 0 is used to recover layer 1, layers 0 and 1 are used to recover layer 2 and so on. The layer grouping in Figure 2 is of course one possibility, and many different groupings can be chosen depending on the size and shape of the lost blocks. Beyond the grouping into layers and associated recovery of layers in stages, the main steps of the algorithm amount to evaluating several complete transforms over the target layer, selective hard-thresholding of transform coefficients, inverse transforming to generate intermediate results, and finally averaging and clipping (see below) to obtain the final recovered layer. Starting with an initial threshold T 0, these steps are carried out iteratively where at each iteration the threshold is reduced and the layers are recovered to finer detail using the new threshold. Prior to the first iteration, pixels in the lost block are assigned initial values, for example, the mean value computed from the surroundings of the outer boundary of layer 1. As will be discussed later, denoising by hard-thresholding is an energy dissipating operation, which makes the success of our algorithm dependent on layering and the selective application of hard-thresholding. Fix an initial threshold T = T 0 > 0
(for example the standard deviation computed from the surroundings of the outer boundary of layer 1) and consider the recovery of layer P using this threshold. For the sake of simplicity assume that we will be using an M M DCT and its M 2 1 overcomplete shifts in the denoising operation. In recovering layer P we will assume that all pixels in the image that are not in layer P, i.e., all pixels that are in preceding and subsequent layers, have fixed values. Suppose the initial DCT corresponds to DCT (M M) TILING 1 in Figure 3. We start by evaluating the DCT coefficients. y k th DCT block o y(k) Outer Border of Layer P o (k) x Hard threshold k thdct block coefficients if o y(k) < M/2 OR o x(k) < M/2 DCT (MxM) TILING 1 x Figure 3: An example DCT tiling and selective hard-thresholding. Then for each DCT block we check to see whether this block overlaps layer P. If there is no overlap, then hard-thresholding of the coefficients of this block is not carried out. If there is an overlap, we calculate the extent of the overlap with the outer border of layer P in x and y directions to yield o x (k) and o y (k) for the k th DCT block (Figure 3). If o x (k) M/2 or o y (k) M/2 then the coefficients of the k th DCT block are hard-thresholded. After this selective hard-thresholding we inverse transform to generate the first, partially denoised, layer P pixels. We then repeat this operation for the remaining M 2 1 overcomplete transforms, each generating a partially denoised result for the pixels in layer P (Figure 4 illustrates the selective thresholding for some other example overcomplete shifts of the M M DCT). The final result for layer P y y y y Coefficients undergo hard thresholding Coefficients are not hard thresholded x x x x DCT (MxM) TILING 1 DCT (MxM) TILING 2 DCT (MxM) TILING 3 DCT (MxM) TILING 4 Figure 4: Example DCT tilings and selective hard-thresholding. is obtained by averaging the partial results followed by clipping the layer pixels to the valid pixel value range (0 255 for grayscale images). Of course our lavish use of computation is for the sake of exposition and not all DCT coefficients need to be evaluated to generate the final result.
Once we generate the denoised results for all layers starting from layer 1, we reduce the threshold (T T T ) and repeat the whole process. The algorithm terminates when T reaches a preselected final threshold T F. Note that we may choose to consider only a subset of the overcomplete shifts and/or utilize different transforms. For other transforms like wavelets, wavelet packets, etc., the operation of the algorithm is the same except that the accounting for the overlap calculations becomes slightly more involved during selective hardthresholding. Since these transforms have basis functions of varying support for each coefficient (depending on the band), we use the support and location of the transform basis function resulting in the particular coefficient during the selective thresholding of that coefficient. 3 Denoising Using Thresholding and Recovery Denoising using thresholding and overcomplete transforms is an established signal processing technique primarily used to combat additive noise that is assumed uncorrelated to the signal of interest. Clearly we can view a lost block as the result of a particular noise process acting on the original image. However, unlike the traditional case, this noise process is not uncorrelated with the original image. Regardless, the basic intuition behind denoising carries forward to the presented algorithm, which tries to keep transform coefficients of high signal to noise ratio (SNR) while zeroing out coefficients having lower SNR. Let c denote a transform coefficient of the original image and let ĉ be the corresponding coefficient in the noisy image, i.e., the image containing the missing block. We have where e denotes the noise. Observe that and we also have ĉ = c + e (1) ĉ 2 = c 2 + 2ce + e 2 c 2 + 2 c e + e 2 (2) σ 2 ĉ = σ 2 c + 2E[ce] + σ 2 e σ 2 c + 2σ c σ e + σ 2 e (3) for the variances, assuming zero mean random variables. The difference from the traditional case manifests itself in the non-zero correlation term, E[ce], in Equation 3. Our primary assumption in this paper is that the linear transformation used to generate the transform coefficients mostly ensures that if ĉ is hard-thresholded to zero with T σ e, then with high probability c << e, i.e., hard-thresholding ĉ removes more noise than signal. It is clear that the algorithm presented in Section 2 makes
changes to the lost block by means of the transform coefficients hard-thresholded to zero. Hence, for the successful operation of the algorithm we need a transformation that typically generates many small valued transform coefficients over image regions of interest, i.e., we need sparse representations of images by the utilized transforms. However, as will be seen in Section 4, sparse image representation is not the only requirement from the utilized transform, and depending on the region surrounding the lost block, further properties will become critical in correct recovery. Assuming that portion of the noise is eliminated after the initial recovery of the layers, the threshold is lowered in order to make further improvements in recovery. Arrange the noisy image into a vector x (N 1), and let H 1, H 2,... H n (N N) denote the orthonormal, overcomplete transformations utilized in the denoising. Let G = [H 1 H 2... H n ] (N nn) denote the compound transformation. The denoised image with respect to these transformations is then given by 1 n GSGT x (4) where S (nn nn) is a diagonal selection matrix of zeros and ones, indicating which coefficients are hard-thresholded to zero. Since coefficient magnitudes are nonincreasing after multiplication with S, it is clear that denoising cannot increase signal energy, and it is in general energy dissipating. A straightforward consequence of this is that the symmetric, positive semidefinite matrix GSG T has all of its eigenvalues between zero and one. Let P 1 denote the projection to layer 1, i.e., the i th component of P 1 x is equal to the i th component of x if it is in layer 1, and zero otherwise. Then the image after the algorithm is run for the first time for layer 1 is given by D 1,T x = (1 P 1 )x + 1 n P 1GS 1,T G T x (5) where the subscript is introduced to keep track of the layer and the threshold used in hard-thresholding. Note that while the general denoising process in Equation 4 is energy dissipating, the presented recovery algorithm does not have this property due to the projections in Equation 5, i.e., since denoising is only allowed to affect a layer at a time, the pixel values in this layer can have increased energy without the consequence of having decreased energy at the remaining pixels. 4 Importance of Transforms We have noted in Section 3 that a sparse image representation by the utilized transform is required for successful operation of the algorithm. Unfortunately, this is not the only requirement. Clearly, recovery of a lost block is an ill-posed problem in the sense that many possible solutions exist and one has to pick among the possibilities
using some criterion. Let us distinguish between two cases: The first case is where the surrounding spatial information provides sufficient clues for the prediction of the lost block, and the second case is where sufficient clues are absent. We defer the latter case to Section 5 and concentrate on the former case here. The reader should consult Section 6 for details regarding the simulation examples. 4.1 Performance Over Periodic Features We begin by considering a locally periodic region surrounding the lost block. The arguments presented here can be straightforwardly extended to the random setting by using cyclostationary processes. In this case it is clear that the transform should have sufficient size to deduce the periodicity from the surroundings. Since spatial periodicity by S in a given direction implies frequency components separated by 1/S in the corresponding direction, it is also clear that the utilized transform should have the corresponding frequency selectivity. This is confirmed in Figure 5 especially on the recovery of the artificially generated pattern (periodic by 8 pixels) which is recovered perfectly with 8 8 DCTs and wavelet packets. ORIGINAL LOST 16x16 BLOCK FILLED WITH LOCAL MEAN RECOVERED 15.02 db 18.03 db 13.26 db (DCT 4x4) Perfect Rec. (DCT 8x8) 15.27 db (Wavelets) Perfect Rec. (Wavelet Packets) 8.91 db 22.59 db 26.24 db (DCT 16x16) 22.92 db 24.63 db 22.88 db (Wavelets) (Wavelet Packets) (Complex Wavelets) Figure 5: Examples of the proposed recovery algorithm over periodic features. Top row artificial pattern, bottom row from standard image Barbara. 4.2 Performance Over Edges Suppose the lost block is over two constant valued regions separated by a line. In this case it is clear that the utilized transformation should have sufficient size to determine the slope of the line and its intersection with the lost block, preferably with directional sensitivity such as that provided by complex wavelets [5]. Note that over constant regions, even very small sized DCTs will provide sparse representations but they will lack the ability to correctly interpolate the separating line as illustrated in the first set of simulations in Figure 6 for 4 4 and 5 5 DCTs, and in the second set for 8 8 DCTs. For wavelets and DCTs we have observed good performance over edge regions primarily when edges are horizontal, vertical and at + 45 0 diagonals.
Directional sensitivity of complex wavelets seems to provide an advantage in other cases. ORIGINAL LOST 16x16 BLOCK FILLED WITH LOCAL MEAN RECOVERED 5.11 db 16.55 db 22.75 db (DCT 4x4) 22.01 db (DCT 5x5) 25.05 db 25.92 db (DCT 16x16) (Complex Wavelets) 5.46 db 14.91 db 13.87 db 27.13 db 25.99 db 31.63 db (DCT 8x8) (DCT 16x16) (Wavelets) (Complex Wavelets) 4.31 db 15.68 db 19.06 db 22.33 db 24.94 db 24.54 db (DCT 8x8) (DCT 16x16) (DCT 24x24) (Wavelets) Figure 6: Examples of the proposed recovery algorithm over edge regions. 5 Spatial Predictability and Thresholds As illustrated in Figure 7 (a), in cases where the spatial surroundings do not provide enough clues about the distribution of pixels in the lost block, the presented algorithm does not have favorable PSNR performance. However, predictability also has an implicit effect even under more favorable circumstances as shown in Figure 7 (b). Observe that the PSNR performance of the algorithm is degraded beyond a certain point, where with reduced thresholds the algorithm tries to recover detail components. As can be seen the added detail beyond iteration 50 is mostly noiselike and appears to be spatially unpredictable. In such cases it is clear that the final threshold T F must be chosen in a way to avoid this problem. 6 Simulation Details In our simulations, we have used the main algorithm of Section 2 with different transforms as follows: All wavelet transforms are evaluated to 4 levels, including wavelet packets (full tree) and complex wavelets (Antonini 7 9, followed by 6-tap H L as in [5]). The (complex) wavelet transforms are fully overcomplete at the first level, and critically decimated thereafter, resulting in 4 redundancy. DCT based results are fully overcomplete. Because of the choice of rectangular shells as layers, and the nature of iterations, four-fold symmetry (left-right/up-down/diagonal) becomes important. While DCTs already have such a symmetry built-in to the basis functions and the utilized complex wavelets lead to four-fold symmetry by construction,
ORIGINAL LOST 16x16 FILLED WITH BLOCK LOCAL MEAN RECOVERED 15.2 15 14.8 14.6 14.4 ORIGINAL LOST 16x16 BLOCK 6.02 db 14.96 db (a) FILLED WITH LOCAL MEAN 13.96 db (DCT 16x16) RECOVERED 14.2 14 PSNR Iterations 13.8 0 50 100 150 200 250 300 350 400 16.5 16 15.5 15. 14.5 14 13.5 5.17 db 11.10 db (b) 15.21 db (DCT 16x16) 16.30 db (DCT 16x16 50 iterations) 13 12.5 12 PSNR Iterations 11.5 0 100 200 300 400 500 600 Figure 7: Spatial Predictability. (a) Recovery is not successful. (b) Successful recovery but spatially unpredictable details added at later iterations reduce performance. 128 128 images (Food.0002, Fabric.0009) from the Vistex [9] database. the orthogonal wavelet banks (Daubechies 8-tap) do not posses such a symmetry. For orthogonal wavelet banks we have thus modified the algorithm to repeat the computations for each layer four times, averaging the results of the regular and the left-right/up-down/diagonal flipped versions of relevant image regions. In all simulations the initial threshold was set as the standard deviation of pixels making up the one pixel boundary immediately outside layer 1. The final threshold T F = 5 and T =.1 throughout, except for the rightmost image in Figure 7 (b), which shows the result after only 50 iterations. Magnitudes of complex wavelet coefficients are thresholded. We have observed that typically most of the PSNR improvements are obtained in the initial iterations. 7 Conclusions and Future Work We have proposed an image recovery algorithm that relies on conventional denoising using thresholding and linear transforms with sparse image representations. Simulation examples demonstrate that the proposed algorithm is particularly effective on locally uniform regions and has good robustness to edges. While not detailed, the algorithm s performance over smooth regions is also very favorable. Over uniform regions, transforms with good frequency selectivity are preferred while over edge separated regions directional sensitivity of a transform also becomes important. The proposed algorithm is mostly intended for ranges of compression that provide good image quality so that the performance of the algorithm on texture regions, etc., (which may become indiscernible at very coarse compression ratios) can be taken advantage
of. Since the algorithm only changes a layer at a time it is clear that one can achieve significant savings in computation by suitable buffering and updating of transform coefficients. Note also that if one views the iterative use of denoising and projection onto layers as a sequence of alternating projections, it can be seen that the proposed method is substantially different from approaches such as [10], since the operation of the presented algorithm cannot be simplified to a sequence of projections onto fixed, data-independent, convex or nonconvex sets. Further issues related to the sensitivity to compression, computational complexity, selection of thresholds, and other results on the optimum transforms to utilize with the presented algorithm will be examined in another paper. References [1] Y. Wang, and Q-F. Zhu Error Control and Concealment for Video Communication: A Review, Proceedings of the IEEE, vol. 86, no. 5, May, 1998, pp. 974-997. [2] Y. Wang, S. Wenger, J. Wen, and A. K. Katsaggelos, Error Resilient Video Coding Techniques, IEEE Signal Processing Magazine, July, 2000, pp. 61-82. [3] ISO/IEC International standard 13818-2, Generic Coding of Moving Pictures and Associated audio information: Part2 - Video, 1995. [4] R. Koenen, Overview of the MPEG-4 standard, ISO/IEC JTC1/SC29/WG11 N1730, July 1997. [5] N. G. Kingsbury, A Dual-Tree Complex Wavelet Transform with improved orthogonality and symmetry properties, in Proc. IEEE Conf. on Processing, Vancouver, September 11-13, 2000, paper 1429. [6] D. L. Donoho, Denoising by soft-thresholding, IEEE Trans. Inf. Theory, vol. 41, pp. 613-627, 1995. [7] R. R. Coifman and D. L. Donoho, Translation invariant denoising, in Wavelets and Statistics, Springer Lecture Notes in Statistics 103, pp. 125-150, New York:Springer-Verlag. [8] P. Moulin and J. Liu, Analysis of Multiresolution Denoising Schemes Using Generalized - Gaussian and Complexity Priors, IEEE Trans. Info. Theory, Vol. 45, No. 3, pp. 909-919, Apr. 1999. [9] http://www-white.media.mit.edu/vismod/imagery/visiontexture/vistex.html [10] H. Sun, and W. Kwok, Concealment of Damaged Block Transform Coded s Using Projections onto Convex Sets, IEEE Trans. on Proc., Vol. 4, No. 4, Aug. 1995.