Single Image Super-Resolution via Sparse Representation over Directionally Structured Dictionaries Based on the Patch Gradient Phase Angle

Transcription

1 2014 UKSim-AMSS 8th European Modelling Symposium Single Image Super-Resolution via Sparse Representation over Directionally Structured Dictionaries Based on the Patch Gradient Phase Angle Mahmoud Nazzal, Faezeh Yeganli and Huseyin Ozkaramanli Electrical and Electronic Engineering Department Eastern Mediterranean University Gazimagusa, via Mersin 10, Turkey Abstract We propose a single-image super-resolution algorithm based on sparse representation over a set of cluster dictionary pairs. For each cluster, a directionally structured dictionary pair is designed. The dominant angle in the patch gradient phase matrix is employed as an approximately scaleinvariant measure. This measure serves for patch clustering and sparse model selection. The dominant phase angle of each low resolution patch is found and used to identify its corresponding cluster. Then, the sparse coding coefficients of this patch with respect to the low resolution cluster dictionary are calculated. These coefficients are imposed on the high resolution dictionary of the same cluster to obtain a high resolution patch estimate. In experiments conducted on several images, the proposed algorithm is shown to outperform the algorithm that uses a single universal dictionary pair, and to be competitive to the state-of-the art algorithm. This is validated in terms of PSNR, SSIM and visual comparison. Keywords single-image super-resolution, sparse representation, cluster dictionary pairs, gradient operator. I. INTRODUCTION Singe-image super-resolution (SR) is the process of reconstructing a high resolution (HR) image (I H ) from a low resolution (LR) image (I L ) corresponding to the same scene. The relationship between I L and I H is typically characterized by a blurring and downsampling operation, as described in (1). I L ΨI H, (1) where Ψ is the blurring and downsampling operator. It is well-known that this process is an ill-posed inverse problem. Approaches to SR can be classified into three main categories. First, interpolation methods which estimate the unknown pixel values based on the known ones. Several advanced interpolation techniques have been proposed to over-come the low-pass nature of the interpolation process. Second is the reconstruction-based methods, where constrains are applied and exploited in reconstructing the HR image. The third category is the learning-based approaches where a training process is carried out and then used during a testing process. The fundamental idea in these approaches is to utilize the correspondence between the training and testing sets. One of the most successful learning-based approaches is the sparse representationbased approach. In this framework, dictionaries are trained over example signals and then used in the testing process. In the context of sparse representation-based SR, sparsity of the representation of LR and HR image patches is used as a natural image prior to regularize the SR problem. This usage allows for assuming that the sparse coding coefficients of an image patch are scale-invariant as proposed by Yang et al. [1], [2]. Another prior that has been successfully employed in regard is the gradient operator. This operator has been employed as a natural image prior to solve ill-posed image processing problems such as SR [3], [4] and denoising [5]. The early usage of the gradient as a prior was proposed by Rudin et al. [6] in the total variation denoising framework. Another line of research considered approximating the gradient distribution such as [7] and [8]. More recent works have considered the invariance or preservation of the gradient histogram and applied it as a prior. An example is the work conducted by Zuo et al. [5] in image denoising, where they enforced the histogram of the denoised image to be similar to that of the noisy one. Another example is the work by Sun et al. [3], [4] which defines a gradient profile prior that is invariant to scale, and uses it as a generic image prior for SR. In view of the success of sparse representation in various signal and image processing applications, it is well-known that its major added-benefit is the signalfitting capability due to the use of learned dictionaries. Compared to fixed basis functions, such dictionaries are proven to better fit natural signals as they are trained over examples signals [9]. Since the representation is intended to be sparse, learned dictionaries have to be overcomplete (redundant) to allow for a good representation quality. However, high redundancy has its own drawbacks. It has been shown that having high dictionary redundancy tends to cause instability of the representation and therefore to degrade the representation quality [10], [11]. In other words, a good dictionary needs to possess a good capability in representing signals at a reasonable /14 $ IEEE DOI /EMS

2 degree of redundancy. Motivated by the above findings, several attempts have recently been made towards designing compact dictionaries without sacrificing the representation power. Intuitively, the variability of signals within a class is less than the general signal variability. A recent research trend considered learning class-dependent dictionaries. Examples of this trend include clustering the training and testing data as proposed by Dong et al. [12], where they applied K-means clustering for separating signals into several clusters, and learned cluster sub-dictionaries. The same clustering criterion is used to classify a signal into a cluster and use its sub-dictionary for the purpose of sparse coding. Along this line, researchers aimed at designing directionally structured dictionaries that correspond to directionally sensitive signal classes. In [13], Yang et al. employed multiple geometric cluster dictionaries. Each cluster is concerned with a certain directional structure. Sparse coding of a signal is carried out over the cluster dictionary that best fits this signal based on its structure. In [14], Yu et al. designed a structural dictionary composed of several orthogonal bases that correspond to different structures. Again, sparse coding of a signal is done by first selecting the best fitting basis (model) according to the signal s structure, and then calculating the sparse coding coefficient with respect to this basis. In this paper, we classify the training and testing signals into a set of directionally structured clusters. We employ the dominant phase angle (DPA) defined via the gradient operator as a classification criterion. We empirically show an acceptable degree of scale-invariance of the DPA as a natural image quantity. For each cluster, a directionally structured coupled dictionary pair is learned and then used for the purpose of sparse coding. We show that the designed dictionaries inherit the intended directional structure of their respective clusters. The proposed SR algorithm is shown to be superior to that algorithm of Yang et al. [2] that uses a universal dictionary pair. Besides, it is shown to be competitive to the state-of-the-art SR algorithms. This result is validated qualitatively in terms of the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) measures, and qualitatively in terms of visual comparisons. II. SINGLE IMAGE SUPER-RESOLUTION VIA SPARSE REPRESENTATION The sparse-representation SR algorithm of Yang et al. [2] is based on using patches of a LR image to reconstruct the underlying patches in the unknown HR image. Sparsity of the representation is applied as a generic natural image property that is invariant with respect to scale. Their work is based on applying two constraints on the reconstructed HR image. First is a reconstruction constraint enforcing the blurred and downsampled version of the HR image estimate to be consistent with the given LR image. Second, is a sparsity constraint which assumes that the sparse coding coefficients of a LR patch are similar to those of the underlying HR one, or scale-invariant in other words. In [1], the authors used a set of sampled patch pairs as a dictionary for sparse coding. However, this paradigm is very slow in practice. Therefore, in [2], Yang et al. employed a pair of coupled dictionaries learned from such patch pairs. They first calculated the sparse coding coefficients of a LR patch with respect to the LR dictionary. Then, they imposed these coefficients on the HR dictionary to find a HR patch estimate. For the sake of local consistency of reconstructed HR patches, it is advantageous to divide the LR image into overlapping patches [1], [2]. Correspondingly, the reconstructed HR patches are also overlapped. They are then merged at the overlap locations to generate a HR image estimate. The LR and HR dictionaries are simultaneously learned in a coupled manner. Patches of HR training images are extracted and column-stacked to form an array of HR training patches. The mean value of each patch is subtracted to allow for a better training. At the same time, LR images are obtained by applying a blurring and downsampling operation of the HR ones. Then, LR images are upsampled to the so called middle resolution (MR) to allow for better feature extraction.. Afterwards, first and second order gradient filters are operated over the MR images to extract the features. The next step is to combine the extracted features of each MR patch into a single column, and then to combine feature vectors column-wise to obtain the corresponding LR training patch array. Eventually, LR and HR training patches are used to train for a pair of coupled LR and HR dictionaries, respectively. This coupling is vital for the validity of the sparse coding coefficient invariance assumption. Given a HR vector patch x H, one may find the sparse coding coefficient vector α H of this patch over a dictionary in the same resolution level D H. Vector selection techniques such the least absolute shrinkage and selection operator (LASSO) [15] can be applied for this purpose. A sparse approximation of x H can be written as x H D H α H, (2) Analogously, one may obtain a sparse approximation for the corresponding LR patch x L. This requires a sparse coding coefficient vector α L over a LR dictionary D L. This approximation can be written as x L D L α L. (3) The same blurring and downsampling operator Ψ shown in (1) can be used to relate x L and x H. With the assumption that Ψ relates also the atoms of D L and D H, one 210

3 may write x L Ψx H ΨD H α H D L α H, (4) Based on the above analysis, the following result in concluded α H α L. Consequently, a reconstruction for x H can be obtained using D H and α L, as follows x H D H α L. (5) III. THE PROPOSED SUPER-RESOLUTION ALGORITHM In this section, we introduce the DPA as an approximately scale-invariant patch measure and empirically study its behavior with respect to scale. Then, we detail the training and testing stages of the proposed algorithm. Intuitively, phase values can characterize the directional structure of an image patch. One may define a dominant phase angle (DPA) in the phase matrix that best describes the directionality. This requires quantizing the the phase matrix and establishing a histogram for the quantized angles. It is empirically found that if a certain angle value is repeated more frequently than half of the number of elements in the phase matrix, then the patch can be considered a directional patch. The directionality of this patch is characterized by that dominant angle. Otherwise, it is considered as a non-directional patch. In this work, we opt to quantize angles into values of 0,45,90 and 135. TABLE I THE NUMBER OF HR PATCHES CLASSIFIED BASED ON DPA IN EACH CLUSTER (TOP) AND DPA SCALE-INVARIANCE RATIO IN THE SAME CLUSTER (BOTTOM). THE LISTINGS OF THE CLUSTER CONTAINING THE LARGEST NUMBER OF IMAGE PATCHES ARE IN BOLD FACE. Image C 1 C 2 C 3 C 4 C 5 Barbara BSDS Buildposter Butterfly Fence Flowers Lena ppt Text Image Average A. Approximate scale-invariance of the dominant gradient phase angle measure The image gradient operator is believed to be crucial to the perception and analysis of natural images [5]. In [3], [4], Sun et al. defined the gradient profile prior as a 1-D profile of gradient magnitudes perpendicular to image structures, and studied its behavior with respect to scale. They reported that this profile follows a certain distribution which is independent of image resolution. Then they applied this prior to the problem of single-image SR. It is well-acknowledged that the phase is generally more informative than the magnitude. It seems thus advantageous to think about exploiting information in the phase of the gradient operator. One can define the phase matrix of the gradient operator based on the horizontal (G h ) and vertical (G v ) gradients as follows Φ = arctan( Gv ). (6) Gh Figure 1. Test images from left to right and top to bottom: Barbara, BSDS , Buildposter, Butterfly, Fence, Flowers, Lena, ppt3 and Text Image 1. To investigate the impact of scale on DPA, the following experiment is conducted on the images shown in Figure 1. Each image is divided into non-overlapping 8 8 patches. A LR version of each image is obtained by filtering it with a bicubic filter and downsampling it by 2 in both dimensions. Clusters C 1 through C 4 are defined corresponding to DPA values of 0,45,90 and 135 respectively, while C 5 corresponds to patches that do not have a specific directional nature. Then the DPA value of each HR and LR patch is calculated as explained earlier. The DPA value of each patch is used to cluster it into one of the clusters. The DPA invariance is defined as the ratio between the number of LR patches correctly classified into a certain cluster and the total number of HR patches in that cluster. DPA invariance ratios for the images shown in Figure 1 are listed in Table I. One can see in Table I that the DPA invariance ratios are generally greater than 50 %. DPA invariance values are high for C 5, they are still high in C 1 and C 3 as compared to the cases of C 2 211

4 C 1 C 2 C 3 C 4 C 5 Figure 2. Reshaped example atoms of HR dictionaries in C 1 through C 5. and C 4. B. Clustering and sparse model selection with the dominant gradient phase angle measure In this work, we classify the training image patch pairs into the aforementioned five clusters. Then, we use the dictionary learning method of Yang et al. [2] to train for coupled LR and HR cluster dictionary pairs. Features extracted from the LR image patches at the MR level are used to train for the LR dictionary. The DPA of each LR patch at the MR level is used to classify it into a certain cluster. Then, the corresponding HR patch and the MR extracted features are inserted to the HR and LR training sets of that cluster, respectively. Algorithm 1 outlines the main steps of the training stage. In this setting, the purpose is to design cluster dictionaries that possess the intended directional structures of their respective clusters. Algorithm 1 The Proposed Dictionary Learning Stage. 1: INPUT: HR Training Image Set. 2: OUTPUT: A Set of Directionally Structured Cluster Dictionary Pairs. 3: Obtain a LR image for each HR one by blurring and downsampling. 4: Upsample each LR image to obtain a MR image. 5: Divide the HR and MR images into patches. 6: Extract features from the MR image by filtering. 7: Divide feature images into column patches. 8: Combine HR patches to form a HR training set and MR features to form a LR training set. 9: for Each patch in the LR training set, do 10: Calculate the DPA of the corresponding patch in the MR image, and identify the cluster number. 11: Set the MR features and the HR patch to the LR and HR training sets of this cluster. 12: end for 13: Learn a pair of coupled dictionaries for each cluster. In the reconstruction stage, a LR image is first upsampled with bicubic interpolation to the MR level. The MR image is then divided into overlapping patches. Next, the DPA of each MR patch is calculated and used to identify the cluster this patch belongs to. The next step is to calculate the sparse coding coefficients of the features extracted from the MR patch over the cluster LR dictionary. The corresponding HR patch is then reconstructed by imposing these coefficients on the HR dictionary of the same cluster. Algorithm 2 The Proposed Reconstruction Stage. 1: INPUT: A LR Image, Cluster Dictionary Pairs. 2: OUTPUT: A HR Image Estimate 3: Upsample the LR image to the required resolution level (MR). 4: Extract feature images from the MR image 5: Extract patches from the feature images and group them columnwise 6: Divide the MR image into overlapping patches. 7: for Each MR patch do 8: Calculate DPA of the MR patch. 9: Determine the cluster this patch belongs to. 10: Calculate the sparse coding coefficients of the corresponding feature vector over the cluster LR dictionary. 11: Reconstruct a HR patch as the product of the cluster HR dictionary and the calculated coefficients. 12: end for 13: Obtain a HR image estimate by merging overlapping HR patches. The final step is to reshape the overplaying HR patches and merge them to form a HR image estimate. A summary of the proposed reconstruction algorithm is outlined in Algorithm 2. Since the proposed algorithm relies on designing cluster dictionaries, they need not to be highly redundant. In his work, we choose to design 600-atom dictionaries as a good compromise between computational complexity and representation quality. This value is empirically determined to meet these requirements. The sparse coding stage is the most computationally expensive stage in the sparse reconstruction framework as it relies on vector selection [6]. Therefore, employing five compact dictionaries is expected to substantially reduce the sparse coding computational complexity and thus reduce the overall SR computational complexity, as compared to the case of employing a single highly redundant dictionary. As an example, Yang et al. [2] use a 1000-atom dictionary pair. However, the proposed algorithm requires calculating the DPA value of each MR patch. In view of these observations, it can be seen that the proposed algorithm s computation complexity is comparable to that of the algorithm of Yang et al. [2]. IV. EXPERIMENTAL RESULTS The performance of the proposed algorithm is compared to bicubic interpolation, the algorithm of Yang et al. [2] which uses a single dictionary pair and the algorithms 212

5 (a) (b) (c) (d) (e) (f) (g) (h) (i) Figure 3. Super-resolution comparison of a scene in the Butterfly image. (a) Original scene, (b) Bicubic interpolation result, (c) Peleg et al. s [16] result, (d) Yang et al. s [2] result, (e) He et al. s [17] result and (d) The proposed algorithm s result. The last row shows the difference between the original scene and (g) Yang et al. s result, (h) He et al. s result and (i) The proposed algorithm s result. of Peleg et al. [16] and He et al. [17] as state-of-theart algorithms. Comparisons are conducted in terms of the PSNR and SSIM [18] measures. In accordance with the common practice, we transform a color image into the luminance and chrominance color space and apply the SR algorithm only to the luminance component. The two chrominance components are reconstructed by bicubic interpolation. Accordingly, PSNR is calculated with the luminance components, where SSIM is calculated as the average of its values between the three color components. Results are obtained by running source codes obtained from the authors of [2], [16] and [17]. It is noted that we have paid attention to make equality of the simulation parameters of the four algorithms. Besides, each algorithm is set to work with the optimal parameters suggested by its authors. In this experiment, the proposed algorithm uses a 6 6 patch size, and five cluster dictionary pairs. We use the images in the Flickr image dataset [19] as a training set. Image patches are classified based on DPA into the five clusters. Then, 40,000 patch pairs are randomly selected and used to train for a cluster dictionary pair. The algorithm of Yang et al. [2] is set to the same parameters with the same training data set to train for a 1000-atom dictionary pair. The algorithms of Peleg et al. [16] and He et al. [17] are set to work according to the default values suggested by the authors. Figure 2 shows example reshaped atoms in the HR cluster dictionaries of the proposed algorithm. The atoms of these dictionaries clearly possess the directional structure of their respective clusters. From left to right, atoms of the first four dictionaries are generally perpendicular to the directions of 0,45,90 and 135 respectively. However, atoms in the fifth dictionary have a chaotic directional in accordance with the non-directional nature of C 5. Table II lists PSNR and SSIM values of the test images shown in Figure 1 with the aforementioned approaches. The performance of the proposed algorithm is shown with dictionaries of 600, 800 and 1000 atoms, as denoted by P 600, P 800 and, P 1000 respectively. With 600-atom dictio- 213

6 TABLE II QUANTITATIVE ASSESSMENT OF BICUBIC INTERPOLATION, THE ALGORITHMS OF PELEG ET AL. [16], YANGETAL. [2], HE ETAL. [17] AND THE PROPOSED ALGORITHM IN TERMS OF PSNR (TOP) AND SSIM (BOTTOM). Image Bicubic Yang [2] Peleg [16] He [17] P 1000 P 800 P 600 Barbara BSDS Buildposter Butterfly Fence Flowers Lena ppt Text Image Average naries, the proposed algorithm outperforms the algorithm of Yang et al. [2] and Peleg et al. [16] with average PSNR improvements of 0.19 db and 0.28 db, respectively. However, it is competitive to the algorithm of He et al. [17]. The same result is concluded in terms of SSIM. It is conclusive that using more redundancy in the dictionaries of the proposed algorithm will not significantly improve the performance. Besides, the proposed algorithm is particularly superior in handling images rich of edges in different orientations. In Figure 3 we compare our approach to bicubic interpolation and the algorithms of Peleg et al. [16], Yang et al. [2] and He et al. [17]. Figure 3 (g), (h) and (i) show the differences between the ground-truth scene and its reconstructions from Yang et al., He et al. and the proposed algorithm, respectively. Bicubic interpolation result is over-smooth. The reconstructions of Peleg et al. and Yang et al. have sharper edges. The algorithm of He et al. has further sharper edges. However, the proposed algorithm s reconstruction is the best to approximate the ground-truth scene with less artifacts. This is particularly observable in the edges and details of the butterfly s wing. This is assured by observing that the difference image of the proposed algorithm is the darkest one. V. CONCLUSION A new single-image super-resolution algorithm is proposed based on sparse coding over directionally-structured cluster dictionary pairs. Clustering is done based on the dominant angle in the gradient operator phase matrix. This angle is shown as an approximately scale-invariant patch measure. The same measure is used as a sparse model selection criterion in the reconstruction phase. With compact cluster dictionaries, the proposed algorithm is shown to be superior to the algorithm of Yang et al. [2] with an average 0.19 db PSNR improvement that uses a single dictionary pair. It is also shown to be competitive with the state-of-the-art algorithms. This result is validated in terms of PSNR and SSIM as quantities metrics and visual comparison. REFERENCES [1] J. Yang, J. Wright, Y. Ma and T. Huang, Image super-resolution as sparserepresentation of raw image patches, Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2008, pp [2] J. Yang, J. Wright, T.S. Huang and Y. Ma, Image super-resolution via sparse representation, IEEE Trans. Image Process., vol. 19, no. 11, 2010, pp [3] J. Sun, Z. Xu and H.Y. Shum, Imagesuper-resolution using gradient profile prior, Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2008, pp [4] J. Sun, Z. Xu and H.Y. Shum, Gradient Profile Prior and Its Applications in Image Super-Resolution and Enhancement, IEEE Trans. Image Process., vol. 20, no. 6, 2011, pp [5] W. Zuo, L. Zhang, C. Song, D. Zhang and H. Gao, Gradient Histogram Estimation and Preservation for Texture Enhanced Image Denoising, IEEE Trans. on Image Process., vol. 23, no. 6, 2014, pp [6] L.I. Rudin, S. Osher, and E. Fatemi, Nonlinear total variation based noise removal algorithms, Physica D, vol. 60, 1992, pp [7] R. Fergus, B. Singh, A. Hertzmann, S.T. Roweis and W.T. Freeman, Removing camera shake from a single photograph, In ACM Transactions on Graphics (TOG). vol. 25, no. 3, 2006, pp [8] A. Levin, Y. Weiss, F. Durand and W. T. Freeman, Efficient marginal likelihood optimization in blind deconvolution, Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2011, pp [9] J. Mairal, F. Bach, J. Ponce and G. Sapiro, Online learning for matrix factorization and sparse coding, J. Mach. Learn. Res., vol. 11, 2010, pp [10] D. Thanou, D. I Shuman and P. Frossard, Parametric dictionary learning for graph signals, IEEE Glob. Conf. Signal Inf. Process. (GlobalSIP), 2013, pp [11] S. Mallat and Y. Guoshen, Super-resolution with sparse mixing estimators, IEEE Trans. Image Process., vol. 19, no. 11, 2010, pp [12] W. Dong, L. Zhang, G. Shi and X. Wu, Image deblurring and super-resolution by adaptive sparsedomain selection and adaptive regularization, IEEE Trans. Image Process., vol. 20, no. 7, 2011, pp [13] S. Yang, M. Wang, Y. Chen and Y. Sun, Single-image superresolution reconstruction via learned geometric dictionaries and clustered sparse coding, IEEE Trans. Image Process., vol. 21, no. 9, 2012, pp [14] G. Yu, G. Sapiro and S. Mallat, Image modeling and enhancement via structured sparse model selection, Proc. IEEE Int. Conf. Image Process. (ICIP), 2010, pp [15] R. Tibshirani, Regression shrinkage and selection via the lasso, J. Royal. Statist. Soc. B, vol. 58, 1996, pp [16] T. Peleg and M. Elad, A statistical prediction model based on sparse representations for single image super-resolution, IEEE Trans. Image Process., vol. 23, no. 6, 2014, pp [17] H. Li, H. Qi and R. Zaretzki, Beta process joint dictionary learning for coupled feature spaces with application to single image super-resolution, Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2013, pp [18] Z. Wang, A.C. Bovik, H.R. Sheikh and E.P. Simoncelli, Image quality assessment: From error visibility to structural similarity, IEEE Trans. Image Process., vol. 13, no. 4, 2004, pp [19] oads.htm 214