Self-Learning of Edge-Preserving Single Image Super-Resolution via Contourlet Transform

Transcription

1 Self-Learning of Edge-Preserving Single Image Super-Resolution via Contourlet Transform Min-Chun Yang, De-An Huang, Chih-Yun Tsai, and Yu-Chiang Frank Wang Dept. Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan Dept. Electrical Engineering, National Taiwan University, Taipei, Taiwan Research Center for Information Technology Innovation, Academia Sinica, Taipei, Taiwan {d , b979000, Abstract We present a self-learning approach for single image super-resolution (SR), with the ability to preserve high frequency components such as edges in resulting highresolution (HR) images. Given a low-resolution (LR) input image, we construct its image pyramid and produce a superpixel dataset. By extracting context information from the super-pixels, we propose to deploy context-specific contourlet transform on them in order to model the relationship (via support vector regression) between the input patches and their associated directional high-frequency responses. These learned models are applied to predict the SR output with satisfactory quality. Unlike prior learning-based SR methods, our approach advances a self-learning technique and does not require the selfsimilarity of image patches within or across image scales. More importantly, we do not need to collect training LR/HR image data in advance and only require a single LR input image. Empirical results verify the effectiveness of our approach, which quantitatively and qualitatively outperforms existing interpolation or learning-based SR methods. Keywords-Super-resolution, self-learning, contourlet transform I. INTRODUCTION Super-resolution (SR) aims at synthesizing a highresolution (HR) image from its multiple or single lowresolution (LR) versions. Generally, SR approaches can be classified into two categories: reconstruction and learning (or example) based methods. Reconstruction-based SR approaches require image patches from several LR images (or within a single LR image) for constructing the SR output. As can be expected, proper registration and alignment techniques need to be performed for this type of SR methods [], [2]. Learning (or example) based SR approaches typically search for similar patches from training LR image data, and the corresponding HR patches are utilized to synthesize the final SR output (e.g., [3], [4]). Moreover, several works have been proposed to learn the relationship between LR/HR image pairs, and thus the observed models can be used for predicting the SR version from the input LR image (e.g., [5], [6], [9], [0], [7]). We note that, in [5], [7], contourlet transform has been applied to exploit directional high frequency components presented in training images. However, since the above learning based methods need to collect proper training LR/HR image data beforehand, the learned SR models and the associated performance will be sensitive to their choice of training data. As a result, these approaches might not generalize well to real-world SR applications. Recently, self-learning based approaches [2], [3], [4] have been proposed to synthesize the SR image using information extracted from a single LR input image. Glasner et al. [2] combines both classical and examplebased SR techniques, which search for similar patches within and across image scales to produce the SR output. Yang et al. extends this idea and learns a joint sparse coding dictionary for LR/HR images to obtain a refined result. Different from [2], Wang et al. [4] does not search for similar patches. Instead, they advocate the learning of context-specific information across images patches and use the observed models to predict the HR output. In this paper, we propose a novel self-learning approach for single image SR. To overcome existing SR approaches with blurred results, we utilize contourlet transform [8] and aim at preserving high frequency components in the SR output. In our proposed framework, we extract and learn context-specific information from a LR input and its image pyramid. We model the relationship between image patches of the same context category in terms of their directional frequency components. Once this learning process is complete, each patch of the input image will be used to predict the final SR output using the associated models. Different from prior SR methods considering contourlet transform [5], [7] or most learning-based SR methods, we do not require the collection of any training LR/HR image data in advance and we do not assume any specific image priors. With the proposed self-learning processing, we can produce satisfactory SR results given a single input LR image. Moreover, based on the use of contourlets for edge-preserving SR, we further integrate multi-processing techniques with a multicore PC, which results in very impressive processing time compared to existing learning-based SR methods. In the following sections, we will detail our proposed method, followed by quantitative and qualitative experimental results and comparisons with state-of-the-art learning-based SR approaches.

2 DFB Decomposition Contourlet Parallel Processing I SR I 0 U i U + H + H i I - C i- 2 C i- 3 C 4 i- C i- 2 Context Category I (a) (b) (c) (d) (e) Figure. Flowchart of our SR framework. (a) Input image I 0 and its lower-resolution versions (e.g., I, etc.). (b) Extraction and clustering of image segments in terms of their context information.(c) Deriving the up-sampled version U i from I i via bicubic interpolation, the associated high frequency component image H i, and four response images of directional frequency components {C j i, j =,..., 4} (Section II-B). (d) Learning of directional frequency and context specific models for SR with parallel processing techniques (Section II-B2). (e) Refining U + into I SR with H + using predicted {C j 0, j =,..., 4}. II. CONTEXT-AWARE SINGLE IMAGE SUPER-RESOLUTION WITH CONTOURLETS A. Context-Constrained Super-Pixel Categorization Inspired by Glasner et al. [2], who constructs an image pyramid from a single LR image input and searches for similar patches for SR, we advance this framework in our proposed method. Different from [2], we do not require the reoccurrence of image patches within or across scales. Instead, we focus on learning context information from such an image pyramid for edge-preserving SR, while no additional training LR/HR image data is needed. We now explain how we build the image pyramid from a single LR input image, and how we collect a super-pixel database from this pyramid for each context category. Given a LR image, we downgrade its resolution by bicubic downsampling to form images with different resolutions {I i }, as shown in Figure (a). Note that the resolution of each down-scaled image I i is a quarter of that of corresponding higher-resolution version I i+. To determine the context information for each image segment in this pyramid, we apply mean-shift [6] to over-segment each {I i } into superpixels. To categorize these super-pixels into different context categories, we extract the textural feature for each superpixel by calculating the responses of derivative filter banks with 6 different orientations and at 3 different scales (as suggested in [5]). The response of each filter is quantized into a histogram with 20 bins, and we obtain the final textural feature as a = 360 dimensional vector. Since the number of context categories for an image cannot be known a priori, we apply affinity propagation (AP) [7] to perform unsupervised clustering on the above super-pixels using their textural features. We note that, since AP divides the data instances into distinct groups simply by calculating the similarity between instances, it is an automatic and unsupervised clustering algorithm without the need to specify the number of clusters beforehand. In our proposed framework, we apply the χ 2 -distance to measure the difference between each super-pixel pair S i and S j, as suggested in [4]. AP thus determines the optimal clustering by maximizing the net-similarity N S between super-pixels: N N NS = c ijs(s i, S j ) () i= j= N N N N γ ( c ii)( c ij) γ ( c ij). i= j= In (), N is the total number of super-pixels in the image pyramid I i, s(s i, S j ) = exp( χ 2 (S i, S j )) measures the similarity between super-pixels S i and S j. The coefficient c ij = indicates that the super-pixel S i is the exemplar (i.e., the cluster representative) of the super-pixel S j, and thus S j is categorized to cluster i (and c ii equals since S i itself is the exemplar cluster i). While the first term in () calculates the similarity between super-pixels within each cluster, the second term penalizes the case when superpixels are assigned to an empty cluster (i.e. c ii = 0 but with N j= c ij ). On the other hand, the third term in () penalizes the condition when super-pixels belong to more than one cluster. In practice, γ is set to + to avoid the aforementioned problems. Once this clustering process is complete, as shown in Figure (b), we obtain the context label information for super-pixels extracted from the image pyramid I i. Note that image patches bounded by red rectangles in Figure (b) indicate the exemplars (representatives) of different context categories. i= j= B. Edge-Preserving SR via Contourlet Transform ) Modeling Directional High-Frequency Components with Contourlets: Since the human visual system is sensitive to the high frequency components of images, our proposed self-learning algorithm aims at recovering high-frequency components for the SR output. Among different techniques which extract edges, etc. high-frequency components from images, the contourlet transform [8] has a unique property of decomposing multi-resolution and multi-directional frequency components for image reconstruction. Compared to the wavelet transform which captures only edges present in

3 horizontal, vertical and diagonal directions only, the contourlet transform provides a higher degree of directionality and anisotropy by utilizing basis functions oriented at more directions and with more flexibility. As a result, it is able to preserve the geometric structure (e.g., edges and higherfrequency components) of an image. We now detail how we apply the contourlet transform to extract context-specific and directional high-frequency components from a given LR input image. From Section II-A, we construct a super-pixel database from the input image pyramid I i, and each super-pixel belongs to a specific context category. Take Figure 2 for example, for each lowerresolution image I i in the image pyramid, we upsample its resolution by bicubic interpolation and obtain a higherresolution version U i, which is of the same size as the image I i in the pyramid. Since the image U i is upsampled from I i, it misses some high-frequency details and thus is blurred. We subtract U i from the image I i in the pyramid, and the resulting image H i would correspond to the highfrequency components to be recovered from I i (when using bicubic interpolation). In order to model these highfrequency components, we apply a two-level directional filter bank (DFB) decomposition with 2 2 = 4 subbands on the image H i to produce a stack of four directional components {C j i, j =,..., 4} (also known as contourlets). More specifically, given the input high-frequency image H i, the four contourlets obtained from the DFB decomposition are calculated as: C j i = downsample(h i E j, S j ), j=,..., 4. (2) where E j is a fan-shaped filter bank with non-zero values in horizontal or vertical directions, and the downsample function downgrades the resolution of the convolution output H i E j using a diagonal matrix S j (see [8] for more details). Since the convolution and downsampling processes are performed once at each level at a particular direction, a total of 2 2 = 4 contourlets {C j i, j =,..., 4} at scale i will be produced (as shown in Figure 2), and their resolution will be a quarter of that of H i. The sample of derived four contoulets is shown in the lower right of Figure 2. It can be seen that, the size of these four contourlets is only a quarter of that of the input image I i, which is exactly the same as the ratio of the resolutions between adjacent image scales I i and I i. In our SR frameowrk, we perform contourlet transform for each image I i in Figure (a), and thus we construct the contourlets pyramid with four contourlets {C j i, i =, 2,... } for each scale, as shown in Figure (c). Together with our context information determined in Section II-A, we now obtain both context information and the associated contourlet for each image patch in the image pyramid (Figure (a)). For SR purposes, if we can properly predict the four directional components {C j 0, j =,..., 4} I i Bicubic Down-Sampling Low-Frequency Components High-Frequency Components U i I i- H i Bicubic Up-Sampling DFB... C i - 3 C i - 2 C i - 4 C i - Figure 2. A contourlet transform example decomposing I i into four contourlets {C j i, j =,..., 4} and the downsampled image I i (see Section II-B). from the LR input I 0, the high frequency component image H + can be synthesized by H + = 4 upsample(c j 0, Sj ) F j. (3) j= In contrast to the directional downsampling process in (2), the above equation reconstructs the image H + by convolving F j with the upsampled image of C j 0 using Sj. Note that F j are the synthesizing filter banks associated with E j and S j are the same operating matrices in (2). As a result, we can reconstruct the SR output using the recovered high-frequency image H + and the bicubic one U + (i.e., the up-sampled version of I 0 by bicubic interpolation). Therefore, the last step of our proposed method is to learn the relationship between the image patches of the same context category and their corresponding contourlet outputs. Once these models are observed, we can determine the context category for each image patch in I 0 (by the results of AP, as discussed in Section II-A), and apply the associated learning model to predict the contourlet. 2) Self-Learning of Context-Aware and Edge-Preserving SR Models: To learn the relationship between the image patches in I i of the same context category and their corresponding contourlet outputs C j i, we apply support vector regression (SVR) [] for this learning process. SVR has been observed an effective algorithm in fitting image data for SR purposes [6], [4]. For each context category given a LR input image, we consider their image patches in Figure (a) as the input data, and the corresponding conourlets in Figure (c) are the output labels to be observed. We extract the sparse representation of each image patch (of the same context category) as its feature vector, since sparse representation has been shown to produce promising results in image related applications especially SR [0]. To learn the sparse representation for an image patch, we solve the following optimization problem:

4 min 2 xg D g α g λ α g, g =, 2,..., k, (4) where x g is a 5 5 image patch (i.e., a 25 dimensional vector) of context label g (out of k categories), D g is the sparse coding dictionary to be learned, α g is the corresponding derived sparse coefficient vector, and the parameter λ balances the sparsity of α g. After the sparse representation features of image patches are calculated, the corresponding pixel values (at the same position of the center for the input patch) of the contourlets {C j i, j =,..., 4.} will be the label to be learned. We note that the pixel values of the contourlets indicate the intensity of the edge at a particular direction, which depicts the directional high frequency components for the image patches of a specific context category. We model the relationship between the sparse representation patches features and the contourlets by SVR, and thus we construct four SVR models for each context category. More precisely, we solve: n min w,b,ξ,ξ 2 wt w + C (ξ i + ξi ) (5) i= s.t. y g i (wt φ(α g i ) + b) ɛ + ξi, ξi 0, (w T φ(α g i ) + b) yg i ɛ + ξ i, ξi 0. In (5), y g i is the pixel values for the contourlet of context category g. The parameter b is the off-set of the regression model, n is the number of patches in context category g, and φ(α g i ) is the sparse representation of the input patch. The parameter w is the norm vector of the nonlinear mapping function, and C is the tradeoff between the generalization and upper/lower training errors ξ i /ξi with precision ɛ. Note that, we will have a total of 4k SVR models to learn given an input image (i.e., 4 contourlets k context categories). To accelerate the training process, we apply parallel processing techniques and significantly reduces the computation time, as shown in Figure (d). The procedure of recovering the SR output with the input I 0 is now described as follows. For each super-pixel in I 0, we first determine its context label using the AP clustering results on the pyramid I i (Figure (a)), and their sparse representation feature vector is calculated by (4). Next, we apply the corresponding SVR model to predict the contourlet for each patch, and four directional highfrequency components images {C j 0, j =,..., 4} will be obtained. These four contourlets will be used to reconstruct the high-frequency image H + with DFB reconstruction. As shown in Figure (e), this H + will be added back to the up-sampled image U + (by bicubic interpolation) to produce the final SR output. The advantage of learning the context-specific highfrequency components of the images without the intervention of the low-frequency ones is that we are able to achieve improved edge-preserving SR, as we verify later in Section III. Moreover, a self-learning framework like ours does not need to collect any training LR/HR image data beforehand (like most learning-based SR methods did), which makes our proposed method preferable in practical SR applications. III. EXPERIMENTAL RESULTS We apply our SR approach on images collected from USC-SIPI image database and Berkeley segmentation database [8]. The size of the test LR input image is pixels and the magnification factor is 2 (in one dimension). We note that the test images are obtained after applying Gaussian blur kernel on the original HR images ([2] also did this). For color images, only the illuminance channel in YIQ color space is used for SR, and the remaining color channels are up-sampled by bicubic interpolation. For learning sparse representation and SVR models (with Gaussian kernels), we apply the SPAMS package 2 and LIBSVM 3, respectively. The DFB decomposition and reconstruction are performed using the contourlet toolbox provided by Matlab 4. Moreover, for parallel processing implementation, we apply matlabpool to create a four-core environment for accelerating the computation time for our algorithm. Table I lists the PSNR values for a variety of images. To compare our approach with other existing interpolation or learning-based SR methods, we consider bicubic interpolation, locally linear embedding (LLE) for SR [4], learning image sparse representation for SR [9], a learning-based single image SR approach [4], and the method of Glasner et al. [2]. From this table, it is clear that our method achieved the best PSNR values for all images except for airplane. Compared to bicubic interpolation, we obtained an average PSNR improvement of 2.9% (or about db), which is remarkably better than other learning-based SR methods. We show example SR results and comparisons in Figures 3, 4, and 5. From these figures, we see that our approach produced HR images with clear edges and less artifacts that other methods did, and thus qualitatively better SR results were achieved. We observe similar remarks applying our method to synthesize higher-resolution images (with a magnification factor > 2). Furthermore, our proposed method achieved a very impressive SR efficiency compared with other learningbased approaches. Our SR algorithm reported an average processing (testing) time of 270 seconds (without parallel processing), which is significantly less than the other two single image learning-based methods [4] and [2]. For other learning-based approaches requiring training data (and thus the training time needs to be considered), ours is still much more computationally efficient ( seconds for [4] and over 0 5 seconds for [9]). It is worth repeating that, even these two methods only need to perform training Available at 2 Software available at 3 C.-C. Chang and C.-J. Lin, LIBSVM: a library for SVMs,

5 Table I PSNR AND COMPUTATION TIME COMPARISONS OF DIFFERENT SR METHODS. NOTE THAT BOTH THE AVERAGE TRAINING AND TESTING TIME (T T rain AND T T est ) ARE LISTED FOR EACH METHOD (IF APPLICABLE), AND THE RUN-TIME ESTIMATES (IN SECONDS) WERE OBTAINED ON AN INTEL QUAD CORE PC WITH 3 GHZ CPU AND 4G RAM. Training boat cars skyview lena person airplane child T T rain T T est Image Bicubic N/A N/A N/A LLE [4] Yes Yang et al. [9] Yes Wang et al. [4] N/A 4800 No Glasner et al. [2] N/A 5400 No Our method (without parallel processing) N/A 270 No Our method (with parallel processing) once, their performance will be sensitive to their collection of training data and thus might not be practical for realworld SR applications. Finally, once we integrate the parallel processing technique into our proposed SR framework, we only require about 80 seconds to synthesize a HR output using a single LR input image. From the above qualitative and quantitative comparisons, we can verify the effectiveness and efficiency of our proposed SR method. IV. CONCLUSION We proposed a self-learning approach for single image super-resolution with contourlet transform. By utilizing the contourlets to model directional high-frequency components present in an input LR image (and its image pyramid), our SR framework is able to learn context-specific highfrequency details of images patches extracted from different context categories. Using sparse representation as image features and support vector regression as the learning models, our proposed method models the relationship between image patches and the associated contourlets for each context category. These observed models are applied to predict the directional high-frequency components for each patch in the input LR image, and the results are used to synthesize/refine the final SR output. Our SR approach is very unique and different from conventional learningbased approaches, since we do not assume any image priors such as the reoccurrence/self-similarity of image patches, nor do we need to collect training LR/HR image data in advance. Our experimental results verified the effectiveness and efficiency of our method, and we showed that our approach qualitatively and quantitatively outperforms stateof-the-art learning-based SR methods. REFERENCES [] R. C. Hardie et al., Joint map registration and high-resolution image estimation using a sequence of undersampled images, IEEE Trans. Image Processing, 997. [2] S. Farsiu et al., Fast and robust multi-frame super-resolution, IEEE Trans. Image Processing, [3] W. T. Freeman, T. Jones, and E. Pasztor, Example-based super-resolution, IEEE Computer Graphics & Applications, [4] H. Chang, D.-Y. Yeung, and Y. Xiong, Super-resolution through neighbor embedding, in IEEE CVPR, [5] C. V. Jiji and S. Chaudhuri, Single-Frame Image Superresolution through Contourlet Learning, EURASIP J. Adv. Sig. Proc., [6] K. Ni and T. Q. Nguyen, Image superresolution using support vector regression, IEEE Trans. Image Processing, [7] K. P. Upla, P. P. Gajjar, M. V. Joshi, A. Banerjee and V. Singh, A fast approach for edge preserving super-resolution, in IEEE ICME, 20. [8] M. N. Do and M. Vetterli, The contourlet transform: an efficient directional multiresolution image representation, IEEE Trans. Image Processing, [9] J. Yang, J. Wright, T. Huang, and Y. Ma, Image superresolution via sparse representation, IEEE Trans. Image Processing, 200. [0] M.-C. Yang, C.-T. Chu and Y.-C. F. Wang, Learning sparse image representation with support vector regression for singleimage super-resolution, in IEEE ICIP, 200. [] V. Vapnik, Statistical Learning Theory, Wiley-Interscience, 998. [2] D. Glasner, S. Bagon, and M. Irani, Super-resolution from a single image, in IEEE ICCV, [3] C.-Y. Yang, J.-B. Huang and M.-H. Yang, Exploiting selfsimilarities for single frame super-resolution, in Asian Conf. Computer Vision, 200. [4] M.-C. Yang, C.-H. Wang, T.-Y. Hu and Y.-C. F. Wang, Learning Context-Aware Sparse Representation for Single Image Super-Resolution, in IEEE ICIP, 20. [5] J. Sun et al., Context-constrained hallucination for image super-resolution, in IEEE CVPR, 200. [6] D. Comaniciu and P. Meer, Mean shift: A robust approach toward feature space analysis, IEEE Trans. Pattern Analysis Machine Intelligence, [7] D. Dueck and B. J. Frey, Clustering by passing messages between data points, Science, [8] D. Martin et al., A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics, in IEEE ICCV, 200.

6 Figure 3. Example SR results (with a magnification factor of 2) and the corresponding PSNR values. Images from left to right: Ground truth, Yang et al. [9] (PSNR: 25.32), Glasner et al. [2] (PSNR: 27.64) and ours (PSNR: 27.92). Note that the method of Yang et al. [9] requires the training LR/HR image data, while Glasner et al. [2] and our approach are learning-based methods for single image super-resolution (i.e., no training LR/HR image data is needed). Figure 4. Example SR results (with a magnification factor 2) and the corresponding PSNR values. Images from left to right: Ground truth, Yang et al. [9] (PSNR: 28.86), Glasner et al. [2] (PSNR: 34.98) and ours (PSNR: 35.6). Figure 5. Example SR results (with a magnification factor of 2) and the corresponding PSNR values. Images from left to right: Ground truth, Yang et al. [9] (PSNR: 3.28), Glasner et al. [2] (PSNR: 25.54) and ours (PSNR: 36.33).