Improved Super-Resolution through Residual Neighbor Embedding

Transcription

1 Improved Super-Resolution through Residual Neighbor Embedding Tak-Ming Chan 1 and Junping Zhang Shanghai Key Laboratory of Intelligent Information Processing, Department of Computer Science and Engineering, Fudan University, , China { ,jpzhang}@fudan.edu.cn 2 The Key Laboratory of Complex Systems and Intelligence Science Institute of Automation, Chinese Academy of Sciences, Beijing, , China Abstract. In this paper we first present the machine learning methods applied to the single-image super-resolution issue. Reviewing the novel learning approach of super-resolution through neighbor embedding based on training images, we propose an revised method learning about the residuals from different level, inspired by image pyramids from multiresolution processing. Experiments on gray-level and color images show that our revised method yields results with lower Root-Mean-Squared errors and preserves high spatial-frequency information better such as sharp edges and high contrast. 1 Introduction Super-resolution is the problem of generating high-resolution enlargements of pixel-based images from one or multiple low resolution images. Applications include resolution enhancement for digital images such as satellite images and images received via CCTV (Closed Circuit Television) or mobile phone, saving storage for web thumbnail images, viewing low-resolution programs on highdefinition television, etc. Super-resolution also has potential value for the preprocessing of low-resolution input for biometrics applications such as face recognition [1]. Traditional resolution enhancement methods concerning spatial frequency mostly refer to interpolation approaches, bilinear and cubic spline, for example, which are common yet suffer from blurring and perceived loss of details. The reason is that they treat different types of image information with the same manner and smoothing all masked regions no matter they consist of edges or textures. Other than traditional interpolation approaches, recent superresolution algorithms based on machine learning take advantage of manifold learning and training images to preserve spatial-frequency and recover information of details such as edges and textures. From this viewpoint, there is loss of information beyond low-resolution images and the task of super-resolution is to retrieve the missing information correctly from training samples and patterns. There are two primary ways of super-resolution based on machine learning: while generating a high-resolution image from a sequence of low-resolution images (a scene) (e.g., [2, 3]) is referred to as super-resolution, generation from

2 2 a single low-resolution image can be specified as single-frame or single-image super-resolution [4, 5]. In this paper, we focus on the latter way of single-image super-resolution. 2 Single-image Super-Resolution with Machine Learning Some research groups have applied machine learning to address the singleimage super-resolution problem. The well known one among them is William T. Freeman s group in the Artificial Intelligence Lab at Massachusetts Institute Technology. Generally speaking, the main concept of single-image superresolution relies on a carefully selected training set of high-resolution images. The training images are down-sampled, divided up into low-resolution patches, each one forming a training pair with the corresponding high-resolution patch. Such super-resolution algorithms try to learn the connection between the lowresolution patches from test images and the ones from training set, and try to reconstruct the high-resolution target employing the according high-resolution training patches. There is a connection to manifold learning and dimensionality reduction here, implicitly existing between the low-resolution and high-resolution training pairs. But the dimensionality transformation is backward in this problem. We are given low-dimensional image patches and we need to interpolate their high-dimensional counterparts. Different single-image super-resolution approaches share similar concepts as mentioned above, but vary in their ways of choosing effective features, designing mechanisms to find appropriate matching patterns, and so on. Freeman et al presented the method which first generates the high-resolution estimate with traditional interpolation, second finds the missing high-frequency information from training images and finally combine the estimate with high-frequency information to obtain the target image [4]. Kim et al proposed an revised method adapting Kernel Hebbian Algorithm for feature representation which had comparable performance particularly for images of faces and natural scenes [6]. Chang et al brought forward a novel method employing YIQ space and gradients as features, as well as searching more than one nearest neighbors to generate high-resolution patches to yield good reconstructions [5]. 3 Super-Resolution through Neighbor Embedding Since our proposed approach is based on the method of super-resolution through neighbor embedding(srne) [5] and it will be briefly introduced for better understanding our work. Chang et. al. developed this method based loosely on LLE [7]. Generally, they match each given low-resolution test image patch to its k nearest neighbors other than only one neighbor in the low-resolution training set, and then form the high-resolution image patch using a weighted sum of the high-resolution training set. Based on simple geometric intuitions of locally linear embedding, the local geometry of one patch in a low-resolution image can be represented by locally linear weighted sum of its nearest neighbors. And optimal weights can be calculated based on least square criterion. From manifold

3 3 learning point of view, data in low-dimensional subspace should have as similar neighborhood relationship as in corresponding high-dimensional space [8, 9]. So the high-resolution patch can be computed by the weighted sum of the highresolution ones corresponding to those low-resolution nearest neighbors we find. SRNE adopts the ideas, and what s more, it employs elaborate techniques, using 1st and 2nd gradient operators as feature vectors and smoothing overlapped patches. More details can be seen in [5]. 4 Proposed Method Using Residuals Neighbor Embedding Results of SRNE are smooth and rich in textures, however, high spacial-frequency details such as edges are not sharply preserved and the Root-Mean-Squared(RMS) errors can be further reduced, which has the have the formulation of RMSe = ( n (ŷ i y i ) 2 ) 1 2 (1) n i=1 Where ŷ i stands for the values of pixel in the ideal target Y and y i stands for the values of corresponding pixels in output Y t. And n stands for the number of total pixels in Y. For preserving the high frequency details such as sharp edges, we are inspired by the multi-resolution processing, especially the residuals from image pyramids, a collection of images with decreasing resolution arranged in the shape of a pyramid [10]. Instead of only using the luminance(y) component from YIQ color space to find the nearest neighbors, our method applies residuals as feature vectors to super-resolution through neighbor embedding. First input is upsampled by traditional interpolation. Second we choose residuals from various levels of images of multi-resolution to do the matching and thus employ the embedding to generate the high-resolution residual. Third we combine the high-resolution residual and the upsampled input to obtained the final result. Suppose we have to reconstruct the nx (n times in each dimension) magnification Y t for the low-resolution input X t. First,we downsample X t to the lower level image (1/nX of X t ) and employ simple interpolation method (e.g.,neighbor replication or bilinear interpolation) to generate its upsampled nx estimate ˆX t, which remains the same size with X t. Second, we compute the residual ResiX t between X t and ˆX t. Similar processes are applied to Y s and X s, generating ResiY s between Y s and estimate upsampled from X s, as well as ResiX s between X s and estimate ˆX s. Ŷt is also prepared using simple interpolation method on X t. And then the algorithm of neighbor embedding is employed to recover each patch of ResiY t from ResiY s based on the weighted sum of its k nearest neighbors from patch to patch between ResiX t and ResiX s. Finally ResiY t is added to Ŷt to finally obtain the Y t. The pseudo-code of our proposed method is tabulated in Table 1.

4 4 Input: low-resolution image X t, high-resolution training image Y s, number of nearest neighbors k, patch Size s and magnification n. Procedure 1: Residuals Generation 1. Compute the lower level(1/nx of X t) image of low input X t. 2. Upsample the lower level image with interpolation methods(e.g., neighbor interpolation) to get the nx estimate ˆX t. 3. Compute the residual ResiX t between X t and ˆX t. 4. Downsample Y s by 1/nX to obtain X s. 5. Under processes similar to 1, 2 and 3 compute ResiY (s) between Y s and estimate upsampled from X s, ResiX(s) between X s and estimate ˆX s. 6. Upsample X t with interpolation to get the nx estimate Ŷt. Procedure 2: Residual Neighbor Embedding 1. Cut ResiX t and also ResiX s into patches of size s by s with overlapping by one or two pixels. 2. Cut ResiY s into patches of size n s by n s with overlapping by n or n 2 pixels accordingly 3. For each patch resix q t from ResiXt, do { Find k nearest neighbors among all patches from ResiX s Compute the reconstruction weights to minimize the error of reconstructing resix q t Compute the high-resolution embedding resiy q t using the reconstruction weights combining the patches in ResiY s corresponding to the k nearest neighbors in ResiX s. } 4. Enforce local compatibility and smoothness constraints between adjacent patches among all resiy q t and get ResiYt. 5. Y t is obtained by adding ResiY t to Ŷt. Table 1. The Pseudo-code of Our Proposed Method 5 Experiments To evaluate the performance of the proposed approach and compare it with the original SRNE method, we implement gray-level version and color version of the both approaches and experiment results are reported in the following subsections. 5.1 Gray-Level Image Experiments The original SRNE paper [5] only uses Y component of YIQ color space to carry out the algorithm. The other two components are enlarged by replication and simply contribute to the transformation from YIQ back to RGB color space thus they are of little effect in the performance. Concerning this we revise the code to a gray-level image implementation in order to evaluate the two approaches more independently, without the interference of other color components. The revision is straightforward, in which Y component are replaced by the gray-level values(which is also the only luminance) and the transformation of color space is waived for the original method. And within our proposed approach, residuals of the gray-level image are used to perform the neighbor embedding algorithm. We use the same training image for both methods. Several experiments are performed on gray-level images taken from the FERET data base. From the experiment of 4X magnification illustrated in Figure 1, we can see both SRNE and residual approaches yield better reconstruction than bilinear and bi-cubic interpolation. And our method is higher in contrast and

5 5 sharper in edges than SRNE. And another 4X example is shown in Figure 2. The training images for both methods are two other faces taken from the FERET data base, respectively. Fig. 1. 4X magnification of girl s face. Left column: Low-Resolution input. Middle column: Residual ResiY t in our method. Right column: Close-up comparison of different methods. From top to bottom: input(pixels 4X enlarged), true high-resolution target, bilinear, bi-cubic, SRNE, our method. RMS error is another notion to compare different super-resolution methods. In Figure 3 we can see that results of residual neighbor embedding have overall lower RMS errors than original SRNE and k is small when the lowest RMS error is obtained, and it s not very sensitive to different values in both methods. The improvement in RMS error of our method to the original one is as much as that of the original one to simple neighbor replication. 5.2 Color Image Experiments For more intuitive evaluation of the two methods, we also implement the color image experiments based on YIQ color space between SRNE and our residual method. In the original method YIQ color space and Y component are kept as the paper [5] does, while in our proposed method residuals of Y component are adopted to perform the neighbor embedding as Table 1 describes. Finally, images in YIQ space are transformed to RGB space with the same manner. Super-resolution results for the the 3X magnification of the bird image example

6 6 Fig. 2. 4X magnification of man s face. (a)input. (b)true high-resolution target. (c)bicubic interpolation. (d)srne. (e)our method. from the original paper are illustrated in Figure 4. We use exactly the same patch size, overlapping degree and training images as the original paper does. It is also can be seen that both our method and SRNE have better results than traditional interpolation methods. What s more, our method has lower RMS errors and sharper contrast and edges than the original one. Also in another 3X magnification example of the plant image from the original paper, our method outperforms SRNE with RMSe = , compared to RMSe = with the same parameter constraints (Figures are not included to save length). 6 Conclusion and Discussion In this paper, we present the application of machine learning methods for singleimage super-resolution. Based on the main concept of learning from the training

7 RMSe RMSe K K Fig. 3. Comparison of RMS errors given different k values between original Neighbor Embedding (lines with +) and our residual method (lines with *). RMS errors of simple neighbor pixel interpolation are shown with real lines. Left: RMSe of super-resolution for girl s face. Right: RMSe for man s face. set, various methods focus on different ways to choose features and finding candidates. We study the novel method of SRNE and develop an improved method with residuals of multi-resolution image pyramids. The proposed approach is lower in RMS errors and better at recovering the sharp contrast and fine edges. It is promising in improving the neighbor embedding based on machine learning for better performance of super-resolution. Several problems deserve to make further research. First, the implementation of residual neighbor embedding has some requirements on image size for the lower levels have to be computed and they can not be too small on pixel-scale. Second, some block effects appear in the region of constant pixel values, for in such regions there are no residuals existing between estimate and the true image. Some approaches are under research to amend this shortcoming. One way is adopting a more elaborate upsample interpolation (e.g. cubic-spline) to obtain finer residuals. While applying original neighbor embedding to the regions with constant pixel values is another promising approach to smooth and improve the quality of images. Acknowledgement This research is sponsored by NSFC under contract No And the authors are grateful to PhD Hong Chang and Professor Dit-Yan Yeung for generous providing source code and invaluable comments. And Portions of the research in this paper use the Gray Level and Color database of the FERET program. References 1. T. M. Chan and J. Zhang, An Improved Super-Resolution with Manifold Learning and Histogram Matching, Proc. IAPR International Conference on Biometric (ICB-2006), LNCS3832, Springer-Verlag Berlin Heidelberg, 2006, pp

8 8 Fig. 4. 3X magnification of the bird image. (a)input. (b)true high-resolution target. (c)bi-cubic interpolation. (d)srne. RMSe = (e)our method. RMSe = S. Baker and T. Kanade, Limits on Super-Resolution and How to Break Them, IEEE Transactions on Pattern Analysis and Machine Intelligence,vol.24, NO.9, September W. T. Freeman, E.C. Pasztor, and O.T. Carmichael, Learning Low-Level Vision, Intl J. Computer Vision,vol. 40, no. 1, Oct. 2000, pp W. T. Freeman, Thouis R. Jones, and Egon C. Pasztor, Example-Based Super-Resolution, in Proceedings of Computer Graphics and Applications, IEEE, March/April 2002, pp H. Chang, D. Y. Yeung, Y. Xiong, Super-resolution through neighbor embedding, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), vol.1, pp , Washington, DC, USA, 27 June - 2 July K. I. Kim, M. O. Franz, and B. Scholkopf, Kernel Hebbian Algorithm for Single-Frame SuperResolution, Statistical Learning in Computer Vision, ECCV 2004 Workshop, Prague, Czech Republic, May 2004, pp S. T. Roweis and K. S. Lawrance, Nonlinear Dimensionality Reduction by Locally Linear Embedding, Science, 290, pp , J. Zhang, S. Z. Li, and Jue Wang, Manifold Learning and Applications in Recognition, in Intelligent Multimedia Processing with Soft Computing. Yap Peng Tan, Kim Hui Yap, Lipo Wang (Ed.), Springer-Verlag, Heidelberg, J. Zhang, Several Problmes in Manifold Learning, Machine Learning and Applications, ZhiHua Zhou et. al. (Eds.), Tsinghua University Press, R. C. Gonzalez and R. E.Woods, Digital Image Processing(Second Edition), Prentice Hall, 2002, pp P. J. Phillips and H. Moon and S. A. Rizvi and P. J. Rauss, The FERET Evaluation Methodology for Face Recognition Algorithms, IEEE Trans. Pattern Analysis and Machine Intelligence, Volume 22, October 2000, pp