Improved Image Super-Resolution by Support Vector Regression

Transcription

1 Proceedings of International Joint Conference on Neural Networks, San Jose, California, USA, July 3 August 5, 0 Imroved Image Suer-Resolution by Suort Vector Regression Le An and Bir Bhanu Abstract Suort Vector Machine (SVM) can construct a hyerlane in a high or infinite dimensional sace which can be used for classification. Its regression version, Suort Vector Regression (SVR) has been used in various image rocessing tasks. In this aer, we develo an image suer-resolution algorithm based on SVR. Exeriments demonstrated that our roosed method with limited training samles outerforms some of the state-of-the-art aroaches and during the suerresolution rocess the model learned by SVR is robust to reconstruct edges and fine details in various testing images. W I. INTRODUCTION ith the wide sread alication of video cameras, surveillance systems and hand-held devices that are equied with moderate image sensors, it is desirable to generate images or video streams with high quality while not increasing the cost of the hardware. The imaging rocess of these sensors can be modeled by I = DHI + n () LR HR where I HR is the high-resolution image that undergoes blurring (H) and downscaling (D) rocedures with additive noise n. The outut is a low-resolution image I that we often observe. Normally the noise n is assumed to be white Gaussian noise. Fig. illustrates this rocess. Often during this rocess, the details of the image (high-frequency comonent) are lost. The urose of image suer-resolution (SR) is to reverse this rocess to recover the high-resolution (HR) image from the low-resolution (LR) observations. SR has other names such as image scaling, image interolation and enlargement. Note that SR is generally an ill-osed roblem due to insufficient rior information about the contents of the HR images (i.e. by downscaling a HR image, the ossible LR image is not unique). Insired by the ioneer work of Tsai and Huang [], there has been extensive work in image and video SR. There are different aroaches for image SR. The first tye of SR algorithms requires multile LR images from the same scene (i.e., consecutive frames taken from a video stream) as inut, L. An is with the Center for Research in Intelligent Systems, Electrical Engineering Deartment, University of California at Riverside, Riverside CA 95 USA ( lan004@ucr.edu). B. Bhanu is with the Center for Research in Intelligent Systems, Electrical Engineering Deartment, University of California at Riverside, Riverside CA 95 USA ( bhanu@cris.ucr.edu). LR then all of those images are registered and fused to generate suer-resolved images based on the assumtion that each LR image contains relevant yet slightly different information that can contribute to the HR reconstruction. Another tye of SR algorithms is single image based interolation. The well-known techniques such as bicubic interolation [] are easy to imlement and fast in rocessing. However, interolation often gives over smooth results due to its incaability to reconstruct the highfrequency comonents of the desired HR image. This could be solved by exloiting the natural image riors such as local structure gradient rofile riors [3]. The disadvantage for this kind of aroaches is that the heuristics about natural images are made, which would not always be valid and for images with fine textures the reconstructed HR image may have the water-color like artifacts. (c) Fig.. Imaging rocess from high-resolution image to low-resolution observation. The original image. Blurred image. (c) Downscaled lowresolution image. (d) Low-resolution image corruted by noise. Recently learning based aroaches have been roosed for image SR [4] [5] [6]. In [4] an examle based learning algorithm is roosed by redicting the HR images from LR images via a Markov Random Field (MRF) model that is comuted by belief roagation. In [5] suort vector regression (SVR) is alied to single image suer-resolution in Discrete Cosine Transform (DCT) domain. In [6], the SVR is alied to find the maing between the LR images and the HR images in the satial domain. In our aroach, the SR is also formulated as a regression roblem which is solved by SVR in the satial domain. However, there are (d) //$ IEEE 696

2 distinctions between our aroach and the aroach in [6]. First, in our aroach we do not aim at estimating the highfrequency comonent of the LR image to be suer-resolved. Instead, the rediction of our algorithm is the ixel value itself. Second, the feature vectors that we choose not only contain the ixel values from a neighborhood but also the local gradient information. Third, the neighboring ixel values are assigned with different weights because they do not contribute equally to generate the outut ixel in the suer-resolved image. Furthermore, in the training rocess we use images of small sizes only to form a relatively small training dataset for efficiency consideration. The exeriments show that even with a small training set our method can still generate visually leasing results. The rest of the aer is organized as follows: The suort vector regression is briefly reviewed in section II. Section III introduces the roosed algorithm for image suerresolution. Section IV shows the exerimental results and gives analysis and comarison to other SR algorithms. Finally, Section V concludes the aer. II. SUPPORT VECTOR REGRESSION Suose we have a training set {( x, )} n y where x = is the feature vector and y is the corresonding observation. Traditional linear regression which seeks a linear function f ( x) =< w, x > + b ( <, > denotes the dot roduct) that minimizes the mean square error is often not caable of searating the nonlinearly distributed inut data while on the other hand by using a transformation function φ ( x), the data is maed into a higher dimensional feature sace in which the data becomes searable. The nonlinear SVR solves the following otimization roblem: n * min w + C ( ) * ξ + ξ () w, b, ξ, ξ = yi ( < w, φ( x ) > + b) ε + ξ, * subect to ( < w, φ( x ) > + b) yi ε + ξ, * ξ, ξ 0, i =, Kn * where ξ and ξ are the slack variables, C is a constant and determines the trade-off between the flatness of the maing function and the amount u to which deviations larger than ε are tolerated. The model generated by SVR deends only on a subset of the training data since the cost function ignores the training data within the threshold of ε. The function k( x, x ) =< φ( x ), φ( x ) > is called the kernel q q function. A nice tutorial on SVR can be found in [7]. In our aroach, x comes from the initial estimation of the LR image and y is from the corresonding HR image. Then a model is learned by SVR. In the rediction rocess, the learned model will be alied to the inut LR image to generate a suer-resolved HR image. III. SUPER RESOLUTION ALGORITHM The algorithm consists of two rocesses: training and rediction, as shown in Fig.. Fig.. Training rocess. Prediction rocess. In the training rocess, we first blur the HR images and then downscale them by a factor of to create the LR images. An initial estimation of the HR image is carried out using bicubic interolation with an uscaling factor of on the generated LR image. For each ixel at location ( ) in the uscaled image, we take a local image atch of size m m centered at ( ). This image atch is then weighted by a matrix of the same size. This matrix is constructed from a -D Gaussian distribution that assigns largest weight to the ixel at ( ) and smaller weight to the other ixels that are further away from the center ixel in the local atch. The weighted image atch is then converted to a row vector: x = vec( W ( R I )) (3) G BI where I is the bicubic interolated image and BI WG ( Ri, I BI ) is the weighted local image atch taken at ( ) by the atch extraction oerator R. Function vec reshaes the matrix into a row vector x of length m. The gradient of the bicubic interolated image is calculated in both horizontal and vertical direction at each ixel. gh ( ) = (( I + I ) + ( Ii+, + Ii+, )) (4) gv ( ) = (( Ii+, I ) + ( Ii+, + I + )) I is the ixel value at ( ). The horizontal gradient magnitude g ( ) and the vertical gradient magnitude h gv ( ) are concatenated to the row vector x. For each ixel in the initially interolated image at ( ), xi, is now a m + dimensional feature vector. The corresonding observation y is the ixel value at osition ( ) in the HR image. We suly SVR with all the feature 697

3 vectors constructed from the training dataset and their corresonding observations. The generated model is then saved for the future use. In the rediction rocess, we first uscale the testing LR image also using bicubic interolation by the same factor of. For each ixel in the interolated image, the local image atch of the same size is taken and the image gradient in two directions is calculated to get the features vectors in the same manner as we do in the training rocess. Now the outut image z is constructed. The last ste is to correct the mean of z since the mean of the uscaled image should be reserved to be the same as that of the inut LR image due to the unchanged image structures and contents in the uscaling rocess. The ixel value of the final outut at ( ) is: mlr z$ = z (5) m where z mlr z is the mean ixel value of the LR image and m is the mean ixel value of z. A. Exerimental setu IV. EXPERIMENTS For the imlementation of SVR we use LibSVM [8]. We choose Gaussian function as the kernel function. The arameters in the SVR are selected by cross-validation ( C = 36, ε = and the standard deviationσ = in the Gaussian kernel function). The downscaling factor for the HR images and the corresonding uscaling factor for the LR images are both. For both training and testing we only consider the luminance comonent of the images. get the LR images for training. The LR images we used in training are of size while for the testing urose the inut LR images are of size 8 8. The HR images of size as the ground truth are used to later evaluate the erformance quantitatively. The size of the image atch we use to get feature vectors is 5 5.We do not add noise deliberately to the LR images because in reality for the case of image SR, the inut LR image is not necessarily corruted by noise and even if it is the case, a re-rocessing with a robust image denoising algorithm can effectively remove the noise [0]. Both eak signal-to-noise ratio (PSNR) and structural similarity (SSIM) [] are used to measure the quality of the suer-resolved images comared to original HR images. PSNR between two images of size M N is calculated by PSNR = 0log M N i= = 0 M N i= = 55 ( x( ) y( )) where 55 is the maximum ossible gray ixel intensity value, x( ) and y( ) are the ixel values at the same location (, ) i from image x and y. SSIM is designed to better match the human ercetion comared to PSNR, which sometimes is inconsistent with the visual observation []. SSIM is defined as: ( µ xµ y + c)( σ xy + c) SSIM = (7) ( µ + µ + c )( σ + σ + c ) x y x y where x and y are the two images to be comared, µ and σ are resectively the average and variance of the ixel values of the images x or y. σ xy is the covariance of x and y. The value of SSIM is a scalar less than or equal to and means the two images in comarison are exactly the same. (6) Fig. 3. Images for training. From left to right and to to bottom: Lena, Peers, Man, Lake, Truck and Car. We use 6 images for training and 9 images for testing. All the images used in the training and testing rocesses are originally taken from the USC-SIPI Image Database [9]. The HR images we used in the training set are all of size 8 8. The LR images are blurred by a 3 3 uniform oint sread function (PSF) and then downscaled in order to Fig. 4. Images for testing. From left to right and to to bottom: Cameraman, Clock, Tree, Elaine, Synthetic, House, Boat, Girl and Mandrill. 698

4 B. Results First we validate our choice of feature selection (ixel intensity values and gradient magnitude) by comaring the result of the roosed method to the outut of SVR with feature vectors that only contain the ixel values as similarly roosed in [6]. Fig. 5 gives an illustrative comarison from a test image. TABLE I PSNR COMPARISON Image BI[] SC[] KR[3] Proosed Cameraman Clock Tree Elaine Synthetic House Boat Girl Mandrill TABLE II SSIM COMPARISON Image BI[] SC[] KR[3] Proosed Cameraman Clock Tree Elaine Synthetic House Boat Girl Mandrill (c) Fig. 5. Result by roosed method. Result by SVR without gradient information. (c) Magnified art from. (d) Magnified art from. It can be seen that by adding image gradient information to the feature vectors, the model learnt by SVR is caable of reconstructing edges and fine details from LR images. We comare the roosed suer-resolution algorithm with state-of-the-art methods including sarse coding [] and kernel regression (KR) [3] with the imlementations rovided by the authors of [] and [3]. Also results by bicubic interolation (BI) [] are rovided as reference. We do not comare our results to [6] directly since without knowing the arameter settings in their SVR training we are not able to roduce results with their algorithm exactly. However as indicated by Fig. 5 the roosed algorithm is better at handling details and edges. Table I and II show the PSNR and SSIM results for the test images. As shown, our method outerforms the other methods with resect to both PSNR and SSIM. Note that the reference methods are secifically designed for single image suer-resolution. We also show some results in Fig. 6. It is suggested to view the results on your comuter. By visual insection, our method roduces shar images. While sarse coding [] is good at reserving highfrequency details, the generated images suffer from noticeable artifacts esecially along the edges and image boundaries. Bicubic interolation [] and kernel regression (d) [3] roduce oututs without many high-frequency comonents. Kernel regression generates ghost artifacts, for examle on the uer triod in Cameraman. The model learned by SVR in our method is able to generate fine details and shar edges which lead to better visual quality. Both obective evaluation (by PSNR and SSIM) and subective evaluation confirm the advantages of our method. V. CONCLUSIONS In this aer, an algorithm for single image suerresolution based on suort vector regression is roosed. By combining the ixel intensity values with local gradient information, the learned model by SVR from low-resolution image to high-resolution image is useful and robust for image suer-resolution. We conducted the exeriments on different tyes of images and the results are romising. Furthermore, the size of the training set is limited which makes the training relatively fast while still achieving good results. By comaring our method to the revious works, we find out that our method is able to roduce better suerresolved images than state-of-the-art aroaches. We believe that by selecting more informative features besides ixel intensity and gradient, the result can be further imroved. By adoting a larger and more comrehensive image dataset for training, the generated model would yield better results for image suer resolution. REFERENCES [] R. Tsa T. Huang, Multi-frame image restoration and registration, Advances in Comuter Vision and Image Processing, vol., no., JAI Press Inc., Greenwich, CT, 984, [] R. Keys, Cubic convolution interolation for digital image rocessing, IEEE Transactions on Acoustics, Seech and Signal Processing, vol. 9, no. 6, ,

5 Fig. 6. From to to bottom: Original HR images, results by bicubic interolation [], results by SC [], results by KR [3], results by the roosed method. [3] [4] [5] [6] [7] J. Sun, Z. Xu, and H.-Y. Shum, Image suer-resolution using gradient rofile rior, In Proc. of the IEEE International Conference on Comuter Vision and Pattern Recognition,. -8, 008. W. T. Freeman, E. C. Pasztor, and O. T. Carmichael, Learning lowlevel vision, International Journal of Comuter Vision, 40 (),. 5-47, 000. K. N S. Kumar, N. Vasconcelos, and T. Q. Nguyen, "Single image suerresolution based on suort vector regression", In Proc. of the IEEE International Conference on Acoustics, Seech and Signal Processing, , 006. D. L S. Simske, and R. Mersereau, Single image suer-resolution based on suort vector regression, In Proc. of International Joint Conferences on Neural Networks, , 007. A. J. Smola, and B. Schölkof, A tutorial on suort vector regression, Statistics and Comuting 4,. 99, 004. [8] [9] [0] [] [] [3] C. -C. Chang, and C. -J. Lin, LIBSVM: a library for suort vector machines, 00. Software available at: htt:// Available at: htt://sii.usc.edu/database/ A. Baudes, B. Coll, and J. M. Morel, A non-local algorithm for image denoising, In Proc. of the IEEE International Conference on Comuter Vision and Pattern Recognition, , vol., 005. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncell Image quality assessment: From error visibility to structural similarity, IEEE Transactions on Image Processing, vol. 3, no. 4, , 004. J. Yang, J. Wright, T. Huang, and Y. Ma. Image Suer-resolution as Sarse Reresentation of Raw Image Patches, In Proc. of the IEEE International Conference on Comuter Vision and Pattern Recognition,. -8, 008. H. Takeda, S. Farsiu, and P. Milanfar, Kernel regression for image rocessing and reconstruction, IEEE Transactions on Image Processing, vol. 6, no. 4, ,