Enhancing DubaiSat-1 Satellite Imagery Using a Single Image Super-Resolution

Transcription

1 Enhancing DubaiSat-1 Satellite Imagery Using a Single Image Super-Resolution Saeed AL-Mansoori 1 and Alavi Kunhu 2 1 Associate Image Processing Engineer, SIPAD Image Enhancement Section Emirates Institution for Advanced Science & Technology (EIAST), Dubai, UAE s_mansoori@eiast.ae 2 Electrical Engineer, ECE Department Khalifa University of Science, Technology and Research (KUSTAR), Sharjah, UAE ABSTRACT DubaiSat-1 (DS1) captures multispectral images, which consist of three visible channels and one NIR with 5-meter resolution and a panchromatic channel with 2.5-meter resolution. Considering these resolutions, some important details might appear blurry on the image. Therefore, the concept of Super Resolution has been introduced to reconstruct the image in a way to overcome the inherent resolution limitations of imaging sensors. The aim of this study is to enhance the quality of the image by artificially increasing the number of pixels within the image to sharpen out-of-focus details or smooth rough edges in DubaiSat-1 images that have been enlarged using a general up-scaling process. Usually, images received from satellites go through many image processing and enhancement steps to increase the quality of these images. Many studies have been conducted in this field, and different techniques were suggested to improve the enhancement procedure. Studies were done to combine low resolution images of the same area to come up with an image with a high resolution and better quality. Image enhancement refers to operations used to get image features such edges, boundaries, or contrast in order to make a digital image more useful for display and analysis. In this paper, the proposed method is based on a well-known approach called Example Based Super-Resolution. A single low resolution DubaiSat-1 image is used to construct its high resolution version. Simulation results illustrate a good performance of super-resolved images over certain magnification factors (i.e. =2,3 and 4). Keywords: DubaiSat-1, Blurry, Super-Resolution, Up-Scaling, Image Enhancement, Bilinear Interpolation, Bicubic Interpolation, Classical Multi-Image Super-Resolution, Example Based Super-Resolution, Kernel Regression. B 1. INTRODUCTION asically, there are three possible ways to increase the resolution of the image; (a) reduce the pixel size, (b) increase the chip size and (c) Super Resolution (SR). Reducing the pixel size leads to increase the number of pixels within the same unit area. This will increase the spatial resolution of the image; nonetheless it will produce a level of Gaussian noise. Obtaining a high resolution images through hardware by increasing, the density of image sensors (i.e. CCD and CMOS sensors) and chip size required high precision optics which are quite costly. To overcome hardware limitations, software engineers look for alternative way to enhance the image resolution through increasing the number of pixels. The well-known approach used to enhance the image resolution is called Super Resolution. The aim of super resolution is to construct a high resolution (HR) enlargement image from one or a set of low resolution (LR) input images. Such approach is designed in a way to increase the number of pixels within the image which leads to enhance image features such as edges, boundaries, or contrast. Therefore, this will provide a detailed version of a given image which can be used in many real-time applications; in scientific (e.g., image compression and communications, medical, feature extraction, satellite imaging) and commercial (e.g., entertainment, high definition television). Interpolation-based super resolution is considered as an early method used in this field. There are three well-known interpolation techniques, namely, nearest neighbor, bilinear, and bicubic [1]. The bicubic interpolation provides smoother edges than other two interpolation techniques. Nonetheless, an interpolation technique alone will construct a blurry image with lack of well details. Thus, other sophisticated super-resolution methods are proposed which can be categorized into two classes: (a) the classical multi-image super resolution and (b) Example based super resolution. Previously, different solutions were used to protect the copyright, such as the legal registration of the digital products with an authority concerned with copyright protection. However, it turned out more complicated compared to applying techniques such as watermarking. Satellite Data Compression, Communications, and Processing IX, edited by Bormin Huang, Antonio J. Plaza, Chein-I Chang, Proc. of SPIE Vol. 8871, 88710N 2013 SPIE CCC code: X/13/$18 doi: / Proc. of SPIE Vol N-1

2 (a) (b) (c) Super Resolution Figure 1. Increasing the number of pixels within the same unit area is the main idea of super resolution (a) The classical multi-image super resolution The classical multi-image super resolution constructing a high resolution version from a set of low resolution images (sub-pixel aligned) of the same scene denoted,,..,. Each low resolution (LR) image imposes a set of linear constraints on the anonymous high resolution (HR) intensity values [2]. If enough low resolution images are taken, a set of linear equations will be solved to construct the high resolution image. Basically, this approach assumes that a set of aligned low resolution images are taken at sub pixel precision and each low resolution image has a known blur kernel denoted. The procedure starts with finding the nearest patch neighbors for each pixel within low resolution image. The next step is to compute their sub-pixel alignment at (1 / scale factor) pixel shifts. By finding the sufficient neighbors, the set of equations becomes determined to fill the unknown pixel intensity values in high resolution image [2]. This approach is limited to minor increase in resolution, therefore, researchers look for different manner to reach to a significant increase in image resolution through a new approach called Example Based Super Resolution. (b) Example based super resolution The example based super resolution recover a high resolution version from a single low resolution image. It exists by recovering the missing high frequencies by searching for highly similar patches in the external database [2].This method is based on learning the correlation between low resolution and the corresponding high resolution patches from a database of known low and high resolution image pairs. The input low resolution image is divided into either overlapping or non-overlapping patches. For each low resolution input patch either one best matched patch or a set of best matched patches is picked from the database. Then the corresponding high resolution patch(s) is used to construct the most likely high resolution image. The database images should be carefully selected to get the expected outcome. Repeating this process several times leads to increase super-resolution factor. Recently, Dalong Li and Steven Simske [3], reported their work based on an example based single-frame image super resolution using support vector regression (SVR). Experimental results indicate that SVR has the ability to identify a set of generally applicable SVs from a very small image database for SR. Moreover, it is noticed that the PSNR improvement is not dramatic, visual assessment recommends that some missed high frequency components are recovered. In addition, Ni et al. [4] proposed a technique using SVR for image SR. Learning based methods differ based on two main criteria s; (a) the input and output images formulation (b) the technique behind learning process. For an Proc. of SPIE Vol N-2

3 instance, Freeman et al. [5] formulated a LR patch and its neighboring HR patch as input and the corresponding HR patch as output. In terms of learning process, the authors developed two approaches to discover the correlation (i.e. neighborhood relationships) in single image SR techniques. Both approaches are based on using a Markov network. The first approach exploits the relationships between low and high resolution patches and between neighboring high resolution patches. The second approach uses the same local relationship information (patch-to-patch mapping) as the Markov network. A survey of example based SR techniques is accessible in [6]. In [7], a class of adaptive Markov Random Fields (MRFs) is proposed to increase the performance of standard examplebased SR. The likely high frequency patches to a certain regions are constrained by adjusting the transition functions. This reduces the Mean Square Error (MSE) related to a typical MRF, as well as, it produces sharper images. In this paper, authors select samples of face images to show how they can adapt the modeling for each image patch consequently to enhance the resolution. 2. PROPOSED METHODOLOGY There are several techniques used to enhance the image resolution as discussed early. In this paper, we proposed a technique based on example based super resolution. A single low resolution DubaiSat-1 image is used to recover a high resolution version. This method is based on learning the correlation between low resolution and the corresponding high resolution patches from a database of known low and high resolution image pairs. In this study, a special database is used and certain steps are followed as demonstrated in figure 2. Up-sample the low resolution image into desired scale using interpolation technique Step1 Low Resolution Image Generate candidate image sets based on Kernel Regression Step2 Combine the candidate sets to construct the image and post-process it to improve the quality Step3 High Resolution Image Figure 2. A Block Diagram of the Proposed Method Step1: Up-sample the low resolution image into desired scale using interpolation technique Two well-known interpolations based super resolution (i.e. Bi-linear interpolation and Bi-Cubic interpolation) are implemented to study their performance on DubaiSat-1 low resolution images. Therefore, a ( ) DubaiSat-1 images are reduced by a factor of 0.5 (i.e. down-sampled by 2) to construct ( ) images. The down-sampled images are then enlarged by a factor of 2 (i.e. up-sampled by 2) to construct ( ) images. It is noticed that images became blurred and most details are disappeared. Original Image ( ) Down-sampled Image ( ) Up-sampled Image ( ) Figure 3. Simple experiment to analyze the performance of interpolation techniques Proc. of SPIE Vol N-3

4 Therefore, two interpolation techniques are applied on 30 DubaiSat-1 up-sampled images in order to reconstruct image details. The resulted bilinear/bicubic up-sampled image is compared to the original image to observe the effect of applying both interpolation techniques. The quality assessment performance is numerically calculated using the Peak Signal-to-Noise Ratio (PSNR) parameter measured in [db]. Table 1 below shows the performance of both Bi-linear and Bi-cubic interpolations. Table 1. PSNR comparison between Bi-linear and Bi-cubic Interpolation Techniques Peak Signal m- Noise Ratio [PSNR in db] Peak Signal to- Noise Ratio (PSNR in db) Sample DS1 Images Image 1 Red Green Blue Red Green Blue Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Bi-linear Interpolation Bi-cubic Interpolation The PSNR of RGB bands differ from one image to another depends if an image contains seas and oceans, urban and green areas. In terms of interpolation techniques, Bi-cubic interpolation gives better results compared to Bi-linear interpolation. This proves the theory discussed in [1,2]. Thus, the bicubic interpolation technique is proposed to upsample the low resolution image into desired scale. Figure 4 below shows a comparison between the enlarging of low resolution image using bilinear and bicubic interpolations. (a) (b) (c) (d) Figure 4. (a) Original Low Resolution Image (50 50), (b) 200% Enlarged Image (Nearest Neighbor Interpolation), (c) 200% Enlarged Image (Bilinear Interpolation), and (d) 200% Enlarged Image (Bicubic Interpolation) Step2: Generate candidate image sets based on Kernel Ridge Regression After enlarging the low resolution DubaiSat-1 image into desired scale using bicubic interpolation function, the image is converted into YC b C r format and the luminance channel (Y) is selected to overcome the time consuming occurs when the same procedure is applied for three image layers (RGB). Then, the high frequency components details are estimated from the Laplacian of the bicubic interpolation enlarged image. The main idea here is to produce the superresolved image by adding the high frequency components and the bicubic interpolation enlarged image (i.e. = Proc. of SPIE Vol N-4

5 + ). To reduce the complexity of regression algorithm to a moderate level, we have used a square-window patch based approach to estimate the high frequency components. The estimation of the values at specific locations (, ) is performed based on the values of at corresponding ( (, )), where represents a square patchwindow cantered at the location (, ) of the image. Then, during the super-resolution, is scanned with a small window of size to generate a patch-valued regression result of size for each pixel. Step3: Combine the candidate sets to construct the image and post-process it to improve the quality Even though it is possible to construct super-resolution images based the scalar-value regression, we have proposed to predict a patch-valued output such a way that different candidates are generated for each pixel. One straightforward way is to construct the final super-resolution images of as a convex combination of candidates based on a certain confidence measure and this method gives better result over scalar-value regression method. A better prediction was obtained when the confidence estimation is obtained based the in-put patches and the context of neighbour reconstructions. We have noticed that the proposed kernel ridge regression-based method is significantly better than the bicubic interpolation. However, detailed visual inspection along the major edges reveals ringing artifacts and postprocessing based on the discontinuity prior of images will removes the ringing artifacts and further enhances edges. 3. RESULTS AND DISCUSSION The proposed super resolution technique is tested on various low resolution color DubaiSat-1 images; Sat1, Sat2, Sat3, Sat4, Sat5 and Sat6. The size of each image is ( ) pixels. To study and analyse the system parameters, the proposed technique is tested under various magnification scaling factors and Patch window sizes. The Magnification scaling factors used are =, = and =, whereas patch window sizes used are( ), ( ) and ( ). Table 2 below illustrates the super-resolved output of six sample images with = and patch window size of ( ). In addition, the proposed technique is tested under = and = as shown in figures 5 and 6 respectively. Table 2. Super-resolved output of six sample images with = and patch window size of ( ) Original Image Enlarged Low Resolution Image Super-Resolved Image ( ) Sat1 Sat2 Proc. of SPIE Vol N-5

6 Sat3 e, Sat4 4 Sat5 Sat6 Proc. of SPIE Vol N-6

7 LR Sat1 ( ) LR Sat2 ( ) 1 Enlarged Low Resolution Image (Sat1) when = 1Ir Enlarged Low Resolution Image (Sat2) when = ; Super-Resolved (Sat1) when = Super-Resolved (Sat2) when = Figure 5. Super-resolved output of Sat1 and Sat2 images with = and patch window size of ( ) Proc. of SPIE Vol N-7

8 LR Sat1 ( ) Enlarged Low Resolution Image (Sat1) when = Figure 6. Super-resolved output of Sat1 with = and patch window size of ( ) Super-Resolved (Sat1) when = Proc. of SPIE Vol N-8

9 By Human Visual System (HVS), it is noticed that as patch window size increases the performance of super-resolved output is increases as well, nonetheless it will takes more time for processing. PSNR analysis of super resolution images under various magnification scaling factors and patch window sizes are show in Table 3. Table 3. PSNR Analysis of Super Resolution Images based Y layer regression method Patch window size = Patch window size = Patch window size = Images ( ) ( ) ( ) = = = = = = = = = Sat Sat Sat Sat Sat Sat Patch Window 5x5 Patch Window 9x9 Patch Window 13x Sat1 Sat2 Sat3 Sat4 Sat5 Sat6 PSNR [db] PSNR [db] Magnification Scaling factor Figure 7. PSNR analysis of super resolved image Sat Magnification Scaling factor Figure 8. PSNR analysis of super resolved images in case of ( ) Patch window 4. CONCLUSION In this paper, the proposed method is based on a well-known technique called Example Based Super-Resolution which is designed to construct a high resolution image from a single low resolution DubaiSat-1 image. Bicubic interpolation technique is applied on the enlarged low resolution image to enhance its quality and reduce the noise. Moreover, a special database is used to generate candidate image sets based on Kernel Regression. The high resolution image is constructed by combining the candidate sets of the image. By post-processing, the quality of the image will be improved. Simulation results illustrate a good performance of super-resolved images over certain magnification factors (i.e. =2,3 and 4). 5. REFERENCES [1] O.Harikrishna and A.Maheshwari, Satellite Image Resolution Enhancement using DWT Technique, International Journal of Soft Computing and Engineering (IJSCE), Vol.2, Issue 5, November [2] Daniel Glasner, Shai Bagon and Michal Irani, Super-Resolution from a Single Image, International Conference on Computer Vision (ICCV), pp , [3] Dalong Li and Steven Simske, Example Based Single-frame Image Super-resolution by Support Vector Regression, Journal of Pattern Recognition Research 1, pp , [4] K. S. Ni and T.Q. Nguyen, Image superresolution using support vector regression, IEEE Trans. Image Process, Vol.16, pp , June Proc. of SPIE Vol N-9

10 [5] W. T. Freeman, T. R. Jones, and E. C. Pasztor, Example-based super-resolution, IEEE Comput. Graphics and Applicat., Vol. 22, No. 2, pp , Mar./Apr [6] M. Elad, D. Datsenko, Example-based regularization deployed to super-resolution reconstruction of single image, The Computer Journal Advance Access published online on April, 20, [7] Todd A. Stephenson and Tsuhan Chen, Adaptive Markov Random Fields for Example-Based Super-resolution of Faces, EURASIP Journal on Applied Signal Processing, Vol. 2006, Article ID 31062, pp Proc. of SPIE Vol N-10