A COMPARATIVE STUDY OF DIFFERENT SUPER- RESOLUTION ALGORITHMS

Transcription

1 A COMPARATIVE STUDY OF DIFFERENT SUPER- RESOLUTION ALGORITHMS Navya Thum Problem Report submitted to the Benjamin M. Statler College of Engineering and Mineral Resources at West Virginia University in partial fulfillment of the requirements for the degree of Master of Science in Electrical Engineering Approved by Dr. Xin Li, Ph.D.,Chair Dr. Matthew Valenti, Ph.D Dr. Vinod Kulathumani, Ph.D Lane Department of Computer Science and Electrical Engineering Morgantown, West Virginia 2013 Keywords: Super resolution, Image processing, ASDS, SME, NE, GUI Copyright 2013 Navya Thum

2 ABSTRACT A COMPARATIVE STUDY OF DIFFERENT SUPER RESOLUTION ALGORITHMS by Navya Thum Image resolution has become very important in many applications today. Generally the images used on the web are low resolution images. When you try to print a low resolution image, either the image is printed at the size of postage stamp or is printed on large scale but looks jagged or blurry. When there are enough pixels per inch, then pixels blend together and look continuous to the human eye. However, as the pixel count decreases, the image will appear jagged or blurry. We should increase the image resolution while printing in order to preserve the details of the image. The process of enhancing the resolution of an image is called as super-resolution. It is also useful in many applications, such as video surveillance and automatic target recognition. This paper presents the results of three super-resolution methods applied to a single frame low resolution image. Quantitative measures including peak signal to noise ratio and structural similarity are used to evaluate the performance of these three methods. The first method is a learning based method where the small patches in low resolution and high resolution images are taken as two distinct feature spaces with similar local geometry. The patch can be reconstructed by its neighbors in the feature space. In the second method, inverse problem estimators are computed by adaptively mixing a family of linear estimators corresponding to different priors. The third method introduces auto regression (AR) and image non-local self-similarity concepts into the sparse representation framework. Finally, the results are displayed in a GUI window for better comparison.

3 Acknowledgements I would like to thank my parents Mr. Vidya Sagar Reddy and Mrs. Sunitha for helping in every walk of my life and being there for me all the time. I would like to express my sincere gratitude to my advisor Dr. Xin Li for his constant support and wonderful encouragement in helping me all through out to complete this work. I wish to express my sincere gratitude to my committee members Dr. Matthew Valenti and Dr. Vinod Kulathumani. I also thank all the faculty members of WVU LCSEE. I would like to convey my thanks to all my friends here, for their support and love. iii

4 Table of Contents Acknowledgement... iii Table of Contents... iv List of Tables..vi List of Figures vii 1 Introduction Overview Super Resolution Low Resolution Image Formation Model Research Objectives Evaluation Methods Peak Signal to Noise Ratio (PSNR) Similarity Index Measure (SSIM) Outline Background Interpolation Types of Interpolation Limitations of Interpolation Classification of Super Resolution Techniques Super Resolution Algorithms Neighbor Embedding Sparse Mixing Estimators Adaptive Sparse Domain Selection and Adaptive Regularization Kernel Super Resolution iv

5 3.5 Cubic Spline Interpolation Experimental Results Results Discussion Graphic User Interface (GUI) Conclusion References v

6 List of Tables Table 4-1 PSNR Values Obtained Using Super Resolution Algorithms Table 4-2 SSIM Values Obtained Using Super Resolution Algorithms Table 4-3 Time In Seconds To Execute The Algorithms On The Image vi

7 List of Figures Figure 1-1 Low Resolution And High Resolution Image... 1 Figure 2-1 Multi-Image Super Resolution and Multi-Image Super Resolution [29]... 9 Figure 4-1 Test Images Taken from Standard Kodak Database Figure 4-2 Super Resolution Algorithm Results of Girl Image Figure 4-3 Super Resolution Algorithm Results of Cropped Girl Image Figure 4-4 Super Resolution Algorithm Results of Parrot Image Figure 4-5 Super Resolution Algorithm Results of Cropped Parrot Image Figure 4-6 Super Resolution Algorithm Results of Bike Image Figure 4-7 Super Resolution Algorithm Results of Cropped Bike Image Figure 5-1 MATLAB Handling Objects [26] Figure 5-2 The Final Output Result Window Figure 5-3 GUI Output result window for bike using ASDS algorithm Figure 5-4 GUI Output result window of cropped above bike image vii

8 Chapter 1 1 Introduction 1.1 Overview Re-sampling of images to change size or resolution has gained greater importance with the popularity of the word resolution. Changing the size, orientation, or resolution is very common in all sorts of devices, like television sets, computers, digital cameras and mobile phones. Due to the improved image quality and easy printing, the use of digital cameras has increased greatly than film cameras. In digital cameras, resolution is the most important feature. The number of pixels in the given area of an image is known as the resolution. When there are enough pixels per inch, the pattern is small and the pixels look continuous to the human eye. On the other hand, as the pixel count per inch drops, the individual pixel will begin to show and the image will appear blurred or jagged. Therefore, a high resolution image has more pixels and helps us to see more details. Unfortunately, increasing the resolution at the sensor level will increase the cost of the camera. The purpose of super resolution is to increase the resolution of the image. Figure 1-1 Low Resolution and High Resolution Image 1.2 Super Resolution High resolution images are always necessary in almost all image processing applications [6]. Super-resolution algorithms obtain an image at higher resolution from its low-resolution observations. Super resolution problem often refers to multiframe super-resolution, where a high 1

9 resolution image is obtained by combining multiple low resolution frames [2]. Multiframe super resolution algorithms are closely related to the problems of image restoration and image interpolation [1]. Image restoration is used to restore the degraded image without changing the dimension of the actual image, whereas, interpolation is used to increase the size of the image. Super resolution is considered a second generation image restoration technique which recovers the degraded image by changing the image dimension [1]. Super resolution can also be applied on a single frame low resolution image which is different from multiframe super resolution. In single frame super resolution, only one low resolution image is taken into consideration. Single frame super resolution is also known as image scaling, interpolation, enlargement and zooming [12]. A common application of the single frame super resolution problem arises when we want to increase the resolution of an image while enlarging it using digital imaging software [9]. And also For example, in web pages to shorten the response time of browsing, images are often shown in thumbnail images or low resolution frames. However a high resolution image should be stored on the web server and downloaded to the user s client machine whenever the user clicks on the corresponding thumbnail [10]. It is desirable to download and then enlarge the low resolution image on the user s machine to save storage space and download time. 1.3 Low Resolution Image Formation Model The general model for low resolution image produced from a single high resolution image is:, where z represents the ordered high resolution image of pixels, D is the decimation matrix, A is the blur matrix, and n is the noise vector [6]. This model holds good for the current study as there is only a single observation f which itself serves as the reference low resolution image. The low resolution image will be noisy, aliased, as well as blurred. This problem is estimating the high resolution image z from the given single observation of f. Ideally, the super resolution algorithm should recover an alias-free upsampled version of low resolution image f by reducing the effects of blur and noise [6]. It will be difficult to deal with all the three problems simultaneously, hence, different forms will be assumed A and n. 2

10 Figure 1-2 Low Resolution Image Formation Model [6] Figure 1-2 is the block diagram for Low Resolution Image Formation Model. The low resolution image is formed when a high resolution image undergoes motion blur, noise, aliasing, or distortion. So, in order to obtain a high resolution image we need to remove all of these disturbances Figure 1-3: Common Imaging System [29] The above Figure 1-3, shows a common imaging system. Whenever a camera or an imaging system is used to capture a picture it undergoes distortion, aliasing, blur and noise, hence, we do not get a complete high resolution image as the captured picture will be blurred, noisy, and aliased.. 3

11 1.4 Research Objectives The objective of this work is to study different existing single frame super resolution algorithms. Three different algorithms along with their mathematical models are studied in detail. PSNR and SSIM values are calculated for a set of standard images from Kodak database. Based on these quantitative measures and computational speed, the performance of super resolution methods is compared with each other. The main aim lies in studying the advantages and disadvantages of these methods and to judge which is the best applicable one for any application. Results are displayed in a separate GUI window. For clear comparison, the user is allowed to crop any portion of the original image. 1.5 Evaluation Methods Applying any image processing technique will reduce the image quality. Image quality can be evaluated using quantitative measures as well as visual assessment. In this work, both quantitative measures and visual assessment are used to evaluate the performance of each method. Performance metrics used in this work are: Peak Signal to Noise Ratio (PSNR) PSNR is the ratio of the signal power to the noise. It is based on the mean squared error (MSE), which is the mean of squared difference for every pixel between the original image and the reconstructed one [3]. PSNR = 10log10 ((Max. ^2) /MSE), where MSE= and Max is the maximum pixel value of the image. When an image is represented by 8-bit the maximum value is 255 and in general for a P-bit representation it is given by 2P-1. Here, I is the original image and is the reconstructed image over pixels [m,n] where M, N is the size of the 2-D image. For a 3-D image the MSE is the sum of all the squared differences of the error divided by size of the image, multiplied by 3. Generally, PSNR lies in between 25dB to 50dB, the higher the better. 4

12 1.5.2 Similarity Index Measure (SSIM) SSIM calculates the similarity in a linked local window by combining differences in average, and variation, and correlation [8]. SSIM gives more accurate results than the traditional methods like peak signal-to-noise ratio (PSNR) and mean squared error (MSE) [4]. SSIM is used because of its higher correlation compared to PSNR and MSE. SSIM for two windows x and y of common size NxN is defined as:, where and are the average of x and y; and are the variance of x and y; is the covariance of x and y. 1.6 Outline Chapter 1 defines super resolution. It explains the need for enhancing the resolution, and also explains the evaluation methods briefly and gives a basic idea about the work. Chapter 2 explains the different types of interpolation and super resolution techniques available to get a high resolution image from low resolution image. Chapter 3 gives the implementation procedure and the algorithms of the three duper resolution techniques in detail. Chapter 4 gives the results of the implementation of these three methods for comparative study, with PSNR and SSIM. Chapter 5 gives a basic overview of Graphic User Interface and the results of this work using GUI. Chapter 6 concludes with understanding of this work and with the future work. 5

13 Chapter 2 2 Background In image processing, interpolation techniques and resolution enhancement methods based on smoothing are commonly practiced to reduce noise in the images. Commonly used interpolation techniques are bi-cubic interpolation and cubic spline interpolation [11]. Smoothing is usually achieved by applying various filters such as Gaussian, Wiener, etc. Linear interpolation techniques usually perform better than simple smoothing but both of them introduce artifacts such as Gibbs oscillations or smoothing of the edges. Both smoothing and linear interpolation cause blurring problems when the image is aliased. Super resolution algorithms are non-linear and can recover high frequency information by taking advantage of prior signal information. Super resolution algorithms take advantage of this aliasing together with some geometric image properties [22]. In general super resolution algorithm includes three tasks: generating missing high frequency components, minimizing aliasing, and removing degradations such as blur and noise which arise during image capture [6]. 2.1 Interpolation Interpolation is the process of constructing new data points within the range of a discrete set of known data points [wiki]. It can also be defined as recovery of continuous data from discrete data within a known range of data points [27]. Image interpolation happens anytime when we want to resize or remap the image from one pixel grid to another. The simplest interpolation is done by locating nearest data value and assigning the same value to intermediate points Types of Interpolation Linear interpolation Linear interpolation is the simplest interpolation which takes two data points and interpolate at the point (x, y) by the given formula: y = y 1 + (y 2 y 1 ) (x x 1), where (x1, y1) and (x2, y2) are two data points. (x 2 x 1 ) 6

14 Linear interpolation is fast and easy, but is not very accurate. The error produced by the linear interpolation is proportional to the square of the distance between the data points. Linear interpolation is best fit for data points in one spatial dimension. For two spatial dimensions, we extend linear interpolation to bilinear interpolation. In bilinear interpolation, linear interpolation is performed in one direction and again in other direction. Though each step is linear, bilinear interpolation as a whole is quadratic in the sample location Spline interpolation Unlike linear interpolation which uses linear function for each interval, spline interpolation uses low degree polynomials in each interval and chooses polynomial pieces for smooth fitting. Spline interpolation suffers a smaller error than linear interpolation. We often use interpolation to interpolate between lists of values. Let P0, P1, P2 and P3 be the values at x=x1, x=x2, x=x3, x=x4 respectively, then cubic interpolation formula is given by: f(p0,p1,p2, P3, x) =(-0.5P0+1.5P1-1.5P2+0.5P3)x 3 +(P0-2.5P1+2P2-0.5P3)x 2 +(-0.5P0+0.5P2)x+P1 Similar to bilinear interpolation, bi-cubic interpolation is an extension of cubic interpolation on a two dimensional regular grid. Cubic interpolation is used to construct bi-cubic interpolation by first interpolating the area vertically and then interpolating the results in the horizontal direction Limitations of Interpolation Though interpolation is easy, fast and useful for some commercial software, it is inappropriate for a natural image with edge and texture as the interpolated kernel is defined based on the assumption of piecewise smooth image. There are many linear interpolation kernel methods which give improved result dealing with edge but all of these methods use a linear equation to estimate interpolation weight [28]. 2.2 Classification of Super Resolution Techniques Super resolution techniques can be classified into two classes. First is a reconstruction based technique which recovers high resolution images by adopting iterated back-projection and deconvolution. This method aims to minimize the reconstruction error. In this technique the unknown high resolution image is obtained by using cues like motion, blur, zoom, etc. Most of the super resolution techniques available today use motion cue to solve for the high resolution 7

15 image. In these techniques, sub-pixel shifts are calculated among different low resolution frames and then interpolated onto high resolution grid, followed by restoration to remove blur and noise. These techniques are good at handling the blur and noise in the low resolution image as the processing is done on the whole of low resolution images simultaneously but aliasing is not handled completely in these techniques [6]. Second is the learning, or example based, Super resolution technique, which uses several other images to learn a prior on the original high resolution image (learning correspondence between low and high resolution image patches from the database) [6]. Some learning based techniques generally use a local approach to learn prior data of the original high resolution image while the others use a set of training images to obtain a high resolution target image. These are good at handling edges but blur and noise present in the low resolution observation are not taken care of properly. These techniques perform poorly in the presence of aliasing though the noise is reduced by considering multiple observations [6]. These methods are suitable for texture images. Super resolution techniques can also be classified into multiframe super resolution techniques and single frame super resolution technique. In multiframe super resolution technique, the high resolution image is recovered from one or more low resolution input images. In this method, a set of low resolution images of the same scene are taken where each low resolution image imposes a set of linear constraints on the unknown high resolution intensity values [29]. The high resolution image can be recovered completely if there are enough low resolution images to make the set of equations to be determined. However, this method is limited only to small increases in resolution [29]. Single frame super resolution is also called example-based super resolution. In this method, correlations between low and high resolution image patches are learned from a database of low and high resolution image pairs and then it is applied to new input low resolution image to recover its high resolution version [29]. In multiframe super resolution method, the high frequency information is assumed to be split across multiple low-resolution images, completely found there in aliased form. However in example-based super resolution, this missing highresolution information is assumed to be available in the high-resolution database patches, and learned from the low-resolution and high-resolution pairs of examples in the database. 8

16 Figure 2-1 Multi-Image Super Resolution and Multi-Image Super Resolution [29] Figure 2-1, shows different types of super resolution. Super resolution can be done using multiple input image or a single input image. In a single input image, recurring patches within a single low resolution image are considered as if they are extracted from multiple different low resolution images of the same high resolution image. 9

17 Chapter 3 3 Super Resolution Algorithms A high resolution image gives high pixel density and thus provides more details about the original image. However, high resolution images are not always available because high resolution imaging is expensive and also not always feasible due to the limitations of the sensor. To overcome these problems, we use image processing algorithms which are relatively inexpensive and utilize the existing low resolution imaging system. Here, in this chapter we will discuss the most popular super resolution algorithms in detail along with underlying mathematics. All the super resolution algorithms discussed here are from a single input image. In these algorithms, the classical multi-image super-resolution is combined with example-based approach to reconstruct a super resolution image from a single input low resolution image. I have chosen neighbor embedding as it the basic learning based algorithm with which the new algorithms can be compared. Sparse Mixing Estimators method is selected as it is an extension of linear and directional interpolators. By using this method, we can use the already existing interpolators to get better results. Finally, Adaptive Sparse Domain Selection is selected as it is the latest method which combines the aforementioned methods. 3.1 Neighbor Embedding This is a learning based super resolution technique which uses a database of several high resolution training images to super resolve the given low resolution image [6]. This is a single frame image super resolution algorithm as it has only one low resolution observation. This type of learning based single frame image super resolution technique is used in investigative criminology, where face and fingerprint database is available but there is only single observation for the suspect [7]. Additionally, it is used to enhance low resolution text image from various high resolution text images [8]. This method is motivated by the locally linear embedding technique. Besides using the training images, we also implement overlapping to take care of smoothness constraints between patches to obtain high resolution target image. In this algorithm [9] the following properties are implemented to satisfy the requirements of accuracy, local compatibility and smoothness: 10

18 a) Each patch in target high resolution image is associated with multiple patch transformations learned from the training set. b) Local compatibility and smoothness constraints between patches in target image are implemented through overlapping and inter-patch relationship. c) Accuracy is determined by the relationship between high resolution image and its corresponding patch in low resolution image. This type of method is motivated by Locally Linear Embedding (LLE). Here, small patches in the low and high resolution images form two distinct feature spaces with same number of patches and similar local geometry. Local geometry is illustrated by how a feature vector can be reconstructed by its neighbors in the feature space corresponding to a patch. In this algorithm, first a set of K nearest neighbors in the training image for each patch in the target low resolution image is found and the reconstruction weights of the neighbors that minimize the error of reconstructing patch are computed. Similarly, the high resolution embedding patch is estimated using the suitable high-resolution features of K nearest neighbors and the reconstruction weights. Then the target high resolution image is constructed by using above three properties to achieve smoothness and local compatibility. Nearest neighbors are found by using Euclidean distance and the best reconstruction weights for each patch in low resolution target image is found by minimizing the local reconstruction error for each patch. As our human eye can spot the changes in luminance more clearly than the changes in color, we use YIQ color model instead of RGB color model. In this algorithm, only the luminance values from the Y channel are used to define features. Whereas the I and Q channels, which represent chromaticity, are copied from the low resolution image to the target high resolution image. Because the features for the low resolution patches cannot reflect the luminance completely, the mean value is subtracted from the luminance-based feature vector of each high-resolution image and it is added to the target low resolution patch while constructing high resolution image. This algorithm can also be applied if we know a small portion of the target high- resolution image without any other training examples. 11

19 3.2 Sparse Mixing Estimators In this algorithm, prior information of the image is used to estimate the signal which has sparse approximation. Sparsity means that the signal is well approximated by its orthogonal projection over a subspace. The orthogonal projection is given by the below equation: f λ = φ(ci λ ) where f λ is the orthogonal projection of the signal f, I λ is the indicator of the approximation set λ, c is the transform coefficient vector, ci λ selects the coefficients in λ and sets the others to zero, ϕ is the dictionary containing the prior information of the signal, and φ(ci λ ) multiplies the matrix ϕ with the vector ci λ. This sparse representation is estimated by decomposing the low resolution image in a transformed dictionary. In this algorithm, we obtain nonlinear inverse estimators for image interpolation by adaptively mixing linear estimators. Non-linear algorithms produce better results over linear interpolators as they remove noise effectively. Combining the general linear observation model y = Uf + w with the above orthogonal projection equation, we get y = Uφ(cI λ ) + w where y is the low resolution measurement, U is the linear operator, f is the signal and w is the additive noise. Inverse sparse mixing estimators are derived from a mixture model by minimizing an l1 norm over blocks of frame coefficients, with weights depending upon quadratic signal priors. Low resolution measurements have an infinite number of possible decompositions in the dictionary. Hence, the transformed dictionary is redundant and has column vectors which are linearly dependent. A sparse approximation with low resolution image can be calculated with a basis pursuit algorithm. This algorithm minimizes a lagrangian penalized by an l1 norm, 1 y Uφc 2 + λ c 2 1. SME computes adaptive signal representation in blocks with linear regularization in each block. This will give accurate estimates for highly unstable inverse problems. In this algorithm, we obtain sparse inverse estimators by adaptive mixing of linear Tikhonov estimators, over blocks of vectors in a frame. Tikhonov regularization imposes a condition that the solution should have a regularity to optimize a linear estimator. In Bayesian terms, Lagrangian is minus the log of the posterior distribution of the signal whose minimization yields the maximum posterior estimator. The first term is proportional to minus the log probability of 12

20 the Gaussian distribution of the noise; the second term is minus the log probability of a Gaussian prior distribution. Thus, the regularity prior is interpreted as a covariance prior. A mixing estimation is obtained depending on the local signal regularity by weighting the different estimators. This is developed by decomposing low resolution measurement over blocks of localized vectors in a frame and then by finding which blocks yield a small energy after transformation. A local mixture decomposition represents a signal as combination of block components plus a residue. Block is selected if the mixing coefficient is close to 1, and is removed if mixing coefficient is close to 0. The above steps will give the mixing estimator of the signal from a mixture model of input data y = Uf +w. This mixture model is calculated from the frame coefficients and mixing coefficients defined over a family blocks. The main concern in this algorithm is to compute mixing coefficients which optimize the inverse problem mixing estimator. The local signal component has a quadratic regularity that is compatible with the prior used to optimize the linear Tikhonov estimator. Thus, a linear mixture estimator is obtained by minimizing the residue energy using signal regularity over all blocks. This minimization recovers the signal with a sparse set of blocks and also regularizes the decomposition by imposing signal regularity within each block. This optimizes a decomposition parameter for a single mixing parameter per block. The blocks should be just large enough to maintain flexibility. On the other hand, every block must be large enough so that we can observe the regularity of the signal specifically. The estimation also depends on each grid of the position indexes of the blocks. Several sets of translated grids are used for computing the estimation to reduce the grid effect. To minimize the computation of mixing coefficient, an upper bound of the lagrangian is computed from the frame coefficients of the signal in each block. To approximate the optimization with fewer computations, orthogonal block matching pursuit algorithms are introduced. Sparse image mixture models in wavelet frame are used to estimate adaptive directional image interpolation. The sub sampled image is decomposed with wavelet transform matrixes whose columns are the vectors of a translation invariant wavelet frame on a single scale but reconstructed with a dual frame matrix whose columns are the dual wavelet frames. Low frequency image projected over the low frequency scaling filters is separated from high frequency image projected over the finest scale wavelets in three directions. The low frequency 13

21 image can be interpolated using cubic spline interpolator as it has very little aliasing whereas high frequency image is interpolated by selecting directional interpolators. If the directional regularity factor is relatively small in the block, directional interpolation is applied over a block of wavelet coefficients. Experiments are performed with 20 angles with blocks having a width of 2 pixels and a length between 6 and 12 pixels. In this algorithm, the original images are down sampled by a factor of 3 with no aliasing filter. Super resolution algorithm is then applied to the resulting low resolution image. In our experiment, we consider lagrangian multiplier to be 0.6. As explained above all the results are obtained with orthogonal block matching pursuits because they require fewer operations. 3.3 Adaptive Sparse Domain Selection and Adaptive Regularization In this algorithm, we use sparse representation where prior knowledge of natural images is used to standardize the image restoration problem. The quality of the restored high resolution image depends on the sparse domain representation. In ASDS, a sub dictionary is assigned adaptively to each local patch as the sparse domain. This uses weighted l1- norm sparse representation model for Image Restoration. The final high resolution image is reconstructed by averaging all the reconstructed patches. The image patches are overlapped to suppress the noise and block the artifacts. The main procedure in ASDS is to determine the best fitted sub dictionary for each local patch. First a dataset of local image patches are constructed for training in order to select a sub- dictionary to each patch. For this, a set of high- quality natural images are collected and cropped into image patches with same size. The cropped image patch will be used only if its intensity variance is greater than a threshold so as to exclude smooth patches and include only meaningful patches with a certain amount of edge structures. The dataset containing image patches is clustered and a sub dictionary from each of the clusters is taken. Clustering is done in feature space and every cluster represents distinct patterns in the dataset. Training dataset is rotation invariant as many patches cropped from training images are rotated versions of the others. High- pass filtering output of each patch is used as a feature for clustering, as image edges convey most of the important information of the image. This will help to increase the accuracy of clustering by allowing us to focus on the edges of the image patches. 14

22 The centroid of each cluster is calculated and is used to find the sub-dictionary from the cluster such that all the elements in cluster represent sub-dictionary. The number of elements is limited and has similar patterns in the selected sub-dictionary. Defining the dictionary is the most important factor as a compact dictionary will decrease the computational cost of coding for a given image patch. Hence, Principal Component Analysis is used for deciding a good dictionary in ASDS. PCA is a signal de-correlation and dimensionality reduction technique that is used in spatially adaptive image denoising. The dictionary is constructed by computing principal components by applying PCA transform to each sub-dataset. By applying PCA to covariance matrix, an orthogonal transformation matrix is obtained and is set as the dictionary. Only the first few most important eigenvectors in orthogonal transformation matrix are used to balance between l1-norm regularization term and l2-norm approximation term. Due to this the reconstruction error will increase with the decrease in number of eigenvectors. Hence, optimal value of r is determined. By applying the above processes to all the sub-datasets, we get sub-dictionaries which will be used in the adaptive sparse domain selection process of each given image patch. We take the initial estimation of each local patch by calculating wavelet bases as the patch is unknown beforehand. The best fitted sub-dictionary for a local patch is selected by comparing its high-pass filtered patch to the centroid. The dictionary for the patch is selected based on the minimum distance between high-pass filtered patch and centroid. The sub-dictionary is determined in sub space of centroid as the initial estimate of patch can be noisy and it will be difficult to calculate the distance between high pass filtered patch and centroid. The above mentioned methods can be further improved by introducing adaptive regularization terms. One of the adaptive regularization terms is auto regression. In this, one autoregressive model is selected to regularize the input image patch from clustered high quality training image patches. In ASDS, a square window autoregressive model is used to predict the central pixel of the window by using the neighboring pixels from all the sample patches inside it. Determining the best order of the AR model is important as high order AR model may cause data over-fitting. Hence, we use AR model of order 8 which is a 3x3 window. The adaptive selection of the AR model for each patch is same as the selection of sub-dictionary for that patch. 15

23 Non-local similarity constraint is also used along with adaptive regularization term to the local autoregressive model to improve image enhancement as there will be many repetitive image structures in natural images. This is done by selecting a similar patch from the first 10 closest patches to the local patch. We use weighted average of central pixel of similar patch to predict the central pixel of the local patch. Contents of different images vary greatly but the micro-structures of images represented by a small number of elementary features are quantitatively similar. Therefore, we can train the dictionaries to represent the natural images by using the extracted patches from several training images which are rich in edges and texture. In our experiment, we use two different training sets where each set has five high quality images. These two sets have different training images. Another important feature in this method is patch size. Smaller patch sizes tend to generate some artifacts in smooth regions where as larger patches cannot represent small number of atoms as there will not be any micro-structures. Hence, to trade off these two, we adopted 7x7 as the image patch size in our execution. We use 727,615 patches of 7x7 sizes which are randomly cropped from each set of training images. We apply 7x7 Gaussian Kernel of standard deviation 1.6 to the original image and then down sampling it by a factor of 3 to get degraded low resolution image. ASDS method is then applied to the low resolution image to get high resolution image. 3.4 Kernel Super Resolution The standard way of obtaining super resolved images is by using kernels [12]. The common problem which arises with kernel super resolution is blurring of sharp edges as kernel filters perform well in smooth areas, but not in edge areas. The second problem with the kernel super resolution is blocking in diagonal edges or lines [12]. A staircase pattern which is caused by horizontal and vertical orientation of resampling kernels appears in the bilinearly super resolved image. Though this approach is easy, the results are not as good as predicted. Another problem with kernel super resolution is the inability to generate high frequency components. The algorithms for super resolution in the past few years differ from each other greatly. Some use machine learning to obtain relationship between low resolution image and its high resolution 16

24 counterpart while others use explicit models to represent natural images. Some algorithms perform super resolution for each pixel to use the concept of pixel classes to perform 3.5 Cubic Spline Interpolation In the interpolation process, first the discrete data is interpolated into a continuous curve and then it is sampled at a different sampling rate. Cubic Spline gives smooth interpolation and is easy to implement [23]. In cubic spline, a series of unique cubic polynomials are fitted between each of the data points to obtain a smooth and continuous curve. Coefficients of this cubic polynomial are weights which are used to interpolate the data by bending the line so that it passes through each data point without any breaks in continuity [24]. Cubic Spline should follow the following four properties: The piece wise function should interpolate all the data points. The function should be continuous First derivative of the function should be continuous Second derivative of the function should be continuous The main application of cubic spline interpolation is curve fitting. 17

25 Chapter 4 We have discussed the algorithms of super resolution methods in detail. Now, let us see the results when these methods are implemented practically on the set of a few standard images. The quality of the high resolution image obtained from the super resolution methods is measured either by visual inspection or by using PSNR and SSIM check. As most of the energy is limited in the lower part of the spectrum, the PSNR measure is heavily inclined towards the lower part. Hence, the PSNR may not be a good measure to evaluate the performance of a super resolution algorithm. But, for better understanding we use both PSNR and SSIM as the comparison parameters. 4 Experimental Results The results of the three super resolution algorithm experiments, performed on various low resolution images is shown in this section. The purpose is to compare the three algorithms for both original and cropped images. The three algorithms are implemented on a Kodak database containing high resolution images. Ten images which are used for the experimental results are shown below. These are selected from the standard Kodak database. PSNR and SSIM values of the three algorithms obtained when employed on the below selected images are tabulated below in Tables 4-1 and 4-2 respectively. 18

26 Figure 4-1 Test Images Taken from Standard Kodak Database 4.1 Results Images NE SME ASDS Bike Butterfly Flower Girl Hat Leaves Parrot Parthenon Plants Raccoon Table 4-1 PSNR Values Obtained Using Super Resolution Algorithms 19

27 Images NE SME ASDS Bike Butterfly Flower Girl Hat Leaves Parrot Parthenon Plants Raccoon Table 4-2 SSIM Values Obtained Using Super Resolution Algorithms PSNR value gives the signal strength relative to the noise level, thus, the higher the ratio the lesser the noise level in the image. The higher value indicates that the signal strength is stronger when compared to the noise. From the above tables, we can clearly see that ASDS method out performs when compared to the SME and NE for all the images. More understanding of these methods can be obtained by looking at the resulting images. The output images obtained when these algorithms are performed are shown below. First we have shown the results for the complete image of girl, parrot and bike and then the cropped portion of the interested area which contains edges. The clipped portion is then shown 20

28 for better understanding. For each image, its original high resolution image, down sampled low resolution image, super resolved image obtained from ASDS, Cubic Spline, NE and SME are shown below for comparison. 21

29 ASDS_AR_NL_TD1_girl CubicSpline_girl HR_girl LR_NE_girl NE_girl SME_girl Figure 4-2 Super Resolution Algorithm Results of Girl Image 22

30 ASDS_AR_NL_TD1_girl CubicSpline_girl HR_girl LR_NE_girl NE_girl SME_girl Figure 4-3 Super Resolution Algorithm Results of Cropped Girl Image 23

31 ASDS_AR_NL_TD1_Parrots CubicSpline_Parrots HR_Parrots LR_Parrots NE_Parrots SME_Parrots Figure 4-4 Super Resolution Algorithm Results of Parrot Image 24

32 ASDS_AR_NL_TD1_Parrots CubicSpline_Parrots HR_Parrots LR_Parrots NE_Parrots SME_Parrots Figure 4-5 Super Resolution Algorithm Results of Cropped Parrot Image 25

33 ASDS_AR_NL_TD1_bike CubicSpline_bike HR_bike LR_bike NE_bike SME_bike Figure 4-6 Super Resolution Algorithm Results of Bike Image 26

34 ASDS_AR_NL_TD1_bike CubicSpline_bike HR_bike LR_bike NE_bike SME_bike Figure 4-7 Super Resolution Algorithm Results of Cropped Bike Image 27

35 4.2 Discussion By simply looking at the above output images we can judge the best algorithm for a particular kind of the image. The best algorithm preserves even the minute details of the images and reconstructing the high resolution version of the image with the removal of blurring and clear edges. Images from the NE algorithm are still blurry and do not produce a clear high resolution image because of the poor PSNR values. The NE algorithm does not produce better results than Cubic Spline. From the above results we can clearly say that sparse representation models produce better results than regular learning based models. Though both neighbor embedding and adaptive sparse domain selection methods use nearest neighbors from the dictionaries of several high resolution training images to super resolve the given low resolution, ASDS performs far better compared to NE because ASDS uses l1- norm to compute the best reconstruction weights for each patch whereas NE uses Euclidean distance. Though SME and ASDS results are pretty close, ASDS performs well as the image patches are overlapped to remove the noise whereas in SME we use directional interpolator to interpolate high frequency regions. Consider the girl image, the output images of the ASDS method are almost the exact replica of the original image with sharp edges. Image obtained from SME method appears to have better quality than that of NE method but still it lost some details specially near at the edges. When we look at the cropped image, nose edges are preserved in both SME and ASDS, but the texture content is lost in SME method. Similar results are produced with parrot and bike images. The features of these images are better reconstructed using ASDS method compared to the other two methods. This can be clearly observed using the cropped images rather than the complete image. Now take a look at the time taken to run each of these algorithms. Images NE SME ASDS Bike Butterfly Flower

36 Girl Hat Leaves Parrot Parthenon Plants Raccoon Table 4-3 Time In Seconds to Execute the Algorithms on the Image This test is performed on a computer having a processor speed of 2.3 GHz, 4 GB RAM and 64 bit operating system. From the table it is clear that the time taken for reconstructing a high resolution image using ASDS method is less when compared to the other two methods. As the number of iterations is more in NE algorithm, it consumes more time. Thus, ASDS method is the fastest algorithm among the three and NE method is the slowest. SME also gives comparatively close results with ASDS and hence can be used for super resolving. 29

37 Chapter 5 5 Graphic User Interface (GUI) Using this new easy to use graphical interface, selecting the type of algorithm and cropping the desired portion can be performed in an intuitive and easy way with the help of the options in the graphical interface. GUI in MATLAB has following advantages over other GUI development tools: [25] High level Script Based Development: MATLAB scripts do not require compilation, installation of variables and low-level memory management. This high-level script based development allows us to focus on the problem rather spending massive time on coding a GUI application using a low-level language. Hence, the code is much simpler. Operating System Independent GUI Applications: MATLAB allows flexibility for code development as it supports almost all operating systems. As GUI applications are not compiled, they may be executed using MATLAB that runs on any supported operating system. User Interactivity and Real Time Measurements By pushing buttons the user is guided through the whole process of output. It is user friendly. GUI window is divided into screen layout and callback handling which shows the difference between how the screen looks versus what it does when you interact with it. To know more about the screen layout, MATLAB Handling Graphic objects are discussed first. 30

38 Main Figure Axes UI Control UI Menu Image Light Line Patch Surface Text Figure 5-1 MATLAB Handling Objects [26] Every object on the screen has a unique identifier, called a handle, which allows to modify the object at any time. The main object is the screen and figure object is figure window. Every figure consists of Axes, UI Controls and UI Menus. There are 9 different styles of UI Controls: push buttons, radio buttons, sliders, checkbox, edit, list box, popup menus, text, and toggle buttons. These UI Controls can be placed anywhere on a window. Though the number of available UI Controls is limited, it is possible to create advanced GUIs. The UI Menus are the menus and submenus at the top of a window: such as File, Help, Edit, etc. The Axes are used to visualize data and they consist of Images, Lines, Text, etc. Latest versions of MATLAB have a tool to graphically create a GUI layout. It is called as GUIDE (Graphical User Interface Development Environment) as it is a GUI to create a GUI. To perform each of these tasks, first a new window is opened and then the user can select desired features of the task, finally the window is closed and the user can return to the main window. Every function follows the following three steps: First a figure window is created together with UI Menus, UI Controls and Axes. The handles of all these objects are stored in properties of figure. By executing the callbacks, handles are retrieved and stored in figure properties if the figure already exists. 31

39 Programming is used to define all the callbacks in one function. In this report, I have used four axes to display input high resolution image, low resolution image, super resolution image and WVU logo. Four push buttons, three for selecting three different algorithms and fourth for resetting all the axes and one popup menu for selecting the input image. Three text boxes for displaying titles for axes and one edit box to display the PSNR and SSIM results. The user can also crop the desired portion in the image for clear comparison. First, the image is selected using the popup menu; it then appears on the input high resolution image axes and down sampled low resolution image is displayed in low resolution image axes. Then the user should select algorithm of his/her choice. Selected super resolution algorithm will be applied to low resolution image and super resolved image will be displayed in super resolution image axes. PSNR and SSIM values are displayed in the edit box for the selected super resolution algorithm. User is then allowed to crop the area of interest better comparison. Cropping in any one axes will perform same cropping in the other two axes. For selecting the second image user should click the reset button. On selecting the reset button all the handles are reset to the default mode by removing the data and setting the horizontal and vertical axis to initial settings. Whenever any algorithm is selected, the axis in the bottom right corner will display the WVU logo after execution. 32

40 Figure 5-2 The Final Output Result Window 33

41 Figure 5-3 GUI Output Result Window for Bike Using ASDS Algorithm 34

42 Figure 5-4 GUI Output Result Window of Cropped above Bike Image 35

43 When the final GUI code is executed, the GUI output result window shown in Figure 5-2 will be displayed where we can select the image and algorithm. After we select one image from the list of ten images, it will be appear in Input high resolution image axis. After selecting the method, low resolution and super resolution image for the selected input image will be displayed along with the PSNR and SSIM values. Result window of ASDS method for bike image is shown in Figure 5-3.The handles for cropping the desired portion will be pop up as soon as the images are displayed. This can be seen in Figure 5-2. Starting and Ending points are selected to crop a desired portion for better comparison. Cropped result will provide better comparison than the complete images. This can be shown in Figure 5-4. In cropped images, the edges can be seen clearly. 36