Anisotropic representations for superresolution of hyperspectral data

Transcription

1 Anisotropic representations for superresolution of hyperspectral data Edward H. Bosch a, Wojciech Czaja a, James M. Murphy a, and Daniel Weinberg a a University of Maryland: Norbert Wiener Center For Harmonic Analysis and Applications, College Park, MD ABSTRACT We develop a method for superresolution based on anisotropic harmonic analysis. Our ambition is to efficiently increase the resolution of an image without blurring or introducing artifacts, and without integrating additional information, such as sub-pixel shifts of the same image at lower resolutions or multimodal images of the same scene. The approach developed in this article is based on analysis of the directional features present in the image that is to be superesolved. The harmonic analytic technique of shearlets is implemented in order to efficiently capture the directional information present in the image, which is then used to provide smooth, accurate images at higher resolutions. Our algorithm is compared to both a recent anisotropic technique based on frame theory and circulant matrices, 1 as well as to the standard superresolution method of bicubic interpolation. We evaluate our algorithm on synthetic test images, as well as a hyperspectral image. Our results indicate the superior performance of anisotropic methods, when compared to standard bicubic interpolation. Keywords: dictionaries. Superresolution, image processing, shearlets, remote sensing, hyperspectral images, anisotropic 1. INTRODUCTION This article develops a novel algorithm for the superresolution of images. The process of superresolution increases the resolution of an image, while attempting to preserve smoothness and information content in the image, and without introducing artifacts. A superresolution algorithm may incorporate additional information, such as lower resolution sub-pixel shifts of the same scene, or images of different modalities. This additional information ideally allows for more accurate superresolution. Several well-established methods for superresolution are based on interpolation, which typically involves some form of weighted local averaging 2 or kernel convolution. More sophisticated methods exploit the geometry inherent in the image to augment these interpolation schemes by improving smoothness. 1 Anisotropic harmonic analysis techniques are useful for analyzing the geometry of an image. These methods provide directionally sensitive computational tools for decomposing an image, and for efficiently and accurately encoding its most salient features. In particular, the construction of shearlets offers a computationally efficient method for analyzing the directional content of an image. This information can be used to provide smoother superresolved images, as our algorithm s results demonstrate; see Section 5. The structure of this article is as follows. We begin with relevant background on the problem of image superresolution in Section 2. We give a brief introduction to anisotropic harmonic analysis in Section 3. Our shearlet-based Further author information: (Send correspondence to J.M.M.) E.B.: bosch e1@verizon.net W.C.: wojtek@math.umd.edu J.M.M.: jmurphy4@math.umd.edu D.W.: daerwe@math.umd.edu

2 superresolution algorithm is detailed in Section 4, and it, along with an anisotropic superresolution technique based on circulant matrices, is tested on synthetic data and a hyperspectral image in Section 5. We conclude and explore future work related to this algorithm in Section BACKGROUND ON SUPERRESOLUTION The problem of superesolution is significant in image processing. The goal of superresolution is to increase the resolution of an image I, while preserving detail and without producing artifacts. The outcome of a superresolution algorithm is an image Ĩ, which is of the same scene as I, but at a higher resolution. In this theoretical discussion, we restrict ourselves to greyscale images. This allows us to consider our images as real-valued matrices. Let I be an M N matrix and Ĩ an M Ñ matrix, with M < M, N < Ñ. We consider the common case where M = 2M and Ñ = 2N, which corresponds to doubling the resolution of the original image. Images with multiple channels, such as hyperspectral images, can be superresolved by superesolving each channel separately. Superesolution can be implemented by using information in addition to I, such as low resolution images at sub-pixel shifts of the scene, 3 or images of the scene with different modalities. The latter method is related to the specific problem of pan-sharpening. 4 In general, the incorporation of additional information sources into the process of superresolution is similar to the approach of image fusion. Alternatively, superresolution can be performed using only I. The first type of superresolution requires additional data, and is thus less desirable than the second type. In this article, we shall develop a superresolution method of the second type, which requires as input only the image itself. There are several standard approaches to superresolving I without using additional information. Among the most common are nearest neighbor interpolation and bicubic interpolation. Let us consider the superresolved version of I = {a m,n }, denoted Ĩ = {ã i,j}. Here, the values ã i,j and a m,n may be understood as entries of a real matrix representing the images. We must compute each pixel value in the new image, namely ã i,j, from the pixel values of the original image, a m,n. In the case of nearest neighbor interpolation, new pixel values are computed replicating current pixel values. This method is simple and computationally efficient, but leads to extremely jagged superresolved images. It is unsuitable when a high-quality, smooth Ĩ is required. Other methods involve convolving the image with an interpolation kernel, which amounts to taking a weighted average of pixel values within some neighborhood. For example, bicubic interpolation determines Ĩ by computing each ã i,j as a weighted average of the 16 nearest neighbors in I; the weights are chosen to approximate the derivative values at the pixels being analyzed. 2 A novel method for improving the smoothness of images superresolved using these methods was recently demonstrated. 1 In this algorithm, local dominant directions are computed using tight frames derived from circulant matrices. After using nearest neighbors or bicubic upsampling, a motion blur filter is applied in the dominant direction. Areas with low variance are assumed to have no dominant direction. This method resulted in superresolved images with much smoother edge features when compared to the interpolation techniques alone. This method proved effective, but required new tight frames to be computed for each application of the algorithm, depending on the image size and the structure of the features present, thus limiting its efficacy. Essentially, this method uses frame theory to compute locally dominant directions; we shall compute locally dominant directions in another, more efficient manner. The aim of this paper is to develop a superresolution algorithm that computes dominant directions efficiently and accurately, using the harmonic analytic construction of shearlets. 5 8 This method is quite general, and can be applied to images of any size, and has few tunable parameters. Both the shearlet superresolution algorithm and the anisotropic circulant matrices algorithm shall be compared to bicubic interpolation. We demonstrate the versatility of these anisotropic harmonic analysis algorithms for image superresolution, and their superior performance when compared to the isotropic method of bicubic interpolation.

3 3.1 Background on wavelet theory 3. BACKGROUND ON HARMONIC ANALYSIS Shearlets are a generalization of wavelets that incorporates a notion of directionality. We thus begin our mathematical discussion of shearlets with some background on wavelets. 9 In a broad sense, wavelet algorithms decompose an image with respect to scale and translation. Mathematically, given a signal f L 2 ([0, 1] 2 ), understood as an ideal image signal, and an appropriately chosen wavelet function ψ, f may be written as f = f, ψ m,n ψ m,n, (1) m Z n Z 2 where: ψ m,n (x) := det A m/2 ψ(a m x n). A GL 2 (R). A typical choice for A is the dyadic isotropic matrix A = ( The set of wavelet coefficients { f, ψ m,n } m Z,n Z 2 describes the behavior of f, our image signal, at different scales (determined by m) and at different translations (determined by n). This infinite scheme is truncated to work with real, finite image signals. 10 Wavelets have been effectively applied to problems in image compression, 11 image fusion, 12 and image registration. 13 ). 3.2 Background on anisotropic harmonic analysis and shearlet theory Beginning in the early 2000s, several efforts were made to extend wavelet theory to be anisotropic, that is, to include a directional character in the coefficients it produces. Among the most prominent of these theoretical constructions are curvelets, 14 contourlets, 15 and shearlets. 6, 7 Shearlets generalize wavelets by decomposing with respect not just to scale and translation, but also direction. Mathematically, given a signal f L 2 ([0, 1] 2 ) and an appropriate shearlet function ψ, we may decompose f as f = i Z where: j Z k Z 2 f, ψ i,j,k ψ i,j,k, (2) ψ i,j,k (x) := 2 3i 4 ψ(b j A i x k). ( ) ( A =, B = ).

4 Note that A is no longer isotropic, hence it will allow our new analyzing functions to be more pronounced in a particular direction. The new matrix B, a shearing matrix, lets us select the direction. The shearlet coefficients { f, ψ i,j,k } i,j Z,k Z 2 describe the behavior of f at different scales (determined by i), translations (determined by k) and directions (determined by j). The anisotropic character of shearlets has proven useful for a variety of problems in image processing, including image denoising, 16 image registration, 17 and image fusion. 18 Preliminary results of the superresolution of images using shearlets have been reported. 19 The ambition of this article is to further study the role of anisotropic harmonic analysis in the problem of image superresolution. 4. DESCRIPTION OF SHEARLET SUPERRESOLUTION ALGORITHM The approach taken in our algorithm is to use anisotropic shearlets to determine the directional content present in an image. This information is then used to smoothly blur the image in certain directions locally, to provide a smoother superresolved image. Our algorithm for shearlet-based superresolution is coded in MATLAB. It is described below. Consider an M N image matrix I; the superresolved image shall be denoted Ĩ as above. 1. Apply the Fast Finite Shearlet Transform 20, 21 to I. This produces shearlet coefficients up to 1 2 log 2(max{M, N}) scales. If we label the scales from coarsest to finest scale starting at j = 1, we have 2 j+1 matrices of size M N at the jth scale, each corresponding to a different direction from 90 to ( (1 1/2 j+1 )), equally spaced. Denote these directional matrices D 1,..., D 2 j+1. The parameter j may be set differently depending on the size of the image and scale of the features being analyzed. 2. Upsample by a factor of 2 each of D 1,..., D 2 j+1, using the upsampling method of bicubic interpolation to acquire D 1,..., D 2 j+1. These contain the directional information that will be used later. 3. Upsample by a factor of 2 the original image I, using both bicubic interpolation and nearest neighbors to acquire Ĩbc and Ĩnn, respectively. The upsampled Ĩnn will be modified using the directional information present in D 1,..., D 2 j Assign each pixel in Ĩnn a local direction based on which matrix contains the shearlet coefficient of largest magnitude, i.e., which entry in that location is maximal among D 1,..., D 2 j+1. Pixels are determined to have no local direction if the magnitude of their largest shearlet coefficient is below some prescribed threshold, T. 5. Apply a motion blur filter of length 5 in each of the 2 j+1 directions to Ĩnn, to produce Ĩ1,..., Ĩ2j Construct the superresolved image as follows. If a pixel was determined to have a local direction, replace the pixel value by its corresponding blurred version based on the assigned local direction, i.e. with the pixel value in Ĩm where the pixel has dominant direction corresponding to m. If a pixel was determined to have no local direction, replace its value by its corresponding bicubic version from Ĩbc. If a pixel is within 10 pixels of an image boundary, replace its value by its corresponding bicubic version from Ĩbc; this is to prevent undesired edge effects. 7. Output the superresolved image Ĩ. We note again that in our algorithm, bicubic interpolation is used to superresolve pixels with no dominant direction, while nearest neighbors is applied to those pixels with a dominant direction, before smoothing. This is because bicubic interpolation is unnecessary for pixels that will be smoothed later. 5. EXPERIMENTS AND RESULTS We considered two anisotropic algorithms: one based on shearlets, as detailed in Section 4, and one based on frame theory and circulant matrices. 1 We compared these algorithms to standard bicubic interpolation, which is isotropic.

5 5.1 Synthetic experiments To test our algorithms, we first considered synthetic experiments. Since our algorithms incorporate anisotropic information, we wanted to study their efficacy on images that have very prominent directional content. We first constructed half planes in MATLAB, at various slopes. Two such half planes appear in Figure 1. Figure 1: Half planes of slope 3 and.5, respectively. We hypothesized our anisotropic algorithms would efficiently capture the directional content, and produce highquality superresolved images of these half planes. Since there is only one major feature in these images, it is difficult to interpret the results via visual inspection. Hence, to judge the quality of our superresolution algorithms for these half planes, we compute the peak signal to noise ratio (PSNR) for each of of the half planes, superresolved with either our shearlets method, circulant matrices, or bicubic. This measures the overall quality of the method by considering superresolution as a recovery problem. Indeed, PSNR requires an ideal or true signal to compare with the superresolved image. In this case, we downsample the planes to be , then apply our superresolution algorithms to double this resolution back to We then compare these images with the original planes to compute the PSNR. The values of PSNR for each of our methods for planes at different slopes appear in Table 1. We test our shearlet superresolution algorithm with 16 (j = 3) and 32 (j = 4) directions, as these methods produce slightly different results. The shearlet superresolution algorithms are run with the threshold parameter set T =.05. Planar Slope PSNR Shearlet (j = 3) PSNR Shearlet (j = 4) PSNR Circulant Matrices PSNR Bicubic Table 1: The PSNR values for various methods on angled half planes. We notice that the anisotropic harmonic analysis methods outperform bicubic interpolation in all cases. We next consider a synthetic experiment on a circle, which appears in Figure 2. We again performed superresolution with our shearlet algorithm, the circulant matrix algorithm, and bicubic interpolation, computing PSNR

6 in a method identical to the method discussed above. The results appear in Table 2. Figure 2: Synthetic circle to be tested. PSNR Shearlet (j = 3) PSNR Shearlet (j = 4) PSNR Circulant Matrices PSNR Bicubic Table 2: The PSNR values for the circle. For the circle experiment, the shearlet algorithm with 16 directions performs best, and all anisotropic methods outperform bicubic interpolation. We conclude that in the case of simple, synthetic experiments, anisotropic superresolution provides superior performance when compared to conventional bicubic interpolation. 5.2 Hyperspectral experiments We also tested our algorithms on real data. We considered a hyperspectral data set of the University of Houston and surrounding area, 22 which consists of 144 bands of size This dataset has a spatial resolution of 2.5 m, and spectral resolution of between 380 nm and 1050 nm. For convenience, we extracted a subset from band 70. Band 70 was chosen for its relatively high contrast. We performed superresolution experiments with our shearlet algorithm with 16 directions, the circulant matrices algorithm, and bicubic interpolation. We note that in accordance with the bound on the number of directions generated by the shearlet algorithm, this image is too small to consider 32 shearlet directions. The images resulting from these experiments appear in Figure 3. The direction map for the shearlet algorithm appears in Figure 4.

7 (a) Original scene of Houston (c) Superresolved with circulant matrices algorithm. (b) Superresolved with shearlet algorithm. (d) Superresolved with bicubic interpolation. Figure 3: Original image of Houston and results of our superresolution experiments. Notice that the algorithms based on anisotropic harmonic analysis produce smoother edges, when compared to bicubic interpolation. Figure 4: Direction map for the shearlet algorithm applied to the Houston scene. There are 16 possible local direction for each pixel, varying from 90 (blue) to 259 (red). The darkest blue corresponds to no assigned local direction. We quantitively analyzed the results of our algorithm as before, by computing PSNR. The results appear in Table 3. We see from these results that our shearlet algorithm gives superior performance over simple bicubic

8 interpolation. PSNR Shearlet (j = 3) PSNR Circulant Matrices PSNR Bicubic Table 3: The PSNR values for the University of Houston scene. 6. CONCLUSIONS AND FUTURE WORK We described a shearlet-based algorithm for the superresolution of images. We performed the algorithm on simple synthetic examples including half-planes and circles. We compared our algorithm at two different scales with bicubic interpolation and a recent circulant matrix method. 1 Our algorithm performed the best in 5 out of 9 of the half-plane examples, while the circulant matrix method performed the best in the the remaining 4 examples. This indicates that algorithms based on anisotropic harmonic analysis outperform bicubic interpolation in all iterations of our half plane experiments. Our shearlet superresolution algorithm also performed the best for the circle experiment. Finally, we applied our algorithm to one channel of a hyperspectral image. Compared to bicubic interpolation, our algorithm produced an image having fewer jagged edges and in fact also has improved PSNR. One of the greatest challenges for our algorithm is superresolving images with many textures without degrading the PSNR. Many textures, especially trees, give rise to large shearlet coefficients and become blurred as a result of our algorithm. In future work, we would like to find a method for filtering out the textures so as to only smooth the edges. In addition, we would like to consider more sophisticated ways of improving edges beyond motion blurring, which tends to decrease image sharpness. Cutting-edge methods, incorporating directional interpolation and statistical methods, exist, 23, 24 and shall be studied. 7. ACKNOWLEDGEMENTS The authors would like to thank the Hyperspectral Image Analysis group and the NSF Funded Center for Airborne Laser Mapping (NCALM) at the University of Houston for providing the data sets used in this study, and the IEEE GRSS Data Fusion Technical Committee for organizing the 2013 Data Fusion Contest. REFERENCES 1. Bosch, E. H., Castrodad, A., Cooper, J. S., Czaja, W., and Dobrosotskaya, J., Multiscale and multidirectional tight frames for image analysis, in [Proceedings of SPIE, vol. 8750], (2013). 2. Keys, R., Cubic convolution interpolation for digital image processing, IEEE Transactions on Acoustics, Speech and Signal Processing 29(6), (1981). 3. Park, S. C., Park, M. K., and Kang, M. G., Super-resolution image reconstruction: a technical overview, IEE Signal Processing Magazine 20(3), (2003). 4. Czaja, W., Doster, T., and Murphy, J. M., Wavelet packet mixing for image fusion and pan-sharpening, in [SPIE Defense+ Security], (2014). 5. Easley, G. R., Labate, D., and Lim, W.-Q., Sparse directional image representations using the discrete shearlet transform, Applied and Computational Harmonic Analysis 25(1), (2008). 6. Labate, D., Lim, W. Q., Kutinyok, G., and Weiss, G., Sparse multidimensional representation using shearlets, in [Proceedings of International Society for Optics and Phototronics: Optics and Phototronics], (2005). 7. Guo, K. and Labate, D., Optimally sparse multidimensional representation using shearlets, SIAM journal on mathematical analysis 39(1), (2007).

9 8. Guo, K., Kutinyok, G., and Labate, D., Sparse multidimensional representations using anisotropic dilation and shear operators, in [Wavelets and Splines (Athens, GA, 2005)], Chen, G. and Lai, M., eds., , Nashboro Press (2006). 9. Daubechies, I., [Ten lectures on wavelets], Society for industrial and applied mathematics (1992). 10. Wickerhauser, M. V., [Adapted Wavelet Analysis from Theory to Software], AK Peters Ltd. (1994). 11. DeVore, R. A., Jawerth, B., and Lucier, B. J., Image compression through wavelet transform coding, IEEE Transactions on Information Theory 38(2), (1992). 12. Amolins, K., Zhang, Y., and Dare, P., Wavelet based image fusion techniques-an introduction, review and comparison, ISPRS Journal of Photogrammetry and Remote Sensing 62, (2007). 13. Zavorin, I. and Le Moigne, J., Use of multiresolution wavelet feature pyramids for automatic registration of multisensor imagery, IEEE Transactions on Image Processing 14(6), (2005). 14. Candès, E. J. and Donoho, D. L., New tight frames of curvelets and optimal representations of objects with piecewise C 2 singularities, Communications on pure and applied mathematics 57(2), (2004). 15. Do, M. N. and Vetterli, M., Contourlets: a directional multiresolution image representation, in [Proceedings of 2002 IEEE International Conference on Image Processing], (2002). 16. Easley, G. R., Labate, D., and Colonna, F., Shearlet-based total variation diffusion for denoising, IEEE Transactions on Image Processing 18(2), (2009). 17. Murphy, J. M. and Le Moigne, J., Shearlet features for registration of remotely sensed multitemporal image, in [Proceedings of IGARSS], (2015). 18. Miao, Q.-g., Shi, C., Xu, P.-F., Yang, M., and Shi, Y.-B., A novel algorithm of image fusion using shearlets, Optics Communications 284(6), (2011). 19. Czaja, W., Murphy, J. M., and Weinberg, D., Superresolution of remotely sensed images with anisotropic features, in [Proceedings of SAMPTA], (2015). 20. Häuser, S. and Steidl, G., Fast finite shearlet transform: a tutorial, Arxiv (2014). 21. Häuser, S. and Steidl, G., Convex multiclass segmentation with shearlet regularization, International Journal of Computer Mathematics 90(1), (2013) IEEE GRSS Data Fusion Contest. technical-committees/data-fusion/. 23. Mallat, S. and Yu, G., Super-resolution with sparse mixing estimators, IEEE Transactions on Image Processing 19(11), (2010). 24. Algazi, V., Ford, G. E., and Potharlanka, R., Directional interpolation of images based on visual properties and rank order filtering, in [IEEE International Conference on Acoustics, Speech, and Signal Processing], (1991).