1418 IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 11, NO. 8, AUGUST 2014

Transcription

1 1418 IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 11, NO. 8, AUGUST 2014 Fusion of Hyperspectral and Multispectral Images: A Novel Framework Based on Generalization of Pan-Sharpening Methods Zhao Chen, Hanye Pu, Bin Wang, Senior Member, IEEE, and Geng-Ming Jiang, Member, IEEE Abstract In many applications, it is imperative to maintain high spectral and spatial resolution of remote sensing images. This letter addresses the issue by fusing low-spatial-resolution hyperspectral images (HSIs) and high-spatial-resolution multispectral images (MSIs) of the same scene collected by the coupled sensors and, thus, present a novel framework that generalizes well-established pan-sharpening algorithms. The main steps of the framework are dividing the spectrum of HSIs into several regions and fusing HSIs and MSIs in each region by the chosen pan-sharpening algorithm. Ratio image-based spectral resampling (RIBSR) is used to interpolate the missing data so that every region is covered by a multispectral band. Therefore, the framework allows most of pan-sharpening algorithms to be extended to HSI and MSI fusion. Synthetic data in accordance with sensor reality are used to test specific methods derived within the framework. Experimental results show that the proposed methods excel the state-of-the-art methods in terms of simplicity, feasibility, efficiency, and effectiveness. Index Terms Fusion, generalization, hyperspectral, multispectral, resolution enhancement, spectral resampling. I. INTRODUCTION HYPERSPECTRAL sensors provide high spectral resolution, which often results in a significant tradeoff in spatial resolution due to the limitation in their design [1], [2]. In this letter, we solve the problem by fusing low-spatial-resolution hyperspectral images (HSIs) and high-spatial-resolution multispectral images (MSIs). This idea is directly motivated by the fact that a Japanese next-generation spaceborne hyperspectral imager suite (HISUI), which is similar to coupled MSI and panchromatic (Pan) sensors (e.g., Quickbird), will provide both HSIs and MSIs obtained over the same region with identical Manuscript received July 13, 2013; revised September 27, 2013 and November 17, 2013; accepted November 26, This work was supported in part by the State Key Laboratory of Earth Surface Processes and Resource Ecology Project under Grant 2013-KF-02, by the National Natural Science Foundation of China under Grant and Grant , by the Innovation Program of the Shanghai Municipal Education Commission under Grant 13ZZ005, and by the Research Fund for Doctoral Program of Higher Education of China under Grant Z. Chen, H. Pu, and B. Wang are with the State Key Laboratory of Earth Surface Processes and Resource Ecology, Beijing Normal University, Beijing , China, and also with the Key Laboratory for Information Science of Electromagnetic Waves (MoE), Fudan University, Shanghai , China ( wangbin@fudan.edu.cn). G.-M. Jiang is with the Key Laboratory for Information Science of Electromagnetic Waves (MoE), Fudan University, Shanghai , China. Color versions of one or more of the figures in this paper are available online at Digital Object Identifier /LGRS atmospheric and illumination conditions [3]. The task of this letter is to find out a proper way to merge the coupled HSIs and MSIs and create high-spatial-resolution HSIs with minimum artifacts. There are many spatial enhancement techniques for HSIs [1] [8]. Among the various HSI and MSI fusion methods, wavelet transform (WT) [1], [4] and maximum a posterior (MAP) [2], [7], [8] are two most intensely studied categories. The WT methods suffer from the inherent ring effect that leads to increased oscillations at the sharp edges in the fused image [1]. Zhang et al. applied ratio image-based spectral resampling (RIBSR) prior to WT [4], which greatly improved the results. However, the method requires MSIs to be spectrally resampled for a number of times, resulting in high computational cost. Meanwhile, the MAP method in [2] often produces smoother and nicer images. It exploits the correlation between the images being enhanced by the MAP estimator based on a spatial varying statistical model, which, however, assumes that the pixels are independent. Attempting to satisfy the condition, Zhang et al. performed WT prior to MAP fusion [8]. Unfortunately, this technique yields no obvious improvement to the image quality. We address the issue from a novel and macroscopic view by proposing a framework based on the generalization of pansharpening algorithms. The framework is not limited to specific methods. Any future pan-sharpening algorithms can also be extended to HSI and MSI fusion within the framework. The proposed strategy consists of three steps. The first step is to divide the whole spectrum covered by hyperspectral bands into several regions. The next step is to apply RIBSR to any region not covered by a multispectral band and interpolate the missing band image in the MSI. Make sure that every region is covered by one and only one multispectral band image. The final step is to fuse the HSI and MSI region to region by the selected pansharpening algorithm. Obviously, the strategy allows for spatial resolution enhancement of HSIs with an arbitrary number of bands. All of the steps are straightforward and easy to implement. On one hand, it maintains local spectral information and minimizes spectral distortions by fusing images of each spectral region one after another. On the other hand, the well-developed pan-sharpening algorithms guarantee the spatial fidelity of the fused images. There are two points supporting the scheme. First, HSI and MSI fusion can be regarded as an analog of the pan-sharpening process. Even in some spectral region, there is no multispectral X 2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See for more information.

2 CHEN et al.: FUSION OF HYPERSPECTRAL AND MULTISPECTRAL IMAGES 1419 band to sharpen all the hyperspectral bands, the missing band image can be estimated by RIBSR. Second, only the algorithms that have already succeeded in enhancing the spatial resolution of MSIs are chosen to be generalized since they are also likely to succeed in this case. In particular, we focus on the generalization of two pansharpening algorithms, namely, Gram Schmidt adaptive (GSA) spectral sharpening [9] and local minimum mean square error (LMMSE) [10]. Accordingly, we present two methods named as generalized GSA (G-GSA) and generalized LMMSE (G-LMMSE). GSA and LMMSE are selected for a number of reasons. First, they give satisfactory performance. Second, both of them are easy to implement. Furthermore, they simplify the process of spectral resampling. The proposed methods require the number of resampled multispectral bands equal to the number of divided hyperspectral regions. Hence, they perform RIBSR once at most in each region. In short, the proposed methods excel the existing methods based on MAP estimator or WT in terms of simplicity, feasibility, efficiency, and effectiveness if RIBSR is properly engaged. The rest of this letter is organized as follows: Section II introduces the basic concepts of GSA, LMMSE, and RIBSR as a preparation for the development of our scheme. Section III presents the framework of generalization and explains the proposed methods. Section IV provides experimental results that demonstrate the efficacy of our methods. Finally, the conclusion is drawn in Section V. II. THEORETICAL BACKGROUND GSA [9] and LMMSE [10] are generalized within the proposed framework as two typical examples. The former is an upgraded version of Gram Schmidt (GS) that falls into the category of the component substitution methods. The latter is an outstanding pan-sharpening algorithm that minimizes the squared error between original MSI and fused MSI. It has two alternative injection models: single spatial-detail (SSD) model and band-dependent spatial-detail (BDSD) model. In this letter, we adopt the first model for efficiency. RIBSR [4] is employed when the generalization process needs to be facilitated. Suppose that remote sensing images X R N Q and Y R N P of N pixels have different spatial resolutions, where Q and P present the number of bands in each image. According to [4], the reflectivity ratio of coupled remote sensing images in two given spectral region is almost changeless, as described in the following: X(:,m 1 ) X(:,m 2 ) = Y(:,n 1) Y(:,n 2 ) = I R (1) where I R is named the ratio image. Based on this property, X(:,m 1 ) can be estimated if X(:,m 2 ), Y(:,n 1 ) and Y(:,n 2 ) are given. Meanwhile, it should be noticed that (1) is only an approximation in real applications. Its reliability depends on the assumption that the spectral regions are sufficiently narrow and close to each other. This condition could be satisfied, and significant distortions would be avoided as we carefully tailor the steps of RIBSR to specific data and fusion method. III. PROPOSED SCHEME A. Synthesis of the HSI and MSI to be Fused Although real coupled HSIs and MSIs of the same scene captured at the same time are unavailable to us, synthetic data simulating the real sensor specifications would suffice for demonstration. HSI Y R N P and MSI X R N Q are derived from the reference HSI Z R N P in the following way. Step 1) Set up the relationship between hyperspectral and multispectral regions covered by real data [3]. For example, hyperspectral bands sampled by the Hyperspectral Digital Imagery Collection Experiment (HYDICE) 1 sensor cover the region of nm, which is also covered by multispectral band 1 of Landsat Thematic Mapper (TM). 2 Step 2) Divide the hyperspectral bands into groups according to Step 1. The HYDICE bands to be sharpened by TM band 1 should be in the same group. Let sets λ k, k =1, 2,...,K denote those groups, where K Z +, K Q. The number of bands grouped in λ k is P k.eachsetofλ q, q = 1, 2,...,Qpresents a region where there is a multispectral band. Meanwhile, λ t,t=1, 2,...,T cover the spectral regions missing in X, where T = K Q. Ensure that every λ t is adjacent to at least one λ q so that RIBSR can be applied. Step 3) Produce X by averaging the band images corresponding to each λ q,as Z(:,p q ) p q λ q X(:,q)=, q =1, 2,,Q. (2) P q We synthesize the low-spatial-resolution HSI Ỹ R M P by degrading each band image of Z by the scale of r 2 through an antialiasing technique called bilinear interpolation. As the proposed methods require the spatial dimensions of the HSIs to be fused match those of MSIs, such data Y is produced by bilinear interpolation of Ỹ. B. Implementation of the Proposed Methods The proposed strategy consists of three steps. 1) Divide the spectrum covered by hyperspectral bands into several regions. 2) Apply RIBSR only to the regions not covered by multispectral bands. Make sure that the wavelength range of HSIs and MSIs matches one another. 3) Select a pan-sharpening algorithm with excellent capacity. Sharpen HSIs with MSIs in each spectral region in the same way as sharpening MSI with Pan by the chosen algorithm. The first two steps maintain local spectral information of every region. The last step guarantees the spatial fidelity of the fused images. In particular, we present two methods, i.e., G-GSA and G-LMMSE derived from GSA and LMMSE, respectively. These two pan-sharpening algorithms are generalized mainly because they outperform most of the other alternatives

3 1420 IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 11, NO. 8, AUGUST 2014 TABLE I PROCEDURE OF G-GSA AND G-LMMSE TABLE II CORRESPONDING RELATIONSHIP BETWEEN THE HYPERSPECTRAL AND MULTISPECTRAL BANDS FOR LANDSAT TM SIMULATION TABLE III CORRESPONDING RELATIONSHIP BETWEEN THE HYPERSPECTRAL AND MULTISPECTRAL BANDS FOR QUICKBIRD SIMULATION Moreover, they are time saving and easy to implement. While assuming that the HSIs and MSIs to be fused are collected by coupled sensors from the same platform or simulated in the same way, the proposed methods can merge the HSIs and MSIs from different platforms as long as they are registered. Such application is precedented [1] but is not the focus in this letter. The procedure of the proposed methods is given by Table I. IV. EXPERIMENTAL RESULTS A. Experimental Setup The reference HSI was captured over Washington, DC by HYDICE. 3 It has 210 bands with a spectral resolution of 10 nm. The low-snr and water absorption bands are removed; hence, 191 bands remain covering the wavelength region of nm. The image is cropped into a subimage of (N) pixels for time saving. r is set as 4; hence, M 3 is equal to Landsat TM 4 bands 1 5 and 7 covering the , , , , , and nm regions are simulated. Four Quikbird 5 multispectral bands, which are the same as the first four TM bands, are also synthesized. Two types of corresponding relationship between hyperspectral and multispectral bands are presented in Tables II and III. All the algorithms are implemented in MATLAB R2010a running on a workstation with an Intel Xeon CPU X5667 at 3 GHz (dual core) and 48 GB of RAM. B. Score Indexes Several score indexes are employed to assess the quality of the fused HSI [11] [14]. Erreur relative globale adimensionnelle de synthése (ERGAS) is originally designed for the

4 CHEN et al.: FUSION OF HYPERSPECTRAL AND MULTISPECTRAL IMAGES 1421 TABLE IV SCORE INDEXES RESULTED FROM TM SIMULATION TABLE VI SCORE INDEXES RESULTED FROM QUICKBIRD SIMULATION TABLE V TIME COSTOFTHEFUSING PROCESS IN TM SIMULATION Fig. 1. Score indexes of the fused HSI. (a) PSNR. (b) SAM. evaluation of the overall quality of the pan-sharpened image, but it allows for any number of bands to be measured as suggested by its formula [10]. The ideal value of ERGAS is 0. Peak signal-to-noise ratio (PSNR) [3] in decibels measures the similarity between the fused and the reference HSI. The higher PSNR is better. Q2 n [12] is an extension of the Universal Image Quality Index (UIQI) [13] and measures quality for images with an arbitrary number of spectral bands. It is made up of different components that account for the correlation, the mean of each spectral band, the intraband local variance, and the spectral angle. The ideal value of Q2 n is 1. Spectral angle mapper (SAM) [14] is the absolute value of the angle between the true and estimated spectral vectors. The best value of SAM is 0. C. Performance Evaluation The proposed methods G-GSA and G-LMMSE are compared with six existing algorithms, including MAP [2], 2-D WT, 3-D WT [1], RIBSR + 2D-WT [4], RIBSR + 3D-WT [4], and 2D-WT + MAP [8]. First of all, we expect to demonstrate the effectiveness of the proposed methods. According to the score indexes in Table IV, G-LMMSE always produces the HSI with the best quality. G-GSA comes in at second place. Then, we compare the efficiency of each method by timing the fusing process in seconds. The time cost by RIBSR or other auxiliary techniques involved is not accounted for. Table V suggests that G-LMMSE and G-GSA are the most time-saving methods. Meanwhile, our framework is also subject to qualifications. 1) The choice of the original pan-sharpening algorithm is vital. In the Quickbird simulation, G-GSA falls behind while G-LMMSE remains the best, as shown by Table VI. Therefore, it is not the general framework but the specific method that is flawed. G-GSA involves a regression step to extract detail image in each divided region. When a region, e.g., the last one, has too many hyperspectral bands to be sharpened by one single resampled multispectral band, the accuracy of the regression is at stake. Nevertheless, G-GSA has its own value for it is easier to implement than G-LMMSE. 2) The reliability of RIBSR depends on the spectral regions. Table VI suggests that even the best method G-LMMSE is compromised by the Quickbird simulation. There is a dilemma that the image quality in the last region heavily relies on RIBSR, but the region itself is too wide (119 bands, nm) to ensure the accuracy of RIBSR. 3) RIBSR provides only approximated data. Fig. 1. gives two indexes, PSNR and SAM for spatial and spectral quality, respectively. Both indexes are generally better in the regions that need no resampling than in those that need so. However, the fourth region is an exception. Although not being covered by any multispectral bands, the region with only two hyperspectral bands is narrow enough to ensure the accuracy of RIBSR. No doubt that SAM in this region is pretty low for both methods while the drop in PSNR for G-GSA is small. As far as the MSI and HSI to be fused comprise enough number of narrow overlapping spectral regions, RIBSR can be undoubtedly relied on. Overall, the experimental results have demonstrated the efficiency and effectiveness of the proposed methods. The generalization framework comprises the following features. 1) The inclusive framework addresses the issue of HSI and MSI fusion from a macroscopic view. Existing and future pan-sharpening algorithms can be extended to HSI and MSI fusion within the framework. Moreover, the framework can sharpen HSI with an arbitrary number of bands.

5 1422 IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 11, NO. 8, AUGUST ) When the pan-sharpening algorithms to be generalized are well chosen, the framework achieves the goal of enhancing HSI with little artifacts. It guarantees spatial quality by exploiting the excellent capacity of the chosen algorithms. Meanwhile, it maintains local spectral information by dividing the spectrum into different regions and fusing images of each spectral region separately. 3) The proposed scheme requires RIBSR only if there is any divided hyperspectral region not covered by a multispectral band. RIBSR is performed in each region once at most. Thus it simplifies implementation and reduces time cost. 4) All the steps of the proposed methods are straightforward and efficient. There is no heavy computational burden or demanding math work to complicate the process of implementation. V. C ONCLUSION In this letter, a novel framework for HSI and MSI fusion based on generalization of some well-established pan sharpening has been proposed. It divides the spectrum into different regions and fuses images of each spectral region in the same way as pan sharpening. RIBSR is involved only if there is any hyperspectral region not covered by multispectral bands; hence, no great spectral distortions are introduced. In particular, two methods, i.e., G-GSA and G-LMMSE, generalized from GSA and LMMSE, respectively, have their functionality tested. The experimental results demonstrate that the proposed methods can outperform the MAP or WT fusion methods if RIBSR is properly engaged. Therefore, the goal of acquiring high-spatialresolution HSI with minimized distortions is accomplished. REFERENCES [1] R. B. Gomez, A. Jazaeri, and M. Kafatos, Wavelet-based hyperspectral and multispectral image fusion, in Proc. SPIE, Orlando, FL, USA, Apr , 2001, vol. 4383, pp [2] R. C. Hardie, M. T. Eismann, and G. L. Wilson, MAP estimation for hyperspectral image resolution enhancement using an auxiliary sensor, IEEE Trans. Image Process., vol. 13, no. 9, pp , Sep [3] N. Yokoya, T. Yairi, and A. Iwasaki, Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion, IEEE Trans. Geosci. Remote Sens., vol. 50, no. 2, pp , Feb [4] Y. Zhang and M. He, Multispectral and hyperspectral image fusion using 3-D WT, J. Electron., vol. 24, no. 2, pp , Mar [5] S. Qian and G. Chen, Enhancing spatial resolution of hyperspectral imagery using sensor s intrinsic keystone distortion, IEEE Trans. Geosci. Remote Sens., vol. 50, no. 12, pp , Dec [6] J. C. Chan, J. Ma, P. Kempeneers, and F. Canters, Superresolution enhancement of hyperspectral CHRIS/Proba images with a thin-plate spline nonrigid transform model, IEEE Trans. Geosci. Remote Sens., vol. 48, no. 6, pp , Jun [7] M. T. Eismann and R. C. Hardie, Hyperspectral resolution enhancement using high-resolution multispectral imagery with arbitrary response functions, IEEE Trans. Geosci. Remote Sens., vol. 43, no. 3, pp , Mar [8] Y. Zhang, S. De Backer, and P. Scheunders, Noise-resistant waveletbased Bayesian fusion of multispectral and hyperspectral images, IEEE Trans. Geosci. Remote Sens., vol. 47, no. 11, pp , Nov [9] B. Aiazzi, S. Baronti, and M. Selva, Improving component substitution pan-sharpening through multivariate regression of MS + Pan data, IEEE Trans. Geosci. Remote Sens., vol. 45, no. 10, pp , Oct [10] A. Garzelli, F. Nencini, and L. Capobianco, Optimal MMSE pan sharpening of very high resolution multispectral images, IEEE Trans. Geosci. Remote Sens., vol. 46, no. 1, pp , Jan [11] Q. Du, N. Younan, R. King, and V. Shah, On the performance evaluation of pan-sharpening techniques, IEEE Geosci. Remote Sens.Lett., vol. 4, no. 4, pp , Oct [12] A. Garzelli and F. Nencini, Hypercomplex quality assessment of multi/hyperspectral images, IEEE Geosci. Remote Sens. Lett., vol. 6, no. 4, pp , Oct [13] Z. Wang and A. C. Bovik, A universal image quality index, IEEE Signal Process. Lett., vol. 9, no. 3, pp , Mar [14] X. Zhu and R. B. Bamler, A sparse image fusion algorithm with application to pan-sharpening, IEEE Trans. Geosci. Remote Sens., vol. 51, no.5, pp , May 2013.