Hyperspectral and Multispectral Image Fusion Using Local Spatial-Spectral Dictionary Pair Yifan Zhang, Tuo Zhao, and Mingyi He School of Electronics and Information International Center for Information Acquisition and Processing Northwestern Polytechnical University, Xi an 71129, China Abstract This paper deals with the spatial resolution enhancement of low resolution (LR) hyperspectral image (HSI) by fusing it with high resolution (HR) multispectral image (MSI) of the same observed scene. A new HSI and MSI fusion approach based on local spatial-spectral dictionary pair is proposed. In the proposed approach, HR MSI and its spatial degradation version (LR MSI) are divided into overlapped subimages for the purpose of HR and LR dictionary pair construction. To incorporate spatial and spectral information simultaneously, spatial-spectral dictionary is generated rather than the traditional spectral or spatial ones. Meanwhile, a localized strategy is employed for dictionary construction to generate a local dictionary rather than a global one, to reduce the computational cost. By appropriately assuming that the LR HSI and desired HR HSI can be collaboratively represented by LR dictionary and HR dictionary respectively sharing the same set of representation coefficients, the desired HR HSI is reconstructed by HR dictionary and the obtained collaborative representation coefficients. Simulative experimental results illustrate that the proposed HSI and MSI fusion approach is capable of producing better or comparable fused results compared with some state-of-the-art fusion approaches using spectral or spatial dictionaries, with much lower computational cost. This makes it quite promising in practical application. I. INTRODUCTION Different from multispectral image (MSI), hyperspectral image (HSI) employs hundreds of contiguous spectral bands to capture and process information and features over a range of wavelengths [1], providing more abundant and detailed spectral information and features which are helpful for interpretation, classification and recognition. However, HSI possesses lower spatial resolution compared with MSI. Image fusion is an effective technique to deal with the limited spatial resolution of HSI when multi-source images are available, which has been explored during the recent decades and is still a very active research area in remote sensing. Considering HSI and MSI are both multi-band images containing spatial and spectral information and features simultaneously, improvement can be highly expected when HSI and MSI are fused. From an application point of view, this problem is also very important, as motivated by recent national programs, e.g., the Japanese next-generation spaceborne HSI suite, which fuses coregistered HSI and MSI acquired over the same scene under the same conditions [2]. Recently, dictionary (including spectral dictionary and spatial dictionary) based fusion methods are proposed to address HSI and MSI fusion problem, in which the fused image is represented as a combination of atoms in dictionary. Spectral dictionary based fusion methods usually employ spectral unmixing techniques to decompose the source HSI and MSI into endmember matrices (mainly representing spectral features) and abundance matrices (mainly representing spatial features), and then endmember matrix generated from source HSI and abundance matrix generated from source MSI are used for fused image reconstruction [3] [5]. Considering the self-similarity of images, spatial dictionary based approaches are also proposed for HSI and MSI fusion [6] [9]. Whereas, neither spectral dictionary nor spatial dictionary can fully represent the spatial and spectral features of source images simultaneously. Furthermore, global dictionaries with a large scale are usually adopted, resulting in high computational cost, which limits their application in practice. To handle these issues, a novel HSI and MSI fusion approach is proposed in this paper. The proposed fusion approach adopts a local spatial-spectral dictionary pair and collaborative representation (CR) based reconstruction, which is capable of producing satisfied fusion results with much lower computational cost. This improvement can be attributed to the effective representation of both spatial and spectral features in the local spatial-spectral dictionary pair with small scale. II. METHODOLOGY In this paper, a P -band low resolution (LR) HSI X R N P and a Q-band high resolution (HR) MSI Y R M Q are fused to obtain an appropriate estimation of the HR HSI Z R M P of the observed scene, denoted as Z. The spatial scale factor is defined as ω = M/N. The two observed image data sets X and Y are obtained under the same atmospheric and illumination conditions and are geometrically coregistered with radiometric correction. It is notable that all the image data sets are lexicographically rearranged into matrices. A. Local spatial-spectral dictionary pair construction The LR HSI X and LR MSI Y L are spatially divided into S partially overlapped subimages in the same manner, to produce {x s R n P s = 1, 2,, S} and {ys L R n Q s = 1, 2,, S}, where n represents the number of pixels in a single spectral band of each subimage. Likewise, the HR MSI is also divided into S partially overlapped HR subimages {y s R nω Q s = 1, 2,, S}.
The fusion of a LR HSI subimage x s and its spatially corresponding HR MSI subimage y s is treated each time. A specific local dictionary pair is constructed. The atoms of local LR dictionary D L and HR dictionary D H are selected from the LR and HR MSI subimages respectively, that is, D L = [d L s,1, d L s,2,, d L s,q] = [y L s,1, y L s,2,, y L s,q] D H = [d H s,1, d H s,2,, d H s,q] = [y s,1, y s,2,, y s,q ]. Different from the commonly used global spectral dictionary or spatial dictionary, local spatial-spectral dictionary with limited number of atoms (equal to the number of spectral bands in HR MSI Y) is produced, which effectively incorporates both spatial and spectral information and features and is with a small scale. B. CR coefficient estimation For a given subimage notation s, the LR HSI patch x s,i can be collaboratively represented by D L x s,i = D L ψ s,i + ε (1) where s = 1, 2,..., S, i = 1, 2,..., P and ψ s,i is the coefficient vector, ε is the model error. The LR subimage x s can be denoted by x s = D L Ψ s + ξ (2) where Ψ s = [ψ s,1, ψ s,2,..., ψ s,p ]. ξ is the model error. Because of the limited number of atoms in dictionary pair, collaborative representation is preferable rather than the commonly employed sparse representation. The coefficient vector ψ s,i can be then estimated as following { } ψ s,i = arg min D L ψ s,i x s,i 2 2 + η Γψ s,i 2 2 (3) ψ s,i where Γ is a biasing Tikhonov matrix specific to each atom of dictionary D L and LR HSI patch x s,i, and η is a global regularization parameter which balances the minimization between the residual and regularization terms. A diagonal Γ in the following form is adopted in this paper: Γ = d L s,1 x s,i 2... d L s,q x s,i 2. (4) As a result, the coefficient vector ψ s,i can be estimated in a closed form ψ s,i = [(D L ) T D L + η 2 Γ T Γ] 1 (D L ) T x s,i. (5) C. HR HSI reconstruction The LR hyperspectral (HS) subimage x s and the HR multispectral (MS) subimage y s are fused to produce an estimation of the corresponding HR HS subimage denoted by z s R nω P. It is appropriate to assume that z s can be collaboratively represented by HR dictionary D H and shares the same representation coefficients as x s is collaboratively represented with LR dictionary D L. As a result, z s can be calculated according to z s = D H Ψs (6) where Ψ s = [ ψ s,1, ψ s,2,..., ψ s,p ]. The tiling up and summation of all S HR HS subimages finally gives the desired estimation of HR HSI Z. Specifically, for the overlapping areas among subimages, an average of all estimations at the corresponding pixel is computed as the final reconstructed value. The proposed HSI and MSI fusion algorithm using local spatial-spectral dictionary pair () is summarized in Algorithm 1. Algorithm 1 algorithm Input: LR HSI X with P channels HR MSI Y with Q channels Output: HR HSI Z with P channels 1: Spatial subdivision: X {x 1, x 2,, x S } with x s = {x s,1, x s,2,, x s,p } Y {y 1, y 2,, y S } with y s = {y s,1, y s,2,, y s,q } Y L {y1 L, y2 L,, ys L } with ys L = {ys,1, L ys,2, L, ys,q} L 2: for each s [1, 2,, S] do 3: Local dictionary pair construction: D L = [d L 1, d L 2,, d L Q] = [y L s,1, y L s,2,, y L s,q] D H = [d H 1, d H 2,..., d H Q ] = [y s,1, y s,2,, y s,q] 4: CR coefficient matrix estimation: Ψ s = [ ψ s,1, ψ s,2,..., ψ s,p ] with ψ s,i = [(D L ) T D L + η 2 Γ T Γ] 1 (D L ) T x s,i 5: HR HS subimage reconstruction: 6: end for z s = D H Ψs 7: HR HSI reconstruction: { z 1, z 2,, z S } Z III. EXPERIMENTAL RESULTS AND ANALYSIS A. Experiment setup The proposed HSI and MSI fusion approach is applied to three synthetic data sets generated from real airborne HSI data, which is also served as the reference: AVIRIS Indian Pines data set with (12 12)pixels 224bands, AVIRIS Cuprite data set with (24 24)pixels 22bands, and HYDICE
Washington DC data set with (24 24)pixels 191bands. To simulate the test LR HSI and HR MSI, the test data construction methods in [3] and [8] are adopted. The LR HSI is constructed by applying a 7 7 Gaussian spatial filter on each band of the reference and downsampling every 6 pixels in both horizontal and vertical directions. The HR MSI is generated by uniform spectral downsampling of the reference corresponding to Landsat TM bands 1-5 and 7, which cover the following spectral regions: -52, 52-6, 63-69, 76-9, 1-175 and 28-2nm. In addition, Gaussian noises are added to the simulated LR HSI and HR MSI, simulating the signal-to-noise ratios (SNRs) of the MS and HS sensors to be 2dB and 3dB respectively. In the simulated experiment, the performance of HSI and MSI fusion is evaluated by comparing the estimated HR HSI (fusion result) with the reference HSI. However, it is difficult to compare the performance of different fusion methods simply by visual assessment. Objective and quantitative analysis can contribute to a more comprehensive evaluation. Therefore, besides subjective visual assessment, two performance metrics, PSNR (peak signal to noise ratio) in decibel and SAE (Spectral Angle Error) in degree are employed for objective evaluation in this paper. B. Comparison experiment and analysis The proposed HSI and MSI fusion approach using local spatial-spectral dictionary pair () is compared with some state-of-the-art HSI and MSI fusion approaches, including based fusion approach proposed in [3] and Q. Wei s fusion approach proposed in [8]. The HSI and MSI fusion approach using a global spatial-spectral dictionary pair and sparse representation () is also employed for comparison. It is worth to note that both based fusion approach and Q. Wei s fusion approach utilize observation models of HS and MS sensor to accomplish fusion. While in both and approaches, only the HS sensor observation model is required. In the comparison experiment, for based fusion approach, the number of endmembers is set to be 4, the maximum iterations in the inner and outer loops are 3 and 5 respectively. In Q. Wei s fusion approach, to acquire the optimal fusion result, the dimension of subspace is set to be 14 for Indian Pines data set, 4 for Washington DC data set and 3 for Cuprite data set, and the regularization parameter is set to be 15, 1 and 3 for the three data sets respectively. In approach, to acquire the optimal fusion results, the HR subimage size is 6 6 for Indian Pines data set, 12 12 for both Washington DC and Cuprite data sets, and the overlapping area size is set to be 18, 12 and 84 pixels for the three data sets respectively. In the proposed approach, HR subimage in the size of 36 36 is adopted for Indian Pines data set, and 24 24 for both Washington DC and Cuprite data sets. And the corresponding LR subimage size would be 6 6 and 4 4 respectively. An overlapping area with 12 pixels is adopted for the three data sets. Regularization parameter is set empirically as following: η =.5 for Indian Pines data set, η =.1 for Washington DC data set and η =.19 for Cuprite data set. Fig. 1 shows PSNR values in function of wavelength. Q. Wei s approach, and approaches generate close enough PSNR curves over wavelengths, and clearly outperform based fusion approach for obvious higher PSNR values over the entire HSI spectrum. Difference images between the reference and actual fused images of different approaches at a specific wavelength for each data set is shown in Fig. 2. It can be observed that, differences between the reference and fused images for based fusion approach are much more obvious than those for the other three approaches. Visible but not salient differences can be observed among difference images generated by Q. Wei s approach, and approaches. Fig. 2. Difference images between the reference and fused images (top row: Indian Pines-1148nm, middle row: Washington DC-1219nm, bottom row: Cuprite-233nm, from left to right:, Q. Wei s, and ). Fig. 3 and 4 show the SAE histograms and SAE distribution maps for the three data sets respectively. It can be observed that based fusion approach seems to produce fused results with higher spectral distortion. Q. Wei s approach, and approaches exhibit comparable spectral reservation ability, while Q. Wei s approach leads to obviously less spectral distortion when applied to Washington DC data set. The globally averaged fusion performance measurements as well as calculation time for different approaches with the three data sets are listed in Table I. It can be observed that Q. Wei s approach, and the proposed approaches produce comparable fusion results with very slight differences, and outperform based fusion approach. While Q. Wei s approach is much more complicated than the approaches employing spatial-spectral dictionary pair ( and approaches), and generally consumes much longer time for computation. The proposed approach is especially less time-consuming, due to its localized dictionary strategy which generates a small scale dictionary pair.
6 5 5 5 4 4 3 4 3 2 15 3 2 1 4 6 8 12 14 16 18 22 24 2 5 15 15 4 6 8 12 14 16 18 22 24 Fig. 1. PSNR in function of wavelength for three data sets. 3 18 16 18 16 14 12 8 6 14 12 8 6 15 4 4 5 2 2.5 1 1.5 2 2.5 3 1 2 3 4 5 6 Fig. 3. SAE histograms for three data sets..5 1 1.5 2 2.5 3 3.5 4 4.5 5 TABLE I PERFORMANCE MEASUREMENTS FOR DIFFERENT FUSION APPROACHES Global averaged PSNR (db) Global averaged Calculation time (sec) Q. Wei Q. Wei Q. Wei Indian Pines 39.85 4.63 4.3 4.44.7693.7161.7475.7324 39.56 363 1.64.97 Washington DC 41.38 43.74 43.33 43.62 1.619.83.9746.9194 173.68 1779 4.97 12.68 Cuprite 43.46.12 44.63.2 1.6933 1.5617 1.666 1.576 173.91 1523 61.26 13.63 IV. CONCLUSIONS To incorporate both spatial and spectral information of source images, a novel LR HSI and HR MSI fusion approach using spatial-spectral dictionary pair () is proposed, in which the HR and LR dictionaries are constructed from the HR and LR version MSI respectively by dividing them into overlapping subimages. Specifically, a localized strategy is adopted for dictionary construction to efficiently reduce the dictionary scale, and hence to greatly reduce the calculation cost. Furthermore, collaborative representation based reconstruction is employed to acquire the desired HR HSI. Simulative experimental results illustrate that the newly proposed is capable of producing better or comparable fusion results compared with some state-of-the-art dictionary-based fusion approaches. Considering its much less computational complexity, the proposed approach is very competitive and promising in practical application. ACKNOWLEDGMENT This work is partially supported by the National Natural Science Foundation of China (Grant No. 6142167 and 6111188), National Aerospace Science Foundation of China (Grant No. 214ZD5347), Fundamental Research Funds for Central Universities (Grant No. 31216ZB29) and Aerospace Science and Technology Foundation of China. REFERENCES [1] C. -I. Chang, Hyperspectral imaging: techniques for spectral detection and classification, Kluwer Academic Publishers, 23. [2] N. Yokoya and A. Iwasaki, Hyperspectral and multispectral data fusion mission on hyperspectral imager suite (HISUI), Proc. IGARSS 213, pp. 486-489, Jul 213. [3] N. Yokoya, T. Yairi and A. Iwasaki, Coupled Nonnegative Matrix Factorization Unmixing for Hyperspectral and Multispectral Data Fusion, IEEE Trans. Geosci. Remote Sens., vol. 5, no. 2, pp. 528-537, Feb 212. [4] M. A. Bendoumi, M. He and S. Mei, Hyperspectral Image Resolution Enhancement Using High-Resolution Multispectral Image Based on Spectral Unmixing, IEEE Trans. Geosci. Remote Sens., vol. 52, no. 1, pp. 6574-6583, Oct 214.
Proceedings of APSIPA Annual Summit and Conference 217 Fig. 4. SAE distribution maps for three data sets (for each subfigure from left to right:,, Q. Wei s and ). [5] Z. H. Nezhad, A. Karami, R. Heylen and P. Scheunders, Fusion of Hyperspectral and Multispectral Images Using Spectral Unmixing and Sparse Coding, IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 9, no. 6, pp. 2377-2389, June 216. [6] C. Grohnfeldt, X. Zhu and R. Bamler, Jointly sparse fusion of hyperspectral and multispectral imagery, Proc. IGARSS 213, pp. 49-493, Jul 213. [7] M. Simes, J. Bioucas-Dias, L. B. Almeida and J. Chanussot, Hyperspectral imagery super-resolution by spatial joint nonlocal similarity, IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 7, no. 6, pp. 26712679, June 214. [8] Q. Wei, J. Bioucas-Dias, N. Dobigeon and J. Tourneret, Hyperspectral and multispectral image fusion based on a sparse representation, IEEE Trans. Geosci. Remote Sens., vol. 53, no. 7, pp. 3658-3668, Jul 215. [9] T. Zhao, Y. Zhang, X. Xue and M. He, Hyperspectral and multispectral image fusion using collaborative representation with local adaptive dictionary pair, Proc. IGARSS 216, pp. 7212-7215, Jul 216. 978-1-5386-1542-3@217 APSIPA APSIPA ASC 217