HYPERSPECTRAL remote sensing imagery provides finer

Transcription

1 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 43, NO. 10, OCTOBER Fusion of Hyperspectral Data Using Segmented PCT for Color Representation and Classification Vassilis Tsagaris, Member, IEEE, Vassilis Anastassopoulos, Member, IEEE, and George A. Lampropoulos, Member, IEEE Abstract Fusion of hyperspectral data is proposed by means of partitioning the hyperspectral bands into subgroups, prior to principal components transformation (PCT). The first principal component of each subgroup is employed for image visualization. The proposed approach is general, with the number of bands in each subgroup being application dependent. Nevertheless, the paper focuses on partitions with three subgroups suitable for RGB representation. One of them employs matched-filtering based on the spectral characteristics of various materials and is very promising for classification purposes. The information content of the hyperspectral bands as well as the quality of the obtained RGB images are quantitatively assessed using measures such as the correlation coefficient, the entropy, and the maximum energy minimum correlation index. The classification performance of the proposed partitioning approaches is tested using the K-means algorithm. Index Terms Color representation, hyperspectral data fusion, image classification, principal components transformation (PCT). I. INTRODUCTION HYPERSPECTRAL remote sensing imagery provides finer resolution in the spectral domain than multispectral data. With the number of bands in the hundreds instead of the tens, a new challenge arises for conventional data analysis techniques. The huge increase in the dimensionality of the vector space requires more sophisticated methods in order to analyze the data. In this work, a novel fusion scheme for efficient representation of a hyperspectral dataset in an informative color image is introduced, employing partitioning of the hyperspectral bands. Additionally, it is experimentally proved that the proposed fusion scheme is efficient for classification purposes. A great variety of fusion methods have been proposed in the literature, due to the complexity of the problem, the different types of data involved, and the different aims of each application [1] [9]. Fusion approaches are usually applied on data that differ spatially, spectrally, or temporally [8]. In general, pixellevel fusion methods aiming at improving spatial resolution can Manuscript received April 9, 2004; revised February 15, The work of the first author was supported in part by the European Social Fund (ESF), in part by the Operational Program for Educational and Vocational Training II (EPEAEK II), and in part by the Program HRAKLEITOS of the Ministry of Education and Religious Affairs, Greece. V. Tsagaris is with the Electronics Laboratory, Department of Physics, University of Patras, Patras 26500, Greece ( tsagaris@upatras.gr). V. Anastassopoulos is with the Electronics Laboratory, Department of Physics, University of Patras, 26500, Greece ( vassilis@physics.upatras.gr). G. A. Lampropoulos is with A.U.G. Signals Ltd., Toronto, ON, M5H 4E8 Canada ( lampro@augsignals.com). Digital Object Identifier /TGRS be classified into linear methods [1], nonlinear methods [2], optimization techniques [3], neural networks [4], [5], and image pyramids [6]. A significant amount of research is performed on linear fusion methods for multispectral and hyperspectral data [7] [9]. Since hyperspectral bands differ spectrally, fusion methods are used to obtain enhanced representation for visualization or classification. The color displays that have been proposed for representation of scientific data [10], [11] are mostly based on the mathematics of the image without considering the human vision system and the way that information is being perceived by the visual expert or end-user. Fusion methods that incorporate transformation of color spaces in order to assess the final color image have been proposed in [7] and [8]. The fusion approach proposed in this work is based on partitioning the hyperspectral dataset into subgroups of bands. The principal components transformation (PCT) [also referred as pincipal components analysis (PCA)] is applied to each subgroup. The derived principal components are suitable for information representation. This approach has significant advantages over the case of conventional PCT applied directly to the entire hyperspectral cube, as far as the computational load is concerned [9]. However, the type of partitioning proposed in this work is placing emphasis on the spectral properties of certain subgroups of bands or the spectral characteristics of specific surface cover types. First, the information in different spectral regions can be conveyed to different final principal components, providing the end-user with the ability to discriminate and classify objects and regions. Second, the proposed approach is computationally less demanding since computational load decreases when the number of subgroups increases. Furthermore, the RGB representation, obtained using three of the principal components, is perceptually more expressive [12] since the components of the final color image are correlated and the energy distribution is uniform as in natural color images. Finally, the proposed partitioning approaches, and especially the last one, which is based on the spectral signature of materials or objects, are promising for classification and detection tasks. In order to fully exploit the statistical characteristics of the data, an analysis based on second-order statistics is carried out. Furthermore, the information content and its distribution among the bands are studied by means of information theory. This analysis takes into account the user s needs and leads to the most appropriate partitioning of the hyperspectral bands. The proposed fusion approach gives emphasis on partitions with three subgroups of hyperspectral bands that result in expressive RGB representations. The approach in [9] is the simplest and gives /$ IEEE

2 2366 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 43, NO. 10, OCTOBER 2005 the best results in terms of reducing the computational complexity. The proposed approach leads to color images with the highest possible contrast in the RGB components and, consequently, in improved representation and visual discrimination of certain end-members that are present in the scene. Thispaperisorganizedasfollows. InSectionII, acompletedata description is provided while the second-order statistical criteria employed for quantitive measures along with information theory tools for the analysis of the hyperspectral data are discussed. In Section III, the dimensionality reduction is addressed focusing on the properties of the segmented PCT. The proposed fusion approaches and the proposed partitioning methods for the hyperspectral bands are presented in Section IV. The experimental results along with the evaluation of the performance of the proposed methods in classification tasks can be found in Section V. Finally, conclusions are drawn in Section VI. II. HYPERSPECTRAL DATA ANALYSIS The hyperspectral dataset employed for analysis in this work was acquired by the AVIRIS instrument. The analysis of the data is an extensive assessment about the way the information is distributed among the electromagnetic spectrum, the importance of each spectral band and its correlation with the others. Accordingly, the correlation coefficient, the maximum energy minimum correlation (MEMC) index [12], the total entropy and the mutual information are employed. A. Data Description The Airborne Visible Infrared Imaging Spectrometer (AVIRIS) is one of the first airborne systems [13] operated by the National Aeronautics and Space Administration (NASA)/Jet Propulsion Laboratory (JPL). In the first dataset the area covered is of Cuprite Mining Distinct, NV. It has been extensively used for remote sensing purposes since it has several exposed minerals of interest, including alunite, buddingtonite, dickite, illite, kaolinite, and quartz. The second hyperspectral dataset has been provided by NASA/JPL as free data for graduate research (namely San Diego 4). The selected part of the scene depicts an urban area with forest vegetation and sea. All the data are radiometrically calibrated and expressed as radiance values. During the data collection, possibly one or two of the spectrometers were not working properly. The result is that certain bands have zero reflectance but they are used in this dataset in order to have a uniform distribution of the wavelengths over the electromagnetic spectrum. B. Second-Order Statistics The high spectral resolution leads to a significant degree of correlation between the bands, which imposes a high level of spectral redundancy. This redundancy can be determined by means of the correlation coefficient, defined as (1) where are two hyperspectral bands and their mean values, respectively. The values of the correlation coefficient satisfy the relation. In order to reveal the spectral redundancy that is present in the hyperspectral dataset, the correlation coefficient was computed for all pairs of spectral bands and the results are shown in Fig. 1 as a grayscale image. The white pixels correspond to high degree of correlation while the black ones to uncorrelated bands. This representation reveals block structures of highly correlated bands [9], a fact that is used advantageously in this work for bands partitioning. C. MEMC Index In multispectral data processing it is often necessary to show the most important directions for the data to be projected on. These directions can be obtained by means of the MEMC index [12]. Actually, MEMC determines the bands which posses the maximum amount of energy (contrast) and have the minimum correlation with the other bands MEMC (2) where is the standard deviation of band and is the correlation coefficient between band and the rest of the bands. A large value of MEMC implies that this band has a high degree of contrast (energy) and low correlation with the other bands. As it is shown in Fig. 1(b), the MEMC index for the AVIRIS data takes its higher value for the bands with wavelength near to 2500 nm. This fact implies that the corresponding hyperspectral bands have high contrast and a low degree of correlation with the rest of the bands. On the other hand, MEMC can be employed to assess the final RGB components that are obtained via the transformation of the hyperspectral bands. D. Information Content of the Hyperspectral Bands Information theory is usually employed to calculate the data compression ratio [14] or to describe the information content of natural images [15]. Each hyperspectral band can be treated as a discrete random variable distributed according to a specific probability density, say. The entropy or total information [16] is defined as For a hyperspectral band, entropy describes the total amount of information contained in this band. In Fig. 1(d) the total entropy of the entire hyperspectral dataset is depicted as a function of the wavelength. It should be mentioned that the conclusions drawn from the study of the entropy are similar with those concerning the MEMC index. The information content in the last bands of the hyperspectral dataset is greater than that in the rest of them. The common information shared by two hyperspectral bands, say and can be studied by means of mutual information that is defined as (3) (4)

3 TSAGARIS et al.: FUSION OF HYPERSPECTRAL DATA 2367 Fig. 1. Hyperspectral data analysis using (a) correlation coefficient, (b) MEMC index, (c) mutual information, and (d) entropy. where is the joint probability density of the two variables. Mutual information is always a positive quantity that vanishes only if. Therefore, it can be interpreted as a measure of the statistical dependence between the variables and, and provides an alternative way for identifying the spectral redundancy between the bands. The mutual information between all pairs of hyperspectral bands is represented in Fig. 1(c) as a grayscale image with white pixels corresponding to the highest values and black pixels corresponding to zero. Mutual information reveals, more apparently than the correlation coefficient, the block structure of bands that have a high degree of statistical dependence. Consequently, it can be used in conjunction with the correlation coefficient to define these blocks. It becomes obvious from Fig. 1(c) that mutual information decreases when the distance of the spectral bands increases. This fact indicates that partitioning should lead to subgroups containing neighboring bands so that the dimensionality of the information in each subgroup is reduced. III. DIMENSIONALITY REDUCTION OF HYPERSPECTRAL DATA In remote sensing, one of the most widely used techniques for dimensionality reduction is the principal components transformation. PCT is effective in compressing information in multivariate datasets. It gives a new orthogonal vector space where the largest amount of energy is concentrated in a few components. Accordingly, the PCT can be considered as being a fusion process since the information of the original multidimensional data is transformed mainly on the first few orthogonal components of the new space. The concentration of the available information on a single band results in maximization of the variance for the pixels and the features in this band. Simultaneously, information does not degenerate, since PCT is an information preserving transformation and consequently the source data can be restored. The main drawback of PCT is its computational burden when it is applied on the complete hyperspectral cube. Furthermore, the use of PCT in the entire dataset may not reveal information that appears only in a few pixels of certain bands. For example an area that will have the biggest reflectance percentage in a certain wavelength will not be represented appropriately in a PCT of the entire hyperspectral cube. This is due to the global statistical nature of the transformation. PCT is optimal in the sense that the first principal component will have the highest contrast and thus it can be displayed as a grayscale image with the larger percentage of visual information.

4 2368 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 43, NO. 10, OCTOBER 2005 Usually the first three principal components convey more than 95% of the total energy of the data. However, these components are not suitable to form an RGB image (first component as red, second as green and third as blue) since the energy is not uniformly distributed and the result will not be optimal for the human visual system. Thus, the first principal component (red) will exhibit a high degree of contrast, the second (green) will display only a limited amount of the available brightness value, whilst the third one (blue) will demonstrate an even smaller range. In addition, the three components displayed as R, G, and B are totally uncorrelated and this is an assumption that does not hold for natural images [15]. Partitioning of the hyperspectral bands prior to PCT has already been suggested in [9]. The basic concept in that work is the reduction of the computational complexity, which can be achieved when the data are highly correlated. Consequently, partitioning was based on the edges of the image representation of the correlation matrix. Simultaneously, this partitioning procedure is assessed in [9], mainly for its classification performance. For this purpose Bhattacharya distance is employed for the principal components obtained and the classification assessment is carried out using different cover types. In this work, partitioning into subgroups of bands for the hyperspectral data prior to PCT is proposed, giving emphasis on the spectral properties of certain subgroups of bands, or the spectral characteristics of specific surface cover types. Thus, the reduction of the required computational effort is not of highest priority as in [9], but is assessed in each of the proposed partitioning approaches. After performing PCT at each subgroup, the first principal component is selected for further use in image representation or classification tasks. If the number of subgroups equals three, the corresponding principal components can be used for RGB visualization. An important issue is that the correlation of these RGB components is similar to correlation between the color components of a natural RGB image. Four different partitioning approaches are presented in the next section. The first of them is general and demonstrates the usefulness of partitioning based on simple spectral characteristics of the bands, especially for RGB representation. The other three approaches are based on specific criteria such as partitioning with equal energy or partitioning based on the spectral characteristics of specific objects, minerals, or surface cover types. As far as the computational load in a dimensionality reduction procedure is concerned, it is worthy mentioning that the implementation of PCT consists of two major tasks: an eigenanalysis to generate the transformation matrix (eigenvectors matrix) and a pixel-by-pixel linear transformation. The first task requires a significant amount of computational work since it requires the evaluation of the covariance matrix of the dataset and the corresponding eigenvectors matrix, which constitutes the transformation matrix. The evaluation of the covariance matrix requires multiplications and additions per pixel for a hyperspectral dataset of bands. In order to find the eigenvalues and the corresponding eigenvectors of the covariance matrix a diagonalization process is required. The total operation count for the diagonalization and the computation of a few eigenvectors is about as indicated in [17]. The pixel-by-pixel transformation is also a time-consuming process TABLE I DESCRIPTION OF SUBGROUPS and for a hyperspectral dataset of bands it requires multiplications and additions per pixel. The overall computational complexity as a function of the number of bands is. The idea of partitioning the hyperspectral bands prior to transformation decreases the computational complexity of all tasks [9]. IV. PROPOSED PARTITIONING APPROACHES The proposed partitioning methods described in this section are based on the spectral characteristics of the bands, the way that the energy or information is distributed along the electromagnetic spectrum as well as on spectral signature characteristics of various objects, minerals or earth surface cover types. These methods can give very good results on RGB image representation, especially for inspection of regions by humans. Additionally, they give color images with improved classification characteristics since information concerning only specific patterns or cover types is brought together (fused) in the final representation. A. Hyperspectral Data Partitioning In the general form, the segmented PCT scheme is applied on the complete dataset by partitioning it into subgroups with the bands in each group possessing high degree of correlation [9]. This results in substantial computational reduction. The number of bands is denoted in subgroup, respectively, with being the total number of bands. The PCT is applied separately on each subgroup and the primary components are selected for further use. In this subsection the hyperspectral bands are partitioned into eight subgroups, according to the perception of primary colors of the human visual system, as well as according to the positions of the atmospheric windows in the infrared region of the electromagnetic spectrum [13]. Specifically, the first three subgroups correspond to the blue, green, and red regions of the visible part of the electromagnetic spectrum. The exact range of each subgroup along with the corresponding number of bands can be found in Table I. In general, more than one of the principal components can be used as features for representation or classification. These features can be regrouped and the steps can be repeated until the required data reduction ratio is achieved. The energy compaction property of the PCT ensures that the first component of

5 TSAGARIS et al.: FUSION OF HYPERSPECTRAL DATA 2369 obtained RGB representation can give to each of the primary colors the electromagnetic information that possesses a special meaning for the humans, the earth reflectance or the atmospheric permeability. Fig. 2. Percentage of energy in the first principal component of each subgroup. TABLE II CORRELATION COEFFICIENT BETWEEN THE FIRST PRINCIPAL COMPONENTS OF EACH SUBGROUP FOR DATASET 1 B. Equal Subgroups The simplest approach in partitioning the bands is to divide the hyperspectral dataset into three groups of equal size (i.e., and ). This partitioning method will be referred to as PCT of equal subgroups (PCT-ES). PCT-ES covers equal regions of the electromagnetic spectrum of the hyperspectral dataset. Using this procedure the RGB image can be formed employingtheorthogonalband(eigenvector) fromeach subgroup that corresponds to the maximum eigenvalue of the group. This band willpossess themaximumpossible energy in the groupsince all the bands of the group are neighbors (i.e., their wavelengths are close) and thus are highly correlated. Moreover, the final RGB components will have a degree of correlation similar to that found in natural color images. In the PCT-ESmethod, the computational load is reduced by a factor of. Actually, this is the highest achievable reduction in the computational complexity. C. Maximum Energy Partitioning A different approach for partitioning the spectral bands is to select the size of each subgroup, so that the eigenvalues corresponding to the first principal component of the group become maximum. For this purpose, the largest eigenvalue corresponding to the first principal component of each subgroup is calculated. An iterative procedure is employed to evaluate the number of bands, in each subgroup so that each subgroup holds the largest percentage of energy. This energy can be expressed as a percentage if the eigenvalue of the first principal component corresponding to the larger eigenvalue of the entire subgroup is divided by the sum of all the eigenvalues obtained by conventional PCT for the entire hyperspectral dataset PE (5) where is the larger eigenvalue for the th subgroup. The first three subgroups have a small number of bands; hence the corresponding PE should be small compared to the rest. The results for all eight subgroups are depicted in Fig. 2. The first principal component of each subgroup can be used as a feature and the resulting eight features hold the 96.11% of the total energy. In the same time, the computational complexity, being, is reduced almost 96%. The principal components of each group can be used as RGB components to form a color representation of the hyperspectral dataset. In that case the final color image should have a high degree of correlation as in the natural color images. The correlation coefficient for the selected eight principal components can be found in Table II. A key advantage of the proposed method is that any triplet can be used to form an RGB representation and the final color image will have a degree of correlation similar to that found in natural color images. In this way, the The use of product instead of summation in (6) ensures that the eigenvalues will be more or less of the same order. Such partitioning implies that the largest amount of energy is conveyed by each subgroup in the first principal component and will be referred to as PCT of maximum eigenvalues (PCT-ME). In this way, the case of a dominating eigenvalue and consequently a dominating color in the RGB representation is avoided. The aim of this partitioning is to convey the highest amount of energy from the original hyperspectral data to the final color image while it distributes the information almost equally among the RGB components so that the final image possesses the properties of natural color images. D. Partitioning Based on Spectral Signature This third approach for segmenting the hyperspectral bands prior to PCT aims to reveal certain spectral characteristics of the hyperspectral dataset and will be referred to as PCT based on spectral signature (PCT-BSS). For each pixel, the spectral distribution of energy depends on what type of earth s surface the pixel contains. The spectral reflectance characteristic of each material of the earth surface is usually referred to as spectral signature of the specific material. The knowledge of the spectral signature for each material is of paramount importance. The proposed partitioning of the hyperspectral bands aims to obtain (6)

6 2370 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 43, NO. 10, OCTOBER 2005 Fig. 3. Spectral signatures of alunite (red line), buddingtonite (green line), and kaolinite (blue line). a final RGB image in which the materials will be distinguishable if their signatures differ. Part of the motivation for high spectral resolution sensors such as AVIRIS is that the bands are narrow enough to coincide with absorption features of individual molecules. As far as the region of exposed material on the surface is comparable to the size of a pixel, such absorption features can provide the means for labeling pixels. The area covered by the available hyperspectral dataset is of Cuprite mining where several exposed minerals of interest, including alunite, buddingtonite, and kaolinite can be found [18], [19]. In this work, the spectral libraries for the materials of interest were obtained from the U.S. Geological Survey ( In order to enhance the visual representation of a known end member a matched-filtering technique is employed. For this purpose, the spectral signatures of alunite, buddingtonite and kaolinite are plotted in Fig. 3. In the proposed partitioning procedure, each of the three materials will be mainly represented in one of the final RGB components. Specifically, the spectral signature of each material is used as a matched-filter to process only the hyperspectral bands in which this specific spectral signature has the highest value among the three. Thereby, the spectral response of the specific material is maximized while the response of the composite unknown background is suppressed [20]. Thus, three subgroups of bands are formed with maximized energy for a certain material in each subgroup. The same partitioning approach has been used for classification purposes. Accordingly, a different AVIRIS dataset from San Diego was used, since it contains calm sea, manmade regions as well as vegetation that presents the general spectral signature of chlorophyll. The single band (principal component in each partition) separability between the aforementioned three classes is measured by means of the Bhattacharya distance in the way it was applied in [9]. Simultaneously, classification is carried out for the same classes using the partitioning procedure described in [9] as well. The experimental results in the next section support the conclusion that the selection of the PCT-BSS partitioning approach is outperforming for the specific classification problem. V. EXPERIMENTS AND RESULTS The methodology of partitioning the hyperspectral bands was explained thoroughly in the previous section in its general form. An example of the proposed partitioning with eight different subgroups was employed, and comparisons with the conventional PCT were carried out in terms of energy preserving and computational complexity. In this way, the dimensionality reduction achieved by the general partitioning method was established. In the subsequent paragraphs partitioning methods, primarily aiming to color representation, are implemented for the Cuprite dataset and the final RGB representations of the hyperspectral data are examined. The measures, provided in Section II, for the analysis of hyperspectral data, are used for objective evaluation of the results along with classification accuracy scores. Moreover, the dimensionality reduction provided by the partitioning approach is compared with the one obtained using the conventional PCT. A. Partitioning Using Equal Subgroups The simplest approach in segmenting the bands of the hyperspectral dataset is to derive three subgroups with equal number of bands. Such an approach reduces significantly the computational load and provides a method for color RGB representation of the hyperspectral data. In this work, the AVIRIS dataset is partitioned in three subgroups, and the first principal component of each subgroup is used as a component of the final RGB image. Thus, the electromagnetic spectrum covered by the AVIRIS instrument is divided into three equal subregions and each of them gives one component to the final RGB image. The first subregion ranges from nm, the second from nm, and the third from nm. In Fig. 4(a) a false-color composite representation of bands 183, 193, and 207 is shown, without any histogram stretching techniques that will distort the color balance of the scene. This combination of bands is used in ENVI Tutorials in order to enhance mineralogical differences. The RGB representation formed by the first principal component of each subgroup is presented in Fig. 4(b). The proposed method results in a color image with enhanced perceivable information without distorted colors. B. Partitioning for Maximum Energy The second method for partitioning of bands prior to PCT is the PCT-ME. The three subgroups of bands, derived from the Cuprite AVIRIS dataset, are chosen in order to provide the maximum product of the largest eigenvalues. The first subgroup covers the region from nm and consists of the first 80 bands. The next 55 bands comprise the second subgroup, which covers the region nm in the electromagnetic spectrum. The third subgroup contains the largest number of bands, namely 89, and ranges from nm. The first principal component of each subgroup is used as a component of the final RGB image. In the PCT-ME method 96.2% of the total energy is conveyed in the final color image by the first principal components of the subgroups. This percentage guarantees that the final color representation of the hyperspectral dataset is meaningful and in the same time suitable for the human visual system.

7 TSAGARIS et al.: FUSION OF HYPERSPECTRAL DATA 2371 Fig. 4. (a) False-color composite of the first AVIRIS dataset, (b) color representation using PCT-ES, (c) false-color composite of the second AVIRIS dataset, and (d) color representation using PCT-ES. C. Partitioning Based on Spectral Signature A partitioning method that aims to enhance the representation of certain materials, which are present in the area covered by the hyperspectral dataset, should take into consideration their spectral signature. The implementation of PCT-BSS method for the specific dataset intends on revealing alunite, kaolinite, and buddigtonite that are present in the area. The first subgroup ranges from nm and in this region of the electromagnetic spectrum the reflectance of alunite is dominant. The spectral signature of alunite is used as the transfer function of a matched-filter applied to the corresponding bands of the hyperspectral dataset. The first principal component of this subgroup is used as the blue component for the final color image. The spectral signature of kaolinite is dominating in the range from nm, as shown in Fig. 3, and the corresponding bands constitute the second subgroup after matched filtering with the spectral signature of this material. The first principal component of this subgroup is used as the green component at the final RGB image. Finally, the spectral signature of buddigtonite is used as a matched filter, in the range from nm, where its reflectance is dominating. The red component of the final RGB image is the first principal component of this subgroup. The final color image representation of the dataset is shown in Fig. 5. A comparison with the previous representation obtained by the other partitioning methods leads to the conclusion that there is a bigger variety of the primary colors (red, green, and blue) due to the matched-filtering procedure. The results of the method can be further improved if it is applied in a certain part of the image in order to maximize the response of specific materials that may be present in the area. D. Discussion on the Proposed Methods The three methods proposed in this work for RGB representation of hyperspectral data are not in competition, and data can be applied in a complementary way when different goals are to be achieved. It is noteworthy that all three methods provide RGB images having correlation properties similar to those of natural color images [12], [15]. This is because the largest principal component used from the different subgroups to form the final

8 2372 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 43, NO. 10, OCTOBER 2005 Fig. 5. Color representation using (a) PCT-ME first dataset, (b) PCT-BSS (alunite-blue, kaolinite-green and buddigtonite-red) first dataset, (c) PCT-ME second dataset, and (d) PCT-BSS (urban-red, vegetation-green, and sea-blue). RGB image are highly correlated. Furthermore, the PCT-ES approach provides a quite natural representation since each RGB component corresponds to equal in length hyperspectral range. The PCT-ME method gives perceptually satisfactory RGB representation due to the equal energy delivered to each of the final bands. The PCT-BSS procedure is very promising for classification purposes, since it incorporates the special spectral characteristics of the materials to be detected. In order to evaluate the performance of the proposed methods in RGB representation, one should examine the properties of the components resulted by partitioning prior to PCT and those obtained by the conventional PCT. For this purpose, the eigenvalues obtained from each subgroup are sorted in descending order, their sum is evaluated and compared with those derived from the conventional PCT. The results are plotted in Fig. 6 for the case of PCT-ME and reveal that while the cumulative eigenvalues are slightly lower compared with those from the conventional PCT for the first five components, they become very close after that point. Specifically, when the first dozen of principal components are employed the difference between PCT-ME and the conventional PCT is only 0.1%. However, Fig. 6. Comparison between cumulative eigenvalues. (Solid line) PCT-ME. (Dashed line) Conventional PCT of the entire dataset. since we are mainly interested in RGB representation, for the first three principal components this difference is 2%, with the

9 TSAGARIS et al.: FUSION OF HYPERSPECTRAL DATA 2373 TABLE III PERFORMANCE EVALUATION BASED ON ENTROPY TABLE V CORRELATION COEFFICIENT BETWEEN THE RGB COMPONENTS OF THE COLOR IMAGES OBTAINED FROM THE PROPOSED PARTITIONING APPROACHES TABLE IV PERFORMANCE EVALUATION BASED ON MEMC PCT-ME method to provide equal in energy RGB bands and perceptually excellent color image. An objective evaluation of the proposed method for color representation could be based on the measures described in Section II. For this reason the entropy and the MEMC index are calculated for each component of the final RGB image and the results for all three proposed partitioning approaches can be found in Tables III and IV, respectively. In addition the correlation coefficient between the RGB components for the three color images formed with the proposed partitioning methods is shown in Table V. The second-order statistics and the information measure calculated for the final image reveal the properties of the proposed partitioning methods. The entropy of the RGB components is significantly increased (to values greater than 4.1) compared to the values of entropy for the source hyperspectral bands, where their mean value is In this way, the information content of each component is richer and thus a meaningful representation can be obtained. As expected, the entropy is more uniformly distributed in the case of the final RGB image obtained by the PCT-ME method. Moreover, the large values of the MEMC index for the final RGB components indicate that the contrast in each component is increased and the correlation is decreased but not vanished. Obviously, the largest values for MEMC are obtained for PCT-BSS method. The way the matched-filtering approach was applied to different spectral bands results in reduced degree of correlation and thus a high MEMC value. The degree of correlation between the RGB components is similar to the correlation found in natural color images [15]. This is an advantageous property of the proposed partitioning methods because the final color image has natural colors and hues. In this way the human visual system can perceive a greater amount of information without the existence of any strange colors often imposed by histogram stretching. The values of the correlation coefficient are smaller in the case of PCT-BSS method as already explained. E. Comparison of Classification Performance In the context of evaluating the performance of the proposed partitioning methods a classification step is employed in order to highlight the applicability of the overall approach in classification problems. The classification procedure is implemented in the ENVI software and unsupervised classification is used. The well-known K-means algorithm with five different classes and five iterations is applied on the first hyperspectral dataset. The first classification approach uses all the 224 hyperspectral bands, and provides 98.2% classification accuracy. The same classification algorithm is applied to the color images formed by PCT-ES, PCT-ME, and PCT-BSS. The overall classification accuracies are 96.9%, 97.3%, and 95.9% for PCT-ES, PCT-ME, and PCT-BSS, respectively. The results reveal that satisfactory classification performance is achieved in all cases while at the same time the computational time and the number of features are reduced. The best classification score is achieved in the case of PCT-ME and this establishes the fact that maximum energy is conveyed by the source hyperspectral bands in the color representation. In the case of PCT-BSS the proposed partitioning aims to demonstrate certain materials so the comparison of this classification map is not straightforward for the case of five classes. In the case of the second hyperspectral dataset three different classes are selected visually, with the same number of pixels in

10 2374 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 43, NO. 10, OCTOBER 2005 TABLE VI CLASSIFICATION ACCURACIES FOR THE SECOND DATASET each class. These classes are urban area, forest or vegetation and calm sea. For this case, a feature selection step similar to that described in [9] is employed. Specifically, the Bhattacharya distance (BD) is calculated and only the eigenbands with BD in each subgroup, are used for classification. In this way, the computational complexity of the classification step is significantly reduced. For example, in the case of the second hyperspectral dataset, 4, 3, and 4 eigenbands are used, for the partitioning approaches PCT-ES, PCT-ME, and PCT-BSS, respectively. The classification results for each partitioning approach and each class can be found in Table VI. The simplest partitioning approach, i.e., PCT-ES, already proposed in [9], provides satisfactory classification results for all classes. The other two partitioning methods can further improve the classification performance for the urban and forest classes. In all cases the computational complexity of the classification step is reduced compared to that of applying the K-means algorithm in the entire hyperspectral cube. It must be noted that for the second hyperspectral dataset the PCT-BSS approach significantly improves the classification performance of the urban and the forest class, thus justifying the use of the matched filtering technique. VI. CONCLUSION Partitioning of the hyperspectral data cube is proposed for efficient fusion and perceptually meaningful RGB images. The obtained color images present desirable perceptual characteristics since their statistical properties are similar to those of natural color images. Dimensionality reduction is achieved through different partitioning methods after an analysis of the source hyperspectral bands. These methods provide a practical and efficient scheme for dealing with hyperspectral data for both improved representation and classification purposes. The type of the partitioning depends on the application and can be based on the number of atmospheric windows, the distribution of spectral energy or the reflectance signature of various materials. Therefore, several final images can be obtained giving different type of information for the area under inspection. Three partitioning methods were proposed for RGB representation, with perceptually significant characteristics. Specifically, the first method namely PCT-ES results in physical representation of the scene. The PCT-ME method gives images that are perceptually more comprehensive. The third method fits better for recognition of specific materials since matched filtering with the material s spectral response is used. The generality of the proposed approach is evident from the fact that each of the above three schemes proposed for RGB representation can be applied to any number of partitions. The performance evaluation based on both statistical and information measures established the method as promising. REFERENCES [1] T. Achalakul and S. Taylor, Real-time multispectral image fusion, Concurr. Computat.: Pract. Exper., vol. 13, no. 12, pp , Sep [2] S. Mukhopadhyay and B. Chanda, Fusion of 2-D grayscale images using multiscale morphology, Pattern Recognit., vol. 34, no. 10, pp , Oct [3] A. Solberg, T. Taxt, and A. Jain, A Markov random field model for classification of multisource satellite imagery, IEEE Trans. Geosci. Remote Sens., vol. 34, no. 1, pp , Jan [4] Z. Zhang, S. Suan, and F. Zheng, Image fusion based on median filters and SOFM neural networks: A three-step scheme, Signal Process., vol. 81, no. 6, pp , Jun [5] Y. V. Shkvarko, Y. S. Shmaily, R. Jaime-Rivas, and M. Torres-Cisneros, System fusion in passive sensing using a modified Hopfield network, J. Franklin Inst., vol. 338, pp , [6] Z. Liu, K. Tsukada, K. Hanasaki, Y. K. Ho, and Y. P. Dai, Image fusion by using steerable pyramid, Pattern Recognit. Lett., vol. 22, no. 9, pp , Jul [7] J. Tyo, A. Konsolakis, D. Diersen, and R. C. Olsen, Principal-components-based display strategy for spectral imagery, IEEE Trans. Geosci. Remote Sens., vol. 41, no. 3, pp , Mar [8] C. Pohl and J. L. van Genderen, Multisensor image fusion in remote sensing: Concepts, methods, and applications, Int. J. Remote Sens., vol. 19, no. 5, pp , [9] X. Jia and J. A. Richards, Segmented principal components transformation for efficient hyperspectral remote sensing image display and classification, IEEE Trans. Geosci. Remote Sens., vol. 37, no. 1, pp , Jan [10] P. K. Robertson, Visualizing color gamuts: A user interface for the effective use of perceptual color spaces in data displays, IEEE Comput. Graph. Appl., vol. 8, pp , Sep [11] P. K. Robertson and J. F. O Callagham, The generation of color sequences for univariate and bivariate mapping, IEEE Comput. Graph. Appl., vol. 6, pp , Feb [12] V. Tsagaris and V. Anastassopoulos, Multispectral image fusion for improved RGB representation based on perceptual attributes, Int. J. Remote Sens., submitted for publication. [13] G. Vane, R. Green, T. Chrien, H. Enmark, E. Hansen, and W. Porter, The airborne visible infrared imaging spectrometer, Remote Sens. Environ., vol. 44, pp , [14] J. C. A. Van der Lubbe, Information Theory. Cambridge, U.K.: Cambridge Univ. Press, [15] A. Turiel, N. Parga, D. Ruderman, and T. Cronin, Multiscaling and information content of natural color images, J. Phys. Rev. E, vol. 62, no. 1, pp , [16] T. Cover and J. Thomas, Elements of Information Theory: Wiley, [17] W. Press, S. Teukolsky, W. Vetterling, and B. Flannery, Numerical Recipes in C. Cambridge, U.K.: Cambridge Univ. Press, [18] D. Landgrebe, Signal Theory Methods in Multispectral Remote Sensing. New York: Wiley, [19] J. A. Richards and X. Jia, Remote Sensing Digital Image Analysis. Berlin, Germany: Springer-Verlag, [20] J. C. Harsanyi and C. I. Chang, Hyperspectral image classification and dimensionality reduction: An orthogonal subspace projection approach, IEEE Trans. Geosci. Remote Sens., vol. 32, no. 4, pp , Jul Vassilis Tsagaris (S 98 M 04) received the B.Sc. degree in physics and the M.Sc. degree in electronics from the University of Patras, Patras, Greece, in 1998 and 2000, respectively. He is currently pursuing the Ph.D. degree in image processing for remote sensing at the University of Patras. His main research interests include image processing, pattern recognition, remote sensing, and data fusion.

11 TSAGARIS et al.: FUSION OF HYPERSPECTRAL DATA 2375 Vassilis Anastassopoulos (M 90) received the B.Sc. degree in physics and the Ph.D. degree in electronics from the University of Patras, Patras, Greece, in 1980 and 1986, respectively. His research interests are within the scope of digital signal processing, image processing, radar signal processing, data fusion, pattern recognition, and classification. He is currently an Associate Professor at the University of Patras. He has worked for two years in Canadian academic and private research institutions and cooperates with a number of scientific-research groups worldwide. Additionally, he has been involved in various R&D programs and scientific mobility activities. He is a reviewer for a number of scientific journals and on the Editorial Board of the journal Pattern Recognition. Dr. Anastassopoulos has been a member of the Scientific, Technical, or Organizing committees of various international conferences. He has served as Associate Editor for the IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS II. George A. Lampropoulos (S 80 M 81) received the B.Sc. degree from the University of Patras, Patras, Greece, the M.Sc. and Ph.D. degrees from Queen s University, Kingston, ON, Canada, in 1979, 1982, and 1985, respectively, all in electrical engineering. His specialization is in digital signal processing, mainly in electro-optical radar. He was with the Electrical Engineering Departments of the Royal Military College, Kingston ( ), Laval University ( ), and the University of Toronto (1999). He gained considerable industrial experience with SPAR Aerospace ( ) and A.U.G. Signals Ltd. (1992-today, as President and CEO). He has supervised more than 70 industrial research projects in the area of signal processing, including image registration, segmentation, detection, classification, noise reduction and restoration, identification, recognition, and fusion. He has been a leading investigator of mineral exploration programs for government and industry using hyperspectral data. He also led projects related to web-based distributed processing with the support of the NRCan GeoInnovations Program. He has published more than 200 articles in journals, conferences, and books, as well as technical reports in the area of signal processing. He has been an invited keynote speaker in major technical events (e.g., 2002 IEE annual meeting), organized and was General Chair in major multisociety conferences (i.e., International Conference on Applications of Photonic Technology or Photonics North, 1994, 1996, 1998) and Vice Chair (2000, 2002, 2003). He is member of the board in several organizations and has been included in several Who s Who including the 1989/1990 Marquis Who s Who in the World.