Integrated analysis of Light Detection and Ranging (LiDAR) and Hyperspectral Imagery (HSI) data

Transcription

1 Integrated analysis of Light Detection and Ranging (LiDAR) and Hyperspectral Imagery (HSI) data Angela M. Kim, Fred A. Kruse, and Richard C. Olsen Naval Postgraduate School, Remote Sensing Center and Physics Department, 833 Dyer Road, Monterey, CA, USA; ABSTRACT LiDAR and hyperspectral data provide rich and complementary information about the content of a scene. In this work, we examine methods of data fusion, with the goal of minimizing information loss due to point-cloud rasterization and spatial-spectral resampling. Two approaches are investigated and compared: 1) a point-cloud approach in which spectral indices such as Normalized Difference Vegetation Index (NDVI) and principal components of the hyperspectral image are calculated and appended as attributes to each LiDAR point falling within the spatial extent of the pixel, and a supervised machine learning approach is used to classify the resulting fused point cloud; and 2) a raster-based approach in which LiDAR raster products (DEMs, DSMs, slope, height, aspect, etc) are created and appended to the hyperspectral image cube, and traditional spectral classification techniques are then used to classify the fused image cube. The methods are compared in terms of classification accuracy. LiDAR data and associated orthophotos of the NPS campus collected during and hyperspectral Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) data collected during 2011 are used for this work. Keywords: LiDAR, Hyperspectral, HSI, Raster Fusion, Point Cloud Fusion 1. INTRODUCTION Identification and mapping of materials for applications such as target detection, trafficability, land use, environmental monitoring, and many others requires accurate characterization of both surface composition and morphology. Typical remote sensing approaches to these problems, however, usually concentrate on one specific data type (modality) and thus don t provide a complete description of the materials of interest. The research described here combines hyperspectral imaging (HSI) and Light Detection and Ranging (LiDAR) in an attempt to enhance remote sensing mapping capabilities. Imaging spectrometers or Hyperspectral Imaging (HSI) sensors are passive systems that rely on reflected sunlight to simultaneously collect spectral data in up to hundreds of image bands with a complete spectrum at each pixel. 1 This allows identification of materials based on their specific spectral signatures and production of maps showing the surface distribution of these materials. The solar spectral range, 0.4 to 2.5 micrometers, provides abundant information about many important Earth-surface materials such as minerals, man-made objects, and vegetation. 2, 3 HSI data provide useful spectral information for detecting targets and/or mapping surface ground cover; however, this is based solely on the surface spectral signature and does not incorporate surface morphology. LiDAR is an active remote sensing method in which pulses of laser energy are emitted, and the time-of-flight is recorded for echoes returning to the sensor. The time-of-flight information, along with the precise location and pointing direction of the sensor in space, is used to determine the exact location from which laser pulses have been reflected. The transmission of many laser pulses enables building up a point cloud dataset representing the 3D arrangement of objects in the scene. Most current LiDAR systems operate at a single wavelength, and the intensity of the reflected pulses is recorded for each of the returned points. This intensity value is typically not calibrated, however, and so provides only a relative measure of reflectivity. Further author information: A.M.K.: amkim@nps.edu, Telephone: Laser Radar Technology and Applications XXI, edited by Monte D. Turner, Gary W. Kamerman, Proc. of SPIE Vol. 9832, 98320W 2016 SPIE CCC code: X/16/$18 doi: / Proc. of SPIE Vol W-1

2 The strengths of hyperspectral imaging and LiDAR are complementary, and the successful fusion of the two data types has huge potential, but multiple challenges to successfully fusing HSI and LiDAR exist. HSI sensors produce 2D raster-based images that have relatively low spatial resolution as compared to the LiDAR data. LiDAR sensors produce 3D, irregularly gridded point clouds. There isn t a direct one-to-one correspondence between the two datasets. A successful fusion approach will maintain the spectral detail of the HSI data, and also the spatially detailed 3D information of the LiDAR point cloud. In the work presented in this paper, two fusion approaches are examined. In the first approach to fusion, data are fused into an integrated 3D point cloud, and processed in the 3D environment. To incorporate the 2D HSI information into the 3D LiDAR point cloud, relevant information from the HSI data are extracted and attributed to each LiDAR point as a vector. This is a one to many relationship, in which one HSI raster attribute is mapped to many irregularly spaced LiDAR points. Additional attributes from the LiDAR point cloud based on statistics of local neighborhoods of points are used to quantify the spatial arrangement of LiDAR points. A supervised machine learning decision tree classifier is then used to classify the point cloud attribute vectors. Working with irregularly gridded point cloud data is more challenging than traditional raster-based approaches, but this approach minimizes information loss due to rasterization of the point cloud data. The second fusion approach rasterizes relevant information from the LiDAR point cloud data (including height, intensity, and the statistical measures of local neighborhoods of points) and fuses these with the spectral data by combining them into a raster datacube. This is a many to one relationship, in which many LiDAR attributes are averaged and mapped to one raster pixel corresponding to the spatial resolution of the HSI data. The fused raster datacube is classifed using standard supervised classification approaches. The raster approach is more computationally feasible than the point cloud approach. Individual LiDAR and HSI analyses along with fusion results are presented in Section 5. An evaluation of the success of the fusion processes is accomplished by examining the classification accuracy the fused data products. A discussion of the relative success of working within the point-cloud domain as compared to the raster-domain is given in Section BACKGROUND Multiple studies have demonstrated the utility of combining hyperspectral imaging and LiDAR data, particularly in a raster-based fashion. An overview of previous work, as well as results of an IEEE GRSS data fusion competition for classification of an urban scene, is given in Debes et al., Previous work fusing LiDAR and spectral information in the 3D domain is not as common, but a series of papers by researchers at the Finnish Geodetic Institute introduce prototype terrestrial laser scanning systems for collecting hyperspectral LiDAR point clouds. In these prototype instruments, a supercontinuum white laser source is combined with a hyperspectral time-of-flight sensor to actively measure reflectance over wavelengths ranging from nm. Results show improved classification of various tree species using the fused data product over either the HSI or the LiDAR data individually. The authors also demonstrate extraction of spectral indices in 3D geometries. 5 7 Buckley 2013 concurrently collected terrestrial laser scanner data (Riegl LMS-Z420i) and terrestrial hyperspectral camera (HySpex SWIR-320 m) data with the goal of integrated geological outcrop mapping. 8 Products from the hyperspectral camera data are projected into 3D space for improved visual analysis. 3. STUDY AREA AND DATA The area chosen for this study covers a small portion of the city of Monterey, California, which includes the campus of the Naval Postgraduate School (NPS). This area was selected because of the diversity of materials and morphologies at the site and the availability of both hyperspectral and LiDAR data. Figure 1 shows the study area and a small subarea selected for ground validation. Proc. of SPIE Vol W-2

3 ''`.~, Figure 1: (Left) AVIRIS true color composite image of the study area. The red box outlines the small subarea shown in greater spatial detail at right. (Right) An aerial photograph with 0.15 m spatial resolution showing the small subarea used for ground validation and accuracy assessment. 3.1 Light Detection and Ranging (LiDAR) data LiDAR data for this study were collected in October and November of 2012 with an Optech Orion C-200 system flown onboard a Bell 206L helicopter from approximately 450 m above ground level, with a 1541 nm laser having a spot size of approximately 50 cm on the ground. The point density is on average approximately 60 points/m 2 across the study area. A scan angle field of view of 30 degrees, as well as multiple overlapping flightlines, gives a point cloud dataset with excellent canopy penetration and some coverage of the sides of vertical objects. 3.2 Hyperspectral Imagery (HSI) data HSI data for this study area were collected at 2.4 m spatial resolution using NASA s Airborne Visible/Infrared Imaging Spectrometer (AVIRIS), an airborne hyperspectral sensor with 224 bands at 10 nm spectral resolution over the range micrometers. 9 The AVIRIS data used for this investigation were acquired 30 September, A total of 80 visible and near infrared (VNIR) bands from micrometers and 51 shortwave infrared (SWIR) bands from micrometers were used in these analyses. The SWIR data between micrometers were not used. 4. APPROACH AND METHODS Two approaches are presented in this study. The point cloud processing approach is discussed first, followed by a description of the raster-based processing approach. Results of each approach are presented in Section Point-cloud fusion approach LiDAR data were processed to determine height above ground level using the LAStools utilities. 10 To facilitate processing the large dataset, the data were first tiled into 200 x 200 m square tiles, with an additional buffer of 2 m to avoid the introduction of tile edge artifacts. The derivation of height values was accomplished using the LAStools lasheight utility, which first uses the points classified as ground to determine a bare earth digital elevation model (DEM), and then calculates the height of each point above the derived DEM surface. In this case, we made use of the ground classification as delivered by the vendor. The height and intensity information associated with the LiDAR points was also used in the point cloud classification process. In addition to the inherent LiDAR attributes of height and intensity, additional attributes based on the statistical distribution of points were calculated. Proc. of SPIE Vol W-3

4 4.1.1 LiDAR point-cloud neighborhood statistics A 1 m radius cylindrical neighborhood is defined around each point in the LiDAR point cloud. Statistics are calculated to quantitatively describe the 3D distribution of the points in each neighborhood. Statistics include the standard deviation of heights and intensities, and features based on a principal components analysis of the LiDAR points. Examples of each of the statistics are given in Figure 2. Principal components of the x, y, z components of neighborhoods of points are used to define a best-fit plane, and the orientation and deviation of points from this plane are recorded for each neighborhood. The percentage of variance explained by each principal component is useful for describing the structure of the points; for example, points that describe linear features such as power lines, fences, or building edges can be described almost completely by a single principal component coordinate, and therefore most of the variance will be explained by a single principal component. In Figure 2 (bottom-left), the image of % Variance Explained - PC1 shows high (bright) values for linear features such as building edges. For planar features, two principal components are required; building roofs and ground surfaces will have high values of this statistic. For globular features such as trees or cars, all three principal components will be required to explain the variance in the data. This use of principal components analysis was presented by Brodu and Lague, The principal components analysis was expanded to include analysis of x, y, z and intensity values of neighborhoods of LiDAR points. LiDAR point cloud statistics were calculated on a per-tile basis. This led to some inconsistencies when viewed across the entire study area as the range of values within each tile varied significantly across multiple tiles. This did not cause problems in the point cloud fusion analysis approach; however, in the raster-based fusion analysis approach, these inconsistencies required the exclusion of some of the LiDAR statistics. % Variance Explained - PC1 Surface Normal Angle Intensity % Variance Explained - PC2 Deviation from best -fit plane Intensity Deviation ij 4e si v = i rt.tte I X i5 _ r',., J % Variance Explained - PC3 % Variance Explained - PC3 (PCA w/ xyz +intensity) -"Jim ' Height Deviation,,g /Ìy 't y.. Figure 2: Statistics of local neighborhoods of LiDAR points Fusion of hyperspectral data and LiDAR point cloud To incorporate information from the hyperspectral data into the point cloud, each LiDAR point was mapped to the nearest image pixel in the 2.4 m spatial resolution hyperspectral raster dataset. Principal components were calculated on the original 224-band AVIRIS hyperspectral image, and the first 7 principal components were added as attributes to each LiDAR point. Additionally, the Normalized Difference Vegetation Index (NDVI) was calculated using the red (638.2 nm) and near infrared (831.5 nm) spectral bands according to the following Proc. of SPIE Vol W-4

5 12, 13 formula: NDV I = NIR RED NIR + RED Classification of the fused point cloud data Ground truth data were created by manually classifying point cloud data from a small subset of the study area (Figure 1). Seven (7) classes were defined: road, sidewalk, grass, trees, buildings, cars and other (Figure 3, top-left). The other class includes isolated objects (such as picnic tables and light fixtures) and unknown points. The 3D classification process enabled the creation of complex training classes; e.g. trees contain canopy and trunks, but exclude the ground underneath. The manually classified points were then divided into training, validation, and test sets. The training set (comprised of a random 60% selection of the points) was used to train a supervised machine learning decision tree classifier, while the validation set (20% of the points) was used to prevent overfitting of the classifier. The remaining 20% of the data points were used as a test dataset. 4.2 Raster fusion approach The HSI data for the study site were analyzed using a standardized approach based on n-dimensional spectral attributes and spectral signatures. 14 HSI results were stacked with raster layers extracted from the LiDAR data, combined signatures were determined using n-dimensional scatterplots, and supervised classification was used to map the distribution of specific surface characteristics Hyperspectral data analysis AVIRIS VNIR and SWIR HSI radiance data were converted to reflectance utilizing the Fast Line-of-sight Atmospheric Analysis of Spectral Hypercubes (FLAASH) atmospheric model. 15 This is a MODTRAN-based model requiring only knowledge about the HSI acquisition parameters (sensor, date, time, altitude), site characteristics (latitude and longitude, elevation), and some basic assumptions (atmospheric model, visibility). FLAASH adjusts the wavelength calibration as required, calculates the atmospheric water vapor concentration from the radiance spectra, determines reflectance utilizing the measured water vapor and the atmospheric model, masks portions of the spectrum with low signal-to-noise ratios (SNR), and removes residual sensor artifacts all without any supporting ground or atmospheric measurements. Image-based endmembers and spectral unmixing were then used for independent analysis of the individual spectral ranges. The selected VNIR and SWIR spectral regions were separately linearly transformed using 14, 16 the Minimum Noise Fraction (MNF) transform, which conditions n-dimensional data in preparation for determination of endmembers and spectral unmixing. A subset of the MNF data were used to determine the spectrally extreme pixels in each dataset using repeated projections of n-dimensional scatterplots (where n is the number of spectral bands) onto a lower dimensional subspace and marking the extreme pixels calculated the Pixel Purity Index (PPI). 14 The PPI results were then analyzed using interactive n-dimensional scatterplots to determine vertices, which typically correspond to unique materials the endmembers. The endmember spectra were visually compared to spectral libraries to determine key spectral features and identify materials. These were then used in a partial unmixing algorithm, Mixture-Tuned-Matched Filtering (MTMF) to map the spatial occurrence and abundance of each endmember for each dataset. 14 It should be noted that this spectral attribute analysis approach is significantly different than the principal component analysis approach for HSI data used in the point cloud fusion. The direct analysis of spectral signatures and endmembers results in many diverse classes that may not necessarily correspond to traditional classification of specific objects such as sidewalks, roads, rooftops, etc. This may in part explain some of the differences between the raster versus point cloud results and the large disparities in accuracy compared to the hand-picked validation classes (Section 5) Raster layer stacking The raster datacube combines spectral information from the HSI data with textural and 3D information extracted from the LiDAR data. Table 1 lists an index value for each of the bands used in the analysis. Spectral endmember abundance images of the AVIRIS data as well as the NDVI raster provide 18 raster images defining surface material characterization. LiDAR height above ground, intensity, and statistical measures (as described in Section 4.1.1) were rasterized to the 2.4 m spatial resolution and identical spatial coverage of the AVIRIS imagery by averaging all of the LiDAR points occurring in each pixel, yielding 10 raster images defining contextual Proc. of SPIE Vol W-5

6 and surface texture information. A subset of 22 raster bands (out of the total of 28) were used in the fused HSI/LiDAR raster-based classification process. Five of the LiDAR raster products were not used in the final fused analyses due to inconsistencies introduced during the tiling process (see Section 4.1). The final PCA (PCA3a) LiDAR raster product was also excluded. Table 1: Raster layers derived from the AVIRIS HSI and LiDAR neighborhood statistics. The index value indicates the arrangement of the layers as ordered in the fused raster layer stack. Layers without an index value were not used in the final raster layer analysis. Index Source Layer 1 AVIRIS VNIR Classification Mean_nD_Classl: Green Artificial Turf`1, Green Tennis Courts (Red) 2 Mean_nD_Class4: Water, Dark Vegetation, Shadows (Blue) 3 Mean_nD_Class5: Asphalt Roads and Asphalt Roofs (Cyan) 4 Mean_nD_Class6: Selected Green Vegetation (Green) 5 Mean_nD_Classl 1: Brown Bare Soil, Asphalt Roads (Yellow2) 6 Mean_nD_Classl7: Bright White Synthetic Roof Material (Red2) 7 Mean_nD_Classl9: Dark Green Artificial Turf#2 (Green2) 8 Mean_nD_Class20: Artificial Turf #3, asphalt roads (Sienna) 9 Mean_nD_Class21: Red Tile Roofs, Red Soil, Red Synthetics (Red3) 10 AVIRIS NDVI Normalized Difference Vegetation Index 1 l AVIRIS SWIR Classification Mean_nD_Classl: Dolomite, Sand Traps, Roads, 2.32 µm (Red) 12 Mean_nD_Class2: Synth Material, Tennis Courts, Rooftops, 2.27µm (Green) 13 Mean_nD_Class3: 2.15, 2.30 pm, Synth Roofs #1 (Blue) 14 Mean_nD_Class6: 2.28, 2.31 pm Synth Roofs #2 (Magenta) 15 Mean_nD_Class8: flat, 2.21 µm, Soils, Sand, Clay (Sea Green) 16 Mean_nD_Class10: 2.06, 2.31, 2.35 p.m, Synth Artificial Turf (Coral) 17 Mean_nD_Classl 1: peak at 2.2 µm, Green (Cyan) 18 Mean_nD_Classl3: 2.01, 2.32 pm, peak at 2.2 µm Dry Veg, Soil (Sienna) 19 LIDAR Rasterized Mean Heights 20 Rasterized Standard Deviation of Heights 21 Rasterized Mean Intensity Rasterized Standard Deviation of Intensity (not used due to tiling artifacts) 22 Rasterized Angle (angular difference from vertical of surface normal of best -fit plane) Rasterized Standard Deviation from Best Fit Plane (not used due to tiling artifacts) Rasterized % Variance explained PCAI (not used due to tiling artifacts) Rasterized % Variance explained PCA2 (not used due to tiling artifacts) Rasterized % Variance explained PCA3 (not used due to tiling artifacts) Rasterized % Variance explained PCA3a (PCA xyz + intensity, not used) 5. RESULTS Results are presented for the point-cloud processing approach first, followed by results for the raster-based approach. Proc. of SPIE Vol W-6

7 5.1 Point cloud analysis results Attributes of LiDAR points (including height and intensity) were combined with attributes based on statistics of local neighborhoods of points, and features from the HSI data (including 7 principal components and NDVI). A supervised machine learning decision tree classifier was trained using the training and validation data, and the point cloud data were classified into 7 classes including road, sidewalk, grass, trees, buildings, cars, and an other containing small isolated objects and unknown points. To quantify the performance of the point cloud data fusion approach, the trained classifier was applied to the test data set (20% of manually classified points) with the LiDAR data alone, the HSI information alone, and then again using the fused data product. Figure 3 shows the true classes and each of the classification results using the test dataset. A measure of overall accuracy was calculated as the number of correctly classified points divided by the total number of ground truth points. The decision tree classified trained on the fused HSI/LIDAR point cloud yields an overall accuracy of 88.8%. This represents a 3.5% increase in overall accuracy over using the LiDAR data alone, and an increase of over 20% using the HSI data alone. The performance of the HSI data alone appears to be very poor, but this can be explained in part due to the large spatial resolution of the HSI data (2.4 m pixels) as compared to the spatial resolution of the ground truth classes (60 points/m 2 ); this is discussed further in Section 5.3. Point Cloud Decision Tree Classification Results for Test Dataset (20% of Manually Classified Points) Road Sidewalk Grass Trees Buildings Cars Other/ unknown True Classes LiDAR only % overall accuracy HSI only % overall accuracy Fused LIDAR & HSI % overall accuracy Figure 3: Results of applying the decision tree classifiers to the test dataset, and the corresponding overall accuracy measurements. Fusing the HSI and LiDAR data resulted in a 3.5% improvement in overall accuracy over using LiDAR data alone. The confusion matrix for the fused HSI and LiDAR point-cloud classification (Table 2) shows classification accuracy on a class-by-class basis. The result of applying the decision tree classifier to a larger portion of the study area is shown in Figure 4. Proc. of SPIE Vol W-7

8 Table 2: The confusion matrix for the fused HSI and LiDAR decision tree classifier as applied to the small test set; the overall accuracy (calculated as the number of correctly classified points divided by the total number of ground truth points) is 88.8%. Predicted Class Sidewalk Grass Road Cars Trees Building Other Sidewalk % of Ground Truth 11% 71% 17% 0% 0% 0% 0% Grass % 90% 10% 0% 0% 0% 0% Road Ñ 0% 13% 86% 0% 0% 0% 0% u d 2 ~ Cars Trees Building Other 0 0% 1 0% 74 1% % % 0% 0% 0% 100% 0% 0% % 0% 0% 0% 2% 98% 0% % 1% 5% 7% 15% 1% 71% 178 2% 83 1% 342 3% Number of Ground Truth Points Number of ,ir.i+''` _- Figure 4: Results of applying the decision tree classifier to a larger subset of the fused HSI and LiDAR point cloud study area. 5.2 Raster analysis results The AVIRIS data were split into two spectral subsets: VNIR ( micrometers) and SWIR ( micrometers). Results are presented for each of the HSI subsets first, followed by results for the fused HSI/LiDAR datacube. Unfortunately, the HSI data s large 2.4 m pixel results in the occurrence of multiple materials per pixel in both the VNIR and SWIR. Only the predominant endmember is shown in the materials maps in Figures 5 and 6. Additionally, the spectral diversity of the study area and the fact that there is significant spectral variability within classes makes it extremely difficult to directly relate the spectral classes to traditional ground truth or to those derived using the higher resolution LiDAR data. The HSI data stand alone to determine surface composition, but we later had to consolidate and simplify classes to produce comparable results. Proc. of SPIE Vol W-8

9 5.2.1 AVIRIS VNIR Eighty-six (86) reflectance corrected spectral bands of AVIRIS data covering the VNIR from micrometers were used for the analysis. These were reduced to 50 linearly combined bands using the MNF transform as described in Section 4.2. N-dimensional scatterplotting was used to select endmembers with unique VNIR spectral characteristics. Repeated endmembers with similar spectral classes were manually merged, 14 resulting in a total of 9 unique VNIR endmembers (Figure 5, Table 1). Pixel-based classification using MTMF (an MTMF MF/Infeasibility ratio thresholded at 0.008) 3 and subsequent 3 x 3 sieving and clumping produced the image map shown in Figure 5. Selected VNIR n -D Visualizer ROI Means lass 11.lass 6 Class Wavelength (Micrometers) Figure 5: (Left) AVIRIS VNIR endmember spectra extracted using the n-dimensional scatterplotting approach. (Bottom Right) MTMF classification result showing distribution of selected VNIR spectral endmembers. Refer to Table 1 for endmember descriptions. The white box shows the location of the small subarea used for validation of results, with the corresponding (Top Center) MTMF classification and (Top Right) true-color AVIRIS image AVIRIS SWIR Forty-eight (48) reflectance corrected spectral bands of AVIRIS data covering the SWIR from micrometers were used for the analysis. These were reduced to 35 linearly combined bands using the MNF transform as described in Section 4.2. N-dimensional scatterplotting was used to select endmembers with unique SWIR Proc. of SPIE Vol W-9

10 spectral characteristics. 14 Repeated endmembers with similar spectral classes were manually merged, resulting in a total of 8 unique SWIR endmembers (Figure 6, Table 1). Pixel-based classification using MTMF (an MTMF MF/Infeasibility ratio thresholded at 0.008) 3 and subsequent 3 x 3 sieving and clumping produced the image map shown in Figure 6. Selected SWIR n-d Visualizer ROI Means ii1rlr r 1 íii ií í I I l r1l ii i *; 0:.zy%.. ' -M' IIIIIIIIIIIII Wavelength (Micrometers) 1 Figure 6: (Left) AVIRIS SWIR endmember spectra extracted using the n-dimensional scatterplotting approach. (Bottom Right) MTMF classification result showing distribution of selected SWIR spectral endmembers. Refer to Table 1 for endmember descriptions. The white box shows the location of the small subarea used for validation of results, with the corresponding (Top Center) MTMF classification and (Top Right) true-color AVIRIS image Combined HSI/LiDAR raster analysis N-dimensional scatterplot analysis of the combined HSI/LiDAR (stacked) raster data cube produced a complex set of endmembers with diverse combinations of spectral and spatial characteristics. A total of 31 different classes were defined. Unfortunately, detailed analysis and ground validation of all of these classes is beyond the scope of this investigation. Many of the classes are small and attributable to unique man-made features such as buildings or artificial surfaces. The validity of some classes is unclear. For the purposes of this study, six spatially coherent classes, roughly corresponding to those defined during the LiDAR point cloud analysis were selected for further analysis and validation. The 22-dimensional signatures for these classes are shown in Figure 7. Their spatial distribution for the full study are shown in Figure 8. Proc. of SPIE Vol W-10

11 Trees Grass UDAR Mean Height Red Roof White Roof LIDAR Mean Height 15 - Road Sidewalk LIDAR Mean Intensity LIDAR UDAR Mean _41 Intensity fo 15 Mean. Intensity 15 LIDAR 10 NDVI 10 Mean SWIR Height 2.2pm 1 VNIR 5 Green. Vegetation A VNIR Asphalt VNIR Red Tile SWIR 2.2pm Peak NDVI Peak '1 SWIR Soils. I lalu Nuwlx:r Layer Nunbef layer Number Figure 7: The six 22-dimensional signatures for selected combined HSI/LiDAR classes similar to those defined during the fused point cloud analysis. (Left) Comparison of signatures for green grass versus green trees; (Center) comparison of signatures for red roofs versus white roofs; (Right) comparison of roads versus sidewalks. Unclassified Roads/Asphalt. Green Grass Buldings - Red Tile Roofs Green Trees INISiderralks Willi - White Synthetic Roofs Figure 8: (Bottom) Minimum Distance classification image map using 22-dimensional endmember signatures of the fused HSI LiDAR dataset. The white box shows the location of the small subarea used for validation of results, with the corresponding (Top Left) classification results and (Top Right) the true-color AVIRIS image. Proc. of SPIE Vol W-11

12 Note in Figure 7 that the combined use of the HSI and rasterized LiDAR data make it possible to separate classes that might otherwise be difficult using only one data modality. For example, the trees and grass, which have similar spectral signatures in the HSI data, are easily mapped as distinct classes when the spectral characteristics are combined with the LiDAR height and intensity information. Also, while rooftops are easily mapped using the LiDAR height data, the addition of HSI information makes it possible to clearly define different roof types. The HSI also contributes additional information towards separation of roads from sidewalks. The two main problems with the rasterized approach with respect to mapping typical ground classes, however, are 1) the HSI spectral mapping results are inherently different than typical ground truth based on human-defined classes, and 2) the reduced resolution precludes full spatial definition of key classes. A confusion matrix (Table 3) was calculated using the small test dataset (Figure 1) and the ground truth data as defined by manually classifying the point cloud data (Figure 9). To calculate a confusion matrix for the 2D raster classification result using 3D ground truth data, each point in the ground truth point cloud was mapped nearest raster pixel. The confusion matrix for the raster result is not directly comparable to the confusion matrix for the point cloud results because not all classes were used in the raster classification approach. The classes used for the raster classification approach include sidewalk, grass, road, trees, and two types of buildings roofs. The red and white roof classes of the raster classification process were combined into a single building class for purposes of calculating overall classification; in the manually classified ground truth data, all buildings were combined into a single class, regardless of roof material. Also, there were pixels that were unclassified in the raster classification results, whereas in the point cloud approach, all points were forced to be classified as one of the ground truth classes. The overall classification accuracy (number of correctly classified points/total number of ground truth points) was calculated using only the classes actually used in the raster-based classification approach (sidewalk, grass, road, trees, and building). The overall classification accuracy for the raster-based approach is poor (51.2%), as expected due to the low 2.4 m spatial resolution of the HSI raster data as compared to the high spatial resolution (60 points/m 2 ) of the manually classified point cloud ground truth. In the 2.4 m HSI pixels, objects such as cars and most sidewalks are represented by mixed spectral signatures. Likewise, within each 2.4 m pixel region, there may be multiple ground truth classes; however, each pixel is assigned an individual class, leading to a high degree of misclassification when measured against the ground truth. It should also be noted that the exclusion due to tiling artifacts of some of the statistical measures used during the point cloud analysis may explain some of the disparities between the raster and point cloud results. Table 3: Confusion matrix for the fused HSI and LiDAR raster-based classification as applied to the small test set. The overall accuracy of the fused raster classification is 51.2%. The ground truth classes that were not used in the raster classification process (cars and other) were excluded from the confusion matrix. Likewise, there was no unclassified class in the ground truth data. Predicted Class Sidewalk Grass Road Trees Building Unclassified Sidewalk %of Ground Truth 10% 12% 57% 19% 0% 2% Number of Ground Truth Points Grass % 21% 46% 28% 0% 1% y Road 2% 0% 87% 8% 0% 2% al Trees it 1% 5% 42% 53% 0% 0% Building % 3% 46% 8% 29% 8% Unclassified % 0% 0% 0% 0% 0% Number of Classified Points Proc. of SPIE Vol W-12

13 5.3 Comparison of fusion results Ground truth classes were manually defined based on the very high spatial resolution (60 points/m 2 ) point cloud data. In the 2.4 m spatial resolution AVIRIS data, some of these classes represent areas too small to be spectrally distinct (e.g., sidewalks and cars are only represented as spectrally mixed pixels). Additionally, the lower spatial resolution of the raster data results in the loss of edges and sharpness in the combined class definitions and mapping. The diverse nature of the HSI data and spectral mixing at 2.4 m spatial resolution further contributes to spatial coherency problems in the fused raster data analysis. These differences in spatial resolution 2.4-m AVIRIS pixels versus sub-meter point spacing for the LiDAR data make a direct comparison of the point-cloud approach versus the raster-based approach challenging. Results of each of the classification schemes as applied to the small test area, with the raster approach classes mapped to the most similar ground truth classes, are shown in Figure 9. The overall classification accuracy of the fused point cloud approach as applied to the test dataset is 88.8% (Table 2), while the accuracy of the fused raster based approach is only 51.2% (Table 3). These numbers are not directly comparable, however, due to the similar (but different) classes used for the two approaches. Aerial Orthophoto (0.15 m pixels) Fused Point Cloud Classification (60 pts /m2) s Manually Classified Point Cloud Ground Truth (60 pts /m2) plull Road Sidewalk Grass Trees Buildings Cars Other/ unknown AVIRIS True -color Image (2.4 m pixels) A. Fused Raster Classification (2.4 m pixels) Figure 9: (Left) Ground truth point cloud data; (Top Center) true color aerial photo; (Top Right) the fused HSI and LiDAR point-cloud classification result (60 points/m 2 ) with an overall classification accuracy of 88.8%; (Bottom Center) AVIRIS true color image; and (Bottom Right) the fused HSI and LiDAR raster classification result (2.4 m pixels) with an overall classification accuracy of 51.2%. Proc. of SPIE Vol W-13

14 6. DISCUSSION, CONCLUSIONS, AND FUTURE WORK This study illustrates the use of fused HSI and LiDAR data to map the distribution and character of selected Earth-surface materials. Individually, HSI data map surface composition and LiDAR data map surface morphology. Analysis of a point cloud fusion approach and a raster fusion approach illustrates that fusing LiDAR and HSI together provides improved characterization and better classification maps. Each of the fusion approaches has strengths and drawbacks. The main advantage to the point cloud processing approach is the ability to maintain the high-spatial resolution of the LiDAR point cloud data. In the point cloud approach, the information from each AVIRIS pixel is mapped to the much higher resolution of the LiDAR point cloud data. In a fashion similar to pan-sharpening, this approach maintains the spatial resolution of the LiDAR point cloud data, while also incorporating information from the HSI analysis. The principal advantage of the raster approach is its simplicity; however, significant losses in classification accuracy resulted from the reduced spatial resolution as compared to the point cloud method. The main challenge in working with the LiDAR point cloud data is the computational complexity of working with irregularly gridded data. The necessity of the tiled processing approach introduced artifacts due to different ranges of values in each tile; while these artifacts did not have a significant effect on the point cloud processing approach, they did cause problems with the raster approach that prohibited use of most of the LiDAR neighborhood statistics. A direct quantitative comparison of the raster fusion approach and the point cloud fusion approach is difficult due to the disparity in spatial resolution of the datasets, but some qualitative observations can be made: Adding the LiDAR information into the raster-based HSI process helped distinguish materials which were spectrally similar, such as trees and grass. It also allowed distinguishing between morphologically similar roofs with different spectral characteristics (such as red and white roofs). For the point cloud approach, adding HSI information into the LiDAR point cloud resulted in an improvement in overall classification accuracy of approximately 3.5% as compared to using LiDAR data only, even though the 2.4 m pixel spatial resolution of the HSI data is very low compared to the LiDAR data (60 pts/m 2 ). The main challenge in combining HSI and LiDAR data in this study was the large disparity in spatial resolution. It is hypothesized that the improvement in classification accuracies for datasets with more similar spatial resolutions would be more obvious than the results shown here. While the raster based approach appears to perform very poorly in this work, this is explained by the differences in spatial resolution between the HSI data and the ground truth data that was derived from the LiDAR point cloud data. The raster approach is a useful method for combining HSI and LiDAR data in a way that enables the use of standard spectral processing tools (including commercial software packages), and derives some benefit from fusing the separate modalities. The ability of the point cloud approach to maintain the spatial resolution of the LiDAR data, while also incorporating the spectral information from lower spatial resolution imagery, has huge potential. More work is needed to make the process more computationally efficient and automated. A large portion of this work was accomplished using code written by the authors. The point cloud and raster based fusion approaches introduced in this work demonstrate the benefit of combining HSI and LiDAR datasets, and also highlight some of the challenges of working with these very different collection modalities. We see huge potential in recently developed spectral LiDAR datasets (such as the Teledyne Optech Titan system 17 ) and the prototype hyperspectral LiDAR systems being developed at the Finnish Geodetic Institute 5 7 as these data eliminate the need to perform fusion! Proc. of SPIE Vol W-14

15 REFERENCES [1] A. F. H. Goetz, G. Vane, J. E. Solomon, and B. N. Rock, Imaging spectrometry for Earth remote sensing, Science 228, pp , [2] R. N. Clark, G. A. Swayze, K. E. Livo, T. Hoefen, R. F. Kokaly, S. J. Sutley, J. B. Dalton, R. R. McDougal, and C. A. Gent, Imaging spectroscopy: Earth and planetary remote sensing with the USGS Tetracorder and expert systems, Journal of Geophysical Research 108(5), pp. 1 44, [3] F. A. Kruse, Integrated visible and near infrared, shortwave infrared, and longwave infrared (VNIR-SWIR- LWIR), full-range hyperspectral data analysis for geologic mapping, Journal of Applied Remote Sensing 9, [4] C. Debes, A. Merentitis, R. Heremans, J. Hahn, N. Frangiadakis, T. van Karsteren, W. Liao, R. Bellens, A. Pizurica, S. Gautama, W. Philips, S. Prasad, Q. Du, and F. Pacifici, Hyperspectral and LiDAR data fusion: Outcome of the 2013 data fusion contest, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 7(6), pp , [5] T. Hakala, J. Suomalainen, S. Kaasalainen, and Y. Chen, Full waveform hyperspectral LiDAR for terrestrial laser scanning, Optics Express 20(7), pp , [6] J. Suomalainen, T. Hakala, H. Kaartinen, E. Raikkonen, and S. Kaasalainen, Demonstration of a virtual active hyperspectral LiDAR in automated point cloud classification, ISPRS Journal of Photogrammetry and Remote Sensing 66, pp , [7] E. Puttonen, J. Suomalainen, S. Kaasalainen, and Y. Chen, Tree species classification from fused active hyperspectral reflectance and LiDAR measurements, Forest Ecology and Management 260, pp , [8] S. Buckle, T. H. Kurz, J. A. Howell, and D. Schneider, Terrestrial LiDAR and hyperspectral data fusion products for geological outcrop analysis, Computers and Geosciences 2013, pp , 54. [9] R. Green, AVIRIS and related 21st century imaging spectrometers for Earth and space science, pp Chapman and Hall/CRC Press, Boca Raton, FL, [10] M. Isenburg, LAStools - efficient tools for LiDAR processing. [11] N. Brodu and D. Lague, 3D terrestrial lidar data classification of complex natural scenes using a multi-scale dimensionality criterion: Applications in geomorphology, ISPRS Journal of Photogrammetry and Remote Sensing 68, pp , [12] J. W. Rouse, R. H. Haas, J. A. Schell, and D. W. Deering, Monitoring vegetation systems in the great plains with ERTS, Third ERTS Symposium, NASA SP-351, pp , [13] C. J. Tucker, Red and photographic infrared linear combinations for monitoring vegetation, Remote Sensing of the Environment 8, pp , [14] J. W. Boardman and F. A. Kruse, Analysis of imaging spectrometer data using n-dimensional geometry and a mixture-tuned matched filtering (MTMF) approach, Transactions on Geoscience and Remote Sensing (TGARS), Special Issue on Spectral Unmixing of Remotely Sensed Data 49(11), pp , [15] M. W. Matthew, S. M. Adler-Golden, A. Berk, G. Felde, G. P. Anderson, D. Gorodetzky, S. Paswaters, and M. Shippert, Atmospheric correction of spectral imagery: Evaluation of the FLAASH algorithm with AVIRIS data, Proceedings of SPIE, Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery IX, Orlando, FL 5093, pp , [16] A. A. Green, M. Berman, P. Switzer, and M. D. Craig, A transformation for ordering multispectral data in terms of image quality with implications for noise removal, IEEE Transactions on Geoscience and Remote Sensing 26(1), pp , [17] Teledyne Optech Incorporated, Optech Titan multispectral lidar system specifications, Proc. of SPIE Vol W-15