ABSTRACT. Keywords: Thermal imaging, MWIR, photogrammetry, computer vision, point cloud

Transcription

1 Evaluation of terrestrial photogrammetric point clouds derived from thermal imagery Jeremy P. Metcalf, Richard C. Olsen, Naval Postgraduate School, 833 Dyer Rd, Monterey, CA, USA ABSTRACT Computer vision and photogrammetric techniques have been widely applied to digital imagery producing high density 3D point clouds. Using thermal imagery as input, the same techniques can be applied to infrared data to produce point clouds in 3D space, providing surface temperature information. The work presented here is an evaluation of the accuracy of 3D reconstruction of point clouds produced using thermal imagery. An urban scene was imaged over an area at the Naval Postgraduate School, Monterey, CA, viewing from above as with an airborne system. Terrestrial thermal and RGB imagery were collected from a rooftop overlooking the site using a FLIR SC8200 MWIR camera and a Canon T1i DSLR. In order to spatially align each dataset, ground control points were placed throughout the study area using Trimble R10 GNSS receivers operating in RTK mode. Each image dataset is processed to produce a dense point cloud for 3D evaluation. Keywords: Thermal imaging, MWIR, photogrammetry, computer vision, point cloud 1. INTRODUCTION Remote Sensing data present fundamental challenges in use for intelligence purposes. The data volume is large, almost defining the term big data. Autonomous processing is difficult, and analysis of imagery largely requires significant human interaction. The primary goal is generally change detection in either the short or long term (day-to-day up to decade-to-decade). Multiple modalities make the process of analysis even more challenging. In particular, extracting results from combinations of panchromatic data, spectral imaging data, synthetic aperture radar (SAR) data, and now light detection and ranging (LiDAR) data presents a problem with few obvious solutions. The primary issue with bringing multiple observations together, and in particular multiple modalities, is that these data are generally processed into two-dimensional (2D) representations, and then analyzed. This makes the merger of optical and SAR imagery particularly difficult, but really any set of images with varying perspective presents the same problem. The solution is simple enough: address the data in three-dimensions (3D). Technology to extract 3D point clouds from EO data, SAR data, and thermal imaging systems exists now, and can be used to resolve the problem of sensor fusion and change detection. In recent years, an increasingly wide range of software solutions combining photogrammetry and computer vision have emerged enabling rapid production of 3D content using imagery collected from both metric and non-metric cameras. This emergence is largely due to advancements in GPU parallel processing and the introduction of semi-global matching 1,2. Current commercial and open source software are capable of determining image depth at the level of individual pixels, producing dense point clouds. Given the availability of high resolution visible to NIR wavelength cameras, photogrammetric point cloud processing has almost exclusively performed on multispectral imagery. Because thermal cameras have much lower resolutions than modern visible cameras, the inclusion of thermal imagery in photogrammetry is often limited to fusion with higher resolution sources 3,4. The goal of this work is to generate point clouds solely from thermal imagery for purposes of sensor fusion in 3D. To accomplish this, a high resolution thermal image dataset was acquired with high overlap and then processed using modern photogrammetry software. A separate visible image dataset was coincidentally acquired and processed for comparison. The accuracy of the resulting thermal and visible point clouds was determined using terrestrial LiDAR scanning data as reference. Thermosense: Thermal Infrared Applications XXXVIII, edited by Joseph N. Zalameda, Paolo Bison, Proc. of SPIE Vol. 9861, SPIE CCC code: X/16/$18 doi: / Proc. of SPIE Vol

2 2. EXPERIMENT DESCRIPTION 2.1 Image Acquisition Photogrammetry software typically requires imagery collected in suitable lighting conditions and with some degree of overlap in order to perform an accurate bundle block adjustment. With this in mind, we approximated an oblique airborne imaging trajectory by marking 10 positions with a distance spacing of 6 m along the roof of a building that overlooks the campus quad area at Naval Postgraduate School in Monterey, CA. The marked positions help to ensure similar image overlap and repeatability between successive image acquisitions. A tripod mounted FLIR SC82000 MWIR thermal camera and a handheld Canon T1i DSLR were used to capture the campus quad area by collecting multiple adjacent images at each marked position. Figure 1 illustrates the thermal image collection pattern from a single marked position. Figure 1. Example of the thermal image collection pattern used to capture the campus quad area from each marked position. Black dots represent the camera centers while black lines show the projected image outlines. The FLIR SC8200 records light in the MWIR spectral range between 3 and 5 µm. The focal plane array is a cooled InSb detector with a pixel pitch of 18 µm and an image resolution of 1024 x At just over a megapixel, this thermal camera has an image resolution much higher than most other thermal cameras available commercially. This camera records 14-bit imagery from -20 C to 500 C at a frame rate of 132 Hz. The attached adjustable focus lens has a 50 mm focal length with a 20.9 by 20.9 field of view. 2.2 Global Navigation Satellite System (GNSS) Survey To ensure that all point clouds would be referenced to the same coordinate system and comparable, we use a GNSS system to collect ground control within the campus quad area. Ground control points (GCPs) weree surveyed using a pair of Trimble R10 GNSS receivers operating in real-time kinematic mode. The layout of GCPs was determined by analyzing the spatial extentt of acquired thermal imagery to ensure adequate coverage. Thirteen locations were identified between concrete slab junctions and then collected after an acceptable real-time precision was achieved. Horizontal and vertical precisions for each point were reported to be below 2 cm at the 95% confidence level. The GCPs were projected in the WGS84 UTM Zone 10N coordinate system with ellipsoidal height. Of the 13 surveyed points, 7 are used for processing while the remaining 6 are held back as check points. 2.3 LiDAR Survey A single LiDAR scan was collected from the roof using a RIEGL VZ-400 terrestrial scanner for the purposes of providing an accurate point cloud reference. The scanner was mounted on a survey tripod and then positioned at the linear center of the image collection positions. A fulll 360-degree scan was performed with a horizontal and vertical angular resolution of 0.04 degrees resulting in a ground density of 1300 ppm 2 at a distance of 50 m. The LiDAR point cloud was georegistered by matching 7 GCPs to their associated ground features using QT Modeler. The final registered point cloud consisted of 42 million points with an RMS error of 1 cm. Figure 2 shows the terrestrial LiDAR point cloud, associated ground control and marked camera positions. Proc. of SPIE Vol

3 0;, Camera1 Positions Ground Points Check F'oints -20 Figure 2. Terrestrial LiDAR point cloud of Naval Postgraduate School campus quad area as viewed from above. Points are colored by a blend of elevation and intensity while the rooftop represents the highest elevation in the scene. The majority of elevations are negative due to the WGS84 height above ellipsoid coordinate system and the proximity to sea level. Camera positions, GNSS ground and check points are illustrated as pushpins. 2.4 Photogrammetric Processing Agisoft Photoscan v1.2.3 was used to produce individual point clouds from the acquired thermal and visible image datasets. Using a single project, each image dataset was assigned a separate chunk within the workspace to allow batch processing to be performed across the entire project. Image alignment was performed using a high accuracy setting resulting in the reconstruction of a sparse point cloud in an arbitrary coordinate system. Seven of the GCPs were then marked on all corresponding images while the remaining 6 GCPs were withheld as check points. GCPs are used for point cloud optimization, whereas check points are used to assess the accuracy of the georegistration. After ground control placement, the sparse point cloud was then optimized, producing associated measurement error in both distance and pixels. Finally, a densee point cloud was calculated using the ultra-high quality setting with aggressive depth filtering. The resulting point clouds are discussed in Section OBSERVATIONS 3.1 Thermal and visible imagery Daytime thermal and visible imagery were collected at 1:30 pm on February 9 th, Nighttime thermal imagery was collected just prior to sunrise at 6:25 am the following day. Figure 3 shows thermal and visible image mosaics from a single marked position for illustration purposes. Each thermal image collection was performed within 15 minutes from start to finish in order to minimize surface temperature variation between image capture. Weather conditions were clear during both day and night image collection times. FLIR s ResearchIR was used to perform temperature calibration for each thermal image datasett producing temperature values in C. Thermal images were then outpu at as radiometric 32- bit TIFFs. To preserve the radiometric values during image processing, thermal images were scaled down to 16 bit TIFFs by multiplying the temperature value by Proc. of SPIE Vol

4 r -a Figure 3. Daytime visible (top), thermal (middle) and nighttime thermal (bottom) image mosaics for a single image collection position. In the grayscale thermal images, cool regions are black and hot regions are white. In the daytime thermal imagery (collected at 1:30 pm), the radiation reaching the thermal camera is dominated by reflected sunlight. Shadows from tall vegetation and buildings are clearly present in both the daytime thermal and visible images. The rooftop and the façade of the center building appear similar in brightness. Towards the bottom of the image, two bright spots represent manhole covers above an active underground steam pipe. The outlines of the manhole lids can clearly be delineated as circular (left) and square (right). The nighttime thermal imagery (collected at 6:25 am prior to sunrise) shows radiation from surfaces occurring well after sunset. Shadows from sunlight are not present, although thermal shadows can be observed from cars that had departed the previous day. Building rooftops appear very dark compared to their façades. The steam pipe manhole covers can be identified, although their shapes can t be recognized as easily as in the daytime thermal imagery. The steam line connecting the manhole covers is also apparent. 3.2 Photogrammetric Point Clouds Photogrammetric processing of daytime and nighttime thermal imagery produced approximately 11 million and 13 million densee points respectively. In comparison, processing of visible imagery resulted in over 49 million dense points, although this is mostly due to differences in image resolutions between the thermal and visible cameras. Figure 4 shows the resulting visible and thermal photogrammetric densee point clouds. Proc. of SPIE Vol

5 ú o,,., I Figure 4. Oblique view of daytime visible (left), daytime thermal (middle), and nighttime thermal (right) photogrammetric point clouds. Thermal point clouds are colored by temperature in C. In all 3 photogrammetric point clouds, the ground surfaces were reconstructed with uniform density where clearly visible gaps are caused by occlusion from elevated objects. Only portions of trees were reconstructed throughout the site. Trees remain difficult to reconstruct in any photogrammetric application due to motion and occlusion. In both thermal point clouds, the majority of the building rooftop is represented while several areas failed to reconstruct in the visible point cloud. During the sparse point cloud optimization, Photoscan produced georegistration error metrics for the marked GCPs and check points which are given in Table 1. In both thermal point clouds, the GCP errors were rather low when compared to those of the visible point cloud. For both the thermal and visible point clouds, the errors in XY were the largest contributor to the total error. The opposite is often true when considering positional errors associated with GNSS hardware used to collect the survey points. For the thermal point clouds, the total check point error was over 3 times higher than the total GCP error, while the visible point cloud GCP and check point errors were consistent. A more detailed analysis of the photogrammetric point cloud errors is given in Section 4. Table 1. Summary of reported GCP and check point errors in XYZ from Photoscan. GO PXY r(m) GCP2 GCP Total Error(m) 1 Error(m) Erro ( í Ó i d CheckXY Error(m) POINT CLOUD ANALYSIS CMeck Z Etrror(m) Check Err GCP and check point errors are useful when considering the accuracy of point clouds at specific locations. To determine the accuracy on a more global scale, the photogrammetric point clouds are compared to the acquired terrestrial LiDAR scan by computing the distance using the Multi Scale Model to Model Cloud Comparison (M3C2) plugin implemented in CloudCompare v Simply put, M3C2 computes the local distance between two point clouds along the surface normal direction without the need for meshing or gridding 5. Surface normals are determined by calculating the scale at which the reference point cloud becomes the most flat. In areas where the compared point clouds do not overlap, the distance is not computed. To reduce computation time and focus on areas with maximum overlap, only points within a 100 m spherical range from the terrestrial LiDAR scanner location are considered. Figure 5 shows the results of calculating the photogrammetric and terrestrial LiDAR point cloud distances using M3C2 while Figure 6 shows the associated histograms. Proc. of SPIE Vol

6 Distance (r Figure 5. M3C2 distance of daytime visible (top), thermal (middle), and nighttime thermal (bottom) photogrammetric point cloud from a single terrestrial LiDAR scan. Areas where the compared point clouds do not overlap are depicted in grey mo msaoo - Meanc m Std Dlev: m Y como Mean: m Std Delv: m COO 000 Mean: 0.01 Std Dey: mom moro ü.s u.zs M 2dspn[ (m) u.s o, a.7s a.s o O. 2 distance (m: ) o <lds ! Mxz&sfanceOn) Figure 6. M3C2 distance histograms for the daytime visible (left), daytime thermal (middle), and nighttime thermal (right) photogrammetric point clouds. Proc. of SPIE Vol

7 For all photogrammetric point clouds, the majority of points were found to be well within a few centimeters from the terrestrial LiDAR scan. The reconstruction accuracy was highest in areas representing impervious surfaces and low vegetation with mean distances close to zero. Both daytime thermal and visible point clouds were highly affected by areas located within shadows. Points created within shadow were typically characterized as having low elevations (blue); this affect is most noticeable on the left side of the daytime thermal distance image in Figure 5. Although shadows were not present in the nighttime thermal point cloud, a similar effect was observed in the building rooftops where temperatures tend to be very low. 5. SUMMARY AND CONCLUSIONS Daytime and nighttime thermal image datasets were acquired with an oblique perspective and high overlap to explore photogrammetric point cloud creation using Agisoft Photoscan. An additional daytime visible image dataset was coincidentally acquired to demonstrate differences and similarities between point clouds reconstructed using visible and thermal image sources. Point clouds created directly from radiometric thermal imagery not only produce similar geometry to those of visible point clouds, valuable temperature information for each point are now embedded and exploitable in 3D. The reconstruction accuracy was determined by calculating the 3D distance between the photogrammetric point clouds and a terrestrial LiDAR scan using M3C2. Point clouds derived from both day and night collected thermal imagery were found to perform similarly to the visible point cloud with mean distances between 1 and 2 cm. An improvement in accuracy was found in the nighttime thermal point cloud largely due to the absence of shadows. REFERENCES [1] Hirschmüller, H., Accurate and efficient stereo processing by semi-global matching and mutual information, Proc. IEEE Computer Society Conference on Computer Vision, , (2005). [2] Hirschmüller, H., Semi-global matching motivation, developments and applications, Photogrammetric Week, 2011, (14 Mar 2016) [3] James, M.R., Robson, S., Pinkerton, H., and Ball, M., Oblique photogrammetry with visible and thermal images of active lava flows, Bulletin of Volcanology Vol. 69, (2006). [4] Lewis, A., Hilley, G.E., and Lewicki, J.L., Integrated thermal infrared imaging and structure-from-motion photogrammetry to map apparent temperature and radiant hydrothermal heat flux at Mammoth Mountain, CA, USA, Journal of Volcanology and Geothermal Research, Vol. 303, (2015). [5] Lague, D., Brodu, N. and Leroux, J., Accurate 3D comparison of complex topography with terrestrial laser scanner: application to the Rangitikei canyon (NZ), ISPRS Journal of Photogrammetry and Remote Sensing 82, (2013). Proc. of SPIE Vol