Accuracy Assessment of an ebee UAS Survey McCain McMurray, Remote Sensing Specialist mmcmurray@newfields.com July 2014
Accuracy Assessment of an ebee UAS Survey McCain McMurray Abstract The ebee unmanned aircraft system (UAS) was used to survey an open-pit mine site in New Mexico. Three survey datasets were generated: a point cloud, an orthomosaic and a digital surface model (DSM). Twenty ground control points (GCPs) within the survey area were measured and marked prior to the ebee survey being conducted. Nine GCPs were used to calibrate the survey datasets and the remaining 11 were used to validate them. The accuracy assessment calculated root-mean-square error (RMSE) values for the x, y and z dimensions for the three survey datasets (where applicable). Four out of six RMSE values were lower (more accurate) than the manufacturer s expectations for absolute accuracy relative to survey ground sample distance (GSD; 1 2 times the GSD horizontally and 2 3 times the GSD vertically). The remaining two RMSE values fell within the expected ranges. The RMSE x, RMSE y and RMSE z for the most accurate survey dataset, the point cloud, were 1.46 in (3.71 cm), 2.58 in (6.55 cm) and 2.76 in (7.01 cm), respectively. The results of this project suggest that a properly conducted ebee survey incorporating accurate ground control is capable of generating survey datasets which meet or exceed the expected accuracy levels based on GSD. 1. Introduction The ebee is a fixed-wing UAS developed by sensefly Ltd., in Switzerland. The UAS is comprised of an unmanned aerial vehicle, a USB radio modem, pre-flight mission planning software and post-flight data processing software, all seamlessly integrated to form an accurate and efficient surveying tool. The ebee flies autonomously, navigating between user-defined waypoints while capturing imagery, and returning to a pre-defined landing area following the survey. Post-processing software, called Postflight, is then used to generate three primary datasets from the survey data: a point cloud, an orthomosaic and a DSM. The ebee was used to conduct a survey of an inactive mine site near Deming, New Mexico in March 2014. The survey captured 412 true color images from approximately 425 ft (130 m) above ground level with an average GSD of 1.71 in (4.33 cm). The images were captured with an along-track overlap of 75% and an across-track overlap of 70%. The survey area was located at 32 11' 15" N, 108 05' 06" W and covered a total area of approximately 0.33 mi 2 (86 ha). At the center of the survey area was an inactive, open-pit mine dug into the side of a hill. The open-pit mine was surrounded by a cleared gravel area within a high desert environment. 2
Figure 1. The survey location near Deming, New Mexico (left) and a view of the inactive pit mine (right). 2. Methods Licensed surveyors were hired to collect 20 GCPs within the survey area. The GCPs were collected with an average accuracy of 1 in (2.5 cm) and were distributed in and around the mine site. The GCP coordinates were collected in reference to the State Plane coordinate system, New Mexico, West Zone in US Survey Feet. To ensure visibility in the aerial imagery, each GCP was marked using a circular orange target measuring approximately 1 ft (30 cm) in diameter. Nine of the GCPs were used to calibrate the survey data prior to processing and the remaining 11 were used to validate the accuracy of the datasets generated by the survey. Following the successful completion of the ebee survey, the aerial imagery was georeferenced using the ebee s GPS data and imported into the Postflight processing software. The ground survey coordinates (easting, northing, elevation) of the calibration GCPs were imported into Postflight s GCP Editor and each was identified by marking the center of the corresponding photogrammetric target in at least four different images. The GCP editor data and the georeferenced aerial imagery were used as inputs for processing within Postflight. 3
Figure 2. The distribution of the nine calibration GCPs and 11 validation GCPs displayed over the orthomosaic produced by the ebee survey. Three survey datasets were produced for this project: a point cloud, an orthomosaic and a DSM. The point cloud contained over 50 million points, with an average point spacing of 6.1 in (15.4 cm). The orthomosaic and DSM both had spatial resolutions of 1.71 in (4.33 cm) and were composed of over 562 megapixels. 4
Figure 3. The distribution of the nine calibration GCPs and 11 validation GCPs displayed over the DSM produced by the ebee survey. Quantitative accuracy assessments were performed for all three survey datasets. Postflight was used to assess the accuracy of the point cloud and a combination of ArcMap and Excel was used to assess the accuracy of the orthomosaic and DSM. RMSE was calculated in order to assess the accuracy of the survey datasets in relationship to expectations based on the survey GSD. RMSE was calculated as the square root of the average of the set of squared differences between the coordinate values of the validation GCPs in the survey datasets and the corresponding validation GCP coordinates collected by the ground survey. In the point cloud accuracy assessment, prior to processing the survey data in Postflight, the GCP Editor was used to add the 11 validation GCPs by the same method described above for the calibration GCPs. The validation GCPs were marked as check points so they were not incorporated into processing the survey data. During processing, Postflight automatically calculated the RMSE x, RMSE y and RMSE z for the check points. 5
Figure 4. The point cloud produced by the ebee survey viewed from the West. In the orthomosaic and DSM accuracy assessments, the ground survey coordinates of the 11 validation GCPs were entered into Excel as GCP Northing, GCP Easting and GCP Elevation. The orthomosaic and DSM were opened in ArcMap and the coordinates of each of the 11 validation GCPs were measured from these datasets and entered into Excel as UAS Northing, UAS Easting and UAS Elevation. The RMSE equation described above was written into Excel and used to calculate the RMSE for both datasets. 3. Results Tables 1 3 display the results of the accuracy assessments conducted for the survey datasets in units of US Survey Feet (the units used to measure the GCPs and process the data). In the point cloud accuracy assessment, the RMSE x was 1.46 in (3.71 cm), RMSE y was 2.58 in (6.55 cm) and the RMSE z was 2.76 in (7.01 cm). In the orthomosaic accuracy assessment, the RMSE x was 1.58 in (4.01 cm) and the RMSE y was 3.11 in (7.90 cm). In the DSM accuracy assessment the RMSE z was 3.04 in (7.72 cm). 6
Table 1. Point Cloud Accuracy Assessment (US ft) According to the Postflight support documentation, the relative RMSE x and RMSE y for the survey datasets are expected to range from 1 2 times the GSD and the relative RMSE z is expected to range from 2 3 times the GSD. The incorporation of GCPs is expected to increase the absolute accuracy of the survey datasets. The impact of GCPs on the absolute accuracy of the survey datasets depends on the number of GCPs, their distribution in relation to the survey area and their accuracy. 7
Table 2. Orthomosaic Accuracy Assessment (US ft) Table 3. DSM Accuracy Assessment (US ft) Table 4 displays the RMSE values for all three survey datasets in relation to the survey GSD. The point cloud RMSE x and RMSE z, the orthomosaic RMSE x and the DSM RMSE z were all below the lower limit of the expected ranges. The point cloud RMSE y and orthomosaic RMSE y were both within the expected range. None of the RMSE values exceeded the upper limit of the expected ranges. 8
The RMSE values for the orthomosaic and DSM were higher than the corresponding values for the point cloud. These discrepancies were most likely the result of the interpolation of the point cloud used to create the DSM, which was subsequently used to create the orthomosaic. The RMSE x values for the point cloud and orthomosaic were substantially lower than the RMSE y values, but no systematic errors could be identified which may have contributed to this. Future work should consider incorporating additional validation GCPs to facilitate more robust accuracy assessments of survey datasets. Table 4. RMSE and GSD This survey and accuracy assessment demonstrate that the ebee is capable of generating survey datasets with accuracy levels that meet and exceed the expected relationship with GSD. 9