Investigation of indoor and outdoor performance of two portable mobile mapping systems

Transcription

1 Investigation of indoor and outdoor performance of two portable mobile mapping systems Erica Nocerino a*, Fabio Menna a, Fabio Remondino a, Isabella Toschi a, Pablo Rodríguez-Gonzálvez b,c a 3D Optical Metrology unit, Bruno Kessler Foundation (FBK), Trento, Italy <nocerino><fmenna><remondino><toschi>@fbk.eu, Web: b TIDOP Research Group, Higher Polytechnic School of Ávila, University of Salamanca, Hornos Caleros, 50, 05003, Ávila, Spain. pablorgsf@usal.es c Dept. of Mining Technology, Topography and Structures, University of León, Avda. Astorga, s/n, 24401, Ponferrada, León, Spain. p.rodriguez@unileon.es ABSTRACT The paper investigates the performances of two portable mobile mapping systems (MMSs), the handheld GeoSLAM ZEB-REVO and Leica Pegasus:Backpack, in two typical user-case scenarios: an indoor two-floors building and an outdoor open city square. The indoor experiment is characterized by smooth and homogenous surfaces and reference measurements are acquired with a time-of-flight (ToF) phase-shift laser scanner. The noise of the two MMSs is estimated through the fitting of geometric primitives on simple constructive elements, such as horizontal and vertical planes and cylindrical columns. Length measurement errors on different distances measured on the acquired point clouds are also reported. The outdoor tests are compared against a MMSs mounted on a car and a robust statistical analysis, entailing the estimation of both standard Gaussian and non-parametric estimators, is presented to assess the accuracy potential of both portable systems. Keywords: Mobile mapping system, backpack, handheld, accuracy, robust statistical analysis, length measurement error, noise 1. INTRODUCTION The ability of acquiring and recording precise, dense and geo-referenced 3D information is a constant request for a vast variety of applications, ranging from civil engineering and construction to cultural heritage, from environment to industry, etc. The most effective way to satisfy this need is represented by mobile mapping systems (MMSs), instruments able to acquire 3D information on-the-move, using a moving platform. A MMS is a compound system consisting of three main components 1 : mapping sensors for the acquisition of 3D/2D data (point coordinates and/or images), a positioning and navigation unit for spatial referencing, and a time referencing unit operating as central system for data synchronization and integration. Since their early development in the late 1980s, MMSs have been progressively improved in order to provide increasingly more precise and denser data, acquired in shorter time. Besides the progresses in optical sensors, one of the key advances in MMS is related to spatial referencing technology. While the very early applications were restricted to environments where the sensor positions were computed using ground control 2, thanks to advantages in satellite and inertial technology, today spatial referencing is commonly possible in previously unknown and undiscovered places. Also, the miniaturization and cost reduction of components have played a fundamental role in the spread of MMS, allowing for more and more flexible, portable and low-cost systems. * nocerino@fbk.eu; phone ;

2 a) b) c) d) Figure 1. The devices under investigation: the GeoSLAM ZEB-REVO (a and b) and Leica Pegasus:Backpack (c and d). The majority of the MMSs rely on light detection and ranging (LiDAR) sensors as 3D mapping unit, but are usually equipped also with cameras for retrieving color information of the scene. Traditionally, MMSs are fitted into vehicles, like vans, cars, planes, boats, etc., which represent the optimal choice for recording vast areas, but hardily permit the acquisition of narrow passages and are ineffective for indoor scenarios. A more efficient alternative for 3D mapping in complex and indoor scenarios, characterized by rough terrain, small obstacles, stairs, entails robotic platforms, which are however still limited to the research domain 3,4,5. More widespread are MMSs designed and adapted to be easily carried or worn by persons walking through and, consequently, mapping the environment of interest. Nowadays, popular solutions available on the market and among research laboratories are: 1. man-portable MMS backpacks: these systems usually feature a conventional positioning and navigation unit integrating GNSS (global navigation satellite system) and IMU (inertial measurement unit) sensors (Akhka R2 6 ; Leica Backpack:Pagasus 7 ; ROBIN 8 ); however, solutions with only IMU or even without positioning and navigation components exist 9 (HERON 10 ). Some systems are also equipped with cameras providing panoramic spherical video or still imagery (UltraCam Panther 11 ); 2. handheld MMSs: the main difference from backpack solutions is that the user holds the scanning device in hand. These systems are usually equipped with IMU based navigation unit (no GNSS), 2D laser profilometer (ims2d 12, ZEB1 13, ZEB-REVO 14 ), and may also feature a very wide angle camera (PX-80 15, ZEB-CAM 16 ); 3. trolley MMSs: in this configuration, mainly designed for indoor mapping applications the different sensors (lidar, IMU, cameras, etc.) are fitted in a cart (ims 3D 17, Timms 18 ). Previous works presented accuracy evaluation of vehicle-based MMSs: (i) using an established urban test field focusing on planimetric and elevation errors 19, (ii) adopting reference values from an existing 3D city model 20, (iii) in terms of target representation 21, (iv) through robust statistical assessment with respect to photogrammetry and terrestrial laser scanner (TLS) data 23,24. Backpack MMSs were also tested: (i) in outdoor scenarios, using as reference data 3D points collected with unmanned aerial vehicle (UAV) equipped with a laser scanner 24, reporting planimetric and elevation errors with respect to a TLS 25 ; (ii) in indoor environment, employing a high-precision laser-based positioning and tracking system Paper overview The aim of the article is to investigate the performance of two portable MMS: the GeoSLAM ZEB-REVO (Figure 1a and 1b) and the Leica Pegasus:Backpack (Figure 1c and 1d). Both systems, described in section 2, are evaluated indoor (section 3) and outdoor (section 4). In the first test, two floors of a building are surveyed with the two portable MMS and independent check measurements are acquired using a ToF (time-of-flight) phase-shift TLS (Leica HDS ). The indoor scene, characterized by smooth and homogenous surfaces, as well as constructive elements like columns, is also used to derive meaningful information about noise on horizontal and vertical planes, along with fitting of geometric primitives for the two systems. The outdoor experiment is carried out in an 80 m by 70 m historical city square, where the two portable MMS are compared against a classical van-based MMS (RIEGL VMX ) whose accuracy potential

3 is discussed by Toschi et al 22. A robust statistical analysis is performed to evaluate the portable MMSs geometric performance in an outdoor environment. 2.1 The GeoSLAM ZEB-REVO device 2. THE TWO PORTABLE MMS The GeoSLAM ZEB-REVO is a lightweight portable MMS, commercialized by the GeoSLAM company. The instrument is the evolution of the ZEB1 device, commercial version of the ZEBedee handheld 3D range sensor developed by the Autonomous Systems Laboratory, CSIRO ICT Centre in Brisbane (Australia) 29. The main difference between the two devices is that the ZEB1 is equipped with a spring that allows the scan head to nod, or oscillate both in a front-back and side-by-side direction; in the updated version of the system, the ZEB-REVO, the spring is no longer present and it has been replaced by an automatic rotating head. The instrument features a 2D infrared laser scanner profilometer (UTM-30LX) coupled with an IMU sensor, without GNSS receiver. The 2D laser profilometer is a compact laser scanning range finder (LRSF), which consume less power, is more compact and light-weight than a classic 3D laser scanner system 30. The UTM-30LX emits pulsed light beams in the near infrared, with a wave length of 905 nm; the travelling time of the pulse from the sensor to the object and back provides the range measurement, according to the well-known time-of-flight measurement principle. The laser source scans a 270 semicircular field, so that the coordinates of the recorded points are calculated using the measured distance and laser pulse step angle 31. The industrial-grade microelectromechanical (MEMS) IMU is mounted beneath the scanner and consists of triaxial gyros and accelerometers, providing measurements of angular velocities and linear accelerations that, combined with the laser data, allow to estimate the sensor trajectory. The IMU also contains a three-axis magnetometer to reduce environmental magnetic interference 32. The scanning head, consisting of the laser and IMU sensors, is connected to a backpack containing the battery and a data-logger unit, where the acquired data are stored in real-time. The 2D laser profiles are aligned through a 3D SLAM (simultaneous localization and mapping) approach that estimates the six degree of freedom (6DoF) of the sensor head motion, and generates the 3D point cloud of the scene (i.e. the map). Details of the implemented SLAM approach are provided in 29,33. Recently, the ZEB-REVO has been equipped with a GoPro camera. The authors haven t had the opportunity to test the new version of the device yet; however, according to the specifications from the manufacturer 16, the camera is intended to provide imagery co-registered with the scanning data, but not to enrich the 3D laser point cloud with RGB information. The authors developed an in-house method to map color information to the 3D data acquired with the GeoSALM devices based on two GoPro cameras attached to the scan head (Figure 2a). The images from the cameras are processed through a photogrammetric workflow and the obtained RGB information is mapped onto the ZEB point cloud (Figure 2b and 2c). a) b) c) Figure 2. An in-house method is developed to map color information based on two GoPro cameras fixed on the GeoSLAM ZEB1 device (a). The original (b) and colored (c) 3D point clouds.

4 The proper acquisition approach with both GeoSLAM instruments requires to leave them stationary on a horizontal, planar surface for the so-called initialization procedure, which allows to define a local coordinate reference system with the vertical axis perpendicular to this planar surface. Then the user preforms the acquisition walking through the scene and, at the end, must go back to the initial position to close the acquisition loop. The collected data are post-processed through proprietary software applications, either running on local machine or via cloud processing. The data presented in this contribution were processed via the desktop software application (version v1), where the processing was totally automatic and no user intervention or alignment refinement was possible. Table 1. Technical specifications of the tested portable MMSs. (*) An optional GoPro camera is now available. GeoSLAM ZEB-REVO Leica Pegasus:Backpack Max range indoor outdoor 30 m m 50 m # Laser scanners 1 2 Measuring principle ToF ToF Scanner data acquisition rate 43,200 points/sec 300,000 points/sec per scanner Scanner Resolution horizontal vertical Scanner angular FOV 270 x x 30 per scanner Channels 1 16 per scanner Laser wavelength 905 nm 903 nm Scanner line speed 100 Hz 5-20 Hz Laser head rotation speed 0.5 Hz - Scanner weight 1.0 kg 0.83 kg per scanner Scanner dimensions 86 x 113 x 287 mm 72 x 72 x 72 mm per scanner # Cameras 0 (*) 5 CCD / pixel size x 2046 / 5.5 um x 5.5 um per camera Focal length mm Cameras total angular FOV x 200 Cameras max frame rate - 8 Hz Absolute position accuracy IMU type MEMS FOG GNSS - Triple band, single and dual antenna support Relative accuracy 2-3 cm 3-5 cm indoor outdoor 3-30 cm (10 mins scanning, 1 loop) 5-50 cm (10 mins scanning, minimum 3 loop closures or double passes conditions) 5 cm Tot system weight 4.1 kg 11.9 kg Backpack dimensions 220 mm x 180 mm x 470 mm 310 mm x 270 mm x 730 mm Operating time 4 hours 3 hours, up to 6 hours with optional batteries

5 2.2 The Leica Pegasus:Backpack The Leica Pegasus:Backpack (summer 2016 version) combines two laser profilometers (Velodyne PUCK TM VLP-16) synchronized with five cameras, a triple band GNSS receiver (NovAtel ProPak6 ), and a fiber optic gyroscope (FOG) IMU. Each VLP-16 features 16 laser/detector pairs, mounted in a rotating housing providing 360 field of view 34. The device is also equipped with five high-dynamic-range cameras, placed to acquire 360 x200 field of view. The cameras provide images co-registered with the laser data and RGB color information projected to the point cloud. The full inertial navigation system (INS) is composed of GNSS and IMU sensors; the absolute and outdoor positioning is delivered by GNSS, while indoor positioning and in GNSS-denied environments is based on the IMU and a SLAM algorithm. The integrated IMU is a FOG type, which is usually more costly and of slightly higher performance than MEMS systems for standalone INS performance 35. The acquired data, i.e. laser profiles and spherical images, are automatically processed via the proprietary software application; however, the user may refine the alignment by either importing ground control points (GCPs) or identifying 3D tie points on the point clouds. A plug-in working in ArcMAP, component of the geospatial processing toolkit ArcGIS by Esri, is available to visualize and analyze the data. To assure a fair investigation of the two MMSs, the Leica Pegasus:Backpack data presented in this study are processed following an automatic approach with minimal user intervention. Evaluating the system calibration and adjustment procedures of the collected raw data is out of the scope of the tests here described. 3. INDOOR PERFORMANCE EVALUATION The method adopted to test the indoor performances of the two portable MMSs is based on two measures for quality assessment, i.e. the root mean square (RMS) of the residuals from fitting of geometry primitives (section 3.1) and length comparison (section 3.2). The scanning operations in the building (Figure 3) are carried out with the two MMSs preforming the initialization procedure outside the building and following a close-loop acquisition path. Figure 3. The two-floor building, test area for the indoor performance evaluation. Left: a picture of the interior. Right: cross section of the ZEB-REVO point cloud. 3.1 Reference data and alignment of point clouds To evaluate the indoor accuracy potential, a single point cloud, consisting of a great portion of the second floor and part of the first floor, is acquired with a phase-shift ToF terrestrial laser scanner (TLS), the Leica HDS7000, whose technical

6 specifications are reported in Table 2. Only one point cloud is used as refence to avoid any additional uncertainty that might be caused, for example, by uncontrolled residual errors in the registration process of several point clouds. Figure 4. Up: top view of second floor walls; down: perspective view of a section of the two floors. In white the HDS7000 point cloud, in cyan the ZEB-REVO, in yellow the Pegasus. The point clouds acquired with the three different devices are aligned in the same coordinate reference system, defined as follows: the z axis coincides with the gravity vector, i.e. the vertical direction provided by leveling the HSD7000 with the digital bubble; the x axis runs along the width of the floors; the y lays along the length. Some views of the aligned point clouds are shown in Figure 4.

7 Table 2. Technical specifications of the Leica HDS7000 TLS. Measuring principle phase-shift ToF Max range 187 m (ambiguity interval) Data acquisition rate Up to 1,016,727 points/sec Scanner resolution (horizontal/vertical) Scanner angular FOV 360 x 320 Laser wavelength 1500 nm Range noise (on a black 14% target, i.e. worst condition) 0.5 / 2.7 mm 10 / 50 m Linearity error 1 mm Angular accuracy (horizontal/vertical) 127 urad 3.2 Noise estimation The device s noise is evaluated through the fitting of geometric primitives (planes and cylinders) to the acquired point clouds: vertical planes are fitted on the walls, horizontal planes on the floors and ceilings, cylinders on the columns. Table 3 reports the RMS results. As expected, the HDS7000 greatly outperforms the two MMSs, which anyway show RMS values significantly within the levels declared by the vendors (see the relative accuracy row in Table 1). Table 3. Noise estimation: RMS of fitting procedures with outliers removal. HDS700 GeoSLAM ZEB-REVO Leica Pegasus:Backpack Vertical planes 0.1 cm 1.1 cm 1.9 cm Horizontal planes 0.2 cm 0.9 cm 1.2 cm Cylindrical columns 0.1 cm 0.9 cm 1.8 cm 3.3 Length measurements The positional or coordinate standard error (S XYZ), theoretical length measurement error (TLME) and relative theoretical length measurement accuracy (RTLMA) are reported in Table 4. The S XYZ for the two MMSs are derived from the datasheet, i.e. the absolute position accuracy in Table 1, whilst for the HDS7000 is computed taking into account the angular accuracy, linearity error and range noise for two distances (10 m and 50 m) in the worst condition (laser reflected from a 14% black surface, Table 2). The TMLE and RTLMA are, respectively, calculated according to (1) 36 and to (2), for two distance values D equal to 3 m and 50 m, considering the best and worst positional standard errors. TLME 3 2 (1) s XYZ D RTLMA 1: ROUND (2) TLME Table 4. Positional or coordinate standard error (SXYZ), theoretical length measurement error (TLME) and relative theoretical length measurement accuracy (RTLMA). The value marked with () are taken from the instrument datasheet (Table 1 - absolute position accuracy indoor ). HDS700 GeoSLAM ZEB-REVO Leica Pegasus:Backpack Min Max Min Max Min Max s XYZ m m 3 cm 30 cm 5 cm () 50 cm () TLME 0.6 cm 2.3 cm 12.7 cm cm 21.2 cm cm D = 3 m 1:500 1:150 1:25 1:2 1:15 1:1 RTLMA D = 50 m 1:9000 1:2200 1:500 1:50 1:250 1:25 The theoretical length measurement accuracy of the HDS7000 is from one to two orders of magnitude better than the two portable MMS, and increases, as expected, with the distance.

8 Figure 5: Distances measured for the indoor performance evaluation. Figure 5 depicts the distance measurements extracted from the point clouds of the two-floor building. The length, L, two width values, B1 and B2, and one height, H1, are measured on the second floor; the height H2 is measured from the first floor to the ceiling of the second. Table 5 reports the analysis performed on the measured distances. Each value is the mean of eight distance measurements between two planes fitted on the opposite walls (L, B1 and B2), and between floor and ceiling (H1 and H2). The relative length measurement error (RLME) is computed according to (3) as the relative difference between the measured distance D m for the GeoSLAM ZEB-REVO and Leica Pegasus:Backpack, and the distance from the HSD7000, assumed as reference length D r. The relative length measurement accuracy (RLMA) is defined as the rounded absolute reciprocal value of the RLME times 100 (4). D RLME m Dr 100 Dr (3) 100 D r RLMA 1: ROUND 1: ROUND RLME Dm Dr (4) The RLMA results always better than the computed theoretical accuracies (Table 4); in particular, the GeoSLAM ZEB- REVO provides distance measurements that are closer to the HDS7000 than the Leica Pegasus:Backpack, which also features higher standard deviation values (σ).

9 Table 5. Indoor performance evaluation: measured distances with standard deviations (σ), relative length measurement errors (RMLE) and accuracies (RMLA). HDS700 GeoSLAM ZEB-REVO Leica Pegasus:Backpack Dr [m] σ [cm] Dm [m] σ [cm] RLME RLMA Dm [m] σ [cm] RLME RLMA L cm % 1: % 1:2000 B cm % 1: % 1:50 B cm % 1: % 1:150 H cm % 1: % 1:50 H cm % 1: % 1:50 4. OUTDOOR PERFORMANCE EVALUATION The outdoor tests are performed in the cathedral square in Trento (Figure 6a). The site represents a challenging scenario for SLAM based systems due to moving objects, people, cars, trucks, buses, and the façades geometry. The city is surrounded by high mountains that might constitute an unfavorable environment also for GNSS based systems, although the square itself is quite big (80 m x 70 m). It is worth noting the evident blunders in the GeoSLAM ZEB-REVO point cloud (Figure 6c, highlighted in yellow): the fountain and tree are duplicated, clearly due to huge error in the alignment of the laser profiles. The point clouds acquired with the two portable MMSs (Figure 6c and 6d) are compared through a robust statistical analysis with data derived from the RIEGL VMX-450 MMS mounted on a van. a) b) c) d) Figure 6. The cathedral square in Trento (Italy), test area for the outdoor performance evaluation (a). RIEGL VMX-450 MMS colored point cloud of the square (b); GeoSLAM ZEB-REVO point cloud (c); Leica Pegasus:Backpack colored point cloud (d).

10 4.1 Reference data The RIEGL VMX-450 MMS platform (Table 6) integrates two synchronously operated VQ-450 laser scanners, a portable control unit (VMX-450-CU) and IMU/GNSS navigation hardware. The system measures up to 1.1 million points and 4 profiles per second, with an online waveform processing that allows to penetrate obstructions, such as fences and vegetation. The platform is also equipped with the modular VMX-450-CS6 camera system, where up to six industrial digital color cameras can be integrated. The cameras complement the acquisition of geometric data from the laser scanners with time-stamped images. A detailed performance investigation of the RIEGL VMX-450 MMS is presented in Toschi et al 22. Table 6. Technical specifications of the RIEGL VMX-450 system. # Laser scanners 2 Measuring principle phase-shift ToF Max range 800 m Data acquisition rate 550,000 points/sec per scanner Scanner angular FOV up to 360 Laser wavelength near infrared # Cameras up to 6 CCD / pixel size 2452 x 2056 / 3.45 um x 3.45 um per camera Focal length 5 mm Cameras angular FOV 80 x 65 per camera Relative position accuracy 1 cm Absolute position accuracy 2-5 cm Tot system weight > 100 kg 4.2 Alignment and signed distance computation Figure 7. Color-coded map of the signed distances computed between the GeoSLAM ZEB-REVO (left) and Leica Pegasus:Backpack (right) point clouds and the RIEGL VMX-450 data. The differences are in m. To evaluate the accuracy potential of the two portable MMSs against the reference data, the three point clouds are first cleaned from noisy elements, such as pedestrians, vegetation, cars, etc. Due to the blunder in the ZEB-REVO point cloud (Figure 6c), the fountain is also removed; consequently, only the building façades overlooking the square are used in the performance evaluation. The blunder-free point clouds from GeoSLAM ZEB-REVO and Leica Pegasus:Backpack are aligned in a local coordinate reference systems to the cleaned RIEGL VMX-450 data by means of the iterative closest point (ICP)

11 registration method implemented in the open source software application CloudCompare v2.8 ( The signed distances between the portable MMSs and RIEGL VMX-450 point clouds are then computed, using the CloudCompare M3C2 plugin, which implements the Multiscale Model to Model Cloud Comparison method 37. It allows a direct comparison of 3D points, without the need of a preliminary meshing or gridding phase. If the data do not contain normal vectors, they are estimated on the basis of the local surface roughness. Then, for each 3D point, the local distance between the two clouds is computed. The color-coded maps of the signed distances for both the portable MMMs are shown in Figure Robust statistical analysis Pegasus:Backpack VS RIEGL VMX-450 ZEB-REVO VS RIEGL VMX-450 Figure 8. Histograms of the signed differences with the superimposed curve for the normal distribution (left). Q-Q plots of the distribution of the signed differences (right). Several studies 38,39,40 have demonstrated that in the accuracy assessment of data provided by laser scanner systems, as well as photogrammetry, the hypothesis that errors follow a Gaussian distribution is hardly verified. This might be due to the presence of residual system errors, but also unwanted objects not correctly filtered out from the data. In the following analyses, the hypotheses (i) that many outliers will exist in comparing data provided by different instruments and (ii) that the normality assumption of distribution of the differences is not valid, are first verified. Then, suitable accuracy measures are computed and reported in Table 7. Two visual diagnostic tests are reported to test the normality assumption (Figure 8), i.e. the histogram of the signed differences with the superimposed curve for the normal distribution, and the quantile-quantile (Q-Q) plot of the distribution of the signed differences 38. The Q-Q plot depicts the quantiles of the empirical distribution plotted against the theoretical quantiles of the normal distribution. If the actual distribution is normal, the Q-Q plot should provide a straight line. Big deviation from the straight line indicates that the distribution of the errors is not normal.

12 Both graphical evaluations show that the differences do not follow a Gaussian distribution: the Q-Q plot relative to the GeoSLAM ZEB-REVO comparison shows a shape significantly far away from the theoretical normal hypothesis, while the histogram for the Leica Pegasus:Backpack a two-peaked distribution with a quite high dispersion. The previous conclusion is also supported by the computed kurtosis (5) and skewness (6) values, which represent, respectively, a measure of whether the data are peaked or flat with respect to a normal distribution, and an indication of departure from symmetry in a distribution (asymmetry around the mean value): x i1 i Kurtosis 4 n 1 Skewness n 4 n x i 1 i n (5) (6) Where n, µ and σ are the sample size (i.e. number of data points x), mean and standard deviation. When the sample distribution follows the normal hypothesis, both values should be equal to zero. The distance distribution for the GeoSLAM ZEB-REVO reveals a highly peaked shape, while the distribution for the Leica Pegasus:Backpack results slightly more asymmetric. In both cases, the distribution of the differences discloses kurtosis, skewness, hence a significant amount of outliers; consequently the classical µ and σ parameters are not adequate to provide the accuracy measures for the two portable MMSs. When this is the case, other non-parametric estimators are to be adopted, such as the median m, normalized median absolute deviation NMAD (7) and the square root of the biweight midvariance BWMV (8): n BWMV NMAD MAD (7) n n i1 a a i1 i i xi m 1 Ui U 1 5U i i 1, if Ui 1 a i (9) 0, if U 1 i xi m U (10) 9MAD (8) being the median absolute deviation MAD (9), i.e. the median (m) of the absolute deviations from the data s median (m x): MAD m x i m x (11) The computed accuracy measures are summarized in Table 7. The value of the median is closer to the mean for the GeoSLAM ZEB-REVO, while is smaller for the Leica Pegasus:Backpack. The values are inside the expected a-priori error of the point clouds analysis (Table 1); they are likely to represent the residual of the registration process. In both the cases, the values of the median are smaller than the absolute position accuracy quoted in the technical sheets (Table 1). Concerning the values of the standard deviations, they are higher than the NMAD and BWMV, and lower than the specifications from the vendors. In accordance to the visual analysis of the histograms (Figure 9), the values for the Leica Pegasus:Backpack indicate a wider dispersion than the GeoSLAM ZEB-REVO. Since the data sample does not follow a Gaussian distribution, the standard deviation values are affected, so it does not represent correctly the error dispersion. It appears clearly in the case of GeoSLAM ZEB-REVO when is compared with the robust measures of the dispersion (NMAD and square root of BWMV), being undervalued (more than two times).

13 Please note also, the asymmetry of the error distribution, being not possible provide a plus-minus range, but an absolute interpercentile range. In the case of Table 7, it is computed according to the real error distribution. They (dispersion measurement and error distribution function) are related, showing for the Leica Pegasus:Backpack a wider dispersion than the GeoSLAM ZEB-REVO, in according to the visual analysis of the histograms (Figure 9). Moreover, in the reported interpercentile range at 95 % of confidence level, both portable MMS differs from their relative declared position precision. As a remark, the 50 % of the observed points (interpercentile range at 50% of confidence level) are in a 3.0 cm size interval for the GeoSLAM ZEB-REVO, and 13.5 cm for Leica Pegasus:Backpack, while the 95% of data points (interpercentile range at 95% of confidence level) are within 30 cm and 51 cm for the GeoSLAM ZEB-REVO and Leica Pegasus:Backpack (respectively). Gaussian assessment Robust assessment Table 7. Statistical analysis of the signed distance computation. ZEB-REVO VS RIEGL VMX-450 Pegasus:Backpack VS RIEGL VMX-450 Sample size n 4,061,074 1,0351,184 Kurtosis Skewness Sample mean µ 2.0 cm 1.9 cm Standard deviation σ 7.1 cm 13.3 cm Median 2.0 cm 1.0 cm NMAD 2.2 cm 9.6 cm Sqrt(BWMV) 2.9 cm 12.5 cm Interpercentile range 50% 3.0 cm 13.5 cm Percentile cm cm Percentile cm 29.1 cm Interpercentile range 95% 23.9 cm 50.9 cm 5. CONCLUSIONS The presented investigation aimed at evaluating the performance of two portable MMSs, the handheld GeoSLAM ZEB- REVO and Leica Pegasus:Backpack, in indoor and outdoor scenarios. The tests were designed to specifically address relevant issues related to mapping environments, such as building interiors or complex city parts, which represent typical applications for such systems. Consequently, the analyses were performed in order to estimate the magnitude of errors in measuring distances and acquiring 3D data. To this end, the data provided by the two MMSs were compared against those acquired by two different reference devices, a TLS for the indoor and a van-based MMS for the outdoor. While indoor the performance assessment was carried out reporting noise evaluation and length measurement errors, a robust statistical analysis was performed on data acquired outdoor. In both scenarios, the two portable MMSs performed within the accuracy specifications provided by the vendors, with the GeoSLAM ZEB-REVO generally outperforming the declared values despite the evident gross error in the point cloud acquired outdoor. Worth to note is that errors in Leica Pegasus:Backpack data might be further mitigated by including control points and constraints within the postprocessing adjustment. The results and statistical analyses presented in this paper were achieved using instruments tested during the summer Other performances could be expected using new releases of the instruments. ACKNOWLEDGMENTS Authors are thankful to Dr. Nadia Guardini (ME.S.A. srl), Mr. Marco Formentini and Mr Simone Oppici (Leica Geosystem Italy) for giving the possibility to use and test the two instruments investigated in this article.

14 REFERENCES [1] Puente, I., González-Jorge, H., Martínez-Sánchez, J. and Arias, P., "Review of mobile mapping and surveying technologies," Measurement 46(7), pp (2013). [2] Schwarz, K.P. and El-Sheimy, N., "Mobile mapping systems state of the art and future trends, "International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 35(Part B), p.10 (2004). [3] Surmann, H., Nchter, A., Lingemann, K. and Hertzberg, J., "Kurt3D-an autonomous mobile robot for modelling the world in 3D," ERCIM NEWS, 55, pp (2003). [4] Nüchter, A., Elseberg, J. and Borrmann, D., "Irma3D An Intelligent Robot for Mapping Applications," IFAC Proceedings Volumes, 46(29), pp (2013). [5] Ziparo, V.A., Zaratti, M., Grisetti, G., Bonanni, T.M., Serafin, J., Di Cicco, M., Proesmans, M., Van Gool, L., Vysotska, O., Bogoslavskyi, I. and Stachniss, C., "Exploration and mapping of catacombs with mobile robots," In Safety, Security, and Rescue Robotics (SSRR), 2013 IEEE International Symposium on, pp. 1-2 (2013). [6] Centre of Excellence in Laser Scanning Research, (April, 2017). [7] Leica, (April, 2017). [8] ROBIN, Three-in-one LiDAR mapping system, 3D Laser Mapping, < (April, 2017). [9] Nüchter, A., Borrmann, D., Koch, P., Kühn, M. and May, S., "A man-portable, IMU-free mobile mapping system," ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences II-3 W, 5, pp (2015). [10] Gexcel, (April, 2017). [11] Vexcel Imaging, (April, 2017). [12] VIAMETRIS, (April, 2017). [13] Geo-SLAM, (April, 2017). [14] Geo-SLAM, (April, 2017). [15] Paracosm, (April, 2017). [16] GeoSLAM, (April, 2017). [17] VIAMETRIS, viametris.info/ims3d/en/, (April, 2017). [18] Applanix, (April, 2017). [19] Kaartinen, H., Hyyppä, J., Kukko, A., Jaakkola, A. and Hyyppä, H., "Benchmarking the performance of mobile laser scanning systems using a permanent test field," Sensors, 12(9), pp (2012). [20] Haala, N., Peter, M., Kremer, J. and Hunter, G., "Mobile LiDAR mapping for 3D point cloud collection in urban areas A performance test," The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 37, pp (2008). [21] Lin, Y., Hyyppä, J., Kaartinen, H. and Kukko, A., "Performance analysis of mobile laser scanning systems in target representation," Remote Sensing, 5(7), pp , (2013). [22] Toschi, I., Rodríguez-Gonzálvez, P., Remondino, F., Minto, S., Orlandini, S. and Fuller, A., "Accuracy evaluation of a mobile mapping system with advanced statistical methods," The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 40(5), p.245 (2015). [23] Rodríguez-Gonzálvez, P., Jiménez Fernández-Palacios, B., Muñoz-Nieto, Á.L., Arias-Sanchez, P. and Gonzalez-Aguilera, D., "Mobile LiDAR System: New Possibilities for the Documentation and Dissemination of Large Cultural Heritage Sites," Remote Sensing, 9(3), p.189 (2017). [24] Rönnholm, P., Liang, X., Kukko, A., Jaakkola, A. and Hyyppä, J., "Quality analysis and correction of mobile backpack laser scanning data," ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, pp (2016). [25] Glennie, C., Brooks, B., Ericksen, T., Hauser, D., Hudnut, K., Foster, J. and Avery, J., "Compact multipurpose mobile laser scanning system Initial tests and results," Remote Sensing, 5(2), pp (2013). [26] Lauterbach, H.A., Borrmann, D., Heß, R., Eck, D., Schilling, K. and Nüchter, A., "Evaluation of a backpackmounted 3D mobile scanning system," Remote Sensing, 7(10), pp (2015). [27] Leica Geosystems, HDS7000_UserManual_en.pdf, (April 2017).

15 [28] RIEGL, (April 2017). [29] Bosse, M., Zlot, R. and Flick, P., "ZEBedee: Design of a spring-mounted 3-d range sensor with application to mobile mapping," IEEE Transactions on Robotics, 28(5), pp (2012). [30] Mader D, Westfeld P and Maas HG, "An integrated flexible self-calibration approach for 2D laser scanning range finders applied to the Hokuyo UTM-30LX-EW," The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 385 (2014). [31] Hokuyo, (April 2017) [32] Zlot, R. and Bosse, M., "Three-dimensional mobile mapping of caves," Journal of Cave and Karst Studies, 76(3), p.191 (2014a). [33] Zlot, R. and Bosse, M., "Efficient Large-scale Three-dimensional Mobile Mapping for Underground Mines," Journal of Field Robotics 31(5), pp (2014b). [34] Velodyne Lidar, %20Programming%20Guide,VLP-16.pdf (April 2017) [35] Goodall, C., Carmichael, S. and Scannell, B., "The battle between MEMS and FOGs for precision guidance." Analog Devices Technical Article MS-2432 (2013). [36] Luhmann, T., Robson, S., Kyle, S. and Boehm, J., [Close-range photogrammetry and 3D imaging]. Walter de Gruyter (2014). [37] Lague, D., Brodu, N. and Leroux, J., "Accurate 3D comparison of complex topography with terrestrial laser scanner: Application to the Rangitikei canyon (NZ)," ISPRS Journal of Photogrammetry and Remote Sensing, 82, pp (2013). [38] Höhle, J. and Höhle, M., "Accuracy assessment of digital elevation models by means of robust statistical method," ISPRS Journal of Photogrammetry and Remote Sensing, 64(4), pp (2009). [39] Hasan, A., Pilesjö, P. and Persson, A., "The use of LIDAR as a data source for digital elevation models a study of the relationship between the accuracy of digital elevation models and topographical attributes in northern peatlands," Hydrology and Earth System Sciences Discussions, 8(3), pp (2011). [40] Rodríguez-Gonzálvez, P., Garcia-Gago, J., Gomez-Lahoz, J. and González-Aguilera, D., "Confronting passive and active sensors with non-gaussian statistics," Sensors, 14(8), pp (2014).