Mobile mapping system and computing methods for modelling of road environment

Mobile mapping system and computing methods for modelling of road environment Antero Kukko, Anttoni Jaakkola, Matti Lehtomäki, Harri Kaartinen Department of Remote Sensing and Photogrammetry Finnish Geodetic Institute PO Box 15 FI-02431 MASALA Antero.Kukko@fgi.fi Abstract - Mobile mapping is a new way of efficiently collecting three-dimensional data from the road environment. Mobile mapping systems are cost efficient and robust technique to acquire information about even highly dynamic environments like highways and urban streets, where the data collection has previously been laborious and even dangerous for the staff performing the surveying. The dynamic mobile mapping systems could access the site with less risk to the personnel and with less need for road closures. The need for high resolution and details captured in to the data for street and road inventories, or city modelling, are the main reasons for the rapid adoption of the mobile mapping techniques in these fields. Lidar based mobile mapping system produces three-dimensional points from the surrounding objects. Typically, two-dimensional profiling scanner is mounted on the system and the third dimension is achieved by the movement of the vehicle. The characteristics of the obtained point cloud depend largely on the sensor arrangement and the sensor properties. The ROAMER, a single-scanner system for road environment mapping presented in this paper, is able to use various tilted scanning planes for the point acquisition with 120 khz point measurement frequency and up to 48 Hz profile measurement rate. The relative point precision for the system is estimated to be a few millimetres, but is eventually defined absolutely by the accuracy of the navigation solution that could be provided in real-time, or more reliably through post-processing. We believe that in the future, lidar based mobile mapping will be used considerably for urban and road environment modelling, as well as in many other applications in the fields of construction, forestry, railways, and even in environmental modelling and monitoring e.g. hydrology and glaciology. In urban context, the main applications of these models could include urban and environmental planning, road safety assessment, road construction planning and navigation. I. INTRODUCTION A modern mobile mapping system, mounted on top of a vehicle, consists of a navigation system, one or more laser scanners, and cameras to provide mapping capacity. Many of the mapping applications could benefit from the improved accuracy and efficiency of the mobile mapping technology, compared to the traditional methods using aerial images and laser scanning as well as geodetic measurements using tacheometers and terrestrial lasers. Lidar based mobile mapping system produces threedimensional points from the surrounding objects. Typically, two-dimensional profiling scanner is mounted on the system and the third dimension is achieved by the movement of the Yuwei Chen Department of Navigation and Positioning Finnish Geodetic Institute PO Box 15 FI-02431 MASALA vehicle. The characteristics of the obtained point cloud depend largely on the sensor arrangement and the sensor properties. The ROAMER, a single-scanner system for road environment mapping, introduced first in [1], is able to use various tilted scanning planes for the point acquisition with 120 khz point measurement frequency and with its updated laser scanner up to 48 Hz profile measurement rate. The relative point precision for the system is estimated to be a subcentimetre, but is eventually defined absolutely by the accuracy of the navigation solution that could be provided in real-time, or more reliably through post-processing. Since the geometry of the scanning and point density for a mobile lidar is different from the airborne laser scanning, new algorithms are needed for information extraction [2],[3]. In addition, due to the huge amount of data produced by the mobile mapping systems, automated processing of the data becomes a necessity. For example, the data consisting of a kilometer of road can include tens of millions of 3D points, and hundreds of images. For a dataset containing tens or hundreds of kilometers of roads and streets, manual processing would be very time consuming, if not impossible. In order to efficiently process mobile mapping data we need to organize it. Laser data from the mobile mapping system is inherently organized as profiles consisting of consecutive points. Another efficient method is to tile the data into regular cubes and store the data in these tiles. This is very helpful when local neighborhood of a point is required, e.g. in nearest neighbor operations and classification. We present a mobile mapping system for lidar and image data acquisition from the road environment, and fully automatic methods for classifying and modelling the road environment. Classified and modeled objects include road markings, curbstones, road surface, and poles. Based on the experimental tests, the integrated system proved to be suitable for this use and it could provide detailed road environment data reliably with sub-decimeter accuracy. II. ROAMER SYSTEM The thread in the design of the ROAMER was to build a versatile, multipurpose platform for research purposes, not only limited to the fields of urban and road mapping. The hardware sub-systems in the ROAMER, seen in Fig. 1, include 1) laser scanner, 2) GPS-INS navigation equipment, 3) camera system, 4) synchronization electronics and 5)

2009 Urban Remote Sensing Joint Event mechanical support structure. The first three (1-3) of the subsystems represent the current state-of-the-art in their respective fields, and were purchased from third party companies. The synchronization electronics was specified and designed in the FGI, and manufactured by a sub-contractor. The platform allows the operator to put the scanner in eight different scanning positions to meet some application specific purposes. Figure 2. Components of a mobile mapping system. III. DATA A. Navigation data The SPAN system receiver takes care of the real time navigation data processing and storage. IMU aided position estimation for event logs are recorded in own logs, separate from the high frequency IMU raw data and GPS observations. In the real-time navigation, the GPS reference station is erected to a benchmark point to deliver correction data for the ROAMER system receiver. Data is also stored in the reference station memory to be used later in post-processing. If the trajectory information is computed in the post-processing, the reference data can be from virtual station, or any of the permanent reference stations, and the adequate data is downloaded from commercial service providers to do the postprocessing. Figure 1. Side views of the platform design and instrumentation. The software component of the ROAMER system deals with the 1) GPS-INS data collection, and 2) real-time or 3) post-processing tools to provide GPS-INS trajectory for the 4) georeferencing of laser and image data. Further pre-processing of the laser data includes 5) noise filtering and 6) map projection before the data is put into the 7) modelling process. Different modelling tasks are greatly affected by the application, and could in road and urban environment deal with building and road structure modelling, detection of road markings and traffic signs, inventory of light poles, fences, and other urban structures. Support component of the ROAMER system takes care of the 1) maintenance and calibration of each sub-system, 2) system calibration to integrate the sub-systems tightly in to a common coordinate frame, 3) maintenance and testing of the system integration, provides 4) GPS base reference and correction data for the navigation data processes, and 5) other relevant target reference data for validation and error assessment of the acquired point and image data. The chart in Fig. 2 describes the hardware needed for mobile mapping, the post-processing steps to refine the raw data in to an applicable format for automated modelling processes and analysis, support activities and reference data essential to reliable data acquisition and quality. IMU trajectory data is recorded, and stored in INSPVA files with operator-selected frequency maximum data rate for the SPAN system in use is 100 Hz. The data contains time (GPS week and second), position, velocity and attitude data of the platform. The trajectory information is used to derive position and attitude information for each point and image acquired with the scanner and cameras in the post-mission georeferencing phase. The data synchronization, i.e. between navigation and mapping sensors, works in a way that the system receiver logs the time signals sent by the TLS for each profile. Event logger data thus contains the unique time stamps for each of the recorded TLS profiles. In the georeferensing phase each data point in the profile the sensor location and attitude are interpolated from the time and trajectory data. Synchronization data accumulates into files at a rate corresponding to the scan frequency. The specialized synchronization device, bi-trigger synchronizer, takes care of splitting the signal frequency, and delivering the signals into the two input channels for the event logging. Synchronizer also delivers external triggering signals to the cameras for synchronized image acquisition. B. TLS data TLS data is stored in files containing operator-selected number of profiles, i.e. data blocks. In the test phase the file split parameter was set to 1000 profiles per file to keep the

amount of data reasonable in the post-processing point of view. One TLS data block can be seen in Fig. 3 as an intensity image, where each of the columns describe one laser profile. On the left of the image, where the scan starts, a few profiles from a non-moving state can be seen. In the middle of the image the platform takes a right angle turn to the right and rolls into a road tunnel, where an oncoming bus and a van pass the scanner FOV. Figure 3. Intensity image produced by the ROAMER from a street corner and a road tunnel. Data storage accumulation at different platform velocities for 15 Hz and 30 Hz scan frequencies was estimated, and is summarized in Table I. Approximate size of one 1000 profile block of data is 20 MB and 10 MB in native scanner data format (FARO fls), respectively. Total time (duration) of the data acquisition and storage size is approximated for 100 km of road scanning. From these we can expect maximum data accumulation rate of 60 MB/km for the TLS system. TABLE I. DATA STORAGE CALCULATION FOR TLS DATA. Velocity Duration Number of Blocks Storage size Scan frequency Native ASCII km/h m/s s 15 30 GB Hz Hz 20 6 17000 260 570 6 200 30 8 12500 200 420 4 100 40 11 9100 140 300 3 65 50 14 7200 110 240 2,5 50 60 17 5900 90 200 2 40 In the pre-processing the TLS data is read to the Matlab directly from the native fls-files. After georeferencing and filtering the data is delivered to modelling phase in two formats. The first stores the data points in an array with ROW, COL, X, Y, Z and intensity information in each line, and where ROW and COL are indexes to the order of points. This format is useful for exporting/importing the data to other point processing software. The other format stores the data in to a three-dimensional array (4 x ROW x COL), i.e. an image of four layers, where the X Y Z and intensity data of each point is stored to the corresponding layers. C. Image data Image data is stored in jpg-format, and named with running numbering to maintain the image acquisition order. The image exposures are extracted from the MARKPOS-data and stored in a separate file. Fig. 4 shows an image captured with the left looking camera aboard the ROAMER system. Figure 4. Street view seen from left looking camera. Image data was captured using the mode 7 with image size of 822 x 1088 pixels. The mode 7 allows image frame frequencies up to 8 Hz. The data format for delivering the image orientations is yet to be defined, but should contain the parameters for the exterior orientation (X, Y, Z, and rotation matrix) for each image. Camera model and calibration parameters are exported from the calibration software for making the automatic computation possible. IV. DATA PROCESSING A. Point data organization In order to efficiently process mobile mapping data we need to organize it. Laser data from the mobile mapping system is inherently organized as profiles consisting of consecutive points. This is a very efficient way to store the data as neighborhood relations between points can be easily deducted and the image-like format enables the use of established image processing methodology. At some point in processing the image-like arrangement of points becomes more of a burden, especially when multiple datasets are combined. Therefore, we need other ways to organize the data. One efficient method is to tile the data into regular cubes and store the data in these tiles. This is very helpful when local neighborhood of a point is required, e.g. in nearest neighbor operations and classification. B. Filtering When 3D scene is mapped by an MMS, a huge number of points is produced by the laser scanner. Due to the operation principle of the laser scanner in use, the acquired data contains noisy points together with the relevant object points. Before further data classification, it is necessary to remove the disturbance of noisy points. Different methods for noise filtering have been developed and tested for the ROAMER data. Such methods are dark point and column-wise stray point filters, and r method based on number of points within cube to distinguish noisy points. In this method the data indexing derived in the data organization phase is used, and cubes with points less than the threshold are removed. These methods are discussed more in [2] and [4].

Isolated point filtering was tested on the ROAMER data. The filter searches for points within radius r from the center point, and if n points or more are found, the center point, it is preserved. The testing considered eight different parameter sets for n and r. Table II summarizes the parameters and results for the testing. The Noise present column gives the description about the amount of noise in the data after the filtering, and the Data points column indicates the removal of correct data points. These indicators are based on visual analysis of the data filtering result. In Fig. 5 we show the result of the isolated point filtering for four of the cases. The isolated point filtering for a data block of 7110000 points is fast, taking only approximately 6-15 seconds for the test cases, and depending on the parameter setting. The processing time is stretched with the increasing number of points n and radius r. Figure 5. Isolated point filtering of the highway data: (a) original, (b) n=10, r=1, (c) n=15, r=1, and (d) n=25, r=2. TABLE II. ISOLATED POINT FILTERING. Case n r [m] Reduction Noise present Data points 1 10 2 336000 Major Not at all 2 10 1 1204000 Minor Minor 3 15 2 500000 Major Not at all 4 15 1 1391000 Medium Medium 5 25 2 806000 Minor Not at all 6 25 1 1661000 Not at all Major 7 50 2 1263000 Minor Minor 8 50 1 2055000 Not at all Major of the isolated point filtering is dependent on the data acquisition parameters, mainly on the driving speed and angular resolution. Distance adaptive filtering for detecting isolated points might help in removing fringe noise from the data with smaller n. C. Pole detection Poles form group of objects present everywhere in the urban environment. Their importance within the urban context is high as the usually support equipment and wires vital for urban traffic and power supply. A whole new approach is to use poles as reference objects for correcting the navigation solution of a MMS, i.e. to aid in the georeferencing and extraction of road side objects. Demands for an algorithm to detect such objects are: Automatic, fast, and efficient. Extraction of pole-like objects in MMS using laser data has not been subject of many studies so far. However, in [3] vertical scan lines were used in MMS application and utility poles were searched for by detecting point groups which form independent vertical lines in profiles. The drawback of the approach was that because vertical scan lines were used, maximum one sweep was received from each pole in normal driving speeds. The faster the scanner moved the smaller the resolution of the laser data in driving direction was achieved. Thus probability of catching a pole with a high speed becomes small. Also poles close to buildings faces are missed. In [5] a method for the extraction of traffic signs and signals in the close neighborhood of the road was introduced. First points were clustered to different objects and then clusters were projected to a vertical plane. Finally poles were detected based on their elongated form in the projection. 1) Algorithm The developed pole detection algorithm makes use of the row and column, i.e., profile information of the scanner. Fig. 6 illustrates well the data characteristics on a pole object. If one can find a sweep of the pole in some profile then he or she knows that there has to be another sweep either below the current sweep in the next profile or above in the previous profile or both of these. This way all the sweeps can be found. Isolated point filtering could preserve points acquired with bad measurement geometry if the search radius r was 2 m for all n tested. Smaller r could be applied if the n is reduced. In general the smaller r could remove the noise more efficiently, but the sparse data points from real object became more easily wasted in these cases. Based on visual interpretation of the filtering results, a parameter set performing relatively well was found. The filtering with number of points n=25 and radius r=2 m, could remove most of the noise, and still not remove point data from the road surface. However, it is expected that the performance Figure 6. A pole picked from the point cloud. Note the nameplate attached to the pole. Algorithm proceeds in three stages. In the first stage possible sweeps of the poles are extracted from the data.

Extraction is done by grouping the points of each profile based on the distances of adjacent points along the profile and removing too long groups. In the second stage extracted sweeps are clustered to form poles by comparing the sweeps found in consecutive profiles. In the third stage actual poles are separated from other clusters which are found during the clustering, for example, from trees leaves and buildings walls because of shadowing from objects between the wall and the scanner. Separation is done by removing too short clusters and clusters that have too many points around them in their close neighborhood. 2) Results One data block consisting of about 450 meters of road and about seven million points was used for the testing of the algorithm. In the data set the road was driven only in one direction. The data set was checked manually and totally 110 poles and 110 trees were found. From now on manually detected poles and trees are called reference data set. Only those trees in which a trunk was visible in the data were taken into the reference data set. In the first stage of the algorithm sweeps with less than five points and whose length was longer than 40 cm were removed. The algorithm was implemented in Matlab software and it took about six minutes to run it with the used data set. Classification result of the algorithm can be found in Table III. 60 percent of the poles and 65 percent of the trees in the reference data set were detected by the algorithm. The correctness of the classification, i.e., proportion of number of true positives to number of all clusters found by the algorithm was 81 percent when poles and trees were taken together. The average of correctness and completeness values or the mean accuracy was 71 percent for poles and trees together. More than half of the undetected poles and trees had missing sweeps or some sweeps contained less than five points and were removed in the first stage of the algorithm. The maximum allowed length for one sweep (40 cm) turned out to be too small and few poles and trees were missed because of this. Missed and detected poles and trees and the route of the scanner can be found in Fig. 7. It can be seen that targets closer to the road are easier to detect, as expected. If only targets closer than 15 meters from the route of the scanner were taken into account, 84 % of the trees and 77 % of the poles were detected. Correctness in that case was 88 % and mean accuracy 84 %. Better classification result is due to higher point density and lesser effect of shadowing from other objects. Figure 7. An upward view of the manually found poles and trees which were detected by the algorithm (green dots) and which were missed (red dots). The route of the scanner is plotted as blue line. In the future it will be under investigation how well trees can be separated from other poles, for example, from lampposts and whether it is possible to utilize data from a twoway drive of the same area in the algorithm. D. Road modeling In FGI, early development and application of automated road modeling for pavement and painting extraction from the ROAMER derived mobile laser scanning data is presented in [2]. The processes described there in more detail utilize the laser derived intensity information for detection of zebra lines and other painted road markings, and iterative triangulation methods for curbstone and pavement delineation. Fig. 8 delineates the point data classification process. Currently automatic recognition of buildings, trees, poles, road surface from MMS data is available as a toolbox for Matlab. Fig. 9 shows the result for automatic pavement and road marking modeling, and detected curbstones. TABLE III. CLASSIFICATION RESULT OF THE ALGORITHM. Completeness Poles 60 % Trees 65 % Correctness Mean accuracy 81 % 71 % The algorithm detected totally 37 false positives, i.e., clusters that did not belong to any pole or tree in the reference data set. 22 of these were poles, trees or pole-like point clusters which were missed in the manual check. The cause for more than half of the remaining 15 false positives was that in the data set the road was driven only in one direction. Planar faces which were perpendicular to the driving direction and which were seen in the opposite of the driving direction from the car had low point density. Thus some corners in the walls, for example, next to a window looked like poles to the algorithm. It might be possible to remove these if the road in the data set was driven also in the opposite direction. Figure 8. Object classification process flow.

Figure 9. Street model with pavement surface (TIN), curbstones (black) and road markings automatically extracted from the ROAMER data. The completeness and correctness of the produced model were assessed with a ground reference obtained using manual classification of the dataset. The acquired modelling accuracies using the same data are shown in Table IV. It is often unclear where the real edge should be and, therefore, the error could as well be in the reference data. Some lines were completely missed because they were too short to be found by the algorithm as it assumes a minimum line length of about 8 m. TABLE IV. ASSESSMENT OF MODELLING ACCURACY. Target Completeness Correctness Mean accuracy Lines 86.6 % 74.6 % 80.6 % Zebra lines 95.1 % 89.5 % 92.3 % Curbstones 73.9 % 85.6 % 79.7 % Later on in processing, when ground points have been classified, the rest of the points can be clustered into separate objects and stored as clusters of points. This clustering makes use of region growing, which in turn takes advantage of the data tiling. After clustering the data, the clusters can then be classified into multiple classes such as buildings, trees, bushes, poles and cars. The classification is based on different attributes that are calculated for each cluster. V. CONCLUSIONS We have developed and built a mobile mapping system for road and urban environment mapping. The system consists of a navigation system, TLS originated laser scanner, camera system, and in-house developer synchronization electronics on a sturdy integration platform. In the paper we present the current status of the ROAMER system, describe the components that are needed for successful operation of an MMS, and describe the data processing methods for filtering and classification of the point data from road environment. The ROAMER system has been successfully employed in various research projects in the fields of forestry, urban mapping, and environmental studies. The ROAMER can measure its surroundings with point measurement frequency of 120 khz, and up to 48 Hz profile measurement frequency. Its easy adaptation to different surveying applications has proven to be of great value. Based on visual interpretation of the filtering results for the isolated point filtering, a parameter set performing relatively well for the ROAMER data was found. The filtering could remove most of the noise, and still not remove point data from the road surface. However, it is expected that the performance of the isolated point filtering is dependent on the data acquisition parameters, mainly on the driving speed and angular resolution. Distance adaptive filtering for detecting isolated points might help in removing fringe noise from the data with smaller n. Classified and modeled objects include road markings, curbstones, road surface, and poles. Based on the experimental tests, the integrated system proved to be suitable for this use and it could provide detailed road environment data reliably with sub-decimeter accuracy. We believe that in the future, lidar based mobile mapping will be used considerably for urban and road environment modelling, as well as in many other applications in the fields of construction, forestry, railways, and even in environmental modelling and monitoring e.g. hydrology and glaciology. REFERENCES [1] A., Kukko, C.-O., Andrei, V.-M. Salminen, H., Kaartinen, Y., Chen, P., Rönnholm, H., Hyyppä, J., Hyyppä, R., Chen,H., Haggrén, I., Kosonen, and K. Čapek, Road environment mapping system of the Finnish Geodetic Institute FGI Roamer. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 36(3/W52), pp. 241-247. [2] A., Jaakkola, J., Hyyppä, H., Hyyppä, and A. Kukko, Retrieval algorithms for road surface modelling using laser-based mobile mapping, Sensors, vol. 8, pp. 5238-5249. [3] D., Manandhar, and R., Shibasaki, Feature extraction from range data, Proceedings of the 22 nd Asian Conference on Remote Sensing, vol. 2, pp. 1113-1118. [4] A. Kukko, Road Environment Mapper - 3D data capturing with mobile mapping, Licentiates thesis, Helsinki University of Technology, 2009. [5] Y.-Z., Chen, H.-J., Zhao, and R., Shibasaki, A Mobile system combining laser scanners and cameras for urban spatial objects extraction, Proceedings of the Sixth International Conference on Machine Learning and Cybernetics, Hong Kong, vol. 3, pp. 1729 1733, 19-22 th August 2007.