Automatic camera to laser calibration for high accuracy mobile mapping systems using INS

Transcription

1 Automatic camera to laser calibration for high accuracy mobile mapping systems using INS Werner Goeman* a+b, Koen Douterloigne a, Sidharta Gautama a a Department of Telecommunciations and information processing (TELIN), Ghent University, St. Pietersnieuwstraat 41, B-9000, Ghent, Belgium; b Geodata & Mobile Mapping group, Grontmij Belgium NV, Meersstraat 138, B-9000, Ghent ABSTRACT A mobile mapping system (MMS) is a mobile multi-sensor platform developed by the geoinformation community to support the acquisition of huge amounts of geodata in the form of georeferenced high resolution images and dense laser clouds. Since data fusion and data integration techniques are increasingly able to combine the complementary strengths of different sensor types, the external calibration of a camera to a laser scanner is a common pre-requisite on today's mobile platforms. The methods of calibration, nevertheless, are often relatively poorly documented, are almost always time-consuming, demand expert knowledge and often require a carefully constructed calibration environment. A new methodology is studied and explored to provide a high quality external calibration for a pinhole camera to a laser scanner which is automatic, easy to perform, robust and foolproof. The method presented here, uses a portable, standard ranging pole which needs to be positioned on a known ground control point. For calibration, a well studied absolute orientation problem needs to be solved. In many cases, the camera and laser sensor are calibrated in relation to the INS system. Therefore, the transformation from camera to laser contains the cumulated error of each sensor in relation to the INS. Here, the calibration of the camera is performed in relation to the laser frame using the time synchronization between the sensors for data association. In this study, the use of the inertial relative movement will be explored to collect more useful calibration data. This results in a better intersensor calibration allowing better coloring of the clouds and a more accurate depth mask for images, especially on the edges of objects in the scene. Keywords: mobile mapping, indirect calibration, laser 3D monoplotting, external calibration, relative calibration, cam calibration. 1. INTRODUCTION A land-based mobile mapping system (MMS) is a state-of-the-art system to collect visual and non-visual information from the environment on a large scale and as fast and complete as possible. In order to accomplish this challenging task, the mobile platform is equipped with several complementary sensors to capture as much information as possible of the surrounding scene. The current technology allows to navigate the platform with high velocities through the requested regions, capturing a huge amount of data to be processed offline for a variety of purposes such as asset management, road inspection, traffic sign detection, etc. Modern MMS systems generates highly redundant sensor data which is exploited by the customized data fusion algorithms with near real-time capabilities [1]. It gives the end-user more and more feedback about completeness and quality assurance. However, the quality of the extracted data highly depends on the quality of the calibration of the sensors. Two sets of parameters can be identified for each sensor, intrinsic and extrinsic. Both parameter sets can be estimated separately and have already been studied extensively. Also in many experiments, extrinsic calibration parameters are treated as already known through pre-calibration, while few details can be found on the used methodology. However, it is common knowledge that the process of extrinsic calibration is labor intensive. For this reason, off-the-shelf platforms such as [4][5][6] try to eliminate the calibration step for the end user by providing an all in one sensor package which can be mounted on a mobile platform. This eliminates the flexibility of a sensor configuration by preventing sensor adaptations customized for more specialized surveying and computer vision tasks, such as road inspection, surveys on rails, offshore operations, infrared heat inspection, visual odometry etc. *Werner.goeman@grontmij.be; Phone: ; Fax: ;

2 To fix these problems we present an extrinsic camera to laser calibration methodology which is automatic, robust, portable and easy to perform. This way, the calibration can be performed without time consuming procedures. There is no need for a target site or unpractical checkerboards under controlled lighting conditions, as presented in [2][9][10]. Only a one dimensional reference model such as a standard ranging pole is needed on which some reflectivity patches are added. The ranging pole can easily be setup to 5m in length while staying practical and portable compared to the checkerboards used in [2][11]. The focus of this work is on extrinsic camera calibration relative to the laser scanner. It calculates the relative rotation and translation between the laser scanner and the pinhole camera. The INS will be used to collect more useful data, thus improving the intersensor calibration. This methodology is preferred when fusing the color information of the camera with the laser range data, resulting in a colored laser point cloud and image depth masks with better accuracy. The procedure is an automated process and can be performed by a non-expert, reducing the operational cost. In previous work, a methodology was presented how direct external camera calibration can be performed for a MMS using an automated extraction of a one dimensional reference object in the field of view of the camera. Here, automated relative camera calibration is considered based on the same feature extraction presented in [8]. The experiments show how the relative process performs better for coloring laser point clouds and 3D monoplotting. Laser scanner and camera direct georeferencing 2. CALIBRATION METHODOLOGY Geometric camera calibration can be seen as the combination of intrinsic and extrinsic calibration, resulting in a set of intrinsic and a set of extrinsic parameters respectively. The intrinsic parameters describe the characteristics inherent to the sensor itself, such as focal length and central point, and do not change over time as long as the internal state of the sensor remains unchanged. Determining the camera s intrinsic parameters and the lens distortion can be done offline, separate from the external calibration, using e.g. the techniques in [3][9]. In this work, the internal calibration parameter set is considered known. This is a reasonable requirement, since experiments show that the internal parameter set does not degenerate over a period of several years when used intensively on a MMS. The extrinsic parameters, on the other hand, describe the relative rotation and translation between different sensor coordinate systems, and need to be recalculated every time the orientation or lever arm of the sensor in relation to the reference frame was changed. Direct georeferencing is defined as direct measurement of extrinsic orientation parameters, using positioning and orientation sensors, such as the Global Navigation Satellite System (GNSS) and the inertial navigation system (INS). It is the preferred and more recent approach to determine the extrinsic parameters of a camera in relation to the GPS/INS positioning system, i.e. lever arm and bore sight calibration. This way, the parameters of each image are determined directly by a combination of satellite and inertial navigation system measurements. Line and laser scanner systems direct georeferencing, on the other hand, is indispensible because of the needed extrinsic orientation information for each single measurement. For the camera direct georeferencing, a pinhole model is used. This way the projection from the 3D space to the image plane can be described by: or (1) Where P w (X w,y w,z w,1) are the coordinates of a 3D point in the world coordinate space W, p are the coordinates of the projection point in pixels, s is a scale factor. A is called a camera matrix, or a matrix of intrinsic parameters, and denotes a matrix gathering the extrinsic parameters (rotation and translation) of the camera. In (1), p is not the actual observed image point since virtually all imaging devices introduce a certain amount of nonlinear distortions. Among the nonlinear distortions, radial distortion is present and increasing along the radial direction from the center of distortion. It has been recognized to be the most severe. In this paper it is assumed that the intrinsic parameter set of the camera is known by an off-line calibration process such as the camera calibration algorithm presented in [10]. The intrinsic parameters define the projection from a point in the camera frame to the pixel coordinates in the image plane.

3 Figure 1. Left: camera frame O C in relation to INS frame O INS during direct georeferencing. Right: camera frame O C at time t 2 capturing point P. The laser scanner O L hits point P at time t 1. The transformation from world coordinate frame to the camera frame is defined by the following equations: ( ) ( ) ( ) (4) (2) (3) [ ], (5) and are the coordinates of the same point in respectively the camera frame, the GPS/INS frame and the world frame. Each rotation is defined by 3 parameters as shown in (4), and each translation is defined by three parameters as shown in (5). is the rotation from world frame to GPS/INS frame. This rotation is known by the positioning system, next to the position of the world reference frame origin in the INS frame, represented by translation. Thus, the position and orientation of the camera frame is defined by the six remaining parameters (t x,t y,t z,α,β,γ) of translation and rotation. These parameters can be estimated using 2D-3D correspondences. For this, a set of object points, their corresponding image projections, the camera matrix and the distortion parameters of the camera are required [12]. The extrinsic camera parameters set is calculated while minimizing the reprojection error, i.e. the sum of the squared distances between the observed image projections and the projected object points. The laser ranger direct georeferencing problem is less complex and is reduced to an absolute orientation problem since 3D measures are acquired directly by this active sensor in the sensor frame. Here, the extrinsic parameter set can be calculated using 3D-3D correspondences. A comparison of four major algorithms can be found in [13]. The mobile platform The platform used here is the MMS constructed by Grontmij Belgium NV. The main advantage of this platform is the flexibility for its use in a wide range of applications. The navigation sensor is an Applanix POS LV 420 tightly coupled GPS/INS system. The INS makes it possible to maintain the positional accuracy during GPS outages due to low satellite visibility, occurring in urban canyons, tunnels, etc.

4 Figure 2. The mobile platform from Grontmij Belgium NV mounted on car, boat and train The platform can handle up to 6 synchronized cameras and 4 laser scanners, a ground penetrating radar, an infrared scanner, a multibeam, and various other types of sensors. The cameras and laser scanners can be oriented conform the application requirements. The navigation data is processed offline together with the best available RINEX data. All sensors are synchronized with the navigation sensor for georeferencing. In the standard configuration setup, direct camera calibration is used conform the methodology presented in [8]. The accuracy of absolute measurements using triangulation is given in Table 1. Table 1 Performance of MMS showing the average error and standard deviation on absolute photogrammetric measurements Distance(m) Average error (cm) Std dev (cm) > The laser scanner mounted on the platform is a Riegl LMS-Q120. It measures point per second in a view angle of 80 degrees with an accuracy of 2.5cm. In the standard configuration, also the laser sensor was calibrated, using direct georeferencing. This results in absolute accuracies better than 4cm up to 30m distance of the scanner. Relative orientation parameter set between pinhole camera and laser scanner When both laser scanner and pinhole camera are calibrated using direct georeferencing, each of the sensor calibration extrinsic parameter set incorporates the error introduced by the inevitable limited accuracy and noise of the positioning sensors. Each point measurement is an extrapolation of the position of the sensor centers which makes each measurement highly sensitive for any type of error or inaccuracy during calibration. In some applications, a dedicated camera-laser sensor couple is used to augment the captured data. By integrating the sensor data, a depth mask can be added to the images or the intensity based grayscale laser point clouds can be converted to full RGB point clouds. In this case it is not desired to use direct georeferencing for both sensors, since the data fusion process will be subject to noise on the positioning data twice. To tackle this problem, the calibration of a pinhole camera in relation to a 2D laser scanner is studied here. The method explained above for calibrating the camera in relation to the INS, Figure 1(left), can be repeated, but now the object points will be collected in the laser sensor frame. The equation (3) is now replaced by:, with a marker reflection laser point in the laser scanner frame, and the unknown extrinsic camera parameter set. Together with equation (1), the correspondence between 3D laser point and 2D image pixel is made. However, there is a practical problem since the detection of the laser marker by the laser scanner and the capturing of the image with the ranging pole by the camera has to be synchronized. Both sensors must see the marker at the exact same moment in time in order to use the 2D-3D correspondence for calibration purposes. When using a 2D laser scanner, this is a very unpractical condition. Not only does the camera have to look in the same direction as the laser scanner, but also it is almost impossible to make sure the camera was not moving between the time when the laser scanner hits the marker, t=t 1 in Figure 3 (right), and the time a frame was captured by the camera, t=t 2. (6)

5 To solve this problem, the inertial navigation system will be used to compensate for the time difference. Since the laser scanner is already calibrated accurately using direct georeferencing, the relative motion of the laser scanning can be calculated. In Figure 3 (right), the marker is hit by the laser scanner at time t 1. The position of the marker in the laser scanner frame is given by, but the image is taken at time t 2, when the marker has position in the camera frame. To use a 2D-3D correspondence at time t 2, is needed, which can be easily calculated using the relative inertial movement of the INS system between time t 1 and t 2 and the laser calibration. Data acquisition The procedure to collect data for calibration has to be easy to perform and practical. Therefore, a standard ranging pole is used as a reference. The pole is set up and vertically leveled at a known ground control point. With the mobile platform a small survey around the pole is performed. This results in a set of georeferenced images and a 3D laser point cloud in which the pole is visible from different viewing angles and at different distances from the camera and the laser sensor. The ranging pole can be seen as a one dimensional sequence of colored segments in the image as is shown in Figure 3. Each transition in color of the ranging pole can be used as a reference point of which the world coordinates are known. On the ranging pole some markers can be added which can be used as high reflectivity reference points in the laser cloud. C L Figure 3: Left: One dimensional reference model. Middle: Top-down view of a typical calibration track, logged by an Applanix POS LV 420, where the MMS drives around the central ranging pole, marked by a cross. The pattern of overlapping ellipses enables the MMS to see the ranging pole from all angles. The overlaid squares measure about 5 x 5 meter each. The ranging pole is viewed from a distance of about 3 to 15 meter. Right: the configuration used for the experiment with cam C and laser L. 3. EXPERIMENT For the experiment, a MMS with configuration shown in Figure 3 was constructed. A 2 MP Scorpion Point Grey color camera with focal length of 4.8 mm and a Riegl LMS-Q120 laser scanner was mounted on the platform. The ranging pole was leveled on a known GCP and a survey capturing and scanning the pole from different viewing angles was performed. The extrinsic parameters set of both sensors were calculated using direct georeferencing. It is important to note that in order to calibrate the camera in relation to the laser scanner, it is not necessary to calibrate the camera using direct georeferencing. But here it is calculated to be able to compare the results. To calibrate the camera relative to the laser scanner, a set of 2D-3D laser point to image pixel correspondences are needed as explained above. The automatic ranging pole extraction tool described in [8] was used to collect useful frames from which the position of the reflecting markers in the image can be automatically detected, based on the invariance of the cross ratio for perspective transformations. Each of these 2D image coordinates of the markers are linked to the nearest (in time) laser point reflection in the laser point cloud, which is a 3D point in the laser frame. This set of 2D and 3D corresponding coordinates was used to calibrate the camera relative to the laser scanner. The maximum allowed time difference between laser marker reflection and frame capture was set to 2s. The data set exists out of frames selected from the survey making sure the ranging pole is in clear field of view of the camera. The time tag of each frame needs to be in a 2 second time interval of the time tag of at least 1 marker laser

6 reflection point. Using the INS inertial data, a 2D to 3D correspondence can then be made using data with different time tags. Based on viewing angle, a set of 36 marker reflection to image correspondences were collected and used for camera to laser calibration. Figure 4. An example of the calibration result. The white plus mark shows the ground truth representing the centre of the marker on the ranging pole. The black cross is the back projection of the 3D marker detected in the laser frame using the relative camera to laser calibration. The black o-mark is the back projection of the 3D marker using direct georeferencing for camera calibration. To analyze the results, the relative camera calibration was compared to the camera calibration using direct georeferencing in Table 2. From the survey, 331 frames were available showing the ranging pole. A set of 178 validation frames were selected with a time tag falling in the 2 seconds time interval of the nearest marker reflection. These frames show the ranging pole from different viewing angles, but were not used in the calibration process. For each of these frames, the ground truth was created by manually selecting the image coordinates of the markers on the ranging pole. The back projection of the 3D coordinates of the markers in these frames can now be compared to the ground truth using both the relative camera calibration and the calibration using direct georeferencing. A result of this procedure is given in Figure 4. Table 2. Results of the back projection error of relative camera to laser calibration and calibration using direct georeferencing Number of frames Number of points Average error(pixels) Maximum error(pixels) Direct georeferencing Relative camera to laser using INS calibration set validation set calibration set validation set When using direct georeferencing, an average pixel error of the back projected markers in the images of 5.4 was observed. The maximum pixel error was 11.8 pixels for the validation set. The method presented here reduces the maximum back projection error to 7.4 pixels, and the mean error to 4.0 pixels. The remaining error is mainly explained by the accuracy of the laser and the fact that the marker size is bigger than one pixel. 4. CONCLUSIONS In this paper, a new methodology was presented to improve and automate the extrinsic camera calibration in relation to a 2D laser range sensor. The calibration process is easy to perform, fast and automatic and uses only a standard ranging pole on a known ground control point. This eliminates the need for calibration grids or a target site used in other calibration methods. The use of inertial data makes it possible to collect more data by eliminating the need to capture the image and marker reflection at exact the same moment in time. A validation experiment shows a significant improvement in maximum and mean back projection error of 3D marker points in the image when comparing extrinsic camera calibrations using the presented method and camera calibration using direct georeferencing.

7 5. ACKNOWLEDGMENTS This work was made possible with the financial support of iminds. The authors would also like to thank Grontmij Belgium NV for the use of their mobile mapping data. REFERENCES [1] Brzezinska, D., Tutorial: Mobile mapping technology: paradigm shift and future trends, 7 th International Syposium on Mobile Mapping Technology (2011) [2] Zhang, Q., Extrinsic calibration of a camera and laser rangefinder, IEEE IROS, (2004) [3] Heikkilä, J.,Silvén,O., A Four-step Camera Calibration Procedure with Implicit Image Correction, 26th IEEE Conference on Computer Vision and Pattern Recognition, (1997) [4] MDL, Dynascan 3D mobile mapper, (11 July 2012) [5] TOPCON, IP-S2 HD Mobile Mapping System, (11 July 2012) [6] GeoVISAT, Data acquisition, (11 July 2012) [7] Hassan,T., El-Sheimy, N., System Calibration of Land-Based Mobile Mapping Systems, ISPRS, (2008) [8] Goeman, W., Douterloigne, K., Bogaert, P., Pires, R., Gautama, S., Automatic and robust external camera calibration for high accuracy mobile mapping, Proc. SPIE Optics and Photonics. San Diego, USA (2012) [9] Zhang, Z., A flexible new technique for camera calibration, IEEE Transactions on Pattern Analysis and Machine Intelligence 22, (2000) [10] Scheller, S., Westfeld, P., Ebersback, D., Calibration of a mobile mapping camera system with photogrammetic method, Proc ISPRS Vol XXXVI, 5 th International conference on mobile mapping technology (2007) [11] Geiger, A., Moosman, F., Car, O., Schutser, B., Automatic camera and range sensor calibration using a single shot, IEEE International Conf on Robotics and Automation, p (2012) [12] OpenCV, opencv. willowgarage.com/documentation/ camera_calibration_ and_3d_reconstruction.html [13] Eggert, D.W., Lorusso, A., Fisher, R.B., Estimating 3-D rigid body transformations: a comparison of four major algorithms, Machine Vision and Applications Vol 9, Issue 5-6, (March 1997)