Ford Campus Vision and Laser Data Set

Transcription

1 1 Ford Campus Vision and Laser Data Set Gaurav Pandey*, Ryan Eustice* and James McBride** *University of Michigan, Ann Arbor, MI **Ford Motor Company Research, Dearborn, MI Abstract This paper describes a dataset collected by an autonomous ground vehicle testbed, based upon a modified Ford F-250 pickup truck (Fig 1). The vehicle is outfitted with, a professional (Applanix POS LV) and consumer (Xsens MTI-G) Inertial Measuring Unit (IMU); along with a 3D Velodyne lidar scanner, two push-broom forward looking Riegl lidars, and a Point Grey Ladybug3 omnidirectional camera system. Here we present the timeregistered data from these sensors mounted on the vehicle, collected while driving the vehicle around the Ford Research campus and downtown Dearborn, Michigan during August-November The vehicle path trajectory in these dataset contain several large and small-scale loop closures, which makes it useful for testing various state of the art computer vision and SLAM (Simultaneous Localization and Mapping) algorithms. be useful in generating rich textured 3D maps of the environment for navigation purposes. Here we present the dataset collected by this vehicle, while driving in and around the Ford research campus and downtown Dearborn in Michigan. The dataset includes various small and large loop closure events, ranging from feature rich downtown areas to featureless empty parking lots. The most significant aspect of this dataset is the precise co-registration of the laser data with the omnidirectional camera imagery data thereby adding visual information to the structure of the environment as obtained from the laser data. This fused vision data along with odometry information constitutes an appropriate framework for mobile robot platforms to enhance various state of the art computer vision and robotics algorithms. We hope that this dataset will be very useful to the robotics and vision community and will provide new research opportunities in terms of using the image and laser data together, along with the odometry information. Fig. 1. Test vehicle showing sensor arrangement I. INTRODUCTION University of Michigan and Ford Motor Company have been collaborating on autonomous ground vehicle research since the 2007 DARPA Urban Grand Challenge, and through this continued collaboration have developed an autonomous ground vehicle testbed based upon a modified Ford F-250 pickup truck. This vehicle was one of the finalists in the 2007 DARPA Urban Grand Challenge and showed capabilities to navigate in the mock urban environment [1], which included moving targets, intermittently blocked pathways, and regions of denied-global positioning system (GPS) reception. This motivated us to collect large scale visual and inertial data of some real world urban environments, which might II. SENSORS We used a modified Ford F-250 pickup truck as our base platform. Although, the large size of the vehicle might look like a hindrance in the urban driving environment, but it has proved useful because it allows strategic placement of different sensors. Moreover, the large space at the back of the truck was sufficient enough to install 8 quad-core processors along with the cooling mechanism. The vehicle was integrated with the following perception and navigation sensors: A. Perception Sensors 1) Velodyne HDL 64E lidar: The HDL-64E has two blocks of lasers each consisting of 32 laser diodes aligned vertically, resulting in an effective 26.8 degree vertical field of view. The entire unit can spin about its vertical axis at speeds up to 900 rpm (15 Hz) to provide a full 360 degree azimuthal field of view. The maximum range till the sensor can see is 120m and it captures about 1 million range points per second. We captured our dataset with

2 2 the laser spinning at 10 Hz. More details about the sensor can be obtained from [2]. 2) Pointgrey Ladybug3 omnidirectional camera: The Pointgrey Ladybug3 (LB3) is a high resolution omnidirectional camera system. It has six 2- Megapixel (1600x1200) cameras, with five CCDs positioned in a horizontal ring and one positioned vertically, that enable the system to collect video from more than 80% of the full sphere. The camera can capture images at multiple resolutions and multiple frame rates, it also supports hardware jpeg compression on the head. We collected our dataset at half resolution (i.e. 1600x600) and 8 fps in raw format(uncompressed). The details of the sensor are available from the manufacturers website [3]. 3) Riegl LMS-Q120 lidar: Two Riegl LMS Q120 LIDARs each having an 80-deg FOV with very fine (0.02 deg per step) resolution are installed at the front of the vehicle [4]. B. Navigation Sensors 1) Applanix POS-LV 420 INS with Trimble GPS: POS LV is a compact, fully integrated, turnkey position and orientation system, utilizing integrated inertial technology to generate stable, reliable and repeatable positioning solutions for land-based vehicle applications. It gives position, orientation, velocity, angular rate and accelerations at a rate of 100 Hz [5]. 2) Xsens MTiG: The MTi-G is a miniature size and low weight 6DOF Attitude and Heading Reference System (AHRS). The MTi-G contains accelerometers, gyroscopes, magnetometers in 3D, an integrated GPS receiver, a static pressure sensor and temperature sensor. Its internal low-power signal processor provides real time and drift-free 3D orientation as well as calibrated 3D acceleration, 3D rate of turn, 3D earth-magnetic field, 3D position and 3D velocity data at maximum update rate of 120 Hz [6] All the sensors are distributed across the 8 quad-core processors installed at the back of the truck, in order to minimize the latency in data capture. As sensor data is received by the host computers, it is timestamped using the host computer clock along with the timestamp associated with the hardware clock of the device. This sensor data available on the host computer is then packaged into a Lightweight Communication and Marshalling (LCM) [7] packet and is transmitted over the network using multicast UDP. This transmitted data is captured by a logging process, which listens for such packets from all the sensor drivers, and stores them on the disk in a single log file. The data recorded in this log file is timestamped again which allows a synchronous playback of the data later on. III. SENSOR CALIBRATION All the sensors (perceptual and navigation) are fixed to the vehicle and are related to each other by a fixed transformation. This transformation which allows the reprojection of any point from one coordinate frame to the other was calculated for each sensor. The coordinate frame of the two navigation sensors (Applanix and Mtig) coincide and is called the body frame of the vehicle and all other coordinate frames are defined with respect to the body frame. Assuming the body frame to be the fixed reference frame the relative transformation of the remaining sensors is calculated either manually or automatically. Here we use the Smith, Self and Cheeseman s [8] notations to represent the 6DOF pose of a sensor coordinate frame. The calibration procedure and the relative transformation between the different coordinate frames is described below. 1) Relative transformation between Velodyne laser scanner and body frame (X bl ): This transformation is obtained from the vehicle CAD model. We have a very accurate CAD model of the vehicle which allows us to precisely obtain the position and orientation of the velodyne laser scanner with respect to the fixed body frame. The transformation thus obtained from the CAD model is: X bl = [2.4m, 0.01m, 2.3m, 180 o, 0 o, o ]. 2) Relative transformation between Velodyne laser scanner and Ladybug3 camera head (X hl ): This transformation allows us to project any 3D point in laser reference frame into the camera head s frame and thereby into the corresponding camera image. The calibration procedure requires a checkerboard pattern to be viewed simultaneously from both the sensors. The 3D points lying on the checkerboard pattern in laser reference frame and the normal to the plane in camera reference frame, obtained from the image using Zhang s method, constrains the rigid body transformation between the laser and the camera reference frame. We can formulate this constrain as a non-linear optimization problem to solve for the optimum rigid body transformation. The details of the procedure can be found in [9]. The transformation obtained using the method described above is-: X hl = [0.34m, 0.01m, 0.42m, 0.02 o, 0.03 o, o ]. 3) Relative transformation between Ladybug 3 camera head and body frame (X bh ): Once we have X bl and X hl the transformation X bh can be calculated using Smith et al [8] compounding operation-: X bh = X bl ( X hl ) (1)

3 3 The transformation thus obtained is-: Xbh = [2.06m, 0m, 2.72m, 180o, 0.02o, 0.8o ]. Main Directory Pose Applanix.log SCANS IMAGES VELODYNE LCM LCM 00.log LCM 01.log Scan1.mat Pose Mtig.log velodyne data 00.pcap velodyne data 01.pcap Scan2.mat Timestamp.log Gps.log PARAM.mat Scan3.mat Cam 0 Cam 1 Cam 2 Cam 3 Cam 4 Fig. 3. The trajectory of the vehicle in one trial around downtown Dearborn. Here we have plotted the GPS data coming from the Trimble on the Google map. FULL Fig. 2. The directory structure containing the dataset. The rectangular blocks represent folders. IV. DATA C OLLECTION The data was collected around the Ford research campus area and downtown area in Dearborn, Michigan, which we will refer to as the test environment in the rest of the paper. It is an ideal dataset representing urban environment and will be very helpful to researchers working on autonomous navigation in an unstructured urban environment. We drove the test vehicle mounted with all the sensors around the test environment several times covering different areas. We call each data collection exercise to be a trial. In every trial we collected the data keeping in mind the requirements of state of the art SLAM algorithms, thereby covering several small and large loops in our dataset. A sample trajectory of the vehicle in one of the trials around downtown Dearborn is shown in Fig 3. The unprocessed data consists of three main files for each trial. One file contains the raw images of the omnidirectional camera, captured at 8fps and the other file contains the timestamps of each image measured in microseconds since 00:00:00 Jan 1, 1970 UTC. The third file contains the data coming from remaining sensors (perceptual/navigational). This file stores the data in a LCM log file format similar to that described in [10]. The unprocessed data consist of raw spherically distorted images from the omnidirectional camera and raw point cloud without accounting for the vehicle motion. Here we present a set of processed data with MATLAB scripts that allow easy access of the dataset to the users. The processed data is organized in folders and the directory structure is as shown in Fig 2. The main files and folders are described below: 1) LCM: This folder contains the LCM log file corresponding to each trial. Each LCM log file contains the raw 3D point cloud from the velodyne laser scanner, the lidar data from Reigal LMS-Q120 and the navigational data from the navigational sensors described in section II. We provide software to playback this log file and visualize the data in an interactive graphical user interface. 2) Timestamp.log: This file contains the unix timestamp, measured in microseconds since 00:00:00 Jan 1, 1970 UTC, of each image captured by the omnidirectional camera during one trial. 3) Pose-Applanix.log: This file contains the 6DOF pose of the vehicle along with the timestamp provided by the Applanix POS-LV 420 INS, during one trial. 4) Pose-Mtig.log: This file contains the 6DOF pose of the vehicle along with the timestamp provided by the Xens MTiG, during one trial. 5) Gps.log: This file contains the GPS data (latitude/longitude/attitude) of the vehicle provided by the Trimble GPS, during one trial. 6) IMAGES: This folder contains the undistorted images captured from the omnidirectional camera system during one trial. The folder is further divided into sub-folders containing images corresponding to individual camera of the omnidirectional camera system. This folder also contains a folder named FULL which contains the undistorted images stacked together in one file as shown in Fig??. 7) PARAM.mat: This contains the intrinsic and extrinsic parameters of the omnidirectional camera system. This is a (1 5) array of structures with

4 4 Fig. 4. The left panel shows the spherically distorted image obtained from the ladybug camera. the right panel shows the corresponding undistorted image obtained after applying the transformation provided by the manufacturers following fields: a) PARAM.K: This is the (3 3) matrix of the internal parameters of the camera. b) PARAM.R, PARAM.t: These are the (3 3) rotation matrix and (3 1) translation vector respectively which transforms the 3D point cloud from laser reference system to camera reference system. c) PARAM.MappingMatrix: This contains the mapping between the distorted and undistorted image pixels. This mapping is provided by the manufacturers of the Ladybug3 camera system and it corrects the spherical distortion in the image. A pair of distorted and undistorted images from the camera is shown in Fig 4. 8) SCANS: This folder contains 3D scans from the Velodyne laser scanner, motion compensated by the vehicle pose provided by Applanix POS-LV 420 INS. Each scan file in this folder is a MATLAB file (.mat) which can be easily loaded into the MATLAB workspace. The structure of individual scan once loaded in MATLAB is shown below: a) Scan.XYZ: is an array (3 N ) of motion compensated 3D point cloud represented in Velodyne s reference frame (described in section III). Here N is the number of points per scan which is typically 80,000-1,00,000. b) Scan.timestamp laser: is the unix timestamp measured in microseconds since 00:00:00 Jan 1, 1970 UTC for the scan captured by the Velodyne laser scanner. c) Scan.timestamp camera: is the unix timestamp measured in microseconds since 00:00:00 Jan 1, 1970 UTC for the closest image captured by the omnidirectional camera. d) Scan.image index: is the index of the image which is closest in time to this scan. e) Scan.X wv: is the 6DOF pose of the vehicle in world reference system when the scan was captured. f) Scan.Cam: is a (1 5) array of structures corresponding to each camera of the omnidirectional camera system. The format of this structure is given below: i) Scan.Cam.points index: is a (1 m) array of index of the 3D points in laser reference frame, in the field of view of the camera. ii) Scan.Cam.xyz: is a (3 m) array of 3D points (m < N ) in camera reference system, in the field of view of the camera. iii) Scan.Cam.pixels: This is a (2 m) array of pixel coordinates corresponding to the 3D points projected onto the camera. 9) VELODYNE: This folder has several.pcap files containing the raw 3D point cloud from velodyne laser scanner in a format which can be played back at desired frame rate using our playback tool. This allows the user to quickly browse through the data corresponding to a single trial. Fig. 5. The top panel is a perspective view of the Velodyne lidar range data, color-coded by height above the estimated ground plane. The bottom panel shows the above-ground-plane range data projected into the corresponding image from the Ladybug cameras. We have provided MATLAB scripts to load and visualize the dataset into MATLAB workspace. We have tried to keep the data format simple so that it can be easily used by other researchers in their works. We have also provided some visualization scripts (written in MATLAB) with this dataset that allow the reprojection of any scan on the corresponding omnidirectional

5 5 imagery as shown in Fig 5. We have also provided a C visualization tool that uses OpenGl to render the textured point cloud as shown in Fig 6. Fig. 6. Screenshot of the viewer showing the textured point cloud [4] RIEGL. Mobile scanning. [Online]. Available: [5] Applanix. Land, pos lv. [Online]. Available: [6] Xsens. Mti-g a gps aided mems based inertial measurement unit (imu) and static pressure sensor. [Online]. Available: [7] A. Huang, E. Olson, and D. Moore, Lightweight communications and marshalling, [Online]. Available: [8] R. Smith, M. Self, and P. Cheeseman, A stochastic map for uncertain spatial relationships, in Proceedings of the 4th international symposium on Robotics Research. Cambridge, MA, USA: MIT Press, 1988, pp [9] G. Pandey, J. McBride, S. Savarese, and R. Eustice, Extrinsic calibration of a 3d laser scanner and an omnidirectional camera, in 7th IFAC Symposium on Intelligent Autonomous Vehicles, 2010, accepted, To Appear. [10] A. S. Huang, M. Antone, E. Olson, L. Fletcher, D. Moore, S. Teller, and J. Leonard, A high-rate, heterogeneous data set from the darpa urban challenge, in International Journal of Robotics Research, 2010, under review. V. SUMMARY We have presented a time-registered vision and navigational dataset of unstructured outdoor environments. We believe that this dataset will be highly useful to the robotics community especially to those who are looking toward the problem of autonomous navigation of ground vehicles in an a-priori unknown environment. The dataset can be used as a bench mark for testing various state of the art computer vision and robotics algorithms like SLAM, ICP (Iterative Closest Point), 3D object detection / recognition etc. Thus its a useful dataset which not only allows the researchers to test the existing algorithms but also provides a new perspective to the problems, in terms of utilizing the image (2D) and laser (3D) data together. ACKNOWLEDGMENTS We are very thankful to the Ford Motor Company for taking up this research initiative and providing us with all the hardware needed for data collection. We are also thankful to Prof. Silvio Savarase of University of Michigan for his continuous feedback throughout the data collection process. REFERENCES [1] J. R. McBride, J. C. Ivan, D. S. Rhode, J. D. Rupp, M. Y. Rupp, J. D. Higgins, D. D. Turner, and R. M. Eustice, A perspective on emerging automotive safety applications, derived from lessons learned through participation in the darpa grand challenges, in Journal of Field Robotics, 25(10), 2008, pp [2] Velodyne, Velodyne s hdl 64e: A high definition lidar sensor for 3d applications, October [3] Pointgrey. (2009) Spherical vision products: Ladybug3. [Online]. Available: