SIFT-Based Localization Using a Prior World Model for Robot Navigation in Urban Environments

Transcription

1 SIFT-Based Localization Using a Prior World Model for Robot Navigation in Urban Environments H. Viggh and K. Ni MIT Lincoln Laboratory, Lexington, MA USA Abstract - This project tested the use of a prior world model for robot navigation in an urban environment where Global Positioning System (GPS) can have poor performance due to multi-path issues. A spatially sampled world model was used that consisted of Scale Invariant Feature Transform (SIFT) keypoints extracted from tourist-like photos and georegistered using airborne LIDAR data. A pushcart emulating a mobile robot platform was equipped with digital cameras and GPS units and then maneuvered through the same area as the prior model. SIFT features detected in the new photos were matched against the prior model s SIFT keypoints and used to localize the cameras in relation to the prior model. The accuracy of these SIFT-based localizations were analyzed and found to have half the geo-localization error as the GPS. Such a model-based geo-location approach can therefore enable navigation performance at least twice as good as GPS-based navigation in urban environments. Keywords: autonomous vehicle; mobile robot; localization; navigation; SIFT; GPS; urban 1 Introduction Autonomous vehicles such as mobile robots are limited in the sensors and data processors that they can carry due to size, weight, and power constraints. When operating in a netcentric environment, robots can access resources on the network to augment their onboard capabilities. These resources can include data collected by sensors on other robots, from airborne platforms, and on the ground, as well as offboard processing resources for world modeling, mission planning, perception, and navigation. When a robot utilizes these offboard resources, the robotic system effectively becomes distributed, with the mobile robot platform acting as an end effector that carries only those sensors and processing capabilities that need to be on the mobile platform to execute its mission. This will be referred as Distributed RObotics in a Netcentric Environment (DRONE). The available communications bandwidth between the robot and the network will drive the overall system design of what needs to be on the robot and what can be on the network. The physics of the sensing and perception process, such as the required aperture, range to target, view angle, etc., will require that certain mission critical sensors be on the mobile robot platform. Similarly, fast reaction for critical This work is sponsored by the United States Department of Defense under Air Force Contract # FA C Opinions, interpretations, recommendations and conclusions are those of the authors and are not necessarily endorsed by the United States Government. sensing-based decision making will drive other sensing and processing capabilities onto the mobile platform as well. However, all other data processing of the sensor data collected by the mobile robot could be sent offboard for processing if enough bandwidth is available. Likewise, data collected by offboard sensors and information extracted from such data could be sent to the robot for processing, especially if it is to be combined with local sensor data. One useful resource on the network is a model of the world built and updated from offboard sensors and other data sources. Such a model could guide the mobile robot and its interactions with the environment, aid in mission planning, provide options for navigation, and segment its understanding of the world into dynamic and static components. Examples of world models have been demonstrated in multi-modal three dimensional (3D) fused datasets built from airborne 3D LIDAR and aerial video or ground video, as described in [1] and [2]. By its nature, LIDAR point clouds are inherently 3D, but incorporating images and video into the world model requires the extraction of features, where the Scale-Invariant Feature Transform (SIFT), developed by Lowe [3], and its variants have been widely used. A recent successful construction of world models from SIFT features, a procedure that is employed in this paper, relies on sparse reconstruction of 3D urban structure from motion (SfM) in random collections of tourist photographs. These models are constructed with techniques pioneered by Snavely [4][5] and implemented in the commercial Photosynth software. In order to exploit the world model, it is necessary for the robot to first be able to reference it in a meaningful way. That is, a robot needs to register itself to the model in order to address its environment. Satellite-based Global Positioning System (GPS) localizations are often used for mobile robot navigation, but in urban environments, tall buildings with reflective surfaces frequently induce GPS multi-path errors. Due to the passive nature of image sensors, localization and registration with photos do not suffer from such errors, and offer a more robust approach to urban navigation. Because of its image matching capabilities, the same SIFT features implemented for world modeling are exceedingly useful for robotic navigation and localization. It is possible for a robot to self-localize by matching and referencing a photo to 3D reconstructions, whose sample points are SIFT features. SIFT feature localization has been

2 popular in robotic applications [6][7][8][9]. Most of these applications have been done indoors and relied on SLAM [10] for building the world model by the robot itself. The original contribution of this paper is a demonstration of robot localization in an urban environment using an offboard prior world model generated from a random tourist photo collection. To compare the performance of SIFT-based localizations to GPS, the authors collected GPS and photos in an urban environment using a pushcart, emulating local data collection from a mobile robot platform. The photos and GPS were collected in an area for which a prior world SIFT model existed which was built using a collection of random photos taken by pedestrians. The GPS localization accuracy was compared to a model-based localization technique that detects Scale Invariant Feature Transform (SIFT) features in the local camera photos and matches them to SIFT features in the prior world model. Multiple feature matches were then used to localize camera positions relative to the geo-referenced prior model. This same technique could also be used for non-robot applications that require localization and navigation in an urban environment. 2 SIFT-based geo-referencing 2.1 Prior world model This project utilized an existing prior world model consisting of a 3D point cloud of SIFT features detected on buildings in and near MIT East Campus in Cambridge, MA [10]. The 128 dimensional SIFT feature vectors, also referred to as keypoints, tend to be invariant to rotation, large changes in image scale, and slight changes in illumination. The 3D keypoint cloud fused with aerial LIDAR was generated under an internally funded Lincoln Laboratory program. Research under that program generated a 3D SIFT keypoint cloud from a prior set of photos and then matched SIFT features from Figure 1. Fused Lidar and SIFT point cloud. new photos to the cloud. Using this approach, new photos with unknown location can be localized within the prior keypoint cloud. A mobile robot could therefore use this same approach to localize itself based on the localizations of photos taken by the robot s cameras. If the keypoint cloud is georegistered, then so will be the robot s location. Localization here refers to finding the location in an arbitrary reference frame, while geo-registration refers to converting the localization into a geo-referenced frame such as UTM. 2.2 Point cloud generation The prior SIFT keypoint cloud can be thought of as a spatially sampled 3D model of the world, where the samples are taken where distinctive SIFT features are located. Such point clouds could be generated from prior sensor data collected by other ground robots, low flying aircraft (manned or unmanned), or by people walking with cameras. The point cloud used for this project was generated using 2317 photos taken randomly on five separate days in July The photos were taken by MIT Lincoln Laboratory staff members using various consumer digital cameras while walking through the area, emulating tourists taking photos. The process for building the 3D cloud of SIFT keypoints from these photos is described in [11]. Briefly, SIFT features common to multiple photos were used to estimate the relative geometry of the cameras and thereby the relative geometry of the common SIFT keypoints. These common SIFT keypoints are then assigned a 3D coordinate along with the 128 dimensional SIFT feature vector. The resulting 3D SIFT keypoint cloud was then geo-registered by manually aligning it to a geo-registered 3D point cloud from an airborne LIDAR sensor. The LIDAR data came from the Army s Topographic Engineer Center and was collected in Figure 1 depicts the SIFT and LIDAR point clouds plotted together. 2.3 Photo localization and geo-registration The prior SIFT keypoint cloud can be exploited for the purpose of localizing and geo-registering new photos with unknown location [12]. This is done by matching SIFT features in a new 2D photo to SIFT keypoints in the prior 3D SIFT keypoint cloud [13]. When there are sufficient feature matches across the new image, the location and pose of the camera that took the new photo can be estimated. Since the world model s SIFT keypoints are geo-registered, features and positions from the new photo will be referenced to absolute geo-coordinates (i.e., UTM ). 3 Data collection In order to compare the relative accuracy of SIFT-based and GPS localizations, a new collection of simultaneous photos and GPS data was required. This data collection was done on November 18, 2010 in Cambridge, MA within the area covered by the prior SIFT keypoint cloud described in Section 2. This SIFT keypoint cloud can be viewed as the

3 prior world model available on the network described above in the DRONE robot navigation use-case. 3.1 Equipment A pushcart was used to carry the cameras and GPS sensors in place of a mobile robot platform, as shown in Figure 2. The cart was equipped with pneumatic tires to reduce shocks while traversing urban sidewalks and streets. Four Nikon D5000 digital cameras were mounted to the cart pointing to the front, rear, left, and right and pointing up approximately 5 degrees. The D5000 is a consumer Digital Single Lens Reflex (DSLR) camera with a 12.3 Mpixel focal plane sensor. The D5000 was selected because of its built-in interval timer that allows the camera to be programmed to continuously take photos at a specified time interval. Each D5000 was equipped with a Nikon DX AF-S Nikkor 18-55mm zoom lens with auto-focus (AF) and vibration reduction (VR) features. Each camera employed a 16 GB SDRAM memory card for photo storage. The internal clocks of all four cameras were synchronized to within 1 second using a watch recently set to GPS time. Each D5000 camera was equipped with a Nikon GP-1 GPS unit. These GPS sensors were used to geo-tag each collected photo with GPS coordinates. In addition, a Garmin Vista HCx GPS unit was mounted on a wooden pole to lift the Garmin GPS approximately 1 meter above the cameras in an attempt to improve the GPS signal reception and provide an independent GPS data set. 4 Analysis and results 4.1 SIFT-based localization Using the software described in Section 2, each of the collected D5000 photos was processed to find SIFT features and match them to the prior world model SIFT keypoint cloud also described in Section 2. The result for each photo was a text output file that included whether or not a valid localization was found for the photo, and if so, the UTM coordinates. 4.2 Data ingest and pre-processing All further analysis software was written in MATLAB (R2010a). The Nikon JPEG meta-data of each D5000 photo was parsed to extract the date, time, and GPS localization if available. The Nikon JPEG meta-data attribute filemoddate was used as the collection date and time of each photo, since not all photos had a valid GPS localization, either due to lack of sufficient satellite signals in the urban environment or due to a loose cable on the front camera s GPS unit. For those photos that did have a valid GPS localization, the filemoddate was automatically synched to the GPS date and time by the camera. GPS latitude-longitude was converted to UTM coordinates. The output text files from the SIFT-based localization of each D5000 photos were parsed and the UTM coordinates of the SIFT localization, if available, were extracted. 3.2 Procedure Data collection was done between approximately 10:30AM and 12:30PM. The weather was generally clear with blue skies and a few high cirrus clouds. The lens on each D5000 camera was set to 24mm, VR off, and AF on. Each D5000 camera as set to store images in Large (12MP) Fine Resolution format and the interval between photos set to 5 seconds for most runs. For one run, a 1-second interval was used. Note that the cameras were started manually and only approximately synchronized. The cart was pushed along the path depicted by solid lines in Figure 3, completing two circuits, one in each direction. On the second circuit, the path was on the opposite side of Hayward Street. The cart was pushed at approximately meters/second, stopping as needed to make turns, and occasionally the cart was spun 360 degrees in place. Both of these circuits were done with a photo interval of 5 seconds for each camera and began and ended on Hayward St. One extra northbound leg was done along Carelton St. in front of MIT Medical Department with the photos collected at 1 second intervals. There were four major pauses during the data collection to take notes, during which time the cameras and Garmin GPS units were halted. Figure 2. Data collection pushcart with cameras and GPS.

4 Figure 3. Data collection route and GPS localizations. Figure 3 also depicts the Garmin GPS locations collect on the first circuit as pushpin icons. Comparing these to the actual route, we see that at the top that there are significant GPS errors compared to the actual route taken. This was due to multi-path errors caused by the tall buildings in that area. The SIFT localizations from all four cameras were merged, as were the GPS localizations. Figure 4 plots the merged SIFT localizations (top) and the merged GPS localizations (bottom), with dashed line segments connecting adjacent localizations to highlight sequential changes in position. Note that both the GPS and SIFT localization techniques had spurious localizations that are physically impossible given the adjacent localizations and the velocity the cart was pushed at. Both types of localizations were filtered to remove spurious localizations using a physically plausible velocity cutoff of 2 meters/second. The filtered localizations are plotted in Figure 5. Note that the GPS multipath errors are still present in the filtered GPS data. 4.3 Ground truth estimation In order to compare the accuracy of the GPS and SIFT localizations, ground truth was needed to compare both of them to. The approach taken to generate ground truth was to first use the collected D5000 photos to identify the straight line-segments followed during the data collection. For each Figure 4. Merged SIFT (top) and GPS (bottom) localizations. line segment, the photos collected along the line segment were identified. Next, a high-resolution geo-registered aerial photograph was downloaded from the Massachusetts Office of Geographic Information (MassGIS). Using MATLAB, line segment endpoints were manually selected in the aerial photo and converted to UTM coordinates. It is estimated that the cross-track error of these ground truth line segments is less than 0.5 meters with respect to the image coordinates. Given the size of the pushcart and the separation of the four cameras on it, the estimated accuracy of the ground truth position of each camera is approximately 1 meter. Thirty-two (32) ground truth line segments were defined using this technique and are plotted in Figure 6. Figure 7 shows the filtered SIFT localizations (top) and filtered GPS localizations (bottom) associated with the line segments plotted with the segments. Note that there are a larger number of GPS localizations than SIFT.

5 the SIFT (*) and GPS (+) cross-track RMSEs for these line segments. Note the large GPS errors associated with line segments 3 and 4. These errors are due to the multi-path errors noted in Figures 3 and 4. The average GPS and SIFT RMSEs were then calculated across all of the line segments and are depicted on Figure 8 as the solid (SIFT) and dashed (GPS) horizontal lines. On average, the SIFT localizations had only half the cross-track error as GPS, demonstrating twice the localization accuracy in the urban environment of the data collection. Figure 6. Ground truth line segments. Figure 5. Filtered SIFT (top) and GPS (bottom) localizations. 4.4 Cross-track error calculation In order to compare the accuracy of the GPS and SIFT localizations, an error metric was needed. Ideally, twodimensional error while traversing the ground truth line segments would be used. However, while the pushcart was known to move along each line segment when the corresponding photos were collected, it was not known where along the segment each photo was taken. Therefore, the most reliable error metric is the cross-track error perpendicular to the line segment. Given that the line segments are on a closed route with nearly equal distances traveled in all directions, any biases would be minimized by averaging the cross-track errors from all line segments. Equation 1 describes the crosstrack root mean squared error (RMSE) calculated over the N localizations (either SIFT or GPS) along each line segment.!"#$ =!!!!!!! Twenty five (25) of the 32 line segments had both valid GPS and valid SIFT localizations. Figure 8 shows a plot of (1) 4.5 Prior SIFT cloud sub-sampling This project used an existing SIFT point cloud of opportunity with a random density of SIFT features that was not designed for any specific system performance level. When designing a distributed robotic system that uses SIFTbased localization, one design question is how dense must the prior SIFT-cloud be to achieve a specified localization performance. Therefore, a natural question is how the SIFT localization performance degrades when the prior SIFT cloud has fewer points to match against. To test this, the prior SIFT point cloud was sub-sampled by removing SIFT points to varying degrees and then re-running the SIFT localization processing of all the push cart collected photos. The subsampling factors used were 0.875, 0.75, 0.5, 0.25, and Given that the initial SIFT cloud had 337,057 keypoints, this resulted in sub-sampled point clouds with 294,924, 252,792, 168,528, 84,264, and 42,132 keypoints. 4.6 Effect of sub-sampling on localization As SIFT keypoints are removed from the prior point cloud, fewer of them are available for matching against SIFT features found in each new photo. While some successful photo localizations will have poorer accuracy due to fewer SIFT feature matches with which to triangulate the photo s position, the greatest effect is in the reduction of photos that have successful localizations due to built in accuracy thresholds in the photo matching software. In other words, the

6 Figure 9. Number of total successful SIFT localizations as a function of SIFT keypoints in the prior world model. number of successful localizations along a line segment will diminish as the density of SIFT keypoints in the prior world model is reduced. The effect on robot navigation will be longer distances between localizations, resulting in larger navigation errors during the dead-reckoning between localizations. Figure 7. SIFT (top) and GPS (bottom) localizations associated with line segments (dashed lines). Figure 9 plots the total number of successful SIFT localizations for all four cameras as a function of the number of SIFT keypoints in the prior world model. To first order, we see that the number of SIFT localizations varies linearly with total keypoints in the prior model. 5 Conclusions The use of a prior world model for robot localization in an urban environment was successfully demonstrated. This validates the DRONE concept of using a world model accessible over a network by a mobile robot, and also demonstrates the potential for robot navigation using prior world models built from tourist photo datasets. Since the SIFT-based localization technique is currently processing intensive, one would likely employ a system design where local photos are sent to a localization service on the network. The SIFT-based localization technique employed had twice the average localization accuracy when compared to GPS in an urban environment. However, there tended to be fewer SIFT localizations than GPS localizations in some areas. As the density of SIFT keypoints in the prior model is reduced, the number of successful SIFT localizations tends to fall off linearly. Figure 8. Cross-track RMSE for SIFT and GPS localizations.

7 6 References [1] Vasile, L. Skelly, K. Ni, R. Heinrichs, O. Camps, and M. Sznaier, Efficient City-sized 3D Reconstruction from Ultra- High Resolution Aerial and Ground Video Imagery, to appear in Presentation at the International Symposium on Visual Computing, 2011, Las Vegas, NV, ISCV-2011D. [2] K. Ni, Z. Sun, N. Bliss, "3-D Image Geo-Registration Using Vision-Based Modeling", Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, 2011, Prague, Czech Republic, ICASSP-2011, pp [12] Z. Sun, N. Bliss, and K. Ni, A 3D feature model for image matching," in IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP-2010, 2010, pp [13] K. Ni, Z. Sun, and N. Bliss, 3D image geo-registration using vision-based modeling," in IEEE International Conference on Acoustics, Speech, and Signal Processing, 2011,pp [3] D. G. Lowe, Distinctive image features from scaleinvariant keypoints," International Journal of Computer Vision, vol. 60, pp , [4] N. Snavely, S. M. Seitz, and R. Szeliski, Photo tourism: Exploring photo collections in 3d," in SIGGRAPH Conference Proceedings. New York, NY, USA: ACM Press, 2006, pp [5] N. Snavely, S. M. Seitz, and R. Szeliski, Modeling the world from Internet photo collections," International Journal of Computer Vision, vol. 80, no. 2, pp , November Available at: [6] Se, S.; Lowe, D.G.; Little, J.J.;, "Vision-based global localization and mapping for mobile robots," Robotics, IEEE Transactions on, vol.21, no.3, pp , June 2005 [7] Stephen Se; Lowe, D.; Little, J.;, "Global localization using distinctive visual features," Intelligent Robots and Systems, IEEE/RSJ International Conference on, vol.1, no., pp vol.1, 2002 [8] Se, S.; Lowe, D.; Little, J.;, "Vision-based mobile robot localization and mapping using scale-invariant features," Robotics and Automation, Proceedings 2001 ICRA. IEEE International Conference on, vol.2, no., pp vol.2, 2001 [9] Santosh, D.; Achar, S.; Jawahar, C.V.;, "Autonomous image-based exploration for mobile robot navigation," Robotics and Automation, ICRA IEEE International Conference on, vol., no., pp , May 2008 [10] Durrant-Whyte, H.; Bailey, T. (2006). "Simultaneous Localization and Mapping (SLAM): Part I The Essential Algorithms". Robotics and Automation Magazine 13 (2): [11] P. Cho and N. Snavely, "3D exploitation of 2D groundlevel and aerial imagery", Applied Imagery Pattern Recognition Workshop, Washington DC, Oct 2011.