8th International DAAAM Baltic Conference "INDUSTRIAL ENGINEERING 19-21 April 2012, Tallinn, Estonia COMPARISION OF AERIAL IMAGERY AND SATELLITE IMAGERY FOR AUTONOMOUS VEHICLE PATH PLANNING Robert Hudjakov & Mart Tamre Abstract: The focus of our research is to improve navigation capabilities of an offroad unmanned ground vehicle (UGV) by improving its perception range. We train an ortophoto classifier by using data gathered by UGV and then extrapolate the data into wider area by using aerial imagery from unmanned aerial vehicles (UAV). The current progress report analyses feasibility of using satellite images instead of or together with aerial imagery and discusses an optimization technique that executes classifier as needed by path planner saving processing resources and time. Keywords: UGV, UAV, cost map, path planning. 1. INTRODUCTION There are battery powered off-road multipurpose UGV platforms developed in Tallinn University of Technology [1] objectives of which contains driving to target GPS coordinate, executing a task and returning to base. By fusing the on-board sensors of an UGV with overhead imagery we can increase the effective perception range of the UGV helping to reduce the time and energy requirements for executing said mission. In addition we can plan a path to objective ahead of time and predict the feasibility of the mission with current battery charge. Our proposed system consists of classifier, cost map generator and path planner. The system uses previously gathered UGV data and aerial imagery as an input and produces a long haul path for UGV, the prior knowledge and aerial imagery is used to train an image classifier that is then used for cost map generation and path planning. We intent to repeat the cycle of data gathering, classifier training, cost map generation and path planning after the mission is dispatched in order to keep the system flexible and to be able to consider new kinds of obstacles after they have been discovered by UGV. In this article we are analyzing the possibility of using satellite imagery together with or instead of aerial imagery. There are two explicit scenarios that we cover: whether it is possible to replace aerial imagery with lower quality satellite imagery and is it possible to switch over from aerial imagery to satellite imagery when UAV goes offline? The aerial imagery is preferred over satellite imagery for it has higher resolution and it can be gathered ahead or during the mission ensuring its freshness. In addition we describe an optimization technique that increases the performance of aerial imagery evaluation by combining cost map generation and path planning steps. The advantages are twofold. First we benefit from reduced processing requirements allowing cheaper and more mobile hardware (in terms of power requirements). Secondly it allows us to generate path faster which in turn allows us regenerate the updated path to
objective more often. 2. AERIAL IMAGERY VS. SATELLITE IMAGERY Thus far we have done all our experiments with aerial imagery and we have estimated that we get comparable results with satellite imagery, this time we decided to put the hypothesis to a bench. For the experiment we chose identical area to our aerial data set from Google maps database [2] [3] [4] [5], up-scaled the satellite imagery to identical resolution of aerial imagery and hand classified it (Illustration 1). It must be noted, however, that the satellite imagery has about 4x lower resolution, was taken during different time of the day, different time of the year and different year. The aerial data set was acquired during a summer time when grass was green and multiple construction projects were ongoing so there was a lot of construction debris. The satellite imagery was acquired a year later during spring when grass was not up yet and there was little debris from construction projects left. Our processing pipeline consists of three distinct steps: image classification, cost map generation and path planning. For image classification we pick patterns from aerial imagery and feed them into a classifier that outputs a feature vector. For the classification we use two approaches that are described in finer detail in our previous publication [5]: with method A we try to detect all features that are present on input pattern and with method B we try to detect to which category belongs the pixel in the middle of input pattern. The outputs of method A are independent meaning that the input pattern may contain multiple features that we are looking for (houses, roads, etc) while the outputs of method B are dependent - the center pixel may belong to only one category. The objectives of this experiment are twofold: We want to verify that our classifier can get comparable classification rates with satellite imagery. We want to verify the feasibility of cross usage of a trained classifier e.g. we want to see if a classifier that has been trained on aerial imagery is usable on satellite imagery and vice versa. Method A Method A is based on the idea that a convolutional artificial neural network is capable of extracting distinct features from input pattern and linear classifier layer can then be trained to report presence of objects in input pattern. In layman's terms each classifier output is trained to answer simple questions such as Is there a house in input pattern?, Is there a tree in input pattern or Is there some grass in input pattern?. All the classifier outputs are independent, there can be multiple features simultaneously present in the input pattern. For the experiment we used a classifier that is based on convolutional neural network and had three 29x29 neuron inputs for a 29x29pixel pattern with three color channels. The second hidden layer had six 13x13 neuron feature maps that were connected to all inputs using 5x5 neuron kernel and the third layer had fifty 5x5 neuron feature maps that were connected to all second layer feature maps using 5x5 neuron kernels. The final hidden layer consisted of a linear classifier with 100 neurons in it and the output layer had 4 neurons for four feature categories: houses, roads, grass and debris. The specific classifier configuration was chosen because we have reported multiple experiments with it in the past [2] [3] [4]. For the experiment we generated two data sets - one from aerial imagery and another from satellite imagery. Next we randomized the internal parameters of our classifier and trained it on aerial data set. Finally we tested the trained classifier on both aerial test set
and satellite test set. The same operation was repeated with a classifier trained on satellite data set and the results were saved into Table 1. Table 1 Aerial imagery classifier data set. Not only the color of the grass was different on the data sets, but also the sharp texture of grass on aerial imagery was replaced with botched texture of mud on Satellite imagery classifier Aerial data set Correctly classified: 97,9% Houses, rate=99,4% Roads, rate=99,7% Grass, rate=99,8% Debris, rate=98,8% Correctly classified: 11,2% Houses, rate=83% Roads, rate=78% Grass, rate=42% Debris, rate=50% Satellite data set Correctly classified: 9,4% Houses, rate=48% Roads, rate=87% Grass, rate=41% Debris, rate=48% As seen from above table the classifier is usable both with aerial imagery and satellite imagery - the classification rate in both cases is well over 90%. The cross use, however, is limited. The classifier that was trained on lower quality satellite imagery was capable of detecting houses on higher quality aerial imagery but the reverse did not hold. Correctly classified: 98,9% Houses, rate=99,7% Roads, rate=99,8% Grass, rate=99,9% Debris, rate=99,6% satellite imagery (Illustration 1). Similarly the aerial imagery had a lot of debris but satellite imagery had next to none and hence the cross use of classifiers for debris detection is not reliable. The roads were similar on both data sets and the classifier could detect the roads on both - this is important results because having a Illustration 1: Path planning using classifier with high uncertainty levels. Cost map (left) and planned path (right). Because on one imagery the grass was up and green while on other imagery it had not yet grown, the classifier trained on one data set can not be used to detect grass on other classifier that can detect roads is better than not having a classifier making our system usable even in unfavorable conditions.
Illustration 2: Grass on aerial imagery vs. mud on satellite imagery Our cost map generation algorithm is built to cope with classifier uncertainties and is capable of producing output even when classifier is only capable of detecting some of the features. By feeding the classifier trained on aerial imagery with satellite imagery we receive a cost map where everything besides roads are marked as unknown terrain and during navigation the roads are preferred. The following illustration (illustration 2) shows the path planning capability of our system with classifier that has been trained with aerial imagery using satellite imagery. Despite on the overall the classifier could only correctly classify 9,4% of patterns, it could find roads with relatively high certainty and thus can be used for path planning. The results are better with the classifier that is trained on satellite imagery and used on aerial imagery (Illustration 3). Both the roads and the buildings are clearly detected and since our proposition is to re-train the classifier while UGV is moving (using freshly gathered data) the classifier trained on satellite imagery has good starting values. To validate that claim we took a classifier that has been trained on satellite imagery and trained it for one single epoch on aerial imagery. The classification rate for all items went up from 11,2% to 66% and in detail the classification rate for houses went from 83% to 98%, for roads from 78% to 96%, for grass from 42% to 88% and for debris it went from 50% to 79%. We saw a similar increase after we took a classifier that had been trained on aerial imagery and used it on satellite imagery. After one epoch of training the total classification rate went up from 9,4% to 54%. Method B Method B will train the classifier to detect Illustration 3: Comparison of cost maps: satellite classifier on satellite imagery (left) vs. satellite classifier on aerial imagery (right). Darker areas have lower traversal cost than lighter areas.
the category of a single pixel in input image, particularly the pixel from the center of the pattern. The classifier will then answer the question Into which category belongs the pixel in the center of this pattern?. The categories are exclusive and thus the classifier outputs are dependent. The experiment setup was identical to method A: we generated two data sets and trained a classifier on each of them. The classifier were then tested and cross tested on the data sets and results are in the Table2. Table 2 Aerial imagery classifier Aerial data set Correctly classified: 88% Houses, rate=97,87% Roads, rate=97,07% Grass, rate=94,13% Debris, rate=97,93% Classifier trained on the aerial imagery works well on the the aerial imagery Classifier trained on the satellite imagery works well on the satellite imagery The classifier trained on lower quality imagery also works well on higher quality imagery but the reverse does not hold. The classifier that has been trained on satellite imagery is a good starting Satellite imagery classifier Correctly classified: 20,4% Houses, rate=66,4% Roads, rate=67,57% Grass, rate=76,23% Debris, rate=52% Satellite data set Correctly classified: 12,7% Houses, rate=55,73% Roads, rate=77,77% Grass, rate=48,67% Debris, rate=57,93% The results are very similar to those of method A. The classifier can be trained on both aerial data set and satellite data set to a nearly identical classification capability. The effects we observed during cross testing of classifiers using method A also hold for method B: the classifier trained on satellite imagery is better at classifying aerial imagery than the classifier trained on aerial imagery is at classifying satellite imagery. All the combinations of classifiers and data sets are capable of finding roads and since our cost map generator is built to deal with uncertainties of classifier we can use them for at least preliminary path planning. The conclusions from comparing the satellite data to aerial imagery: Correctly classified: 89,9% Houses, rate=97,37% Roads, rate=96,93% Grass, rate=96,33% Debris, rate=98,63% point for re-training on aerial imagery. Reverse also holds true; the classifier that has been trained on aerial imagery is a god starting point for re-training on satellite imagery. When using a trained classifier on a different imagery we must take into account the changes in nature - where there is grass on summer there is mud on spring and snow in winter. Each has different color and texture. Because the roads are easily distinguishable it is better to use non-matching classifier on terrain where roads are present than not to use overhead imagery at all. 3. INTEGRATION OF COST MAP GENERATION AND PATH PLANNING
The cost map is a grid on nodes; each node has a cost associated with it. For generating the cost map we divide an aerial photo into a grid of 29x29 pixel patterns and feed the patterns to our classifier which outputs feature vectors for the patterns. The pattern size is determined by the size of classifier input layer; to enhance the cost map resolution we use overlapping patterns. The feature vectors are mapped into a probability vectors using a threshold vector. For cost calculation the probability vectors are multiplied with a weight vector and a constant bias is added [4] [5]. cost = d f i, t i w i + c u, i where F is classifier output vector, T is threshold vector, d is the transformation function that maps classifier output to probability [5], W is the weight vector, c u is constant bias. It is important to note here that for calculating the cost of each node we need to evaluate the pattern associated with the node using a classifier. The evaluation of the pattern is relatively expensive operation when compared to the cost calculation. For path planning we use A* algorithm [6] which is improvement over Dijkstra's algorithm [7] and generates an optimal path from start node to end node. There s an excellent tutorial on A* algorithm by Patrick Lester: 1) Add the starting square (or node) to the open list. 2) Repeat the following: a) Look for the lowest F cost square on the open list. We refer to this as the current square. b) Switch it to the closed list. c) For each of the 8 squares adjacent to this current square If it /.../ is on the closed list, ignore it. Otherwise do the following. If it isn t on the open list, add it to the open list. Make the current square the parent of this square. Record the F, G, and H costs of the square. If it is on the open list already, check to see if this path to that square is better, using G cost as the measure. A lower G cost means that this is a better path. If so, change the parent of the square to the current square, and recalculate the G and F scores of the square. /.../ d) Stop when you: Add the target square to the closed list, in which case the path has been found /.../, or Fail to find the target square, and the open list is empty. In this case, there is no path. 3) Save the path. Working backwards from the target square, go from each square to its parent square until you reach the starting square. That is your path. [8] H is an heuristic value that estimates the cost of moving from current node to end square, G is the cost of moving from start node to current node and F = G + H. When calculating the cost G for any given neighboring node we add the cost G of current node to value taken from cost map. Because the A* algorithm does not necessarily use all nodes in cost map and because the generation of whole cost map is relatively slow process (when compared to path planning stage) we can get significant improvements to our algorithm by combining the path planner and the cost map generation steps. Illustration 4: red line presents the used path and the nodes evaluated by path planner are marked using dark area. Overall there was need to evaluate 7% of the aerial image for the path planning When we calculated the cost of a node only when it was needed by path planner we can occasionally reduce the combined time of cost map generation and path planning by
order of magnitude - especially when the path from start to end point is clear (illustration 1, illustration 2). However, this optimization technique does not affect the worst case scenario - there are cases where we still need to evaluate the whole imagery for path planning particularly in scenarios where there are no clear pathways to designated target position. Illustration 5: red line presents the used path and the nodes evaluated by path planner are marked using dark area. Overall there was need to evaluate 25% of the aerial image for the path planning. 4. CONCLUSIONS Our proposed system of extending UGV perception distance by extending UGV local experiences to wider area using aerial imagery works equally well using satellite imagery. We can also use classifier that has been trained for satellite imagery on aerial imagery and using of a classifier that is trained on similar imagery is a good starting point for re-training it for specific imagery in terms of quick results. By combining the cost map generation and path planning stages we can reduce the total time it takes for cost map generation and path planning up to 10x. The performance increase comes from executing the classifier on overhead imagery (expensive operation) as needed by path planner. The optimization, however, does not affect the worst case scenario where we still need to generate whole cost map. 5. BIBLIOGRAPHY [1] M. Hiiemaa and M. Tamre, "Semiautonomous Motion Control Layer for UGV-Type Robot," pp. 203-207, 2010. [2] R. Hudjakov and M. Tamre, "Aerial Imagery Terrain Classification for Long-Range Autonomous Navigation," Proc. of International Symposium on Optomechatronic Technologies, pp. 88-91, 2009. [3] R. Hudjakov and M. Tamre, "Aerial Imagery Terrain Classification for Long-Range Autonomous Navigation," Proc. of the 7th International Conference of DAAAM Baltic Industrial Engineering, pp. 530-535, 2010. [4] R. Hudjakov and M. Tamre, "Ortophoto Analysis for UGV Long-Range Autonomous Navigation," Estonian Journal of Engineering, no. 17, p. 17 27, 2011. [5] R. Hudjakov and M. Tamre, "Aerial Imagery Based Long-Range Path Planning for Unmanned Ground Vehicle," in Mechatronic Systems and Materials, Kaunas, 2011. [6] P. E. Hart, N. J. Nilsson and B. Raphael, "Correction to A Formal Basis for the Heuristic Determination of Minimum Cost Paths," SIGART Bull., no. 37, pp. 28--29, December 1972. [7] E. W. Dijkstra, "A note on two problems in connexion with graphs," Numerische Mathematik, vol. 1, no. 1, pp. 269-271, 1959. [8] P. Lester, "A* Pathfinding for Beginners," 18 July 2005. [Online]. Available: http://www.policyalmanac.org/games/a StarTutorial.htm. [Accessed 28 02 2012].