Sensor Scheduling Using Ant Colony Optimization

Transcription

1 Sensor Scheduling Using Ant Colony Optimization Dan Schrage Charles River Analytics, Inc. 625 Mt. Auburn St. Cambridge, MA 02138, USA x512 Paul G. Gonsalves Charles River Analytics, Inc. 625 Mt. Auburn St. Cambridge, MA 02138, USA x528 Abstract The basic problem of collection management is to schedule a group of sensor assets over a series of mission objectives in a way that minimizes resource usage and maximizes the likelihood that all the mission objectives will be completed. We present an approach to collection management, specifically sensor scheduling, that relies on Ant Colony Optimization (ACO), a biologically-inspired search algorithm. This approach offers agent-based modeling of the search resources and environment to ensure realism. We extend the traditional ACO algorithm, which relies on a single agent for search, to accommodate coordinated multi-sensor search teams made up of heterogeneous sensor assets. Keywords: Sensor scheduling, collection management, Ant Colony Optimization, multi-agent systems 1 Introduction Sensor management [1] seeks to fulfill a series of collection tasks using a given set of sensor assets. The number of assets and collection requirements creates a huge number of combinations to be searched for producing a scheduling solution that optimizes asset utilization and collection needs. Additionally, the resulting schedules must satisfy complex domain constraints and rules and must try to satisfy all high-priority requests. Moreover, during the course of a mission, specific events may occur to warrant the re-allocation or replanning of the sensor assets. For example, certain collection requirements may have been satisfied already or have become obsolete, assets may be required to be redirected to unsatisfied or novel requirements, cross-cueing opportunities may arise, and mission objectives and constraints may have changed (e.g. change from reconnaissance to battle damage assessment, time constraints, asset operational status, etc.). What is needed is an approach to scheduling and retasking that: 1) provides a user-friendly, flexible and declarative problem modeling environment; 2) is capable of producing optimized schedules and incorporating dynamic changes; and 3) is embeddable within other C4ISR application software. Currently available scheduling software cannot meet all of these requirements [2]. We propose an approach to scheduling that relies on a biologically-inspired optimization algorithm known as Ant Colony Optimization (ACO). This approach is easily combined with agent-based modeling to produce an accurate representation of the search environment. Moreover, it is one of the best known algorithms for solving a number of prioritized path-planning problems, which is exactly the type of problem we have in this case. Finally, ACO distinguishes itself from other search algorithms with its ability to adapt to dynamicallychanging circumstances in the search environment. Thus, not only is it well suited for the initial planning problem, but it also offers the option of dynamic replanning of a solution in response to changing battlespace conditions. One of our main contributions here is our extension of ACO from an algorithm primarily used to solve singleagent problems to an approach that supports multi-agent teams of coordinated, heterogeneous sensors, which was an important part of applying ACO to the collection management domain. This paper presents preliminary results from our ongoing effort to solve the sensor scheduling problem in a dynamic environment. As such, the results are intended simply to be a proof-of-concept that ACO can successfully deal with many of the challenges we describe. 2 ACO Background 2.1 Ant Colony Optimization (ACO) Biologically-inspired metaphors have become both a popular and a valuable area of research within Artificial Intelligence in recent years. One line of research in particular has focused on how social insects collectively solve problems, which has come to be known as Swarm Intelligence [3,4]. ACO, as its name suggests, is based on the collective foraging and food-gathering behavior of ants. A number of experiments have shown that ants have the ability to find the shortest path between their nest and a food source. This is most clearly demonstrated in an experiment where a food source is reachable from an ant colony s nest by two different bridges (branches), one of which is longer than the other [5]. Ants are known to move by random chance; so, for example, given the choice of

2 two bridges, the ants will randomly choose each with a probability of 0.5. However, those probabilities can change dramatically in the presence of pheromones, chemical substances laid by ants that signal other ants to follow their trail. Pheromones act as positive feedback reinforcement that tells other ants to follow a marked trail, which has a feedback effect since more ants that follow leave more pheromones on the trail, so the value of that trail is further reinforced. In the two-bridge experiment, the first ants to arrive at the choice point between the bridges face a trail unmarked by pheromones, so half the ants choose one bridge and half the ants choose the other. However, the ants who randomly chose the shorter bridge return more quickly than those who chose the longer bridge, so the shorter trail is marked with pheromones before the longer trail can be marked. This causes the ants immediately following to choose the shorter bridge since it has a strong pheromone presence. Thus, the shorter path quickly gains an even higher concentration of pheromones as the following ants also mark it. Convergence occurs rapidly and soon none of the ants follow the longer path. In practice, two problems exist with this. First, the longer bridge must be about twice as long (or more) than the shorter path in order for the ants to converge on the short path. Otherwise the short path does not build enough of a lead in relative pheromone concentration for convergence to occur, and the ants are split between the two paths. The second problem is that ants cannot adapt to changing circumstances: if the experiment begins with only the longer bridge in place and the shorter bridge is added only after the ants have begun to follow the longer bridge, the ants never select the shorter bridge since the longer bridge has already been marked with pheromones. Fortunately, computer scientists are not tied to the realities of ant behavior in formulating their algorithms, and a simple modification solves both of these problems and creates a very powerful optimization algorithm. The necessary change is to add pheromone decay to the algorithm so that the pheromone artificially evaporates over time. The effect of this is that pheromone concentration decreases at a given point if that point is not continually marked with new pheromones by the ants. Pheromone decay solves both of the problems described above. First, with fast pheromone decay, even very small differences in the length of two possible paths can be exploited to quickly build up a large difference in pheromone concentration. This is a useful property for an optimization algorithm, because the algorithm needs to be able to choose the best results from competing alternatives that are often very similar. Second, pheromone decay allows the algorithm to adapt to dynamically changing circumstances. If the pheromone is set to decay quickly enough (which is advantageous in an optimization context), it is hard to maintain pheromone trails at all and even harder on a very long trail that is marked at a low frequency. In the case where a shorter path is added after the ants have begun to follow a longer path, a very small number of ants will continue to probabilistically explore other paths; this is essential in an optimization algorithm to keep from converging prematurely on mediocre solutions. Since the pheromone on the longer path is constantly decaying, ants finding the new shorter path can build up the pheromone concentration on it. Since it is easier to sustain a pheromone trail on a shorter path (since the ants can cross it and return at a faster rate), the shorter path will overtake the longer path, so the ants find the optimal solution even when conditions are not fixed. From an algorithmic standpoint, we can view the ants in these examples as artificial agents traversing a search space, looking for optimal solutions. As mentioned above, these agents differ somewhat from their natural counterparts. One important difference is that these artificial ants should follow pheromone-saturated trails with a lower probability than natural ants. In other words, we want to have a greater number of artificial ants that randomly explore the search space than we would see in the behavior of natural ants. The reason for this is familiar from other optimization algorithms: we want to avoid premature convergence on locally optimal solutions and instead search for the globally optimal solution. We do not want to settle on a solution discovered early in the search that seems good enough ; instead, we want to continuously improve upon our solutions. ACO has been applied successfully to a number of classic optimization and scheduling problems. One of its first successful applications was to the Traveling Salesman Problem (TSP), which is the problem of finding the shortest path for an agent to travel through n connected cities, where each city can only be visited once and the path must begin and end at a given starting point. This is a difficult problem with no known analytical solution, so it has been used as a test point for a number of optimization algorithms. ACO has been demonstrated to be one of the best solutions to this problem in a number of different variations [6]. Additionally, ACO is the best known solution to the sequential ordering problem (SOP), which is an asymmetric version of the TSP that includes prioritization of the individual cities to be visited [7]. This kind of priority-based scheduling problem is very similar to the scheduling and allocation problem that is the subject of our approach here, so there is strong evidence that ACO is the best candidate to apply to this problem as we have represented it. Moreover, ACO has been successfully used in military contexts for the control of unmanned robotic vehicles, which further supports its utility to this type of domain [8]. 2.2 ACO For Sensor Scheduling Because ACO has been so widely studied and applied to a variety of scheduling problems, its application in our approach to sensor scheduling is straightforward. Moreover, its success in priority-based scheduling makes it uniquely suited for such a problem. After preprocessing

3 to turn the search area into a discrete space, the problem given to the sensor scheduling module is this: given a few different ways to partition a geographic area, and given that individual partitions have priorities that determine their value as search targets, find the optimal sequence in which to search those individual partitions. At first glance, this problem seems almost identical to the sequential ordering problem (SOP) described in the previous section. This similarity is good, since ACO is the best known heuristic for solving this problem. In our case, however, the problem is more complicated than that. Two factors complicate the problem. First, we may be coordinating multiple sensor assets instead of a single sensor to search the area. Second, those sensor assets could be heterogeneous, with different speeds, sensor ranges, and other characteristics that differentiate their search capabilities. Fortunately, ACO is easily applied to such a situation, since the artificial ants are actually agents, and we can give those agents any characteristics necessary to fit the constraints of the problem. Using this agent-based approach, we can create a large number of search agents for each type of sensor available for our sensing task. Using the ACO method described in the previous section, these teams will converge on optimal solutions that coordinate the efforts of all sensor types, since each sensor agent still functions as an artificial ant as described above. This agent-based approach provides one more advantage of ACO over other optimization techniques. It allows our system to be at once very general and at the same time extensible to any number of specific types of sensors. Using this technique, we could, for example, coordinate the efforts of a number of SURTASS sensors for scanning underwater regions with the efforts of a number of UAV-based RF sensors for scanning the remaining air and surface regions. All that is necessary is some basic information about the search capabilities of each sensor type available to the system, and the integration of a heterogeneous group of sensors becomes seamless. 2.3 Extending ACO To Multiple Agents One of the main innovations in our approach to scheduling is its ability to optimize teams of sensor assets, instead of just individual assets. Traditional implementations of ACO have been limited to optimizing a single agent s search path. In order to achieve multiagent optimization, they had to cheat in a sense by artificially dividing the search space among the multiple agents and running a separate optimization for each agent. This is problematic because it is impossible to optimally divide the search space unless the optimal search solution is known in advance, which is obviously not the case in any real optimization. Moreover, since optimizations are run separately for each agent, there is no mechanism for introducing coordination or cooperation among multiple sensors. Our method is similar to the approach taken by Iredi, Merkle, and Middendorf [9] for coordinating ant teams, though we rely on more problem-specific methods of coordination. As such, our solution is less general, but it is better-suited to solving the sensor scheduling problem. Our approach moves beyond the traditional ACO algorithm by grouping agents into teams. Each sensor asset is represented by a number of ant agents searching in parallel for the optimal solution. In our approach, individual ants continue to act autonomously, but each ant is connected to a group of other ants, one for each sensor. Their movements remain independent, but their utility functions are dependent upon the actions of the others so that multiple agents do not receive credit for searching the same targets. In other words, in a given cycle, as soon as one agent has searched a target and received credit (in the form of utility) for that action, no other agent will receive any additional utility if it also happens to search the target. This might seem limiting at first glance, but recall that there are many teams of ants all working this way in parallel, so at any given point, a large number of paths are being explored simultaneously. Moreover, these teams work for a number of cycles, and in each cycle a new set of paths is explored, so this turns out to be a very effective way of exploring a large portion of the search space. In this way, search targets are adaptively divided among the sensors in the most optimal manner. In addition to providing an optimal coordination of a group of sensor assets over a number of targets, this approach also allows for a lot of flexibility and extensibility into more complex collection requirements. For example, some sensor types require synchronization with other sensors, which is a simple extension in our approach. Traditional ACO implementations would be unable to accommodate such requirements. Moreover, our approach makes it simple to add sensors that only collect particular types of targets. With such additions, targets can still be divided adaptively in the same way as they are in the current implementation, whereas such requirements under a traditional ACO implementation would require a complete system overhaul. Thus, our approach not only solves a number of previously-unsolved problems, but it provides a flexible way to continue to extend the system to accommodate more complex requirements in a future follow-on effort. 3 Implementation The C++ core of our system implements Ant Colony Optimization (ACO) for sensor scheduling, which was outlined in Section 2. ACO is an agent-based approach to optimization, where each sensor asset is represented by an individual agent that models the characteristics of that asset. The implementation itself uses three major classes: AntAlgorithm, which is the central overseer of the scheduling algorithm; AntGraph, which models the environment in which the scheduling takes place; and Ant,

4 which represents an individual ant agent corresponding to a sensor asset that needs to be scheduled. AntAlgorithm contains the main program loop, and it is responsible for organizing the interaction between the individual ant agents and their environment. The algorithm divides time into a sequence of turns. In each turn, every ant is allowed to make a move within the environment. A cycle is a sequence of turns long enough for each ant to move as far as the maximum path length, which is a parameter settable by the user. At the end of each full cycle, the ants return to their start locations, ready to begin the next cycle. More importantly, pheromones are distributed onto the map at the end of each cycle. Pheromone trails reinforce successful paths discovered by the ants so that those paths are more likely to be followed over future cycles. This is done by assigning a utility to each ant s path at the end of every cycle. The utility function for the ants represents a measure of how well they met their objectives; in this case, their main objective is to find the search targets in the shortest path possible. Avoidance of threats can also be factored in to the utility. After calculating the utility for each path, pheromones are distributed on the map in proportion to the utility of each ant s path, thus, more pheromone is laid for paths with greater utility, so those paths become the ones most likely to be chosen in future cycles. In addition to adding pheromones at the end of each cycle, AntAlgortihm controls pheromone evaporation, which takes away a certain percentage of the pheromones on the map. Evaporation prevents the pheromones from converging prematurely to a sub-optimal path, because it means a path requires constant reinforcement in order to maintain its level of pheromone. Pheromone evaporation is another parameter specified by the user. After pheromones have been distributed and evaporated, the cycles continue and the paths progress until the maximum number of cycles has been reached, which is also a user-defined parameter. At this point, AntAlgorithm returns the best set of paths found for the sensors, which it tracks continuously over the course of the optimization. AntGraph provides a discrete representation of the continuous search space represented by a user-defined area on the map. For maximum flexibility, AntGraph creates a grid over the search space. The level of detail of this grid (i.e. how fine or coarse it is) is determined by parameters set by the user. By reducing the search space to a discrete grid, the size of the search space is dramatically reduced, so the optimization problem becomes much more manageable. AntGraph creates a grid where ants are allowed to move between any two adjacent nodes. The edges between these nodes are weighted according to the utility an ant can receive for traveling across those edges. For example, edges that lead to sensor targets (named areas of interest, or NAIs) are weighted with strong positive utility. Edges that pass over enemy air defenses are given strong negative utility, discouraging ants from getting near them. AntGraph is also responsible for making this representation spatially accurate, so distances between nodes represent real distances between the corresponding points on the map. This ensures a degree of spatial and temporal accuracy in the optimization. Note that this does not reduce to a simple greedy approach. Ants gain utility for each target they search, but they must be within search range of the target before any utility is awarded. This prevents the system from producing solutions where the ants always move only to the nearest target. Instead, it simply guarantees that if an ant passes by a target within its range, it searches that target, which can be thought of as a local optimization. However, the global solution is stochastic, which produces a great deal of complexity beyond the simple greedy solution. The third major class in the the system core, Ant, represents an individual ant agent that moves through the search space represented by AntGraph, searching for targets to help construct an optimal search path. Each sensor asset it represented by a number of ants working in parallel to find the best path for that individual asset. Ants move probabilistically throughout this search space. At the very beginning of the optimization, before any pheromones have been distributed on the graph, the ants move randomly, influenced only by relative strength of the edge weights they traverse. However, as pheromones begin to build up on the graph, the ants movements are weighted towards moving over edges with high pheromone concentrations. The probability of making a particular move thus depends on a weighted combination of the edge weight for each possible move and the pheromone concentration of each move possibility. Ants function autonomously in the search space, choosing a move at each turn and then executing that move. Because of this architecture, ants can represent any possible sensor asset characteristic, such as sensor range, speed, physical constraints, etc. This makes it a powerful and extensible architecture for supporting a variety of heterogeneous sensor assets. 4 Results We tested our ACO approach to scheduling on a variety of basic problems, ranging from single-sensor problems used to test the basic workings of the approach to more complex mult-sensor problems that used our multi-agent sensor team approach. We again stress that these are preliminary results meant to prove the feasibility of our approach. Research is still ongoing to extend this approach to more complex search tasks using more realistic mission constraints and physical models. The simplest example is that of a single sensor asset assigned to find a set of named areas of interest (NAIs) that are geographically clustered in a way that makes the optimal search path obvious, as is the case in Figure 1.

5 optimization process by examining changes in the pheromone distribution as the path evolves. Figure 1. Single Sensor Search Example. In this example, the 10 targets, or named areas of interest (NAIs), are shown and labeled as red circles on the map. The single sensor s start position is shown by the blue square. The search area is the black rectangle surrounding the area containing the NAIs; the sensor is not allowed to search outside of this rectangle. The obvious path here to search the ten targets is to follow the NAIs in order from zero to nine, and that is exactly what the optimization produced. The pheromone distribution converged fairly rapidly to reflect this single, optimal path. A second example involves two sensors starting from opposite corners of the search area, shown in Figure 2, which we will allow us to examine our approach to multisensor coordination in more detail. Figure 3. Pheromone Distribution After 68 Iterations. Figure 3 shows the pheromone distribution after 68 iterations. The pheromone distribution display corresponds to the search area inside the rectangle in Figure 2, so the pheromone distribution should gradually change to mirror the shape of the NAIs that the sensor is searching for. In the figure, pheromones for the sensor starting in the upper left are represented in shades of red, and pheromones from the sensor starting in the lower right are shown by shades of green. Brighter, more saturated colors represent higher pheromone concentrations. In Figure 3, both sensors have begun to form a path along the top and bottom rows, respectively, corresponding to the rows of NAIs that cover those areas. However, the middle of the search area is only beginning to be explored, as evidenced by the faint colors in the center. Figure 2. Two-Sensor Search Example. This example also possesses an obvious optimal search path, with the sensor in the upper left traversing the NAIs in ascending order and the sensor in the lower right starting at NAI 16 and searching them in descending order. However, this problem is more difficult, not only because there are two sensors, but because the cluster of targets in the center of the search area are just as close to the sensors as are the NAIs on the edges opposite the sensors initial positions. We can trace the path of the Figure 4. Pheromone Distribution After 202 Iterations. In Figure 4, we see the progress of the search after 202 iterations. Notice how the sensor in the bottom right has formed a preliminary path to the middle of the search area in addition to the path along the bottom row of NAIs. In this case, it is finding paths with as much value that follow the first few NAIs in the lower right and then jump to the NAIs in the center of the search area, neglecting the targets along the left edge. It is common over the course of

6 exploration that such false routes will be taken, but those routes should be corrected by the time the optimization is complete. Figure 5. Pheromone Distribution After 436 Iterations. In Figure 5, the situation has been corrected. This shows the final pheromone distribution after the optimization has finished running. The path to the center of the search area has been abandoned by the lower right sensor in favor of pursuing the logical route across the bottom and then up the left edge before continuing around to the middle. The upper left sensor has also refined its route along the top of the search area. Figure 6. Search Paths Resulting From Optimization. Figure 6 shows the resulting search paths discovered by this particular run of the optimization algorithm. Note that the paths cover slightly more territory in the middle than was evident on the final pheromone distribution. This is because the agents constantly explore new paths, and the pheromone distribution may never converge completely on the optimal solution. Instead, the algorithm keeps track of the best paths found so far at any point, and those are the paths returned at the conclusion, not some set of paths derived from the resulting pheromone distribution. All the targets have been found in their logical order in this solution. One important result from this solution is that the sensors never both try to search the same target. This shows that our multi-agent team approach is effective in adaptively dividing the targets among the appropriate sensors. This result extends to more complicated search paths and a greater number of sensor assets. The paths shown in this solution do exhibit some slack time, time where the paths appear to wander between NAIs rather than following a more direct, straight path. This occurs when the sensors have more time available than is necessary for them to complete their missions. Slack time is built into the schedules so that dynamic changes in the search environment can be more easily accommodated. This slack time also results from the problem being under-constrained. The system was built with the goal of extending the approach to handle a more realistic physical model of both the sensors and the battlespace and to handle more complicated tasking requirements. All of these factors will create more constraints on the problem, and the representation of the search space was designed to incorporate these extensions. In simple problems such as this, however, this results in a set of constraints too loose to produce a singular optimal solution. This sacrifice should prove valuable when the paths are extended to more complex problems. 5 Conclusions Our basic Ant Colony Optimization (ACO) approach proved valuable when applied to a wide array of search problems like the ones presented here. We successfully extended the ACO algorithm to support multi-agent coordinated teams of sensors, which is an important requirement for solving problems in the collection management domain. Finally, we provided a flexible architecture that provides fertile ground for future extensions of the approach to more complex tasks and models. Future extensions to this approach should focus on two main tasks : First, ACO is well-suited to problems that require dynamic adaptation to changing conditions in the search environment. Future work should focus on incorporating this property into the approach, so the algorithm is capable not only of initial planning but of dynamic replanning to account for changing information and objectives. Second, this system was built with the goal of extending it to more complex physical models of both the sensors and the battlespace and to handle more complex collection requirements. The flexible representation provided by our approach should perform well when faced with this added complexity. References [1] Army. (1994). Collection Management and Synchronization Planning (Army Field Manual (FM) 34-2): Army

7 [2] Pritchard, D. E. R. (2000). Dynamic Route Replanning and Retasking of Unmanned Aerial Reconnaisance Vehicles. Master's Thesis, Air Force Institute of Technology. AFIT/GAE/ENY/00M-10. [3] Dorigo M., V. Maniezzo & A. Colorni (1991). The Ant System: An Autocatalytic Optimizing Process. Technical Report No Revised, Politecnico di Milano, Italy. [4] Bonabeau, E., Dorigo, M., & Theraulaz, G. (1999). Swarm Intelligence: From Natural to Artificial Systems. New York: Oxford Univ. Press. [5] Deneubourg, J. L., & Goss, S. (1989). "The ant system: optimization by a colony of cooperating agents". Ethology, Ecology, and Evolution, 1, [6] Dorigo, M., & Gambardella, L. M. (1997). "Ant colonies for the traveling salesman problem". BioSystems, 43, [7] Bonabeau, E., Dorigo, M., & Theraulaz, G. (2000). "Inspiration for optimization from social insect behavior". Nature, 406(6 July), [8] Parunak, H. V. D., Brueckner, S., Sauter, J., & Posdamer, J. (2001). "Mechanisms and Military Applications for Synthetic Pheromones". Workshop on Autonomy Oriented Computation, Montreal. pp [9] Iredi, S., Merkle, D., & Middendorf, M.: "Bi- Criterion Optimization with Multi Colony Ant Algorithms." EMO 2001: