Load Balancing in Individual-Based Spatial Applications.

Transcription

1 Load Balancing in Individual-Based Spatial Applications Fehmina Merchant, Lubomir F. Bic, and Michael B. Dillencourt Department of Information and Computer Science University of California, Irvine Abstract Individual-based spatial simulations are a class of applications in which a collection of entities interact locally with one another within a simulated space to generate some global collective behavior. An Eulerian implementation of such a system partitions the simulated space and assigns each partition, together with the corresponding entities, to a dierent physical node. Load balancing is achieved by dynamically adjusting the decomposition of the simulated space, which forces the corresponding autonomous entities to automatically migrate among physical nodes. This paper presents three load balancing algorithms suitable for such applications. Their primary advantages are (1) full integration into the application, which permits load balancing to be performed without suspending the application and without any additional messages, and (2) use of only near-neighbor communication, which facilitates scalability. Keywords: Dynamic Load Balancing; Domain Decomposition; Individual-Based Simulation; Mobile Objects; Distributed Computing. 1 Introduction Individual-based simulations are a class of applications in which a collection of entities (particles) interact with one another within a two or three-dimensional simulated space. Each entity in such a system has only a very limited local knowledge of the \problem" and has a set of simple rules to follow. A global collective behavior emerges from the interactions among individual entities. Typical examples of such applications are interactive battle simulations, particle-level simulations in physics, trac modeling, evolution and behavior modeling in biology/ecology, articial life, and advanced graphics and animation. For complex individual-based systems, the necessary performance can only be achieved by using multiple machines. This is especially true if the model is to be animated in real time giving the user the opportunity to change the model's behavior interactively. Individual-based systems can be implemented in a distributed fashion by making each node responsible for a xed portion of the problem domain. This xed portion can be assigned using either the Lagrangian method or the Eulerian method [3, 6]. In the Lagrangian method, each node is responsible for a xed set of entities. Since entities usually move in the simulated space by coordinating with their neighboring entities, this approach requires that each node keeps a snapshot of the complete simulated space and updates it throughout the simulation. It also requires that each node must communicate with all other nodes in order to detect potential proximity of entities in the simulated space. Hence this approach is not at all scalable. A conventional dynamic load balancing algorithm may be applied to correct the load imbalance, which can arise as entities perform dierent tasks or get created or destroyed during the simulation. The second approach, using the Eulerian method, assigns to each node a portion of the simulated space, together with the set of entities currently residing in that area. Unlike the Lagrangian method, this approach is scalable since each node only requires a snapshot of its neighboring nodes' simulated space. It also requires only near-neighbor communication for the purpose of proximity detection. However, since entities migrate among nodes, a load imbalance is much more likely to occur, even if all entities perform the same type and amount of work. To correct the imbalance, a load balancing algorithm must periodically re-distributes the simulated space among the nodes, together with the entities residing in the various partitions [1, 3, 6, 7, 2, 9]. In this paper we focus on the Eulerian approach due to its scalability. We used the Messengers en-

2 vironment [5, 4], which is based on the principles of autonomous self-migrating threads, called Messengers. This paradigm is very suitable for individualbased simulations, since each entity can directly be implemented as a separate Messenger, which navigates through the underlying network and interacts with other such Messengers in its proximity. We have implemented three dierent algorithms for spatial load balancing and have investigated their effect on performance. The algorithms were incorporated into a test application, which simulates the behavior of a school of sh in a simulated 2-dimensional body of water [8]. The simulated environment is partitioned using either a 1-dimensional grid (i.e., into strips) or a 2-dimensional grid (i.e., into rectangles). The individual cells are distributed over the network of computing nodes. The simulated entities (sh) are then implemented as individual Messengers, which move in the simulated environment by periodically recomputing their respective positions. The new position of each sh is based on the proximity and the vector of motion of other sh with a certain radius of visibility. When the new position of an entity is outside of the current node's environment, the entity automatically migrates to the appropriate neighboring node. Dynamic load balancing is then performed in an elegant way by moving the partitioning boundaries, which automatically forces the entities to migrate. The three load-balancing algorithms described in this paper all share the following key characteristics. 1. No centralized information. All load balancing decision are made strictly based on information exchanged among nodes sharing a logical boundary. In a 1-dimensional decomposition, each node needs to communicate with two neighbors while in a 2- dimensional decomposition it needs to communicate with eight neighbors. These numbers remain constant regardless of the size of the problem and the number of computers involved. 2. No additional messages. The load balancing algorithms introduce no new messages. All load balancing information is exchanged as part of the normal periodic information exchange among nodes, i.e., it is \piggy-backed" on messages used to migrate entities and exchange boundary information. 3. Preservation of topology. The load balancing algorithms preserve the topology of the decomposition (grid). Thus the size and shape of a grid cell may change, but it retains the same neighbors at all times. 2 Framework for Spatial Load Balancing We have developed a framework for dynamic load balancing for individual-based spatial systems. This consists of the following eight phases: 1. Exchange load information: Each node determines its load value, which is equal to the number of entities in the simulated space of the node, and sends this information to its neighboring nodes. These load values are then used to detect load imbalances and to make simulated space adjustments. 2. Determine local simulated space adjustment: Based on the load information gathered about the neighboring nodes, the imbalance factor is estimated and weighed against the load balancing overhead to determine the protability of load balancing. If it is advantageous to try to correct the load imbalance, the desired local simulated space adjustment is determined. 3. Poll neighbors for simulation space adjustment approval: If a node decides to make local space adjustments, it polls the neighbors whose local space would be aected for approval. A consensus among all the aected nodes needs to be established for consistent global space adjustments. 4. Determine neighbors whose space adjustment requests are acceptable: Each node that has been polled by any of its neighbors decides if the requested space adjustment is acceptable. 5. Exchange replies from previous poll regarding space adjustment requests: Each polled node responds regarding acceptable space adjustments. The replies from all polled nodes are gathered by the corresponding polling nodes. 6. Decide if space adjustment is to be made: Each node decides if space adjustment can be made by examining the responses received from its neighbors. It decides to make an adjustment if responses from all neighbors were armative. 7. Disseminate space adjustments: The results of the previous step are disseminated to all the nodes that might be aected by a local space adjustment. (For load balancing algorithms that require establishing consensus with near-neighbors only, this phase is not needed.) 8. Update local space: Each node updates its local simulated space to correct the load imbalance.

3 PHASES Dynamic Strips Dynamic Rectangles Dynamic Quadrilaterals D D = diameter of the logical network Table 1. Phases of Load Balancing Schemes 3 Dynamic Load Balancing Schemes We have developed three load balancing schemes for individual-based systems using the above framework. Table 1 captures the essence of each algorithm by showing the phases it consists of. Phases 1, 3, 5, and 7 require information to be exchanged among neighbors while phases 2, 4, 6, and 8 are purely computational. The necessary information exchange is \piggy-backed" on the normal messages exchanged among neighboring nodes as part of the application. The number in each entry of Table 1 gives the number of communication exchanges required for the phase. That is, phases 2, 4, 6, and 8 require zero communication exchanges. Phases 1, 3, and 5 each require one exchange. Phase 7 does not exist for dynamic strips (shown as a dash); for dynamic rectangles, it requires D exchanges to propagate the information along a row or column, where D is the diameter of the grid; and for dynamic quadrilaterals it requires two exchanges. 3.1 Dynamic Strips The 2-dimensional simulated space is initially decomposed into N strips as shown in Figure 1. Each strip is mapped onto a single workstation. Load balancing is then achieved by the overloaded nodes (senders) \pulling" the boundaries that they share with their underloaded neighbors (receivers) inward. In other words, each sender node tries to shrink its strip causing the neighboring receiver node strip to become larger, thereby forcing some of its entities to automatically hop to the receiver node. The algorithm starts with each node informing its two neighbors of its load level (phase 1). If the node is overloaded, it randomly selects one of its neighbors n1 n2 n3 n4 Figure 1. Dynamic Strips and determines how protable it will be to move the selected boundary. If this is not protable, it tries the other neighbor and determines if this would be profitable (phase 2). Load balancing protability is determined by calculating the average load of the sender and receiver nodes. If the sender node's load exceeds the average by a prespecied amount, it determines how much to move the boundary inward and then proceeds with the third phase of the load balancing process. In this phase, the sender node queries the receiver if it is okay for it to move the boundary they share with each other (phase 3). The receiver node decides on which (if any) request to accept (phase 4) and replies to the corresponding neighbor (phase 5). The decision is based on whether it has itself polled one of its neighbors (in which case it denies the request) or if it has been polled by both neighbors (in which case it selects one at random). Finally, each sender node that has received a positive response pulls its boundary inward thereby forcing some of its entities to hop to the neigh-

4 n11 n12 n13 n14 n21 n22 n23 n24 n31 n32 n33 n34 n41 n42 n43 n44 Figure 2. Dynamic Rectangles boring receiver node (phase 8). Figure 1 illustrates the scheme. The simulated space is divided into four strips n1, n2, n3 and n4. Assuming node n2 is overloaded and n1 and n3 are less loaded, then n2 selects one of them, say n1, at random. It polls n1 regarding moving the northern boundary that it shares with it to move inward. If node n1 agrees, both n1 and n2 update the boundary thereby correcting the imbalance. 3.2 Dynamic Rectangles The 2-dimensional simulated space is initially decomposed into N rectangles. Each rectangle is mapped onto a single workstation. The simulated space can be thought of in terms of 1-dimensional horizontal and vertical strips. Load is then balanced by nodes, belonging to a particular horizontal or vertical strip, pulling the same local boundaries inward by a certain amount. This is done in unison to preserve the topology of the logical network used by the simulation. The scheme starts with each node informing its eight neighbors of their load level (phase 1). Phases 2-5 are the same as in the case of dynamic strips since nodes in a row or column are trying to adjust horizontal or vertical strips. Assuming that vertical strips are being adjusted, each node selects either its eastern or western boundary that it shares with the lighter neighbor and determines how protable it will be to move the selected boundary (phase 2). In case of adjusting the horizontal strips, each overloaded node will either select its northern or southern boundary to be moved inward. The load balancing protability is determined by calculating the average load of the sender and receiver nodes. If the sender node's load exceeds the average by a prespecied amount, it determines how much to move the boundary by and proceeds with the third phase of the load balancing process. In this phase, the sender node queries the receiver if it is okay for it to move the boundary they share with each other by a certain amount (phase 3). The receiver node determines which (if any) request to accept (phase 4) and replies to the corresponding neighbor (phase 5). At this point, all the nodes along a particular column know if they are allowed to move their eastern or western boundaries. Each node along a column then propagates this information to the rest of the nodes in the same column (phase 7). This is accomplished by each node sending the information to its northern and southern neighbors. The northern and southern neighbors, in turn, passing it on to their immediate north and south neighbors and so on until the information ows to all the nodes in a particular column. After all the nodes in a particular column are made aware of each other's wish to move their eastern or western boundaries, each node determines by how much to move its boundary using the following rule: if at least half of the total number of nodes in a column wish to move their eastern boundaries inward, then each node in a column decides to move its eastern boundary by the average of the total eastern inward move requested by the rest of the nodes in a column. The nodes in each column then update their eastern and western boundaries in unison thereby adjusting the vertical strips (phase 8). An analogous rule is used for the western boundaries and the algorithm keeps alternating between adjusting vertical and horizontal strips. Figure 2 illustrates the dynamic rectangles scheme graphically. The simulated space is divided into 16 rectangles (four horizontal strips and four vertical strips). All nodes rst inform their neighbors of their load level. Assuming that nodes n12, n22, n31 and n42 are overloaded, each selects an eastern or western boundary that it shares with its lighter neighbor to move. Assume further that nodes n12, n22, and n42 selected their western boundaries and node n31 selected its eastern boundary to move inward. Nodes n12, n22, n42, and n31 then ask their neighbors n11, n21, n41, and n32 if it is okay for them to move the boundaries they share with each other. If all the receivers send positive replies, each node makes the rest of the nodes sharing its column aware of the result it has received from its underloaded neighbor. This is done by each node propagating its information in the northern and southern direction to the rest of the nodes in its column. Since at least half of the nodes in column two (n12, n22, and n42) want to move their western bound-

5 n11 n12 n13 n14 n21 n22 n23 n24 n31 n32 n33 n34 n41 n42 n43 n44 Figure 3. Dynamic Quadrilaterals aries inward, all nodes in columns 1 and 2 will do so by the average amounts requested by the nodes n12, n22, and n42, thereby correcting the imbalance while preserving the topology of the logical network. 3.3 Dynamic Quadrilaterals The 2-dimensional simulated space is initially decomposed into N rectangles. Each rectangle is mapped onto a single workstation as in the case of the dynamic rectangles algorithm. Load is then balanced by nodes \pulling" their vertices inward as long as all the affected quadrilaterals stay convex. The topology of the logical network is preserved as in the case of dynamic rectangles. When the scheme starts, it is assumed that every node, in addition to the shape of its simulated space, knows about all its neighbors' shapes. This information is needed to preserve the convexity of all the quadrilaterals. The algorithm begins by each node informing its eight neighbors of their load level (phase 1). Each node chooses a vertex to move that it shares with a diagonally underloaded node. For each vertex, the node with the largest load becomes the sender. If two or three nodes have the same (high) load then the one with the lightest diagonally opposite node is chosen. The load balancing protability is determined by calculating the average load of all four nodes incident on the same vertex. If the sender node's load exceeds the average by a prespecied amount, it calculates how much to move the vertex inward along a vector bisecting the angle formed by its two boundaries incident on the vertex (phase 2). The sender node queries the other three nodes incident on the vertex if it is okay for it to move it (phase 3). Every node that received at least one inquiry or has itself sent inquiries to its neighbors will select either itself or one of its polling neighbors at random to be the sender (that is, the node that pulls the vertex inward) (phase 5). At this point, each sender node knows if it is okay for it to move a vertex and it makes the decision on whether to move it (phase 6). A node can move its vertex only if all three of its neighboring nodes that share the vertex agree with the move. The sender node now propagates the position of its new vertex to all the nodes that will be aected by it (phase 7). The sender rst sends the position of the new vertex to its three neighboring nodes that are incident on it. Then, the three nodes incident on the new vertex propagate it to their neighboring nodes that are diagonally opposite from the new vertex. Finally, all aected nodes update their vertices and the imbalance is corrected (phase 8). Figure 3 illustrates the dynamic quadrilaterals scheme graphically. The simulated space is again divided into 16 rectangles. All the nodes rst inform their neighbors of their load level. Assume that nodes n23 and node n34 are overloaded and their diagonally opposite neighbors n32 and n43 are underloaded and that it is protable to move the vertices they share. Then, node n23 asks its neighbors n22, n32, and n33 if it is okay for them to move the vertex they all share. Similarly, n34 asks the nodes n33, n43 and n44 about their shared vertex. Nodes n22, n32, n43, and n44 reply positively as each of them has received only one request for a vertex movement. Assuming that node n33 replies positively to node n23 and denies node n34's request to move the vertex (using the rule to choose one of the nodes randomly), node n23, at this point, knows that it can move its vertex. Node n23 now propagates the new position of this vertex to all the nodes that have this information cached. Node n23 accomplishes this by rst sending the new vertex position to its neighbors n22, n32, and n33, each of which in turn propagate the information to their neighbors that are incident on the vertex diagonally opposite to the new vertex (n21, n11, n12, n13, n14, n24, n34, n44, n43, n42, n41, and n31). Finally, all the nodes update their vertices. 4 Performance Evaluation We have applied the three dynamic load balancing techniques to a generic individual-based simulation running on a cluster of Sun SPARCstations connected by a 10Mbps Ethernet. The simulation was programmed using Messengers [5, 4], an autonomousobjects-based system developed by our research group. The simulation space was a 2-dimensional 128 by 128

6 Static Strips Static Rectangles Dynamic Strips Dynamic Rectangles Dynamic Quadrilaterals 70 Execution Time (min) Number of Workstations Figure 4. Effect of load balancing under highest communication overhead Static Strips Static Rectangles Dynamic Strips Dynamic Rectangles Dynamic Quadrilaterals 9 8 Speedup Computation to Communication Ratio Figure 5. Effect of computation to communication ratio toroidal grid in which 25,000 entities moved as a single school. The school occupied an area of 10% of the total simulated body of water. The reason for choosing 25,000 entities is that we wanted to have as large a problem as possible without incurring the eects of paging. The single most important measure of eectiveness for any load balancing technique is its impact on the

7 Static Strips Static Rectangles Dynamic Strips Dynamic Rectangles Dynamic Quadrilaterals Average Maximum Occupancy Number of Workstations Figure 6. Level of load balance achieved application's total execution time. Figure 4 shows the results for dierent numbers of workstations ranging from one to 25. The curves labeled static strips and dynamic strips represent the 1-dimensional decomposition of the space; the static one is the execution without load balancing while the dynamic one uses the algorithm described in Section 3.1. The other three curves represent a 2-dimensional space decomposition. Static rectangles means no load balancing while dynamic rectangles and dynamic quadrilaterals correspond to the two load balancing methods described in Sections 3.2 and 3.3. We make the following observations. First, dynamic strips is by far the best method. The only exception is the case of four workstations, where dynamic rectangles are best. The latter provides a more favorable space decomposition but has the same message-passing overhead. At the same time, four workstations are too small a conguration to be of much interest. The reason for the favorable outcome of the dynamic strips method over the other two is communication overhead. A 1-dimensional decomposition requires signicantly fewer messages than a 2-dimensional one, since each grid cell has only two neighbors in one dimension versus eight in two dimensions. This is further supported by the observation that all methods except the dynamic strips improve execution time for up to 16 workstations but yield worse overall performance when 25 workstations are used. This deterioration is not due to the load balancing overhead, which does not introduce any additional message in any of the cases. Rather, the deterioration is due to the application itself the computation performed by each sh is simply too small compared to the necessary message communications to be able to utilize 25 workstations. To study this further, we have held the number of workstation constant at 25 and have increased the computation-to-communication ratio. That is, the length of the computation between any two moves (position adjustments) was increased by articially repeating it multiple times. Figure 5 shows the results. The x-axis represents the computation-to-communication ratio expressed as multiples of the minimal (base) execution time of Figure 4. That is, a 1 on the x-axis represents the same computation-to-communication ratio as that of Figure 4; a 2 means the computation was twice as long, and so on, all the way to 50. We make the following observations. For a small computation to communication ratio, the dynamic strips method continues to perform best. When a critical threshold is passed, both of the 2-dimensional decomposition methods surpass the 1-dimensional one. Of these, the dynamic quadrilateral method performs signicantly better than all the others. The reason is that it has much more exibility to adjust the grid than the other methods (the cells do not have to remain

8 rectangular) and thus achieves the very best load balance. This is shown in Figure 6, which plots the average maximum occupancy, that is, the maximum number of entities in any grid cell averaged over the length of the experiment. The lower the number, the better the load distribution. This is an important measure since all nodes always need to wait for the slowest one the one with the largest number of entities to process. The dynamic quadrilaterals algorithm achieves a consistently better load balance than all the other schemes. It achieves this at the expense of a more complex computation than the other methods. However, the more accurate balance, combined with the fact that it needs fewer communication phases than the dynamic rectangles methods, make it a clear winner for large problems as long as the computation-to-communication ratio is suciently large. 5 Concluding Remarks We have proposed a framework for dynamic load balancing in individual-based spatial systems and presented three specic algorithms based on this framework: (1) dynamic strips (2) dynamic rectangles and (3) dynamic quadrilaterals. The algorithms are based on known concepts and principles but their implementation in the context of individual-based spatial applications is new. Specically, all three algorithms have been designed to satisfy the following constraints: 1. The algorithms work in a fully integrated fashion with the application. Therefore, load balancing is performed without suspending the application and without introducing any additional messages; 2. The algorithms involve only near-neighbor communication, which makes them arbitrarily scalable. Due to these features, load balancing incurs very little overhead. We applied the three dynamic load balancing schemes to a generic individual-based simulation and studied its eect on performance. Our experiments show that all three algorithms are eective in improving overall performance. For simple problems and those involving a low computation-to-communication ratio, a one-dimensional space decomposition with the corresponding dynamic strips algorithm is best. For more complex problems, which are able to mask the communication overhead with adequate amounts of computation, the dynamic quadrilaterals algorithm achieves the best load balance and results in the best overall performance of the application. Additional performance results and other details may be found in [10] and on our Web page: References [1] S. Baden. Run-Time Partitioning of Scientic Continuum Calculations Running on Multiprocessors. Dissertation, U. of California, Berkeley, [2] T.W. Clark, R.v. Hanxleden, J.A. McCammon, and L.R. Scott. Parallelizing molecular dynamics using spatial decomposition. Proceedings of the IEEE Scalable High-Performance Computing Conference, pages 95{102, [3] G.C. Fox. Domain decomposition in distributed and shared memory environments. In Proc. 1st Conf. Supercomputing, Lecture Notes in Computer Sci. 287, [4] M. Fukuda, L.F. Bic, M. Dillencourt, and F. Merchant. Distributed coordination with messengers. Science of Computer Programming, 31(2), Special Issue on Coordination Models, Languages, Applications. [5] M. Fukuda, L.F. Bic, and M.B. Dillencourt. Messages versus messengers in distributed programming. In Proc. Int'l Conf. on Distributed Computing Systems (ICDCS-97), Baltimore, MD, May [6] R.V. Hanxleden and L.R. Scott. Load balancing on message passing architectures. Journal of Parallel and Distributed Computing, 13:312{324, [7] D.Y. Hinz. A run-time load balancing strategy for highly parallel systems. Acta Informatica, 29:63{ 94, [8] A. Huth and C. Wissel. The simulation of the movement of sh schools. Journal of Theoretical Biology, 156:365{385, [9] H. Lorek and M. Sonnenschein. Using parallel computers to simulate individual-oriented models in ecology: A case study. In Proceedings of the 1995 European Simulation Multiconference (ESM), pages 526{531, June [10] Fehmina Merchant. Load Balancing in Spatial Individual-based Systems using Autonomous Objects. PhD thesis, Dept. of Information and Computer Science, University of California, Irvine, Irvine, CA, 1998.