An Examination of Routing Algorithms for Parallel Computing Environments

Size: px

Start display at page:

Download "An Examination of Routing Algorithms for Parallel Computing Environments"

Martina Mills
5 years ago
Views:

1 A case can be made that the Achilles heel of parallel processing networks and clusters is that they all have to deal with the unavoidable problem of communication over the System Area Network. In distributed computing networks computers (nodes) are connected in a System Area Network (SANs). Communications between nodes are inherently slower than communications inside a node due to factors as simple as distance, power, and communications medium, and thus can slow down an entire calculation if one node is dependent on receiving data from another one. Complex calculations may be performed faster if you have more than one node working on the problem at the same time, but if the necessary communications between those nodes take too long, the most powerful processors in the world, running parallel in the most powerful computers in the world, running in parallel clusters are not going to be giving you your results any quicker. Thus, a great deal of effort in high-performance parallel-network architecture is geared towards optimizing these inter-node communications, to increase their speed and reduce the overhead that can bring down a potentially faster program. Since not much can be done about power and communications medium, we programmers tend to focus our optimizing efforts on path-selection algorithms, or algorithms to determine the most ideal route to get data (i.e. a message) from one node in a network to another. Optimizing the path selection algorithm for lower latency and increased throughput will decrease the overall execution time required to run programs on a distributed computing network. The shorter and quicker (two different concepts which will be explained below) the path a given message takes, the less time is spent by the receiving node waiting around, and thus slowing up the final result. Shorter paths

2 invariably are as close to a straight-line path as possible from source to destination, while quicker paths take the least time. As any person who has had to sit in half and hour of traffic to travel three blocks knows shorter is not always quicker, and vice versa. And this is where a lot of path-selection algorithms focus most of their efforts, in avoiding those traffic jams which threaten to derail the quickness of a shortest-path route, while not sending a message on an unnecessarily circuitous route to its destination because it has less of a chance of being clogged. These traffic jams can be caused by congestion (other nodes, sending other messages over the same routes) or deadlocks (where two or more messages are basically waiting infinitely for the other one to pass). Deterministic algorithms plot an entire route in advance, typically using global network-state data, in order to avoid those congestions. Adaptive algorithms typically need little to no global network-state data, and instead change their routes on-the-fly depending on conditions they encounter. Some algorithms can take advantage of Virtual Channels (VC), if the hardware and algorithm allow, catching deadlocks ahead of time. We are going to be taking a look at a few path-selection algorithms designed for routing data and messages in networks, especially parallel-processing clusters. These algorithms run the spectrum in the different ways to approach the path-selection problem, tackling regular and irregular topologies, deterministic and adaptive routing, even virtual channels or the lack thereof. One huge hurdle in plotting out ideal paths for messages is the topography of the network itself. Regular networks correspond to some sort of geographical graph structure (1), and thus are easier to deterministically plot out. Typically these geometrical

3 shapes are meshes (a generalization of a multi-dimensional cube, think grid-system) or tori (ring-layouts). In their 2003 paper, Yoshinaga et al. describe an algorithm for just such regular topographies as meshes and tori (1). Yoshinaga and his cohorts endeavored to create a router which was highly fault tolerant, and came up with a corresponding algorithm for it, which they titled Detour-NF, which could switch from deterministic routing to adaptive, and back, depending on a given situation and user preferences. Furthermore, they sought to expand the capabilities of the adaptive portion of their algorithm by providing a context in which both minimal and non-minimal path-selections could be available: Adaptive routing algorithms can be classified into two categories: minimal and non-minimal path selections [4]. Minimal adaptive algorithms are usually used to increase performance by improving the utilization of network resources, such as channels. However, they don t provide fault-tolerance. Therefore, non-minimal adaptive routing is required for reliable communication against faults. Once a non-minimal adaptive routing algorithm is supported, it is not so difficult to use the router as a minimal router by restricting path selection to the minimal paths. Design and Evaluation of a Fault-Tolerant Adaptive Router for Parallel Computers (Yoshinaga 1) In short, they aimed to create a fault-tolerant router that could take advantage of a wide spectrum of path-selection options with minimum necessary hardware and memory commission. They hoped to achieve this by taking any already-existing fully-adaptive routing algorithm, and changing its behavior with the use of some restrictions as well as freedoms. Fault-tolerance was added by taking the algorithm and adding one or more

4 VC s (in addition to any VC s the algorithm may already require) to any existing channels, which are treated as non-minimal-channels, as well as a function which details how to deal with these new VC s. The original algorithm and said function are then combined so that the new algorithm favors paths chosen by the original algorithm, but misroutes a message to a non-minimal VC channel if there is an unavoidable block in the path chosen by the original algorithm and no alternate minimal-routes. When a message s channel is blocked for some reason, the algorithm forces it to take a Westward, or negative turn, followed by any necessary turns, in order to leapfrog the faulty channel or blockage. Each leapfrog is started by heading in a negative direction, followed by appropriate positive or negative direction. Positive turns as the first step in a leapfrog, are not allowed. This is illustrated more clearly in their graphical representations, reproduced here. One unintended aspect of this algorithm is that if there is a blockage in the final turn of such a leapfrog, in a message headed in the Northerly, or positive direction, the route becomes unnecessarily longer, since a leapfrog, doesn t (Yoshinaga 3)

5 initiate from a positive position. Therefore they opted to allow 180 degree turns, also illustrated to the left, to allow a message to get to its destination quicker. The version of this algorithm for a torus structure fixes a weakness that does not allow the previous version (created for mesh-networks), to handle blockages on the negative edge. The torus algorithm overcomes this weakness by by utilizing wraparound channels that connect the nodes across the dateline. (3), by either preparing two of the non-minimal VC s in such a way as to eliminate torus cycles, or limiting the use of wraparound channels to a single hop per message. In essence, the torus algorithm ends up treating the network as a mesh anyway. Overall, though the hardware router meant to implement their algorithm is more expensive to build (a 29% increase in cost, measured in Flip-Flops, and a 56% increase in cell area), and the more accepted Duato s Protocol server does best in speed in each round, the Detour-NF routers have a much higher capacity for fault tolerance (which DP lacks), which gives it an advantage in random-traffic surge situations (5-7). Unfortunately, we don t get the chance to see the Detour-NF s fault-tolerance capabilities go up against other fault-tolerant routing algorithms, only against a Duato s run without any faults (which bests it every time), so we can t really get an answer as to it s comparative worth. The Detour-NF router is an admirable goal, which promised to see an increase in efficiency in any given situation due to said router s inherent pliability. A user could determine just what kind of message-routing algorithms would be used by his or her high-performance job, and presumably pick the most efficient options. Unfortunately, it

6 was an algorithm created for their specific router, and despite its potential for swappablealgorithms, it cannot be fully applied to a wider range of problems, unless the router comes with the deal, and with its marginal speedup in MHZ against a Duato s Protocol run, it may not be worth the investment. Another, more general, adaptive algorithm for regular mesh topographies comes from Ge-Ming Chiu of the IEEE Computer Society. His algorithm, the Odd-Even Turn Model, which functions without Virtual Channels, promises to greatly increase speedup compared to typical deadlock-avoiding, non-vc based algorithms, while sacrificing none of the deadlock protection. Chiu attempts to solve the un-evenness of previous non-vc, deadlock avoiding algorithms like the xy, turn model, west-first, north-last, and negativefirst algorithms, which work by either prohibiting certain turns a message can make, closely controlling them, or limiting their numbers in a general scope. Chiu redresses this unevenness by prohibiting very specific turns only in very specific situations, based on the current location. (1-2). Picture a mesh network as a grid, and label the top of the network North, the left West, the right East, and the bottom South. Then start labeling each column and row, left to right, top to bottom respectively, starting with an index of 0. Now consider North-West (NW) turn to be one where a message is traveling in a northern direction, then turns on a node, and begins traveling in a western direction. With this basic setup, you re ready to understand Chiu s algorithm, which consists of two basic rules: Rule 1. Any packet is not allowed to take an EN turn at any nodes located in an even column, and it is not allowed to take an NW turn at any nodes located in an odd column. Rule 2. Any packet is not allowed to take an ES turn at any

7 nodes located in an even column, and it is not allowed to take an SW turn at any nodes located in an odd column. The Odd-Even Turn Model for Adaptive Routing (Chiu 2) The basic premise behind these rules is that two messages taking mirror-opposite turns (for example, a message that s traveling East-to-South, and another traveling Eastto-North, but one row down), won t end up running into each other in the same column and causing a deadlock, a concept Chiu illustrated in his paper, reproduced below (3). Technically, the designations of even and odd in those rules can be interchanged, thus making the algorithm even more generally applicable. (Chiu3) The one weakness he acknowledges in his algorithm is that 180-degree turns cannot be allowed, lest it undermine the entire deadlock-avoidance system, standing in contrast, once again, to Detour-NF s insistence that they be allowed. The Odd-Even Turn Model, at the end, is just a small part of what makes a complete routing algorithm. But Chiu insists that with his turning-rules, it is easy to construct a fully-fledged routing algorithm that remains deadlock free, without the use of virtual-channels, by letting the turn-model give an adaptive algorithm options of where to

8 send the message for each hop Chiu provides such an algorithm, called ROUTE, a minimal-route algorithm. (3) Under Chiu s test, the odd-even turn model based algorithm performed well under high traffic conditions, and even handled hot-spot traffic tests the best. While the xy algorithm performed best in most simulations, its un-evenness and inadaptability proved to cause fluctuations in the networks performance, while the odd-even model held up reasonably well. The Odd-Even Turn Model stands in sharp contrast to the Detour-NF approach to routing. Where Detour-NF leaves room for third-party algorithms to potentially be swapped in, the Odd-Even Model is essentially itself a swappable portion of a larger algorithm. Where Detour seeks to limit routing, Odd-Even seeks to allow as much choice as possible, and where Detour insists on allowing a turn (namely a 180 degree turn), Odd- Even forbids it. Perhaps the most striking difference is Odd-Even s lack of Virtual Channels, which Detour-NF is dependent on. Citing the high-price of implementing VC s, Chiu determines that non-vc based, deadlock-free algorithms are important to produce. (1) The biggest advantage of Chiu s model is that it has the potential to be scaled and adapted for a variety of different and useful situations. The model itself, while providing a big piece of a deadlock-free routing algorithm, is also just a single part of one, allowing it to be integrated into more complex, or more streamlined and efficient proper routing algorithms. When discussing the importance of creating a non-vc-based routing algorithm, he points out that this very model could be used in other algorithms as a way

9 to manage VC s themselves. In addition, the model readily extends to higher-dimensional meshes, allowing it to be adapted for bigger and more expansive parallel highperformance environments. Now that we ve seen some algorithms for regular network topology, let s focus our attention on irregular topology. Where regular topology allows a network to be easily mapped to a set geometric shape or graph, irregular simply refuses to. A good example of an irregular network is the Internet, and its vast, uncounted, and difficult-totopographically-analyze structure. Let s take a look some routing algorithms that attempt to tackle this difficult topography. The Koibuchi article discusses algorithms for SANs with virtual channels and without virtual channels and compares them to show the advantages of SANs with virtual channels.(koibuchi 1) The irregular-topology algorithms we will be looking at fall into two basic categories. Low port first, random, Sancho s and Low virtual-channel first can all be done without virtual channels. Low port first, random,, low virtual-channel first, along with high virtual-channel first, high physical-channel first, and low physical-channel first are also able to take advantage of virtual channels. The first algorithm is the random selection. This algorithm doesn t take into account network congestions so while it is easy to implement is not an optimal solution. The low port first method, when given more than one available paths simply selects the port with the smallest port unique identifier. The low port first method also doesn t take into account network congestion and is only a simplistic method for directing network traffic.

10 The Up*/Down* routing algorithm proposed by Sancho is a method is a way to avoid network congestion. Up*/Down* routing does not have virtual channels for communication. It starts by going through the network and labeling all the channels with a counter that is equal to the number of legal paths crossing its channel. When faced with two or more possible paths you then remove the path with the largest counter. Then the network is relabeled with new counters and you continue to remove channels until you are left with a single path. This algorithm takes O(n^2*diameter) where n is the number of switches.(119) This may take a while but the time will be made up for in the reduction of congestion. The following algorithms use the Descending Layers (DL) routing method proposed by the Koibuchi article. DL routing improves routing over Up*/Down* by using virtual channels. DL routing uses smaller sub networks that work individually the same as Up*/Down* routing. DL routing then establishes paths across each of the different sub networks in order to insure there is a path from all nodes in the network to all other nodes in the network. For networks with routers capable of virtual channels there are better algorithms than Sanchos. Algorithms such as high virtual-channel first, high physical-channel first, low virtual-channel first, low physical-channel first can be better for avoiding network congestion. High virtual-channel first is different from Sancho s algorithm in that it selects the virtual channel with the largest counter and removes it where Sancho s removed the

11 regular channel with the largest counter. This is basically a way to use Sancho s algorithm on a system with virtual channels. (4) High physical-channel first is different in that it selects the physical channel with the largest sum of counters from every virtual channel. This algorithm avoids bottlenecks of virtual and physical channels while high virtual-channel first only avoids virtual channel bottlenecks. Bottlenecks occur when too much network traffic is sent through a single channel. This requires some messages to have to wait for the channel to clear to send and will decrease the time it takes to send a message. Low virtual-channel first is a bit different. It takes the virtual channel with the smallest count and removes all other paths making the selected one the chosen path. Similarly, low physical-channel first selects the smallest counter value on the physical channel based on the sum of the counters on each virtual channel. It also then fixes the channel and uses that path for communication. Low virtual-channel first spreads out the work and used the virtual network to keep congestion down. Low physical-channel first not only spreads the work over virtual channels it also spreads it out over physical channels.(4) High virtual channel first shows the best traffic throughput. There is a 92% increase when comparing the throughput of simplistic routine of low port first and high virtual-channel first on a DC routing network. This shows that the high virtual-channel first can send more data of the network per clock cycle. If virtual channels aren t available then you can still get a throughput increase of 90.5% by using high physicalchannel first. (6-14)

12 Low virtual-channel first and low physical-channel first do not show the speedup of the high algorithms but they do still show a significant improvement over low port first and random. Low physical-channel first gives an increase of 68.6% increase when comparing to low port. Low virtual-channel first gives an increase of 67.4% when also comparing to low port. (6-14) All algorithm in the DL routing system have higher throughput than the Up*/Down* routing algorithms. They Up*/Down* algorithms have smaller standard deviations meaning they are more consistent but the overall throughput is so much higher for DL routing that they are still the obviously better choice. This is because there are fewer channels crossing paths in the DL routing leading to shorter path hops and the algorithms are pages on the number of hops. Throughput is not the only way to measure the effectiveness of the path selection algorithms. Latency is also a central issue in evaluating different path selection algorithms. The two algorithms with the lowest latency are high virtual-channel and high physical-channel first. This means high virtual-channel and high-physical-channel not only give you the highest throughput but they also have the lowest latency of the algorithm we are looking at. The Up*/Down* routing algorithms all have worse latencies than the DC algorithms. These conclusions may not be the same if these algorithms are applied to a regular network structure. All the tests used for this discussion of path selection were based on irregular network structures. The structures of these algorithms are made for irregular network structures so results on regular topology could be different. In general, though

13 Sancho s, high virtual-channel first and high physical-channel first all perform better than low physical-channel first and low virtual-channel first for irregular topologies. Though there doesn t seem to be any single algorithm that can always be counted on for optimum optimization of SAN communications, each of these articles and algorithms show how, depending on your regular traffic patterns, network layout, and communications needs, you can usually find a way to adapt one or more algorithms (and sometimes combine them) to achieve satisfactory optimization, and keep your parallel network running with a minimum of overhead.

14 Works Cited Tsutomu Yoshinaga, Hiroyuki Hosogoshi, Masahiro Sowa, University of Electro- Communications, Tokyo, Japan, Design and Evaluation of a Fault-Tolerant Adaptive Router for Parallel Computers, Innovative Architecture for Future Generation High-Performance Processors and Systems , 2003, Kauai, HI Ge-Ming Chiu, Member, IEEE Computer Society. The Odd-Even Turn Model for Adaptive Routing. IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 11, NO. 7, JULY 2000, pp Michihiro Koibuchi *, Akiya Jouraku, Hideharu Amano. Department of Information and Computer Science, Keio University. Path selection algorithm: the strategy for designing deterministic routing from alternative paths. Parallel Computing 31 (2005) Networks and Topologies,

Design and Evaluation of a Fault-Tolerant Adaptive Router for Parallel Computers

Design and Evaluation of a Fault-Tolerant Adaptive Router for Parallel Computers Tsutomu YOSHINAGA, Hiroyuki HOSOGOSHI, Masahiro SOWA Graduate School of Information Systems, University of Electro-Communications,