Adaptive Routing. Claudio Brunelli Adaptive Routing Institute of Digital and Computer Systems / TKT-9636

Transcription

1 1 Adaptive Routing Adaptive Routing Basics Minimal Adaptive Routing Fully Adaptive Routing Load-Balanced Adaptive Routing Search-Based Routing Case Study: Adapted Routing in the Thinking Machines CM-5

2 2 Adaptive Routing Basics Adaptive routing algorithms use info about the state of the network to select among alternative paths to deliver a packet Since these algorithms make use of the state of the network, they are tightly coupled with flow-control mechanism In general, adaptive routing algorithms have poor worstcase performance: they use only local state information, so they balance local load but cause imbalance on a global scale Their local nature leads also to delay in responding to a change in traffic patterns Anyway, a flow control method with stiff backpressure (shallow queues) allows faster adaptation to remote congestion

3 3 Adaptive Routing Basics 8 node ring (fig. 10.1) Node 5 sends a packet to node 6, and node 3 wants to send to node 7. The problem consists in choosing the best path. How does the adaptive routing algorithm sense the state of the network? Does it use local or global information? Does it use current or historical information? Most of adaptive routing algorithms use the state of the queues at the present node to estimate local congestion (no info on the state of links elsewhere in the network One way which routers can use to sense congestion elsewhere consists in using backpressure. A backpressure signal stops transmission from the preceding node when the queues on current node fill up. Backpressure propagates backwards in the network; anyway it propagates only in case of traffic (in absence of traffic, no propagation and no info on remote congestion) (fig. 10.2)

4 4 Adaptive Routing Basics Fig demonstrates how adaptive routing performs better with stiff flow control: If a queue can hold F=4 packets, then node 3 senses the congestion just after sending 8 packets in the wrong direction; the network is load-balanced quite quickly Problem of information currency Node 3 senses the state of channel (5,6) after some time (H*F packets before, where H is the hop count to 5 and F is the capacity of the input buffer). If in the meantime node 1 starts to send packets to node 0, node 3 would mistakenly start routing packets on the newly congested channel.

5 5 Adaptive Routing Basics More complex topologies: fig Routing decisions are made at every step (not only at the source) Still sub-optimal routing due to the local nature of congestion information Fig 10.3 (a good local decision can lead to bad global route) Route a packet from 00 to 23. At node 01 we move towards node 11 to avoid slight congestion at node 02; then we face much worse condition on channels or 21-22

6 6 Minimal Adaptive Routing Routing decisions are taken at each hop At each hop a routing function generates a productive output vector The vector indentifies the output channel which will move the packet closer to the destination Network state (usually queue length) is used to select the output channel For ex. In fig 10.3 for node 01 the output vector is (1,0,1,0) because only 2 directions are useful. Then direction +x is taken, because has less congestion Minimal adaptive routing is good at locally balancing, but poor at global load balance (fig 10.4). Unable to avoid congestion for source-destination pairs with no minimal path diversity (route from node 20 to 23). Due to the fact that there is only one productive output channel (Non-minimal adaptive routing can be used to avoid this problem)

7 7 Fully Adaptive Routing Fully adaptive routing (or non-minimal routing) no longer restricts choice to the shortest path to the destination Packets may be put on channels which increase s-d distance but avoid congested channels (fig 10.5) Gives priority to the productive outputs Packets are routed towards the destination in absence of congestion, but can be routed to unproductive outputs to increase path diversity For example, a threshold is fixed for the queue length on productive outputs: a packet is sent to a productive output with a queue length below the threshold. Otherwise the packet is routed to the output (productive or not) with the shortest queue length Can lead to livelock A packet travels indefinately in the network never reaching the destination Livelock can occur if a packet is misrouted on an unproductive channel at least half of the time (fig 10.6)

8 8 Fully Adaptive Routing To avoid livelock, fully adaptive routing must guarantee progress over time Allow misrouting only a fixed number of times; after that the algorithm reverts to minimum adaptive routing If the packet starts H hops from the destination, it will traverse at most H+2M channels to reach the destination Alternative 1: allow misrouting one hop for every H' > 1 productive hops Alternative 2: based on cahotic routing. No bound on H, but makes a probabilistic argument that a packet will eventually be delivered

9 9 Load-Balanced Adaptive Routing Adaptive routing algorithms have the problem of local optimization leading to global imbalance hybrid routing algorithm, where the quadrant to route in is selected obliviously (section 9.3). Within this quadrant, adaptive routing without backtracking is used. Oblivious selection of quadrant balances the load globally, while the adaptive routing performs local balancing within the quadrant This hybrid algorithm provides very good load balance and also very good worst-case performance; anyway, on local traffic patterns it's not good as minimal or fully adaptive algorithms. Packets always make progress towards their destination, so no livelock Once the routing quadrant is selected, H (number of hops to reach the dfestination) is determined and also guaranteed

10 10 Search-based Routing The previous routing strategies do not backtrack: once they have taken a channel, they keep it They are conservative: they send a packet just along only one path Non-greedy routing: similar to a search problem. The packet is instructed to search for the best path to the destination This may involve backtracking (in case of congested channel) or broadcast headers along several paths and then transmit the data over the best one Such search-based routing algorithms are slow and use a lot of resources, so they are not much used in practice. Anyway, they are useful off line to build routing tables (for finding paths in networks)

11 11 Adaptive routing in thinking machines CM-5 CM-5 consisted of up to 16K processing nodes Node: 32-MHz SPARC processor and 4-wide vector unit 3 separate interconnection networks Data network, control network, diagnostic network Folded Clos topology with duplex connections to the processors and 2:1 concentration in the first 2 stages of switches (fig 10.8) Channel: 20Mbytes/s (4 bits wide at 40MHZ) in each direction; differential signaling Switch: 8x8 router Each node is connected to 2 separate switches via a pair of channels (aggregate per-node interface bandwidth of 40MBytes/s)

12 12 Adaptive routing in thinking machines CM-5 Duplex connection => single-point fault tolerant network If a router connected to a node breaks, then the node can continue sending and receiving messages via the second channel Each processor injects data via a memory-mapped interface Messages can contain up to 5 words of data (32-bit each) 2 switches at level 1 (attached to 4 processors) logically act as a single node, connected to (each of) 4 switches at level 2. Similarly, these 4 switches at level 2 collectively connect to each of 8 switches at level 3. So, a switch at level i can access (4)i nodes by sending messages downstream Messages from s to d are routed in 2 phases: First the message is routed upstream (right) till a common ancestor for s and d. This upstream routing is adaptive, and the message chooses randomly among the idle upstream links. When the ancestor is reached, the message is routed deterministically downstream along the unique path to d (using destination-tag routing)

13 13 Adaptive routing in thinking machines CM-5 A message has the format shown in fig. 10.9: The message is divided in 4-bit flits One flit is delivered each cycle if the sending node has credit The first flit is a height flit (specifies the height h of the message) h specifies how far (right) in the network the message must travel to reach a common ancestor of s and d After the height flit, there are h/2 route flits in the message: they contain two 2-bit route fields, each specifying one step of the downstream route

14 14 Adaptive routing in thinking machines CM-5 The h field controls the upstream routing When an upstream message enters each router, h is compared to the level l of the router If l < h the upstream phase continues; random selection of an idle upstream link to forward the message If all links are busy, the message is blocked till a link becomes idle If h = l the common ancestor has been found => downstream routing begins At each step of downstream route, one field of the leading route flit (r) is consumed and h is decremented; the LSB of h serves to select which field in r to use for output port selection h=0 when the message arrives at d The adaptivity of the upstream routing is governed by the flitlevel blocking flow control (section 13.2) To regulate flow over channels, CM-5 router uses a variant of on/off flow control (section 13.3)

15 15 Adaptive routing in thinking machines CM-5 To regulate flow over channels, CM-5 router uses a variant of on/off flow control (section 13.3) When there is space in the buffer of an input port, the receiving router sends a token to the sending router The sender uses the token to send a flit immediately (but cannot bank the tokens) If no flit to send, the token expires When the buffer is full, no tokens are sent and traffic is blocked Each CM-5 output port has a buffer which can hold a 5-word message If the output buffer of an upstream port is empty, the port is considered idle Since each router waits until an output port buffer can accept an entire message, a message can never be blocked across the router's switch