Outline Networks: Routing and Design Routing Switch Design Case Studies CS 5, Spring 99 David E. Culler Computer Science Division U.C. Berkeley 3/3/99 CS5 S99 Routing Recall: routing algorithm determines which of the possible paths are used as routes how the route is determined R: N x N -> C, which at each switch maps the destination node n d to the next channel on the route Issues: Routing mechanism» arithmetic» source-based port select» table driven» general computation Properties of the routes Deadlock feee Routing Mechanism need to select output port for each input packet in a few cycles Simple arithmetic in regular topologies ex: Dx, Dy routing in a grid» west (-x) Dx <» east (+x) Dx >» south (-y) Dx =, Dy <» north (+y) Dx =, Dy >» processor Dx =, Dy = Reduce relative address of each dimension in order Dimension-order routing in k-ary d-cubes e-cube routing in n-cube 3/3/99 CS5 S99 3 3/3/99 CS5 S99 4 Routing Mechanism (cont) Source-based message header carries series of port selects used and stripped en route CRC? Packet Format? CS-, Myrinet, MIT Artic Table-driven message header carried index for next port at next switch» o = R[i] table also gives index for following hop» o, I = R[i ] ATM, HPPI P 3 P P P Properties of Routing Algorithms Deterministic route determined by (source, dest), not intermediate state (i.e. traffic) Adaptive route influenced by traffic along the way Minimal only selects shortest paths Deadlock free no traffic pattern can lead to a situation where no packets mover forward 3/3/99 CS5 S99 5 3/3/99 CS5 S99 6 NOW Handout Page CS5 S99
Deadlock Freedom How can it arise? necessary conditions:» shared resource» incrementally allocated» non-preemptible think of a channel as a shared resource that is acquired incrementally» source buffer then dest. buffer» channels along a route How do you avoid it? constrain how channel resources are allocated ex: dimension order How do you prove that a routing algorithm is deadlock free 3/3/99 CS5 S99 7 Proof Technique resources are logically associated with channels messages introduce dependences between resources as they move forward need to articulate the possible dependences that can arise between channels show that there are no cycles in Channel Dependence Graph find a numbering of channel resources such that every legal route follows a monotonic sequence => no traffic pattern can lead to deadlock network need not be acyclic, on channel dependence graph 3/3/99 CS5 S99 Example: k-ary D array Channel Dependence Graph Thm: x,y routing is deadlock free Numbering +x channel (i,y) -> (i+,y) gets i similarly for -x with as most positive edge +y channel (x,j) -> (x,j+) gets N+j similary for -y channels any routing sequence: x direction, turn, y direction is increasing 3 7 6 3 7 3 3 9 3 3 7 3 7 3 6 9 3 3 3 33 3 7 7 7 7 3 7 7 7 7 3 6 9 6 9 6 9 6 9 3 3 3 3 33 3/3/99 CS5 S99 9 3/3/99 CS5 S99 More examples: Why is the obvious routing on X deadlock free? butterfly? tree? fat tree? Any assumptions about routing mechanism? amount of buffering? What about wormhole routing on a ring? Deadlock free wormhole networks? Basic dimension order routing techniques don t work for k-ary d-cubes only for k-ary d-arrays (bi-directional) Idea: add channels! provide multiple virtual channels to break the dependence cycle good for BW too! Cross-Bar 3 7 4 5 6 3/3/99 CS5 S99 Do not need to add links, or xbar, only buffer resources This adds nodes the the CDG, remove edges? 3/3/99 CS5 S99 NOW Handout Page CS5 S99
Breaking deadlock with virtual channels Packet switches from lo to hi channel Up*-Down* routing Given any bidirectional network Construct a spanning tree Number of the nodes increasing from leaves to roots UP increase node numbers Any Source -> Dest by UP*-DOWN* route up edges, single turn, down edges Performance? Some numberings and routes much better than others interacts with topology in strange ways 3/3/99 CS5 S99 3 3/3/99 CS5 S99 4 Turn Restrictions in X,Y +Y Minimal turn restrictions in D +y -X +X -x +x West-first -Y XY routing forbids 4 of turns and leaves no room for adaptive routing Can you allow more turns and still be deadlock free north-last -y negative first 3/3/99 CS5 S99 5 3/3/99 CS5 S99 6 Example legal west-first routes Adaptive Routing R: C x N x S -> C Essential for fault tolerance at least multipath Can improve utilization of the network Simple deterministic algorithms easily run into bad permutations Can route around failures or congestion Can combine turn restrictions with virtual channels 3/3/99 CS5 S99 7 fully/partially adaptive, minimal/non-minimal can introduce complexity or anomolies little adaptation goes a long way! 3/3/99 CS5 S99 NOW Handout Page 3 CS5 S99 3
Switch Design How do you build a crossbar I o I o I I I 3 Receiver Buffer Buffer Transmiter I O O i Cross-bar O I O 3 Routing, Scheduling I 3 phase RAM addr Din Dout I o I I I 3 O O i O O 3 3/3/99 CS5 S99 9 3/3/99 CS5 S99 buffered swtich Buffered Switch R R R R Cross-bar R R3 R Scheduling Independent routing logic per input FSM Scheduler logic arbitrates each output priority,, random Head-of-line blocking problem 3/3/99 CS5 S99 R3 How would you build a shared pool? 3/3/99 CS5 S99 Example: IBM SP vulcan switch scheduling CRC check Port Route control CRC check Port Route control Deserializ er Deserializer 64 64 Central Queue RAM 64x Many gigabit ethernet switches use similar design without the cut-through In Arb Out Arb x Crossbar 3/3/99 CS5 S99 3 64 Serializer Ser ializer Ouput Port XBar Arb Ouput Port CRC Gen XBar CRC Arb Gen Buffers R R R R3 Cross-bar n independent arbitration problems? static priority, random, round-robin simplifications due to routing algorithm? general case is max bipartite matching 3/3/99 CS5 S99 4 O O O NOW Handout Page 4 CS5 S99 4
Stacked Dimension Switches Dimension order on 3D cube? Cube connected cycles? Zin Yin Host In x x Zout Yout What do you do when push comes to shove? ethernet: collision detection and retry after delay FDDI, token ring: arbitration token TCP/WAN: buffer, drop, adjust rate any solution must adjust to output rate Link-level flow control Xin x Xout Ready Host Out Data 3/3/99 CS5 S99 5 3/3/99 CS5 S99 6 Examples Smoothing the flow Short Links F/E Ready/Ack Req F/E Incoming Phits -control Symbols Full Source Data Destination Stop Go High Mark Low Mark long links several flits on the wire Empty Outgoing Phits How much slack do you need to maximize bandwidth? 3/3/99 CS5 S99 7 3/3/99 CS5 S99 Link vs global flow control Example: T3D Hot Spots Global communication operations Natural parallel program dependences Read Req Read Resp Read Resp Write Req Write Req Write Resp - no cache - cached - Proc - proc 4 - cache - BLT - BLT 4 - prefetch - fetch&inc - fetch&inc Route Tag Route Tag Route Tag Route Tag Route Tag Route Tag Dest PE Dest PE Dest PE Dest PE Dest PE Dest PE Command Command Command Command Command Command Addr Word Word Addr Addr Addr Word Addr Addr Src PE Word Src PE Src PE Word 3 Word Word Word Word Word 3 Packet Type req/resp coomand BLT Read Req Route Tag Dest PE Command Addr Addr Src PE Addr Addr 3/3/99 CS5 S99 9 3 3D bidirectional torus, dimension order (NIC selected), virtual cut-through, packet sw. 6 bit x 5 MHz, short, wide, synch. rotating priority per output logically separate request/response 3 independent, stacked switches 6-bit flits on each of 4 VC in each directions 3/3/99 CS5 S99 3 NOW Handout Page 5 CS5 S99 5
Example: SP 6-node Rack Switch Board Inter-Rack External Switch E E E E 3 E 5 P P P P 3 P 5 Intra-Rack Host Multi-rack Configuration -port switch, 4 MB/s per link, -bit phit, 6-bit flit, single 4 MHz clock packet sw, cut-through, no virtual channel, source-based routing variable packet <= 55 bytes, 3 byte fifo per input, 7 bytes per output, 6 phit links -byte chunks in central queue, LRU per output 3/3/99 run in shadow mode CS5 S99 3 Summary Routing Algorithms restrict the set of routes within the topology simple mechanism selects turn at each hop arithmetic, selection, lookup Deadlock-free if channel dependence graph is acyclic limit turns to eliminate dependences add separate channel resources to break dependences combination of topology, algorithm, and switch design Deterministic vs adaptive routing Switch design issues input/output/pooled buffering, routing logic, selection logic control Real networks are a package of design choices 3/3/99 CS5 S99 3 NOW Handout Page 6 CS5 S99 6