EE 6900: Interconnection Networks for HPC Systems Fall 2016

Transcription

1 EE 6900: Interconnection Networks for HPC Systems Fall 2016 Avinash Karanth Kodi School of Electrical Engineering and Computer Science Ohio University Athens, OH Acknowledgement: Interconnection Networks Computer Architecture: A Quantitative Approach 4 th Edition, Appendix E Timothy Mark Pinkston University of Southern California José Duato Universidad Politécnica de Valencia with major presentation contribution from José Flich, UPV (and Cell BE EIB slides by Tom Ainsworth, USC) 1

2 Outline Introduction Interconnecting Two Devices Interconnecting Many Devices Network Topology Network, Arbitration, and Switching - 3 Switch Microarchitecture Practical Issues for Commercial Interconnection Networks Examples of Interconnection Networks Emerging Technologies Photonics, Wireless, 3D 3 Deadlock Freedom How can it arise? necessary conditions: shared resource incrementally allocated non-preemptible think of a channel as a shared resource that is acquired incrementally source buffer then dest. buffer channels along a route How do you avoid it? constrain how channel resources are allocated ex: dimension order How do you prove that a routing algorithm is deadlock free 2

3 Proof Technique Resources are logically associated with channels Messages introduce dependences between resources as they move forward Need to articulate the possible dependences that can arise between channels Show that there are no cycles in Channel Dependence Graph find a numbering of channel resources such that every legal route follows a monotonic sequence => no traffic pattern can lead to deadlock Network need not be acyclic, on channel dependence graph Channel Dependencies SAF & VCT dependency dependency Header flit Data flit Wormhole dependency When a packet holds a channel and requests another channel, there is a direct dependency between them Channel dependency graph D = G(C,E) For deterministic routing: single dependency at each node For adaptive routing: all requested channels produce dependencies, and dependency graph may contain cycles 3

4 Some Observations Deadlock is not synonymous with the occurrence of cyclic dependencies Note the goal of guaranteeing message delivery Are there exits from cycles? The set of dependencies that actually occur (wait-for) are a subset of the statically determined dependencies that can occur as determined by the routing function Breaking Cycles in Rings/Torii c 10 n0 c 0 n1 n0 n1 c 3 c 1 c 00 c 13 c 03 c 11 c 01 c 12 n3 n2 n3 n2 c 2 c 02 The configuration to the left can deadlock Add (virtual) channels We can make the channel dependency graph acyclic via routing restrictions (via the routing function) function is c 0i when j<i, c when j >i 1i 4

5 Breaking Cycles in Rings/Torii (cont.) c 10 c 10 n0 n1 c 13 c 03 c 00 c 11 c 01 c 03 c 01 c 11 c 12 n3 n2 c 02 c 02 Channels c 00 and c 13 are unused function breaks cycles in the channel dependency graph c 12 Extension to Adaptive c 10 c 10 c 00 n0 n1 c 03 c 00 c 11 c 01 c 03 c 01 c 11 c 12 n3 n2 c 02 c 02 Note the acyclic subgraph function: is c 0i j iand c 1i when j >i How can we formalize this? c 12 Additional dependencies due to adaptive routing 5

6 Key Idea Informally: The deterministic routing sub-function is used to provide an escape path for an adaptive routing protocol Packets are guaranteed to escape cyclic dependencies Formally the escape path is defined by a connected routing sub-function defined over a subset of channels sub-function supplies only escape channels In practice, this concept only used as a proof mechanism R 1 (x,y) R(x,y) for all x, y and C 1 is the set of channels supplied by R 1 Theorems Theorem: A connected routing function R for an interconnection network I is deadlock free iff there exists a routing sub-function R 1 that is connected and has no cycles in its extended channel dependency graph Holds for both adaptive and deterministic routing functions In the case of the latter, R = R1 and the extended channel dependency graph = channel dependency graph Dally & Seitz 1987 for deterministic routing Duato 1996 for adaptive routing 6

7 Deadlock Avoidance in SAF &VCT Networks c c H0 H0 n0 n1 direct dependency direct cross dependency c A3 c A0 c A1 c H1 escape channel for n0 c A3 c A1 c H1 c A2 c A2 n3 n2 c H2 Direct cross dependency c H2 sub-function R 1 is c Ai when j<i, c when j >i Hi A closer look at escape channel dependencies c A1 is an escape channel for n0 but not n3 c H2 is an escape channel for n3 Extended channel dependency graph must be acyclic Example: Constructing Deadlock Free Algorithms N1 N2 Extra virtual channel in the north direction R is deadlock free if R1 is deadlock free Making the North-last algorithm fully adaptive Add an extra adaptive channel in the North Direction The extra channel has no turn restrictions 7

8 Some More Observations Can have one escape network for all packets or multiple escape networks Construct dependency graph on escape channels Deadlock freedom: no cycles in the extended channel dependency graph of R 1 Must consider all direct and indirect dependencies There still remain dependencies we have not yet considered! Indirect Channel Dependencies Network c i request c k Direct and indirect dependencies between escape channels Channel c i is an escape channel for some destination Now we can focus all of our attention on the escape channels Add other channels for adaptive routing Define the extended channel dependency graph for escape channels Capture the use of all channels for adaptive routing 8

9 Deadlock Avoidance in Wormhole Networks c A0 c B0 c H0 c H1 c B1 c A1 n0 n1 n2 n5 n4 n3 indirect dependency function R is c Ai or c Bi j i c Hi when j >I sub-function R 1 is c Ai when j<i, c when j >i Hi Note the addition of indirect dependencies for Wormhole switching Cross and direct dependencies between escape channels transmitted through adaptive channels Deadlock Avoidance in Wormhole Networks (cont.) c H0 n0 n1 c A0 direct dependency c H0 direct cross dependency indirect dependency c A3 c H1 c A1 c A3 c H1 c A2 n3 n2 c A1 c H2 c A2 function: is c Ai j i and c Hi when j >i sub-function R 1 is c Ai when j<i, c when j >i Hi c H2 9

10 Example: Fully Adaptive Adaptive channel supplied by R S Deterministic channel supplied by R 1 D Each physical channel has a fully adaptive channel a, and DOR channel b Look at the channels supplied by R at node S Extensions to Irregular Networks Create an appropriate routing sub-function Example: Identify a distinguished node and construct a spanning tree Label each channel as up/down depending on proximity of end points to root (break ties with unique node number) restriction: a valid path traverses zero or more links in the up direction followed by zero or more links in the down direction How does this avoid cycles? Note the absence of virtual channels! 5 10

11 Extensions Extending the domain of the routing function Including source, channel, or history information Extension to central buffers Channels are really queues Associate routing functions with central queues and use queue dependencies to analyze deadlocks Domain of the routing function may be or NxN or QxN Example 2-D mesh: four queues/node, one corresponding to each direction in each dimension. Queue dependency graph is acyclic Mixed resource types (edge buffers and central queues) utilizes similar analysis techniques the extended resource dependency graph, Arbitration, and Switching Performed at each switch, regardless of topology Defines the allowed path(s) for each packet (Which paths?) Needed to direct packets through network to intended destinations Ideally: Supply as many routing options to packets as there are paths provided by the topology, and evenly distribute network traffic among network links using those paths, minimizing contention Problems: situations that cause packets never to reach their dest. Livelock Arises from an unbounded number of allowed non-minimal hops Solution: restrict the number of non-minimal (mis)hops allowed Deadlock Arises from a set of packets being blocked waiting only for network resources (i.e., links, buffers) held by other packets in the set Probability increases with increased traffic & decreased availability 22 11

12 , Arbitration, and Switching Common forms of deadlock: -induced deadlock of packets in a 2D mesh c i = channel i s i = source node i d i = destination node i p i = packet i Channel dependency graph p 1 p 1 s 1 c 3 s 2 c 0 c 1 c 2 c 0 c 4 c 1 c d 2 3 d 4 s 5 c 3 p 2 p 2 c 4 c 5 p 1 d 5 c 11 d 2 c 8 c 5 c d 7 1 c 12 p 5 p 3 p 3 p 2 c 10 c 6 c 12 c 6 c 7 c 8 s 4 c 9 s 3 p 4 p 3 p 4 p 4 c 9 c 10 c 11 A Formal Model of Message Blocking and Deadlock Resolution in Interconnection Networks, S. Warnakulasuriya 23 and T. Pinkston, IEEE Trans. on Parallel and Distributed Systems, Vol. 11, No. 3, pp , March, 2000., Arbitration, and Switching Common forms of deadlock: Protocol (Message)-induced deadlock Network End Node Interconnection Network C Hi = high-ordered channel i C Li = low-ordered channel i Q Ni,RQ = node i Request Q Q Ni,RP = node i Reply Q Memory / Cache Controller Read Request N 0 Reply Q Message Coupling Request Q Request-Reply Dependency Protocol-Induced Deadlock C H0 N3 Reply with Data Memory / Cache Controller N 0 N 1 C H0 N3 RQ RP RQ RP C L3 C L1 C H1 R 0 R 1 C L0 R3 Q N3,RP C L2 C L1 C H1 C H3 C L3 N N 2 3 C L2 R 3 R2 C L2 C H2 Crossbar Q N3,RQ C L3 C H3 C H2 C H2 A Progressive Approach to Handling Message-Dependent Deadlocks in Parallel Computer Systems, Y. Song and T. Pinkston, IEEE Trans. on Parallel and Distributed Systems, Vol. 14, No. 3, pp , March,

13 , Arbitration, and Switching Common forms of deadlock: Fault (Reconfiguration)-induced deadlock XY YX YX XY The transition from one routing function (YX routing) to another routing function (XY routing) in order to circumvent faults can create cyclic dependencies on resources that are not present in either routing function alone! Part I: A Theory for Deadlock-free Dynamic Reconfiguration of Interconnection Networks, J. Duato, O. Lysne, 25 R. Pang, and T. Pinkston, IEEE Trans. on Parallel and Distributed Systems, Vol. 16, No. 5, pp , May, 2005., Arbitration, and Switching Common strategies to deal with all forms of deadlock Deadlock avoidance: restrict allowed paths only to those that keep the global state deadlock-free Duato s Protocol: always guarantee an escape path from deadlock» Establish ordering only on a minimal (escape) set of resources» Grant escape resources in a partial or total order» Cyclic dependencies cannot form on escape resources, although cycles may form on larger set of network resources DOR (dimension-order routing) on meshes and hypercubes» Establish ordering on all resources based on network dimension DOR on rings and tori (k-ary n-cubes with wrap-around links)» Ordering on all resources between and within each dimension» Apply to multiple virtual channels (VCs) per physical channel» Alternatively, keep resources along each dimension from reaching full capacity by ensuring the existence of a bubble(s) 26 13

14 , Arbitration, and Switching Common strategies to deal with deadlock Deadlock avoidance: Deadlock avoidance in 2D mesh using DOR n i = node i c i = physical channel i c 1i = high-ordered VC i c 0i = low-ordered VC i Deadlock avoidance in ring using VCs Network graph Channel dependency graph c 13 c 00 2 VCs per physical channel A General Theory for Deadlock-free Adaptive Using a Mixed Set of Resources, J. Duato and 27 T. Pinkston, IEEE Trans. on Parallel and Distributed Systems, Vol. 12, No. 12, pp , December, 2001., Arbitration, and Switching Common strategies to deal with all forms of deadlock Deadlock recovery: allow deadlock to occur, but once a potential deadlock situation is detected, break at least one of the cyclic dependencies to gracefully recover A mechanism to detect potential deadlock is needed Regressive recovery (abort-and-retry): remove packet(s) from a dependency cycle by killing (aborting) and later re-injecting (retry) the packet(s) into the network after some delay Progressive recovery (preemptive): remove packet(s) from a dependency cycle by rerouting the packet(s) onto a deadlockfree lane Deterministic routing: routing function always supplies the same path for a given source-destination pair (e.g., DOR) Adaptive routing: routing function allows alternative paths for a given source-destination pair (e.g., Duato s Protocol, Bubble Adaptive, Disha ) Increases routing freedom to improve network efficiency, r 28 14

15 , Arbitration, and Switching Dimension-order routing (deterministic) Comparison of Freedom Legend: source destination permissible paths restricted paths freedom can increase r ( i.e., r R ) Duato s Protocol (adaptive) Recovery routing (true fully adaptive) 29, Arbitration, and Switching in centralized switched (indirect) networks Least common ancestor (LCA) routing Applicable to fat tree and other bidirectional MINs Use resources in some partial order to avoid cycles, deadlock Reach any LCA switch through any one of multiple paths Traverse down the tree to destination through a deterministic path Self routing property: switch output port at each hop is given by shifts of the destination node address (least significant bit/digit) Up*/down* routing: Universally applicable to any topology: map a tree graph onto it Assign up and down directions to network links (or VCs) Allowed paths to destination consist of zero or more up traversals followed by zero or more down traversals Up-down traversals impose partial order to avoid cycles, deadlocks 30 15

16 , Arbitration, and Switching Implementing the routing: source routing vs distributed routing Source routing (offset-based or could use absolute output port #) control unit in switches is simplified; computed at source Headers containing the route tend to be larger increase overhead payload Source Node Source routing Destination Node port identifiers offsets packet header routing unit 31, Arbitration, and Switching Implementing the routing: source routing vs distributed routing Distributed routing Next route computed by finite-state machine or by table look-up Look-ahead routing is possible: the route one hop away is supplied payload 60 table Source Node Distributed routing Destination Node port identifiers packet header routing table 32 16

17 , Arbitration, and Switching Arbitration Performed at each switch, regardless of topology Determines use of paths supplied to packets (When allocated?) Needed to resolve conflicts for shared resources by requestors Ideally: Maximize the matching between available network resources and packets requesting them At the switch level, arbiters maximize the matching of free switch output ports and packets located at switch input ports Problems: Starvation Arises when packets can never gain access to requested resources Solution: Grant resources to packets with fairness, even if prioritized Many straightforward distributed arbitration techniques for switches Two-phased arbiters, three-phased arbiters, and iterative arbiters 33, Arbitration, and Switching Arbitration request phase grant phase Only two matches out of four requests (50% matching) Two-phased arbiter request phase grant phase accept phase Now, three matches out of four requests (75% matching) Three-phased arbiter Optimizing the matching can increase r ( i.e., r A ) 34 17

18 , Arbitration, and Switching R, A, & S Characteristics of Commercial Machines Compan y Intel IBM Intel Cray System [Network] Name ASCI Red Paragon ASCI White SP Power3 [Colony] Thunter Itanium2 Tiger4 [QsNet II ] XT3 [SeaStar] Max. compute nodes [x #CPUs] 4,510 [x 2] 512 [x 16] 1,024 [x 4] 30,508 [x 1] Basic network topology 2-D mesh 64 x 64 BMIN w/8-port bidirect. switches (fat-tree or Omega) fat tree w/8-port bidirectional switches 3-D torus 40 x 32 x 24 Network routing algorithm distributed dimension-order routing source-based LCA adaptive, shortest-path routing source-based LCA adaptive, shortest path routing distributed table-based dimension-order Switch arbitration scheme 2-phased RR, distributed across switch 2-phased RR, centralized & distributed at outputs for bypass paths 2-phased RR, priority, aging, distributed at output ports 2-phased RR, distributed at output ports Network switching technique wormhole w/ no virtual channels buffered WH & VCT for multicasting, no VCs WH with 2 VCs VCT with 4 VCs Cray X1E 1,024 [x 1] 4-way bristled 2-D torus (~ 23 x 11) with express links distributed table-based dimension-order 2-phased RR, distributed at output ports VCT with 4 Vcs IBM ASC Purple pseries 575 [Federation] >1,280 [x 8] BMIN w/8-port bidirect. switches (fat-tree or Omega) source and distrib. table-based LCA adapt. shortest path 2-phased RR, centralized & distributed at outputs for bypass paths buffered WH & VCT for multicasting, 8 VCs IBM Blue Gene/L eserver Sol. [Torus Net] 65,536 [x 2] 3-D torus 32 x 32 x 64 distributed adaptive with bubble escape Duato s Protocol 2-phased SLQ, distributed at input & output VCT with 4 VCs 35 18