Reading W. Dally, C. Seitz, Deadlock-Free Message Routing on Multiprocessor Interconnection Networks,, IEEE TC, May 1987 Deadlock F. Silla, and J. Duato, Improving the Efficiency of Adaptive Routing in Networks with Irregular Topology, HIPC 1997 J. Duato A New Theory of Deadlock-Free Adaptive Routing in Wormhole Networks, IEEE TPDS, December 1993 Chapter 3 of Duato, Yalamanchili and Ni Sudhakar Yalamanchili, Georgia Institute of Technology (except as indicated) ECE 8813a (2) Overview: The Problem Guaranteeing message delivery v Fundamentally an issue of finite resources (buffers) and the manner in which they are allocated o How are distributed structural hazards handled? Waiting for resources can lead to v Deadlock: configuration of messages that cannot make progress v Livelock: messages that never reach the destination v Starvation: messages that never received requested resources o Even though the requested resource does periodically become available Deadlock v Prevention v Avoidance v Recovery Livelock v Minimal paths v Restricted non-minimal paths v Probabilistic Starvation v Resource allocation Ensuring Packet Delivery ECE 8813a (3) ECE 8813a (4) 1
Focus Deadlocked Configuration of Messages We are primarily concerned with deadlock v Starvation is addressed via fair sharing v Livelock is addressed (at the moment) by constraining messages to minimal paths Relationship between resources, traffic and conditions v Some conditions can produce others o Deadlocked configuration à livelock for other messages Routing in a 2D mesh What can we infer from this figure? v Routing v Dependencies or Wait-for relationships ECE 8813a (5) ECE 8813a (6) Some Definitions Router and Network Model s i c i d i Blocked packets wait for first available channel Fair access to channel bandwidth Routing function R delivers a set of channels v Memoryless, coherent, and connected v Note we define the domain as N N Selection function picks a channel v Determines performance properties Arbitrary topology Fair access to routing and arbitration unit Edge buffers v Theory can be extended to central buffers (we will not) Wait for any channel in a set For wormhole routing, allocated channel must be empty v See Appendix A for detailed list of assumptions Consumption Assumption ECE 8813a (7) ECE 8813a (8) 2
Concepts and Definitions Message Configurations No assumptions about the paths delivered by R v Expect that minimal paths are always included in R Coherent routing function v If a routing function supplies a path, it also supplies all sub-paths of the path Configuration v Assignment of packets to a set of queues/buffers Connected routing function S D Channels delivered by a routing function Canonical message configurations v All messages deadlocked Legal, and reachable message configurations v A legal configuration must be possible with the routing function o E.g., non-minimal paths v Since routing functions are memoryless, all legal configurations are reachable ECE 8813a (9) ECE 8813a (10) Deadlock Channel Dependencies A canonical, deadlocked, reachable, legal configuration is one which no message can advance v SAF and VCT o All packets are in transit o All adjacent buffers are full v Wormhole o Messages are in transit o Header flits and data flits cannot advance since all adjacent buffers are full Framework for proving deadlock freedom SAF & VCT dependency Wormhole dependency dependency Header flit Data flit When a packet holds a channel and requests another channel, there is a direct dependency between them Channel dependency graph D = G(C,E) For deterministic routing: single dependency at each node For adaptive routing: all requested channels produce dependencies, and dependency graph may contain cycles ECE 8813a (11) ECE 8813a (12) 3
Some Observations Deadlock is not synonymous with the occurrence of cyclic dependencies v Note the goal of guaranteeing message delivery v Are there exits from cycles? The set of dependencies that actually occur (wait-for) are a subset of the statically determined dependencies that can occur as determined by the routing function c 3 c 0 c 2 Breaking Cycles in Rings/Torii c 1 c 00 c 13 The configuration to the left can deadlock Add (virtual) channels v We can make the channel dependency graph acyclic via routing restrictions (via the routing function) Routing function is c 0i when j<i, c 1i ECE 8813a (13) ECE 8813a (14) Breaking Cycles in Rings/Torii (cont.) Extension to Adaptive Routing c 13 c 00 c 00 c 00 Cyclic dependency? Channels c 00 and c 13 are unused Routing function breaks cycles in the channel dependency graph Note the acyclic subgraph Routing function: is c 0i j iand c 1i How can we formalize this? Additional dependencies due to adaptive routing ECE 8813a (15) ECE 8813a (16) 4
Key Idea Informally: The deterministic routing sub-function is used to provide an escape path for an adaptive routing protocol v Packets are guaranteed to escape cyclic dependencies Formally the escape path is defined by a connected routing sub-function defined over a subset of channels v Routing sub-function supplies only escape channels v In practice, this concept only used as a proof mechanism v R 1 (x,y) R(x,y) for all x, y and C 1 is the set of channels supplied by R 1 Theorems Theorem: A connected routing function R for an interconnection network I is deadlock free iff there exists a routing sub-function R 1 that is connected and has no cycles in its extended channel dependency graph Holds for both adaptive and deterministic routing functions v In the case of the latter, R = R1 and the extended channel dependency graph = channel dependency graph v Dally & Seitz 1987 for deterministic routing v Duato 1996 for adaptive routing ECE 8813a (17) ECE 8813a (18) Properties Use of escape channels by a message is not unidirectional v If a message enters the escape network it can move back to the adaptive network Do not need to wait on a specific channel v Can wait on any channel including escape channels - the latter guaranteed to show up eventually if allocation is fair Application: start with a deterministic routing function and add channels to be used for adaptive routing Additional restriction required for wormhole (coherent) while this holds for SAF and VCTs Robustness of the Model No assumptions are made about paths v At least one minimal path is necessary for performance The use of NxN routing functions are practical v What are the alternatives? No assumptions about packet generation rate, packet length and destinations (other than connectivity) For wormhole, only one message can occupy a buffer v Buffers at both ends are single buffer v Thus head flits always block occupying the head of the queue head flit Cannot access escape channels ECE 8813a (19) ECE 8813a (20) 5
Canonical legal configuration Revisit Some Properties Deadlock Avoidance in SAF &VCT Networks direct dependency direct cross dependency Which legal configurations are reachable from an empty network for a given routing function? What is the importance of a legal configuration? v Can model a legal configuration as one that occurred by packet injection at the blocked node (VCT/SAF) or the packet with the tail node (wormhole) c A0 Direct cross dependency escape channel for when j<i, c Hi A closer look at escape channel dependencies v is an escape channel for but not v is an escape channel for Extended channel dependency graph must be acyclic ECE 8813a (21) ECE 8813a (22) c A0 Deadlock Avoidance in SAF &VCT Networks escape channel for direct dependency direct cross dependency Deterministic network for Example: Constructing Deadlock Free Routing Algorithms Channels returned by R Channels returned by R 1 Direct cross dependency Deterministic Network Adaptive Network Deterministic network for when j<i, c when j>i Hi Routing function: is c Ai and c Hi j i ECE 8813a (23) ECE 8813a (24) 6
Example: Constructing Deadlock Free Routing Algorithms Some More Observations N1 N2 Extra virtual channel in the north direction Can have one escape network for all packets or multiple escape networks Construct dependency graph on escape channels R is deadlock free if R1 is deadlock free Making the North-last algorithm fully adaptive v Add an extra adaptive channel in the North Direction v The extra channel has no turn restrictions v N1, E, S, and W implement North-Last Deadlock freedom: no cycles in the extended channel dependency graph of R 1 v Must consider all direct and indirect dependencies o There still remain dependencies we have not yet considered! ECE 8813a (25) ECE 8813a (26) Indirect Channel Dependencies Deadlock Avoidance in Wormhole Networks Network c A0cB0 c B1 c i request c k n5 n4 Direct and indirect dependencies between escape channels Channel c i is an escape channel for some destination Now we can focus all of our attention on the escape channels v Add other channels for adaptive routing v Define the extended channel dependency graph for escape channels o Capture the use of all channels à for adaptive routing ECE 8813a (27) indirect dependency j i Routing function R is c Ai or c Bi c Hi when j >I when j<i, c Hi Note the addition of indirect dependencies for Wormhole switching v Cross and direct dependencies between escape channels transmitted through adaptive channels ECE 8813a (28) 7
Deadlock Avoidance in Wormhole Networks (cont.) c A0cB0 c B1 c A0 Deadlock Avoidance in Wormhole Networks (cont.) direct dependency direct cross dependency indirect dependency n5 n4 indirect cross dependency j i Routing function R is c Ai or c Bi c Hi when j >I when j<i, c Hi Note the addition of indirect dependencies for Wormhole switching v Cross and direct dependencies between escape channels transmitted through adaptive channels ECE 8813a (29) j i Routing function: is c Ai and c Hi when j<i, c Hi ECE 8813a (30) Example: Fully Adaptive Routing Extensions to Irregular Networks S Adaptive channel supplied by R Deterministic channel supplied by R 1 0 3 4 7 1 2 6 Each physical channel has a fully adaptive channel a, and DOR channel b Look at the channels supplied by R at node S D Create an appropriate routing sub-function Example: v Identify a distinguished node and construct a spanning tree v Label each channel as up/down depending on proximity of end points to root (break ties with unique node number) v Routing restriction: a valid path traverses zero or more links in the up direction followed by zero or more links in the down direction How does this avoid cycles? Note the absence of virtual channels! 5 ECE 8813a (31) ECE 8813a (32) 8
Review The extended channel dependency graph must capture all dependencies v Direct and indirect cross dependencies For wormhole switching routing function must be coherent, otherwise the preceding theorem provides only a sufficient condition Complete the list of dependencies accommodated by the theorem ECE 8813a (33) channel Class 0 1 2 3 4 5 6 7 8 Extensions: Channel Classes Y1+ Y0+ X0+ X1+ X0- X1- Create equivalence classes of channels v For example, with no intra-class dependencies If the extended class dependency graph is acyclic, routing is deadlock free (sufficient condition) Y0- Y1- ECE 8813a (34) Extensions Extending the domain of the routing function v Including source, channel, or history information Extension to central buffers v Channels are really queues o Associate routing functions with central queues and use queue dependencies to analyze deadlocks v Domain of the routing function may be or NxN or QxN o Example 2-D mesh: four queues/node, one corresponding to each direction in each dimension. v Queue dependency graph is acyclic v Mixed resource types (edge buffers and central queues) utilizes similar analysis techniques à the extended resource dependency graph Summarize Solution Philosophies Defined permitted dependencies (prevention) v Correct by construction (acyclic) v Minimal routing freedom Weaken the preceding requirement (avoidance) v Ensure cycles cannot lead to deadlock v Improve routing freedom v Some creative avoidance techniques based on the theory (next) Detect and recover (Next) v Predicated on unlikelihood of occurrence v Maximal routing freedom ECE 8813a (35) ECE 8813a (36) 9