Reading Assignment T. M. Pinkston, Deadlock Characterization and Resolution in Interconnection Networks, Chapter 13 in Deadlock Resolution in Computer Integrated Systems, CRC Press 2004 Deadlock: Part II V. Puente et.al, Adaptive Bubble Router: A Design to Improve Performance in Torus Networks, Proceedings of the 1999 International Conference on Parallel Processing Sudhakar Yalamanchili, Georgia Institute of Technology (except as indicated) ECE 8813a (2) Deadlock: A Closer Look Deadlock conditions arise from more than just routing packets, e.g., routing induced deadlock v Just making use of the topology in a deadlock free manner is insufficient Other solutions to deadlock freedom beyond strictly guaranteeing avoidance v Recovery vs. avoidance Types of Deadlock Routing induced v Created by the routing function v This is what we have studied so far, but are there weaker solutions? Message induced v Addition of dependencies to an existing routing function via message type dependencies at the endpoints Reconfiguration induced v Addition of dependencies when transitioning (reconfiguration) from one routing function to another ECE 8813a (3) ECE 8813a (4) 1
Routing Induced Approaches Deadlock freedom in the wide sense v Strict avoidance + routing freedom à Duato s protocol Deadlock freedom in the weak sense v Permitted resources (cycle or knot) are never filled o Injection limitation and bubble flow control o Two-phase routing protocols n Circuit switched and PCS networks v Deflection routing o SAF and VCT only Deadlock Avoidance: Weak Sense ECE 8813a (5) Sudhakar Yalamanchili, Georgia Institute of Technology (except as indicated) Key Idea: Move Empty Buffers Local Scheme: Injection Limitation Need to ensure that empty buffers exist in every potential cycle Couple with cycle detection? Randomize & influence propagation of bubbles? Key to deadlock avoidance is guaranteeing forward progress What if we ensured there would always be at least one buffer in any cycle? v Injection limitation at sources à O(PxM) buffers/ni For SAF and VCT only Many ways to ensure the bubble condition From T. Pinkston, Chapter 13: Deadlock Resolution in Computer Integrated Systems, Macel Dekker(pubs), M. C. Zhou and M. P. Fanti (eds), 2004 ECE 8813a (7) ECE 8813a (8) 2
Bubble Flow Control Extension to Multiple Dimensions Packet injection requires two free packet buffers v One empty buffer left after injection Selection function is biased towards adaptive channels Dimension traversal is treated as an injection Bubble flow control only apply to escape channels Priority is given to traversal over injection v Traversal to a new dimension must meet buffer requirements From V. Puente et.al, Adaptive Bubble Router: A Design to Improve Perfomance in Torus Networks, From V. Puente et.al, Adaptive Bubble Router: A Design to Improve Perfomance in Torus Networks, Proceedings of the 1999 International Conference on Parallel Processing ECE 8813a (9) Proceedings of the 1999 International Conference on Parallel Processing ECE 8813a (10) Behavior Under Load Some Consequences Number of VCs required for deadlock free routing reduced to 1 Number of VCs required for fully adaptive routing is 1 Behave as in a deterministically routed network Flow control overhead is less in VCT routers Simple fast routers v Applicable only to VCT and Packet switched routers From V. Puente et.al, Adaptive Bubble Router: A Design to Improve Perfomance in Torus Networks, Proceedings of the 1999 International Conference on Parallel Processing ECE 8813a (11) ECE 8813a (12) 3
Types of Deadlock Message & Reconfiguration Induced Deadlock Routing induced v Created by the routing function v This is what we have studied so far, but are there weaker solutions? Message induced v Addition of dependencies to an existing routing function via message type dependencies at the endpoints Reconfiguration induced v Addition of dependencies when transitioning (reconfiguration) from one routing function to another Sudhakar Yalamanchili, Georgia Institute of Technology (except as indicated) ECE 8813a (14) Message Induced Deadlock Directory-Based Coherence Protocols Consider that message types have dependencies between them v For example, request à reply v Dependencies are manifested at the end points Message sequences utilize resources Local node generates a memory reference P + C Generating the request Dir Memory Network Remote node has a copy of block P + C Dir Memory Message type dependencies are transferred to resource dependencies v Message dependencies or protocol dependencies v Remember the consumption assumption Observe the message sequence between local, remote and home nodes P + C Dir Memory Home node is the physical memory location of a memory reference ECE 8813a (15) ECE 8813a (16) 4
Message Induced Deadlock: Example Message Induced Deadlock: Example Consider the chain of dependencies between message types required to implement a transaction Key: dependencies prevent consumption! Consider the request-reply sequence from R1 à R3 From T. Pinkston, Chapter 13: Deadlock Resolution in Computer Integrated Systems, Macel From T. Pinkston, Chapter 13: Deadlock Resolution in Computer Integrated Systems, Marcel Dekker(pubs), M. C. Zhou and M. P. Fanti (eds), 2004 ECE 8813a (17) Dekker(pubs), M. C. Zhou and M. P. Fanti (eds), 2004 ECE 8813a (18) Message Induced Deadlock Avoidance Using Virtual Networks for Avoidance Deadlock Free in the weak sense v Knots/cycles exist but are never filled v Size total buffer space at end points O(PxM) buffers at each node Number of nodes Number of outstanding v Ensures knots/cycles are never filled messages from a node Separate message types by resource usage v Use virtual networks to avoid cycles v This can get expensive RQ & Reply Deadlock Avoidance in the Strict Sense RQ/Reply Reply RQ & Reply RQ RQ RQ ECE 8813a (19) ECE 8813a (20) 5
Example: Alpha 21364 Message dependency chain of length 7 Torus network Two VCs for deadlock freedom and one VC for adaptive routing à 3 VCs per network v For six networks Total number of VCs = 19 v Any message type can only use at most 2 VCs! Using Virtual Networks for Avoidance Channel dependencies when R1 and R3 are requester/responder Total #VCs/ channel Routing Freedom = Message dependency chain length 1 C E L + r Minimum #VCs for avoiding routing induced deadlock Note: C E = L m E r ECE 8813a (21) ECE 8813a (22) Types of Deadlock Reconfiguration Induced Deadlock Routing induced v Created by the routing function v This is what we have studied so far, but are there weaker solutions? Message induced v Addition of dependencies to an existing routing function via message type dependencies at the endpoints Reconfiguration induced v Addition of dependencies when transitioning (reconfiguration) from one routing function to another ECE 8813a (23) Messages are routed under two different routing functions Problem when packet holds resources provided under the old routing function Produces ghost dependencies From T. Pinkston, Chapter 13: Deadlock Resolution in Computer Integrated Systems, Marcel Dekker(pubs), M. C. Zhou and M. P. Fanti (eds), 2004 ECE 8813a (24) 6
Deadlock Freedom Solutions Static reconfiguration v Flush the network v Update the routing function v Resume message transmissions relying on upper layer protocols Dynamic reconfiguration v Incremental, partial and more difficult v Packet dropping schemes v Routing function update schemes o Partitioned resources such as virtual networks We will not cover this in detail (see paper) Deadlock Recovery ECE 8813a (25) Sudhakar Yalamanchili, Georgia Institute of Technology (except as indicated) Formation of Deadlocked Configurations Approach Routing Freedom Topology Employ recovery mechanisms to break deadlocked configurations of messages Buffer Resources Packet injection What are the opportunities to create cycles? What determines the likelihood that cycles will occur? v Routing freedom + resources + injection rate Relationship between hardware complexity and routing function Predicated on the following hypothesis v Deadlocks are rare à make them rare v Therefore recovery costs are acceptable v Recovery is less resource intensive than avoidance ECE 8813a (27) ECE 8813a (28) 7
Key Questions Probability of Occurrence Probability of occurrence Characterization On-line detection Recovery techniques Influential factors v Routing freedom o Exponential decrease in probability of occurrence v Number of blocked packets o Correlated blocking patterns v Number of resource dependency cycles o Increases with routing freedom v Presence of virtual channels o Reduces the probability of blocking ECE 8813a (29) ECE 8813a (30) Characterization of Deadlocks Deadlock set v Set of messages that are deadlocked Resource set v Set of buffer resources occupied by the deadlocked set Knot cycle density v Number of unique cycles within a knot o Captures complexity of formation of deadlocked message configurations On-Line Deadlock Detection Discovery of Cycles v Typically use some form of flooding protocol v Exact detection vs. heuristics Local vs. centralized detection Use of time-outs v At nodes with packet headers o Counter + comparator v Optimal value depends on message length ECE 8813a (31) ECE 8813a (32) 8
Deadlock Recovery Principles Progressive deadlock Recovery v Robin hood approach de-allocate resources from normal packets and assign to the recovery packet o Remove any message from a deadlocked cycle o Ensure its progress towards the destination Regressive deadlock recovery v De-allocate resources from deadlocked packets v Typically destroy packets v Need some recovery mechanism, for example, endto-end flow control Trade-offs Deadlock recovery at the source is typically regressive v De-allocate and re-inject Deadlock recovery in the network can be both v Regressive propagate de-allocation signals upstream to release resources and abort the packet v Progressive re-allocation of resources from normal to deadlocked packets ECE 8813a (33) ECE 8813a (34) Progressive Deadlock Recovery Recovering from Deadlock Important to be on the output side Forms the recovery lane across routers Floating virtual channel can be used by all VCs Utilized via a separate control path v Effectively cycle stealing on the physical channels Use of recovery lanes must be deadlock free control to steal physical channel cycles ECE 8813a (35) ECE 8813a (36) 9
Number of deadlock buffers v Impact on crossbar size Location Implementation Issues v Centralized vs. edge o Rate at which deadlocked packets can drain v At the switch output vs. switch input Optimizations Sequential progressive recovery v Only one packet is permitted to enter the deadlock recovery lane v Mutual exclusion via a circulating token v Recovery lane implements a connected routing subfunction with no cyclic dependencies Concurrent recovery v Hamiltonian path based v Spanning tree based ECE 8813a (37) ECE 8813a (38) Ping and Bubble Scheme Ping and Bubble Scheme: Example Ping propagation to trace cycles Bubble insertion to permit progress Performance issues v Each insertion will permit forward progress by at least one step v Routing freedom increases the probability that deadlocks are broken quickly v Minimal adaptive routing ensures all messages are eventually delivered Permits True Fully Adaptive Routing ECE 8813a (39) From T. Pinkston, Chapter 13: Deadlock Resolution in Computer Integrated Systems, Marcel Dekker(pubs), M. C. Zhou and M. P. Fanti (eds), 2004 ECE 8813a (40) 10
Performance of Recovery Protocols Extends the range of achievable performance Resources devoted to deadlock management is minimized in recovery more resources available for performance enhancement Performance Issues Cyclic non-deadlocks can form v Extended blocking sludgelock v Occurs when available routing freedom is being extensively exploited at high loads Deadlock avoidance places acyclic dependency guarantees before routing freedom Deadlock recovery emphasizes routing freedom first, and hence can have a performance advantage From T. Pinkston, Chapter 13: Deadlock Resolution in Computer Integrated Systems, Marcel Dekker(pubs), M. C. Zhou and M. P. Fanti (eds), 2004 ECE 8813a (41) ECE 8813a (42) Routing protocols must be designed to be correct v Applications to fault tolerance and multicast Summary Recovery vs. Avoidance v Resource commitment vs. latency impact Customized solutions can favor recovery ECE 8813a (43) 11