Distributed Deadlock Detection

Size: px

Start display at page:

Download "Distributed Deadlock Detection"

Shannon Simpson
6 years ago
Views:

1 Distributed Deadlock Detection COMP 512 Spring 2018 Slide material adapted from Dist. Systems (Couloris, et. al) and papers/lectures from Kshemkalyani, Singhal, 1

2 Distributed Deadlock What is deadlock? Exclusive access, hold and wait, no preemption, circular wait (4 necessary/sufficient conditions) How does deadlock in a distributed system differ from deadlock in a centralized system? Messages may be delivered out of order, garbled, or lost Processors may fail; communication may go down

3 Distributed Deadlock Detection Assumptions Reusable resources Exclusive access Only one copy of each resource 3

4 Wait For Graph The state of the system can be modeled by a directed graph called a wait for graph (WFG) [sometimes called a transaction WFG or TWF] In a WFG, nodes are processes and there is a directed edge from node P1 to node P2 if P1 is blocked and is waiting for P2 to release some resource A system is deadlocked if and only if there exists a directed cycle or knot in the WFG WFG is just general a reduced general resource graph (GRG)

5 Deadlocks Resource Deadlocks Most common A process needs resources for an activity Communication Deadlocks Processes wait to communicate with other processes in a set 5

6 Request Models Single Unit Request Model AND Request Model OR Request Model AND/OR Request Model P-out-of-Q Request Model 6

7 Detecting Deadlock In a general resource graph, a cycle is a necessary condition for deadlock If the GRG is expedient, then a knot is a sufficient condition for deadlock Deadlock existence of a cycle Knot (in exp system) deadlock Absence of knot Absence of deadlock 7

8 Deadlock Handling Strategies Deadlock prevention Deadlock avoidance Deadlock detection Deadlock detection algorithms must satisfy two conditions: No undetected deadlocks (progress) No false (phantom) deadlocks (safety) Deadlock resolution (fix it) 8

9 Deadlock Prevention Wait-die (older waits on younger; younger dies if needs to wait on older) Wound-wait (older preempts younger; younger waits on older) 9

10 Centralized Deadlock Detection A control site( coord ) constructs wait-for graphs (WFGs) and checks for directed cycles Simple, easy to implement Poor availability, lack of fault tolerance Large response time Congestion Can suffer from phantom deadlock 10

11 Ho-Ramamoorthy 2-phase Algorithm Each site maintains a status table of all processes initiated at that site (includes resources locked and resources requested by processes) Phase 1: Coord periodically requests the status table from each site and constructs WFG from these tables, searches for cycle(s) Phase 2: If cycle detected, requests status tables again, constructs WFG based onlyon transactions common to both reports If the same cycle is detected again, system is declared deadlocked Later proven that cycles in 2 consecutive reports need not result in a deadlock; hence, this algorithm detects phantom deadlocks 11

12 Ho-Ramamoorthy 1-phase Algorithm Each site maintains 2 tables: resource statustable and process status table Resource table: tracks all transactions that have locked or are requesting resources stored at that site Process table: tracks resources locked or requested by all transaction at that site Coord site periodically collects these tables from each site and constructs a WFG from transactions common to both the tables by adding a request/assignment edge between P and Q if and only if P and Q appear consistently in both of the relevant process and resource status tables Faster than 2 phase but requires more storage and bigger messages 12

13 Deadlock Detection Distributed Control (DDD) WFG is spread over different sites; any site can initiate the deadlock detection process Hierarchical Control Sites are arranged in a hierarchy A site checks for cycles only in descendants 13

14 DDD Characteristics Responsibility shared equally Global state graph spread over many sites and several sites participate in detection of cycle Detection initiated when process suspects deadlock (several sites may initiate detection for same deadlock) Proof of correctness very difficult Deadlock resolution cumbersome (several sites can detect the same deadlock and may not be aware each other) 14

15 Classes of DDD Algorithms Edge-chasing: special messages or probes circulated along edges of WFG; deadlock exists if the probe is received back by the initiator Examples: Chandy-Misra-Haas for AND model, Mitchell-Merritt Path-pushing: resource dependency information disseminated/propagated through designated paths (in the graph); basic idea: build global WFG Examples: Manasce-Muntz, Obermarck 15

16 DDD Algorithms Diffusion computation: queries diffused through WFG ( are you blocked probes sent along edges) Examples: CMH for OR model, Chandy-Herman Global state detection: exploit the ability to get a consistent snapshot of the system without freezing underlying computation Examples: Bracha-Toueg, Kshemkalyani-Singhal 16

17 Edge-Chasing Algorithm Chandy-Misra-Haas Algorithm: Based on the AND model No global WFG in constructed A probe(i, j, k) is used for deadlock detection. This probe is initiated by P i, being sent by P j, and being set to P k. This probe message is circulated via the edges of the graph; A probe returning to P i implies deadlock detection. Terms used: P j is dependenton P k, if a sequence of P j, P i1,.., P im, P k exists and each in the sequence except P k is blocked and waiting on the next. P j is locally dependenton P k, if above condition + P j, P k at same site. Each process maintains an array dependent i : dependent i (j) is true if P i knows that P j is dependent on it (initially set to false for all i & j). 17

18 Chandy-Misra-Haas Algorithm Formally, sending the probe: IF (P i is locally dependent on itself) THEN declare deadlock ELSE (for all P j and P k such that P i is locally dependent upon P j AND P j is waiting on P k AND P j and P k are on different sites) send probe(i,j,k) to the home site of P k 18

19 Chandy-Misra-Haas s Algorithm Formally, receiving the probe: IF (P k is blocked AND dependent k (i)is falseand P k has not replied to all requests of P j ) THEN dependent k (i) = true; IF (k = i) THEN declare P i is deadlocked ELSE (for all P m and P n such that P k is locally dependent upon P m AND P m is waiting on P n AND P m and P n are on different sites) send probe(i,m,n) to the home site of P n ENDIF 19

20 Chandy-Misra-Haas s Algorithm Performance: For a deadlock that spans m processes over n sites, how many messages are needed? Size of the message is 3 words Delay in deadlock detection is O(n) 20

21 Mitchell-Merritt Each node has two labels : Public & Private Private label is unique to node but may change Initially both private and public label values are same Guarantees that only one process will detect the deadlock Detects deadlock by propagating the public label backwards through the WFG When a blocked transaction reads the public label of a blocking process, it replaces its public label if the blocking process s public label is larger When a initiator process reads the message with public label equals to its own then deadlock is detected 21

22 u Mitchell-Merritt Algorithm The algorithm exhibits 4 nondeterministic state transitions State Before v Block State State After x v x Value x is any value which is larger than both u,v 1. This block step occurs when a process begins to wait on some resource held by other [ Creates an edge in WFG] 2. Label change occurs in this step for waiting process 3. Both public and private labels of the waiting process are increased to a value greater than their previous values & greater than the public label of the process being waited on

23 State Before u v Transmit State If u < v State After v v 1.Blocked transaction periodically reads public label of blocking transaction and replaces its own public label with it (provided blocking transaction s public label is larger) 2.Waiting process s private label remains unchanged

24 Detection State Before u u u State After u u u 1. Deadlock is detected when a transaction receives its own public label 2. When a process reads a public label of the waiting upon process and finds that the public label value of waiting upon process is equals to its own public label value then it determines that a cycle exists and declares deadlock

25 Path-pushing Obermarck s Algorithm (AND model) Path propagation-based algorithm Based on a database model using transaction processing (transactions totally ordered) Sites which detect a cycle in their partial WFG views convey the paths discovered to members of the (totally ordered) transaction Algorithm can detect phantoms due to its asynchronous snapshot method 25

26 Obermarck s Algorithm A site waits for deadlock-related information (produced in previous iteration) from other sites The site combines the received information with its local wait for graph to build an updated graph Non-local portion of graph is abstracted by a node called Ex(External) The site detects all cycles and breaks localcycles, i.e., those that do not contain Ex Cycles with Ex node are potential global deadlocks 26

27 Obermarck salgorithm (cont) For cycles with Ex nodes: Site builds a string Ex, T1, T2, Ex for possible global cycles This string is transmitted to all other sites where a subtransaction of T2 is waiting for a message from another transaction in the other site Reducing message exchanges: a string Ex, T1, T2, T3, Ex is sent to other sites only if T1 is (lexically) higher than T3 (otherwise, wait for another site to initiate the message) Priorities among transactions: higher priority transaction detects the deadlock Obermarck s algorithm can detect false deadlocks as the snapshot of the distributed system taken is asynchronously by different sites (global cycles can change with time and may not by reflected in the local information) Example 27

28 Obermarck salgorithm Example 28

29 Diffusion-based Algorithm (CMH for OR Model) NOT COVERED IN CLASS Initiation by a blocked process P i send query(i,i,j) to all processes P j in the dependent set DS i of P i ; num i (i) = DS i ; wait i (i) = true; /* num is local variable that represents number of query msgssent */ Blocked process P k receiving query(i,j,k) if this is engagingquery for process P k /* first query from P i */ then send query(i,k,m) to all P m in DS k ; num k (i) = DS k ; wait k (i) = true; else if wait k (i) then send a reply(i,k,j) to P j Process P k receiving reply(i,j,k) if wait k (i) then num k (i) = num k (i) -1; if num k (i) = 0 then if i = k then declare a deadlock /* reply for each query msg*/ else send reply(i, k, m) to P m, which sent the engaging query 29

30 Diffusion Algorithm: Example reply(1,6,2) P2 P1 reply(1,1,7) query(1,7,1) P3 query(1,3,4) query reply P7 P6 P5 P4 30

31 Engaging Query How to distinguish an engaging query? query(i,j,k) from the initiator contains a unique sequence number for the query apart from the tuple (i,j,k) This sequence number is used to identify subsequent queries. (e.g.,) when query(1,7,1) is received by P1 from P7, P1 checks the sequence number along with the tuple P1 understands that the query was initiated by itself and it is not an engaging query Hence, P1 sends a reply back to P7 instead of forwarding the query on all its outgoing links 31

32 Hierarchical Deadlock Detection Ho-Ramamoorthy s Algorithm Site grouped into clusters Periodically, 1 site chosen as central control site (which chooses control sites for other clusters) Each control site applies 1-phase algorithm to detect (intracluster) deadlocks Central site collects information from control sites, combines WFGs, and performs cycle search Control site Central Site Control site Control site 32

33 Hierarchical Deadlock Detection Menasce-Muntz Algorithm Sites (controllers) organized in a tree structure Leaf controllers manage local WFGs Upper controllers handle deadlock detection Each parent node maintains a global WFG (union of WFGs of its children); deadlock is detected for its children Changes are propagated upwards in the tree 33

34 Persistence and Resolution Deadlock persistence: Average time a deadlock exists before it is resolved Implication of persistence Resources unavailable for this period (affects utilization) Processes wait for this period unproductively (affects response time) Deadlock resolution Involves aborting at least one process/request involved in the deadlock Efficient resolution of deadlock requires knowledge of all processes and resources If every process detects a deadlock and tries to resolve it independently, this is highly inefficient, and several processes might be aborted unnecessarily 34

35 Deadlock Resolution Priorities for processes/transactions can be useful for resolution Consider priorities introduced in Obermarck s algorithm Highest priority process initiates and detects deadlock (initiations by lower priority ones are suppressed) When deadlock is detected, lowest priority process(es) can be aborted to resolve the deadlock After identifying the processes/requests to be aborted All resources held by the victims must be released; state of released resources restored to previous states; released resources granted to deadlocked processes All deadlock detection information concerning the victims must be removed at all the sites 35

36 Deadlock Resolution Efficient resolution of deadlock requires knowledge of all processes and resources involved in the deadlock In many algorithms The process detecting the deadlock does not know all process/resources involved More than one process may detect the same deadlock 36

37 Performance Metrics Trade off between Communication overhead Deadlock persistence time Storage overhead Computation/processing overhead Resolution overhead Theory of correctness (need for sophisticated methods to prove correctness) 37

38 Perspective There are few areas of Computer Science in which theory and practice diverge as much as in distributed deadlock detection algorithms. Discovering yet another distributed deadlock detection algorithm is the goal of many a researcher. Unfortunately, these models often have little relation to reality worst of all, a fraction of all the published algorithms in this area are just plain wrong, including those proven to be correct. 38

39 Summary Centralized Deadlock Detection Algorithms Large communication overhead Coordinator is performance bottleneck Possibility of single point of failure Not very common or effective Distributed Deadlock Detection Algorithms High complexity Detection of phantom deadlocks possible Hierarchical Deadlock Detection Algorithms Most common Efficient Solve problems introduced by other approaches 39

Distributed Deadlock

Distributed Deadlock 9.55 DS Deadlock Topics Prevention Too expensive in time and network traffic in a distributed system Avoidance Determining safe and unsafe states would require a huge number of messages