Distributed Synchronization: outline

Transcription

1 Distributed Synchronization: outline Introduction Causality and time o Lamport timestamps o Vector timestamps o Causal communication Snapshots This presentation is based on the book: Distributed operating-systems & algorithms by Randy Chow and Theodore Johnson 1

2 Distributed systems A distributed system is a collection of independent computational nodes, communicating over a network, that is abstracted as a single coherent system o o o o o Grid computing Cloud computing ( infrastructure as a service, software as a service ) Peer-to-peer computing Sensor networks A distributed operating system allows sharing of resources and coordination of distributed computation in a transparent manner (a), (b) a distributed system (c) a multiprocessor 2

3 Distributed synchronization Underlies distributed operating systems and algorithms Processes communicate via message passing (no shared memory) Inter-node coordination in distributed systems challenged by o Lack of global state o Lack of global clock o Communication links may fail o Messages may be delayed or lost o Nodes may fail Distributed synchronization supports correct coordination in distributed systems o May no longer use shared-memory based locks and semaphores 3

4 Distributed Synchronization: outline Introduction Causality and time o Lamport timestamps o Vector timestamps o Causal communication Snapshots 4

5 Distributed computation model Events o o o Sending a message Receiving a message Timeout, internal interrupt Processors send control messages to each other o send(destination, action; parameters) Processes may declare that they are waiting for events: o Wait for A 1, A 2,., A n A 1 (source; parameters) code to handle A 1.. A n (source; parameters) code to handle A n 5

6 Causality and events ordering A distributed system has no global state nor global clock no global order on all events may be determined Each processor knows total orders on events occurring in it There is a causal relation between the sending of a message and its receipt Lamport's happened-before relation H 1. e 1 < p e 2 e 1 < H e 2 (events within same processor are ordered) 2. e 1 < m e 2 e 1 < H e 2 (each message m is sent before it is received) 3. e 1 < H e 2 AND e 2 < H e 3 e 1 < H e 3 (transitivity) Leslie Lamport (1978): Time, clocks, and the ordering of events in a distributed system 6

7 e 1 e 2 Causality and events ordering (cont'd) P 1 P 2 P 3 Time e 3 e 4 e5 e 6 e 7 e8 e 1 < H e 7? Yes. e 1 < H e 3? Yes. e 1 < H e 8? Yes. e 5 < H e 7? No. 7

8 Lamport's timestamp algorithm 1 Initially my_ts=0 2 Upon event e, 3 if e is the receipt of message m 4 my_ts=max(m.ts, my_ts) 5 my_ts++ 6 e.ts=my_ts 7 If e is the sending of message m 8 m.ts=my_ts To create a total order, ties are broken by process ID 8

9 Lamport's timestamps (cont'd) P 1 P 2 P 3 Time 1.1 e 1 e 2 e e e 5 e e 7 e

10 Distributed Synchronization: outline Introduction Causality and time o Lamport timestamps o Vector timestamps o Causal communication Snapshots 10

11 Vector timestamps - motivation Lamport's timestamps define a total order o o e 1 < H e 2 e 1.TS < e 2.TS However, e 1.TS < e 2.TS e 1 < H e 2 does not hold, in general. (concurrent events ordered arbitrarily) Definition: Message m 1 casually precedes message m 2 (written as m 1 < c m 2 ) if s(m 1 ) < H s(m 2 ) (sending m 1 happens before sending m 2 ) Definition: causality violation occurs if m 1 < c m 2 but r(m 2 ) < p r(m 1 ). In other words, m 1 is sent to processor p before m 2 but is received after it. Lamport's timestamps do not allow to detect (hence nor prevent) causality violations. 11

12 Causality violations an example P 1 P 2 P 3 Migrate O to P 2 Where is O? On p 2 M2 M1 M3 Where is O? I don t know ERROR! Causality violation between M 1 and M 3. 12

13 Vector timestamps 1 Initially my_vt=[0,,0] 2 Upon event e, 3 if e is the receipt of message m 4 for i=1 to M 5 my_vt[i]=max(m.vt[i], my_vt[i]) 6 my_vt[self]++ 7 e.vt=my_vt 8 if e is the sending of message m 9 m.vt=my_vt e 1.VT V e 2.VT if and only if e 1.VT[i] e 2.VT[i], for every i=1,,m e 1.VT < V e 2.VT if and only if e 1.VT V e 2.VT and e 1.VT e 2.VT For vector timestamps it does hold that: e 1 < VT e 2 e 1 < H e 2 13

14 An example of a vector timestamp P 1 P 2 P 3 P e 6 e.vt=(3,6,4,2) 14

15 Comparison of vector timestamps P 1 P 2 P 3 P e e1 5 e2 6 e1.vt=(5,4,1,3) e2.vt=(3,6,4,2) e3.vt=(0,0,1,3) 15

16 VTs can be used to detect causality violations P 1 P 2 P 3 Migrate O to P 2 Where is O? (2,0,1) (1,0,0) (0,0,1) On p 2 (3,0,1) M2 (3,0,2) M1 (3,1,3) I don t know (3,2,3) (3,3,3) M3 Where is O? (3,0,3) ERROR! (3,2,4) Causality violation between M 1 and M 3. 16

17 Distributed Synchronization: outline Introduction Causality and time o Lamport timestamps o Vector timestamps o Causal communication Snapshots 17

18 Preventing causality violations A processor cannot control the order in which it receives messages But it may control the order in which they are delivered to applications Application Deliver in FIFO order Message arrival order FIFO ordering x 1 x 2 x 4 x 5 x 3 Message passing Assign timestamps Deliver x 1 Deliver x 2 Buffer x 4 Buffer x 5 Deliver x 3 x 4 x 5 Protocol for FIFO message delivery (as in, e.g., TCP) 18

19 Preventing causality violations (cont'd) Senders attach a timestamp to each message Destination delays the delivery of out-of-order messages Hold back a message m until we are assured that no message m < H m will be delivered o o For every other process p, maintain the earliest timestamp of a message m that may be delivered from p Do not deliver a message if an earlier message may still be delivered from another process 19

20 Algorithm for preventing causality violations 1 earliest[1..m] initially [ <1,0, 0>, <0,1,,0>,, <0,0,,1> ] 2 blocked[1 M] initially [ {},, {} ] 3 Upon the receipt of message m from processor p 4 Delivery_list = {} 5 If (blocked[p] is empty) 6 earliest[p]=m.timestamp 7 Add m to the tail of blocked[p] 8 While ( k such that blocked[k] is non-empty AND i {1,,M} (except k and Self) not_earlier(earliest[i], earliest[k]) 9 remove the message at the head of blocked[k], put it in delivery_list 10 if blocked[k] is non-empty 11 earliest[k] m'.timestamp, where m' is at the head of blocked[k] 12 else 13 increment the k'th element of earliest[k] 14 End While 15 Deliver the messages in delivery_list, in causal order For each processor p, the earliest timestamp with which a message from p may still be delivered 20

21 Algorithm for preventing causality violations 1 earliest[1..m] initially [ <1,0, 0>, <0,1,,0>,, <0,0,,1> ] 2 blocked[1 M] initially [ {},, {} ] 3 Upon the receipt of message m from processor p 4 Delivery_list = {} 5 If (blocked[p] is empty) 6 earliest[p]=m.timestamp 7 Add m to the tail of blocked[p] 8 While ( k such that blocked[k] is non-empty AND i {1,,M} (except k and Self) not_earlier(earliest[i], earliest[k]) 9 remove the message at the head of blocked[k], put it in delivery_list 10 if blocked[k] is non-empty 11 earliest[k] m'.timestamp, where m' is at the head of blocked[k] 12 else 13 increment the k'th element of earliest[k] 14 End While 15 Deliver the messages in delivery_list, in causal order For each process p, the messages from p that were received but were not delivered yet 21

22 Algorithm for preventing causality violations 1 earliest[1..m] initially [ <1,0, 0>, <0,1,,0>,, <0,0,,1> ] 2 blocked[1 M] initially [ {},, {} ] 3 Upon the receipt of message m from processor p 4 Delivery_list = {} 5 If (blocked[p] is empty) 6 earliest[p]=m.timestamp 7 Add m to the tail of blocked[p] 8 While ( k such that blocked[k] is non-empty AND i {1,,M} (except k and Self) not_earlier(earliest[i], earliest[k]) 9 remove the message at the head of blocked[k], put it in delivery_list 10 if blocked[k] is non-empty 11 earliest[k] m'.timestamp, where m' is at the head of blocked[k] 12 else 13 increment the k'th element of earliest[k] 14 End While 15 Deliver the messages in delivery_list, in causal order List of messages to be delivered as a result of m s receipt 22

23 Algorithm for preventing causality violations 1 earliest[1..m] initially [ <1,0, 0>, <0,1,,0>,, <0,0,,1> ] 2 blocked[1 M] initially [ {},, {} ] 3 Upon the receipt of message m from processor p 4 Delivery_list = {} 5 If (blocked[p] is empty) 6 earliest[p]=m.timestamp 7 Add m to the tail of blocked[p] 8 While ( k such that blocked[k] is non-empty AND i {1,,M} (except k and Self) not_earlier(earliest[i], earliest[k]) 9 remove the message at the head of blocked[k], put it in delivery_list 10 if blocked[k] is non-empty 11 earliest[k] m'.timestamp, where m' is at the head of blocked[k] 12 else 13 increment the k'th element of earliest[k] 14 End While 15 Deliver the messages in delivery_list, in causal order If no blocked messages from p, update earliest[p] 23

24 Algorithm for preventing causality violations 1 earliest[1..m] initially [ <1,0, 0>, <0,1,,0>,, <0,0,,1> ] 2 blocked[1 M] initially [ {},, {} ] 3 Upon the receipt of message m from processor p 4 Delivery_list = {} 5 If (blocked[p] is empty) 6 earliest[p]=m.timestamp 7 Add m to the tail of blocked[p] 8 While ( k such that blocked[k] is non-empty AND i {1,,M} (except k and Self) not_earlier(earliest[i], earliest[k]) 9 remove the message at the head of blocked[k], put it in delivery_list 10 if blocked[k] is non-empty 11 earliest[k] m'.timestamp, where m' is at the head of blocked[k] 12 else 13 increment the k'th element of earliest[k] 14 End While 15 Deliver the messages in delivery_list, in causal order m is now the most recent message from p not yet delivered 24

25 Algorithm for preventing causality violations 1 earliest[1..m] initially [ <1,0, 0>, <0,1,,0>,, <0,0,,1> ] 2 blocked[1 M] initially [ {},, {} ] 3 Upon the receipt of message m from processor p 4 Delivery_list = {} 5 If (blocked[p] is empty) 6 earliest[p]=m.timestamp 7 Add m to the tail of blocked[p] 8 While ( k such that blocked[k] is non-empty AND i {1,,M} (except k and Self) not_earlier(earliest[i], earliest[k]) 9 remove the message at the head of blocked[k], put it in delivery_list 10 if blocked[k] is non-empty 11 earliest[k] m'.timestamp, where m' is at the head of blocked[k] 12 else 13 increment the k'th element of earliest[k] 14 End While 15 Deliver the messages in delivery_list, in causal order If there is a process k with delayed messages for which it is now safe to deliver its earliest message 25

26 Algorithm for preventing causality violations 1 earliest[1..m] initially [ <1,0, 0>, <0,1,,0>,, <0,0,,1> ] 2 blocked[1 M] initially [ {},, {} ] 3 Upon the receipt of message m from processor p 4 Delivery_list = {} 5 If (blocked[p] is empty) 6 earliest[p]=m.timestamp 7 Add m to the tail of blocked[p] 8 While ( k such that blocked[k] is non-empty AND i {1,,M} (except k and Self) not_earlier(earliest[i], earliest[k]) 9 remove the message at the head of blocked[k], put it in delivery_list 10 if blocked[k] is non-empty 11 earliest[k] m'.timestamp, where m' is at the head of blocked[k] 12 else 13 increment the k'th element of earliest[k] 14 End While 15 Deliver the messages in delivery_list, in causal order Remove k'th earliest message from blocked queue, make sure it is delivered 26

27 Algorithm for preventing causality violations 1 earliest[1..m] initially [ <1,0, 0>, <0,1,,0>,, <0,0,,1> ] 2 blocked[1 M] initially [ {},, {} ] 3 Upon the receipt of message m from processor p 4 Delivery_list = {} 5 If (blocked[p] is empty) 6 earliest[p]=m.timestamp 7 Add m to the tail of blocked[p] 8 While ( k such that blocked[k] is non-empty AND i {1,,M} (except k and Self) not_earlier(earliest[i], earliest[k]) 9 remove the message at the head of blocked[k], put it in delivery_list 10 if blocked[k] is non-empty 11 earliest[k] m'.timestamp, where m' is at the head of blocked[k] 12 else 13 increment the k'th element of earliest[k] 14 End While 15 Deliver the messages in delivery_list, in causal order If there are additional blocked messages of k, update earliest[k] to be the timestamp of the earliest such message 27

28 Algorithm for preventing causality violations 1 earliest[1..m] initially [ <1,0, 0>, <0,1,,0>,, <0,0,,1> ] 2 blocked[1 M] initially [ {},, {} ] 3 Upon the receipt of message m from processor p 4 Delivery_list = {} 5 If (blocked[p] is empty) 6 earliest[p]=m.timestamp 7 Add m to the tail of blocked[p] 8 While ( k such that blocked[k] is non-empty AND i {1,,M} (except k and Self) not_earlier(earliest[i], earliest[k]) 9 remove the message at the head of blocked[k], put it in delivery_list 10 if blocked[k] is non-empty 11 earliest[k] m'.timestamp, where m' is at the head of blocked[k] 12 else 13 increment the k'th element of earliest[k] 14 End While 15 Deliver the messages in delivery_list, in causal order Otherwise the earliest message that may be delivered from k would have previous timestamp with the k'th component incremented 28

29 Algorithm for preventing causality violations 1 earliest[1..m] initially [ <1,0, 0>, <0,1,,0>,, <0,0,,1> ] 2 blocked[1 M] initially [ {},, {} ] 3 Upon the receipt of message m from processor p 4 Delivery_list = {} 5 If (blocked[p] is empty) 6 earliest[p]=m.timestamp 7 Add m to the tail of blocked[p] 8 While ( k such that blocked[k] is non-empty AND i {1,,M} (except k and Self) not_earlier(earliest[i], earliest[k]) 9 remove the message at the head of blocked[k], put it in delivery_list 10 if blocked[k] is non-empty 11 earliest[k] m'.timestamp, where m' is at the head of blocked[k] 12 else 13 increment the k'th element of earliest[k] 14 End While 15 Deliver the messages in delivery_list, in causal order Finally, deliver set of messages that will not cause causality violation (if there are any). 29

30 Execution of the algorithm as multicast P 1 P 2 P 3 (0,1,0) m 1 p 3 state earliest[1] earliest[2] earliest[3] (1,0,0) (0,1,0) (0,0,1) Upon receipt of m 2 (1,1,0) m 2 (1,1,0) (0,1,0) (0,0,1) m 2 Upon receipt of m 1 (1,1,0) (0,1,0) (0,0,1) m 2 m 1 deliver m 1 (1,1,0) (0,2,0) (0,0,1) m 2 deliver m 2 (2,1,0) (0,2,0) (0,0,1) Since the algorithm is interested only in causal order of sending events, vector timestamp is incremented only upon send events. 30

31 Distributed Synchronization: outline Introduction Causality and time o Lamport timestamps o Vector timestamps o Causal communication Snapshots 31

32 Snapshots motivation phantom deadlock Assume we would like to implement a distributed deadlock-detector We record process states to check if there is a waits-for-cycle P 1 r 1 r 2 P 2 request r 1 OK p 1 p 2 1 request 2 OK r 2 Observed waits-for graph release request r 1 OK 3 request 4 p 1 p 2 r 2 Actual waits-for graph 32

33 What is a snapshot? Global system state: o S=(s 1, s 2,, s M ) local processor states o The contents L i,j =(m 1, m 2,, m k ) of each communication channel C i,j (channels assumed to be FIFO) These are messages sent but not yet received Global state must be consistent o If we observe in state s i that p i received message m from p k, then in observation s k, k must have sent m. o Each L i,j must contain exactly the set of messages sent by p i but not yet received by p j, as reflected by s i, s j. Snapshot state must be consistent, one that might have existed during the computation. Observations must be mutually concurrent that is, no observation casually precedes another observation (a consistent cut) 33

34 Snapshot algorithm informal description Upon joining the algorithm, a process records its local state The process that initiates the snapshot sends snapshot tokens to its neighbors (before sending any other messages) o Neighbors send them to their neighbors broadcast Upon receiving a snapshot token: o o a process records its state prior to sending/receiving additional messages Must then send tokens to all its other neighbors How shall we record sets L p,q? o o q receives token from p and that is the first time q receives token: L p,q ={ } q receives token from p but q received token before: L p,q ={all messages received by q from p since q received token} 34

35 Snapshot algorithm data structures The algorithm supports multiple ongoing snapshots one per process Different snapshots from same process distinguished by version number Per process variables integer my_version initially 0 integer current_snap[1..m] initially [0,,0] integer tokens_received[1..m] processor_state S[1 M] channel_state [1..M][1..M] The version number of my current snapshot 35

36 Snapshot algorithm data structures The algorithm supports multiple ongoing snapshots one per process Different snapshots from same process distinguished by version number Per process variables integer my_version initially 0 integer current_snap[1..m] initially [0,,0] integer tokens_received[1..m] processor_state S[1 M] channel_state [1..M][1..M] current_snap[r] contains version number of snapshot initiated by processor r 36

37 Snapshot algorithm data structures The algorithm supports multiple ongoing snapshots one per process Different snapshots from same process distinguished by version number Per process variables integer my_version initially 0 integer current_snap[1..m] initially [0,,0] integer tokens_received[1..m] processor_state S[1 M] channel_state [1..M][1..M] tokens_received[r] contains the number of tokens received for the snapshot initiated by processor r 37

38 Snapshot algorithm data structures The algorithm supports multiple ongoing snapshots one per process Different snapshots from same process distinguished by version number Per process variables integer my_version initially 0 integer current_snap[1..m] initially [0,,0] integer tokens_received[1..m] processor_state S[1 M] channel_state [1..M][1..M] processor_state[r] contains the state recorded for the snapshot of processor r 38

39 Snapshot algorithm data structures The algorithm supports multiple ongoing snapshots one per process Different snapshots from same process distinguished by version number Per process variables integer my_version initially 0 integer current_snap[1..m] initially [0,,0] integer tokens_received[1..m] processor_state S[1 M] channel_state [1..M][1..M] channel_state[r][q] records the state of channel from q to current processor for the snapshot initiated by processor r 39

40 Snapshot algorithm pseudo-code execute_snapshot() Wait for a snapshot request or a token Snapshot request: 1 my_version++, current_snap[self]=my_version 2 S[self] my_state 3 for each outgoing channel q, send(q, TOKEN, my_version) 4 tokens_received[self]=0 TOKEN(q; r,version): 5. If current_snap[r] < version 6. S[r] my state 7. current_snap[r]=version 8. L[r][q] empty, send token(r, version) on each outgoing channel 9. tokens_received[r] else 11. tokens_received[r] L[r][q] all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished 40

41 Snapshot algorithm pseudo-code execute_snapshot() Wait for a snapshot request or a token Snapshot request: 1 my_version++, current_snap[self]=my_version 2 S[self] my_state 3 for each outgoing channel q, send(q, TOKEN, my_version) 4 tokens_received[self]=0 TOKEN(q; r,version): 5. If current_snap[r] < version 6. S[r] my state 7. current_snap[r]=version 8. L[r][q] empty, send token(r, version) on each outgoing channel 9. tokens_received[r] else 11. tokens_received[r] L[r][q] all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished 41

42 Snapshot algorithm pseudo-code execute_snapshot() Wait for a snapshot request or a token Increment version number of this snapshot, record my local state Snapshot request: 1 my_version++, current_snap[self]=my_version 2 S[self] my_state 3 for each outgoing channel q, send(q, TOKEN, my_version) 4 tokens_received[self]=0 TOKEN(q; r,version): 5. If current_snap[r] < version 6. S[r] my state 7. current_snap[r]=version 8. L[r][q] empty, send token(r, version) on each outgoing channel 9. tokens_received[r] else 11. tokens_received[r] L[r][q] all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished 42

43 Snapshot algorithm pseudo-code execute_snapshot() Wait for a snapshot request or a token Snapshot request: 1 my_version++, current_snap[self]=my_version 2 S[self] my_state 3 for each outgoing channel q, send(q, TOKEN; self, my_version) 4 tokens_received[self]=0 Send snapshot-token on all outgoing channels, initialize number of received tokens for my snapshot to 0 TOKEN(q; r,version): 5. If current_snap[r] < version 6. S[r] my state 7. current_snap[r]=version 8. L[r][q] empty, send token(r, version) on each outgoing channel 9. tokens_received[r] else 11. tokens_received[r] L[r][q] all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished 43

44 Snapshot algorithm pseudo-code execute_snapshot() Wait for a snapshot request or a token Snapshot request: 1 my_version++, current_snap[self]=my_version 2 S[self] my_state 3 for each outgoing channel q, send(q, TOKEN; self, my_version) 4 tokens_received[self]=0 Upon receipt from q of TOKEN for snapshot (r,version) TOKEN(q; r,version): 5. If current_snap[r] < version 6. S[r] my state 7. current_snap[r]=version 8. L[r][q] empty, send token(r, version) on each outgoing channel 9. tokens_received[r] else 11. tokens_received[r] L[r][q] all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished 44

45 Snapshot algorithm pseudo-code execute_snapshot() Wait for a snapshot request or a token If this is first token for snapshot Snapshot request: (r,version) 1 my_version++, current_snap[self]=my_version 2 S[self] my_state 3 for each outgoing channel q, send(q, TOKEN; self, my_version) 4 tokens_received[self]=0 TOKEN(q; r,version): 5. If current_snap[r] < version 6. S[r] my state 7. current_snap[r]=version 8. L[r][q] empty, send token(r, version) on each outgoing channel 9. tokens_received[r] else 11. tokens_received[r] L[r][q] all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished 45

46 Snapshot algorithm pseudo-code execute_snapshot() Wait for a snapshot request or a token Snapshot request: Record local state for r'th snapshot 1 my_version++, current_snap[self]=my_version Update version number of r'th snapshot 2 S[self] my_state 3 for each outgoing channel q, send(q, TOKEN; self, my_version) 4 tokens_received[self]=0 TOKEN(q; r,version): 5. If current_snap[r] < version 6. S[r] my state 7. current_snap[r]=version 8. L[r][q] empty, send token(r, version) on each outgoing channel 9. tokens_received[r] else 11. tokens_received[r] L[r][q] all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished 46

47 Snapshot algorithm pseudo-code execute_snapshot() Wait for a snapshot request or a token Snapshot request: 1 my_version++, current_snap[self]=my_version 2 S[self] my_state 3 for each outgoing channel q, send(q, TOEKEN; self, my_version) 4 tokens_received[self]=0 TOKEN(q; r,version): 5. If current_snap[r] < version 6. S[r] my state 7. current_snap[r]=version 8. L[r][q] empty, send token(r, version) on each outgoing channel 9. tokens_received[r] else 11. tokens_received[r]++ Set of messages on channel from q is empty Send token on all outgoing channels Initialize number of received tokens to L[r][q] all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished 47

48 Snapshot algorithm pseudo-code execute_snapshot() Wait for a snapshot request or a token Snapshot request: 1 my_version++, current_snap[self]=my_version 2 S[self] my_state 3 for each outgoing channel q, send(q, TOEKEN; self, my_version) 4 tokens_received[self]=0 Not the first token for (r,version) TOKEN(q; r,version): 5. If current_snap[r] < version 6. S[r] my state 7. current_snap[r]=version 8. L[r][q] empty, send token(r, version) on each outgoing channel 9. tokens_received[r] else 11. tokens_received[r] L[r][q] all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished 48

49 Snapshot algorithm pseudo-code execute_snapshot() Wait for a snapshot request or a token Snapshot request: 1 my_version++, current_snap[self]=my_version 2 S[self] my_state 3 for each outgoing channel q, send(q, TOEKEN; self, my_version) 4 tokens_received[self]=0 Yet another token for snapshot (r,version) TOKEN(q; r,version): 5. If current_snap[r] < version 6. S[r] my state 7. current_snap[r]=version 8. L[r][q] empty, send token(r, version) on each outgoing channel 9. tokens_received[r] else 11. tokens_received[r] L[r][q] all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished 49

50 Snapshot algorithm pseudo-code execute_snapshot() Wait for a snapshot request or a token Snapshot request: 1 my_version++, current_snap[self]=my_version 2 S[self] my_state 3 for each outgoing channel q, send(q, TOEKEN; self, my_version) 4 tokens_received[self]=0 TOKEN(q; r,version): 5. If current_snap[r] < version 6. S[r] my state 7. current_snap[r]=version 8. L[r][q] empty, send token(r, version) on each outgoing channel 9. tokens_received[r] else 11. tokens_received[r]++ These messages are the state of the channel from q for snapshot (r,version) 12. L[r][q] all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished 50

51 Snapshot algorithm pseudo-code execute_snapshot() Wait for a snapshot request or a token Snapshot request: 1 my_version++, current_snap[self]=my_version 2 S[self] my_state 3 for each outgoing channel q, send(q, TOKEN; self, my_version) 4 tokens_received[self]=0 TOKEN(q; r,version): 5. If current_snap[r] < version 6. S[r] my state 7. current_snap[r]=version 8. L[r][q] empty, send token(r, version) on each outgoing channel 9. tokens_received[r] else 11. tokens_received[r]++ If all tokens of snapshot (r,version) arrived, snapshot computation is over 12. L[r][q] all messages received from q since first receiving token(r,version) 13. if tokens_received = #incoming channels, local snapshot for (r,version) is finished 51

52 Sample execution of the snapshot algorithm A token passing system Each processor receives, in its turn, a privilege token, uses it to perform privileged operations and then passes it on Assume processor p did not receive the token for a long duration so it invokes the snapshot algorithm to find out why p q 52

53 Sample execution of the snapshot algorithm 1. p requests a snapshot p t q State(p)={} 2. q sends privilege token p t q State(p)={} 3. Snapshot token arrives at q p q State(p)={} t State(q)={}, L p,q ={} 4. Snapshot token arrives at p p q State(p)={} L q,p ={ } State(q)={}, L p,q ={} Observed state: p q 53