Rollback-Recovery p Σ Σ

Size: px

Start display at page:

Download "Rollback-Recovery p Σ Σ"

Beverly Wilkins
5 years ago
Views:

1 Uncoordinated Checkpointing Rollback-Recovery p Σ Σ Easy to understand No synchronization overhead Flexible can choose when to checkpoint To recover from a crash: go back to last checkpoint restart m 8 m 8

2 m 8

4 How to Avoid the Domino Effect Coordinated Checkpointing No independence Synchronization Overhead Easy Garbage Collection Communication Induced Checkpointing : detect dangerous communication patterns and checkpoint appropriately Less synchronization Less independence Complex

5 Coordinated checkpoint for every output commit High overhead if frequent I/O with external environment

6 Distributed Checkpointing at a Glance Message Logging Can avoid domino effect Works with coordinated checkpoint Independent + Simplicity + Autonomy + Scalability - Domino effect Coordinated + Consistent states + Good performance + Garbage Collection - Scalability Communicationinduced + Consistent states + Autonomy + Scalability - None is true Works with uncoordinated checkpoint Can reduce cost of output commit How Message Logging Works Logging Message Determinants To tolerate crash failures: periodically checkpoint application state; log on stable storage determinants of non-deterministic events executed after checkpointed state. Determinants for message delivery events: message m = <m.dest, m.rsn, m.data> receive sequence number Recovery: restore latest checkpointed state; replay non-deterministic events according to determinants

7 Logging Message Determinants Pessimistic Logging Determinants for message delivery events: message m = <m.dest, m.rsn, m.data> logs synchronously to stable storage the determinants of and receive sequence number before sending. Or alternatively: message m = <m.dest, m.rsn, m.source, m.ssn> Never creates orphans pointer to the message data may incur blocking straightforward recovery Sender Based Logging Optimistic Logging (Johnson and Zwaenepoel, FTCS 87) 2 sends Message log is maintained in volatile storage at the sender. A message m is logged in two steps: logging determinants. If fails before logging the i) before sending m, the sender logs its content: m is partially logged determinants of and, ii) the receiver tells the sender the receive sequence number of m, and the sender adds this information to its log: m is fully logged. becomes an orphan. q p m partially logged (m.data, m.ssn) m fully logged (ACK, m.rsn) (m.ssn, m.rsn) q blocks? Eliminates orphans during recovery non-blocking during failure-free executions rollback of correct processes complex recovery q knows m is fully logged

8 Causal Logging No blocking in failure-free executions No orphans No additional messages Tolerates multiple concurrent failures Keeps determinant in volatile memory Localized output commit Given a message m sent from m.source to m.dest, Depend(m): Log(m): { p P (p = m.dest) and p delivered m ( e p :(deliver m.dest (m) e p )) set of processes with a copy of the determinant of m in their volatile memory p orphan of a set C of crashed processes: (p C) m :(Log(m) C p Depend(m)) } The No-Orphans Consistency Condition No orphans after crash C if: m :(Log(m) C) (Depend(m) C) No orphans after any C if: m :(Depend(m) Log(m)) The Consistency Condition m :( stable(m) (Depend(m) Log(m))) Optimistic and Pessimistic No orphans after crash C if: m :(Log(m) C) (Depend(m) C) Optimistic weakens it to: m :(Log(m) C) (Depend(m) C) No orphans after any crash if: m :( stable(m) (Depend(m) Log(m))) Pessimistic strengthens it to: m :( stable(m) Depend(m) 1)

9 Causal Message Logging No orphans after any crash of size at most f if: m :( stable(m) (Depend(m) Log(m))) An Example Causal Logging: m :( stable(m) (Depend(m) Log(m))) If f = 1, stable(m) Log(m) 2 Causal strengthens it to: m : ( stable(m) ( (Depend(m) Log(m)) (Depend(m) =Log(m)) )) <#,# > <# >

Hypervisor-based Fault-tolerance. Where should RC be implemented? The Hypervisor as a State Machine. The Architecture. In hardware

Hypervisor-based Fault-tolerance. Where should RC be implemented? The Hypervisor as a State Machine. The Architecture. In hardware Where should RC be implemented? In hardware sensitive to architecture changes At the OS level state transitions hard to track and coordinate At the application level requires sophisticated application