Tardigrade: Leveraging Lightweight Virtual Machines to Easily and Efficiently Construct Fault-Tolerant Services

Size: px

Start display at page:

Download "Tardigrade: Leveraging Lightweight Virtual Machines to Easily and Efficiently Construct Fault-Tolerant Services"

Phoebe Fox
5 years ago
Views:

1 Tardigrade: Leveraging Lightweight Virtual Machines to Easily and Efficiently Construct Fault-Tolerant Services Jacob R. Lorch Andrew Baumann Lisa Glendenning Dutch T. Meyer Andrew Warfield

2 Our goal: Turn existing binaries into faulttolerant services. Jay Lorch, Microsoft Research Tardigrade 2

3 Example: FDS Metadata Service FDS Metadata server FDS Cluster [Nightingale et al., OSDI 2012] Jay Lorch, Microsoft Research Tardigrade 3

4 Example: FDS Metadata Service FDS Metadata server Paxos leader election FDS Cluster [Nightingale et al., OSDI 2012] Jay Lorch, Microsoft Research Tardigrade 4

5 Techniques for making code fault-tolerant have limitations Use state machine replication library Explicitly persist state to reliable back-end Better: Transparently make the binary Requires development resources fault-tolerant Potential for oversight Non-determinism Failing to persist state Exposing non-persisted data Bugs in crash recovery Jay Lorch, Microsoft Research Tardigrade 5

6 Outline Motivation Background: Asynchronous VM replication Our solution: Lightweight VM replication Challenges and solutions Evaluation Jay Lorch, Microsoft Research Tardigrade 6

7 Outline Motivation Background: Asynchronous VM replication Our solution: Lightweight VM replication Challenges and solutions Evaluation Jay Lorch, Microsoft Research Tardigrade 7

8 Asynchronous virtual machine replication - Remus [Cully et al., NSDI 2008] Δ primary backup Primary can crash at any time; backup is always a bit behind. Jay Lorch, Microsoft Research Tardigrade 8

9 Asynchronous virtual machine replication - Remus [Cully et al., NSDI 2008] primary backup Output buffer Jay Lorch, Microsoft Research Tardigrade 9

10 Asynchronous virtual machine replication - Remus [Cully et al., NSDI 2008] Ack(Δ) primary backup Output buffer Jay Lorch, Microsoft Research Tardigrade 10

11 Latency of ping (ms) High VM activity can delay packets Baseline Safety Scan Search Indexer Update Deduplication Processes unrelated to the service 10 can balloon client-perceived latency. 1 50th quantile 95th quantile 99th quantile 99.9th quantile Jay Lorch, Microsoft Research Tardigrade 11

12 Outline Motivation Background: Asynchronous VM replication Our solution: Lightweight VM replication Challenges and solutions Evaluation Jay Lorch, Microsoft Research Tardigrade 12

13 Our solution: Use lightweight VMs instead Other processes Lightweight VM system examples Xax [Douceur et al., OSDI 2008] Native Client [Sehr et al., IEEE S&P 2009] Drawbridge [Porter et al., ASPLOS 2011] Embassies [Howell et al., NSDI 2013] Bascule [Baumann et al., Eurosys 2013] Service process Narrow API (e.g., ~45 calls in Bascule) LVM host Host OS Jay Lorch, Microsoft Research Tardigrade 13

14 Lightweight VMs can support unmodified binaries via a library OS Service process LVM host LVM API Jay Lorch, Microsoft Research Tardigrade 14

15 Lightweight VMs can support unmodified binaries via a library OS Service process Service binary Library OS OS API Bascule has a Windows LibOS and a Linux LibOS LVM host LVM API Jay Lorch, Microsoft Research Tardigrade 15

16 A lightweight VM is encapsulated by virtue of having a narrow interface Service process Service binary Library OS OS API LVM host LVM API Jay Lorch, Microsoft Research Tardigrade 16

17 Our approach: Checkpoint by interposing on existing LVM API Checkpoint Service process Service binary Library OS OS API Interposition using existing API means LVM and LibOS don t have to change Checkpointer LVM host LVM API LVM API Jay Lorch, Microsoft Research Tardigrade 17

18 [Cully et al., NSDI 2008] Asynchronous Virtual Machine Replication Lightweight Virtual Machine Replication Service Library OS Checkpointing Host Service Library OS Checkpointing Host primary backup primary backup Jay Lorch, Microsoft Research Tardigrade 18

19 [Cully et al., NSDI 2008] Asynchronous Virtual Machine Replication Our implementation of LVMR is called Tardigrade Lightweight Virtual Machine Replication Guest (service+os) Checkpointing Host Guest (service+os) Checkpointing Host primary backup primary backup Jay Lorch, Microsoft Research Tardigrade 19

20 Outline Motivation Background: Asynchronous VM replication Our solution: Lightweight VM replication Challenges and solutions Evaluation Jay Lorch, Microsoft Research Tardigrade 20

21 Practical LVMR poses challenges See paper for details Challenges Maintaining consistency across reconfigurations Achieving performance potential Checkpointing via an existing LVM API Solutions Vertical Paxos Incremental checkpointing, Lessons checkpoint for LVM capping, parallelism, API designers scaling send buffer size Quiescing, pre-checkpointing, enforcing determinism, terminating connections Jay Lorch, Microsoft Research Tardigrade 21

22 Checkpointing uses certain LVM API Feature Ability to track changed memory pages features Purpose Efficiently compute checkpoint deltas Ability to suspend and inspect other threads Determinism when API calls are replayed Host state either replayable or regeneratable Capture consistent snapshot Prevent divergence on failover Recreate host state on backup Jay Lorch, Microsoft Research Tardigrade 22

23 Features may not always be in LVM Feature Ability to track changed memory pages APIs Workaround Missing Ability ability to suspend to suspend and and inspect other other threads Non-determinism Determinism when when API calls API calls are are replayed Host state not either replayable or or regeneratable Use exceptions, precheckpointing Hide non-determinism Expose divergence as error condition Jay Lorch, Microsoft Research Tardigrade 23

24 To capture a checkpoint, we must quiesce and capture all threads state. Guest (service + library OS) Memory Checkpointing layer What if the API doesn t let a thread suspend and inspect another thread? Host primary Jay Lorch, Microsoft Research Tardigrade 24

25 We can use exceptions to quiesce guest threads Guest (service + library OS) Checkpoint Checkpointing layer Host primary Jay Lorch, Microsoft Research Tardigrade 25

26 Exception handler quiesces and captures each guest thread s state Guest (service + library OS) Checkpoint ExceptionHandler(, ) Memory Checkpointing layer Host primary Jay Lorch, Microsoft Research Tardigrade 26

27 Synchronous system calls complicate quiescence Guest (service + library OS) Checkpointing layer Host primary Jay Lorch, Microsoft Research Tardigrade 27

28 The wait system call is easy to deal with Guest (service + library OS) Checkpointing layer Host select() file descriptor list 0x1AC 0x3BB 0x907 time-to-checkpoint primary Jay Lorch, Microsoft Research Tardigrade 28

29 General synchronous system calls require pre-checkpointing Guest (service + library OS) Checkpointing layer Host primary Jay Lorch, Microsoft Research Tardigrade 29

30 API non-determinism undermines replay CreateSemaphore() returns descriptor 0xAAA CreateSemaphore() returns descriptor 0xBBB primary backup Jay Lorch, Microsoft Research Tardigrade 30

31 An indirection table can hide nondeterminism Guest (service + library OS) Guest (service + library OS) Checkpointing layer Guest descriptor Host descriptor 0x001 0xAAA 0x002 0x932 Host Checkpointing layer Guest descriptor Host descriptor 0x001 0xBBB 0x002 0x909 Host primary backup Jay Lorch, Microsoft Research Tardigrade 31

32 State external to guest needs to be replayable or regeneratable Guest (service + library OS) LVM API Checkpointing layer API provides sockets, not packets Host LVM API TCP session state Checkpointer can t capture TCP session state! primary backup Jay Lorch, Microsoft Research Tardigrade 32

33 System-specific modifications may be necessary Guest (service + library OS) Guest (service + library OS) TCP connections get dropped on a failover. Checkpointing layer Checkpointing layer Fixing this requires a major API change to make it use Host packets rather than sockets Host TCP session state primary backup Jay Lorch, Microsoft Research Tardigrade 33

34 Outline Motivation Background: Asynchronous VM replication Our solution: Lightweight VM replication Challenges and solutions Evaluation Jay Lorch, Microsoft Research Tardigrade 34

35 Latency of ping (ms) Effect of external processes - Remus Baseline Safety Scan Search Indexer Update Deduplication th quantile 95th quantile 99th quantile 99.9th quantile Jay Lorch, Microsoft Research Tardigrade 35

36 Latency (ms) Effect of external processes - Tardigrade Baseline Safety Scan Search Indexer Update Deduplication % 95% 99% 99.9% Quantile Jay Lorch, Microsoft Research Tardigrade 36

37 Latency (ms) Effect of external processes - Tardigrade 25 Baseline Safety Scan Search Indexer Update Deduplication % 95% 99% 99.9% Quantile Jay Lorch, Microsoft Research Tardigrade 37

38 CDF (%) Memory dirtying affects checkpoint latency No dirtying 10% of net b/w 20% of net b/w 30% of net b/w 40% of net b/w 50% of net b/w Latency (ms) Jay Lorch, Microsoft Research Tardigrade 38

39 CDF (%) FDS metadata service Metadata server initially idle Cluster operating normally Checkpoint delta average size: 0.9 MB Cluster starting up Checkpoint delta average size: 1.8 MB Checkpoint interval (ms) Jay Lorch, Microsoft Research Tardigrade 39

40 CDF (%) ZKLite, a simple non-fault-tolerant Java implementation of the Zookeeper API Client request latency (ms) Jay Lorch, Microsoft Research Tardigrade 40

41 Conclusions No changes to binaries needed, making deployment simple Replicating processes rather than VMs substantially Lightweight reduces worst-case VM Examples replication latencyof good is targets: practical for making Metadata existing services Lightweight virtual machine API Coordination designers should services service consider binaries effect on fault-tolerant replication Niche web services Reasonable performance if memory dirtying rate and load are low Jay Lorch, Microsoft Research Tardigrade 41

Services in the Virtualization Plane. Andrew Warfield Adjunct Professor, UBC Technical Director, Citrix Systems

Services in the Virtualization Plane Andrew Warfield Adjunct Professor, UBC Technical Director, Citrix Systems The Virtualization Plane Applications Applications OS Physical Machine 20ms 20ms in in the