Verdi: A Framework for Implementing and Formally Verifying Distributed Systems

Transcription

1 Verdi: A Framework for Implementing and Formally Verifying Distributed Systems Key-value store VST James R. Wilcox, Doug Woos, Pavel Panchekha, Zach Tatlock, Xi Wang, Michael D. Ernst, Thomas Anderson

2 Challenges

3 Challenges Distributed systems run in unreliable environments

4 Challenges Distributed systems run in unreliable environments Many types of failure can occur

5 Challenges Distributed systems run in unreliable environments Many types of failure can occur Fault-tolerance mechanisms are challenging to implement correctly

6 Challenges Distributed systems run in unreliable environments Many types of failure can occur Fault-tolerance mechanisms are challenging to implement correctly

7 Challenges Contributions Distributed systems run in unreliable environments Formalize network as operational semantics Many types of failure can occur Build semantics for a variety of fault models Fault-tolerance mechanisms are challenging to implement correctly Verify fault-tolerance as transformation between semantics

8 Verdi Workflow Build, verify system in simple semantics Client I/O Key-value store

9 Verdi Workflow Build, verify system in simple semantics Client I/O Key-value store V Apply verified system transformer S T KV Consensus Client I/O KV Consensus KV Consensus

10 Verdi Workflow Build, verify system in simple semantics Client I/O Key-value store V Apply verified system transformer T S KV End-to-end correctness by composition Client I/O KV Consensus Consensus KV Consensus

11 Contributions Formalize network as operational semantics Build semantics for a variety of fault models Verify fault-tolerance as transformation between semantics

12 Contributions General Approach Formalize network as operational semantics Find environments in your problem domain Build semantics for a variety of fault models Formalize these environments as operational semantics Verify fault-tolerance as transformation between semantics Verify layers as transformations between semantics

13 Verdi Successes Applications Key-value store Lock service Fault-tolerance mechanisms Sequence numbering Retransmission Primary-backup replication Consensus-based replication linearizability

14 Verdi Successes Applications Key-value store Lock service Fault-tolerance mechanisms Sequence numbering Retransmission Primary-backup replication Consensus-based replication linearizability

15 Important data

16 Replicated KV store Replicated KV store Replicated KV store Replicated for availability

17 Replicated KV store Replicated KV store Replicated KV store

18 Crash Reorder Drop Duplicate Partition... Replicated KV store Replicated KV store Replicated KV store

19 Crash Reorder Drop Duplicate Partition... Replicated KV store Replicated KV store Replicated KV store Environment is unreliable

20 Crash Reorder Drop Duplicate Partition... Replicated KV store Replicated KV store Replicated KV store Decades of research; still difficult to implement correctly Implementations often have bugs

21 Bug-free Implementations

22 Bug-free Implementations

23 Bug-free Implementations Several inspiring successes in formal verification CompCert, sel4, Jitk, Bedrock, IronClad, Frenetic, Quark

24 Bug-free Implementations Several inspiring successes in formal verification CompCert, sel4, Jitk, Bedrock, IronClad, Frenetic, Quark Goal: formally verify distributed system implementations

25 Formally Verify Distributed Implementations

26 Formally Verify Distributed Implementations Separate independent system components

27 Formally Verify Distributed Implementations App Fault tolerance App Fault tolerance App Fault tolerance Separate independent system components

28 Formally Verify Distributed Implementations App Fault tolerance App Fault tolerance App Fault tolerance Separate independent system components Verify application logic independently from fault-tolerance tolerance

29 Formally Verify Distributed Implementations KV Consensus KV Consensus KV Consensus Separate independent system components Verify application key-value store logic independently from consensus

30 Formally Verify Distributed Implementations KV KV 1. Verify application logic Consensus 2. Verify fault tolerance mechanism KV 3. Run the system! Consensus Consensus Separate independent system components Verify application key-value store logic independently from consensus

31 1. Verify Application Logic Client I/O Key-value store

32 1. Verify Application Logic Simple model, prove good map Client I/O Key-value store

33 1. Verify Application Logic Simple model, prove good map Client I/O Key-value store

34 2. Verify Fault Tolerance Mechanism Simple model, prove good map Client I/O Key-value store

35 2. Verify Fault Tolerance Mechanism Simple model, prove good map Client I/O Key-value store V S T

36 2. Verify Fault Tolerance Mechanism Simple model, prove good map Client I/O Key-value store V S T KV Consensus Client I/O KV Consensus KV Consensus

37 2. Verify Fault Tolerance Mechanism Simple model, prove good map Client I/O Key-value store V Apply verified system transformer, prove properties preserved S T KV Consensus Client I/O KV Consensus KV Consensus

38 2. Verify Fault Tolerance Mechanism Simple model, prove good map Client I/O Key-value store V Apply verified system transformer, prove properties preserved S T KV Consensus Client I/O KV Consensus KV Consensus

39 2. Verify Fault Tolerance Mechanism Simple model, prove good map Client I/O Key-value store V Apply verified system transformer, prove properties preserved S T KV End-to-end correctness by composition Client I/O KV Consensus Consensus KV Consensus

40 3. Run the System! KV Consensus KV Consensus KV Consensus Extract to OCaml, link unverified shim Run on real networks

41 Verifying application logic

42 Simple One-node Model Key-value State: {}

43 Simple One-node Model Set k v" Key-value State: {}

44 Simple One-node Model Set k v" Key-value State: { k : v }

45 Simple One-node Model Set k v" Key-value State: { k : v } Resp k v

46 Simple One-node Model Set k v" Key-value State: { k : v } Resp k v Trace: [Set k v", Resp k v ]

47 Simple One-node Model System State: σ H inp (, i)=( 0, o) (, T) s ( 0, T ++ hi, oi) Input

48 Simple One-node Model Input:! System State: σ H inp (, i)=( 0, o) (, T) s ( 0, T ++ hi, oi) Input

49 Simple One-node Model Input:! System State: σ H inp (, i)=( 0, o) (, T) s ( 0, T ++ hi, oi) Input

50 Simple One-node Model Input:! System State: σ State: σ H inp (, i)=( 0, o) (, T) s ( 0, T ++ hi, oi) Input

51 Simple One-node Model Input:! System State: σ State: σ Output: o H inp (, i)=( 0, o) (, T) s ( 0, T ++ hi, oi) Input

52 Simple One-node Model Input:! System State: σ State: σ Output: o H inp (, i)=( 0, o) (, T) s ( 0, T ++ hi, oi) Input

53 Simple One-node Model Input:! System State: σ State: σ Output: o H inp (, i)=( 0, o) (, T) s ( 0, T ++ hi, oi) Input

54 Simple One-node Model Input:! System State: σ State: σ Output: o Trace: [!, o] H inp (, i)=( 0, o) (, T) s ( 0, T ++ hi, oi) Input

55 Simple One-node Model Input:! System State: σ State: σ Output: o Trace: [!, o] H inp (, i)=( 0, o) (, T) s ( 0, T ++ hi, oi) Input

56 Simple One-node Model Input:! System State: σ State: σ Output: o Trace: [!, o] H inp (, i)=( 0, o) (, T) s ( 0, T ++ hi, oi) Input

57 Simple One-node Model

58 Simple One-node Model Spec: operations have expected behavior (good map) Set, Get Del, Get

59 Simple One-node Model Spec: operations have expected behavior (good map) Set, Get Del, Get Verify system against semantics by induction Safety Property

60 Verifying Fault Tolerance

61 The Raft Transformer Consensus provides a replicated state machine Raft Same inputs on each node Calls into original system Raft Raft

62 The Raft Transformer Log of operations Consensus provides a replicated state machine Raft Same inputs on each node Calls into original system Raft Raft

63 The Raft Transformer Log of operations Consensus provides a replicated state Original machine system Raft Same inputs on each node Calls into original system Raft Raft

64 The Raft Transformer When input received: Add to log Send to other nodes Raft When op replicated: Apply to state machine Send output Raft Raft

65 The Raft Transformer For KV store: Ops are Get, Set, Del State is dictionary Raft Raft Raft

66 Raft Correctness V Correctly transforms systems T S Preserves traces Raft Linearizability Raft Raft

67 Fault Model Model global state Model internal communication Model failure

68 Fault Model: Global State

69 Fault Model: Global State Machines have names 1 2 3

70 Fault Model: Global State Machines have names Σ maps name to state 1 Σ[1] 2 3 Σ[2] Σ[3]

71 Fault Model: Messages 1 Σ[1] 2 3 Σ[2] Σ[3]

72 Fault Model: Messages 1 Vote? Σ[1] Vote? 2 3 Σ[2] Σ[3]

73 Fault Model: Messages 1 Network <1,2, Vote? > Vote? Σ[1] Vote? <1,3, Vote? > 2 3 Σ[2] Σ[3]

74 Fault Model: Messages 1 Network Vote? Σ[1] Vote? <1,3, Vote? > <1,2, Vote? > 2 3 Σ[2] Σ[3]

75 Fault Model: Messages 1 Network Vote? Σ[1] Vote? <1,3, Vote? > <1,2, Vote? > 2 3 Σ[2] Σ[3] H net (dst, [dst], src, m)=( 0, o, P 0 ) 0 = [dst 7! 0 ] ({(src, dst, m)} ] P,, T) r (P ] P 0, 0, T ++ hoi)

76 Fault Model: Messages 1 Network Vote? Σ[1] Vote? <1,3, Vote? > <1,2, Vote? > 2 3 Σ[2] Σ [2] = σ Σ[3] H net (dst, [dst], src, m)=( 0, o, P 0 ) 0 = [dst 7! 0 ] ({(src, dst, m)} ] P,, T) r (P ] P 0, 0, T ++ hoi)

77 Fault Model: Messages 1 Network Vote? Σ[1] Vote? <1,3, Vote? > <1,2, Vote? > 2 3 Σ[2] Σ [2] = σ Σ[3] Output: o H net (dst, [dst], src, m)=( 0, o, P 0 ) 0 = [dst 7! 0 ] ({(src, dst, m)} ] P,, T) r (P ] P 0, 0, T ++ hoi)

78 Fault Model: Messages 1 Network Vote? Σ[1] Vote? <1,3, Vote? > <1,2, Vote? > <2,1, +1 > 2 3 Σ[2] Σ [2] = σ Σ[3] Output: o H net (dst, [dst], src, m)=( 0, o, P 0 ) 0 = [dst 7! 0 ] ({(src, dst, m)} ] P,, T) r (P ] P 0, 0, T ++ hoi)

79 Fault Model: Messages 1 Network Vote? Σ[1] Vote? <1,3, Vote? > <2,1, +1 > <1,2, Vote? > 2 3 Σ[2] Σ [2] = σ Σ[3] Output: o H net (dst, [dst], src, m)=( 0, o, P 0 ) 0 = [dst 7! 0 ] ({(src, dst, m)} ] P,, T) r (P ] P 0, 0, T ++ hoi)

80 Fault Model: Messages 1 Network Vote? Σ[1] Vote? <1,3, Vote? > <2,1, +1 > <1,2, Vote? > 2 3 Σ[2] Σ [2] = σ Σ[3] Output: o H net (dst, [dst], src, m)=( 0, o, P 0 ) 0 = [dst 7! 0 ] ({(src, dst, m)} ] P,, T) r (P ] P 0, 0, T ++ hoi)

81 Fault Model: Messages 1 Network Vote? Σ[1] Vote? <1,3, Vote? > <2,1, +1 > <1,2, Vote? > 2 3 Σ[2] Σ [2] = σ Σ[3] Output: o H net (dst, [dst], src, m)=( 0, o, P 0 ) 0 = [dst 7! 0 ] ({(src, dst, m)} ] P,, T) r (P ] P 0, 0, T ++ hoi)

82 Fault Model: Messages 1 Network Vote? Σ[1] Vote? <1,3, Vote? > <2,1, +1 > <1,2, Vote? > 2 3 Σ[2] Σ [2] = σ Σ[3] Output: o H net (dst, [dst], src, m)=( 0, o, P 0 ) 0 = [dst 7! 0 ] ({(src, dst, m)} ] P,, T) r (P ] P 0, 0, T ++ hoi)

83 Fault Model: Messages 1 Network Vote? Σ[1] Vote? <1,3, Vote? > <2,1, +1 > <1,2, Vote? > 2 3 Σ[2] Σ [2] = σ Σ[3] Output: o H net (dst, [dst], src, m)=( 0, o, P 0 ) 0 = [dst 7! 0 ] ({(src, dst, m)} ] P,, T) r (P ] P 0, 0, T ++ hoi)

84 Fault Model: Failures Network <1,2, Vote? > <1,3, Vote? > 1 Σ[1] 2 3 Σ[2] Σ[3]

85 Fault Model: Failures Network <1,3, Vote? > Message drop 1 Σ[1] 2 3 Σ[2] Σ[3]

86 Fault Model: Failures Network <1,3, Vote? > Message drop <1,3, Vote? > Message duplication 1 Σ[1] 2 3 Σ[2] Σ[3]

87 Fault Model: Failures Network <1,3, Vote? > Message drop <1,3, Vote? > Message duplication 1 Σ[1] Machine crash 2 3 Σ[2] Σ[3]

88 Fault Model: Drop Network <1,2, hi > <1,3, hi > ({p} ] P,, T) drop (P,, T) Drop

89 Fault Model: Drop Network <1,2, hi > <1,3, hi > ({p} ] P,, T) drop (P,, T) Drop

90 Fault Model: Drop Network <1,2, hi > <1,3, hi > ({p} ] P,, T) drop (P,, T) Drop

91 Toward Verifying Raft General theory of linearizability 1k lines of implementation, 5k lines for linearizability State machine safety: 30k lines Most state invariants proved, some left to do

92 Verified System Transformers Functions on systems Transform systems between semantics Maintain equivalent traces Get correctness of transformed system for free

93 Verified System Transformers

94 Verified System Transformers Raft Consensus App

95 Verified System Transformers Raft Consensus Primary Backup App App

96 Verified System Transformers Raft Consensus Primary Backup Seq # and Retrans App App

97 Verified System Transformers Raft Primary Seq # and Ghost Consensus Backup Retrans Variables App App

98 Running Verdi Programs

99 Running Verdi Programs Coq extraction to Ocaml Thin, unverified shim Trusted compute base: shim, Coq, Ocaml, OS

100 Performance Evaluation Compare with etcd, a similar open-source store 10% performance overhead Mostly disk/network bound etcd has had linearizability bugs

101 Previous Approaches EventML [Schiper 2014] Verified Paxos using the NuPRL proof assistant MACE [Killian 2007] Model checking distributed systems in C++ TLA+ [Lamport 2002] Specification language and logic

102 Contributions Formalize network as operational semantics Build semantics for a variety of fault models Verify fault-tolerance as transformation between semantics

103 Contributions Formalize network as operational semantics Build semantics for a variety of fault models Verify fault-tolerance as transformation between semantics

104 Contributions Formalize network as operational semantics Build semantics for a variety of fault models Verify fault-tolerance as transformation between semantics Thanks!