Transparent TCP Recovery

Transcription

1 Transparent Recovery with Chain Replication Robert Burgess Ken Birman Robert Broberg Rick Payne Robbert van Renesse October 26, 2009

2 Motivation Us:

3 Motivation Them: Client

4 Motivation There is a connection... Client

5 Motivation Client We keep some application state...

6 Motivation Client... and so do they

7 Motivation The server fails! Client

8 Motivation... and is revived Client But two things are still missing!

9 Motivation Look! A persistent store! Client

10 Motivation checkpoint! Client

11 Motivation Client recover!

12 Motivation Client recovers application state

13 Motivation Client What recovers connection state?

14 Why bother? As soon as the client tries to send... Client

15 Why bother? RST! Client

16 Why bother? Client I ll just try again!

17 Why bother? Client

18 Why bother? Client

19 The client Client

20 The client Them: Client

21 The client Humans see Connection Reset Them: Client

22 The client Humans see Connection Reset When should the client retry? Them: Client

23 The client Humans see Connection Reset When should the client retry? Them: When should the client give up? Client

24 The client Humans see Connection Reset When should the client retry? Them: When should the client give up? What session maps to a new connection? Client

25 The client Humans see Connection Reset When should the client retry? Them: When should the client give up? What session maps to a new connection? Client Some protocols would need re-authentication

26 The client Humans see Connection Reset When should the client retry? Them: When should the client give up? Client What session maps to a new connection? Some protocols would need re-authentication Some protocols already respond actively... BGP assumes link is lost! Resync is slow

27 What are the possibilities?

28 What are the possibilities?

29 What are the possibilities?

30 The network stack

31 The network stack Has the state and the logic

32 The network stack Has the state and the logic No copies or context switches

33 The network stack Has the state and the logic No copies or context switches Fork network stack

34 The network stack Has the state and the logic No copies or context switches Fork network stack Redundant logging

35 The network stack Has the state and the logic No copies or context switches Fork network stack Redundant logging Synchronous replication

36 The network stack Has the state and the logic No copies or context switches Fork network stack Redundant logging Synchronous replication End-to-end, only use kernel for efficiency

37 Network stack wrappers [FT-]

38 Network stack wrappers [FT-] Still some kernel advantages

39 Network stack wrappers [FT-] Still some kernel advantages No change to server or

40 Network stack wrappers [FT-] Still some kernel advantages No change to server or Two kernel modules

41 Network stack wrappers [FT-] Still some kernel advantages No change to server or Two kernel modules Must interpose on socket calls

42 Network stack wrappers [FT-] Still some kernel advantages No change to server or Two kernel modules Must interpose on socket calls Synchronous replication

43 The server

44 The server User-level networking

45 The server User-level networking state includes

46 The server User-level networking state includes Can t leverage OS

47 The server User-level networking state includes Can t leverage OS Significant server changes

48 A proxy [CRAFT, I-]

49 A proxy [CRAFT, I-] Splice (spoof) separate connections

50 A proxy [CRAFT, I-] Splice (spoof) separate connections Replicate for fault-tolerance

51 A proxy [CRAFT, I-] Splice (spoof) separate connections Replicate for fault-tolerance Recover by reconnecting

52 A proxy [CRAFT, I-] Splice (spoof) separate connections Replicate for fault-tolerance Recover by reconnecting state machine replication

53 A proxy [CRAFT, I-] Splice (spoof) separate connections Replicate for fault-tolerance Recover by reconnecting state machine replication Connections aren t really connected

54 A man in the middle [Morris, ST-]

55 A man in the middle [Morris, ST-] Little or no overhead

56 A man in the middle [Morris, ST-] Little or no overhead Guesswork

57 A man in the middle [Morris, ST-] Little or no overhead Guesswork No control over server

58 A man in the middle [Morris, ST-] Little or no overhead Guesswork No control over server What about recovery?

59 R

60 R Little or no overhead

61 R Little or no overhead No guesswork

62 R Little or no overhead No guesswork Can control server

63 R Little or no overhead No guesswork Can control server Chain replication

64 R Little or no overhead No guesswork Can control server Chain replication Machine-independent

65 R Little or no overhead No guesswork Can control server Chain replication Machine-independent Replicas don t grok

66 R Little or no overhead No guesswork Can control server Chain replication Machine-independent Replicas don t grok But what about recovery?

67 Recovery How we recover determines what we must replicate

68 Recovery If we replicate state machines

69 Recovery If we replicate state machines and we control the stack

70 Recovery If we replicate state machines and we control the stack Can failover TCBs

71 Recovery We must restart the connection

72 Recovery We must restart the connection with an unchanged stack

73 Recovery Recovery We must restart the connection with an unchanged stack Recovery process notifies chain

74 Recovery Recovery We must restart the connection with an unchanged stack Recovery process notifies chain and makes a new connection

75 Recovery Recovery We must restart the connection with an unchanged stack Recovery process notifies chain and makes a new connection Packets modified to spoof real client

76 Recovery Recovery We must restart the connection with an unchanged stack Recovery process notifies chain and makes a new connection Packets modified to spoof real client handshakes normally (chain maintains spoofing)

77 Recovery Recovery Connection replayed from checkpoint

78 Recovery Recovery Connection replayed from checkpoint Client traffic can continue ignored until it catches up

79 Recovery Recovery Connection replayed from checkpoint Client traffic can continue ignored until it catches up Connection recovered!

80 Replaying client data

81 Replaying client data Replay from beginning No need for server to tell recovered from new must be deterministic Memory-intensive in common case Slow recovery

82 Replaying client data Replay from beginning No need for server to tell recovered from new must be deterministic Memory-intensive in common case Slow recovery Replay from explicit checkpoint Can be requested from server or administrator may need to distinguish recovered connections (getpeername) Bounded data to store and replay

83 Replaying client data Replay from beginning No need for server to tell recovered from new must be deterministic Memory-intensive in common case Slow recovery Replay from explicit checkpoint Can be requested from server or administrator may need to distinguish recovered connections (getpeername) Bounded data to store and replay One step further:

84 Replaying client data Replay from beginning No need for server to tell recovered from new must be deterministic Memory-intensive in common case Slow recovery Replay from explicit checkpoint Can be requested from server or administrator may need to distinguish recovered connections (getpeername) Bounded data to store and replay One step further: Hold ACKs until checkpointed

85 Replaying client data Replay from beginning No need for server to tell recovered from new must be deterministic Memory-intensive in common case Slow recovery Replay from explicit checkpoint Can be requested from server or administrator may need to distinguish recovered connections (getpeername) Bounded data to store and replay One step further: Hold ACKs until checkpointed No need to store or replay packets at all!

86 It can t be that simple...

87 It can t be that simple... Initial Sequence Numbers We can pick the correct client sequence number The server picks its own We ll have to patch up all future packets Change sequence, recompute checksum

88 It can t be that simple... Initial Sequence Numbers We can pick the correct client sequence number The server picks its own We ll have to patch up all future packets Change sequence, recompute checksum Fragmentation Assume that endpoints use a reasonable MSS Or that driver program handles reassembly

89 It can t be that simple... Initial Sequence Numbers We can pick the correct client sequence number The server picks its own We ll have to patch up all future packets Change sequence, recompute checksum Fragmentation Assume that endpoints use a reasonable MSS Or that driver program handles reassembly Selective ACKnowledgements SACKs are advisory only Let SACKs flow normally Do nothing to recover those packets if lost

90 It can t be that simple... Initial Sequence Numbers We can pick the correct client sequence number The server picks its own We ll have to patch up all future packets Change sequence, recompute checksum Fragmentation Assume that endpoints use a reasonable MSS Or that driver program handles reassembly Selective ACKnowledgements SACKs are advisory only Let SACKs flow normally Do nothing to recover those packets if lost MD5 security Adds an additional checksum with symmetric key Administrator must provide key information

91 Implementation

92 Implementation librtcp or administrator functions Set up server (with key information) Send checkpoints to replicas Recover connection Replica functions Process packet (agnostic of transport)

93 Implementation librtcp or administrator functions Set up server (with key information) Send checkpoints to replicas Recover connection Replica functions Process packet (agnostic of transport) Potentially many driver programs Current rtcp program uses Linux netfilter QUEUE 1. Pull packet off queue 2. Let library process packet 3. Permit delivery

94 Implementation librtcp or administrator functions Set up server (with key information) Send checkpoints to replicas Recover connection Replica functions Process packet (agnostic of transport) Potentially many driver programs Current rtcp program uses Linux netfilter QUEUE 1. Pull packet off queue 2. Let library process packet 3. Permit delivery In the future, Feather-Weight Pipes!

95 Implementation status Replicas log and forward packets Connections are tracked and kept up-to-date No checkpointing or recovery yet No MD5 option handling yet

96 Conclusion R Chain replication enables cheap consistency Leverages existing stacks Platform-independent Can run on any machine in network Perhaps minor changes to server (or automatic administrator) to checkpoint and recover

97 Conclusion R Chain replication enables cheap consistency Leverages existing stacks Platform-independent Can run on any machine in network Perhaps minor changes to server (or automatic administrator) to checkpoint and recover Simple, simple, simple, simple. Fast?