CGAR: Strong Consistency without Synchronous Replication. Seo Jin Park Advised by: John Ousterhout

Transcription

1 CGAR: Strong Consistency without Synchronous Replication Seo Jin Park Advised by: John Ousterhout

2 Improved update performance of storage systems with master-back replication Fast: updates complete before replication to backups Safe: save RPC requests and retry if master crashes Two variants: CGAR-C: save RPC requests in client library CGAR-W: save RPC requests in a different server (Witness) Performance Result Overview RAMCloud: 0.5x latency, 4x throughput Redis: strongly consistent (cost: 12% latency é) Slide 2

3 CGAR s Role in Platform Lab Granular Computing Platform Cluster Scheduling Low-Latency RPC Scalable Notifications Thread/ App Mgmt Hardware Accelerat ors Low- Latency Storage CGAR Slide 3

4 Consistency in Master- Master-backup replication: client send updates to a master and master replicate state to backups. Consistency after crash Responses for update operations must wait for backup replications (synchronous replication) Must not reveal non-replicated value client write x = 1 X: 1 ok X: 1 ok X: 1 Master Slide 4

5 Waiting for Replication is Not Cheap Synchronous replication increases latency of updates Alternative: asynchronous replication Non-replicated data can be lost Sacrifice consistency if master crashes Enables batched replication (more efficient) Client Master Processing time for RAMCloud WRITE operation 4 µs 3 µs 8 µs Asynchronous update: 7 µs Slide 5

6 Consistency over Performance: RAMCloud RAMCloud uses synchronous replication Consistent even after crash Write: 14.3 µs vs. Read: 5 µs Focused on minimizing latency while consistent Polling wait for replication è Write throughput is only 18% of read throughput Client Write Ok Master(s) s Durable Log Write Slide 6

7 Performance over Consistency: Redis Redis uses asynchronous replication to a file in disk Default: fsync every second Lose data if a master crashes Option for strong consistency: fsync-always On SSDs, 1~2 ms delay Without fsync, SET takes 25 µs. Client SET Ok Server Memory Fsync Log File Server Disk Can we have both consistency and performance? Slide 7

8 Consistency Guaranteed Asynchronous Replication Asynchronous Replication à performance For consistency Save RPC requests in 3 rd -party server (Witness) Replay RPCs in Witness if master crashes Witness Client RPC Master recover Slide 8

9 Witness Record Operation Client multicasts RPC request to master and witness Witness vouches the RPC will be retried if master crash write x = 1 Witness (8MB) client X: 1 Master async X: 0 Slide 9

10 Recovery Steps of CGAR-W Step 1: recover from backups Step 2: retry update RPCs in witness write x = 1 Witness retry X: 01 Y: 7 New Master client recover replicate X: 1 Y: 7 Master X: 0 Y: 7 1 Slide 10

11 Challenges in Using Witness for Recovery Witness may receive RPCs in a different order than master Solution: witness saves only 1 record per key Concurrent operations on same key? Witness rejects all but first Retry may re-execute an RPC Solution: use RIFL to ignore already completed RPC. Update may depend on unreplicated value in master Master cannot assume witness saved the RPC request Solution: delay update if current value is not yet replicated Slide 11

12 Example: RPCs in a Different Order Witness write x = 2 Client Red write x = 1 Client Blue write x = 2 Master 1 Slide 12

13 Example: RPCs in a Different Order Witness write x = 2 Client Red Must wait for replication write x = 1 Client Blue ok Can complete as soon as master returns write x = 2 Master Slide 13

14 Garbage Collection Witness must drop a record before accepting new one with same key client write x = 2 accept write x = 1 client Witness Drop write x = 1 [use RPC ids assigned by RIFL] X: 1 Master async X: 1 Slide 14

15 Using Multiple Witnesses A system can use multiple witnesses per each master Higher availability (recovery can use any witnesses) To use async update, all witnesses must accept client write x = 1 write x = 1 Witnesses X: 1 Master Slide 15

16 Evaluation of CGAR Ø RAMCloud implementation Performance improvement Latency reduction Ø Redis implementation Supports wide range of operations Slide 16

17 RAMCloud s Latency after CGAR Writes are issued sequentially by a client to a master Median 14.3 μs Median 6.6 μs, 7.1 μs Slide 17

18 RAMCloud s Throughput after CGAR Batching replication improved throughput Slide 18

19 Making Redis Consistent with Small Cost SET: write to key-value store HMSET: write to a member of hashmap INCR: increment an integer counter Slide 19

20 Conclusion Fast: updates don t wait for replication Consistent: CGAR saves RPC requests in witness; If server crashes, retry the saved RPCs to recover High throughput: replication can be batched Slide 20

21 Questions Slide 21

22 Latency under Skewed Workloads YCSB-A: Zipfian dist (1M objects, p = 0.99) Slide 25

23 CGAR Decoupled Replication from Update Delay replication RPC s completion time Slide 26