Decibel: Isolation and Sharing in Disaggregated Rack-Scale Storage

Transcription

1 Decibel: Isolation and Sharing in Disaggregated Rack-Scale Storage Mihir Nanavati, Jake Wires, and Andrew Warfield (Coho Data and University of British Columbia)

2 Redundancy and high availability Transparent device aggregation Cluster-accessibility Decoupling and migration

3 Splendid cheeses they were with a two hundred horse-power scent that might have been warranted to knock a man over at two hundred yards. Extract from Three Men in a Boat, Jerome K Jerome (1889)

4 because the data is remote (and often replicated under the covers), there s typically a noticeable latency cost involved in using it rather than local storage [...] the ideal would be to use local disk for each replica for the sake of lower latency. Low throughput EBS has got a lot better... [At Netflix] we [still] Latency overheads don't quite trust it for Kafka workloads [ ] going for instance type Shared è Crosstalk unless you want to add more complexity for your operations team, choose [ ] direct-attached storage.

5

6 (Roz Chast, The New Yorker, April 21) [with Local SSDs] You cannot stop and restart an instance only suitable for temporary storage...

7

8 DECIBEL STORAGE GOALS vs. Managed Storage Redundancy and high availability Transparent device aggregation Cluster-accessibility Local Storage Throughput and latency No crosstalk Tight host coupling Decoupling and migration

9 Compute Drawer Make all local storage cluster-accessible! Storage Brick (Adapted from Intel Rack Scale Architecture)

10 Decibel is a storage service that remotely serves local storage Challenges Workload dynamism Virtualization Virtualized Storage Abstraction + Runtime Maintain device performance

11 They are neither man nor woman They are neither brute nor human They are Ghouls Extract from The Bells, Edgar Allan Poe (1849)

12 dvols Network Boundary Virtual Machines Preserve device performance Virtualizing onlyof the capacity Vertical slice the entire of Full-system resources devicessystem is insufficient! Performance isolation Hardware-like interfaces Isolation and sharing Access control Multiplex on fast hardware! From the application Client Library perspective, storage resemblestcp-based a network-attached RPC disk Network Storage Service D e c i b e l Runs locally on R u n t i m e dvols

13 N:1 multiplexing User space Core Core Core Decibel Logic Volume Volume dvol dvol Volume dvol TCP Block I/O DPDK SPDK Virtualization and Scheduling Kernel Hardware Queues OS Thread Flow Director

14 Decouple placement from scheduling (Mirador, FAST 217) (PD Lankovsky) dvols are bound to a single core and a single device Capacity Migrations Performance Migrations Limits mimic local SSDs

15 Device Partitioning Contention and fragmentation Per-core caches (Hoard/tcmalloc) Scheduling (Guido Mocafico, Movement) Device throughput and latency dvol isolation

16 Throughput (K IOPS) Single Device, 4K Mixed (7/3) RW Time dvols provide No Isolation hooks (FCFS) for conveniently implementing policies Regular Client (A) Fair Share (3%) Regular Client (B) Bursty Client Decibel (DRR) Latency (μs) Time

17 THE DECIBEL RUNTIME Resource Management Scheduling Address Virtualization Access Control Atomicity and Data Integrity

18 Decibel vs. Local Devices Performance Single-core Performance

19 THE SETUP 1U Client 1U Server 2 x Fortville 4GbE 4 x P37 Clients/Core : 2 2 x Fortville 4GbE QD/client : 16 4K 1.8 random M IOPS requests (6 Gbps) Independent remote dvol

20 Performance (7/3 Mixed Workload) 9 Throughput (K IOPS) 7 Latency (μs) vs 45 vs 49μs Local Decibel (DPDK) Decibel (Legacy) Local Decibel (DPDK) Decibel (Legacy) Number of Cores Number of Cores

21 Performance (All Reads Workload) 2 Throughput (K IOPS) 35 Latency (μs) vs 221 vs 29μs Local Decibel (DPDK) Decibel (Legacy) Local Decibel (DPDK) Decibel (Legacy) Number of Cores Number of Cores

22 SINGLE CORE PERFORMANCE Latency (μs) Read (Idle) Read (Saturation) Mixed 7/3 (Idle) Mixed 7/3 (Saturation) Local Decibel

23 Local storage as a service Encapsulate full-system resources 2-3μs overhead to local devices Isolation on shared storage

24 Backup Slides

25 Performance (DRAM Backed Storage) 25 Throughput (K IOPS) 8 Latency (μs) Saturation: 23 vs 68μs Decibel (DPDK) Decibel (Legacy) Decibel (DPDK) Decibel (Legacy) Number of Cores Number of Cores

26 LATENCY CONSISTENCY (READS) Latency (μs) Throughput (K IOPS) Throughput Avg. Latency 95th Percentile Queue Depth

27 LATENCY CONSISTENCY (WRITES) Latency (μs) Throughput (K IOPS) Throughput Avg. Latency 95th Percentile Queue Depth