ReFlex: Remote Flash Local Flash

Transcription

1 ReFlex: Remote Flash Local Flash Ana Klimovic Heiner Litz Christos Kozyrakis NVMW 18 Memorable Paper Award Finalist 1

2 Flash in Datacenters Flash provides 1000 higher throughput and 100 lower latency than disk PCIe Flash: 1,000,000 IOPS 70 µs read latency Flash is often underutilized due to imbalanced resource requirements Solution: share SSD between remote tenants 2

3 Existing Approaches Remote access to disk (e.g. iscsi) Remote access to DRAM or NVMe over RDMA There are 2 main issues: 1. Performance overhead 2. Interference on shared remote flash device 3

4 Issue 1: Performance Overhead kB random read p95 read latency (us) latency increase 4x throughput drop Local Flash iscsi (1 core) libaio+libevent (1core) IOPS (Thousands) Traditional network storage protocols and Linux I/O libraries (e.g. libaio, libevent) have high overhead 4

5 Issue 2: Performance Interference p95 read latency (us) %read 99%read 95%read 90%read 75%read 50%read Total IOPS (Thousands) Latency depends on IOPS load Writes impact read tail latency To share Flash, we need to enforce performance isolation 5

6 How does ReFlex achieve high performance? Linux vs. ReFlex User Space Remote Storage Application User Space Remote Storage Application Linux Filesystem ReFlex Block I/O Device Driver Control Plane Data Plane Hardware Hardware Network Interface Flash Storage Network Interface Flash Storage 6

7 How does ReFlex achieve high performance? Linux vs. ReFlex User Space Remote Storage Application Linux Filesystem Block I/O Device Driver User Space Remote Storage Application ReFlex Control Plane Remove SW bloat by separating control & data plane Data Plane Hardware Hardware Network Interface Flash Storage Network Interface Flash Storage 7

8 How does ReFlex achieve high performance? Linux vs. ReFlex User Space Remote Storage Application Linux Hardware Filesystem Block I/O Device Driver Network Interface Flash Storage User Space Remote Storage Application ReFlex Control Plane Hardware Data Plane DPDK SPDK Network Interface Direct access to hardware Flash Storage 1 data plane per CPU core 8

9 How does ReFlex achieve high performance? Linux vs. ReFlex User Space Remote Storage Application User Space Remote Storage Application Linux Filesystem Block I/O Device Driver Polling vs. interrupts ReFlex Control Plane Data Plane Hardware IRQ Network Interface Flash Storage Hardware Network Interface Flash Storage 9

10 How does ReFlex achieve high performance? Linux vs. ReFlex User Space Remote Storage Application User Space Remote Storage Application Linux Filesystem Block I/O Device Driver Polling vs. interrupts ReFlex Control Plane Data Plane Run to completion Hardware IRQ Network Interface Flash Storage Hardware Network Interface Flash Storage 10

11 How does ReFlex achieve high performance? Linux vs. ReFlex User Space Remote Storage Application User Space Remote Storage Application Linux Filesystem Block I/O Device Driver Polling vs. interrupts ReFlex Control Plane Data Plane Adaptive batching Hardware IRQ Network Interface Flash Storage Hardware Network Interface Flash Storage 11

12 How does ReFlex achieve high performance? Linux vs. ReFlex User Space User Space Remote Storage Application Linux Hardware Filesystem 2. Block I/O Device Driver 1. Network Interface Flash Storage Remote Storage Application Zero-copy ReFlex device-to-device Control Plane Hardware Data Plane Network Interface Flash Storage 12

13 How does ReFlex enable performance isolation? Request cost based scheduling Determine the impact of tenant A on the tail latency and IOPS of tenant B Control plane assigns tenants with a quota Data plane enforces quotas through throttling 13

14 Request Cost Modeling Compensate for read-write asymmetry For this device: Write == 10x Read p95 read latency (us) %read 99%read 95%read 90%read 75%read 50%read Total IOPS (Thousands) p95 Read Latency (us) %read 99%read 95%read 90%read 75%read 50%read Weighted IOPS (x 10 3 tokens/s ) 14

15 Request Cost Based Scheduling 15

16 Request Cost Based Scheduling 1ms tail latency SLO 16

17 Request Cost Based Scheduling 1ms tail latency SLO Device max IOPS: 510K 17

18 Request Cost Based Scheduling 1ms tail latency SLO 200K IOPS SLO Device max IOPS: 510K 18

19 Request Cost Based Scheduling 1ms tail latency SLO 200K IOPS SLO 310K Slack Device max IOPS: 510K 19

20 Results: Local Remote Latency p95 Read Latency (us) Linux: 75K IOPS/core ReFlex: 850K IOPS/core Local-1T ReFlex-1T Linux-1T Libaio-1T IOPS (Thousands) 20

21 Results: Local Remote Latency p95 Read Latency (us) Latency Local Flash 78 µs ReFlex 99 µs Linux 200 µs Local-1T ReFlex-1T Linux-1T Libaio-1T IOPS (Thousands) 21

22 Results: Local Remote Latency p95 Read Latency (us) ReFlex: saturates Flash Local-1T Local-2T ReFlex-1T ReFlex-2T Linux-1T Libaio-1T Libaio-2T Linux-2T IOPS (Thousands) 22

23 Results: Performance Isolation Read p95 latency (us) I/O sched disabled I/O sched enabled Latency SLO IOPS (Thousands) I/O sched disabled Tenant A IOPS SLO I/O sched enabled Tenant B IOPS SLO 0 0 Tenant A Tenant B Tenant C Tenant D Tenant A Tenant B Tenant C Tenant D 100%rd 80%rd 95%rd 25%rd 100%rd 80%rd 95%rd 25%rd Tenants A & B: latency-critical; Tenant C + D: best effort 23

24 Results: Performance Isolation Read p95 latency (us) I/O sched disabled I/O sched enabled Latency SLO IOPS (Thousands) I/O sched disabled Tenant A IOPS SLO I/O sched enabled Tenant B IOPS SLO 0 0 Tenant A Tenant B Tenant C Tenant D Tenant A Tenant B Tenant C Tenant D 100%rd 80%rd 95%rd 25%rd 100%rd 80%rd 95%rd 25%rd Tenants A & B: latency-critical; Tenant C + D: best effort Without scheduler: latency and bandwidth QoS for A/B are violated 24

25 Results: Performance Isolation Read p95 latency (us) I/O sched disabled I/O sched enabled Latency SLO IOPS (Thousands) I/O sched disabled Tenant A IOPS SLO I/O sched enabled Tenant B IOPS SLO 0 0 Tenant A Tenant B Tenant C Tenant D Tenant A Tenant B Tenant C Tenant D 100%rd 80%rd 95%rd 25%rd 100%rd 80%rd 95%rd 25%rd Tenants A & B: latency-critical; Tenant C + D: best effort Without scheduler: latency and bandwidth QoS for A/B are violated Scheduler rate limits best-effort tenants to enforce SLOs 25

26 ReFlex Summary 1. Enables Flash disaggregation à improve utilization Performance: remote local Commodity networking, low CPU overhead 2. Guarantees QoS in shared resource deployments Quality of Service aware request scheduling 26

27 Impact of ReFlex Open source: Works on AWS i3 cloud instances with NVMe Flash Integrated as a remote Flash dataplane in the Apache Crail distributed storage system (collaboration with IBM Research) Broadcom is porting ReFlex to ARM-based SoC 27

28 Thank You! Download the source code at: Original paper presented at ASPLOS