Shiqin Yan, Huaicheng Li, Mingzhe Hao, Michael Hao Tong, Swaminathan Sundararaman *, Andrew Chien, and Haryadi S. Gunawi

Transcription

1 Shiqin Yan, Huaicheng Li, Mingzhe Hao, Michael Hao Tong, Swaminathan Sundararaman *, Andrew Chien, and Haryadi S. Gunawi *

2 2 if your read is stuck behind an erase you may have wait 10s of milliseconds. That s a 100x increase in latency variance The Tail at Scale [CACM 13]

3 3 Convert to CDF NoGC 100% Reads + Writes Read Latency Percentile Clean/Empty SSD 0.3ms Time 80% 0.3ms Read Latency

4 4 Reads + Writes Read Latency 80 ms! NoGC Objective: cut tail 100% 3% 5 ms with GC Percentile Long tail! Aged/Full SSD 0.3ms Time 80% 0.3ms Read Latency 80ms

5 5 A fast Read A B delayed! B A GC moves tens of valid pages! which makes channel/chips busy for tens of ms! Channel Chip

6 6 Tail-tolerant techniques in distributed/storage systems: Leverage redundancy to cut tail! Full Stripe Read A B C C = XOR (A, B, P) fast tail! RAID: A B C P Slow / busy

7 7 Error rate increases à RAIN (Redundant Array of Independent NAND) Similarly, we leverage RAIN to cut tails! Full Stripe Read A B C fast slow! C = XOR (B, C, P) SSD: A B C P GC

8 8 New techniques: I. Plane-Blocking GC II. III. GC-Tolerant Read Rotating GC IV. GC-Tolerant Flush Current SSD technology: RAIN (Parity-based Redundancy)

9 9 NoGC 100% CDF (Percentile) +Rotating GC +GC-Tolerant Read +Plane-Blocking 95% 0.3ms Latency 80ms

10 10 100% Tiny tail! CDF (Percentile) 95% 0.3ms Overall results achieved: Between th percentiles: ttflash 1-3x slower than NoGC Base Latency5-138x slower 80ms than NoGC

11 11 q Introduction q Background q Tiny-Tail Flash Design q Evaluation, limitations, conclusion

12 12 C 0 C 1 C N Chip Die [0] Die [1] Plane[0] Plane[N] Chip

13 13 C 0 C 1 C N Block[0] Plane Block[N] Valid Page Chip Plane

14 14 for (1 # of valid pages): 1. read to controller (check with ECC) 2. write to another block SSD Controller 1 2 Old block blocked! Empty block GCed pages block the channel

15 15 SSD Controller 3. Erase the old block Old block Empty block Erase operation block the plane Erase!

16 16 blocked! A B C Channel blocking GC Base approach GCing plane

17 17 NoGC 100% CDF (Percentile) 95% 0.3ms Latency 80ms

18 18 q Introduction q Background q Tiny-Tail Flash Design Plane-Blocking GC GC-Tolerant Read Rotating GC GC-Tolerant Flush q Evaluation, limitations, conclusion

19 blocked! FAST 17 Base: Channel Blocking Plane Blocking 19 A B C Unblock the channel A B C Leverage intra-plane copyback support GCing plane GCing plane

20 Base GC Logic: FAST 17 for (every valid page) 1. flash read+write (over channel) Plane Blocking 2. wait GC Logic: for (every valid page) 1 flash read+write (inside plane) 2 serve other user I/Os Plane Blocking SSD Controller Read Page 2 A B C Old block 1 Empty block 20 Intra-plane copyback Overlap 1 intra-plane copyback with 2 channel usage for other non-gcing planes

21 21 3% of I/Os are blocked by GC NoGC 100% Only 1.5% +Plane-Blocking 95% 0.3ms Latency 80ms

22 22 q Issue 1: No ECC check for garbage-collected pages (will discuss later) q Issue 2: Read X X Read Y Y Read Z delayed! Z GC-ing plane still blocks

23 23 q Introduction q Background q Tiny-Tail Flash Design Plane-Blocking GC RAIN + GC-Tolerant Read Rotating GC GC-Tolerant Flush q Evaluation, limitations, conclusion

24 24 PG 0 PG 1 C 0 C 1 C P 0,1,2 3 4 P 3,4,5 5 C 3 LPN (Logical Page #) Static mapping: LPN0 à C[0]PG[0] LPN1 à C[1]PG[0] Add parity: PG 2 6 P 6,7,8 7 8 LPN 0, 1, 2 à P0,1,2 Rotating parity as RAID 5

25 25 RAIN enables GC-Tolerant Read Full Stripe Read fast tail 2 = XOR (0, 1, P 0,1,2 ) 1 Read in parallel + XOR cost ~0.01 ms P 0,1,2 GC vs. Wait for GC 2 to 10s of ms

26 26 GC-Tolerant Read Issue: partial stripe read Partial stripe read: slow! 2 2 = XOR (0, 1, P 0,1,2 ) Must generate extra N-1 reads! P 0,1,2 Add contention to other N -1 channels and planes Convert to full stripe if: T extra-reads < T GC

27 27 NoGC 100% CDF (Percentile) 0.5% +GC-Tolerant Read +Plane-Blocking 95% 0.3ms Latency 80ms

28 28 Issue: more than 1 GCs in a plane group? Full-stripe read tails! One parity à cut one tail Can t cut two tails! DOES NOT HELP! PG P 0,1,2 GC GC

29 29 q Introduction q Background q Tiny-Tail Flash Design Plane-Blocking GC GC-Tolerant Read Rotating GC GC-Tolerant Flush q Evaluation, limitations, conclusion

30 30 PG 0 Postpone! P 0,1,2 Rotating GC: Anytime, at most 1 plane per plane group can perform GC

31 31 Rotating! PG P 0,1,2 Rotating GC: Anytime, at most 1 plane per plane group can perform GC

32 32 PG 0 PG P 0,1,2 Rotating GC: Anytime, at most 1 plane per plane group can perform GC PG 2 Concurrent GCs in different PGs are permitted.

33 33 100% 0.5% +Rotating GC Tiny tail! Why still tiny tails? Small/partial-stripe read à Sometimes may be better to wait for GC than adding extra reads/ contentions! CDF (Percentile) 95% 0.3ms Latency 80ms

34 34 q Tiny-Tail Flash Design Plane-Blocking GC GC-Tolerant Read Rotating GC GC-Tolerant Flush (in paper) q Evaluation q Limitations q conclusion

35 35 q SSDsim (~2500 LOC) Device simulator q VSSIM (~900 LOC) QEMU/KVM-based Run Linux and applications q OpenSSD Many limitations of the simple programming model q Future: ttflash on OpenChannel SSD

36 36 q Simulator: SSDsim (verified against hardware) q Workload: 6 real-world traces from Microsoft Windows q Settings and SSD parameters: SSD size: 256GB, plane group width = 8 planes (1 parity, 7 data)

37 37 Developer Tools Release Server Trace NoGC 100% CDF (Percentile) +Rotating GC +GC-Tolerant Read +Plane-Blocking ttflash 99.99th Result: th percentile: ttflash 3x slower than NoGC Base 138x slower than NoGC 95% 0.3ms Latency 80ms

38 38 Evaluated on 6 windows workload traces with various characteristics Display Ads Server (DAPPS) Live Maps Server (LMBE) Dev. Tools Release (DTRS) MSN File Server (MSNFS) Exchange Server (Exch) TPC-C Reduced blocked I/Os (total) from 2 7% to % %: x slower for ttflash and x for Base

39 39 q Filebench on VSSIM+ttFlash ttflash achieves better average latency than base case q Vs. Preemptive GC ttflash is more stable than semi-preemptive GC - (If no idle time, preemptive GC will create GC backlogs, creating latency spikes) Avg Latency (ms) Latency (s) File Server Base +PB +GTR +RGC ttflash Preempt Network FS OLTP Varmail Video Server Elapsed time (s) 4522 Web Proxy ttflash stable

40 40 q ttflash depends on RAIN 1 parity for N parallel pages/channels We set N = 8, so we lose one channel out of 8 channels. Average latencies are x slower than NoGC, No-RAIN case q RAID à more writes (P/E cycles) ttflash increases P/E cycles by 15 18% for most of workloads Incur > 53% P/E cycles for TPCC, MSN (random write) q ECC is not checked during GC Suggest background scrubbing (read is fast & not as urgent as GC) Important note: in ttflash, foreground/user reads are still ECC checked

41 41 ttflash 55MB/s Latency CDF w/ Write Bursts 90% CDF (Percentile) 20% ttflash 64MB/s 0 Latency (ms) 80 ms Under write burst and at high watermark, ttflash must dynamitcally disable Rotating GC to ensure there is always enough number of free pages.

42 42 New techniques: I. Plane-Blocking GC II. III. IV. technology: GC-Tolerant Read Rotating GC GC-Tolerant Flush CDF (Percentile) Powerful Controller RAIN (parity-based redundancy) Capacitor-backed RAM ttflash Overall results achieved: GC-induced long tail Between th percentiles: ttflash 1-3x Latencyslower than NoGC Base 5-138x slower than NoGC

43 43