Manylogs Improving CMR/SMR Disk Bandwidth & Latency. Tiratat Patana-anake, Vincentius Martin, Nora Sandler, Cheng Wu, and Haryadi S.

Transcription

1 Manylogs Improving CMR/SMR Disk Bandwidth & Latency Tiratat Patana-anake, Vincentius Martin, Nora Sandler, Cheng Wu, and Haryadi S. Gunawi

2 M SST Got 100% of the read bandwidth User 1 Big Read

3 Many 3 Got 50% of the read bandwidth User 1 Fair Share Big Read User 2 Big Read

4 N M SST Still Fair! 2 3 1/N bandwidth

5 M SST Our Oh no! 5% bandwidth?! User 1 Big Read User 2 Small Durable Writes

6 M SST Our More Bandwidth Please! m Faster Latency Please! User 1 Big Read User 2 Small Durable Writes

7 M SST Ordered ournaling Data ournaling ournal Big Read Small Writes Seek

8 M SST Ordered ournaling Data ournaling ournal Big Read Small Writes Seek

9 M SST Problems with Current ournaling Ordered ournaling Data ournaling

10 M SST Problems with Current ournaling Ordered ournaling Data o First Write Second Write

11 M SST Introducing Manylogs Single Log Manylogs 10 MB 100 MB

12 M SST Manylogs Small writes made durable to the nearest log without seeking ournal Big Read Small Writes Seek

13 M SST Manylogs q Reserved log spaces uniformly across the disk 10 MB every 100 MB q Follow the disk head (last big I/O) q Redirect Small Writes (e.g. 256 KB) Nearest log: log closest to last big I/O q Sequential Writes are left untouched 10 MB 100 MB

14 M SST Increased Read Throughput Manylogs Reduced Write Latency

15 M SST Where are logs on the disk? 10 MB 100 MB The log space = whole platter

16 M SST Where are logs on the disk? 10 MB 100 MB The log space = whole platter

17 M SST Same cylinder = No seek! Log for others in the same cylinder

18 M SST Ratio of Max Read Bandwidth Latency (ms) User 1 128MB Sequential Reads User 2 4KB Random Writes Ordered vs. Data vs. Adaptive vs. Manylogs At different intensities writes/s writes/s writes/s 320 writes/s Disk

19 M SST Adaptive ournaling q Middle ground between ordered journaling and data journaling q Single-log design q Prabhakaran et al., ATC 05

20 M SST Results Ordered Adaptive Data Manylogs 100% Ratio of Max Bandwidth 80% 60% 40% 20% 0% Random Writes per Second (IOPS)

21 M SST Results Ordered Adaptive Data Manylogs 100 Ratio of Max Bandwidth 80% 60% 40% 20% 0% 57% at 40 IOPS 5% at 320 IOPS Random Writes per Second (IOPS)

22 M SST Results Ordered Adaptive Data Manylogs 100% Ratio of Max Bandwidth 80% 60% 40% 20% 73% at 40 IOPS 9% at 320 IOPS 0% Random Writes per Second (IOPS)

23 M SST Results Ordered Adaptive Data Manylogs 100% Ratio of Max Bandwidth 80% 60% 40% 20% 0% Random Writes per Second (IOPS)

24 M SST of Max Bandwidth Manylogs gives the most bandwidth 100% 80% 60% Hey! What about my latency? Ordered daptive Data Manylogs 50% of max bandwidth at extreme IOPS Random Writes per Second (IOPS)

25 M SST Results Ordered Adaptive Data Manylogs Average sync Latency (ms) Random Writes per Second (IOPS)

26 M SST Results Ordered Adaptive Data Manylogs Average sync Latency (ms) Random Writes per Second (IOPS) 0

27 M SST Results Ordered Adaptive Data Manylogs Average sync Latency (ms) Random Writes per Second (IOPS) 0

28 M SST Results Ordered Adaptive Data Manylogs Average sync Latency (ms) Random Writes per Second (IOPS) 0

29 M SST Results Ordered Adaptive Data Manylogs WOW! Fast latency at extreme IOPS! Average sync Latency (ms) Random Writes per Second (IOPS) 0

30 M SST Results Ordered Adaptive Data Manylogs 100% 600 Ratio of Max Bandwidth 80% 60% 40% 20% Average sync Latency (ms) 0% Random Writes per Second (IOPS) 0

31 Results M SST 16 31

32 M SST Ratio of Max Read Bandwidth Latency (ms) User 1 128MB Sequential Reads User 2 fileserver Using Filebench Multi-threaded 2, 4, 8 instances

33 M SST Ordered Manylogs provides the best outcomes! 100% Data ogs 4000 Ratio of Max Bandwidth 80% 60% 40% 20% Operation Latency (ms) 0% fileserver Instances 0

34 M SST Checkpointing Data ournaling q Periodically Usually every 5 secs q ournal can get filled fast because all writes are in the journal! Manylogs q Lazy or Off-hours q Rarely full because just small writes are redirected q Log Swapping

35 M SST Log Swapping Hot Area! Cold Area! ournal Big Read Small Writes Seek

36 M SST Integrations q File System (MLFS) Durability-Only Mode (O_DUR) q SMR Disk (MLSMR) q RAID

37 M SST Cassandra Write Path Writes Memtable Commit Log SSTable Memory Disk

38 M SST Cassandra Write Path Writes Memtable Commit Log SSTable Memory Disk

39 M SST Cassandra Write Path Writes Memtable Requires Fast Durability Commit Log Ideally SYNC write SSTable Random writes are the problem Flush Triggered But in practice, background write e.g. every 10 seconds Memtable Commit Log SSTable Temp File No location needed Memory Disk

40 M SST open(file, O_DUR); q Need fast durability but not location constraints q Content of files will be put in Manylogs regardless of the write size q Never checkpoint their content q Random writes are not a problem anymore!

41 M SST Ratio of Max Read Bandwidth Latency (ms) User 1 HDFS User 2 MongoDB 1 instance 2 instances 4 instances

42 M SST Most Bandwidth & Lowest Latency with Manylogs Data Manylogs 250 Ratio of Max Bandwidth 80% 60% 40% 20% MongoDB Write Latency (ms) 0% 1DB 2DB 4DB MongoDB Instances (w/default flush period) 0

43 M SST Manylogs & SMR One non-shingled surface = log space

44 M SST Manylogs & SMR Non-Shingled Tracks Shingled Band

45 M SST Latency (ms) Latency (ms) User 1 Build Server (WBS) Trace User 2 Back-end Server (LM-TBE) Trace

46 M SST SMR Manylogs SMR (MLSMR) Single-log SMR (SLSMR) Percentile Latency (ms) ML-SMR SL-SMR

47 M SST Manylogs & RAID User 1 Big Read User 2 Small Durable Writes Disk #1 Disk #2 Disk #3 Disk #4 Max Bandwidth! Bandwidth drops up to 50% Mingzhe Hao, Gokul Soundararajan, Deepak Kenchammana-Hosekote, Andrew A. Chien, and Haryadi S. Gunawi. "The Tail at Store: A Revelation from Millions of Hours of Disk and SSD Deployments." FAST 16.

48 M SST Ratio of Max Read Bandwidth Latency (ms) User 1 128MB Sequential Reads User 2 4KB Random Writes At different intensities 40 writes/s 80 writes/s 160 writes/s 320 writes/s

49 M SST Ordered 10x Bandwidth Speed-up 14x Latency Speed-up at 320 IOPS! Data Manylogs 600 Ratio of Max Bandwidth 80% 60% 40% 20% Average sync Latency (ms) 0% Random Writes per Second (IOPS) per Disk 0

50 M SST More in the paper q Block-Level Manylogs q Other workloads Sequential Writes varmail More Traces q Log Size q Logged Write Size q Mapping Table

51 M SST Manylogs q Reserved log spaces uniformly across the disk q Redirect small writes to the nearest log q Can help with NoSQL, SMR, RAID, and more! q Provide up to 5x speed-up on average

52 M SST Manylogs Bandwidth Speed-up vs. Ordered 3.7x 5.7x vs. Adaptive 2.7x 2.0x vs. Single-log SMR 1.3x Latency Speed-up

53 M SST Thank you! Questions?