Gecko: Contention-Oblivious Disk Arrays for Cloud Storage

Transcription

1 Gecko: Contention-Oblivious Disk Arrays for Cloud Storage Ji-Yong Shin Cornell University In collaboration with Mahesh Balakrishnan (MSR SVC), Tudor Marian (Google), and Hakim Weatherspoon (Cornell) FAST 2013

2 Throughput (MB/s) Cloud and Virtualization What happens to storage? Guest VM Guest VM Guest VM Guest VM Disk RAID0 Seq - VM+EXT4 Rand - VM+EXT # of Seq Writers Shared Disk SEQUENTIAL Shared Disk VMM RANDOM Shared Disk Shared Disk I/O CONTENTION causes throughput collapse 2

3 Existing Solutions for I/O Contention? I/O scheduling: reordering I/Os Entails increased latency for certain workloads May still require seeking Workload placement: positioning workloads to minimize contention Requires prior knowledge or dynamic prediction Predictions may be inaccurate Limits freedom of placing VMs in the cloud 3

4 Log-structured File System to the Rescue? [Rosenblum et al. 91] Write Log Tail Write Write Log all writes to tail Addr N Shared Disk 4

5 Log Head Log Tail GC Read Challenges of Log-Structured File System Garbage collection is the Achilles Heel of LFS [Seltzer et al. 93, 95; Matthews et al. 97] Garbage Collection (GC) from Log Head N Shared Disk Shared Disk Shared Disk GC and read still cause disk seek 5

6 Throughput (MB/s) Challenges of Log-Structured File System Garbage collection is the Achilles Heel of LFS 2-disk RAID-0 setting of LFS GC under write-only synthetic workload Max Sequential Throughput RAID 0 + LFS Throughput falls by 10X during GC GC Aggregate App Time (s) 6

7 Problem: Increased virtualization leads to increased disk seeks and kills performance RAID and LFS do not solve the problem 7

8 Rest of the Talk Motivation Gecko: contention-oblivious disk storage Sources of I/O contention New technique: Chained logging Implementation Evaluation Conclusion 8

9 Write Write Read Read Write Read What Causes Disk Seeks? Write-write Read-read Write-read VM 1 VM2 9

10 Write GC Read What Causes Disk Seeks? Write-write Read-read Write-read VM 1 VM2 Logging Write-GC read Read-GC read 10

11 Principle: A single sequentially accessed disk is better than multiple randomly seeking disks 11

12 GC Gecko s Chained Logging Design Separating the log tail from the body Eliminates write-gc read contention Physical Addr Space Log Tail Garbage collection from Log Head to Tail Eliminates write-write and reduces write-read contention Disk 0 Disk 1 Disk 2 GC reads do not interrupt the sequential write 1 uncontended drive >>faster>> N contended drives 12

13 GC Gecko s Chained Logging Design Smarter Compact-In-Body Garbage Collection Physical Addr Space Garbage collection from Log Head to Tail Log Tail Disk 0 Disk 1 Disk 2 Garbage collection from Head to Body 13

14 Read Read Write Read Gecko Caching What happens to reads going to tail drives? Reduces read-read contention Body Cache (Flash ) Tail Cache Reduces write-read contention LRU Cache RAM Hot data Disk 0 Disk 1 Disk 2 MLC SSD Warm data 14

15 Read Read Write Gecko Properties Summary Reduced read-write and read-read contention No write-write contention, No GC-write contention, and Reduced read-write contention Body Cache (Flash ) Tail Cache Fault tolerance Disk 0 Disk 1 Disk + 2 Read performance Mirroring/ Striping Disk 0 Disk 1 Disk 2 Power saving w/o consistency concerns 15

16 head tail head tail Gecko Implementation Primary map: less than 8 GB RAM for a 8 TB storage Logical-to-physical map (in memory) 4-byte entries W W R W W 4KB pages Data (in disk) R W W Disk 0 Disk 1 Disk 2 empty filled Physical-to-logical map (in flash) Inverse map: 8 GB flash for a 8 TB storage (written every 1024 writes) 16

17 Evaluation 1. How well does Gecko handle GC? 2. Performance of Gecko under real workloads? 3. Effect of varying Gecko chain length? 4. Effectiveness of the tail cache? 5. Durability of the flash based tail cache? 17

18 Evaluation Setup In-kernel version Implemented as block device for portability Similar to software RAID User-level emulator For fast prototyping Runs block traces Tail cache support Hardware WD 600GB HDD Used 512GB of 600GB K RPM SATA-600 Intel MLC (multi level cell) SSD 240GB SATA

19 Throughput (MB/s) Throughput (MB/s) How Well Does Gecko Handle GC? 2-disk setting; write-only synthetic workload Gecko Log + RAID GC Max Sequential Aggregate Aggregate throughput 150 Throughput of 1 disk Max Sequential = Max throughput App 150 Throughput of 2 disks Throughput collapses GC Aggregate App 0 0 Time (s) Time (s) Gecko s aggregate throughput always remains high 3X higher aggregate & 4X higher application throughput 19

20 Throughput (MB/s) Throughput (MB/s) How Well Does Gecko Handle GC? Gecko GC Aggregate App Time (s) App throughput can be preserved using smarter GC 0 No GC Moveto-Tail GC Compactin-Body GC 20

21 MS Enterprise and MSR Cambridge Traces Trace Name Estimated Addr Space Total Data Accessed (GB) Total Data Read (GB) Total Data Written (GB) TotalIOReq NumReadReq NumWriteReq prxy 136 2,076 1, ,157, ,797,984 70,359,948 src ,107 2, ,069,608 65,172,645 19,896,963 proj 4,102 2,279 1, ,841,031 55,809,869 10,031,162 Exchange 4, ,179,165 26,324,163 34,855,002 usr 2,461 2,625 2, ,091,915 50,906,183 7,185,732 LiveMapsBE 6,737 2,344 1, ,766,484 35,310,420 9,456,064 MSNFS 1, ,345,085 19,729,611 9,615,474 DevDivRelease 4, ,195,701 12,326,432 5,869,269 prn ,819,297 9,066,281 7,753,016 [Narayanan et al. 08, 09] 21

22 Throughput (MB/s) Throughput (MB/s) What is the Performance of Gecko under Real Workloads? Mix of 8 workloads: prn, MSNFS, DevDivRelease, proj, Exchange, LiveMapsBE, prxy, and src1 6 Disk configuration with 200GB of data prefilled Gecko Log + RAID Read Write 150 Read Write Gecko Time (s) Mirrored chain of length 3 Tail cache (2GB RAM + 32GB SSD) Body Cache (32GB SSD) Log + RAID10 Time (s) Mirrored, Log + 3 disk RAID-0 LRU cache (2GB RAM + 64GB SSD) Gecko showed less read-write contention and higher cache hit rate Gecko s throughput is 2X-3X higher 22

23 Throughput (MB/s) What is the Effect of Varying Gecko Chain Length? Same 8 workloads with 200GB data prefilled Read Write errorbar = stdev RAID 0 Single uncontended disk Separating reads and writes better performance 23

24 Hit Rate (%) Hit Rate (%) How Effective Is the Tail Cache? Read hit rate of tail cache (2GB RAM+32GB SSD) on 512GB disk 21 combinations of 4 to 8 MSR Cambridge and MS Enterprise traces SSD RAM Application Mix Amount of Data in Disk (GB) At least 86% of read hit rate RAM handles most of hot data Amount of data changes hit rate Still average 80+ % hit rate Tail cache can effectively resolve read-write contention 24

25 SSD Lifetime (Days) How Durable is Flash Based Tail Cache? Static analysis of lifetime based on cache hit rate Use of 2GB RAM extends SSD lifetime Application Mix 2X-8X Lifetime Extension Lifetime at 40MB/s 25

26 Conclusion Gecko enables fast storage in the cloud Scales with increasing virtualization and number of cores Oblivious to I/O contention Gecko s technical contribution Separates log tail from its body Separates reads and writes Tail cache absorbs reads going to tail A single sequentially accessed disk is better than multiple randomly seeking disks 26

27 Question? 27