Lightweight KV-based Distributed Store for Datacenters

Transcription

1 Lightweight KV-based Distributed Store for Datacenters Chanwoo Chung, Jinhyung Koo*, Arvind, and Sungjin Lee Massachusetts Institute of Technology (MIT) Daegu Gyeongbuk Institute of Science & Technology (DGIST) * Inha University 9th USENIX Workshop on Hot Topics in Storage and File Systems Wild and Crazy Ideas (WACI) July 10-11, 2017

2 Storage Computation Existing Distributed Storage Systems Datacenter Network (e.g., Ethernet, InfiniBand, ) Xeon CPUs Disk Array w/ RAID GB Storage Node 0 Storage Node 1 Storage Node N It is not a mere storage it is another high-end server!!! High-end Xeon CPUs Several GBs of An array of s Large form-factor Power Hungry (e.g., 800 W) Expensive (e.g., $42,000 w/o s) Large Volume (e.g., 2-4 U) High TCO (e.g., Cooling) 2

3 Storage Node is Designed for s is slow require large and array of disks 10 ms latency & 100~300 MB/s throughput is dumb the host system makes it smarter Xeon CPUs with advanced algorithms Aggr. Network Throughput = 4 GB/s Storage Host Prefetching Caching/Buffering Parity Mgmt Dedup/Compresion Xeon CPUs Array w/ RAID GB Local File System (e.g., EXT4, WAFL, ) 1~300 MB/s 1~300 MB/s 3

4 Storage Node is Designed for s is slow require large and array of disks 10 ms latency & 100~300 MB/s throughput is dumb the host system makes it smarter Xeon CPUs with advanced algorithms Aggr. Network Throughput = 4 GB/s Storage Host Prefetching Caching/Buffering Parity Mgmt Dedup/Compresion Xeon CPUs 3 Array w/ RAID GB Local File System (e.g., EXT4, WAFL, ) 1~300 MB/s Aggr. Throughput = 1~3 GB/s (with 10 s) 1~300 MB/s

5 Storage Node is Designed for s is slow require large and array of disks 10 ms latency & 100~300 MB/s throughput is dumb the host system makes it smarter Xeon CPUs with advanced algorithms Aggr. Network Throughput = 4 GB/s Xeon CPUs 3 Array GB w/ RAID 1~300 MB/s Aggr. Throughput = 1~3 GB/s (with 10 s) Storage Host Prefetching Caching/Buffering Parity Mgmt Dedup/Compresion Local File System (e.g., EXT4, WAFL, ) 1~300 MB/s

6 Unfortunately, Such an Architecture is Invalid with s is slow require large and array of disks 10 ms latency & 100~300 MB/s throughput s are not a bottleneck Network/CPU are new bottlenecks is dumb the host system makes it smarter Xeon CPUs with advanced algorithms Aggr. Network Throughput = 4 GB/s Bottleneck!!! Xeon CPUs 4 Array GB w/ RAID 1~6 GB/s Storage Host Prefetching Caching/Buffering Parity Mgmt Dedup/Compresion Local File System (e.g., EXT4, WAFL, ) Aggr. SDD Throughput = 10~60 GB/s (with 10 s) 1~6 GB/s

7 Unfortunately, Such an Architecture is Invalid with s is slow require large and array of disks 10 ms latency & 100~300 MB/s throughput s are not a bottleneck Network/CPU are new bottlenecks is dumb the host system makes it smarter Xeon CPUs with advanced algorithms More Network Ports? Aggr. Network Throughput = 40 GB/s Bottleneck!!! Xeon CPUs 4 Array GB w/ RAID 1~6 GB/s Storage Host Prefetching Caching/Buffering Parity Mgmt Dedup/Compresion Local File System (e.g., EXT4, WAFL, ) Aggr. SDD Throughput = 10~60 GB/s (with 10 s) 1~6 GB/s

8 Unfortunately, Such an Architecture is Invalid with s is slow require large and array of disks 10 ms latency & 100~300 MB/s throughput s are not a bottleneck Network/CPU are new bottlenecks is dumb the host system makes it smarter Xeon CPUs with advanced algorithms More Network Ports? Aggr. Network Throughput = 40 GB/s Bottleneck!!! Xeon CPUs 4 Array Bottleneck!!! GB w/ RAID 1~6 GB/s Storage Host Prefetching Caching/Buffering Parity Mgmt Dedup/Compresion Local File System (e.g., EXT4, WAFL, ) Aggr. SDD Throughput = 10~60 GB/s (with 10 s) 1~6 GB/s

9 Unfortunately, Such an Architecture is Invalid with s is slow require large and array of disks 10 ms latency & 100~300 MB/s throughput s are not a bottleneck Network/CPU are new bottlenecks is dumb the host system makes it smarter Xeon CPUs with advanced algorithms s are smarter than you think Aggr. Network Throughput = 4 GB/s Xeon CPUs 4 Array GB w/ RAID 1~6 GB/s Storage Host Prefetching Caching/Buffering Parity Mgmt Dedup/Compresion Local File System (e.g., EXT4, WAFL, ) Ctrl Ctrl Ctrl Ctrl Ctrl Ctrl Ctrl Ctrl 1~6 GB/s

10 Let s Look into s 4 embedded CPUs running at 700 MHz to 1.4 GHz and >2~4GB that a desktop system had 10 years ago Those resources are required for firmware/ftl algorithms Host-to-PCIe Controller PCIe Interface (1~6 GB/s) ARM Block CPUI/O-to-Flash I/O Interfacing ARM CPU (Max 1.4 GHz) (Max 1.4 GHz) Remapping Wear-Leveling Cleaning ARM CPU ARM CPU Deduplication (Max 1.4 GHz) Compression (Max Parity 1.4 GHz) Mgmt. (>4 GB) RAID NAND CHIP NAND CHIP NAND CHIP NAND CHIP NAND CHIP NAND CHIP NAND CHIP NAND CHIP But, many of the algorithms are redundant in storage host server 5

11 So What is the Crazy Idea? 1. Get rid of a space-consuming, expensive, power-hungry host server 2. Put and run everything in s 3. Attach s to a datacenter network 4. Let application servers directly talk to s Datacenter Network Prefetching Caching/Buffering Parity Mgmt Local File System (e.g., EXT4, WAFL, ) Ctrl Ctrl Ctrl Ctrl Ctrl Ctrl Ctrl 6

12 So What is the Crazy Idea? 1. Get rid of a space-consuming, expensive, power-hungry host server 2. Put and run everything in s 3. Attach s to a datacenter network 4. Let application servers directly talk to s Datacenter Network Prefetching Caching/Buffering Parity Mgmt Local File System (e.g., EXT4, WAFL, ) Ctrl Ctrl Ctrl Ctrl Ctrl Ctrl Ctrl 6

13 So What is the Crazy Idea? 1. Get rid of a space-consuming, expensive, power-hungry host server 2. Put and run everything in s 3. Attach s to a datacenter network 4. Let application servers directly talk to s Datacenter Network Host-to-PCIe Controller High-level Flash Management Low-level Flash Management (2~4 GB) Ctrl Ctrl Ctrl Ctrl Ctrl Ctrl Ctrl RAID NAND NAND NAND NAND NAND NAND NAND NAND 6

14 So What is the Crazy Idea? 1. Get rid of a space-consuming, expensive, power-hungry host server 2. Put and run everything in s 3. Attach s to a datacenter network 4. Let application servers directly talk to s Datacenter Network Host-to-PCIe Ethernet Controller High-level Flash Management Low-level Flash Management (2~4 GB) Ctrl Ctrl Ctrl Ctrl Ctrl Ctrl Ctrl RAID NAND NAND NAND NAND NAND NAND NAND NAND 6

15 So What is the Crazy Idea? 1. Get rid of a space-consuming, expensive, power-hungry host server 2. Put and run everything in s 3. Attach s to a datacenter network 4. Let application servers directly talk to s Datacenter Network Host-to-PCIe Ethernet Controller High-level Flash Management Low-level Flash Management (2~4 GB) Ctrl Ctrl Ctrl Ctrl Ctrl Ctrl Ctrl RAID NAND NAND NAND NAND NAND NAND NAND NAND 6

16 So What is the Crazy Idea? 1. Get rid of a space-consuming, expensive, power-hungry host server 2. Put and run everything in s 3. Attach s to a datacenter network 4. Let application servers directly talk to s Datacenter Network An x86 storage server with N s is replaced with N s Ctrl Ctrl Ctrl Ctrl Ctrl Ctrl Ctrl Low Power (e.g., 100 W) Cheap (e.g., Zero server cost) Small Volume (e.g., Less than 1U) Low TCO (e.g., Less Cooling) Scalability 6

17 Technical Challenges? Which protocol is suitable for host interfacing? Redis KV Protocol Simple, Lightweight, Easy-to-implement, widely used in many applications, Datacenter Redis Protocol Network Redis 7 Nodes

18 Technical Challenges? How to support various applications (e.g., KVS, NFS, SQL)? Protocol Adaptors on the application server side For file systems: NFS Redis For DB servers: DB Redis NFS Adapter CIFS Adapter DB Adapter KVS Adapter Redis Protocol Redis 7 Nodes

19 Technical Challenges? How to manage flash media LSM-tree algorithm: Suitable for Redis-based KV store; Removes almost all of FTL functions Hardware FTL: and CPU-less flash management NFS Adapter CIFS Adapter DB Adapter KVS Adapter Redis Protocol Redis LSM-Tree HW FTL 7 Nodes

20 Technical Challenges? How to support CPU-intensive operations (e.g., dedup and compression) Use FPGA accelerator Low power & High performance sits between cores and flash chip data preprocessed in-path NFS Adapter CIFS Adapter DB Adapter KVS Adapter Redis Protocol Redis LSM-Tree HW FTL FPGA 7 Nodes

21 Technical Challenges? How about data persistency/data availability? Use a data replication feature of Redis, instead of RAID inside nodes NFS Adapter CIFS Adapter DB Adapter KVS Adapter Redis Protocol Redis LSM-Tree HW FTL FPGA Original Replica Replica 7 Nodes

22 Conclusion A storage server with N s is replaced with N new -sized nodes Low Power (e.g., 100 W) Cheap (e.g., Zero server cost) Small Volume (e.g., Less than 1U) Low TCO (e.g., Less Cooling) NFS Adapter CIFS Adapter DB Adapter KVS Adapter Datacenter Network Redis LSM-Tree HW FTL FPGA Original Replica Replica 8 Nodes

23 9 Thank You!