A Design for Networked Flash

Transcription

1 A Design for Networked Flash (Clusters Of Raw Flash Units) Mahesh Balakrishnan, John Davis, Dahlia Malkhi, Vijayan Prabhakaran, Michael Wei*, Ted Wobber Microso- Research Silicon Valley * Graduate student at UCSD

2 The I/O Story? Processors Main Memory Storage 2

3 The I/O Story Disk Capacity 1980s 30 MB Transfer Rate 1980s 2 MB/s Latency 1980s 20 ms Capacity/Bandwidth Large Transfers 1980s 15s Capacity/Bandwidth Small Transfers 1980s 600s 3

4 The I/O Story Disk Capacity s 30 MB 2 TB Transfer Rate 1980s 2 MB/s Latency 1980s 20 ms Capacity/Bandwidth Large Transfers 1980s 15s Capacity/Bandwidth Small Transfers 1980s 600s 3

5 The I/O Story Disk Capacity s 30 MB 2 TB Transfer Rate s 2 MB/s 150 MB/s Latency s 10 ms 20 ms Capacity/Bandwidth Large Transfers 1980s 15s Capacity/Bandwidth Small Transfers 1980s 600s 3

6 The I/O Story Disk Capacity s 30 MB 2 TB Transfer Rate s 2 MB/s 150 MB/s Latency s 10 ms 20 ms Capacity/Bandwidth Large Transfers s 15s 5,000s 300x Worse Capacity/Bandwidth Small Transfers 1980s 600s 3

7 The I/O Story Disk Capacity s 30 MB 2 TB Transfer Rate s 2 MB/s 150 MB/s Latency s 10 ms 20 ms Capacity/Bandwidth Large Transfers s 15s 5,000s 300x Worse Capacity/Bandwidth Small Transfers s 600s 58 days 8,000x Worse 3

8 512 Gb ~50 MB/s 1,280 4k The I/O Story 128 Mb ~50 MB/s byte 4 Mb ~200 MB/s 1 byte NAND Flash Phase Change STT- RAM 4

9 ~320GB $7,000 $21/GB 500GB- 10TB $10,000+ $20/GB 2 TB $88,000 $44/GB 5

10 PCIe 3.0 x16: 16 GB/s iscsi: 10 Gb/s SAS : 12 Gb/s ~320GB $7,000 $21/GB 500GB- 10TB $10,000+ $20/GB 2 TB $88,000 $44/GB 5

11 Ethernet PCIe 3.0 x16: 16 GB/s iscsi: 10 Gb/s SAS : 12 Gb/s ~320GB $7,000 $21/GB 500GB- 10TB $10,000+ $20/GB 2 TB $88,000 $44/GB 5

12 Ethernet Bo^leneck Single Point of Failure Difficult to Scale Power- Inefficient How fast does it need to be? PCIe 3.0 x16: 16 GB/s iscsi: 10 Gb/s SAS : 12 Gb/s ~320GB $7,000 $21/GB 500GB- 10TB $10,000+ $20/GB 2 TB $88,000 $44/GB 5

13 Outline The I/O Story CORFU Overview Hardware Plaaorm Conclusion 6

14 flash in the data center How can we leverage flash in distributed systems? Can flash clusters eliminate the trade- off between consistency and performance? What new abstracdons are required to manage and access flash clusters? 7

15 The CORFU Architecture Cluster of raw flash units 8

16 The CORFU Architecture Cluster of raw flash units 8

17 The CORFU Architecture Cluster of raw flash units No Bo^lenecks Fault Tolerant Highly Scalable Low Power (10W /unit) Cheap Cost of Flash) 8

18 what is CORFU? flash cluster 9

19 what is CORFU? read from anywhere append to tail flash cluster 9

20 what is CORFU? Replicated, fault- tolerant append read from anywhere append to tail flash cluster 9

21 what is CORFU? CORFU Replicated, fault- tolerant append read from anywhere append to tail flash cluster 9

22 what is CORFU? infrastructure applicayons: key- value stores databases SMR filesystems virtual disks CORFU Replicated, fault- tolerant append read from anywhere append to tail flash cluster 9

23 what is CORFU? infrastructure applicayons: key- value stores databases SMR filesystems virtual disks TOFU CORFU Replicated, fault- tolerant append read from anywhere append to tail flash cluster 9

24 what is CORFU? infrastructure applicayons: key- value stores databases SMR filesystems virtual disks TOFU CORFU Replicated, fault- tolerant append read from anywhere append to tail flash cluster cluster of 32 server- a^ached SSDs, 1.5 TB 0.5M reads/s, 0.2M appends/s (4KB) 9

25 what is CORFU? infrastructure applicayons: key- value stores databases SMR filesystems virtual disks TOFU CORFU Replicated, fault- tolerant append read from anywhere append to tail network- a^ached flash flash 1 cluster Gbps, 15W, 75 µs reads, 200 µs writes (4KB) cluster of 32 server- a^ached SSDs, 1.5 TB 0.5M reads/s, 0.2M appends/s (4KB) 9

26 the case for CORFU s append/read data why a shared log interface? 1.easy to build strongly consistent (transacdonal) s 2.effecdve way to pool flash: 1. SSD uses logging to avoid write- in- place 2. random reads are fast 3. GC is feasible 10

27 CORFU is a distributed SSD with a shared log interface gelng 10 Gbps random- IO from 1TB flash farm:

28 the CORFU design CORFU API: read(o) append(v) trim(o) read from anywhere append to tail 4KB log entry 12

29 the CORFU design CORFU API: read(o) append(v) trim(o) read from anywhere append to tail 4KB log entry each logical entry is mapped to a replica set of physical flash pages 12

30 the CORFU design mapping resides at the client CORFU API: read(o) append(v) trim(o) read from anywhere append to tail 4KB log entry each logical entry is mapped to a replica set of physical flash pages 12

31 the CORFU design mapping resides at the client CORFU API: read(o) append(v) trim(o) read from anywhere append to tail sequencer 4KB log entry each logical entry is mapped to a replica set of physical flash pages 12

32 the CORFU design mapping resides at the client CORFU API: read(o) CORFU append throughput: # of append(v) 64- bit tokens issued per second trim(o) read from anywhere append to tail sequencer 4KB log entry each logical entry is mapped to a replica set of physical flash pages 12

33 Striping

34 Striping

35 Striping

36 Striping

37 Striping

38 Striping

39 Striping

40 Striping

41 Striping

42 Striping

43 Chaining (Replicadon)

44 Chaining (Replicadon)

45 Chaining (Replicadon)

46 Chaining (Replicadon)

47 Chaining (Replicadon)

48 Chaining (Replicadon)

49 Chaining (Replicadon)

50 Chaining (Replicadon)

51 Chaining (Replicadon)

52 Chaining (Replicadon)

53 Chaining (Replicadon)

54 Chaining (Replicadon)

55 Chaining (Replicadon)

56 Chaining (Replicadon)

57 Chaining (Replicadon)

58 Chaining (Replicadon)

59 Chaining (Replicadon)

60 Holes

61 Holes

62 Holes

63 Holes

64 Holes

65 Holes

66 Holes

67 Holes

68 Holes

69 Holes

70 Holes

71 Holes

72 Holes

73 Holes

74 Holes X

75 Holes X

76 Holes X

77 Holes X

78 Holes X

79 Holes X

80 CORFU throughput (server+ssd) [CORFU: A Shared Log Design for Flash Clusters, NSDI 2012] 10 Gbps router X 16 servers (1 Gbps) X 2 SSDs per server Reads bounded by network bandwidth Writes are replicated, and bounded by sequencer throughput 16

81 The CORFU Hardware Append Read 17

82 The CORFU Hardware Append Read sequencer 17

83 The CORFU Hardware Append Read sequencer 17

84 The CORFU Hardware Append Read sequencer SATA Ethernet FPGA (or ASIC) NAND 17

85 The CORFU Hardware Append Read sequencer Write- Once, Address Read Trim SATA Management Ethernet FPGA (or ASIC) NAND 17

86 The CORFU Hardware Plaaorm 2 Prototype Systems XUPv5 Virtex5 XC5VLX110T 2 GB DDR2 RAM 2x SATA 2.0 BEE3 Virtex5 XC5VLX155T x4 8GB DDR2 RAM 8x SATA /64GB Flash DIMM 18

87 The CORFU Hardware Plaaorm 2 Prototype Systems XUPv5 Virtex5 XC5VLX110T 2 GB DDR2 RAM 2x SATA 2.0 BEE3 Virtex5 XC5VLX155T x4 8GB DDR2 RAM 8x SATA /64GB Flash DIMM 18

88 Hardware Design SSD DDR2 Memory SATA Core DDR2 Controller Write Core Token Ring Read Core Metadata Core Packet Processing Core FPGA prototype: 1 Gbps Ethernet Corfu protocol (UDP) SSD, not raw flash 1 Gbps PHY Ethernet Core System Core Other Cores GPIO RS232 Status

89 Our hardware

90 Our hardware Simple, low power, cheap, naturally pipelined No interrupts; no thread synchronizadon; polling throughout No data copying Can implement dme- sensidve funcdons in logic Frequency: 100 MHz

91 Our hardware Simple, low power, cheap, naturally pipelined No interrupts; no thread synchronizadon; polling throughout No data copying Can implement dme- sensidve funcdons in logic Frequency: 100 MHz power: 15W

92 Our hardware Simple, low power, cheap, naturally pipelined No interrupts; no thread synchronizadon; polling throughout No data copying Can implement dme- sensidve funcdons in logic Frequency: 100 MHz power: 15W tput: ~27,000 reads /sec (4KB) ~ 920 Mbps

93 Our hardware Simple, low power, cheap, naturally pipelined No interrupts; no thread synchronizadon; polling throughout No data copying Can implement dme- sensidve funcdons in logic Frequency: 100 MHz power: 15W tput: ~27,000 reads /sec (4KB) ~ 920 Mbps end- to- end latency (measured at NIC): reads: 75 µsecs non- mirrored appends: 200 µsecs

94 Applicadons Key- Value Store Virtual Block Device Shared Full rollback Distributed Synchronizadon (ZooKeeper)

95 Conclusion

96 Conclusion From Paxos to CORFU.. Corfu Paxos

97 Conclusion CORFU is a distributed SSD: 1 million write IOPS, linearly scalable read IOPS Corfu From Paxos to CORFU.. CORFU is a shared log: strong consistency at wire speed Paxos CORFU uses network- a^ached flash to construct inexpensive, power- efficient clusters