Purity: building fast, highly-available enterprise flash storage from commodity components

Transcription

1 Purity: building fast, highly-available enterprise flash storage from commodity components J. Colgrove, J. Davis, J. Hayes, E. Miller, C. Sandvig, R. Sears, A. Tamches, N. Vachharajani, and F. Wang 0

2 Gala has already introduced HDD What is exactly SSD? Heroes of this talk 1

3 Outline Introduction to SSD Unique properties in SSD The pure storage system System processes Real world deployments 2

4 SSD SSD is composed of flash memory arrays Block: 2-16 MB Page: 0.5-4KB 3

5 Write/Read/Erase Operations in SSD Full bucket means 1 Empty (or almost empty) bucket means 0 We can read/write rows However must erase the entire block 1 level 0 level 4

6 Multi-level flash memories 5

7 Write/Read/Erase Operations in MLC SSD Data is represented by the amount of electrical charge Cells can be written individually However, can only be erased by erasing an entire block q-1 level 0 level 6

8 7

9 Google s Data Centers 8

10 It even determines the location 9

11 So why not always use SSD? Traditionally, data center s software and design were optimally developed for HDD 10

12 Purity All SSD enterprise storage system Claims to be cheaper Higher performance 11

13 Outline Introduction to SSD Unique properties in SSD The pure storage system System processes Real world deployments 12

14 SSD - unique properties 13

15 SSD unique properties #1 Wear For each P/E cycle the media becomes worse Y. cai, et al. Threshold Voltage Distribution in MLC NAND Flash Memory: Characterization, Analysis, and Modeling DATE13 14

16 Pages SSD unique properties #2 Garbage collection We can write a page but can only erase a block Typical numbers: pages per block ~2048 blocks per flash drive Blocks

17 Pages SSD unique properties #2 Garbage collection We can write a page but can only erase a block Typical numbers: pages per block ~2048 blocks per flash drive Blocks we want to update block First option: erase the block Second option: use overprovisioning 16

18 Pages SSD unique properties #2 Garbage collection Blocks we want to also update pages 4, 5, We are full! And now what? Tradeof:Space vs. lifetime Garbage collection 18

19 Main conclusions for SSD Random fast access (unlike disks) Performance heavily depends on workload The response of the device is not uniform We wish to write 3 : Scenario A Scenario B

20 SSD unique properties #2 Wear Leveling Let us assume we keep updating pages 0, 1, 2, What will happen? 20

21 System benefits from fast access Some processes are no longer required to run on cache HDD requires much more (expensive) DRAM for cache Can perform system processes (many IOPS) that are hard to do in HDD: Deduplication Read speedup Log structure file system etc. 21

22 Outline Introduction to SSD Unique properties in SSD The pure storage system System processes Real world deployments 22

23 Pure Storage storage system 23

24 Comparison between Purity and disk based system We will try to explain this magic 24

25 Implementation 12x(10-16) Gb/s MLC drives This actually SLC flash 25

26 Basic Architecture Each segment is striped across multiple SSDs Reed-Solomon is used in order to overcome two SSD failures The parity pages enable correcting a single corrupted page without reading the rest of the SSDs (???) 7+2 Drives 1MB write block 8MB taken from a single SSD 26

27 Outline Introduction to SSD Unique properties in SSD The pure storage system System processes Real world deployments 27

28 Processes in the storage system 28

29 Compression 101: Column-oriented database management system Insights: Reducing seek time in HDD (for example if we wish to count how many people get >48000) Many repeating patterns 29

30 Compression 101: Run-Length Encoding 30

31 Compression 101: Entropy encoding Data centers use lossless compression Let us consider a text file: aabacfffeedcbaaa The storage needed in order to store the file: ( )*1000*3 = 300,000 bits 31

32 Compression 101 cont. But what if The storage needed is given by: (45*1+13*3+12*3+16*3+9*4+5*4)*1000=224,000bits 25% reduction in needed storage! We pay in using encoder and decoder for each read/write operation This is the Huffman code 32

33 Deduplication Elimination of duplicate copies of repeating data Replacing duplicated files/data blocks by pointers Deduplication ratio of 16/11=

34 Deduplication cont. Examples from real life for cases in which a single file is saved many times on the same server: Downloading homework1 in algebra to 1000 Dropbox accounts Mail attachments Streaming music files from the cloud Backups! What is the practical storage reduction by deduplication and compression? 3 10 (can even reach 50 for backups) 34

35 Block Deduplication Deduplication can be performed by blocks of information (and not necessarily by files) For example: 35

36 Deduplication in Purity Tracks deduplication blocks at 512B granularity Keeps the hash value of every 8 th block In case of a match, the block is verified byte by byte Can detect 8*512B=4KB duplicates Purity performs: Inline deduplication: detecting duplicates before writing it on SSD Looks only in recently written data, and in frequently duplicated data In charge on most of the deduplication ratio Deduplication during garbage collection 36

37 Log Structure File System In a nut shell: in order to overcome the high seek times of disks, data is buffered in a log structure to create long sequential writes: File#1 File#2 Update File#1 File#2 Update File#2 File#3 This structure supports the way we write to SSD! No need to buffer the data in expensive RAM Garbage collection is inherent process in SSD Wear leveling algorithms in flash memories avoids fragmentation 37

38 Data Read speedup Vast majority of slow SSD reads happen while the SSD is in the middle of a write process Purtiy avoids writing to more than two SSDs per ECC group in the same time Time Threshold Reconstructing using the erasure codes Request Latency 38

39 Real world deployments 39

40 Reliability The company collects telemetry data form its customers Telemetry data includes: I/O request rates Request sizes Deduplication ratio etc. The company analyzed the data and can foresee failures and replace components before they fail They reach % availability (5min/year) 40

41 Reliability cont. SSDs are very reliable in all of Purity s data centers (the number is not given) only two SSDs have failed Most customers never approach the P/E ratings of consumer MLC drives The company offers free SSD replacement due to wear In order to fix errors, Purity uses: ECC Rewriting un-accessed data 41

42 Database deployments Customers usually deploy dozens/hundreds of database instances on top of a single Purity array The 5min rule: Data accessed more often than once every 5min belonged to RAM Colder data belonged on disk Several assumptions for analysis: The deduplication ratio is usually 3-8 for documents database, it is ~10 I/Os are 55KB on average 42

43 Relative Cost based on data form customers Disks are no good for high performance needs Without reduction, store everything that you can afford to lose in RAM With data reduction, never cache cold data (accessed less frequently than 30m) 43

44 Summary SSDs are fast random storage devices SSDs have unique properties like: Wear Garbage collection etc. By using all SSD storage system, it is possible to enhance some system processes which are gamechanging Therefore, although SSDs are more expensive, the Purity system is cost-effective 44

45 Questions? 45