Storage Systems : Disks and SSDs. Manu Awasthi CASS 2018

Transcription

1 Storage Systems : Disks and SSDs Manu Awasthi CASS 2018

2 Why study storage? Scalable High Performance Main Memory System Using Phase-Change Memory Technology, Qureshi et al, ISCA 2009

3 Trends Total amount of data created Type of data Image courtesy -

4 Cost versus access time for DRAM and HDD

5 Storage Systems : Today DRAM DRAM DRAM CPU SATA/SAS I/O Bridge PCIe/ NVMe HDD HDD Storage SSD 5

6 System Organization

7 First Things First Storage devices are accessed differently than memory devices Access granularity Block minimum 512B, typically 4 8 KB Access is usually through a deeper, software stack Much higher access latencies Milliseconds vs ns Interfaces are slower than everything that we have seen so far SATA, SAS, PCIe Metrics for comparison Latency, Bandwidth, IOPS

8 I/O Workloads Types Random and Sequential

9 What Does Random Mean? Q U I A K C KEYS ARE ALL OVER A QUICK BROWN FOX JUMPED OVER A LAZY DOG IMAGINE THAT THE KEYBOARD IS A DISK DRIVE Slide courtesy

10 What Does Sequential Mean? EVERY KEY IS NEXT TO PREVIOUS IMAGINE THAT THE KEYBOARD IS A DISK DRIVE Slide courtesy

11 Sequential Read Example SEQUENTIAL READ Slide courtesy

12 I/O Workloads Types Random and Sequential Most workloads are a combination of the above X% sequential; (100-x)% random, or some combination thereof Varies over time as well Most workloads also have varying granularity of request sizes Picture courtesy - Characterization of Storage Workload Traces from Production Windows Servers Swaroop Kavalanekar et al., IISWC 2008

13 Magnetic Disks/ Hard Disk Drives/ HDDs A magnetic disk consists of 1-12 platters (metal or glass disk covered with magnetic recording material on both sides), with inch diameter Each platter is comprised of concentric tracks (5-30K) and each track is divided into sectors ( per track, each about 512 bytes) A movable arm holds the read/write heads for each disk surface and moves them all in tandem a cylinder of data is accessible at a time dule4/lec28/images/image001.jpg

14 Hard Disk Drive bms/diskorg.htm

15 Disk Latency To read/write data, the arm has to be placed on the correct track this seek time usually takes 5 to 12 ms on average can take less if there is spatial locality Rotational latency is the time taken to rotate the correct sector under the head average is typically more than 2 ms (15,000 RPM) Transfer time is the time taken to transfer a block of bits out of the disk and is typically 3 65 MB/second A disk controller maintains a disk cache (spatial locality can be exploited) and sets up the transfer on the bus (controller overhead)

16 Shingled Magnetic Recording HDDs Typical Drive Reduce the guard space between the tracks to increase density Overlapping tracks look like roof shingles, hence the same SMR Hard Disk Drive Figure courtesy -

17 SSDs Why now? NAND Flash has been around since 1980s why the sudden increase in interest? Slide courtesy : Michael Freedman, COS 518: Advanced Computer Systems Lecture 8, Harvard University

18 Flash And NAND Gates EVERY NAND CAN BE SET TO 0 INDIVIDUALLY TO SET BACK TO 1, AN ENTIRE GROUP NEEDS TO BE RESET WRITE 0 WRITE INPUT WRITE INPUT COMMON INPUT ERASE INPUT COMMON INPUT WRITE 1 ERASE INPUT Slide courtesy 18

19 NAND Flash: Architecture Architecture: Pages: 4-16 KB, assembled into Blocks: 128KB, 256KB, 4MB, 8 MB Slide courtesy : Michael Freedman, COS 518: Advanced Computer Systems Lecture 8, Harvard University 19

20 NAND Flash Basics Storing bits Single Level Cell (SLC) encodes single bit Multi-level Cell (MLC) 2 bits Triple Level Cell (TLC) three bits Levels of Organization Banks/planes Blocks/Erase Blocks ( KB) Page (4 KB) ge2011/dsc_3756.jpg 4-advanced-functionalities-and-internal-parallelism/

21 NAND Flash: Reads/ Writes Always read an entire page: Can only read entire aligned page from SSD Always write an entire page: To change single byte, need to write entire page Pages cannot be overwritten Page can be written only if the free / erased state. Updating: Read page to internal register, modify, then write to free page Erases are aligned on block size To make a page free, need to erase it Erasures can only occur at block boundary Slide courtesy : Michael Freedman, COS 518: Advanced Computer Systems Lecture 8, Harvard University 21

22 Flash Operations Read (Page) Read any page in device Fast - 10s of microseconds, irrespective of page number Erase (Block) To write a page, the entire block has to be erased content destroyed every bit changed to 1. Implication all data needs to be copied safely before erase Slow few millisecs to complete Program (Page) Can write to a page after block erase Fast(er) - 100s of microseconds

23 Example

24 Flash Based SSDs Turning Raw Flash to a storage device requires provide the standard block interface to applications/os SSD contains Flash chips (of course) Volatile memory (SRAM/DRAM) Control logic for operations FTL Flash Translation Layer Takes logical rd/wr ops; converts them into ops for device

25 Flash Translation Layer Flash translation layer (FTL) Provides abstraction of flash memory characteristics Maintains logical to physical address mapping Carries out cleaning operations Conducts wear leveling Flash Memory Flash Request Host Machine FTL Flash Request IO Request Flash Request FTL in multiple flash chip environment Manages parallelism and wear level among chips Flash Memory module Flash Memory module Flash Memory module Slide coutresy : FTL Design Exploration in Reconfigurable High-Performance SSD (RHPSSD) for Server Applications, ICS 2009

26 Garbage Collection FLASH DEVICE GARBAGE COLLECTION LOGICAL TO PHYSICAL REDIRECTION MAP 0X00 0X01 0X02 0XAA BLOCK BLOCK BLOCK BLOCK CLEAN DATA DIRTY DATA BLOCK BLOCK BLOCK BLOCK DIRTY DIRTY DIRTY DIRTY ERASE BLOCK BLOCK BLOCK BLOCK DIRTY DIRTY DATA DIRTY NUMBER OF BLOCKS DEFINES CAPACITY BLOCK BLOCK BLOCK BLOCK CLEAN DIRTY DIRTY DIRTY ERASE 1 DIRTY BLOCK AT A TIME (WHEN NUMBER OF CLEAN BLOCKS IS LOW) Slide courtesy 26

27 Garbage Collection Example Erase Block 0 Block 1 Block Write! A1 A2 B1 B2 C1 C2 A1 A2 B1 B2 D1 D2 VE I VE I VE I VE I V V VE VE E E V V

28 Write Amplification G moved to new block by the garbage collector Stale pages Cleaned cannot be block can overwritten now be or erased rewritten individually AK BL C Block X D G E A F B C D E F G A B Block ZY C F D H E I J A B Once all pages have been written, valid pages must be consolidated to free up space Write amplification: a write triggers garbage collection/compaction One or more blocks must be read, erased, and rewritten before the write can proceed Slide courtesy : Christo Wilson 28

29 SSD Controllers SSDs are extremely complicated internally All operations handled by the SSD controller Maps LBAs to physical pages Keeps track of free pages, controls the GC May implement background GC Performs wear leveling via data rotation Controller performance is crucial for overall SSD performance Slide courtesy : Christo Wilson 29

30 Flavors of NAND Flash Memory Multi-Level Cell (MLC) Multiple bits per flash cell For two-level: 00, 01, 10, 11 2, 3, and 4-bit MLC is available Higher capacity and cheaper than SLC flash Lower throughput due to the need for error correction write cycles Consumes more power Single-Level Cell (SLC) One bit per flash cell 0 or 1 Lower capacity and more expensive than MLC flash Higher throughput than MLC write cycles Expensive, enterprise drives Consumer-grade drives Slide courtesy : Christo Wilson 30

31 Other Concerns Wear Leveling Each cell has limited number of P/E cycles Keep track of how many times physical pages have been erased/written Try to make sure all pages have similar numbers

32 The Simplified I/O Software Stack Application File System System Call Interface Page Cache Device Mapper Block Layer Device Driver 32

33 Linux I/O Stack

34 DRAM inside HDDs/SSDs DRAM can cache I/O data within an SSD What should you cache?

35 Bonus Slide # 2 Caching I/O Data in DRAM Main memory is used as a I/O cache, among other things What data should be cached? Who should cache it? The application? The OS?

36 Hybrid SSD + HDD Architectures HDDs are cheap; have larger capacities SSDs are fast, are much more expensive Is there a mechanism to get the best of both worlds? Improving Flash-based Disk Cache with Lazy Adaptive Replacement, Huang et al., ACM Transactions on Storage (TOS), 2016

37 Thanks!

38 RAID Reliability and availability are important metrics for disks RAID: redundant array of inexpensive (independent) disks Redundancy can deal with one or more failures Each sector of a disk records check information that allows it to determine if the disk has an error or not (in other words, redundancy already exists within a disk) When the disk read flags an error, we turn elsewhere for correct data

39 RAID 0 and RAID 1 RAID 0 has no additional redundancy (misnomer) it uses an array of disks and stripes (interleaves) data across the arrays to improve parallelism and throughput RAID 1 mirrors or shadows every disk every write happens to two disks Reads to the mirror may happen only when the primary disk fails or, you may try to read both together and the quicker response is accepted Expensive solution: high reliability at twice the cost rd_raid_levels

40 RAID 3 Data is byte-interleaved across several disks and a separate disk maintains parity information for a set of bytes For example: with 8 disks, byte 0 is in disk-0, byte 1 is in disk-1,, byte 7 is in disk-7; disk-8 maintains parity for all 8 bytes For any read, 8 disks must be accessed (as we usually read more than a byte at a time) and for any write, 9 disks must be accessed as parity has to be recalculated High throughput for a single request, low cost for redundancy (overhead: 12.5%), low task-level parallelism rd_raid_levels

41 RAID 4 and RAID 5 Data is block interleaved this allows us to get all our data from a single disk on a read in case of a disk error, read all disks Block interleaving reduces throughput for a single request (as only a single disk drive is servicing the request), but improves task-level parallelism as other disk drives are free to service other requests On a write, we access the disk that stores the data and the parity disk parity information can be updated simply by checking if the new data differs from the old data

42 RAID 5 If we have a single disk for parity, multiple writes can not happen in parallel (as all writes must update parity info) RAID 5 distributes the parity block to allow simultaneous writes

43 Solid State Devices SSD consists of a set of flash memory packages that are connected to a controller Package has one or more dies Die has multiple planes Planes have multiple blocks Blocks have multiple 4KB pages Difference from Hard Drives Absence of mechanical moving parts Can not overwrite data as is; need to erase blocks first Multiple levels of parallelism ge2011/dsc_3756.jpg _papers/rajimwale/rajimwale_html/

44 ~2016 Seagate ($50) 1TB HDD 7200RPM Model: STD1000DM003-1SB10C Samsung ($330) 512 GB 960 Pro NVMe PCIe M.2 Model: MZ-V6P512BW Operation HDD Performance SSD Performance Sequential Read 176 MB/s 2268 MB/s Sequential Write 190 MB/s 1696 MB/s Random Read 4KiB Random Write 4KiB MB/s 121 IOPS MB/s 224 IOPS 44.9 MB/s 10,962 IOPS 151 MB/s 36,865 IOPS DQ Random Read 4KiB DQ Random Write 4KiB MB/s 292 IOPS MB/s 227 IOPS MB/s IOPS 399 MB/s 97,412 IOPS 44

45 Log Structured FTL Similar to concepts used in log structured file systems On a write to logical block N, append the write to the next free spot in the currentlybeing-written-to block

46 Final State

47 Garbage Collection Overwriting blocks happens out of place Need to claim pages with older, invalid versions of pages Before GC After GC

48 Mapping Tables Page level mappings become impractical as size of SSD grows Block Based Mapping Similar to having bigger page sizes in a virtual memory system Hybrid Mapping Always keep a few blocks erased and write to them, keep those mappings in log-table also maintains per-page mappings in data-table On lookup, consult log-table first, then data table