Computer Systems Laboratory Sungkyunkwan University

Transcription

1 Storage Jin-Soo Kim Computer Systems Laboratory Sungkyunkwan University

2 Storage: A Logical View Abstraction given by block device drivers: 512B 512B 512B 0 1 N 1 Operations Identify(): returns N Read(start sector #, # of sectors) Write(start sector #, # of sectors) Source: Sang Lyul Min (Seoul National Univ.) 2

3 Storage Storage-level techniques RAM Drive NBD (Network Block Device) DRBD (Distributed Replicated Block Device) Distributed Storage LVM (Logical Volume Manager) Disk mirroring RAID (Redundant Array of Inexpensive Disks) Storage Area Network Solid State Drives 3

4 Hard Disk Drives (HDDs)

5 Secondary Storage Secondary storage usually is anything that is outside of primary memory. does not permit direct execution of instructions or data retrieval via machine load/store instructions. Characteristics It s large: 100GB and more It s cheap: 1TB SATA2 disk costs 80,000 It s persistent: data survives power loss. It s slow: milliseconds to access. 5

6 HDDs (1) Electromechanical Rotating disks Arm assembly Electronics Disk controller Buffer Host tinterface 6

7 HDDs (2) Seagate Barracuda ST AS (1TB) 4 Heads, 2 Discs Max. recording density: 1413K BPI (bits/inch) Avg. track density: 236K TPI (tracks/inch) Avg. areal density: 329 Gbits/sq.inch Spindle speed: 7200rpm (8.3ms/rotation) Average seek time: < 8.5ms (read), < 9.5ms (write) Max. internal data transfer rate: 1695 Mbits/sec Max. I/O data transfer rate: 300MB/sec (SATA-2) Max. sustained data transfer rate: 125MB/sec Internal cache buffer: 32MB Max power-on to ready: < 10.0 sec 7

8 HDDs (3) Hard disk internals Our Boeing 747 will fly at the altitude of only a few mm at the speed of approximately 65mph periodically landing and taking off. And still the surface of the runway, which consists of a few mm-thick layers, will stay intact for years. 8

9 Managing Disks (1) Disks and the OS Disks are messy physical devices: Errors, bad blocks, missed seeks, etc. The job of the OS is to hide this mess from higherlevel software. Low-level device drivers (initiate a disk read, etc) Higher-level abstractions (files, databases, etc.) The OS may provide different levels l of disk access to different clients. Physical disk block (surface, cylinder, sector) Disk logical block (disk block #) Logical file (filename, block or record or byte #) 9

10 Managing Disks (2) Interacting with disks Specifying disk requests requires a lot of info: Cylinder #, surface #, track #, sector #, transfer size, etc. Older disks required the OS to specify all of this The OS needs to know all disk parameters. Modern disks are more complicated. Zoned bit recording (ZBR), sectors are remapped, etc. Current disks provide a higher-level interface (e.g., SCSI) The disks exports its data as a logical array of blocks [0..N-1] Disk maps logical blocks to cylinder/surface/track/sector. Only need to specify the logical block # to read/write. e As a result, physical parameters are hidden from OS. 10

11 Managing Disks (3) Disk performance Performance depends on a number of steps Seek: moving the disk arm to the correct cylinder depends on how fast disk arm can move (increasing very slowly) Rotation: waiting for the sector to rotate under head depends on rotation rate of disk (increasing, but slowly) Transfer: transferring data from surface into disk controller, sending it back to the host. depends on density of bytes on disk (increasing, and very quickly) Disk scheduling: Because seeks are so expensive, the OS tries to schedule disk requests that are queued waiting for the disk. 11

12 FCFS FCFS (= do nothing) Reasonable when load is low. Long waiting times for long request queues. 12

13 SSTF Shortest seek time first Minimizes arm movement (seek time) Maximizes request rate Unfairly favors middle blocks May cause starvation of some requests 13

14 SCAN Elevator algorithm Service requests in one direction until done, then reverse Skews wait times non-uniformly 14

15 C-SCAN Circular SCAN Like SCAN, but only go in one direction (e.g. typewritters) Uniform wait times 15

16 LOOK / C-LOOK LOOK / C-LOOK Similar to SCAN/C-SCAN, but the arm goes only as far as the final request in each direction. C LOOK 16

17 Disk Scheduling Selecting a disk scheduling algorithm SSTF is common and has a natural appeal. SCAN and C-SCAN perform better for systems that place a heavy load on the disk. Either SSTF or LOOK is a reasonable choice for the default algorithm. Performance depends on the number and types of requests. Requests for disk service can be influenced by the file allocation method. In general, unless there are request queues, disk scheduling does not have much impact. Important for servers, less so for PCs Modern disks often do the disk scheduling themselves. Disks know their layout better than OS, can optimize better. Ignores, undoes any scheduling done by OS. 17

18 Modern Disks Intelligent controllers A small CPU + many kilobytes of memory. They run a program written by the controller manufacturer to process I/O requests from the CPU and satisfy them. Intelligent features: Read-ahead: the current track Caching: frequently-used dblocks Command queueing Request reordering: for seek and/or rotational optimality Request retry on hardware failure Bad block/track identification Bad block/track remapping: onto spare blocks and/or tracks 18

19 I/O Schedulers I/O scheduler s job Improve overall disk throughput Merging requests to reduce the number of requests Reordering and sorting requests to reduce disk seek time Prevent starvation Submit requests before deadline Avoid read starvation by write Provide fairness among different processes Guarantee quality-of-service (QoS) requirement 19

20 Linux I/O Schedulers (1) Linus elevator scheduler Merges adjacent I/O requests Sorts I/O requests in ascending block order Writes-starving-reads: Stop insertion-sorting if there is a sufficiently old request in the queue Trades fairness for improved global throughput Not really an elevator: puts requests with a low sector number at the top of the queue regardless of the current head position Real-time? Was the default I/O scheduler in Linux

21 Linux I/O Schedulers (2) Deadline scheduler Two standard Read/Write sorted queues (by LBA) + Two Read/Write FIFO queues (by submission time) Each FIFO queue is assigned an expiration value. Read: 500 msec Write: 5 sec Normally, send I/O requests from the head of the standard d sorted queue. If the request at the head of one of the FIFO queues expires, services the FIFO queue Emphasizes average read request response time No strict guarantees over request latency 21

22 Linux I/O Schedulers (3) Anticipatory scheduler We will see shortly Was the default I/O scheduler in the 2.6 kernel Dropped in the Linux Kernel e in favor ao of te the CFQ scheduler 22

23 Linux I/O Schedulers (4) Noop scheduler Minimal overhead I/O scheduler Only yperforms merging g For random-access access devices such as RAM disks s and solid state drives (SSDs) For storage with intelligent HBA or externally attached controller (RAID, TCQ drives) 23

24 Linux I/O Schedulers (5) CFQ Q( (Complete Fair Queuing) scheduler Assigns incoming I/O requests to specific queues based on the process originating the I/O request Within each queue, requests are coalesced with adjacent requests and insertion sorted Service the queues round robin, plucking a configurable number of requests (by default, four) from each queue before continuing on to the next Fairness at a per-process level Initially for multimedia applications, but works well across many workloads Subsumes anticipatory scheduling New default scheduler for Linux 2.6 (from ) 24

25 NAND Flash Memory

26 Memory Types FLASH High density Low cost High speed Low power High reliability DRAM High density Low cost High speed High power EPROM Non volatile High density Ultraviolet light for erasure EEPROM Non volatile Lower reliability Higher cost Lowest density Electrically byte erasable ROM High density Reliable Low cost Suitable for high production with stable code Source: Intel Corporation. 26

27 Flash Memory Characteristics Flash memory Non-volatile, Updateable, High-density Low cost, Low power consumption, High reliability Erase-before-write Read Write or Program: 1 0 Erase: Read faster than write/erase Bulk erase Erase unit: block write (program) erase Program unit: byte or word (NOR), page (NAND) 27

28 NOR Flash NOR flash Random, direct access interface Fast random reads Slow erase and write Mainly for code storage Intel, Numonyx, Spansion, STMicro,... 28

29 NAND Flash NAND flash I/O mapped access Smaller cell size Lower cost Smaller size erase blocks Better performance for erase and write Mainly for data storage Samsung, Toshiba, Hynix,... 29

30 NAND Flash Architecture 2Gb NAND flash device organization Source: Micron Technology, Inc. 30

31 NAND Flash Types SLC NAND 1 SLC NAND 2 (small block) (large block) MLC NAND 3 Page size (Bytes) , , Pages / Block Block size 16KB 128KB 512KB t R (read) 15 μs (max) 20 μs (max) 50 μs (max) t PROG (program) t BERS (erase) 200 μs (typ) 500 μs (max) 2 ms (typ) 3 ms (max) 200 μs (typ) 700 μs (max) 1.5 ms (typ) 2 ms (max) 600 μs (typ) 1,200 μs (max) 3 ms (typ) NOP 1 (main), 2 (spare) 4 1 Endurance Cycles 100K 100K 10K ECC 1 bit ECC 1 bit ECC 4 bits ECC (per 512Bytes) 2 bits EDC 2 bits EDC 5 bits EDC 1 Samsung K9F1208X0C (512Mb) 2 Samsung K9K8G08U0A (8Gb) 3 Micron Technology Inc. 31

32 NAND Applications Universal Flash Drives (UFDs) Flash cards CompactFlash, MMC, SD, Memory stick, Embedded devices Cell phones, MP3 players, PMPs, PDAs, Digital TVs, Set-top boxes, Car navigators, Hybrid HDDs Intel Turbo Memory SSDs (Solid-State Disks) 32

33 SSDs (1) HDDs vs. SSDs 1.8 HDD Flash SSD (78.5x54x4.15mm) HDD Flash SSD (101x70x9.3mm) Top Bottom 33

34 SSDs (2) Feature SSD (Samsung) HDD (Seagate) Model MMDOE56G5MXP (PM800) ST AS (Momentus ) Capacity Form factor Host interface Power consumption Performance Measured performance 1 (On MacBook Pro, 256KB for sequential, 4KB for random) 256GB (16Gb MLC x 128, 8 channels) 2.5 Weight: 84g Serial ATA 2 (3.0 Gbps) Host transfer rate: 300MB Active: 0.26W Idle/Standby/Sleep: 0.15W 500GB (2 Discs, 4 Heads, 7200RPM) 2.5 Weight: 110g Serial ATA 2 (3.0 Gbps) Host transfer rate: 300MB Active: 2.1W (Read), 2.2W (Write) Idle: 0.69W, Standby/Sleep: 0.2W Sequential read: Upto 220 MB/s Power onto on ready: 4.5sec Sequential write: Up to 185 MB/s Average latency: 4.17 msec Sequential read: MB/s Sequential write: MB/s Random read: MB/s Random write: 2.93 MB/s Price 2 759,000 won 140,000 won Sequential read: MB/s Sequential write: MB/s Random read: 0.61 MB/s Random write: 1.28 MB/s 1 Source: 2 Source: (As of Jul. 24, 2010) 34

35 NAND Constraints (1) No in-place update Require sector remapping (or address translation) Bit errors Require the use of error correction codes (ECC) Bad blocks Factory-marked & run-time bad blocks Require bad block remapping Limited program/erase cycles < 100K for SLCs < 10K for MLCs Require wear-leveling 35

36 NAND Constraints (2) Limited NOP (Number of Programming) g) 1 / sector for most SLCs (4 for 2KB page) 1 /page for most MLCs Sequential page programming For large block SLCs and MLCs Pair-page programming in MLCs Two pages inside a block are linked together Performance difference Interference 36

37 FTL (1) Flash cards internals 37

38 FTL (2) SSDs internals Source: Mtron Technology 38

39 FTL (3) Flash Translation Layer (FTL) A software layer to make NAND flash fully emulate traditional block devices (e.g., disks). Why FTL? No in-place-update Bulk erase File System Read Sectors Wit Write Sectors Mismatch! Read Write Erase Device Driver + File System Read Sectors Wit Write Sectors Read Sectors Write Sectors FTL + Device Driver + Flash Memory Flash Memory 39

40 FTL (4) For performance Sector mapping (or address translation) Garbage collection Interleaving over multiple channels & flash chips Request scheduling Buffer management For reliability Power-off recovery Wear-leveling Bad block management Error correction code (ECC) 40

41 Sector Mapping (1) General page mapping Most flexible Efficient handling of small writes Large memory footprint One mapping entry per page: 32MB for 32GB MLC (4KB page) Bitmap for page validity Per-block invalid page counter Sensitive to the amount of reserved blocks Performance affected as the system ages LPN 0 LPN 1 LPN 2 LPN 3 LPN 4 LPN 5 LPN 6 LPN 7 LPN 8 LPN 9 LPN 10 LPN 11 LPN 12 LPN 13 LPN 14 LPN 15 Data blocks Log blocks W = <1, 2, 8, 1, 2, 12, 13, 9> 41

42 Sector Mapping (2) Naïve block mapping Each table entry maps one block Small RAM usage Data blocks Inefficient handling of small writes Log blocks LBN 0 LBN 1 LBN 2 LBN W = <4, 5, 6, 7, 1>

43 Sector Mapping (3) Log block scheme [IEEE TOCE 2002] A small number of log blocks 1+ log block(s) per data block Data blocks Page mapping for log blocks 0 12 Full/partial/switch merge 4 5 LBN 0 6 Switch merge for sequential LBN 1 7 LBN 2 LBN 3 updates Log blocks Low log block utilization W = <1, 2, 8, 1, 2, 12, 13, 9> 43

44 Sector Mapping (4) FAST [ACM TECS 2007] Log blocks shared by all data blocks Sequential/random log blocks Improved po log ogboc block utilization ato Data blocks Log blocks Increased merge time 4 56 LBN 0 LBN 1 LBN 2 LBN W = <1, 2, 8, 1, 2, 12, 13, 9> 44

45 Sector Mapping (5) Superblock FTL [ACM EMSOFT 2006] Superblock = logically adjacent N blocks A superblock shares log blocks Up to M log blocks per superblock Page mapping within a superblock LSBN 0 LSBN 1 LPN 0 LPN 1 LPN 2 LPN 3 LPN 4 LPN 5 LPN 6 LPN 7 Data blocks Log blocks Hot/cold pages separation 8 LPN 8 9 The amount of mapping information increased LPN 9 LPN 10 LPN 11 LPN 12 LPN 13 LPN 14 LPN W = <1, 2, 8, 1, 2, 12, 13, 9> 45

46 Sector Mapping (6) μ-ftl [ACM EMSOFT 2008] Page mapping Multiple mapping granularities Based on extents Reduce the amount of mapping information Requires more sophisticated index structure μ-tree is used to store the mapping information Tunable memory footprint Frequently accessed mapping information i cached in memory LPN 0, 1 LPN 1, 1 LPN 2, 1 LPN 3, 1 LPN 4, 4 LPN 8, 1 LPN 9, 1 LPN 10, 2 LPN 12, 2 LPN 14, 2 Data blocks Log blocks W = <1, 2, 8, 1, 2, 12, 13, 9> 46

47 Performance (1) Simulation environment 4GB flash memory Large block SLC NAND (2KB page, 128KB block) FTL schemes Naïve block mapping Replacement block Log block Superblock Workload Trace from PC using NTFS 47

48 Performance (2) Extra erase and write operations 256 extra blocks Naive Sain Replacement Logblock Superblock 0 Erase 0 Write(GC) 48

49 OS Implications (1) NAND flash has different characteristics compared to disks No seek time Asymmetric read/write access times No in-place-update p Good sequential read/sequential write/random read performance, but bad random write performance Wear-leveling Traditional operating systems have been optimized for disks. What should be changed? 49

50 OS Implications (2) SSD support in Microsoft Windows 7 Turn off defragmentation for SSDs New TRIM command Remove-on-delete Align file system partition with SSD layout Larger block size proposal (4KB) 50

51 Summary Software support for NAND flash memory is essential for improving performance & reliability of the system We need to revisit OS policies and mechanisms which are optimized for disks 51