Yet other uses of a level of indirection...! Log-structured & Solid State File Systems Nov 19, Garth Gibson Dave Eckhardt Greg Ganger

Transcription

1 Yet other uses of a level of indirection...! Log-structured & Solid State File Systems Nov 19, 2009 Garth Gibson Dave Eckhardt Greg Ganger 1 L33_Adv_Filesystem , F10

2 Recall Unix multi-level per-file index Attributes Direct Block 1 Data (lbn 576) Direct Block 2... Direct Block 12 Data Data (lbn 344) (lbn 968) Indirect Block Data Block 13 Data (lbn 632) Data Block Data Block N Data Data (lbn 1944) (lbn 480) 2 Double-Indirect Block Indirect Block 1 Indirect Block 2... Data Block N+1 Data Block N+2... Data Block Q+1... Data Data Data (lbn 96) (lbn 176) (lbn 72) , F10

3 Why a multi-level index? Performance Sequential bandwidth by careful allocation of adjacent blocks Random bandwidth by O(lg) lookup structures to any address Fragmentation When space is low and fragmented into lots of small sequences of free blocks, can still grow any file Albeit, sequential bandwidth will suffer because of lots of seeks Metadata space proportional to file size Small files (with just a inode) can grow to terabytes and a pointer for every block so random access is still O(lg) , F10

4 Log-structured File Systems - Motivation Back in the days of the ideas of RAID (1980s) CS postulated that read performance would become less important because big caches work well for read But that writing must still be done to make changes durable to crashes, power down And RAID showed us that small writes were going to get more expensive RAID 5 does four disk positionings for 1 small random write RAID 6 does six! , F10

5 Recall RAID 5 Single-sector writes Modifying a single sector is hard Must maintain parity invariant for stripe Could read rest of stripe then do full stripe write Cheaper to fetch old version of sector & parity Change a sector of 0's to a sector of 1's without reading X, Y Old condition: 0 X Y = 0 New condition: 1 X Y = 1 Every bit flip in data causes a bit flip in parity Four disk accesses two read, two write , F10

6 Recall RAID 6 P+Q Redundancy Protects against multiple failures using Reed-Solomon codes Similar to codes used per sector inside disk, but simpler Uses 2 parity disks P is parity Q is a second code Its two equations with two unknowns, just make bigger bits Group bits into nibbles Add different coefficients to each equation two independent equations in two unknowns (erasures) For small writes, requires 6 I/Os!! Read old data, old parity1, old parity2 Write new data, new parity1, new parity , F10

7 Log-structured File Systems Mendel Rosenblum, John Ousterhout proposed (1990) Log-structured File Systems (LFS) to improve this: Make all writes append to the end of the log, sequentially, so there are no positionings everything is sequential! Still uses same multi-level index structure but blocks move as they are updated Delay and gather small writes from many different files into same end of log, large enough to be a large write (RAID parity calculated in memory, not by reading old data and parity) ACM Trans. on Computer Systems. Vol 10, No 1, , F10

8 Data & Metadata Layout (for 1 block files) Updated structures written to end of log Buffer as long as possible to seek less than once per file create Compare to FFS: seeks for 2 file inode writes, 1 file data write, 1 directory inode write, 1 directory data write (& free bitmap writes) , F10

9 Write instead of Read Optimized Alternative viewpoint: non-overwrite vs overwrite Constantly re-allocating on change can improve layout Simplifies crash recovery because old is available if new is not completely updated, and new is forgotten Importance of disk write BW overstated Successful read caching makes read unimportant (false) Unfortunately, larger DRAM offset by larger files, bigger disks & their metadata structures, larger app VM working sets But consecutive write order good for read back unless seq reads after random write (ugh) And SSDs may be answer for too many small random reads Synchronous writing for recovery is bad (true) Part answers: battery backup writeback caches for power fail LFS led into new era of disk layout NetApp WAFL leader in distrd file systems (PanFS is LFS too) , F10

10 Potential Performance Reducing seeks goes long way to make file system faster No fragment cleaning done in benchmark , F10

11 Cleaning the log But space freed by file deletes is needed Divide into segments & thread log thru segments , F10

12 Reconsidering Cleaning LFS uses continuous cleaning to defragment Seltzer95 (Usenix95) disagrees Fixed some old non-optimal FFS actions (update 1984 technology to LFS compared to weak target) Finds cleaning costs often outweigh benefits; background cleaning may defray cost to user workload Very workload dependent -- benefits of LFS best in small file create loads Maybe don t clean automatically Admin tool for defrag as needed (if needed) , F10

13 Recovery after Crash To avoid fsck (find & test all metadata) Checkpoint top of FS to fixed location (superblock) Contains pointer to last written segment, stats Double buffer checkpoint, use latest timestamp Eg. Every 30sec (better: min(30sec, X MB written)) Roll-forward log-segments after checkpoint Embed write-ahead log for directory changes (create, link, unlink, rename) so interrupted log writing can correct directory/inode pointers/refcounts , F10

14 On to Solid-State Disks Solid-state storage technology: alternative to disks Non-volatile memory chips, disk interface, no moving parts Faster access (no seeks) but more costly than disks Lower power (~none if not accessing), noise free (laptops!) Types of solid-state storage Flash (& RAMdisk, phase change, magnetic RAM) Performance differences No seeks means high rates of small random access Reliability differences Device fatigue (stops being programmable) Device retention (information leaks away) , F10

15 Why would we want solid-state disks? 15 Mechanical disk seeks are slow A few milliseconds to move head, a few more for rotate At most positionings per second Solid-state is transistors, word lines, RAM-like Accesses per second limited by RPC overhead Measurements: Random read SSD NAND flash vs magnetic disk Laptop (Mtron, Memoright) Server (Intel X25, Fusion-io iodrive) IOPS 100,000 10,000 1, K 100X 8K 16K 32K Request size 64K Memoright MTron X25 M X25 E iodrive 15000rpm 10000rpm 7200rpm , F10

16 Why would we not want solid-state disks? Cost per byte of Flash much higher than of Disk Base price of material & control (small computer) ~ fixed Number of bits on a disk much higher than on a Flash Disk amortizes base costs per byte better Data is not as durable or rewritable as with disk In comparison, disk data last ~ forever, infinite writes , F10

17 SSD Technologies RAMdisks: a PC w/ lots of DRAM and a battery Very expensive, very fast, limited poweroff data retention Flash: Flash!= SSD, but most SSD is NAND flash NOR: lower density, word accessible, slow write, ~EEPROM NAND: write in pages, erase in blocks of pages Embed a computer to manage wear/retention Future technologies of note Phase Change: amorphous vs lattice depend on heating Magnetic RAM: magnetic bit in middle of DRAM cell , F10

18 NAND Flash circuits stuff 18 Add a floating gate in a MOSFET cell, insulating it so charge on FG persists w/o voltage Vg applied to control gate above it Last maybe a year = long DRAM refresh J Read: apply a little control voltage & stored charge, if any, modifies Vg so drain current either flows or doesnt N cells in series, only 1 enabled (all others read 1) SLC: see only on/off; MLC: see 3 levels of on (2 bits) Erase (1) uses large (-) voltage to flush charge Slow, applied to all cells together Program (0) applies large positive voltage to inject charge Source-drain current common to all cells, so select one to write Reading & writing wears out insulation around floating gate Source: Intel , F10

19 Flash storage organization A flash block is a grid of cells " " " bytes per page hidden for ECC, addressing Source: Wobber, Microsoft , F10

20 NAND Flash SSD is a little computer Storage: flash chips Access: multiple independent access channels Interface: SATA Controller: computer + RAM 20 Processes cmds Drives channels Write behind Allocation Wear leveling , F 10

21 NAND Flash SSD is a little computer Storage: flash chips Access: multiple independent access channels Interface: SATA Controller: computer + RAM Processes cmds Drives channels Write behind Allocation Wear leveling 21 Source: Wobber, Microsoft , F10

22 Physical organization & specs (SLC, 4GB) PLANE 0 PLANE 1 PLANE 2 PLANE 3 PLANE Block 0 Size PLANE 1 PLANE PLANE KB 3 REG REG REG REG REG Plane REG REG 512 REG MB DIE 0 Data Register Page Size Die Size Erase Cycles DIE 1 4 KB 4 KB 2 GB 100K Page Read 25µs Page Program 200µs Serial Access 100µs Block Erase 1.5ms MLC (multiple bits in cell): slower, less durable 22 Source: Wobber, Microsoft , F10

23 Performance: Sequential (Small) Access 23 Read bandwidth depends on interface speed and chip parallelism Product decisions BW costs! $2400 $800 $400 Read (top) & write Measured on Linux with ext3 mounted Megabytes per second Megabytes per second K 4K 8K 8K 16K 32K Request size 16K 32K Request size 64K 64K Memoright MTron X25 M X25 E iodrive 15000rpm 10000rpm 7200rpm Memoright MTron X25 M X25 E iodrive 15000rpm 10000rpm 7200rpm , F10

24 Performance: Random small access (1) Read w/o seeks! 100X faster Some products have multiple chips, multiple channels (parallelism) IOPS 100,000 10,000 1, K 8K 16K 32K Request size 64K Memoright MTron X25 M X25 E iodrive 15000rpm 10000rpm 7200rpm , F10

25 Performance: Random small access (2) Read w/o seeks! 100X faster Some products have multiple chips, multiple channels (parallelism) Writes w/o seeks? Not faster at all? 100X faster? What is going on? IOPS IOPS 100,000 10,000 1, ,000 10,000 1, K 8K 16K 32K Request size 64K Memoright MTron X25 M X25 E iodrive 15000rpm 10000rpm 7200rpm Memoright MTron X25 M X25 E iodrive 15000rpm 10000rpm 7200rpm 10 4K 8K 16K 32K Request size 64K , F10

26 Performance-price Effectiveness Sequential access in SSD primarily limited by cost of sufficient parallelism, because flash chips are slow Magnetic disks are serial but very fast (tiny bits, whizzg by) Random access has those zero time seeks! 10X lower cost per access per second (IOPS) Less sophisticated SSDs dont do writes well , F10

27 Write Amplification Host writes 2 pages, how much does NAND write? Erase DRAM Block Copy (EB) Page 63 Page 62 Page 61 Page 63 Page 62 Page 61 Page 3 Page 2 Page 1 Page 0 Page 3 Page 2 Page 1 Page 0 Source: Intel pages (here): read-erase-modify-write costs 32X! Big difference between simple & sophisticated SSDs Here is where log-structured FS design comes back *Simplified example to illustrate the write amplification effect. Specific algorithms vary greatly , F10

28 Reducing Write Amplification (think LFS) Amplification worst for small, random writes Strategy: freely remap host address & NAND address Write blocks to some place other than where it was before Group bunch of different small writes into full blocks Write at sequential write rates Leaves holes in other blocks (where old data was) At some point, clean out the holes by reading a bunch of old blocks and writing back a smaller number of whole blocks Rate of cleaning depends amount of unallocated space Controller reserves X% hidden space (ie. 10, 20, 50%) , F10

29 Log writing (map granularity = 1 block) P P P 0 P 1 Block(P) 29 Source: Wobber, Microsoft , F10

30 Logging writes (map granularity = 1 page) P Q P P 0 Q 0 P 1 Page(P) Page(Q) 30 Source: Wobber, Microsoft , F10

31 Cleaning a page at a time P Q R P Q R P 0 Q 0 R 0 P 0 Q 0 R 0 Page(P) Page(Q) Page(R) 31 Move remaining valid pages so block can be erased Efficiency: Choose blocks to minimize page traffic This is a hard problem LFS says not just emptiest block because the valid data in emptiest block is likely to be deleted soon, making the cleaning work wasted Source: Wobber, Microsoft , F10

32 Cleaning often not free, far far from free! , F10

33 Making cleaning easier Over-provisioning Advertise fewer blocks than are really there (10% or 50%?) Pre-clean/background clean lots of blocks Leave partially deleted blocks longer to gather overwrites Host delete notification Disks (really SCSI and SATA) store containers, not info When OS decides some info is no longer needed, it puts that disk container on a free list, but doesnt tell disk!! Disk cant free deleted info until eventual overwrite Add TRIM command to SCSI/SATA to delete info Host-controlled free space reduces over-provisioning , F10

34 Wear leveling: spread writing evenly in SSD Recall: floating gate insulator degrades, accumulating charge that interferes, even when just reading Stored charge leaks over months to years Blocks not written recently need to be overwritten Read-erase-write or read-move/clean-erase After some number of erase/program cycles, its over 10K (consumer), 100K (enterprise), 1M (too costly?) Each block wears independently, so a heavily written block can wear out long before a mostly-read block Wear leveling is remapping addresses to balance number of erase/program cycles seen by each block , F10

35 Wear leveling: some numbers. 35 After some number of erase/program cycles, its over How many is enough? With limited BW into SSD, lifetime is reasonable N M MB/s = 1000*N/M secs to write all pages once * 100K /60/60/24/365 years to write all pages 100K times 128 GB, 100 MB/s, > 4 years 100% writing 24x7 IF WRITING EVENLY APPLIED TO ALL BLOCKS Really need to include write amplification writes and wear leveling writes Most sophisticated SSD controllers claim 10% extra Use up over-provisioning as blocks wear out Degrades performance, then stops offering full size , F10

36 One algorithm for leveling/cleaning Expiry Meter for block A Cold content Block A Block B Q R P Q R Q 0 R 0 P 0 Q 0 R 0 If If If Remaining(A) < >= < Throttle-Threshold, Migrate-Threshold, clean reduce A, but probability migrate clean A of cold cleaning data into A A 36 Source: Wobber, Microsoft , F10

37 SSDs use lower power than disks Nothing needs to be spun (always!) Nothing needs to be seeked Serial bits use faster logic than parallel 37 Source: Intel , F10

38 SSDs use lower power than disks (2) SSD uses almost no power most of the time 38 Source: Intel , F10

39 Summary 39 Log-structuring Write all changed blocks to a new address, sequentially highly concurrent random or small accesses get high write bandwidth; read bandwidth is mostly unchanged Rotating disks SSDs Slow seeks, low cost per bit Mostly constant map from LBA to on-disk location (simpler) Fast seeks, lower power/noise, higher cost per bit Dynamic LBA map, write amplification, wear-leveling. complicated , F10