Storage Systems for Shingled Disks

Transcription

1 Storage Systems for Shingled Disks Garth Gibson Carnegie Mellon University and Panasas Inc Anand Suresh, Jainam Shah, Xu Zhang, Swapnil Patil, Greg Ganger

2 Kryder s Law for Magnetic Disks Market expects ever more dense disks Future is multi-terabit per square inch Real challenge is making money at $100/disk when engineering is this hard 2

3 Directions in High Capacity Disks Heat-Assisted (HAMR) Small bits need high coercivity media to retain orientation High coercivity can t be changed by normal writing Heated media lowers coercivity Include lasers? Bit-Patterned (BPM) Small bits retain orientation easier if bits kept apart Pattern media so only write a single dot per bit Tera-dots per sq. inch? 3

4 Shingled Magnetic Recording (SMR) 4

5 5

6 File systems do far too much small random writing 6

7 Disk becomes tape!! File systems do far too much small random writing 7

8 What About Reading? Read head is possibly thinner than write head If target is 2-3 X density, maybe not too hard Targeting higher density sees lots of crosstalk Signal processing in two dimensions (TDMR) One approach to TDMR involves gathering signal from 1-2 adjacent tracks on both sides Means 3 to 5 revs to read a single sector Not likely to be accepted by marketplace Safe plan is to see residual track w/ only 1 head 8 G. Gibson, Sept 2012

9 Geometry Model: Getting a handle on the parameters 9

10 Shingled writing: need big bands Reason for doing it: density Shingling projected at X track density Can mix shingled and non-shingled so, e.g., separate sequential from random just lose some of the density gains Can break up sets of shingled tracks ( bands ) allowing overwrite of individual bands but, they need to be big like 32 to 256 MB 10

11 Simple Geometry Model $!#!" # % & % ' ( & ( ' ) SMR allows wider write heads, w >w SMR reduces gaps, g, per track to per band (B tracks) Residual (readable) track width (r) after overlapping is a key factor A fraction of tracks not shingled, f, allows some random sector writing * 11

12 Simple Geometry Model $!#!" # % & % ' ( & ( ' ) * SMR allows wider write heads, w >w SMR reduces gaps, g, per track to per band (B tracks) Residual (readable) track width (r) after overlapping is a key factor A fraction of tracks not shingled, f, allows some random sector writing SMR increase in areal density given by simple model 12

13 Areal Density Favors Large Bands Eg. w=25, g=5, w =70, r=10,13,20 nm, f=0%,1%,10% 13

14 Areal Density Favors Large Bands Eg. w=25, g=5, w =70, r=10,13,20 nm, f=0%,1%,10% 1% unshingled is affordable 10% if r<w small B bad news r~=w needs large B (~100+) r<w allows smallish B (~10) But not soon. Systems should plan for large bands 14

15 Coping with SMR at the system level 15

16 Convergence with Flash 16

17 Transparent STL/FTL approach Shingled disks implement translation Same types of algorithms as Flash Data will be correct using existing program code But, not performance transparent Erase block: X bigger Read-erase-write: X longer Sure to exceed long tolerable latency thresholds And, not cost transparent Disk margins < flash margins Yet disk STL needs more resources 17

18 Non-transparent SMR interface Define an interface exposing key differences Bands, non-shingled regions, trim, Modify systems software to avoid and minimize read-modify-write Log-structured files systems 20 years old STL-like technology not costly in host Cloud storage writes in 64 MB chunks (HDFS) Flash, PCM, etc may be available to host 18

19 Non-transparent SMR interface Standards processes in T13 and T10 exist Key idea: disk attribute says sequential writing Each band has a write cursor for next write LBA Writes before and reads after cursor are bad Software can reset cursor, mostly to start of band Software can ask for map of bands & cursors 19

20 Experimenting with File Systems for SMR 20

21 Project Plan Demonstrate systems using SMR interface Cloud/BigData initial target application space Mock interface models SMR device Hadoop / HDFS first example Chunks ~= Bands HDFS is write once, so easier to pack frags Log-structured Merge Tree/LFS? Implement directories and inodes as table entries Logs of changes in tables written as bands 21

22 Start w/ class project framework 1) 2) Application 3) 1 4 4) Your FUSE file system (melangefs) 5) ) 2 <F1> SSD (ext2) HDD (ext2) 7) App does create(f1.txt) MelangeFS creates f1.txt in SSD Ext2 on SSD returns a handle for f1.txt to FUSE FUSE translates that handle into another handle which is is returned to the app App uses the returned handle to write to f1.txt on the SSD When f1.txt grows big, MelangeFS moves it to HDD, and f1.txt on the SSD becomes a symlink to the file on HDD Because this migration has to be transparent, app continues to write as before (all writes go to the HDD). 22

23 Experimental Platform Today Write to open Open file cache open(f2,w) file go to cache F1 F2 Metadata ops for F2 Unshingled partition F2 written to SMR on close() Shingled partition User USER-LEVEL EMULATOR Hadoop/ HDFS FUSE Shingledfs SMR File-to-Band/Block translation Model T13 interface model SMR Device Emulator ext3 To disk 23

24 Prototype Banded Disk API CMU view of API essentials Edi_modesense() Discover band information (number, size) Edi_managebands(OP, band, offset, length) GET: where is next_write_offset? SET: change next_write_offset (mostly to 0) Edi_read(band, Offset must be less than next_write_offset Edi_write(band, Offset offset, length) offset, length) must be next_write_offset (else reject) 24

25 Hadoop Sort Benchmark 6 node Hadoop cluster: write, sort, verify X GBs Compare local disk, FUSE-local, FUSE-SMR FUSE causes most overhead No cleaning during tests SMR file system can support Big Data apps 25

26 Ongoing Project Directions 26

27 Future Work: General Workloads Compile Linux on SMR Bigger overheads Especially untar Lots of small files, lots of directory operations, etc 27

28 Future Work: Pack Metadata Change traditional file systems in unshingled region Use LSM Tree for directories, inodes Eg. LevelDB Most metadata on disk in SSTable blobs Initial experiments reduce disk seeks for metadata ops 28

29 Summary of status Experiments: SMR appropriate for Big Data apps Deployment: embed in HDFS DataNode servers or local file system Implementation greatly simplified by one file: one band files open for write held in memory until close Hadoop/HDFS is write-once Next steps: Cleaning overhead, cluster soon-to-delete Log-structured Merge Tree to pack metadata 29

30 Further reading technical reports: CMU-PDL : Big Data experiments CMU-PDL : Principles of operations CMU-PDL : TableFS approach Thanks to our sponsors: Seagate and the PDL Consortium (Actifio, APC, EMC, Emulex, Facebook, Fusion-IO, Google, HP Labs, Hitachi, Huawei, Intel, Microsoft, NEC, NetApp, Oracle, Panasas, Riverbed, Samsung, STEC, Symantec, VMWare, Western Digital) 30