Fully journaled filesystems. Low-level virtualization Filesystems on RAID Filesystems on Flash (Filesystems on DVD)

Transcription

1 RAID_and_Flash Page 1 Beyond simple filesystems 4:33 PM Fully journaled filesystems. Low-level virtualization Filesystems on RAID Filesystems on Flash (Filesystems on DVD)

2 RAID_and_Flash Page 2 Network appliances Tuesday, December 08, :55 PM Our transactional filesystem: A network appliance fileserver (netapp02.cs.tufts.edu) 20 terabytes (last I checked) Fully transactional (proprietary) filesystem Used for: your home directories /g (course directories) Your home directories contain an invisible directory ".snapshot" You can travel back in time up to a month to recover deleted files! Reason this works: The "journal" doesn't just contain current files, it contains all versions, up to the point where it gets rewritten for old versions. To find an old version: chop the journal at a specific real time, ignore what's after it.

3 RAID_and_Flash Page 3 How a transactional filesystem works Tuesday, December 07, :40 AM Whole disk is one huge transaction queue. Writes of blocks are always in change order. So all writes are contiguous! So updates can be aggressive! The center of the disk contains a hash that maps from logical block addresses to addresses in the queue. Transaction queue is periodically "cleaned up" by clearing out unused entries and compacting. An entry is "unused" when it is no longer referred to by inodes/directories. Inodes and directories refer to positions in the transaction queue itself, and not that of an underlying filesystem.

4 RAID_and_Flash Page 4 4:43 PM How are blocks reclaimed? A block is in use if it is part of any file that persists over the past month. A block is unused if it is part of a version of a file that was replaced more than one month ago. Result: internal fragmentation; But as well, we update the hash. Goal: have a large number of contiguous blocks to use in the transaction queue.

5 4:39 PM RAID_and_Flash Page 5

6 RAID_and_Flash Page 6 Access Tuesday, December 08, :04 PM Access from multiple machines Our netapp talks two protocols for file access NFS (sun): network file system CIFS (Microsoft): common internet file system Can "mount" a filesystem from a remotely located machine. A local driver talks to a remote server. One instance of client-server architecture. Did you know: Your home directory on linux is your My Documents on windoze.

7 RAID_and_Flash Page 7 So far, Tuesday, December 07, :43 AM So far, we've dealt with the issue of robustness against power failures. But disk failures are also a problem. Solution: RAID. redundant array of inexpensive disks

8 RAID Thursday, December 02, :04 PM Raid: redundant arrays of inexpensive disks Idea: access time is slow because data is only on one disk. Speed up access times by making multiple disks emulate one disk. Treat a group of devices as one device. Build a regular filesystem on top. Raid levels: (Patterson's characterization) *0: no raid/striping: "stripe" files onto multiple disks with no redundancy. *1: mirroring: two disks, same content Twice as fast on read. Must do every write twice. 2,3: parallel access: arms track in parallel 2: words with error-correcting code. 3: parity bit (faster to compute) 4: independent access: Each disk independent. Data striping: block groups span disks. Parity disks check for errors. *5: parity strips distributed on disks. single error correction! Keys to raid: redundant data decreases access time (but not write time) single-disk failures don't result in I/O failures. (use parity disks to recover data) Virtual disk still works, but reads slow down! RAID_and_Flash Page 8

9 RAID 1: 4:55 PM RAID_and_Flash Page 9

10 RAID 5 Tuesday, December 07, :26 PM RAID_and_Flash Page 10

11 Picture of RAID Thursday, December 02, :04 PM Raid 0: no redundancy Raid 1: can remove a mirror, and disk remains ok. Raid 5: 1/5 of all disks can fail, and array keeps working. This is the basis of large file servers. Basic idea: fault-tolerant system; gives us time to replace a bad disk before system fails. The basic idea: reduce the probability of system failure by 1/5. RAID_and_Flash Page 11

12 RAID_and_Flash Page 12 Which RAID 5:00 PM RAID 1 for speed (but 1/2 the disk) RAID 5 for reliability and yield (but slower)

13 RAID_and_Flash Page 13 Special needs Tuesday, December 07, :49 AM So far, we've considered filesystems on devices that can be read and written with little concern for wear. Not all devices are like that. Two problems: How to construct a filesystem on a device in which writes degrade the device (flash). How to construct a filesystem on a device in which one can only write once (CDROM, DVD), or where writes must be orchestrated with erases.

14 RAID_and_Flash Page 14 Dealing with the physical Thursday, December 03, :32 PM Dealing with physical limits The ext2 format is optimized for physical disk: no limit on rewriting superblock re-written for every block allocation Flash is a very different medium than physical disk limited number of rewrites before failure (1000/block) how does one deal with the super-block in this case? A very strange answer: add a logical-to-physical layer between the filesystem and the flash! Flash drive maintains a mapping between logical flash blocks and physical flash blocks. Flash drive attempts to write new data into a new physical block, then remaps it! Thus we optimize the life of the flash! This is called "load leveling". Q: where is the logical to physical mapping kept? A: a very complex scheme ensues, partly on disk, partly in memory.

15 Dealing with Flash Memory 12:50 PM Dealing with Flash Memory Only truly portable file system is FAT. Most flash drives use it. But it was built for disks. Q: How do we cope with the flash rewrite problem? A: block virtualization. Basic strategy is as follows: Virtualize the raw device. Underneath the file system. Result is a raw filesystem that is robust against multiple writes to the same block. How: we keep moving the (physical) block! RAID_and_Flash Page 15

16 RAID_and_Flash Page 16 Properties of flash memory 5:58 PM All flash memory: Blocks start out as all 1's. Can program a bit by changing it to a 0. Bits are written in sectors. Can erase a block by changing it back to all 1's. Typical parameters: 1 (writable) sector = 4 KB. 1 (erasable) block = 128 KB.

17 RAID_and_Flash Page 17 Two kinds of flash 5:57 PM Two kinds of flash: NOR flash (legacy): Each bit can be individually cleared once per erase cycle. Each block can be erased (set to 1's) in entirety. NAND flash (modern): Must write a sector. Sector writes (4 KB) can only make 1->0 transitions; (0->1 transitions are ignored). Thus one can write via a masking strategy: write 1 to preserve existing sector contents. Limited number of sector writes per erase cycle. (We then have to copy the block and erase the original) Block erases (128 KB) can make the whole block 1's.

18 RAID_and_Flash Page 18 Basic flash writing strategies 5:56 PM Basic flash writing strategies: Descriptor arrays: if something has to change, record the changes in contiguous sectors. Whiteout: set a descriptor to all 0's: tells controller to use the next descriptor in the array. Backing stores: keep two copies of a thing, and write to the second one if the first one has been written to already. Wear-leveling: virtualize all addresses of sectors on the flash; when rewriting, move the sector and change the mapping.

19 RAID_and_Flash Page 19 Page descriptors 5:55 PM The page descriptor: contiguous to the physical page Virtual page number that this physical page represents (inverse mapping). Three status bits in the page descriptor: free/used: 1 if free, 0 if used. pre-valid/valid: 1 if pre-valid, 0 if valid. valid/obsolete: 1 if valid, 0 if obsolete. Three states of a page: free valid obsolete state unused allocated valid obsolete

20 RAID_and_Flash Page 20 Writing a page 5:55 PM Q: Why this much detail? A: Atomicity of block writes. Writing a page: Find a blank page (virtual page number is all 1's) Mark it allocated. Write contents into it. Write logical page number into it. Mark it valid. Enter it into the page table (logical to physical): a memory object. Mark old version (if any) obsolete.

21 RAID_and_Flash Page 21 The mapping table 5:54 PM The mapping table A memory object. Initialized when flash drive is inserted. Contains virtual-to-physical page mapping. Updated as pages change. Older flash drives were small=> initialize all virtual blocks during insertion. Newer flash drives are large => need some way to speed up that access.

22 RAID_and_Flash Page 22 FTL 5:53 PM Problem: virtualization takes time to initialize. Solution: an on-flash image of block mapping. Infrequently modified. Using a white-out strategy to invalidate old maps. One solution: Flash Translation Layer (FTL) FTL Two-layer mapping Virtual block # in filesystem (via first mapping, to) Logical erase unit #, block # (via second mapping, to) Physical erase unit #, block #

23 RAID_and_Flash Page 23 Backup maps 5:53 PM Backup maps Always keep two copies of each mapping table. When changing a map, First see if the backup map location is all 1's If it is, Clear the main map entry to all 0's Put the new location into the backup map entry. Else rewrite the main map. Moral of story: the structure of manipulation for a filesystem is very much dependent on the limitations of the media.

24 RAID_and_Flash Page 24 Backup stores 5:16 PM Minimize erases, extend life.

25 FTL Structure 5:25 PM RAID_and_Flash Page 25

26 RAID_and_Flash Page 26 Some cosmic considerations Thursday, December 03, :42 PM Some cosmic considerations When a file is deleted, its blocks are not erased. All that happens is that its directory entry is X'd out. It is possible to reconstruct files from what is left on disk A colleague of mine based his Ph.D. thesis on recovering personal information from discarded hard drives for sale on ebay! Then he called each person and told them he had, e.g., their credit card numbers!

27 RAID_and_Flash Page 27 Rethinking filesystems Tuesday, December 07, :54 AM Rethinking filesystems The cloud has forced us to think of files in a radically different way. Inside google, There are no "files". There are distributed objects with multiple instances. To store an instance of an object, one throws it into the cloud. To reconstruct something like a file, one makes a query into the cloud to return matching instances. In other words, files are a convenient illusion, maintained for backward compatibility.

28 RAID_and_Flash Page 28 Example: our mail versus google mail Tuesday, December 07, :58 AM Our mail: INBOX is a file. Consisting of a concatenation of mail messages. Where deleting one requires rewriting the whole file. Google mail: Mail is a distributed object. Messages are distributed object instances. Your inbox is a query. "All mail" is the universal query. There is no "file" where all of your messages are stored. They're spread among over 10,000 machines! More about this in "Cloud Computing"!