Tolera'ng File System Mistakes with EnvyFS

Transcription

1 Tolera'ng File System Mistakes with EnvyFS Lakshmi N. Bairavasundaram NetApp, Inc. Swaminathan Sundararaman Andrea C. Arpaci Dusseau Remzi H. Arpaci Dusseau University of Wisconsin Madison

2 File Systems in Today s World Modern file systems are complex Tens of thousands of lines of code (e.g., XFS 45K LOC) Storage stack is also gevng deeper Hypervisor, network, logical volume manager Need to handle a gamut of failures Memory alloca'on, disk faults, bit flips, system crashes Preserve integrity of its meta data and user data 2

3 File System Bugs Bug reports for Linux 2.6 series from Bugzilla ext3: 64, JFS: 17, ReiserFS: 38 Some are FS corrup'on causing permanent data loss FS bugs broadly classified into two categories fail stop : System immediately crashes Solu'ons: Nooks [Swi/ 04], CuriOS [David08] fail silent : Accidentally corrupt on disk state Many such bugs uncovered [Prabhakaran05, Gunawi08, Yang04, Yang06b] 3

4 Bugs are inevitable in file systems Challenge: how to cope with them? 4

5 N Version File Systems Based on N version programming [Avizienis77] NFS servers [Rodrigues01], databases [Vandiver07], security [Cox06] EnvyFS: Simple solware layer Store data in N child file systems Opera'ons performed on all children Rely on a simple so-ware layer Challenge: reducing overheads while retaining reliability SubSIST: Novel Single Instance Store Applica'on EnvyFS layer Child 1 Child 2 Disk SIS layer driver Disk Child N 5

6 Results Robustness Tradi'onal file systems handle few corrup'ons (< 4%) EnvyFS 3 tolerates 98.9% of single file system mistakes Performance Desktop workloads: EnvyFS 3 has comparable performance I/O intensive workloads: Normal mode: EnvyFS 3 + SubSIST acceptable performance Under memory pressure: EnvyFS 3 + SubSIST large overheads Poten'al as a debugging tool for FS developers Pinpoint the source of fail silent bug in ext3 6

7 Outline Introduc'on Building reliable file systems Reducing overheads with SubSIST Evalua'on Conclusion 7

8 N Version Systems Development process: 1. Producing the specifica'on of solware 2. Implemen'ng N versions of the solware 3. Crea'ng N version layer Executes different versions Determines the consensus result 8

9 1. Producing Specifica'on Our own specifica'on? Imprac'cal: Requires wide scale changes to file systems Specifica'ons take years to get accepted Can we leverage exis'ng specifica'on? Yes, can leverage VFS, but there are some issues VFS not precise for N versioning purpose Needs to handle cases where specifica'on is not precise e.g., Ordering directory entries, inode number alloca'on 9

10 Imprecise VFS Specifica'on Ordering directory entries Issue: No specified return order Can t blindly compare entries Solu'on: Read all entries from a directory (dir: test in our case) from all FSes Match entries from FSes Return majority results File 1 File 2 File 3 Dir: test Readdir: test FS X EnvyFS layer File 1 File 2 File 3 File 2 File 3 File 1 Dir: test File 1 File 2 File 3 Dir: test FS Y File 1 File 2 No File Entries 3 File 3 File 1 File 2 Dir: test FS Z 10

11 Imprecise VFS Specifica'on (cont) Inode number alloca'on Inode numbers returned through system calls Each child file system issues different inode numbers Possible solu'on: Force file systems to use same algorithm? Our solu'on: Issue inode numbers at EnvyFS layer Stat: File 1 File 1?? 15 EnvyFS layer File 1 10 File 1 36 File 1 65 Virt # FS 1 FS 2 FS File 1 10 File 2 15 File 3 16 FS X File 2 04 File 3 44 File 1 36 FS Y File 3 99 File 1 65 File 2 43 Dir: test Dir: test Dir: test Inode Numbers FS Z Inode Mapping Table Inode Mapping Table not persistently stored 11

12 2. Implemen'ng N versions of FS Painful process High cost of development, long 'me delays Lucky! Hard work already done for us 30 different disk based file systems in Linux 2.6 Which file systems to use? ext3, JFS, ReiserFS in a three version FS Others should work without modifica'ons 12

13 3. Crea'ng N Version Layer N Version layer (EnvyFS) Inserted beneath VFS Simple design to avoid bugs Example: Reading a file Allocate N data buffers Read data block from the disk Compare: data, return code, file posi'on Return: data, return code Issues: Allocate memory for each read opera'on Extra copy from allocated buffer to applica'on Comparison overheads Read (file, 1 block) Applica'on VFS layer Read (file, 1 block) err, D EnvyFS Inode Mapping Table Layer Wrappers Comparators err = Read ( ) D err = Read ( ) D err = Read ( ) D F F F pos: x pos: x pos: x ext3 ReiserFS JFS D D D Disk err, D 13

14 Reading a File in EnvyFS Solu'on: Same applica'on buffer for all FS TCP like checksums for data comparison Compare: checksums, return code, file posi'on Read data un'l majority FS 1 # FS 2 # FS N # Checksums Read (file, 1 block) Applica'on VFS layer Read (file, 1 block) err, D EnvyFS Inode Mapping Table Layer Wrappers Comparators err = Read ( ) D err = Read ( ) D err = F F F pos: x ext3 ReiserFS pos: x JFS D D D Disk err, D Read ( ) D pos: x 14

15 Outline Introduc'on Building reliable file systems Reducing overheads with SubSIST Evalua'on Conclusion 15

16 Case for Single Instance Storage (SIS) Ideal: One disk per FS Prac'cal: One disk for all FS Overheads Effec've storage space: 1/N N 'mes more I/O (Read/write) Challenge: Maintain diversity while minimizing overheads Applica'on VFS layer 1 EnvyFS layer 1 2 N FS 1 FS N Disk Req. Queue FS N Disk 1 Disk 2 Disk N Part 1 Part Disk 2 Part N 16

17 SubSIST: Single Instance Store Variant of an Single Instance Store Selec'vely merges data blocks Block addressable SIS Exports virtual disks to FSes Manages mapping, free space info. Not persistently stored on disk EnvyFS writes through N file systems N data blocks merged to 1 data block Content hashes not stored persistently Meta data blocks not merged Inter FS blocks and not intra FS D Applica'on VFS layer D EnvyFS layer FS 1 FS 2 D FS N D M D M D M D Vdisk 1 Vdisk 2 Vdisk N CHash Layer Read Cache Free Space Management SubSIST 17 Disk

18 Handling Data Block Corrup'ons? Corrup'on to data in a single FS Due to bugs, bit flips, storage stack Corrupt data blocks not merged All other N 1 data blocks merged Corrupt data block fixed at next read Corrup'on to data block inside disk Applica'on VFS layer EnvyFS layer D D FS 1 FS 2 D FS N D D D Single copy of data Different code paths Different on disk structures Vdisk 1 Vdisk 2 Vdisk N D CHash Layer Read Cache Free Space Management SubSIST D Disk 18

19 Outline Introduc'on Building reliable file systems Reducing overheads with SubSIST EvaluaHon Reliability Performance Conclusion 19

20 Reliability Evalua'on: Fault Injec'on Robustness of EnvyFS in recovering from a child file system s mistake? VFS EnvyFS layer Corrup'on: bugs in FS / storage stack Types of disk blocks superblock, inode, block bitmap, file data, Perform different file ops Type aware fault injechon [Prabhakaran05] mount, stat, creat, unlink, read, Report user visible results All results are applicable with SubSIST except corruphon to data blocks ext 3 Pseudo Device Driver B B B JFS Block Driver B B Disk B B B ReiserFS 20

21 ext3 E INODE DIR BMAP IMAP INDIRECT DATA SUPER path traversal SET 1 (stat, ) SET 2 (chmod) read readlink getdirentries creat link mkdir rename symlink write truncate rmdir unlink mount SET 3 (fsync) umount Result Matrix Normal Data loss Cannot mount Ops fail Data corrupt Crash Read only JSUPER GDESC Depends e 21 N/A

22 ext3 E INODE DIR BMAP IMAP INDIRECT DATA SUPER JSUPER GDESC path traversal SET 1 (stat, ) SET 2 (chmod) read readlink getdirentries creat link mkdir rename symlink write truncate rmdir unlink mount SET 3 (fsync) umount Ext3 stores many superblock copies; but, does not handle superblock corrup'on Data loss Cannot mount N/A 22

23 ext3 E path traversal SET 1 (stat, ) SET 2 (chmod) read readlink getdirentries creat link mkdir rename symlink write truncate rmdir unlink mount SET 3 (fsync) umount INODE DIR BMAP IMAP INDIRECT DATA SUPER JSUPER GDESC In addi'on to opera'ons failing, inode corrup'on leads to data loss Unlink: system crash during unmount Data loss Cannot mount Ops fail Crash N/A 23

24 ext3 E INODE DIR BMAP IMAP INDIRECT DATA SUPER path traversal SET 1 (stat, ) SET 2 (chmod) read readlink getdirentries creat link mkdir rename symlink write truncate rmdir unlink mount SET 3 (fsync) umount Normal Data loss Cannot mount Ops fail Data corrupt Crash Read only JSUPER GDESC Depends e 24 N/A

25 EnvyFS 3 EnvyFS E J R INODE DIR BMAP IMAP INDIRECT DATA SUPER JSUPER path traversal SET 1 (stat, ) SET 2 (chmod) read readlink getdirentries creat link mkdir rename symlink write truncate rmdir unlink mount SET 3 (fsync) umount Kernel panic in ext3 Normal N/A GDESC EnvyFS 3 works in every scenario 25

26 Poten'al for Bug Isola'on ext3 EnvyFS 3 Time Unlink on corrupt inode: ext3_lookup (bug) ext3_unlink Time Unlink on corrupt inode: ext3_lookup (bug) ext3 inode does not match others Further ops not issued Unmount (panic) In typical use, a problem is no'ced only on panic In EnvyFS 3, a problem is no'ced the first 'me child file system returns wrong results 26

27 JFS J INODE DIR BMAP IMAP INTERNAL DATA SUPER JSUPER JDATA AGGR INODE IMAPDESC IMAPCNTL path traversal SET 1 SET 2 read readlink getdirentries creat link mkdir rename symlink write truncate rmdir unlink mount SET 3 umount Normal Data loss Cannot mount Ops fail Data corrupt Crash Read only Depends 27 N/A a

28 EnvyFS 3 EnvyFS E J R path traversal SET 1 SET 2 read readlink getdirentries creat link mkdir rename symlink write truncate rmdir unlink mount SET 3 umount INODE DIR BMAP IMAP INTERNAL DATA SUPER JSUPER JDATA AGGR INODE IMAPDESC IMAPCNTL Normal Crash N/A Kernel panic in EnvyFS 3 28

29 OpenSSH Benchmark Performance Evalua'on Elapsed Time (in Seconds) 30 3 % overhead Experimental setup AMD Opteron 2.2 GHz Processor 2GB RAM 80 GB Hitachi Deskstar 7200 rpm SATA disk Linux GB disk par''on for each file system ext3 JFS ReiserFS EnvyFS EnvyFS+SIS CPU Intensive OpenSSH 4.5 Copy, untar and make Performance of EnvyFS File Systems 3 is comparable to a single file system 29

30 Postmark Benchmark Elapsed Time (in Seconds) 900 I/O Intensive Mimics busy mail server workload Transac'on: creates, deletes, reads, appends, Postmark Configura'on EnvyFS 3 : 3.3x + SubSIST: 32% 2500 files File size: 4Kb 40Kb EnvyFS 3 : 8x + SubSIST: 4x No of transac'ons: K 34 and 100K Postmark 10K Postmark 100K Postmark 100K* ext3 JFS ReiserFS EnvyFS EnvyFS+SIS EnvyFS 3 : 1.7x + SubSIST: 11.5%

31 Summary of Results Robustness Tradi'onal file systems vulnerable to corrup'ons EnvyFS 3 tolerates almost all mistakes in one FS Performance Desktop workloads: EnvyFS 3 has comparable performance I/O intensive workloads: Regular Opera'ons: EnvyFS 3 + SubSIST acceptable performance Memory pressure: EnvyFS 3 + SubSIST has large overhead 31

32 Outline Introduc'on Building reliable file systems Reducing overheads with SubSIST Evalua'on Conclusion 32

33 Conclusion Bugs/mistakes are inevitable in any solware Must cope, not just hope to avoid EnvyFS: N version approach to tolera'ng FS bugs Built using exis'ng specifica'on and file systems SubSIST: single instance store Decreases overheads while retaining reliability 33

34 Thank You! Advanced Systems Lab (ADSL) University of Wisconsin Madison hxp:// 34

35 Future Work Debugging tool for developers Run older and newer version of file systems Compare results with older version File system repair Simple repair: copy data from other file system Complex repair: recreate en're file system tree How to do micro repair? 35

36 ext3 E path traversal SET 1 (stat, ) SET 2 (chmod) read readlink getdirentries creat link mkdir rename symlink write truncate rmdir unlink mount SET 3 (fsync) umount INODE DIR BMAP IMAP INDIRECT DATA SUPER JSUPER GDESC Data loss Read only Ext3 detects corrup'on for rmdir, unlink N/A creat, mkdir, symlink cause ext3 to reuse an inode, resul'ng in data loss 36