File Systems. CS 450: Operating Systems Sean Wallace Computer Science. Science

Transcription

1 File Systems Science Computer Science CS 450: Operating Systems Sean Wallace

2 What is a file? Some logical collection of data Format/interpretation is (typically) of little concern to OS 2

3 A filesystem is a collection of files Supports a managed namespace of data Maps & managed file metadata (automatically & explicitly) 3

4 Different (overlapping) classes of filesystems: Traditional : hierarchy of on-disk data Database-backed storage (rich metadata) Distributed storage (e.g., for MapReduce) Namespace for everything (e.g., Plan 9) 4

5 We will limit most of our discussion to traditional filesystems and regular files Modern filesystem implementations are almost all hybrids (of the classes mentioned) 5

6 Agenda Filesystem goals & requirements Filesystem API Filesystem implementation Filesystem robustness Case study: xv6 (Unix) 6

7 Filesystem Goals 7

8 1. File CRUD API: Create Read Update Delete 8

9 2. Protection & Security Access control Ownership & permissions Encryption 9

10 3. Robustness Crashes should not affect filesystem validity Also, try to mitigate data loss (e.g., uncommitted changes) 10

11 4. Flexibility & Scaleability Different ways of accessing data E.g., stream vs. memory mapped Support exponential growth in drive capacity 11

12 5. Decoupling of OS & FS FS not tied to OS (or vice versa) Multiple FS s on a single OS (at once) 12

13 6. Device agnosticism FS shouldn t assume/optimize for a certain type of storage device E.g., HDD vs. SSD vs. RAM disk 13

14 7. Good throughput & responsiveness Throughput (in MB/s or IOPS) Responsiveness request latency 14

15 8. Good disk utilization Often least important! Usually preferable to trade spatial inefficiency for robustness & speed 15

16 Filesystem API 16

17 File attributes (file as an ADT): Name/path (convenient for humans) Identifier (unique, system-wide) Type (e.g., executable) Protection & access control Creator/owner, size, timestamp Possibly much more! (e.g., log, tags, ) 17

18 Basic operations: Create at some location, with specified mode(s), possibly truncating Read Update: write content, metadata; adjust position in file (need to track) Delete = remove from filesystem 18

19 Typical data structures: File descriptor Open file structure Namespace structure (e.g., directory) Access control metadata 19

20 A. File descriptor Process-held pointer to an open file Used to identify file to OS/FS for user initiated file operations Enables OS encapsulation of file data 20

21 B. Open file structure Essentials: position in file & count of referring processes (via FDs) May permit multiple positions Flush in-memory struct if count == 0 Also, per open-file access mode(s) 21

22 C. Namespace structure (e.g., directory) Tracks position of data in filesystem May function as all-purpose OS namespace (E.g., even for off-disk data) E.g., full path from FS root : /home/sean/.zshrc 22

23 D. Access-control metadata E.g., rwx bits in Unix Separate bits for owner/group/other Or more granular ACLs E.g., read/write/append/readacl/writeacl/ delete/ect., based on user 23

24 e.g., Unix file syscalls int open (const char *path, int oflag,...); int creat (const char *path, mode_t mode); int close (int fd); int link (char *oldpath, char *newpath); int unlink (char *path); int chdir (char *dirpath); ssize_t read (int fd, void *buf, size_t nbytes); ssize_t write (int fd, void *buf, size_t nbytes); off_t lseek (int fd, off_t offset, int whence); int int fchmod (int fd, mode_t mode); fstat (int fd, struct stat *buf); 24

25 struct stat { dev_t st_dev; /* ID of device containing file */ ino_t st_ino; /* File serial number */ mode_t st_mode; /* Mode of file (see below) */ nlink_t st_nlink; /* Number of hard links */ uid_t st_uid; /* User ID of the file */ gid_t st_gid; /* Group ID of the file */ dev_t st_rdev; /* Device ID */ struct timespec st_atimespec; /* time of last access */ struct timespec st_mtimespec; /* time of last data modification */ struct timespec st_ctimespec; /* time of last status change */ time_t st_atime; /* Time of last access */ long st_atimensec; /* nsec of last access */ time_t st_mtime; /* Last data modification time */ long st_mtimensec; /* last data modification nsec */ time_t st_ctime; /* Time of last status change */ long st_ctimensec; /* nsec of last status change */ off_t st_size; /* file size, in bytes */ blkcnt_t st_blocks; /* blocks allocated for file */ blksize_t st_blksize; /* optimal blocksize for I/O */ uint32_t st_flags; /* user defined flags for file */ uint32_t st_gen; /* file generation number */ }; 25

26 Unix convention of mapping fixed file descriptor values to standard in/out is widely copied allows for I/O redirection 26

27 int main(int argc, const char * argv[]) { int fd = open("foo.txt", O_CREAT O_TRUNC O_RDWR, 0644); dup2(fd, 1); /* set fd 1 (stdout) to be "foo.txt" */ printf("arg: %s\n", argv[1]); } 27

28 int main(int argc, const char * argv[]) { int fd = open("foo.txt", O_CREAT O_TRUNC O_RDWR, 0644); dup2(fd, 1); /* set fd 1 (stdout) to be "foo.txt" */ printf("arg: %s\n", argv[1]); } } (by default: terminal) OFD empty file file descriptors (process-local) 28

29 int main(int argc, const char * argv[]) { int fd = open("foo.txt", O_CREAT O_TRUNC O_RDWR, 0644); dup2(fd, 1); /* set fd 1 (stdout) to be "foo.txt" */ printf("arg: %s\n", argv[1]); } (output) OFD empty file 29

30 int main(int argc, const char * argv[]) { int fd = open("foo.txt", O_CREAT O_TRUNC O_RDWR, 0644); dup2(fd, 1); /* set fd 1 (stdout) to be "foo.txt" */ printf("arg: %s\n", argv[1]); } $./a.out Hello! $ ls -lh -rwxr-xr-x 1 sean staff -rw-r--r-- 1 sean staff $ cat foo.txt Arg: Hello! 18K Nov 1 16:48 a.out 12B Nov 1 16:49 foo.txt 30

31 int main(int argc, const char * argv[]) { int fd = open("foo.txt", O_CREAT O_TRUNC O_RDWR, 0644); if (fork() == 0) { dup2(fd, 1); execlp("echo", "echo", "Hello!", NULL); } close(fd); } $./a.out $ cat foo.txt Hello! 31

32 Filesystem Implementation 32

33 system call interface (API) OS-FS interface FS implementation FS-Device interface device drivers devices (HDDs, SSDs) (in reality, not so tidy!) 33

34 1. Mass storage (disk) systems 2. Volumes and Partitions 3. Names and Paths 4. File space allocation 5. Free space tracking 34

35 Names and Paths 35

36 Requirement: a fully qualified filename uniquely identifies a set of data blocks on disk Big filenames & flat namespace work, but are hard to reason about Prefer hierarchical namespaces Fully qualified filename = name + path 36

37 /Users/sean/cs450/09_File_Systems.pdf Absolute path From /Users/sean, relative path is./cs450/09_file_systems.pdf (. = current directory,.. = previous directory) 37

38 One or more root name spaces Typically can mount additional filesystems onto global namespace Support for multiple filesystems 38

39 E.g., Windows: C:\foo.txt vs. D:\foo.txt E.g., Unix: /home/sean/foo.txt vs. /mnt/cdrom/foo.txt 39

40 What s in a name? Path file must be unique File path?? Consider aliases/shortcuts: /bin/prog /home/sean/foo_prog Different paths may refer to the same file 40

41 Directories provide linking structures Directory maps name file identifier File ID is implementation specific Directories are also files (recursive definition) 41

42 Link types: Hard link: different names (possibly in different directories) map to the same file Remove all hard links = removing file Soft/symbolic link: file containing the name of another file Independent of whether file exists 42

43 Demo: Linking 43

44 Note: soft links are possible across partitions/volumes, but hard links aren t (usually) 44

45 To find a file: Just need location of root directory Search recursively for path components Tricker with multiple filesystems Each logical volume of data contains its own high level metadata 45

46 File Space Allocation 46

47 Mapping problem: for a given file (by path or ID), find (ordered) list of data blocks 47

48 Considerations: Good disk utilization Efficiency (with respect to HDD seeks) Random access Scaleability 48

49 Basic strategies: Contiguous Linked (decentralized) Centralized Linked Indexed 49

50 Directory may double as metadata store, too (e.g., mode, owner) Contiguous allocation 50

51 Pros: Ideal for sequential HDD reads: reduce seeks fast! Random access is trivial Cons: Clear disadvantage: fragmentation Affects utilization, placement ( all or nothing ), resizing 51

52 Not used on its own, but contiguous extents are used in most modern file systems Multiple of block size variable size Reserve in advance during allocation Balance fragmentation & efficiency 52

53 block metadata block data Linked allocation (decentralized) 53

54 Pros: Good utilization + allows resizing Cons: Fragmentation lot of seeks = slow! No random access Hard to protect file metadata! 54

55 stored as per-volume metadata! Linked allocation (centralized) 55

56 Pros: Allows for random access Used with extents, can limit fragmentation Disadvantages: Centralized file metadata (robustness?) Overhead incurred by central FAT Hard limit on volume size! 56

57 Also, unless directories maintain metadata, central structure has limited space E.g., where to put mode, ownership, ACL, timestamp, etc.? 57

58 E.g., MS-DOS file-allocation table (FAT) - FAT12, FAT16, FAT32 variants (based on sizes of FAT entry) 58

59 Some MS FAT terminology: Sector physical disk block (512 bytes) Cluster fixed-size extent of sectors (512 bytes - 128KB) 59

60 Some limits: FAT12: 4K clusters x 512 = 2MB FAT16: 64K clusters x 8K = 512MB FAT32: only 28-bits of FAT entry useable, 268M clusters x 8K = 2TB 60

61 FAT12 requirements : 3 sectors on each copy of FAT for every 1,024 clusters FAT16 requirements : 1 sector on each copy of FAT for every 256 clusters FAT32 requirements : 1 sector on each copy of FAT for every 128 clusters FAT12 range : 1 to 4,084 clusters : 1 to 12 sectors per copy of FAT FAT16 range : 4,085 to 65,524 clusters : 16 to 256 sectors per copy of FAT FAT32 range : 65,525 to 268,435,444 clusters : 512 to 2,097,152 sectors per copy of FAT FAT12 minimum : 1 sector per cluster 1 clusters = 512 bytes (0.5 KiB) FAT16 minimum : 1 sector per cluster 4,085 clusters = 2,091,520 bytes (2,042.5 KiB) FAT32 minimum : 1 sector per cluster 65,525 clusters = 33,548,800 bytes (32,762.5 KiB) FAT12 maximum : 64 sectors per cluster 4,084 clusters = 133,824,512 bytes ( 127 MiB) [FAT12 maximum : 128 sectors per cluster 4,084 clusters = 267,694,024 bytes ( 255 MiB)] FAT16 maximum : 64 sectors per cluster 65,524 clusters = 2,147,090,432 bytes ( 2,047 MiB) [FAT16 maximum : 128 sectors per cluster 65,524 clusters = 4,294,180,864 bytes ( 4,095 MiB)] FAT32 maximum : 8 sectors per cluster 268,435,444 clusters = 1,099,511,578,624 bytes ( 1,024 GiB) FAT32 maximum : 16 sectors per cluster 268,173,557 clusters = 2,196,877,778,944 bytes ( 2,046 GiB) [FAT32 maximum : 32 sectors per cluster 134,152,181 clusters = 2,197,949,333,504 bytes ( 2,047 GiB)] [FAT32 maximum : 64 sectors per cluster 67,092,469 clusters = 2,198,486,024,192 bytes ( 2,047 GiB)] [FAT32 maximum : 128 sectors per cluster 33,550,325 clusters = 2,198,754,099,200 bytes ( 2,047 GiB)] 61

62 File size limit theoretically = disk limit, but directory implementation constrains file sizes to 4GB in FAT32 62

63 Indexed allocation 63

64 Files identified by index block number a.k.a. inode number Directory is an inode registry Index of file name inode # Each entry is a hard link Directories are files, too, so they also have inodes 64

65 Pros: Allows for random access Natural metadata store Used with extents, can limit fragmentation Disadvantages: Overhead incurred by index nodes Limit on file size (# block references) 65

66 E.g., Unix File System (UFS), and all its descendants 66

67 "super" block inodes data blocks 67

68 Superblock contains filesystem metadata Size of logical blocks Location & number of inodes inodes section contains per-file metadata # inodes = max # files 68

69 inode block file metadata (e.g., type, ownership, access time, #links) Note: indirect blocks are stored in data area of volume! direct pointers data block single indirect pointer double indirect pointer triple indirect pointer data block direct pointers data block data block single indirect pointers direct pointers direct pointers data block data block 69

70 E.g., UFS properties: 32-bit i-node pointers 4KB i-node/data blocks 8 direct, 2 single indirect, 1 double indirect pointer per i-node Max disk / file size? 70

71 32-bit i-node pointers 4KB i-node/data blocks 8 direct, 2 single indirect, 1 double indirect pointer per i-node Max disk size = 4G x 4KB = 16TB 71

72 32-bit i-node pointers 4KB i-node/data blocks 8 direct, 2 single indirect, 1 double indirect pointer per i-node Directly addressed: 8 x 4KB = 32KB 72

73 32-bit i-node pointers 4KB i-node/data blocks 8 direct, 2 single indirect, 1 double indirect pointer per i-node Each indirect block can hold 4KB / 4 bytes = 1K pointers 73

74 32-bit i-node pointers 4KB i-node/data blocks 8 direct, 2 single indirect, 1 double indirect pointer per i-node Single indirect pointer = 1K x 4KB = 4MB Two single indirect = 8MB 74

75 32-bit i-node pointers 4KB i-node/data blocks 8 direct, 2 single indirect, 1 double indirect pointer per i-node Double indirect pointer = 1K x 1K x 4KB = 4GB 75

76 32-bit i-node pointers 4KB i-node/data blocks 8 direct, 2 single indirect, 1 double indirect pointer per i-node Max file size = 32KB + 8MB + 4GB Variable # block requests per data request (depending on location in file!) 76

77 How to keep filesystem decoupled from OS? 77

78 Need a middle layer a mediator between filesystem specific constructs & abstract OS file-related operation 78

79 VFS: Virtual File System layer Unix centric API between syscall API (open/ close/read/write) & filesystems Every filesystem must implement generic analogues of: inode, file, superblock, dentry 79

80 Each filesystem object has a table of function pointers (e.g., open/close/read/ write) that are used by VFS to map system calls 80

81 Free Space Tracking 81

82 1. Linked free blocks 2. Free space bitmap 3. General disk-based data structures 82

83 1. Linked free blocks No overhead But expensive to traverse! Can optimize as a skip list Useful for extent search 83

84 2. Free space bitmap n-1 bit[i] = ( 0, block[i] occupied 1, block[i] free Simple to maintain & fast! Use machine instruction to locate first 1 84

85 Block size = 2 12 bytes (4KB) Disk size = 1TB = 2 40 bytes Free space bitmap = 2 38 bits (32MB) Small enough to keep in memory But beware synchronization issues 85

86 Optimization: Break bitmap into subsets & build index of # free blocks subset Speed up extent search Can lock subsets separately 86

87 3. General disk-based data structures e.g., B+ tree Balanced search tree with very large branching factor (# pointers per block) worth it? 87

88 Mass storage systems 88

89 Magnetic disks (HDDs) provide bulk of secondary storage - rotating magnetic platters 89

90 Motor & belt driven 90

91 Smaller & denser, but still mechanical 91

92 ?! 92

93 We will focus on traditional HDDs for now Still a valuable discussion HDDs will remain the mass storage device of choice for some time to come NEW for S2016, we will talk about SSDs. 93

94 Idealized addressing: Cylinder, Head, Sector 94

95 A sector, historically, maps to a fixed 512-byte block of disk space Minimum disk transfer size Recently, drives are moving to 4K block sizes (but still supporting old mapping) 95

96 Disk access times = S + R + T S: seek time (head movement) R: rotational latency (depends on angular velocity usually constant for HDDs) T: transfer time (relatively small) + spin-up time (discount for long I/O) 96

97 Disk access times = S + R + T S: move to correct cylinder R: wait for sector to rotate under head T: move head across adjacent blocks 97

98 Some numbers: Seek time = 3ms - 15ms Typical RPM = 7200 (range of ) Rotational latency = 1/2 of period e.g., ms 98

99 99

100 By contrast, each channel of DDR memory has max theoretical throughput: 2133 MHz 8 bytes = 17,064 MB/s only ~100x more than disk throughput? 100

101 123 MB/s is sustained rate Unlike when dealing with random, fragmented data on disk 6 Gb/s (750 MB/s) is buffer to memory not indicative of HDD speed 101

102 HDDs are best leveraged by reading contiguous sectors i.e., w/o seeking 102

103 Idea: optimize order of block requests to minimize seeks (most expensive operation) Goals: Maximize throughput Minimize latency per response 103

104 Territory of disk head scheduler 104

105 Cylinder-head-sector is useful for discussion: Bigger difference in cylinders = larger head movement Note: heads move as single unit 105

106 But, CHS is unrealistic in modern drives: low density in outer cylinders A C B D 106

107 Modern drives use logical block addressing (LBA) Number of blocks starting from 0 (innermost) to outermost, then back in on reverse side Problem: no disk geometry info! Not so bad: LBAi, LBAi+1 are at most 1 cylinder apart 107

108 Disk head scheduling problem: Given requests B1, B2, from processes, what seek order to send to disk controller? 108

109 Analogs to scheduling approaches: First come, first served (FCFS) Shortest Seek Time First (SSTF) Nearest Block Number First (NBNF) 109

110 As before, SSTF can result in starvation or at best poor request latency! 110

111 How to alleviate starvation problem, and optimize wait time, responsiveness, etc.? 111

112 Elevator Algorithms 112

113 SCAN: Track from spindle edge of disk Only service requests in the current direction of travel Keep heading towards spindle/edge even if no requests in that direction 113

114 Variants of SCAN: C-SCAN: circular tracking F-SCAN: freeze request queue on direction change 114

115 LOOK: Reverse direction when no more requests Variants: C-LOOK, F-LOOK 115

116 Demo: UTSA disk-head simulator 116

117 Solid-State Storage 117

118 No mechanical or moving parts like HDDs. But not quite DRAM either Retains (non-volatile) data despite power loss. Based on a technology known as flash (NANDbased flash). All sorts of new terms: Flash page, flash block, wear out, etc 118

119 Flash Based Memory 119

120 vs. 120

121 Store one or more bits in a single transistor. The level of the transistor determines the number of bits: Single-level cell: 0 or 1 Multi-level cell: 00, 01, 10, and

122 So just jam a bunch of these together and you get a storage system? 122

123 If only it were that easy flash chips are organized into banks or planes consisting of a large number of cells. 123

124 A bank is comprised of blocks which are comprised of pages. 124

125 Banks are accessed in two different sized units: blocks and pages. Block: Page: Content: To write within a block, you have to erase the entire block! 125

126 Flash based operations: Read Erase Program 126

127 Read (a page): Get data at a specific location (i.e., page). Extremely fast! Because of the indexed nature of pages, random access is trivial. 127

128 Erase (a block): Before writing to a page, you must erase (set each bit to 1) the entire block. Therefore, you need to be sure that any data in that block is erasable. Very slow (relatively speaking)! 128

129 Program (a page): Since the erase command must come first and defaults to all 1 s, just change some to 0 s. Faster than erasing, but slower than reading. 129

130 A day in the life of a page: INVALID ERASED VALID 130

131 iiii Initial: pages in block are invalid(i) Erase() EEEE State of pages in block are set to erased(e) Program(0) VEEE Program page 0; state set to valid(v) Program(0) error Cannot re-program page after programming Program(1) VVEE Program page 1 Erase() EEEE Contents erased; all pages programmable 131

132 8-bit pages 4-page block Page 0 Page 1 Page 2 Page VALID VALID VALID VALID Write to page

133 8-bit pages 4-page block Page 0 Page 1 Page 2 Page ERASED ERASED ERASED ERASED Program page 0 with

134 8-bit pages 4-page block Page 0 Page 1 Page 2 Page VALID ERASED ERASED ERASED But what about these!? Collateral damage. 134

135 Need to move collateral data out of the way before erasing. Pros/cons? 135

136 Next:

137 Territory of the flash translation layer. 137

138 Goals: Maintain idea of logical blocks but turn them into low-level read, erase, and program commands on underlying physical blocks and pages. Deliver excellent performance. Reduce write amplification. Total traffic issued by FTL / total write traffic issued by client. 138

139 Host Interface Logic Memory Flash Flash Flash Flash Controller Flash Flash Flash Multiple flash chips means writing in parallel. 139

140 Approach 1 - Direct Mapped Reading is simple, to read logical page N, just read physical page N. Writing still needs to erase, though Read entire block that page N is contained in. Erase the entire block. Program the old pages as well as the new one. 140

141 Pros: It s simple Cons: Performance Have to read entire block, erase it, and then program it. This can be even slower than typical hard drives! Reliability Unnecessary wear to blocks. 141

142 Approach 2 - Log-Structured FTL When writing to block N, append to the next free spot in the currently-being-writting-to block. To read this block, device keeps a mapping table which stores the physical address of each logical block. 142

143 For example: 4KB sized write chunks. 16KB sized blocks each divided into four 4KB pages. Client issues the following operations: Write(100) with contents a1 Write(101) with contents a2 Write(2000) with contents b1 Write(2001) with contents b2 143

144 Write(100) (physical block 0) with contents a1 Block: Page: Content: State: i i i i i i i i i i i i 144

145 Write(100) (physical block 0) with contents a1 Block: Page: Content: State: E E E E i i i i i i i i Erase block 0 145

146 Write(100) (physical block 0) with contents a1 Block: Page: Content: a1 State: V E E E i i i i i i i i Program physical page 0 146

147 In order to read logical block 100, FTL records in an in-memory mapping table. Table: Memory Block: Page: Content: a1 Flash Chip State: V E E E i i i i i i i i 147

148 Now rest of write just populate block contents and record logical-to-physical mapping. Table: Memory Block: Page: Content: a1 Flash Chip State: V E E E i i i i i i i i 148

149 Now rest of write just populate block contents and record logical-to-physical mapping. Table: Memory Block: Page: Content: a1 a2 Flash Chip State: V V E E i i i i i i i i 149

150 Now rest of write just populate block contents and record logical-to-physical mapping. Table: Memory Block: Page: Content: a1 a2 b1 Flash Chip State: V V V E i i i i i i i i 150

151 Now rest of write just populate block contents and record logical-to-physical mapping. Table: Memory Block: Page: Content: a1 a2 b1 b2 Flash Chip State: V V V V i i i i i i i i 151

152 Pros Vastly improved performance. Greatly enhanced reliability (i.e., wear leveling) Cons: In memory table scales with size of device. Overwrites of logical blocks lead to garbage. 152

153 Writing to blocks 100 and 101 again with c1 and c2. Table: Memory Block: Page: Content: a1 a2 b1 b2 Flash Chip State: V V V V i i i i i i i i 153

154 Writing to blocks 100 and 101 again with c1 and c2. Table: Memory Block: Page: Content: a1 a2 b1 b2 c1 c2 Flash Chip State: V V V V V V E E i i i i Garbage 154

155 Need some form of garbage collection. Read in non-garbage pages from block. Write out those pages to the log. Reclaim the entire block. Therefore, need to keep track of alive and dead pages. Commonly know as the TRIM command. 155

156 What about that in memory table size? If we have a single entry for each 4KB page: With 1TB SSD, a single 4 byte entry per page: 1 GB of overhead! Page-level FTL just isn t practical. 156

157 Block-level mapping greatly reduces tables size. Table: Memory Block: Page: Content: a b c d Flash Chip State: i i i i V V V V i i i i 157

158 Reading is trivial, just compute base + offset. For example, to read logical address 2002: Extract the logical chunk number (500) which gives page 4. Add 2 to get page 6. What about writing? 158

159 Write(2002) with contents c Table: Memory Block: Page: Content: a b c d Flash Chip State: i i i i V V V V i i i i Read in 2000, 2001, and

160 Write(2002) with contents c Table: Memory Block: Page: Content: a b c d a b c d Flash Chip State: i i i i V V V V V V V V Write out all four logical blocks and update mapping. 160

161 Write(2002) with contents c Table: Memory Block: Page: Content: a b c d Flash Chip State: i i i i E E E E V V V V Erase block

162 Improved memory footprint At the cost of performance problems: Physical blocks are 256KB or larger. When writes are smaller than this, lots of of overhead! Block-level mapping isn t going to cut it either! 162

163 Hybrid Mapping Keep a few blocks (log blocks) erased and direct all writes to them. Keep a per-page mapping for these log blocks. Two levels of mappings: Small set of per-page mappings (log table). Larger set of per-block mappings (data table). 163

164 To update: Log Table: Data Table: Memory Block: Page: Content: a b c d Flash Chip State: i i i i i i i i V V V V Overwrite with a, b, c, d. 164

165 To update: Log Table: Memory Data Table: Block: Page: Flash Content: a b c d a b c d Chip State: V V V V i i i i V V V V Because write happened in same manner, can perform switch merge. 165

166 To update: Log Table: Data Table: Memory Block: Page: Content: a b c d Flash Chip State: V V V V i i i i E E E E What about writes not in the same manner? 166

167 To update only logical blocks 0 and 1: Log Table: Data Table: Memory Block: Page: Content: a b a b c d Flash Chip State: V V E E i i i i V V V V Partial merge 167

168 To update only logical blocks 0 and 1: Log Table: Data Table: Memory Block: Page: Content: a b c d a b c d Flash Chip State: V V V V i i i i i i i i 168

169 To update only logical blocks 0 and 1: Log Table: Data Table: Memory Block: Page: Content: a b c d Flash Chip State: V V V V i i i i E E E E 169

170 but filesystems may span more than just one storage device! 170

171 Volumes and Partitions 171

172 Why volumes & partitions? Separate logical & physical storage layers Allow M:N mapping between filesystems & disks 172

173 A volume is a logical storage area. A partition is a slice of a physical disk. A disk may have zero or more partitions A partition may contain a volume A volume may span one or more partitions A volume may exist independently of a partition (e.g., ISO/DMG files) 173

174 GUID Partition Table Scheme 174

175 (Typically) partition volume filesystem Inter-partition/inter-volume filesystem operations are more expensive! Separate metadata structures Separate caches 175

176 Filesystem Robustness 176

177 We like to think of the filesystem (unfortunately) as the rock of the OS when things go wrong (e.g., BSoD/ panic), hard restart and count on persisted data to save us. 177

178 That is, filesystem can t count on OS to play nice! E.g., unannounced crashes, incomplete operations, unfleshed buffers, etc. 178

179 Cannot ensure durability of in-memory data, but want to preserve validity of the file system when possible E.g., file metadata is accurate, persisted data is not corrupted, etc. 179

180 Q: What might happen when a crash occurs? 180

181 Important: differentiate between inmemory (cached) and on-disk (persisted) structures Note: filesystem aggressively caches data! 181

182 E.g., disk block allocation: 1. Update free bitmap 2. Update inode 182

183 1. Update cached free bitmap 2. Update vnode 3. Write back inode 4. Write back disk bitmap crash (durability problem) 183

184 User responsibility; e.g., Unix fsync system call 184

185 1. Update cached free bitmap 2. Update vnode 3. Write back inode 4. Write back disk bitmap crash ( free space in use!) 185

186 1. Update cached free bitmap 2. Update vnode 3. Write back disk bitmap 4. Write back inode crash (lost space) 186

187 E.g., file deletion (number of links = 0) 1. Free inode & data blocks 2. Remove directory link crash ( free space in use!) 187

188 E.g., file deletion (number of links = 0) 1. Remove directory link 2. Free inode & data blocks crash ( orphaned inodes) 188

189 Imminent data corruption vs. storage leak (lesser of two evils) 189

190 Soft updates: order software updates so that, in worst case, we only ever leak free space generally speaking, update freespace structures last 190

191 Leaked space isn t permanent! Can perform manual consistency check of filesystem 191

192 E.g., UFS: Manually walk through all inodes and directory structures Allocated inodes with 0 links can be reused Allocated blocks with no referencing inodes can be garbage collected 192

193 The notorious fsck can report: Unreferenced inodes Link counts in inodes too large Missing blocks in the free map Blocks in the free map also in files Counts in the super-block wrong 193

194 BUT! Soft updates aren t trivial to implement, and may also conflict with caching needs No good! Filesystem is already messy to begin with! 194

195 Another approach to filesystem robustness: journaling / logging 195

196 1. Say what you re about to do 2. Do it 3. Say that you did it 196

197 1. Record what you re about to do 2. Indicate that you finished (a) 3. Do it 4. Record that you did it 197

198 1. Record filesystem update in journal entry 2. Ensure journal entry is persisted 3. Perform filesystem update crash 4. Commit/delete journal entry No journal entry on reboot; No possibility of filesystem inconsistency 198

199 1. Record filesystem update in journal entry 2. Ensure journal entry is persisted 3. Perform filesystem update crash 4. Commit/delete journal entry On reboot, find partial journal entry; No filesystem data corruption possible 199

200 1. Record filesystem update in journal entry 2. Ensure journal entry is persisted 3. Perform filesystem update crash 4. Commit/delete journal entry On reboot, journal shows incomplete filesystem update; replay entry to ensure filesystem consistency 200

201 1. Record filesystem update in journal entry 2. Ensure journal entry is persisted 3. Perform filesystem update crash 4. Commit/delete journal entry Detect completed operation; Commit/delete entry 201

202 Journal enables filesystem transitions Crash replay journal; skip incomplete entries 202

203 Drawbacks? Huge overhead write-twice penalty *Cannot delay persisting journal entries 203

204 Ease overhead: physical vs. semantic journals Physical = record block-level data in journal Semantic = record logical intent when possible 204

205 Also, ensuring filesystem consistency arguably more important that short-term data loss Complete vs. metadata-only journal 205

206 Q: Is there a way to eliminate the writetwice penalty and still get transactional behavior? 206

207 Hint: think back to persistent data structures used to implement MVCC 207

208 There is no spoon (the filesystem is the journal) 208

209 Long-structured filesystem: All filesystem updates are persisted to the end of the journal File updates are effectively copy-on-write Current filesystem state = log replay 209

210 For efficiency, periodically: Garbage collect unreachable blocks, deleted files, etc. from log Write filesystem checkpoints to avoid full replay 210

211 Interesting benefit of LFS: most writes are sequential (but reads are scattered throughout the log) 211

212 Nifty idea, but horrible fragmentation! Impractical with HDDs, but what about SSDs? Robustness without write-twice penalty. Hmmmmmmmmm. 212

213 Interestingly: SSDs already kind of do LFS with TRIM wear leveling writes occur elsewhere on the disk from replaced block Long term performance of SSDs has similar pattern to LFSs SSDs are also fast-to-read, slower-to-write 213

214 Soft updates, journaling, and LFS s = software based solutions 214

215 Hard drive crash? 215

216 Hardware Level Robustness 216

217 Mean Time To Failure 217

218 1,000,000+ hours! 218

219 Crap 219

220 220

221 221

222 222

223 SSDs fare better: Issue is each cell has a limited number of changes it can endure over a lifetime. By example, 100GB SSD designed to accommodate 10 drive writes per day consistently: 1TB of writing each and every single day, 365 days a year for five years 223

224 Hard drive failure: Question of when, not if! 224

225 Redundancy 225

226 Prevent downtime Prevent data loss 226

227 Redundant Array of Independent Disks 227

228 Data robustness 228

229 Secondary objectives: Increased capacity Improved performance 229

230 RAID array = one logical disk 230

231 Transparent to OS/FS (ideally) 231

232 Software vs. hardware RAID 232

233 RAID levels 233

234 Combination of techniques Mirroring Striping Parity 234

235 Type of bit parity Even parity Odd parity Successful transmission scenario A wants to transmit: 1001 A computes parity bit value: (mod 2) = 0 A adds parity bit and sends: B receives: B computes parity: (mod 2) = 0 B reports correct transmission after observing expected even result. A wants to transmit: 1001 A computes parity bit value: (mod 2) = 1 A adds parity bit and sends: B receives: B computes overall parity: (mod 2) = 1 B reports correct transmission after observing expected odd result. 235

236 236

237 237

238 238

239 239

240 240

241 bottleneck! Update: A1 A2 A3 A3 A3 AP 241

242 242

243 Write penalty 243

244 Battle Against Any Raid Five 244

245 Data & parity updates are separate 245

246 Failure in between? 246

247 Write hole 247

248 248

249 249

250 Case Study: xv6 (Unix) 250