42. Crash Consistency: FSCK and Journaling

Transcription

1 42. Crash Consistency: FSCK and Journaling Oerating System: Three Easy Pieces 1

2 Crash Consistency 2

3 Crash Consistency Unlike most data structure, file system data structures must ersist w They must survive over the long haul, stored on devices that retain data desite ower loss. One major challenge faced by a file system is how to udate ersistent data structure desite the resence of a ower loss or system crash. We ll begin by examining the aroach taken by older file systems. w fsck(file system checker) w journaling(write-ahead logging) AOS@UC 3

4 A Detailed Examle (1) Workload w Aend of a single data block(4kb) to an existing file w oen() à lseek() à write() à close() Inode Bitma 8bit, 1/inode Data Bitma 8bit, 1/data block Inodes 8 total, sread across 4 block Data Blocks 8 total I[v1] Da Before aend a single data block w single inode is allocated (inode number 2) w single allocated data block (data block 4) w The inode is denoted I[v1] AOS@UC 4

5 A Detailed Examle (2) Inside of I[v1] (inode, before udate) owner : remzi ermissions : read-only size : 1 ointer : 4 ointer : null ointer : null ointer : null Inode Bitma 8bit, 1/inode Data Bitma 8bit, 1/data block Inodes 8 total, sread across 4 block Data Blocks 8 total I[v1] Da w Size of the file is 1 (one block allocated) w First direct ointer oints to block4 (Da) w All 3 other direct ointers are set to null(unused) AOS@UC 5

6 A Detailed Examle (3) After udate owner : remzi ermissions : read-only size : 2 ointer : 4 ointer : 5 ointer : null ointer : null Inode Bitma 8bit, 1/inode Data Bitma 8bit, 1/data block Inodes 8 total, sread across 4 block Data Blocks 8 total I[v2] Da Db w Data bitma is udated w Inode is udated (I[v2]) w New data block is allocated (Db) AOS@UC 6

7 A Detailed Examle (end) To achieve the transition, the system erform three searate writes to the disk. w One each of inode I[v2] w Data bitma B[v2] w Data block (Db) These writes usually don t haen immediately w dirty inode, bitma, and new data will sit in main memory w age cache or buffer cache If a crash haens after one or two of these write have taken lace, but not all three, the file system could be left in a funny state AOS@UC 7

8 Crash Scenario (1) Imagine only a single write succeeds; there are thus three ossible outcomes 1. Just the data block(db) is written to disk The data is on disk, but there is no inode Thus, it is as if the write never occurred This case is not a roblem at all (for the filesystem) 2. Just the udated inode(i[v2]) is written to disk The inode oints to the disk address (5, Db) But, the Db has not yet been written there We will read garbage data(old contents of address 5) from the disk The bitma says block isn t allocated by inode otherwise n Problem : file-system inconsistency AOS@UC 8

9 Crash Scenario (2) 3. Just the udated bitma (B[v2]) is written to disk The bitma indicates that block 5 is allocated But there is no inode that oints to it Thus, the file system is inconsistent again Problem : sace leak, as block 5 would never be used by the file system AOS@UC 9

10 Crash Scenario (3) There are also three more crash scenarios. In these cases, two writes succeed and the last one fails 1. The inode(i[v2]) and bitma(b[v2]) are written to disk, but not data(db) The file system metadata is comletely consistent Problem : Block 5 has garbage in it 2. The inode(i[v2]) and the data block(db) are written, but not the bitma(b[v2]) We have the inode ointing to the correct data on disk Problem : inconsistency between the inode and the old version of the bitma(b1) AOS@UC 10

11 Crash Scenario (end) 3. The bitma(b[v2]) and data block(db) are written, but not the inode(i[v2]) Problem : inconsistency between the inode and the data bitma We have no idea which file it belongs to AOS@UC 11

12 The Crash Consistency Problem What we d like to do ideally is move the file system from on consistent state to another atomically Unfortunately, we can t do this easily w The disk only commits one write at a time w Crashes or ower loss may occur between these udates We call this general roblem the crash-consistency roblem AOS@UC 12

13 Solution #1: The File System Checker 13

14 The File System Checker (1) The File System Checker (fsck) w fsck is a Unix tool for finding inconsistencies and reairing them. w fsck check suer block, Free block, Inode state, Inode links, etc. w Such an aroach can t fix all roblems examle : The file system looks consistent but the inode oints to garbage data. w The only real goal is to make sure the file system metadata is internally consistent. AOS@UC 14

15 The File System Checker (2) Basic summary of what fsck does: w Suerblock fsck first checks if the suerblock looks reasonable n Sanity checks : file system size > number of blocks allocated Goal : to find susect suerblock In this case, the system may decide to use an alternate coy of the suerblock w Free blocks fsck scans the inodes, indirect blocks, double indirect blocks, The only real goal is to make sure the file system metadata is internally consistent. AOS@UC 15

16 The File System Checker (3) Basic summary of what fsck does: (Cont.) w Inode state Each inode is checked for corrution or other roblem n Examle : valid tye file, directory, symbolic link, etc) If there are roblems with the inode fields that are not easily fixed. n The inode is considered susect and cleared by fsck w Inode Links fsck also verifies the link count of each allocated inode n To verify the link count, fsck scans through the entire directory tree If there is a mismatch between the newly calculated count and that found within an inode, corrective action must be taken n Usually by fixing the count with in the inode AOS@UC 16

17 The File System Checker (4) Basic summary of what fsck does: (Cont.) w Inode Links (Cont.) If an allocated inode is discovered but no directory refers to it, it is moved to the lost+found directory w Dulicates fsck also checks for dulicated ointers Examle : Two different inodes refer to the same block n n If on inode is obviously bad, it may cleared Alternately, the ointed-to block could be coied AOS@UC 17

18 The File System Checker (5) Basic summary of what fsck does: (Cont.) w Bad blocks A check for bad block ointers is also erformed while scanning through the list of all ointers A ointer is considered bad if it obviously oints to something outside it valid range Examle : It has an address that refers to a block greater than the artition size n In this case, fsck can t do anything too intelligent; it just removes the ointer AOS@UC 18

19 The File System Checker (6) Basic summary of what fsck does: (Cont.) w Directory checks fsck does not understand the contents of user files n n However, directories hold secifically formatted information created by the file system itself Thus, fsck erforms additional integrity checks on the contents of each directory Examle n n n making sure that. and.. are the first entries each inode referred to in a directory entry is allocated? ensuring that no directory is linked to more than once in the entire hierarchy AOS@UC 19

20 The File System Checker (end) Buliding a working fsck requires intricate knowledge of the filesystem fsck have a bigger and fundamental roblem: too slow w scanning the entire disk may take many minutes or hours w Performance of fsck became rohibitive. as disk grew in caacity and RAIDs grew in oularity At a higher level, the basic remise of fsck seems just a tad irrational. w It is incredibly exensive to scan the entire disk w It works but is wasteful w Thus, as disk(and RAIDs) grew, researchers started to look for other solutions w search-the-entire-house-for-keys recovery algorithm AOS@UC 20

21 Solution #2: Journaling 21

22 Journaling (1) Journaling (Write-Ahead Logging) w When udating the disk, before over writing the structures in lace, first write down a little note describing what you are about to do w Writing this note is the write ahead art, and we write it to a structure that we organize as a log w By writing the note to disk, you are guaranteeing that if a crash takes laces during the udate of the structures you are udating, you can go back and look at the note you made and try again w Thus, you will know exactly what to fix after a crash, instead of having to scan the entire disk AOS@UC 22

23 Journaling (Cont.) We ll describe how Linux ext3 incororates journaling into the file system. w Most of the on-disk structures are identical to Linux ext2 w The new key structure is the journal itself w It occuies some small amount of sace within the artition or on another device Suer Grou 0 Grou 1 Grou N Fig.1 Ext2 File system structure Suer Journal Grou 0 Grou 1 Grou N Fig.2 Ext3 File system structure AOS@UC 23

24 Data Journaling (1) Data journaling is available as a mode with the ext3 file system Examle : our canonical udate again w We wish to udate inode (I[v2]), bitma (B[v2]), and data block (Db) to disk w Before writing them to their final disk locations, we are now first going to write them to the log(a.k.a. journal) Transaction AOS@UC 24

25 Data Journaling (2) Examle : our canonical udate again (Cont.) Journal TxB I[v2] B[v2] Db TxE w TxB: Transaction begin block It contains some kind of transaction identifier(tid) w The middle three blocks just contain the exact content of the blocks themselves This is known as hysical logging w TxE: Transaction end block Marker of the end of this transaction It also contain the TID AOS@UC 25

26 Data Journaling (3) Checkoint w Once this transaction is safely on disk, we are ready to overwrite the old structures in the file system w This rocess is called checkointing w Thus, to checkoint the file system, we issue the writes I[v2], B[v2], and Db to their disk locations AOS@UC 26

27 Data Journaling (4) Our initial sequence of oerations: 1. Journal write Write the transaction to the log and wait for these writes to comlete TxB, all ending data, metadata udates, TxE 2. Checkoint Write the ending metadata and data udates to their final locations 27

28 Data Journaling (5) When a crash occurs during the writes to the journal 1. Transaction each one at a time 5 transactions (TxB, I[v2], B[v2], Dnb, TxE) This is slow because of waiting for each to comlete 2. Transaction all block writes at once Five writes -> a single sequential write : Faster way However, this is unsafe n n Given such a big write, the disk internally may erform scheduling and comlete small ieces of the big write in any order Write barriers (many disk just ignore them) AOS@UC 28

29 Aside: Enforcing disk write order In old disks was easy to guarantee write ordering w Write A and wait for interrut w Write B Not the case in modern disks w Write buffering in the disk (immediate reorting) in volatile RAM w The disk inform that the data has been written when is coies in the volatile RAM w Disable write buffering?? Write barriers w Instruct to the disk to dum all buffer content before to go ahead w Manufacturers doesn't t necessarily honor this feature (to increase erformance) AOS@UC 29

30 Data Journaling (6) When a crash occurs during the writes to the journal (Cont.) 2. Transaction all block writes at once (Cont.) Thus, the disk internally may (1) write TxB, I[v2], B[v2], and TxE and only later (2) write Db Unfortunately, if the disk loses ower between (1) and (2) Journal TxB I[v2] B[v2]?? TxE Transaction looks like a valid transaction. n n Further, the file system can t look at that forth block and know it is wrong. It is much worse if it haens to a critical iece of file system, such as suerblock. AOS@UC 30

31 Data Journaling (7) When a crash occurs during the writes to the journal (Cont.) 2. Transaction all block writes at once (Cont.) To avoid this roblem, the file system issues the transactional write in two stes First, writes all blokes excet the TxE block to journal Journal TxB id=1 I[v2] B[v2] Db Second, The file system issues the write of the TxE Journal TxB id=1 An imortant asect of this rocess is the atomicity guarantee rovided by the disk. I[v2] B[v2] Db TxE id=1 n n The disk guarantees that any 512-byte write either haen or not Thus, TxE should be a single 512-byte block AOS@UC 31

32 Data Journaling (8) When a crash occurs during the writes to the journal (Cont.) 2. Transaction all block writes at once (Cont.) Thus, our current rotocol to udate the file system, with each of its three hases labeled: 1. Journal write : write the contents of the transaction to the log 2. Journal commit (added) : write the transaction commit block 3. Checkoint : write the contents of the udate to their locations AOS@UC 32

33 Data Journaling (end) Recovery w If the crash haens before the transactions is written to the log The ending udate is skied w If the crash haens after the transactions is written to the log, but before the checkoint Recover the udate as follow: n n Scan the log and lock for transactions that have committed to the disk Transactions are relayed AOS@UC 33

34 Batching Log Udates If we create two files in same directory, same inode, directory entry block is to the log and committed twice. To reduce excessive write traffic to disk, journaling manage the global transaction. w Write the content of the global transaction forced by synchronous request. w Write the content of the global transaction after a timeout (v.gr. 5 seconds). AOS@UC 34

35 Making The log Finite (1) The log is of a finite size w Two roblems arise when the log becomes full Journal Tx1 Tx2 Tx3 Tx4 Tx5 1. The larger the log, the longer recovery will take Simler but less critical 2. No further transactions can be committed to the disk Thus making the file system less than useful AOS@UC 35

36 Making The log Finite (2) To address these roblems, journaling file systems treat the log as a circular data structure, re-using it over and over w This is why the journal is referred to as a circular log. To do so, the file system must take action some time after a checkoint w Secifically, once a transaction has been checkointed, the file system should free the sace AOS@UC 36

37 Making The log Finite (3) journal suer block w Mark the oldest and newest transactions in the log. w The journaling system records which transactions have not been check ointed. Journal Journal Suer Tx1 Tx2 Tx3 Tx4 Tx5 AOS@UC 37

38 Making The log Finite (end) journal suer block (Cont.) w Thus, we add another ste to our basic rotocol 1. Journal write 2. Journal commit 3. checkoint 4. Free n Some time later, mark the transaction free in the journal by udating the journal Suerblock AOS@UC 38

39 Metadata Journaling (1) There is a still roblem : writing every data block to disk twice w Commit to log (journal file) w Checkoint to on-disk location. Peole have tried a few different things in order to seed u erformance. w Examle : A simler form of journaling is called ordered journaling (metadata journaling) User data is not written to the journal AOS@UC 39

40 Metadata Journaling (2) Thus, the following information would be written to the journal Journal TxB I[v2] B[v2] TxE The data block Db, reviously written to the log, would instead be written to the file system roer 40

41 Metadata Journaling (3) The modification does raise an interesting question: when should we write data blocks to disk? Let s consider an examle 1. Write Data to disk after the transaction Unfortunately, this aroach has a roblem The file system is consistent but I[v2] may end u ointing to garbage data 2. Write Data to disk before the transaction It avoids the roblems AOS@UC 41

42 Metadata Journaling (end) Secifically, the rotocol is as follows: 1. Data Write(added): Write data to final location 2. Journal metadata write(added): Write the begin and metadata to the log 3. Journal commit 4. Checkoint metadata 5. Free 42

43 Tricky case: Block Reuse (1) Some metadatas should not be relayed. Examle 1. Directory foo is udated. Journal TxB id=1 I[foo] tr:1000 D[foo] [final addr:1000] TxE id=1 2. Directory foo id deleted. block 1000 is freed u for reuse 3. User Create a file foobar, reusing block 1000 for data Journal TxB id=1 I[foo] tr:1000 D[foo] [final addr:1000] TxE id=1 TxB id=2 I[foobar] tr:1000 TxE id=2 AOS@UC 43

44 Tricky case: Block Reuse (2) 4. Now assume a crash occurs and all of this information is still in the log. Journal TxB id=1 I[foo] tr:1000 D[foo] [final addr:1000] TxE id=1 TxB id=2 I[foobar] tr:1000 TxE id=2 5. During relay, the recovery rocess relays everything in the log Including the write of directory data in block The relay thus overwrites the user data of current file foobar with old directory contents AOS@UC 44

45 Tricky case: Block Reuse (2) Solution w What Linux ext3 does instead is to add a new tye of record to the journal, Known as a revoke record w When relaying the journal, the system first scans for such revoke records w Any such revoked data is never relayed AOS@UC 45

46 Data Journaling Timeline TxB Journal contents File System TxE (metadata) (data) Metadata Data issue issue issue comlete comlete comlete issue comlete issue comlete issue comlete Data Journaling Timeline 46

47 Metadata Journaling Timeline TxB Journal contents TxE File System (metadata) Metadata Data issue issue Issue comlete comlete comlete issue comlete issue comlete Metadata Journaling Timeline 47

48 Disclaimer: Disclaimer: This lecture slide set is used in AOS course at University of Cantabria by V.Puente. Was initially develoed for Oerating System course in Comuter Science Det. at Hanyang University. This lecture slide set is for OSTEP book written by Remzi and Andrea Araci-Dusseau (at University of Wisconsin) 48