Caching and reliability

Transcription

1 Caching and reliability

2 Block cache Vs. Latency ~10 ns 1~ ms Access unit Byte (word) Sector Capacity Gigabytes Terabytes Price Expensive Cheap Caching disk contents in RAM Hit ratio h : probability of data existing in cache speedup = 10h h for h=0.9, 2 for h=0.5

3 Block cache operation in OS Process A Process B Process Z File A File B File A FP a FP b FP c Inode A Inode B File A Directory A File B Inode A Inode B Directory A File A File B

4 Best scenario What to cache? Caching data to access in near future: high hit rate Impossible: cannot foresee the future Replacement algorithms FIFO / LIFO Least-Recently Used Simple and known to be universal solution Least-Frequently Used Second chance algorithms: clock, multi-queue,

5 x86 page table LRU-approximate algorithms Accessed / dirty bits per page Check page table regularly identify recently-accessed pages

6 Cache eviction algorithms Clock Round robin page scanning Check accessed flag If set, clear If not set, evict accessed accessed accessed Second chance accessed Improved clock algorithm Give second chance to the not accessed pages Evict at the second chance accessed

7 Cache eviction algorithms Multi-queue Most Recent Multiple LRU/FIFO queues Q 3 Promote when accessed Demote at the end of Q Q 2 Q 1 Promote Demote Evict the lowest Q s LRU entry Evict Swapping vs. eviction Cached page: page with an associated inode and a file offset Anonymous page: page allocated by an application

8 Active list Inactive list Clock algorithm for each queue Linux 2-queue algorithm mark_page_accessed() lru_cache_add() Head Head Tail Tail Reclamation Eviction (write-back)

9 Adding to page cache Linux page cache API void add_to_page_cache(struct page * page, struct address_space * mapping, unsigned long offset) struct page * page_cache_alloc(struct address_space *x) int page_cache_read(struct file * file, unsigned long offset) Removing from page cache Case: file is erased while it is in page cache void remove_inode_page(struct page *page) void page_cache_release(struct page *page) c.f.) void page_cache_get(struct page *page)

10 Out-of-order writes Caching and consistency A transaction of pages update completes an atomic I/O Ex. Append to a file: Data block new data Free block bitmap mark allocated blocks Inode increase size, add allocated blocks Superblock decrease the number of free blocks RAM s i b D D FS s i b D D Higher-level objects are updated frequently prone to write Buffer cache evicts pages in unpredictable order

11 Failure in-writing Consistency problem Power failure Device failure: bad blocks, bit-flip, wear-out, S/W Bugs RAM s i b D D FS s i b D D

12 Surviving hardware failures Replication (RAID, backup) Checksum (ECC) Remapping and warning (SMART) UPS Low-level mechanisms Out-of-order writes Blocking I/O I/O Barrier (NCQ) I/O barrier

13 Static mapping Enforcing Reliability Read-only file system, root directory of FAT, Reliable but inflexible Synchronous write Low performance Reliable? ex. rename(a, b) a file b

14 Enforcing Reliability <Journaling> Journaling : write-ahead logging [Hagmann, SOSP 87] Log file system operation Redo in recovery time Fast disaster recovery Additional overhead Log and checkpoint: after checkpoint, discards logs RAM s i d D D Log d i s Logging FS s Check-pointing i d D D

15 Enforcing Reliability <Shadow paging> Shadow paging [Chamberlin, ACM Comm 81] inode data data data Btrfs, XFS, DFS, Avoid in-place updates Cascaded updates propagation Log-structured file system [Rosenblum, ACM ToCS 92] Transform every file system updates to log operations Good for write operations Heavy garbage collection overhead

16 Ext* File System Journaling Mechanism

17 Goal of file system journaling Until ext2, on every system crash File can be lost or corrupted File system itself is corrupted File system checker (fsck) must be executed before mount Long time to check and correct the file system Journaling: a safety measure All data is conserved Some data can be lost, but file system must be consistent

18 Ex. Append to a file File system I/O = a transaction Data block new data Free block bitmap mark allocated blocks Inode increase size, add allocated blocks Superblock decrease the number of free blocks RAM s i b D D FS s i b D D

19 Write ahead logging Write log of updates before applying them to FS RAM s i d D D Log Logging FS File system update d i s D D s i d D D System failure cases Crash while logging Ignore the log Crash after logging: during file system update Replay log to apply Crash after file system updates No need to replay

20 Log records Elements of transaction Low-level operation of ext3fs: every updated blocks Whole block or changes only? Buffer_head in ext3/jbd journal_head bh buf bh buf buf Transaction An atomic group of log records Modified blocks consisting an atomic operation One transaction for each FS operation

21 Problem: High overhead for small I/Os A few blocks updates for each transaction Transaction group Solution Group some transactions to update journal Descriptor block Transaction 1 Transaction 2 Transaction 3 Commit block Tx ID block offsets Modified blocks End of transaction

22 Journal structure File or external device Special inode (5) for journal Journal file: circular log At the center of partition sb sb journal S_blocksize S_maxlen S_first S_sequence S_start S_errno Block size Nr. Blocks in journal file First block of log information First commit ID Block no. of start of log Errno of last journal abort

23 Commit Write a transaction group to journal Commit and Checkpoint Two stage: (descriptor block, log blocks) (commit block) After commit complete, they can be recovered by replay Cached updated blocks remain in memory after commit File system update are done using the cached blocks Checkpoint Force flush modified blocks to file system No need to read from log (cached in memory) Blocks can be written between commit and checkpoint

24 Commit a transaction Condition: every 5 sec / sync I/O / too many buffers kernel thread kjournald Running Transaction group Locked Committing Commit Running Transaction group Trans 1 Trans 2 Trans 3 handle Descriptor block Transact 1 Transact 2 Transact 3 Commit block Descriptor block Transact 1 Now, updated blocks in the transaction can be written to FS

25 Flushing dirty blocks in journal To restrict journal size and recovery time Checkpoint tx0 tx1 tx2 tx3 tx4 tx5 tx6 tx7 tx8 tx9 Replay: logs after the checkpoint Optimal checkpoint interval? Ext3: circular log Checkpoint! Checkpoint! tx0 tx9 tx1 tx2 tx3 tx4 tx5 tx6 tx7 tx8 Write dirty blocks (in page cache) in the oldest transaction

26 Journal data Journal metadata and data blocks Highest safety with high overhead Ordered data (default) Journaling modes Journal metadata only Data blocks must be updated before commit Writeback data Journal metadata only Don t care when the data blocks are written Unsafe but fast

27 Journaling modes in ext3

28 Recovery When: mount time Procedure Check superblock: nothing to do on clean unmount Round 1: find start and end of the log Round 2: update revoke table Round 3: write valid journal blocks in to FS Start a new transaction at the end of the journal Update superblock Checkpoint! tx9 tx1 tx2 tx3 tx4 tx5 tx6 tx7 tx8