Junfeng Yang. EXPLODE: a Lightweight, General System for Finding Serious Storage System Errors. Stanford University
|
|
- Vivien Palmer
- 5 years ago
- Views:
Transcription
1 EXPLODE: a Lightweight, General System for Finding Serious Storage System Errors Junfeng Yang Stanford University Joint work with Can Sar, Paul Twohey, Ben Pfaff, Dawson Engler and Madan Musuvathi
2 Mom, Google Ate My GMail! 2
3 3
4 Why check storage systems? Storage system errors: some of the most serious machine crash data loss data corruption Code complicated, hard to get right Conflicting goals: speed, reliability (recover from any failures and crashes) Typical ways to find these errors: ineffective Manual inspection: strenuous, erratic Randomized testing (e.g. unplug the power cord): blindly throwing darts Error report from mad users 4
5 Goal: build tools to automatically find storage system errors Sub-goal: comprehensive, lightweight, general 5
6 EXPLODE [OSDI06] Comprehensive: adapt ideas from model checking General, real: check live systems Can run (on Linux, BSD), can check, even w/o source code Fast, easy Check a new storage system: 200 lines of C++ code Port to a new OS: 1 kernel module + optional modification Effective 17 storage systems: 10 Linux FS, Linux NFS, Soft-RAID, 3 version control, Berkeley DB, VMware Found serious data-loss in all Subsumes FiSC [OSDI04, best paper] 6
7 Outline Overview Checking process Implementation Example check: crashes during recovery are recoverable Results 7
8 Long-lived bug fixed in 2 days in the IBM Journaling file system (JFS) Serious Loss of an entire FS! Fixed in 2 days with our complete trace Hard to find 3 years old, ever since the first version Dave Kleikamp (IBM JFS): I really appreciate your work finding and recreating this bug. I'm sure this has bitten us before, but it's usually hard to go back and find out what causes the file system to get messed up so bad 8
9 Events to trigger the JFS bug creat( /f ); flush /f 5-char system call, crash! not a typo fsck.jfs File system recovery utility, run after reboot Buffer Cache (in mem) Disk / / Orphan file removed. Legal behavior for file systems f 9
10 Events to trigger the JFS bug creat( /f ); bug under low mem (design flaw) flush / crash! fsck.jfs File system recovery utility, run after reboot Buffer Cache (in mem) Disk / f dangling / pointer! fix by zeroing, entire FS gone! 10
11 Overview User-written Checker } creat( /f ) EXPLODE Runtime void mutate() { void check() { Toy creat( /old ); checker: crash after int fd creat( /f ) = open( /old, sync(); O_RDONLY); should not lose any old file that is already creat( /f ); User-written persistent checker: on if disk (fd < 0) check_crash_now(); can be either very sophisticated or very simple close(fd); crash-disk f fsck.jfs } error( lost old! ); /root check( ) old JFS /root / f EKM / fsck.jfs /root check( ) old Linux Kernel Hardware / f fsck.jfs check( /root ) old f User code Our code EKM = EXPLODE kernel module 11
12 Outline Overview Checking process Implementation Example check: crashes during recovery are recoverable Results 12
13 One core idea from model checking: explore all choices Bugs are often triggered by corner cases How to find? Drive execution down to these tricky corner cases Principle When execution reaches a point in program that can do one of N different actions, fork execution and in first child do first action, in second do second, etc. Result: rare events appear as often as common ones 13
14 Crashes (Overview slide revisit) User-written Checker EXPLODE Runtime crash-disk f fsck.jfs /root check( ) old JFS /root old / f EKM / / f fsck.jfs fsck.jfs /root check( ) check( /root ) old f Linux Kernel Hardware 14
15 External choices Fork and do every possible operation /root a c r e at m kd i r unl ink rm dir /root a b /root a c /root /root Explore generated states as well Users write code to check FS valid. EXPLODE amplifies a 15
16 Internal choices Fork and explore all internal choices /root /root a ca ch e h i t miss d is k re ad s ucce e d f a i l a b /root a /root struct block* read_block (int i) { struct block *b; if ((b = cache_lookup(i))) return b; return disk_read (i); } a b 16
17 Users expose choices using choose(n) To explore N-choice point, users instrument code using choose(n) (also used in other model checkers) choose(n): N-way fork, return K in K th kid struct block* read_block (int i) { struct block *b; if ((b = cache_lookup(i))) return b; return disk_read (i); } cache_lookup (int i) { if(choose(2) == 0) return NULL; // normal lookup } Optional. Instrumented only 7 places in Linux 17
18 Crash X External X Internal /root a b /root a /root a c /root /root a 18
19 Speed: skip same states /root a reat( /b ) c m k dir( /c ) /root a b /root a c /root /root m k di r ( /c ) c reat ( / b ) /root a b c Abstract and hash a state, discard if seen. a 19
20 Outline Overview Checking process Implementation FiSC, File System Checker, [OSDI04], best paper EXPLODE, storage system checker, [OSDI06] Example check: crashes during recovery are recoverable Results 20
21 Checking process S0 S0 = checkpoint() enqueue(s0) while(queue not empty){ S = dequeue() restore(s) for each action in S { do action S = checkpoint() if(s is new) enqueue(s ) } } How to checkpoint and restore a live OS? 21
22 FiSC: jam OS into tool User-written Checker JFS User Mode Linux FiSC Linux Kernel Hardware User code Our code Pros Cons Comprehensive, effective No model, check code Checkpoint and restore: easy Intrusive. Build fake environment. Hard to check anything new. Months for new OS, 1 week for new FS Many tricks, so complicated that we won best paper OSDI 04 22
23 EXPLODE: jam tool into OS User-written Checker JFS User-written Checker EXPLODE Runtime User Mode Linux FiSC Linux Kernel Hardware JFS EKM Linux Kernel Hardware User code Our code EKM = EXPLODE kernel module 23
24 EKM lines of code OS Lines of code Linux 2.6 1,915 FreeBSD 6.0 1,210 EXPLODE kernel modules (EKM) are small and easy to write 24
25 How to checkpoint and restore a live OS kernel? Checker Hard to checkpoint live kernel memory EXPLODE Runtime Virtual machine? No JFS VMware: no source Xen: not portable EKM Linux Kernel Hardware heavyweight There s a better solution for storage systems 25
26 Checkpoint: save actions instead of bits state = list of actions checkpoint S = save (creat, cache miss) restore = re-initialize, creat, cache miss re-initialize = unmount, mkfs Utility that clears Utility to creat S0 Redo-to-restore-state in-mem state of a idea create used an in storage system model checking to trade empty time FS for space (stateless search). c ache m i s s S We use it only to reduce intrusiveness 26
27 Deterministic replay Storage system: isolated subsystem Non-deterministic kernel scheduling decision Opportunistic fix: priorities Non-deterministic interrupt Fix: use RAM disks, no interrupt for checked system Non-deterministic kernel choose() calls by other code Fix: filter by thread IDs. No choose() in interrupt Worked well in practice Mostly deterministic Worst case: auto-detect & ignore non-repeatable errors 27
28 Outline Overview Checking process Implementation Example check: crashes during recovery are recoverable Results 28
29 Why check crashes during recovery? Crashes are highly correlated Often caused by kernel bugs, hardware errors Reboot, hit same bug/error What to check? fsck once == fsck & crash, re-run fsck fsck(crash-disk) to completion, /a recovered fsck(crash-disk) and crash, fsck, /a gone Bug! Powerful heuristic, found interesting bugs (wait until results) 29
30 How to check crashes during recovery? crash-disk fsck.jfs EXPLODE Runtime fsck.jfs fsck.jfs same as? same as? fsck.jfs same as? crash-crash-disk Problem: N blocks 2^N crash-crash-disks. Too many! Can prune many crash-crash-disks 30
31 Simplified example fsck(000) Read(B1) = 0 Write(B2, 1) Write(B3, 1) Read(B3) = 1 Write(B1, 1) 3-block disk, B1, B2, B3 each block is either 0 or 1 crash-disk = 000 (B1 to B3) buffer cache: B2=1 buffer cache: B2=1, B3=1 buffer cache: B2=1, B3=1, B1=1 fsck(000) =
32 Naïve strategy: 7 crash-crash-disks crash-disk = 000 fsck(000) = 111 Read(B1) = 0 Write(B2, 1) Write(B3, 1) Read(B3) = 1 Write(B1, 1) buffer cache: B2=1, B3=1, B1= {B2=1} fsck(010) == 111? fsck(001) == 111? fsck(011) == 111? fsck(100) == 111? fsck(110) == 111? fsck(101) == 111? fsck(111) == 111? 32
33 Optimization: exploiting determinism crash-disk = 000 fsck(000) = 111 Read(B1) = 0 Write(B2, 1) Write(B3, 1) Read(B3) = 1 Write(B1, 1) For all practical purposes, fsck is deterministic read same blocks write same blocks {B2=1} fsck(010) == 111? fsck(000) doesn t read B2 So, fsck(010) =
34 What blocks does fsck(000) actually read? crash-disk = 000 fsck(000) = 111 Read(B1) = 0 Write(B2, 1) Write(B3, 1) Read(B3) = 1 Write(B1, 1) Read of B3 will get what we just wrote. Can t depend on B3 fsck(000) reads/depends only on B1. It doesn t matter what we write to the other blocks. fsck(0**) =
35 Prune crash-crash-disks matching 0** crash-disk = 000 buffer cache: B2=1, B3=1, B1=1 fsck(000) = 111 Read(B1) = 0 Write(B2, 1) Write(B3, 1) Read(B3) = 1 Write(B1, 1) fsck(010) == 111? fsck(001) == 111? fsck(011) == 111? fsck(100) == 111? fsck(110) == 111? fsck(101) == 111? fsck(111) == 111? Can further optimize using this and other ideas 35
36 Outline Overview Checking process Implementation Example check: crashes during recovery are recoverable Results 36
37 Bugs caused by crashes during recovery Found data-loss bugs in all three FS that use logging (ext3, JFS, ReiserFS), total 5 Strict order under normal operation: First, write operation to log, commit Second, apply operation to actual file system Strict (reverse) order during recovery: First, replay log to patch actual file system Second, clear log No order corrupted FS and no log to patch it! 37
38 Bug in fsck.ext3 recover_ext3_journal( ) { // retval = -journal_recover(journal); // // clear the journal e2fsck_journal_release( ) // } journal_recover( ) { // replay the journal // // sync modifications to disk fsync_no_super ( ) } // Error! Empty macro, doesn t sync data! #define fsync_no_super(dev) do {} while (0) Code directly adapted from the kernel But, fsync_no_super defined as NOP: hard to implement 38
39 FiSC Results (can reproduce in EXPLODE) Error Type VFS ext2 ext3 JFS ReiserFS total Data loss N/A N/A False clean N/A N/A Security Crashes Other Total in total, 21 fixed, 9 of the remaining 11 confirmed 39
40 EXPLODE checkers lines of code and errors found Storage System Checked 10 file systems CVS Checker 5, Bugs 18 1 Storage Subversion 69 1 applications EXPENSIVE Berkeley DB RAID FS Transparent subsystems NFS VMware GSX/Linux FS FS 4 1 Total 6, bugs per 1,000 lines of checker code 40
41 Related work FS Testing Static (compile-time) analysis Software model checking 41
42 Conclusion EXPLODE Comprehensive: adapt ideas from model checking General, real: check live systems in situ, w/o source code Fast, easy: simple C++ checking interface Results Checked 17 widely-used, well-tested, real-world storage systems: 10 Linux FS, Linux NFS, Soft-RAID, 3 version control, Berkeley DB, VMware Found serious data-loss bugs in all, over 70 bugs in total Many bug reports led to immediate kernel patches 42
43 Future work Apply EXPLODE to other systems (e.g. distributed system) Design and build easy-to-check systems Apply program analysis techniques to other system-building stages (e.g. debugging) 43
44 Other projects Automatically generating malicious disks using symbolic execution. J. Yang, C. Sar, P. Twohey, C. Cadar, and D. Engler [Oakland Security 06] MECA: an extensible, expressive system and language for statically checking security properties. J. Yang, T. Kremenek, Y. Xie, and D. Engler [ACM CCS 03] An empirical study of operating system errors. A. Chou, J. Yang, B. Chelf, S. Hallem, and D. Engler. [SOSP 01] Kinesis: a case for multi-hash replica placement in distributed storage systems. J. MacCormick, N. Murphy, V. Ramasubramanian, U. Wieder, J. Yang, and L. Zhou. [MSR TR 2006] Correlation exploitation in error ranking. T. Kremenek, K. Ashcraft, J. Yang, and D. Engler [FSE 2004] 44
45 45
46 Conclusion EXPLODE: comprehensive, lightweight, general, effective Checked 17 real-world storage systems and found serious data loss in every system checked Future work Apply EXPLODE to other systems (e.g. distributed sys) How to design and build easy-to-check systems Adapt program analysis techniques to other systembuilding stages (e.g. debugging) 46
47 Crude benchmarking result Physical machine: P4 3.2G with 2G mem Virtual machine: 1G mem 20 hours 230,744 crashes, 3 crashes/second 327 different FS topologies 47
48 FS Sync checking results : observed data-loss Applications (such as your mailer) rely on sync operations, yet they are broken 48
49 ext2 fsync bug due to design flaw Events to trigger bug truncate A A B creat B Mem write B fsync B crash! Disk inode A B fsck.ext2 indirect block 49
50 50
51 Checking crashes EXPLODE Kernel Log: mark_dirty(b1) make_request(b2) end_request(b1,b2) Log { 1 buffer } cache operations. Use this log to recreate cache 1 2 { } { } 1 2 Generate Crashes
52 Checking crashes EXPLODE Kernel Log: mark_dirty(b1) mark_dirty(b1) make_request(b2) end_request(b1,b2) { 1 } { 1 } { 1 2 } { } Keep old value 1 2 Generate Crashes
53 Classic app mistake: atomic rename All three version control app. made this mistake Atomically update file A to avoid corruption fd = creat(a_tmp, ); write(fd, ); fsync(fd); // missing! close(fd); rename(a_tmp, A); Problem: rename guarantees nothing abt. Data 53
54 Checking a storage stack Checker EXPLODE Runtime JFS RAID EKM Linux Hardware User code Our code EKM = EXPLODE kernel module 54
55 What EXPLODE provides check_crash_now(): check all crashes that can happen at the current moment Paper talks more ways for checking crashes error(): record trace for deterministic replay choose(n): conceptual N-way fork, return K in K th child execution 55
56 Checker : What users do mutate(): drive JFS to do something check(): verify what JFS did was correct storage component: JFS RAID initialize, set up, repair and tear down JFS, RAID. Write once per system assemble a checking stack JFS RAID 56
57 0 Checker mutate JFS Component Stack choose(4) creat file rm file mkdir rmdir sync fsync 57
58 Checker Check file exists check JFS Component Check file contents match Stack Found yet another JFS fsync bug, caused by re-using directory inode as file inode Checkers can be simple (50 lines) or very complex(5,000 lines) Whatever you can express in C++, you can check 58
59 FS Checker JFS Component storage component: initialize, repair, set up, and tear down your system Mostly wrappers to existing utilities. mkfs, fsck, mount, umount threads(): returns list of kernel thread IDs for deterministic error replay Stack Write once per system, reuse to form stacks Easy to write: proof next slide 59
60 FS Checker JFS Component Stack 60
61 FS Checker assemble a checking stack JFS Component Stack JFS Let EXPLODE know how subsystems are connected together, so it can initialize, set up, tear down, and repair the entire stack Easy to write: proof next slide RAID 61
62 FS Checker JFS Component Stack JFS RAID 62
63 Internal choices Fork and explore all internal choices /root a ca ch e h i t miss d is k re ad s ucce e d f a i l /root a b /root a /root a b 63
64 Naïve strategy: 7 crashes crash-disk = 000 crash-crash-disk = 001 fsck(000) = 111 Read(B1) = 0 Write(B2, 1) Write(B3, 1) Read(B3) = 1 Write(B1, 1) fsck(001) =? Read(B1) = 0 Write(B2, 1) Write(B3, 1) Read(B3) = 1 Write(B1, 1) blocks fsck reads empty {B1=0} {B1=0} {B1=0} {B1=0, B3=1} {B1=0, B3=1} 64
65 Optimization: exploiting determinism For all practical purposes, fsck is deterministic read same blocks write same blocks Want prune without running fsck such crashcrash-disk that fsck(crash-crash-disk) == fsck(crash-disk) crash-crash-disk = crash-disk + W, where W is a subset of fsck-modified blocks If W doesn t change any blocks that fsck(crash-disk) reads, then fsck(crash-disk + W) == fsck(crash-disk) 65
66 Future work Short-term: EXPLODE follow-on projects Long-term: checking Implementation Deployment Design Testing Patching Monitoring Debugging Crash analysis 66
67 Future work Short-term: EXPLODE follow-on projects Long-term: checking Design Implementation Debugging Testing Or, whatever good ideas that strike me 67
68 Optimization: exploiting determinism For all practical purposes, fsck is deterministic read same blocks write same blocks Can prune without running fsck such crashcrash-disk that fsck(crash-crash-disk) == fsck(000) Example: fsck(010) == fsck(000) 010 = {B2=1} {B2=1} doesn t change anything fsck(000) reads fsck(010) will read the same blocks as fsck(000), therefore write the same blocks 68
EXPLODE: a Lightweight, General System for Finding Serious Storage System Errors. Junfeng Yang, Can Sar, Dawson Engler Stanford University
EXPLODE: a Lightweight, General System for Finding Serious Storage System Errors Junfeng Yang, Can Sar, Dawson Engler Stanford University Why check storage systems? Storage system errors are among the
More informationEXPLODE: A Lightweight, General Approach to Finding Serious Errors in Storage Systems
EXPLODE: A Lightweight, General Approach to Finding Serious Errors in Storage Systems Junfeng Yang, Paul Twohey, Ben Pfaff, Can Sar, Dawson Engler Computer Systems Laboratory Stanford University Stanford,
More informationAdvanced file systems: LFS and Soft Updates. Ken Birman (based on slides by Ben Atkin)
: LFS and Soft Updates Ken Birman (based on slides by Ben Atkin) Overview of talk Unix Fast File System Log-Structured System Soft Updates Conclusions 2 The Unix Fast File System Berkeley Unix (4.2BSD)
More informationW4118 Operating Systems. Instructor: Junfeng Yang
W4118 Operating Systems Instructor: Junfeng Yang File systems in Linux Linux Second Extended File System (Ext2) What is the EXT2 on-disk layout? What is the EXT2 directory structure? Linux Third Extended
More informationMEMBRANE: OPERATING SYSTEM SUPPORT FOR RESTARTABLE FILE SYSTEMS
Department of Computer Science Institute of System Architecture, Operating Systems Group MEMBRANE: OPERATING SYSTEM SUPPORT FOR RESTARTABLE FILE SYSTEMS SWAMINATHAN SUNDARARAMAN, SRIRAM SUBRAMANIAN, ABHISHEK
More informationSystem Errors. Junfeng Yang, Can Sar, and Dawson Engler Computer Systems Laboratory Stanford University
EXPLODE: a Lightweight, General System for Finding Serious Storage System Errors Junfeng Yang, Can Sar, and Dawson Engler Computer Systems Laboratory Stanford University Abstract Storage systems such as
More informationFile System Consistency. Jin-Soo Kim Computer Systems Laboratory Sungkyunkwan University
File System Consistency Jin-Soo Kim (jinsookim@skku.edu) Computer Systems Laboratory Sungkyunkwan University http://csl.skku.edu Crash Consistency File system may perform several disk writes to complete
More informationCS 318 Principles of Operating Systems
CS 318 Principles of Operating Systems Fall 2017 Lecture 17: File System Crash Consistency Ryan Huang Administrivia Lab 3 deadline Thursday Nov 9 th 11:59pm Thursday class cancelled, work on the lab Some
More informationExt3/4 file systems. Don Porter CSE 506
Ext3/4 file systems Don Porter CSE 506 Logical Diagram Binary Formats Memory Allocators System Calls Threads User Today s Lecture Kernel RCU File System Networking Sync Memory Management Device Drivers
More informationFile System Consistency
File System Consistency Jinkyu Jeong (jinkyu@skku.edu) Computer Systems Laboratory Sungkyunkwan University http://csl.skku.edu EEE3052: Introduction to Operating Systems, Fall 2017, Jinkyu Jeong (jinkyu@skku.edu)
More informationCrash Consistency: FSCK and Journaling. Dongkun Shin, SKKU
Crash Consistency: FSCK and Journaling 1 Crash-consistency problem File system data structures must persist stored on HDD/SSD despite power loss or system crash Crash-consistency problem The system may
More informationCaching and reliability
Caching and reliability Block cache Vs. Latency ~10 ns 1~ ms Access unit Byte (word) Sector Capacity Gigabytes Terabytes Price Expensive Cheap Caching disk contents in RAM Hit ratio h : probability of
More informationMembrane: Operating System support for Restartable File Systems
Membrane: Operating System support for Restartable File Systems Membrane Swaminathan Sundararaman, Sriram Subramanian, Abhishek Rajimwale, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, Michael M.
More informationò Very reliable, best-of-breed traditional file system design ò Much like the JOS file system you are building now
Ext2 review Very reliable, best-of-breed traditional file system design Ext3/4 file systems Don Porter CSE 506 Much like the JOS file system you are building now Fixed location super blocks A few direct
More informationFS Consistency & Journaling
FS Consistency & Journaling Nima Honarmand (Based on slides by Prof. Andrea Arpaci-Dusseau) Why Is Consistency Challenging? File system may perform several disk writes to serve a single request Caching
More informationFile Systems: Consistency Issues
File Systems: Consistency Issues File systems maintain many data structures Free list/bit vector Directories File headers and inode structures res Data blocks File Systems: Consistency Issues All data
More informationFile Systems Management and Examples
File Systems Management and Examples Today! Efficiency, performance, recovery! Examples Next! Distributed systems Disk space management! Once decided to store a file as sequence of blocks What s the size
More informationCase study: ext2 FS 1
Case study: ext2 FS 1 The ext2 file system Second Extended Filesystem The main Linux FS before ext3 Evolved from Minix filesystem (via Extended Filesystem ) Features Block size (1024, 2048, and 4096) configured
More informationCase study: ext2 FS 1
Case study: ext2 FS 1 The ext2 file system Second Extended Filesystem The main Linux FS before ext3 Evolved from Minix filesystem (via Extended Filesystem ) Features Block size (1024, 2048, and 4096) configured
More informationAnnouncements. Persistence: Crash Consistency
Announcements P4 graded: In Learn@UW by end of day P5: Available - File systems Can work on both parts with project partner Fill out form BEFORE tomorrow (WED) morning for match Watch videos; discussion
More informationCS3210: Crash consistency
CS3210: Crash consistency Kyle Harrigan 1 / 45 Administrivia Lab 4 Part C Due Tomorrow Quiz #2. Lab3-4, Ch 3-6 (read "xv6 book") Open laptop/book, no Internet 2 / 45 Summary of cs3210 Power-on -> BIOS
More informationAnnouncements. Persistence: Log-Structured FS (LFS)
Announcements P4 graded: In Learn@UW; email 537-help@cs if problems P5: Available - File systems Can work on both parts with project partner Watch videos; discussion section Part a : file system checker
More informationCOS 318: Operating Systems. NSF, Snapshot, Dedup and Review
COS 318: Operating Systems NSF, Snapshot, Dedup and Review Topics! NFS! Case Study: NetApp File System! Deduplication storage system! Course review 2 Network File System! Sun introduced NFS v2 in early
More informationCrashMonkey: A Framework to Systematically Test File-System Crash Consistency. Ashlie Martinez Vijay Chidambaram University of Texas at Austin
CrashMonkey: A Framework to Systematically Test File-System Crash Consistency Ashlie Martinez Vijay Chidambaram University of Texas at Austin Crash Consistency File-system updates change multiple blocks
More information[537] Journaling. Tyler Harter
[537] Journaling Tyler Harter FFS Review Problem 1 What structs must be updated in addition to the data block itself? [worksheet] Problem 1 What structs must be updated in addition to the data block itself?
More informationCS122 Lecture 15 Winter Term,
CS122 Lecture 15 Winter Term, 2017-2018 2 Transaction Processing Last time, introduced transaction processing ACID properties: Atomicity, consistency, isolation, durability Began talking about implementing
More informationmode uid gid atime ctime mtime size block count reference count direct blocks (12) single indirect double indirect triple indirect mode uid gid atime
Recap: i-nodes Case study: ext FS The ext file system Second Extended Filesystem The main Linux FS before ext Evolved from Minix filesystem (via Extended Filesystem ) Features (4, 48, and 49) configured
More informationTricky issues in file systems
Tricky issues in file systems Taylor Riastradh Campbell campbell@mumble.net riastradh@netbsd.org EuroBSDcon 2015 Stockholm, Sweden October 4, 2015 What is a file system? Standard Unix concept: hierarchy
More informationUsing Model Checking to Find Serious File System Errors
Using Model Checking to Find Serious File System Errors JUNFENG YANG, PAUL TWOHEY, and DAWSON ENGLER Stanford University and MADANLAL MUSUVATHI Microsoft Research This article shows how to use model checking
More informationOperating System Concepts Ch. 11: File System Implementation
Operating System Concepts Ch. 11: File System Implementation Silberschatz, Galvin & Gagne Introduction When thinking about file system implementation in Operating Systems, it is important to realize the
More informationTowards Efficient, Portable Application-Level Consistency
Towards Efficient, Portable Application-Level Consistency Thanumalayan Sankaranarayana Pillai, Vijay Chidambaram, Joo-Young Hwang, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau 1 File System Crash
More informationTopics. File Buffer Cache for Performance. What to Cache? COS 318: Operating Systems. File Performance and Reliability
Topics COS 318: Operating Systems File Performance and Reliability File buffer cache Disk failure and recovery tools Consistent updates Transactions and logging 2 File Buffer Cache for Performance What
More informationAdvanced File Systems. CS 140 Feb. 25, 2015 Ali Jose Mashtizadeh
Advanced File Systems CS 140 Feb. 25, 2015 Ali Jose Mashtizadeh Outline FFS Review and Details Crash Recoverability Soft Updates Journaling LFS/WAFL Review: Improvements to UNIX FS Problems with original
More informationNetwork File System (NFS)
Network File System (NFS) Nima Honarmand User A Typical Storage Stack (Linux) Kernel VFS (Virtual File System) ext4 btrfs fat32 nfs Page Cache Block Device Layer Network IO Scheduler Disk Driver Disk NFS
More informationNFS: Naming indirection, abstraction. Abstraction, abstraction, abstraction! Network File Systems: Naming, cache control, consistency
Abstraction, abstraction, abstraction! Network File Systems: Naming, cache control, consistency Local file systems Disks are terrible abstractions: low-level blocks, etc. Directories, files, links much
More informationLecture 18: Reliable Storage
CS 422/522 Design & Implementation of Operating Systems Lecture 18: Reliable Storage Zhong Shao Dept. of Computer Science Yale University Acknowledgement: some slides are taken from previous versions of
More informationTolera'ng File System Mistakes with EnvyFS
Tolera'ng File System Mistakes with EnvyFS Lakshmi N. Bairavasundaram NetApp, Inc. Swaminathan Sundararaman Andrea C. Arpaci Dusseau Remzi H. Arpaci Dusseau University of Wisconsin Madison File Systems
More informationDistributed File Systems
Distributed File Systems Today l Basic distributed file systems l Two classical examples Next time l Naming things xkdc Distributed File Systems " A DFS supports network-wide sharing of files and devices
More informationAdministrative Details. CS 140 Final Review Session. Pre-Midterm. Plan For Today. Disks + I/O. Pre-Midterm, cont.
Administrative Details CS 140 Final Review Session Final exam: 12:15-3:15pm, Thursday March 18, Skilling Aud (here) Questions about course material or the exam? Post to the newsgroup with Exam Question
More informationJOURNALING FILE SYSTEMS. CS124 Operating Systems Winter , Lecture 26
JOURNALING FILE SYSTEMS CS124 Operating Systems Winter 2015-2016, Lecture 26 2 File System Robustness The operating system keeps a cache of filesystem data Secondary storage devices are much slower than
More informationOperating Systems. File Systems. Thomas Ropars.
1 Operating Systems File Systems Thomas Ropars thomas.ropars@univ-grenoble-alpes.fr 2017 2 References The content of these lectures is inspired by: The lecture notes of Prof. David Mazières. Operating
More informationCSE506: Operating Systems CSE 506: Operating Systems
CSE 506: Operating Systems Block Cache Address Space Abstraction Given a file, which physical pages store its data? Each file inode has an address space (0 file size) So do block devices that cache data
More informationLecture 10: Crash Recovery, Logging
6.828 2011 Lecture 10: Crash Recovery, Logging what is crash recovery? you're writing the file system then the power fails you reboot is your file system still useable? the main problem: crash during multi-step
More informationCOS 318: Operating Systems. Journaling, NFS and WAFL
COS 318: Operating Systems Journaling, NFS and WAFL Jaswinder Pal Singh Computer Science Department Princeton University (http://www.cs.princeton.edu/courses/cos318/) Topics Journaling and LFS Network
More informationDistributed File Systems. Directory Hierarchy. Transfer Model
Distributed File Systems Ken Birman Goal: view a distributed system as a file system Storage is distributed Web tries to make world a collection of hyperlinked documents Issues not common to usual file
More informationFilesystem. Disclaimer: some slides are adopted from book authors slides with permission
Filesystem Disclaimer: some slides are adopted from book authors slides with permission 1 Recap Directory A special file contains (inode, filename) mappings Caching Directory cache Accelerate to find inode
More informationCSE506: Operating Systems CSE 506: Operating Systems
CSE 506: Operating Systems File Systems Traditional File Systems FS, UFS/FFS, Ext2, Several simple on disk structures Superblock magic value to identify filesystem type Places to find metadata on disk
More informationJournaling versus Soft-Updates: Asynchronous Meta-data Protection in File Systems
Journaling versus Soft-Updates: Asynchronous Meta-data Protection in File Systems Margo I. Seltzer, Gregory R. Ganger, M. Kirk McKusick, Keith A. Smith, Craig A. N. Soules, and Christopher A. Stein USENIX
More informationPERSISTENCE: FSCK, JOURNALING. Shivaram Venkataraman CS 537, Spring 2019
PERSISTENCE: FSCK, JOURNALING Shivaram Venkataraman CS 537, Spring 2019 ADMINISTRIVIA Project 4b: Due today! Project 5: Out by tomorrow Discussion this week: Project 5 AGENDA / LEARNING OUTCOMES How does
More informationEIO: Error-handling is Occasionally Correct
EIO: Error-handling is Occasionally Correct Haryadi S. Gunawi, Cindy Rubio-González, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, Ben Liblit University of Wisconsin Madison FAST 08 February 28, 2008
More informationFile Systems II. COMS W4118 Prof. Kaustubh R. Joshi hdp://
File Systems II COMS W4118 Prof. Kaustubh R. Joshi krj@cs.columbia.edu hdp://www.cs.columbia.edu/~krj/os References: OperaXng Systems Concepts (9e), Linux Kernel Development, previous W4118s Copyright
More informationFilesystem. Disclaimer: some slides are adopted from book authors slides with permission 1
Filesystem Disclaimer: some slides are adopted from book authors slides with permission 1 Storage Subsystem in Linux OS Inode cache User Applications System call Interface Virtual File System (VFS) Filesystem
More informationFile System. Minsoo Ryu. Real-Time Computing and Communications Lab. Hanyang University.
File System Minsoo Ryu Real-Time Computing and Communications Lab. Hanyang University msryu@hanyang.ac.kr File Concept Directory Structure File System Structure Allocation Methods Outline 2 2 File Concept
More informationReview: FFS [McKusic] basics. Review: FFS background. Basic FFS data structures. FFS disk layout. FFS superblock. Cylinder groups
Review: FFS background 1980s improvement to original Unix FS, which had: - 512-byte blocks - Free blocks in linked list - All inodes at beginning of disk - Low throughput: 512 bytes per average seek time
More informationò Server can crash or be disconnected ò Client can crash or be disconnected ò How to coordinate multiple clients accessing same file?
Big picture (from Sandberg et al.) NFS Don Porter CSE 506 Intuition Challenges Instead of translating VFS requests into hard drive accesses, translate them into remote procedure calls to a server Simple,
More informationNFS. Don Porter CSE 506
NFS Don Porter CSE 506 Big picture (from Sandberg et al.) Intuition ò Instead of translating VFS requests into hard drive accesses, translate them into remote procedure calls to a server ò Simple, right?
More informationLogical disks. Bach 2.2.1
Logical disks Bach 2.2.1 Physical disk is divided into partitions or logical disks Logical disk linear sequence of fixed size, randomly accessible, blocks disk device driver maps underlying physical storage
More informationVirtual File System. Don Porter CSE 306
Virtual File System Don Porter CSE 306 History Early OSes provided a single file system In general, system was pretty tailored to target hardware In the early 80s, people became interested in supporting
More informationKLEE: Effective Testing of Systems Programs Cristian Cadar
KLEE: Effective Testing of Systems Programs Cristian Cadar Joint work with Daniel Dunbar and Dawson Engler April 16th, 2009 Writing Systems Code Is Hard Code complexity Tricky control flow Complex dependencies
More informationDa-Wei Chang CSIE.NCKU. Professor Hao-Ren Ke, National Chiao Tung University Professor Hsung-Pin Chang, National Chung Hsing University
Chapter 11 Implementing File System Da-Wei Chang CSIE.NCKU Source: Professor Hao-Ren Ke, National Chiao Tung University Professor Hsung-Pin Chang, National Chung Hsing University Outline File-System Structure
More informationEECS 482 Introduction to Operating Systems
EECS 482 Introduction to Operating Systems Winter 2018 Harsha V. Madhyastha Multiple updates and reliability Data must survive crashes and power outages Assume: update of one block atomic and durable Challenge:
More informationNetwork File Systems
Network File Systems CS 240: Computing Systems and Concurrency Lecture 4 Marco Canini Credits: Michael Freedman and Kyle Jamieson developed much of the original material. Abstraction, abstraction, abstraction!
More informationCS5460: Operating Systems Lecture 20: File System Reliability
CS5460: Operating Systems Lecture 20: File System Reliability File System Optimizations Modern Historic Technique Disk buffer cache Aggregated disk I/O Prefetching Disk head scheduling Disk interleaving
More informationReview: FFS background
1/37 Review: FFS background 1980s improvement to original Unix FS, which had: - 512-byte blocks - Free blocks in linked list - All inodes at beginning of disk - Low throughput: 512 bytes per average seek
More informationCSE 451: Operating Systems Winter Module 17 Journaling File Systems
CSE 451: Operating Systems Winter 2017 Module 17 Journaling File Systems Mark Zbikowski mzbik@cs.washington.edu Allen Center 476 2013 Gribble, Lazowska, Levy, Zahorjan In our most recent exciting episodes
More informationVirtual File System. Don Porter CSE 506
Virtual File System Don Porter CSE 506 History ò Early OSes provided a single file system ò In general, system was pretty tailored to target hardware ò In the early 80s, people became interested in supporting
More informationjvpfs: Adding Robustness to a Secure Stacked File System with Untrusted Local Storage Components
Department of Computer Science Institute of Systems Architecture, Operating Systems Group : Adding Robustness to a Secure Stacked File System with Untrusted Local Storage Components Carsten Weinhold, ermann
More informationTo Everyone... iii To Educators... v To Students... vi Acknowledgments... vii Final Words... ix References... x. 1 ADialogueontheBook 1
Contents To Everyone.............................. iii To Educators.............................. v To Students............................... vi Acknowledgments........................... vii Final Words..............................
More informationOPERATING SYSTEM TRANSACTIONS
OPERATING SYSTEM TRANSACTIONS Donald E. Porter, Owen S. Hofmann, Christopher J. Rossbach, Alexander Benn, and Emmett Witchel The University of Texas at Austin OS APIs don t handle concurrency 2 OS is weak
More information6.828 Fall 2007 Quiz II Solutions
Department of Electrical Engineering and Computer Science MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.828 Fall 2007 Quiz II Solutions The average was 80. 1 I Virtual Machines The x86 SIDT and LIDT instructions
More informationReminder from last time
Concurrent systems Lecture 7: Crash recovery, lock-free programming, and transactional memory DrRobert N. M. Watson 1 Reminder from last time History graphs; good (and bad) schedules Isolation vs. strict
More informationECE 598 Advanced Operating Systems Lecture 19
ECE 598 Advanced Operating Systems Lecture 19 Vince Weaver http://web.eece.maine.edu/~vweaver vincent.weaver@maine.edu 7 April 2016 Homework #7 was due Announcements Homework #8 will be posted 1 Why use
More informationCS5412: TRANSACTIONS (I)
1 CS5412: TRANSACTIONS (I) Lecture XVII Ken Birman Transactions 2 A widely used reliability technology, despite the BASE methodology we use in the first tier Goal for this week: in-depth examination of
More informationJFS Log. Steve Best IBM Linux Technology Center. Recoverable File Systems. Abstract. Introduction. Logging
JFS Log How the Journaled File System performs logging Steve Best sbest@us.ibm.com IBM Linux Technology Center Note: This paper is to appear in the Proceedings of the 4th Annual Linux Showcase & Conference,
More informationWhat is a file system
COSC 6397 Big Data Analytics Distributed File Systems Edgar Gabriel Spring 2017 What is a file system A clearly defined method that the OS uses to store, catalog and retrieve files Manage the bits that
More informationRCU. ò Dozens of supported file systems. ò Independent layer from backing storage. ò And, of course, networked file system support
Logical Diagram Virtual File System Don Porter CSE 506 Binary Formats RCU Memory Management File System Memory Allocators System Calls Device Drivers Networking Threads User Today s Lecture Kernel Sync
More informationLinux Filesystems Ext2, Ext3. Nafisa Kazi
Linux Filesystems Ext2, Ext3 Nafisa Kazi 1 What is a Filesystem A filesystem: Stores files and data in the files Organizes data for easy access Stores the information about files such as size, file permissions,
More informationPush-button verification of Files Systems via Crash Refinement
Push-button verification of Files Systems via Crash Refinement Verification Primer Behavioral Specification and implementation are both programs Equivalence check proves the functional correctness Hoare
More informationFall 2017 :: CSE 306. File Systems Basics. Nima Honarmand
File Systems Basics Nima Honarmand File and inode File: user-level abstraction of storage (and other) devices Sequence of bytes inode: internal OS data structure representing a file inode stands for index
More informationFile System Performance (and Abstractions) Kevin Webb Swarthmore College April 5, 2018
File System Performance (and Abstractions) Kevin Webb Swarthmore College April 5, 2018 Today s Goals Supporting multiple file systems in one name space. Schedulers not just for CPUs, but disks too! Caching
More informationIn This Lecture. Transactions and Recovery. Transactions. Transactions. Isolation and Durability. Atomicity and Consistency. Transactions Recovery
In This Lecture Database Systems Lecture 15 Natasha Alechina Transactions Recovery System and Media s Concurrency Concurrency problems For more information Connolly and Begg chapter 20 Ullmanand Widom8.6
More informationDepartment of Electrical Engineering and Computer Science MASSACHUSETTS INSTITUTE OF TECHNOLOGY Fall 2008.
Department of Electrical Engineering and Computer Science MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.828 Fall 2008 Quiz II Solutions 1 I File System Consistency Ben is writing software that stores data in
More informationCLASSIC FILE SYSTEMS: FFS AND LFS
1 CLASSIC FILE SYSTEMS: FFS AND LFS CS6410 Hakim Weatherspoon A Fast File System for UNIX Marshall K. McKusick, William N. Joy, Samuel J Leffler, and Robert S Fabry Bob Fabry Professor at Berkeley. Started
More informationTxFS: Leveraging File-System Crash Consistency to Provide ACID Transactions
TxFS: Leveraging File-System Crash Consistency to Provide ACID Transactions Yige Hu, Zhiting Zhu, Ian Neal, Youngjin Kwon, Tianyu Chen, Vijay Chidambaram, Emmett Witchel The University of Texas at Austin
More informationDistributed Systems. Lec 9: Distributed File Systems NFS, AFS. Slide acks: Dave Andersen
Distributed Systems Lec 9: Distributed File Systems NFS, AFS Slide acks: Dave Andersen (http://www.cs.cmu.edu/~dga/15-440/f10/lectures/08-distfs1.pdf) 1 VFS and FUSE Primer Some have asked for some background
More informationCSE 333 Lecture 9 - storage
CSE 333 Lecture 9 - storage Steve Gribble Department of Computer Science & Engineering University of Washington Administrivia Colin s away this week - Aryan will be covering his office hours (check the
More informationThe Google File System
October 13, 2010 Based on: S. Ghemawat, H. Gobioff, and S.-T. Leung: The Google file system, in Proceedings ACM SOSP 2003, Lake George, NY, USA, October 2003. 1 Assumptions Interface Architecture Single
More information1 / 23. CS 137: File Systems. General Filesystem Design
1 / 23 CS 137: File Systems General Filesystem Design 2 / 23 Promises Made by Disks (etc.) Promises 1. I am a linear array of fixed-size blocks 1 2. You can access any block fairly quickly, regardless
More informationVFS Interceptor: Dynamically Tracing File System Operations in real. environments
VFS Interceptor: Dynamically Tracing File System Operations in real environments Yang Wang, Jiwu Shu, Wei Xue, Mao Xue Department of Computer Science and Technology, Tsinghua University iodine01@mails.tsinghua.edu.cn,
More informationECE 550D Fundamentals of Computer Systems and Engineering. Fall 2017
ECE 550D Fundamentals of Computer Systems and Engineering Fall 2017 The Operating System (OS) Prof. John Board Duke University Slides are derived from work by Profs. Tyler Bletsch and Andrew Hilton (Duke)
More informationGFS: The Google File System. Dr. Yingwu Zhu
GFS: The Google File System Dr. Yingwu Zhu Motivating Application: Google Crawl the whole web Store it all on one big disk Process users searches on one big CPU More storage, CPU required than one PC can
More information2. PICTURE: Cut and paste from paper
File System Layout 1. QUESTION: What were technology trends enabling this? a. CPU speeds getting faster relative to disk i. QUESTION: What is implication? Can do more work per disk block to make good decisions
More informationAdvanced UNIX File Systems. Berkley Fast File System, Logging File System, Virtual File Systems
Advanced UNIX File Systems Berkley Fast File System, Logging File System, Virtual File Systems Classical Unix File System Traditional UNIX file system keeps I-node information separately from the data
More informationProcesses and Threads Implementation
Processes and Threads Implementation 1 Learning Outcomes An understanding of the typical implementation strategies of processes and threads Including an appreciation of the trade-offs between the implementation
More informationLinux Journaling File System: ext3 Shangyou zeng Physics & Astronomy Dept., Ohio University Athens, OH, 45701
Linux Journaling File System: ext3 Shangyou zeng Physics & Astronomy Dept., Ohio University Athens, OH, 45701 Abstract In Red Hat Linux 7.2, Red Hat provides the first officially supported journaling file
More informationLecture 21: Reliable, High Performance Storage. CSC 469H1F Fall 2006 Angela Demke Brown
Lecture 21: Reliable, High Performance Storage CSC 469H1F Fall 2006 Angela Demke Brown 1 Review We ve looked at fault tolerance via server replication Continue operating with up to f failures Recovery
More informationFile Systems. Chapter 11, 13 OSPP
File Systems Chapter 11, 13 OSPP What is a File? What is a Directory? Goals of File System Performance Controlled Sharing Convenience: naming Reliability File System Workload File sizes Are most files
More informationFilesystem. Disclaimer: some slides are adopted from book authors slides with permission 1
Filesystem Disclaimer: some slides are adopted from book authors slides with permission 1 Recap Blocking, non-blocking, asynchronous I/O Data transfer methods Programmed I/O: CPU is doing the IO Pros Cons
More informationMidterm evaluations. Thank you for doing midterm evaluations! First time not everyone thought I was going too fast
p. 1/3 Midterm evaluations Thank you for doing midterm evaluations! First time not everyone thought I was going too fast - Some people didn t like Q&A - But this is useful for me to gauge if people are
More informationMotivation. What s the Problem? What Will we be Talking About? What s the Solution? What s the Problem?
1 Checking System Rules Using System-Specific, Programmer-Written Compiler Extensions Dawson Engler Benjamin Chelf Andy Chou Seth Hallem Stanford University Matthew Thornton November 9, 2005 2 Motivation
More information