File Systems. CS170 Fall 2018

Transcription

1 File Systems CS170 Fall 2018

2 Table of Content File interface review File-System Structure File-System Implementation Directory Implementation Allocation Methods of Disk Space Free-Space Management Contiguous allocation Block-oriented indexing Unix inode structure

3 Files File concept: Contiguous logical address space in a persistent storage (e.g. disk). File structure None - sequence of words, bytes Simple record structure Lines Fixed length Variable length Complex Structures: Formatted document Who decides the structure: Operating system Program

4 File Attributes Name only information kept in human-readable form Identifier unique tag (number) identifies file within file system Type needed for systems that support different types Location pointer to file location on device Size current file size Protection controls who can do reading, writing, executing Time, date, and user identification data for protection, security, and usage monitoring Information about files are kept in the directory structure, which is maintained on the disk

5 File Operations Create Open(F i ) search the directory structure on disk for entry F i move the content of entry to memory Close (F i ) move the content of entry F i in memory to directory structure on disk Write Read Reposition within file (e.g. seek) Delete Truncate

6 Access Methods Sequential Access read next write next reset Direct Access n = relative offset read n write n position to n read next write next rewrite n

7 File System Abstraction Directory Group of named files or subdirectories Mapping from file name to file metadata location Path String that uniquely identifies file or directory Ex: /cse/www/education/courses/cse451/12au Links Hard link: link from name to metadata location Soft link: link from name to alternate name Mount Mapping from name in one file system to root of another

8 UNIX File System API create, link, unlink, createdir, rmdir Create file, link to file, remove link Create directory, remove directory open, close, read, write, seek Open/close a file for reading/writing Seek resets current position fsync File modifications can be cached fsync forces modifications to disk (like a memory barrier)

9 File System Interface filedescriptor = open(pathname, flags) UNIX file open is a Swiss Army knife: Open the file, return file descriptor Options: if file doesn t exist, return an error If file doesn t exist, create file and open it If file does exist, return an error If file does exist, open file If file exists but isn t empty, nix it then open If file exists but isn t empty, return an error Memory Does child process inherit file descriptors of parent process? Yes Duplicate Parent Child 4 5 File Metadata on a file

10 Example of Linux read, write, and lseek int main() { int file=0; char buffer[15]; if((file=open("testfile.txt",o_rdonly)) < -1) return 1; if(read(file,buffer,14)!= 14) return 1; printf("%s\n",buffer); } if(lseek(file,5,seek_set) < 0) return 1; if(read(file,buffer,19)!= 14) return 1; printf("%s\n",buffer); return 0; $ cat testfile.txt This is a test file $./testing This is a test is a test file

11 Protection File owner/creator should be able to control: what can be done by whom Types of access Read Write Execute Append Delete List Example in Linux

12 Access Lists and Groups in Linux Mode of access: read, write, execute Three classes of users RWX a) owner access 7 Þ RWX b) group access 6 Þ RWX c) public access 1 Þ Ask manager to create a group (unique name), say G, and add some users to the group. For a particular file (say game) or subdirectory, define an appropriate access. owner group public chmod 761 game Attach a group to a file chgrp G game

13 Windows Access-Control List Management

14 Directory Structure A collection of nodes containing information about all files Directory Files F 1 F 2 F 3 F 4 F n Both the directory structure and the files reside on disk Backups of these two structures are kept on tapes

15 A Typical File-system Organization on a Disk Partition

16 Operations Performed on Directory Search for a file Create a file Delete a file List a directory Rename a file Traverse the file system

17 Directory with single-level or two-level A single directory for all users Two -level

18 Tree-Structured Directories

19 Table of Content File interface review File-System Structure File-System Implementation Directory Implementation Allocation Methods of Disk Space Free-Space Management Contiguous allocation Block-oriented indexing Unix inode structure

20 Building a File System File System: Layer of OS that transforms block interface of disks (or other block devices) into Files, Directories, etc. File System Components Disk Management: collecting disk blocks into files Naming: Interface to find files by name, not by blocks Protection: Layers to keep data secure Reliability/Durability: Keeping of files durable despite crashes, media failures, attacks, etc User vs. System View of a File User s view: Durable Data Structures System call interface: Collection of Bytes (UNIX) System s view (inside OS): Collection of blocks (a block is a logical transfer unit, while a sector is the physical transfer unit on disk) Block size ³ sector size; in UNIX, block size is 4KB Kubiatowicz s cs162 UCB

21 Design Challenges for Performance, Flexibility and Reliability Index structure, index granularity, and free space How do we locate the blocks of a file? What block size do we use? How do we find unused blocks on disk? Easy management of files (growth, truncation, etc) Performance and Flexibility Maximize sequential performance while supporting efficient random access to file? How do we preserve spatial locality? Reliability What if machine crashes in middle of a file system op?

22 File System Workload Studying workload characteristics can help feature prioritization or optimization of design What should be considered? File sizes Are most files small or large? Which accounts for more total storage: small or large files? Some files have a pre-defined size at creation Some files start small and grow over time Ex: program stdout, system logs File access patterns Random access vs sequential access? Most files are read/written sequentially Some files are read/written randomly Ex: database files, swap files

23 File System Design For small files: Small blocks for storage efficiency Concurrent ops more efficient than sequential Files used together should be stored together For large files: Storage efficient (large blocks) Contiguous allocation for sequential access Efficient lookup for random access May not know at file creation Whether file will become small or large Whether file is persistent or temporary Whether file will be used sequentially or randomly

24 File-System Implementation Directories and index structure Special root block at a specific location contains the root directory Directory structure organizes the files Given file name, find a file number Given a file number which contains the file structure info, locate blocks of this file. Per-file File Control Block (FCB) contains many details about the file Called i-node in Linux/Unix Typical information

25 Layered File System Virtual File Systems (VFS) provide an object-oriented way of implementing file systems. VFS allows the same system call interface (the API) to be used for different types of file systems. The API is to the VFS interface, rather than any specific type of file system.

26 Schematic View of Virtual File System

27 Directory Implementation Linear list of file names with pointer to the data blocks. simple to program time-consuming to execute Hash Table linear list with hash data structure. decreases directory search time collisions situations where two file names hash to the same location Search tree

28 How do we actually access files? All information about a file contained in its file header File control block: UNIX calls this an inode Inodes are global resources identified by index ( inumber, or inode number) Once you load the header structure, all blocks of file are locatable The maximum number of inodes is fixed at file system creation, limiting the maximum number of files the file system can hold. A typical allocation heuristic for inodes is one percent of total file system size.

29 Unix File System with inode structure Inode Array Inode Triple Indirect Blocks Double Indirect Blocks Indirect Blocks Data Blocks file_number File Metadata Direct Pointers Indirect Pointer Dbl. Indirect Ptr. Tripl. Indrect Ptr. 2 Minimum I/O accesses to a file: i-node header access + data block access

30 Question: how does the user ask for a particular file? One option: user specifies an inode by a number (index). Imagine: open( ) Better option: specify by textual name Have to map name inumber Another option: Icon This is how Apple made its money. Graphical user interfaces. Point to a file and click A. Joseph UCB CS162. Spr 2014

31 Named Data in a File System Directories are files!

32 Directory Layout Directory stored as a file Linear search to find filename (small directories)

33 Large Directories: B Trees

34 Recursive Filename Lookup: /home/tom/foo.txt Directory lookup: / Directory lookup: home Directory lookup:tom File fetch

35 How many disk accesses to read the first data block of /home/tom/foo.txt? 8 Read in file header for root / (fixed spot on disk) Read in first data block for root / Table of file name/index pairs. Search linearly ok since directories typically very small Read in file header for home Read in first data block for home ; search for tom Read in file header for tom Read in first data block for tom ; search for foo.txt Read in file header for count.txt Fetch #1 data block of count.txt Current working directory: Per-address-space pointer to a directory (inode) used for resolving file names Allows user to access relative filename quickly

36 In-Memory File System Structures Open system call: Resolves file name, finds file control block (inode) Makes entries in per-process and system-wide tables Returns index (called file descriptor or file handle ) in open-file table

37 Open Files Several pieces of data are needed to manage open files: File pointer: pointer to last read/write location, per process that has the file open File-open count: counter of number of times a file is open to allow removal of data from open-file table when last processes closes it Disk location of the file: cache of data access information Access rights: per-process access mode information Open file locking is provided by some systems Mediates access to a file

38 In-Memory File System Structures Read/write system calls: Use file handle (descriptor) to locate inode Perform appropriate reads or writes

39 Allocation of Disk Blocks An allocation method refers to how disk blocks are allocated for files: Contiguous allocation Linked allocation Indexed allocation

40 Contiguous Allocation of Disk Space

41 Contiguous Allocation Each file occupies a set of contiguous blocks on the disk Advantages: Simple only starting location (block #) and length (number of blocks) are required Fast Random access Disadvantages: Not easy to grow files. Waste in space (e.g. external fragmentation)

42 Linked Allocation Each file is a linked list of disk blocks: blocks may be scattered anywhere on the disk.

43 Microsoft File Allocation Table (FAT) Linked list index structure Simple, easy to implement Still widely used (e.g., thumb drives) File table: Linear map of all blocks on disk Each file is a linked list of blocks

44 FAT

45 FAT Pros: Easy to find free block Easy to append to a file Easy to delete a file Cons: Small file access is slow Random access is very slow Fragmentation File blocks for a given file may be scattered Files in the same directory may be scattered Problem becomes worse as disk fills

46 Direct Data Pointers: One-level Indexed Allocation Place all direct data pointers together into the index block

47 Example of One-level Indexed Allocation

48 One-level Indexed Allocation (Cont.) Advantages Support random access No external fragmentation. Disadvantages: Space overhead: need 1 block for index table Maximum file size? Assume each block is 4KB index block holds 1024 entries (4B/entry) 1024x block size = 4MB

49 Two-level Indexed Allocation: Single indrection 1K entries 1K entries 4KB data! Level 1index Indirect pointers Maximum size? 4GB index table: Direct pointers File data

50 Hybrid multi-level scheme: UNIX file system Key idea: efficient for small files, but still allow big files File header contains pointers called an inode in UNIX File Header format: First 10-12: direct data pointer 1 indirect block 1 doubly indirect block 1 triple indirect block

51 Berkeley UNIX FFS (Fast File System) i-node metadata File owner, access permissions, access times, Each file block: 4KB i-node header contains 15 pointers Set of 12 direct data pointers With 4KB blocks => max size of 48KB files 1 indirect block pointer Indirect block: 4KB contains 1K entries àdata blocks => 4MB (+48KB) 1 double indirect pointer 1K*1K blocks 1 triple indirect pointer 1K*1K*1K blocks Maximum size: 4TB + 4GB + 4MB + 48KB

52 Unix Design follows Characteristics of Files Most files are small, growing numbers of files over time. Most of the space is occupied by the rare big ones.

53 Benefits for small files Small files: 12 pointers direct to data blocks Direct pointers Inode Array 4kB blocks Þ sufficient for files up to 48KB File Metadata Inode Triple Indirect Blocks Double Indirect Blocks Indirect Blocks Data Blocks Direct Pointers Indirect Pointer Dbl. Indirect Ptr. Tripl. Indrect Ptr.

54 Handling of large files Large files: 1,2,3 level indirect pointers Indirect pointers Inode Array - point to a disk block containing only pointers - 4 kb blocks => 1024 ptrs => 4 level 2 => 4 level 3 => 4 level 4 File Metadata Direct Pointers Inode Triple Indirect Blocks Double Indirect Blocks Indirect Blocks Data Blocks Indirect Pointer Dbl. Indirect Ptr. Tripl. Indrect Ptr. 48 KB +4 MB +4 GB +4 TB

55 Free-Space Management Bitmap (n blocks) n-1 "#$ bit[i] = 0 Þ block[i] free 1 Þ block[i] occupied Block number calculation (number of bits per word) * (number of 0-value words) + offset of first 1 bit

56 Performance Optimization Disk cache separate section of main memory for frequently used blocks Read-ahead (prefetching) techniques to optimize sequential access improve PC performance by dedicating section of memory as virtual disk, or RAM disk

57 Disk Buffer Cache Caches frequently used disk blocks Speedup file system I/O Memory-mapped I/O can also use

58 Question: File Systems Q1: True _ False _ inumber is the id of a block Q2: True _ False _ inumber is a file description returned in open system call. Q3: True _ False _ Typically, directories are stored as files Q4: True _ False _ With FAT, pointers are maintained in the data blocks Q5: True _ False _ Unix file system is more efficient than FAT for random access

59 Question: File Systems Q1: True _ False _x inumber is the id of a block Q2: True _ False _ x inumber is a file description returned in open system call. Q3: True _x False _ Typically, directories are stored as files Q4: True _ False _x With FAT, pointers are maintained in the data blocks Q5: True _x False _ Unix file system is more efficient than FAT for random access

60 Summary File access sequential random File-System Structure Layered file system Multi-level directory Allocation Methods of Disk Space Linked allocation Contiguous allocation Block-oriented indexing and maximum file size One-level vs. multi-level Unix inode, inumber