Content File Systems Hengming Zou, Ph.D. Files Disk scheduling, file structure, indexed files Directories File system implementation File system performance Example file systems 2006-4-29 1 2006-4-29 2 Motivation All applications need store or retrieve information Where can information be kept? Motivation Process s address space while it is running Limited size, volatile, not sharable Or it can be stored on disk Persistent, sharable, large size But using disk is not easy for ordinary users The solution is: file system 2006-4-29 3 2006-4-29 4 File Systems File Systems What is a file system? A file system is an OS abstraction to make the disk (or other devices) easier to use Shield users from the details of how and where the information is stored, and how disks actually work Users File system Disks Virtual interface Physical interface 2006-4-29 5 2006-4-29 6 Information Server 1
File System Characteristics Store large amounts of data Survive the termination of the process using it Multiple processes can access it concurrently File Systems Files are managed by operating system How are files structured? How are files named? Used? Protected? How are files implemented? 2006-4-29 7 2006-4-29 8 Views of File Systems Views of File Systems Users concern how files appear to them: What constitute a file? How files are named? What operations are allowed on files? Etc. OS designers concern how files are implemented How are files structured? How to keep track of free storage? How many sectors are in a logical block? First let s look from the users perspective 2006-4-29 9 2006-4-29 10 Naming How to specify which file you want to access? Naming Typically, the user provides a symbolic name e.g. English name, icon OS translates name to a numeric file header or disk location of the file header 2006-4-29 11 2006-4-29 12 Information Server 2
Naming Naming Alternative is to describe the contents of the file have the OS find the name by looking through the files for that content usually done by databases on top of file systems Lots of ways to translate from symbolic name to file header s disk block # e.g. hash table, expandable array, etc. the structure that does the mapping is called a directory 2006-4-29 13 2006-4-29 14 Naming Exactly rules for naming differing by systems Normally 2-8 characters But can be 255 characters A file name can also have a extension To convey information about the file.c for C file,.bak for backup file, etc. File Structure File can be structured in any of several ways Common ways to structure files are: byte sequence, record sequence, tree 2006-4-29 15 2006-4-29 16 File Structure File Types Most OS supports several types of files Regular files, directories UNIX also have character and block special files To model serial I/O devices and disks 2006-4-29 17 2006-4-29 18 Information Server 3
File Types Regular files are either ASCII or binary ASCII files contain lines of text Binary file are encoded files i.e. need special programs to open it File Types Executable file Archive file 2006-4-29 19 2006-4-29 20 File Access Only two types of accesses available: sequential or random access Early OS provides only sequential access Sequential Access Read all bytes/records from the beginning Cannot jump around, could rewind or back up Convenient when medium was tape 2006-4-29 21 2006-4-29 22 Random Access Bytes/records read in any order Essential for data base systems Read can be Move file marker (seek), then read or Read and then move file marker File Attributes Every file has a name and its data In addition, all OS associate other info to files These other information are called attributes 2006-4-29 23 2006-4-29 24 Information Server 4
File Attributes File Attributes Attribute Meaning Attribute Meaning Protection Who can access the file and in what way Temporary flag 0 for normal; 1 for delete file on process exit Password Password needed to access the file Lock flags 0 for unlocked; nonzero for locked Creator ID of the person who created the file Record length Number of bytes in a record Owner Current owner Key position Offset of the key within each record Read-only flag 0 for read/write; 1 for read only Key length Number of bytes in the key field Hidden flag 0 for normal; 1 for do not display in listings Creation time Date and time the file was created System flag 0 for normal files; 1 for system files Time of last access Date and time the file was last accessed Archive flag 0: has been backed up; 1: need to be backed up Time of last change Data and time the file has last changed ASCII/binary flag 0 for ASCII file; 1 for binary file Current size Number of bytes in the file Random access flag 0 for sequential access only; 1 for random access Maximum size Number of bytes the file may grow to 2006-4-29 25 2006-4-29 26 File Operations An Example Using File System Calls Create/delete Open/close Read/write/append Seek Get attributes /set attributes Rename /* file copy program. Error checking and reporting is minimum */ #include <sys/types.h> /* include necessary header files */ #include <fcntl.h> #include <stdlib.h> #include <unistd.h> int main(int argc, char *argv[]); /*ANSI prototype */ #define BUF_SIZE 4096 /* use a buffer size of 4096 bytes */ #define OUTPUT_MODE 0700 /* protection bits for output file */ 2006-4-29 27 2006-4-29 28 An Example Using File System Calls An Example Using File System Calls int main(int argc, char *argv[]) { int int_fd, out_fd, rd_count, wt_count; char buffer[buf_size]; if (argc!=3) exit(1) /*syntax error if argc is not 3 */ /* open input file and create output file */ in_fd = open(argv[1],o_rdonly); /* open source file*/ if (in_fd<0) exit(2) /*if cannot open, exit */ out_fd=create(argv[2],output_mode); /* create destination file*/ if (out_fd<0) exit(3); /* if cannot create, exit */ while (TRUE) { /* copy loop */ rd_count=read(in_fd,buffer,buf_size); /*read a block of data*/ if (rd_count<=0) break; /*if end of file or error, exit loop*/ wt_count=write(out_fd,buffer,rd_count); /*write data*/ if(wt_count<=0) exit(4); /*wt_count<=0 is an error */ } close (in_fd); close(out_fd); /*close the files */ if(rd_count==0) exit(0); /*no error on last read */ else exit(5); /*error on last read */ } 2006-4-29 29 2006-4-29 30 Information Server 5
Memory-Mapped Files The previous example is cumbersome to some Because have make many system calls One solution is use memory-mapped files Started by MULTICS Key idea Map a file into memory at a virtual address Memory-Mapped Files After mapping, access a file like access memory Ex: 64KB file is mapped to virtual address 512K Reads the content at byte 512K gets byte 0 of the file Write to 512K+10 modifies byte 10 of the file When process terminates, modified file is left on disk 2006-4-29 31 2006-4-29 32 Directories Directories Directories are used to keep track of files Also called folders Directories are themselves files! A directory contains a mapping for a set of files name => file header s disk block # for that file Often a simple array of entries (name, file header s disk block #) The table is stored in a normal file as normal data E.g. ls can be implemented by reading this file and parsing its contents 2006-4-29 33 2006-4-29 34 Directories Can often treat directories and files in same way Use same storage structure to store their data Directory entries can point to either a file or another directory Directories Directories are typically organized into hierarchical structure Directory A has a mapping for a bunch of files and directories in directory A There are always a root directory in system 2006-4-29 35 2006-4-29 36 Information Server 6
Hierarchical Directory Systems Directories For example: /zou/cs307/names / is root directory contains a list of files and other directories for each file/directory in /, / has a mapping from name to file header s disk block # one of these entries is zou 2006-4-29 37 2006-4-29 38 Directories zou is a directory entry within the / directory contains a list of files and directories one of the directories in /zou is cs307 Directories /zou/cs307 is a directory within the /zou directory contains a list of files and other directories one of the files it lists is names 2006-4-29 39 2006-4-29 40 Directories How many disk I/Os to access the first byte of /zou/cs307/names? Path Names Need a way to specifying file names in the tree Absolutely path name A path from root to the file, i.e. /zou/cs307/file1 Relatively path name A path relative from current work directory Assume /zou is the work directory, then cs307/file1 2006-4-29 41 2006-4-29 42 Information Server 7
Current Working Directory Path Names Allows a user to specify a file name relative to the current directory Allows the system to access a file without traversing the entire directory from / Reducing positioning overhead by taking advantage of usage patterns A UNIX directory tree 2006-4-29 43 2006-4-29 44 Directory Operations Create, Delete Opendir, Closedir Readdir Rename Link, Unlink Link and Unlink Link allows a file to appear in more than 1 place Link specifies a existing file and a path name Link creates a link from the file to path name It also increment a counter in the file s i-node 2006-4-29 45 2006-4-29 46 Link and Unlink Unlink removes a directory entry for a file If the last link for the file, the file is removed If file is in multiple directory, decrement counter File System Implementation Now let s look at from designers perspective 2006-4-29 47 2006-4-29 48 Information Server 8
File System Layout File systems are stored on disks Most disks are divided into partitions Each partition has its own file systems File System Layout Sector 0 of a disk is called MBR Mater Boot Record MBR is used to boot the computer The end of MBR contains the partition table Giving starting and ending addresses of each partition One of the partitions is marked as active in the table 2006-4-29 49 2006-4-29 50 File System Layout At boot time, BIOS reads in the MBR MBR then locates the active partition and reads in the first block, the boot block, and execute it Boot block in turns loads the OS in the partition File System Layout Other than the boot block, layout of disk partition varies greatly from file system to file system Often a file system have a super block right after the boot block that contains the key parameters about the file system on the disk 2006-4-29 51 2006-4-29 52 File System Layout Implementing Files Keep track of which disk block belongs to which file is an important issues in file implementation How to organize files on disk? What data structure is the right one to use when storing a file on disk? Three common approaches are possible: Contiguous allocation, Linked list allocation, i-node A possible file system layout 2006-4-29 53 2006-4-29 54 Information Server 9
Contiguous Allocation Contiguous Allocation Store a file in one contiguous segment on disk Sometimes called an extent User must declare the size of the file in advance Use some kind of fitting algorithm to locate an area of disk that can store the file Contiguous allocation of disk space for 7 files State of the disk after files D and E have been removed 2006-4-29 55 2006-4-29 56 Contiguous Allocation File header is very simple: Starting block #, file size Equivalent to base & bounds for memory management Contiguous Allocation Pros and Cons + fast sequential access no seeks between consecutive disk blocks + easy random access can easily calculate the disk location of any file block 2006-4-29 57 2006-4-29 58 Contiguous Allocation Pros and Cons - external fragmentation - wastes space if user declares a bigger size - hard to grow files Linked List Allocation Keep each file as a linked list of disk blocks First word of each block points to the next one Disk blocks need not be adjacent 2006-4-29 59 2006-4-29 60 Information Server 10
Linked List Allocation Linked List Allocation Pros and Cons + no space lost to disk fragmentation + Fast sequential access - extremely slow random access - amount of data in block is not a power of 2 Storing a file as a linked list of disk blocks 2006-4-29 61 2006-4-29 62 Linked List Allocation Linked List Allocation How to solve the problems? Use File Allocation Table (FAT) Take the pointer word from each disk block Putting it in a table in memory Linked list allocation using a file allocation table (FAT) in RAM 2006-4-29 63 2006-4-29 64 Indexed Files Indexed Files User (or system) declares max # of blocks in file; System allocates a file header with an array of pointers big enough to point to that number of blocks The header is called an i-node The file header is load into memory when open File Header Also called i-node File block # 0 1 2 3 Disk block # 18 50 8 15 Equivalent to a page table for MMU Usually contain additional information such as attributes 2006-4-29 65 2006-4-29 66 Information Server 11
Indexed Files Indexed Files Pros and Cons + can easily grow file up to the # of blocks allocated in file header + easy random access - lots of seeks for sequential access - hard to grow file bigger than initially allocated in the file header How to grow files sizes? The solution is to use multi-level index files Level 1 node Level 2 node Disk block 18 Disk block 50 Disk block 8 Disk block 15 Level 2 Node 2006-4-29 67 2006-4-29 68 Multi-level Indexed Files This allows really big files without wasting header space for small files How many accesses to get 1 block of data? how to solve? Use non-uniform depth multi-level indexed files Multi-level Indexed Files Non-uniform depth multi-level indexed files Level 1 node Disk block 18 Disk block 50 Level 2 node Disk block 8 Disk block 15 Disk block 75 Disk block 40 2006-4-29 69 2006-4-29 70 Multi-level Indexed Files Implementing Files Pros and cons + files can easily expand + small files don t pay full overhead of deep trees - lots of indirect blocks for big files - lots of seeks for sequential access An example i-node 2006-4-29 71 2006-4-29 72 Information Server 12
Implementing Directories Directory contains entries about files in it Main function is to map the ASCII name of the file onto the information needed to locate the data Name disk address of the entire file Name # of first block (for linked list schemes) Name the number of the i-node Implementing Directories Where to keep the attributes of files? Two approaches available: In directory entry In the i-node structure 2006-4-29 73 2006-4-29 74 Implementing Directories Implementing Directories disk addresses and attributes in directory entry Directory in which each entry just refers to an i-node What happens if a file name is long and cannot fit into the fixed size directory entry? Two ways of handling long file names in directory (a) In-line (b) In a heap 2006-4-29 75 2006-4-29 76 Implementing Directories Implementing Directories If very long directories, various techniques can be employed to speed up the search process Use hashing table Cache the results of searches In-line In heap 2006-4-29 77 2006-4-29 78 Information Server 13
Shared Files Shared Files Shared files are implemented with symbolic links Two users share a file by linking to the same file # of links to a file is kept in i-node File system containing a shared file 2006-4-29 79 2006-4-29 80 Shared Files Shared Files Symbolic links cause problems that must be solved Inconsistencies with block count of the file File remains while owner has deleted the file Problems with backup program Prior to linking After link is created Original owner removes file 2006-4-29 81 2006-4-29 82 Mounting Combining (mounting) multiple disks into a single file system Each disk has a file system each disk s file system starts at its own root (/) Mounting Can tie several disks together into a single file system have an entry in one directory point to the root directory of another disk s file system this entry is called a mount point 2006-4-29 83 2006-4-29 84 Information Server 14
Example: zou s Workstation % ls / afs/ kernel/ sbin/ bin@ lib@ template-server@ core lost+found/ template-server-ro@ dev/ nfs/ tmp/ devices/ opt/ usr/ etc/ platform/ var/ export/ proc/ vol/ home/ root/ Example: zou s Workstation /bin is a normal directory in / /usr points to the root file system of disk 1 /var points to the root file system of disk 2 /afs points to the root of a distributed file system 2006-4-29 85 2006-4-29 86 Mounting Windows: disks are made visible under my_computer/ C,D,E Unix: can mount disks at any place in the directory hierarchy Disk Space Management Files are normally stored on disks Thus disk space management is important The problem is similar to memory space management Contiguous allocation is hard to file to grow Linked list allocation is slow on sequential access But more severe because disk access is much slower 2006-4-29 87 2006-4-29 88 Block Size Block Size Most file systems use fixed-block allocation scheme How big should a file block be? Too big means internal waste Low space utilization Too small means frequent disk seek (may be rotation) Low data transfer rate Block size bytes Assume all files are of size 2KB 2006-4-29 89 2006-4-29 90 Information Server 15
Keep Track of Free Blocks The second problem in disk space management Two methods can be employed: Linked list implementation Free blocks are linked in a list Bitmap implementation A bitmap indicating if a block is free or not Disk Quotas Prevents people from getting too much disk space Use a second table in the open file table structure 2006-4-29 91 2006-4-29 92 Disk Quotas File System Reliability Two meanings: Durability: Data in a file system must be permanent Implemented through backup and replication Consistency: Data in a file system must be good Implemented through logging, transactions, and shadowing Quotas for keeping track of each user s disk use 2006-4-29 93 2006-4-29 94 File System Durability File System Durability From the data s point of view, there are two backup approaches available: Logical backup Backups only specified directories and files Physical backup Backup everything on a disk File that has not changed A file system to be dumped 2006-4-29 95 2006-4-29 96 Information Server 16
File System Durability File System Consistency Multi-step updates cause problems for consistency if crash happens in the middle E.g. transfer $100 from my account to Janet s 1. deduct $100 from my account 2. add $100 to Janet s account What happens if you crash between step 1 and 2? Bit maps used by the logical dumping algorithm 2006-4-29 97 2006-4-29 98 File System Consistency E.g. create (empty) new file 1. update the directory to point to the new file header 2. write new file header to disk E.g. move file from 1 directory to another 1. delete file from old directory 2. add file to new directory Block Read Ahead Read more blocks than are requested In the hope that they will be needed soon Since data transfer time is negligible compared to disk seek and rotation time (to an extent) Read ahead impose little or no overhead This technique helps only if access is sequential 2006-4-29 99 2006-4-29 100 Reduce Disk Arm Motion File System Performance Seek time dominates disk access time Reducing disk arm motion improved performance Idea is to put i-node closer to actual file data Normally all i-nodes go together, and all data go together I-nodes placed at start of disk Disk divided into cylinder groups each with its own blocks and i- nodes 2006-4-29 101 2006-4-29 102 Information Server 17
Log-Structured File Systems With CPUs faster, memory larger disk caches can also be larger increasing # of read requests can come from cache thus, most disk accesses will be writes Log-Structured File Systems LFS Strategy structures entire disk as a log have all writes initially buffered in memory periodically write these to the end of the disk log when file opened, locate i-node, then find blocks 2006-4-29 103 2006-4-29 104 Example File Systems The MS-DOS File System Windows 98 File System UNIX V7 File System MS-DOS File Systems Use 8+3 upper case file names DOS 1.0 limited to one directory Later versions allows hierarchical file system Directory can nested to an arbitrary depth Directories are variable sized But directory entries are fixed size of 32 bytes 2006-4-29 105 2006-4-29 106 MS-DOS File Systems MS-DOS File Systems The highest expressible year in MS-DOS is 2107 The counter starts at 1980 Use file allocation table to keep track of file addresses Each directory entry only gives the first block number Which is used to index into a the FAT table The MS-DOS directory entry 2006-4-29 107 2006-4-29 108 Information Server 18
MS-DOS File Systems Maximum Partition in MS-DOS FAT file system has three versions for MS-DOS: Block size FAT-12 FAT-16 FAT-32 FAT-12, FAT-16, FAT-32 Each differs from the bits a disk address contains FAT-32 introduced with 2nd release of Windows 95 Disk block numbers are multiples of 512 bytes Can be different for each partition Allowable block size differing by variants 0.5 KB 1 KB 2 KB 4 KB 8 KB 16 KB 2 MB 4 MB 8 MB 16 MB forbidden forbidden forbidden forbidden 128 MB 256 MB 512 MB 1024 MB forbidden forbidden forbidden 1 TB 2 TB 2 TB 32 KB forbidden 2048 MB 2 TB 2006-4-29 109 2006-4-29 110 Windows 98 File System Windows 98 File System To accommodate long file names i.e. up to 255 characters To provide backwards compatibility to MS-DOS files i.e. must use the MS-DOS directory structure Idea: keep two names for each file i.e. create a MS-DOS name from the given (long) name Both name are then stored in the directory structure Bytes The extended MOS-DOS directory entry used in Windows 98 2006-4-29 111 2006-4-29 112 Windows 98 File System Windows 98 File System Bytes Checksum An entry for (part of) a long file name in Windows 98 An example of how a long name is stored in Windows 98 2006-4-29 113 2006-4-29 114 Information Server 19
The UNIX V7 File System The UNIX V7 File System Introduced in PDP-11 Organized in the form of a tree starting from root File name are 14 characters Each directory entry has two fields: File name, and the # of i-node for that file (2 bytes) Number of files in the file system limited to 64K 2 i-node # 14 File Name A UNIX V7 directory entry 2006-4-29 115 2006-4-29 116 The UNIX V7 File System Steps in Looking Up /usr/ast/mbox A UNIX i-node 2006-4-29 117 2006-4-29 118 Questions or Comments? Information Server 20