File Systems. Information Server 1. Content. Motivation. Motivation. File Systems. File Systems. Files

Transcription

1 Content File Systems Hengming Zou, Ph.D. Files Disk scheduling, file structure, indexed files Directories File system implementation File system performance Example file systems Motivation All applications need store or retrieve information Where can information be kept? Motivation Process s address space while it is running Limited size, volatile, not sharable Or it can be stored on disk Persistent, sharable, large size But using disk is not easy for ordinary users The solution is: file system File Systems File Systems What is a file system? A file system is an OS abstraction to make the disk (or other devices) easier to use Shield users from the details of how and where the information is stored, and how disks actually work Users File system Disks Virtual interface Physical interface Information Server 1

2 File System Characteristics Store large amounts of data Survive the termination of the process using it Multiple processes can access it concurrently File Systems Files are managed by operating system How are files structured? How are files named? Used? Protected? How are files implemented? Views of File Systems Views of File Systems Users concern how files appear to them: What constitute a file? How files are named? What operations are allowed on files? Etc. OS designers concern how files are implemented How are files structured? How to keep track of free storage? How many sectors are in a logical block? First let s look from the users perspective Naming How to specify which file you want to access? Naming Typically, the user provides a symbolic name e.g. English name, icon OS translates name to a numeric file header or disk location of the file header Information Server 2

3 Naming Naming Alternative is to describe the contents of the file have the OS find the name by looking through the files for that content usually done by databases on top of file systems Lots of ways to translate from symbolic name to file header s disk block # e.g. hash table, expandable array, etc. the structure that does the mapping is called a directory Naming Exactly rules for naming differing by systems Normally 2-8 characters But can be 255 characters A file name can also have a extension To convey information about the file.c for C file,.bak for backup file, etc. File Structure File can be structured in any of several ways Common ways to structure files are: byte sequence, record sequence, tree File Structure File Types Most OS supports several types of files Regular files, directories UNIX also have character and block special files To model serial I/O devices and disks Information Server 3

4 File Types Regular files are either ASCII or binary ASCII files contain lines of text Binary file are encoded files i.e. need special programs to open it File Types Executable file Archive file File Access Only two types of accesses available: sequential or random access Early OS provides only sequential access Sequential Access Read all bytes/records from the beginning Cannot jump around, could rewind or back up Convenient when medium was tape Random Access Bytes/records read in any order Essential for data base systems Read can be Move file marker (seek), then read or Read and then move file marker File Attributes Every file has a name and its data In addition, all OS associate other info to files These other information are called attributes Information Server 4

5 File Attributes File Attributes Attribute Meaning Attribute Meaning Protection Who can access the file and in what way Temporary flag 0 for normal; 1 for delete file on process exit Password Password needed to access the file Lock flags 0 for unlocked; nonzero for locked Creator ID of the person who created the file Record length Number of bytes in a record Owner Current owner Key position Offset of the key within each record Read-only flag 0 for read/write; 1 for read only Key length Number of bytes in the key field Hidden flag 0 for normal; 1 for do not display in listings Creation time Date and time the file was created System flag 0 for normal files; 1 for system files Time of last access Date and time the file was last accessed Archive flag 0: has been backed up; 1: need to be backed up Time of last change Data and time the file has last changed ASCII/binary flag 0 for ASCII file; 1 for binary file Current size Number of bytes in the file Random access flag 0 for sequential access only; 1 for random access Maximum size Number of bytes the file may grow to File Operations An Example Using File System Calls Create/delete Open/close Read/write/append Seek Get attributes /set attributes Rename /* file copy program. Error checking and reporting is minimum */ #include <sys/types.h> /* include necessary header files */ #include <fcntl.h> #include <stdlib.h> #include <unistd.h> int main(int argc, char *argv[]); /*ANSI prototype */ #define BUF_SIZE 4096 /* use a buffer size of 4096 bytes */ #define OUTPUT_MODE 0700 /* protection bits for output file */ An Example Using File System Calls An Example Using File System Calls int main(int argc, char *argv[]) { int int_fd, out_fd, rd_count, wt_count; char buffer[buf_size]; if (argc!=3) exit(1) /*syntax error if argc is not 3 */ /* open input file and create output file */ in_fd = open(argv[1],o_rdonly); /* open source file*/ if (in_fd<0) exit(2) /*if cannot open, exit */ out_fd=create(argv[2],output_mode); /* create destination file*/ if (out_fd<0) exit(3); /* if cannot create, exit */ while (TRUE) { /* copy loop */ rd_count=read(in_fd,buffer,buf_size); /*read a block of data*/ if (rd_count<=0) break; /*if end of file or error, exit loop*/ wt_count=write(out_fd,buffer,rd_count); /*write data*/ if(wt_count<=0) exit(4); /*wt_count<=0 is an error */ } close (in_fd); close(out_fd); /*close the files */ if(rd_count==0) exit(0); /*no error on last read */ else exit(5); /*error on last read */ } Information Server 5

6 Memory-Mapped Files The previous example is cumbersome to some Because have make many system calls One solution is use memory-mapped files Started by MULTICS Key idea Map a file into memory at a virtual address Memory-Mapped Files After mapping, access a file like access memory Ex: 64KB file is mapped to virtual address 512K Reads the content at byte 512K gets byte 0 of the file Write to 512K+10 modifies byte 10 of the file When process terminates, modified file is left on disk Directories Directories Directories are used to keep track of files Also called folders Directories are themselves files! A directory contains a mapping for a set of files name => file header s disk block # for that file Often a simple array of entries (name, file header s disk block #) The table is stored in a normal file as normal data E.g. ls can be implemented by reading this file and parsing its contents Directories Can often treat directories and files in same way Use same storage structure to store their data Directory entries can point to either a file or another directory Directories Directories are typically organized into hierarchical structure Directory A has a mapping for a bunch of files and directories in directory A There are always a root directory in system Information Server 6

7 Hierarchical Directory Systems Directories For example: /zou/cs307/names / is root directory contains a list of files and other directories for each file/directory in /, / has a mapping from name to file header s disk block # one of these entries is zou Directories zou is a directory entry within the / directory contains a list of files and directories one of the directories in /zou is cs307 Directories /zou/cs307 is a directory within the /zou directory contains a list of files and other directories one of the files it lists is names Directories How many disk I/Os to access the first byte of /zou/cs307/names? Path Names Need a way to specifying file names in the tree Absolutely path name A path from root to the file, i.e. /zou/cs307/file1 Relatively path name A path relative from current work directory Assume /zou is the work directory, then cs307/file Information Server 7

8 Current Working Directory Path Names Allows a user to specify a file name relative to the current directory Allows the system to access a file without traversing the entire directory from / Reducing positioning overhead by taking advantage of usage patterns A UNIX directory tree Directory Operations Create, Delete Opendir, Closedir Readdir Rename Link, Unlink Link and Unlink Link allows a file to appear in more than 1 place Link specifies a existing file and a path name Link creates a link from the file to path name It also increment a counter in the file s i-node Link and Unlink Unlink removes a directory entry for a file If the last link for the file, the file is removed If file is in multiple directory, decrement counter File System Implementation Now let s look at from designers perspective Information Server 8

9 File System Layout File systems are stored on disks Most disks are divided into partitions Each partition has its own file systems File System Layout Sector 0 of a disk is called MBR Mater Boot Record MBR is used to boot the computer The end of MBR contains the partition table Giving starting and ending addresses of each partition One of the partitions is marked as active in the table File System Layout At boot time, BIOS reads in the MBR MBR then locates the active partition and reads in the first block, the boot block, and execute it Boot block in turns loads the OS in the partition File System Layout Other than the boot block, layout of disk partition varies greatly from file system to file system Often a file system have a super block right after the boot block that contains the key parameters about the file system on the disk File System Layout Implementing Files Keep track of which disk block belongs to which file is an important issues in file implementation How to organize files on disk? What data structure is the right one to use when storing a file on disk? Three common approaches are possible: Contiguous allocation, Linked list allocation, i-node A possible file system layout Information Server 9

10 Contiguous Allocation Contiguous Allocation Store a file in one contiguous segment on disk Sometimes called an extent User must declare the size of the file in advance Use some kind of fitting algorithm to locate an area of disk that can store the file Contiguous allocation of disk space for 7 files State of the disk after files D and E have been removed Contiguous Allocation File header is very simple: Starting block #, file size Equivalent to base & bounds for memory management Contiguous Allocation Pros and Cons + fast sequential access no seeks between consecutive disk blocks + easy random access can easily calculate the disk location of any file block Contiguous Allocation Pros and Cons - external fragmentation - wastes space if user declares a bigger size - hard to grow files Linked List Allocation Keep each file as a linked list of disk blocks First word of each block points to the next one Disk blocks need not be adjacent Information Server 10

11 Linked List Allocation Linked List Allocation Pros and Cons + no space lost to disk fragmentation + Fast sequential access - extremely slow random access - amount of data in block is not a power of 2 Storing a file as a linked list of disk blocks Linked List Allocation Linked List Allocation How to solve the problems? Use File Allocation Table (FAT) Take the pointer word from each disk block Putting it in a table in memory Linked list allocation using a file allocation table (FAT) in RAM Indexed Files Indexed Files User (or system) declares max # of blocks in file; System allocates a file header with an array of pointers big enough to point to that number of blocks The header is called an i-node The file header is load into memory when open File Header Also called i-node File block # Disk block # Equivalent to a page table for MMU Usually contain additional information such as attributes Information Server 11

12 Indexed Files Indexed Files Pros and Cons + can easily grow file up to the # of blocks allocated in file header + easy random access - lots of seeks for sequential access - hard to grow file bigger than initially allocated in the file header How to grow files sizes? The solution is to use multi-level index files Level 1 node Level 2 node Disk block 18 Disk block 50 Disk block 8 Disk block 15 Level 2 Node Multi-level Indexed Files This allows really big files without wasting header space for small files How many accesses to get 1 block of data? how to solve? Use non-uniform depth multi-level indexed files Multi-level Indexed Files Non-uniform depth multi-level indexed files Level 1 node Disk block 18 Disk block 50 Level 2 node Disk block 8 Disk block 15 Disk block 75 Disk block Multi-level Indexed Files Implementing Files Pros and cons + files can easily expand + small files don t pay full overhead of deep trees - lots of indirect blocks for big files - lots of seeks for sequential access An example i-node Information Server 12

13 Implementing Directories Directory contains entries about files in it Main function is to map the ASCII name of the file onto the information needed to locate the data Name disk address of the entire file Name # of first block (for linked list schemes) Name the number of the i-node Implementing Directories Where to keep the attributes of files? Two approaches available: In directory entry In the i-node structure Implementing Directories Implementing Directories disk addresses and attributes in directory entry Directory in which each entry just refers to an i-node What happens if a file name is long and cannot fit into the fixed size directory entry? Two ways of handling long file names in directory (a) In-line (b) In a heap Implementing Directories Implementing Directories If very long directories, various techniques can be employed to speed up the search process Use hashing table Cache the results of searches In-line In heap Information Server 13

14 Shared Files Shared Files Shared files are implemented with symbolic links Two users share a file by linking to the same file # of links to a file is kept in i-node File system containing a shared file Shared Files Shared Files Symbolic links cause problems that must be solved Inconsistencies with block count of the file File remains while owner has deleted the file Problems with backup program Prior to linking After link is created Original owner removes file Mounting Combining (mounting) multiple disks into a single file system Each disk has a file system each disk s file system starts at its own root (/) Mounting Can tie several disks together into a single file system have an entry in one directory point to the root directory of another disk s file system this entry is called a mount point Information Server 14

15 Example: zou s Workstation % ls / afs/ kernel/ sbin/ bin@ lib@ template-server@ core lost+found/ template-server-ro@ dev/ nfs/ tmp/ devices/ opt/ usr/ etc/ platform/ var/ export/ proc/ vol/ home/ root/ Example: zou s Workstation /bin is a normal directory in / /usr points to the root file system of disk 1 /var points to the root file system of disk 2 /afs points to the root of a distributed file system Mounting Windows: disks are made visible under my_computer/ C,D,E Unix: can mount disks at any place in the directory hierarchy Disk Space Management Files are normally stored on disks Thus disk space management is important The problem is similar to memory space management Contiguous allocation is hard to file to grow Linked list allocation is slow on sequential access But more severe because disk access is much slower Block Size Block Size Most file systems use fixed-block allocation scheme How big should a file block be? Too big means internal waste Low space utilization Too small means frequent disk seek (may be rotation) Low data transfer rate Block size bytes Assume all files are of size 2KB Information Server 15

16 Keep Track of Free Blocks The second problem in disk space management Two methods can be employed: Linked list implementation Free blocks are linked in a list Bitmap implementation A bitmap indicating if a block is free or not Disk Quotas Prevents people from getting too much disk space Use a second table in the open file table structure Disk Quotas File System Reliability Two meanings: Durability: Data in a file system must be permanent Implemented through backup and replication Consistency: Data in a file system must be good Implemented through logging, transactions, and shadowing Quotas for keeping track of each user s disk use File System Durability File System Durability From the data s point of view, there are two backup approaches available: Logical backup Backups only specified directories and files Physical backup Backup everything on a disk File that has not changed A file system to be dumped Information Server 16

17 File System Durability File System Consistency Multi-step updates cause problems for consistency if crash happens in the middle E.g. transfer $100 from my account to Janet s 1. deduct $100 from my account 2. add $100 to Janet s account What happens if you crash between step 1 and 2? Bit maps used by the logical dumping algorithm File System Consistency E.g. create (empty) new file 1. update the directory to point to the new file header 2. write new file header to disk E.g. move file from 1 directory to another 1. delete file from old directory 2. add file to new directory Block Read Ahead Read more blocks than are requested In the hope that they will be needed soon Since data transfer time is negligible compared to disk seek and rotation time (to an extent) Read ahead impose little or no overhead This technique helps only if access is sequential Reduce Disk Arm Motion File System Performance Seek time dominates disk access time Reducing disk arm motion improved performance Idea is to put i-node closer to actual file data Normally all i-nodes go together, and all data go together I-nodes placed at start of disk Disk divided into cylinder groups each with its own blocks and i- nodes Information Server 17

18 Log-Structured File Systems With CPUs faster, memory larger disk caches can also be larger increasing # of read requests can come from cache thus, most disk accesses will be writes Log-Structured File Systems LFS Strategy structures entire disk as a log have all writes initially buffered in memory periodically write these to the end of the disk log when file opened, locate i-node, then find blocks Example File Systems The MS-DOS File System Windows 98 File System UNIX V7 File System MS-DOS File Systems Use 8+3 upper case file names DOS 1.0 limited to one directory Later versions allows hierarchical file system Directory can nested to an arbitrary depth Directories are variable sized But directory entries are fixed size of 32 bytes MS-DOS File Systems MS-DOS File Systems The highest expressible year in MS-DOS is 2107 The counter starts at 1980 Use file allocation table to keep track of file addresses Each directory entry only gives the first block number Which is used to index into a the FAT table The MS-DOS directory entry Information Server 18

19 MS-DOS File Systems Maximum Partition in MS-DOS FAT file system has three versions for MS-DOS: Block size FAT-12 FAT-16 FAT-32 FAT-12, FAT-16, FAT-32 Each differs from the bits a disk address contains FAT-32 introduced with 2nd release of Windows 95 Disk block numbers are multiples of 512 bytes Can be different for each partition Allowable block size differing by variants 0.5 KB 1 KB 2 KB 4 KB 8 KB 16 KB 2 MB 4 MB 8 MB 16 MB forbidden forbidden forbidden forbidden 128 MB 256 MB 512 MB 1024 MB forbidden forbidden forbidden 1 TB 2 TB 2 TB 32 KB forbidden 2048 MB 2 TB Windows 98 File System Windows 98 File System To accommodate long file names i.e. up to 255 characters To provide backwards compatibility to MS-DOS files i.e. must use the MS-DOS directory structure Idea: keep two names for each file i.e. create a MS-DOS name from the given (long) name Both name are then stored in the directory structure Bytes The extended MOS-DOS directory entry used in Windows Windows 98 File System Windows 98 File System Bytes Checksum An entry for (part of) a long file name in Windows 98 An example of how a long name is stored in Windows Information Server 19

20 The UNIX V7 File System The UNIX V7 File System Introduced in PDP-11 Organized in the form of a tree starting from root File name are 14 characters Each directory entry has two fields: File name, and the # of i-node for that file (2 bytes) Number of files in the file system limited to 64K 2 i-node # 14 File Name A UNIX V7 directory entry The UNIX V7 File System Steps in Looking Up /usr/ast/mbox A UNIX i-node Questions or Comments? Information Server 20