Outline Unix File and I/O (USP Chapters 4 and 5) Instructor: Dr. Tongping Liu Basics of File Systems Directory and Unix File System: inode UNIX I/O System Calls: open, close, read, write, ioctl File Representations: FDT, SFT, inode table fork and inheritance, Filters and redirection File pointers and buffering Directory operations Links of Files: Hard vs. Symbolic 1 2 Storing Information Applications may store information in the process address space. But it is a bad idea. Ø Size is limited to size of virtual address space ü May not be sufficient, e.g. airline reservations, banking Ø The data is lost when the application terminates ü Even when a computer doesn t crash! Ø Multiple processes might want to access the same data ü Imagine a telephone directory of one company File Systems For long-term information storage: Ø Should be able to store very large amount of information Ø Information must survive the processes of using it Ø Should provide concurrent accesses to multiple processes Solution: Ø Store information to disks in units called as files Ø Files are persistent until users delete it explicitly Ø Files are managed by the OS Refer the slides of Profs. Sirer and George at Cornell File Systems: How the OS manages files! 1
File Naming Motivation: Ø People cannot remember block, sector, Ø They should have human-readable names Basic idea: Ø Process creates a file, and gives it a name ü Other processes can access the file by the name Ø Naming conventions are OS dependent ü Usually names are allowed to be less than 255 characters ü Digits and special characters are sometimes allowed ü MS-DOS and Windows are not case sensitive, but UNIX-like OSs are File Attributes File-specific info maintained by the OS Ø File size, modification date, creation time, etc. Ø Varies a lot across different OSs Some examples: Ø Name human-readable form Ø Identifier a unique tag (number) identifies a file within the file system Ø Type required for systems supporting different types (e.g. ro, exe) Ø Location pointer to the file location on device Ø Size Ø Protection controls who can read, write, execute the file Ø Time, date, and user identification data for protection, security, and usage monitoring File Protection File owner/creator should be able to control: Ø what can be done Ø by whom Types of access Ø Read Ø Write Ø Execute Ø Append Ø Delete Ø List Access Example in Linux Mode of access: read, write, execute Three classes of users: Ø Owner: Ø Group: Ø Public: owner!group! public! chmod!761!game! RWX 7: 1 1 1 6: 1 1 0 1: 0 0 1 2
Outline Basics of File Systems Directory and Unix File System: inode (SGG 12) UNIX I/O System Calls: open, close, read, write, ioctl File Representations: FDT, SFT, inode table fork and inheritance, Filters and redirection File pointers and buffering Directory operations Links of Files: Hard vs. Symbolic Directory: An Example An example tree structure of the file system. 9 10 Structure of a Unix File System A directory entry contains only a name and an index into a table with the information of a file (inode) Definition of inode The inode is a data structure in a Unix-style file system that describes a filesystem object, such as a file or directory. Each inode stores the attributes and disk block location(s) of the object's data. Inode attributes may include metadata (times of last change, access, modification), as well as owner and permission data. 11 12 3
More about inode Traditional inode Structure Each file system (such as ext2 or ext3) maintains an array of inodes-- the inode table, which contains list of all files. Each individual inode has a unique number (unique to that filesystem): the inode number. An inode stores: Ø File type: regular file, directory, pipe etc. Ø Permissions to that file: read, write, execute Ø Link count: The number of hard link relative to an inode Ø User ID: owner of file Ø Group ID: group owner Ø Size of file: or major/minor number in case of some special files Ø Time stamp: access time, modification time and (inode) change time Ø Attributes: immutable' for example Ø Access control list: permissions for special users/groups Ø Link to location of file Ø Other metadata about the file 13 Pointers to location of file 14 File Size and Block Size Block size is 8K and pointers are 4 bytes One single indirect pointer Ø A block contains 2K pointers to identify file s 2K blocks Ø File size = 2K * 8K = 16MB One double indirect pointer Ø 2K pointers à 2K blocks, where each block contains 2K pointers to identify blocks for the file Ø Number of blocks of the file = 2K*2K = 4M Ø File size = 4M * 8K = 32 GB One triple indirect pointer Ø Number of blocks of the file = 2K*2K*2K = 8 *K*K*K Ø File size à 64 TB ( = 2^33) ; 15 File Size and Block Size Block size is 8K and pointers are 8 bytes Ø How large a file can be represented using only one single indirect pointer? Ø What about one double indirect pointer? Ø What about one triple indirect pointer? 16 4
Outline Basics of File Systems Directory and Unix File System: inode UNIX I/O System Calls: open, close, read, write, ioctl File Representations: FDT, SFT, inode table fork and inheritance, Filters and redirection File pointers and buffering Directory operations Links of Files: Hard vs. Symbolic Unix I/O Related System Calls Device independence: uniform device interface I/O through device drivers with standard interface Use file descriptors (FDs) Ø FD is an abstract indicator (handle) used to access a file or other input/output resource 3 file descriptors open when a program starts Ø STDIN_FILENO (0), STDOUT_FILENO (1), STDERR_FILENO (2) 6 main system calls for I/O Ø open, close, read, write, ioctl, lseek Ø Return -1 on error and set errno 17 18 File position An open file has its file position that keeps track of where the next character is to be read or written. Ø On GNU systems, and all POSIX.1 systems, the file position is simply an integer representing the number of bytes from the beginning of the file. Ø The file position is normally set to the beginning of the file when it is opened, and each time a character is read or written, the file position is incremented. In other words, accessing to a file is normally sequential. Ø File position can be changed by lseek open #include <fcntl.h> #include <sys/stat.h> int open(const char *path, int oflag); int open(const char *path, int oflag, mode_t mode); Oflag: O_RDONLY, O_WRONLY, O_RDWR, O_APPEND, O_CREAT, O_EXCL, O_NOCTTY, O_NONBLOCK, O_TRUNC; 19 20 5
open (cont.) O_CREAT flag: must use the 3-parameter form of open and specify permissions POSIX Symbolic Names for Permissions (mode) defined in sys/stat.h Historical layout of the permissions mask. 21 22 Program 4.9 (p106): copyfilemain.c copy a file. close and its usage #include <unistd.h> int close(int fildes); Open files are closed when program exits normally. 23 24 6
System Call: read #include <unistd.h> ssize_t read(int fildes, void *buf, size_t nbyte); An example to read in a line Read one char at a time! Need to allocate a buffer *buf to hold the bytes read; size_t is an unsigned long type; ssize_t is a signed long type; Can return fewer bytes than requested; Return value of -1 with errno set to EINTR is not usually an error. 25 26 System Call: write #include <unistd.h> ssize_t write(int fildes, const void *buf, size_t nbyte); Example for write?? Function copyfile reads from one file and writes out to another Not error if return value > 0 but less than nbyte Ø Must restart write if it returns fewer bytes than requested Return value of -1 with errno set to EINTR is not an error usually 27 28 7
Outline Basics of File Systems Directory and Unix File System: inodes UNIX I/O System Calls: open, close, read, write, ioctl File Representations: FDT, SFT, inode table fork and inheritance, Filters and redirection File pointers and buffering Directory operations Links of Files: Hard vs. Symbolic 29 File Representation File Descriptor Table: Ø An array of pointers indexed by the file descriptors Ø The pointers point to entries in System File Table System File Table: Ø Contains entries for each opened file Ø Entries contain pointers to a table of inodes kept in memory Ø Other information in an entry: current file offset; file mode; count of file descriptors using this entry; Ø When a file is closed, the count is decremented. The entry is freed when the count becomes 0. In-Memory Inode Table: copies of in-use inodes 30 File Representation (cont.) What Happened for File Read/Write Per Process All Processes All Processes Figure 4.2 (page 120): Relationship between the file descriptor table, the system file table and the in-memory inode table. System file table: dynamic information 31 1. The process passes the file descriptor to the kernel through a system call (read/write) 2. The kernel will access the entry in file table on behalf of the process, by getting the pointer to the inode 3. For read/write, after obtaining the file pointer (offset) in the file table, then it can compute the block/sector information. 4. In the end, it will issue IO request. IO scheduler may combine multiple requests with the close location together. 32 8
inode: Store File Information Traditional UNIX inode structure. Questions about file-related tables 1. If a file is opened twice in the same process, how many entries will be created in file descriptor table? How many will be created in open file table? How many in inode table? 2 entries in file descriptor table and open file table, but only one inode 2. What about one file is opened separately by two processes? 33 34 Inheritance of File Descriptors File Representation (cont.) When fork creates a child Ø child inherits a copy of the parent's address space, including the file descriptor table 35 36 9
Inheritance of File Descriptors When fork creates a File child is opened BEFORE fork Ø child inherits a copy of the parent's address space Ø including the file descriptor table 37 Parent s FDT 0 stdin 1 stdout 2 stderr 3 File_A Child s FDT 0 stdin 1 stdout 2 stderr 3 File_A both parent and child share the same system file table entry System File Table stdin stdout stderr File_A Question: how many stdin, stdout, stderr in system file table? 38 File is opened AFTER fork parent and child use different system file table entries 39 40 10
Example of File Read int main(void) { char c = '!'; char d = '!'; int myfd; if (fork() == -1) { return 1; } if ((myfd = open("my.dat", O_RDONLY)) == -1) { return 1; } read(myfd, &c, 1); read(myfd, &d, 1); printf("process %ld got %c\n", (long)getpid(), c); printf("process %ld got %c\n", (long)getpid(), d); return 0; } Suppose the file my.dat contains "abcdefghijklmnop and no errors occur; what will be the possible outputs? Process 17514 got a Process 17514 got b Process 17515 got a Process 17515 got b Process 17515 got a Process 17515 got b Process 17514 got a Process 17514 got b Process 17515 got a Process 17514 got a Process 17515 got b Process 17514 got b 41 Read Example (open/fork are switched) int main(void) { char c = '!'; char d = '!'; int myfd; if ((myfd = open("my.dat", O_RDONLY)) == -1) { return 1; } if (fork() == -1) { return 1; } read(myfd, &c, 1); read(myfd, &d, 1); printf("process %ld got %c\n", (long)getpid(), c); printf("process %ld got %c\n", (long)getpid(), d); return 0; } Suppose the file my.dat contains "abcdefghijklmnop and no errors occur; what will be the possible outputs? Process 17514 got a Process 17514 got b Process 17515 got c Process 17515 got d Many possibilities on the order 42 Exercise fd = open("myinfile",o_rdonly); fork(); read(fd,buf,1); read(fd,buf+1,1); printf("%c%c",buf[0],buf[1]); Suppose myinfile is a regular file containing wxyz. Consider the following code segment and assume that it is executed without error. Explain why it is not possible: xwzy Redirection A program can modify the file descriptor table entry so that it points to a different entry in the system file table. This action is known as redirection. 43 44 11
A Redirection Example Example 4.35 (p129) Ø cat > my.file More file redirection examples Symbol Redirection > Output redirect >> append output pipe output to another command < input redirection Output redirection takes the output of a command and places it into a named file. Input redirection reads the file as input to the command. 45 46 Redirection: dup2 Copy one file descriptor table entry into another Can be done with dup2 system call #include <unistd.h> int dup2(int fildes, int fildes2); Ø First close fildes2 silently if fildes2 exists Ø Then, copy the pointer of entry fildes into entry fildes2 File positions for dup() vs. open() if you open a descriptor and then duplicate it to get another descriptor, these two descriptors share the same file position: changing the file position of one descriptor will affect the other. if you open a file twice even in the same program, you get two streams or descriptors with independent file positions. 47 48 12
Example Implementation: cat > my.file FDT for Redirection Example Figure 4.7 (page 131): The status of the file descriptor table during the execution of Program 4.18. 49 50 Outline Basics of File Systems Directory and Unix File System: inodes UNIX I/O System Calls: open, close, read, write, ioctl File Representations: FDT, SFT, inode table fork and inheritance, Filters and redirection File pointers and buffering Directory operations Links of Files: Hard vs. Symbolic File Pointers and Buffering Use fopen, fclose, fread, fwrite, fprintf, fscanf, etc. Example: open a file for output using file pointers 51 52 13
File Pointers and Buffering (cont.) File Pointers and Buffering (cont.) 53 I/O using file pointers à read/write from/to buffer; Buffer is filled or emptied when necessary Buffer size may vary, depend on different OS write may fill part of the buffer without causing any physical I/O to the file; If a write is done to standard output, and program crashes, data written may not show up on screen Standard error is NOT buffered Interleaving output to standard output and error à output to appear in an unpredictable order Force the physical output to occur with an fflush 54 fopen() vs. open() Exercise: bufferout.c 1. fopen() returns a C standard FILE pointer, while open() returns a file descriptor. 2. fopen is a library function while open is a system call. 3. Fopen() provides you with buffering IO if you are mainly reading or writing a file sequentially, and a big speed improvement since fread and fget typically read a buffer size (4K, 8K). But it is not true if you are not accessing sequentially. 4. A FILE * gives you the ability to use fscanf and other stdio functions. printf( a ); 55 56 14
Exercise: bufferinout.c Fileiofork.c What is the output? #include <stdio.h> #include <unistd.h> int main(void) { printf("this is my output."); fork(); This is my output.this is my output. return 0; } Fork will copy the original buffer to the new process. 57 58 Fileioforkline.c What is the output? #include <stdio.h> #include <unistd.h> int main(void) { This is my output. printf("this is my output.\n"); fork(); return 0; } The buffering of standard output is usually line buffering. This means that the buffer is flushed when it contains a newline. Current Working Directory #include <unistd.h> int chdir(const char *path); char *getcwd(char *buf, size_t size); PATH_MAX to determine the size of the buffer needed 59 60 15
Functions for Directory Access Lists Files in a Directory #include <dirent.h> DIR *opendir(const char *filename); Ø provides a handle for the other functions struct dirent *readdir(dir *dirp); Ø gets the next entry in the directory void rewinddir(dir *dirp); Ø restarts from the beginning int closedir(dir *dirp); Ø closes the handle. Functions are not re-entrant (like strtok) 61 62 Functions to Access File Status Contents of the struct stat #include <sys/stat.h> int lstat(const char *restrict path, struct stat *restrict buf); Ø Same as stat; for symbolic link à information about link, not the file it links to int stat(const char *restrict path, struct stat *restrict buf); Ø use the name of a file int fstat(int fildes, struct stat *buf); Ø used for open files System dependent At least the following fields 63 64 16
Example: Last Access Time of A File Access and Modified Times 65 66 File Mode Check Whether a File is A Directory Use the following macros to test the st_mode field for the file type. S_ISBLK(m) block special file S_ISCHR(m) character special file S_ISDIR(m) directory S_ISFIFO(m) pipe or FIFO special file S_ISLNK(m) symbolic link S_ISREG(m) regular file S_ISSOCK(m) socket 67 68 17
Outline Basics of File Systems Directory and Unix File System: inodes UNIX I/O System Calls: open, close, read, write, ioctl File Representations: FDT, SFT, inode table fork and inheritance, Filters and redirection File pointers and buffering Directory operations Links of Files: Hard vs. Symbolic Links in Unix Link: association between a filename and an inode Two types of links in Unix: Hard vs. Symbolic/Soft Ø Hard link: A hard link just creates another file with a link to the same underlying inode. Ø Soft link: a link to another filename in the file system When a file is created Ø A new inode is created Ø inode tracks the number of hard links to the inode 69 70 Creating links Create a hardlink Create a softlink Links in Unix (cont.) New hard link to an existing file Ø creates a new file entry Ø no other additional disk space Ø Increment the link count in the inode Remove a hard link Ø the rm command or the unlink system call Ø decrement the link count in the inode When inode s link count à 0 Ø The inode and associated disk space are freed 71 72 18
Example: Hard Link Example: Hard Link (cont.) Creating command: ln /dira/name1 /dirb/name2 Figure 5.4 (page 163): A directory entry, inode, and data block for a simple file; Figure 5.5 (page 165): Two hard links to the same file; 73 74 File Modification with Multiple Hard Links File Modification (cont.) Exercise 5.17, page 166: What would happen to Figure 5.5 after the following operations: open("/dira/name1"); read close modify memory image of the file unlink("/dira/name1"); open("/dira/name1"); write close After unlink, the name1 is deleted, while name1 still points to the original file 75 Figure 5.6 (page 167): The situation after editing the file. A new file with the new inode is created; 76 19
Symbolic Link Symbolic link: special type of file that contains the name of another file A reference to the name of a symbolic link Ø OS use the name stored in the file; not the name itself. Create a symbolic link Ø ln s /dira/name1 /dirb/name2 Symbolic links do not affect link count in the inode Symbolic links can span filesystems (while hard links cannot) Example: A Symbolic Link Figure 5.8 (page 170): An ordinary file with a symbolic link to it. 77 Create a new inode that the content links to the original file 78 Hard vs. Symbolic Links Hard links Ø Hard links are restrict to a given filesystem Ø References a physical file (unless the filesystem is corrupt) Ø A count of hard links is kept in the inode Ø Removing the last hard link frees the physical file Symbolic links Ø Can make a symbolic link to a file that does not exist Ø Even if it did exist once, a symbolic link might not reference anything Ø Removing a symbolic link cannot free a physical file 79 Summary Basics of File Systems Directory and Unix File System: inodes UNIX I/O System Calls: open, close, read, write, ioctl File Representation Ø File descriptor table, System File Table, In-memory inode table File pointers and buffering fork and file operations Filters and redirection Links of Files: Hard vs. Symbolic 80 20