Files and Directories Filesystems from a user s perspective

Files and Directories Filesystems from a user s perspective Unix Filesystems Seminar Alexander Holupirek Database and Information Systems Group Department of Computer & Information Science University of Konstanz December, 04 2007 Alexander Holupirek (U KN) Files & Directories December, 04 2007 1 / 57

Introduction Introduction Last session: Additional features of the filesystem. Properties of a file. The stat functions. Members of the stat structure. (ZFS interview) Today: The structure of a Unix filesystem. Remaining members of the stat structure. Filesystem structure (i-nodes etc.). Next: The Standard I/O library Streams and FILE Objects Buffering Formatted I/O Alternatives to Standard I/O Alexander Holupirek (U KN) Files & Directories December, 04 2007 2 / 57

Members of the stat struct The structure mode_t st_mode ; /* inode s mode */ uid_t st_uid ; /* user ID of owner */ gid_t st_gid ; /* group ID of owner */ off_t st_size ; /* file size, in bytes */ int64_t st_blocks ; /* blocks allocated for file */ u_int32_t st_blksize ;/* optimal file sys I/ O ops blocksize */ dev_t st_dev ; /* device inode resides on */ ino_t st_ino ; /* inode s number */ nlink_t st_nlink ; /* number of hard links to the file */ dev_t st_rdev ; /* device type, for special file inode */ struct timespec st_atimespec ; /* time of last access */ struct timespec st_mtimespec ; /* time of last data modification */ struct timespec st_ctimespec ; /* time of last file status change */ Alexander Holupirek (U KN) Files & Directories December, 04 2007 4 / 57

st size, st blocks, st blksize The size-related fields struct stat { off_t st_size ; /* file size, in bytes */ int64_t st_blocks ; /* blocks allocated for file */ u_int32_t st_blksize ;/* optimal file sys I/ O ops blocksize */ }; st size Regular file. File size of 0 is allowed (EOF on read). Symbolic link. Actual number of bytes of the targetstring w/o terminating null byte (explicit length is given). st blocks The actual number of blocks allocated for the file in 512-byte units (non-portable!). As short symbolic links are stored in the i-node, this number may be zero. st blksize The optimal I/O block size for the file (standard I/O library). Alexander Holupirek (U KN) Files & Directories December, 04 2007 6 / 57

Write a single byte file. The stat structure The size-related fields Snippets from write single byte file.c: const char * fname = "./ single_byte_file "; const char * abc = "a"; if (( fd = open ( fname, O_CREAT O_TRUNC O_RDWR, 0600 )) == -1) err ( errno, " can not creat %s [%d]", fname, errno ); if (( wb = write (fd, abc, strlen ( abc ))) == -1) err ( errno, " can not write to fd %d ", fd ); Examine written file: $ ls -l single_byte_file -rw ------- 1 holu holu 1 Dec 11 10:22 single_byte_file $ du single_byte_file 2 single_byte_file Alexander Holupirek (U KN) Files & Directories December, 04 2007 8 / 57

getbsize(3) and du(1) getbsize(3) and du(1) getbsize(3) - get user block size The getbsize(3) function determines the user s preferred block size. # include < stdlib.h> char * getbsize ( int * headerlenp, long * blocksizep ); du(1) - display disk usage statistics The du(1) utility displays the file system block usage for each file argument and for each directory in the file hierarchy rooted in each directory argument. Alexander Holupirek (U KN) Files & Directories December, 04 2007 10 / 57

statfs(2) - get file system statistics # include <sys / param.h> # include <sys / mount.h> int statfs ( const char * path, struct statfs * buf ); struct statfs { u_int32_t f_flags ; /* copy of mount flags */ int32_t f_bsize ; /* fundamental file system block size */ u_int32_t f_iosize ; /* optimal transfer block size */ u_int32_t f_blocks ; /* total data blocks in file system */ u_int32_t f_bfree ; /* free blocks in fs */ int32_t f_bavail ; /* free blocks avail to non - superuser */ u_int32_t f_files ; /* total file nodes in file system */ u_int32_t f_ffree ; /* free file nodes in fs */ fsid_t f_fsid ; /* file system id */ uid_t f_owner ; /* user that mounted the file system */ u_int32_t f_syncwrites ; /* count of sync writes since mount */ u_int32_t f_asyncwrites ; /* count of async writes since mount */ u_int32_t f_ctime ; /* last mount [- u] time */ u_int32_t f_spare [3]; /* spare for later */ char f_fstypename [ MFSNAMELEN ]; /* fs type name */ char f_mntonname [ MNAMELEN ]; /* dir on which mounted */ char f_mntfromname [ MNAMELEN ]; /* mounted file system */ union mount_info mount_info ; /* per - fs mount options */ }; Alexander Holupirek (U KN) Files & Directories December, 04 2007 12 / 57

Write a single byte file (cont.) statfs(2) - get file system statistics Get some background information: $./ write_single_byte_file getbsize : 512 /* block size preferred by user */ st_size : 1 st_blocks : 4 /* 4 x 512 = 2048 */ st_blksize : 16384 f_bsize : 2048 /* fundamental filesystem block size */ f_iosize : 16384 Compare it: $ ls -l single_byte_file -rw ------- 1 holu holu 1 Dec 11 09:45 single_byte_file $ du single_byte_file 2 single_byte_file /* 2 * 1024 = 2048 */ File occupies one filesystem data block (2K) to store the single byte. Alexander Holupirek (U KN) Files & Directories December, 04 2007 14 / 57

Write a 2K file The stat structure Writing files in different sizes $./ write_2k_file getbsize : 512 st_size : 2048 st_blocks : 4 st_blksize : 16384 f_bsize : 2048 f_iosize : 16384 $ ls -l 2 K_file -rw ------- 1 holu holu 2048 Dec 11 11:06 2 K_file $ du 2 K_file 2 2 K_file $ ls -l single_byte_file -rw ------- 1 holu holu 1 Dec 11 11:05 single_byte_file $ du single_byte_file 2 single_byte_file No change in disk usage. Alexander Holupirek (U KN) Files & Directories December, 04 2007 16 / 57

Writing files with holes Holes in a file - write(2) and lseek(2) write(2) - reposition read/write file offset # include <sys / types.h> # include < unistd.h> ssize_t write ( int d, const void * buf, size_t nbytes ); lseek(2) - reposition read/write file offset # include < unistd.h> off_t lseek ( int fildes, off_t offset, int whence ); If whence is SEEK_SET, offset is set to offset bytes. If whence is SEEK_CUR, offset is current location plus offset bytes. If whence is SEEK_END, offset is size of file plus offset bytes. Alexander Holupirek (U KN) Files & Directories December, 04 2007 18 / 57

Writing files with holes /* hole.c */ const char * fname = "./ file_with_hole "; const char * abc = " abcdefghij "; const char * ABC = " ABCDEFGHIJ "; if (( fd = open ( fname, O_CREAT O_TRUNC O_RDWR, 0600 )) == -1) err ( errno, " can not creat %s [%d]", fname, errno ); if (( wb = write (fd, abc, strlen ( abc ))) == -1) err ( errno, " can not write to fd %d ", fd ); else printf (" wrote %ld bytes.\n", wb ); if ( lseek (fd, 20, SEEK_CUR ) == -1) err ( errno, " can not seek [%d].", errno ); else printf (" seek 20 bytes.\n" ); } if (( wb = write (fd, ABC, strlen ( ABC ))) == -1) err ( errno, " can not write to fd %d ", fd ); else printf (" wrote %ld bytes.\n", wb ); Alexander Holupirek (U KN) Files & Directories December, 04 2007 20 / 57

Writing files with holes Holes in a file - write(2) and lseek(2) $./ hole wrote 10 bytes. lseek 20 bytes. wrote 10 bytes. getbsize : 512 st_size : 40 st_blocks : 4 st_blksize : 16384 f_bsize : 2048 f_iosize : 16384 $ ls -l file_with_hole check its size -rw ------- 1 holu holu 40 Nov 25 17:24 file_with_hole $ od -c file_with_hole examine actual content 000 a b c d e f g h i j \0 \0 \0 \0 \0 \0 020 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 A B 040 C D E F G H I J Holes are created by seeking past the current end of file. Alexander Holupirek (U KN) Files & Directories December, 04 2007 22 / 57

64K files (with holes) The stat structure Writing files with holes # define NBLK 4 # define BSIZ 16384 /* snippet to write a 64K file */ for (i = 0; i < NBLK ; i ++) { for (j = 0; j < BSIZ ; j ++) { if (( wb = write (fd, abc, strlen ( abc ))) == -1) err ( errno, " can not write to fd %d ", fd ); } } # define NBLK 2 /* snippet to write a 64K file with holes */ for (i = 0; i < NBLK ; i ++) { if ( lseek ( fd, BSIZ, SEEK_CUR ) == -1) err ( errno, " can not seek [%d].", errno ); for (j = 0; j < BSIZ ; j ++) { if (( wb = write (fd, abc, strlen ( abc ))) == -1) err ( errno, " can not write to fd %d ", fd ); } } Alexander Holupirek (U KN) Files & Directories December, 04 2007 24 / 57

Write 64K files (with holes) Writing files with holes $./ write_64k_file st_size : 65536 st_blocks : 128 /* 128 x 512 = 65536 */ $./ write_64k_file_with_holes st_size : 65536 st_blocks : 64 /* 64 x 512 = 32768 */ $ od -c 64 K_file 0000000 a a a a a a a a a a a a a a a a * 0200000 $ od -c 64 K_file_with_holes 0000000 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0040000 a a a a a a a a a a a a a a a a * 0100000 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0140000 a a a a a a a a a a a a a a a a * 0200000 Alexander Holupirek (U KN) Files & Directories December, 04 2007 26 / 57

File with holes (cont.) The stat structure Writing files with holes $ ls -l 64 K_file * -rw ------- 1 holu holu 65536 Dec 11 12:41 64 K_file -rw ------- 1 holu holu 65536 Dec 11 12:41 64 K_file_with_holes $ du 64 K_file * 64 64 K_file /* 64 x 1024 = 65536 */ 32 64 K_file_with_holes /* 32 x 1024 = 32768 */ $ cat 64 K_file_with_holes > 64 K_file_with_holes. copy $ od -c 64 K_file_with_holes. copy 0000000 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0040000 a a a a a a a a a a a a a a a a * 0100000 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0140000 a a a a a a a a a a a a a a a a * 0200000 Alexander Holupirek (U KN) Files & Directories December, 04 2007 28 / 57

File with holes (cont.) The stat structure Writing files with holes $ ls -l 64 K_file * -rw ------- 1 holu holu 65536 Dec 11 12:41 64 K_file -rw ------- 1 holu holu 65536 Dec 11 12:41 64 K_file_with_holes -rw -r--r-- 1 holu holu 65536 Dec 11 12:42 64 K_file_with_holes. copy $ du 64 K_file * 64 64 K_file 32 64 K_file_with_holes 64 64 K_file_with_holes. copy Alexander Holupirek (U KN) Files & Directories December, 04 2007 30 / 57

Filesystem structure Device number, i-node and hard links struct stat { dev_t st_dev ; /* device inode resides on */ ino_t st_ino ; /* inode s number */ nlink_t st_nlink ; /* number of hard links to the file */ }; To understand these members, we need to know the difference between an i-node and a directory entry that refers to an i-node. This will be of importance when we talk about the concept of links to a file. Alexander Holupirek (U KN) Files & Directories December, 04 2007 32 / 57

Filesystem structure Disk drive, partitions, and a filesystem Abbildung: We can think of a disk drive being divided into one or more partitions. Each partition can contain a filesystem. i-nodes are fixed-length entries that contain most of the information about a file. Alexander Holupirek (U KN) Files & Directories December, 04 2007 33 / 57

A closer look at the filesystem Filesystem structure Abbildung: A filesystem in more detail (boot blocks and super block ignored). Two directory entries point to the same i-node. Alexander Holupirek (U KN) Files & Directories December, 04 2007 34 / 57

Filesystem structure link count, remove and unlink, hard link Every i-node has a link count. Alexander Holupirek (U KN) Files & Directories December, 04 2007 36 / 57

Filesystem structure link count, remove and unlink, hard link Every i-node has a link count. The link count contains the number of directory entries that point to the i-node. Alexander Holupirek (U KN) Files & Directories December, 04 2007 36 / 57

Filesystem structure link count, remove and unlink, hard link Every i-node has a link count. The link count contains the number of directory entries that point to the i-node. Only when the link count goes to zero can the file be deleted (i.e., can the associated data blocks be released). This is why unlinking a file does not always mean deleting the blocks associated with the file. This is why the function that removes a directory entry is called unlink(2) and not delete. Alexander Holupirek (U KN) Files & Directories December, 04 2007 36 / 57

Filesystem structure link count, remove and unlink, hard link Every i-node has a link count. The link count contains the number of directory entries that point to the i-node. Only when the link count goes to zero can the file be deleted (i.e., can the associated data blocks be released). This is why unlinking a file does not always mean deleting the blocks associated with the file. This is why the function that removes a directory entry is called unlink(2) and not delete. These type of links are called hard links. Alexander Holupirek (U KN) Files & Directories December, 04 2007 36 / 57

Filesystem structure link count, remove and unlink, hard link Every i-node has a link count. The link count contains the number of directory entries that point to the i-node. Only when the link count goes to zero can the file be deleted (i.e., can the associated data blocks be released). This is why unlinking a file does not always mean deleting the blocks associated with the file. This is why the function that removes a directory entry is called unlink(2) and not delete. These type of links are called hard links. In the stat structure the link count is called st nlink and has type nlink t with limit LINK MAX. Alexander Holupirek (U KN) Files & Directories December, 04 2007 36 / 57

Filesystem structure link count, remove and unlink, hard link Every i-node has a link count. The link count contains the number of directory entries that point to the i-node. Only when the link count goes to zero can the file be deleted (i.e., can the associated data blocks be released). This is why unlinking a file does not always mean deleting the blocks associated with the file. This is why the function that removes a directory entry is called unlink(2) and not delete. These type of links are called hard links. In the stat structure the link count is called st nlink and has type nlink t with limit LINK MAX. (There is another link type, called symbolic link. Later.) Alexander Holupirek (U KN) Files & Directories December, 04 2007 36 / 57

Filesystem structure inode(5) - format of file system volume The i-node contains all information about a file the file type the file s access permission bits the size of the file pointers to the data blocks for the file Most stat entries are obtained from the i-node Which not? Alexander Holupirek (U KN) Files & Directories December, 04 2007 37 / 57

Filesystem structure inode(5) - format of file system volume The i-node contains all information about a file the file type the file s access permission bits the size of the file pointers to the data blocks for the file Most stat entries are obtained from the i-node Which not? file name and i-node number ino t are stored in the directory entry. Alexander Holupirek (U KN) Files & Directories December, 04 2007 37 / 57

Filesystem structure i-nodes, directory entries, and hard links i-node is filesystem bound It is assumed that the i-node number in the directory entry points to an i-node in the same filesystem. We cannot have a directory entry point to an i-node in a different filesystem. This is why the ln(1) command (make a new directory entry that points to an existing file) cannot cross filesystems. Alexander Holupirek (U KN) Files & Directories December, 04 2007 39 / 57

link(2) - make a hard file link Function that operate on links # include < unistd.h> int link ( const char * name1, const char * name2 ); The link(2) function atomically creates the specified directory entry (hard link) name2 with the attributes of the underlying object pointed at by name1. If the link is successful: the link count of the underlying object is incremented; name1 and name2 share equal access and rights to the underlying object. If name1 is removed, the file name2 is not deleted and the link count of the underlying object is decremented. name1 must exist for the hard link to succeed and both name1 and name2 must be in the same file system. As mandated by POSIX.1 name1 may not be a directory. Alexander Holupirek (U KN) Files & Directories December, 04 2007 41 / 57

unlink(2) - remove directory entry Function that operate on links # include < unistd.h> int unlink ( const char * path ); The unlink(2) function removes the link named by path from its directory. The link count of the file which was referenced by the link is decremented. If that decrement reduces the link count of the file to zero, and no process has the file open, then all resources associated with the file are reclaimed. If one or more processes have the file open when the last link is removed, the link is removed, but the removal of the file is delayed until all references to it have been closed. Alexander Holupirek (U KN) Files & Directories December, 04 2007 43 / 57

Function that operate on links remove(3) - remove a file or directory # include <stdio.h> int remove ( const char * path ); The remove(3) function removes the file or directory specified by path. If path specifies a directory, remove(path) is the equivalent of rmdir(path). Otherwise, it is the equivalent of unlink(path). Alexander Holupirek (U KN) Files & Directories December, 04 2007 45 / 57

Function that operate on links rename(2) - change the name of a file # include <stdio.h> int rename ( const char * from, const char * to ); rename(2) causes the link named from to be renamed as to. If to exists, it is first removed. Both from and to must be of the same type (that is, both directories or both non-directories), and must reside on the same file system. rename(2) guarantees that if to already exists, an instance of to will always exist, even if the system should crash in the middle of the operation. If the final component of from is a symbolic link, the symbolic link is renamed, not the file or directory to which it points. Alexander Holupirek (U KN) Files & Directories December, 04 2007 47 / 57

Notes on rename(2) The stat structure Function that operate on links When renaming a file without changing filesystems... the actual content of the file need not be moved. a new directory entry needs to point to the existing i-node the old directory entry has to be removed Example: rename( /usr/lib/foo, /usr/foo ) Assumption: /usr/lib and /usr are on the same filesystem Consequence: the contents of foo need not be moved This is how mv(1) usually operates Alexander Holupirek (U KN) Files & Directories December, 04 2007 49 / 57

Link count for directories Making a new directory in the working directory (mkdir testdir Abbildung: Filesystem state after the creation of testdir Alexander Holupirek (U KN) Files & Directories December, 04 2007 50 / 57

Link count for directories i-node 2549 has a type field of directory Alexander Holupirek (U KN) Files & Directories December, 04 2007 51 / 57

Link count for directories i-node 2549 has a type field of directory i-node 2549 has a link count equal to 2 Alexander Holupirek (U KN) Files & Directories December, 04 2007 51 / 57

Link count for directories i-node 2549 has a type field of directory i-node 2549 has a link count equal to 2 Any leaf directory always has a link count of 2 Alexander Holupirek (U KN) Files & Directories December, 04 2007 51 / 57

Link count for directories i-node 2549 has a type field of directory i-node 2549 has a link count equal to 2 Any leaf directory always has a link count of 2 One from the directory entry that names the directory (testdir) One from the dot entry in that directory. i-node 1267 has a type field of directory Alexander Holupirek (U KN) Files & Directories December, 04 2007 51 / 57

Link count for directories i-node 2549 has a type field of directory i-node 2549 has a link count equal to 2 Any leaf directory always has a link count of 2 One from the directory entry that names the directory (testdir) One from the dot entry in that directory. i-node 1267 has a type field of directory i-node 1267 has a link count greater than or equal to 3 Alexander Holupirek (U KN) Files & Directories December, 04 2007 51 / 57

Link count for directories i-node 2549 has a type field of directory i-node 2549 has a link count equal to 2 Any leaf directory always has a link count of 2 One from the directory entry that names the directory (testdir) One from the dot entry in that directory. i-node 1267 has a type field of directory i-node 1267 has a link count greater than or equal to 3 Why? Alexander Holupirek (U KN) Files & Directories December, 04 2007 51 / 57

Link count for directories i-node 2549 has a type field of directory i-node 2549 has a link count equal to 2 Any leaf directory always has a link count of 2 One from the directory entry that names the directory (testdir) One from the dot entry in that directory. i-node 1267 has a type field of directory i-node 1267 has a link count greater than or equal to 3 Why? The directory entry that names it, dot, and dot dot. Alexander Holupirek (U KN) Files & Directories December, 04 2007 51 / 57

Symbolic link The stat structure Symbolic link $ ls -l foobar lrwxr - xr - x 1 holu holu 3 Dec 11 09:08 foobar -> foo $ du foobar 0 foobar The actual content of the file (the data blocks) contains the name of the file that the symbolic link points to In the example above the filename in the directory entry is the six-character string foobar The three bytes of data in the file are foo The file type in the i-node is S IFLINK to denote the symbolic link Alexander Holupirek (U KN) Files & Directories December, 04 2007 53 / 57

Symbolic link The stat structure Symbolic link $ ls -l foobar lrwxr - xr - x 1 holu holu 3 Dec 11 09:08 foobar -> foo $ du foobar 0 foobar The actual content of the file (the data blocks) contains the name of the file that the symbolic link points to In the example above the filename in the directory entry is the six-character string foobar The three bytes of data in the file are foo The file type in the i-node is S IFLINK to denote the symbolic link What about the du result? Alexander Holupirek (U KN) Files & Directories December, 04 2007 53 / 57

Symbolic link The stat structure Symbolic link $ ls -l foobar lrwxr - xr - x 1 holu holu 3 Dec 11 09:08 foobar -> foo $ du foobar 0 foobar The actual content of the file (the data blocks) contains the name of the file that the symbolic link points to In the example above the filename in the directory entry is the six-character string foobar The three bytes of data in the file are foo The file type in the i-node is S IFLINK to denote the symbolic link What about the du result? Short symbolic links are stored inlined in the i-node $ ls -l a lrwxr - xr - x 1 holu holu 124 Dec 11 17:22 a -> bbb... $ du a 2 a Alexander Holupirek (U KN) Files & Directories December, 04 2007 53 / 57

Device numbers The stat structure Special Device Files dev_t st_dev ; /* device inode resides on */ dev_t st_rdev ; /* device type, for special file inode */ Every filesystem is known by its major and minor device number This device number is encoded in the primitive data type dev t Why? A disk drive often contains several filesystems The st dev value for every filename on a system is the device number of the filesystem containing that filename and its corresponding i-node Only character and block special files have an st rdev value This value contains the device number for the actual device Alexander Holupirek (U KN) Files & Directories December, 04 2007 55 / 57

The three time fields The stat structure File Times struct timespec st_atime ; /* time of last access */ struct timespec st_mtime ; /* time of last data modification */ struct timespec st_ctime ; /* last - change time of i- node status */ Three time fields are maintained for each file The modification time (mtime is when the contents of the file was last modified The changed-status time (ctime) is when the i-node of the file was last modified Many operations (changing the file access permissions, changing the user ID, changing the number of links... ) do affect the i-node without changing the actual content Since the i-node is stored separate from the actual content we need both Note that there is no last-access time for an i-node This is why functions like access(2) and stat(2) do not change any of the three values. Alexander Holupirek (U KN) Files & Directories December, 04 2007 57 / 57