Systems Programming 08. Standard I/O Library Alexander Holupirek Database and Information Systems Group Department of Computer & Information Science University of Konstanz Summer Term 2008
Last lecture: File I/O Low-Level I/O: Schedule for Today 2 Unbuffered I/O and control functions on file descriptors. Filesystem Interface: Functions for operating on directories and for manipulating file attributes such as access modes and ownership. Today: Standard I/O Library Standard I/O library (ISO C) aka I/O on streams. High-level functions that operate on streams, including formatted input and output. Discussion of the lecture evaluation. Discussion of project part I.
The Standard I/O Library 3 Input and output functionality of the ISO C standard library Specified by the ISO C standard. Has been implemented on many OSs other than UNIX. Additional interfaces defined as extensions by SUSv3. Handles details such as buffer allocation and performing I/O in optimal-sized chunks (no need to worry about using the correct buffer size). Ease of use. Initially written by Dennis Ritchie around 1975.
Streams and FILE Objects 4 Unbuffered I/O File descriptors So far I/O centered around file descriptors. When a file is opened a file descriptor is returned. It was used for all subsequent I/O operations. Standard I/O Library Streams Standard I/O centers around streams. When opening or creating a file we say that we associate a stream with the file (fopen(3) returns a pointer to FILE). FILE contains all the information required by the standard I/O library to manage the stream.
The FILE object 5 Typical members of the FILE structure The file descriptor used for actual I/O. A pointer to a buffer for the stream. The size of the buffer. Count of the number of characters currently in the buffer. An error flag. Incidental Remark In general there is no need to examine a FILE object, just pass the pointer as an argument to each standard I/O function. A pointer with type FILE * is referred to as a file pointer.
Single- and Multibyte Character Sets 6 Standard I/O file streams can be used with single-byte and multibyte ( wide ) character sets. ASCII character set: A single character is represented by a single byte. International character sets: A character can be represented by more than one byte. A stream s orientation determines whether the characters that are read and written are single-byte or multibyte. Initially, when a stream is created, it has no orientation. If a multibyte I/O function ( wchar.h) is used on a stream without orientation, the stream s orientation is set to wide-oriented, and byte-oriented in case of byte I/O functions.
Predefined streams 7 Standard Input, Standard Output, and Standard Error 3 predefined streams are automatically available to a process. They refer to the same files as the file descriptors: STDIN FILENO, STDOUT FILENO, and STDERR FILENO. They are referenced through the predefined file pointers stdin, stdout, and stderr, defined in <stdio.h>. /* <stdio.h> */ BEGIN_DECLS extern FILE sf []; END_DECLS # define stdin (& sf [0]) # define stdout (& sf [1]) # define stderr (& sf [2])
Buffering 8 The standard I/O library provides buffering Goal is to minimize the number of read and write calls. Buffering is tried to be automatically associated to streams. Applications should not worry about it. Different buffering modes can lead to confusions. Three types of buffering provided by the standard I/O library Fully buffered Line buffered Unbuffered
Fully (block) buffered I/O 9 The fully buffered I/O provided by the standard I/O library: Actual I/O takes place when the standard I/O buffer is filled. Files residing on disk are normally fully buffered by the library. The buffer is obtained by one of the I/O functions. Usually by calling malloc(3) the first time I/O takes place. The term flush describes the writing of a standard I/O buffer. A buffer can be flushed automatically by the standard I/O routines such as when a buffer fills. Explicitly, by using the function fflush(3).
Line buffered I/O 10 Line buffered I/O provided by the standard I/O library: Actual I/O takes place, when a newline character is encountered on input or output. This allows us to output a single character at a time (e.g., with fputc(3)), knowing that actual I/O will take place only when we finish writing each line. Line buffering is typically used on a stream when it refers to a terminal (e.g., standard input and standard output). However, the size of the buffer is fixed, so I/O might take place if the buffer is filled before a newline is seen.
Unbuffered I/O 11 Unbuffered I/O provided by the standard I/O library: The standard I/O library does not buffer the characters. When an output stream is unbuffered, information appears on the destination file/terminal as soon as written write(2). Standard error stream is normally unbuffered. Any error messages are displayed as quickly as possible (regardless whether they contain a newline or not).
ISO C Buffering Requirements 12 ISO C requires the following buffering characteristics: Standard input and output are fully buffered, if and only if they do not refer to an interactive device. Standard error is never fully buffered. Should standard input and output be unbuffered or line buffered, if they refer to an interactive device? Should standard error be line buffered or unbuffered? System dependent (for instance OpenBSD) If stdin and stdout refer to a terminal they are line buffered. Standard error is initially unbuffered.
Turn Buffering On and Off 13 # include <stdio.h> void setbuf ( FILE * stream, char * buf ); setbuf turns buffering on or off. It may be implemented similar to: /* / usr / src / lib / libc / stdio / setbuf.c */ # include <stdio.h> void setbuf ( FILE *fp, char * buf ) { ( void ) setvbuf ( fp, buf, buf? _IOFBF : _IONBF, BUFSIZ ); }
# include <stdio.h> Alter Buffering Behaviour 14 int /* 0 if OK else EOF ( but stream is still functional ) */ setvbuf ( FILE * stream, char * buf, int mode, size_t size ); setvbuf is used to alter the buffering behavior of a stream. May only be used after sucessful open and before first I/O. mode must be one of the following three macros: # define _IOFBF 0 /* setvbuf should set fully buffered */ # define _IOLBF 1 /* setvbuf should set line buffered */ # define _IONBF 2 /* setvbuf should set unbuffered */ For an unbuffered stream, buf and size are ignored. For line or fully buffered streams buf and size can optionally specify a buffer and its size. If buf is NULL the system chooses an apt size 1. 1 System-dependent: BUFSIZE (stdio.h), st blksize (stat.h)
Buffer Options and Flushing a Stream 15 The setbuf and setvbuf functions and their options: Function mode buf Buffer & length Type of buffering setbuf setvbuf nonnull user buf of length BUFSIZ fully buffered or line buffered NULL (no buffer) unbuffered IOFBF nonnull user buf of length size NULL system buffer of apt length fully buffered IOLBF nonnull user buf of length size NULL system buffer of apt length line buffered IONBF (ignored) (no buffer) unbuffered At any time, a stream can be flushed: # include <stdio.h> int /* 0 if OK, EOF on failure and errno set */ fflush ( FILE * stream ); Any unwritten data for the stream is passed to the kernel. If stream is NULL, all output streams are flushed.
# include <stdio.h> Opening a Stream 16 FILE * fopen ( const char * path, const char * mode ); FILE * freopen ( const char * path, const char * mode, FILE * stream ); FILE * /* all : fpointer if OK, NULL on failure with errno */ fdopen ( int fildes, const char * mode ); fopen opens a specified file freopen opens a specified file on a specified stream. The original stream (if it exists) is always closed. Change the file associated with stderr, stdin, stdout. fdopen is part of Posix.1 not ISO C, as standard I/O does not deal with file descriptors.
Modes to Open a Standard I/O Stream 17 ISO C specifies 15 values for opening a standard I/O stream: mode Description r or rb open for reading w or wb truncate to 0 length or create for writing a or ab append; open for writing at end of file, or create for writing r+ or r+b or rb+ open for reading and writing w+ or w+b or wb+ truncate to 0 length or create for reading and writing a+ or a+b or ab+ open or create for reading and writing at end of file Using b allows to differentiate between text and binary files. UNIX kernels do not differentiate between these types of files it has no effect. With fdopen, the meaning of mode differs slightly.
Meanings of Open Modes with fdopen 18 fdopen associates a stream with an existing file descriptor Opening for write does not truncate the file. Example: Descriptor was created by open(2). The file already existed. O TRUNC flag is in control whether file is to be truncated. fdopen can not simply truncate any file it opens for writing. Opening for append can not create the file. The file has to exist if a descriptor refers to it.
Appending to a Stream 19 Append mode guarantees atomic operation Opening for append: Each write is at the current end of file. If multiple processes open the same file using append mode 2, data from each process will be correctly written to the file. Older versions of UNIX didn t support the append mode, so programs were coded as follows: if ( lseek (fd, 0L, 2) < 0) /* position to EOF... */ err ( errno, " lseek error "); if ( write (fd, buf, 100)!= 100) /*... and write */ err ( errno, " write error "); This works fine for a single process, but problems arise if multiple processes use this technique to append to the same file appending messages to a logfile, for instance. 2 Same holds for O APPEND with open(2) function.
Sharing a Single File 20 Figure: Two processes with the same file open [Apue, Fig. 3.7]
Lost Update on Append 21 Assume processes A and B append to the same file. 3 Each process has its own file table entry, but they share a single v-node table entry (see Figure 3.7). A performs lseek to EOF and sets current file offset to 1500. Kernel switches and schedules B to run. B performs lseek to 1500 (EOF). B performs write and increments current file offset to 1600. Since the file size has been extended, the kernel also updates the current file size in the v-node to 1600. Kernel switches and A resumes. When A calls write, the data is written at current file offset for A, which is 1500. Data wrote by process B is overwritten. Lost update. 3 Without using append mode.
Atomic Operation with Append Mode 22 Problem: Logical operation position to EOF and write causes two seperate function calls. Solution: Positioning to the current end of file and the write has to be an atomic operation with regard to other processes. Any operation that requires more than one function call cannot be atomic (kernel can suspend the process in-between). Append mode is an atomic way. Each time a write is performed for a file with this append flag set, the current file offset in the file table entry is first set to the current file size from the i-node table entry. Each every write appends to the (updated) current end of file.
Final Remarks on Opening a Stream 23 Six different ways to open a standard I/O stream: Restriction r w a r+ w+ a+ file must already exist x x previous contents of file discarded x x stream can be read x x x x stream can be written x x x x x stream can be written only at end x x Creating a new file with mode w or a, there is no way to specify file s access permission bits, as with open(2). Any created files will have mode S IRUSR S IWUSR S IRGRP S IWGRP S IROTH S IWOTH (0666). With a file opened for reading and writing (+ sign in mode) reads and writes cannot be arbitrarily intermixed. Output shall not be directly followed by input without an intervening fflush. Input shall not be followed by output without repositioning.
Closing a stream with fclose(3) 24 # include <stdio.h> int /* 0 if OK, else EOF / errno ( no further access ) */ fclose ( FILE * stream ); An open stream is closed by calling fclose. Any buffered output data is flushed before the file is closed. Any buffered input data is discarded. Any automatically allocated buffers are released. When a process terminates normally (calling exit or returning from main), all open standard I/O streams are closed.
Reading and Writing a Stream 25 Once opened a stream there are three different types of I/O: Character-at-a-time I/O. Read and write one character at a time, with the standard I/O functions handling all the buffering (if the stream is buffered). Line-at-a-time I/O. To read or write a line at a time, we use fgets(3) and fputs(3). Each line is terminated with a newline character, and we have to specify the maximum line length we can handle. Direct I/O 4. Provided by fread(3) and fwrite(3). For each operation we read or write some number of objects, where each object is of specified size. These types of I/O are refered to as unformatted I/O. Formatted I/O is done by functions, such as printf or scanf. 4 aka binary I/O, object-at-a-time I/O, record/structure-oriented I/O
Character-at-a-time Input Functions 26 # include <stdio.h> int fgetc ( FILE * stream ); int getc ( FILE * stream ); int /* equivalent to getc () with the argument stdin. */ getchar ( void ); Return the next requested object from the stream. Next character as an unsigned char converted to int. The input functions return the same value whether an error occurs or EOF (feof and ferror are used to distinguish).
Check Stream Status 27 # include <stdio.h> int /* non - zero if it is set */ feof ( FILE * stream ); int /* non - zero if it is set */ ferror ( FILE * stream ); int clearerr ( FILE * stream ); Most implementations have two flags for each stream in FILE. An error flag. An end-of-file flag. Both flags are cleared by clearerr.
# include <stdio.h> Push-Back Characters 28 int /* c if OK, EOF on failure */ ungetc ( int c, FILE * stream ); Characters pushed back return by subsequent reads on the stream in reverse order of their pushing (FILO). One character of push-back is guaranteed, but as long as there is sufficient memory, an effectively infinite amount of pushback is allowed. If a character is successfully pushed-back, the end-of-file indicator for the stream is cleared. Pushing back EOF will fail and the stream remains unchanged. Pushed characters don t get written back to file or device. They are kept incore.
Character-at-a-time Output Functions 29 # include <stdio.h> int fputc ( int c, FILE * stream ); int putc ( int c, FILE * stream ); int /* All : c if OK, EOF / errno on failure */ putchar ( int c); The functions write the character c (converted to an unsigned char) to the output stream. EOF is returned if a write error occures, or if an attempt is made to write a read-only stream.
# include <stdio.h> Line-at-a-time Input Functions 30 char * fgets ( char * str, int size, FILE * stream ); char * /* should NEVER be used - > unknown buffer size */ gets ( char * str ); /* Both return str if OK, NULL on EOF or error */ str specifies the address of the buffer to read the line into. gets reads from stdin and fgets from stream. With fgets the size of the buffer is specified. The buffer is always null-terminated, i.e., at most size 1 is read. If the line is longer, a partial line is returned. The next call will read what follows.
Line-at-a-time Output Functions 31 # include <stdio.h> int /* 0 on success and EOF on error */ fputs ( const char * str, FILE * stream ); int puts ( const char * str ); /* >=0 on success and EOF or error */ fputs writes the string pointed to by str to the stream pointed to by stream. puts writes the string str, and a terminating newline character, to the stream stdout.
Binary I/O 32 Motivation for binary I/O Read or write an entire structure at a time. With character-at-a-time functions, such as getc or putc we have to loop through an entire structure. Line-at-a-time functions will not work. fputs stops writing when it hits a null byte. fgets won t work right on input with null or newline bytes.
# include <stdio.h> Binary I/O Functions 33 size_t fread ( void * ptr, size_t size, size_t nmemb, FILE * stream ); size_t fwrite ( const void * ptr, size_t size, size_t nmemb, FILE * stream ); /* Return number of objects read or written */ fread reads nmemb objects, each size bytes long. Input is taken from stream and stored at the location ptr. Both return number of objects read or written. For read it can be less than nmemb if error occurs or EOF. 5 For write an error has occured if it is not equal to nmemb. 5 ferror and feof must be called to determine.
Binary I/O Write an Array 34 The functions have two common cases: Read or write a binary array float data [10]; if ( fwrite (& data [2], sizeof ( float ), 4, fp)!= 4) err (1, " fwrite error."); size as the size of each element of the array. nmemb as the number of elements.
Binary I/O Write a Structure 35 Read or write a structure struct tuple { unsigned int size ; unsigned int level ; enum kind kind ; void * cnt ; } tup ; if ( fwrite (& tup, sizeof ( tup ), 1, fp)!= 1) err (1, " fwrite error."); size as the size of structure. nmemb as one (the number of objects to write).
Fundamental Problems with Binary I/O 36 Binary formats change between compilers and architectures Binary formats used to store multibyte integers and floating-point values differ among machine architectures. The offset of a member within a structure can differ between compilers and systems. Even on a single system, the binary layout of a structure can differ, depending on compiler options. To exchanging binary data among different systems a higher-level protocol is probably the better choice.
Positioning a Stream 37 There are three ways to position a standard I/O stream: ftell and fseek. File position stored as long. (Historic) ftello and fseeko. File position stored as off t. (SUSv3) fgetpos and fsetpos. File position stored as fpos t. (ISO C) They work similar to lseek(2) and the whence options (SEEK SET etc.) are the same.
Obtaining a File Descriptor 38 # include <stdio.h> int /* file descriptor assoc. with the stream */ fileno ( FILE * stream ); On UNIX, the standard I/O library ends up calling the low-level I/O routines. Each standard I/O stream has an associated file descriptor. fileno can obtain the descriptor (SUSv3, not ISO C).