File Descriptors and Piping CSC209: Software Tools and Systems Programming Furkan Alaca & Paul Vrbik University of Toronto Mississauga https://mcs.utm.utoronto.ca/~209/ Week 8
Today s topics File Descriptors Low-level I/O with system calls Pipes and Inter-Process Communication Bitwise operators Acknowledgement: These slides are built upon materials originally prepared by Dan Zingaro and Andrew Petersen. 2 / 36
File Descriptors If we open a file in a process and then fork or exec, the file is also open in the child. But we have to use file descriptors and low-level file I/O, using system calls instead of standard C library calls. open returns a file descriptor (a positive integer that serves a similar purpose as the FILE * returned by fopen) 3 / 36
open and close int open ( const char * fname, int flags, [ mode_t mode ]); int close ( int fd); Some flags O_RDONLY, O_WRONLY, O_RDWR (like "rb", "wb" (but without the truncation), and "r+b" to fopen). flags can also include O_CREAT which creates a file if it does not exist. If O_CREAT is included, mode must be supplied and be set to the permissions used when the file gets created. 4 / 36
read and write 1 // Returns # bytes actually read, 0 for EOF, -1 for error 2 ssize_t read ( int fd, void * buffer, size_t maxbytes ); 3 // e.g. 4 char buf [1024]; 5 int num = read ( STDIN_FILENO, buf, 1024) ; 6 7 // Returns # bytes actually written, -1 for error 8 ssize_t write ( int fd, void * buffer, size_t maxbytes ); Note: The C standard libraries use these system calls internally. To see during runtime what system calls your program makes, run strace./myprog (run sudo apt install strace on Ubuntu, if you don t have strace installed). 5 / 36
File Descriptors and fork (forkfd.c) 1 int fd; char buf [6]; 2 if (( fd = open (" blah ", O_RDONLY )) == -1) { 3 perror (" open "); 4 exit (1) ; 5 } 6 if ( fork () == 0) { 7 read (fd, buf, 6); 8 write ( STDOUT_FILENO, buf, 6); 9 } else { 10 wait ( NULL ); 11 read (fd, buf, 6); 12 write ( STDOUT_FILENO, buf, 6); 13 read (fd, buf, 6); 14 write ( STDOUT_FILENO, buf, 6); 15 } 16 return 0; parent child filename pos "blah" 6 A child process created via fork inherits duplicates of its parent s file descriptors, which refer to the same open file descriptions (see man 2 open). 6 / 36
File Descriptors and fork... If the child closes a file descriptor (FD) inherited from fork, and re-opens the file, the kernel creates a new open file description (OFD): parent child filename pos filename pos "blah" 6 "blah" 6 Now if the child reads six more bytes, only the child s read position (and not the parent s) moves forward: parent child filename pos filename pos "blah" 6 "blah" 12 7 / 36
dup and dup2 int dup ( int oldfd ); int dup2 ( int oldfd, int newfd ); dup returns a new FD that refers to the same OFD as oldfd (As with fork) Both FDs share a common file position. You have to close both FDs to really close the file. dup2 recycles the FD newfd (whereas dup uses the lowest-numbered unused FD) dup2 first closes newfd if it is open Check out man dup for more details. 8 / 36
Example: Output Redirection (execl2.c) 1 int main ( void ) 2 { 3 int fd = open (" lsout ", O_ WRONLY O_CREAT, 0600) ; 4 if (fd == -1) { 5 perror (" open "); 6 exit (1) ; 7 } 8 9 dup2 (fd, STDOUT_FILENO ); 10 execl ("/ bin /ls", "ls", "-l", ( char *) NULL ); 11 perror (" execl "); 12 return 1; 13 } 9 / 36
Inter-Process Communication (IPC) After a fork we have two independent processes which have separate address spaces. Most importantly, each process has separate copies of variables and thereby the processes cannot use variables to communicate. Communication using files is possible, but coordination is difficult. 10 / 36
IPC with Pipes We are going to use pipes to let the processes communicate: Remember that any FD (file descriptor) that the parent opens before calling fork is also open in the child! This is crucial for pipes to work as an inter-process mechanism. 11 / 36
Definition (Pipe) A pipe is a one-way, first-in first-out (FIFO) communication channel. Usually, one process writes into a pipe, and another process reads from the pipe. Usage We pass a reference to an array of two integers, and pipe fills it with two newly-opened FDs. 1 int pipe ( int fds [2]) ; 12 / 36
1 int p [2]; 2 if ( pipe (p) == -1) { 3 perror (" pipe "); 4 exit (1) ; 5 } p[1] pipe p[0] p[0] is now an FD open for reading p[1] is now an FD open for writing Whatever you write to p[1] can be read from p[0] Do not try reading from p[1] or writing to p[0]! 13 / 36
Pipe in a single process Example (pipe1.c) 1 # define MSG_ SIZE 13 2 char * msg = " hello, world \n"; 3 4 int main ( void ) 5 { 6 char buf [ MSG_SIZE ]; 7 int p [2]; 8 if ( pipe (p) == -1) { 9 perror (" pipe "); 10 exit (1) ; 11 } 14 / 36
Example (pipe1.c, continued) 1 write (p[1], msg, MSG_SIZE ); 2 read (p[0], buf, MSG_SIZE ); 3 write ( STDOUT_FILENO, buf, MSG_SIZE ); 4 return 0; 5 } 15 / 36
Child talks and parent listens Example (pipe2.c) 1 # include < unistd.h> 2 # include <stdio.h> 3 # include < stdlib.h> 4 5 # define MSG_ SIZE 13 6 char * msg = " hello, world \n"; 7 int main ( void ) 8 { 9 char buf [ MSG_SIZE ]; 10 int p [2]; 16 / 36
1 if ( pipe (p) == -1) { 2 perror (" pipe "); 3 exit (1) ; 4 } 5 if ( fork () == 0) // Child writes 6 write (p[1], msg, MSG_SIZE ); 7 else { // Parent reads from pipe and prints 8 read (p[0], buf, MSG_SIZE ); 9 write ( STDOUT_FILENO, buf, MSG_SIZE ); 10 } 11 return 0; 12 } 17 / 36
pipe and eof parent hangs. Example (pipe3.c) The buffer is too small for the parent to read everything at once. So it reads in a loop... but parent hangs! 1 # include < unistd.h> 2 # include <stdio.h> 3 # include < stdlib.h> 4 5 # define MSG_SIZE 13 6 char * msg = " hello, world \n"; 7 8 # define BUF_SIZE 4 // Small! 18 / 36
1 int main ( void ) 2 { 3 char buf [ BUF_SIZE ]; 4 int p[2], nbytes ; 5 if ( pipe (p) == -1) { 6 perror (" pipe "); 7 exit (1) ; 8 } 9 if ( fork () == 0) 10 write (p[1], msg, MSG_SIZE ); 11 else { 12 while (( nbytes = read ( p[0], buf, BUF_SIZE )) > 0) 13 write ( STDOUT_FILENO, buf, nbytes ); 14 } 15 return 0; 16 } 19 / 36
Closing Ends of Pipes When a process does a read on a pipe it blocks until data is available in the pipe. As long as any process has the writing end open, a read on a pipe will wait forever if the pipe is empty. If no process has the writing end open, read detects eof and returns 0. Like all FDs, a process closes any open pipe FDs when it exits. 20 / 36
Pipe and eof again Example (pipe4.c) The parent closes its writing end. When the child terminates, no process has the writing end open, and the parent s read returns 0. 1 # define BUF_ SIZE 4 // Small! 2 # define MSG_ SIZE 13 3 char * msg = " hello, world \n"; 4 5 int main ( void ) 6 { 7 char buf [ BUF_SIZE ]; 21 / 36
1 int p[2], nbytes ; 2 if ( pipe (p) == -1) { 3 perror (" pipe "); 4 exit (1) ; 5 } 6 if ( fork () == 0) 7 write (p[1], msg, MSG_SIZE ); 8 else { 9 close (p [1]) ; 10 while (( nbytes = read (p[0], buf, BUF_SIZE )) > 0) 11 write ( STDOUT_FILENO, buf, nbytes ); 12 } 13 return 0; 14 } 22 / 36
Closing Ends of Pipes... SIGPIPE Attempting to write to a pipe when no process has the reading end open produces the SIGPIPE signal that terminates your program. Generally each process will either read or write a pipe but not both always close the end that it is not using. 1. When reading from a pipe, close p[1] before reading. 2. When writing to a pipe, close p[0] before writing. 3. Close both ends when done with the pipe. 23 / 36
I/O on Pipes From man 7 pipe, under I/O on pipes and FIFOs heading: 1. Pipes have no message boundaries. If two writes occur and then you read, do not assume that you will get just the first one! 2. writes into a pipe are guaranteed to be atomic only if they are smaller than a certain number of bytes (on Linux, 4KB) If two processes write to a pipe, and both write less than 4K with a write call, the pipe is guaranteed to hold all data from one process and then all data from the other. If two processes write to a pipe, and one writes more than 4K with a write call, the data could be interleaved in the pipe. 24 / 36
Pipe Capacity From man 7 pipe, under Pipe capacity heading: 3. Pipes have a finite size: Default size is 64KB on Linux 4. If a write would overflow a pipe, the process blocks (or fails, if O NONBLOCK flag is set) until another process empties some of the pipe with read. 25 / 36
Worksheet In the last worksheet we wrote a program that forked one child for each command line argument. The child computed the length of the command line argument and exits with that integer as the return value. The parent sums these return codes and reports the total length of all the command line arguments together. For this worksheet: write a similar program where each child communicates the length to the parent through a pipe rather than via a return code. Use the framework on the next slide. 26 / 36
1 int main ( int argc, char ** argv ) { 2 // Declare any variables you need 3 // write the code to loop over the command line arguments 4 for ( int i = 1; i < argc ; i ++) { 5 // Before we call fork. Call pipe. 6 int result = fork (); 7 if ( result < 0) { 8 perror (" fork "); 9 exit (1) ; 10 } else if ( result == 0) { // child process 11 // close child writing fds 12 // close parent 's reading fds 13 exit (0) ; 14 } else { 15 // in the parent before next loop iteration 16 } 17 } 18 // Only the parent gets here 27 / 36
Bitwise Shift Operators Bitwise operations will help us with UNIX signals and FD flags, among other things. i << j shifts each bit in i to be j places to the left. For each position shifted off to the left side, a 0 bit is filled in from the right-hand side. 1 char i, j; 2 3 i = 13; /* binary 00001101 */ 4 j = i << 2; /* binary 00110100 */ 28 / 36
Bitwise Shift Operators (cont d) i >> j shifts each bit in i to be j places to the right. For each position shifted to the right, a new leading bit is filled in from the left. Unsigned types: new leading bit is always a 0 Signed types: new leading bit is a copy of the old leading bit (to preserve the sign) 29 / 36
Bitwise Shift Operators (cont d) 1 char i, j; 2 i = 13; /* binary 00001101 */ 3 j = i >> 2; /* binary 00000011 */ 4 5 i = -13; /* binary 11110011 */ 6 j = i >> 2; /* binary 11111100 */ 7 8 i = -13; /* binary 11110011 */ 9 j = ( unsigned char ) i >> 2; /* binary 00111100 */ 30 / 36
Bitwise AND &: bitwise AND operator 1 unsigned char i, j, k ; 2 3 i = 2 1 ; / binary 00010101 / 4 j = 5 6 ; / binary 00111000 / 5 k = i & j ; / binary 00010000 / 31 / 36
Bitwise OR : bitwise OR operator 1 unsigned char i, j, k ; 2 3 i = 2 1 ; / binary 00010101 / 4 j = 5 6 ; / binary 00111000 / 5 k = i j ; / binary 00111101 / 32 / 36
Bitwise NOT 1 unsigned char i, k ; ~: bitwise NOT operator 2 3 i = 2 1 ; / binary 00010101 / 4 k = i ; / binary 11101010 / 33 / 36
Bitwise XOR ^: bitwise XOR operator 1 unsigned char i, j, k ; 2 3 i = 2 1 ; / binary 00010101 / 4 j = 5 6 ; / binary 00111000 / 5 k = i ˆ j ; / binary 00101101 / 34 / 36
Setting and Clearing a Bit Bits are numbered from the right, starting at 0.... b16 b15... b1 b0 Example To set bit b of integer n to 1: n = n (1 << b); Example To clear bit b of integer n to 0: n = n & ~(1 << b); 35 / 36
Question How can you test whether a particular bit b is set in integer n? 36 / 36