IPC SAP A Family of Object-Oriented Interfaces for Local and Remote Interprocess Communication

Transcription

1 IPC SAP A Family of Object-Oriented Interfaces for Local and Remote Interprocess Communication Douglas C. Schmidt schmidt@cs.wustl.edu Department of Computer Science Washington University, St. Louis (TEL) , (FAX) An earlier version of this paper appeared in the November/December 1992 issue of the C++ Report magazine. An expanded version of this paper [1] that includes performance results over Ethernet and ATM networks is available at the WWW URL 1 Introduction This is part of a series that describes techniques for encapsulating existing operating system (OS) interprocess communication (IPC) mechanisms within object-oriented C++ wrappers. This paper focuses on the IPC SAP ( Inter- Process Communication Service Access Point ) C++ class categories. IPC SAP provides applications with a family of object-oriented service access points that encapsulate standard network programming interfaces such as BSD sockets [2], the System V transport layer interface (TLI) [3], SVR4 STREAM pipes [4], UNIX FIFOs [5], and Windows NT named pipes [6]. In turn, these network programming interfaces shield developers and applications from nonportable details of the underlying OS local and remote IPC mechanisms. IPC mechanisms include standard connectionoriented and connectionless protocols (such as TCP, UDP, and IPX/SPX) available in modern operating systems (such as BSD/SVR4 UNIX, Windows, Windows NT, and OS/2). IPC SAP utilizes object-oriented techniques and C++ features to provide a rich set of components that simplify the development of communication software. This paper is organized as follows: Section 2 outlines the levels of abstraction for programming communication software; Section 3 describes existing network programming interfaces and discusses their limitations; Section 4 presents the object-oriented design and implementation of IPC SAP and explains how it overcomes limitations with existing interfaces; Section 5 examines the object-oriented design of the C++ wrappers for sockets, TLI, STREAM pipes, and FIFOs in detail; Section 6 illustrates several examples that use IPC SAP to implement a client/server application; and Section 7 discusses the advantages and disadvantages of using C++ to develop object-oriented wrappers. 2 Network Programming Interfaces Writing robust, extensible, and efficient communication software is a complex and challenging task. Developers must master many complex OS and communication concepts such as: Network addressing and service identification; Presentation conversions (e.g., encryption, compression, and network byte-ordering conversions between heterogeneous end-systems with alternative processor byte-orderings); Process and thread creation and synchronization; System call and library routine interfaces to local and remote interprocess communication (IPC) mechanisms. Various programming tools and interfaces have been created to help simplify the development of communication software. Figure 1 illustrates the IPC interfaces available on modern OS platforms (such as SVR4 UNIX). As shown in the figure, applications may access several levels of network programming interfaces for local and remote IPC. The remainder of this section outlines each level of abstraction, ranging from high-level RPC toolkits to transport layer interfaces to low-level kernel programming mechanisms. 2.1 RPC Toolkits Applications that exchange data with clients in a requestresponse fashion (such as network file servers) are often developed using code produced by remote procedure call (RPC) toolkits [7]. RPC toolkits provide a set of high-level tools (such stub generators and directory services) that support request-response and oneway communication. RPC toolkits are defined broadly to include the Open Network Computing (ONC+) transport-independent RPC facility [7], the Common Object Request Broker Architecture (CORBA) [8], the OSF Distributed Computing Environment (DCE) 1 [9] and OODCE [10], and Microsoft s OLE/COM [11]. RPC toolkits automate many tedious and error-prone aspects of these application development including 1 DCE also provides support for bulk data transfer via its pipe datatype, which is not provided by CORBA or ONC RPC. 1

2 Figure 1: Levels of Abstraction for Network Programming Authentication, authorization, and data security; Service location and binding; Service registration and activation; Demultiplexing and dispatching in response to events; Implementing message framing atop bytestreamoriented communication protocols like TCP; Presentation conversion issues involving network byteordering and parameter marshaling. In addition, RPC toolkits shield developers from the complexities of lower-level OS system calls that transmit and receive packets across a network. 2.2 Transport Layer Interfaces RPC toolkits are typically built upon transport layer interfaces such as BSD UNIX sockets [2], the System V UNIX Transport Layer Interface (TLI) [3], or Windows NT named pipes [6]. Compared with the higher-level RPC toolkits, there are several advantages to directly programming applications via transport layer interfaces: Omit unnecessary functionality Applications may omit unnecessary functionality (such as presentation layer conversions for ASCII data); Allow fine-grained control over behavior Transport interfaces enable finer-grain control over behavior (such as permitting multicast transmission and signal-driven asynchronous I/O); Increase Portability Transport interface (particularly sockets) are available across a wider range of OS platforms compared with RPC toolkits. The request-response and oneway communication mechanisms provided by RPC toolkits are not well-suited for a certain class of applications, known as streaming applications. Streaming applications are characterized by high-bandwidth, long-duration communication of untyped bytestreams or relatively simple datatypes that possess stringent communication performance requirements. Interactive teleconferencing, medical imaging, and video-on-demand are examples of streaming applications. The quality of service requirements of streaming applications frequently cannot tolerate the performance overhead caused by RPC. This overhead stems from non-optimized presentation format conversions, non-optimized memory management, inefficient receiver-side demultiplexing, stopand-wait flow control, synchronous send-side method invocations, and non-adaptive retransmission timer schemes. Traditionally, meeting the requirements of streaming applications has involved direct access to network programming interfaces such as sockets [2] or System V TLI [3]. 2.3 Lower-level Programming Interfaces Lower-level IPC mechanisms provide interfaces to an OS kernel s communication subsystem. For example, the SVR4 putmsg and getmsg system calls may be used to directly access the transport provider interface (TPI) [12] and the data-link provider interface (DLPI) [13] available in System V STREAMS [14]. It is also possible to develop network services (such as routers or network file systems) that reside entirely within the OS kernel [4]. However, programming at this level is usually not portable between different OS platforms nor often even different versions of the same OS. 2.4 Evaluation Compared with RPC toolkits, it is more challenging to program distributed applications using transport-level or lowerlevel IPC interfaces. Conventional transport layer programming libraries (such as sockets and TLI) lack type-safe, portable, re-entrant, and extensible interfaces. For instance, socket endpoints are implemented via weakly-typed descriptors that increase the potential for subtle errors to occur at run-time [15]. The IPC SAP class category 2 provides an object-oriented interface that encapsulates much of the complexity of transport layer interfaces. The goals of IPC SAP are to improve the correctness, ease of use, and portability/reusability of communication software, without adversely affecting its performance. IPC SAP is currently being used on many commercial projects including Motorola, Ericsson, Bellcore, Siemens, and Credit Suisse. 3 Survey of Network Programming Interfaces 3.1 Background This section surveys the behavior and limitations of widely available network programming interfaces. In BSD/SVR4 UNIX and Windows NT, the communication protocol stacks reside within the protected address space of the OS kernel. Application programs running in user-space access the 2 A class category is a collection of components that collaborate to provide a set of related interfaces and services [16]. 2

3 kernel-resident protocol stacks via interfaces such as BSD UNIX sockets, System V UNIX TLI, or Win32 named pipes. These interfaces manage local and remote endpoints of communication by allowing applications to open connections to remote hosts, negotiate and enable/disable certain options, exchange data, and close all or part of the connections when transmissions are complete. In UNIX, sockets and TLI are loosely modeled on the standard file I/O interface, which includes the open, read, write, close, ioctl, lseek, and select routines [14]. However, sockets and TLI provide additional functionality that is not supported directly by the standard UNIX file I/O interfaces. This functionality stems from certain syntactic and semantic differences between file I/O and network I/O. For example, the pathnames use to identify files on a UNIX system are not globally unique across hosts in a largescale distributed environment. Therefore, a different naming scheme (such as IP host addresses) is used to uniquely identify network applications. The socket and TLI interfaces provide similar functionality. They support a general-purpose interface to multiple communication domains. A domain specifies a protocol family and an address family. Each protocol family contains a stack of protocols that implement certain types of communication in the domain. Common protocol stacks provide reliable, bi-directional, connection-oriented, message and bytestream services (based on protocols such as TCP, TP4, and SPX), as well as unreliable, connectionless datagram service (based on protocols such as UDP, CLNP, and IPX). An address family defines an address format (e.g., the address size in bytes, number and type of fields, and order of fields) and a set of kernel-resident subroutines that interpret the address format (e.g., to determine which subnet an IP datagram is destined for). Section 3.2 gives an overview on BSD sockets, Section 3.3 briefly outlines TLI, Section 3.4 covers STREAM pipes, and Section 3.5 discusses UNIX FIFOs. A complete discussion of these interfaces is beyond the scope of this paper (see [4, 2, 6, 5, 17] for additional details). 3.2 BSD Sockets The socket interface was originally developed in BSD UNIX to provide an interface to the TCP/IP protocol suite [2]. From an application s perspective, a socket is a local endpoint of communication that is bound to an address residing on a local or a remote host. Sockets are accessed via descriptors. A descriptor is an unsigned integer handle that indexes into a table of descriptors. This table is maintained in the OS kernel on a per-process basis. In UNIX, socket descriptors share the same name space as other descriptors (such as file, pipe, and terminal device descriptors). Descriptors provide an encapsulation mechanism that shields applications from knowledge of internal kernel data structures. Descriptor tables reside within an OS kernel. Thus, hardware memory protection mechanisms are used to prevent accidental cor- Figure 2: Socket Interface ruption by applications. Note that this type of encapsulation is fairly heavyweight since an application must perform a mode switch to change from user-mode to kernel-mode. The socket interface shown in Figure 2 contains around two dozen system calls that may be classified into the following general categories: Local Context Management: The socket interface provides the following routines for managing local context information: socket which allocates the lowest unused socket descriptor; bind which associates a socket descriptor with a local or remote address; getsockname and getpeername which determine the local or remote address, respectively, a socket is connected with; close which deallocates a socket descriptor, making it available for subsequent reuse. Connection Establishment and Connection Termination: The socket interface provides the following routines for establishing and terminating connections: connect a client typically uses connect to actively establish a connection with a server; listen a server uses listen to indicate it willingness to listen passively for incoming client connection requests; accept a server uses accept to create a new endpoint of communication to service a client; shutdown which selectively terminates the read-side and/or write-side stream of a bi-directional connection. Data Transfer Mechanisms: The socket interface provides the following routines that send and receive data: read/write which receive and transfer buffers of data via a particular descriptor; send/recv which are similar to read/write, though they provide an extra parameter that controls certain socket-specific operations (such as exchanging urgent data or peeking at data in a receive queue without removing it from the queue); 3

4 sendto/recvfrom which exchange connectionless datagrams; readv/writev which support scatter-read and gather-write semantics, respectively (these operations optimize user/kernel mode switching and simplify memory management); sendmsg/recvmsg which are general-purpose routines that subsume the behavior of all the other data transfer routines. For UNIX domain sockets, the sendmsg and recvmsg routines also provide the ability to pass access rights (such as open file descriptors) between arbitrary processes on the same host machine. Note that these interfaces may also be used for other types of I/O such as files and terminals. Options Management: The socket interface defines the following routines that allow users to alter the default semantics of socket behavior: setsockopt and getsockopt which modify or query options within different layers in a protocol stack. Options include multicasting, broadcasting, and setting/getting the size of send and receive transport buffers; fcntl and ioctl which are UNIX system calls that enable asynchronous I/O, non-blocking I/O, and urgent message delivery on sockets. In addition to the socket routines describe above, communication software may use the following standard library routines and system calls: gethostbyname and gethostbyaddr which handle various aspects of network addressing such as mapping host names to IP addresses; getservbyname which identifies services by their port numbers or humanly-readable names; ntohl, ntohs, htonl, htons which perform network byte-order transformations; select which performs I/O-based and timer-based event demultiplexing on sets of open descriptors. 3.3 System V TLI TLI is an alternative interface for accessing communication protocol stacks. TLI provides basically the same set of services that sockets does. However, it places greater emphasis on shielding applications from the details of the underlying transport provider. [4] discusses TLI in detail. 3.4 STREAM Pipes STREAM pipes are an enhancement to the original UNIX pipe mechanism. Earlier generation UNIX pipes provided a single uni-directional stream of bytes from a writer endpoint to a reader endpoint. STREAM pipes support bi-directional delivery of bytestream and prioritized message data between processes and/or threads executing on the same host machine [17]. Although the pipe system call interface remains the same, STREAM pipes offer additional functionality that is roughly equivalent to UNIX-domain SOCK STREAM sockets. They are somewhat more flexible than UNIX-domain sockets, however, since they enable STREAM modules to be pushed and popped to and from pipe endpoints. By default, a STREAM pipe provides only a single channel of data between its two endpoints. Therefore, if multiple senders write to the pipe all the messages are placed into the same communication channel. This is often too restrictive since it is hard to multiplex data from multiple clients over a single channel. For example, each messages must include an identifier that enables the receiver to determine which sender transmitted which message. By using mounted STREAM pipes and the connld module [18], applications may dedicate a separate non-multiplexed I/O channel between a server and each instance of a client. STREAM pipes and connld work as follows. The server invokes the pipe system call, creating a bi-directional endpoint of communication. The fattach system call mounts a pipe descriptor at a designated location in the UNIX file system. A server application may be created by pushing the connld STREAM module onto the mounted end of the STREAM pipe. A client application running on the same host machine as the server subsequently opens the filename associated with the mounted pipe. At this point, the connld module ensures that the client and server each receive a unique I/O descriptor identifying a non-multiplexed, bi-directional channel of communication. 3.5 FIFOs UNIX FIFOs (also called named pipes) are a restricted form of a STREAM pipe. Unlike STREAM pipes, FIFOs offer only a uni-directional data channel from one or more senders to a single receiver. Moreover, messages from different senders are all placed into the same communication channel. Therefore, some type of demultiplexing identifier must be included explicitly in each message to enable the receiver to determine which sender transmitted the message. The STREAMS-based implementation of FIFOs in SVR4 UNIX provides both message and bytestream data delivery semantics. In contrast, earlier versions of UNIX (such as SVR3 and SunOS 4.x) only provide bytestream-oriented FI- FOs. Therefore, unless fixed length messages are always used, each message sent via a FIFO must be distinguished by some form of byte count or special termination symbol. This allows a receiver to extract messages from the FIFO bytestream. FIFOs are described further in [4, 5, 17]. 3.6 Limitations with Existing IPC interfaces Sockets, TLI, STREAM pipes, and FIFOs provide a wide range of interfaces for accessing local and remote IPC mechanisms. These interfaces share several limitations. This 4

5 #include <sys/types.h> #include <sys/socket.h> const int PORT_NUM = 10000; int buggy_echo_server (void) sockaddr s_addr; int length; // (1) uninitialized variable. char buf[bufsiz]; int s_fd, n_fd; // Create a local endpoint of communication. if (s_fd = socket (PF_UNIX, SOCK_DGRAM, 0) == -1) return -1; // Set up the address information to become a server. // (2) forgot to "zero out" structure first... s_addr.sin_family = AF_INET; // (3) used the wrong address family... s_addr.sin_port = PORT_NUM; // (4) forgot to use htons() on PORT_NUM... s_addr.sin_addr.s_addr = INADDR_ANY; if (bind (s_fd, (sockaddr *) &s_addr, sizeof s_addr) == -1) perror ("bind"), exit (1); // (5) forgot to call listen() // Create a new endpoint of communication. // (6) doesn t make sense to accept a SOCK_DGRAM! if (n_fd = accept (s_fd, &s_addr, &length) == -1) // (7) Omitted a crucial set of parens... int n; // (8) doesn t make sense to read from the s_fd! while ((n = read (s_fd, buf, sizeof buf)) > 0) // (9) forgot to check for "short-writes" write (n_fd, buf, n); // Remainder omitted... Figure 3: Buggy Echo Server following discussion focuses on sockets, though many of the limitations also apply to the other network programming interfaces, as well. High Potential for Error: In UNIX, descriptors for sockets, files, pipes, terminals, and other devices are identified using weakly-typed integer values. This weak type checking enables subtle errors to occur at run-time. For example, the socket interface cannot enforce the correct use of socket routines for different communication roles (such as active vs. passive connection establishment or datagram vs. stream communication). A compiler cannot detect or prevent erroneous use of descriptors since descriptors are weakly typed. Thus, operations may be applied incorrectly on descriptors (e.g., invoking a data transfer operation on a descriptor set up for establishing connections). Figure 3 depicts the following subtle (and all-too-common) bugs that occur when using the socket interface: 1. Forgetting to initialize the len parameter to accept to the size of struct sockaddr in; 2. Forgetting to initialize all bytes in the socket address structure to 0 ; 3. Using an address family type that is inconsistent with the protocol family of the socket; 4. Neglecting to use the htons library function to convert port numbers from host byte-order to network byte-order and vice versa; 5. Omitting the listen system call when creating a passive-mode SOCK STREAM socket; 6. Applying the accept function on a SOCK DGRAM socket; 7. Erroneously omitting a key set of parentheses in an assignment expression; 8. Trying to read from a passive-mode socket that should only be used to accept connections; 9. Failing to properly detect and handle short-writes that occur due to buffering. Several of the problems listed above are classic problems with C. For instance, by omitting the parentheses in the expression if (n_fd = accept (s_fd, &s_addr, &length) == -1) the value of n fd will always be set to either 0 or 1, depending on whether accept() == -1. A deeper problem is that C data structures lack adequate abstraction. For example, the generic sockaddr address structure forces the use of typecasts to provide a form of inheritance for Internet domain and UNIX domain addresses. These subclass address structures (sockaddr in and sockaddr un, respectively) overlay the sockaddr base class. In general, the use of typecasts, combined with the weaklytyped descriptor-based socket interface, makes it very hard for a compiler to detect mistakes at compile-time. Instead, error checking is deferred until run-time, which complicates error handling and reduces application robustness. Complex Interface: Sockets support multiple protocol families (such as TCP/IP, IPX/SPX, ISO OSI, and UNIX domain sockets) with a single interface. The socket interface contains many functions to support different communication roles (such as active vs. passive connection establishment), communication optimizations (such as writev that sends multiple buffers in a single system call), and options for infrequently used operations such as broadcasting, multicasting, asynchronous I/O, and urgent data delivery. Although sockets combine this functionality into a common library, the resulting interface is complex and hard to master. This complexity stems from the overly broad and one-dimensional design of the socket interface. That is, all the routines appear at a single level of abstraction (as shown in Figure 2). This design increases the amount of effort required to learn and use sockets correctly. In particular, programmers must understand the entire the socket interface in order to use only part of it effectively. If sockets are examined carefully, however, it becomes clear that the interface may be decomposed into clusters of functionality. Figure 4 clusters the related socket routines according to the following three criteria: 1. Type of communication service i.e., stream vs. datagram vs. connected datagram; 5

6 IPC_SAP SOCK_SAP TLI_SAP SPIPE_SAP FIFO_SAP A Figure 4: Socket Dimensions 2. Communicationrole i.e., active vs. passive (clients are typically active, whereas servers are typically passive); 3. Communication domain i.e., local vs. local/remote. Since the interface is one-dimensional, however, this natural clustering is obscured. Section 5 illustrates how this classification is restructured into a hierarchy of classes that simplify the socket interface and enhance the type-safety of communication software. Another problem with the socket interface is that its several dozen routines lack a uniform naming convention. Nonuniform naming makes it hard to determine the scope of the socket interface. For example, it is not immediately obvious that socket, bind, accept, and connect are related. Other network programming interfaces solve this problem by prepending common prefix before each routine. For example, a t is prepended before each routine in the TLI library. However, the TLI interface also contains operations with overly complex semantics. For example, unlike the socket interface, the TLI option handling interface is not specified in a standard manner. This makes it hard to write portable applications that access standard TCP/IP options. Likewise, subtle application-level code is necessary to handle the nonintuitive, error-prone behavior of t listen and t accept in a concurrent server with a qlen > 1 [4]. 4 Overview of IPC SAP IPC SAP encapsulates the standard descriptor-based IPC interfaces such as sockets, TLI, STREAM pipes, and FI- FOs. As shown in Figure 5, IPC SAP is designed as a forest of class categories that includes SOCK SAP (which encapsulates the socket interface), TLI SAP (which encapsulates the TLI interface), SPIPE SAP (which encapsulates the UNIX SVR4 STREAM pipe interface), and FIFO SAP (which encapsulates the UNIX FIFO interface). SOCKET API TRANSPORT LAYER INTERFACE API STREAM PIPE API Figure 5: IPC SAP Class Category Relationships NAMED PIPE API Each class category is organized as an inheritance hierarchy. Every subclass provides a well-defined interface to a subset of existing IPC mechanisms. Together, the subclasses within a hierarchy comprise the overall functionality of a particular communication abstraction (such as the Internet-domain or UNIX-domain protocol families). This section describes the design goals of IPC SAP, outlines its class categories, and discusses the principles that underly its object-oriented design. 4.1 IPC SAP Design Goals IPC SAP is designed to improve the correctness, ease of learning and ease of use, portability, and reusability of communication software, while maintaining a high level of performance. This section discusses how IPC SAP achieves these goals. Correctness: Several of the problems with sockets are related to its weak typechecking. IPC SAP improves the correctness of application networking code by permitting only type-safe operations on instances of its classes. To enforce type-safety, IPC SAP ensures all of its objects are properly initialized via constructors. In addition, only well-defined operations are permitted on IPC SAP objects. IPC SAP is specifically designed to prevent accidental violations of typesafety. For example, components in the SOCK SAP class category prevent the accidental use of connection-oriented operations on datagram objects. Therefore, it is impossible to invoke the accept method on a datagram object, recv or send data on a connector or acceptor factory object, or invoke a sendto operation on a connection-oriented object. Since IPC SAP classes are strongly typed, any attempts to perform invalid operations are rejected at compile-time rather than at run-time. This point is illustrated in the revised SOCK SAP revision of buggy echo server shown in Figure 6. This example fixes all the problems with sockets identified in Figure 3. Ease of Learning and Ease of Use: Simplifying the use of common IPC operations is a goal related to correctness. By providing simpler interfaces, developers are able to concentrate on writing applications, instead of wrestling with 6

7 const int PORT_NUM 10000; int buggy_echo_server (void) // Address of local server. INET_Addr local_addr (PORT_NUM); // Initiate the passive mode server. SOCK_Acceptor acceptor (local_addr); // Check to make sure acceptor is active. if (acceptor.get_handle () == -1) perror ("open"), exit (1); SOCK_Stream new_stream; // Data transfer object. INET_Addr client_addr; // Client address object. // Accept a new connection. if (acceptor.accept (new_stream, &client_addr)) char buf[bufsiz]; ssize_t n; while ((n = new_stream.recv (buf, sizeof buf)) > 0) if (new_stream.send_n (buf, n)!= n) // Remainder omitted. Figure 6: SOCK SAP Version of the Buggy Echo Server Figure 8: Using the IPC SAP C++ Wrapper to Increase Portability Figure 7: The IPC SAP Address Class Hierarchy the networking code. In general, IPC SAP simplifies its network programming interfaces as follows: Designing auxiliary classes that shield applications from error-prone details e.g., IPC SAP contains the Addr class hierarchy (shown in Figure 7). This hierarchy supports several diverse network addressing formats via a type-secure C++ interface. The Addr hierarchy eliminates several common programming errors (such as forgetting to zero-out a sockaddr addressing structure) associated with using the C-based struct sockaddr data structures directly; Combining several operations to form a single operation e.g., the SOCK Acceptor is a factory for passive connection establishment. Its constructor performs the various socket system calls (such as socket, bind, and listen) required to create a passive-mode server endpoint; Supplying default parameters for typical method argument values e.g., the addressing parameters to accept are frequently NULL pointers. To simplify programming, these values are given as defaults in SOCK Acceptor::accept, so that programmers don t have to provide them. Reusability: Inheritance-based hierarchical decomposition is used in IPC SAP to increase the amount of common code that is shared amongst the various IPC mechanisms (such as the C++ interface to the lower-level OS device control system calls like fcntl and ioctl). Inheritance enhances reuse within the IPC SAP implementation by sharing code between different subclasses. For example, the IPC SAP root base class provides standard methods and data that is shared by the other derived classes. These shared components provide an descriptor variable and its related set/get methods, as well as methods that enable and disable asynchronous I/O, non-blocking I/O, and urgent message delivery on the descriptor. Portability: Several C++ features are used to enhance portability. For example, IPC SAP provides a platformindependent interface that improves communication software portability by using C++ templates. As illustrated in Figure 8, a subset of the SOCK SAP and TLI SAP classes offer the same object-oriented interface. Each platform may possess a different underlying interface for local and remote network programming (e.g., sockets vs. TLI). However, it is possible to write applications that may be transparently parameterized with either class. This enhances application portability across OS platforms that may not support both sockets and TLI. The use of classes (as opposed to stand-alone functions) helps to simplify network programming by allowing applications to be parameterized by the type of IPC mechanism they requires. As discussed in Section 4.2, parameterization helps to improve portability among platforms that support different network programming interfaces (such as sockets or TLI). Performance: To encourage developers to substitute IPC SAP for existing interfaces, the implementation is designed to operate efficiently. Several techniques help improve performance without sacrificing clarity and modularity. 7

8 First, most methods are specified as C++ inline functions. This eliminates the extra run-time overhead of calling IPC SAP methods. Inlining is a reasonable approach since each method is very short (averaging approximately 3 lines per method). Second, virtual functions are not used in the IPC SAP inheritance hierarchy. This improves performance since (1) it eliminates indirect vtable function pointer dispatching and (2) it facilitates the direct inlining of certain short, frequently-accessed methods (such as sending and receiving user data). 4.2 IPC SAP Design Principles The following design principles were applied throughout the IPC SAP architecture: Simplify for the Common Case: The first principle is make it easy to use IPC SAP correctly, hard to use it incorrectly, but not impossible to use it in ways the class designers did not anticipate originally. This principle is exemplified by the get handle and set handle operators provided by the IPC SAP root class. These methods which extract and assign the underlying descriptor, respectively. By providing get handle and set handle, IPC SAP allows applications to circumvent its type-checking mechanisms for infrequent situations where applications must interface directly with UNIX system calls (such as select) that expect a descriptor. Define Parsimonious Interfaces: The second principle involves interface parsimony, which attempts to localize the costs of using a particular abstraction. The IPC SAP interfaces provide developers with precisely the functionality they need for various types of communication (such as connection-oriented vs. connectionless) and various roles (such as active vs. passive). This is beneficial since it limits the amount of details that application developers must remember to use each class properly. For example, to reduce the change of error, the SOCK Acceptor class only permits operations that apply for applications playing passive roles and the SOCK Connector class only permits operations that apply for applications playing an active role. In addition, sending and receiving open file descriptors has a much simpler calling interface using IPC SAP compared with using the highly-general sendmsg/recvmsg routines. Replace One-dimensional Interfaces with Hierarchically-related class categories: A third design principle employs hierarchically-related class categories to restructure existing one-dimensional network programming interfaces. C++ s inheritance mechanisms support different subsets of functionality for the IPC SAP class categories. For instance, since not all operating systems support passing open file descriptors (e.g., MS-DOS), it is possible to omit the LSOCK class (described in Section 5.1.1) from the inheritance hierarchy without disturbing the interfaces of other classes in the IPC SAP design. In general, inheritance and parameterization are used heavily to develop IPC SAP class categories. These features improve modularity and increase code reuse. Base classes are used to express similarities between class category components and derived classes are used to express the differences. For example, the SOCK SAP design places shared mechanisms towards the root of the inheritance hierarchy. These mechanisms include operations for opening/closing and setting/retrieving the underlying socket descriptors, as well as certain option management functions that are common to all the derived SOCK SAP classes. Subclasses located towards the bottom of the inheritance hierarchy implement specialized operations that are customized for the type of communication provided (such as stream vs. datagram communication or local vs. remote IPC). This approach avoids unnecessary duplication of code since the more specialized derived classes reuse the more general mechanisms provided at the root of the inheritance hierarchy. C++ templates are useful since they enable reusable code without spending excessive effort developing an interface that encompasses all potential functionality. Instead, templates are used to parameterize application code that is carefully designed to invoke only a subset of methods that are common to the various communication abstractions (e.g., open, close, send, recv, etc.). 5 The Object-Oriented Design of IPC SAP This section describes the object-oriented design of the C++ class categories that comprise the IPC SAP class categories. Particular emphasis is given to the design of the SOCK SAP C++ wrapper for sockets. SOCK SAP has been ported to many UNIX platforms, as well as to the WinSock transport layer programming interface available on Windows, Windows NT, Chicago, and OS/2 [19]. 5.1 SOCK SAP The SOCK SAP class category provides applications with an object-oriented interface to the Internet-domain and UNIXdomain protocol families [5]. Applications may access the functionality of the underlying Internet-domain or UNIXdomain socket types by inheriting or instantiating the appropriate SOCK SAP subclasses shown in Figure 9. The SOCK* subclasses encapsulate Internet-domain functionality and the LSOCK* subclasses encapsulate UNIX-domain functionality. As shown in Figure 9, the subclasses may be further decomposed into the following components: *Dgram vs. *Stream subclasses the *Dgram subclasses provide unreliable, connectionless, message functionality whereas the *Stream components provide reliable, connection-oriented, bytestream functionality; 8

9 *Acceptor and *Connector vs. *Stream subclasses passive and active connection establishment functionality is provided to servers by the *Acceptor and *Connector subclasses, respectively whereas the *Stream components provide bidirectional bytestream data transfer functionality used by applications that play both client and server roles. Using C++ wrappers to encapsulate the socket interface helps to (1) detect many subtle application type system violations at compile-time, (2) facilitate a platform-independent transport-level interface that improves application portability, and (3) greatly reduce the amount of application code and development effort expended upon lower-level network programming details. To illustrate the latter point, the following example program implements a simple client application. This application uses the SOCK Dgram Bcast class to broadcast a message to all servers listening on a designated port number in a LAN subnet: int main (int argc, char *argv[]) SOCK_Dgram_Bcast b_sap (sap_any); char *msg; unsigned short b_port; msg = argc > 1? argv[1] : "hello world\n"; b_port = argc > 2? atoi (argv[2]) : 12345; if (b_sap.send (msg, strlen (msg), b_port) == -1) perror ("can t send broadcast"), exit (1); return 0; It is instructive to compare this concise example with the dozens of lines of C source code required to implement broadcasting using the socket interface directly The SOCK SAP Class Category As shown in Figure 9, SOCK SAP consists of around one dozen C++ classes that are related by multiple inheritance and composition. These components and their relationships are illustrated via Booch notation [16]. Dashed clouds indicate classes and directed edges indicate inheritance relationships between these classes. For example, SOCK Stream inherits from SOCK. Each class provides an abstract interface for a particular subset of mechanisms that together comprise the overall SOCK SAP interface. The functionality of various types of Internet-domain and UNIX-domain sockets is achieved by inheriting mechanisms from the appropriate subclasses. The criteria for decomposing sockets into the particular class hierarchy described below involve the following Identify related classes of behavior in the existing socket interface and clustering these into three separate groups of classes shown in Figure 10: active vs. passive, localonly vs. local and remote, and stream vs. datagram; Maximize the reusability and sharing of class components. The design also distinguishes between noninstantiatable classes (i.e., abstract base classes) and instantiatable classes, as described below. Non-Instantiatable Classes: The following classes anchor the inheritance hierarchy and provide a basis for subsequent derivation and code sharing. In particular, it is not possible to instantiate objects of these classes, since their constructors are declared in the protected section of the class definition. SOCK: this class is the root of the SOCK SAP hierarchy. It provides mechanisms common to all other classes, such as opening and closing local endpoints of communication and handling options (e.g., selecting internal system buffer sizes, enabling multicasting and broadcasting, etc.). LSOCK: provides mechanisms that allow applications to send and receive open file descriptors between unrelated processes on the same host machine. Note that System V and BSD UNIX both support this feature, though Windows NT does not. Other classes inherit from LSOCK to obtain this functionality. Instantiatable Classes: User applications may instantiate the following classes to form objects that exchange data between clients and servers. Figure 10 depicts the following: Type of communication service i.e., stream-oriented vs. datagram-oriented vs. connection datagram; Communication role certain classes are involved with active connection establishment (typically performed by clients), where as other classes are involved with passive connection establishment (typically performed by servers); Communication domain the LSOCK* classes are intended for local host communication only, whereas the SOCK* classes may communicate both locally and remotely. LSOCK* and SOCK* classes are distinguished in the SOCK SAP design on the basis of network address formats and communication semantics. In particular, the LSOCK* classes use UNIX pathnames as addresses and are intended for intra-machine IPC only. The SOCK* classes, on the other hand, use Internet Protocol (IP) addresses and port numbers and allow both intra- and inter-machine IPC. It is instructive to compare Figure 4 with Figure 10. The latter is less cluttered since it uses C++ wrappers to encapsulate the behavior of multiple socket mechanisms within individual classes Connectionless Classes SOCK CODgram and LSOCK CODgram: These classes provide a connected-datagram mechanism, which 9

10 Figure 9: The SOCK SAP Class Categories allows the send and recv operations to omit the address of the service when exchanging datagrams. Note that the connected-datagram mechanism is only a syntactic convenience since there are no additional semantics associated with the data transfer (i.e., datagram delivery is still unreliable). SOCK CODgram inherits mechanisms from the SOCK Connector class. LSOCK CODgram inherits mechanisms from both SOCK CODgram and LSOCK (which provides the ability to pass file descriptors). SOCK Dgram and LSOCK Dgram: These classes provides mechanisms for exchanging datagrams between processes running on local and/or remote hosts. Unlike the connected-datagram classes described above, each send and recv operation must provide the address of the service with every datagram sent or received. LSOCK Dgram inherits all the operations of both SOCK Dgram and LSOCK. It only exchanges datagrams between processes on the same host. The SOCK Dgram class, on the other hand, may exchange datagrams between processes on local and/or remote hosts. SOCK Dgram Bcast: This class provides mechanisms for broadcasting UDP datagrams to processes running on local and/or remote hosts attached to local subnets. The interface for this class supports the broadcast of datagrams to (1) all network interfaces connected to the host machine or (2) a particular network interface. This class shields the end-user from the low-level details required to utilize broadcasting effectively. SOCK Dgram Mcast: This class provides mechanisms for multicasting UDP datagrams to processes running on local and/or remote hosts attached to local subnets. The interface for this class supports the multicast of datagrams to a particular multicast group. This class shields the end-user Figure 10: Taxonomy of SOCK SAP Communication Services from the low-level details required to utilize multicasting effectively Connection-Oriented Classes As noted in Section 3, communication software is typified by asymmetric relationships. In general, servers listen passively for clients to initiate connections actively. However, once a connection is established (either passively or actively), communication may proceed in a peer-to-peer manner. That is, data may be exchanged between the two endpoints in any manner. The structure of passive/active connection establishment and data transfer relationships are captured by 10

11 the connection-oriented SOCK SAP classes in the following manner. SOCK Acceptor and LSOCK Acceptor: The *Acceptor classes are factories [20] that passively establish new endpoints of communication in response to connection requests. The SOCK Acceptor factory produces a SOCK Stream connection endpoint object and the LSOCK Acceptor object produces new LSOCK Stream connection endpoint object. SOCK Connector and LSOCK Connector: The *Connector classes are factories that actively establish new endpoints of communication. These classes (1) identify the endpoints of communication, (2) establish connections with these endpoints, and (3) produce the appropriate *Stream object when a connection is established. A connection may be initiated either synchronously (the default) or asynchronously (using the ACE NONBLOCK flag). The SOCK Connector factory produces a SOCK Stream connection endpoint object and the LSOCK Connector object produces new LSOCK Stream connection endpoint object. SOCK Stream and LSOCK Stream: These two classes are produced by the *Acceptor or *Connector factories described above. The *Stream classes provide mechanisms for transferring data between two processes. The overloaded send and recv methods provide standard UNIX write and read semantics. Thus, a send may write less (and a recv may read more) than the requested number of bytes. This behavior is due to flow control and buffering that occurs in the OS kernel and intervening networks. To reduce programmer effort, the send n and recv n methods allow the transfer of exactly n bytes. There are also methods that provide scatter-read and gather-write semantics for sending and receiving multiple buffers of data, respectively. LSOCK Stream objects exchange bytestreams only between processes on the same host machine; SOCK Stream objects exchange data between processes that may reside on different host machines. Unlike the *Stream classes, the *Acceptor and Connector classes do not provide any methods that send or receive data. They are factories that are used solely to create new endpoints of communication. Note how the use of strongly-typed interfaces detects accidental mix-up of local and non-local SOCK Stream objects at compile-time. In contrast, the BSD socket interface detects these type mismatches at run-time Network Addressing Designing an efficient, general-purpose network addressing interface is surprisingly complicated. A major challenge arises from trying to represent different network address formats with a space efficient, uniform interface. Different address formats usually store different types of information that require different numbers of bytes. For example, an Internet-domain service (such as ftp or telnet) is identified using two fields: (1) a four-byte IP address (which uniquely identifies the remote host machine throughout the Internet) and (2) a two-byte port number (which is used to demultiplex incoming protocol data units to the appropriate client or server process on the remote host machine). In contrast, UNIX-domain sockets rendezvous via UNIX pathnames (which may be up to 108 bytes in length and are meaningful only on a single local host machine). The existing sockaddr-based network addressing structures provided by the socket interface is cumbersome and error-prone. It requires developers to explicitly initialize all the bytes in the address structure to 0 and to use explicit casts. In contrast, the IPC SAP addressing classes shown in Figure 7 contain mechanisms for manipulating network addresses. The constructors for the Addr base class ensure that all fields are automatically initialized correctly. Moreover, the different sizes, formats, and functionality that exist between different address families are encapsulated in the derived address subclasses. The UNIX Addr subclass is associated with the LSOCK* classes and the INET Addr subclass is associated with the SOCK* classes. 5.2 TLI SAP The TLI SAP class category provides a C++ interface to the System V Transport Layer Interface (TLI). The TLI SAP inheritance hierarchy for TLI is almost identical to the SOCK SAP C++ wrapper for sockets. The primary difference is that TLI and TLI SAP do not define an interface to the UNIXdomain protocol family. By combining C++ features (such as default parameter values and templates) together with the tirdwr (the read/write compatibility STREAMS module), it becomes relatively straight-forward to develop applications that may be parameterized at compile-time to operate correctly over either a socket-based or TLI-based transport interface. The following code illustrates how C++ templates may be applied to parameterize the IPC mechanisms used by an application. This code was extracted from the distributed logging facility described in [21]. In the code below, a subclass derived from Event Handler is parameterized by a particular type of transport interface and its corresponding protocol address class: // Logging_Handler header file. template <class XPORT_SAP, class ADDR> class Logging_Handler : public Event_Handler public: Logging_Handler (void); virtual Logging_Handler (void); virtual int handle_close (int); virtual int handle_input (int); virtual int get_handle (void) const return this->xport_sap.get_handle (); 11

12 protected: XPORT_SAP xport_sap; ; Depending on certain properties of the underlying OS platform (such as whether it is BSD-based SunOS 4.x or System V-based SunOS 5.x), the logging application may instantiate the Client Handler class to use either SOCK SAP or TLI SAP, as shown below: // Logging application. class Logging_Handler : public #if defined (MT_SAFE_SOCKETS) Logging_Handler<SOCK_Stream, INET_Addr> #else Logging_Handler<TLI_Stream, INET_Addr> #endif // MT_SAFE_SOCKETS. //... ; The increased flexibility offered by this template-based approach is extremely useful when developing an application that must run portability across multiple OS platforms. In particular, the ability to parameterize applications according to transport interface is necessary across variants of SunOS platforms since the socket implementation in SunOS 5.2 is not thread-safe and the TLI implementation in SunOS 4.x contains a number of serious defects. TLI SAP also shields applications from many peculiarities of the TLI interface. For example, the accept method in the TLI Acceptor class encapsulates the subtle application-level code required to handle the non-intuitive, error-prone behavior of t listen and t accept in a concurrent server with a qlen > 1 [4]. The accept method passively establishes client connection requests. Through the use of C++ default parameter values, the standard method for calling the accept method is syntactically equivalent for both TLI SAP-based and SOCK SAP-based applications. 5.3 SPIPE SAP The SPIPE SAP class category provides a C++ wrapper interface for mounted STREAM pipes and connld [18]. The SPIPE SAP inheritance hierarchy mirrors the one used for SOCK SAP and TLI SAP. It offers functionality that is similar to the SOCK SAP LSOCK* classes (which themselves encapsulate UNIX-domain sockets). However, SPIPE SAP is more flexible than the LSOCK* interface since it enables STREAM modules to be pushed and popped to and from SPIPE SAP endpoints, respectively. SPIPE SAP also supports bi-directional delivery of bytestream and prioritized message data between processes and/or threads executing within the same host machine [17]. #define PORT_NUM /* BSD socket client. */ int main (int argc, char *argv[]) struct sockaddr_in saddr; struct hostent *hp; char *host = argc > 1? argv[1] : "ics.uci.edu"; unsigned short port_num = htons (argc > 2? atoi (argv[2]) : PORT_NUM); char buf[bufsiz]; int s_fd; int w_bytes; int r_bytes; int n; /* Create a local endpoint of communication */ if ((s_fd = socket (PF_INET, SOCK_STREAM, 0)) == -1) perror ("socket"), exit (1); /* Determine IP address of the server */ if ((hp = gethostbyname (host)) == 0) perror ("gethostbyname"), exit (1); /* Set up address information to contact the server */ memset ((void *) &saddr, 0, sizeof saddr); saddr.sin_family = AF_INET; saddr.sin_port = port_num; memcpy (&saddr.sin_addr, hp->h_addr, hp->h_length); /* Establish connection with remote server */ if (connect (s_fd, (struct sockaddr *) &saddr, sizeof saddr) == -1) perror ("connect"), exit (1); /* Send data to server (correctly handles "incomplete writes" due to flow control) */ while ((r_bytes = read (0, buf, sizeof buf)) > 0) for (w_bytes = 0; w_bytes < r_bytes; w_bytes += n) if ((n = write (s_fd, buf + w_bytes, r_bytes - w_bytes)) < 0) perror ("write"), exit (1); /* Explicitly close the connection */ if (close (s_fd) == -1) perror ("close"), exit (1); return 0; Figure 11: Socket-Based Client Example 5.4 FIFO SAP The FIFO SAP class category encapsulates the UNIX FIFO mechanism. 12