Rx: Extended Remote Procedure Call

Transcription

1 Rx: Extended Remote Procedure Call

2 1 1 Introduction Please note that this is a preliminary version of this document, and is greatly subject to change. Rx is under development. Rx is a Remote Procedure Call protocol extended to allow arbitrary amounts of data in a request or a response and to provide general support for end-to-end authentication and security. In addition, the protocol is rich enough to allow clients and servers to adapt to differences in their relative performance, to lossy links, and to network delays. Bulk data transfer is accomplished in this protocol by sending or receiving the entire bulk object in a single request or response. This protocol guarantees exactly-once RPC semantics in the absence of crashes or prolonged network failure. In the presence of server crashes, however, a call may potentially execute more than once (in different instantiations of the server), without any direct notification to the client. Optionally, the client may request to be notified on server crash detection. In this case, all of the calls on a connection will be executed by a single server instance. Once the server crashes, all calls on that connection will fail. Prolonged network failure will be treated by the client as if a server crash had occured. A typical Rx RPC call consists of the following stages: The client sends a number of data packets, constituting the request, to a selected server. The low-level Rx protocol ensures that these packets are delivered to the server if it is capable of receiving them. The server computes a result and transmits this back to the client as a sequence of response data packets. These are received by the client, acknowledged as appropriate, and the call completes. Note that any given call is "half-duplex": the server does not send a response until the entire set of call packets has been received from the client. The client is not expected to begin the next call until it has received the results of the current call. Rx is currently implemented on top of the UDP datagram protocol from the Internet protocol suite. This choice was made based on the requirements of both a user-level implementation and a kernel-level implementation of Rx for 4.3 BSD UNIX, with the additional requirement that these implementations be able to coexist (the in-kernel version of Rx is intended for use by inkernel clients, only). 1.1 Services An Rx-based server exports services. A service is addressed by a client using the triple (host address, UDP port, service Id). The host address and UDP port number are artifacts of our IP environment, but could be replaced by a more general addressing scheme. The service ID allows multiple, independently specified services to be exported by a single server process. For example, a file server might export the basic file service, a file server administration service, and an error-logging service (called by a central error monitoring program). Each service independently specifies which security methods clients may elect to use for any connections to the service, and must be supplied with at least one instance of a security class to actually implement security procedures at the server. Scheduling parameters are also implemented by Rx for each service.

3 2 1.2 Connections In order to access a remote service, a client makes a new connection to the service, providing the addressing information described above. In addition, the client specifies which security class (if any) will be used to implement any end-to-end security measures, and an instance of the security class (a "security object") to be used for the lifetime of the connection. A security object typically contains enough information/code to authenticate a single user to the remote service, and to secure (if desired) the conversation with the remote service against eavesdropping and/or tampering. Each connection is capable of supporting up to four simultaneous calls to the remote service. Additional calls above this limit are either held up by Rx or aborted, at the user s option. Rx allows calls to be aborted after a maximum time interval has passed, or after a maximum time interval with "no work accomplished" has passed. Each call consists of an arbitrary number of packets sent from the client to the server, followed by an arbitrary number of response packets sent from the server to the client. Clients and servers typically use procedure "stubs" to hide the packet level activity, and may also write or read the ingoing or outgoing packet streams directly, or through a byte stream interface. A call can be aborted at any time by the client or server with an end-to-end abort mechanism. 1.3 Security A slight twist to the basic RPC protocol is added in order to provide for secure, authenticated conversations. If the server has never heard from a client, or has discarded any information it previously held about the client, then Rx may decide to send a challenge packet to the client before generating a response and before all of the request packets have arrived. The client is expected to respond appropriately to this challenge packet with a challenge response packet filled in appropriately according to the security needs of the conversation. In addition, depending upon the degree of security desired, the data packets may be encrypted or securely marked so that an arbitrary degree of security is possible, within the constraints of this model. Rx implements security policies using the concept of an externally provided security class. Compatible code for a given security class must exist both at the client and the server, if it is to be used, and the security class must be explicitly associated with each service at the server which wishes to allow connections using that security class. Multiple security classes may be used for any service. Only a single security object, an instantiation of a security class, is, however, useable with any given connection. Security classes are defined externally to Rx; Rx simply calls on the security modules at the appropriate times. 2 An Example Client Application and Server As a programmer, you may write client code or server code, or possibly both. If you are only writing client code, then life will be considerably simpler: the server application writer will have provided the appropriate interface definitions for you to use in building your client application. So we ll look first at the steps involved in building a client application to interface to an existing server.

4 3 2.1 Building a Client Application The existing server for our example provides two interfaces: an add and a subtract interface, called SAMPLE_Add and SAMPLE_Sub, respectively. Whenever the client calls either of these procedures, the arguments are packaged up, shipped to a remote site, and delivered to a server process. The server process deciphers the arguments, performs the appropriate operation (add or subtract) and ships the results back to the original client. The client s call blocks until the operation completes. Here is the text of the client application, minus the routine GetIpAddress, which simply returns the network-byte-order IP address of the host: #include <sys/types.h> #include <netdb.h> #include <stdio.h> #include "sample.h" main(argc, argv) char **argv; { struct rx_connection *conn; u_long host; struct rx_securityclass *null_securityobject; int i; rx_init(0); host = GetIPAddress(argv[1]); null_securityobject = null_newclientsecurityobject(); conn = rx_newconnection( host, SAMPLE_SERVER_PORT, SAMPLE_SERVICE_ID, null_securityobject, SAMPLE_SC_NULL); for(i=1;i<10;i++) { int error, result; printf("add(%d,%d)",i,i*2); error = SAMPLE_Add(conn, i,i*2,&result); printf(" ==> %d, error %d\n", result, error); printf("sub(%d,%d)",i,i*2); error = SAMPLE_Sub(conn, i, i*2, &result); printf(" ==> %d, error %d\n", result, error); } } The Rx package is initialized by calling rx_init. Since this application is only going to run client code, no UDP port needs to be specified (a zero indicates this). The client then prepares to make a connection to the server. To make a connection, the server process has to be addressed (the code retrieves the Internet host address, and the UDP port number of the server, which both must be in network byte order), and the service type within that process must be specified. The service type is simply a constant defined in the header file exported by the server (the service type is not in network byte order!). Finally, a security object must be obtained for use with the connection. In Rx, all connections are potentially secure, authenticated connections. That is, the connection may be secured against eavsdropping or malicious alteration of data. This is accomplished by encrypting or otherwise obscuring the data using a shared secret between the client and the server. A shared secret is necessary so the server can decrypt the encrypted data. Often the client will obtain a secret from an authentication server, in the form of a "ticket". This ticket will probably be passed into to a

5 4 security class to obtain an instance of that class (a "security object") good for that particular ticket. In this example, however, the null security class is used. This is a simple class implementing no security, that is, the connections are unauthenticated and not secure. The function null_newclientsecurityobject() does not require any parameters: no password or ticket is required, since no security is implemented. Having obtained the service address and the security object, a new connection is made by calling rx_newconnection with the appropriate parameters. Once a connection is obtained, the client is able to call the server routines, remotely, by simply invoking them as if they were locally defined, with the exception that the connection pointer returned from rx_newconnection must be provided as the first parameter to each of these routines. 2.2 Building a Server To build the server and to create the appropriate header files and library routines for use by both the client and server, the program rxgen is used. Rxgen, described in detail in an accompanying document, generates stubs, which are functions which convert between function call arguments and RPC messages and back. The rxgen input file, sample.rg, is shown below: package SAMPLE_ #include <rx/rx.h> #include <rx/rx_sc_null.h> #define SAMPLE_SERVER_PORT htons(5000) #define SAMPLE_SERVICE_PORT 0 #define SAMPLE_SERVICE_ID 4 /* Maximum number of requests that will be handled by this service simultaneously. This number will also be guaranteed to execute in parallel if no other service s requests are being processed */ #define SAMPLE_MAX 2 /* Minimum number of requests that are guaranteed to be handled immediately */ #define SAMPLE_MIN 1 /* Index of the "null" security class with respect to the sample service. This must be 0 (there are N classes, numbered from 0. In this case, N is 1) */ #define SAMPLE_SC_NULL 0 /* The routines exported by this interface. The 1 and 2 are the RX operation codes to assign to the routines */ Add(IN int a, int b, OUT int *result) = 1; Sub(IN int a, int b, OUT int *result) = 2; Here is the text of the sample server application. The procedure Quit is not shown: #include <sys/types.h> #include <netdb.h> #include <stdio.h> #include "sample.h"

6 5 #define N_SECURITY_OBJECTS 1 extern SAMPLE ExecuteRequest(); main() { struct rx_securityclass *(securityobjects[n_security_objects]); struct rx_service *service; /* Initialize Rx, telling it the port number this server will use for its single service */ if (rx_init(sample_server_port) < 0) Quit("rx_init"); /* Create a single security object, in this case the null security object, for unauthenticated connections, which will be used to control security on connections made to this server */ securityobjects[0] = null_newserversecurityobject(); if (securityobjects[0]==(struct rx_securityclass *)0) Quit("none_NewServerSecurityObject"); /* Instantiate a single sample service. The rxgengenerated procedure which is called to decode requests is passed in here */ service = rx_newservice( 0, SAMPLE_SERVICE_ID, "sample", securityobjects, N_SECURITY_OBJECTS, SAMPLE_MIN, SAMPLE_MAX, 0, SAMPLE ExecuteRequest); if (service == (struct rx_service *) 0) Quit("rx_NewService"); } rx_startserver(1); /* Donate this process to the server process pool */ Quit("StartServer returned?"); int SAMPLE_Add(call, a, b, result) struct rx_call *call; int a,b; int *result; { printf("sample_add(%d,%d)\n", a,b); *result = a + b; return 0; } int SAMPLE_Sub(call, a, b, result) struct rx_call *call; int a,b; int *result; { printf("sample_sub(%d,%d)\n", a,b); *result = a - b; return 0; } In this example, rx_init is called, specifying SAMPLE_SERVER_PORT as the port at which to listen for requests, by default, for any installed services. The server side of a null security object is then created and passed into rx_newservice as the single supported security object. Some scheduling parameters and a dont-care value for the stack size are also passed in. Finally, the

7 6 rxgen-created stub routine, SAMPLE ExecuteRequest, responsible for decoding requests received for this service, is provided. Rx will invoke this routine whenever a call is received, passing the routine the rx_call structure it has allocated to handle the incoming call. Having exported all of its services (in this case, one), the server code now calls rx_startserver, passing it a "1" to indicate that it should donate the current thread of control to the cache of processes used for executing server routines. The rest is automatic: the routines SAMPLE_Add and SAMPLE_Sub are called by SAMPLE ExecuteRequest whenever a new call arrives. They are required to return a status of 0 if the call succeeds, which they do. The actual function result is returned by reference, through the result parameter. 3 Programming Interface Specification This section describes those aspects of the RX programming interface needed by a prorammer using the rxgen stub-generator. There will, eventually, also be a number of interfaces that may be called to set various Rx operating parameters. These are currently undocumented, since they have not yet been finalized. There exist routines to allocate packets, and send and receive packets (rx_allocpackets, rx_senddata, and rx_readdata) which are not documented here, since they will not normally be used directly. The stub generator is described briefly. 3.1 The Rx Interface rx_init(port) u_short port; Rx_Init must be called before any other RX procedure. The port number is specified (that is, it is non-zero), in net byte order, for server applications or mixed server/client applications, only. If a port number is specified, it becomes the default UDP port number of any service exported by this server, unless a service specifically registers its own port number. If the port number is not specified to this routine, it must be specified to every call to rx_newservice. This routine retuns 0 for success, -1 on error. rx_startserver(donateme) int donateme; rx_startserver is called by a server application after all other server initialization has been completed (in particular, after all calls to rx_newservice). The server will not accept any RPC calls until this routine has been invoked. If donateme is non-zero, then rx_startserver will never return, rather it will be pressed into service handling incoming RPC requests. A client application (i.e. an application which has not called rx_newservice) need not call this routine.

8 7 struct rx_service *rx_newservice( port, serviceid, servicename, securityobjects, nsecurityobjects, minrequests, maxrequests, stacksize, serviceproc ) u_short port; u_short serviceid; char *servicename; struct rx_securityclass *securityobjects; int nsecurityobjects; int minrequests; int maxrequests; int stacksize; short (*serviceproc)(); Rx_NewService must be called to install a new service at a server. The service will be advertised at this host on port port (which must be in network byte order), as service serviceid. A wellknown name of the service should be specified. This may be used to allow a remote program to probe this server for statistics regarding a particular service. The caller also provides a securityobjects array, and a corresponding size. This array contains pointers to instantiated instances of security classes that will be supported by this service. For more details, see the section on security classes, below. MinRequests and maxrequests are provided to control the scheduling of processes at the server. MinRequests gives the number of processes (kernel processes or light-weight LWP processes, depending upon the implementation of Rx) that must be made available at all times for this service. That is, when a request for this service arrives, then it will be guaranteed a process to service it if the current number of processes allocated to the service is less than the minimum number specified for the service. MaxRequests implements two constraints: it specifies the maximum number of requests that will ever be serviced in parallel for this service, and, in addition, it specifies the number of processes that will actually be able to run in parallel for this service, if no other service is presently competing for process resources. StackSize specifies the minimum number of bytes of stack space that are required for any request on this service. Rx computes the maximum of all of the stack size numbers provided, as well as the default value (4096) and uses this maximum value for all processes that are created (in the LWP version of Rx). Finally, the field serviceproc is the function which is to be called whenever a new call is received for this service. The function will be called by Rx with the call structure pointer as its single argument. It is responible for reading the request packets for the call, using rx_readdata, generating response packets, using rx_senddata, and finally returning with an error code greater than or equal to 0. If it returns a non-zero error code, the call will be aborted and the response packets may not actually reach the client, although the error code normally will. Typically, rxgen is used to generate the Rx stubs. Included with the stubs is an "execute request" function, which should be passed into rx_newservice as the serviceproc argument.

9 8 struct rx_connection *rx_newconnection( host, port, service, securityobject, servicesecurityindex ) u_long host; u_short port; u_short service; struct rx_securityclass *securityobject; int servicesecurityindex; A client makes a new connection to a service by calling rx_newconnection. The (host, port, service) triple must be supplied to address the service, the host and port supplied in network byte order. An instance of a security class, a security object, must also be provided to implement security/authentication details at this end of the connection. Finally, the index of the security class with respect to the desired service must be specified (each security class may define a different index for each service). If any problems arise creating this connection, the call returns the null pointer. No check is made that the specified host actually exists, or that the service is currently exported. No communication with the remote service is initiated as a result of creating a new connection. For its entire lifetime, a connection is associated with a single security object. If that security object ceases to be useful (e.g. the rights carried by the object expire) a new connection must be made using a different security object. void rx_destroyconnection(conn) struct rx_connection *conn; A client connection may be destroyed by calling this rx_destroyconnection. The connection will not really be destroyed until the last active call has completed. Once this routine has been called, however, new calls may not be started on the specified connection. struct rx_call *rx_makecall(conn, wait, maxtime) struct rx_connection *conn; int wait; int maxtime; Once a client establishes a connection to a service, it may use the connection to make calls. When using stubs generated by rxgen this is done automatically (the stubs call this routine). Rx_MakeCall is passed a connection, a flag specifying whether Rx should wait if all four of the connection s channels are currently busy, and the maximum time, in seconds, that this call should be allowed to run before preemptively terminating it 1. The returned call reference may be used in subsequent rx_senddata and rx_readdata requests to send data to the server and to retrieve the results. 1 There should also be a way to specify the maximum time that a call may live without work being done (e.g. data packets transmitted), and there should be a way to specify these parameters from the server end of the connection. These interfaces were yet not specified at the time of writing.

10 9 int rx_endcall(call, error) struct rx_call *call; long error; A call is terminated at either the client or the server by invoking rx_endcall, with the call reference and an error code. If the call is already in an error state, for some reason, this call simply releases the call channel for further calls (it can accept new incoming calls at the server, or generate new outgoing calls from the client). In this case, the return value from this function is the original error state of the call. If the call is not in error, and the supplied error code is zero, the call completes normally. If the caller is a server, it will arrange to send any remaining response packets to the client. If the supplied error code is non-zero (it should be positive), the call is terminated, and an abort message is sent to the peer. In this case the supplied error code is also returned as the function result. 3.2 The Rx Stub Generator The Rx programmer typically will make use of rxgen, a stub generator based upon Sun Microsystem s rpcgen program. Rxgen is described elsewhere in detail, and will be described only briefly here. Rxgen generates C-code to marshal and unmarshal arguments, and to interface with Rx, for each routine described in an interface specification. The generated code uses Sun s External Data Representation, XDR, to encode/decode the parameter lists for each routine. It uses a special XDR object of type xdr_rx to interface to the Rx stream package, described below. For a client, the generated routine first initiates a call by calling rx_makecall, then calls xdrrx_create in ENCODE mode to create an XDR object, fills in the parameters of the call, and destroys the object. It then creates another object in DECODE mode to extract the returned results, destroys the object, and terminates the call by calling rx_endcall. The Rx stream interface takes care of the rest: it streams bytes generated by XDR into packets, sends them to the server, and routes bytes back from response packets, through XDR to the client. The server code does essentially the opposite, but is not responsible for creating the call: this is done automatically by Rx when a call is received from the client. The client stub is additionally responsible for transmitting an operation code appropriate to the call. Rxgen also generates a routine to decode this operation code and call the appropriate server stub. 3.3 The Rx Stream Interface The Rx stream interface is used by the rxgen-generated stubs, and is also used to implement bulk transfer of data 2. To use a stream, a reference to an active call is required. 2 It has not yet been determined how best to provide direct access to Rx streams while using rxgen-generated stubs.

11 10 void rx_stream_initread(sd, call) struct rx_stream *sd; struct rx_call *call; Attach the stream descriptor sd to the specified call, and prepare to read. int rx_stream_read(sd, buf, nbytes) struct rx_stream *sd; char *buf; int nbytes; Read up to nbytes bytes from the supplied stream descriptor (initialized in read mode) into the specified buffer. Returns the number of bytes actually read. Returns zero on end of data. If the number of bytes returned is less than the number requested, the next read will return zero. Warning: this is implemented as a macro. int rx_stream_readiov(sd, iovlenp, iov, nbytes) struct rx_stream *sd; int *iovlenp; struct iovec *iov; int nbytes; This is a concession to 4.3 BSD UNIX. The "io vector" pointed to by iovp is filled in with enough information to address nbytes worth of data, scattered amongst several incoming Rx packets. The vector is assumed to be able to accomodate (*iovlenp) items, and this value is updated after the call to indicate the precise number of entries in the resulting iov. The actual number of bytes available, but not greater than that requested, is returned. A writev system call can now be used to write out the data in these packets. This routine allows blocks of data of optimal size for a file system to be written directly from the packet buffers to the disk, with no intervening copy operation. void rx_stream_finishread(sd) struct rx_stream *sd; This routine must be called pairwise with rx_initread, in order to guarantee that any resources used by the stream are released. It is normally called after end of file is detected on the stream, but may be called at any time after rx_initread. void rx_stream_initwrite(sd, call) struct rx_stream *sd; struct rx_call *call; Attach the stream descriptor sd to the specified call, and prepare to write. int rx_stream_write(sd, buf, nbytes) struct rx_stream *sd; char *buf; int nbytes; Write nbytes bytes from the supplied stream descriptor (initialized in write mode) into the specified buffer. Returns the number of bytes actually sent. If less bytes are sent than expected, then there is an error on the call, and any further writes will return 0. Warning: this is

12 11 implemented as a macro. If the buffer pointer supplied is null, the internal stream pointer is merely advanced by the specified number of bytes. This is used in conjunction with rx_stream_allociov. int rx_stream_allociov(sd, iovlenp, iovp, nbytes) struct rx_stream *sd; int *iovlenp; struct iovec *iovp; int nbytes; This is another 4.3 BSD concession. The "io vector" pointed to by iovp is filled in with enough information to address nbytes worth of data, scattered amongst several Rx packets. The vector is assumed to be able to accomodate (*iovlenp) items, and this value is updated after the call to indicate the precise number of entries in the resulting iov. The actual number of bytes allocated is returned. A readv system call can now be used to fill in these packets, and the data can be written by calling rx_stream_write with a null buffer pointer (this write is optional, and can be done with less than the number of bytes returned from rx_stream_allociov. The stream buffer pointer is not actually advanced unless rx_stream_write is called). This routine allows blocks of data of optimal size for a file system to be read into packet buffers using a single system call, and without an additional copy of the data. void rx_stream_finishwrite(sd) struct rx_stream *sd; This routine must be called pairwise with rx_stream_initwrite, in order to guarantee that any resources used by the stream are released. It may be called at any time after the stream descriptor is initialized for write. 3.4 Security Classes A security class is an implementation of an end-to-end security model for the Rx environment. A number of different security models may exist. For the purposes of this discussion, we will assume that we are building a security class named sec. Sec is required to export at least two interfaces: struct rx_securityclass *sec_newclientsecurityobject(); struct rx_securityclass *sec_newserversecurityobject(); These two routines return instantiations of the corresponding security class for the client and server ends of the connection, respectively. Most security classes will take initialization parameters here: in a Kerberos authentication environment, for example, sec_newserversecurityobject might accept the server s service password, previously registered with Kerberos, as the service key for that service, whereas sec_newclientsecurityobject might accept a ticket to use that service from the user. Additionally, these routines may take other parameters specifying such things as the level of security to be implemented. A security class object can be destroyed by calling the following routine. The object will not be destroyed immediately if it is currently in use by a connection:

13 12 void rx_destroysecurityobject(obj) struct rx_securityclass *obj; The security object returned by the NewClientSecurityObject or NewServerSecurityObject routine contains a pointer to a structure containing a pointer to an array of functions: struct rx_securityops { int (*op_close)(); int (*op_newconnection)(); int (*op_preparepacket)(); int (*op_sendpacket)(); int (*op_checkauthentication)(); int (*op_createchallenge)(); int (*op_getchallenge)(); int (*op_getresponse)(); int (*op_checkresponse)(); int (*op_getheadersizes)(); int (*op_reserved1)(); int (*op_reserved2)(); int (*op_reserved3)(); int (*op_reserved4)(); int (*op_reserved5)(); } *ops; The functions are called by Rx at strategic times. They will be described in a future version of this document. 3.5 System Error Codes The Rx implementation, the rxgen stubs, and the security objects can all return "system level" errors. The Rx interface will try to ship the error code to the connected peer. All system error codes are defined to be less than zero, and user code may not return errors in that range Rx Errors The currently defined Rx errors are: RX_CALL_DEAD RX_INVALID_OPERATION RX_CALL_TIMEOUT RX_EOF RX_PROTOCOL_ERROR A network partition or server crash has occured. An invalid operation was attempted, such as a client attempting to send data after having received the beginning of a reply from the server. The timeout specified for a call has expired, before the call was able to complete. End of data on a read (returned to Rx_ReadData). An error in the protocol was detected. Caused by a bug in the protocol or the implementation of one of the ends of a conversation Rxgen Errors Rxgen introduces the concept of badly marshalled or unmarshalled arguments. For example, if there are not enough bytes transmitted for some reason, the opposite side will get an unmarshalling error. User code is not expected to interpret these codes, but they may be handy for debugging the software (so the codes should be printed if they re not understood).

14 13 RXGEN_CC_MARSHAL RXGEN_SS_MARSHAL RXGEN_CC_UNMARSHAL RXGEN_SS_UNMARSHAL Failure to correctly marshal an argument at the client. Failure to correctly marshal an argument at the server. Failure to correctly unmarshal an argument at the client. Failure to correctly unmarshal an argument at the server Security Class Errors These will be specified in a future version of this document. 4 The Wire Protocol The basic Rx protocol is described in the introduction to this document. To add detail to the description, the packet formats are described, below, together with a brief description of the various fields. An Rx packet consists of a IP/UDP header followed by a basic Rx header, common to all Rx packets. These headers are followed by data which is dependent upon the type of the packet. The basic Rx header is 28 bytes, all values in network byte order, and is described below: (4 bytes) unsigned long EPOCH (4 bytes) unsigned long CID EPOCH is the start time of the client s Rx package, and CID is a connection identifier assigned by the client whenever it generates a new connection. The lower 2 bits of CID are a CHANNEL identifier, representing one of the four sub-channels for this connection, each of which may have a currently active call. (4 bytes) unsigned long CALL CALL indicates which call this packet is concerned with. The call number is incremented by the client on each new call for a given connection channel, beginning with call number 1. All of the packets concerned with a call are labeled with the same call number. If a packet is received by the server with a call number greater than any previously received packets, then the server will accept the call, otherwise it will ignore the packet. This prevents calls from being executed twice, and allows calls to be missed (because of temporary network partition, for example). If a client requests the optional server crash detection, then the server will not perform the operation until the client is told the epoch value of the server--the start time of the server. This is accomplished by piggy-backing the epoch on the security challenge packet, which will be described later. (4 bytes) unsigned long SEQUENCE SEQUENCE is the call-relative sequence number of a given data packet. The sequence number begins with a value of one for the first packet of each call. A packet may be retransmitted any number of times, but retains the same sequence number on each retransmission. Sequence numbers are, of course, used in the usual manner to piece together incoming packets into a welldefined, ordered packet-stream.

15 14 (4 bytes) unsigned long SERIAL SERIAL is a connection-relative number stamped on each packet as it is transmitted for a given connection. This number is used as an aid in optimizing the retransmission of dropped packets. (1 byte) unsigned char TYPE TYPE is the type of the packet. The currently defined packet types are listed below. DATA A vanilla data packet, part of either a request or a response stream. The body of the packet may or may not be encrypted, depending upon the security methods selected. The length of the data is determined from the UDP encapsulation, and the size and shape of any additional security data is determined by context by the associated security class for the connection. ACK An acknowledge packet. Acknowledge packets are discussed in detail, below. PING BUSY ABORT ACKALL CHALLENGE A ping packet. Ping packets are used to announce that one end of a call is alive, and request an ACK packet to be sent, acknowledging the PING packet and demonstrating that the opposite number is alive. PING packets are sent whenever packets haven t been sent for a while, while a call is in progress. A busy packet is sent when a server cannot accept a call. This may be because the server side of a connection channel is still in use from a previous call (it hasn t been released by the user-supplied server code) or that there are not enough resources (i.e. packet buffers or processes) to handle this call right now. The client should try again later. This call will never be sent from a client to a server; a client is supposed to have the resources to accept a reply to a call. The busy packet includes a long word with a suggested minimum time-to-wait (in seconds) before retrying the call. An abort packet is sent whenever a non-zero return code is returned by the client or server (when the call is ended), or whenever a protocol problem should be reflected to the other end of the connection. This packet contains a single 4-byte word of error return information, in addition to the standard header. This packet is sent by either server or client to acknowledge all of the packets of a call or response have been received. This is sent by the server shortly after receiving the last packet of a call, if there is to be no immediate response to the call, and by the client after receiving the last packet of a reply, if there is to be no immediate response to the reply. When many calls are being generated by a client, and the responses are close to immediate, there will be no ACKALL packets generated. This packet is sent by the server to the client to establish the client s identify for authentication purposes. The contents of the packet are filled in by the security object used for the connection. RESPONSE The response to the challenge packet. This packet is filled in by the client side s security object, and the contents are unknown to Rx. (1 byte) unsigned char FLAGS

16 15 This byte contains various flags for the call: CLIENT_INITIATED This packet was sent from the client-side of a call. REQUEST_ACK LAST_PACKET The peer is requested to acknowledge this data packet. This is the last of the request packets from the client, or the last of the response packets from the server. (1 byte) unsigned char STATUS This is a 1 byte quantity which can be used by the client or the server to convey some userdefined meaning to the peer. The value will be piggy-backed on whatever packets are going out (typically data packets or acknowledge packets) and can be used to indicate the ongoing status of the call, in an asynchronous manner. Since calls are half-duplex (no response data can be sent to a client before all of the request packets have been received) this provides a limited means for the two parties to communicate "out-of-band". Each end can monitor the remote end s status, by asking Rx for its current value. The value returned is simply the latest value of this field from an incoming packet for that particular connection. If packets are received out-of-order, as can be determined by comparing the serial numbers of the packets, then the lower numbered packet s status is ignored. (1 byte) unsigned char SECURITY This byte allows the client to specify which security mechanism will be used for a connection. Every packet the client sends is required to have the same value for the security field (the server will use the first value it sees for all subsequent requests). The mapping of a security id to a security class implementing a particular type of security is a per-service mapping. Each service specifies its own set of security types. (2 bytes) unsigned short SERVICE_ID This specifies the service number at the server to connect to. The complete service address includes the IP destination address and UDP port number, both contained in the IP/UDP header. SERVICE_ID is the number of the service requested by the client at the server. This is logically concatenated with the IP address and UDP port number provided in IP/UDP header to complete the address of a service. This field is filled in by the client and read by the server. When a packet from a previously unknown client arrives, the client s CID and EPOCH fields are associated with a connection structure hooked into the specified service. Subsequent packets are routed on the basis of the clients UDP/IP address and the CID and EPOCH fields. (2 bytes) RESERVED Reserved for future extensions. Must be zero Acknowledge Packets An acknowledge (ack) packet acknowledges the receipt of data packets from the peer. Actually, two types of acknowledgement are included in an ack packet: the sequence number of the last packet sent up to the user code is provided (and, hence, all packets with lower sequence numbers are implicitly acknowledged). This packet and its predecessors will never have to be transmitted

17 16 again. In addition, another set of packets within the current receive window is acknowledged. Since these may not all have been received, they are each acknowledged explicitly. This acknowledgement does not, however, imply that these packets can be dropped from the sender s transmit queue; on the contrary, the packets could be dropped from the receive side at any time and still have to be be retransmitted by the sender. Note that there are three other ways ways to implicitly acknowledge data: An ACKALL packet acknowledges all packets sent by either the client or server. It is typically sent if there is a delay in sending a reply from the server, or in making a new request at the client. A new request from a client acknowledges all previous packets sent by the server. Reply packets from the server acknowledge all the packets sent from the client, for the current request. An ack packet has the following fields, beyond the Rx header: (2 bytes) u_short BUFFER_SPACE The number of packet buffers available at the receiver. That is, the number of buffers that the sender of the ack packet believes it is willing to provide for data, on this or subsequent calls. (2 bytes) u_short SKEW This is the maximum difference between the serial number of the packet acknowledged and the highest packet yet received by the sender of the ack packet. This number is used to determine whether it is reasonable to assume that a packet has been dropped, based upon missing serial numbers in a sequence. If packets are routinely delivered out-of-order, then such an assumption is only true if the expected serial number of a packet is not greater than the actual serial number of the packet by more than this number. (4 bytes) u_long FIRST This is the first packet sequence number acknowledged by this ack packet. That is, the first element of the ack array within this packet refers to this sequence number. (4 bytes) u_long PREVIOUS The previous packet number received. This is not used by the current Rx implementation. (4 bytes) u_long SERIAL The serial number of the packet which prompted the acknowledge. The sequence number of this packet is provided in the packet header sequence field.

18 17 (1 byte) u_char REASON The reason for the acknowledgement was provided. Note that an acknowledge is not required for the reasons described below, but may be issued for one of these reasons: REQUESTED DUPLICATE OUT_OF_SEQ NOSPACE PING_RESPONSE The peer requested an acknowledge. This is a duplicate packet (a packet with the same call and sequence numbers, etc. has already been received). Packet arrived out of sequence. This reason is not currently employed. There is no buffer space available. We re acking because we got a ping packet. (1 byte) u_char NACKS (NACKS bytes) u_char[nacks] ACKS NACKS is the number of preliminary acknowledges specified in this packet, up to 255 of them. ACKS actually holds the acknowledges. Each value in the ACKS array corresponds to one packet, starting from the packet named by FIRST. Currently, a zero value indicates that the corresponding packet is not acknowledged (or has been downgraded from temporarilyacknowledged to not-acknowledged), and a value of one indicates that the corresponding packet is acknowledged.

19 18 References

20 i Table of Contents 1 Introduction Services Connections Security 2 2 An Example Client Application and Server Building a Client Application Building a Server 4 3 Programming Interface Specification The Rx Interface The Rx Stub Generator The Rx Stream Interface Security Classes System Error Codes Rx Errors Rxgen Errors Security Class Errors 13 4 The Wire Protocol Acknowledge Packets 15