Application. Protocol Stack. Kernel. Network. Network I/F

Transcription

1 Real-Time Communication in Distributed Environment Real-Time Packet Filter Approach 3 Takuro Kitayama Keio Research Institute at SFC Keio University 5322 Endo Fujisawa Kanagawa, Japan takuro@sfc.keio.ac.jp Tetsuya Saito Graduate School of Media and Governance, Keio University 5322 Endo Fujisawa Kanagawa, Japan saimune@sfc.keio.ac.jp Akihiko Miyoshi Keio Research Institute at SFC Keio University 5322 Endo Fujisawa Kanagawa, Japan miyos@sfc.keio.ac.jp Hideyuki Tokuda Faculty of Environmental Information Keio University 5322 Endo Fujisawa Kanagawa, Japan hxt@sfc.keio.ac.jp Abstract Recent modern operating system technology enables protocol processing in user space using in-kernel packet lter and user-level protocol processing library for exibility without sacricing the performance of traditional kernelized protocol processing. This technology can be adapted to build a highly preemptable protocol processing mechanism for distributed real-time environment. In this paper, we discuss the structural dierence of various operating systems from the protocol processing point of view, and propose an extended mechanism of packet lter and user-level protocol processing library for real-time communication. Using this mechanism, priority of the client can be handed o to the server without priority inversion problem while protocol processing. 1. Introduction In distributed real-time systems, applications on each node must communicate and coordinate each other across a network. In such environment, even if client and server applications are well written, it is not sucient. Communication mechanism under these applications must support real-time functionalities to make the system predictable. In such communication, the problem can be divided into two parts. One is the 3 This research is conducted under the fund of Informationtechnology Promotion Agency, Japan (IPA). network bandwidth and throughput, and the other is protocol processing in both sender and receiver side. Many traditional operating systems use FIFO queuing for network processing. This causes serious priority inversion problems, if a higher priority thread is trying to send a message while lower priority thread is sending a message. In addition, the protocol stacks are implemented in the kernel, so any high priority activities can be blocked by an incoming lower priority packets. Many modern operating systems, however, have protocol stack implemented in the user space as a server for high exibility. Furthermore, protocol stack implemented as a user-level library provides both high performance and high exibility. Coincidently, these mechanisms can be suitable for real-time environment to make protocol processing preemptable compared to traditional operating system architecture. We address the problems on sender and receiver side protocol processing and designed a real-time communication mechanism on Real-Time Mach microkernel. Real-Time Mach(RT-Mach) has been developed by Carnegie Mellon University(CMU), and Keio University, and Japan Advanced Institute of Science and Technology(JAIST). The main objectives of RT- Mach is to provide a common real-time computing environment[18]. In RT-Mach, we have real-time thread where a periodic activity can be easily dened by explicitly specifying timing attributes. It also provides real-time synchronization[17], real-time IPC(RT- IPC)[5], high resolution clocks and timers[16], and a processor capacity reservation mechanism[11].

2 In this paper, we propose a distributed real-time communication mechanism to support distributed realtime applications. In Section 2, we explain our distributed real-time communication model. In Section 3, we discuss some of the design choices illustrating previous approaches. Section 4 shows the implementation overview of our system. In Section 5, the evaluation result of the current mechanism is presented and we conclude in Section Communication Model Kernel Application I/F The abstraction of our real-time communication model is that the priority is associated with processing whether the processing is done on a client machine or server machine. We dene the real-time communication model in distributed system as: Client-server communication. Point-to-point communication. Broadcast and multi-cast messages are not considered in this paper, but is a future consideration. A server consists of single or multiple thread activities. Server's priority is dynamically changed based on the priority of requests. A server has its own priority for its initialization. Once the server receives a request, it processes with the priority of the request. Priority of the request is propagated from client's priority. The priority needs to be transfered from the client machine to the server machine through a network. The meaning of priority is consistent among all clients and servers in the system. The scheduling policy of client and server machines must be the same. Communications are either synchronous or asynchronous. Duration of a server priority propagation begins when the server receives a request from a client, and lasts until the server replies to the client in case of synchronous communications. Remote Procedure Call(RPC) and remote method invocation mechanism can be mapped to this model. In the case of asynchronous communication case, priority of the server is propagated until the server receives a next request. Figure 1. In-Kernel Protocol Processing 3. Design Choices In this section, we consider ve approaches of protocol processing in distributed real-time environment, and discuss the advantages and disadvantages In-Kernel Protocol Processing Many traditional operating systems such as 4.3BSD[7] use this approach. It provides high throughput and fast response to the network. Figure 1 illustrates this protocol processing model. In this approach, all protocol processing is in the kernel and non-preemptable, i.e. kernelized. This gives the protocol processing activity highest priority in the system. Higher priority activity can be preempted by an incoming lower priority packet. Sending large amount of packets by lower priority activity may take over the processor during protocol processing regardless of the importance of any other activities. This approach is less exible compared to the rest of the four approaches explained in the following sections Using netmsgserver netmsgserver[2] is a message server for communicating across a network on the Mach microkernel. It has a name service facility and extends the local Mach Inter-Process Communication(IPC) abstraction over a network. The security mechanism, remote procedure calls using Mach Interface Generator(MIG), copy-on-reference operation over a network are provided. Figure 2 shows the communication mechanism of netmsgserver over a network using TCP/IP. 2

3 Application netmsgserver Application Protocol Server Unix Server Kernel I/F Kernel I/F Figure 2. Using netmsgserver Figure 3. Shared Protocol Processing Server First, an application program sends a local message using Mach-IPC to netmsgserver, then netmsgserver converts the data to the network representation. Then, it sends the data to the protocol processing server(in this case, UNIX server) to process TCP/IP. Finally, the packet goes out to the network. Receiving a packet is vice versa. Thus, the problem in this model is the overhead. Sending and receiving data requires many context switching and data copying. Hutchinson, et al.[3] proposed fast Mach IPC implementation over a network using xrpc protocol. They reduced the overhead, but an issue still remains. The priority of netmsgserver and the protocol processing server can be assigned explicitly when sending a packet. But when receiving a packet, netmsgserver and the protocol processing server must run with the highest priority, otherwise priority inversion problem may occur if the packet has higher priority than other schedulable activities Shared Protocol Processing Server This approach is used particularly in microkernel architecture. The overhead of this approach is less than that of netmsgserver approach, but higher than inkernel protocol processing. Since the protocol stack exists in the user space, this structure provides better exibility than the in-kernel protocol processing model. x-kernel[3] and UNIX server on Mach microkernel use this architecture. Figure 3 shows this architecture. Using this architecture, protocols can be created or modied easily without changing the microkernel. The priority of the protocol processing server can be assigned explicitly by application programs or prior- ity inheritance can be applied for sending packets and it is preemptable. When receiving packets, however, the server's priority cannot be inherited by incoming packet from the network. So, the priority of protocol processing must be highest among the thread activities which use the protocol stack. Nakajima, et al.[14] proposed a Prioritized IP(PIP) which extends IP protocol using IP option eld. NPS is implemented as a real-time network engine on Real- Time Mach. By adding priority elds in the IP option, processing of UDP can be handled with the priority of the message. So this processing is preemptable by any other higher thread activities. Processing IP still needs the highest priority and it only provides UDP/IP protocol User-Level Protocol Processing The mechanism of protocol processing in user-level library is proposed for high performance and high exibility on microkernel[8]. All TCP/IP and UDP/IP protocol processing are done in the user-level library. Figure 4 depicts the mechanism of this approach. The user-level library directly accesses a network device driver in the microkernel to send and receive packets. When receiving a packet, the packet lter[10, 19] in the microkernel dispatches incoming network packets to the receiver. Unlike the shared protocol processing server approach, this mechanism does not need context switching to send and receive packets from a network. An operating system server is required only for maintaining sockets. This approach minimizes the overhead of protocol processing, but the priority inversion problem similar as in shared protocol processing 3

4 Socket Library Application In Kernel Protocol Processing Shared Protocol Server User-Level Protocol Processing Packet Filter Assigning Priority Performance Flexibility Preemptability Good Bad Bad Bad Good Good Good Good Good Good Good Good Kernel Packet Filter I/F Table 1. Protocol Processing Comparison(1) Figure 4. User-Level Protocol Processing server still remains when receiving incoming packets. Mercer, et al.[12] and Lee, et al.[6] extend this mechanism for predictable communication protocol processing on Real-Time Mach microkernel using processor capacity reservation[11]. This is conceptually the same as real-time publisher/subscriber(rt/ps)[15]model. In their model, a server has its own processor reservation or priority, and protocol processing is done using its reservation or priority. Protocol processing is fully preemptable both for sending and receiving packets. A problem arises when the server reservation is running out. Under this condition, the requests are processed with lower priority(time shared) even if the request has the highest priority. With the xed priority scheduling policy, server work including protocol processing is done by the server priority regardless of the client priority. Thus, the priority of the client request is ignored and may lead to a priority inversion problem Packet Filter Assigning the Priority The nal approach which we actually choose for our real-time communication in distributed environment is basically the same structure as the user-level protocol processing model. The user-level library directly accesses a network interface to send and receive packets. To solve the priority inversion problem during the protocol processing in the server side, the packet lter in the microkernel should assign the server priority based on the priority described in incoming packets when the packet lter dispatches the packet to the server. At the client side, when a client sends a request, the outgoing packets take the client priority. This makes the client priority to be handed o to the server over the network. This approach provides high performance, high exibility, and fully preemptable protocol processing. Also, another benet is that servers can run with the client priority from the beginning of the protocol processing Comparison To summarize the characteristics of the protocol processing implementation, Table 1 shows the comparison of each protocol processing explained in this section from the point of performance, exibility, and preemptability. In-kernel protocol processing provides better performance where shared protocol processing server on microkernel architecture has higher exibility and preemptability, but has bad performance. User- Level protocol processing provide high performance, exibility, and preemptability, and our approach also has such characteristics. Figure 5 illustrates the processing priority in each phase of protocol processing. The problem in userlevel protocol processing is illustrated in this gure. Inkernel protocol processing needs to be kernelized when sending and receiving packets. Shared protocol processing server and user-level protocol processing partially solve this problem. Sending packets can be done with the client priority. Receiving packets needs to be done with the server priority. To avoid blocking by lower priority activities during protocol processing, servers must have the highest priority among the activities which need to communicate with the servers. Our approach solve this problem. Both sending and receiving protocol processing can be done with request priority Other Related Works Many other mechanisms has been proposed to support real-time communication. An attempt has been done by Tenet Group[1] to support continuous media 4

5 In-Kernel Protocol Processing Cc Cpc Cnc Cl Cns Cps Cs Shared Protocol Processing Server Cc Cpc Cnc Cns Cps Cs User-Level Protocol Processing Cl Cc Cpc Cnc Cns Cps Cs Packet Filter Assigning Priority Cc Cpc Cnc Cns Cps Cs t MK Server Server Packet Filter RT-IPC I/F P ri o Client Client Client Packet Filter MK I/F Kernelized Computation with Server Priority Computation with Client Priority Cc Client Computation Cpc Protocol Processing in Client Side Cnc Processing in Client Side Cns Processing in Server Side Cps Protocol Processing in Server Side Cs Server Computation Cl Local Communication Overhead Figure 5. Protocol Processing Comparison(2) applications with high-speed computer network. D. Kandlur, et al.[4] has built a real-time communication systems to guarantee the maximum delivery time for messages. These attempts are supported by high-speed networks which provide the notion of priorities. Our approach is to build a real-time communication mechanism even if the underlying network does not support priorities like the most popular network in the world. To support such kind of networks, we are focusing on the protocol processing in host machines. 4. Implementation In this section, we explain the implementation of the real-time communication mechanism in distributed environment. Figure 6 illustrates the overview of the system. We are implementing real-time communication system using many real-time functionalities provided by RT-Mach. Our real-time communication mechanism is based on the user-level protocol processing method described in Section 3.4, and provides the priority propagation mechanism from clients to servers without priority inversions while protocol processing. This mechanism also provides good performance, preemptability, and exibility as the user-level protocol processing method. To add the priority information into packets, we use Figure 6. Real-Time Communication Prioritized IP(PIP)[14]. PIP is an extension of the IP protocol to transfer priority information. It uses a new IP option eld so that PIP packet can be routed and received by traditional IP systems since unknown IP option is just ignored. In this case, priority propagation cannot be used. In the added priority eld, priority, period, and deadline are contained to support various RT-Mach scheduling policies. The modications for the real-time communication mechanism are as follows. Packet Filter The current packet lter implemented in RT-Mach peeks incoming packet from the network to search the destination of the communication end-point(socket port). We are extending the packet lter, called real-time packet lter, to make the packet lter check whether the incoming packet is PIP or not. If the packet is PIP, the real-time packet lter peeks the PIP packet option eld to nd out the priority. Socket Library In the client side, when a client sends a packets to a network, the priority of the client is copied into the PIP packet by the extended version of the socket library. In the server side, on the other hand, the original socket library waits for an IPC message using a IPC port. Our extension is to use RT-IPC. The server is waiting for a RT-IPC message using a RT-IPC port. RT-IPC is an extension of original Mach IPC with various real-time mechanisms. It provides prioritized queuing for messages, priority hand-o between sender and receiver, and priority inheritance protocol with the integration of real-time synchronization. These mechanisms can be enabled, disabled, or selected with real-time port attributes. 5

6 Using RT-IPC, the specied priority is easily propagated to the server thread. Microkernel After the packet lter nds the destination socket port, and the request priority in PIP option eld, the RT-Mach microkernel sends a RT- IPC message where the original microkernel sends a IPC message to dispatch incoming packets. 4.4 BSD Lite Server Since the original socket library needs 4.4 BSD Lite Server to maintain socket ports, we needed to modify 4.4 BSD Lite Server. The IPC port which is used in socket library to receive incoming packets by a server thread is allocated by 4.4 BSD Lite Server. We modied 4.4 BSD Lite Server to allocate RT-IPC port instead of IPC port. Another modication of 4.4 BSD Lite Server is the packet lter programming to handle the PIP option eld. Currently, the implementation of the real-time communication mechanism is under way. The experimental version of the real-time communication mechanism has been implemented. This version only support UDP/IP with the extension of PIP. The priority propagation between clients and servers is working. In following section, we will present the evaluation result of the current real-time communication mechanism. 5. Evaluation In this section, we will show the evaluation result of the experimental implementation of our real-time communication mechanism. The measurements are done on two IBM PC/AT compatible machines. Each machine has 166MHz Intel Pentium processor and 32 megabytes of memory and 3Com EtherLinkIII(ISA, 3c509) for Ethernet interface. We used RDTRC (read time stamp counter) instruction[9] on the Pentium processor for measurements. To show the eect of the protocol processing mechanism by the dierent implementation, we have measured four implementations. The rst one is the inkernel protocol processing, and we used FreeBSD RELEASE. It is represented as FreeBSD in the gures. The second implementation uses 4.4 BSD Lite Server running on the RT-Mach microkernel as an implementation of shared protocol processing server. BSD Lite Server in the gures indicates this implementation. The third implementation uses user-level protocol processing library on RT-Mach. It also uses 4.4 BSD Lite Server for maintaining sockets, but protocol processing is done in the library. It is shown as Socket Library Round Trip Cost (Milli-Seconds) Transfer Size (Bytes) FreeBSD BSD Lite Server Socket Library RT Communication Figure 7. Round Trip Cost of Client-Server Communication in the gures. The last one is our real-time communication mechanism on RT-Mach. RT Communication indicated in the gures is this mechanism. We used the xed priority scheduling policy on RT-Mach to measure BSD Lite Server, Socket Library, and RT Communication Basic Performance Figure 7 show the round trip cost between a client and a server machine across Ethernet using UDP/IP. The measurement was repeated 1000 times and the averages are taken. 4.4 BSD Lite Server is the worst in the four implementation, because of the overheads of local IPC cost between the benchmark program and the 4.4 BSD Lite Server on both client and server side. The local IPC happens when the client sends a request, the server receives the request, the server sends a reply, and the client receives the reply, so it occurs four times in one round trip. Also, context switching is necessary four times. The total overhead is more than 1ms. The socket library implementation is about 0.2ms slower than FreeBSD. The overhead comes from the packet lter. It needs to search the receiver of incoming packets. IPC from microkernel to the receiver task is 6

7 Time (Milli-Seconds) Time (Milli-Seconds) FreeBSD BSD Lite Server Socket Library RT Communication Iterations (Times) Figure 8. Without Background Job Iterations (Times) FreeBSD BSD Lite Server Socket Library RT Communication also expensive, where the receiver just returns from trap call in FreeBSD. But these overheads are less than 0.2ms, and relatively smaller then that of 4.4 BSD Lite Server. Our Real-Time communication mechanism is slightly slower than the socket library implementation. This causes the priority propagation when the server receives a packet Eect of Priority Propagation To show the eects of priority propagation between a client to a server, we created a client-server benchmark test. The benchmark is very simple. A client sends a request packet to a server, then the server consumes 100ms of computation time, then sends a reply packet to the client. The measurement starts before the client sends the request packet, and ends after the client receives the reply packet from the server. We executed this cycle 100 times with two dierent conditions and show elapsed time of each iteration. Figure 8 shows the result of the benchmark when the server machine is not under heavy load. Figure 9 depicts the benchmark result with a heavy load condition on server machine. We executed RT-Mach kernel build as the background job on the server machine. When the server machine is not under the loaded condition, the elapsed time of all four implementation is very stable. But once the server machine is heavily Figure 9. With Background Job loaded, FreeBSD, 4.4 BSD Lite Server, and Socket Library compete with the background job, so the elapsed time becomes very unstable. The unstability of Real- Time communication mechanism with loaded condition is about 20ms caused by I/O interrupts, and other methods are about 250ms 300ms because they compete with the back ground job. 6. Conclusion In this paper, we addressed the problems which arise in protocol processing of distributed real-time communication. To examine the current available protocol processing technologies, we have compared the structure of a traditional operating system and some modern operating systems from protocol processing point of view, and discussed the advantages and disadvantages. Recent modern operating system architectures which use user-level socket library and in-kernel packet lter meet the demands of real-time communication protocol processing to build a preemptable protocol processing. We are extending such technology to be a fully preemptable and priority inversion free protocol processing mechanism maintaining high performance and high exibility. The main idea of our system is that the server priority can be handed o from the client before 7

8 protocol processing begins by using real-time packet lter. Currently, we have implemented the real-time communication mechanism. By comparing the performance of dierent protocol processing mechanism, our real-time mechanism is about 0.2ms slower than In- Kernel protocol processing, but about more than 1ms faster than shared protocol processing server architecture. Under heavy loaded condition in server machine, our real-time communication mechanism gets stable performance compared to other protocol processing. Currently, our real-time communication mechanism is still experimental, we will complete the implementation and evaluate in more detail. We are building a real-time Java environment on top of the RT-Mach microkernel[13]. One of the facility of real-time Java is real-time remote method call across a network. To implement remote method call in this environment, we are planning to use the mechanism described in this paper. Acknowledgment We would like to thank the members of the MKng Project for their variable inputs and comments. References [1] D. Ferrari and D. Verma. A Scheme for Real-Time Channel Establishment in Wide-Area s. IEEE Journal on Selected Areas in Communications, pages 368{379, Apr [2] M. N. Group. Server Design, Sept [3] N. C. Hutchinson and L. L. Peterson. The x- Kernel: An architecture for implementing network protocols. IEEE Transactions on Software Engineering, 17(1):64{76, Jan [4] D. Kandlur, K. G. Shin, and D. Ferrari. Real-time communication in multi-hop networks. IEEE Trans. on Parallel and Distributed Systems, pages 1044{1056, Oct [5] T. Kitayama, T. Nakajima, and H. Tokuda. RT-IPC: An IPC extension for Real-Time Mach. In Proceedings of the USENIX Symposium on Microkernel and Other Kernel Architectures, [6] C. Lee, K. Yoshida, C. Mercer, and R. Rajkumar. Predictable communication protocol processing in realtime mach. In the proceedings of IEEE Real-time Technology and Applications Symposium, June [7] S. J. Leer, M. K. McKusic, M. J. Karels, and J. S. Quarieran. The Design and Implementation of the 4.3BSD UNIX Operating System. Addison Wesley, [8] C. Maeda and B. N. Bershad. Protocol Service Decomposition for High-Performance ing. In Proceedings of the 14th Symposium on Operating Systems Principles, [9] T. Mathisen. Pentium secrets. Byte magazine. [10] S. McCanne and V. Jacobson. The BSD Packet Filter: A New Architecture for User-level Packet Capture. In Proceedings of the 1993 Winter USENIX Conference, Jan [11] C. W. Mercer, S. Savage, and H. Tokuda. Processor Capacity Reserves for Multimedia Operating Systems. In Proceedings of the IEEE International Conference on Multimedia Computing and Systems, May [12] C. W. Mercer, J. Zelenka, and R. Rajkumar. On Predictable Operating System Protocol Processing. Technical Report CMU-CS , Carnegie Mellon University, [13] A. Miyoshi and H. Tokuda. Real-Time Java Server for Real-Time Mach. In Proceedings of the Fifth International Workshop on Parallel and Distributed Real- Time Systems, Apr [14] T. Nakajima and H. Tokuda. User-level Real-Time System on Real-Time Mach. In Proceedings of 4th International Workshop on Parallel and Distributed Real-Time System, [15] R. Rajkumar, M. Gagliardi, and L. Sha. The Real- Time Publisher/Subscriber Inter-Process Communication Model for Distributed Real-Time Systems: Design and Implementation. In Proceedings of the IEEE Real-time Technology and Applications Symposium, June [16] S. Savage and H. Tokuda. RT-Mach Timers: Exporting Time to the User. In Proceedings of USENIX 3rd Mach Symposium, Apr [17] H. Tokuda and T. Nakajima. Evaluation of Real-Time Synchronization in Real-Time Mach. In Proceedings of 2nd USENIX Mach Workshop, Nov [18] H. Tokuda, T. Nakajima, and P. Rao. Real-Time Mach: Towards a Predictable Real-Time System. In Proceedings of USENIX 1st Mach Workshop, Oct [19] M. Yuhara, B. N. Bershad, C. Maeda, and J. E. B. Moss. Ecient Packet Demultiplexing for Multiple Endpoints and Large Messages. In Proceedings of the 1994 Winter USENIX Technical Conference, Jan