Development and Evaluation of QoS Measurement System for Internet Applications by Client Observation

Transcription

1 JOURNAL OF INFORMATION SCIENCE AND ENGINEERING 18, (2002) Development and Evaluation of QoS Measurement System for Internet Applications by Client Observation Department of Information Systems and Multimedia Design School of Engineering Tokyo Denki University Chiyoda-ku, Tokyo, , Japan Various applications provide convenient services in the Internet. As these applications are widely used, it is important that the quality of these services (QoS), such as the response time and throughput, be high. Several ways to measure network performance have been developed. However, QoS management is still difficult because of the lack of effective tools to evaluate network system usability. The goal of our study was to develop a new QoS measurement system for network system usability. In our approach, we measure system usability and performance through mechanisms installed in the clients. The purpose of our system is to make clear the behavior of the client applications, and this allows us to measure system performance and system usability. In this paper, we describe the fundamental concept behind client observation and its performance indices, and the design and implementation of a performance evaluation system implemented through observation. In addition, we confirm the effectiveness of our system through experiments. Keywords: quality of service, quality evaluation, performance measurement, performance management, network management, internet 1. INTRODUCTION Many kind of applications on the Internet are widely used. So it is important that quality of these services, such as response time and throughput, be high. To achieve high quality the service in network systems, system administrators should evaluate how their systems are working and should operate their systems so as to perform users requests and optimize performance. To manage network system performance, it is important for administrators to be aware of system usability factors, such as access delay, and processing time, and data transfer throughput. Several ways to evaluate the performance of network systems have been developed: Statistical analysis is applied to activity logging in servers. This is a popular way to evaluate server performance. However, this analysis cannot provide any hints about network links or clients. Benchmarking is another performance measurement method. It can provide various indices of server performance. However, the benchmark requires a special environment, and the results are valid only for that environment. Network monitoring allows us to evaluate network usage at the datalink level. However, It focuses net- Received August 30, 2001; accepted April 15, Communicated by Jang-Ping Sheu, Makoto Takizawa and Myongsoon Park. 891

2 892 work equipment management in the network layer. Its results do not indicate the quality of application services. This is because the application level performance includes not only the characteristics of the datalink, but also many other performance factors. Therefore, we need an evaluation method for end point application performance. The goal of our study was to develop a new quality evaluation system. In our approach, the performance through measurement mechanisms installed in clients are indices the systems quality. 2. QUALITY EVALUATION FOR INTERNET APPLICATIONS 2.1 Quality of Internet Applications To evaluate the quality of Internet applications from the point of view of usability, system administrators must know how their services are working and must improve them to satisfy user requests. The quality of Internet applications with regard to usability is determined by how the client provides performance to the user; that is, the system administrator should be aware of the performance factors in client system. These factors are as follows: Response time: 1. time that elapses while a transaction is performed in the clients; 2. time that elapses while the clients send a request message to the servers; 3. time elapses while the servers process the request message and prepare a response message; 4. time elapses while the servers send a response message to the clients. Throughput: 1. amount of data processing per seconds in a transaction; 2. amount of data transfer per seconds in request message transferring; 3. amount of data transfer per seconds in response message transferring. Therefore, an evaluation tool are needed to measure mentioned above indices. 2.2 Necessary Functions of Evaluation Tools We consider a common framework to evaluate the performance of an end point application. The evaluation tool should have the following functions: 1. The tool is able to measure the throughput and response speed at an end-point application, which impact the user. The performance evaluation tool aims to improve the performance of the end-point application. The total system performance is not always related to the performance of the network path. Therefore, the network system performance should be evaluated in the client applications. 2. The tool handle various kinds of datalinks. The Internet has been implemented on various datalinks, such as Ethernet, ATM, FDDI, Token Ring, X.25, xdsl, and so on. TCP/IP technology is a set of proto-

3 QOSMESUREMENT SYSTEM FOR INTERNET APPLICATIONS 893 cols of the upper layers of these datalinks. Therefore, the measurement method should be independent of datalinks. 3. The measurement tool be independent of applications. Various applications and application protocols are utilized on the Internet. The performance measurement approach should be a standard framework, and it should not be dependent on a single application or a single application protocol. 4. The measurement tool should be applicable to existing applications without any modification. It is costly to modify application software to operate a measurement tool. Also, a number of applications would be difficult to modify for use with a measurement tool. 5. The measurement tool should be applicable to running systems. Using computer simulation, it is difficult to calculate all the performance factors, and the benchmark is only valid under specified conditions. It is more effective to evaluate running systems by measuring performance in a real situation. 2.3 Conventional Performance Measurement Several tools to evaluate network performance have been developed: 1. Statistical analysis of server logs. Statistical analysis of server access logs allows us to determine the operation status of the servers, such as the number of accesses, the number of data transfers, and the processing time. However, the results of analysis of server logs only indicates the performance of the servers themselves. The performance of the clients and the network links is not included in the results. 2. Measurement of network usage and Round Trip Time (RTT). The Simple Network Management Protocol (SNMP) is widely used to measure network usage [1, 2]. System administrators can use simple tools, such as ping and traceroute, todiagnose networks. However, TCP performance degrades as network usage increases, and it is also affected by the characteristics of the network links [3, 4]. The performance of an end-point application is affected by not only the network usage, but also by the characteristics, end-to-end throughput, and capacity of the servers and clients. 3. Benchmark. Some benchmark tools are available, such as SPECweb [5], WebStone [6], ttcp, DBS [7] and so on. These tools provide many indices of exact application performance. However, it is difficult to set up suitable benchmark conditions in order to reproduce those of servers under actual operating conditions. 4. Packet dumping. Analysis of packet dumping provides many indices of the datalink level, such as tcpdump [8] and snoop. Some tools, such as RMONv2 and ENMA [6], can calculate the indices in the TCP layer. However, packet dumping can only be applied to specific datalinks and traffic shared networks, and it requires much CPU power. Even in the Ethernet environment, packets can not be observed in a switched network.

4 894 Accordingly, we need a new performance evaluation tool for network systems that can be applied to actual systems operation under various configurations. 3. EVALUATION METHOD BY CLIENT OBSERVATION 3.1 Fundamental of Client Observation The goal of our performance evaluation method is to provide a suitable, common framework to measure network system performance usability. We do this by measuring the actual network system performance from the user s point of view. The proposed method involves observing the behavior of the client applications, as shown in Fig. 1. To access the network facility and enable communication from a client application to a server, applications always use the system calls in the Operating System (OS). A layer attached to observe the applications, called the Observation layer, reveals all the system call level behavior in the client applications. Fig. 1. Fundamental of client observation. The observation layer can effectively evaluate the network system quality because the client application stands on the end-user side directly. Moreover, the observation layer interfaces are the same as the standard Application Programming Interfaces (APIs), such as the BSD Socket, Transport Layer Interface (TLI), and Winsock. In other words, applications can access the network via the observation layer through standard system calls without modification.

5 QOS MESUREMENT SYSTEM FOR INTERNET APPLICATIONS Communication Model in the Observation Layer In this section, we describe the target communication model for measurement through the observation layer. In popular Internet application protocols, such as the Hypertext Transfer Protocol (HTTP) [10], Simple Mail Transfer Protocol (SMTP) [11], Post Office Protocol 3 (POP3) [12] and so on, the clients the perform following procedures, that are called the Request-Response Communication Model : 1. Connection Establishment connect() system call. 2. Sending of request write() system call. 3. Receiving of result read() system call. 4. Repeat 2 and Connection Close close() system call. For example, Fig. 2 shows the communication flow diagram in HTTP. The client sets up the connection, sends the server a request as GET, PUT and POST, then receives and the results that the server processed. Fig. 2. Communication flow in HTTP. 3.3 Performance Index The observation layer can measure various performance indices by observing system calls. Fig. 3 shows the performance indices based on the Request-Response Communication Model. The contents of the measurement are the time period of T1 T7 and the amount of data transferred.

6 896 Fig. 3. Performance indices. Our system is able to measure the following performance indices: Transaction time (Tt) (T1 T7) The transaction time (Tt) is the time that elapses during a TCP connection. Specifically, it is defined as the period from the time when the open() system call is invoked (T1 in Fig. 3) to the time when the close() system call is returned (T7 in Fig. 3). This index is the processing time of a transaction. Connection establishment time (Te) (T1 T2) The connection establishment time (Te) is the time that elapses during connection setup. It is defined as being the time from when the connect() system call is invoked (T1 in Fig. 3) to when it is returned (T2 in Fig. 3). This index is the RTT (Round Trip Time) of the network and the TCP/IP performance of the clients and servers. Request data transfer time (Ts) (T3 T4) The request data transfer time (Ts) is the time that elapses for the data transfer to send the servers to the request data in the Request-Response Communication Model in above mentioned. It is defined as the time from when the write() system call is invoked (T3 in Fig. 3) to when it is returned (T4 in Fig. 3). Response data transfer time (Tr) (T5 T6) The response data transfer time (Tr) is the time elapsed for the data transfer to receive the processed data in the servers, defined as the time from when the read() system call is invoked (T5 in Fig. 3) to when it is returned (T6 in Fig. 3). It includes not only the

7 QOSMESUREMENT SYSTEM FOR INTERNET APPLICATIONS 897 time that elapses during for the data transfer, but also the time needed to process a request in the servers. Therefore, this value indicates the performance of the server systems and the network. Request data size (D s ) The request data size (D s ) is the amount of data transferred to the servers as the request. It is defined as the number of bytes transmitted by write() system call. Response data size (D r ) The response data size (D r ) is the amount of data transferred from the servers as the response. It is defined as the number of bytes received by read() system call. Request data transfer rate (R s ) The request data transfer rate (R s ) is the transfer rate of the requested data sent to the servers. It is defined as Ds (Request data size) R s =. (1) T (Request data transfer time) s Response data transfer rate (R r ) The response data transfer rate (R r ) is the transfer rate of the data returned by the servers. It is defined as Dr (Response data size) R r =. (2) T (Response data transfer time) r Transaction transfer rate (R a ) The transaction transfer rate (R a ) is the rate of all data transferred during the time that elapses during the entire process. It is a performance index based on the data size of the transaction. It is defined as R a Ds (Request data size) + Dr (Response data size) =. (3) T (Transaction time) t 4. DESIGN OF PERFORMANCE EVALUATION SYSTEM 4.1 System Model Fig. 4 shows the system model of the performance evaluation system implemented through client observation. The system consists of four components: an observation layer, a data slicer, a database, and a manager.

8 898 Fig. 4. System structure Observation layer The observation layer monitors the system calls between the applications and the OS. The observation layer has proxy system call functions. The proxy system call functions stands for the application instead of the actual system call functions in the OS. When the application calls system calls, the proxy system call procedures in the observation layer are invoked. Then the observation layer records the time of each state (T 1 T 7 ), counts the amount of data transferred in each TCP connection, and simultaneously, invokes the actual system call in the OS Data slicer The data slicer processes the measured raw data in the observation layer with the conditions such as the IP address and the port number, and performs basic analysis Database The data slicer stores the processed data in the database. The database is used for further analysis by the manager. The database consists of a matrix into which are put the server address, port number, time elapsing in each state, data size, and transfer rate in time stamp order. The structure of the database is quite simple because a data entry for each TCP connection forms a

9 QOSMESUREMENT SYSTEM FOR INTERNET APPLICATIONS 899 single row of data. Therefore, it reduces the amount of overhead for measurement in the clients Manager The manager collects measurement data from the target of the remote clients. It analyzes the data and provides a graphical interpretation of the results. Currently, the manager is implemented as the interface to Gnuplot to view the analysis results. 4.2 Implementation We are currently implementing the performance evaluation system on World Wide Web (WWW) systems. The observation layer, data slicer, and database are implemented in ANSI C on Sun Solaris. The manager is implemented in shell programs and C programs. The observation layer has a proxy BSD Socket interface for invoking from the target application. The proxy interface is implemented as a shared library that uses dynamic linking. It enables replacement of the actual Socket interface in the OS with the proxy Socket interface in the observation layer when the application is executed dynamically. Therefore, no modification of the application program is needed for measurement. Even if the target system does not have dynamic linking facilities, we do not have to modify the application; we recompile/link to the observation layer module as a static library. The proxy Socket interface is invoked when the application calls a Socket system call to access the network. The observation layer monitors connection open, close, write, and read, records the time that elapses, and counts the data transfers. It continuously calls actual Socket system calls in the OS. 5. EXPERIMENTS CONDUCTED TO EVALUATE THE EFFECTIVENESS OF OUR APPROACH 5.1 Comparison of Our System With Server Log Analysis In order to evaluate our implementation, we applied our system to actual running systems. To confirm the measurement accuracy of our system, we compared the measure of our system with the analysis results of the server access log. Statistical analysis of the server access log was used to evaluate the performance. The configuration of the target system is shown in Table 1. The target system was configured on an Apache HTTP server, a Sun Ultra 60 workstation and an ApacheBench client program on Sun Ultra 10 workstations. These devices were connected on 100Base-T Ethernet. In the server, we prepared 20,000 objects which had the same distribution based on data size in a WWW server in the Department of Computer Science, Meiji University. Fig. 5 shows the frequency distribution of object sizes in the server. The client randomly accessed the server objects 10,000 times. We measured T t (Transaction time) in our system and the process time of the server log. Fig. 6 shows the frequency distribution of time that elapsed (milliseconds).

10 900 Table 1. Configuration of experiment. Server Client Server program Client program Sun Ultra 60 (Two UltraSPARC-II 360MHz) (Solaris 2.6) Sun Ultra 10 (UltraSPARC-Iii 333MHz) (Solaris 2.6) Apache 1.3 ApacheBench 1.3 Size (bytes) Fig. 5. Frequency of file size in server. Fig. 6. Comparison T t in observation layer with process time in server log.

11 QOSMESUREMENT SYSTEM FOR INTERNET APPLICATIONS 901 The results were distributed around 200ms for T t of our system and concentrated around 10ms in the server log. The average of our system was 815ms, and one of the server logs was 418ms. The difference reached 51 percent. T t in our system is the entire time that elapses during the transaction, including the time needed for connection setup, data transfer, and connection close procedures in the Socket layer. In contrast, the server log records only the server side application activity. Therefore, the server log cannot measure performance with respect to connection setup/closing. Moreover, the relation between data size and elapsed time is shown in Fig. 7 (for T t in our system) and Fig. 8 (in the server log). In our system, the elapsed time increased depending on the data size. In contrast, in the server log, the elapsed time was distributed around 10ms when the data size was from 0bytes to 8,500bytes, distributed around 200ms when the data size was from 8,500bytes to 17,000bytes, and distributed around 330ms when the data size was from 17,000bytes to 26,000bytes. Time elaps (ms) Fig. 7. Distribution of data size and process time in observation layer. Time elaps (ms) Fig. 8. Distribution of data size and process time in server log.

12 902 The server logs records the performance in the application level on the server side. Once the data are put into the buffer in the OS, before an actual transmission occurs, it appears that data transfers are completed early in the server programs. Since the size of the socket buffer is configured at about 8kbytes in the Solaris environment, the transmission process is carried out in blocks of 8kbytes. Generally, because almost all WWW contents on the Internet consist of 8kbyte or smaller objects [13], server logs are inaccurate records. Therefore, our system is more accurate than the conventional method based on the experimental results. 5.2 Comparison of Server Performance We conducted other experiments to measure the performance of the server. We compared a measurement of the Fast server with one of the Slow server. The configuration of the target system is shown in Table 2. Server A was connected on 100Base-T Ethernet, and Server B was connected on 10Base-T Ethernet. We prepared 20,000 objects in the servers as mentioned in section 5.1. The client randomly accessed the server objects 10,000 times. We measured T t (Transaction time) in each server. Server A Server B Client Table 2. Configuration of experiment. Server program Client program Sun Ultra 60 (Two UltraSPARC-II 360MHz) (Solaris 2.6) Sun SPARCstation 5 (HyperSPARC 170MHz) (Solaris 2.6) Sun Ultra 10 (UltraSPARC-Iii 333MHz) (Solaris 2.6) Apache 1.3 ApacheBench 1.3 Fig. 9 shows the frequency distribution of T t (milliseconds). The results distributed around 200ms in Server A, and distributed around 1,600ms in Server B. Therefore, our system can measure differences in server performance. 6. SUMMARY In this paper, we have explained why a new method to evaluate the QoS of Internet applications is needed, and we have proposed a QoS measurement method based on client observation. Our method involves observing the behavior of the client application and exactly measuring its performance. Our system can effectively evaluate the network system quality because the client application stands on the end-user side directly. In addition, we have used our system on an actual WWW system and confirmed its effectiveness through experiments.

13 QOSMESUREMENT SYSTEM FOR INTERNET APPLICATIONS 903 Fig. 9. Comparison T t in server A with T t in server B. REFERENCES 1. A. Leinwand, Accomplishing performance management with SNMP, in Simple Times, Vol. 1, 1992, pp A. Leinwand and K. F. Conroy, Network Management: A Practical Perspective, 2nd ed., Addison-Wesley, M. Allman, C. Hayes, H. Kruse, and S. Osterman, TCP performance over satellites links, in Proceedings of the 5th International Conference on Telecommunications Systems, 1997, pp B. Vandalore, S. Kalyanaraman, R. Jain, R. Goyal, and S. Fahmy, Simulation study of world wide web traffic over the ATM ABR service, in Proceedings of SPIE Symposium on Voice, Video and Data Communications, Vol. 3530, 1998, pp SPECweb99 Benchmark, Standard Performance Evaluation Corporation, spec.org/osg/ web99/. 6. WebStone, The Benchmark for Web Servers, webstone/. 7. Y. Murayama and S. Yamaguchi, DBS: powerful tool for TCP performance evaluation, in Proceedings of SPIE, International Society for Optical Engineering, Vol. 3231, 1997, pp tcpdump, LBNL s Network Research Group, 9. Y. Nakamura, K. Chinen, H. Sunahara, S. Yamaguchi, and Y. Oie, ENMA: The WWW server performance measurement system via packet monitoring, in Proceedings of INET 99, Internet Society, R. Fielding, J. Getty, J. Mogul, H. Frystyk, and T. Berners-Lee, Hyper Text Transfer Protocol HTTP/1.1, RFC2068, J. Postel, Simple Mail Transfer Protocol, RFC821, J. Myers and M. Rose, Post Office Protocol Version 3, RFC1939, M. Nabe, K. Baba, M. Murata, and H. Miyahara, Analysis and modeling of www

14 904 traffic for designing internet traffic, IEICE Transaction, Vol. J80-B1, 1997, pp H. Saito and T. Chusho, Design and implementation of network performance evaluation system through client observation, in Proceedings of INET2000, Internet Society, H. Saito and T. Chusho, Quality evaluation of network services by client observation, in Proceedings of the 15th International Conference on Information Networking 01 (ICOIN15 01), pp Hiroki Saito was born in Tokyo. He received B.S., M.E. and Ph. D degrees in Computer Science from Meiji University, Japan, in 1996, 1998 and 2001, respectively. He is since 2002 an instructor in the Department of Information Systems and Multimedia Design, School of Engineering, Tokyo Denki University, Japan. He is a member of Internet Society, Information Processing Society Japan and Japan Society for Software Science and Technology. His research interest includes performance measurement in the Internet, network management and Internet architecture.