10-Gbps IP Network Measurement System Based on Application-generated Packets Using Hardware Assistance and Off-the-shelf PC

Transcription

1 10-Gbps IP Network Measurement System Based on Application-generated Packets Using Hardware Assistance and Off-the-shelf PC Kenji Shimizu Katsuhiro Sebayashi Seiki Kuwabara Mitsuru Maruyama NTT Network Innovation Laboratories {shimizu.kenji, sebayashi.katuhiro, kuwabara.seiki, Abstract Targeting high-bandwidth applications such as video streaming services, we discuss advanced measurement systems for high-speed 10-Gbps networks. To verify service stability in such high-speed networks, we need to detect network quality under real environmental conditions. For example, test traffic injected into networks under-test for measurements should have the same complex characteristics as the video streaming traffic. For such measurements, we have built Internet protocol (IP) stream measurement systems by using our 10-Gbps network interface card with hardware-assisted active/passive monitor extensions based on low-cost off-the-shelf personal computers (PCs). After showing hardware requirements and our implementation of each hardware-assisted extensions, we report how we build pre-service and in-service network measurement systems to verify the feasibility of our hardware architecture. A traffic-playback system captures packets and stores traffic characteristics data without sampling any packets and then sends them precisely emulating the complex characteristics of the original traffic by using our hardware assistance. The generated traffic is useful as test traffic in pre-service measurement. A distributed in-service network monitoring system collects traffic characteristics at multiple sites by utilizing synchronized precise timestamps embedded in video streaming packets. The results are presented on the operator s display. We report on their effectiveness by measuring 1.5-Gbps uncompressed high-definition television traffic flowing in the high-speed testbed IP network in Japan. Index Terms 10 Gbps, network measurement, network interface card, stream I. INTRODUCTION To provide stable services based on high-quality streaming applications, which are growing rapidly, there is strong demand for advanced Internet protocol (IP) network measurement systems that focus on the complex characteristics of applications. In our previous experiment on deploying 1.5-Gbps uncompressed high-definition television (HDTV) transmission systems[1], we experienced network trouble such as packet dropping even though the network quality had been verified beforehand using existing test traffic generators. Furthermore, after the service started, it became instable when the network bandwidth was fluctuated by competing traffic in the same network. Targeting such high-bandwidth applications, we have developed a novel 10-Gbps network interface card (NIC), which has hardware-assisted network measurement extensions[2]. In this paper, by using our NIC, we discuss how to build measurement systems for pre-service and in-service network measurements. The progress from the paper[2] includes the development of concrete application systems of the NIC. In particular, to realize an active measurement system, we have developed newly hardware assistance to control the probepacket characteristics in a packet-by-packet manner. For pre-service measurements which are conducted before services start, we have implemented a traffic-playback system that can generate test traffic with exactly same characteristics as that from real streaming servers, which is useful for reliable evaluations of service availability. For in-service measurement which are conducted during the services, we have implemented a distributed in-service network monitoring system that collects traffic characteristics among multiple sites in real time by using the precise synchronized timestamps embedded by our NIC. We built the systems based on low-cost off-theshelf personal computers (PCs) which enables us to deploy our systems in multiple measurement sites. We conducted experiments using the developed systems built upon a high-speed R&D testbed network in Japan. The results show the feasibility of our systems. The rest of this paper is organized as follows: section II shows related work. Section III details and classifies measurement systems. Then, section IV describes requirements for building measurement systems in 10-Gbps networks based on low-cost off-the-shelf PCs. Section V shows our hardware design and implementation, while section VI proposes novel measurement systems based on our hardware-assisted extensions. Section VII reports the results of experiments performed on a testbed network. Finally, section VIII concludes the paper. II. RELATED WORK We discuss measurement technique and systems from the viewpoint of whether they can be applied to measurements of high-bandwidth IP streaming service in 10-Gbps networks. Packet sampling such as sflow is used in high-speed networks to collect traffic information[3]. However, the network qualities such as jitter and burstiness are calculated by using the continuous packets. Therefore, the measurement accuracy decline in sampling technique, which is not appropriate for streaming services. Recently, lots of high-performance 10-Gbps NIC have been proposed. For example, Dag and TILExpress provide measurement system [4] which can be used for 10-Gbps non-sampling /11/$ IEEE

2 Active measurement system Traffic generator sends test traffic at an arbitrary bitrate. Passive measurement system Router/switch Mirrored port Fig. 1. Timestamp at sender IP network Optical splitter Traffic capturing systems Flow-by-flow analysis systems Overview of measurement systems. Test-traffic analysis system. passive network measurement. Although their approaches are based on hardware assistance like ours, they do not have active measurement extensions such as injecting test traffic into under-test networks, which we consider important to evaluate networks for streaming services. The paper[5] reporting on active measurements takes advantage of a NIC s interrupt coalescing feature to make bursty packet trains, which is useful for available bandwidth estimation. The available bandwidth is important network quality metric, which is, however, out of scope of this paper. Traffic playback feature discussed in this paper can be implemented by software such as libpcap-based tcpreplay[6] and by fully-customized hardware such as Spirent Avalanche[7], which proves the strong demand for such a technique. The software implementation lacks accuracy in controlling the traffic characteristics. Simple hardware extensions combined with low-cost PCs like our systems can balance the cost and accuracy. III. HOW NETWORK MEASUREMENT SYSTEMS ARE DEPLOYED? Here, we classify network measurement systems required for streaming services. A. Pre-service Measurements Recent streaming services use large amount of bandwidth exceeding 1 Gbps [1]. Such high-bandwidth traffic may cause networks congestion resulting in troubles such as packet droppings. When packets drop during video conference services, for example, user experiences badly degrade because of distorted video and audio communication. Therefore, we should verify whether the network can transmit the test traffic at the same bitrate as the service beforehand. In figure 1, active measurement (AM) systems are used to inject test traffic into the networks under test. An operator can evaluate the availability by seeing whether the test traffic flows without problems of delay, jitter and packet dropping. However, this method becomes inaccurate in high-bandwidth streaming services because the test traffic characteristics are different from that of real streaming traffic which have complex behaviors. It is difficult to generate test traffic with such complex characteristics by using an existing traffic generator. B. In-service Measurements Even after streaming service starts, network measurements should be carried out to guarantee the service quality. For example, when file transfer applications are used at the same time with the streaming service, a bandwidth shortage might make the streaming service unstable. In figure 1, passive measurement (PM) systems are used for flow-by-flow bandwidth analysis, which is useful for identifying flows that are undesirable for service stability. 1) Application-coexistent network monitor: In paper [2], we proposed a novel in-service measurement system that appends precise timestamps to all the packets that streaming servers send and receive. It means that the packets delivering video data also work as probe packets of AM systems. By leveraging this approach, we can determine how the application s packet detects the network characteristics exactly. The key component for building such systems is our NIC, which has various network measurement extensions. C. Measurement at multiple sites Wide-area deployment of measurement systems enables us to evaluate the service stability more reliably because the network quality metrics such as one-way delay[8] and jitter can be determined among multiple sites. Such deployment, however, become costly because current measurement systems supporting 10-Gbps networks are still very expensive. This is partly because lots of complex hardware functions are implemented in each network interface with application-specific integrated circuits (ASICs), which take over almost all of the protocol processing[9]. The low-cost way to deploy the systems is to build measurement systems on an off-the-shelf PC. However, the PC architecture lacks performance for 10-Gbps network measurement as explained in the next section. IV. HARDWARE REQUIREMENTS AND CHALLENGES In this section, we discuss the requirements and challenges for network measurement systems targeting 10-Gbps networks. A. 10-Gbps wire-speed traffic handling Network measurement systems analyze the traffic after capturing and generating packets even if they flows at any bitrate with instantaneous bursty bandwidth. Therefore, measurement systems must handle up-to-10- Gbps wire-speed traffic. With an off-the-shelf PC system, however, the packet capturing and generating throughput is limited severely by the packet size, system load, physical bus bandwidth, and so on[10]. B. Precise timestamping To measure delay and jitter accurately in 10-Gbps networks, the timestamps should be more precise than those of an offthe-shelf PC. Our measurements require microsecond precision. In addition, they should be exactly synchronized with each other even in multiple measurement sites.

3 PCI-bus width in bits L PCI Transferred data TS Wait LBurst Time TS Idle TS Set TS Bd Fig Gbps network interface card with active/passive network measurement extensions. C. Controlling packet sending intervals In active measurements, inter-packet gaps (IPGs) must be controlled exactly to make arbitrary kinds of packet pairs and packet trains. For example, to estimate the available bandwidth in a 10-Gbps networks, the IPGs may be configured around a certain number of microseconds. A PC cannot adjust the IPG to such a small value because of system load and bandwidth limitations[5], which would lead to inaccurate results. V. HARDWARE DESIGN AND IMPLEMENTATION We intend to implement them in our NIC (figure 2). The NIC works as a network interface for three 10-Gbps protocols including 10GbE LAN-PHY, WAN-PHY, and OC-192c POS when installed in the PCI-X bus of PCs. A. 10-Gbps wire-speed traffic capture/generation We have already proposed a new direct memory access (DMA) algorithm suitable for capturing and generating traffic in high-speed networks [2]. Although our NIC can work as the normal network interface, it can switch to network measurement mode instantly. 1) Modified DMA algorithm: We have modified the DMA algorithm for both capturing and generating packets. Here, we show the algorithm focusing on the capturing procedure. When packets arrive, the NIC temporary stores only the configured size from their header, discarding the remains. Then, the monitoring data, which includes the packet s original size, is appended to each extracted header. Finally, headers are concatenated into a single large data block in the onboard memory until the summation of concatenated-headers reaches the pre-configured size. The resulting single data block is transferred to the main memory all at once. All the DMA transfer procedures above are managed by the NIC s device driver by using buffer descriptors (BDs), which are data structures allocated in initialization. The NIC s DMA controller finds out where it should transfer the data block by referring to the address field in the BDs. Therefore, the NIC retrieves the BD from the main memory before receiving packets. This method helps to reduce per-packet BDs processing overheads and amount of data to be handled, resulting in highly efficient DMA transfer and a lighter central processing unit (CPU) load. Fig. 3. Estimation of onboard memory size from PCI bus utilization. 2) Onboard memory size: By analyzing the utilization efficiency of the PCI-bus, we estimated the required size of NIC s onboard memory. The PCI bus usage is shown in figure 3 when a single large block (explained in the previous section) is transferred. The effective DMA bandwidth V DMA [bps] is 1 V DMA = V PCI 1+ L PCI {TS G +(N 1) TS W ait } L Pack (1) where V PCI [bps] is the PCI-bus bandwidth, L PCI [bit]isthe PCI-bus width in bits, and L Pack [bit] is the data block size transferred by a single DMA. TS Wait is the number of idletime slots between continuous data transfers. Moreover, TS G = TS Idle + TS Set + TS Bd (2) LPack L Pack 1 N = L Burst L Burst (3) L Pack 0 < 1 L Burst are given. Here, L Burst [bit] is the amount of continuously transferred data. TS Idle, TS Set, and TS Bd are the number of idle-time slots consumed after a single DMA transfer, consumed for registering the next DMA transfer, and consumed for transferring the next BD from the main memory, respectively. On the other hand, required DMA bandwidth V DMA can be expressed as: V DMA = L Header V Wire, (4) L Packet where L Packet [bit] is the average size of incoming packets, V Wire [bps] is the wire speed of the network, and L Header [bit] is the size of an extracted header. Furthermore, when L MRU [bit] is the maximum receive unit (MRU) size, the required size of onboard memory is given by L Memory = L Pack + L MRU. (5) From equations (1), (4), and (5), we obtain L Memory = L MRU + L PCI {TS G +(N 1) TS W ait }. L Packet V PCI 1 L Header V Wire (6) The number of idle-time slots can be determined with a PCIbus analyzer. And when the MRU size is 9 kbytes, the average,

4 Buffer Descriptors (BDs) IPG value for a single DMA transfer. Fig. 4. Buffers to be sent } header data A single buffer can contain multiple headers. IPG value and size of packet to be sent. Buffer structure for traffic generator. packet size of streaming services is 300 bytes [11] and the extracted header size is 54 bytes 1, the required size of onboard memory is 12.3 kbytes, which is small enough to be allocated only in a field-programmable gate array (FPGA) without any external memory, contributing to low-cost hardware implementation. B. Precise timestamping We implemented external inputs so that our NIC can receive external timing pulses synchronized to coordinated universal time (UTC). By using them, it generates highly precise timestamps at a resolution of 100 ns synchronized around the world. This is widely used in time server systems such as network time protocol (NTP) servers[12]. Our NIC also appends timestamps to all the capturing and generating packets. C. Controlling packet sending intervals The sending data structure in the device driver is shown in figure 4. Although device drivers, in general, contain only a single packet in each buffer, our driver stores multiple packets headers ( header data in the figure) for the highly efficient DMA transfer to achieve 10-Gbps wire-speed traffic generating. In the first sending step, the BD s address is registered in the NIC s DMA controller by the device driver. After the NIC transfers the contents of the BD, it transfers the entire buffer pointed to by the buffer address in the BD to the onboard memory. To control the packet-sending intervals, we add the following parameters: 1) IPG values and sending packet size for each header data. 2) An IPG value in each BD for the interval of each DMA transfer. Based on the parameters, the NIC manipulates the sending data as follows: 1) It splits each header data from the buffer, then appends dummy data (padding by 0 ) to the headers so that the packet is sent with the designated size. In general, network measurement systems generates test traffic only with designated routing headers such as IP and media access control (MAC) addresses. Therefore, the payload part can be padded by 0 by the NIC which reduces the required bus bandwidth. 1 Ethernet header: 14 bytes, TCP header: 20 bytes, IPv4 header: 20 bytes. Traffic-playback system (capturing) Video server Client Traffic-playback systems (generating phase) Network under test Receiver (traffic analyzer) Fig. 5. Overviews of traffic-playback system (step 1, step 2). Extraction of packet headers Capturing phase (step 1) To be discarded Fig. 6. Timestamp and original packet size Concatenated Store traffic characteristics Generating phase (step 2) Values of Inter-packet gap (IPG) Original packet size Append dummy data based on Split Implementation of traffic-playback system. Control the sending packet intervals based on 2) The resulting data is sent to the network as a packet, while the NIC controls the IPG as designated in the BD by using the hardware clock. The packet-sending intervals can be configured at a resolution of 5 ns, which depends on the 200-MHz clock frequency of the FPGA. The 5-ns interval corresponds to about 6 bytes in a 10- Gbps network, which is small enough because the minimum inter-frame gap in the MAC layer is 12 bytes in the standard for 10 gigabit Ethernet[13]. VI. DEVELOPED MEASUREMENT SYSTEMS Our newly developed measurement systems are described in this section. A. Traffic-playback system On the left of figure 5, the video server transmits a video stream to a client in a local experiment. At the same time, the traffic-playback system operating in capturing mode collects the characteristics of the streaming traffic. Next, on the right, located in multiple sites, the traffic-playback systems generate test traffic precisely emulating the complex characteristics of the streaming traffic collected in step 1. Delays, jitters, and packet losses can be measured accurately by using it. 1) Implementation by using hardware-assisted extensions: Our implementation is shown in figure 6. Capturing phase (step 1), left of figure 6 Our NIC collects only the header parts of all the incoming packets, and then appends monitoring data including arrival timestamps into each collected headers, finally concatenates multiple headers in the onboard memory into a large data block for highly efficient DMA transfers. The captured data is stored in hard disk drives. Analysis of collected packets in the software Using the headers and timestamps, the software calculates

5 IPGS and stores the results in the disk. The IP and MAC addresses of the captured header are rewritten so that the generated packets can travel the network under test in the next step. Generating phase (step 2), right of figure 6 The software prepares multiple headers, IPGs and their original sizes in a buffer. After the data has been transfered to the onboard memory, the NIC splits each header, appends dummy data, and sends them while controlling IPGs as described in section V-C. Generated traffic emulating original streaming traffic is used as test traffic for an active measurement. B. Distributed in-service network monitoring system The distributed in-service network monitoring system is composed of our NIC installed in streaming servers, a network probe software, and control/display software. Our NIC in the streaming server appends precise timestamps to all the sending packets containing video data which is referred to in the measurement sites for calculating one-way delays. Network probe software This software is installed in capturing PCs at multiple sites. At each site, it captures the headers of all incoming packets without sampling any packets. It also has a role to configure the hardware-assisted extensions. After calculating the delays, jitters, IPGs distribution, and flowing traffic bitrate by using the embedded timestamps, the results are sent to the monitor control/display software at operator s site. Control/display software This software is installed in the operator s control terminal. It connects with all the network probe software using TCP/IP and sends and receives commands over TCP/IP. It provides the following functions: It shows the current configuration of the hardwareassisted extensions of each network probe. It receives statistical data and visualizes the delays, jitters, and IPGs on a time line and in a histogram. It also displays counters that indicate how many packets were received for each protocol. It show the link status of the NICs at each site. VII. EXPERIMENTS We conducted an experiment to evaluate the effectiveness of our newly developed systems. A. Traffic-playback system The basic evaluation of the traffic-playback system was done experimentally with i-visto systems [1]. The i-visto gateway sends and receives an IP-packetized uncompressed HDTV stream at 1.5 Gbps in real time. The measurement results are shown in figure 7. The upper panel in the figure shows the characteristics of traffic captured in step 1. The lower panel shows those of traffic played back in step 2. The bitrate of the traffic generated by the i-visto gateway is controlled to be exactly constant. However, various different IPG values were observed by our Frequency counter [%] Fig. 7. IPG [ s] Original traffic (upper) and played-back traffic (lower). Tokyo site 1 Tokyo site 1 Distributed in-service network monitor JGN2Plus Tokyo site 2 Displays characteristics of streaming traffic JGN2Plus 1.5-Gbps uncompressed HDTV stream Fig. 8. Experimental setup. Osaka site Video server measurement systems. Furthermore, the detailed fluctuation of the IPG was played back exactly by the system thanks to the precise timestamps and mechanism for controlling the packetsending interval. B. Distributed in-service network monitoring system We evaluated the operation of the distributed in-service network monitoring systems by deploying them in the highspeed test-bed network for R&D in Japan (JGN2Plus). The network connects multiple remote sites with a 10-Gbps IP network, in which we constructed a high-quality video sharing environment for video production use. The video-on-demand streaming servers have been used to deliver a 1.5-Gbps uncompressed HDTV stream over the network. The network setup is shown in figure 8. We captured the travelling packets by using optical splitters installed in the measurement sites. Altough such splitters may decrease an optical power resulting in some limitations of the transmission length, we took advantage of 15:85- or 20:80-ratio optical splitters in order not to induce any transmission errors of application s packets. The experiment aimed to evaluate the system s ability to measure up-to-10-gbps traffic and obtain high-resolution measurement results utilizing precise timestamps. We attempted to visualize the behavior of traffic bitrate at a fine resolution when the streaming server changes the traffic characteristics by enabling/disabling a traffic shaping function. The results are shown in figures 9 and 10. 1) Traffic burstiness (IPG distribution) without shaping The most frequently observed value was 12.3 µs, which

6 Throughput [Mbps] Throughput [Mbps] Frequency counter [%] Fig e-04 1e e-04 1e IPG [ s] IPG distribution of uncompressed HDTV streaming server. Without shaping With shaping Time (1-ms resolution) [ms] Time (1-ms resolution) [ms] Time (10-ms resolution) [ms] Fig Time (10-ms resolution) [ms] Traffic bitrate of uncompressed HDTV over IP server. corresponds to about 5.8 Gbps when 9-kbyte packets were used (upper panel in figure 9). The streaming server reads out the video data from storage at a constant interval and then sends it into the network all at once resulting in the bitrate fluctuation. This is why small IPG values were frequently observed[14]. 2) Traffic burstiness (IPG distribution) with shaping With traffic shaping, the most frequently observed IPG was around 42.3 µs, which corresponds to 1.7 Gbps (lower panel in figure 9). Considering the overheads of headers including IP, UDP, and others, this means that the 1.5-Gbps uncompressed HDTV stream was transmitted stably at a constant bitrate over the network. 3) high-resolution analysis of the traffic bitrate In figure 10, the left (right) graphs show the bitrate behavior of traffic without (with) shaping. The bursty characteristics could not be observed at the 10-ms time resolution, although we could see the detailed shape of bursty traffic at the finer resolution of 1 ms thanks to the hardware assistance of our NIC. achieve them with a low-cost off-the-shelf PC, we introduced simple hardware-assisted functions. Our experiments, which were conducted on a 10-Gbps testbed network, showed that the traffic-playback system could generate traffic exactly emulating real streaming server s traffic, which had complex characteristics. In addition, the distributed in-service network monitoring system could collect traffic characteristics from multiple sites repeatedly, which is very useful for real-time detection of network quality degradation. ACKNOWLEDGMENTS This research is partially funded by the National Institute of Information and Communications Technology (NiCT). REFERENCES [1] K. Harada, K. Hasebe, H. Kimiyama, and T. Ogura, Activities on uncompressed HD over IP using i-visto, 21th Asia-Pacific Advanced Network(APAN) Meeting, Tokyo, Japan, Jan. 24th, [2] K. Shimizu, T. Ogura, T. Kawano, H. Kimiyama, M. Maruyama, and K. Koyanagi, Application-coexistent Wire-Rate Network Monitor for 10 Gigabit-per-Second Network, IEICE Trans. Inf. & Syst., vol.e89-d, no.12, pp , Dec [3] P. Phaal, S. Panchen, and N. McKee, InMon Corporation s sflow: A Method for Monitoring Traffic in Switched and Routed Networks, RFC 3176, Sept [4] IP Traffic Monitoring at 10 Gbit and above, TELENA 2nd NGN Workshop, Municon, Germany, April 4th, [5] Cao Le Thanh Man, G. Hasegawa, and M. Murata, ICIM: An Inline Network Measurement Mechanism for Highspeed Networks, Proc. the Endto-End Monitoring Techniques and Services (E2EMON), Vancourver, Canada, pp.66-73, April [6] Tcpreplay, [7] Spirent Avalanche - Application Layer Network Testing, [8] G. Almes, S. Kalidindi, and M. Zekauskas, A One-way Delay Metric for IPPM, RFC 2679, Sept [9] Endace - DAG Network Monitoring Cards, [10] K. Shimizu, T. Ogura, T. Kawano, H. Kimiyama, and M. Maruyama, OC-48c High-speed Network PCI Card: Implementation and Evaluation, IEICE Trans. Inf. & Syst., vol.e86-d, no.11, pp , Nov [11] D.J. Parish, K. Bharadia, and A. Larkum, Using packet size distributions to identify real-time networked applications, IEE Proc.-Commun., vol.150, no.4, pp , [12] D.L. Mills, Network Time Protocol (Version 3) Specification, Implementation and Analysis, RFC 1305, March [13] IEEE Std 802.3ae-2002, Media Access Control (MAC) Parameters, Physical Layers, and Management Parameters for 10 Gb/s Operation, IEEE Computer Society, Aug [14] H. Kimiyama, K. Shimizu, T. Kawano, and M. Maruyama, Real-time processing method for an ultra high-speed streaming server running PC Linux, Proc. the Advanced Information Networking and Applications (AINA) 2004, Fukuoka, Japan, vol.2, pp , March 29th-31th, VIII. CONCLUSION We discussed network measurement systems focusing of IP streaming services, especially targeting 10-Gbps highspeed networks. After showing the detailed requirements of measurement systems, we proposed traffic-playback systems and a distributed in-service network monitoring system. To