Hybrid Network Traffic Engineering Software (HNTES)

Transcription

1 Hybrid Network Traffic Engineering Software (HNTES) Malathi Veeraraghavan and Zhenzhen Yan University of Virginia Feb. 28, 2010 Abstract: This document defines the purpose of the Hybrid Network Traffic Engineering Software (HNTES) system, lists its functional requirements, and describes its high-level architecture. This is a revised version of the HYbrid Network Engineering Software (HYNES) document, dated Dec. 30, The software has been renamed to better reflect its purpose as a traffic engineering solution. 1 Introduction Today s ESnet offers two types of connectivity services, IP-routed datagram service and virtual circuit service. This work addresses the problem of traffic engineering in this hybrid network deployment. Traffic engineering can be loosely defined as techniques that enable the best use of the deployed resources. Given that most end-host applications run on TCP/IP, most traffic is directed to the IP-routed datagram service. Some of this traffic is served better with the virtual circuit service rather than the IProuted datagram service. There are two techniques to redirect certain types of traffic arriving at an IP router to the virtual-circuit service. The first is called Lambdastation 2, which has been implemented and demonstrated. In this approach, applications signal a Lambdastation server, which is deployed in site networks, before initiating a long flow. The Lambdastation server issues a create-reservation request to the On-Demand Secure Circuits and Advance Reservation System (OSCARS) Inter-Domain Controller (IDC) 3, which reserves a circuit and at the time of the scheduled request, provisions the circuit. The virtual circuit endpoint is typically a customer or provider edge IP router within a site. Using a policy based route configuration of the router, packets corresponding to the long flow are directed to the newly established virtual circuit. When the flow completes, the circuit is released. A second technique is a complementary method in which packets for predetermined long flows are automatically redirected to virtual circuits without any triggers from applications. Our software implementation of this technique is named Hybrid Network Traffic Engineering Software (HNTES). It uses a combination of data-plane actions for identifying flows for which circuits are appropriate along with control-plane actions to provision and release circuits. Unlike management-plane actions, the time scale of operation is much shorter. The goal is not to achieve optimal network topologies but rather to improve user performance metrics such as delays and jitter for mice flows without compromising throughput for elephant flows. The advantage of this approach over the application-triggered Lambdastation approach is that users do not need to modify their applications or run shell scripts that invoke circuit setup prior to application execution. The drawback however is that it is difficult to predict which flows are good candidates for circuits, which could be lead to under-utilization of circuits and/or less than satisfactory performance improvements. Section 2 identifies a set of functional requirements for the HNTES system, and Section 3 describes a high-level architecture for HNTES. Among the components identified, the Offline Flow Analysis Tool (OFAT) is a critical module since the success of this method depends upon the accuracy of flow-length predictions. Our current findings and work on the OFAT module is described in Section 4. An example deployment of HNTES within ESnet is used to illustrate various system procedures in Section 5. Finally, our plan for testing HNTES and evaluating its performance is described in Section 6. Source code for the HNTES modules will be made available to the community on our project web site 4. 1

2 2 HNTES requirements What is the purpose of HNTES, i.e., what are the functional requirements? The purpose of HNTES is traffic engineering, i.e., its goal is to improve usage efficiency of network resources while offering improved user performance. It should trigger circuit setup when IP datagrams of a set of predetermined flows arrive at a router, and cause subsequent packets of these flows to be rerouted to the newly established circuits. Circuit release should be initiated by this software when these flows are deemed completed. HNTES should support a user interface to allow human administrators or external software modules to provide information characterizing the flows suited for such redirection. Performance requirements of HNTES relate to the speed of execution of control-plane functions such as the addition of policy-based routes to reroute packets of certain flows to newly established circuits. These will be determined with experimentation. 3 HNTES architecture Figure 1: HNTES software architecture and interfaces Figure 1 shows the HNTES software architecture and its interfaces. It interfaces with the Inter-Domain Controller (IDC), which is the virtual circuit scheduler, and IP routers. It consists of several modules as described below: Offline Flow Analysis Tool (OFAT): This tool collects Netflow data from the routers, analyzes the data to find long flows, and populates the Monitored Flow Data Base (MFDB) with flow identifiers for long flows through the user-interface module. The circuit duration field is populated by OFAT. User-interface module: The purpose of this module is to allow two types of users, human administrators and software systems, such as the offline flow analysis tool, to enter information about flows that should be redirected to circuits whenever possible. We refer to these flows as monitored flows. This module will support a graphical interface for human users as well as a programmatic interface for software, such as the offline flow analysis tool. Information entered through this module is saved in the MFDB. All users should be authenticated. 2

3 Monitored-Flow Data Base (MFDB): The structure of this database (one per router) is as follows: Source IP address Flow identifiers Status Layer-2 Destination IP address Protocol Source port Destination port circuit endpoints Of monitored flow (not all fields are required for each flow) Monitored Redirected Disabled IP addresses and VLAN IDs Circuit duration Circuit rate Flow identifiers are subsets of the 5-tuple, source IP address, destination IP address, protocol, source port and destination port, which typically characterize flows. The Status field shows whether the flow is currently just being monitored or whether it has already been redirected to a circuit. It could be Disabled if the user who added this flow information decides to stop monitoring this flow. It is useful to store information for a disabled flow if there is a possibility that it will monitored again in the future. The Layer-2 circuit endpoints corresponding to each flow are computed by the initialization module. Finally, the circuit duration and rate desired for each flow should be precomputed and stored. Router-control interface module: This module is triggered by the user-interface module to configure the router (through its control port) to mirror monitored flows to a data-plane interface leading to the server on which the flow-monitoring module is executed. In other words, as packets are forwarded by the IP router to the normal data-plane interface as determined by its routing table, the packets from monitored flows are also mirrored to the flow-monitoring module. Flow-monitoring modules: One instance of this module is run per monitored router. When this module receives packets of a monitored flow, it continues monitoring for a certain preconfigured duration. If packets continue to arrive for this flow past this duration, it concludes that the flow is a long flow, and initiates the reservation and provisioning of a circuit between the Layer-2 circuit endpoints identified for that flow in the MFDB. It does this by sending a message to the IDC-interface module. For heavy flow volumes, a multiple node cluster implementation may be required for this module. These modules also update the MFDB entries based on observations of flow durations, and change the Status field of the flow. They initiate path termination by sending a message to the IDC interface module if the packets seen for a redirected flow start trickling down to a light rate. IDC-interface module: Upon receiving messages from the flow-monitoring modules, the IDCinterface module formulates IDCP SOAP messages 5 and requests the reservation and provisioning of circuits and release of circuits. In a circuit request, it passes the circuit endpoints (referred to as Layer-2 information in IDCP) as well as the flow identifier (subset of 5-tuple, source IP address, destination IP address, source port number, destination port number, IP protocol type), referred to as Layer-3 information in IDCP, to the IDC. The IDC configures policy based routes in the routers at the ends of circuits after circuit provisioning and release. Initialization module: This module computes the endpoints of Layer-2 circuits corresponding to each monitored flow. This information can be obtained from the IDC and/or routing information bases (RIBs) in routers. The initialization module also computes through some means an appropriate value for the circuit rate desired for each monitored flow and enters this information in the MFDB. 4 Offline flow analysis tool (OFAT) Several flow-classification techniques have been developed Most of them note that port-based classifications do not work because applications such as P2P do not use well-known port numbers. However, first, for our purpose, the goal of flow classification is just to identify long, fat flows not the 3

4 associated applications. Flows have to be long so that the overhead of circuit setup and router configuration is relatively low. Flows have to be fat, which means that the number of bytes sent within the duration has to be large enough to warrant the use of a dedicated circuit. Second, on ESnet, most of the long fat flows are generated by scientific applications run from servers with well-known IP addresses. Furthermore, scientific applications that generate long fat flows, such as GridFTP, have well-defined characteristics that can be exploited for flow identification. Therefore, we hypothesize that a generic scheme is not required for flow-length prediction. Instead, at least some of these flows can be characterized a priori by site administrators and/or automated tools. The feasibility of this approach rests on our ability to make such flow characterizations a priori. Through an analysis of Netflow data obtained from Internet2 routers, a few scientific applications have been identified as generating long flows. For example, consider the Unidata Local Data Manager (LDM) application 11. This is used by the Earth systems scientific community. Netflow data at the KANS router from five 5-min flow files showed the LDM flows listed in Table 1. Netflow is configured in Internet2 routers to report out on a flow that reaches 60sec in length. Each Netflow file consists of flow information exported from the router to a collector over a 5-minute interval. An examination of five consecutive 5-minute files showed multiple flow-export reports on the same flow as shown in Table 1. The source TCP port number is 388, which corresponds to the Unidata LDM application. A list of Unidata LDM Internet Data Distribution (IDD) servers is provided in 12. Of these, server idd.unidata.ucar.edu has an IP address As Internet2 anonymizes addresses by masking the last 11 bits, the /21 address corresponding to is , which is the source IP address in the flows listed in Table 1. Table 1: An example set of Unidata LDM flows srcip dstip srcport dstport protocol firstunix lastunix flowlength The above table has two columns called firstunix and lastunix. These are the unix seconds (number of seconds since standard epoch of 1/1/1970, 00:00:00) at which the first and last packets, respectively, of the flow were observed by the router. An examination of these two columns reveals that except for a few gaps, the router observed packets from this flow almost continuously. Netflow on Internet2 routers is configured to capture 1-in-100 packets. 4

5 Consider a second application, GridFTP. GridFTP uses a well-know port number, 2811, for its control TCP connection. Netflow data shows that this connection is short-lived, but the data connections, which do not use well-known port numbers are often long lived. An approach is to use packets sent on the control connection as a signal for an upcoming set of GridFTP data connections. As an example, an analysis of a 5-minute Netflow file collected at the CHIC Internet2 router (July 6, 2009) showed the following GridFTP control flow: srcip dstip srcport dstport protocol firstunix lastunix flowlength Next, we looked for long flows between the same two IP addresses soon after the time instant, , when the control flow packets were exchanged, and found the following (presumably, since these ports are unassigned) GridFTP data flows: srcip dstip srcport dstport protocol firstunix lastunix flowlength These GridFTP data flows were obtained by filtering out flows with flow lengths longer than the 95th percentile. The firstunix and lastunix columns indicate that a number of flows occurred in parallel. Due to address anonymization, it is unclear whether multiple hosts within a cluster were involved at both ends or whether they were single nodes. Recall that Internet2 Netflow is configured to report flows every 60 seconds, and thus the 5% flows have durations in the range shown. Many of these flows could be much longer, but as this data is extracted from a single 5-minute file, the flow concatenation step is not yet executed. In a striped GridFTP implementation, the server protocol interpreter (PI), which terminates the control connection could be executed on the head node of a cluster, while the Data Transfer Processes (DTP) are executed on other nodes of the cluster 13. This implies that packets whose source and destination IP addresses match a set of addresses associated with the control-connection addresses should be redirected to the circuit since the data TCP connections are the ones likely to be long. As soon as GridFTP control flow packets are detected, a circuit can be provisioned and the router configured to reroute packets whose IP addresses correspond to those in the associated set so that even data flow packets are rerouted. In another experiment, we wrote an R program to concatenate flows from consecutive 5-minute files to determine the actual length of flows. Since a report is exported for a flow every 60 secs, to determine the true length of long flows, several 5-minute Netflow files need to be analyzed. A whole day s Netflow (5-min) files were downloaded for the CHIC router. Only flows with lengths greater than or equal to 59 seconds were retained for this analysis. Each flow record reported by Netflow shows the time instant at which the first packet of a flow entered the router and the time instant at which the last packet entered. Since Netflow does 1-in-100 packet sampling, there can be gaps in the inter-packet arrival times for 5

6 packets counted within this record. Ignoring these gaps, we estimate flow length by subtracting firstunix from lastunix. Now when the same flow is identified in several consecutive 5-min files, the question is how to treat gaps. For example, consider Table 1. There is a gap of 181 seconds between the second and first flow records, but the immediately following few records show no gaps. Later, we see a 120 sec gap. These gaps may arise because of the 1-in-100 sampling issue, or possibly due to genuine gaps in packet emission by the source. Assuming circuit setup delays are 1 min, we arbitrarily decided to ignore gaps smaller than 5 mins for this analysis. With this assumption, our analysis of this one-day set of flows showed a total of distinct long flows. The longest flow duration was 5 hours. The fattest flow, i.e., the flow in which the most number of bytes were transferred, consisted of a transfer of bytes. The duration of this flow was sec, or 3.17 hours. The top ten fat flows are listed below. bytes srcip dstip srcport dstport protocol firstunix lastunix flow length Of these top ten flows, three are encapsulated (50: ESP; 47: GRE), five are ssh (port 22), one is Unidata LDM (port 388) and one is rsync (port 873). There are two long ssh flows to a University of Texas at Austin Texas Advanced Computing Center server ( ) from a Fermilab ( ) host. The address corresponds to NCSA (National Center for Supercomputing Applications) for two other ssh flows. The Unidata LDM flow is from NCAR (National Center for Atmospheric Research) with address The offline analysis tool can implement this type of flow length and flow width analysis algorithms and execute them on a daily basis. Some subset of the 5-tuple can be used per flow to configure the database for monitoring (described in Section 3). Our next step is to correlate long-flow information obtained for consecutive days. More broadly, flow analysis will be carried out to identify temporal patterns with which long fat flows arrive and terminate so that circuits can be reserved on a periodic basis in anticipation of specific long fat flows. If such predictions can be made for a significant fraction of long fat flows, it becomes less important to detect the start of the flows by live packet monitoring 14. 6

7 5 An illustrative example of HNTES deployment Figure 2: Illustration of HNTES procedures Figure 2 shows an illustrative example of how HNTES may be deployed by ESnet. Two ESnet site networks, SLAC and FNAL, are used as examples. Sites networks are shown to connect into the core network through the circuit switches. IP-access links are provisioned as static circuits from Provider Edge (PE) routers located at the sites through core circuit switches to core IP routers (shown with dashed lines in Figure 2). An OSCARS IDC is shown in the core network for handling dynamic virtual circuit scheduling, provisioning, and release. HNTES flow monitoring modules are shown to be deployed at the sites for monitoring long flows through ESnet-owned Provider Edge (PE) routers. A data-plane link connects the servers running the flow monitoring modules to their corresponding PE routers. The other HNTES modules can be deployed in a centralized manner anywhere within ESnet. Servers running the HNTES modules, including the flow monitoring modules, are connected to ESnet s IP-routed network. Several procedures will be executed by HNTES systems, a few of which are explained below. Monitored-flow configuration: When a user enters the description of one or more flows for monitoring and rerouting to circuits, HNTES updates its MFDB through the user-interface module. Through the router-control interface module, HNTES configures the PE routers to mirror packets matching the monitored flow identifiers in the MFDB to the flow-monitoring module servers. This update is done through the control port of the routers. Software developed for an NSF-funded project called CHEETAH to interface with a Force10 E600 system 15 will be reused in the router-control interface module. Layer-2 circuit endpoints identification: When information for a monitored flow is entered into the MFDB via the user-interface module, the latter notifies the initialization module. By obtaining routing data, either from the routers or from the IDC, the initialization module determines the Layer-2 circuit endpoints corresponding to each monitored flow. It also estimates circuit rates. Detected-flow actions: When a flow-monitoring module receives packets corresponding to a monitored flow, it will initiate the reservation and provisioning of a circuit by communicating with its corresponding IDC-interface module. If circuit setup is successful, OSCARS IDC will configure policy based routes in the PE routers to redirect packets from this flow to the newly established virtual circuit. 7

8 The flow-monitoring module changes the status of the monitored flow in the MFDB to redirected so that any administrative queries about flow states can be responded to correctly. Terminated-flow actions: HNTES continues to monitor redirected flows. If the rate of packet reception for a redirected flow is determined to be low, then the flow monitoring module will signal the IDC interface module asking it to initiate circuit termination. The IDC will release the circuit and initiate a reconfiguration of the policy-based route in its router for that flow. The flow-monitoring module updates the status of the flow in the MFDB, and if the time between circuit termination and the circuit reservation end time is greater than some predetermined value, it will send a message to the IDC-interface module to have it generate a cancel-reservation IDCP message. 6 HNTES testing and performance evaluation The HNTES system will be tested on the Tabletop testbed/long Island MAN 16 provided by ESnet. The time taken to complete various tasks will be measured carefully. Flow lengths could be on the order of a few minutes. If the time taken to set up a circuit and configure the routers is itself on the order of minutes, then flows with total durations on the same scale should not be candidates for redirection. Therefore, every attempt will be made to keep these configuration times short. An important question is to study how changes in the path latency affect TCP performance of redirected flows. Will changes in latency cause retransmission timeouts to vary wildly, causing either premature retransmissions or excessive delay? References 1 M. Veeraraghavan and Z. Yan, HYbrid Network Engineering Software (HYNES), Dec. 30, The Lambda Station Project. [Online]. Available: 3 On-Demand Secure Circuits and Advance Reservation System (OSCARS), 4 UVA Hybrid Networking Project, 5 Inter-domain Controller (IDC) Protocol Specification, Version 1.1, February 9, Laurent Bernaille, Renata Teixeira, Ismael Akodkenou, Augustin Soule, and Kave Salamatian. Traffic classification on the fly. ACM SIGCOMM Comput. Commun. Rev., 36(2):23 26, Andrew W. Moore and Denis Zuev. Internet traffic classification using Bayesian analysis techniques. In SIGMETRICS 05: Proceedings of the 2005 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, pages 50 60, New York, NY, USA, ACM. 8 T. Auld, A.W. Moore, and S.F. Gull. Bayesian neural networks for internet traffic classification. Neural Networks, IEEE Transactions on, 18(1): , Jan J. Erman, A. Mahanti, and M. Arlitt. Qrp05-4: Internet traffic identification using machine learning. In Global Telecommunications Conference, GLOBECOM 06. IEEE, pages 1 6, Dec Thomas Karagiannis, Konstantina Papagiannaki, and Michalis Faloutsos. Blinc: multilevel traffic classification in the dark. In SIGCOMM 05: Proceedings of the 2005 conference on Applications, technologies, architectures, and protocols for computer communications, pages , New York, NY, USA, ACM. 11 Unidata Local Data Manager, 12 Unidata IDD site contact list, 8

9 13 William Allcock, John Bresnahan, Rajkumar Kettimuthu, Michael Link, Catalin Dumitrescu, Ioan Raicu, Ian Foster, The Globus Striped GridFTP Framework and Server, Proceedings of the 2005 ACM/IEEE SC 05 Conference (SC 05) paper Conversation with Brian Tierney, Dec. 18, M. E. McGinley, T. Li, M. Veeraraghavan, On Virtualizing Ethernet Switches, Proc. of IEEE ICCCN 2008, Aug. 3-7, St. Tomas, US Virgin Islands. 16 B. Tierney, The ARRA ANI Network Testbed Project, Feb 2, 2010, 9