Title: Collaborative research: End-to-End Provisioned Optical Network Testbed for Large-Scale escience Applications

Transcription

1 Year 2Activities report for the NSF project EIN Title: Collaborative research: End-to-End Provisioned Optical Network Testbed for Large-Scale escience Applications Date: July 15, 2005 PI: Malathi Veeraraghavan, Please reiterate the goals and objectives of your efforts, and summarize the research and education activities you have engaged in that aim to achieve these objectives. Include experiments you have conducted, the simulations you have run, the collecting you have done, the observations you have made, the materials you have developed, and major presentations you have made about your efforts. In a later section you will list more formally any publications and other specific products (database, collections, software, inventions, etc.) that have resulted. The goal of this project is to develop the infrastructure and networking technologies to support a broad class of escience projects and specifically the Terascale Supernova Initiative. Our objectives are to design and deploy a high-performance, experimental optical network infrastructure and to test application/ middleware/transport protocol software, developed specifically for escience projects, on this network. Our two target applications are file transfers and remote visualization. The objective of our research activities is to determine whether call-by-call dynamic sharing of bandwidth resources can be used for data applications. Specifically, can such sharing reduce the costs of highspeed circuits, thus increasing their availability for escience applications such as the TSI project. In addition, we are designing and implementing transport protocols that handle the mismatch between the characteristics of multi-tasking general-purpose end hosts with off-the-shelf disks with the characteristics of circuits. To achieve high circuit utilization, the sending end host should be programmed to send data at a monotonic rate equal to that of the circuit, and the receiving end host should be programmed to move data from kernel buffers to the disk at that same rate. The objective of the educational activities is to prepare a new generation of engineers who are knowledgeable about the new technological advents in the general area of optical networks and specifically in how to leverage the high-speed capabilities of these networks to architect and design new applications. In addition, our students have the opportunity to acquire inter-disciplinary skills spanning operating systems, distributed systems, parallel computing and networking.

2 Below is a summary list of all our activities in this project to date (Aug July 2005), which includes experiments we have conducted, and some observations we have made (more details on our observations are listed in the findings attachment): CHEETAH wide-area network (primarily work of postdoctoral fellow, Xuan Zheng, with support from graduate student, Xiangfei Zhu): We completed design of the network, negotiated with vendors, tested equipment, purchased and installed equipment, and finally interconnected network nodes with circuits. Activities included testing switches, ordering equipment (including SONET-based MSPPs, high-end computers, Ethernet switches, Netscreen security devices, GbE network interface cards, console servers and power distribution units (for remote control), fiber cables and other minor associated pieces of equipment), negotiating with four service providers to purchase various wide-area optical circuits, finalizing order agreements and purchasing these circuits, negotiating and purchasing collocation services to locate our switch equipment in two PoPs (RTP area in NC and Atlanta). We now have a running highspeed circuit-switched network in place as shown in Fig. 1. [We request the reader to view Fig. Figure 1 CHEETAH network as of July 7, on a computer screen rather than on a black-and-white printout for two reasons: (a) color and (b) ability to enlarge the figure.] We have deployed three Sycamore SN16000 switches as shown in Fig. 1, using collocation agreements with MCNC and SLR/SOX (space and power).

3 Each SN16000 has multiple GbE ports (the ORNL switch additionally has a 10GbE port), and multiple OC192 ports. These nodes run GMPLS signaling and routing protocol software, version 7.6. We also purchased and located computers (wukong in MCNC) and zelda1, zelda2, zelda3 (at SOX), and zelda4 and zelda5 at ORNL. These computers run Redhat Linux version 2.6 and have two GbE NICs each, one connected to the Internet (primary) and the second connected to the CHEETAH network. We purchased Internet access as part of our collocation service agreements from NCNI (MCNC) and SLR/SOX. The wide-area OC192 circuits were purchased from (1) MCNC (from MCNC in the Research Triangle Park to NLR PoP in Raleigh) via Cisco MSTPs, (2) NLR (for the Raleigh-Atlanta segment) via Cisco 15808s, (3) SLR (for intra-atlanta segment from the NLR PoP to the SoX PoP) via Movaz Ray Express equipment, and (4) ORNL (for the SOX-ORNL segment) via Ciena Corestream equipment. Non-recurring costs include the purchase of OC192 transponders for all four segments. The first three are functional and in use; only the last segment still needs to be activated. In its place we are using 1Gb/s MPLS tunnels as circuits (up to 3 Gbps) on an existing OC192 circuit extending between two Juniper M320 routers. The circuits from NCSU (Centaur Lab), where our co-pi John Blondin s compute cluster, orbitty, is located, to MCNC, are realized using five 1Gbps VLANs. A 10GbE circuit was already in place between NCSU s Centaur Lab and NCSU s CMDF building, and between the CMDF building and MCNC. These five VLANs were provisioned through cooperation between NCSU and MCNC. The Foundry FESX448 and the two Cisco 7600 Ethernet switches shown in Fig. 1 needed to be manually configured for the VLANs. We have successfully tested communication between various pairs of computers across various segments of our network, e.g., orbitty to wukong, orbitty to zelda1/2/3, orbitty to zelda 4/5. Application support for TSI (primarily work of graduate student, Xiuduan Fang): We downloaded and tested PVFSv2 (Parallel Virtual File System) and GridFTP on a local cluster in our CS department. We found several problems in adapting these to the CHEETAH network and modified the source code accordingly. The two main problems were (1) PVFSv2 tools did not actually support the -s option that was available in the API to set the stripe size, and (2) there was no way to specify the stripe order and control the physical location of stripes. In the CHEETAH concept, we set up dedicated circuits host-to-host. Thus if there are two clusters orbitty and zelda, and circuits are set up between orbitty1 and zelda1, orbitty2 and zelda2 and so on, then we wanted to control how PVFS striped a file on zelda (located in ORNL) so that the stripe on zelda1 was transferred to orbitty1, and so on. Using strace, Xiuduan determined

4 what parts of the PVFS code needed modification and did so. The GridFTP striped transfer mode is used to enable multiple nodes to work together on a single file and act as a single GridFTP server. It relies on an underlying parallel file system (e.g., PVFS2) to allow all nodes to see the same file system. It uses third-party transfer, in which a controller running on an external end host initiates a transfer from one cluster to another cluster. The GridFTP protocol extends the FTP PASV and PORT commands to SPAS and SPOR commands respectively, setting the receiving cluster to operate in passive mode, and obtaining the port numbers on which this cluster s servers are listening, and providing these port numbers to the sending cluster allowing the latter s servers to initiate data connections to the receiving servers. In the current GridFTP code, the process of assigning the port numbers returned in response to the SPAS command to the sending cluster is non-deterministic and in each run a different pair of hosts could be required to connect. This makes the current implementation unsuitable for CHEE- TAH unless the SPAS and SPOR code is modified. We have currently suspended this activity for two reasons. First, our TSI project co-pi, John Blondin, clarified that while the TSI simulations generate TB-sized datasets, these datasets are created in the form of several files, each on the order of gigabytes. Since each host on his orbitty cluster has three 146GB disks, different files can be stored on different compute nodes of orbitty. In other words, the TSI project does not necessarily need to store single large files striped across the local disks of several computers, which is the primary reason for needing PVFS. In general, even if a file can fit into the disk of one host, there are advantages to striping it across the disks of multiple end hosts and then use striping to achieve parallelism in the transfer. However, in the TSI project since there are multiple GB-sized files, we exploit parallelism across the multiple files generated in a simulation run rather than within a single file transfer. Second, we extended the CHEETAH solution to allow the concept of a cluster-to-cluster dedicated circuit. Since cluster computers nowadays increasingly use GbE switches, we propose to connect one port of a cluster GbE switch to the CHEETAH network and other ports to the second NICs on the cluster computers. By plugging the switch port into the CHEETAH Sycamore switch, we can create Ethernet switch-to-ethernet switch connections. This would allow Ethernet frames destined to any single cluster node at one end from any node on the far-end cluster to reach successfully without requiring pairwise dedicated circuits. We are now experimenting with performance results from FTP over tuned TCP and FRTP, as well as bbcp and SecureFTP applications. After a dedicated circuit is set up, there is almost no possibility of packet loss with the use of TCP because the only buffer involved is the receiver s buffer, and TCP s window based flow con-

5 trol adequately warns the sender. With GB sized files, the loss of throughput in the initial Slow Start is marginal. General-use applications (primarily work of graduate student, Xiuduan Fang): We used the Common Gateway Interface (CGI) mechanism to integrate CHEETAH software into the web browsing application to trigger the automatic set up of a circuit and the use of it for downloading the requested web page. This was demonstrated at SuperComputing Signaling software (primarily work of graduate student, Xiangfei Zhu): We designed and implemented an RSVP-TE client for Linux hosts starting with RSVP-TE Kom/Dragon code. We tested interoperability of this client with Sycamore s SN16000 RSVP-TE implementation. This test was a success. We found a few bugs in our code and fixed them. The availability of RSVP-TE decoding in the sniffer tool, Ethereal, greatly helped us in this task. Sycamore s implementation of the GMPLS protocol suite is quite good. However, a significant part of our activities were used to identify the boundaries of their implementation and to work within these boundaries. For example, given there is no standard yet for signaling for heterogeneous connections, and in the CHEETAH network we necessarily have heterogeneous connections (Ethernet to hosts and SONET between switches), we needed GMPLS support for heterogeneous connections. Sycamore developers were highly cooperative and provided us a proprietary solution for this problem, with a commitment to evolve this to a standards based solution once that is defined. Other features we found lacking in the Sycamore implementation are as follows: we cannot define multiple TE links per port with corresponding capacity levels (split port capacity among TE links), specify a subnet mask for remote IP addresses when configuring TE-links, and cannot specify multiple IP addresses per TE-link. This is because the Sycamore software was developed for a transit node, not for an ingress/egress node, i.e., nodes on which the circuit terminates. These limitations do pose constraints, but we have designed our network and software within these constraints. We also integrated the Dynamic TL1 library purchased from Monfox with RSVP-TE software to control the Cisco externally. We demonstrated the setup/release of circuits through the at SuperComputing We borrowed high-speed GbE cards from MCNC/Cisco and later purchased an ML GbE card for our laboratory testbed. Transport protocol (primarily work of graduate student, Anant P. Mudambi): The main problem we tackled this year was to have the sending end of the transport protocol implementation maintain a constant sending rate to match the rate of the circuit. This is important to achieve high circuit utilization. We started off with the FRTP1.0 (Fixed Rate Transport Protocol) code,

6 which was modified from a SABUL implementation. But this implementation used a busywait mechanism at the sender to maintain the required inter-packet gap. This not only made it a CPU-intensive implementation, but also made the sender s rate very sensitive to any interruptions (e.g., if another process becomes scheduled to run or the sending process blocks in some system call like while reading from disk). We tried a second approach to this problem of maintaining a constant sending rate by using a larger time frame; in other words, have the sender send a fixed number of packets every T time units. We tried this by modifying the FRTP1.0 code to send a fixed number of packets and then sleep for some time. The packets are then sent in a burst. To reduce burstiness, we would like T to be as small as possible. In Linux 2.4, we found that the minimum value for T is 10ms, but none of the system calls related to sleep were accurate at such small time intervals. What we needed was a way for the user-space sending process to get a reliable interrupt every T time units. Linux offers a way for a program to set a timer and have the OS send a signal to the process whenever the timer expires. We used this mechanism to implement a more accurate way for the sending process to be invoked periodically. This worked fairly well, except the burst sending of packets caused a problem if the SONET wide-area circuit rate was lower than the end host NIC rate of 1Gb/s. In the CHEETAH solution, we proposed such a circuit to handle the issue that receive disk rates are more commonly in the 400Mbps range, and hence wanted to limit the bandwidth wasted in the wide-area circuit. Luckily, both the Cisco MSPP and Sycamore s SN16000 Ethernet cards support the GbE PAUSE feature if the GbE port is mapped to a lower rate SONET circuit. This PAUSE signal, sent by the GbE card in the MSPP stops the sender s Ethernet driver from sending frames. While this solution appears to be perhaps the best choice for a user-space based FRTP implementation, it is not very accurate in maintaining a constant rate if the sender is multi-tasking. If the file transfer sending process is not running when the timer signal expires, the process will not be invoked immediately and hence it will miss being strictly monotonic. Another problem we identified with a user-space implementation is that windowbased flow control is difficult to implement because the size of the UDP buffer, held in the kernel, is the one that needed reporting back to the sender and this size is difficult to obtain in time before the receive window overflows. Therefore, we spent the second half of the year on a kernel based implementation of FRTP using Web100. The goal was to reuse window control implemented as part of the TCP kernel implementation. To remove TCP s congestion control (given this is detrimental when using a circuit), we downloaded Net100/Web100, which provides a way to tune the AIMD parameters of TCP. Web100 does not provide a way to set the

7 AI and MD parameters to 0, which is what we require to be able to maintain a steady congestion window. We added code to the Linux kernel TCP stack to be able to do this. The shift to a kernel-based implementation reusing TCP and Web100 code provides another important advantage: TCP s self-clocking provides a way to maintain a steady sending rate. Once a congestion window s worth of data is sent into the network, the sender sends the next packet only when a returning ACK slides the window to the right. The ACKs come back at the bottleneck link rate, which in our case, is the circuit rate, and so the sending rate equals the circuit rate. We thus have two new implementations of FRTP, one user-space UDP based, and a second kernel-space TCP based. We are currently measuring the performance of these implementations and integrating them with applications. Routing decision software (primarily work of graduate student, Zhanxiang Huang): The goal of the routing decision module is to decide whether a transfer should attempt the setup of a circuit or directly use the Internet path. The models for computing the cost of circuit setup and the TCP/IP connectionless path from the delay and utilization aspect have already been described in the CHEETAH Opticomm 2003 paper and Dr. Xuan Zheng s dissertation. This year we focussed on designing and implementing the routing decision module. We defined the requirements for the routing decision module as follows. It should handle multiple queries from application clients simultaneously. It should maintain a cache of frequently queried endpoints to reduce query overhead. The design is based on multithreading and synchronization. We specified the interfaces between sub-modules, and identified the parameters that need to be measured and also selected appropriate measurement tools after experimenting with several tools, such as pchar, pathload, pipechar, iperf, etc. We designed the cache structure, and implemented the delay model for TCP and circuit setup/transfer. The parameters we measure include round trip time (RTT) on both the connectionless and circuit path, the packet loss rate on the connectionless path, and TCP client side maximum window size. Analysis of remote visualization application traffic (primarily work of graduate student, Zhanxiang Huang): The goal of monitoring the traffic of the TSI visualization application, Ensight, is to help us determine what type of circuit should be set up for remote visualization. The monitoring environment is as follows. Professor Blondin runs Ensight client on his office desktop to connect to the Ensight server running on a powerful remote cluster called orbitty. We placed a fiber splitter between the orbitty cluster and its outgoing switch and the splitter forwarded all the traffic from and to orbitty to another host named cheetah.ncren.net, on which we captured and analyzed the traffic. We used tcpdump as the capturing tool, and with its fil-

8 tering function, we filtered out other traffic irrelevant to the Ensight application. Since we already knew that the Ensight server sends one geometry structure data at a time, the main goal of the analysis was to find out the size of a single geometry structure. We first used Ethereal s post-processing tool to separate the server-to-client and client-to-server traffic, and then used the statistical tool provided in Ethereal to determine the peak burst rate. However, Ethereal could not directly provide us the size of a typical geometry structure. Thus we had to write our own post-processing tool to analyze it. Our key assumption was that one geometry structure is sent by the server application to the kernel TCP buffer at once, and expected this to be reflected in the traffic as a sequence of contiguous full-mtu length packets. Further, we expected control messages to be sent in smaller packets interleaved among the sequence of packets carrying the geometry data. Thus the first effort of our post-processing program was to find the sequence of contiguous full-mtu length packets. But it turned out that our assumption that the Ensight server application sends a whole geometry structure to the TCP buffer was wrong. In fact it breaks this up into smaller blocks. Using this assumption, we induced there should be at most one packet whose size is less than the full MTU length in between blocks, which is highly consistent with our observation of the captured trace file. Furthermore, we used the tool mergecap, to synchronize the server-to-client and client-to-server traces and then plotted the TCP congestion window size against time to check the window size when the smaller packets occur with the tool tcptrace and xplot. This confirmed that the smaller packets are not caused by the change of TCP congestion window size. Thus we modified our own post-processing tool to find the contiguous sequence of full-mtu length packets with at most one smaller sized packet in between. The analysis results showed us that the maximum geometry structure data we observed is around 3.216MB and the time it took to transfer this geometry is around 0.33 second. The interval in between sending geometries from server to client depends on the frequency of the user request from the client. Thus our preliminary conclusion is that remote visualization does not require a high-bandwidth circuit, and if we can predict the user request model, we can decide when to set up and when to tear down a circuit. Project management: We coordinated activities between all four participating organizations. We supported the CUNY team in their development of VLSR (with design and OSPF testing), and security solution for CHEETAH. We worked with the ORNL and NCSU teams to facilitate the TSI application support for file transfers and remote visualization. Education activities include the following. Xiangfei Zhu obtained his Masters degree in May He is continuing on with support of this project for a Ph.D. Two undergraduate students, Andrew Love and

9 Yan Peng (Roger) Guo, did their undergraduate senior thesis projects in our laboratory. Students have taken full advantage of the opportunities provided in realizing this wide-area testbed and learning the use of all the equipment we have procured. Significant learning about network security devices was an important aspect of our educational growth this year. Materials: The materials we developed include papers and software. The publications are listed in the Products part of this report. We published two journal papers, one conference paper and one workshop paper. Software developed in the project along with presentations are being posted on our project web site: cheetah.cs.virginia.edu/. Presentations (the first by Xuan Zheng, and the rest by Malathi Veeraraghavan): Title of Presentation Meeting Place Date End-To-End Provisioned Optical Network Testbed for Large-Scale escience Applications CHEETAH CHEETAH Joint Engineering Team Review of Optical Networking Testbeds, NSF, Washington, DC NSF Sponsored Planning Workshop on The Future of Optical Communications: Understanding the Choices Virginia Tech High-Performance Computing Conference Blacksburg Washington, DC Santa Barbara May 26-27, 2005 April 19, 2005 April 12 & 13, 2005 Enabling a connection-oriented internet Georgia Tech Atlanta March 30, 2005 Enabling a connection-oriented internet Duke University Durham, Feb. 11, 2005 NC CHEETAH Exhibition SuperComputing 2005 Pittsburgh Nov. 6-12, 2004 Immediate-request vs. scheduled calls, Short-duration vs. long-duration calls Served as panelist MCNC Workshop on GMPLS Control Plane Optical Networks and Grid Computing Panel, Broadnets, 2004 Pittsburgh November 12, San Jose Oct , 2004