No. 200235 March 2001 StreamCore StreamGroomer Module 200 Flow Regulator and StreamGroomer Manager (SGM) TCP/IP WAN Access Link Flow Regulation System Performance Evaluation Premise: Network professionals who need to ensure acceptable response times for mission-critical traffic across a Wide Area Network (WAN) can optimize the performance of their network by deploying flow regulation devices to provide Quality of Service (QoS). Such solutions can allocate bandwidth to user-defined traffic types to provide users of critical applications acceptable and consistent response time even when the WAN backbone is congested. StreamCore S.A. commissioned The Tolly Group to evaluate its StreamGroomer Module 200 (M200) Flow Regulator along with its StreamGroomer Manager (SGM), a WAN traffic shaping and optimizing system. Tests were conducted to identify the improvement rate of transactions per second (tps) when introducing the StreamGroomer system into a simulated WAN consisting of links with varying degrees of introduced latency. For these tests, The Tolly Group configured a WAN network with varying links, speeds and latencies and measured the baseline performance of the WAN in this uncontrolled environment. Following baseline tests, Tolly engineers measured the transactions across the WAN when the StreamGroomer was added and configured to give priority to interactive transactions. To deliver Quality of Service (QoS), the StreamGroomer builds a set of software "tunnels" between the various endpoints and relays the TCP/IP traffic inside these tunnels. To prioritize, the StreamGroomer system relays the TCP/IP stream of octets T H E TOLLY G R O U P Test Highlights Test Summary Improves throughput by up to 10 transactions per second in tests across a 2 Mbit/s link with 10 millisecond WAN latency Precisely manages relative bandwidths of a variety of traffic types Provides consistent response time by alleviating TCP's bandwidth randomness on fully utilized links Effectively regulates both directions of access link flows Transactions per second Bandwidth Management Across WAN Access Links (256 Kbit/s, 2 Mbit/s) with Varying Degrees of Introduced Latency: Chariot SAP R/3 Critical Traffic 12 10 8 6 4 2 0 0.37 2001 The Tolly Group Page 1 2.38 from remote server 10 ms delay (site C) 0.32 from remote server 100 ms delay (site D) Inbound traffic 256 Kbit/s link (site A) Without StreamGroomer 1.3 1.13 to remote client 10 ms 10 ms delay delay (site (site C) C) 10.56 With StreamGroomer 0.79 2.21 to remote client 100 ms delay (site D) Outbound traffic 2 Mbit/s link (site A) Source: The Tolly Group, March 2001 Figure 1
into an internal protocol with credit-based flow control. This protocol then creates a session between devices that can be managed by the StreamGroomer Manager software. Tolly engineers measured the StreamGroomer's capabilityto stabilize the response time when sending successive files across a simulated WAN. Testing was performed in October 2000. Results show that the StreamGroomer flow regulation system successfully manages the bandwidth to prevent batch traffic from monopolizing the network; and allocates this bandwidth in conformance with a user-specified ratio. Results also show that the StreamGroomer can stabilize response times. Results Bandwidth Management (SAP R/3 Critical Traffic) The Tolly Group engineers measured the average tps of a complex set of both batch and interactive traffic in a WAN comprised of both 256 Kbit/s and 2 Mbit/s links. The same test was repeated with the StreamGroomer system incorporated into the WAN to control bandwidth with Weighted Fair Queuing configured to give priority to the interactive traffic. In both tests, the simulated WAN was configured to mimic a "far" remote site by introducing 100 millisecond (ms) latency and to mimic a "near" remote site by introducing 10 ms latency. the unmanaged WAN across a 256 Kbit/s segment from the simulated "near" remote site with 10 ms latency, throughput of critical SAP R/3 traffic was only 0.37 tps. With the StreamGroomer system in place to manage traffic, the throughput of 2.38 tps. Across the 256 Kbit/s segment of the unmanaged WAN from the simulated "far" remote site with 100 ms latency, throughput of SAP R/3 Bandwidth Management Across WAN Access Link (2 Mbit/s) with Varying Degrees of Introduced Latency: Chariot HTTP Critical Traffic Throughput, Kbit/s 140 120 100 80 60 40 108 104 Critical 77 critical traffic was measured at an average of 0.32 tps. With the StreamGroomer system in place to manage traffic, the throughput of 1.3 tps. the unmanaged WAN across a 2 Mbit/s segment to the simulated "near" remote site with 10 ms latency, throughput of critical SAP R/3 traffic was only 1.13 tps. With the StreamGroomer system in place to manage traffic, the throughput of 10.56 tps. Across the 2 Mbit/s segment of the unmanaged WAN to the simulated "far" remote site with 100 ms latency, throughput of SAP R/3 critical traffic was measured at an average of 0.79 tps. With the StreamGroomer system in place to manage traffic, the throughput of 2.21 tps. See figure 1. Bandwidth management (HTTP Critical Traffic) an unmanaged WAN across a 2 Mbit/s segment with 10 ms latency introduced, throughput of HTTP graphical traffic deemed "critical" consisting of a series of 63-Kbyte files, averaged 108 Kbit/s and HTTP text traffic deemed "critical" of 55- Kbyte files averaged 104 Kbit/s. FTP traffic of 100-Kbyte files averaged 132 Kbit/s and SMTP traffic of 32- Kbyte files averaged 56 Kbit/s. When Tolly engineers added the StreamGroomer system to the WAN to prioritize both the HTTP graphical and text traffic, thus ensuring a throughput ratio of 4 between "critical" and "non-critical" traffic, results demonstrate that the HTTP graphical traffic of 63-Kbyte files now averaged 140 Kbit/s of throughput and the HTTP text traffic of 55-Kbyte files averaged 137 Kbit/s of throughput. The non-critical FTP traffic of 100-Kbyte 2001 The Tolly Group Page 2 74 132 56 Non- Critical 82 40 20 B-C B-D B-C B-D B-C B-D B-C B-D 0 10 ms* 100 ms** 10ms* 100 ms** 10 ms* 100 ms** 10 ms* 100 ms** Without StreamGroomer SMTP (32 Kbytes) FTP (100 Kbytes) HTTP graphical object (63 Kbytes) HTTP text (55 Kbytes) 140 137 125 Critical 121 Non-Critical 37 37 37 With StreamGroomer *10 millisecond delay **100 millisecond delay Source: The Tolly Group, March 2001 Figure 2 30
files and SMTP traffic of 32-Kbyte files both decreased to an average of 37 Kbit/s of throughput. an unmanaged WAN across a segment with 100 ms latency introduced, throughput of HTTP graphical traffic of 63-Kbyte files averaged 77 Kbit/s and HTTP text traffic of 55-Kbyte files averaged 74 Kbit/s. FTP traffic of 100-Kbytes files averaged 82 Kbit/s and SMTP traffic of 32-Kbyte files averaged 40 Kbit/s. When Tolly engineers added the StreamGroomer system to the WAN to prioritize both the HTTP graphical and text traffic (with the same ratio of 4), results demonstrate that the HTTP graphical traffic of 63-Kbyte files now averaged 125 Kbit/s of throughput and the HTTP text traffic of 55-Kbyte files averaged 121 Kbit/s of throughput. The noncritical FTP traffic of 100-Kbyte files and SMTP traffic of 32-Kbyte files both decreased to an average of 37 Kbit/s and 30 Kbit/s of throughput respectively. See figure 2. Variability in Response Time Engineers tested the average response time of the WAN by sending successive 100-Kbyte files. Results demonstrate that when using the StreamGroomer system, the average response time had little variability. In baseline tests, the average response time had greater variability. See figure 3. Analysis Quality of Service (QoS) is defined as the capacity of an infrastructure to deliver a specific service for particular traffic. QoS mechanisms allow network architects to determine the absolute or relative amount of bandwidth allocated to a particular type of traffic. The StreamGroomer system relays the TCP/IP data stream for user equipment into an internal protocol with credit-based flow control. This protocol essentially creates a tunnel between the StreamGroomer devices across which traffic can then be managed. The network administrator configures the StreamGroomer by defining the types of traffic by its TCP port number, and then assigning a priority to the traffic by defining the relative weights for each of the connections. Each type of traffic receives a predefined portion of the tunnel based upon its relative weight. This prevents any one stream from monopolizing the entire bandwidth, causing stream starvation. Bandwidth allocated but unused can be absorbed by other streams, thereby allowing for full-bandwidth utilization. Results in this set of tests demonstrates that on fully used access links, effective bandwidth usage of locally constrained application flows are proportional to their assigned "relative weights," in both directions of the link, so that performance of critical applications can be tailored to fit the needs of the user. StreamGroomers can help eliminate the randomness of effective data rates, thus making performance guarantees possible. This is due to the control of the characteristics inherent in the technique of Weighted Fair Queuing (WFQ). WFQ protects low-volume traffic streams from high-volume traffic streams thereby providing equity for all data flows. This results in more consistent data flows, since the higher volume streams can no longer monopolize the bandwidth and effectively "starve" the low volume streams. With StreamGroomers, enforcement of local policy, concerning bandwidth management of applications, is largely insensitive to unequal backbone delays that affect different flows. The Tolly Group manipulated the parameters of the StreamGroomer test environment to introduce delays of 10 ms and 100 ms, and demonstrated that these parameters had little effect on the results of the StreamGroomers flow regulation capabilities. StreamCore StreamGroomer Flow Regulator Performance Evaluation StreamCore StreamGroomer Module 200 (M200) Flow Regulator and StreamGroomer Manager (SGM) Product Specifications* Total Bandwidth Management on Intranet and Internet Access Links For interactive applications, delay minimization For volume transfers with flexible data rates, effective bandwidths of application connections are precisely proportional to their assigned weights (1 to 1024) Automatic optimization of applications that alternate volume transfers and interactions For streaming applications, guaranteed bandwidths and minimized jitter For fluctuating characteristic links (e.g. frame relay), protection against packet discards Comprehensive Application Classes Class identifications based on parameter values of TCP and UDP, including port numbers, IP precedence, address ranges Proxy option for FTP, HTTP, HTTP-S Explicit support of Real Audio, Real Video, Windows Media Player, and other Streaming Applications Powerful Link Traffic monitoring Full history of link utilization, and of the ratio between actual bandwidths and assigned parameters Application class statistics in bytes, packets, and connections per second Current connections reporting Configurations for ISP access links, private Intranets and MPLS based VPNs A CPE-M200 (Customer Private Equipment) can be attached to a LAN, or placed between a LAN and its access router, or replace an access router A COE-M200 (Central Office Equipment), supporting several CPE-M200s via a central router, is attached to the router either via a LAN port or, to support several MPLS-VPNs, via an ATM or frame relay port For more information contact: StreamCore S.A. 2, place des Hauts Tilliers 92300 Gennevilliers - France Phone: +33 (0) 1 41 21 46 26 Fax: +33 (0) 1 41 21 46 27 URL: www.streamcore.com * Vendor-supplied information not verified by The Tolly Group 2001 The Tolly Group Page 3
Response time/average response time 1.4 1.3 1.2 1.1 1.0 0.9 0.8 Response Time Variability on a WAN Access Link (2 Mbit/s) With and Without StreamGroomer 1 3 5 7 9 11 13 15 17 19 21 23 Successive sample measurements (24 samples) Without StreamGroomer With StreamGroomer Source: The Tolly Group, March 2001 Figure 3 StreamGroomer bandwidth shaping largely improves response times of short interactive batch traffic on access links that are simultaneously used for bulk data transfers and are fully utilized. Part of the test group consisted of small database transfers, which were assigned the highest priority level. The response time of these transactions doubled in some tests with the StreamGroomer bandwidth shaping in place. This response improvement was due to the guarantee that the highest priority traffic was always given the necessary bandwidth. The test bed diagram (figure 4) demonstrates sites intended to represent typical user transaction locations to and from near (10 ms delay) and far (100 ms delay) locations. The test bed diagram is intended to represent: site A, a local Web client; site B, a local Web server; a central office switch; remote site C, as a "near" client and server ("near" being simulated by the 10 ms delay); and finally, remote site D, as a "far" client and server ("far" being simulated by the 100 ms delay). In tests demonstrating the StreamGroomers consistency of response times, figure 3 demonstrates the variability from the mean (average) for each of the test iterations for the first 24 samples with, and without, the StreamGroomer enabled. The X-axis shows the variability from the average, by means of "response time/average response time" ratios. This means that some file transfers are faster, and some slower than the average. With the StreamGroomer enabled, there is very little variability in the amount of time it takes for the file to be transferred, therefore, the StreamGroomer has provided predictability and stability in the rate of the file transfer. The average time it takes to transfer a 100-Kbyte file with the StreamGroomer in place is 21.958 seconds, with the minimum at 21.578 and the maximum at 22.226. This variability is shown in figure 3 as the "response time/average response time" ratio: minimum 0.982696, maximum 1.012207. Without the StreamGroomer providing traffic shaping, results demonstrate greater variability in the samples. The average file transfer time takes 8.154 seconds, while the minimum was 7.30 and the maximum 10.985 seconds. This variability is shown in figure 3 as the "response time/average response time" ratio: minimum 0.896179 maximum 1.347088. Note: As the FTP traffic was deemed "non critical", and as StreamGroomers improve the bandwidth of "critical" traffic, there are a total of 24 samples in a three-minute period, while without the StreamGroomer enabled, there are 56 samples in a three-minute period. For the latter, the first 24 samples have been used for the comparison. Test Configuration and Methodology The Tolly Group configured three StreamCore StreamGroomer Module 200 Flow Regulators into a simulated WAN network for this series of tests. All three devices were hardware version 1-3.b13 and were configured for 100 Mbit/s half duplex. A StreamCore Management and Directory Server version 1-2.b02 running on a standard PC managed the StreamGroomer devices. Two of the StreamGroomer devices were configured on two simulated sites: one simulated a Web client (site A); and the other simulated a Web server (site B). The third StreamGroomer was configured on a simulated central office site (central site). Two additional remote sites connected to the network as a "near 2001 The Tolly Group Page 4
Test Bed Source: The Tolly Group, March 2001 Figure 4 server that was simulated by a 10 millisecond delay (site C); and a "far server that was simulated by a 100 millisecond delay (site D). The central office StreamGroomer Flow Regulator connected to a 3Com Corp. SuperStack II 3300 10/100 Mbit/s Ethernet switch version 2.52 that connected to four 500-MHz generic PCs running Shunra Software, Ltd., The Cloud software version 2.1, software-based WAN simulator. Each PC had 128 Mbytes of RAM, was equipped with a 3Com 905-network adapter, and was running Microsoft Windows 2000 Professional. Two Acterna Corp. DominoPLUS chassis' were connected to two ports of the central switch between two of the Shunra PCs. Each PC also connected to a 3Com SuperStack II 3300 10/100 Ethernet switch version 2.52 in both site A and site B and the switches both connected to the remaining two StreamGroomers. All ports on both switches were configured for 100 Mbit/s full duplex with the exception of the StreamGroomer ports that were configured for 100 Mbit/s half duplex. Site A had another Acterna DominoPLUS Chassis between its switch and the StreamGroomer device. The switches in site A and site B also connected to a generic PC running NetIQ Chariot Endpoint 3.3 software. Site A endpoint served as an FTP, Web and SAP R/3 client and site B served as an FTP, Web and SAP R/3 server. The central office StreamGroomer connected to the generic PCs running Shunra's The Cloud WAN simulator that connected to sites C and D. Site C hosted NetIQ Chariot Endpoint 3.2 software running on a generic PC that served as the FTP client and server, Web browser and server and SAP R/3 server and client. Site D consisted of a generic PC also running NetIQ Chariot Endpoint 3.2 software that also served as the FTP client and server, Web browser and server and SAP R/3 server and client. An Acterna DominoPLUS Chassis was configured between the WAN simulator and the Chariot Endpoint. 2001 The Tolly Group Page 5
Each Chariot Endpoint connected to a Compaq Computer Corp. Intelligent 24-port, 10/100 Ethernet hub that connected to a generic PC with 64 Mbytes of RAM and 2.1 Gbytes of fixed-disk space. This PC served as the DominoPLUS and Chariot Endpoint console and was equipped with a 3Com 905-network adapter and ran Microsoft Windows NT 4.0 Service Pack 5. To test the bandwidth management of the StreamGroomer, engineers used the Chariot test tool to generate 36 traffic streams consisting of interactive and non-interactive traffic simultaneously across a simulated 256 Kbit/s link and a 2 Mbit/s link configured for zeropercent packet loss. Tests were conducted with 10 ms delay and 100 ms delay on each link (latency introduced by The Cloud). The DominoPLUS monitored and captured traffic for protocol verification. Tests ran for three minutes and the average of three iterations were recorded as the results as reported by Chariot. For traffic shaping tests of the StreamGroomer, Tolly engineers used the Chariot test tool to generate 36 traffic streams consisting of interactive and non-interactive traffic simultaneously across a simulated 256 Kbit/s link and a 2 Mbit/s link configured for zeropercent packet loss. HTTP graphical object and HTTP text traffic types with a weight of four and SAP traffic types with a weight of 16 were identified as critical; and SMTP and FTP types with a weight of one were considered non-critical. Engineers used The Cloud to introduce 10 and 100 ms delays into the simulated WAN and then tested the baseline with no shaping using mixed traffic. Results were recorded in Kbit/s. Tolly engineers repeated the test with the StreamGroomer enabled and recorded results. Tests ran for three minutes and results of three iterations were averaged as reported by Chariot. To test the variability in response time, engineers used the Chariot test tool to generate 36 traffic streams consisting of interactive and noninteractive traffic and measured the variability of one filesendlong 100,000-byte traffic stream sent for three minutes from site B to D. The engineers then reported the first 24 successive transfers as reported by Chariot. Engineers repeated this test with the StreamGroomer enabled and averaged the results. The DominoPLUS monitored and captured traffic. The Tolly Group gratefully acknowledges the providers of test equipment used in this project. Vendor Product Web address Acterna Corp. DominoPLUS DA-360 http://www.acterna.com NetIQ Corp. Chariot 3.2 http://www.netiq.com NetIQ Corp. Chariot Endpoint 3.3 http://www.netiq.com Shunra Software, Ltd. The Cloud 2.1 http://www.shunra.com Since its inception, The Tolly Group has produced highquality tests that meet three overarching criteria: All tests are objective, fully documented and repeatable. We endeavor to provide complete disclosure of information concerning individual product tests, and multiparty competitive product evaluations. As an independent organization, The Tolly Group does not accept retainer contracts from vendors, nor does it endorse products or suppliers. This open and honest environment assures vendors they are treated fairly, and with the necessary care to guarantee all parties that the results of these tests are accurate and valid. The Tolly Group has codified this into the Fair Testing Charter, which may be viewed at http://www.tolly.com. Project Profile Sponsor: StreamCore Document number: 200235 Product Class: WAN traffic shaper and optimizer Products under test: StreamGroomer Module 200 Flow Regulator StreamGroomer Manager (SGM) Testing window: October 2000 Software versions tested: Version 1-3.b13 Software status: Readily available For more information on this document, or other services offered by The Tolly Group, visit our World Wide Web site at http://www.tolly.com, send E-mail to info@tolly.com, call (800) 933-1699 or (732) 528-3300. Internetworking technology is an area of rapid growth and constant change. The Tolly Group conducts engineering-caliber testing in an effort to provide the internetworking industry with valuable information on current products and technology. While great care is taken to assure utmost accuracy, mistakes can occur. In no event shall The Tolly Group be liable for damages of any kind including direct, indirect, special, incidental, and consequential damages which may result from the use of information contained in this document. All trademarks are the property of their respective owners. The Tolly Group doc. 200235 rev. kco 07 March 01 2001 The Tolly Group Page 6