Lab Testing Summary Report April 2014 Report 140305 Product Category: Ethernet Services Edge Routers Vendors, Products Tested: Key findings and conclusions: The Cisco ASR 9922 reroutes around failed links quicker up to 97 percent quicker for some services and with much less user data loss than does the Alcatel-Lucent 7750 SR-12e The Cisco ASR 9922 consistently delivers line-rate throughput and QoS traffic handling, and low, consistent latency, even while handling 50 percent more service-processing complexity The Alcatel-Lucent 7750 SR-12e exhibited per-port VLAN restrictions, disappointing handling of high-priority QoS traffic and the FP3 failed to support its 200Gbps claim The Juniper MX960 3D could not handle the high service scale configuration required for some tests. It exhibited wide latency variability and resiliency issues with replicated-packet handling, and failed to meet the vendor's own 10GE throughput claims with even light service processing T his report summarizes the dramatic results of Miercom's months' long, hands-on testing of leading Provider Edge (PE) routers, which are collectively called Ethernet Services Edge Routers. The products tested, from Alcatel-Lucent, Cisco and Juniper, are these vendors' current top-of-the-line access routers for service providers. This bold comparative study is among the first to apply such 'multi- Aggregate Throughput for Vendor's Highest Density 10GE Line Card Tested with Maximum VLANs per Port and HQoS applied, with 99 Percent-Load, Bidirectional Traffic Streams 250 7750 SR-12e Throughput 200 150 100 Dropped Packets Ingress ASR 9922 ALU 7750 MX 960 ASR 9922 50 0 ASR 9922 ALU 7750 MX 960 Throughput (Gbps) 222.9 36.4 16.8 Dropped Packets 250 MX960 3D Throughput 200 150 100 Dropped Packets Egress ASR 9922 ALU 7750 MX 960 50 Dropped Packets 0 ASR 9922 ALU 7750 MX 960 Throughput (Gbps) 222.9 164 40.6 Only Cisco delivered the full load of traffic received on its 24-port 10GE line card, with heavy VLAN and HQoS processing. Alcatel-Lucent forwarded just 54 percent via its 20-port card, and Juniper just 52 percent via its 8-port card.
dimensional' tests to high-end, modular and multislot routers. A diverse but consistent set of Ethernet-based services was configured on each of the devices under test (DUTs), up to scale, or the maximum extent the system would allow, and then a staggering array of test equipment delivered Gigabits/s of traffic load to each, while carefully measuring different aspects of the DUTs' performance. The different performance aspects of the routers examined included: Recovery from a variety of failed-path scenarios, in terms of simulated user data lost and elapsed time until resumption of normal service data flows. The maximum sustainable throughput of the vendors' highest density 10-Gig Ethernet Services line cards, first with minimal per-port processing and then with maximum VLANs and Hierarchical Quality of Service (HQoS) applied on all ports to assess the handling of HQoS services under load. Point-to-multipoint and replicated-packet handling, in terms of point-to-multipoint throughput, replicated-packet latency and protection of replicated packet streams in the event of a router card failure. Full details of the testing recapped here can be found in a companion White Paper, Report DR140121, issued along with this summary. Rerouting Around a Failed Path A battery of tests measured how quickly the device under test (DUT) could reroute, or reconverge, around an internal-network link failure when one or more equal-cost links exist for the different services running. Also measured was how much user data was lost as a result of such a link outage. The tests were done using bi-directional traffic; test traffic was delivered at below a 50 percent rate, to prevent oversubscription after the link fails. The links that were failed assessed the device under test's (DUT's) ability to respond to: a single high-speed core link member failure, in an Ethernet Link Aggregation group (LAG) and then the failure of a whole Ethernet LAG. The test data streams were bi-directional flows for each unicast service, and unidirectional outbound (PE-to-CE) for multicast services which, by packet addresses and type, exercised most of the different services running on the DUT. Half of the test traffic was being carried over two 100GE links that together comprised an aggregated Ethernet bundle (the Link Aggregation Group, or LAG). To determine the percent of traffic impacted by the failure scenario we caused, we presumed that each DUT load-balanced equally between equal-cost multiple paths and link members within a bundle. As shown in the diagram, 25 percent of the test traffic flows were being carried over each 100- gigabit/s Ethernet (100GE) link for the single highspeed core link failure. To simulate the failure of the whole Ethernet group, both links were simultaneously failed, either by physically popping the line card in the DUT that handled the bi-directional traffic over these two 100GE links, or by gracefully shutting down the links on the core router. Three Cisco CRS core routers were used in exactly the same manner for all the tests. In this battery of tests, the Juniper MX960 3D router was not included. In attempting to configure the number of concurrent services and high-speed links that our test bed specified, we encountered software issues with the MX960: Error messages indicated insufficient resources. The Juniper router could be successfully configured for less complex configurations, but in order to present a comparable "apples to apples" test environment, we deemed that including the Juniper MX960 in these tests would be inappropriate. Juniper was afforded the opportunity to address the issue but did not offer a resolution to the configuration issue. Frame Loss by Service, Single-Link Failure Given the same multi-service flows of simulated customers' bidirectional traffic, frame loss by service was typically less with the Cisco ASR 9922 than with the Alcatel-Lucent 7750. For redirecting traffic around a single 100GE backbone link failure, loss for VPLS service was 73 percent less with Cisco than Alcatel-Lucent. EoMPLS service loss was nearly 80 percent less with Cisco. Copyright 2014 Miercom Ethernet Services Edge Routers Page 2
Table 1: Loss by Service, from a Failed 100GE Link Service Service (VPLS), LDPcontrolled Service (VPLS), BGPcontrolled Ethernet over Multiprotocol Label Switching (EoMPLS) Cisco ASR 9922 (No. of Frames Lost) Alcatel-Lucent 7750 (No. of Frames Lost) 3,196 8,599 3,209 8,778 599 2,864 High-Speed Internet (HIS) 844 2,410 Layer 3 Virtual Private Network (L3VPN) 8,085 22,890 IPv6 VPN 816 2,223 Multicast VPN 180 80 Broadband Network Gateway 2,320 2,505 Table 2: Service Resumption Times after 100GE Link Failure Service Service (VPLS), LDPcontrolled Service (VPLS), BGPcontrolled Ethernet over Multiprotocol Label Switching (EoMPLS) Cisco ASR 9922 (Outage time in milliseconds) Alcatel-Lucent 7750 (Outage time in milliseconds) 4.4 15.6 5.7 16.0 5.5 30.0 High-Speed Internet (HIS) 6.5 15.5 Layer 3 Virtual Private Network (L3VPN) 15.5 15.5 IPv6 VPN 5.5 5.0 Multicast VPN 30.0 17.5 Broadband Network Gateway 56.4 60.9 Table 3: Loss by Service, from a Remote Ethernet Bundle Failure Service Service (VPLS), LDPcontrolled Service (VPLS), BGPcontrolled Ethernet over Multiprotocol Label Switching (EoMPLS) Cisco ASR 9922 (No. of Frames Lost) Alcatel-Lucent 7750 (No. of Frames Lost) 153,042 4,481,307 153,003 5,109,156 25,318 670,859 High-Speed Internet (HIS) 92,982 1,323,069 Layer 3 Virtual Private Network (L3VPN) 464,575 25,088,092 IPv6 VPN 47,017 1,978,832 Multicast VPN 3,984 46,568 Broadband Network Gateway 185,516 671,460 Table 1 shows the amount of data loss, incurred by service, as a result of a single core-link outage, with the Cisco ASR 9922 and the Alcatel-Lucent 7750, before traffic could be redirected around the failed link and the service resumed normal traffic flow. Given the same multi-service flows of simulated customers' bidirectional traffic, frame loss by service was typically less with the Cisco ASR 9922 than with the Alcatel-Lucent 7750. For redirecting traffic around a single 100GE backbone link failure, loss for VPLS service was 73 percent less with Cisco than Alcatel-Lucent. EoMPLS service loss was nearly 80 percent less with Cisco. Table 2 shows the duration of service outage that is, the time, in milliseconds, from the single 100GE core-link failure, until the device under test (DUT) successfully redirected traffic and full traffic flow for that service resumed. With the same multi-service flows of simulated customers' bidirectional traffic, the Cisco ASR 9922 rerouted most services around a failed network-core link faster than the Alcatel-Lucent 7750, enabling services to resume full traffic rates quicker. Full traffic levels for VPLS service, for example, resumed up to 65 percent faster with Cisco, and Ethernet-over-MPLS service 80 percent faster, than Alcatel-Lucent. Also tested was the device under test's (DUT's) ability to recover from an entire failed Ethernet bundle. When all links in a bundle fail, the path between the two points is severed, including IPpath connectivity. This is a higher-layer and more severe failure, requiring IP alternate-path re-routing and re-convergence. The Ethernet bundle failure was tested "locally," by popping the card within the device under test (DUT) that handled both links of the bundle; and "remotely," where the bundle was administratively shut down at the CRS core router end. As shown in Table 3, the amount of data loss from such a bundle failure can be substantial. However, loss was much less, by service, with the Cisco ASR than with the Alcatel-Lucent 7750. Loss for VPLS and Layer 3 VPN services were more than 99 percent less with Cisco than with Alcatel-Lucent. Also significant were the differences in the duration of service outages resulting from the failure of an entire Ethernet bundle. In the case Copyright 2014 Miercom Ethernet Services Edge Routers Page 3
of the local bundle outage, the Cisco ASR 9922 was able to reconverge VPLS service, allowing full-rate traffic to resume, 97 percent quicker than the Alcatel-Lucent 7750. High-Speed Internet (HSI) service resumed 96 percent quicker with the Cisco ASR and Multicast VPN service 90 percent faster, than with the Alcatel-Lucent 7750. With the remote Ethernet bundle failure, the Cisco ASR 9922 allowed full-rate traffic to resume for most services in less than 200 milliseconds. By comparison, reconverging around the same remote Ethernet bundle outage typically took the Alcatel- Lucent 7750 from 20 to 500 times longer. Multi-Path Reconvergence Another series of tests assessed the device under test's (DUT's) ability to promptly address the failure of multiple remote routes, what is termed multi-path reconvergence. The state of the IP routes is learned from remote Provider Edge routers (PEs) using Multi-protocol BGP. In these tests the routes across the service provider's network are MPLS-based Virtual Private Network (VPN) routes. This convergence scenario for IP VPNs is commonly referred to as BGP PIC Edge. The topology for this testing was complex, and readers are referred to the full White Paper report for complete test details. Customer-edge routers were dual-homed to different PE routers. Each PE had a unique route distinguisher, and one of the two PEs within each dual-homed CE "customer" connection had a higher local preference value for traffic routing. VLANs (per IEEE 802.1q) were used to provide IP-path separation between the primary and backup paths to each CE, which the test equipment emulated. Test traffic was delivered bi-directionally for all services. However, the traffic of interest in these tests was IPv4 VPN traffic from the customers on the device under test (DUT) to the emulated PE customers connected at the far end of the network. We exercised two path-failure scenarios: When two remote PE paths fail we failed the path to the higher-preference-vpn PE; and when multiple paths fail we failed paths to multiple-vlan PEs. This testing was conducted with each DUT configured at the same scale as the Rerouting Around a Failed Path test. As the scale tested for each DUT was based on its number of ports, the Cisco ASR 9922 had to manage 50 percent more prefixes impacted by the outages than the Alcatel- Lucent 7750 SR-12e. A notable difference in frame loss was observed between the tested routers, resulting from a single remote-pe-path failure, for Layer 3 (IP-based) Virtual Private Network (L3VPN) service. With a single remote-link failure, our testing found that VPN IP traffic through the Alcatel-Lucent 7750 suffered 1.67 million lost frames, compared to just 180,000 lost packets with the Cisco ASR 9922. In other words, a simulated customer's Layer 3 VPN incurred nine times more loss through the Alcatel- Lucent 7750 than through the Cisco ASR 9922. The duration of service outage for the Layer 3 VPN service, before reconvergence, is shown below for the Cisco ASR 9922 (top) and the Alcatel-Lucent 7750 (bottom), in response to the multiple-remotepath-link failure. Duration of Outage before Multi-path Reconvergence Cisco = 124 ms Alcatel-Lucent = 1,892 ms Over 15 times faster recovery. Reconvergence after a remote multi-path failure took 1,892 milliseconds via the Alcatel-Lucent 7750, nearly two full seconds, before Layer 3 VPN traffic levels resumed normally. The same outage time for the Cisco ASR 9922 was 124 milliseconds over 15 times faster than the Alcatel-Lucent 7750. Copyright 2014 Miercom Ethernet Services Edge Routers Page 4
Overall Performance Another set of tests sought to measure the vendors' claimed performance of these Ethernet Services Edge Routers. We focused on the throughput of each vendor's highest density 10- Gigabit Ethernet (10GE) Services Edge line card. The vendors offer quite different 10GE line cards. The below table shows the cards we tested, the highest density 10GE cards shipping at the time of our testing, and the vendor claims as far as the number of ports per card concurrently supporting line-rate throughput. That is the number of ports we tested in this round. Device Under Test: Line Card Alcatel-Lucent 7750: IMM-2PAC- FP3 20x10GE Cisco ASR 9922: 24x10GE A9K- 24x10GE-SE Juniper MX960: MPC2E-3D-EQ with 8x10Ge No. of 10GE Ports on Line Card Claimed Line Rate, No. of 10GE Ports No. of 10GE Ports Tested 20 20 20 24 24 24 8 6 6 Bi-directional test traffic streams were applied to each port. An IMIX traffic pattern a blend of IP packet types and packet sizes, from minimal 66 to 1,518 bytes was used. The test systems generated about 3.82 million packets per second, per 10GE interface, delivering the specified rate of 99 percent of each 10GE port's capacity. Tests were run initially with minimal additional processing per port just a single VLAN per port was defined. Afterwards, we added serviceprocessing complexity: we configured 850 VLANS per port with a shaper and basic three class Hierarchical Quality of Service (HQoS) processing on each VLAN. Cisco and Juniper supported this; Alcatel-Lucent, we learned, supported only 714 IPv4-enabled VLANs per port. Of the 99 percent of full 10GE capacity applied on each port, the three HQoS classes (assured forwarding, expedited forwarding and best effort) accounted for equal 33 percent volumes of the traffic capacity. Traffic of each HQoS class was split equally to and from every VLAN configured. The aggregate throughputs achieved for the vendors' 10GE line cards are shown on the front page of this report. These throughputs reflect the most complex per-port service-processing environment: with VLANs at scale and with HQoS processing applied. "Ingress" reflects throughput in one direction, "egress" in the other relative to the line card under test. We observed considerable loss and improper handling of HQoS classes by both Alcatel-Lucent and Juniper. Throughputs in the egress direction are shown in the figures below. Alcatel-Lucent 7750, Egress Throughput Cisco ASR 9922, Egress Throughput Juniper XM960, Egress Throughput Copyright 2014 Miercom Ethernet Services Edge Routers Page 5
With just one VLAN configured per port and no other exertive processing, we observed appreciable packet loss with both the Juniper MX960 3D and the Alcatel-Lucent 7750 SR-12e. Juniper dropped 7 percent of the traffic in one direction, and so failed to support line rate on six 10GE ports, as claimed in the vendor's datasheets. The Alcatel-Lucent 7750 SR-12e dropped 26 percent of the traffic in one direction and 28 percent in the other on the line card under test. No packets were dropped by the Cisco ASR-9922 in the same configuration and with the same 99- percent bi-directional load. Cisco System s claim to be able to handle line rate, full duplex, on 24 x 10GE ports was therefore validated. Under heavy load, with hundreds of VLANs configured per port and with Hierarchical QoS enabled and running, Alcatel-Lucent processed and sent just 30 percent of the high-priority traffic in one direction, along with 5 percent of the default Best Effort. In the other direction it delivered 94 percent of the Assured Forwarding and 80 percent of the Expedited Forwarding. Under load, Juniper processed and forwarded about 40 percent of the priority traffic and about 13 percent of the low-priority Best Effort traffic. In the other direction Juniper managed 90 percent of the high priority traffic, and about 38 percent of Best Effort traffic. Only Cisco managed to correctly process 100 percent of all three HQoS traffic classes, in both directions, and with no loss. Packet Replication, Multipoint A final battery of tests focused on packet replication, the key component in point-tomultipoint applications. We sought to exercise and observe the DUTs' ability to effectively replicate and deliver a high volume of traffic to output ports. We checked, too, to see how resilient point-to-multipoint traffic handling was when disruptions, such a line-card failure, occur within the router. Tests were devised to compare DUT performance with regard to: Packet-replication-induced latency. Throughput performance for replicating a single 9.5-Gbps flow. The DUT's performance was tested with a single multicast stream. Packet-replication resilience. We intentionally failed a customer/receiver-side line card to observe what impact it would have on other customers/receivers, connected via other line cards, who also relied on packet replication from the DUT. These tests were performed with scaled (maximum configurable) services all enabled and active, but with no unicast traffic flowing. The number of 10GE output ports closely monitored for this testing were: Alcatel-Lucent: 48 x 10GE ports Cisco: 72 x 10GE ports Juniper: 60 x 10GE ports The figure below shows the latency differences of a multicast stream on the DUT's output ports. Multicast Latency by Port Latency for replicated traffic should be low and consistent across all output ports. In Juniper's case, latency was two or three times as much on some ports. Alcatel-Lucent's latency exhibited smaller variation. Cisco's latency was consistently low. Cisco Alcatel Juniper Copyright 2014 Miercom Ethernet Services Edge Routers Page 6
The Cisco ASR 9922 exhibited remarkable consistency, delivering all point-to-multipoint traffic to all output ports with the same, consistent, sub-50- microsecond latency. The Alcatel 7750 exhibited variable latency, depending on the output port. The most inconsistent and variable latency was experienced by replicated-packet traffic through the Juniper MX960 3D. Latency varied considerably depending on output port. Juniper exhibited the most variation in latency, as the chart below shows. The latency of replicated-packet traffic through the Alcatel-Lucent 7750 varied much less. Cisco delivered consistent latency; multicast traffic through the Cisco ASR 9922 varied barely 2 microseconds from the average of 48 microseconds. Minimum / Average / Maximum Latency Our last test, of point-to-multipoint resiliency the ability of the DUT to protect replicated packet streams revealed an issue with Juniper. In this test we sent a unidirectional input traffic stream at 9.5 Gbps, a near-maximum rate for large-sized, 1,518-byte packets, to a 10- Gigabit/s Ethernet (10GE) port. The DUT then has to replicate and forward this stream to every other 10GE port on the router. All services were active and running. Test equipment measured how much replicatedpacket traffic was delivered on each output port. Then we intentionally failed one of the DUT's customer/receiver-side line cards, to see what impact it would have on other customers/ receivers, connected via other line cards. With Cisco and Alcatel-Lucent there was no impact whatever on the other, unaffected output ports. On the Juniper MX960 3D, however, every output port experienced loss, over 8,000 large-size packets, when the line card was failed, until the other, unaffected, replicated-packet streams resumed normal, flow. This loss on all Juniper ports is shown in the chart below. Multicast Resiliency - Juniper Cisco and Juniper were both able to replicate and deliver the full 9.5-Gbit/s multicast input stream onto all output ports. However, Alcatel-Lucent exhibited serious issues in processing replicated-packet traffic, replicating and delivering just 20 percent of the input stream, as the chart below shows. Packet Loss Multicast Throughput Ports Gbps Cisco Alcatel Juniper The Bottom Line The testing of high-end Provider Edge (PE) access routers in this study revealed crucial performance differences, especially under heavy traffic loads while running multiple concurrent services. These multi-dimensional tests, among the first to scientifically apply such loads to these systems, caution service providers in accepting vendors' isolated performance claims. Copyright 2014 Miercom Ethernet Services Edge Routers Page 7
Miercom Performance Verified The Cisco ASR 9922 has demonstrated in independent testing that it delivers wirespeed throughput and performance under conditions where competitors cannot. Stringent tests also found that the Cisco ASR 9922 recovers from failed network links much faster, and with much less loss of user data, than competitors. We especially laud the Cisco router's impeccable latency and QoS-handling performance. Testing proved that the Cisco ASR 9922 has earned the Miercom Performance Verified Certification. Cisco ASR 9000 Router Model ASR 9922 Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 1-800-553-6387 www.cisco.com About Miercom s Product Testing Services Miercom has hundreds of product-comparison analyses published over the years in leading network trade periodicals including Network World, Business Communications Review, Tech Web - NoJitter, Communications News, xchange, Internet Telephony and other leading publications. Miercom s reputation as the leading, independent product test center is unquestioned. Miercom s private test services include competitive product analyses, as well as individual product evaluations. Miercom features comprehensive certification and test programs including: Certified Interoperable, Certified Reliable, Certified Secure and Certified Green. Products may also be evaluated under the NetWORKS As Advertised program, the industry s most thorough and trusted assessment for product usability and performance. Report 140305 reviews@miercom.com www.miercom.com Before printing, please consider electronic distribution Product names or services mentioned in this report are registered trademarks of their respective owners. Miercom makes every effort to ensure that information contained within our reports is accurate and complete, but is not liable for any errors, inaccuracies or omissions. Miercom is not liable for damages arising out of or related to the information contained within this report. Consult with professional services such as Miercom Consulting for specific customer needs analysis. Copyright 2014 Miercom Ethernet Services Edge Routers Page 8