H3C S12516X-AF Sets New World Records for Data Center Performance. June 2017

H3C S12516X-AF Sets New World Records for Data Center Performance June 2017

T A B L E O F C O N T E N T S Executive Summary... 3 About This Test.... 4 Performance Test Results....................................................... 5 IPv4 Unicast Throughput With BGP Routing... 5 Routing Scalability and Throughput... 7 IPv4 Unicast Latency and Jitter With BGP Routing.... 9 Routing Scalability Latency and Jitter With BGP Routing.... 11 IPv6 Unicast Throughput With BGP Routing... 12 IPv6 Unicast Latency and Jitter With MP-BGP Routing.... 14 EVPN Scalability With VXLAN Tunnels... 15 ISSU/ISSD Failover Times... 18 Conclusion.... 21 Appendix A: About Network Test.... 22 Appendix B: Hardware and Software Releases Tested.... 22 Appendix C: Disclaimer.... 22 Page 2

H3C S12516X-AF: A New World Record for Data Center Performance Executive Summary Data centers keep growing in two ways: They re larger, with more servers connected than ever before, and they re faster, with server NIC speeds of 10 gigabits and higher now common. For the core switches that tie everything together, one requirement stands out above all else: Scale to unprecedented heights. That s exactly what H3C has done with its S12516X-AF data center core switch, setting a new world record by demonstrating the highest bandwidth capacity for a single switch with 768 100G Ethernet interfaces. H3C commissioned independent test lab Network Test to validate the performance and scalability of its data center core switch. This is, by far, the largest assessment of 100G Ethernet switch performance yet conducted. In addition to performing rigorous stress tests on the switch fabric, Network Test assessed the H3C switch in terms of control-plane metrics such as BGP and MP-BGP routing with IPv4 and IPv6 traffic; Ethernet VPN (EVPN) scalability using VXLAN tunnels; and the impact of in-service software upgrades (ISSU). In all these areas, the H3C switch delivered stellar performance: The highest throughput ever recorded from a single switch chassis (76.8 terabits per second) Throughput of more than 100 million frames per second per port on each of 768 100G Ethernet ports Support for nearly 1 million unique routes learned via BGP Identical throughput when routing to 768 routes and nearly 1 million routes The highest EVPN scalability ever recorded with a single chassis (768 concurrent VXLAN tunnels) ISSU failover times measured in tens of microseconds This report is organized as follows. This section provides an overview of the test results. The About This Test section provides an overview of test cases and briefly covers the importance of each metric used and briefly describes issues common to all test cases. The Performance Test Results section provides full results from individual test cases. Appendix B provides software versions used in testing. Figure 1: The H3C S12516X-AF test bed, with Spirent TestCenter and 768 100G Ethernet ports Page 3

About This Test The device under test for this project was an H3C S12516X-AF data center core switch fully loaded with two supervisor modules and 16 line-card modules, each supporting 48 100-gigabit Ethernet interfaces 1. This project assessed the H3C S12516X-AF using eight test cases, most involving 768 100G Ethernet interfaces: IPv4 unicast throughput with BGP routing (1 route per port) IPv4 route scalability throughput (nearly 1 million routes total) IPv4 unicast latency and jitter with BGP routing IPv4 routing scalability latency and jitter IPv6 unicast throughput with MP-BGP routing IPv6 unicast latency and jitter with MP-BGP routing EVPN scalability with VXLAN tunnels ISSU / ISSD failover times The test bed, as seen in Figure 1, also included the Spirent TestCenter traffic generator/analyzer with dx3 10/25/40/50/100G hardware modules and direct-attached (DACS) copper cables. The Spirent test instrument can offer traffic at wire speed concurrently on all ports with timestamp resolution of 2.5 nanoseconds. The primary metrics in performance tests were throughput, latency, and jitter. EVPN tests examined VXLAN tunnel scalability as well as throughput, latency, and jitter. ISSU tests used frame loss to derive failover time during software image upgrades and downgrades. RFC 2544, the industry-standard methodology for network device performance testing, determines throughput as the limit of system performance. In the context of lab benchmarking, throughput describes the maximum rate at which a device forwards all traffic with zero frame loss. Describing real-world performance is explicitly a nongoal of RFC 2544 throughput testing. Indeed, production networks load are typically far lower than the throughput rate. Latency and jitter respectively describe the delay and delay variation introduced by a switch. Both are vital, and arguably even more important than throughput, especially for delay-sensitive applications such as video and voice. In all tests, engineers configured the H3C data center core switch in store-and-forward mode, its default setting. This is a common industrywide practice for data center core switches. Each test described in this document uses a three-part structure. Why It Matters explains the significance of the metric or feature being tested in the context of data center networking. What We Did describes the test procedure. And What We Found covers test results. 1 The supervisor modules were LSXM1SUPB1 main processing units (MPUs). The line cards were LSXM1CGQ48HB1 interface modules. Page 4

Performance Test Results IPv4 Unicast Throughput With BGP Routing Why It Matters: No task is more important for a data center core switch than moving traffic at maximum speed with zero frame loss. None of a switch s other attributes its features list, its supported protocols, its high-reliability attributes will matter if the switch cannot efficiently move traffic under even the heaviest loads. How We Tested: We measured throughput by stressing both the data-plane fabric and control-plane routing capabilities of the H3C switch. RFCs 2544 and 2889 describe a data-plane stress test, and have long been the industry-standard methodologies for router and switch performance testing with unicast traffic. To load up the control plane, engineers configured Spirent TestCenter to emulate 768 Border Gateway Protocol (BGP) routing peers, each using a unique Autonomous System Number (ASN). Each Spirent BGP router brought up a peering session with the H3C S12516X-AF switch, then advertised a total of 768 unique routes. The Spirent test tool then offered fully meshed traffic among all networks learned using BGP. A fully meshed pattern means each port exchanges traffic with all other ports. This is the most stressful traffic pattern for a switch fabric. Test engineers determined the throughput rate the highest speed at which the switch correctly forwarded all traffic, in sequence and without frame loss. Frame sizes ranged from the Ethernet minimum of 64 bytes to the maximum of 1,518, and beyond to 9,216-byte jumbo frames. Engineers used a duration of 60 seconds for each test. Note that the choice of 768 routes is due to a limit in the number of trackable receive streams supported by the Spirent dx3 test modules, and not of the H3C switch s routing capacity. As demonstrated in the Routing Scalability and Throughput test in this report, the H3C S12516X-AF offers the same high performance to nearly 1 million routes using a different traffic pattern. Other Spirent test modules also support higher trackable stream counts. With the dx3 module, a higher route count also would have been possible using fewer than 768 ports. What We Found In nearly all test cases, the H3C switch moved traffic at the theoretical maximum rate to all 768 100-gigabit Ethernet ports. These tests involved traffic in a fully meshed pattern the most stressful possible load. In some of these tests, traffic moved so fast that the switch processed more than 100 million frames per second per port simultaneously, on each of 768 ports. Aggregate layer-1 throughput was nearly 77 terabits per second. Page 5

Figure 2 presents results from the IPv4-with-BGP throughput tests. Although the industry-standard methodology calls for seven frame lengths to be tested, H3C also included two extra frame sizes. Engineers included 102-byte frames to demonstrate that theoretical maximum rates are possible with this relatively short frame size, not far up from the 64-byte minimum. Engineers also tested with 9,216-byte jumbo frames to demonstrate that the H3C switch can handle the large payloads commonly found in data-center applications. Figure 2: IPv4 throughput with BGP routing, 768 100G Ethernet ports Page 6

Routing Scalability and Throughput Why It Matters: It s not just the global Internet that continues to expand at a rapid pace, requiring ever-larger routing tables. Inside the data center, huge numbers of routes are now increasingly common. Both for external and internal connectivity, data center core switches must be able to route traffic among huge numbers of unique networks. This test involved nearly 1 million routes learned via BGP. To put this number in context, consider that the entire global Internet consists, at this writing, of fewer than 700,000 BGP routes. For internal routing within even very large data centers, a core switch capable of routing to 1 million networks has a very high capacity. How We Tested: This test is conceptually similar to the previous IPv4-plus-BGP test, with the Spirent test instrument emulating 768 BGP peers attached to the H3C switch. But instead of advertising just 1 route per port, test engineers this time advertised nearly 1 million routes, all emulated by the Spirent test instrument. The advertised routes included nearly 750,000 networks with a /24 prefix length. The remaining networks, approximately 250,000 in number, used a pseudorandom distribution of prefix lengths between /8 and /32. After verifying that the H3C switch had learned all routes, engineers again configured Spirent TestCenter to offer traffic at the throughput rate using a variety of frame sizes. One difference from previous tests is that engineers configured a port-pair traffic pattern, in which 384 pairs of ports exchanged bidirectional traffic, instead using a fully meshed pattern. Engineers made this change in traffic pattern to accommodate the maximum number of trackable receive streams supported on the test instrument s dx3 modules; the change does not reflect a limitation on the part of the H3C switch. What We Found Throughput was identical for all frame sizes when routing to nearly 1 million networks as it was when routing to just 1 network per port. The H3C data center core switch exhibited wire-speed throughput for all frame sizes of 102 bytes and larger, regardless of the number of routes and the prefix lengths involved. Page 7

Figure 3 presents throughput results from the routing scalability tests. Although the industry-standard methodology calls for seven frame lengths to be tested, H3C also included two extra frame sizes. Engineers included 102- byte frames to demonstrate that theoretical maximum rates are possible with this relatively short frame size, not far up from the 64-byte minimum. Engineers also tested with 9,216-byte jumbo frames to demonstrate that the H3C switch can handle the large payloads commonly found in data-center applications. Figure 3: BGP routing scalability throughput, 1 million routes and 768 100G Ethernet ports Page 8

IPv4 Unicast Latency and Jitter With BGP Routing Why It Matters: For some applications, latency and jitter are even more important than throughput. Spikes in latency and/or jitter can degrade performance not only for voice and video, but also for the mission-critical applications used in some vertical markets. For example, in the financial-services sector, many trading applications require the lowest possible latency to ensure rapid order fulfillment. Further, while throughput is a measure of switch performance at its maximum speed, latency and jitter affect application at every speed, regardless of load. Network device architecture is yet another factor that may affect latency and jitter. Some top-of-rack switches may reduce latency and jitter by using cut-through designs that begin forwarding a frame as soon as they receive the very beginning of a frame (the Ethernet header). In contrast, virtually all data center core switches, including the H3C S12516X-AF, use a store-and-forward design that caches the entire incoming frame before deciding where to forward it. A store-and-forward design is mandatory in situations involving routing, as was the case in these tests, since the switch must look beyond the Ethernet header before making a forwarding decision. How We Tested: As required by RFC 2544, we measured latency at the throughput rate in all tests. All network devices have a load-vs.-delay curve that spikes up sharply as offered loads approach the throughput rate. In production networks, where utilization rates seldom hit 100 percent, both average and maximum delay will be lower than results obtained from RFC 2544 tests. To illustrate this, we also ran an additional test to measure delay, using 64-byte frames at 65 percent of line rate, slightly below the throughput rate equivalent to 71 percent of line rate. This additional test served to show how delay is lower just below the throughput rate. Because we measured latency and jitter at the same time as throughput, the same test conditions applied 768 100G Ethernet interfaces, with Spirent TestCenter offering traffic in a fully meshed pattern. What We Found Average and maximum latency and jitter are very consistent across test cases for most frame sizes, both for average and maximum measurements. In addition, variations in latency and jitter are relatively small across most frame sizes. Delay for 64-byte frames, when measured just below the throughput rate, is significantly lower than at the throughput rate. Page 9

Table 1 presents average and maximum latency and jitter measurements for the H3C data center core switch when handling IPv4 routed traffic. Again, the switch ran BGP routing and learned 1 unique route on each of its 768 100G Ethernet ports. Latency Table 1: IPv4 with BGP routing latency and jitter Jitter Frame size (bytes) Minimum (us) Average (us) Maximum (us) Average (us) Maximum (us) 64 1.830 22.716 462.640 6.260 448.240 102 1.800 10.592 87.360 0.476 64.480 128 1.820 4.992 7.180 0.022 2.570 256 1.810 4.721 6.660 0.014 2.260 512 1.820 6.576 10.220 0.014 5.060 1,024 1.830 4.795 6.840 0.010 2.050 1,280 1.850 5.275 8.200 0.015 2.920 1,518 1.830 4.816 6.860 0.013 2.100 9,216 1.920 5.949 8.260 0.129 3.960 Table 2 shows the differences in delay for 64-byte frames between traffic offered at and just below the throughput rate. As the table shows, there are significant reductions in delay and jitter at the slightly lower rate, especially for maximum delay and jitter measurements. Delay Jitter Intended load (% of line rate) Minimum (us) Average (us) Maximum (us) Average (us) Maximum (us) 65 1.790 4.379 357.640 0.262 348.260 71 1.830 22.716 462.640 6.260 448.240 Delta -0.040-18.337-105.000-5.998-99.980 Table 2: Delay for 64-byte frames at 65% load and 71% load for IPv4 with BGP routing Page 10

Routing Scalability Latency and Jitter With BGP Routing Why It Matters: In a network, frames don t know if they are being switched or routed latency and jitter are still significant factors in application performance. If anything, the extra overhead involved with lookups in large routing tables may increase latency and jitter, making measurement of these metrics even more critical than in situations with little or no routing. How We Tested: We measured latency and jitter at the same time as throughput, using the same test conditions: 768 100G Ethernet interfaces, with Spirent TestCenter offering traffic in a port-pair pattern. The test instrument advertised nearly 1 million routes to the H3C switch, including nearly 750,000 networks with a /24 prefix length. The remaining networks, approximately 250,000 in number, consisted of a pseudorandom distribution of prefix lengths between /8 and /32. As in the scenario with 1 route, test engineers also compared delay and jitter for 64-byte frames at the throughput rate, and just below the throughput rate. What We Found Even when routing traffic to nearly 1 million unique routes, latency and jitter for the H3C data center core switch remained very similar to tests with just 1 route per port. In addition, variations in latency and jitter are relatively small across most frame sizes. Delay for 64-byte frames, when measured just below the throughput rate, is significantly lower than at the throughput rate. Table 3 presents average and maximum latency and jitter measurements for the H3C data center core switch. Latency Table 3: BGP routing scalability latency and jitter Jitter Frame size (bytes) Minimum (us) Average (us) Maximum (us) Average (us) Maximum (us) 64 1.830 22.322 471.330 6.148 447.240 102 2.730 3.887 4.580 0.004 0.450 128 2.770 3.861 4.560 0.004 0.450 256 2.800 3.822 4.510 0.005 0.430 512 2.790 3.921 4.600 0.004 0.440 1,024 2.810 3.897 4.600 0.005 0.370 1,280 2.820 4.021 4.750 0.005 0.550 1,518 2.910 4.511 5.440 0.005 1.130 9,216 1.920 5.949 8.260 0.129 3.960 Page 11

Table 4 shows the differences in delay for 64-byte frames between traffic offered at and just below the throughput rate. As the table shows, there are significant reductions in delay and jitter at the slightly lower rate, especially for maximum delay and jitter measurements. Delay Jitter Intended load (% of line rate) Minimum (us) Average (us) Maximum (us) Average (us) Maximum (us) 65 1.790 4.379 357.640 0.262 348.260 71 1.830 22.322 471.330 6.148 447.240 Delta -0.040-17.943-113.690-5.886-98.980 Table 4: Delay for 64-byte frames at 65% load and 71% load for BGP routing scalability IPv6 Unicast Throughput With BGP Routing Why It Matters: Now that the pool of routable IPv4 allocations has been exhausted, enterprises and service providers are turning in record numbers to IPv6. This naturally raises the question of whether throughput and routing performance will be the same as with IPv4 traffic. How We Tested: We measured throughput by stressing both the data-plane fabric and control-plane routing capabilities of the H3C switch using IPv6 traffic. RFC 5180 builds upon the foundation in RFCs 2544 and 2889 to describe an IPv6-specific data-plane stress test. Conceptually RFC 5180 is very similar to the previous work for IPv4, especially in its use of maximally stressful IPv6 traffic patterns to measure throughput. To load up the control plane, engineers configured Spirent TestCenter to emulate 768 Multiprotocol-Border Gateway Protocol (MP-BGP) routing peers, each using a unique Autonomous System Number (ASN). Each Spirent MP-BGP router brought up a peering session with the H3C S12516X-AF switch, then advertised a total of 768 unique routes. The Spirent test tool then offered fully meshed IPv6 traffic between all networks learned using MP-BGP. A fully meshed pattern means each port exchanges traffic with all other ports. This is the most stressful traffic pattern for a switch fabric. Test engineers determined the throughput rate the highest speed at which the switch correctly forwarded all traffic, in sequence and without frame loss. Frame sizes ranged from 86 bytes to the Ethernet maximum of 1,518, and beyond to 9,216-byte jumbo frames. Engineers used a duration of 60 seconds for each test. For the IPv6 tests, test engineers used a minimum frame size of 86 bytes rather than 64 bytes to accommodate IPv6 s larger header size and the signature field added to each test frame by the Spirent test instrument. Note that the choice of 768 routes is due to a limit in the number of trackable receive streams supported by the Spirent dx3 test modules, and not of the H3C switch s routing capacity. Other Spirent test modules also support higher trackable stream counts. With the dx3 module, a higher route count also would have been possible using fewer than 768 ports. Page 12

What We Found In every test case involving frames of 102 bytes and larger, the H3C switch moved IPv6 traffic at the theoretical maximum rate to all 768 100-gigabit Ethernet ports, with zero frame loss. These tests involved traffic in a fully meshed pattern the most stressful possible load and MP-BGP routing on all ports. In some IPv6 tests, traffic moved so fast that the switch processed more than 100 million frames per second per port simultaneously, on each of 768 ports. Aggregate layer-1 throughput was nearly 77 terabits per second. Figure 4 presents results from the IPv6 with MP-BGP throughput tests. Although the industry-standard methodology calls for seven frame lengths to be tested, H3C also included two extra frame sizes. Engineers included 102- byte frames to demonstrate that theoretical maximum rates are possible with this relatively short frame size, not far up from the 86-byte minimum. Engineers also tested with 9,216-byte jumbo frames to demonstrate that the H3C switch can handle the large payloads commonly found in data-center applications. Figure 4: IPv6 throughput with MP-BGP routing, 768 100G Ethernet ports Page 13

Figure 5 compares throughput from the IPv4 and IPv6 tests with 102-byte frames and larger (so results are directly comparable). For both address families, throughput is line rate, with zero frame loss. Thus, there is no throughput penalty in moving from IPv4 to IPv6. Figure 5: IPv4 and IPv6 throughput compared, 768 100G Ethernet ports IPv6 Unicast Latency and Jitter With MP-BGP Routing Why It Matters: Latency and jitter can be even more important significant considerations than throughput with IPv6 traffic, given the longer frame size of the newer address family. Longer frames potentially means more time being cached and de-cached by the switch, with every extra bit of delay holding the potential to degrade application performance. Thus, a key question in migrating to IPv6 in the data center is whether core switches can deliver the same latency and jitter as for IPv4 traffic. How We Tested: Engineers measured latency and jitter at the same time as throughput, using the same test conditions: 768 100G Ethernet interfaces, with Spirent TestCenter offering traffic to all ports in a fully meshed pattern. Each Spirent test port emulated one MP-BGP peer and advertised 1 unique route. As required by RFCs 2544 and 5180, we measured latency and jitter at the throughput rate in all tests. Page 14

What We Found Average and maximum latency and jitter for IPv6 traffic are very slightly higher than for IPv4 traffic, typically by less than 1 microsecond for average latency with most frame sizes. The additional size of the IPv6 packet may help explain the slight increase. In addition, variations in latency and jitter are relatively small across most frame sizes. Table 5 presents results from the IPv6 with MP-BGP latency and jitter tests. With 128-byte and larger frames, latency and jitter are fairly consistent as payload size increases. Latency Jitter Frame size (bytes) Minimum (us) Average (us) Maximum (us) Average (us) Maximum (us) 86 1.780 19.483 399.470 4.613 379.520 102 1.820 8.328 81.820 0.300 59.940 128 1.810 4.836 7.090 0.021 2.350 256 1.820 4.753 6.780 0.018 2.200 512 1.830 4.725 6.710 0.015 2.050 1,024 1.820 4.814 6.840 0.011 2.030 1,280 1.810 4.789 6.720 0.009 2.020 1,518 1.790 4.841 6.930 0.009 2.430 9,216 1.920 5.342 8.180 0.018 3.560 Table 5: IPv6 with MP-BGP routing latency and jitter EVPN Scalability With VXLAN Tunnels Why It Matters: Ethernet virtual private networks (EVPNs) offer a powerful new method for creating Layer-2 overlay networks across MPLS and other types of backbones. Through use of VXLAN tunnels, EVPN simplifies network designs and operations for data center interconnect (DCI). Scalability is a key question in assessing EVPN implementations, specifically the number of concurrent VXLAN tunnels a device can support, and the performance of traffic flowing through EVPN-capable data center core switches. As far as Network Test is aware, this is the largest single-switch EVPN demonstration ever conducted. Page 15

How We Tested: The test bed modeled a scenario in which the H3C S12516X-AF connected 768 different sites, each using different EVPN instances and VXLAN tunnels. Figure 6 shows the EVPN/VXLAN test bed topology. Figure 6: The EVPN with VXLAN test bed In this configuration, 384 pairs of hosts communicated across EVPN tunnels set up using VXLAN and BGP routing. Although each host resided in a different Layer-3 IP subnet, the hosts reached one another across transports set up with VXLAN tunneling. Engineers configured a loopback interface on the H3C switch with the address 1.1.1.1, which served as the VX- LAN tunnel endpoint (VTEP) for all VXLAN tunnels. The switch also ran BGP, which specified the loopback address as the source for routing updates and brought up a BGP peering session with each VTEP advertised by the Spirent test instrument. After bringing up tunnels and BGP peers and advertising networks across the EVPN tunnels, engineers then configured the Spirent test instrument to offer bidirectional traffic streams between all hosts. Engineers measured throughput, latency, and jitter for small, medium, and large frame sizes, but omitted 64-byte frames due to the tunneling encapsulation overhead added by UDP and VXLAN headers. Page 16

What We Found The H3C data center core switch seamlessly set up EVPN connectivity among 768 different sites, and moved traffic at wire speed in all test cases with zero frame loss. Latency and jitter were low and consistent across frame sizes. Figure 7 presents throughput results from the EVPN with VXLAN scalability tests. Although EVPN and VXLAN are control-plane technologies, the tests results show no impact on data-plane forwarding capabilities. Figure 8: EVPN with VXLAN throughput Table 6 presents latency and jitter measurements from the EVPN tests. In fact, latency and jitter with EVPNs and VXLAN are virtually identical to tests that used BGP routing and 1 million routes, as described in the Routing Scalability Latency and Jitter With BGP routing section. Since both tests used port-pair traffic patterns, results are directly comparable. Table 7 illustrates the negligible differences in latency and jitter between the BGP routing and EVPN/VXLAN test cases. This figure presents differences between the two sets of measurements, in the form of EVPN measurements minus routing measurements. Page 17

Latency Table 6: EVPN with VXLAN latency and jitter Jitter Frame size (bytes) Minimum (us) Average (us) Maximum (us) Average (us) Maximum (us) 128 2.780 3.914 4.600 0.003 0.280 256 2.770 3.888 4.600 0.003 0.320 512 2.800 3.849 4.530 0.004 0.300 1,024 2.840 3.948 4.640 0.005 0.220 1,280 2.840 3.939 4.700 0.005 0.220 1,518 2.840 4.049 4.800 0.004 0.270 9,216 2.950 4.536 5.490 0.005 0.780 Frame size (bytes) Minimum delta (us) Latency Average delta (us) Maximum delta (us) Average delta (us) Jitter Maximum delta (us) 128 0.050 0.027 0.020-0.001-0.170 256 0.000 0.027 0.040-0.001-0.130 512 0.000 0.027 0.020-0.001-0.130 1,024 0.050 0.027 0.040 0.001-0.220 1,280 0.030 0.042 0.100 0.000-0.150 1,518 0.020 0.028 0.050-0.001-0.280 9,216 0.040 0.025 0.050 0.000-0.350 Table 7: EVPN with VXLAN and route scalability latency and jitter compared (deltas in microseconds) There are two items of note in comparing the two sets of results. First, the differences with the routing scalability tests are minuscule less than 100 nanoseconds in most cases. Second, the EVPN and VXLAN results show significantly lower maximum jitter than BGP routing, likely because routing control-plane messages add extra overhead. ISSU/ISSD Failover Times Why It Matters: Downtime is not an option in modern data centers, especially for core switches handling traffic that may represent vast sums and life-saving data. It s also important to keep switch software up-to-date as security patches and new features become available. To balance these requirements, equipment makers offer in-service software upgrades (ISSU), updating software via redundant modules with little or no change visible to users. Equally important, however, is the ability to downgrade software versions via in-service software downgrade (ISSD). Downgrades are necessary for a variety of reasons: Orchestration software might need to put all units on the same version of software. Bugs and/or security flaws in software may occur. And network engineers may try an upgrade temporarily to see if a new feature works as expected. For both upgrades and downgrades, the same question applies: What impact will ISSU or ISSD have on user traffic? Page 18

How We Tested: This test bed used two H3C S12516X-AF data center core switches, each running an alpha version of Comware, to be upgraded to a released version of Comware. This test involved two core switches; a future software release will support ISSU and ISSD within a single switch chassis with redundant supervisor modules. Figure 9 illustrates the test bed topology. H3C s Intelligent Resilient Framework (IRF) technology connected the two core switches and maintained control-plane state between them; with IRF, the two core switches appeared to the rest of the network as a single logical switch. Two other H3C switches redundantly connected to each core switch. And the Spirent test instrument attached to each of these switches, emulating hosts sending traffic across the data center backbone. Engineers configured the switches with two sets of VLANs and configured traffic flows within and across VLAN boundaries. The use of separate Layer-2 and Layer-3 flows would determine whether ISSU and ISSD had any impact when switching and routing traffic. Figure 9: The ISSU/ISSD test bed Page 19

Test engineers configured Spirent TestCenter to offer test traffic continuously throughout the test, and then initiated a software upgrade on one of the two core switches. The switch to be upgraded, previously designated at the master, went into backup mode, with the other switch taking over as master. After the upgrade completed, engineers stopped the test instrument and noted any frame loss. Since engineers offered traffic at 1 million frames per second in each direction, it was possible to derive failover time from frame loss, with one lost frame equal to 1 microsecond. Using the display version command, engineers also verified that the software upgrade was complete. Engineers then repeated the same test with a software downgrade back to the original alpha image. Again, engineers derived failover time from frame loss, and verified that software versions had changed. What We Found For both ISSU and ISSD, there was zero frame loss on at least two out of four links used in this test. Frame loss did exist on the remaining links, but in minuscule amounts equivalent to 20 and 28 microseconds for the ISSU test and 1.8 milliseconds for the ISSD test. Thus, in both ISSU and ISSD tests, failover was less than 2 milliseconds, but only on some links, with no service interruption for users on other links. Table 8 presents results from the ISSU and ISSD test cases. Test case Traffic direction Switched (L2) or routed (L3) Table 8: ISSU and ISSD failover times Failover time (usec) ISSU North -> South L2 20 ISSU North -> South L3 0 ISSU South -> North L2 28 ISSU South -> North L3 0 ISSD North -> South L2 0 ISSD North -> South L3 0 ISSD South -> North L2 1,880 ISSD South -> North L3 0 Page 20

Conclusion With this largest-ever test of 100G Ethernet networking, the H3C S12516X-AF sets a new high-water mark for data-center core networking. In an extensive set of stressful benchmark tests, the S12516X-AF pumped traffic through 768 100G Ethernet interfaces and set several records along the way: Line-rate, lossless performance for IPv4 and IPv6 traffic at nearly all frame sizes using BGP routing and fully meshed traffic (the most stressful possible test pattern) Line-rate, lossless performance using BGP routing to nearly 1 million unique routes Virtually no difference in IPv4 and IPv6 performance, so no cost for IPv6 migration Record-high scalability for EVPN, with 768 concurrent tunnels established using VXLAN and line-rate, zero-loss performance for every frame size Virtually no difference in latency and jitter between BGP routing and EVPN/VXLAN test cases Minimal to no impact on user traffic during ISSU and ISSD software upgrades and downgrades As these test results demonstrate, the H3C S12516X-AF is a highly capable performer even under the most demanding conditions. Such high performance on such an unprecedented scale offers a measure of future proofing for tomorrow s data center networks. Data centers will continue to grow ever larger; as these test results demonstrate, the H3C S12516X-AF is well positioned to serve as the engine of that growth. Page 21

Appendix A: About Network Test Network Test is an independent third-party test lab and engineering services consultancy. Our core competencies are performance, security, and conformance assessment of networking equipment and live networks. Our clients include equipment manufacturers, large enterprises, service providers, industry consortia, and trade publications. Appendix B: Hardware and Software Releases Tested This appendix describes the software versions used on the test bed. Network Test conducted all benchmarks in May 2017 at H3C s labs in Beijing, China. Component H3C S12516X-AF Version Spirent TestCenter 4.73.0706.0000 Appendix C: Disclaimer Comware 7.1.070, Feature 2702; Comware 7.1.070, Alpha 0718 (ISSU/ISSD tests only) Network Test Inc. has made every attempt to ensure that all test procedures were conducted with the utmost precision and accuracy, but acknowledges that errors do occur. Network Test Inc. shall not be held liable for damages which may result for the use of information contained in this document. All trademarks mentioned in this document are property of their respective owners. Version 2017060100. Copyright 2017 Network Test Inc. All rights reserved. Network Test Inc. 31324 Via Colinas, Suite 113 Westlake Village, CA 91362-6761 USA +1-818-889-0011 http://networktest.com info@networktest.com