QFX GbE Fiber Optic Interface Interoperability Testing

Transcription

1 QFX GbE Fiber Optic Interface Interoperability Testing Tech Note October 2017 Juniper Networks, Inc Innovation Way Sunnyvale, California USA Copyright 2017, Juniper Networks, Inc. All rights reserved Juniper Networks, Inc.

2 Table of Contents Introduction... 3 Importance of Interoperability Testing... 3 Forward Error Correction (FEC)... 4 Technical Overview... 4 Implementation... 5 Configuration... 6 Verification... 6 Interoperability Testing Methodology... 7 Internal Test Results... 7 Optics/Cable Testing... 8 Interoperability Test Matrix... 8 University of New Hampshire Interoperability Lab Test Results... 9 Summary Appendix Test Methodology for Internal Testing Juniper Networks, Inc.

3 Introduction The explosive growth of cloud applications, video and mobile traffic is driving the demand for faster and faster server uplink speeds. The 10GbE sever access connectivity is getting saturated as operators add more workloads or VMs per server. Increasingly network architects are looking to the emerging 25GbE standard as a solution for their increased bandwidth requirements. A number of factors make 25GbE attractive to datacenter operators. The 25GbE standard doesn't require a new cable infrastructure; it runs over the same 10GbE cable installation. Compared to 40GbE, 25GbE is both more power efficient and cost effective. Since it s designed backward from 100GbE technology using 4x25GbE SERDES, it requires only 1x25GbE SERDES just like the 10GbE. Juniper Networks QFX5200 switches offer 10GbE, 25GbE, 40GbE, 50GbE, and 100GbE interfaces. Users can have 100GbE by default or can also configure the interfaces as 40GbE. 100GbE can be channelized to two independent 50GbE or to four independent 25GbE (the channels are often called lanes ). Similarly, 40GbE can be channelized to four independent 10GbE using breakout cables. One of the advantages of 25GbE is that 4-lane 40G NIC has the same price as 2-lane 25GbE NIC, which means customers get higher throughput for the same fiber count. Importance of Interoperability Testing Why is interoperability testing so important? Almost everything implemented today in networking is covered by some type of standard, either a commonly seen de facto standard or legally established de jure standard. For example, in the United States, locating the car steering wheel on the left or right is a de facto standard, and many postal vehicles locate the steering wheel on the right. However, in the United States, driving on the left side of the road is a de jure standard governed by law and a violation (and dangerous). If one vendor says my product complies with all relevant standards and another vendor says my product also complies with the same standards, then there should be no need to test for interoperability, right? Not necessarily. There are at least five reasons why a simple declaration of standards compliance is not a guarantee of interoperability. 1. Standards are interpreted This is always the number one reason for interoperability issues. Engineers ponder the text and often puzzle over the precise meaning of a term or condition. Sometimes different groups read and interpret things the same way, but often they do not. 2. Standards have options Options allow standards to apply in different situations. However, if a vendor applies Option Set A and another applies Option Set B, it s not always clear what they 2017 Juniper Networks, Inc. 3

4 have done (they often list their product as complying with the overall standard, and in general it does), or what might happen when they try to talk to each other. 3. Standards are incomplete Standards do not always cover every circumstance and application. Technology moves quickly, and many RFCs and ITU specifications include extensions or addenda to cover things that have come up once implementation begins. 4. Standards are wrong It s better to say standards have typos, or that long standards sometimes seem to have later sections that appear to contradict earlier ones (see Standards are interpreted above). Many RFCs and ITU specifications have errata attached the correct unintentional errors. 5. On purpose This does not imply that a vendor is trying to be difficult. Sometimes a vendor will purposely restrict operation to its own product groups until the standardized product is more mature. The vendor might also be trying out an extension to the standard that might be added to the standard at a later time, but which for now is not supported by other vendors. In a perfect world, all of the cases outlined above would have documentation and would be well known. But often they are not completely known to everyone. In fact, it is often interoperability testing that detects many of the conditions listed above in the first place. Put these reasons all together, and you see why interoperability testing is essential, especially for newer technologies. Forward Error Correction (FEC) Technical Overview As systems get faster signal integrity becomes critical, both internally and over transmission links. New 25GbE and 100GbE systems are a good example of this. Such systems are expected to have lower tolerance for latency and errors. There are many techniques that can be used to enhance data reliability. To correct for errors and protect the data integrity, FEC is used in 25Gbps and higher speeds to improve data transfer throughput without the need for re-transmitting the data. FEC is a digital pre- and post-processing technique that greatly reduces the number of uncorrected errors and helps to extend the reach capability of both Copper and Optical PMDs. There are a number of FEC modes, offering differing degrees of error detection and recovery. The IEEE 802.3by standard has defined Clause 91 with Reed-Solomon FEC (RS-FEC) for 25Gb Ethernet to support most of the copper and optical physical media dependent devices (PMDs). Clause 74 of the 100GbE standard specifies BASE-R FEC also known as fire-code FEC (Fire code has nothing to do with flames, but is named for Philip Fire, who described this method in 1959), and is available in most 100GbE capable switch silicon. FC-FEC offers a weaker error correction than RS-FECs, especially with random errors instead of bursts, but with lower latency compared to RS-FEC Juniper Networks, Inc.

5 To enable FEC on a communication link, the same type of FEC has to be configured and enabled on both ends of the link. Figure 1: FEC Pre- and Post-Processing. FEC was developed originally for situations where re-transmissions are expensive or impossible, such as broadcast and multicast communications or for deep-space probes. As the error detecting and correcting capabilities of FEC increase with faster and more powerful pre- and post-processing, data transmission over noisy channels became more reliable and all but error-free. This use of FEC leads to improved bandwidth efficiency, lower costs, and higher achievable data rates. In fact, without FEC, many higher-rate line speeds would be impossible to sustain. Implementation Starting with Junos OS Release 15.1X53-D230, forwarding error correction (FEC) is supported on QFX5200 switches; users can disable or re-enable FEC clause 74 (CL74) as well as FEC clause 91 (CL91) using CLI commands. FEC CL74 is supported for 25Gbps port configuration, when the ports are channelized either in 4x25GbE or 2x50GbE, FEC CL74 is enabled. FEC CL91 is supported for the 100GbE configuration and enabled by default Juniper Networks, Inc. 5

6 Configuration To disable FEC: set interfaces <interface-name> gigether-options fec none To enable FEC: set interfaces <interface-name> gigether-options fec <fec91/fec74> or delete interfaces <interface-name> gigether-options fec none Verification To check FEC status: show interfaces et-0/0/0 Physical interface: et-0/0/0, Enabled, Physical link is Up Interface index: 649, SNMP ifindex: 505 Link-level type: Ethernet, MTU: 1514, LAN-PHY mode, Speed: 40Gbps, BPDU Error: None, Loop Detect PDU Error: None, Ethernet-Switching Error: None, Source filtering: Disabled Ethernet-Switching Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Flow control: Disabled, Media type: Fiber Device flags : Present Running Interface flags: SNMP-Traps Internal: 0x4000 Link flags : None CoS queues : 10 supported, 10 maximum usable queues Current address: 5c:45:27:37:55:c3, Hardware address: 5c:45:27:37:55:c3 Last flapped : :14:22 PDT (00:00:57 ago) Input rate : 0 bps (0 pps) Output rate : 0 bps (0 pps) Active alarms : None Active defects : None PCS statistics Seconds Bit errors 0 Errored blocks 0 Ethernet FEC statistics Errors FEC Corrected Errors 0 FEC Uncorrected Errors 0 FEC Corrected Errors Rate 0 FEC Uncorrected Errors Rate 0 PRBS Statistics : Disabled Interface transmit statistics: Disabled Logical interface et-0/0/ (Index 554) (SNMP ifindex 506) Flags: Up 0x Encapsulation: ENET2 Input packets : 0 Output packets: Juniper Networks, Inc.

7 Interoperability Testing Methodology To ensure that Juniper products are deployable in customer environments, extensive testing is performed on a set of critical devices. This internal testing focuses on network interface cards and cables that are commonly deployed with the QFX5200. A second level of interoperability testing is done through our participation in the University of New Hampshire Interoperability Labs and their 25GbE/100GbE test events. This industry wide interoperability event tests the QFX5200 with devices and cables from a number of other vendors. Details of the test methodology are presented in the appendix. Internal Test Results These are the NIC cards that were used during the test: 1. QLogic FastLinQ QL45212-DE 25GbE Adapter The QLogic FastLinQ QL45212-DE Series Intelligent Ethernet 25GbE SFP28 Adapters has dual ports. It supports standard MTU size and jumbo frames up to 9600 Bytes. It also supports VXLAN, NVGRE and L2/l3 tunneling options Mellanox connect-4 Lx Ethernet Adapter Mellanox adapter has dual 10/25/40/50 Gbs ports. PCIe Gen 3.0 compliant and supports Passive copper cable with ESD protection. It supports SR-IOV, 1K ingress and egress QOS level Broadcom NetXtreme E-Series Dual Port 25GbE SFP28 Ethernet PCIe Adapter Broadcom Adapter has Dual SFP28 interfaces - compliant to SFF-8402, IEEE 802.3by and 25G Ethernet Consortium. It supports SRIOV with 128 VF. Port-25GbE.pdf 2017 Juniper Networks, Inc. 7

8 4. Netronome Smart NIC Adapters We tested the QFX5200 switch with Agilio CX 2x25GbE. Netronome Agilio CX 10GbE, 25GbE and 40GbE standard low-profile PCIe SmartNICs are designed for general-purpose x86 commercial off-theshelf (COTS) rack servers, fitting needed power and form factor requirements. Optics/Cable Testing QSFP-100G-SR4 with breakout cables QSFP-100G-LR4 with breakout cables DAC JNP-100G-4x25G-1M DAC JNP-100G-4x25G-3M DAC JNP-100G-4x25G-5M (The DAC Vendors are Leoni and Amphenol) Interoperability Test Matrix QLogic FastLinQ Mellanox connect-4 Lx Broadcom NetXtreme Netronome Smart NIC Ping test 25G Channelized Default FEC Channelized disable CL74 Not Tested Optics Hot Swap Not Tested Fiber pull Not Tested File transfer Juniper Networks, Inc.

9 Attenuation using VOA 25G FEC with daemon restart Pre-Provisioned 25G/50G FEC configuration Not Tested Not Tested Not Tested MTU settings Not Tested RFC 2544 test 100G/25G speed toggle test Link Speed mismatch Not Tested Not Tested FEC mismatch Note: A check mark ( ) means the configuration passed the interoperability test. Some tests were not performed on the Netronome product due to availability. University of New Hampshire Interoperability Lab Test Results In May 2017, the UNH IoL lab held an open industry wide interoperability test event for vendors with 25GbE and 100GbE solutions. The tests were matrixed between different vendors and different cables and exercised the various FEC modes for the types of links. The Juniper QFX5200 was tested for interoperability with the following third party 25GbE solutions: Marvell X5113 Huawei SmartIO-2 Arista 7280SR2-48YC6 Arista 7060CX2-32S Arista AR-ALH Intel FVL-25 For additional test data details, please contact your Juniper Networks representative Juniper Networks, Inc. 9

10 Summary In conclusion, all tested configurations were interoperable, showing the stability and the reliability of the Juniper Networks devices 25GbE interfaces. Appendix Test Methodology for Internal Testing 1. Ping test a. Self-interface Ping i. Connect two QFX5200 back-to-back and ping 100 packets. b. Back-to-back ping verification i. Connect QFX GbE interface to a server NIC and ping 100 packets GbE Channelized Default FEC settings i. Configure QFX GbE interface with IPv4. ii. Run traffic and verify the stats. 3. Base-R FEC (25GbE) a. Channelized disable CL74 settings i. Configure QFX GbE interface with IPv4. ii. Disable the FEC and verify the CLI. b. Optics Hot Swap i. Configure QFX GbE interface with IPv4. ii. Hot Swap the optics and verify FEC settings. Make sure the FEC is still enabled. iii. Disable FEC and repeat the above test. c. Fiber pull test i. Configure 25GbE interface with IPv4. ii. Unplug and plug fiber back in and verify FEC settings. iii. Disable FEC and repeat the test. d. File transfer to server i. Configure 25GbE interface with IPv4. ii. Initiate file transfer between Server and QFX5200 with FEC enabled. e. Introduce some attenuation using Variable Optical Attenuator (VOA) i. Configure 25GbE interface with IPv4. ii. Attenuate the signal using VOA Juniper Networks, Inc.

11 iii. Test the link signal quality against the fixed increment of attenuation. iv. Repeat the test without FEC enabled and make sure link goes down sooner compared to previous case. f. 25GbE FEC with daemon restart i. Configure 25GbE interface with IPv4. ii. Restart various daemons like chassisd/routing daemon/l2 learning daemon. Make sure FEC settings are not changed after daemon restart. iii. Disable FEC and repeat the above tests. g. Pre-Provisioned 25GbE/50GbE port in channelized mode with default/non-default FEC settings i. After Bring up 25GbE/50GbE port channelized to 4x25GbE/2x50GbE verify default FEC settings. ii. Hot Swap the optics and verify FEC settings. Make sure the FEC is still enabled. iii. Disable FEC and repeat the above tests. h. Different MTU settings i. Configure QFX GbE/50GbE interface with IPv4. ii. Configure different MTU sizes and run traffic. iii. Make sure there isn t any packet loss. i. RFC 2544 test i. Configure QFX GbE/50GbE interface with IPv4. ii. Configure Spirent to run RFC 2544 test. j. 100GbE/25GbE speed toggle test with FEC enabled/disabled i. Bring up 25GbE/50GbE port channelized to 4x25GbE/2x50GbE with default FEC settings. ii. Change speed to 100Gbps and make sure FEC changes to CL91. iii. Again, change channelized port to 25GbE and check FEC (CL74). k. Speed mismatch on both ends i. Bring up near end port in 25GbE/50GbE port channelized and far end in 100GbE mode with default FEC settings. ii. Verify link is down. iii. Repeat the test with FEC disabled and speed change on both sides to cause mismatch. l. FEC mismatch 2017 Juniper Networks, Inc. 11

12 i. Bring up near end port in 25GbE/50GbE port channelized default FEC settings. ii. Disable CL74 FEC on one side to cause FEC mismatch. iii. Verify link goes down. iv. Repeat the test with 100GbE mode for CL91 FEC Juniper Networks, Inc.