Fast Lanes and Speed Bumps on the Road To Higher-Performance, Higher-Speed Networking

Transcription

1 Fast Lanes and Speed Bumps on the Road To Higher-Performance, Higher-Speed Networking Bruce Tolley Linux Foundation End User Summit, 24 June 2014 Copyright 2013 Solarflare Communica9ons, Inc. All rights reserved. June

2 Abstract With Ethernet server adapters running at 40 gigabits per second, Linux users are still determining the best way to connect network servers to keep up with these speeds and maximize network performance. While the move from 10G to 40G sped up Ethernet 4X, the move from PCIe 2.0 to 3.0, just sped up the server bus 2 X. This presentation will look at the network interface and the server bus from a standards perspective, and present an overview of what is possible over the next 2 to 3 years to get the maximum throughput through servers with PCIe 2.x or 3.0 busses. Intel has not publicly communicated either the the dates or which chipsets/servers will support PCIe 4. We can make some predictions based on past experience with the time it takes from when the specification is complete to when we see adapter cards and servers implementing the next generation. Along the way we will briefly compare and contrast, multiple different method of driving innovation: The IEEE working group, an industry, multivendor driven standards development organization (SDO), the PCIe SIG, an industry consortium led by Intel and others, the Open Source Linux community, and vendor-specific hardware and software implementations June

3 Introduction Quick look backwards at 10Gb Ethernet Status of 40, 100, 400 Gb Ethernet, and PCIe 3.0 and 4.0 Examples of some proprietary point product innovations Observations and Analysis June

4 Higher Speed Networking: Time to Market Name Start (PAR) STD Approved Products 1000BASE- X BASE- T GBASE- R GBASE- T GBASE- R GBASE- R GBASE- T 2013 NA 2017e Lower cost 40/100G R NA 2017e 400GBASE NA 2018e Time to Market Notes: Product 9ming based on Dell Oro Group reported switch port shipments. 2017e and other es9mates are the opinion of the author, not Solarflare or the IEEE June

5 Why the Demand for Even Higher Speeds (Google s Network) Source: Google Presenta9on to Ethernet Alliance TEF June

6 State of the Higher Speed Ethernet (Switching Ports 1000s) 450,000.0# 400,000.0# 350,000.0# 300,000.0# 250,000.0# 200,000.0# 150,000.0# 100#Gbps# 40#Gbps# 10#Gbps# 1000#Mbps# 100,000.0# 50,000.0# 0.0# 2006# 2007# 2008# 2009# 2010# 2011# 2012# 2013# 2014# 2015# 2016# 2017# 2018# Source: Dell Oro Group June

7 What Can We Learn From 10 Gigabit Ethernet ,000.0 $40,000 30, , , , , , GbE: ASP vs. Unit Shipments $35,000 $30,000 $25,000 $20,000 $15,000 $10,000 $5,000 $0 Ports ASP Too many options Too expensive Too high power Google built its own connectivity solution Solution that dominates market today not even defined by IEEE Direct attached copper June

8 10G Ethernet: Multiplicity of Port Types and Technologies 10GBASE- SR Media Distance XENPAK X2 XFP SFP+ MM Fiber 300 to 400 m 10GBASE- LR SM Fiber 10 km 10GBASE- ER DAC 10GBASE- T 10GBASE- CX4 10GBASE- LRM 10GBASE- LX4 SM Fiber 40 km 2 pair Twinax Twisted pair 100 m 8 pair Twinax FDDI grade MM fiber FDDI grade MM/SM 15 km 15 m 220 m 300 m/ 10Km Pluggable Form factors: XENPAK, XPAK, X2, XFP, SFP+ June

9 Overview of 40 and 100G Options Port Type 40 Gigabit Ethernet 100 Gigabit Ethernet Backplane 40GBASE- KR4 100GBASE- KP4 100G Backplane 7 m over twinax copper cable 30 m over Cat 8 twisted pair 40GBASE- CR4 40GBASE- T 100GBASE- KR4 100GBASE- CR m over OM3 MMF 40GBASE- SR4 100GBASE- SR m over OM4 MMF 40GBASE- SR4 100GBASE- SR10 2 km over SMF, serial 40GBASE- FR 10 km over SMF 40GBASE- LR4 100GBASE- LR4 40 km over SMF 100GBASE- ER4 June

10 Cisco Trying Something Different: Bidirectional Optics to Support Installed Fiber Patch Panel Cisco QSFP BiDi 40Gbps Transceiver: Duplex MMF with LC Connectors at Both Ends hdp:// series- switches/white- paper- c html June

11 IEEE Gb/s Ethernet Study Group Objectives Support a MAC data rate of 400 Gb/s Support a BER of better than or equal to at the MAC/PLS service interface (or the frame loss ratio equivalent) Support full-duplex operation only Preserve the Ethernet frame format utilizing the Ethernet MAC Preserve minimum and maximum FrameSize of current Ethernet standard Provide appropriate support for OTN Specify optional Energy Efficient Ethernet (EEE) capability for 400Gb/s PHYs Support optional 400 Gb/s Attachment Unit Interfaces for chip- to-chip and chip-to-module applications Provide physical layer specifications which support link distances of: At least 100 m over MMF At least 500 m over SMF At least 2 km over SMF At least 10 km over SMF June

12 400 Gb Ethernet Options Fibers/Connectors (#) Lamdas (#) Data rate (Gbps) Symbols/bit (#) Trade offs: cost, power, density Source: John D Ambrosia, Dell June

13 The Other Side of the Ethernet Connection PCI Express 1a Signaling rate 2.5 GT/s 5 GT/s 8 GT/s 16 GT/s Encoding 8b/10b 8b/10b 128b/130/b 128/130b Encoding Efficiency Max BW per lane BW on x8 PCIe card Date Spec Finished 80% 80% 98.5% 98.5% 2 Gbps 4 Gbps 7.88 Gbps Gbps 16 Gbps 32 Gbps 63 Gbps 126 Gbps July 2002 Dec 2006 Nov 2010 Late 2015 Server support 2H2003 1H2008 End2011 Q June

14 Solarflare Flareon SFN7142Q 40GbE Server I/O Adapter Launched 19 June 2014 Availability: NOW Dual-port 40GbE QSFP+ or Quad-port 10GbE SFP+ Maximizes multi-core CPU efficiency OpenOnload application acceleration middleware to bypass kernel and networking overheads Certified with leading 40G swtiches from Arista and Cisco Red Hat Enterprise Linux, Red Hat Enterprise Linux Open Stack Platform, Red Hat Enterprise Virtualization, SUSE Linux Enterprise Server, VMware ESXi, Microsoft Windows, Microsoft Hyper-V Server June

15 What Can We Do Above the Network Layer: Kernel Bypass Performance like Infiniband But regular Ethernet Standard UDP/TCP Performance Very easy to use June

16 What Can We Do With Software Abstraction + Hardware Hypervisor Bypass with SR-IOV Each VM gets its own VNIC Now the VM kernel is in direct contact with the network hardware Gives you back most of the na9ve performance Some limita9ons June

17 Simultaneous Hypervisor and Kernel Bypass Technology preview- - not announcing product (today) Ultra- low latency Extremely fast message rates June

18 Kernel + Hypervisor Bypass -- 10GbE Performance June

19 Different Organizational Strategies for Driving Innovation Example (s) Type Advantage Legal SDO IEEE Closed user group Interoper ability SIG consor9um Open Source Proprietary single vendor PCIe SIG Linux OpenOnload,Ci sco 40G bidi Monopolis9c closed user group Intel leadership. Market 9ming, 2X Open user group Mul9ple innovators, contributors Disadvantage Slow Intel Mul9ple distros Time to market Single source June

20 June

21 Questions, Summary, and Conclusions Proprietary QSFP+ bidi modules Return to structured cabling 16 PCIe lanes for 100g 100GbE at the server edge Do standards organizations move fast enough? Virtualization June

22 END June

23 June