Datacenter Network Innovation. Freek Dijkstra, SURFsara Colloquium UvA SNE 15 November 2017

Transcription

1 Datacenter Network Innovation Freek Dijkstra, SURFsara Colloquium UvA SNE 15 November 2017

2 Outline Cabling Also: TCP Performance OpenCompute Clos Network Bare Metal Switches Network Function VirtualisationTraffic Engineering Fabric Networks & Overlays Future Trends Named Data Networking

3 1. Cabling Picture: Yuri Demchenko

4 Shannon-Hartley Limit 10 3 Gb/s in 50 GHz Not Possible 200 Gb/s Gb/s 40 Gb/s Gb/s OSNR in 0.1 nm [db] Source: Inder Monga 30 40

5 More cables (or wavelengths) Source: The Fiber Optics Association

6 More bits in a baud How? Suggestions? Source: Optical Engineering, Vol 52, Issue 11 (2013)

7 Dispersion

8 Latency Original cable Buried fiber-optic cable Mid-1980s ~ 1600 km RTT 14.5 ms Spread Networks Buried fiber-optic cable August km RTT 13.1 ms Source: McKay Brothers Microwave beam through air 4 July km RTT 9 ms Tradeworx Microwave beam through air Winter km RTT 8.5 ms

9 2. TCP Performance

10 c-latency Causes: 1. TCP transfer 2. DNS 3. Requestresponse 4. TCP handshake 5. Router Source: Ankit Singla Towards a Speed of Light Internet (2015)

11 TCP Initial Window 400 NumberofRoundTripstoDeliver NSegments Transfered Data 500 kb kb kb kb 93 Segments kb kb 1 2 Source: John Rauser TCP and the Lower Bound of TCP Performance (2010) Round Trips 6 7

12 TCP Initial Window with Selective Acknowledgements (SACK) 400 NumberofRoundTripstoDeliver NSegments Transfered Data 500 kb kb Segments 300 kb 200 kb RFC 6928 increases initial window (IW) from 2-4 to 10 packets kb kb 1 2 Source: John Rauser TCP and the Lower Bound of TCP Performance (2010) Round Trips 6 7

13 TCP Congestion Control Sending Rate(cwd) Additive increase Multiplicative decrease Slow start Transmission number See RFC 5681

14 TCP Recovery Times Capacity RTT MTU Window Recovery Time 100 Mb/s 2ms 1500 bytes 17 packets 17 ms 1 Gb/s 24ms 9000 bytes 333 pkts 4s 1 Gb/s 182ms 1500 bytes pkts 23 min 10 Gb/s 262ms 9000 bytes pkts 1hr 19 mins Mathis equation: rate< MSS 1 * RTT errorrate Source: Matthew Mathis - The Macroscopic Behavior of the TCP Congestion Avoidance Algorithm (1997)

15 TCP Congestion Measurements Throughput vs. increasing latency on a 10Gb/s link with % packet loss No packet loss Reno (theory & measured) H-TCP (measured) Source: Alternative congestion control algorithms must be TCP-friendly. (UDP is not!)

16 TCP Incast Source: Philip Brighten Godfrey and Ankit Singla

17 Bufferbloat

18 Ideal Buffer Size? Large? Small? High throughput: buffers absorb bursts Low latency: large, full buffers cause delays Source: Philip Brighten Godfrey and Ankit Singla

19 DCTCP, PCC TCP DCTCP+ Mean 4.01 ms ms Median 4.06 ms ms Maximum 4.20 ms ms Minimum 3.32 ms ms σ ms ms Per-packet latency Source: Glenn Judd - Attaining the Promise and Avoiding the Pitfalls of TCP in the Datacenter (2015)

20 Explicit Congestion Notification Treshold Buffer 3. The sender reduces the TCP Congestion Window in half each window an ACK with ECN echo bit set was received. Actual details: RFC The switch sets the Congestion Encountered bits of a packet when buffer size exceeds threshold 2. The receiver sets the ECN Echo bit in the ACK packets.

21 DCTCP Treshold Buffer 3. The sender reduces the TCP Congestion Window proportional to the amount of ACKs with ECN echo bit set. Source: Mohammad Alizadeh (Microsoft) et alli Datacenter TCP (2010) 1. The switch sets the Congestion Encountered bits of a packet when buffer size exceeds threshold 2. The receiver sets the ECN Echo bit in the ACK packets.

22 3. Bare Metal Switches a.k.a. White Label Switches, a.k.a. Merchant Silicon Switches

23 IBM PC Compatible Vendor OS DEC VAX VMS Sun Solaris HP HP-UX Vendor Network OS IBM A/IX Cisco ios, NX-OS Apple ][ Integer BASIC Juniper JunOS Tandy TRSDOS Arista EOS Commodore KERNAL Dell DNOS, OS10 Prior to generic OSes

24 Bare Metal ( White Label ) Switches

25 Merchant Silicon Switches ASICs (Network Vendors Operating Systems Edgecore (with ONIE bootloader ) Processors) Broadcom Apollo2 (1G) Quanta (QCT) Broadcom Firebolt3 Wiwynn Cumulus Linux Broadcom Helix4 Agema Broadcom Triumph2 Penguin Big Switch Network Switch Light OS Broadcom Trident2 Dell Broadcom Tomahawk2 HPE Broadcom Maverick Mellanox Broadcom Jericho Juniper (some) Broadcom Qumran Huawei Mellanox Spectrum Arista Cavium XPliant Cisco Pica8 PicOS Dell OS 10 IP Infusion OcNOS Pluribus Networks Open Netvisor Linux Microsoft SONiC Microsoft Azure Cloud Switch (ACS)

26 Many options to choose from! Open networking vendors Quanta (QCT) Wiwynn Edgecore Corsa Barefoot Dell Mellanox Penguin Pica8 HPE Open networking OSes Big Switch Cumulus Linux Dell OS 10 MLNX-OS IP Infusion OcNOS Pluribus Microsoft SONiC Linux with switchdev Network Processors Broadcom Firebolt Broadcom Helix Broadcom Triumph Broadcom Tomahawk Broadcom Trident Broadcom Maverick Broadcom Jericho Broadcom Qumran Mellanox Spectrum Cavium Xpliant

27 Cumulus Linux Linux on a switch No more vendor proprietary CLI Install your own software when desired Manageable using many configuration management systems Makes use of switchd kernel driver for communicating with Broadcom ASIC

28 Port configuration Cumulus vs Cisco

29 Network management approach Northbound interface Management plane Control plane Data plane API Network Controller API

30 Forget SDN controllers, Think Configuration Management NCS Network Controller Managed Network Devices

31 Forget SDN controllers, Think Configuration Management P4 OpenFlow NETCONF Network Controller REST Managed Network Devices

32 Forget SDN controllers, Think Configuration Management Network Controller Managed Network Devices

33 Forget SDN controllers, Think Configuration Management Configuration Manager Linux Cluster

34 Forget SDN controllers, Think Configuration Management Configuration Manager Linux Cluster

35 Forget SDN controllers, Think Configuration Management Ansible Configuration Manager SSH NETCONF REST Managed Network Devices

36 Forget SDN controllers, Think Configuration Management API? Ansible Configuration Manager Managed Network Devices

37 Switch features in Cumulus Linux Switching feature Linux App / Daemon L2 switching switchd L2 bridging (VLANs) Linux bridge brctl routing Free range Routing / Quagga Access lists iptables snmp Net-snmp syslog rsyslog MLAG (Multi-chassis LACP) clagctl LLDP (link discovery) Lldpd Authentication LDAP (PAM) / tacplus

38 Workflow at SURFsara Zero Touch Provisioning (ZTP) workflow: MAC in CMDB Generate DHCP Generate DNS Generate monitoring switch in rack ONIE install ZTP script Ansible playbook

39 How does SURFsara use bare metal switches? Open networking devices in SURFsara 120% 100% 80% 57% 45% 30% 60% 100% 99% 96% 40% 43% 20% 0% 0% % % 2015 Open networking vendors % 2017 Legacy vendors 70% 2018

40 4. Datacenter Fabrics

41 Clos Network

42 Traditional Spine-Leaf Architecture External Core Spine (aggregation) Leaf (access)

43 Fat Tree Fabric Source: Mohammed Al-Fares A Scalable, Commodity Data Center Network Architecture (2008)

44 Complex Fabric Example Source: (2014)

45 Location Independence Without overlay, the IP of a VM is locked to a topological location Source: Philip Brighten Godfrey and Ankit Singla

46 Naming and addressing The name of a resource indicates what we seek, an address indicates where it is, and a route tells us how to get there Source: John F. Shoch Inter-Network Naming, Addressing, and Routing, IEN#19 (1978)

47 Overlay Network Typically L2 over L3 E.g. vxlan with EVPN Integrated with cloud management software (e.g. OpenStack) Routing questions - Where VTAP? In switch, or hypervisor? - How to deal with BUM traffic?

48 VXLAN OSI layer 2 encapsulation in UDP

49 Two Earlier Examples Microsoft VL2 (2009) Anycast IP of Interconnect IP of ToR device (address) IP of VM (name) Data Azure SmartNIC Source: Albert Greenberg (Microsoft) SDN for the Cloud Facebook (2014) No Layer 2 overlay, IP only Use EBGP within the datacenter because it is known to scale RFC 7983 Use of BGP for Routing in Large-Scale Data Centers

50 5. Future Trends

51 The Future? Automate A Apple, Facebook, Google, Microsoft Look at the big five: Amazon, u e t a to A m o t u u A tom te e m t a Intent engine m a o a t ate u m A te Automatic rollback o t u e Network: An Evolution E.g. The Zero A Touch of SDN at Google by Bikash t a m o Koley. t e t u a A m o t A u u t A om a t e A u to Automate m at e e t A a u t o m m e a t t o e a t m u o A Automat Au t e Automate to m a om Au Aut at e Automate te

52 Network Function Virtualisation (NFV) Source: Dave Neary NFV for beginners

53 Multi-path Traffic Engineering How to get from A to B, avoiding congestion

54 Multi-path Traffic Engineering Centralized Distributed e.g. Hedera, B4, SWAN Slow to react for DCs In Network Local (Stateless or Congestion-Aware) Host-Based e.g. MPTCP Hard to deploy, increases incast Global, Congestion-Aware e.g. ECMP, Flare, LocalFlow, LetFlow Poor with asymmetry, especially with TCP traffic LetFlow (2017) is promising Leaf to Leaf General, Link State Near-optimal for 2-tier, deployable with an overlay Complex, hard to deploy Per Flow Per Flowlet Per Packet e.g. round-robin Suboptimal for heavy flow distributions (with large flows) e.g. CONGA No TCP modifications, resilient to flow distribution Optimal, needs reordering-resilient TCP Source: Mohammad Alizadeh (Microsoft) et al. CONGA: Distributed Congestion-Aware Load Balancing for Datacenters (2014)

55 Programmable Switches Source: Changhoon Kim (Programming the Network Dataplane in P4)

56 OCP Hardware: Racks, Servers, Switches Source: Wiwynn Tioga at opencompute.org Marketplace

57 6. Named Data Networking

58 Content Centric Networking (CCN) aka Named Data Networking (NDN) Telephony (1876) by Alexander Graham Bell et al. System creates circuits But users want to communicate Packet switching (1965) Content Centric by Leonard Kleinrock et al. Networking (2007) by Van Jacobson et al. System allows communications Systems allows data CoreIdea CoreIdea retrieval Today s networks aretobased Modern communication consi Today s networks are based on on Modern communication consists But users want get host-to-host connections of for named host-to-host connections of requests for named data data data ( , movie,...) Just asrequests users want Source: Van Jacobson A New Way to look at Networking (2006) InterestInterest data data NDN:NDN: a general-purpose network a general-purpose network protocol supporting requests for named protocol supporting requests for n

59 IP and NDN Hourglass Source: Lixia Zhang, Van Jacobson, et alli Named Data Networking (2014)

60 NDN Interest Forwarding TwoPacketTypes Data packet Interest packet Interest kettypesdataname Data name meta information (maycarryafew opuonal parameters) Data Data packet Crypto signature Data name meta information Publisherbindsnametodata; receiversverify Data No address! Crypto signature NDN Tutorial ACM ICN 2016 Data data; receiversverify CM ICN Source: /ucla.edu/lixia/talks/icn16-arch.pptx 12

61 Freek Dijkstra