High Performance Cloud-native Networking K8s Unleashing FD.io

Size: px

Start display at page:

Download "High Performance Cloud-native Networking K8s Unleashing FD.io"

Imogen Wilkins
5 years ago
Views:

1 High Performance Cloud-native Networking K8s Unleashing FD.io Giles Heron Principal Engineer, Cisco Maciek Konstantynowicz FD.io CSIT Project Lead Distinguished Engineer, Cisco Jerome Tollet Distinguished Engineer, Cisco

2 DISCLAIMERs 'Mileage May Vary' Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your opinion and investment of any resources. For more complete information about open source performance and benchmark results referred in this material, visit and/or Trademarks and Branding This is an open-source material. Commercial names and brands may be claimed as the property of others.

3 Internet Mega Trends.. Portability and Efficiency Scalability and Self-Healing Cloud Software Defined Networking Cloud Native Designs NFV SDN Open Source Platforms

4 THE SOFTWARE DEFINED OPERATOR DO YOU REMEMBER..? IP NFV SDN Deutsche Telekom AG, NOV-203 4

5 Pillars of Next Generation Software Data Planes Blazingly Fast Process the massive explosion

innovation in cloud-native networking No compromise on performance (zero-tolerance) Software

Predictable performance Dataplane performance must be deterministic Predictable for a number of

to count everything for detailed cross-layer operation and efficiency monitoring Enables

5 5 Pillars of Next Generation Software Data Planes Blazingly Fast Process the massive explosion of East-West traffic Process increasing North-South traffic Truly Extensible Foster pace of innovation in cloud-native networking No compromise on performance (zero-tolerance) Software First Cloud means running everywhere Cloud means hardware and physical infra agnostic Predictable performance Dataplane performance must be deterministic Predictable for a number of VMs, Containers, virtual topology and (E-W, N-S) traffic matrix Measureable Counters everywhere to count everything for detailed cross-layer operation and efficiency monitoring Enables feedback loop to drive optimizations FD.io VPP meets these challenges How can one use it in large scale Cloud-native networks?

6 The Way Applications Are Developed and Deployed.. Has Changed.. Time

7 The Way Networks are Deployed and Used has Changed Corporate LAN/WAN 80:20 rule Internet Internet exchanges & public peering Data-Center Core/Distr/Access, VLAN based LAN WAN Network Network 2 Network 3 VLAN n VLAN 2 VLAN Time Intranets & Internet LAN WAN WWW Tiered Transit & Private Peering Tier A Tier C Tier B Spine/Leaves & L3 Core/L2 Access VLAN 2 VLAN Core VLAN 2 Internet Tier2 D Tier2 E Tier2 F SD-WAN & BeyondCorp Cloud Provider Telco/Cable Access &, OTT/CDN Content L3 Fabric/SW Overlay & Virt Networking WiFi Internet Cloud Cloud Provider Provider Global 2 Global Backbone Backbone Layer 3 Fabric Telco Telco 2 MSO

8 Aside: A Trip Down Memory Lane (Transporting Data vs. Processing Data) Year 202 Internet service provider comment at IETF: processing bits is cheaper than transporting bits, computing and networking - networking is becoming st order citizen on compute platforms. Year 203 RIPE67 Terastream been fixing the cost of transporting bits - 96 of 00GE coherent lambdas per fibre span - transporting is getting cheaper, so challenging the compute part again more bandwidth delivered to Data Centres Most/all network services in Data Dentres NFV SDN IP

9 Remember 965 Moore s Law.. Moore s Law Electromechanical Solidstate Vacuum Tube Transistor Relay Integrated Circuit 0 6 Calculations per Second per $, Source: Ray Kurzweil, "The Singularity Is Near: When Humans Transcend Biology", Page 67, The Duckworth Publishers Data points between 600 and 900, and after 2000 represent Maciek's perspective and approximations.

10 Remember 965 Moore s Law.. Popular pre 900 Moore s Law 965 Mechanical Electromechanical Solidstate Vacuum Tube Transistor Relay Integrated Circuit 0 6 Calculations per Second per $, Binary Code Boolean Logic Source: Ray Kurzweil, "The Singularity Is Near: When Humans Transcend Biology", Page 67, The Duckworth Publishers Data points between 600 and 900, and after 2000 represent Maciek's perspective and approximations.

11 Remember 965 Moore s Law Is It Still Applicable? Popular pre 900 Moore s Law 965 Mechanical Electromechanical Solidstate Vacuum Tube Transistor Relay Integrated Circuit 0 6 Calculations per Second per $, Binary Code Boolean Logic Source: Ray Kurzweil, "The Singularity Is Near: When Humans Transcend Biology", Page 67, The Duckworth Publishers Data points between 600 and 900, and after 2000 represent Maciek's perspective and approximations.

12 Remember 965 Moore s Law Is It Still Applicable? Popular pre 900 Moore s Law 965 Popular in 207 Xeon Skylake Mechanical Electromechanical Solidstate Vacuum Tube Transistor Relay Integrated Circuit Modern µprocessor 0 6 Calculations per Second per $, Binary Code Boolean Logic Era of Modern µprocessors Source: Ray Kurzweil, "The Singularity Is Near: When Humans Transcend Biology", Page 67, The Duckworth Publishers Data points between 600 and 900, and after 2000 represent Maciek's perspective and approximations.

13 Remember 965 Moore s Law Yes, It Surely Is.. Ramble On.. Popular pre 900 Moore s Law 965 Popular in 207 Xeon Skylake Mechanical Electromechanical Solidstate Vacuum Tube Transistor Relay Integrated Circuit Modern µprocessor 0 6 Calculations per Second per $, Binary Code Boolean Logic Era of Modern µprocessors Source: Ray Kurzweil, "The Singularity Is Near: When Humans Transcend Biology", Page 67, The Duckworth Publishers Data points between 600 and 900, and after 2000 represent Maciek's perspective and approximations.

14 Processing Packets: How to Use Compute.. Resources to Get Performance SATA B I O S PCH DMI Socket 0 Skylake Server CPU x6 x6 x6 00GE 00GE 00GE ETH frames CyclesPerPacket [ClockCycles] = #FGHIJKLIMNGH OPLQRI Th#$%&h'%( [''+] = 2 UPI UPI UPI 4 #STLURH FGHIJKLIMNG Socket Skylake Server CPU x6 x6 x6 00GE 00GE 00GE ETH frames V OPLQRI_OJNLRHHMGW_XMYR[HRL ] = SOZ_[JR\[]^] STLURH RJ_OPLQRI 2 ) Processor and CPU cores for performing packet processing operations _ 2) 2 Memory bandwidth for moving data (packets, lookup) and instructions (packet processing) 3) I/O bandwidth 3 for moving packets to/from NIC interfaces _ 4) Inter-socket bandwidth 4 for handling inter-socket operations _ Th#$%&h'%( [*'+] = Th#$%&h'%( ''+ /023(_5673[''+]

15 Processing Packets: What Improves in Compute.. Resources to Get Performance SATA B I O S PCH DMI Socket 0 Skylake Server CPU x6 x6 x6 00GE 00GE 00GE ETH frames CyclesPerPacket [ClockCycles] = #FGHIJKLIMNGH OPLQRI Th#$%&h'%( [''+] = 2 UPI UPI UPI 4 Socket Skylake Server CPU x6 Th#$%&h'%( [*'+] = Th#$%&h'%( ''+ /023(_5673[''+] x6 x6 00GE 00GE 00GE ETH frames #STLURH FGHIJKLIMNG V OPLQRI_OJNLRHHMGW_XMYR[HRL ] = SOZ_[JR\[]^ ] Moore s Law in STLURH RJ_OPLQRI Action 2 ) Processor and CPU cores FrontEnd: faster instr. decoder (4- to 5-wide) BackEnd: faster L cache, bigger L2 cache, deeper OOO* execution Uncore: move from ring to X-Y fabric mesh 2) 2 Memory bandwidth ~50% increase: channels (4 to 6), speed (DDR- 2666) 3) I/O bandwidth 3 >50% increase: lanes (40 to 48), redesigned IO blocks 4) Inter-socket bandwidth 4 ~60% increase: QPI to UPI (2x to 3x), interface speed (9.6 to 0.4 GigTrans/sec)

16 FD.io VPP Vector Packet Processing Compute-Optimised SW Networking Platform Bare-Metal / VM / Container Dataplane Management Agent Packet Processing Network IO Packet Processing Software Platform High performance Linux user space Runs on compute CPUs: And knows how to run them well! Shipping at volume in server & embedded products 6

17 FD.io VPP The Magic of Vectors Compute Optimized SW Network Platform Packet processing is decomposed into a directed graph of nodes 2 packets move through graph nodes in vector 3 graph nodes are optimized to fit inside the instruction cache vhost-userinput af-packetinput dpdk-input Packet 0 Packet Microprocessor arp-input cdp-input lldp-input mpls-input l2-input ip4-lookup* ip4-input ip4-lookupmulitcast ethernetinput...-nochecksum ip6-input Packet 2 Packet 3 Packet 4 Packet 5 Packet 6 Packet Instruction Cache Data Cache interfaceoutput ip4-loadbalance mpls-policyencap ip4-rewritetransit ip4- midchain Packet 8 Packet 9 Packet 0 4 packets are pre-fetched into the data cache. * Each graph node implements a micro-nf, a micro-networkfunction processing packets. Makes use of modern Intel Xeon Processor micro-architectures. Instruction cache & data cache always hot è Minimized memory latency and usage.

Cloud-native Network Micro-Services For Native Cloud Network Services Contiv LIGATO Production-Grade Container Orchestration Performance-Centric Container Networking Cloud-native Network Function

18 Cloud-native Network Micro-Services For Native Cloud Network Services Contiv LIGATO Production-Grade Container Orchestration Performance-Centric Container Networking Cloud-native Network Function Orchestration Containerized Fast Data Input/ Output Enabling Production-Grade Native Cloud Network Services at Scale Service Policy Service Topology Lifecycle Production-Grade Container Orchestration Kubernetes API Proxies Network Function and Network Topology Orchestration SFC Controller Container Networking Contiv Netmaster Containerized Network Data Plane Networking Plugin Contiv Netmaster CNI CRI Kubelet Agent FD.io VPP Container Switch Agent Agent Agent Container Network Function CNF CNF

Contiv-VPP Architecture K8s Master High Performance

Reflector Cloud-Native NFs Legacy Apps High

Pod Pod App Pod Pod Pod CNF Kubelet Contiv-VPP Etcd

Pod Envoy Sidecar VPP TCP Stack Kernel Host stack

CNI memif Kernel Host stack tapv2/veth VPP TCP Stack

19 Contiv-VPP Architecture K8s Master High Performance Apps Legacy Apps Cloud-Native NFs K8s State Reflector Cloud-Native NFs Legacy Apps High Performance Apps Pod Pod Pod Envoy Sidecar App Pod Pod Pod App Pod Pod Pod CNF Kubelet Contiv-VPP Etcd Kubelet Pod Pod Pod CNF Pod Pod Pod App App Pod Pod Pod Envoy Sidecar VPP TCP Stack Kernel Host stack tapv2/veth memif CNI K8s policy & state distribution CNI memif Kernel Host stack tapv2/veth VPP TCP Stack VPP Contiv-VPP vswitch Agent Agent VPP Contiv-VPP vswitch Data Centre Fabric

20 Service Function Chaining with Ligato Logical Representation Ingress Network Ingress Classifier Ingress Router NF NF 2 NF 3 Egress Router Egress Network Egress Classifier Placement (K8s) Rendering Topology Physical Representation CNF VPP CNF 2 VPP Host CNF VPP CNF VPP Server CNF 3 VPP CNF VPP VPP Vswitch Vswitch VPP Overlay Tunnel Overlay Tunnel Overlay Tunnel Ingress Classifier Egress Classifier

Ligato Cloud-native NFs (CNFs) Kubernetes does not provide a way to stitch micro-services together

can now become part of the service topology Dedicated Telemetry Engine in VPP to enable closed-loop

Master Define Topology High Performance Apps Legacy Apps Cloud-Native VNFs Pod Pod Pod Sidecar

TCP Stack Kernel Host stack tapv2/veth memif CNI CRI Contiv-VPP Etcd Smarts in NICs VPP Contiv-VPP

21 Ligato Cloud-native NFs (CNFs) Kubernetes does not provide a way to stitch micro-services together today Ligato enables you to wire the data plane together into a service topology Network functions can now become part of the service topology Dedicated Telemetry Engine in VPP to enable closed-loop control Offload functions to NIC but via vswitch in host memory Define Topology Define Services K8s Master Define Topology High Performance Apps Legacy Apps Cloud-Native VNFs Pod Pod Pod Sidecar Proxy App Pod Pod Pod App Pod Pod Pod Agent VPP Kubelet Contiv-VPP Netmaster Ligato Controller VPP TCP Stack Kernel Host stack tapv2/veth memif CNI CRI Contiv-VPP Etcd Smarts in NICs VPP Contiv-VPP vswitch Telemetry Engine Agent Back Propagation Loop For Reactive Placement/Rsrc Mgmt Data Centre Fabric

22 Without data, you're just another person with an opinion. W. Edwards Deming

23 Open Source Benchmarking Guiding Principles Discover the limits and know them Assess based on externally measured data and behavior (black-box) Guide benchmarking by good understanding of the whole system (white-box) Provide a feedback loop to hardware and software engineering One can t violate the laws of physics, but one can stretch them..

24 Benchmarking Data and Public References: FD.io CSIT-CPL Per release test and performance reports Multi-Platform/-Vendor Intel & ARM (WiP) Packet Throughput & Latency Non-Drop & Partial Drop Rates Data Plane Workloads FD.io VPP DPDK L3fwd, Testpmd Scaling Single-, Multi-Core MACs, IPs, Flows, ACLs etc. Performance Test Suites (#s) L2: 58 L3 (IPv4 / IPv6): 63 VM vhostuser: 26 Containers memif: 0 Crypto: 3 SRv6: 3 Total: 73 DPDK FD.io VPP 0ge2px520-ethip4-ip4scale2m 0ge2px520-ethip4-ip4base 0ge2px520-eth-l2bdbasemaclrn 0ge2px520-eth-l2xcbase 0ge2px520-ethip4-ip4base-l3fwd 0ge2px520-eth-l2xcbase-testpmd T/C x520 Packet Throughput Tests * Newer data available!! * Selection of testcases from the FD.io CSIT 8.0, 7.0 and 7.07 reports Mpps Throughput Breath (# of test cases), Depth (of measurement) and Repeatability (every release, repeatable locally)

FD.io CSIT-8.07: Packet Throughput Results Source: https://docs.fd.

Baseline 8 Mpps Baseline 8 Mpps 7 Mpps 200k of /32 hfib prefixes 20k of /32 hfib prefixes

25 FD.io CSIT-8.07: Packet Throughput Results Source: IPv4 Routing (ip4) L2 Switching with MAC Learning (l2bd) 9 Mpps Baseline 8 Mpps Baseline 8 Mpps 7 Mpps 200k of /32 hfib prefixes 20k of /32 hfib prefixes 2M of /32 hfib prefixes 6 Mpps 4 Mpps 2 Mpps 00k of /48 L2FIB MACs 0k of /48 L2FIB MACs M of /48 L2FIB MACs 6 Mpps

FD.io CSIT-8.07: Throughput Speedup Results Source: https://docs.fd.

cores Follows Amdahl s Law * * * * * * * * * * * * * * Capped by 4.

26 FD.io CSIT-8.07: Throughput Speedup Results Source: VPP Multi-Core Speedup Properties: Predictable performance Linear scaling with cores Follows Amdahl s Law * * * * * * * * * * * * * * Capped by 4.88 Mpps 0GE 64B link rate limit 200k of /32 hfib prefixes Baseline 2M of /32 hfib prefixes Baseline 0k of /48 00k of /48 L2FIB MACs L2FIB MACs 20k of /32 hfib prefixes M of /48 L2FIB MACs

27 VPP: Multi-Core Speedup Properties Source: VPP Multi-Core Speedup Properties: Predictable performance Linear scaling with cores Follows Amdahl s Law

28 Packet Vectors are Good for You! Netgate shipping product(s) [] Alibaba [2] Netgate: TNSR, hw appliances Alibaba: Network Service Optimization with the VPP Platform [] [2]

29 Baremetal Data Plane Performance Limit FD.io benefits from increased Processor I/O YESTERDAY PCle Packet Forwarding Rate [Gbps] TODAY Ethernet 4 Socket 0 2 Broadwell Server CPU 3 QPI QPI 4 Socket Broadwell 2 Server CPU 3 Server 2x [2 Sockets] %,20* Gbps 4 Socket 0 2 Skylake Server CPU 3 UPI UPI 4 Socket Skylake 2 Server CPU 3 SATA B I O S PCH x8 50GE x6 00GE x8 50GE x6 00GE x6 00GE Server [2 Sockets] +75% SATA B I O S Lewisburg PCH x8 40GE x8 50GE x6 00GE x8 x8 50GE 50GE x6 00GE x6 00GE Network I/O: 60 Gbps Core ALU: 4-wide parallel µops Memory: 4-channels 2400 MHz Max power: 45W (TDP) Server [ Socket] +75% FD.io Takes Full Advantage of Faster Intel Xeon Scalable Processors No Code Change Required Network I/O: 280 Gbps Core ALU: 5-wide parallel µops Memory: 6-channels 2666 MHz Max power: 205W (TDP) Intel Xeon E5-2699v4 22 Cores, 2.2 GHz, 55MB Cache Intel Xeon v3, v4 Processors Intel Xeon Platinum 880 Processors * On compute platforms with all lanes from the Processors routed to slots. Intel Xeon Platinum Cores, 2.7 GHz, 33MB Cache Breaking the Barrier of Software Defined Network Services Terabit Services on a Single Intel Xeon Server!

30 Internet Mega Trends.. Portability and Efficiency Scalability and Self-Healing Cloud Software Defined Networking Cloud Native Designs NFV SDN Open Source Platforms

31 Internet Mega Trends Being Addressed.. Portability and Efficiency Scalability and Self-Healing Cloud Software Defined Networking Cloud Native Designs NFV SDN Open Source Platforms

32 Internet Mega Trends Being Addressed.. NFV Cloud SDN PORTABILITY AND EFFICIENCY Portability and Efficiency Scalability and Self-healing SCALABILITY and SELF-HEALING Software Defined Networking SOFTWARE DEFINED NETWORKING Cloud Native Designs CLOUD NETWORK SERVICES Open Source Platforms LINUX FOUNDATION 32

33 High Performance Cloud-Native Networking K8s Unleashing FD.io THANK YOU!

34 References FD.io VPP, CSIT and related projects VPP: CSIT-CPL: pma_tools - Benchmarking Methodology Kubecon Dec-207, Benchmarking and Analysis.., Benchmarking and Analysis of Software Network Data Planes by M. Konstantynowicz, P. Lu, S.M. Shah,

35 Opportunities to Contribute We invite you to Participate in FD.io Get the Code, Build the Code, Run the Code Try the vpp user demo Install vpp from binary packages (yum/apt) Read/Watch the Tutorials Join the Mailing Lists Join the IRC Channels Explore the wiki Join FD.io as a member Thank you! 35

High Performance Cloud-native Networking K8s Unleashing FD.io

High Performance Cloud-native Networking K8s Unleashing FD.io Giles Heron Principal Engineer, Cisco giheron@cisco.com Maciek Konstantynowicz FD.io CSIT Project Lead Distinguished Engineer, Cisco mkonstan@cisco.com