High Performance Cloud-native Networking K8s Unleashing FD.io

Size: px

Start display at page:

Download "High Performance Cloud-native Networking K8s Unleashing FD.io"

Franklin Thornton
5 years ago
Views:

High Performance Cloud-native Networking K8s Unleashing FD.io Giles Heron Principal Engineer, Cisco giheron@cisco.

1 High Performance Cloud-native Networking K8s Unleashing FD.io Giles Heron Principal Engineer, Cisco Maciek Konstantynowicz FD.io CSIT Project Lead Distinguished Engineer, Cisco Jerome Tollet Distinguished Engineer, Cisco

2 DISCLAIMERs 'Mileage May Vary' Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your opinion and investment of any resources. For more complete information about open source performance and benchmark results referred in this material, visit and/or Trademarks and Branding This is an open-source material. Commercial names and brands may be claimed as the property of others.

3 Internet Mega Trends Portability and Efficiency Cloud Scalability and Self-Healing Software Defined Networking NFV SDN Cloud Native Designs Open Source Platforms

5 Pillars of Next Generation Software Data Planes Blazingly Fast Process the

traffic Truly Extensible Foster pace of innovation in cloud-native

Cloud means running everywhere Cloud means hardware and physical infra

Predictable for a number of VMs, Containers, virtual topology and (E-W, N-S)

detailed cross-layer operation and efficiency monitoring Enables feedback

4 5 Pillars of Next Generation Software Data Planes Blazingly Fast Process the massive explosion of East-West traffic Process increasing North-South traffic Truly Extensible Foster pace of innovation in cloud-native networking No compromise on performance (zero-tolerance) Software First Cloud means running everywhere Cloud means hardware and physical infra agnostic Predictable performance Dataplane performance must be deterministic Predictable for a number of VMs, Containers, virtual topology and (E-W, N-S) traffic matrix Measureable Counters everywhere to count everything for detailed cross-layer operation and efficiency monitoring Enables feedback loop to drive optimizations FD.io meets these challenges How can one use it in large scale Cloud-native networks?

High Performance Cloud-native Networking K8s Unleashing FD.

App Micro-Services Network-as-a-Service Network-as-a-Service NF

Orchestration De facto standard for container orchestration

Networking Seamless cloud-native integration Extends K8s with

Scalability, performance and agility Marries K8s with flexible

5 High Performance Cloud-native Networking K8s Unleashing FD.io Cloud-native Networking Platform Micro-Services Micro-Services App Micro-Services Network-as-a-Service Network-as-a-Service NF Micro-Services Contiv LIGATO Production-Grade Container Orchestration De facto standard for container orchestration Superior extensibility and self-healing Performance Container Networking Seamless cloud-native integration Extends K8s with userspace networking Cloud-native Network Micro-services Scalability, performance and agility Marries K8s with flexible network topologies Fast Data Input/Output Most efficient on the planet Top performance, flexibility and extensibility

6 Time The Way Applications Are Developed and Deployed.. Has Changed..

Tier A Tier C Tier B Spine/Leaves & L Core/L2 Access VLAN 2 VLAN Core VLAN 2 Internet Tier2 D Tier2 E Tier2 F SD-WAN & BeyondCorp Cloud Provider

7 The Way Networks are Deployed and Used has Changed Corporate LAN/WAN 80:20 rule Internet Internet exchanges & public peering Data-Center Core/Distr/Access, VLAN based LAN WAN Network Network 2 Network VLAN n VLAN 2 VLAN Time Intranets & Internet LAN WAN WWW Tiered Transit & Private Peering Tier A Tier C Tier B Spine/Leaves & L Core/L2 Access VLAN 2 VLAN Core VLAN 2 Internet Tier2 D Tier2 E Tier2 F SD-WAN & BeyondCorp Cloud Provider Telco/Cable Access &, OTT/CDN Content L Fabric/SW Overlay & Virt Networking WiFi Internet Cloud Cloud Provider Provider Global 2 Global Backbone Backbone Layer Fabric Telco Telco 2 MSO

8 Microservices & Containers have changed many things Applications are being developed and deployed very differently today. Microservices split applications into modular pieces with the network stitching the pieces together The interconnection of the pieces increases East / West traffic Network performance is therefore critical to the composite application performance

9 Microservices & Containers have changed many things How do application trends impact SDN / NFV? Network Functions can be implemented using containers Traffic Traffic The microservice model enables decomposition of Network Functions into finer-grained Cloud-native Network Functions (CNFs) Agent Agent Agent Agent Agent Since network functions are I/O bound network performance is even more critical than for applications Agent Agent Traffic Data Plane Microservices

10 Remember 965 Moore s Law.. Moore s Law Electromechanical Solidstate Vacuum Tube Transistor Relay Integrated Circuit 0 6 Calculations per Second per $,

11 Processing Packets: How to Use Compute.. Resources to Get Performance SATA B I O S PCH DMI Socket 0 Skylake Server CPU x6 x6 x6 00GE 00GE 00GE 280 ETH frames CyclesPerPacket [ClockCycles] = #FGHIJKLIMNGH OPLQRI Th#$%&h'%( [''+] = 2 UPI UPI UPI 4 #STLURH FGHIJKLIMNG Socket Skylake Server CPU x6 x6 x6 00GE 00GE 00GE 280 ETH frames V OPLQRI_OJNLRHHMGW_XMYR[HRL ] = SOZ_[JR\[]^] STLURH RJ_OPLQRI 2 ) Processor and CPU cores for performing packet processing operations _ 2) 2 Memory bandwidth for moving data (packets, lookup) and instructions (packet processing) ) I/O bandwidth for moving packets to/from NIC interfaces _ 4) Inter-socket bandwidth 4 for handling inter-socket operations _ Th#$%&h'%( [*'+] = Th#$%&h'%( ''+ /02(_567[''+]

12 Processing Packets: What Improves in Compute.. Resources to Get Performance SATA B I O S PCH DMI Socket 0 Skylake Server CPU x6 x6 x6 00GE 00GE 00GE 280 ETH frames CyclesPerPacket [ClockCycles] = #FGHIJKLIMNGH OPLQRI Th#$%&h'%( [''+] = 2 UPI UPI UPI 4 Socket Skylake Server CPU x6 Th#$%&h'%( [*'+] = Th#$%&h'%( ''+ /02(_567[''+] x6 x6 00GE 00GE 00GE 280 ETH frames #STLURH FGHIJKLIMNG V OPLQRI_OJNLRHHMGW_XMYR[HRL ] = SOZ_[JR\[]^ ] Moore s Law in STLURH RJ_OPLQRI Action 2 ) Processor and CPU cores FrontEnd: faster instr. decoder (4- to 5-wide) BackEnd: faster L cache, bigger L2 cache, deeper OOO* execution Uncore: move from ring to X-Y fabric mesh 2) 2 Memory bandwidth ~50% increase: channels (4 to 6), speed (DDR- 2666) ) I/O bandwidth >50% increase: lanes (40 to 48), redesigned IO blocks 4) Inter-socket bandwidth 4 ~60% increase: QPI to UPI (2x to x), interface speed (9.6 to 0.4 GigTrans/sec)

13 FD.io The Magic of Vectors Compute Optimized SW Network Platform Packet processing is decomposed into a directed graph of nodes 2 packets move through graph nodes in vector graph nodes are optimized to fit inside the instruction cache vhost-userinput af-packetinput dpdk-input Packet 0 Packet Microprocessor arp-input cdp-input lldp-input mpls-input l2-input ip4-lookup* ip4-input ip4-lookupmulitcast ethernetinput...-nochecksum ip6-input Packet 2 Packet Packet 4 Packet 5 Packet 6 Packet 7 4 Instruction Cache Data Cache interfaceoutput ip4-loadbalance mpls-policyencap ip4-rewritetransit ip4- midchain Packet 8 Packet 9 Packet 0 4 packets are pre-fetched into the data cache. * Each graph node implements a micro-nf, a micro-networkfunction processing packets. Makes use of modern Intel Xeon Processor micro-architectures. Instruction cache & data cache always hot è Minimized memory latency and usage.

14 K8s perspective: Orchestation/K8s incl. placement for scale-out We have all of this performance, how do we use it How do we make it work for us at scale How do we intelligently make an efficient use of available resource to deliver intended application and networking services

15 Cloud-native Network Micro-Services For Native Cloud Network Services Contiv LIGATO Production-Grade Container Orchestration Performance-Centric Container Networking Cloud-native Network Function Orchestration Containerized Fast Data Input/ Output Enabling Production-Grade Native Cloud Network Services at Scale

16 Kubernetes Overview Kubernetes is a portable, extensible open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. Terminology: Node: a host or VM on which pods can be scheduled : one or more co-resident linux containers with a single IP address (typically a /24 of address space is provided per node) Deployment: a set of pods implementing the same application Service: an abstraction providing a single persistent IP address for a deployment (Kubernetes provides mechanisms to load balance across multiple pods) Kubernetes assumes seamless connectivity between pods, wherever it places them A networking plugin is needed to abstract the network

17 Contiv- Overview Contiv- is a networking plugin for Kubernetes that: Allocates IP addresses to s (IPAM) Programs the underlying infrastructure it uses ( and the Linux TCP/IP stack) to connect the pods to other pods in the cluster and/or to the external world Implements K8s network policies that define which pods can talk to each other Implements K8s services using a stateful load-balanced NAT function Contiv- is a user-space, high-performance, high-density networking plugin for Kubernetes - leveraging FD.io/ as the industry s highest performance software data plane

Contiv- Architecture Can deliver complete container networking solution entirely from

Apps can add VCL library for Higher Performance (bypass Kernel host stack and use TCP stack)

Performance Apps Legacy Apps Cloud-Native NFs K8s State Reflector Cloud-Native NFs Legacy

App Envoy Sidecar TCP Stack Kernel Host stack tapv2/veth memif CNI K8s policy & state

18 Contiv- Architecture Can deliver complete container networking solution entirely from user-space Replace all eth/kernel interfaces with memif/user-space interfaces. Apps can add VCL library for Higher Performance (bypass Kernel host stack and use TCP stack) Legacy apps can still use the kernel host stack in the same architecture K8s Master High Performance Apps Legacy Apps Cloud-Native NFs K8s State Reflector Cloud-Native NFs Legacy Apps High Performance Apps Envoy Sidecar App App CNF Kubelet Contiv- Etcd Kubelet CNF App App Envoy Sidecar TCP Stack Kernel Host stack tapv2/veth memif CNI K8s policy & state distribution CNI memif Kernel Host stack tapv2/veth TCP Stack Contiv- vswitch Agent Agent Contiv- vswitch Data Centre Fabric

19 What Container Networking Lacks for NFV Use-Cases Networking HTTP or NAT-based load balancing isn t suitable for NFV use-cases No support for multiple interfaces or IP addresses per pod No support for high-speed wiring of NFs Policy No support for QoS, network-aware placement etc. Isolation Applications run in user-space kernel networking is unsuited to NFV Performance Polling mode drivers (e.g. DPDK) required for maximum throughput

20 Service Function Chaining with Ligato Logical Representation Ingress Network Ingress Classifier Ingress Router NF NF 2 NF Egress Router Egress Network Egress Classifier Placement (K8s) Rendering Topology Physical Representation CNF CNF 2 Host CNF CNF Server CNF CNF Vswitch Vswitch Overlay Tunnel Overlay Tunnel Overlay Tunnel Ingress Classifier Egress Classifier

Ligato Cloud-native NFs (CNFs) Kubernetes does not provide a way to

plane together into a service topology Network functions can now become

closed-loop control Offload functions to NIC but via vswitch in host

Performance Apps Legacy Apps Cloud-Native VNFs Sidecar Proxy App App

stack tapv2/veth memif CNI CRI Contiv- Etcd Smarts in NICs Contiv-

21 Ligato Cloud-native NFs (CNFs) Kubernetes does not provide a way to stitch micro-services together today Ligato enables you to wire the data plane together into a service topology Network functions can now become part of the service topology Dedicated Telemetry Engine in to enable closed-loop control Offload functions to NIC but via vswitch in host memory Define Topology Define Services K8s Master Define Topology High Performance Apps Legacy Apps Cloud-Native VNFs Sidecar Proxy App App Agent Kubelet Contiv- Netmaster Ligato Controller TCP Stack Kernel Host stack tapv2/veth memif CNI CRI Contiv- Etcd Smarts in NICs Contiv- vswitch Telemetry Engine Agent Back Propagation Loop For Reactive Placement/Rsrc Mgmt Data Centre Fabric

22 Unleashing Innovation in Networking Container networking requires fast innovation cycles to deploy new models Kernel upgrades and ad hoc modules may cause problems in production environments Multiple options are being explored in the industry OVS Technology Programmability Model Execution Context OpenFlow lets users configure very granular data path behaviour Primarily kernel based Openflow model ebpf+xdp FD.io / Packets handled by user coded ebpf programs running in a sandbox Regular user-mode C program with user loadable plugins Bypass kernel networking stack but still in kernel-mode Bytecode + JIT + kernel helpers User-mode, no kernel dependencies Native NIC drivers, Linux APIs, DPDK

640 +75%,20* Gbps 4 Socket 0 2 Skylake Server CPU UPI UPI 4 Socket Skylake 2 Server CPU SATA B I O S PCH x8 50GE x6 00GE x8 50GE x6 00GE x6 00GE Server [2 Sockets] +75% 20 560 SATA B I O S Lewisburg

23 Baremetal Data Plane Performance Limit FD.io benefits from increased Processor I/O YESTERDAY PCle Packet Forwarding Rate [Gbps] TODAY Ethernet 4 Socket 0 2 Broadwell Server CPU QPI QPI 4 Socket Broadwell 2 Server CPU Server 2x [2 Sockets] %,20* Gbps 4 Socket 0 2 Skylake Server CPU UPI UPI 4 Socket Skylake 2 Server CPU SATA B I O S PCH x8 50GE x6 00GE x8 50GE x6 00GE x6 00GE Server [2 Sockets] +75% SATA B I O S Lewisburg PCH x8 40GE x8 50GE x6 00GE x8 x8 50GE 50GE x6 00GE x6 00GE 2 4 Network I/O: 60 Gbps Core ALU: 4-wide parallel µops Memory: 4-channels 2400 MHz Max power: 45W (TDP) Server [ Socket] +75% FD.io Takes Full Advantage of Faster Intel Xeon Scalable Processors No Code Change Required 2 4 Network I/O: 280 Gbps Core ALU: 5-wide parallel µops Memory: 6-channels 2666 MHz Max power: 205W (TDP) Intel Xeon E5-2699v4 22 Cores, 2.2 GHz, 55MB Cache Intel Xeon v, v4 Processors Intel Xeon Platinum 880 Processors * On compute platforms with all lanes from the Processors routed to slots. Intel Xeon Platinum Cores, 2.7 GHz, MB Cache Breaking the Barrier of Software Defined Network Services Terabit Services on a Single Intel Xeon Server!

24 High Performance Cloud-Native Networking K8s Unleashing FD.io Most Portable and Most Efficient on the Planet Scalability and Self-healing PORTABILITY AND EFFICIENCY SCALABILITY and SELF-HEALING Rich NFV Functionality SOFTWARE DEFINED NETWORKING Cloud Native CLOUD NETWORK SERVICES Open Source LINUX FOUNDATION 24

25 High Performance Cloud-Native Networking K8s Unleashing FD.io THANK YOU!

High Performance Cloud-native Networking K8s Unleashing FD.io

High Performance Cloud-native Networking K8s Unleashing FD.io Giles Heron Principal Engineer, Cisco giheron@cisco.com Maciek Konstantynowicz FD.io CSIT Project Lead Distinguished Engineer, Cisco mkonstan@cisco.com