Improve Performance of Kube-proxy and GTP-U using VPP

Transcription

1 Improve Performance of Kube-proxy and GTP-U using VPP Hongjun Ni Danny Zhou Johnson Li Network Platform Group, DCG, Intel Acknowledgement: Jianfeng Tan

2 Agenda Enabling high performance kube-proxy Six ways to improve the performance of GTP-U Key takeaway 2

3 Introducing Vector Packet Processor Packet processing development platform Runs on commodity CPUs and leverages DPDK Runs as a Linux user-space application

4 VPP Graph Node and Plugin Framework dpdk-input Packet Vector ethernetinput mpls-ethernetinput ip4-input ip6-input arp-input ip4-lookup ip4-local ip4-udplookup ip4-loadbalance Kube-proxy ethernetoutput ip4- rewrite GTP-U

5 Enabling high performance kube-proxy Introducing kube-proxy Current pain point Proposed kube-proxy solution Kube-proxy dataplane GoVPP Performance 5

6 Introducing kube-proxy Watches addition and removal of Service and Endpoints. Installs iptables rules Captures traffic and select Pod Redirects traffic Reference:

7 Current Pain Point Supports userspace and iptables Uses kernel iptables NAT Performance degrades when service/endpoint pairs increase 7

8 Proposed kube-proxy solution API Server watch watch watch Kube-proxy CP Kube-proxy CP Kube-proxy CP GoVPP GoVPP GoVPP CP: Control Plane DP: Data Plane Kube-proxy DP Kube-proxy DP Kube-proxy DP Node 1 Node 1 Node 1

9 Kube-proxy Data Plane K8s-node Backend Pod 1 Pod IP: x.x.x.x Backend Pod 2 Pod IP: y.y.y.y Distribute traffic evenly virtio-user virtio-user Per flow stick to Pod vhost vhost Three service types: VPP DNAT Load balancing SNAT (1). ClusterIP: Port (2). NodeIP: NodePort (3). External LoadBalancer eth

10 Kube-proxy Data Plane Internals Supported Graph Nodes Other input Graph nodes ip-lookup kp4-nat4 kp6-nat6 kp4-nat6 kp4- nodeport kp6-nat4 udplookup kp6- nodeport ip4 -lookup input Other Graph nodes VIP->Pod Table ip6 -lookup Stores VIP->Pod session

11 Kube-proxy: External LoadBalancer VPP Kube-proxy K8s-node-1 Pod 1 VPP LB eth1 Kube-proxy DP Pod N VPP Router K8s-node-2 Pod 1 eth1 Kube-proxy DP Pod N K8s-node-3 Pod 1 eth1 Kube-proxy DP Pod N

12 GoVPP Golang toolset for VPP management VPP binary API (JSON) Go Structure Handle 250,000 binary API requests per second Reference:

13 Throughput (Mpps) Performance Kube-proxy Throughput For 64-Byte Packet Linux iptables perf: < 400 kpps Scaling Phy NIC vhost-user Test Case Input packet size (bytes) Output packet size (bytes) Load balance + DNAT + Routing Reference:

14 Six Ways to Improve Performance of GTP-U Cache table lookup result Bypass second ip-input Bypass first route lookup Dual-loop and Quad-loop Packet prefetching Bypass second route lookup 14

15 GTP-U Plugin Internals Typical GTP-U packet processed by VPP and GTP-U plugin Outer MAC header Outer IP header UDP header GTP-U header Inner IP header L4 header Payload GTP-U Plugin Supported Graph Nodes Other input Graph nodes ip-lookup udp-lookup gtpu-input gtpu-encap gtpu-bypass IP4 -input input Other Graph nodes GTP-U tunnel IP6 -input GTP-U Plugin Stores PDP context

16 Device Under Test for Performance Network Topology Hardware Configuration BIOS Configuration Traffic Gen (Trex) CPU DIMM Intel(R) Xeon(R) CPU E GHz 2133 MHz, 64GB Total Enhanced Intel Speedstep Turbo Boost Processor C3 Enabled Enabled Disabled Port A Port B NIC 2x 82599ES 10-Gigabit SFI/SFP+ Network Connection Processor C6 Hyper-Threading Disabled Disabled PacketGen Ixia* 10 Gigabit Ethernet Traffic Generator Intel VT-d CPU Power and Performance Policy Enabled Performance Memory Freq MHz Port A Port B Software Configuration Total Memory Size Memory RAS and Performance Configuration -> NUMA Optimized 64 GB ENABLED Core 0 OS Ubuntu LTS QPI B/W 9.6 GT/s DUT (VPP + GTP-U) Kernel Linux version generic DPDK MLC Streamer MLC Spatial Prefetcher ENABLED ENABLED Legend Flow A Flow B VPP GTP-U rc rc0 DCU Data Prefetcher DCU Instruction Prefetcher Direct Cache Access (DCA) 16ENABLED ENABLED ENABLED

17 Cache table lookup result Gtpu-input Read IP and TEID N key == stored key? Y Tunnel Lookup Retrieve stored tunnel index Store key and tunnel index Read tunnel entry

18 Bypass second ip-input dpdk-input Packet Vector ethernetinput ip4-input ip6-input arp-input Pros: Boost performance ip4-lookup ip4-local ip4-udplookup Cons : Security issue ip4-loadbalance gtpu-decap ethernetoutput ip4- rewrite ip4-lookup ip4-input

19 Bypass first route lookup dpdk-input ethernetinput Packet Vector gtpu packet: accelerate decap processing gtpu-bypss ip4-input none-gtpu packet ip4-lookup ip6-input ip4-local arp-input ip4-udplookup none-gtpu packet: an overhead with about 13 clocks per packet gtpu packet ip4-loadbalance gtpu-decap ethernetoutput ip4- rewrite ip4-lookup ip4-input

20 Throughput (Mpps) GTPU-Decap Performance and Analysis GTPU-Decap Throughput For 64-Byte Packet raw cache table lookup bypss ip-input bypass route lookup Test Case Input packet size (bytes) Output packet size (bytes) Transport IP Routing + GTPU-Decap + Inner IP Routing 98 64

21 Dual-loop and Quad-loop, Packet Prefetching Reduce read latency Process packets in parallel

22 Bypass second route lookup dpdk-input Packet Vector ip4-input ip6-input arp-input ip4-lookup ip4-rewrite ip4-loadbalance gtpu-encap ethernetoutput ip4- rewrite ip4-lookup ethernetinput

23 Throughput (Mpps) GTPU-Encap Performance and Analysis GTPU-Encap Throughput For 64-Byte Packet One-loop Dual-loop Quad-loop Quad-loop w Prefetch Bypass route Test Case Input packet size (bytes) Output packet size (bytes) Inner IP Routing + GTPU-Encap + Transport Routing

24 Key Takeaway Easy-to-use and flexible VPP plugin framework Kube-proxy plugin to boost DP s performance in Cloud environment Combination of a set of ways to optimize data plane performance Better platform for developing open source dataplane ingredients 24

25 Q&A