Fast packet processing in the cloud. Dániel Géhberger Ericsson Research

Transcription

1 Fast packet processing in the cloud Dániel Géhberger Ericsson Research

2 Outline Motivation Service chains Hardware related topics, acceleration Virtualization basics Software performance and acceleration Packet Processing Execution Environment Conclusion Ericsson Internal Page 2

3 Motivation? 5G! High bandwidth Up to 10Gbps but a few 100Mbps should be generally available Low latency 1 ms 5G radio latency Optimal network deployment for low end-to-end latency Remote control use-cases High spectrum (NX) radio technology, massive MIMO, beamforming Network slicing Create separated networks in seconds Fully cloud based deployment from the antenna site Network Function Virtualization and service chaining Support for IoT (the smart hype) Ericsson Internal Page 3

4 Service chains Splitting up legacy network components Core network examples: per user charging, rate limitation Radio example: PDCP functionality IPSec termination and GTP de-tunneling Sequence numbering, buffering Header compression Air-crypto and transmission towards the antenna site Possibly per user service chains Dynamic configuration 3 rd party components Optimization for performance Hardware capabilities and acceleration, software technologies Placement: physical location, noisy neighbors Ericsson Internal Page 4

5 Modern Ethernet NICS Throughput: 10/25/40/100 Gbps, dual port Interconnect (RJ45+CAT7 for 10Gbps) / SFP+ / QSFP+ Latency! PCIe counts PCIe v2, 16 lanes: 64Gbps PCIe v3, 16 lanes: 128Gbps Offloading Checksum calculation TCP/UDP reassembling Tunneling RDMA over Ethernet Receive Side Scaling (RSS) Ericsson Internal Page 5

6 NIC Queues 1 CPU core can handle ~10Gbps in the best (and simple) case Distribute the incoming packets between cores RSS hash: IP addresses, transport type, transport ports Assign the packet to an RX queue (descriptor ring) Default: # of queues = # of CPU cores, interrupts Sometimes it cannot work: encrypted IPSec flow We have TX queues as well SR-IOV uses these as well RX queue 1 CPU core 1 RX queue 2 CPU core 2 NIC Hash RX queue n CPU core n Ericsson Internal Page 6

7 NON-Uniform memory access NUMA More than one CPU sockets Dedicated memory and PCIe Slower access from other sockets Where to schedule the packet processing tasks? Use the local memory Ericsson Internal Page 7

8 Commodity hardware? Clouds will be built with off-the-shelf hardware elements Partially true, but Amazon, Google, Facebook and the other big players do not really follow this They can ask Intel for custom designs and build their own servers Hardware acceleration (GPU) Cryptography Packet re-ordering Using ARM and Network Processing Units instead of Intel Smart NICs Ericsson Internal Page 8

9 Smart NIC example Netronome Open vswitch data path in the NIC Ericsson Internal Page 9

10 Rack Scale architecture Hyperscale datacenter Intel Ericsson joint datacenter hw/sw development Today exists only in a limited version, research topic Separated sleds for compute, networking, storage, (memory) Optical backplane interconnect Smart NICs Software defined hardware Give me physical computer with 3 cores, 5GB memory, and 1TB disk CPU slicing New type of OS? Latencies? Virtualization? Ericsson Internal Page 10

11 Virtualization Technologies Standard VM Well-known hypervisors KVM, Xen, ESX, etc. Full kernel Linux container Common kernel 1 application Docker Packaging! Data to container Library OS Lightweight kernel Only kernel space Focus on 1 application Can be a JVM Well-known hypervisors again! Ericsson Internal Page 11

12 Hypervisors Ericsson Internal Page 12

13 Packet forwarding with OVS 2.0 Figure: mega packet per second! millions of 64 byte packets per second on 10G interface A packet arrives every ~67.2 ns 3.5 GHz 1 cycle = 0.28 ns 240 cycles for everyting for line-rate Ericsson Internal Page 13

14 Latency Simple ping RTT measurement with ~10000 samples Not tuned for low latency Ericsson Internal Page 14

15 Default packet processing with Linux Kernel overhead System calls Context switching on blocking I/O Data copying from kernel to user space Interrupt handling in kernel User space App Kernel space Socket Driver Ring buffers NIC RX/TX queues Ericsson Internal Page 15

16 DPDK Intel Data Plane Development Kit Fast packet processing solution from Intel Not only for Intel NICs any more Other vendors, virtual interfaces Set of libraries for applications Switch, VNF, etc. Polling packets Uses a dedicated CPU for 100% User space App DPDK Ring buffers Direct user-space access Lockless buffer rings Kernel space UIO driver Batch packet handling Huge memory pages Pre-allocated buffers NIC RX/TX queues Ericsson Internal Page 16

17 OVS 2.0 with DPDK and KVM The price: DPDK has to support the NIC DPDK aware switch, hypervisor and application 3 CPU cores with a VM 1 in the switch for NIC VM 1 in the VM 1 in the switch for VM NIC (So the figure is not completely fair) Ericsson Internal Page 17

18 kpps / W Power consumption Packet size BananaPI (ARM) Atom Xeon Ericsson Internal Page 18

19 switching to Ericsson Research ER Flow Switch 12 Mpps for packet forwarding through a VM or container JIT linking Packet Processing Execution Environment JIT linking, custom callbacks in the switch By extending OpenFlow Heavy PPs in VMs/containers Specific PPs on NPU, ARM, etc. Ericsson Internal Page 19

20 Conclusion 5G networks will be cloud based High bandwidth, low latency Acceleration is required NIC extra features Dedicated hardware acceleration User space fast packet processing Ericsson Internal Page 20

21