OPNFV Plugfest Jun Hardware Acceleration over NFV in China Mobile. Wang Xu, China Mobile

Transcription

1 OPNFV Plugfest Jun Hardware Acceleration over NFV in China Mobile Wang Xu, China Mobile 1

2 Problems we face today Why we need hardware acceleration in NFV Which VNFs need to be accelerated? Which Functions need to be accelerated? Which accelerator do we need? ASIC or FPGA or GPU Look-aside or In-line? 1 card with 1 function or n functions? Detailed Spec of accelerator Multiple PRs 2

3 ITU 5G Vision 1 million/ km2 1 ms 3

4 mmtc: massive Machine Type Communication 1 million/ km2 Traditional terminals Sensor-based Medical instruments Home appliances Smart phones Forest-fire prevention Environment monitoring Intelligent agriculture Smart city 4

5 urllc : ultra-reliable & Low Latency Communications 1 ms Current situation: 40-50ms business latency V2X: driverless vehicles <= 1ms, drive-assistance vehicles<=20ms. Intelligent parking, Intelligent motorcade, driverless vehicles 0.75GB/s Unmanned aerial vehicle NW <=2ms, VR Image rendering Industrial automation : Sensors-based NW <= 1ms 5

6 embb: Enhanced Mobile Broadband User experience rate : 1Gbps Peak rate 10Gbps Tbps per km2 AR/VR, 3D/4K video, Telemedicine, Distance learning Large number of flows, large traffic, low latency<20ms Forwarding capability MBB FBB Facing embb and URLLC requirements, Local DC and edge DC near to end users Handle large bandwidth and operations to save core DC resources 6

7 MEC Vision scpe CRAN -CU MECP 5GUP CDN SD- WAN Access DC County DC City DC SAE- GW 5GUP Province Core DC Nation Core DC Access Layer Aggregation Layer Core Layer Province NW Nation NW Distance Latency (transfer) Latency (title) Hops Bandwidth 1~10km 5~50km 80~300km <70us <290us 0.45ms~1.55ms 2.2ms 2.44ms 2.6~3.7ms ~4hops ~8hops ~10hops 50GE 100GE 200GE 7

8 Why we need hardware acceleration in NFV More VNFs will be located in edge DC (MEC). - Limited compute resources - Limited power/cooling - Limited space Higher performance require by 5G - Lower latency/jitter - Predictable performance - More secure - High throughput packet forwarding, real-time media, encrypt, complex computing, etc. - Auto NW optimization, failure analysis & prediction, deeplearning, big-data in future 8

9 Current situation Low development of media VNFs - Lack of performance requirements - Low development of new business HW accelerators lack of standardization - So many acceleration solutions(fpga, SoC, NP, GPU ) - Accelerators depend on VNF, NFVI - VNFs use their own accelerator China Mobile needs of acceleration - CDN, SD-WAN, SAE-GW, 5G core UP in city-level DC - scpe, MECP, 5G edge UP, CRAN-CU in county-level DC - Various accelerations DPDK, SR-IoV, GPU, SmartNIC, FPGA 9

10 Acceleration points VNF Func Scale Acceleration points vepc Volte ims SAE- GW SBC 1.2M bearers 1user=1.1~1.2bearer 1M users 35,000 concurrent sessions L2/L3/L4 forwarding: throughput 50Gbps DPI:100% packets need DPI Media transcoding AMR-WB AMR Call after SRVCC AMR-WB G.711/G.729 VoLTE & Fixed-line IMS AMR-WB EVS VoLTE terminal support EVS codec Ipsec: Only signaling plane Forwarding 5G UPF 1M users L2/L3/L4 forwarding: throughput 200Gbps DPI: 100% packets need DPI Ipsec: Only signaling plane BRAS 512K L2 L3 MPLS forwarding: Overall 2-way forward capacity >2000Gbps, Overall throughput >1000Gbps, Overall packet forward capacity >1500MPPS Pppoe/ipoe tunnel: 2000session/up/s HQoS:Support FQ,SQ,GP,PQ 4-level control Support multicast Traffic statistics 10

11 Acceleration targets Functions Key words Describe CheckSum TCP/UDP/SCTP Generate a Sum value to check data-integrity Vlan VLAN tag offload Tunnel Offload VxLan VxLan tag offload NVGRE NVGRE tag offload TCP TCP Segmentation TCP segmentation due to MTU RSC(ReceiveSideCo alescing) Receive side offload Service Offload GTP GTP encap/decap offload PacketLable Filter Packet Filter offload FW Filter FW ACL offload NAT NAT NAT translation offload 11

12 Carrier Network first mile Access enodeb ONT Node OLT node Aggregation Region A EPC CO & Cloud vepc vcpe valg vbng vsaegw vmonitor Central Office Residence ucpe/sdwan/mec Optical Splitter DPU/MDU ONT Node OLT Carrier Grade NW Aggregation Embedded Radio Modem Region B enodeb MEC CPE/IoT on Radio Region C Community Hotel 12

13 Accelerator types Smart NIC: Accelerator is a part of NIC, ChinaMobile is considering this one NW attached accelerator: Accelerators can be used by hosts in NW PCIe accelerator: Accelerators are linked by PCIe bus with CPU 13

14 Which accelerator The growth of AI/high-traffic complex service forwarding/big data X86 performance growth is slow After decoupling, performance is even worse Heterogeneous acceleration is the way CPU + FPGA: mainstream technical routes flexible high performance low power low price FPGA GPU ASIC NP Flexibility Performance Power Price Ecosphere 14

15 Industry trends INTEL 2020: 1/3 of Cloud DC FPGA servers 2017: A10 FPGA PCIe 2015: Buy Altera $16.7B Microsoft 2017: AI platform BrainWave based on S10 FPGA 2016: Nearly all new buy servers with FPGA : Bing and Azure use FPGA 2012: Large scale pilot with FPGA(1,632 servers) 2011: Catapult plan engine on Baidu : AI platform XPU based on KU115 FPGA : FPGA service on Baidu Cloud 2016: SQL-ACC using FPGA in inside analysis Intel A10 FPGA(GX660) BrainWave Architecture XPU Architecture 15

16 X86+FPGA X86 CPU L1 Cache L1 Cache L1 Cache L1 Cache CPU CPU CPU CPU L2Cache L2Cache L2Cache L2Cache L3Cache Message Queue Data Buffer DRAM DIMM Memory Controller Home Agent QPI Coherent Interface QPI PCIe HW acc driver data dissemination PCIe Interface Message Control Message Stream Data Stream 16

17 Look-aside vs. In-line CPU CPU PCIe*8/16 PCIe*8/16 快路径分发和协业 DPI 40Ge Standard NIC 40Ge 100Ge Smart NIC 100Ge LB 议务 加速网卡 Look-aside CPU handle data plane In-line Offload data plane to NIC NIC offload main data path in In-Line mode Dual server can realize 100Gbps GPT offload 17

18 Acceleration Architecture NFVO VM1 (Trancoding) Virtio-Frontend VM2 (DPI) Virtio-Frontend VNFM Nova- Compute Cyborgagent L2- agent Hypervisor Virtio- Backend Openstack Horizion Nova Neutron Cinder KVM FPGA device driver Cyborg Keystone Ceilomete r Server Static Region FPGA PR1 PR2 PR3 PRn Dynamic load 1, Use FPGA in-line 2, Cyborg to manage and orchestrate FPGA 3, Common API to implement mutiple FPGA 18

19 19 vepc Acceleration Link IP FWD LB Secu rity GTP Rule Look up DPI QoS Char ging IP FWD Link RAN PTN L % L3 4.30% L4 6.98% L % L4 4.19% L7 6.28% L % L3 2.68% L % L % CPU? Func OVS IPv4v6 MAC Tunnel MPLS ACL Sec multica st Load balance Packet monitor Ddos Antispoofin g APN ACL Flow match Flow oper GTP encap GTPdecap GTP-ctrl proto match proto analysis Policy Car Remark Shaping CG AAA OCS IPv4v6 MAC Tunnel MPLS ACL multica st OVS

20 vepc Acceleration Business BG: GW-U/UPF Forward capability(fc) 300Gbps City-level DC 1.5M user access Average rate 200Kbps per user Xilinx VU9P LCs: 2,586 K CLB Flip-Flops: 2,364 Memory: Mb I/O: 832 DPI Forwarding L3~L7 policy FC/ VM X Gbp s VMs Pow er kw SPI Forwarding L3~L4 policy FC/ VM 7Gb ps VMs Pow er kw Base Forwarding Only encap FC/ VM 30G bps VMs Pow er kw Save at least 16 VMs X86 +FP GA 5Gb ps kw 10G bps kW Gb ps kw 20

21 Telco need E2E Acc Solution not only HW Provide Acc Image, handle Acc req, Or&Ma Acc HW VNFs use Acc API Common API Acc Virtualize and Pool Manage Acc using Cyborg VIM virtualizes Acc, and create pool Accelerators like FPGA 21

22 Cyborg Queens ; Rocky Queens Accelerators to mainstream Optimize cyborg-db Add report & trait, interact with Placement Rocky Extend Nova API to provide Acc Add FPGA versions Add Pythonclient-cyborg API Extend Placement, describe Acc with Func 22

23 China Mobile testing on FPGA(Smart NIC) Servers E v4 256G Mem Spec&Number Controller node Huawei 2288*1 Hypervisor with Smart NIC ZTE R5300*1 + Huawe 2288*1 Hypervisor with 40G SRIOV NIC ZTE R5300*1 Hypervisor with normal NIC ZTE R5300*1 + Huawei 2288*1 Tester: IXIA XM2 VIM: TECS v (Openstack M) 11 Nov 2015 OPNFV Summit 23

24 Testing result SR-IOV vswitch Smart NIC Functions Only L2 FWD DVR, SFC, Security Group, Mirror, SDN VTEP DVR, SFC, Security Group, Mirror, SDN VTEP Live Migration Not support Support Support CPU usage None Very high Low Decouplability Tightly coupled Common VirtIO driver, decoupled Performance High Low High Common VirtIO driver, decoupled 24

25 Testing result - performance 480 Flows (Control plane VNFs) NIC type Line rate(82b/256b/512b)% Latency(82B/256B/512B)us SR-IOV 25.5/73.3/ /7.2/10.0 vswitch(3 cores) 11.0/30.6/ /45.6/50.4 Smart NIC(3 cores) 18.8/50.5/ /45.8/ ,000 Flows (Data plane VNFs) NIC type Line rate(82b/256b/512b)% Latency(82B/256B/512B)us SR-IOV 25.8/73.6/89 6.5/7.2/10 vswitch(3 cores) 6.6/19.0/ /57.1/45.6 Smart NIC(3 cores) 18.9/50.7/ /46.7/

26 Testing Conclusion Smart NIC vs. SR-IOV Line rate: 512B Smart NIC = SR-IOV; 256B & 84B Smart NIC = SRIOV*70% Latency: Smart NIC>SRIOV,(due to software processing), Virtio-BE offload Smart NIC vs. vswitch When Flows grow, Smart NIC=vSwitch*3 If simply calculate, vswitch needs 9 cores to achieve same performance of Smart NIC, Concidering across NUMA, it needs over 10 physical cores, there will be no enough CPU for VNFs. 26

27 Future research 1. FPGA? FPGA+NP? 2. GPU + FPGA resource pool architecture design 3. Acceleration point deep mining 4. Start vepc acceleration testing 5. Design Accelerator common API & push to community 6. Design accelerators management spec 7. FPGA detailed spec that we need 8. GPU usage in MEC, AR/VR 9. OPNFV project research on 4G/5G FWD plane & Data plane, including APIs 27