DPDK Summit China 2017

Transcription

1 Summit China 2017

2 Embedded Network Architecture Optimization Based on Lin Hao T1 Networks

3 Agenda Our History What is an embedded network device Challenge to us Requirements for device today Our solution T1 unique embedded network architecture(t1-system) Model of embedded network architecture History of T1-system Business layer of T1-system An optimization case dual-socket system T1-system as a NFV

4 Our History T1 Networks Professional application delivery & High-performance fusion of network security products Harbor Networks Corp. Venustech Corp. T1 networks Corp. Product: Router Product: UTM Product: ADC Product: NGFW HW: Freescale + Intel NP HW: Cavium OCTEON HW: X86 HW: X86 SW: vxworks + ucode SW: cvm excutivesdk SW: Linux+Netmap SW: Linux

5 Challenge to our system 1. Falling cost on network bandwidth 10Gbps 40Gbps 2.Hardware is varied and iteration fast 3.Features expansion 100Gbps 10/100/1000 Mbps Xeon I350 Atom VPN Core Policy Situation Anti-virus XL710 QOS IPS RRC X552 Compress require for our system Performance! Compatibility! Scalability!

6 Model of ENA Model of Embed Network Architecture Network traffic System components Timer Mem Business Layer Be responsible for Scalability Ring Mbuf Traffic distribute layer Be responsible for Performance Packets IO SW HW Drivers Hardware adaptation layer Be responsible for Compatibility HW I XL710

7 History of T1-system 1st Generation kernel driver based system 2nd Generation Muti-Core MIPS64 3rd Generation Dispatcher-application system 4th Generation Balanced-dispatcher -equipped system 5th Generation +FPGA system Why we need?how to use?

8 1st Gen Kernel driver based User- Space Kernel High-level Application 3 System call 2 Low-level traffic forward Advantage: Easy to get Problem: 1 Bottleneck of Linux IRQ Driver 1 2 Difficult to develop and optimize HW NIC 3 Inefficient system call Network traffic

9 2rd Gen Muti-Core MIPS64 Ctrl_plane: Linux OS Data_plane: Non-OS app code Advantage: Excellent throughput performance Cores Problem: SW 1 Performance decline on complex feature 2 Hard to develop HW Packet Schedule Unit

10 3rd Gen Dispatcher-Application User-space Core group 1 ctrl_plane APP-C APP-C APP-D APP-D Kernel with netmap patched driver HW 3 NIC APP-D APP-D APP-D APP-D Disp Disp Disp Disp 1 RSS 2 Core group 2: data_plane with netmap Core group 3: Dispatcher with netmap 1 RSS-binded packets handle 2 5-tuple hash dispatcher 3 control-plane Vs dataplane

11 3rd Gen Dispatcher-Application Advantage:Reduced Muti-core competition SHARED FLOW TABLE SEPARATED FLOW TABLE APP1 APP2 APP3 APP4 APP1 APP2 APP3 APP4 A B C D without Dispatcher Dispatchers A B C D with Dispatcher

12 3rd Gen Dispatcher-Application Problem:Bottleneck in different situation App App App App App App Bottleneck Dispatcher Inefficient Dispatcher Heavy traffic, low complexity Light traffic, high complexity

13 4th Gen -equipped system Core group : ctrl_plane Single core with APP-D Single core with APP-D Single core with APP-D Two improvement: APP-C APP-C 2 1 DISP DISP DISP 1 -equipped. 2 balance-dispatcher system. NIC RSS

14 5th Gen Maybe in the future Core group : ctrl_plane Single core with Single core with Single core with APP-C APP-C APP-D APP-D APP-D Release CPU cost from dispatcher. Avoid packets deliver between cores. Intel NIC RSS More efficient on cache scheduling. Intel FPGA Dispatcher

15 Why dispatcher in software Can not use RSS hash, why? IP_D2:port_D2 IP_S:port_S Precondition: Client Forwarding Device Server 1. HASH value of both sides must be consistent 2. port_d2 can be decided IP_C:port_C IP_D1:port_D1 Calculate process: HASH_VALUE = hash(ip_c, port_c, IP_D1, port_d1) It is difficult to perform a inverse hash based on hardware RSS HASH port_d2 = hash_inverse(hash_value, IP_D2, IP_S, port_s)

16 History of T1-system 1st Generation kernel driver based system 2nd Generation Muti-Core MIPS64 3rd Generation Dispatcher-application system 4th Generation Balanced-dispatcher -equipped system 5th Generation +FPGA system Why we need?how to use?

17 Why?? vs netmap 1. Performance: E5-2670V3 24cores/1000 policies/64-bytes throughput Throughput Latency Average 64bytes (ns) Netmap 27 Gbps Gbps 20601

18 Why?? vs netmap 2. Performance: CPU cost analysis by oprofiler Netmap lib lib System with System with Netmap

19 Why?? vs netmap 3. Code maintenance costs : Code block we should take care of User application with User application with Netmap User Intel PMD User Netmap IO Libs Kernel Linux UIO driver Kernel E1000E-netmap IGB-netmap IxGB-netmap I40e-netmap IGBVF-netmap Other drivers dr

20 Application with SW routine Pkt flow Execute flow Pkt queue functions Dispatcher rnic APP vnic queue A queue B queue C queue D MBUF Get Pkts from rnic dispatch Pkts Get Pkts from vnic Pkts Handle Flush Pkts to vnic Flush Pkts to rnic RX Ring Ring TX

21 Agenda Our History What is an embedded network device Challenge to us Requirements for device today Our solution T1 unique embedded network architecture(t1-system) Model of embedded network architecture History of T1-system Business layer of T1-system An optimization case dual-sockets system T1-system as a NFV

22 Business layer of T1-System: Multi-path traffic handle system proxy-client L7 features: SSL VPN/HTTP Comp/ TCP payload transaction proxy-server Hi-function path (Base on proxy) Hi-function path User-space Socket TRAFFIC client Performance path Server local-in L4 features: FIREWALL/L4 LB/ local-out Fast path route pre-routing forwarding post-routing Miss 5-tuple dispatch Create/Del fast patch 5-tuple Analysis forwarding table Hit PMD Performance path (Base on netfilter) Fast path (Base on dispatcher) Aim of Multi-path system: Reduce CPU cost on traffic processing. App App App Dispatcher

23 Agenda Our History What is an embedded network device Challenge to us Requirements for device today Our solution T1 unique embedded network architecture(t1-system) Model of embedded network architecture History of T1-system Business layer of T1-system An optimization case dual-sockets system T1-system as a NFV

24 Optimization on Dual-sockets platform NUMA 0 NUMA 1 QPI APP APP APP APP APP APP APP APP Basic environment: 1 Separated buffers and queues initialization on each Numa node Dispatcher Dispatcher Dispatcher 1 Mbufs RX TX 2 X 3 Dispatcher Dispatcher Dispatcher Mbufs RX TX 2 Ethernet ports bind with a singlenode. Case 1: Packets cross-qpi 3 In case of simple handle of packets, such as IP forwarding. PCIE PCIE PCIE PCIE PCIE PCIE PCIE PCIE PCIE PCIE PCIE PCIE

25 Optimization on Dual-sockets platform NUMA 0 NUMA 1 QPI APP APP APP APP APP APP APP APP Basic environment: 1 Separated buffers and queues initialization on each Numa node Dispatcher Dispatcher Dispatcher 1 Mbufs RX TX 3 2 X Dispatcher Dispatcher Dispatcher Mbufs RX TX 2 Ethernet ports bind with a singlenode. Case 2: Packets copy mode 3 In case of complex handle of packets, such as traffic audit. PCIE PCIE PCIE PCIE PCIE PCIE PCIE PCIE PCIE PCIE PCIE PCIE

26 Agenda Our History What is an embedded network device Challenge to us Requirements for device today Our solution T1 unique embedded network architecture(t1-system) Model of embedded network architecture History of T1-system Business layer of T1-system An optimization case dual-sockets system T1-system as a NFV NFV resource pool Fusion gateway New solution: OVS with

27 NFV Case1:NFV Resource pool Guest3 Intel Xeon Blade Guest2 VMa(FW) VMb(WAF) Guest1 VMc(IPS) 1 VMa(FW) VMb(WAF) VMc(IPS) NFVa(FW) NFVb(WAF) NFVc(IPS) Intel NIC with VFs VF2 VF1 VF1 VF2 VF1 VF2 VF1 VF1 VF2 VF2 VF1 VF2 VF1 VF1 VF2 VLAN1 VF1 VF2 VLAN2 VF2 VLAN3 NFV Resource pool: 1 Multiple NFVs for each guest 2 Traffic between NFVs in the same guest is forwarding by HW switch fabric 3 Traffic is isolated by vlan tag between guests Update ports Outside ports 2 HW SWITCH FABRIC VLAN1 VLAN2 VLAN3 3 scene: Multi-tenant in data-center/ same flow-define template for each tenant/elastic expansion

28 NFV Case2:Fusion gateway Intel Xeon Blade Fusion gateway: NFVa(FW) NFVb(ADC) NFVc(WAF) 1 Passthrough mode for IO Virtualization 1 Passthrough 2 Flexible flow-define rules: Intel NICs Update ports Trunk1 Trunk2 Trunk3 FW ADC WAF Web UI 2 HW SWITCH FABRIC Outside ports User defined flow rules, for example: Rule 1: Trunk1 < > Trunk2, tag value=100 Rule 2: Trunk2 < > Trunk3, tag value=101 Rule 3: p1 < > trunk1, tag value=200 Rule 4: p2 < > trunk3, tag value=201 P1 P2 P3 P4 SDN Openflow P1 scene: P2 Gateway position/face to network/high performance/feature fusion

29 About NFV-Comparison Comparison of two scenarios IO Virtualization Face to performance requirement number of VMs Configuration focus NFV resource pool VF(SR-IOV) Guest Low High Virtual machine management Fusion gateway Passthrough Network High Low flow-define rules configuration Limitation:Rely on Hardware fabric

30 New solution OVS with Guest T1 NFV T1 NFV T1 NFV T1 NFV VF Virtio VF Virtio VF Virtio VF Virtio T1 NFV T1 NFV T1 NFV T1 NFV VF Virtio VF Virtio VF Virtio VF Virtio T1 NFV VF Virtio T1 NFV VF Virtio OVS with is a low cost, more flexible alternative. Host PF Virtio backend OVS-vswitchd OVSdbserver Flow-define rules VF PF Intel NICs with SRIOV

31 Thank you!