Ed Warnicke, Cisco. Tomasz Zawadzki, Intel

Transcription

1 Ed Warnicke, Cisco Tomasz Zawadzki, Intel

2 Agenda SPDK iscsi target overview FD.io and VPP SPDK iscsi VPP integration Q&A 2

3 Notices & Disclaimers Intel technologies features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. For more complete information about performance and benchmark results, visit Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit Benchmark results were obtained prior to implementation of recent software patches and firmware updates intended to address exploits referred to as "Spectre" and "Meltdown." Implementation of these updates may make these results inapplicable to your device or system. Intel Advanced Vector Extensions (Intel AVX)* provides higher throughput to certain processor operations. Due to varying processor power characteristics, utilizing AVX instructions may cause a) some parts to operate at less than the rated frequency and b) some parts with Intel Turbo Boost Technology 2.0 to not achieve any or maximum turbo frequencies. Performance varies depending on hardware, software, and system configuration and you can learn more at Intel's compilers may or may not optimize to the same degree for non-intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction. Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate Intel Corporation. Intel, the Intel logo, and Intel Xeon are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as property of others.

4 4

5 Moving to Userspace Alternate solutions (RDMA) are moving in strides TCP/IP transport has been present for much longer There are still use cases for TCP/IP Even NVMe-oF transport will be using it in the future Intel Builders 5

6 SPDK iscsi target overview Using POSIX sockets for data path negates benefits of userspace storage services by: K E R NIC Driver Having syscalls go to kernel and back Adding back interrupts N E L S P L2/L3 MAC/IP L4 TCP Kernel A C E U POSIX sockets S E R iscsi target S P A C E Block Device Abstraction NVMe Driver SPDK Intel Builders 6

7 7

8 FD.io: The Universal Dataplane Project at Linux Foundation Multi-party Multi-project Software Dataplane High throughput Low Latency Feature Rich Resource Efficient Bare Metal/VM/Container Multiplatform Fd.io Scope: Network IO - NIC/vNIC <-> cores/threads Packet Processing Classify/Transform/Prioritize/Forward/Terminat e Dataplane Management Agents - ControlPlane Bare Metal/VM/Container Dataplane Management Agent Packet Processing Network IO FD.io Foundation 8

9 VPP Vector Packet Processing Compute Optimized SW Network Platform Bare-metal / VM / Container Dataplane Management Agent Packet Processing Network IO Packet Processing Software Platform High performance Linux user space Runs on compute CPUs: - And knows how to run them well! Shipping at volume in server & embedded products 9

10 VPP How does it work? Compute Optimized SW Network Platform 1 Packet processing is decomposed into a directed graph of nodes 2 packets move through graph nodes in vector 3 graph nodes are optimized to fit inside the instruction cache vhost-userinput af-packetinput dpdk-input Packet 0 Packet 1 Microprocessor arp-input cdp-input lldp-input mpls-input l2-input ip4-lookup* ip4-input ip4-lookupmulitcast ethernetinput...-nochecksum ip6-input Packet 2 Packet 3 Packet 4 Packet 5 Packet 6 Packet Instruction Cache Data Cache Packet 8 interfaceoutput ip4-loadbalance mpls-policyencap ip4-rewritetransit ip4- midchain Packet 9 Packet 10 4 packets are pre-fetched into the data cache. * Each graph node implements a micro-nf, a micro-networkfunction processing packets. Makes use of modern Intel Xeon Processor micro-architectures. Instruction cache & data cache always hot Minimized memory latency and usage.

11 VPP Architecture: Packet Processing Packet n Vector of n packets dpdkinput vhost-user-input ethernet-input af-packet-input Input Graph Node Graph Node Packet Processing Graph ip6-input ip4-input mpls-input arp-input ip6- lookup ip4- lookup ip6-rewrite ip6-local ip4-local ip4- rewrite

12 VPP Architecture: Splitting the Vector Packet n Vector of n packets dpdkinput vhost-user-input ethernet-input af-packet-input Input Graph Node Graph Node Packet Processing Graph ip6-input ip4-input mpls-input arp-input ip6- lookup ip4- lookup ip6-rewrite ip6-local ip4-local ip4- rewrite

13 VPP Architecture: Plugins Packet n Hardware Plugin Vector of n packets hw-accel-input dpdkinput vhost-user-input ethernet-input af-packet-input Input Graph Node Graph Node Packet Processing Graph Skip sftw nodes where work is done by hardware already ip6-rewrite ip6-input ip6- lookup ip6-local ip4-input ip4- lookup ip4-local mpls-input ip4- rewrite arp-input Plugin /usr/lib/vpp_plugins/foo.so custom-1 custom-2 custom-3 Plugins are: First class citizens That can: Add graph nodes Add API Rearrange the graph Can be built independently of VPP source tree

14 FD.io Foundation 14

15 K8s Networking Microservice: Contiv-VPP K8s Master Node Node Pod Pod Pod tapv2 Pod Pod Pod vet h Kubelet CNI CNI Kubelet Pod Pod Pod vet h Pod Pod Pod tapv2 VPP VPP Agent VPP Agent VPP Contiv-VPP vswitch CNF Pod Contiv-VPP IPv4/IPv6/SRv6 Network

16 Motivation: Container networking PID 1234 PID 4321 glibc send() recv() FIFO kernel FIFO TCP TCP IP (routing) IP (routing) device device FD.io Mini-Summit at KubeCon Europe 2018

17 Why not this? PID 1234 PID 4321 send() recv() FIFO FIFO Session TCP IP DPDK VPP FD.io Mini-Summit at KubeCon Europe 2018

18 VPP Host Stack App Binary API rx tx shm segment Session TCP IP, DPDK VPP FD.io Mini-Summit at KubeCon Europe 2018

19 VPP Host Stack: SVM FIFOs App Binary API rx tx shm segment Session TCP IP, DPDK VPP Allocated within shared memory segments with or without file backing (ssvm/memfd) Fixed position and size Lock free enqueue/dequeue but atomic size increment Option to dequeue/peek data Support for out-of-order data enqueues FD.io Mini-Summit at KubeCon Europe 2018

20 VPP Host Stack: TCP App Clean-slate implementation Complete state machine implementation Connection management and flow control (window management) Timers and retransmission, fast retransmit, SACK NewReno congestion control, SACK based fast recovery Checksum offloading Linux compatibility tested with IWL TCP protocol tester Binary API Session TCP IP, DPDK rx tx shm segment VPP FD.io Mini-Summit at KubeCon Europe 2018

21 SPDK w/ VPP Host Stack: More network option iscsi/spdk App Binary API rx tx shm segment Session IPv4, IPv6 Bridging/Routing MPLSoX, SRv6 VXLAN{-GPE}, Geneve, GRE Much much more TCP IP, DPDK VPP SCTP UDP TLS FD.io Mini-Summit at KubeCon Europe 2018

22 Future: Storage Unified Storage/Networking Graph - Unified Storage Networking Graph allows hyper efficient processing of blocks to packets and packets to blocks - Avoid copies - Avoid cache misses - Utilize other VPP performance tricks - Most Storage IO is connected to Network IO - Can extend with additional protocols like ROCEv2 arp-input cdp-input lldp-input mpls-input l2-input ip4-lookup* dpdk-input ip4-input vhost-userinput af-packetinput ip4-lookupmulitcast ethernetinput...-nochecksum spdk-input ip6-input tcp-output block processing iscsi ROCEv2 ip4-loadbalance mpls-policyencap ip4-rewritetransit ip4- midchain interfaceoutput FD.io Foundation 22

23 23

24 iscsi target architecture Extension K E R N E L S P A C E U S E R NIC Driver L2/L3 MAC/IP L4 TCP POSIX sockets iscsi target Kernel DPDK NIC Driver VPP Graph nodes TCP host stack VPP API iscsi target VPP U S E R S P A C E S P A C E Block Device Abstraction NVMe Driver SPDK Block Device Abstraction NVMe Driver SPDK Network Services API Storage Services Intel Builders 24

25 iscsi target architecture with VPP SPDK iscsi target is using VPP Communications Library (VCL): No kernel syscalls from top to bottom Better CPU utilization Extensive VPP networking capabilities available USERSPACE PROCESS USERSPACE PROCESS DPDK NIC Driver VPP Graph nodes TCP host stack VPP Shared memory VCL API iscsi target Block Device Abstraction NVMe Driver SPDK Network Services API Storage Services Intel Builders 25

26 net framework abstraction iscsi target is not aware of socket types used All net framework types can be used at the same Kernel VPP time POSIX sockets VPP API POSIX sockets are still available VPP support - optional at compile time Enables usage in other libraries in the future iscsi target Net framework NVMe-oF target SPDK (such as NVMe-oF target) Planned API Storage Services Intel Builders 26

27 VPP integration Key steps for running SPDK iscsi target with VPP: 1. Build SPDK with VPP support 2. Run VPP process 3. Configure interfaces using VPPCTL utility 4. Start SPDK iscsi target, which can now utilize VPP interfaces All configuration steps can be found on spdk.io iscsi target documentation Intel Builders 27

28 What about performance DATA? Intel Builders 28

29

30 Backup 30

31