NFV Infrastructure Manager with High Performance Software Switch Lagopus

Transcription

1 NFV Infrastructure Manager with High Performance Software Switch Lagopus Hirofumi Ichihara NTT Software Innovation Center Collaborator: Tomoya Hibi(NTT), Hiroki Kumazaki(NTT) Copyright 2015 NTT corp. All Rights Reserved.

2 Agenda NFV NFV Infrastructure Manager OpenStack Performance requirement for NFV OpenvSwitch and Lagopus Lagopus vs OVS Lagopus advantage Demo(video) Copyright 2015 NTT corp. All Rights Reserved. 2

3 Who Hirofumi Ichihara Engineer at NTT Software Innovation Center OpenStack developer Neutron and Devstack contributor in OpenStack projects Copyright 2015 NTT corp. All Rights Reserved. 3

4 What is NFV? NFV(Network Function Virtualization) Rapid development Reducing CAPEX Reducing OPEX Avoiding vendor lock in ref: white_ paper.pdf Copyright 2015 NTT corp. All Rights Reserved. 4

5 ETSI NFV Today main topic ref: gs/nfv/001_ 099/002/ _ 60/gs_ nfv002v010101p.pdf Copyright 2015 NTT corp. All Rights Reserved. 5

6 NFV Infrastructure Manager Manage infrastructure for VNF Compute Network Storage IaaS Softwares OpenStack CloudStack Eucalyptus VMware VM VM VM VM Hypervisor Virtual Network FW Network LR LR Physical Switch Storage FW Firewall LR Logical Router VM Virtual Machine LV LV LV LV Logical Volume Copyright 2015 NTT corp. All Rights Reserved. 6

7 OpenStack Cloud Operating System Provides API to users Multi Hypervisor support KVM, QEMU, Xen, VMware, Hyper- V, LXC and others Integrated many network appliance and storage appliance Ref: Copyright 2015 NTT corp. All Rights Reserved. 7

8 OpenStack Conceptual Architecture Ref: guide- cloud/content/conceptual- architecture.html Copyright 2015 NTT corp. All Rights Reserved. 8

9 OpenStack Release Schedule 1 release / half a year Apr 30, 2015 Kilo release Major release Oct 21 Austin Feb 3 Bexar Apr 15 Cactus Sep 22 Diablo Apr 5 Essex Sep 27 Folsom Apr 4 Grizzly Oct 17 Havana Apr 17 Icehouse Oct 16 Juno Apr 30 Kilo Provided functions Virtual environment provisioning Virtual network Block storage Horizon Web user interface Metering/Monitoring Quantum Cinder Horizon Quantum Cinder Horizon Database Heat Ceilometer Neutron Cinder Horizon Hadoop Trove Heat Ceilometer Neutron Cinder Horizon Sahara Trove Heat Ceilometer Neutron Cinder Horizon Neutron Cinder Horizon Authentication service Keystone Keystone Keystone Keystone Keystone Keystone Keystone Glance Virtual machine image Glance Glance Glance Glance Glance Glance Glance Swift Swift Object storage Swift Swift Swift Swift Swift Swift Swift Nova Nova Virtual machine Nova Nova Nova Nova Nova Nova Nova Copyright 2015 NTT corp. All Rights Reserved. 9

10 OpenStack Nova Provides API enables to define Virtual Machine Supporting a wide variety of virtualization technologies, including KVM, Xen, Docker, etc VM VM Container VM VM Container VM VM Container KVM Host Docker Host Copyright 2015 NTT corp. All Rights Reserved. 10

11 OpenStack Neutron Provides API enables to define virtual network Neutron core resources Network: Virtual network (L2 Switch) Subnet: Manage IP address assign to network Port: Port connected with network Service plugin resources Router(include Floating IP) Load balancer VPN Firewall Router VPN Firewall L2 Switch Virtual Machine Tenant A Load balancer Tenant B Copyright 2015 NTT corp. All Rights Reserved. 11

12 NFV History in OpenStack OpenStack has been designed for Data Center use cases Past Telco and vendor expect to use OpenStack to build NFV infrastructure They didnʼ t realized Hong Kong Summit 2014 They realized OpenStack NOT NFV orchestrator for CARRIER What is NFV requirement? Present Cloud Provider Telco Carrier Copyright 2015 NTT corp. All Rights Reserved. 12

13 Gaps between Cloud and Telco What is NFV requirement? Cloud Provider Canʼ t help failure User must HA A few network down User gets angry Guaranteed by service contract Sometimes Fail Telco Carrier Must help failure Provider must HA Donʼ t down network Government gets angry too Guaranteed by law 24/365 on Ready! Copyright 2015 NTT corp. All Rights Reserved. 13

14 NFV Telco Requirements Performance Packet processing speed with short packet Low latency High Availability Interface for management Stable Monitoring Fault detection Security Fault tolerance Copyright 2015 NTT corp. All Rights Reserved. 14

15 NFV History in OpenStack OpenStack has been designed for Data Center use cases Past Telco and switch vendor expects to use OpenStack to build NFV infrastructure They didnʼ t realized Hong Kong Summit 2014 After Summit Oct Juno, Kilo release They realized OpenStack NOT NFV orchestrator for CARRIER Telco Working Group and NFV subteam was organized OPNFV was organized Some NFV requirements was merged in OpenStack Present Copyright 2015 NTT corp. All Rights Reserved. 15

16 OpenStack Activity for NFV Proposed in OpenStack VLAN aware VM VM Scheduler High Availability method Service chaining API Driver for network high performance Liberty summit sessions related to NFV 41 sessions (my grep) OPNFV Days Copyright 2015 NTT corp. All Rights Reserved. 16

17 # of packet to be proceeded for 10Gbps with 1 CPU core # of packets per seconds 16,000,000 14,000,000 12,000,000 10,000,000 8,000,000 6,000,000 4,000,000 2,000,000 0 Short packet 64Byte MPPS, 67.2 ns 2Ghz: 134 clocks 3Ghz: 201 clocks Computer packet 1KByte 1.2MPPS, 835 ns 2Ghz: 1670 clocks 3Ghz: 2505 clocks Packet size (Byte) Copyright 2015 NTT corp. All Rights Reserved. 17

18 OpenvSwitch/Lagopus OpenvSwitch Multilayer software switch Supports OpenFlow 1.4 protocol Including DPDK implement Source code: Lagopus High performance software switch with DPDK Supports OpenFlow 1.3 protocol Source code: Copyright 2015 NTT corp. All Rights Reserved. 18

19 OpenFlow Conventional NW node OpenFlow OpenFlow controller Control plane (Routing / switching) Control plane (routing/ switching) OpenFlow Protocol Data- plane (ASIC, FPGA) Flow match OpenFlow switch OpenFlow switch agent Flow Table action Data- plane counter Flow match action counter Flow Table #2 Flow Table #3 Flexible flow match pattern (Port #, VLAN ID, MAC addr, IP addr, TCP port #) Action (frame processing) (output port, drop, modify/pop/push header) Flow statistics (# of packet, byte size, flow duration, ) Copyright 2015 NTT corp. All Rights Reserved. 19

20 high- performance user- space packet processing with Intel DPDK Userspace packet processing (Event- based) vswitch agent Dataplane User space DPDK apps (polling- based) vswitch DPDK agent dataplane Socket API 2. system call (read) 3. system call (write) Socket API sk_ buf Kernel space 1. DMA Write 2. DMA READ 1. Interrupt & DMA Ethernet Driver API 4. DMA Ethernet Driver API NIC packet buffer Driver packet NIC buffer Copyright 2015 NTT corp. All Rights Reserved. 20

21 Issues about high performance packet processing on x86 Many rx interrupt causes overhead Polling- based packet receiving Lock with multiple thread causes bottleneck Thread local storage and lockless- queue High TLB miss rates causes slowing performance Huge DTLB for miss of memory controller Context switch and memory copy causes overhead Direct data copy from NIC buffer to CPU Kernel stack bypass of network Copyright 2015 NTT corp. All Rights Reserved. 21

22 About Lagopus OpenFlow software switch Run on x86 server High performance packet processing > 10Gbps Multiple protocol Conform OpenFlow1.3 Protocol(MPLS, PBB) for WAN OSS OpenFlow Controller Control Plane OpenFlow Protocol Flow Pattern Flow Table OpenFlow Switch Action Data Plane Counter Flow Pattern Action Counter Flow Table #2 Flow Table #3 Flow Table #4 Copyright 2015 NTT corp. All Rights Reserved. 22

23 Switch Performance Conditions OVS- DPDK master Back for DPDK commit 66cabc46ecc09eeae536277a0fc7d5e44836f845 Lagopus v0.1.2 DPDK System CPU: Xeon 2.70GHz(12 cores to switch) Memory: 64GB(hugepages: 1GBx8) NIC: Intel X520 10GNIC OpenFlow Controller Ryu ( Traffic Generator IXIA commit a41cbceffaac30ace2c66b6e489eb359c8 Author: Mark Kavanagh Date: Mon Apr 20 12:37: DPDK: add support for v2.0.0 Update relevant artifacts to add support for DPDK v INSTALL.DPDK.md - travis build script - acinclude.m4: add 'mssse3' flag to OVS_ CFLAGS - netdev- dpdk: fix build with unified offload types in DPDK v2.0.0 Note that this breaks compatibility with DPDK v1.8.0 Copyright 2015 NTT corp. All Rights Reserved. 23

24 DPDK config diff - - git a/config/common_ linuxapp b/config/common_ linuxapp index 2f9643b dc a/config/common_ linuxapp +++ b/config/common_ - 81,7 CONFIG_ RTE_ BUILD_ SHARED_ LIB=n # # Combine to one single library # - CONFIG_ RTE_ BUILD_ COMBINE_ LIBS=n +CONFIG_ RTE_ BUILD_ COMBINE_ LIBS=y CONFIG_ RTE_ LIBNAME="intel_ dpdk" - 160,7 CONFIG_ RTE_ LIBRTE_ IXGBE_ DEBUG_ TX_ FREE=n CONFIG_ RTE_ LIBRTE_ IXGBE_ DEBUG_ DRIVER=n CONFIG_ RTE_ LIBRTE_ IXGBE_ PF_ DISABLE_ STRIP_ CRC=n CONFIG_ RTE_ LIBRTE_ IXGBE_ RX_ ALLOW_ BULK_ ALLOC=y - CONFIG_ RTE_ IXGBE_ INC_ VECTOR=y +CONFIG_ RTE_ IXGBE_ INC_ VECTOR=n CONFIG_ RTE_ IXGBE_ RX_ OLFLAGS_ ENABLE=y - 372,7 CONFIG_ RTE_ KNI_ VHOST_ DEBUG_ TX=n # fuse- devel is needed to run vhost. # fuse- devel enables user space char driver development # - CONFIG_ RTE_ LIBRTE_ VHOST=n +CONFIG_ RTE_ LIBRTE_ VHOST=y CONFIG_ RTE_ LIBRTE_ VHOST_ DEBUG=n # Copyright 2015 NTT corp. All Rights Reserved. 24

25 Test scenario Full route Set about 51,000 flow rule (IPv4) packets: 1flow, random ipv4_ dst flow Port VLAN Set port VLAN to two port (2 flow rule for round trip) packets: 1flow, random ipv4_ dst 10000flow Copyright 2015 NTT corp. All Rights Reserved. 25

26 Evaluation - full route Lagopus wins with 1flow packets OVS- DPDK wins with flow packets Result 1flow packets Result flow packets Wire- rate Gbps Wire- rate OVS- DPDK Lagopus Gbps OVS- DPDK Lagopus byte/packet byte/packet Copyright 2015 NTT corp. All Rights Reserved. 26

27 Evaluation port VLAN Lagopus wins with 1flow packets Lagopus wins with 10000flow packets Result 1flow packets Result 10000flow packets Gbps Wire- rate OVS- DPDK Lagopus byte/packet Gbps Wire- rate OVS- DPDK Lagopus byte/packet 1500 Copyright 2015 NTT corp. All Rights Reserved. 27

28 Lagopus vs OVS- DPDK Lagopus has some advantages Telco friendly network protocol QinQ, PBB, MPLS Comfortable more OpenFlow protocols than OVS Important protocol is Meter for NFV OVS is used by general services but lagopus is designed for network services Copyright 2015 NTT corp. All Rights Reserved. 28

29 OpenFlow 1.3 Conformance Status Type Action Set field Match Group Meter Total # of test scenario (mandatory, optional) Lagopus OVS (kernel) OVS (netdev) IVS ofswitch LINC Trema (3, 53) 56 (3, 53) 34 (3, 31) 34 (3, 31) 17 (3, 14) 50 (3, 47) 24 (3, 21) 50 (3, 47) 161 (0, 161) 161 (0, 161) 96 (0, 96) 102 (0, 102) 46 (0, 46) 100 (0, 100) 68 (0, 68) 159 (0, 159) 714 (108, 606) 714 (108, 606) 534 (108, 426) 467 (93, 374) 323 (108, 229) 708 (108, 600) 428 (108, 320) 708 (108, 600) 15 (3, 12) 15 (3, 12) 6 (3, 3) 8 (3, 5) 3 (0, 2) 15 (3, 12) 3 (3, 0) 15 (3, 12) 30 (0, 30) 30 (0, 30) 0 (0, 0) 0 (0, 0) 0 (0, 0) 30 (0, 30) 4 (0, 4) 34 (0, 34) 991 (114, 877) 976 (114, 862) 670 (114, 556) 611 (99, 556) 402 (111, 291) 962 (114, 848) 523 (114, 409) 966 (114, 854) Copyright 2015 NTT corp. All Rights Reserved. 29

30 Meter Demo System Intel(R) Core(TM) i GHz Memory 16GB(hugepages: 1GBx8) NIC Intel X520 10GNIC Traffic Generator Pktgen DPDK(master ) Lagopus Pktgen DPDK Flow Pattern Data Plane Flow Table Action in:port1 Meter kbps out:port2 port1 port2 Copyright 2015 NTT corp. All Rights Reserved. 30

31 Using Lagopus for NFV How do we setup lagopus for NFV? Lagopus is designed as network switch not switch for VM on hypervisor Big issue Canʼ t add/delete port OpenFlow controller essential Canʼ t try easily Copyright 2015 NTT corp. All Rights Reserved. 31

32 Gondola Made by me + other developers Tomoya Hibi: Lagopus developer Hiroki Kumazaki: Lock free master What is Gondola? Manage KVM and Docker with Hybrid tenants and support for various tunneling protocols(vlan, MPLS) Try to run Lagopus as Hypervisor switch easily Exhibited in SDN/Cloud Program Contest 2014 Okinawa Not OSS yet Copyright 2015 NTT corp. All Rights Reserved. 32

33 Gondola Architecture Using etcd for DB and notification Master node REST/GUI Scheduler Alive monitoring Manage agent nodes Agent node Manage VMs and containers Docker API gondola agent Libvirt API GUI NW IF API REST API gondola gondola gondola master management etcd db Gondola SW App. Ryu heart beat User App write VM Virtual/Phisical port Physical machine Network change notification Container VM HV KVM+Docker HV KVM+Docker HV KVM+Docker HV KVM+Docker HV KVM+Docker lagopus DC Network Copyright 2015 NTT corp. All Rights Reserved. 33

34 Demo video Demo1 Create VM/Container and ping/ssh between them Isolation between multi- tenant networks Demo2 Host (ubuntu02) kvm- test OpenStack Integration OpenStack API/GUI lagopus Host (ubuntu03) docker- test docker- test lagopus Host (ubuntu02) docker- kvm Host (ubuntu03) kvm- openstack2 docker- openstack mistake -> kvm-openstack lagopus lagopus Copyright 2015 NTT corp. All Rights Reserved. 34

35 Summary OpenStack as NFV Infrastructure Manager NFV requirements discussion DPDK for high performance packet processing Advantage of Lagopus Copyright 2015 NTT corp. All Rights Reserved. 35