NFV Infrastructure Manager with High Performance Software Switch Lagopus

NFV Infrastructure Manager with High Performance Software Switch Lagopus Hirofumi Ichihara NTT Software Innovation Center Collaborator: Tomoya Hibi(NTT), Hiroki Kumazaki(NTT) Copyright 2015 NTT corp. All Rights Reserved.

What is NFV? NFV(Network Function Virtualization) Rapid development Reducing CAPEX Reducing OPEX Avoiding vendor lock in ref: https://portal.etsi.org/nfv/nfv_ white_ paper.pdf Copyright 2015 NTT corp. All Rights Reserved. 4

NFV Infrastructure Manager Manage infrastructure for VNF Compute Network Storage IaaS Softwares OpenStack CloudStack Eucalyptus VMware VM VM VM VM Hypervisor Virtual Network FW Network LR LR Physical Switch Storage FW Firewall LR Logical Router VM Virtual Machine LV LV LV LV Logical Volume Copyright 2015 NTT corp. All Rights Reserved. 6

OpenStack Cloud Operating System Provides API to users Multi Hypervisor support KVM, QEMU, Xen, VMware, Hyper- V, LXC and others Integrated many network appliance and storage appliance Ref: http://www.openstack.org/software/ Copyright 2015 NTT corp. All Rights Reserved. 7

OpenStack Release Schedule 1 release / half a year Apr 30, 2015 Kilo release 2010 2011 2012 2013 2014 Major release Oct 21 Austin Feb 3 Bexar Apr 15 Cactus Sep 22 Diablo Apr 5 Essex Sep 27 Folsom Apr 4 Grizzly Oct 17 Havana Apr 17 Icehouse Oct 16 Juno Apr 30 Kilo Provided functions Virtual environment provisioning Virtual network Block storage Horizon Web user interface Metering/Monitoring Quantum Cinder Horizon Quantum Cinder Horizon Database Heat Ceilometer Neutron Cinder Horizon Hadoop Trove Heat Ceilometer Neutron Cinder Horizon Sahara Trove Heat Ceilometer Neutron Cinder Horizon Neutron Cinder Horizon Authentication service Keystone Keystone Keystone Keystone Keystone Keystone Keystone Glance Virtual machine image Glance Glance Glance Glance Glance Glance Glance Swift Swift Object storage Swift Swift Swift Swift Swift Swift Swift Nova Nova Virtual machine Nova Nova Nova Nova Nova Nova Nova Copyright 2015 NTT corp. All Rights Reserved. 9

OpenStack Nova Provides API enables to define Virtual Machine Supporting a wide variety of virtualization technologies, including KVM, Xen, Docker, etc VM VM Container VM VM Container VM VM Container KVM Host Docker Host Copyright 2015 NTT corp. All Rights Reserved. 10

OpenStack Neutron Provides API enables to define virtual network Neutron core resources Network: Virtual network (L2 Switch) Subnet: Manage IP address assign to network Port: Port connected with network Service plugin resources Router(include Floating IP) Load balancer VPN Firewall Router VPN Firewall L2 Switch Virtual Machine Tenant A Load balancer Tenant B Copyright 2015 NTT corp. All Rights Reserved. 11

NFV History in OpenStack OpenStack has been designed for Data Center use cases Past Telco and vendor expect to use OpenStack to build NFV infrastructure They didnʼ t realized Hong Kong Summit 2014 They realized OpenStack NOT NFV orchestrator for CARRIER What is NFV requirement? Present Cloud Provider Telco Carrier Copyright 2015 NTT corp. All Rights Reserved. 12

Gaps between Cloud and Telco What is NFV requirement? Cloud Provider Canʼ t help failure User must HA A few network down User gets angry Guaranteed by service contract Sometimes Fail Telco Carrier Must help failure Provider must HA Donʼ t down network Government gets angry too Guaranteed by law 24/365 on Ready! Copyright 2015 NTT corp. All Rights Reserved. 13

NFV Telco Requirements Performance Packet processing speed with short packet Low latency High Availability Interface for management Stable Monitoring Fault detection Security Fault tolerance Copyright 2015 NTT corp. All Rights Reserved. 14

NFV History in OpenStack OpenStack has been designed for Data Center use cases Past Telco and switch vendor expects to use OpenStack to build NFV infrastructure They didnʼ t realized Hong Kong Summit 2014 After Summit Oct. 2014 Juno, Kilo release They realized OpenStack NOT NFV orchestrator for CARRIER Telco Working Group and NFV subteam was organized OPNFV was organized Some NFV requirements was merged in OpenStack Present Copyright 2015 NTT corp. All Rights Reserved. 15

OpenStack Activity for NFV Proposed in OpenStack VLAN aware VM VM Scheduler High Availability method Service chaining API Driver for network high performance Liberty summit sessions related to NFV 41 sessions (my grep) OPNFV Days Copyright 2015 NTT corp. All Rights Reserved. 16

# of packet to be proceeded for 10Gbps with 1 CPU core # of packets per seconds 16,000,000 14,000,000 12,000,000 10,000,000 8,000,000 6,000,000 4,000,000 2,000,000 0 Short packet 64Byte 14.88 MPPS, 67.2 ns 2Ghz: 134 clocks 3Ghz: 201 clocks Computer packet 1KByte 1.2MPPS, 835 ns 2Ghz: 1670 clocks 3Ghz: 2505 clocks 0 256 512 768 1024 1280 Packet size (Byte) Copyright 2015 NTT corp. All Rights Reserved. 17

OpenvSwitch/Lagopus OpenvSwitch Multilayer software switch Supports OpenFlow 1.4 protocol Including DPDK implement Source code: https://github.com/openvswitch/ovs Lagopus High performance software switch with DPDK Supports OpenFlow 1.3 protocol Source code: https://github.com/lagopus/lagopus Copyright 2015 NTT corp. All Rights Reserved. 18

OpenFlow Conventional NW node OpenFlow OpenFlow controller Control plane (Routing / switching) Control plane (routing/ switching) OpenFlow Protocol Data- plane (ASIC, FPGA) Flow match OpenFlow switch OpenFlow switch agent Flow Table action Data- plane counter Flow match action counter Flow Table #2 Flow Table #3 Flexible flow match pattern (Port #, VLAN ID, MAC addr, IP addr, TCP port #) Action (frame processing) (output port, drop, modify/pop/push header) Flow statistics (# of packet, byte size, flow duration, ) Copyright 2015 NTT corp. All Rights Reserved. 19

high- performance user- space packet processing with Intel DPDK Userspace packet processing (Event- based) vswitch agent Dataplane User space DPDK apps (polling- based) vswitch DPDK agent dataplane Socket API 2. system call (read) 3. system call (write) Socket API sk_ buf Kernel space 1. DMA Write 2. DMA READ 1. Interrupt & DMA Ethernet Driver API 4. DMA Ethernet Driver API NIC packet buffer Driver packet NIC buffer Copyright 2015 NTT corp. All Rights Reserved. 20

Issues about high performance packet processing on x86 Many rx interrupt causes overhead Polling- based packet receiving Lock with multiple thread causes bottleneck Thread local storage and lockless- queue High TLB miss rates causes slowing performance Huge DTLB for miss of memory controller Context switch and memory copy causes overhead Direct data copy from NIC buffer to CPU Kernel stack bypass of network Copyright 2015 NTT corp. All Rights Reserved. 21

About Lagopus OpenFlow software switch Run on x86 server High performance packet processing > 10Gbps Multiple protocol Conform OpenFlow1.3 Protocol(MPLS, PBB) for WAN OSS OpenFlow Controller Control Plane OpenFlow Protocol Flow Pattern Flow Table OpenFlow Switch Action Data Plane Counter Flow Pattern Action Counter Flow Table #2 Flow Table #3 Flow Table #4 Copyright 2015 NTT corp. All Rights Reserved. 22

Switch Performance Conditions OVS- DPDK master Back for DPDK 1.8.0 commit 66cabc46ecc09eeae536277a0fc7d5e44836f845 Lagopus v0.1.2 DPDK 1.8.0 System CPU: Xeon E5-2697v2@ 2.70GHz(12 cores to switch) Memory: 64GB(hugepages: 1GBx8) NIC: Intel X520 10GNIC OpenFlow Controller Ryu (https://github.com/osrg/ryu) Traffic Generator IXIA commit 543342a41cbceffaac30ace2c66b6e489eb359c8 Author: Mark Kavanagh <mark.b.kavanagh@intel.com> Date: Mon Apr 20 12:37:14 2015-0700 DPDK: add support for v2.0.0 Update relevant artifacts to add support for DPDK v2.0.0 - INSTALL.DPDK.md - travis build script - acinclude.m4: add 'mssse3' flag to OVS_ CFLAGS - netdev- dpdk: fix build with unified offload types in DPDK v2.0.0 Note that this breaks compatibility with DPDK v1.8.0 Copyright 2015 NTT corp. All Rights Reserved. 23

DPDK config diff - - git a/config/common_ linuxapp b/config/common_ linuxapp index 2f9643b..78738dc 100644 - - - a/config/common_ linuxapp +++ b/config/common_ linuxapp @@ - 81,7 +81,7 @@ CONFIG_ RTE_ BUILD_ SHARED_ LIB=n # # Combine to one single library # - CONFIG_ RTE_ BUILD_ COMBINE_ LIBS=n +CONFIG_ RTE_ BUILD_ COMBINE_ LIBS=y CONFIG_ RTE_ LIBNAME="intel_ dpdk" # @@ - 160,7 +160,7 @@ CONFIG_ RTE_ LIBRTE_ IXGBE_ DEBUG_ TX_ FREE=n CONFIG_ RTE_ LIBRTE_ IXGBE_ DEBUG_ DRIVER=n CONFIG_ RTE_ LIBRTE_ IXGBE_ PF_ DISABLE_ STRIP_ CRC=n CONFIG_ RTE_ LIBRTE_ IXGBE_ RX_ ALLOW_ BULK_ ALLOC=y - CONFIG_ RTE_ IXGBE_ INC_ VECTOR=y +CONFIG_ RTE_ IXGBE_ INC_ VECTOR=n CONFIG_ RTE_ IXGBE_ RX_ OLFLAGS_ ENABLE=y # @@ - 372,7 +372,7 @@ CONFIG_ RTE_ KNI_ VHOST_ DEBUG_ TX=n # fuse- devel is needed to run vhost. # fuse- devel enables user space char driver development # - CONFIG_ RTE_ LIBRTE_ VHOST=n +CONFIG_ RTE_ LIBRTE_ VHOST=y CONFIG_ RTE_ LIBRTE_ VHOST_ DEBUG=n # Copyright 2015 NTT corp. All Rights Reserved. 24

Test scenario Full route Set about 51,000 flow rule (IPv4) packets: 1flow, random ipv4_ dst 100000flow Port VLAN Set port VLAN to two port (2 flow rule for round trip) packets: 1flow, random ipv4_ dst 10000flow Copyright 2015 NTT corp. All Rights Reserved. 25

Evaluation - full route Lagopus wins with 1flow packets OVS- DPDK wins with 100000flow packets Result 1flow packets Result 100000flow packets 10 512 1024 1500 10 512 1024 1500 9 8 7 64 9 8 7 64 Wire- rate Gbps 6 5 4 3 Wire- rate OVS- DPDK Lagopus Gbps 6 5 4 3 OVS- DPDK Lagopus 2 2 1 1 0 0 500 1000 1500 byte/packet 0 0 500 1000 1500 byte/packet Copyright 2015 NTT corp. All Rights Reserved. 26

Evaluation port VLAN Lagopus wins with 1flow packets Lagopus wins with 10000flow packets Result 1flow packets Result 10000flow packets 10 512 1024 1500 10 512 1024 1500 Gbps 9 8 7 6 5 4 3 2 1 0 64 Wire- rate OVS- DPDK Lagopus 0 500 1000 1500 byte/packet Gbps 9 8 7 6 5 4 3 2 1 0 64 Wire- rate OVS- DPDK Lagopus 0 500 1000 byte/packet 1500 Copyright 2015 NTT corp. All Rights Reserved. 27

Lagopus vs OVS- DPDK Lagopus has some advantages Telco friendly network protocol QinQ, PBB, MPLS Comfortable more OpenFlow protocols than OVS Important protocol is Meter for NFV OVS is used by general services but lagopus is designed for network services Copyright 2015 NTT corp. All Rights Reserved. 28

OpenFlow 1.3 Conformance Status Type Action Set field Match Group Meter Total # of test scenario (mandatory, optional) Lagopus 2015.3.19 OVS (kernel) 2014.08.08 OVS (netdev) 2014.11.05 IVS 2015.02.11 ofswitch 2015.01.08 LINC 2015.01.29 Trema 2014.11.28 56 (3, 53) 56 (3, 53) 34 (3, 31) 34 (3, 31) 17 (3, 14) 50 (3, 47) 24 (3, 21) 50 (3, 47) 161 (0, 161) 161 (0, 161) 96 (0, 96) 102 (0, 102) 46 (0, 46) 100 (0, 100) 68 (0, 68) 159 (0, 159) 714 (108, 606) 714 (108, 606) 534 (108, 426) 467 (93, 374) 323 (108, 229) 708 (108, 600) 428 (108, 320) 708 (108, 600) 15 (3, 12) 15 (3, 12) 6 (3, 3) 8 (3, 5) 3 (0, 2) 15 (3, 12) 3 (3, 0) 15 (3, 12) http://osrg.github.io/ryu/certification.html 30 (0, 30) 30 (0, 30) 0 (0, 0) 0 (0, 0) 0 (0, 0) 30 (0, 30) 4 (0, 4) 34 (0, 34) 991 (114, 877) 976 (114, 862) 670 (114, 556) 611 (99, 556) 402 (111, 291) 962 (114, 848) 523 (114, 409) 966 (114, 854) Copyright 2015 NTT corp. All Rights Reserved. 29

Meter Demo System Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz Memory 16GB(hugepages: 1GBx8) NIC Intel X520 10GNIC Traffic Generator Pktgen DPDK(master 2015.4.28) Lagopus Pktgen DPDK Flow Pattern Data Plane Flow Table Action in:port1 Meter 100000kbps out:port2 port1 port2 Copyright 2015 NTT corp. All Rights Reserved. 30

Using Lagopus for NFV How do we setup lagopus for NFV? Lagopus is designed as network switch not switch for VM on hypervisor Big issue Canʼ t add/delete port OpenFlow controller essential Canʼ t try easily Copyright 2015 NTT corp. All Rights Reserved. 31

Gondola Made by me + other developers Tomoya Hibi: Lagopus developer Hiroki Kumazaki: Lock free master What is Gondola? Manage KVM and Docker with Hybrid tenants and support for various tunneling protocols(vlan, MPLS) Try to run Lagopus as Hypervisor switch easily Exhibited in SDN/Cloud Program Contest 2014 Okinawa Not OSS yet Copyright 2015 NTT corp. All Rights Reserved. 32

Gondola Architecture Using etcd for DB and notification Master node REST/GUI Scheduler Alive monitoring Manage agent nodes Agent node Manage VMs and containers Docker API gondola agent Libvirt API GUI NW IF API REST API gondola gondola gondola master management etcd db Gondola SW App. Ryu heart beat User App write VM Virtual/Phisical port Physical machine Network change notification Container VM HV KVM+Docker HV KVM+Docker HV KVM+Docker HV KVM+Docker HV KVM+Docker lagopus DC Network Copyright 2015 NTT corp. All Rights Reserved. 33

Demo video Demo1 Create VM/Container and ping/ssh between them Isolation between multi- tenant networks Demo2 Host (ubuntu02) kvm- test 192.168.0.10 OpenStack Integration OpenStack API/GUI lagopus Host (ubuntu03) docker- test2 192.168.0.5 docker- test 192.168.0.5 lagopus Host (ubuntu02) docker- kvm Host (ubuntu03) kvm- openstack2 docker- openstack mistake -> kvm-openstack lagopus lagopus Copyright 2015 NTT corp. All Rights Reserved. 34