Nova Scheduler: Optimizing, Configuring and Deploying NFV VNF's on OpenStack

Size: px

Start display at page:

Download "Nova Scheduler: Optimizing, Configuring and Deploying NFV VNF's on OpenStack"

Rhoda Bishop
6 years ago
Views:

3 Our software has been deployed in over 2 billion devices; into environments, systems, and applications subject to the highest standards of safety, security and performance WIND RIVER. ALL RIGHTS RESERVED.

Deployed Throughout Wired and Wireless Networks Secure, reliable, low-latency networking for service providers IoT Infrastructure Radio Access Network Virtual CPE Networks Network Appliances Content

4 Deployed Throughout Wired and Wireless Networks Secure, reliable, low-latency networking for service providers IoT Infrastructure Radio Access Network Virtual CPE Networks Network Appliances Content Delivery Networks Mobile Edge Computing Proprietary Open 4G 5G Vertically Integrated Multivendor Physical Virtual Appliances Servers Titanium Cloud Private Cloud for Critical Infrastructure WIND RIVER. ALL RIGHTS RESERVED.

6 Areas To Cover Requirements of typical NFV application Nova scheduler is very capable, but complex. We will show how to tune guests to maximize performance of applications for NFV. We will compare performance with and without leveraging the guest tuning parameters that are available and highlight the delta WIND RIVER. ALL RIGHTS RESERVED.

7 Requirements of an NFV Application Deterministic network throughput and latency Typically higher throughput than a kernel based vswitch can provide Packet Latency below 50us Compute profiles with low interrupt latency variation Predictable access to CPU performance Predictable memory access throughput and latency Typically a large number of vcpus are required Typically running Linux in the guest WIND RIVER. ALL RIGHTS RESERVED.

9 Enhanced Platform Awareness Optimize VM placement NUMA Memory requirements NIC support Acceleration engines Hyper-threading Configure flavours for best performance Predictable Performance for Applications WIND RIVER. ALL RIGHTS RESERVED.

10 PCI Bus, Network Contention PCI bus contention Each PCI bus is connected to a single host NUMA node Instances using PCI devices will be affined to the same NUMA node as the device Cross-NUMA traffic should be avoided Vswitch processes should run on the same NUMA node as the physical NICs they re using PT or SRIOV will be impacted by crossing NUMA nodes. Instances should run on the same NUMA node as the NIC Planning to submit a blueprint to add this functionality. Network bandwidth contention All instances connecting to the same host network will use the same host NIC 10G or higher NICs are desirable WIND RIVER. ALL RIGHTS RESERVED.

11 Host/Guest Network Bandwidth Limitation Emulated NICs (e1000, rtl8139, etc.) are slow Paravirtualized NICs are faster if the guest supports it (virtio) PCI-passthrough/SR-IOV is faster yet PCI device passed through into the guest (this can be complex) pci_alias={ "vendor_id": "8086", "product_id": "1528", "name": "my_pf"} in nova.conf "pci_passthrough:alias=my_pf:1 in flavor extra-spec Need suitable device driver in the guest Still have overhead of virtual interrupts Can be tricky to configure everything initially. A guest that leverages DPDK is fastest Poll-mode driver, works best with dedicated core(s) on host and in guest. Leveraging a poll-mode driver has the guest in a tight loop draws more power WIND RIVER. ALL RIGHTS RESERVED.

Memory Contention Host TLB contention The TLB caches the virtual page to physical page mapping Shared by all processes on a host CPU core Dedicated CPUs means less contention between guests -- still

12 Memory Contention Host TLB contention The TLB caches the virtual page to physical page mapping Shared by all processes on a host CPU core Dedicated CPUs means less contention between guests -- still have contention within guest Use hugepages to minimize contention fewer mappings, more likely to stay within the cache Memory contention (defaults to 1.5x overcommit) Use hugepages this disables overcommit completely Set in flavor extra spec or image properties: hw:mem_page_size=<large > Requires hugepages to be reserved on host, ideally at boot time Memory bandwidth contention Each guest NUMA node is mapped to a unique host NUMA node Spread the VM across NUMA nodes to spread memory access Impact: may limit viable hosts for the scheduler WIND RIVER. ALL RIGHTS RESERVED.

Host CPU Contention Between Guests By default, nova allows 16x overcommit of CPU Each host CPU can run up to 16 vcpu threads For better performance, reduce overcommit Per-compute-node Set

13 Host CPU Contention Between Guests By default, nova allows 16x overcommit of CPU Each host CPU can run up to 16 vcpu threads For better performance, reduce overcommit Per-compute-node Set cpu_allocation_ratio = <value> in nova.conf Per-host-aggregate Run nova aggregate-set-metadata <aggregate> cpu_allocation_ratio=<value> For best performance, ensure each host CPU is owned by a single guest Disables overcommit, also reduces host scheduling overhead Set in flavor extra-spec or image properties: hw:cpu_policy = dedicated Can t mix instances with dedicated/shared CPUs on one compute node Use host aggregates to manage, with flavor key matching host aggregate metadata key WIND RIVER. ALL RIGHTS RESERVED. 13

14 Physical Contention Between Guests Application performance is maximized when you have a single guest running on each host physical core Different guests running on hyperthread siblings are likely to impact performance In most cases application performance is maximized with a single guest vcpu per host physical core Set in flavor extra-spec or image properties: hw:cpu_thread_policy = isolate This reserves any other siblings (if any) of the same host core and keeps them idle In some cases application performance benefits from running on hyperthread siblings (due to shared cache or increased pipeline efficiency, for example) Set in flavor extra-spec or image properties: hw:cpu_thread_policy = require Highly recommend actually testing to see if hyperthreading is beneficial WIND RIVER. ALL RIGHTS RESERVED.

CPU Cycle Stealing By Host SMI (system management interrupt) avoidance SMIs on host can cause significant latency spikes May need to tweak BIOS or select different hardware Created by Dooder -

15 CPU Cycle Stealing By Host SMI (system management interrupt) avoidance SMIs on host can cause significant latency spikes May need to tweak BIOS or select different hardware Created by Dooder - Freepik.com CPU stealing by host processes, kernel threads, etc. Dedicate specific CPUs to host management, reserve the rest for instances Set vcpu_pin_set=<guest cpu range> in nova.conf Useful kernel boot args: isolcpus, rcu_nocbs, nohz_full Generally these would be matched to vcpu_pin_set WIND RIVER. ALL RIGHTS RESERVED.

Performance Comparison Baseline Setup The basic test involves two instances with 2 vcpus, dedicated, with the isolate CPU thread policy The image used was based on CentOS 7 In all cases vswitch is

19 Performance Comparison Baseline Setup The basic test involves two instances with 2 vcpus, dedicated, with the isolate CPU thread policy The image used was based on CentOS 7 In all cases vswitch is running on 2 host CPUs on host NUMA node 0. On one VM iperf was run as "taskset -c 1 iperf -s", and on the other it was run as "taskset -c 1 iperf -c <ip_addr> -t i WIND RIVER. ALL RIGHTS RESERVED.

20 Performance Comparison Virtual Interface Model Memory vswitch VM-0 Socket 0 VM-1 QPI Socket 0 Memory NUMA Node 0 PCIe PCIe NUMA Node 1 Compute Node Changed from e1000 to virtio vif_model Result was 4X faster Avoid emulated hardware if possible! WIND RIVER. ALL RIGHTS RESERVED.

21 Performance Comparison Memory Backing Page Size Memory vswitch VM-0 Socket 0 VM-1 QPI Socket 0 Memory NUMA Node 0 PCIe Changed page size from 4KiB to 2MiB Result was 4.3X faster Changed page size from 2MiB to 1GiB No significant performance change Compute Node PCIe NUMA Node 1 2MiB page size is the sweet spot WIND RIVER. ALL RIGHTS RESERVED.

22 Performance Comparison Cross-NUMA Traffic Memory vswitch Socket 0 QPI Socket 0 VM-0 VM-1 Memory NUMA Node 0 PCIe PCIe NUMA Node 1 Compute Node Move both VMs to same host NUMA node as vswitch Result was 1.9X faster Minimize cross-numa traffic! WIND RIVER. ALL RIGHTS RESERVED.

23 Performance Comparison Hyperthread Contention Memory Thread 1 Thread 0 0 Thread 1 Thread 0 1 NUMA Node 0 vswitch Socket 0 vcpu0 from both guests on HT siblings of same host core, vcpu1 from both guests on HT siblings of another host core Reduced performance by 37% PCIe WIND RIVER. ALL RIGHTS RESERVED. Thread 1 Thread 0 2 Thread 1 Thread 0 3 Thread 1 Thread 0 4 Thread 1 Thread 0 5 Thread 1 Thread 0 6 Only one Pipeline VM-0 VM-1

24 Summary NFV applications have very stringent performance requirements There are many options for configuring OpenStack and the guest CPU isolation can give 16X higher performance Huge page backing yields 4X higher Network using para-virtualized NIC gives 4X Leverage EPA feature NUMA awareness gives ~2X These are interdependent, changing one may not help if bottlenecks remain. Predictable performance will speed adoption of NFV Don t forget about storage but that is another talk for a future summit Come see our talk on Thursday on what is happening under the covers 4:10pm Ballroom A, 3 rd floor WIND RIVER. ALL RIGHTS RESERVED.

Achieve Low Latency NFV with Openstack*

Achieve Low Latency NFV with Openstack* Yunhong Jiang Yunhong.Jiang@intel.com *Other names and brands may be claimed as the property of others. Agenda NFV and network latency Why network latency on NFV