OneCore Storage Performance Tuning

Transcription

1 OneCore Storage Performance Tuning Overview To improve Emulex adapter performance while using the OneCore Storage Linux drivers in a multi-core CPU environment, multiple performance tuning features can be employed by using the ocs_config.py shell script, such as: Specifying CPU affinity settings: o o Mapping a driver s interrupt requests (IRQs) to individual CPU cores (CPU affinity). Mapping driver threads to CPU cores. Setting CPU frequency scaling governor to performance mode, if appropriate. Using OCS driver port to non-uniform memory access (NUMA) node mapping to minimize latency by running OCS driver processes on local memory (versus remote memory). The ocs_config.py shell script is provided in the tools/bin directory in the SDK release package. This script should be run after the driver is loaded. Note: The ocs_config.py script combines and replaces the previous ocsmknod, ocs_perf_conf, and ocs_thread_perf_config.py scripts. The following sections describe the performance tuning details. MSI-X Interrupts and CPU Affinity Introduction The OneCore Storage (OCS) Linux reference drivers use MSI-X interrupts. Each PCI function uses one MSI-X assigned interrupt vector. In a multi-core CPU environment, it is generally expected that each OCS interrupt handler instance is run on its own CPU core. However, on some Linux systems, multiple OCS interrupt handlers can share CPU cores. This condition is not optimal for leveling CPU utilization across CPU cores, and can negatively impact input/output operations per second (IOPS) performance. To insure OCS interrupt handlers are distributed across CPU cores and optimize IOPS performance, each OCS interrupt handler can be explicitly bound to its own CPU core. 1 OneCore Storage Performance Tuning April 2015

2 Determining if Multiple Interrupt Handlers Share the Same CPU Core Before explicitly binding OCS interrupt handlers, you must determine whether multiple interrupt handlers are sharing the same CPU core. The interrupt counts for each CPU/driver can be observed by examining the /proc/interrupts file after running IOs on all OCS adapter ports. To display the number of interrupts that each OCS interrupt handler has used on each CPU, issue a grep ocs /proc/interrupts command. The following example shows a grep ocs /proc/interrupts output, where two OCS interrupt handlers are sharing the same CPU core (CPU2): CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 30: PCI-MSI-edge ocs 31: PCI-MSI-edge ocs In contrast, the following output shows two OCS interrupt handlers that are using different CPU cores (CPU2 and CPU3): CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 30: PCI-MSI-edge ocs 31: PCI-MSI-edge ocs Note: The number in the first column (30 and 31 in the previous examples) is the MSI-X interrupt vector assigned to each OCS interrupt handler. Binding an Interrupt Handler (IRQ) to Its Own CPU Core Each interrupt vector is represented by an entry in /proc/irq/<vector>. The /proc/irq/<vector>/smp_affinity entry is a read/write value that indicates the vector-to-cpu binding. This value represents a bitmask of CPUs that may service the interrupt. For example, a value of 1 in bit position 0 indicates that the interrupt handler only runs on CPU0. By default, a value of 1 is set for all the bit positions. For example, on an 8-core CPU, the typical value is 0xff, allowing the interrupt handler to run on any CPU. If you write a single bit value (1, 2, 4, 8 and so on) to the smp_affinity entry, the interrupt handler only runs on that corresponding CPU core. The interrupt vectors may be determined from examining the proc/ocs_* entries: # cat /proc/ocs_fc_ramd 247,4 0,113, ,114, ,115, OneCore Storage Performance Tuning April 2015

3 3,116,10719 In this example, there are four devices, using interrupt vectors 113, 114, 115, and 116. The CPU affinity may be set to CPUs 2-5 as follows: echo 4 > /proc/irq/113/smp_affinity echo 8 > /proc/irq/114/smp_affinity echo 10 > /proc/irq/115/smp_affinity echo 20 > /proc/irq/116/smp_affinity When binding interrupts to CPU cores, there are a few recommendations that may improve performance: Avoid binding to CPU0 and CPU1 (depending on the kernel version). If using hyper-threaded processors with physical cores and virtual hyper-threaded cores, bind the per-port interrupts to physical cores. That is, do not bind interrupts to a hyper-threaded core on the same physical core. Binding Threads to CPU Cores and Setting Thread Priority By default, the OneCore Storage drivers execute within kernel threads. The kernel thread priority and CPU affinity may be set to improve performance. The taskset and chrt programs are used to specify the CPU affinity and priority. The first step is to determine the process IDs for the OCS kernel threads: # cat /proc/ocs_fc_ramd 247,4 0,113, ,114, ,115, ,116,10719 In this example, there are four devices, with process IDs 10716, 10717, 10718, and The taskset command may be used to bind the threads to CPU cores 2-5. taskset c p taskset c p taskset c p taskset c p OneCore Storage Performance Tuning April 2015

4 The thread scheduler policy and priority may be set using the chrt command. In this example, the threads are set to a policy of SCHED_FIFO with a priority of 99. chrt f p chrt f p chrt f p chrt f p Setting the CPU Frequency Scaling Governor to Performance Mode The code in the ocs_config.py script sets the CPU frequency scaling governor to performance mode. CPU frequency scaling (also known as CPU throttling) is a technique where the microprocessor is regulated to run at less-than-maximum frequency to conserve power. When the CPU frequency scaling governor is set to performance mode, the CPU runs at the highest frequency within the specified minimum and maximum frequency limits. Using OCS Driver Port to NUMA Node Mapping Note: This feature is applicable only on systems that support NUMA and have NUMA enabled. In some systems/applications, it is possible that OCS driver port processes are run on remote memory (memory that is not on the same bus as the adapter). In this scenario, the processes are running on a longer signal path as compared to processes running on local memory (memory that is on the same bus as the adapter). This longer signal path not only increases latency but can become a throughput bottleneck if the signal path is shared by multiple CPUs. NUMA was designed to alleviate these latency issues and bottlenecks by grouping compute resources and memory into nodes. A NUMA node typically includes multiple CPUs, shared memory, and a memory controller running on the same bus. Memory located in the same NUMA node as the CPU currently running the process is referred to as local memory, while any memory that does not belong to the node on which the process is currently running is considered remote. Using OCS driver port to NUMA node mapping helps to keep the processes running locally within a NUMA node, which minimizes cross-node remote memory access. The ocs_config.py script recognizes the available NUMA nodes and ensures that CPU mapping is balanced on the appropriate OCS driver port. 4 OneCore Storage Performance Tuning April 2015

5 Example The following output shows a system with two NUMA nodes (two physical processors), each having eight CPUs: # lscpu grep NUMA NUMA node(s): 2 NUMA node0 CPU(s): 0-3,8-11 NUMA node1 CPU(s): 4-7,12-15 The OCS driver port-to-numa mapping (physical adapter ports to physical CPUs) can be viewed using the elxsdkutil utility s list command. In the following output, the OCS driver port 0 (in the example, [0] Emulex/Skyhawk FCoE ) is located on NUMA node 0, whereas the OCS driver ports 1-4 are located on NUMA node 1. #./elxsdkutil list egrep "Emulex NUMA" [0] Emulex/Skyhawk FCoE modeldesc: Emulex OneConnect FCoE/NIC Adapter Physical NUMA Node: 0 NUMA Map: cpu0:node0 [1] Emulex/Lancer FCoE NUMA Map: cpu4:node1 [2] Emulex/Lancer FCoE NUMA Map: cpu5:node1 [3] Emulex/Lancer FCoE NUMA Map: cpu6:node1 [4] Emulex/Lancer FCoE NUMA Map: cpu7:node1 When the ocs_config.py script is executed, it maps the OCS driver ports and the CPUs according to the following NUMA association: #./ocs_config.py --- Checking for ocs drivers ocs_fc_ramd driver found 5 OneCore Storage Performance Tuning April 2015

6 /dev/ocs_0 already exists /dev/ocs_1 already exists /dev/ocs_2 already exists /dev/ocs_3 already exists /dev/ocs_4 already exists Bind ocs:0 vec 79 to cpu_id 0 << OCS port 0 maps to cpu 0 from NUMA node 0 Bind ocs:1 vec 80 to cpu_id 4 << OCS port 1 maps to cpu 4 from NUMA node 1 Bind ocs:2 vec 81 to cpu_id 5 Bind ocs:3 vec 82 to cpu_id 6 Bind ocs:4 vec 83 to cpu_id 7 Bind ocs:0 pid 6939 to cpu_id 0 Bind ocs:1 pid 6945 to cpu_id 4 Bind ocs:2 pid 6951 to cpu_id 5 Bind ocs:3 pid 6957 to cpu_id 6 Bind ocs:4 pid 6974 to cpu_id 7 Restarting irqbalance Copyright Emulex. All rights reserved worldwide. This document refers to various companies and products by their trade names. In most, if not all cases, their respective companies claim these designations as trademarks or registered trademarks. This information is provided for reference only. Although this information is believed to be accurate and reliable at the time of publication, Emulex assumes no responsibility for errors or omissions. Emulex reserves the right to make changes or corrections without notice. This report is the property of Emulex and may not be duplicated without permission from the Company. 6 OneCore Storage Performance Tuning April 2015