Netchannel 2: Optimizing Network Performance

Transcription

1 Netchannel 2: Optimizing Network Performance J. Renato Santos +, G. (John) Janakiraman + Yoshio Turner +, Ian Pratt * + HP Labs - * XenSource/Citrix Xen Summit Nov 14-16, Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice

2 Motivation TCP performance for GigE today (PV dirvers) Throughput (Mb/s) CPU utilization (%) RX TX RX TX 60 linux xen linux xen linux xen linux xen Xen PV driver can sustain peak throughput on GigE But Xen uses significantly more CPU cycles than Linux Less available cycles for application. 10 Gig networks: CPU saturation prevents achieving line rate Need to reduce I/O virtualization overhead in Xen networking page 2

3 Netchannel 2 Netchannel2: New I/O channel protocol to enable Xen networking design changes for improving performance Work in progress Software optimizations Implementation optimizations Software design changes Devices with direct access (Direct I/O, PCI-IOV) (Device exposes multiple virtual interfaces accessed directly by ) Multi-queue devices Device has multiple RX queues Avoids data copy on receive page 3

4 Performance Analysis of Xen networking Identified main sources of overhead Results guided design choices in Netchannel 2 Emphasis on RX results higher overhead than TX page 4

5 Performance Analysis of Xen Networking (RX) RX, UDP traffic, large msg (48KB) (Xen 64bit on Intel core2 duo) 60 CPU (%) usercopy kern xen1 grantcopy kern0 xen0 0 xen linux usercopy: data copy from kernel to user buffer kern: kernel code in xen1: xen code executed in the context of the grantcopy: data copy using Xen grant (only memory copy cost) kern0: kernel code in domain 0 xen0: xen code executed in the context of domain 0 page 5

6 Implementation optimizations on RX CPU (%) usercopy kern xen1 grantcopy kern0 xen0 0 current xen no bridge netfilter align copy linear skb grant opt. linux Implementation optimizations (also possible in netchannel 1) Disabling netfilter on bridge Fix grant copy alignment problem Avoid fragments on single page packets A few optimizations in grant code page 6

7 Background: receive path on Xen today driver domain netback IO rings netfront skb GR grant copy RX TX post buffer rx skb device driver Xen NIC Hardware page 7

8 Netchannel 2: Moving grant copy to driver domain netback IO rings netfront skb RX TX rx GR skb grant copy device driver NIC Xen Hardware page 8

9 Netchannel 2: Moving grant copy to Improve performance by placing packet in the CPU cache Improve 2 nd data copy from kernel to user Avoid polluting dom0 CPU cache Improve resource management Assigns CPU cost of data copy to the Better scalability Eliminates dom0 as data copy bottleneck page 9

10 Netchannel 2: Multi queue device support Device has multiple RX queues Dedicate one RX queue to a particular Program device to demultiplex incoming packets to the dedicated queue using MAC address Post RX descriptors pointing to memory Device places received packet directly into memory avoiding data copy page 10

11 Netchannel 2: Multi-queue device support dedicated RX queue driver domain device driver netback skb GR IO rings Post RX RX TX netfront RX buffer RX NIC Xen Hardware page 11

12 Netchannel 2: Multi-queue device support Grant copy still used for inter- packet (also multicast, broadcast) other dedicated RX queue driver domain device driver netback skb GR IO rings Post RX RX TX netfront RX buffer GR skb Xen NIC Hardware page 12

13 Netchannel 2: Multi-queue device support Need to extend linux netdev interface (for native device driver) Buffer posted on dedicated queue must be allocated from the respective pool Need a new buffer allocation function that selects the memory pool based on the queue id In Xen this will be mapped to a function in netback page 13

14 Netchannel 2: Grant caching in driver domain Avoid unmapping grants, expecting that the same memory will be reused in the future Reduces grant mapping/unmapping overhead RX buffers should have high locality Windows and Linux tend to recycle RX buffers TX buffer recycling behavior is uncertain Need to evaluate experimentally In Linux we could promote buffer recycling if we can modify the skb allocator page 14

15 Netchannel 2: Grant caching in driver domain driver domain netback GR skb skb GR IO rings Post RX RX TX netfront RX buffer RX TX packet device driver GR GR GR grant cache Xen NIC Hardware page 15

16 Netchannel 2: Grant caching in driver domain driver domain device driver netback skb skb GR GR GR GR GR grant cache IO rings Post RX RX TX Control netfront RX buffer RX TX packet invalidate grant NIC Xen Hardware page 16

17 Netchannel 2: Grant extensions 1. Grant transitivity Guest authorizes grant to be transferred to other domain For to communication TX issues a grant for driver domain Driver domain transfers grant rights to RX RX uses grant to copy packet to local memory 2. New grant copy operation with range limit Grant does not give access to a full page access limited to a range specified by (offset, size) Prevents from snooping on other packets previously received on the same page page 17

18 Netchannel 2: RX performance improvements usercopy kern xen1 grantcopy kern0 xen0 0 current xen implem opt. copy Guest copy improvements linux usercopy: cached packet reduces copy overhead grant copy: lower overhead on copy (possibly because copy is cache aligned) kern: kernel also benefits from cached packet for packet header access xen: grant copy cost moved from dom0 (xen0) do (xen1); also grant optimizations are more effective on side (pin read-only page) page 18

19 Netchannel 2: RX performance improvements usercopy kern xen1 grantcopy kern0 xen0 0 current xen implem opt. copy mqueue linux multi queue multi-queue behavior emulated by modifying device driver of traditional NIC and dedicating device RX queue to page 19

20 Netchannel 2: RX performance improvements usercopy kern xen1 grantcopy kern0 xen0 0 current xen implem opt. copy mqueue linux multi queue improvements grant copy eliminated kern0: reduced network processing cost in dom0: no bridge overhead no socket buffer allocation/deallocation.overhead page 20

21 Netchannel 2: RX performance improvements CPU (%) usercopy kern xen1 grantcopy kern0 xen0 0 current xen implem opt. copy mqueue mqueue + grant reuse linux multi queue + grant cache grant caching emulated assuming 100% hit rate on cache Grant cache benefit: eliminates grant operation overhead in dom0 (xen0) page 21

22 Interrupt throttling Netback process RX packets in batches Number of packets in each batch determined by the number of packets received in each hardware interrupt Most remaining Xen overhead proportional to number of batches Interrupts, event notification/delivery, Xen scheduler runs Increasing batch size should reduce Xen overhead NIC can be configured to throttle interrupt rate (coalescing) RX interrupt delayed until N packets received (limit interrupt rate at high throughput) Or after a given timeout (limit latency at low throughput) Latency sensitive applications should not be significantly affected by larger batch size Latency effect is limited by latency coalescing parameter page 22

23 Performance impact of interrupt throttling Default configuration (6 pkt/intr) CPU (%) % copy 193% mqueue +grant cache 126% direct I/O 100% linux CPU (%) Interrupt throttling config (64 pkt/intr) % copy 123% 109% mqueue + grant cache direct I/O 100% linux usercopy kern xen1 grantcopy kern0 xen0 Interrupt throttling significantly improves Xen performance Default configuration is good for Linux (but not optimal for Xen) Xen users should use different device coalescing settings Large batches achieve almost native performance with multi-queue (23% overhead) page 23

24 Conclusion Netchannel 2 will provide: significant performance improvement for traditional NICs Overhead over linux reduced by 3.7 times on RX (370% to 100%) near native performance for RX on multi-queue devices 23% overhead over linux Multi-queue devices can be a good alternative to direct I/O devices (direct access) Slightly higher CPU cost But no hardware dependency on single device driver to maintain, test, debug, etc. Easier to migrate, easier to monitor/enforce traffic policies (firewall, rate control, etc) page 24

25 page 25