To Grant or Not to Grant

Transcription

1 To Grant or Not to Grant (for the case of Xen network drivers) João Martins Principal Software Engineer Virtualization Team July 11, 2017

2 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 2

3 Motivation 3

4 Increasing networking speeds 10 Gbit/s is the new 1Gbit/s 25/50 Gbit/s NICs increasingly more common on servers1 The rise of 100 Gbit/s and even crazier packet budgets NVDIMM shortening storage latencies Exposes virt. backend network processing overhead Xen scaling nicely thanks to scalability work on grant tables But performance per queue hasn't changed much [1] 4

5 Xen dealing with increasing networking speeds Performance per queue ~15 Gbit/s Aggregate throughput improvements in past releases Requires a backend with a reasonable amount of VCPUs Only bulk data transfer seem to perform well (32K..64K sizes) Grants handling represent 50% of Xen backend processing Challenge: Reach 100 Gbit/s and with the least amount of vcpus 5

6 Background 6

7 netif protocol in a nutshell Two rings (fixed slot size) A ring for control purposes Currently for RSS hash keys Separate TX and RX interrupt Guest produces requests, backend responses once network stack has finished delivery Data represented as grants 7

8 Grant Tables Table with round robin recycled 'tickets' visible by Xen Owned by granter, table grown on guest request Indexes within this table are known as grant refs Contains gfn (frame), permissions(flags), domain(dom) 512 entries per 4K page Two grant table versions and protocols Granter populates a ref and fills it in shared ring Grantee changes flags to notify frontend of ref usage 8

9 Grant Tables Handling Ballooned/Fictitious pages used for mapping foreign memory Backends map/unmap/copy grefs Priviliged backends can access the whole guest addr space Only Xen sees granter/grantee tables Xen maintains private state for in-use entries Maptrack keeps track of those mapped by grantee (backend) Grows with usage, by stealing from other vcpus Grantee is given an handle which are indexes in this table 9

10 Grants and the I/O path Xen active grants (private) Maptrack vcpu 0 Maptrack vcpu 1... Frame Maptrack vcpu n Frame Frame Frame N X..X+512 grant table unmap: handle=10 guestrx (kthread) status: 84 Frame RX Pick Grant Ref vifx.y NAPI Dealloc (softirq) (kthread) Frame Frame N X..X+512 write frame=<gfn> handle=10 map: ref=500 copy: ref=2015 len=84 gfn xmit ref: 500 Backend (Linux) TX ref: 500 Frontend (Linux) 10

11 Proposal 11

12 Staging grants - Idea Backend maps/unmaps a permanent set of grefs Initially planned for the linear packet region only Frontend copies or... Alternatively, recycles the permanent grefs (Rx-only on Linux) Similar to persistent grants except: No periodic purging of grants i.e. backend doesn't try to map all incoming grefs Guest has more control on managing grants The protocol is weighted in favour of guest recycling buffers 12

13 Staging grants - Implementation Side-channel for co-managing long-lived grants Introduce 3 new control ring messages: CTRL_TYPE_{ADD,PUT}_GREF_MAPPING CTRL_TYPE_GET_GREF_MAPPING_SIZE Frontend requests <ring-size> (256) grefs mapping on state Connected Alternatively uses a TX RX slot flag for shortcutting the map Backend keeps a hash table of ref page 13

14 Grants and the I/O path (staging grants) grant table memcpy: ref=700 len=84 Frame Frame Frame N X..X+512 Premapped Frames Table ref: 500 ro ref: 700 rw... status: 84 RX vifx.y NAPI (softirq) Backend ref: 500 (fetch from pregranted tbl) TX memcpy: ref: 500 gfn 14

15 Staging grants - Page Recycling Having a permanent set of grefs requires either strategies: 1) Use a fixed set of buffers and copy from/to [implemented] 2) Recycle grefs to avoid copy [implemented] Recycling implemented on Intel drivers and Mellanox drivers Rx only; Tx requires copy to the mapped grefs Page reference count tells driver when page is reusable Userspace zerocopy RX proposed recently1 [1] [RFC PATCH 2/2] ixgbe: add af_packet direct copy support, 15

16 Staging grants - Page Recycling The trick is to chase our own tail i.e. The bigger the ring the longer it takes to consume the oldest entry Case for multi-page rings? I/O path to perform as if no grants had to be used! No copy to permanent grant CPU usage decreased 16

17 Staging grants and Fast Packet I/O engines Fast Packet I/O engines (netmap, DPDK, XDP) are known for: Preallocation of buffers / Static DMA mappings Packet batching Minimum amount of metadata per packet (compared to skb or mbufs in Linux/FreeBSD) Express Data Plane (XDP) introduced in Linux 4.9 to compete with DPDK Aiming at improving netif protocol for these cases 17

18 Experimental Setup ConnectX-4 100Gbit/s (mlx5) Xen Baremetal Intel(R) Xeon(R) CPU E GHz (2x10 cores) Assign each RX TX event channel to a separate CPU Pin Guest/Dom0 CPUs to separate cores, Dom0 has 5 CPUs. Linux v Xen 4.10-unstable (commit 3d2010f9ff) 18

19 Staging grants Throughput 19

20 Staging grants Rx Page Recycling Efficiency 20

21 Staging grants Packet I/O *DomU Dom0 (Local, no bridge/nic involved) 21

22 Staging grants Remarks Database being the perfect storm in perf. requirements 2.4x better handling OLTP workloads with short payloads CPU utilization on backend decreased from 38 % 22% XDP (with staging grants) does with 1 core what the stack does with 4! 22

23 What IF we didn't have grants at all? 23

24 Project Zerogrant Is it that crazy? All powerful Domain-0 can access pretty much the whole gfn space For typical Domain-0 setup there's not really a violation of policy With Paging and Ballooning it is hard to cope with P2M changes Grant tables are private to DOMID_SELF or Xen Need to extend ABI to allow map/unmap foreign grant tables Keeping guest mapped adds up a significant memory requirements to Domain-0 struct page overhead (64 bytes i.e. 4GB for a 256 GB guest) plus any added per-page metadata 24

25 Project Zerogrant Idea for Discussion Map the whole guest on Domain-0 Restricted to PV backends Map the guests grant table Allow a gref_to_pfn(...) semantics on Domain-0 Only for the privileged domain Frontend allowing blacklisting certain pfns to be mapped Come discuss on design session Zerogrant on Xen, Wed 4:35PM! 25

26 Zerogrant PoC Throughput * Unmodified v4.10 guest like Baseline 26

27 Summary of presented schemes Normal grants for heavy consolidated systems Staging grants for performance No grants for highest performance Map + unmap as needed Backend keeps fixed amount of grants mapped Exclusively for Domain-0 centric setups Copy and/or guests recycling Rx buffers (Guest choice) Grants become just a lookup for guest frame numbers Balance between I/O performance and medium consolidated environments Totally transparent to guests Add more Domain-0 CPUs for higher throughput Amount of grants mapped proportional to workload Higher memory requirements per VM 27

28 Conclusions & Future Work V2 design doc/code (XDP included!) targeting Xen 4.10 Project zerogrant for extreme performance on PV drivers Other issues identified so far: Lack of interrupt coalescing or better batching Memory locality per queue (NUMA) Infra for properly negotiating features on PV backends 28

29 29

30