VGPU ON KVM VFIO BASED MEDIATED DEVICE FRAMEWORK Neo Jia & Kirti Wankhede, 08/25/2016

Size: px

Start display at page:

Download "VGPU ON KVM VFIO BASED MEDIATED DEVICE FRAMEWORK Neo Jia & Kirti Wankhede, 08/25/2016"

Brendan Lewis
5 years ago
Views:

1 V ON K BASED MEDIATED DEVICE FEWORK Neo Jia & Kirti Wankhede, 08/25/2016

2 Background / Motivation Framework Overview AGENDA Framework Deep-Dive Current Status Demo Future work 2

TODAY, HOW PRESENTED INSIDE K device pass-through [1]

inside the virtual machine Poor density limited by PCI-E

generic vfio_pci owns this device, and only perform

Difficult to cover all graphics workload either

MACHINE VIRTUAL VIRTUAL MACHINE OS MACHINE OS Apps Apps

3 TODAY, HOW PRESENTED INSIDE K device pass-through [1] Great performance Full API compatibility vendor driver inside the virtual machine Poor density limited by PCI-E resource Minimal visibility of the device on the host generic vfio_pci owns this device, and only perform enable/disable/route interrupts, reset the device Difficult to cover all graphics workload either underutilized or too small to scale Pass-through VIRTUAL MACHINE VIRTUAL VIRTUAL MACHINE OS MACHINE OS Apps Apps Remote Apps Vendor Graphics Remote Driver Graphics NVIDIA Driver NVIDIA Driver K host _PCI DMA remapper 3

multiple virtual machines Great performance

hypervisor/host allows for device-specific

tuning performance, detailed error reporting,

MACHINE OS MACHINE OS Apps Apps Apps Remote

4 WHAT IS V? High level overview Physical shared among multiple virtual machines Great performance and suitable for different workload Full API compatibility vendor driver inside the virtual machine Full device visibility to the hypervisor/host allows for device-specific features such as dynamically monitoring and tuning performance, detailed error reporting, etc. v VIRTUAL MACHINE VIRTUAL VIRTUAL MACHINE OS MACHINE OS Apps Apps Apps Remote Vendor Graphics Remote Driver Graphics NVIDIA Driver NVIDIA Driver Vendor Driver K host DMA remapper 4

5 I/O VIRTUALIZATION SR-IOV and mediated solutions SR-IOV devices - supported by standard PCI (Direct Assignment) today Established /PCI driver, K agnostic and well-defined Virtualized PCI config /MMIO space access, interrupt delivery Modular IO, pin and map memory for DMA devices non SR-IOV, require vendor-specific drivers to mediate sharing Leveraging existing framework, - managing device s internal I/O resource 5

6 MEDIATED DEVICE FEWORK A common framework for mediated I/O devices core module (new) bus driver, create mediated device Physical device interface for vendor driver callbacks Generic mediate device management user interface (sysfs) device module (new) Manage created mediated device, fully compatible with user API IO driver (enhancement) IO API TYPE1 compatible, easy to extend to non-type1 6

7 MEDIATED DEVICE FEWORK Register CBs Vendor Vendor driver driver IO 7

8 MEDIATED DEVICE FEWORK - INITIALIZATION 8

9 MEDIATED DEVICE FEWORK With Driver Initialization Registers as driver Register CBs IO 9

10 MEDIATED DEVICE FEWORK With Driver Initialization Registers as driver registers devices Register CBs Vendor Vendor driver driver IO 10

11 MEDIATED DEVICE FEWORK With Driver Initialization Registers as driver registers devices registers CBs Register CBs IO 11

12 MEDIATED DEVICE FEWORK sysfs After vendor driver device registration, under physical device sysfs: mdev_create mdev_destroy : create a virtual device (aka mdev device) : destroy a mdev device mdev_supported_types : supported mdev and configuration of this device Mdev node: /sys/bus/mdev/devices/$mdev_uuid/ online: start and stop virtual device 12

13 MEDIATED DEVICE FEWORK With Initialization echo UUID > /sys/bus/ /<device>/ mdev_create Registers as driver registers devices registers CBs Register User writes mdev sysfs to create mdev device CBs IO 13

14 MEDIATED DEVICE FEWORK With Initialization /sys/bus/mdev/devices/$mdev_uuid Registers as driver Mdev fd registers devices registers CBs Register User writes mdev sysfs to create mdev device CBs IO calls API to add dev to IO container, group, get fd back 14

15 MEDIATED DEVICE FEWORK Vendor driver With Initialization /sys/bus/mdev/devices/$mdev_uuid Mdev fd Registers as driver registers devices registers CBs Register User writes mdev sysfs to create mdev device CBs IO calls API to add dev to IO container, group, get fd back access device fd and present it into 15

16 MEDIATED DEVICE ACCESS - EMULATED 16

17 MEDIATED DEVICE ACCESS Emulated vs Passthrough Virtual device memory region are presented inside guest for consistent view of vendor driver Access to emulated regions are redirected to mediated vendor driver for virtualization support Access to passthrough region are directly sent to device corresponding region for max performance 1 st access redirected to mediated vendor driver for CPU page table setup 17

18 MEDIATED DEVICE ACCESS Vendor driver Emulated /sys/bus/mdev/devices/$mdev_uuid Mdev fd gets region info via from vendor driver thru and CBs Register CBs K IO 18

19 MEDIATED DEVICE ACCESS Vendor driver MMIO Register CBs Emulated /sys/bus/mdev/devices/$mdev_uuid K Mdev fd gets region info via from vendor driver thru and CBs accesses MMIO trapped region backed by mdev fd triggers EPT violation /EPT IO 19

20 MEDIATED DEVICE ACCESS Vendor driver MMIO Register CBs Emulated /sys/bus/mdev/devices/$mdev_uuid K Mdev fd gets region info via from vendor driver thru and CBs accesses MMIO trapped region backed by mdev fd triggers EPT violation K services EPT violation and forwards to PCI driver /EPT IO 20

21 MEDIATED DEVICE ACCESS Vendor driver MMIO Register CBs IO Emulated /sys/bus/mdev/devices/$mdev_uuid K Mdev fd gets region info via from vendor driver thru and CBs accesses MMIO trapped region backed by mdev fd triggers EPT violation K services EPT violation and forwards to PCI driver convert request from K to R/W access to fd 21

MEDIATED DEVICE ACCESS Vendor driver MMIO Register CBs IO Emulated /sys/bus/mdev/devices/$mdev_uuid K Mdev fd gets region info via from vendor driver thru and CBs accesses MMIO trapped

22 MEDIATED DEVICE ACCESS Vendor driver MMIO Register CBs IO Emulated /sys/bus/mdev/devices/$mdev_uuid K Mdev fd gets region info via from vendor driver thru and CBs accesses MMIO trapped region backed by mdev fd triggers EPT violation K services EPT violation and forwards to PCI driver convert request from K to R/W access to fd RW handled by vendor driver via CBs and 22

23 MEDIATED DMA TRANSLATION 23

24 MEDIATED DMA TRANSLATION With Memory Tracking Starts Register CBs IO 24

25 GFN MEDIATED DMA TRANSLATION With Memory Tracking Starts Memory regions gets added by Register CBs IO 25

26 GFN GFN GFN MEDIATED DMA TRANSLATION With Memory Tracking Starts Memory regions gets added by Register calls _DMA_MAP via Memory listener CBs IO 26

27 GFN GFN GFN MEDIATED DMA TRANSLATION With Memory Tracking Starts Memory regions gets added by Register calls _DMA_MAP via Memory listener tracks <, GFN> CBs GFN IO 27

28 GFN GFN GFN MEDIATED DMA TRANSLATION With Runtime Memory pinning Starts Memory regions gets added by Register calls _DMA_MAP via Memory listener tracks <, GFN> CBs GFN pin/translate GFN by to get PFN PFN IO 28

GFN GFN GFN MEDIATED DMA TRANSLATION With Runtime Memory pinning Starts Memory regions gets added by Register calls _DMA_MAP via Memory listener

29 GFN GFN GFN MEDIATED DMA TRANSLATION With Runtime Memory pinning Starts Memory regions gets added by Register calls _DMA_MAP via Memory listener tracks <, GFN> PFN PFN PFN CBs IO DMA BFN BFN BFN GFN GFN GFN pin/translate GFN by to get PFN call pci_map_sg to map PFNs to BFN, program DMA 29

30 MEDIATED DEVICE - INTERRUPT 30

31 MEDIATED DEVICE FEWORK With Interrupt Setup /sys/bus/mdev/devices/$mdev_uuid Mdev fd query supported interrupt type, provided by vendor driver Register CBs K irqfd IO 31

32 MEDIATED DEVICE FEWORK With Interrupt Setup /sys/bus/mdev/devices/$mdev_uuid Mdev fd query supported interrupt type, provided by vendor driver setups up K IRQFD Register CBs K irqfd IO 32

33 MEDIATED DEVICE FEWORK Register CBs With Interrupt Setup /sys/bus/mdev/devices/$mdev_uuid K irqfd Mdev fd query supported interrupt type, provided by vendor driver setups up K IRQFD notifies the vendor driver IRQFD via PCI IO 33

34 ISR Vendor driver MEDIATED DEVICE FEWORK Register CBs With Interrupt injection in runtime /sys/bus/mdev/devices/$mdev_uuid K irqfd Mdev fd query supported interrupt type, provided by vendor driver setups up K IRQFD notifies the vendor driver IRQFD via PCI inject interrupt by signaling on eventfd, trigger guest ISR IO 34

35 MEDIATED DEVICE CURRENT STATUS 35

36 CURRENT STATUS Upstream [PATCH v7] is sent out by Kirti Wankhede on 08/24/2016 vfio: device Core driver vfio: driver for mediated devices vfio iommu: Add support for mediated devices docs: Add Documentation for Mediate devices Tested with Linux kernel 4.7 Multiple mediated device per Multiple passthru device per Mixed mediated device and passthru device 36

37 DEMO: NVIDIA V 37

38 MEDIATED DEVICE FEWORK FUTURE WORK 38

39 MEDIATED DEVICE FEWORK Future work Register CBs K IO 39

40 MEDIATED DEVICE FEWORK Future work POWER support by extend pin/unpin to SPAPR TCE v2 IO Register CBs K SPAPR TCE V2 IO IO 40

41 MEDIATED DEVICE FEWORK Future work POWER support by extend pin/unpin to SPAPR TCE v2 IO Libvirt integration Register CBs K SPAPR TCE V2 IO IO 41

42 REFERENCE [1] An Introduction to PCI Assignment with - Alex Williamson, Red Hat [Qemu-devel] [PATCH v7 0/4] Add device support [libvirt] [RFC] libvirt v integration 42

43 QUESTIONS?

Live Migration with Mdev Device

Live Migration with Mdev Device Yulei Zhang yulei.zhang@intel.com 1 Background and Motivation Live Migration Desgin of Mediated Device vgpu Live Migration Implementation Current Status and Demo Future