Vhost and VIOMMU. Jason Wang (Wei Xu Peter Xu

Size: px

Start display at page:

Download "Vhost and VIOMMU. Jason Wang (Wei Xu Peter Xu"

Lester West
5 years ago
Views:

Vhost and VIOMMU Jason Wang <jasowang@redhat.

1 Vhost and VIOMMU Jason Wang (Wei Xu Peter Xu

2 Agenda IOMMU & Qemu viommu background Motivation of secure virtio DMAR (DMA Remapping) Design Overview Implementation illustration Performance optimization vhost device iotlb IR (Interrupt Remapping) Performance results & status 08/18/16 VHOST AND VIOMMU 2

3 IOMMU & Qemu viommu Revisit What is IOMMU? A hardware component provides two main functions: IO Translation and Device Isolation. How IO Translation and Device Isolation are supported by IOMMU DMA Remapping(DMAR), IO space address presented by devices are translated to physical address coupled with access permission on the fly, so the ability of devices are limited to access specific regions of memory. Interrupt Remapping (IR), Some architectures also support interrupt remapping, in a manner similar to memory remapping. What's qemu viommu? An emulated IOMMU which behaves as a real one. The functionality is always a subset of a physical unit depending on implementation. Only Intel, ppc, sun4m iommus are support in qemu currently. 08/18/16 VHOST AND VIOMMU 3

4 IOMMU and viommu HOST VM VM Memory Memory viommu vmmu viommu vmmu Emulated Devices vcpu Emulated Devices vcpu Host Memory IOMMU MMU Hardware Devices CPU 08/18/16 VHOST AND VIOMMU 4

5 Motivation Security, Securtiy and security. DPDK: The Userspace Polling-Mode drivers (DPDK) for virtio net devices are vastly used in NFV. Vhost is the popular backend for most of user cases. Vhost is still out of IOMMU scope. 08/18/16 VHOST AND VIOMMU 5

6 DMA Remapping (DMAR)

7 Virtio-Net Device Address Space Overview Virtio-Net Guest gpa Memory API Qemu gpa Guest pages Virtio-Net Backend Service gpa-to-hva tx/rx Vhost-user Vhost-net Vring Other virtio-net backends Virtio-Net Backends 08/18/16 VHOST AND VIOMMU 7

8 Virtio-Net Design of Secure Virtio-Net Device Driver Guest Qemu iova Memory API dma api Virtio-Net Backend Service IOMMU Driver Guest Pages iotlb entry lookup viommu IOTLB API iova tx/rx Vhost-user iova-to-hva Vhost-net Other virtio-net backends Vring Virtio-Net Backends 08/18/16 VHOST AND VIOMMU 8

9 Implementation: Guest Guest Boot guest with a viommu assigned. VIRTIO_F_IOMMU_PLATFORM, if this feature bit is provided in the device, then the guest virtio driver is forced to use dma api to manage all corresponding dma memory access, otherwise the device will be disabled by system compulsorily. 08/18/16 VHOST AND VIOMMU 9

10 Implementation: Qemu and Backends Qemu DMA address translation for viommu has been fully supported, unfortunately, virtio-pci devices is still using memory address space and never use iova at all, switch to use dma address(iova). Backends All address access to vring must be translated from guest iova to hva, this is done via iotlb lookup with interfering of viommu. 08/18/16 VHOST AND VIOMMU 10

11 More optimization: Vhost Device IOTLB Cache Why it comes to vhost? Vhost-net is the most powerful and reliable in-kernel network backend, and is widely used as a preferred backend. What problem does vhost encounter? IOTLB api of viommu is implemented in qemu, while vhost works in kernel, high frequency of iotlb translations which traverse between kernel and userspace will impact performance dramatically. How does vhost survive? Kernel-Side device iotlb cache(ats). 08/18/16 VHOST AND VIOMMU 11

12 Address Translation Services(ATS) Overview Memory Translation Agent (TA) ats request Root Complex ats completion PCIe Device B PCIe Device A device iotlb cache 08/18/16 VHOST AND VIOMMU 12

13 Why Address Translation Services(ATS)? Alternative An individual VT-d in vhost, drawbacks: Code duplication. Vendor and architecture specific. New api for error reporting. Benefits of ATS PCIe spec Platform independent. Easily achieved based on current iommu infrastructure. 08/18/16 VHOST AND VIOMMU 13

14 Vhost Device IOTLB Cache Workflow legal address range Qemu new illegal address range IOTLB API guest unmap 'c' (d, size, wo) error report lookup update 'd' Vhost IOTLB API iotlb-update 'd' iotlb-miss 'd' iotlb invalidate 'c' (a, size, ro) Vhost translate iova 'd' Tx/Rx a (b, size, wo) (d, size, wo) (c, size, rw) Vring device iotble cache entries interval tree 08/18/16 VHOST AND VIOMMU 14

15 Vhost Device IOTLB Implementation Summary Implementation - Save device iotlb cache entries in kernel. - Lookup entry from the cache when accessing virtio buffers. - Request qemu to translate for any tlb miss on demand. - Process update/invalidate message from qemu and manage the kernel cache correctly. Data Structure and Userspace/Kernel Interface - An interval tree is chosen to save the dynamica device iotlb caches. - A message mechanism via vhost 'fd' read/write is used to pass vats request and reply. 08/18/16 VHOST AND VIOMMU 15

16 Interrupt Remapping (IR)

17 X86 system interrupts Processor Processor... Processor Local APIC Local APIC Local APIC System Bus Bridge PCI Bus Kinds of interrupts: Line-based (edge/level) Signal-based (MSI/MSI-X) Signal-based Interrupts (MSI/MSIX) IOAPIC Line-based Interrupts IRQ chips IOAPIC Local APICs (LAPICs) 08/18/16 VHOST AND VIOMMU 17

18 IR challenges for vhost Interrupt remapping (IR) still not supported for x86 viommu MSI and IOAPIC interrupts Kernel irqchip support: How to define interface between user and kernel space? How to enable vhost fast irq path (irqfd)? Performance impact? Interrupt caching 08/18/16 VHOST AND VIOMMU 18

19 IOAPIC interrupt delivery Workflow before IR: Fill in IOAPIC entry with interrupt information (trigger mode, destination ID, destination mode, etc.). When line triggered, interrupt sent to CPU with information stored in IOAPIC entry. Workflow after IR (IRTE: Interrupt Remapping Table Entry): Fill in IRTE with interrupt information (in system memory). Fill in IOAPIC entry with IRTE index. When line triggered, fetch IRTE index from IOAPIC entry, send the interrupt with information stored in specific IRTE. 08/18/16 VHOST AND VIOMMU 19

20 MSI/MSI-X delivery Interrrupt Request (MSI) Delivered MSI Delivery without IR Interrupt Remapping Table IRTE IRTE IRTE IRTE IRTE IRTE IRTE IRTE IRTE IRTE IRTE IRTE IRTE IRTE IRTE IRTE Indexing Lookup Interrrupt Request (MSI with IR) Interrrupt Remapping Table Entry (IRTE) Parse Interrrupt Request (MSI) Delivered MSI Delivery with IR 08/18/16 VHOST AND VIOMMU 20

21 IR with kernel-irqchip We want interrupts as fast as before. Current implementation: Leverage existing GSI routing table in KVM Instead of translate on the fly, translate during setup Easy to implement (no KVM change required) Little performance impact (slow setup, fast delivery) Only support split off kernel irqchip, not on 08/18/16 VHOST AND VIOMMU 21

22 Remap irqfd interrupts Fast IRQ path for vhost devices: without remapping GSI Routing Table QEMU Setup MSI Message 1 MSI Message 2 MSI Message 3 Setup MSI Message 4 Event vhost Guest Notifier KVM IRQ injection Guest 08/18/16 VHOST AND VIOMMU 22

23 Remap irqfd interrupts (cont.) Fast IRQ path for vhost devices: with remapping GSI Routing Table QEMU Setup Translated MSI Message 1 Translated MSI Message 2 Translated MSI Message 3 Setup Translated MSI Message 4 Event vhost Guest Notifier KVM IRQ injection Guest 08/18/16 VHOST AND VIOMMU 23

24 All in all... To boot guest with DMAR and IR enabled: (Possibly one extra flag to enable DMAR for guest virtio driver) qemu-system-x86_64 -M -M q35,accel=kvm,kernel-irqchip=split \ -device intel-iommu,intremap=on \ -netdev tap,id=tap1,script=no,downscript=no,vhost=on \ -device virtio-net-pci,netdev=tap1,disable-modern=off,ats=on 08/18/16 VHOST AND VIOMMU 24

25 Vhost + VIOMMU Performance For dynamic DMA mapping (e.g., using generic Linux kernel drivers): Performance dropped drastically TCP_STREAM: Mbps 600 Mbps TCP_RR: trans/s trans/s For static DMA mapping (e.g., DPDK based application like l2fwd) Around 5% performance drop for throughput (pktgen) Still more work TBD... 08/18/16 VHOST AND VIOMMU 25

26 DMAR/IR upstream status: Current status & TBDs QEMU: IR merged (Peter Xu), DMAR still RFC (Jason Wang will post formal patch soon) Vhost & Virtio driver: merged (Michael S. Tsirkin/Jason Wang) DPDK: vhost-user IOTLB is being developed (Victor Kaplansky) TBDs Performance tuning for DMAR Quite a few enhancements for IR: explicit cache invalidations, better error handling, etc. 08/18/16 VHOST AND VIOMMU 26

27 Thanks! 08/18/16 VHOST AND VIOMMU 27

28 Appendix 08/18/16 VHOST AND VIOMMU 28

29 Command line interface: Kernel-irqchip: a review qemu-system-x86_64 -M -M q35,kernel-irqchip={on off split} Supported modes Mode IOAPIC APIC ON In kernel SPLIT In userspace In kernel In kernel In userspace OFF In userspace 08/18/16 VHOST AND VIOMMU 29

viommu/arm: full emulation and virtio-iommu approaches Eric Auger KVM Forum 2017

viommu/arm: full emulation and virtio-iommu approaches Eric Auger KVM Forum 2017 Overview Goals & Terminology ARM IOMMU Emulation QEMU Device VHOST Integration VFIO Integration Challenges VIRTIO-IOMMU