Development of I/O Pass-through: Current Status & the Future. Nov 21, 2008 Yuji Shimada NEC System Technologies, Ltd.

Size: px

Start display at page:

Download "Development of I/O Pass-through: Current Status & the Future. Nov 21, 2008 Yuji Shimada NEC System Technologies, Ltd."

Sherilyn Merritt
5 years ago
Views:

1 Development of I/O Pass-through: Current Status & the Future Nov 21, 2008 Yuji Shimada NEC System Technologies, Ltd.

2 Agenda 1.Implementation of I/O Pass-through 2.Future Enhancement Plan 3.Challenges for I/O Pass-through 4.Conclusion 2 NEC Corporation 2008

3 1. Implementation of I/O Pass-through

What is I/O Pass-through Assigning I/O device to domain and directly use from guest software Assign HBA High performance access to storage Assign NIC High

4 What is I/O Pass-through Assigning I/O device to domain and directly use from guest software Assign HBA High performance access to storage Assign NIC High performance access to network Domain #1 HBA#1 HBA#2 Dual Port NIC Assign Domain #2 In this presentation I will talk about I/O pass-through for HVM domain 4 NEC Corporation 2008

5 Overview of I/O Pass-through(1) MMIO Virtualization The mapping from guest physical address to machine address is done per-page Reassign Memory Resources to I/O Device If BIOS assigned non-page-aligned MMIO resource, then domain0 linux reassign page-aligned MMIO resource to I/O device Enhancement 1 Configuration Register Access Virtualization ioemu traps configuration register access from guest software, and emulates or pass-through as to bit type Enhancement 2 5 NEC Corporation 2008

6 Summary of I/O Pass-through(2) DMA Virtualization Device driver writes guest physical address to the I/O device I/O device executes DMA by using guest physical address IOMMU on chipset translates guest physical address to machine address Port I/O Virtualization Hypervisor traps IN/OUT instructions and accesses I/O device instead of guest software Interrupt Virtualization Hypervisor receives interrupt from I/O device, emulates interrupt controller, and injects interrupt to HVM domain 6 NEC Corporation 2008

7 Net work Enhancement 1 Reassign Memory Resources to I/O Device Problem If the resources assigned by BIOS is not page-aligned, assigning I/O device to HVM domain fails Error: pci: 0000:00:1d.0: non-page-aligned MMIO BAR found. Enhancement Release resources set by BIOS, and reassign page-aligned resources to I/O device DISK Add boot parameter to specify I/O device to reassign resources NIC HBA 00:1d.1 00:1d.0 00:01.0 page size h h h Memory Resource assigned by BIOS (Machine Address Space) reassigndev=00:1d.0, 00:1d.1, 00:01.0 Boot Xen 00:1d.1 00:1d.0 00: h h h Reassigned Memory Resource (Machine Address Space) Boot HVM Domain We can assign I/O device to HVM domain regardless of the initial resource assignment by BIOS. No Error 7 NEC Corporation 2008

8 Enhancement 2 Configuration Register Access Virtualization(1) Problem I/O device which can be assigned to HVM domain was limited because the registers other than Configuration Header Type 0 were read only Enhancement Allow guest software to read/write access to the following registers, and hide other registers Configuration Header Type 0 MSI Capability Structure MSI-X Capability Structure PCI Express Capability Structure PCI Power Management Capability Structure Vital Product Data Capability Structure Vendor Specific Capability Structure Device Specific Register (excluding Header & Capability Structure) 8 NEC Corporation 2008

9 Enhancement 2 Configuration Register Access Virtualization(2) Enhancement (cont.) Pass-through the directly controllable bit by guest software Emulate the bit that BIOS initial settings should be protected and the bit which has special reasons Configuration Access by Guest Software ioemu Emulated Configuration Registers I/O Device Physical Configuration Registers Pass-through type Emulate type PCI Express Capability Structure PCI Express Capability Register Device Status Link Capabilities Link Status Slot Capabilities Slot Status Root Capabilities Root Status Device Capabilities Next Pointer Device Control Link Control Slot Control Root Control Capability ID Pass-through type Emulate type offset Capability Pointer Capability Pointer + 04h Capability Pointer + 08h Capability Pointer + 0Ch Capability Pointer + 10h Capability Pointer + 14h Capability Pointer + 18h Capability Pointer + 1Ch Capability Pointer + 20h We can assign several types of I/O device, USB and SAS HBA etc. Protect the whole system from invalid guest software behavior. 9 NEC Corporation 2008

10 2. Future Enhancement Plan

11 Future Plan (1) Manage I/O device s power from Guest software Direct management with PMCSR (Power Management Control/Status Register) of Device Support D0, D1, D2, and D3hot Save power of I/O device. high power low power Reduce interrupt redirection On Xen 3.3.0, an interrupt is delivered to one processor (vcpu#0) If that processor is not the one specified by guest software, then interrupt is redirected Support interrupt delivery to processor specified by guest software in order to reduce redirection Improve performance of interrupt delivery. 11 NEC Corporation 2008

12 Improve ioemu log Attach date and pid information to the log of ioemu example Future Plan (2) [ :10: ] can't store dev vc name for domid 1 in /parallel/0 from a stub domain [ :10: ] qemu_map_cache_init nr_buckets = size [ :10: ] shared page at pfn 1fffe Make it easy to analyze the log when any trouble happens in a system. Support I/O pass-through using stub domain Stub domain emulates I/O instead of domain0 Improve stub domain to enable I/O pass-through Improve the scalability and make domain0 smaller. Dom0 HVM Dom Hypervisor Hardware Stub Dom HVM Dom ioemu 12 NEC Corporation 2008

13 3. Challenges for I/O Pass-through

Issue 1 - Unsuitable Device for I/O Pass-through I/O device which accesses internal memory with physical address obtained from device driver does not work Device Driver Processor Access with Machine

Bus Internal Memory I/O Device Access with Guest Physical Address Some Interface Internal Memory Access DMA When we use I/O pass-through, physical address used by device driver is guest physical

14 Issue 1 - Unsuitable Device for I/O Pass-through I/O device which accesses internal memory with physical address obtained from device driver does not work Device Driver Processor Access with Machine Address Pass Guest Physical Address Pass Guest Physical Address Translate Chipset (IOMMU) System Memory Access with Guest Physical Address PCIe Host Interface BAR Firmware Internal Processor Internal Bus Internal Memory I/O Device Access with Guest Physical Address Some Interface Internal Memory Access DMA When we use I/O pass-through, physical address used by device driver is guest physical address. It is distinguished from machine address. On DMA, IOMMU translates guest physical address to machine address. On internal memory access, IOMMU does not translate addresses. It causes access failure. 14 NEC Corporation 2008

15 Issue 1 - Virtualization Hole Causing the Issue. No one translates guest physical address to machine address if I/O device uses guest physical address obtained from device driver for the purpose other than DMA This is common problem among IOMMU based Hypervisors We expect the adapter vender to design I/O device suitable for I/O pass-through. 15 NEC Corporation 2008

16 Issue 2 ioemu does not Support PCIe ioemu does not have following functions MMCFG mechanism Accessing configuration registers by accessing memory space Needed to access configuration register at offset from 100h to FFFh Capability Structure from 100h to FFFh e.g. Device Serial Number Capability Structure PCI Express VSEC Structure Root Port Device with registers to control PCIespecific function Advanced Error Reporting (AER) Reporting PCIe error to software PCIe Hot-plug It is newer mechanism than ACPI hot-plug As a result Guest software can t access register or capability structure it needs Topology at guest software s view is not PCIe machine Guest software can t recover PCIe error Guest software has to use ACPI hot-plug 16 NEC Corporation 2008

17 Issue 2 - Development to Support PCIe First, add MMCFG mechanism(1) and Root Port emulator(2) into ioemu Then, implement others Guest Physical Address Space FFF...FFFh F h E h MMCFG <256MB> (1) Memory Processor Root Port Virtual Machine Topology Host Bridge Legacy Device CD-ROM Display KB / Mouse RTC / PI DMAC I/O APIC PIT / HPET COM ISA Bridge IDE Cont. ACPI Cont. VGA Cont. Platform Device 0h Passthrough Device (2) Assigned Processor / Memory / I/O Emulated I/O Device It is difficult for one developer or a single team to do all the implementations. We need cooperation from all of you! 17 NEC Corporation 2008

18 4. Conclusion

19 Conclusion I/O pass-through becomes practicable now We will continue developing the functions related to I/O pass-through There are still some issues for I/O pass-through We need many developers cooperation for progress 19 NEC Corporation 2008

20 Acknowledgements Thank you very much to all of you: All the people concerned with Xen Summit Tokyo 2008 All the developers in Xen Community 20 NEC Corporation 2008

Intel Graphics Virtualization on KVM. Aug KVM Forum 2011 Rev. 3

Intel Graphics Virtualization on KVM Aug-16-2011 allen.m.kay@intel.com KVM Forum 2011 Rev. 3 Agenda Background on IO Virtualization Device Operation on Native Platform QEMU IO Virtualization Device Direct