VT-d and FreeBSD. Константин Белоусов 21 сентября 2013 г. Revision : Константин Белоусов VT-d and FreeBSD

Size: px

Start display at page:

Download "VT-d and FreeBSD. Константин Белоусов 21 сентября 2013 г. Revision : Константин Белоусов VT-d and FreeBSD"

Gavin Adams
5 years ago
Views:

1 VT-d and FreeBSD Константин Белоусов 21 сентября 2013 г. Revision : 1.12

2 PCIe Example PCI Express Topology Root & Switch CPU Bus CPU Root RCRB Bus 0 PCIe Root Complex PCIe PCIe Memory Virtual PCI Bridge Virtual PCI Bridge Virtual PCI Bridge PCIe Endpoint PCIe PCIe PCIe Endpoint Switch Legacy Endpoint PCIe Endpoint PCIe PCIe Bridge To PCI/PCI-X PCI/PCI-X Switch PCI Express Links Virtual PCI Bridge Legend PCI Express Device Downstream Port PCI Express Device Upstream Port Virtual PCI Bridge Virtual PCI Bridge Virtual PCI Bridge PCI-SIG Developers Conference Copyright 2007, PCI-SIG, All Rights Reserved 8

3 PCIe TLP - Transaction Layer Packets I/O Host access to device (BARs) Device access to memory (DMA) Peer to peer GPU RDMA over Infiniband Nvidia Optimus Messaging: Interrupts, Errors Configuration I/O.

4 Device DMA engines Features and Limitations Scatter/Gather: number of segments DMA engine restrictions Address width Dead bits (alignment) Segment length Streaming Coherence (Snoop) Traffic Prioritization

5 ocument describes the Intel Virtualization Technology for Directed I/ O ( Intel V ; specifically, it describes the components supporting I/ O virtualization as it applies VT-d se Intel processors and core logic chipsets complying with Intel platform speci e 1-1 illustrates the general platform topology. P rocessor P rocessor S ystem B us N orth B ridge D M A R e m a p ping D R A M Integrated D evices P C I E xpre ss D e vice s S outh B ridge P C I, L P C, Legacy devices Figure 1-1. General Platform Topology

6 VT-d Source I dent ifier nterrupt DMAR requests appearing at the address-translation hardware is required to ide inating the request. The attribute identifying the originator of an I/ O transaction is source-id in this document. The remapping hardware may determine the sourcein implementation-specific ways. For example, some I/ O bus protocols may prov device identity as part of each I/O transaction. In other cases (for Root-Complex devices, for example), the source-id may be derived based on the Root-Complex ation. Process TLPs from devices accessing memory Performs Address Translation and Access Control Snoop Control Prioritization press devices, the source-id is the requester identifier in the PCI Express transacti e requester Based identifier on the of a originator device, which of the is TLP composed of its PCI Bus/ Device/ Functio assigned by configuration software and uniquely identifies the hardware function e request. Figure 3-6 illustrates the requester-id 1 as defined by the PCI Express on. Requester Identifier Bus # Device # Function # Figure 3-6. Requester I dentifier Format

7 VT-d DMA Remapping I ntel Virtualization Technology for Directed I / O DMAR translation structures (Dev 31, Func 7) Context entry 255 (Dev 0, Func 1) (Bus 255) (Bus N) Root entry 255 Root entry N (Dev 0, Func 0) Context entry 0 Context-entry Table for Bus N Address Translation Structures for Domain A (Bus 0) Root entry 0 Root-entry Table Context entry 255 Context entry 0 Context-entry Table for Bus 0 Address Translation Structures for Domain B Figure 3-7. Device to Domain Mapping Structures

8 VT-d Hardware Nehalem+ Xeons Desktop Core i7 CPUs: not -K, BIOS Core2 gen: G45, 5500 Documentation Intel R Virtualization Technology for Directed I/O, D External Design Specification (EDS) BIOS Write Guide (BWG) Chipset erratas

9 VT-d Compatibility SMI handlers, USB legacy UMA GPU: GTT and VGA framebuffer Service processor for BMC (AMT, IPMI, ilo, DRAC etc) Bugs Hardware bugs, Specification Updates BIOS bugs

10 VT-d How to detect acpidump -t DMAR: Length=368, Revision=1, Checksum=7, OEMID=DELL, OEM Table ID=PE_SC3, OEM Revision=0x1, Creator ID=DELL, Creator Revision=0x1 Host Address Width=46 Flags={INTR_REMAP,X2APIC_OPT_OUT}

11 VT-d Other features Interrupt remapping MSI, MSI-X: memory write IO-APICs FSB interrupts: HPET ATS (Address Translation Service): IO TLB in devices Hypervisors PCI pass-through

12 IOMMU PCI-era Architectures SPARC4u POWER: DART coarse domains

13 DMA in FreeBSD Busdma(9) layer FreeBSD KPI abstracting access to DMA implementations from NetBSD Busdma(9) overview Tags: device capabilities Maps: Accessible memory Loads and unloads: maps activation and deactivation

14 Busdma implementations Bounce buffers Allocate memory to satisfy device constraints contigmalloc(9) Low 16MB, low 4GB Copy to/from Flush cache on non-coherent platforms

15 Busdma implementations IOMMU: pro Performance: No bouncing Stability: No memory corruption Privacy: Only sanctioned access to memory Driver debugging: Reports of violations IOMMU: contra Performance: Page table setup Performance: Translation overhead

16 Busdma over VT-d Layers Page tables and TLB invalidation Fault handler Context and domain Busdma emulation Integration ACPI: DMAR table parsing DMAR discovery RMRR and BIOS bugs newbus: bus_get_dma_tag() fallback to bounce, enabling pass-through

17 Busdma over VT-d Busdma KPI problems Locking BUS_DMA_NOWAIT abuse bus_dmamap_unload(9) cannot sleep No I/O direction Tag specification of alignment

18 Busdma over VT-d Current state Drivers Storage: ahci(4), mfi(4) USB: uhci(4), ehci(4) Network: em(4), igb(4) (*), bce(4) Platforms Xeon 5400, 5500 NB Xeon Romely-EP (E5-26XX) Haswell (Core i7 4770) Not supported yet Intel GPUs Not tested HDA Discrete GPUs (Radeon, Nvidia) Everything else (HW bugs)

Intel Virtualization Technology for Directed I/O Architecture Specification

Intel Virtualization Technology for Directed I/O Architecture Specification February 2006 Order Number: D51397-001 INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE,