On the DMA Mapping Problem in Direct Device Assignment

Size: px

Start display at page:

Download "On the DMA Mapping Problem in Direct Device Assignment"

Virgil Allison
6 years ago
Views:

1 On the DMA Mapping Problem in Direct Device Assignment Ben-Ami Yassour Muli Ben-Yehuda Orit Wasserman IBM Research Haifa On the DMA Mapping Problem in Direct Device Assignment p. 1/32

2 Happy Birth Day Tomer Yassour On the DMA Mapping Problem in Direct Device Assignment p. 2/32

3 Virtual Machine I/O I/O virtualization is the final frontier How many of you would run your production database in a VM today? Virtual machines use one of three models for I/O: Device emulation Para-virtualized drivers Device assignment On the DMA Mapping Problem in Direct Device Assignment p. 3/32

4 I/O: Device Emulation HOST GUEST 1 device driver 2 4 device driver 3 device emulation On the DMA Mapping Problem in Direct Device Assignment p. 4/32

5 I/O: Para-virtualized Drivers HOST GUEST front end virtual driver 1 3 device driver 2 back end virtual driver On the DMA Mapping Problem in Direct Device Assignment p. 5/32

6 I/O: Direct Device Assignment HOST GUEST device driver On the DMA Mapping Problem in Direct Device Assignment p. 6/32

7 I/O: Direct Device Assignment cont Give VM direct access to a hardware device Without any software intermediaries between the virtual machine and the device Examples: Legacy adapters [Ben-Yehuda06] Self-virtualizing (SRIOV) adapters [Liu06], [Willman07] Best performance[santos08,09] but at a price. On the DMA Mapping Problem in Direct Device Assignment p. 7/32

8 The Linux/KVM Hypervisor A hypervisor extension for the Linux kernel [Kivity07] Makes extensive use of Intel and AMD hardware virtualization extensions Full featured, open source, and hacker friendly On the DMA Mapping Problem in Direct Device Assignment p. 8/32

9 DMA physical memory device A device B On the DMA Mapping Problem in Direct Device Assignment p. 9/32

10 DMA Security Untrusted guest programs a device, without any supervision. Device is DMA capable (all modern devices are). Which means the guest can program the device to overwrite any memory location.... including where the hypervisor lives... game over. On the DMA Mapping Problem in Direct Device Assignment p. 10/32

11 IOMMU On the DMA Mapping Problem in Direct Device Assignment p. 11/32

12 IOMMU to the rescue physical memory IOMMU device A device B IOMMU think MMU for I/O devices separate address spaces, protection from malicious devices! IOMMUs enable direct assignment for VMs. Intra-VM vs. Inter-VM protection [Willman08] But: IOMMUs have costs too [Ben-Yehuda07] On the DMA Mapping Problem in Direct Device Assignment p. 12/32

13 The Intel VT-d IOMMU On the DMA Mapping Problem in Direct Device Assignment p. 13/32

14 IOMMU Protection Strategies As defined by Willman, Rixner and Cox [Willman08]: Single-use Intra-guest protection, expensive! Shared Relaxed protection, expensive. Persistent Inter-guest protection, pins all of memory. Direct-map Inter-guest protection, no run-time cost, pins all of memory. On the DMA Mapping Problem in Direct Device Assignment p. 14/32

15 Direct-map Performance Send Note: with direct mapping, all memory is pinned! On the DMA Mapping Problem in Direct Device Assignment p. 15/32

16 Direct-map Performance Receive Note: with direct mapping, all memory is pinned! On the DMA Mapping Problem in Direct Device Assignment p. 16/32

17 The DMA Mapping Problem Single-use expensive! Shared expensive. Persistent pins all of memory. Direct-map pins all of memory. The DMA mapping problem: When should a memory page be mapped or unmapped for DMA? On the DMA Mapping Problem in Direct Device Assignment p. 17/32

18 On-Demand Mapping Strategy IOMMU remappings are expensive (world switch, IOTLB flush: over 10K cycles per remapping) A map-cache for caching IOMMU mappings when they are first created. How big should it be? Observation: all guests have some memory pinned anyway. Second observation: common workloads do not need to use all of the guest s memory address space. Define a quota for DMA mappings: the amount of memory the guest can pin for DMA. On the DMA Mapping Problem in Direct Device Assignment p. 18/32

19 The Map Cache Mappings are created when the guest first DMAs to that page. Mappings are either pinned (in use) or candidates for eviction. Implemented using a red-black tree. Mappings are removed from the cache when the quota is reached and a new mapping needs to be created. On the DMA Mapping Problem in Direct Device Assignment p. 19/32

20 Quota control Cooperative guests: defining a quota that is equal to their current memory requirements leads to no run-time IOMMU remappings best performance! Un-cooperative guests: smaller quota, hypervisor enforced. Now the question becomes: for a given quota that is smaller than the working set size, how to efficiently replace IOMMU mappings? Close resemblance to the classical page replacement problem.... except I/O devices do not have page faults. On the DMA Mapping Problem in Direct Device Assignment p. 20/32

21 Batching Mapping Requests If the driver batches together multiple mapping and unmapping requests, the map cache only goes to the hypervisor once. Downside: requires changing the drivers. Piggbyacking unmap requests on top of new mappings. On the DMA Mapping Problem in Direct Device Assignment p. 21/32

22 Prefetching No driver changes necessary! Same concept as FAR [Borodin91] and FARL [Fiat97] paging algorithms in the access graph model. Which pages were recently mapped after a given page? When mapping a new page, also opportunistically map its followers. Choose pages to evict using standard LRU. On the DMA Mapping Problem in Direct Device Assignment p. 22/32

23 Evaluation hit rate FIFO Evict pages in a first in first out order. LRU Evict the least recently used page. OPT Evict the page that is going to be used later then any other page in the cache. This is the optimal offline algorithm without batching. Optimal batching The optimal offline algorithm, but with batching. Prefetching. On the DMA Mapping Problem in Direct Device Assignment p. 23/32

24 netperf send hit rate per quota On the DMA Mapping Problem in Direct Device Assignment p. 24/32

25 apache hit rate per quota On the DMA Mapping Problem in Direct Device Assignment p. 25/32

26 Evaluation CPU utilization LRU Default LRU algorithm where no batching or caching is used. Piggyback Piggybacking unmaps on top of maps. Prefetching. Batching LRU with the map and unmap batching optimizations. On the DMA Mapping Problem in Direct Device Assignment p. 26/32

27 netperf send CPU utilization On the DMA Mapping Problem in Direct Device Assignment p. 27/32

28 apache CPU utilization On the DMA Mapping Problem in Direct Device Assignment p. 28/32

29 Summary & Conclusions Direct device assignment gives best performance of all I/O virtualization methods.... but also poses new problems. In particular, how to balance DMA mapping memory consumption and performance? We propose the on-demand DMA mapping strategy. Mappings are cached, with the size of the cache limited by a quota. Remappings are batched, and new mappings prefetched. Same run-time performance as persistent mapping, better memory utilization for common workloads. On the DMA Mapping Problem in Direct Device Assignment p. 29/32

30 Bibliography Barham03: Xen and the Art of Virtualization, SOSP 03 Bellard05: QEMU, a Fast and Portable Dynamic Translator, USENIX 05 Ben-Yehuda06: Utilizing IOMMUs for Virtualization in Linux and Xen, OLS 06 Ben-Yehuda07: The Price of Safety: Evaluating IOMMU Performance, OLS 07 Bhargave08: Accelerating two-dimensional page walks for virtualized systems, ASPLOS 08 On the DMA Mapping Problem in Direct Device Assignment p. 30/32

31 Bibliography cont. Borodin91: Competitive paging with locality of reference, STOC 91 Chen08: Overshadow: A Virtualization-Based Approach to Retrofitting Protection in Commodity Operating Systems, ASPLOS 08 Fiat97: Experimental studies of access graph based heuristics: beating the LRU standard?, SODA 97 Liu06: High Performance VMM-Bypass I/O in Virtual Machines, USENIX 06 Kivity07: kvm: The Kernel-Based Virtual Machine for Linux, OLS 07 Popek74: Formal Requirements for Virtualizable Third Generation Architectures, CACM 17(7), 74 On the DMA Mapping Problem in Direct Device Assignment p. 31/32

32 Bibliography cont. Santos08: Bridging the Gap between SW & HW Techniques for I/O Virtualization, USENIX 08 Santos 09: Achieving 10Gbps using Safe and Transparent Network Interface Virtualization, VEE 09 Willman07: Concurrent Direct Network Access for Virtual Machine Monitors, HPCA 07 Willman08: Protection Strategies for Direct Access to Virtualized I/O Devices, USENIX 08 Yassour08: Direct Device Assignment for Untrusted Fully-Virtualized Virtual Machines, Ben-Ami Yassour, Muli Ben-Yehuda, Orit Wasserman, IBM Research Report H-0263, 2008 On the DMA Mapping Problem in Direct Device Assignment p. 32/32

The Price of Safety: Evaluating IOMMU Performance

The Price of Safety: Evaluating IOMMU Performance Muli Ben-Yehuda 1 Jimi Xenidis 2 Michal Ostrowski 2 Karl Rister 3 Alexis Bruemmer 3 Leendert Van Doorn 4 1 muli@il.ibm.com 2 {jimix,mostrows}@watson.ibm.com