Nested Virtualization Update From Intel. Xiantao Zhang, Eddie Dong Intel Corporation

Size: px

Start display at page:

Download "Nested Virtualization Update From Intel. Xiantao Zhang, Eddie Dong Intel Corporation"

Trevor Bailey
5 years ago
Views:

1 Nested Virtualization Update From Intel Xiantao Zhang, Eddie Dong Intel Corporation

2 Legal Disclaimer INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. INTEL PRODUCTS ARE NOT INTENDED FOR USE IN MEDICAL, LIFE SAVING, OR LIFE SUSTAINING APPLICATIONS. Intel may make changes to specifications and product descriptions at any time, without notice. All products, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice. Intel, processors, chipsets, and desktop boards may contain design defects or errors known as errata, which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. *Other names and brands may be claimed as the property of others. Copyright 2012 Intel Corporation.

3 Agenda Motivation and Goals History Nested VMX Architecture Previous status Latest status and new features Stability Enhancement Virtual EPT Virtual VT-d Preliminary Performance Call to Action 3

4 Motivation and Goals Why nested virtualization? Ordinary OS are adopting VMX now Windows 7 XP compatibility mode Windows 8 Hyper-V Other Commercial VMMs requires VMX for better performance vmware vmm Anti-virus software depends on VMX McAfee Deep Defender What is the goal? To make VMX-based system software run smoothly in a Xen guest. Guest Guest VMM VMM Hardware Platform with VMX Enabled 4

5 Agenda Motivation and Goals History Nested VMX Architecture Previous status Latest status and new features Stability Enhancements Virtual EPT Virtual VT-d Preliminary Performance Call to Action 5

6 Nested VMX Architecture VM entry/exit Virtual VM entry/exit Nested Guest L2 Dom0 HVM Guest vvmcs Nested VMM L1 Shadowing Xen VMM VMCS Shadow VMCS Virtual EPT L0 Virtual VMX Virtual VT-d VMX-Enabled Platform 6 (VT-d, EPT etc.)

7 History Nested VMX Xen Summit Asia (Nov. 2009) Nested VMX design is presented Showed Initial Status Nested guest can boot up to BIOS early stage with limitations single vcpu/single nested guest/ No vcpu migration Refined nested VMX support was pushed into upstream Support multiple nested guests Also includes supporing SMP nested guests However, experimental & preliminary support Very limited configurations can work KVM on Xen, Linux guest can successfully boot up Xen on Xen does not work No virtual VT-d, virtual EPT 7

8 Previous Status Only one combination can work L2 Guest OS L0-VMM L1-VMM 32Bit PAE OS 64Bit OS RHEL6.0 RHEL5.4 Win7 RHEL6.0 Win7 Win2012 Server Ubuntu Xen X X X X X X X Xen KVM X X X X X X 8

9 Agenda Motivation and Goals History Nested VMX Architecture Previous status Latest status and new features Stability Enhancements Virtual EPT Virtual VT-d Preliminary Performance Call to Action 9

10 Stability Enhancement Greatly enhanced stability, with several critical bugs fixed! L2 Guest L2 Guest OS(SMP) L0-VMM L1-VMM 32Bit PAE OS 64Bit OS Win2012 RHEL6.0 RHEL5.4 Win7 RHEL6.3 RHEL6.0 Win7 Server Ubuntu Xen X X X X X X X Xen KVM X X X X X X 10

11 Performance Without Optimizations L2 Guest2(Xen on Xen) L1 Guest Native Platform: SNB-EP OS: RHEL6.3 Guest MEM:4GB Memory CPU: 2 VCPU SpecJBB Unixbench Kernel Build CPU-INT 11

12 Agenda Motivation and Goals History Nested VMX Architecture Previous status Latest status and new features Stability Enhancements Virtual EPT Virtual VT-d Preliminary Performance Call to Action 12

13 Shadowing Shadowing Virtual EPT Architecture VM entry/exit Virtual VM entry/exit L2 L2 Guest L1 vvmcs L1 VMM L1 EPT L2-GPA-> L1 GPA L0 L0 VMM svmcs-l2 Shadow EPT L2-GPA->HPA VMCS-L1 L0 EPT L1-GPA-> HPA Switch to Shadow virtual vmentry 13

14 Virtual EPT: Using EPT Shadowing No write-protection to L1-EPT (Guest EPT paging structure) Flexibility is good. Trap-and-emulate guest s INVEPT Update the shadow EPT entries Better SMP Scalability No global lock is required Requires page-level INVEPT Individual address invalidation 14

15 Enhanced INVEPT Instruction for Virtual EPT INVEPT limitations No Individual address invalidation Only single context and all context invalidation Little performance impact, however, hurt nested performance sharply! Has to drop shadow EPT table for L1 s each INVEPT(with single context) Performance loss if frequent INVEPT in VMM For example, KVM Enhance it in Software Way Add Individual address invalidation for virtual EPT Expose it to nested VMM through PV approach Need to enhance VMMs Easy implementation for Xen and VMM Benefits Reduce frequent shadow EPT paging structure flush 15

16 Performance Evaluation For Virtual EPT w/o virtual EPT with virtual EPT Kernel Build SpecJBB Netperf CPU-INT 16

17 Agenda Motivation and Goals History Nested VMX Architecture Previous status Latest status and new features Stability Enhancements Virtual EPT Virtual VT-d Preliminary Performance Call to Action 17

18 Virtual VT-d: Expose VT-d Capability to L1VMM I/O performance for L2 guest is very slow Due to extremely long device emulation path through all the way to L1 & L0 VMMs How to fix that? Present virtual VT-d engine to L1 VMM So, device can be directly assigned to L2 guest High I/O performance, because of minimum VMM intervention. Must-to-have features in Virtual VT-d DMA Remapping & Queue Invalidation: Exposed Interrupt remapping: Not Exposed 18

VMM VT-d page table L1 Hardware VT-d IO TLB Xen Hypercall Bus 0 Hypercall Virtual

19 Virtual VT-d Architecture Domain 0 Nested Guest Guest view of VT-d L2 Dev Q: Qemu device Dev P: PT device HVM loader DMA Engine Qemu Bus 0 DevQ,Funy DevP,Funn Nested VMM VT-d page table L1 Hardware VT-d IO TLB Xen Hypercall Bus 0 Hypercall Virtual VT-d vdmar vqi Shadowing L0 Dev M FunN Bus N Device ATC VMM Shadow VT-d page table Hypervisor 19

20 Two types of guest devices Pass through device DMA (IOVA->GPA) is handled by hardware VT-d engine Remap guest root/context structure Use physical remapping table to emulate guest remapping table IOVA -> L0 HPA, + audit (use a dummy page for Out of Bound gpn) Maybe cached by IOTLB and ATC IOTLB/Context Cache Synchronization Track guest invalidation of IOTLB Invalidate physical IOTLB, and may invalidate ATC as well if the device has ATC Track guest invalidation of Context Cache Qemu device DMA (IOVA->PA remapping) is emulated by Qemu 2 Options: Caching the remapping table, or No-Caching Starting from simple solution: No caching Qemu device is already slow 20

21 Mb/s Performance Evaluation of virtual VT-d Bandwidth of Nested Guest Ideal Bandwidth TCP Send TCP Receive UDP Send UDP Receive Iperf testing with the assigned NIC to nested Guest Bandwidth is good enough! 21

22 ms Latency Evaluation of virtual VT-d 120 Latency 100 Latency Native Nested Guest Still have room to tune Latency 22

23 Agenda Motivation and Goals History Nested VMX Architecture Previous status Latest status and new features Stability Enhancements Virtual EPT Virtual VT-d Preliminary Performance Call to Action 23

24 Preliminary Performance Based on Xen #25467 L1 Guest L2 Guest Platform: SNB-EP OS: RHEL5.4 Guest MEM:2GB Memory CPU: 2 VCPU 24

25 Agenda Motivation and Goals History Nested VMX Architecture Previous status Latest status and new features Stability Enhancements Virtual EPT Virtual VT-d Preliminary Performance Call to Action 25

26 Call to Action Support more L1 VMMs McAfee Deep Defender VMware VMM Hyper-V Virtual Box Virtual APIC-V New Features for Interrupt/APIC Virtualization are coming For more information, please come to Nakajima Jun s talk Intel Update this afternoon. Improve interrupt virtualization efficiency for both L1 and L2 Performance Tuning 26

27 Reference Nested Virtualization on Xen Qing He: Xen Summit 2009: Virtual APIC-V Jun Nakajima: Intel Update Xen Summit 2012: 27

28 Questions? Or contact 28

Making Nested Virtualization Real by Using Hardware Virtualization Features

Making Nested Virtualization Real by Using Hardware Virtualization Features May 28, 2013 Jun Nakajima Intel Corporation 1 Legal Disclaimer INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL