COSC6376 Cloud Computing Lecture 14: CPU and I/O Virtualization

Size: px

Start display at page:

Download "COSC6376 Cloud Computing Lecture 14: CPU and I/O Virtualization"

Lucinda Norris
5 years ago
Views:

1 COSC6376 Cloud Computing Lecture 14: CPU and I/O Virtualization Instructor: Weidong Shi (Larry), PhD Computer Science Department University of Houston

2 Outline CPU Virtualization I/O Virtualization

3 Types of virtualization Container virtualization Para-virtualization Full-virtualization

4 Xen architecture

5 Clustered Xen environment

6 Network flow in Xen

7 Linux bridge The old version of Citrix XenServer (before v5.6 FP1) using simple Linux Bridge. eth0 VM1 eth1 (Vlan 2) (Vlan 30) (Vlan 30) VM2 VM3 VM4 eth0 eth1 eth0 eth1 eth0 eth1 Xen Server domu Many hypervisor based virtualization also apply Linux Bridge model, such as KVM, libvirt. All of bridging work are done by brctl. Provide simple L2 switching functions. vif1.0 vif1.1 xenbr0 xenbr1 xapi1 xapi2 xapi30 xenbr2 Xenbr3 eth1.1 (insert vlan tag) eth1.2 eth1.30 eth0 eth1 eth2 eth3 XenMgmt vif2.0 (untagged traffic) Switch (trunk port) vif2.1 DataNetwork (vlan) (172.v.v.h/16) vif3.0 vif3.1 (Vlan 1) (Vlan 2) (Vlan 30) (tagged traffic) vif4.0 Internet ( /16) vif4.1 dom0 VmMgmt ( /16)

8 Xen network environment peth0 This is the port that connects to the physical network interface in your system. vif0.0 This is the bridge port that is used by traffic to/from Domain 0. vifx.0 This is the bridge port that is used by traffic to/from Domain X.

9 VMware Infrastructure 3 VMware Infrastructure 3 provides a rich set of networking capabilities. Virtual switches are the key networking components, up to 248 virtual switches on each ESX Server 3 host. They provide core Layer 2 forwarding engines. Physical Ethernet adapters (uplinks) serve as bridges between virtual and physical networks.

VMware vsphere s vds vnic is logically connected to a dvport shown as black squares. Each dvport is implemented by the proxy switch on the host where the VM runs.

10 VMware vsphere s vds vnic is logically connected to a dvport shown as black squares. Each dvport is implemented by the proxy switch on the host where the VM runs. vsphere s vnetwork distributed switch (vds) functions as a single switch across all associated hosts. This enables you to set network configurations that span across all member hosts, and allows virtual machines to maintain consistent network configuration as they migrate across multiple hosts (the vds centrally managed by vcenter).

11 Intel virtualization technology evolution Vector 3: I/O Focus PCI-SIG Standards for IO-device sharing: Multi-Context I/O Devices Endpoint Address Translation Caching Under definition in the PCI-SIG* IOVWG Vector 2: Platform Focus VT-d Hardware support for IO-device virtualization Device DMA remapping Direct assignment of I/O devices to VMs Interrupt Routing and Remapping Vector 1: Processor Focus VT-x VT-i Establish foundation for virtualization in the IA-32 and Itanium architectures followed by on-going evolution of support: Micro-architectural (e.g., lower VM switch times) Architectural (e.g., Extended Page Tables) VMM Software Evolution Software-only VMMs Binary translation Paravirtualization Past No Hardware Support Simpler and more Secure VMM through foundation of virtualizable ISAs Today Increasingly better CPU and I/O virtualization performance and functionality as I/O devices and VMMs exploit infrastructure provided by VT-x, VT-i, VT-d VMM software evolution over time with hardware support

12 VT-x Overview: Intel Virtualization Technology For IA-32 Processors

13 VT-x overview Operating modes Guest SW VMM Transitions Virtual-machine control structure Principal causes of VM Exits Benefits

14 Operating modes VMX root operation: Fully privileged, intended for VM monitor VMX non-root operation: Not fully privileged, intended for guest software Reduces Guest SW privilege w/o relying on rings Solution to Ring Aliasing

15 VM entry and VM exit VM Entry Transition from VMM to Guest Enters VMX non-root operation Loads Guest state and Exit criteria from VMCS VM Exit VMEXIT instruction used on transition from Guest to VMM Enters VMX root operation Saves Guest state in VMCS Loads VMM state from VMCS VM 0 App App... App... VM 1 App App... App Guest OS 0 Guest OS 1 VM Exit VM Entry VM Monitor Physical Host Hardware

16 VT-x operations VM 1 VM 2 VM n VMX Non-root Operation Ring 3 Ring 0 Ring 3 Ring 0... Ring 3 Ring 0 VM Exit VMCS 1 VMCS 2 VMCS n IA-32 VMX Root Operation VMXON VMLAUNCH VMRESUME Ring 3 Ring 0

17 Virtual machine control structure (VMCS) VMCSs are Control Structures in Memory Only one VMCS active per virtual processor at any given time VMCS Payload: VM execution, VM exit, and VM entry controls Guest and host state VM-exit information fields VMCS Format not defined and may vary VMPTRLD: Establishes a pointer to a desired VMCS VMREAD/VMWRITE: New VMCS Access instructions

18 Principal causes of VMEXIT Paging state exits allow page-table control CR3 accesses, INVLPG cause exits Selectively exit on page faults CR0/CR4 controls allow exiting on changes to selected bits State-based exits allow function virtualization CPUID, RDMSR, WRMSR, RDPMC, RDTSC, MOV DRx Selective exception and I/O exiting reduce unnecessary exits 32-entry exception bitmap, I/O-port access bitmap Controls provided for asynchronous events Host interrupt control allows delivery to VMM even when guest blocking interrupts Detection of guest inactivity to support VM scheduling HLT, MWAIT, PAUSE

19 Benefits: VT helps improve VMMs VT Reduces guest OS dependency Eliminates need for binary patching / translation Facilitates support for Legacy OS VT improves robustness Eliminates need for complex SW techniques Simpler and smaller VMMs Smaller trusted-computing base VT improves performance Fewer unwanted Guest VMM transitions

20 Extended page tables (EPT) A VMM must protect host physical memory Multiple guest operating systems share the same host physical memory VMM typically implements protections through page-table shadowing in software Page-table shadowing accounts for a large portion of virtualization overheads Goal of EPT is to reduce these overheads

21 What Is EPT? CR3 EPT Base Pointer (EPTP) Guest Linear Address Guest IA-32 Page Tables Guest Physical Address Extended Page Tables Host Physical Address Extended Page Table A new page-table structure, under the control of the VMM Defines mapping between guest- and host-physical addresses EPT base pointer (new VMCS field) points to the EPT page tables Guest has full control over its own IA-32 page tables

22 EPT translation: details All guest-physical memory addresses go through EPT tables (CR3, PDE, PTE, etc.) Above example is for 2-level table for 32-bit address space Translation possible for other page-table formats (e.g., PAE)

23 VT-d Overview: Intel Virtualization Technology For Directed I/O

24 Q35 chipsets system block diagram

PCI Express 3 rd generation high-performance I/O bus Used to interconnect peripheral devices Point-to-point connection as opposed to bus PCIe interconnect consists of either a x1, x2, x4, x8, x12,

25 PCI Express 3 rd generation high-performance I/O bus Used to interconnect peripheral devices Point-to-point connection as opposed to bus PCIe interconnect consists of either a x1, x2, x4, x8, x12, x16 or x32 point-to-point link if you have x16 link, there are 64 physical lines (16 * 2 (both directions) * 2 (differential signaling)) 1 st generation ISA, EISA, VESA and Micro Channel buses 2 nd generation PCI, PCI-X, and AGP

26 PCIe-based system topology Root Complex Denote the root of I/O hierarchy that connects the CPU/memory subsystem to the I/O May support one or more PCIe ports as shown Endpoint devices other than root complex and switches that are requesters or completers of PCIe transactions Souce: PCIe specification 2.0

27 Three IA-32 address-spaces accessed using a large variety of processor instructions (mov, add, or, shr, push, etc.) and virtual-to-physical address-translation memory space (4GB) accessed only by using the processor s special in and out instructions (without any translation of port-addresses) i/o space (64KB) PCI configuration space (16MB) PCIe supports the same address spaces as PCI Memory space IO space Configuration space PCIe provides a 4KB space per a function as opposed to 256B in PCI i/o-ports 0x0CF8-0x0CFF dedicated to accessing PCI Configuration Space

28 PCI configuration header 16 doublewords 31 0 BIST Status Register Header Type Latency Timer Base Address 1 Command Register Cache Line Size 31 0 Device ID Class Code Class/SubClass/ProgIF Base Address 0 Vendor ID Revision ID Dwords Base Address 3 Base Address Base Address 5 Base Address Subsystem Device ID Subsystem Vendor ID CardBus CIS Pointer reserved capabilities pointer Expansion ROM Base Address Maximum Latency Minimum Grant Interrupt Pin Interrupt Line reserved 15-14

29 Typical NIC packet main memory TX FIFO buffer CPU B U S RX FIFO nic transceiver LAN cable

30 PCI devices and functions A PCI device may include between 1 and 8 functions Function numbers range from 0 to 7 Function 0 must always be present Classified as single-function and multi-function devices

31 DMA (Direct Memory Access) DMA has the ability to transfer large blocks of data directly to/from the memory without involving the processor The processor initiates the DMA transfer by supplying source and destination addresses, the number of bytes to transfer The DMA controller manages the entire transfer (possibly thousand of bytes in length), arbitrating for the bus When the DMA transfer is complete, the DMA controller interrupts the processor to inform that the transfer is complete

32 DMA (Direct Memory Access) DMA has the ability to transfer large blocks of data directly to/from the memory without involving the processor

33 Options for I/O virtualization Monolithic Model Service VM Model Pass-through Model Service VMs Guest VMs VM 0 VM n VM 0 VM n Guest OS and Apps Guest OS and Apps I/O Services VM 0 VM n Guest OS and Apps Guest OS and Apps I/O Services Device Drivers Guest OS and Apps Device Drivers Device Drivers Device Drivers Hypervisor Shared Devices Hypervisor Shared Devices Hypervisor Assigned Devices Pro: Higher Performance Pro: I/O Device Sharing Pro: VM Migration Con: Larger Hypervisor Pro: High Security Pro: I/O Device Sharing Pro: VM Migration Con: Lower Performance Pro: Highest Performance Pro: Smaller Hypervisor Pro: Device assisted sharing Con: Migration Challenges VT-d Goal: Support all Models

34 VT-d overview VT-d is platform infrastructure for I/O virtualization Defines architecture for DMA remapping Implemented as part of platform core logic Will be supported broadly in Intel server and client chipsets CPU CPU System Bus Integrated Devices North Bridge VT-d PCIe* Root Ports DRAM PCI Express South Bridge PCI, LPC, Legacy devices,

35 Each VM thinks it is 0 address based GPA (Guest Physical Address) But mapped to a different address in the system memory HPA (Host Physical Address) VTd does the address mapping between GPA and HPA Catches any DMA attempt to cross VM memory boundary How VTd works? VM0 VM1 VM

36 VT-d usage Basic infrastructure for I/O virtualization Enable direct assignment of I/O devices to unmodified or paravirtualized VMs Improves system reliability Contain and report errant DMA to software Enhances security Support multiple protection domains under SW control Provide foundation for building trusted I/O capabilities Other usages Generic facility for DMA scatter/gather Overcome addressability limitations on legacy devices

VT-d architecture detail DMA Requests Device ID Virtual Address Length Fault Generation Bus 255 Bus N Bus 0 Dev 31, Func 7 Dev P, Func 2 Dev P, Func 1 Dev 0, Func 0 4KB Page Tables Page Frame DMA

37 VT-d architecture detail DMA Requests Device ID Virtual Address Length Fault Generation Bus 255 Bus N Bus 0 Dev 31, Func 7 Dev P, Func 2 Dev P, Func 1 Dev 0, Func 0 4KB Page Tables Page Frame DMA Remapping Engine Translation Cache Device Assignment Structures Device D1 Device D2 Address Translation Structures Context Cache Address Translation Structures Memory Access with System Physical Address Memory-resident Partitioning And Translation Structures

38 VT-d: hardware page walk Requestor ID DMA Virtual Address 15 Bus Device Func b b Level-4 table offset 30 Level-3 table offset Level-2 Level-1 table offset table offset 11 Page Offset 0 Base Device Assignment Tables Level-4 Page Table Example Device Assignment Table Entry specifying 4-level page table Level-3 Page Table Level-2 Page Table Level-1 Page Table Page

39 VT-d applied to pass-through model Direct Device Assignment to Guest OS Guest OS directly programs physical device For legacy guests, hypervisor sets up guest- to host-physical DMA mapping For remapping aware guests, hypervisor involved in map/unmap of DMA buffers Pass-through Model VM 0 Guest OS and Apps Device Drivers VM n Hypervisor Guest OS and Apps Device Drivers PCI-SIG I/O Virtualization Working Group Activity towards standardizing natively sharable I/O devices IOV devices provide virtual interfaces, each independently assignable to VMs Assigned Devices Pro: Highest Performance Pro: Smaller Hypervisor Pro: Device-assisted sharing Con: VM Migration Limits

40 DMA remapping: IOTLB scaling Address Translation Services (ATS) extensions to PCIe * enable IOTLB scaling ATS endpoint implements Device IOTLBs Device-IOTLBs can be used to improve performance E.g., Cache only static translations (e.g. command buffers) Pre-fetch translations to reduce latency Minimizes dependency on root-complex caching Support device-specific demand I/O paging *Other names and brands may be claimed as the property of others

41 Translation Request Translation Response Translated DMA Request Address Translation Services (ATS) ATS Translation Flows Device issues Translation Requests to root-complex Root-complex provides Translation Response Device caches translation locally in Device IOTLB Devices can issue DMA with translated address Translated DMA from enabled devices bypass address translation Root Complex Translate Address Remap Hardware Device IOTLB Endpoint Device IOTLB DMA using Translated Address VT-d supports per-device control of ATS *Other names and brands may be claimed as the property of others

42 VT-x & VT-d working together Virtual Machines Virtual Machine Monitor (VMM) Logical Processors VT-x Binary Translation Paravirtualization Page-table Shadowing Physical Memory IO-Device Emulation Interrupt Virtualization DMA Remap VT-d I/O Devices Hardware Virtualization Mechanisms under VMM Control

43 Mapping to VMM software challenges Virtual Machines (VMs) VM 0 VM 1 VM 2 Apps Apps Apps OS OS OS VM n Apps OS VMM (a.k.a., hypervisor) Higher-level VMM Functions: Resource Discovery / Provisioning / Scheduling / User Interface Processor Virtualization Memory Virtualization I/O Device Virtualization Ring Virtual CPU Binary Deprivileging Configuration Translation Page-table EPT Configuration Shadowing I/O DMA and Interrupt Interrupt Remapping Configuration Remapping I/O Device Emulation Physical Platform Resources VT-x CPU 0 CPU n VT-x2 VT-d VT-d2 PCI SIG Storage VMDq Network Processors Memory I/O Devices

Intel Virtualization Technology Roadmap and VT-d Support in Xen

Intel Virtualization Technology Roadmap and VT-d Support in Xen Jun Nakajima Intel Open Source Technology Center Legal Disclaimer INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS.