COSC6376 Cloud Computing Lecture 15: IO Virtualization

Transcription

1 COSC6376 Cloud Computing Lecture 15: IO Virtualization Instructor: Weidong Shi (Larry), PhD Computer Science Department University of Houston

2 IOV Outline

3 PCI-E Sharing

4 Terminology System Image 1 Virtual System 1 Virtual I/O I/O Virtualization Intermediary Physical System Physical I/O System Image 2 Virtual System 2 Virtual I/O I/O Virtualization (IOV) - the capability for a single physical I/O unit to be shared by more than one System Image I/O Virtualization Intermediary (IOVI) - software or firmware that is used to support IOV by intervening on one or more of the following: Configuration, I/O, and Memory operations from a System Image; and DMA, completion, and interrupt operations to a System Image

5 PCI endpoint (EP) sharing Host CPU set 1 2 Host CPU set RCVE - RC Virtualization Enablers PCI Root IO VI IOVI - IO Virtualization Intermediary 1 2 PCI Root IO VI RCVE ATPT - Address Translation and Protection Table Interrupt Table Switch Switch EP Shared Thru Intermediary RC has no virtualization enablers One or more System Images EPs shared through IO VI IOVI is involved in all IO transactions and performs all IO Virtualization Functions, for example Multiplexes s IO queues onto a single queue in the adapter EP is not required to support any virtualization functions IOV IOV IOV Natively Shared s RC has virtualization enablers One or more System Images EPs shared through IO VI Same as IOVI without ATPT IOV enabled EPs are directly shared IOVI involved in config ops Data transfers are direct

6 IOV mechanisms within a single physical system Dedicated (No Virtualization) Shared Through Intermediary Natively Shared System Image 1 System Image 1 System Image 2 System Image 1 System Image 2 Graphic Depiction I/O Virtualization Intermediary I/O Virtualization Intermediary Physical I/O Physical I/O Physical I/O Physical System Physical System Physical System Intermediary Role None Virtualizes physical I/O by intervening on configuration and data transfer operations Manages assignment of Virtual Resources by intervening on configuration operations Configuration Operation Path direct to VI serves as proxy ( to VI; VI to ) VI serves as proxy ( to VI; VI to ) Data Transfer Operation Path direct to VI serves as proxy ( to VI; VI to ) direct to

7 Single Root IOV

8 Single RC IOV enabled endpoint requirements Base 1.x System Host CPU set PCI Root Switch IOV Enabled System Host CPU set 1 N PCI Root Switch PCIM Only endpoints shall be specified for IOV enabled s Native based PCI SR OV enabled s shall be backwards compatible, in a non virtualized mode, with base 1.x SPEC PCI Bridge IOV IOV PCI PCI Switch Switch Bridge IOV IOV

9 Single root IOV endpoint reqs Configuration Management IOV Internal Routing Non-separable Resources Host CPU set 1 N PCI Root Switch Port PCIM IOV Enabled System VF (Virtual Function 1 ) : VF (Virtual Function N ) Physical Resources 1 Physical Resources N Sharable Resource Pool 1. A mechanism shall be provided to allow a VF to be associated with an, such that data movement operations are enabled and can be performed directly between the and its associated VF, without VI involvement 2. The virtualization mechanisms defined in this specification may require a VI (such as a PCI Configuration Manager) to be involved for configuration operations performed on a VF

10 SR-IOV Physical Functions (PFs) are full devices that include the SR- IOV capabilities. Physical Functions configure and manage the SR-IOV functionality by assigning Virtual Functions. Virtual Functions (VFs) are simple functions that only process I/O. A single Ethernet port, the Physical Device, may map to many Virtual Functions that can be shared to virtualized guests. The number of Virtual Functions a device may have is limited by the device hardware.

11 Multi-Root IOV

12 Multi-Root IOV endpoint requirements Physical View Host CPU set Host CPU set Host CPU set Root Root Root Switch Switch The multi-root solution Shall give each RC its own Virtual Hierarchy Shall enable each switch, bridge, function, and VF to be uniquely represented in the configuration space of each RC Host CPU set IOV Virtual View Switch Root Switch Switch Bridge PCI Bridge IOV Switch PCI Bridge PCI Device IOV Device PCI Device PCI Device PCI Device

13 Multi-Root IOV endpoint requirements Host CPU set 1 2 MRA Root MRA Switch MRA Switch Optional SMP Fabric PCI MRA Bridge Host CPU set PCIM x Root MRA Switch PCI Bridge 3 The multi-root solution shall 1. Provide the same characteristics to its IOV enabled s as the single-root solution relative to separate s 2. Enable use of existing 1.x or later RC 3. Enable existing 1.x Switches, s, and to PCI/ Bridges to each be bound to a single RC 4. Enable an IOV enabled endpoint to be shared amongst multiple RC s using a Multi-Root Aware (MRA) switch

14 Multi-Root IOV endpoint requirements 8. Host CPU set 1 MRA Root MRA Switch 7. 2 MRA Switch 9. PCIM Optional SMP Fabric PCI MRA Bridge Host CPU set 1.x Root MRA Switch PCI Bridge The multi-root solution shall 5. Enable a 1.x MRA bridge to PCI/ to assign each of its independent busses to a separate RC 6. Enable multiple MRA switches to be connected together 7. Enable an IOV enabled to be shared by multiple s within one or more RC s 8. Enable hot-plug of physical EPs and VFs 9. Enable a management entity (PCI Configuration Manger, PCIM) to manage the fabric

15 Network Device IOV

16 Network virtualization Virtual I/O in Native and Guest OS Limitations of sharing I/O device in software prevent HPC applications from running in VMs Virtualization overhead Loss of native features Lack of SLA guarantees

17 Evolution of hardware IOV multi-port and multi-function solutions Single-Root (SR IOV) PCI-G specification Direct Hardware Access (pass-thru)

18 I/O virtualization and hardware SLA Over-Provisioning Bandwidth No or Limited QoS Bandwidth Over-Provisioning VM1: VM1 VM1 VM1 2 Gbps VM1: VM1 VM1 VM1 VM2: VM2 VM2 VM2 2 Gbps Bandwidth Buffer VM3: VM3 VM3 VM3 2 Gbps VM2: VM2 VM2 VM4: VM4 VM4 VM2 VM4 2 Gbps Bandwidth Buffer VM2 VM5: VM5 VM5 VM5 2 Gbps Time T0 T1 T2 Time T0 T1 T2 Example: 5 Virtual Machines SLAs require 2 Gbps per VM Low Priority VM2 Receive: 5 Gbps spike Incoming traffic surge and no QoS results in SLAs being violated Traditional Solution: Over-provision bandwidth to attempt to meet SLAs Reduces VM count by 50% or more

19 NIC 1 NIC 0 NIC 3 NIC 4 NIC 2 Exar Guest OS Driver VMxnet Driver Exar Guest OS Driver VMxnet Driver Network virtualization with IOQoS Optimum Resource Utilization Single X3120 VM1 Application O/S VMn Application O/S Replace multiple HBA and NICs with single dual port 10GbE adapter Clustering RoCE IP Storage Data 10GbE IOQoS Virtual L2 Switch Easy management of Prioritization and Bandwidth Allocation X3120 configured with Multiple PCI Functions Neterion X3120 Exar Host Driver KVM Host Direct Access for Latency sensitive (clustering) fabric Integrated L2 Switch

20 One GPU one user Guest Guest Guest Guest VMCS1 VMCS2 VMCS3 VMCS4 Hypervisor VGA Passthrough VGA Passthrough VGA Passthrough VGA Passthrough

21 One GPU multiple users (emulation) GPU GPU GPU Emulation GPU Emulation Emulation Emulation GPU Emulation GPU Emu Driv GPU Emu Driv GPU Emu Driv GPU Emu Driv Guest Guest Guest Guest VMCS1 VMCS2 VMCS3 VMCS4 Hypervisor

22 Guest Guest Guest Guest One GPU multiple users (GPU IOV) VMCS1 VMCS2 VMCS3 VMCS4 Hypervisor Virtual function Virtual function Virtual function Virtual function Video RAM

23

24 Network Virtualization

25 Due: Next week Reading Assignment