Live Migration of vgpu Aug 2016 Xiao Zheng xiao.zheng@intel.com Kevin Tian kevin.tian@intel.com
Agenda GPU Virtualization and vgpu Live Migration vgpu Resources Design and Solution Current Status Summary 2
GPU Virtualization Usage Cases IT Desktop Media Could Desktop#N 2D/3D Office Productivity CADs Media Process VDI 3D/Media Acceleration Network Remote Framebuffer Streaming 3
XENGT Architecture Mediated Pass-through Host Linux Qemu vpci layout VGA VM2 VM1 pass-through for performance critical resource i915 Driver Hooks I/O Hooks Address space balloon vgt vgpu vgpu GFX Driver Address space balloon Trap and emulate for privileged resource Time-shared among VMs XEN MPT Services GPU 4
vgpu Live Migration Live Migration: Load balance, Maintenance, Fault recovery Unfortunately most of vgpu solutions do not support migration except Intel GVT-g Intel GVT-g with mediated pass-through Typically a GPU pass-through solution Typically a GPU SRIOV solution Host Linux Qemu vpci V layo G ut A VM2 VM1 Guest vgpu Guest vgpu i915 Drive r Hook s I/O Hook s Address space balloon vgt vgp vgp U GFX Driver Addre ss space balloo n Hypervisor HW GPU GPU GPUGPU Hypervisor SRIOV HW GPU PF VF VF XEN GPU MPT Services Intel GVT-g architecture (Mediation) make it possible for seamless live migration 5
Live Migration of vgpu in Intel GVT-g Highlight feature: Intel GVT-g is Open Source project, upstream ongoing vgpu Live Migration follows existing hypervisor migration flow 3D/2D/Media graphics workload seamless migrated between Servers or Local machine Support Linux/Windows Guest Live Migration Service downtime latency < 0.3 sec (Guest RAM 2GB, assigned 512MB vgpu memory, 10Gpbs adapter) VM1 Office uage VM2 CADs VM3 Media Process How? VM3* Media Process VM4 VDI vgpu Server1 vgpu Server2 6
Demo: vgpu Live Migration with 3D workload https://www.youtube.com/watch?v=y2sku5jodiy 7
Agenda GPU Virtualization and vgpu Live Migration vgpu Resources Design and Solution Current Status Summary 8
Inside of vgpu instance pass-through for performance critical resource Trap and emulate for privileged resource Time-shared among VMs GPU Interrupt Registers vgpu Render Engine Graphics Memory XenGT Interrupt Registers vgpu Render Engine Graphics Memory Interrupt Registers Render Engine GTT page table Graphics Memory 9
Challenge of Migrating vgpu Instance When and how to migrate Graphics Memory When and how to migrate Guest Graphics Page Table When and how to migrate Render Engine State Pre-copy GPU dirty page logging Stop and Copy Remove vgpu from scheduler Save and migrate all vgpu state Resume/Post-Copy Restore vgpu state Add vgpu into scheduler Service downtime Total migration time 10
Migration Policies for Different vgpu Resources Registers GTT page table Graphics Memory Context: Render Engine Copy and Restore Recreate Shadowing Track Dirty and Copy Recreate Shadowing 11
Agenda GPU Virtualization and vgpu Live Migration vgpu Resources Design and Solution Current Status Summary 12
Guest GTT Page Table Migration Guest VM GTT view VM1 GTT 0 ~ 512MB VM2 GTT 0 ~ 512MB Both GGTT and PPGTT are shadowed for Guest GGTT required rebasing due to GGTT partition among VMs GMA rebasing GMA rebasing Migration process actually: A. Copy entire Guest GTT page table B. Re-create the shadow page table for Guest HW GTT view VM0 GM address = 0xA VM1 VM2 on Target side C. Rebasing GGTT for GPU commands Migration GMA rebasing GTT = GGTT or PPGTT GMA = Graphics Mem Address Graphics Memory Address rebasing: Target HW GGTT view VM0 GM address = 0xB VM1* All vgpu cmds from Guest need to be rebased on new address in GVT-g before send to real GPU HW 13
Guest Graphics Memory Migration Pre-copy: Logging dirty graphics memory pages Stop-and-Copy: Migrate contents to target Resume/Post-copy: Recreate GTT page table based on target mfn Guest VM GTT view VM1 GTT 0 ~ 512MB Source host Physical Mem Source mfn Migrate page content Problem: Intel GPU page table entities has no Dirty or Accessed flags to track dirty pages Solution: Copy all used graphics memory to target. Target host Physical Mem Target mfn 14
Render Engine State Migration GGTT address vgpu Guest submitted CTX Render Context4 Render Context3 Render Context2 Context in queue o o o Intel GPU HW is context based CTX locates in GGTT memory Render engine state is contained within CTX Migration happens at this point Server1 Render Engine HW in idle Server2 Render Engine HW in idle Context1 completed Context0 completed CTX N Render Context required to be migrated 15
Agenda GPU Virtualization and vgpu Live Migration vgpu Resources Design and Solution Current Status Summary 16
Current Status Experimental support both KVMGT and XENGT Platforms: Intel 5 th /6 th Generation Intel Core Processors Benchmarks covered: Windows guest: Heaven, 3Dmark06, Trophic, Media encoding/decoding, Linux guest: lightsmark, 2D Quality: 12hours overnight testing, migrating every 30sec Timing: (Guest RAM 2GB including 512MB Graphics memory, 10Gbps adapter) Service downtime ~0.3sec Total migration time: ~1.7sec Pre-copy GPU dirty page logging Stop and Copy Remove vgpu scheduler Save and migrate all vgpu state Resume/Post-Copy Restore vgpu state Add vgpu into scheduler Service downtime ~0.3sec Total migration time ~1.7sec 17
Summary Need 3D/2D/Media workload in virtualization? GVT-g is the choice Need GPU virtualization with migration support? GVT-g is the choice 18
Resource Links Project webpage and release: https://01.org/igvt-g Project public papers and document: https://01.org/group/2230/documentation-list Intel IDF: GVT-g in Media Cloud: https://01.org/sites/default/files/documentation/sz15_sfts002_100_engf.pdf XenGT introduction in summit in 2015: http://events.linuxfoundation.org/sites/events/files/slides/xengt- Xen%20Summit-REWRITE%203RD%20v4.pdf XenGT introduction in summit in 2014: http://events.linuxfoundation.org/sites/events/files/slides/xengt- LinuxCollaborationSummit-final_1.pdf 19
Legal Notices and Disclaimers Intel technologies features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. No computer system can be absolutely secure. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance. Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction. This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps. Statements in this document that refer to Intel s plans and expectations for the quarter, the year, and the future, are forward-looking statements that involve a number of risks and uncertainties. A detailed discussion of the factors that could affect Intel s results and plans is included in Intel s SEC filings, including the annual report on Form 10-K. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate. Intel, Core and the Intel logo are trademarks of Intel Corporation in the United States and other countries. OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission by Khronos. *Other names and brands may be claimed as the property of others. 2016 Intel Corporation. 20