Live Migration of vgpu Aug 2016 Xiao Zheng xiao.zheng@intel.com Kevin Tian kevin.tian@intel.com
Agenda GPU Virtualization and vgpu Live Migration vgpu Resources Design and Solution Current Status Summary 2
GPU Virtualization Usage Cases IT Desktop Media Could Desktop#N 2D/3D Office Productivity CADs Media Process VDI 3D/Media Acceleration Network Remote Framebuffer Streaming 3
XENGT Architecture Mediated Pass-through Host Linux Qemu vpci layout VGA VM2 VM1 pass-through for performance critical resource i915 Driver Hooks I/O Hooks Address space balloon vgt vgpu vgpu GFX Driver Address space balloon Trap and emulate for privileged resource Time-shared among VMs XEN MPT Services GPU 4
vgpu Live Migration Live Migration: Load balance, Maintenance, Fault recovery Unfortunately most of vgpu solutions do not support migration except GVT-g GVT-g with mediated pass-through Typically a GPU pass-through solution Typically a GPU SRIOV solution Host Linux Qemu vpci V layo G ut A VM2 VM1 Guest vgpu Guest vgpu i915 Drive r Hook s I/O Hook s Address space balloon vgt vgp vgp U GFX Driver Addre ss space balloo n Hypervisor HW GPU GPU GPUGPU Hypervisor SRIOV HW GPU PF VF VF XEN GPU MPT Services GVT-g architecture (Mediation) make it possible for seamless live migration 5
Live Migration of vgpu in GVT-g Highlight feature: GVT-g is Open Source project, upstream ongoing vgpu Live Migration follows existing hypervisor migration flow 3D/2D/Media graphics workload seamless migrated between Servers or Local machine Support Linux/Windows Guest Live Migration Service downtime latency < 0.3 sec (Guest RAM 2GB, assigned 512MB vgpu memory, 10Gpbs adapter) VM1 Office uage VM2 CADs VM3 Media Process How? VM3* Media Process VM4 VDI vgpu Server1 vgpu Server2 6
Demo: vgpu Live Migration with 3D workload https://www.youtube.com/watch?v=y2sku5jodiy 7
Agenda GPU Virtualization and vgpu Live Migration vgpu Resources Design and Solution Current Status Summary 8
Inside of vgpu instance pass-through for performance critical resource Trap and emulate for privileged resource Time-shared among VMs GPU Interrupt Registers vgpu Render Engine Graphics Memory XenGT Interrupt Registers vgpu Render Engine Graphics Memory Interrupt Registers Render Engine GTT page table Graphics Memory 9
Challenge of Migrating vgpu Instance When and how to migrate Graphics Memory When and how to migrate Guest Graphics Page Table When and how to migrate Render Engine State Pre-copy GPU dirty page logging Stop and Copy Remove vgpu from scheduler Save and migrate all vgpu state Resume/Post-Copy Restore vgpu state Add vgpu into scheduler Service downtime Total migration time 10
Migration Policies for Different vgpu Resources Registers GTT page table Graphics Memory Context: Render Engine Copy and Restore Recreate Shadowing Track Dirty and Copy Recreate Shadowing 11
Agenda GPU Virtualization and vgpu Live Migration vgpu Resources Design and Solution Current Status Summary 12
Guest GTT Page Table Migration Guest VM GTT view VM1 GTT 0 ~ 512MB VM2 GTT 0 ~ 512MB Both GGTT and PPGTT are shadowed for Guest GGTT required rebasing due to GGTT partition among VMs GMA rebasing GMA rebasing Migration process actually: A. Copy entire Guest GTT page table B. Re-create the shadow page table for Guest HW GTT view VM0 GM address = 0xA VM1 VM2 on Target side C. Rebasing GGTT for GPU commands Migration GMA rebasing GTT = GGTT or PPGTT GMA = Graphics Mem Address Graphics Memory Address rebasing: Target HW GGTT view VM0 GM address = 0xB VM1* All vgpu cmds from Guest need to be rebased on new address in GVT-g before send to real GPU HW 13
Guest Graphics Memory Migration Pre-copy: Logging dirty graphics memory pages Stop-and-Copy: Migrate contents to target Resume/Post-copy: Recreate GTT page table based on target mfn Guest VM GTT view VM1 GTT 0 ~ 512MB Source host Physical Mem Source mfn Migrate page content Problem: Intel GPU page table entities has no Dirty or Accessed flags to track dirty pages Solution: Copy all used graphics memory to target. Target host Physical Mem Target mfn 14
Render Engine State Migration GGTT address vgpu Guest submitted CTX Render Context4 Render Context3 Render Context2 Context in queue o o o Intel GPU HW is context based CTX locates in GGTT memory Render engine state is contained within CTX Migration happens at this point Server1 Render Engine HW in idle Server2 Render Engine HW in idle Context1 completed Context0 completed CTX N Render Context required to be migrated 15
Agenda GPU Virtualization and vgpu Live Migration vgpu Resources Design and Solution Current Status Summary 16
Current Status Experimental support both KVMGT and XENGT Platforms: Intel 5 th /6 th Generation Intel Core Processors Benchmarks covered: Windows guest: Heaven, 3Dmark06, Trophic, Media encoding/decoding, Linux guest: lightsmark, 2D Quality: 12hours overnight testing, migrating every 30sec Timing: (Guest RAM 2GB including 512MB Graphics memory, 10Gbps adapter) Service downtime ~0.3sec Total migration time: ~1.7sec Pre-copy GPU dirty page logging Stop and Copy Remove vgpu scheduler Save and migrate all vgpu state Resume/Post-Copy Restore vgpu state Add vgpu into scheduler Service downtime ~0.3sec Total migration time ~1.7sec 17
Summary Need 3D/2D/Media workload in virtualization? GVT-g is the choice Need GPU virtualization with migration support? GVT-g is the choice 18
Resource Links Project webpage and release: https://01.org/igvt-g Project public papers and document: https://01.org/group/2230/documentation-list Intel IDF: GVT-g in Media Cloud: https://01.org/sites/default/files/documentation/sz15_sfts002_100_engf.pdf XenGT introduction in summit in 2015: http://events.linuxfoundation.org/sites/events/files/slides/xengt- Xen%20Summit-REWRITE%203RD%20v4.pdf XenGT introduction in summit in 2014: http://events.linuxfoundation.org/sites/events/files/slides/xengt- LinuxCollaborationSummit-final_1.pdf 19
Notices and Disclaimers INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. INTEL PRODUCTS ARE NOT INTENDED FOR USE IN MEDICAL, LIFE SAVING, OR LIFE SUSTAINING APPLICATIONS. Intel may make changes to specifications and product descriptions at any time, without notice. All products, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice. Intel, processors, chipsets, and desktop boards may contain design defects or errors known as errata, which may cause the product to deviate from published specifications. Current characterized errata are available on request. No computer system can provide absolute security under all conditions. Intel Trusted Execution Technology (Intel TXT) requires a computer with Intel Virtualization Technology, an Intel TXT-enabled processor, chipset, BIOS, Authenticated Code Modules and an Intel TXT-compatible measured launched environment (MLE). Intel TXT also requires the system to contain a TPM v1.s. For more information, visit http://www.intel.com/technology/security Intel, Intel logo, Xeon, and Xeon Inside are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. *Other names and brands may be claimed as the property of others. Copyright 2016 Intel Corporation. All rights reserved. 20