협업환경을위한 NVIDIA 의 GPU 가상화및원격가시화기술. 류현곤 (Hyungon Ryu)

Size: px

Start display at page:

Download "협업환경을위한 NVIDIA 의 GPU 가상화및원격가시화기술. 류현곤 (Hyungon Ryu)"

Garry Bell
6 years ago
Views:

1 협업환경을위한 NVIDIA 의 GPU 가상화및원격가시화기술 류현곤 (Hyungon Ryu)

2 GPU Cloud Agenda Remote Visualization issues Trends of Virtualization Technical Issues for GPU Virtualization PCI-e Virtualization NVIDIA s GPU Virtualization OpenXen

3 Part1. Remote Visualization

4 H/W solution : Teradici 3D (PCoIP) Ethernet port

5 VirtualGL on Linux Remote CAE visualization with OpenGL Acceleration

6 Remote OpenGL on Linux Feb Exceed on Demand via VirtualGL Launch linux version Houdini on Windows XP laptop.

7 Remote OpenGL on Windows Tesla M2070Q NVIDIA s remote solution Checkpoint : full OpenGL 4.x function supports Perf. (latency/bandwidth)

8 RealityServer for Mobile Configurators Source:

9 Part2. GPU Virtualization Trends

10 Ecosystem for GPU Virtualization Server venders IBM, Cray, SGI, HP, Oracle CPU venders Intel, AMD, IBM, Oracle GPU Virtualization PCI-SIG NVIDIA, PLX VMM Parallels, Xen, CITRIX, VMware, MS Hyper-V, VirtualBox, KVM

11 Graphics Need Desktop Virtualization Spectrum O&G 3D Anim Workstation Knowledge Worker Task Worker CAE Pre/Post Flash Web Video WPF Data Entry

12 Trends on Virtualization (1) Single user running multiple Desktop OS Windows on Mac Parallels Desktop for Mac VMware fusion Professional Graphics Parallels Workstation VMware Workstation

13 GUI/OGL/DX on GuestOS (Citrix OpenGL demo 2008) VMware Fusion ( Parallels Desktop)

14 NVIDIA Quadro and Tesla M-Q with RemoteFX Enables multiple Win7 VMs SP1 Windows 7 Virtual Machines Applications GDI, DX9 on a single server, each with DX9 graphics acceleration Physical GPU used for render and compression of graphics, with enhanced RemoteFX codec. DX9/10 NVIDIA Quadro Drivers for Win2k8 R2 Access via RDP Available with Win2k8 R2 / Win7 SP1 Primary market is enterprise desktop virtualization (VDI) Tesla M2070-Q Quadro 6000 / 5000 / 4000 Potential opportunities with DX9 midrange pro graphics apps under evaluation

15 Quadro SLI MultiOS for Professional Graphics Quadro SLI MultiOS Quadro SLI MultiOS (Parallels Workstation)

Trends on Virtualization (2) Server platforms running multiple server OS VMware ESX / Virtual Infrastructure Citrix XenServer (Xen-based) Microsoft Windows Hyper-V Oracle / Solaris

16 Trends on Virtualization (2) Server platforms running multiple server OS VMware ESX / Virtual Infrastructure Citrix XenServer (Xen-based) Microsoft Windows Hyper-V Oracle / Solaris (Xen and VirtualBox-based) Red Hat Linux (KVM-based) Parallels Containers Server platforms running multiple desktop OS VMware View (formerly VDI) Citrix XenDesktop MS remotefx with Hyper-V

17 Amazon EC2 via Xen Linux Linux ssh vgpu vgpu XenServer IOMMU ( VT-d) pgpu pgpu can launch CUDA app.

18 Xen GPU Pass-Through for XenDesktop Apr Windows 7 Windows 7 XenDesktop vgpu vgpu Citrix XenServer IOMMU ( VT-d) pgpu pgpu can launch OpenGL CAD app.

19 Part3. NVIDIA s GPU Virtualization

20 2008 Slides for VMware s Virtual GPU Micah Dowty, Jeremy Sugerman, GPU Virtualization on VMware s Hosted I/O Architecture

21 History of GPU Virtualization Internal Proposal Internal Research Investigate the requirement H/W(MCP, PCI-e, PCB, GPU) S/W(OS, driver, OGL/DX/CUDA) Engagement Developing, bugfix Demo & feedback Parallels, VMWare, MS, CITRIX, HP, DELL, SGI, CRAY Product release [ SLI MultiOS] Parallels CITRIX MS VMWARE

GPU virtualization technologies VM API intercept VM Direct-assigned GPU VM VM

OS NVIDIA Driver OS NVIDIA Driver OS NVIDIA Driver Translation, Execution,

Future NVIDIA Fast GPU Hypervisor Path Control Path?

22 GPU virtualization technologies VM API intercept VM Direct-assigned GPU VM VM NVIDIA Virtual GPU VM VM OS API Intercept OS API Intercept OS NVIDIA Driver OS NVIDIA Driver OS NVIDIA Driver OS NVIDIA Driver Translation, Execution, Readback Hypervisor NVIDIA Driver DMAR Hypervisor Direct Path DMAR Hypervisor Future NVIDIA Fast GPU Hypervisor Path Control Path? NVIDIA Quadro with Microsoft RemoteFX NVIDIA Tesla / Quadro with NVIDIA Multi-OS unannounced future product

23 Part4. Backup PCI-e Virtualization

24 PCI-e Virtualization Chipset I/O remap (CPU, M/B venders) Alternate Routing ID Interpretation (ARI) Address Translation Services (ATS) Access Control Services (ACS) Single Root I/O Virtualization (SR-IOV) Multi-Root I/O Virtualization (MR-IOV)

25 NV Device Client Model Resource allocation abstracted for drivers by the RM Direct access to resources are controlled and protected Abstraction and indirection are used throughout the software model for historical reasons (portability, leverage, maintenance) Driver Driver BAR0 for register access BAR1 for frame buffer access BAR2 for instance space access OS RM BAR0 BAR1 BAR2 MMU GPU

26 Sample of PCI-e tree for GPU Intel IOH -[0000:00] [01-04] [02-04] [03] \-00.1 \-02.0-[04] \ a.0 +-1b.0 +-1c.0-[05]-- +-1c.7-[06] d.0 +-1e.0-[07] f.0 +-1f.2 +-1f.3 \-1f.5 NV BR04 PCI-e switch 2 GPU

27 fff : reserved efff : System RAM 0008f ffff : reserved 000e fffff : reserved cf62bfff : System RAM d1ed4 : Kernel code 014d1ed5-018a7d2f : Kernel data 019e c9eff7 : Kernel bss ffffff : Crash kernel cf62c000-cf633fff : reserved cf cf66efff : System RAM cf66f000-cf6befff : reserved cf6bf000-cf6d3fff : System RAM cf6d4000-cf7befff : ACPI Non-volatile Storage cf7bf000-cf7e5fff : System RAM cf7e6000-cf7eefff : ACPI Tables cf7ef000-cf7effff : System RAM cf7f0000-cf7fefff : ACPI Tables cf7ff000-cf7fffff : System RAM cf cfffffff : reserved d d01fffff : PCI Bus 0000:05 d d03fffff : PCI Bus 0000:05 d f1ffffff : PCI Bus 0000:01 d f1ffffff : PCI Bus 0000:02 d e1ffffff : PCI Bus 0000:04 d dfffffff : 0000:04:00.0 e e1ffffff : 0000:04:00.0 e f1ffffff : PCI Bus 0000:03 e efffffff : 0000:03:00.0 f f1ffffff : 0000:03:00.0 f f50fffff : PCI Bus 0000:01 f f50fffff : PCI Bus 0000:02 f f30fffff : PCI Bus 0000:04 f f2ffffff : 0000:04:00.0 f f2ffffff : nvidia f f3003fff : 0000:04:00.1 f f3003fff : ICH HD audio f f30fffff : 0000:04:00.0 f f50fffff : PCI Bus 0000:03 f f4ffffff : 0000:03:00.0 f f4ffffff : nvidia f f5003fff : 0000:03:00.1 f f5003fff : ICH HD audio f f50fffff : 0000:03:00.0 f f51fffff : PCI Bus 0000:07 f f5103fff : 0000:07:03.0 f f51047ff : 0000:07:03.0 f f51047ff : firewire_ohci f f52fffff : PCI Bus 0000:06 f f52003ff : 0000:06:00.0 f f531ffff : 0000:00:19.0 f f531ffff : e1000e f f5323fff : 0000:00:1b.0 f f5323fff : ICH HD audio f f5324fff : 0000:00:19.0 f f5324fff : e1000e f f53253ff : 0000:00:1d.0 f f53253ff : ehci_hcd f f53257ff : 0000:00:1a.0 f f53257ff : ehci_hcd f f53258ff : 0000:00:1f.3 f ffffffff : reserved f fbffffff : PCI MMCONFIG 0 [00-3f] fec00000-fec00fff : IOAPIC 0 fed00000-fed003ff : HPET 0 fee00000-fee00fff : Local APIC fffffff : System RAM cf62c000-cf633fff : reserved cf cf66efff : System RAM cf66f000-cf6befff : reserved cf6bf000-cf6d3fff : System RAM cf6d4000-cf7befff : ACPI Non-volatile Storage cf7bf000-cf7e5fff : System RAM cf7e6000-cf7eefff : ACPI Tables cf7ef000-cf7effff : System RAM cf7f0000-cf7fefff : ACPI Tables cf7ff000-cf7fffff : System RAM cf cfffffff : reserved d d01fffff : PCI Bus 0000:05 d d03fffff : PCI Bus 0000:05 d f1ffffff : PCI Bus 0000:01 I/O Memory Map of NVIDIA GPU d f1ffffff : PCI Bus 0000:02 d e1ffffff : PCI Bus 0000:04 d dfffffff : 0000:04:00.0 e e1ffffff : 0000:04:00.0 e f1ffffff : PCI Bus 0000:03 e efffffff : 0000:03:00.0 f f1ffffff : 0000:03:00.0 f f50fffff : PCI Bus 0000:01 f f50fffff : PCI Bus 0000:02 f f30fffff : PCI Bus 0000:04 f f2ffffff : 0000:04:00.0 f f2ffffff : nvidia f f3003fff : 0000:04:00.1 f f3003fff : ICH HD audio f f30fffff : 0000:04:00.0 f f50fffff : PCI Bus 0000:03 f f4ffffff : 0000:03:00.0 f f4ffffff : nvidia f f5003fff : 0000:03:00.1 f f5003fff : ICH HD audio f f50fffff : 0000:03:00.0 f f51fffff : PCI Bus 0000:07 f f5103fff : 0000:07:03.0 f f51047ff : 0000:07:03.0 f f51047ff : firewire_ohci f f52fffff : PCI Bus 0000:06 f f52003ff : 0000:06:00.0 f f531ffff : 0000:00:19.0 f f531ffff : e1000e f f5323fff : 0000:00:1b.0 f f5323fff : ICH HD audio Reference : Tesla S1070: 16 MB (BAR0) for register access 64 MB (BAR1) for frame buffer access 32 MB (BAR2) for instance space access Tesla S20xx: 32 MB (BAR0) for register access 64 MB (BAR1) for frame buffer access 64 MB (BAR2) for instance space access 04:00 Region 0: Memory at f (32-bit, non-prefetchable) [size=16m] Region 1: Memory at d (64-bit, prefetchable) [size=128m] Region 3: Memory at e (64-bit, prefetchable) [size=32m] 03:00 Region 0: Memory at f (32-bit, non-prefetchable) [size=16m] Region 1: Memory at e (64-bit, prefetchable) [size=128m] Region 3: Memory at f (64-bit, prefetchable) [size=32m]

28 BAR0 BAR1 BAR fffff : PCI Bus # ffff : 0000:20: fffff : PCI Bus # ffff : 0000:30: a4ffffff : PCI Bus #b a4ffffff : PCI Bus #b effffff : PCI Bus #b effffff : PCI Bus #b3 +-[0000:80] [0000:90-9f] bffffff : PCI Bus #b ffff [0000:a0-af]-- : 0000:b4: fff [0000:b0-cf] [0000:b1-cf] [0000:b2-b8] [0000:b3-b8] [0000:b4-b5] : 0000:b4:00.1 9a bffffff : 0000:b4:00.0 \ c effffff : PCI Bus #b6 \-02.0-[0000:b6-b8] c dffffff : 0000:b6:00.0 9e e07ffff : 0000:b6:00.0 \ e e083fff : 0000:b6: [0000:b9-bf]-- 9f a4ffffff : PCI Bus #c [0000:c0-c7] [0000:c1-c7] [0000:c2-c4] f a4ffffff : PCI Bus #c1 \ f a1ffffff : PCI Bus #c2 \-02.0-[0000:c5-c7] f f07ffff : 0000:c2:00.0 \ f f083fff : 0000:c2:00.1 \-03.0-[0000:c8-cf]-- I/O Memory Map of S2070 a a1ffffff : 0000:c2:00.0 a a4ffffff : PCI Bus #c5 a a3ffffff : 0000:c5:00.0 a a407ffff : 0000:c5:00.0 a a4083fff : 0000:c5:00.1 afd00000-afdfffff : PCI Bus #04 afe00000-afefffff : PCI Bus #0a aff00000-aff03fff : 0000:00:16.7 aff04000-aff07fff : 0000:00:16.6 aff08000-aff0bfff : 0000:00:16.5 aff0c000-aff0ffff : 0000:00:16.4 aff10000-aff13fff : 0000:00:16.3 aff14000-aff17fff : 0000:00:16.2 aff18000-aff1bfff : 0000:00:16.1 aff1c000-aff1ffff : 0000:00:16.0 aff20000-aff200ff : 0000:00:1f.3 b f3ffffff : PCI Bus #b0 b f3ffffff : PCI Bus #b1 b d3ffffff : PCI Bus #c0 b d3ffffff : PCI Bus #c1 b c3ffffff : PCI Bus #c5 b bfffffff : 0000:c5:00.0 c c3ffffff : 0000:c5:00.0 c d3ffffff : PCI Bus #c2 c cfffffff : 0000:c2:00.0 d d3ffffff : 0000:c2:00.0 d f3ffffff : PCI Bus #b2 d f3ffffff : PCI Bus #b3 d e3ffffff : PCI Bus #b6 d dfffffff : 0000:b6:00.0 e e3ffffff : 0000:b6:00.0 e f3ffffff : PCI Bus #b4 e efffffff : 0000:b4:00.0 f f3ffffff : 0000:b4:00.0 f f4003fff : 0000:80:16.7 BAR0 for register access BAR1 for frame buffer access BAR2 for instance space access NV BR04 PCI-e switch Tesla S2070 b4:00.0 BAR0 Memory at 9a (32-bit, non-prefetchable) [size=32m] BAR1 Memory at e (64-bit, prefetchable) [size=128m] BAR2 Memory at f (64-bit, prefetchable) [size=64m] b6:00.0 BAR0 Memory at 9c (32-bit, non-prefetchable) [size=32m] BAR1 Memory at d (64-bit, prefetchable) [size=128m] BAR2 Memory at e (64-bit, prefetchable) [size=64m] c2:00.0 BAR0 Memory at a (32-bit, non-prefetchable) [size=32m] BAR1 Memory at c (64-bit, prefetchable) [size=128m] BAR2 Memory at d (64-bit, prefetchable) [size=64m] c5:00.0 BAR0 Memory at a (32-bit, non-prefetchable) [size=32m] BAR1 Memory at b (64-bit, prefetchable) [size=128m] BAR2 Memory at c (64-bit, prefetchable) [size=64m]

29 Chipset I/O remap Intel VT-d AMD IOMMU Intel VT-d DMA remapping IOMMU Data Structures

30 AMD and Intel IOMMU AMD and Intel IOMMU retranslate DMA requests. IOMMU translates requests to system memory. Intel IOMMU translates peer-to-peer requests. Peer address decoding must be done on translated address. The address in the packet changes, therefore new CRC needed. AMD IOMMU spec does not mention peer-to-peer. IOMMU does not see local memory requests. IOMMU supports translation request. Translation requests return IOMMU page table entries. IO Device can cache the entries. IO Device must support invalidation requests from the IOMMU. Could be used to translate local memory requests. Translation failures are not recoverable. IOMMU can kill requests that fail translation. IOMMU can return obfuscated data on reads that fail translation. For PCIE devices, IOMMU returns Unsupported Request Number of page table levels depends on the address space size One level for 21-bit address. One additional level per 9 additional address bits.

31 Multi-Root I/O Virtualization (MR-IOV) Multi-Root I/O Virtualization defines how a single PCIe endpoint can expose multiple virtual functions for simultaneous use by different operating system images across multiple physical systems SuperMicro 7046GT has dual IOH with QPI

w/o re-compile on thin client access modeling data w/o data-download.

32 당신의가까운미래 CAD on GPU Cloud Tablet & Laptop on Yacht Internet (4G LTE) 500kbps Launch any app. w/o re-compile on thin client access modeling data w/o data-download. Seamless user-interaction [ color, pin-point, latency] Cloud GPU Virtualization Server Side Rendering Remote Visualization H.264

33 감사합니다.

How NVIDIA GRID Brings Amazing Graphics to the Virtualized Experience

How NVIDIA GRID Brings Amazing Graphics to the ized Experience Who is NVIDIA AGENDA GRID For VDI GRID Enabled Solutions User Profiles and Experiences From Super Phones to Super Cars GPU NVIDIA Brands Mobile