3D Graphics Dev Day. Copyright Khronos Group Page 1

Transcription

1 3D Graphics Dev Day Copyright Khronos Group Page 1

2 Khronos 3D Graphics Dev Day - Sessions 1:20pm - Vulkan Game Development on Mobile 2:40pm - Vulkan on Desktop Deep Dive 4:00pm - When Vulkan was One: Looking Forward, Looking Back Collect them all! Copyright Khronos Group Page 2

3 Vulkan on Desktop Deep Dive Vulkan in Xenko - Jörg Wollenschläger (Silicon Studio) Unity - Jesse Barker (Unity) Vulkan Multi-GPU - Jeff Bolz (NVIDIA) Vulkan on the Desktop - Dan Baker (Oxide) Copyright Khronos Group Page 3

4 Copyright Khronos Group Page 1 Vulkan Multi-GPU Jeff Bolz, NVIDIA Corp. February 2017

5 Copyright Khronos Group Page 2 What/Why? A device group is a set of physical devices that support multi-gpu rendering Assumes a certain system configuration - Similar/identical GPUs - WDDM must be in linked display adapter mode - In short, an SLI/Crossfire system - Not dgpu/igpu New Device and Instance extensions: - KHX_device_group(_creation) Goal: Support the common multi-gpu rendering techniques (AFR/SFR/VR) Design philosophy: - Single logical device - Hide/share duplicated object creation - Make adding device group support as non-invasive as possible

6 Copyright Khronos Group Page 3 Device Group Enumeration/Creation Each GPU is still advertised as a distinct VkPhysicalDevice Advertise groups of physical devices from which a single logical device can be created: typedef struct VkPhysicalDeviceGroupPropertiesKHX {... uint32_t physicaldevicecount; VkPhysicalDevice physicaldevices[vk_max_device_group_size_khx]; } VkPhysicalDeviceGroupPropertiesKHX; vkenumeratephysicaldevicegroupskhx(..., /*out array*/vkphysicaldevicegrouppropertieskhx*); Physical Devices GTX 1080 GTX 1080 igpu Device Groups Group 0 (1080 x 2) Group 1 Once the logical device is created, it is used for object creation and work submission for all GPUs

7 Copyright Khronos Group Page 4 Memory Allocation Device-local heap has an instance for each GPU (still advertised as one heap) Each VkDeviceMemory has a number of memory instances - Common case one instance for non-local, N instances for local Most device-local memory allocations should be made with N instances - For static resources, you want a fast local copy of the data - For dynamic resources, you need a separate copy for each GPU/frame (for AFR) Example: Single instance non-device-local alloc, multi-instance device-local alloc Non-local heap Alloc 0 Shared Local heap Alloc 1 Alloc 1 GPU 0 GPU 1

8 Resource Binding Each resource has an instance for each GPU Mental Model: A resource has a single virtual address across all GPUs, which can be bound to a local or peer memory instance on each GPU New (extensible) bindings commands: vkbind{image,buffer}memory2khx - Add control over which memory instance each resource instance is bound to typedef struct VkBindImageMemoryInfoKHX {... uint32_t deviceindexcount; const uint32_t* pdeviceindices; } VkBindImageMemoryInfoKHX; pdeviceindices = {0,1} Image Instance 0 Image Instance 1 Memory Instance 0 Memory Instance 1 GPU 0 GPU 1 Copyright Khronos Group Page 5

9 Resource Binding Descriptor sets have a single instance - Expected use is for different bindings to be hidden behind same virtual address - Put image X in the set, and X is bound to some memory instance on each GPU Example: Peer binding for remote texturing Descriptor Set pdeviceindices = {1,0} Sampled Image Image Instance 0 Image Instance 1 Memory Instance 0 Memory Instance 1 Shared GPU 0 GPU 1 Copyright Khronos Group Page 6

10 Resource Binding Peer Memory feature bits advertise what operations are supported on peer bindings: typedef enum VkPeerMemoryFeatureFlagBitsKHX { VK_PEER_MEMORY_FEATURE_COPY_SRC_BIT_KHX = 0x , VK_PEER_MEMORY_FEATURE_COPY_DST_BIT_KHX = 0x , VK_PEER_MEMORY_FEATURE_GENERIC_SRC_BIT_KHX = 0x , VK_PEER_MEMORY_FEATURE_GENERIC_DST_BIT_KHX = 0x , } VkPeerMemoryFeatureFlagBitsKHX; - COPY_DST is required, NVIDIA also supports COPY_SRC and GENERIC_SRC Copy to/from peer memory by using a resource with peer memory bound Image A Instance 0 vkcmdcopyimage(src=imagea, dst=imageb); Execute on GPU 0 Image B Instance 0 Local Instance Peer Instance Copyright Khronos Group Page 7

11 Copyright Khronos Group Page 8 SFR Image Binding Map rectangular regions of image instances to local/peer memory instances typedef struct VkBindImageMemoryInfoKHX {... uint32_t SFRRectCount; const VkRect2D* psfrrects; } VkBindImageMemoryInfoKHX; SFRRectCount == N^2 - One rect per (resource,memory) instance pair - Element i*n+j is the rect in resource instance i that maps to memory instance j Example: left/right split Image Instance 0 Image Instance 1 Local Peer Peer Local VkRect2D sfrrects[4] = { { { 0, 0 }, { w/2, h } }, { { w/2, 0 }, { w/2, h } }, { { 0, 0 }, { w/2, h } }, { { w/2, 0 }, { w/2, h } }, };

12 Copyright Khronos Group Page 9 Command Buffer Recording/Submission Commands in a command buffer can be directed to a subset of the devices: vkcmdsetdevicemaskkhx(vkcommandbuffer commandbuffer, uint32_t devicemask); CommandBuffer Mask 0x1 vkcmd vkcmd Mask 0x2 vkcmd vkcmd Mask 0x3 vkcmd vkcmd Furthermore, command buffers can be submitted to a subset of devices: typedef struct VkDeviceGroupSubmitInfoKHX {... uint32_t commandbuffercount; const uint32_t* pcommandbufferdevicemasks; } VkDeviceGroupSubmitInfoKHX; Allows directing work to devices at coarse or fine granularity

13 Copyright Khronos Group Page 10 State Divergence Should state be allowed to diverge between GPUs? - Yes and no Rules: - State can be set on a subset of GPUs based on vkcmdsetdevicemask - Scissor and Viewport state can vary between GPUs at Draw time - For other state, all draws/dispatches use the most recently set state - BUT: when recording a draw/dispatch on GPU <i>, the most recent setting of any relevant state must include GPU <i> in the device mask Valid: Valid: Mask 0x1 BindPipeline(A) Draw Mask 0x2 BindPipeline(B) Draw BindPipeline(A) Mask 0x1 Draw Mask 0x3 BindPipeline(B) Mask 0x2 Draw Invalid: Mask 0x1 BindPipeline(A) Mask 0x2 BindPipeline(B) Mask 0x3 Draw - Different pipeline for each GPU in a single draw

14 Copyright Khronos Group Page 11 SFR/VR Features Split Frame Rendering: - Scissor state is allowed to diverge between GPUs (e.g. render to local instance) - Per-GPU render areas control loadop, storeop, AA resolve typedef struct VkDeviceGroupRenderPassBeginInfoKHX {... uint32_t devicerenderareacount; const VkRect2D* pdevicerenderareas; } VkDeviceGroupRenderPassBeginInfoKHX; Image Instance 0 Image Instance 1 Local Peer Peer Local New GLSL input variable: gl_deviceindex - Zero-based index of the physical device within the logical device - E.g. Used to select per-eye transform matrix in VR VkRect2D renderareas[2] = { { { 0, 0 }, { w/2, h } }, { { w/2, 0 }, { w/2, h } }, };

15 Copyright Khronos Group Page 12 Synchronization Fences/events/semaphores all have a single instance of the signaled state - Fence transitions to signaled when all GPUs complete the submission - Event can only be waited on by the GPU that signaled it - Each semaphore wait/signal operation occurs on a single GPU, but a semaphore can be used by multiple GPUs (e.g. signal on GPU 0 then wait on GPU 1) - Queue submission structures are extended to indicate which GPU(s) perform each operation typedef struct VkDeviceGroupSubmitInfoKHX {... uint32_t waitsemaphorecount; const uint32_t* pwaitsemaphoredeviceindices; uint32_t signalsemaphorecount; const uint32_t* psignalsemaphoredeviceindices; } VkDeviceGroupSubmitInfoKHX; Pipeline barriers normally perform synchronization within a GPU - Flag to opt in to pipeline barriers syncing between GPUs

16 Copyright Khronos Group Page 13 Swapchain Creation and Binding Can use vkcreateimage+vkbindimagememory2 in lieu of vkgetswapchainimages - Think of the swapchain as owning the memory, and images are separately created/bound to that memory - Bound by swapchain/imageindex rather than memory/memoryoffset typedef struct VkBindImageMemorySwapchainInfoKHX {... VkSwapchainKHR swapchain; uint32_t imageindex; } VkBindImageMemorySwapchainInfoKHX; Swapchain Image 0 Swapchain Image 1 Swapchain Image 2 Swapchain memory imageindex 0 imageindex 1 imageindex 2 - Allows creating peer/sfr bindings of swapchain memory (for rendering, not presentation)

17 Copyright Khronos Group Page 14 Presentation Modes Support local/remote/split presentation on systems that support them Device group advertises supported modes/devices typedef struct VkDeviceGroupPresentCapabilitiesKHX {... uint32_t presentmask[vk_max_device_group_size_khx]; VkDeviceGroupPresentModeFlagsKHX modes; } VkDeviceGroupPresentCapabilitiesKHX; vkgetdevicegroupsurfacepresentmodeskhx returns just-in-time supported modes for a given surface - Can vary based on window move/resize/etc. - Expect remote scanout only for fullscreen-exclusive

18 Copyright Khronos Group Page 15 Acquire/Present Acquiring a swapchain image takes a mask and returns an index available on those devices typedef struct VkAcquireNextImageInfoKHX {... uint32_t devicemask; } VkAcquireNextImageInfoKHX; Presenting an image takes a mask (must equal the acquire mask) and a mode typedef struct VkDeviceGroupPresentInfoKHX {... uint32_t swapchaincount; const uint32_t* pdevicemasks; VkDeviceGroupPresentModeFlagBitsKHX mode; } VkDeviceGroupPresentInfoKHX; Acquire Mask Present Mode AFR Alternate GPUs REMOTE SFR All GPUs SUM VR All GPUs LOCAL_MULTI_DEVICE

19 Copyright Khronos Group Page 16 Conclusion It sounds like a lot of work, but it s not - Most memory and resources want the default behavior - Submit render passes only to one GPU (AFR) or add per-device scissor (SFR) - A bit of WSI work to present the results Open Source vk_device_group sample in VRWorks (credit: Ingo Esser) - SFR stereo rendering

20 Vulkan in Xenko Vulkan on Desktop Deep Dive Jörg Wollenschläger Software Engineer, Silicon Studio

21 What is Xenko? Next-Level Game Engine Developed by Silicon Studio, Tokyo Release in April (currently free beta) Open-source Written in C# Cross-platform (Windows, UWP, Android, ios, etc.) OpenGL/ES, D3D 11, Vulkan, D3D 12

22 Rendering Architecture Redesign for next-gen APIs Previously D3D11-like Fully accessible and extensible renderer Focus on multi-threading Separation of renderer into highly parallel phases Ground work for completely asynchronous rendering New API abstraction is a mix of Vulkan and D3D 12 Best of both worlds Emulating new concepts in legacy APIs where possible Abstracting differences on a higher level (renderer is more API aware)

23 Rendering Architecture Newly exposed concepts First class: Descriptor sets, pipelines, command buffers Preferring Vulkan s descriptor sets over D3D 12, as they are more intuitive Partially: Only some explicit barriers. Many are still implicit and use state tracking Mostly implicit fences. Recycling of per-frame resources is directly exposed Not yet: Render passes, push constants, transfer queues, etc. Render passes are similar to our render stages, which describe attachments and rendering technique, but will require more high-level changes

24 Rendering Architecture Newly exposed concepts Emulated concepts Staging resources and resource renaming are support on Vulkan through suballocations of upload buffers Descriptor sets and pipelines are fake in legacy APIs API specific code paths Special path on old APIs where resource renaming is needed, e.g. for constant buffer update Special path on Vulkan that uses dynamic descriptor offsets and skipping updates

25 Shaders Current infrastructure Xenko Shader Language Extended HLSL Modular (classes, inheritance, composition, mixins, etc.) Resource groups (similar to cbuffers for resources) D3D Compiler Mixer D3D bytecode D3D XKSL HLSL OpenGL Problems Mixer and GLSL converter are AST transforms Reflection on AST GLSL dialects (e.g. bindings for Vulkan) AST transform Each shader permutation executes the complete pipeline Hard to maintain, complex, costly GLSL SPIR-V glslang OpenGL ES Vulkan

26 Shaders Investigation into new infrastructure Extended SPIR-V Intermediate format for compilation steps New OpCodes for custom features Converter based on glslang Extended glslang D3D bytecode D3D XKSL Extended SPIR-V Mixer/Linker OpenGL GLSL Advantages Compilation per shader class Efficiently working at bytecode level Reflection and optimization in bytecode Lower maintenance due to open-source projects SPIR-V OpenGL ES Vulkan

27 Performance CPU time savings Test scene with ~20000 draw calls Vulkan running at 60 FPS ~75% CPU load across cores D3D 11 Vulkan Time spent per frame Simulation (transformation, culling, sorting, etc.) Preparation (descriptor updates, resource upload, etc.) Drawing Driver thread

28 Summary Requires explicitness and design considerations Works out of the box on Android and Linux May consolidate shader infrastructure Very good debugging experience Decent performance gains Lots of potential for improvements xenko.com

29

30 Established 2013 Founders Leads of Civ V Cumulative 100+ years Independent Studio Game Technology Authors of Nitrous Engine

31 Key Differences Descriptor management differences Shader Stack is HLSL SPIRV Render Passes No explicit memory management Pipeline Barriers Validator more thorough in Vulkan Most of rest is similar to D3D12 Never been easier to support both APIs!

32 More descriptor types buffers and textures are different Descriptors are not free ranged like D3D12, live inside a specific descriptor set with a specific number of descriptors Descriptor set s layout must match pipeline objects exactly cannot be a superset For Ashes meant some descriptor set waste always create max size descriptor sets and bind with empty descriptors

33 D3D12 Global Descriptors Vulkan Descriprtor Set Descriptor Set Descriptor Set Descriprtor Set Descriprtor Set Descriptor Set

34 Similar to D3D12 s root layout Conceptually simpler, a set of descriptor sets get bound at certain slots Strict rules of D3D12 are not present, so need to be careful about hitting glass jaw of hardware We use a single dynamic descriptor set by binding and updating with a new offset for each draw call Have one giant buffer fill with data per frame Don t use too many dynamic descriptors Dynamic Descriptor Constant Memory

35 Buffer Texture Need to create a buffer, and then do copy via vkcmdcopybuffertoimage Must create a layout yourself we use layout of a DDS file Probably be community code for this pretty soon. The exact layout doesn t matter, just needs to be consistent Same applies for doing CPU readback of texture copy to a mappable buffer and read Advantage: No hidden stalls, once semaphore is clear can copy memory seamlessly back and forth Should be careful about high water mark e.g. lots of transient memory needed and discarded for memory upload We do a series of vkcmdcopybuffertoimage calls to copy the data in our CPU mappable buffer into the actual texture

36 Nitrous States Vulkan is more explicit then D3D12 But allows to switch to any mode if contents can be discarded We use our own states that are more explicit then D3D12, but less then Vulkan Having mapping tables to Vulkan pipeline barriers Engine can auto track previous state to help make it simpler RSTATE_DEFAULT_SRV RSTATE_DEFAULT_UAV RSTATE_DATA_READ RSTATE_DATA_WRITE RSTATE_SHADER_READ_SRV RSTATE_SHADER_READ_UAV RSTATE_SHADER_WRITE RSTATE_SHADER_READWRITE RSTATE_COMPUTE_READ_SRV RSTATE_COMPUTE_READ_UAV RSTATE_COMPUTE_WRITE RSTATE_COMPUTE_READWRITE RSTATE_UNINITIALIZED RSTATE_RENDER_OPTIMAL RSTATE_DEPTH_OPTIMAL RSTATE_SHADER_OPTIMAL RSTATE_COLOR_CLEAR RSTATE_DEPTH_CLEAR RSTATE_UAV_CLEAR RSTATE_INDICES RSTATE_INDIRECT_ARGS RSTATE_RT_RESOLVE_SOURCE RSTATE_RT_RESOLVE_DEST RSTATE_PRESENTABLE RSTATE_COMMON RSTATE_GENERIC_READ RSTATE_PREINITIALIZED

37 More important for mobile then desktop Main cause of structural changes in our code base Originally our draw sets had a separate texture and zbuffer Created concept of a RenderGroup in engine, which combines resources needed for rendering together, currently most emulates D3D12 method Don t currently do anything complex with render groups don t do much multi-pass rendering as more common for deferred renderer

38 Somehow, viewport in Vulkan is flipped vertically from D3D Can be solved by extension of negating the height in the viewport depthclampenable is opposite of depthcull! Cull is opposite of clamp

39 Traditionally the biggest hurdle in cross-api support Hurdle is (mostly) gone in Vulkan! Background: OpenGL used GLSL, accepting native text strings and generally a difficult and poor for production Vulkan creates SPIRV standardizing the front end of shading language, and opening the possibility for different languages

40 Major effort from Google, Valve, LunarG and others Oxide helped a tiny tiny bit, mostly freeloading of others Thanks to LunarG for helping us get 30k lines of HLSL code compiled! Still work to do, but good enough to start production work Special thanks to

41 Pipe HLSL syntax into GLSLangvalidator - currently about 99% compatible with HLSL native HLSL syntax HLSL syntax is not HLSL it s a recursive decent parser with more context sensitivity, thus has a few language syntax features not included in HLSL but needed for Vulkan support Outputs SPIRV code from HLSL, which can be consumed by Vulkan Can use exactly the same shaders across platforms!

42 Only issue is dealing with variable bindings different declarations In D3D12, use a remapping table from old registers to descriptor locations Specify these native in Vulkan Use a simple # define trick to define both styles of bindings #define layout(a,b) #else #define register(a) blank #endif layout(set=9,binding=0) sampler SS_DEFAULT : register(s0); layout(set=9,binding=1) sampler SS_DEFAULT_CLAMP : register(s1); layout(set=9,binding=2) sampler SS_ANISO : register(s2); layout(set=9,binding=3) sampler SS_ANISO_CLAMP : register(s3); layout(set=9,binding=4) sampler SS_POINT : register(s4); layout(set=9,binding=5) sampler SS_POINT_CLAMP : register(s5); layout(set=0,binding=0) cbuffer DecalMapParams : register(b0) { row_major float4x4 s_worldtolocal : packoffset(c0.x); float4 s_destoffsetandsize : packoffset(c4.x); float2 s_invdestres : packoffset(c5.x); float s_time : packoffset(c5.z); }; //ResourceSet DecalVB layout(set=1,binding=0)buffer<float4> V_Position : register(t0); layout(set=1,binding=1)buffer<float4> V_TexCoord : register(t1); layout(set=1,binding=2)buffer<float4> V_UserData : register(t2); //EndTexture Set

43 SPRIV is bigger then D3D bytecode by ~10-20x native Current pipeline Dead Code strip (spirv-remap) -> SMOLV -> LZ4 = 2x size of D3D12 More work happening here SPIRV->SPRIV optimizer may be wanted, high level optimizer in HLSL cleans up some things driver may miss

44 Ashes of the Singularity: Escalation in Vulkan Sorry, no perf data, just barely got everything debugged

45 Vulkan is ready for primetime for desktop D3D11 to Vulkan is about the same complexity of D3D11 to D3D12 For us, application/engine level makes no distinction between APIs Only base level graphics layer knows what API is being used If moving from D3D11 to next gen APIs, both could be supported simultaneously without massive extra effort

46

47 Democratize development!

48

49 Graphics Jobs!

50

51

52

53 Graphics Command Buffers Fixed render loop with callout points

54 Unity, Infinite Dreams, and ARM Thursday, March 2, 10am-11am West Hall 3022

55