Ray Tracing in Games with NVIDIA RTX Ignacio Llamas, Sr. Manager of Real-time Rendering SW Edward Liu, Sr. Real-time Rendering Engineer

Transcription

1 Ray Tracing in Games with NVIDIA RTX Ignaci Llamas, Sr. Manager f Real-time Rendering SW Edward Liu, Sr. Real-time Rendering Engineer Bth #223 - Suth Hall

2 Agenda Intrductin t Ray Tracing fr Games RTX, DXR and GameWrks Real-time Ray Tracing and Denising Path Tracing and Light Baking Integrating Ray Tracing int yur engine 2

3 Intrductin t Ray Tracing fr Games 3

4 Ray Tracing 101 What is Ray Tracing Ray Tracing: a primitive t query the intersectins f rays against sme gemetry Hit t = ~1.8 Ray Origin 4

5 Ray Tracing 101 Applicatins in Games Higher quality in-game rendering Reflectins Ambient Occlusin Shadws Cntent Creatin Wrkflws Light baking Cinematic rendering Path traced reference Nn-rendering applicatins Audi simulatin in VR (VRWrks Audi) Physics / Cllisin detectin AI 5

6 Ray Tracing 101 Applicatins in Games Higher quality in-game rendering Reflectins Ambient Occlusin Shadws Cntent Creatin Wrkflws Light baking Cinematic rendering Path traced reference Nn-rendering applicatins Audi simulatin in VR (VRWrks Audi) Physics / Cllisin detectin AI 6

7 Ray Tracing 101 Rendering Cmmnly used t slve the rendering equatin with Mnte Carl sampling L ω = L e ω + න δ L ω i f ω, ω i ω i n dω i Natural slutin fr Shadws, AO, Reflectins, Light Baking, Film Rendering 7

8 Ray Tracing 101 Ray Tracing in Rendering Tday Similar primitives used tday in games in real-time in varius frms: Screen space ray tracing, distance field ray tracing, vxel cne tracing In this talk: ray tracing against full scene gemetry (mstly triangles) Rendering with Ray Tracing usually requires many samples fr high quality results Frm a few hundred t a few thusand Depending n the cmplexity f the integrand (scene) 8

9 Ray Traced Shadws 9

10 Ray Traced Ambient Occlusin 10

11 Ray Traced Reflectins 11

12 Ray Tracing s f samples per pixel are nt practical in real time Small number f rays per pixel pssible n current HW Is that useful enugh? 12

13 Ray Tracing s f samples per pixel are nt practical in real time Small number f rays per pixel pssible n current HW Is that useful enugh? Previus images generated with 1-2 samples per pixel + real-time denising Yes: high quality results pssible with few samples per pixel + denising 13

14 RTX, DXR and GameWrks 14

15 NVIDIA RTX & DirectX Raytracing (DXR) Cntext NVIDIA RTX : ray-tracing technlgy fr Vlta GPUs The result f a decade f GPU ray tracing R&D at NVIDIA Expsed via Micrsft s new DirectX Raytracing API (DXR) fr DirectX 12 New pipeline: the Ray Tracing Pipeline 15

16 RTX & DXR Pwerful and flexible prgramming mdel, similar t NVIDIA OptiX Easy t write efficient ray tracing algrithms DXR makes it easy t integrate int engines that already use DirectX 12 DXR prvides IHV-agnstic abstractin Benefits RTX prvides an efficient implementatin n NVIDIA Vlta GPUs RTX n Vlta delivers perfrmance needed fr real-time ray tracing Fr mre DXR details see Matt Sandy s talk at 3:30 pm, and NVIDIA blg 16

17 GameWrks Ray Tracing Mre cntext GameWrks: tls, simulatin and rendering technlgy fr develpers Annuncing GameWrks Ray Tracing Denising mdules Ray Traced Shadws, Ambient Occlusin and Reflectins Denising Available sn 17

18 Related Wrk at GDC Be sure t check these ut t Advanced Graphics Day (Mnday): Micrsft, Futuremark and Remedy Epic-NVIDIA-ILMxLAB Reflectins dem earlier tday Additinal talks later tday in this rm: NV Research, EA SEED, EA Frstbite Talk by Micrsft + EA abut future f DirectX (DXR) 18

19 This Talk What t expect A shwcase f pssibilities enabled by real-time ray tracing with RTX Details abut NVIDIA s new GameWrks Ray Tracing Denising mdules Our gal: inspire yu t start experimenting with real-time ray tracing 19

20 Real-Time Ray Tracing and Denising 20

21 Surce f Nise in Ray Tracing The rendering equatin is slved with Mnte Carl sampling L ω = න δ L ω i f ω,ω i ω i n dω i = L ω i f ω, ω i ω i n /p(ω i ) Every term in the estimatr is a cmplicated functin ver the hemisphere n i=0 Incming radiance, visibility, BRDF, and sampling Pdf Insufficient sampling leads t high variance in the estimatr 21

22 Previus Denising Wrk Very active area f research Spatitempral Variance-Guided Filtering: Real-Time Recnstructin fr Path-Traced Glbal Illuminatin [Schied, et al. 17] Interactive Recnstructin f Mnte Carl Image Sequences using a Recurrent Denising Autencder [CHAITANYA, el al. 17] Multiple Axis-Aligned Filters fr Rendering f Cmbined Distributin Effects [Wu, et al. 17] Kernel-predicting Cnvlutinal Netwrks fr Denising Mnte Carl Renderings [Bak, et al. 17] And many thers Als prducts like NVIDIA OptiX 5.0 AI Deniser 22

23 Previus Denising Wrk Input ften cntains 10s r hundreds f samples Or rely n tempral reprjectin(ghsting issue withut crrect mtin vectrs) Denising cst frm ~100ms t minutes Nisy primary visibility samples (DF, Mtin blur) Results ften really clse t grund truth, measured in rmse 23

24 Real-Time Denising Requirement Input image with extremely lw sample cunt We target 1 sample per pixel Ghsting free, temprally stable Denising budget f ~1 ms fr 1080p target reslutin n gaming class GPU Perceptually clse t grund truth, able t preserve: physically crrect cntact hardening Elngatin and anistrpy Smth G-Buffer (i.e.: n DOF, r mtin blur, as cmmn in film rendering) 24

25 Deniser Design Chices Image space vs. Lightmap space vs. Path space vs. Light field space Tempral / histry data vs. single-frame Image data vs. Scene data: ray hit distance, surface rughness, nrmal, light psitin, size etc. Denising f individual terms separately vs. denising all the lighting at nce AI based vs. hand-crafted Many ptins 25

26 State f the Art Real-Time Denising One sample per pixel can achieve a lt! Exciting results with glssy reflectins, area light sft shadws, and AO Effect-specific denising, using scene infrmatin t estimate ptimal filter ft print State f the art quality at extremely lw sample cunt Wrks well with even 1spp N tempral reprjectin invlved ghsting/lagging free 26

27 Ray Traced Shadws, 1spp Denised 27

28 Ray Traced Ambient Occlusin, 2spp Denised 28

29 Ray Traced Reflectins, 1spp Denised 29

30 RTX Dem Vide fr slides 31-70: 30

31 Ray Traced Area Light Shadws and Denising 31

32 Ray Traced Area Light Sft Shadw Why Ray Traced Shadws? Mre physically crrect sft shadw visuals Higher quality cntact hardening even fr really large light surce Nt pssible with shadw mapping based algrithms Mre accurate gemetry than character capsule shadws Supprts nn-rigid mtin (skinning, etc) unlike Distance Field Shadws Can be cmbined with analytical area lighting Eliminate cmmn shadw mapping artifacts: Sampling rate mismatch, shadw acne, peter panning, CSM seams, etc 32

33 Sft Shadws with 1spp 34

34 Sft Shadws 1spp Denised 35

35 Sft Shadws Grund Truth 36

36 Shadw Mapping 37

37 Area Light Sft Shadw Denising Deniser Overview Denising nly applied t visibility term (split sum apprximatin) L ω = δ f ω,ω i L d ω i V(ω i ) ω i n dω i δ V(ω i )dω i δ f ω, ω i L d ω i ω i n dω i Denised! Analytical Apprx. Per light surce denising Inputs: Hit distance, Scene depth, Nrmal, Light size and directin Separated crss bilateral filter with variable filter radius and weights Different denisers fr radial lights, directinal lights and rectangular lights Output: filtered visibility term 38

38 Ray Traced Ambient Occlusin and Denising 39

39 Ray Traced Ambient Occlusin Why Ray Tracing? Higher quality results cmpared with existing techniques Physically crrect ambient cclusin SSAO leaves dark hal arund depth discntinuities Screen Space techniques miss cclusin frm gemetry that is nt visible: Off screen bundaries Behind the camera Surfaces clse t parallel t the view vectr (e.g.: tables, chairs, very cmmn) 40

40 Screen Space AO 41

41 Ambient Occlusin 2spp Denised 42

42 Ambient Occlusin Grund Truth 43

43 Ambient Occlusin with 2spp 44

44 Denising Ray Traced Ambient Occlusin Separated crss bilateral filter Estimate filter kernel size in wrld space based n hit T Using ideas frm Axis-Aligned Filtering fr Interactive Physically-Based Diffuse Indirect Lighting [Mehta, el al. 13] Larger kernel size in pen regin, smaller in cntact regin Visibility changes slwer in pen regin, thus can share mre spatially Preserve cntact darkening Deniser Overview At 1spp, achieved smth far field cclusin With 2-4 spp, can recver detailed cclusin in cntact regin 45

45 Ray Traced Reflectins and Denising 46

46 Ray Traced Reflectins + Denising Screen Space Ray Traced Reflectins vs. Screen Space Reflectins Missing data due t ff screen ray hits Incrrect specular n reflectins Specular f primary shading pint being reused was cmputed with view directin instead f reflected ray directin 47

47 Screen Space Reflectins 48

48 Ray Traced Reflectins 49

49 Ray Traced Reflectins + Denising vs. pre-integrated light prbes Pre-integrated light prbes / envirnment maps Usually static. Dynamic is rarely practical Can be very wrng if nt placed crrectly Rughness range discretized int fixed number f levels Pre-integratin with view vectr perpendicular t receiver plane misses NDF anistrpy 50

50 Ray Traced Reflectins + Denising Planar reflectins vs. Planar Reflectins Only wrks fr planar surfaces Des nt scale t many f these N crrect glssy reflectins 51

51 Glssy Reflectins with 1spp. Rughness =

52 Glssy Reflectins 1spp Denised 53

53 Glssy Reflectins Grund Truth 54

54 Stchastic Screen Space Reflectins + Prbe 55

55 Glssy Reflectins Denising Deniser verview Denising applied nly t the incming radiance L ω = δ L ω i f ω, ω i ω i n dω i δ L ω i dω i δ f ω, ω i ω i n dω i Denised! Pre-Integrated Specular albed included in pre-integratin term s wn t be verblurred Denising nly the radiance term, n nise frm BRDF sampling Crss bilateral filter with anis kernel ftprint estimated based n prjecting the BRDF lbe ft print back t screen space 56

56 Limitatins f current denising technlgy Shadws deniser results can be f lwer quality fr verlapping penumbra frm tw ccluders with very different distances frm the receiver Reflectins deniser is further frm grund truth as rughness increases (e.g.: squared rughness > 0.4) AO deniser may need at 2 r mre rays t capture finer details Expect t cntinue imprving ver time t match mre clsely grund truth with better perfrmance 57

57 Future Wrk Deep learning based denisers Using same input as these hand-crafted denisers Expect this culd be the best slutin Denising all lighting cmpnent tgether t save time and memry Hybrid appraches cmbining screen space with wrld space tracing 58

58 Reference In-Engine Path Tracer 59

59 Path Traced Reference Image Next Gen Rendering Wrk Flw Pwerful tl t imprve cntent creatin pipeline Render path traced image as target fr tuning real-time rendering techniques Material tuning, light prbe placement, lighting setup, verall brightness Extremely useful fr rendering prgrammers t Tuning and validating the lk f real-time rendering algrithms Adjusting shading f area lights Tuning denisers Knw what yu are missing in yur hacks 60

60 In-Engine Path Tracing Easier tday than it has ever been Pwerful prgramming mdel in DXR abstracts all the cmplexity, making it easy t write light transprt algrithms. RTX prvides ptimized implementatins f Acceleratin Structure build/update, as well as traversal and scheduling f ray tracing and shading Ability t use HLSL means yu can reuse much f yur existing shading cde Physically based shading cmmnly used in game engines tday makes building path tracer in game engine easier than it used t be 61

61 In-Engine Path Tracing Experimental Path Tracer written by us as sn as DXR was integrated and wrking with ray traced reflectins. Nt prductin quality at this pint What we did in UE4 Separate ray generatin shader runs path tracing lp Simplified Pixel Shader that nly evaluates the material graph, but n lighting Output returned t path tracing ray gen shader via paylad Engine-generated material shaders can be used directly (except where using ddx/ddy) 62

62 In-Engine Path Tracing What we did in UE4 BRDF imprtance sampling n tp f UE4 s shading mdel Light surce sampling fr all light types t d NEE Analytical ray-light intersectin t supprt MIS initially Imprtance-sampled SkyLight cubemap Light culling when sampling frm many light surces 63

63 DL Based Path Tracing Denising NVIDIA OptiX 5.0 AI Deniser Integrated NVIDIA OptiX AI Deniser, recently released in 5.0 Imprves quality f the rendered image frm 0.8 ssim t 0.99 ssim (with respect t target reference image) Fast n Vlta GPUs thanks t Tensr Cres Blend smthly between nisy image and denised image starting at sme set number f iteratins (depending n scene) 64

64 In-Engine Path Tracing + AI Denising 65

65 Light Baking 66

66 In-Engine Light Baking Leverages a lt f the same cde written fr a reference path tracer Different types f light baking 2D Lightmaps vlumetric lightmaps envirnment capture cubemaps) Different mdes f baking Preview mde Batch mde 67

67 Lightmap Baking Preview Mde Fcus n updating nly the lightmap texels cntributing t the current image. Enables quick iteratin fr artists & lighters. Adjust lights, mve bjects and see results instantly. Our experience: Tried apprach that launches path tracing wrk in lightmap space In the end fund it easier t get gd results by path tracing in screen space while accumulating utput int lightmap texels using NvInterlckedAddFp32 (via NVAPI) Prgressive lightmap cmputatin, cmbined with smart tempral reuse and lightmap space denising allws near real-time updates 68

68 Lightmap Baking Batch Mde Utilize ne r mre GPUs t bake all the lightmaps in a level until cnverged Can dispatch a ray generatin shader 2D grid per lightmap texture, r per lightmap atlas Tw ways t get t bject space data at a lightmap texel: Traditinal: cnservative rasterizatin f interplated vertex attributes in lightmap space New: ray tracing an Acceleratin Structure built ver the 2D lightmap UVs Each has prs and cns. New apprach is cnvenient but lses benefits f cnservative raster 69

69 Instant Lightmap Baking Preview 70

70 Game Engine Ray Tracing Integratin 71

71 Game Engine Integratin High Level Plan Extend Graphics API abstractin with RT shader types and RT cmmands Build and Update Acceleratin Structures Create shaders and create the Ray Tracing Pipeline State Object Update Shader Parameters Start experimenting with ray tracing techniques 72

72 Game Engine Integratin Build and Update Acceleratin Structures Build Acceleratin Structure fr static gemetry nce Rebuild Tp Level Acceleratin Structure every frame Update Bttm Level Acceleratin Structure fr defrming gemetry every frame Use Cmpute shader t prcess defrmable gemetry int a temprary buffer t use as input t Acceleratin Structure Update and shaders 73

73 Game Engine Integratin Shaders and Shader Parameters Use dxcmpiler utility functin t cnvert VS+PS pair t Hit shaders Supprts generatin f bth Clsest Hit and Any Hit shader (alpha-test nly) Register shaders and create the Ray Tracing Pipeline State Object Update shader parameters: Naïve apprach: update shader parameters every frame fr every bject Optimal: update nly the shader parameters that changed 74

74 Game Engine Integratin Start Experimenting, then Optimize Replace existing deferred passes fr shadws, AO and reflectins with ray generatin shaders + denising Add new techniques such as path tracing and lightmap baking Sme ptimizatin tips: Allw artists t create simplified materials fr reflectin shading Build a single bttm level Acceleratin Structure fr an bject with multiple subbjects with different materials verlapping spatially Use RAY_FLAG_ACCEPT_FIRST_HIT_AND_END_SEARCH fr shadw and AO rays even if yu need the hit distance fr denising purpses 75

75 Game Engine Integratin Challenges and slutins Decals: early prttype utside engine shwed prmise Tessellatin: start by disabling it if pssible, therwise limit update rate, utput tessellated gemetry t buffers and prvide as input t AS builder Texture LOD: tried appraches such as replicating part f the Pixel Shader cde t cmpute UVs at tw auxiliary pints t be able t cmpute derivatives Des nt seem wrth it in mst scenes, unless tracing primary rays r flat mirrr rays and texture has high frequencies TAA can eliminate a lt f the aliasing 76

76 Wrapping Up 77

77 Takeways RTX and DXR brings real-time ray tracing t develpers Biggest change in Graphics APIs since prgrammable shaders A few rays per pixel practical tday fr lighter weight effects Mre expensive nes (path tracing) can prvide useful wrkflw imprvements We shwed a few examples f what s pssible Expect yu t cme up with bright new ideas Hpe this was inspiring and exciting 78

78 Thanks & Acknwledgements NVIDIA: Alex Bgmjakv, Evan Hart, Matthias Hllaender, Matthew Jhnsn, Martin-Karl Lefrancis, Adam Marrs, Laurent Ruhlmann, Jacp Pantaleni, Fredrik Liljegren, Cem Cebenyan, Martin Stich, Steve Parker, Tny Tamasi, Jn Stry, Luis Bavil, Iain Cantlay, Nun Subtil, Aarn Lefhn, Jn Hasselgren, Jacb Munkberg, Chris Wyman, Marc Salvi, Alex Keller, Carsten Wachter, Magnus Anderssn, Rbert Tth, Pascal Gautrn, Shivan Taher, the rest f the SW team wh made RTX pssible Epic Games: Marcus Wassmer, Brian Karis, Patrick Kelly, Juan Canada, Arne Schber, Uriel Dyn, Yuriy O Dnnell, Jerme Platteaux, Kim Libreri and the rest f the team wh wrked n the Reflectins dem 79

79 Thank yu! 80

80 Ray Tracing Gems A new bk series with fcus n real-time and interactive ray tracing fr game develpment using the DXR API. We invite articles n the fllwing tpics: Basic ray tracing algrithms, effects (shadws, reflectins, etc), nn-graphics applicatins, recnstructin, denising, & filtering, efficiency and best practices, baking & preprcessing, ray tracing API & design, rasterizatin and ray tracing, glbal Illuminatin, BRDFs, VR, deep learning, etc. Imprtant dates Call fr Papers 15th f Octber 2018: submissin deadline fr full papers GDC 2019: publicatin f Ray Tracing Gems (paper versin + e-bk) Eric Haines and Tmas Akenine-Möller will lead the editrial team 81

81 Questins? Ignaci Llamas, illamas AT nvidia.cm Edward Liu, edliu AT nvidia.cm Bth #223 - Suth Hall