Scaling of 3D Game Engine Workloads on Modern Multi-GPU Systems. Jordi Roca Monfort (Universitat Politècnica Catalunya) Mark Grossman (Microsoft)

Transcription

1 Scaling of 3D Game Engine Workloads on Modern Multi-GPU Systems Jordi Roca Monfort (Universitat Politècnica Catalunya) Mark Grossman (Microsoft) 0

2 Outline Introduction on Multi-GPU rendering RTT surface synchronization alternatives Multi-GPU performance models Scaling results Conclusion 1

3 Multi-GPU rendering Main purpose: Several GPUs collaborate to render frames of the same 3D scene. Rendering task is massive parallel. High resolution scenes require a lot of pixel fillrate and memory BW. GPUs partial renders are composed to the final frame sequence. Other different usages: Multi-display rendering: each GPU renders a different viewport/screen. GPUs take different tasks: graphics rendering, physics and AI processing. 2

4 A choice for enthusiast gaming Performance/scaling: Usually high for few GPUs and high resolutions (1600x1200 minimum) Greatly depends on driver maturity and the game workload: Crysis Warhead hits 1.6x and Lost Planet hits 1.9x with 2-GPU systems. Power: Two graphics cards spend more than the equivalent single GPU solution targeting the same performance. Upgrade cost: Double performance for next game generation by acquiring a second graphics card (same GPU family and similar range counterpart). Extra cost of acquiring a high end motherboard with multiple PCIe ports. 3

5 Rendering workload balance Split Frame Rendering Alternate Frame Rendering SuperAA Dynamic split line Fixed tiles (32x32) Geometry scaling problem Decreased interactivity problem Play at High AA modes (x16) 4

6 What do GPUs communicate to each other? Today s 3D engines don t just render the main backbuffer: Draw commands can render to special surfaces used later as textures: reflections, shadow maps, lens flare, post-filtering ops,.. New draw dependency chain with render-to-texture surfaces as synchronization points. GPUs must exchange updated surface contents at this points to ensure data integrity Inter-GPU syncs. 5

7 Render-to-Use sync analysis RTT copies diverge Start sync operation Sync cycles RTT copies are synchronized Frame 0 Swap frame RTT copies diverge Frame 1 SFR GPU0 GPU1 Render[0] Use[0] Repeat Use[0] Use[1] Render[1] RtU intraframe RtU interframe dep AFR GPU0 GPU1 RTT copies diverge Render[0] RtU intraframe dep RtU interframe dep Use[1] Use[0] RTT copies are synchronized Sync cycles Repeat Use[1] Frame 0 Swap frame RTT copies diverge Render[1] Frame 1 Command Buffer events 6

8 Render-to-Use sync analysis Game/Timedemo Engine Release date Screen resolution Frames RTT surfaces % RTT time 3DMark06/Canyon Flight Proprietary 2006/01 16 x 12 (1AA) % FEAR/PerformanceTest LithTech 2005/10 16 x 12 (1AA) % Call Of Duty 2/carentan Proprietary 2005/10 16 x 12 (1AA) % Call Of Duty 2/demo5 Proprietary 2005/10 16 x 12 (1AA) % Company Of Heroes/Intro Essence 2006/09 16 x 12 (1AA) % Half Life 2 Lost Coast/VST Source 2005/10 25 x 16 (8AA) % BattleField 2142/suez canal Refractor2 2006/10 25 x 16 (1AA) % BattleField 2/abl-chini Refractor2 2005/06 16 x 12 (1AA) % 7

9 Render-to-Use sync analysis RTT Surface Id Intraframe (SFR) Interframe (AFR) RtU deps syncs RtU deps syncs 2-GPU Game/Timedemo 306 Engine Release Screen 7405 Frames 0 RTT % RTT0 307 date35722resolution surfaces 0 time 0 3DMark06/Canyon Flight 308Proprietary 2006/ x (1AA) % 0 FEAR/PerformanceTest 309LithTech 2005/ x (1AA) % 0 Call Of Duty 2/carentan 30aProprietary 2005/ x 12 (1AA) % 0 Call Of Duty 2/demo5 30bProprietary 2005/ x 12 (1AA) % 0 Company Of Heroes/Intro 30dEssence 2006/ x (1AA) % 9 Half Life 2 Lost Coast/VST30eSource 2005/ x (8AA) % 0 BattleField 2142/suez canal Refractor2 2006/10 25 x 16 (1AA) % BattleField 2/abl-chini Refractor2 2005/06 16 x 12 (1AA) % Track per-surface RtU dependencies and required syncs. The more intraframe syncs, the worse SFR performs. The more interframe syncs, the worse AFR performs. 8

10 The contributions of this work Which characteristics make 3D Games suitable for multi-gpu systems? Render-to-texture sync analysis. Do they enable any optimization? (See next section) Can we measure multi-gpu scaling based on 3D game workload characteristics? Using a simplified model. Using real 3D workload data. Evaluate SFR, AFR and combined modes (4+ GPUs). 9

11 RTT surface synchronization alternatives

12 Leverage RtU gap: Early Copy Render Render Draw Draw Use GPU0 GPU1 D D D D Determine last render: Use Prediction table, Look ahead command buffer. Early Start of sync operation Swap frame Game/ Timedemo syncs RtU gap wrt frame duration RtU sync Gap % Pixel shading bound 3DM % 52.85% FEAR % 65.08% COD2c % 96.28% COD2d % 94.63% COH % 96.80% HL % 21.95% BF % 87.16% BF % 44.44% Penalization cost = % of time the RTT draw is pixel shading bound Sync cycles (delayed copy) RtU gap cycles 11

13 Pixel Shading bounds: Concurrent Update Pixel Shader R600 Shader: Clock: 750MHz ALUs: 4 x 16 SIMD Pixel Shader Pixel Shader Shader output PCIe 2.0 x16 6GB/sec Shader output Shader output ROP ROP ROP Local Mem Local Mem Local Mem Local GPU Remote GPUs Shading a thousand fragments (35 instructions) in the R600 SPUs takes 1000 x 35 / 64 = 547 cycles. Sending the corresponding color outputs through the PCIe 2.0 bus takes 1000 x 4 bytes x 750 MHz / 6 GB/s = 500 cycles Penalization cost = Remote sent cycles Pixel shading cycles 12

14 Multi-GPU Performance Models

15 EMPATHY analysis tool API state changes Vtx HW counters (N/I) GPUs in SFR mode /(N/I) Game execution in real Hw Downscaled Pixel counters Real app fps (FRAPS) Correlation (for a GPU arch. File describing the same real hardware) Estimated exec time (Single GPU) GPU arch. description file: R600 API state changes ` execution trace Pixel HW counters Collect data Propietary Analytical tool Vtx HW counters Single GPU arch. description file: R600 GPU Interconnection BW I GPU clusters in AFR mode Propietary Analytical tool Min SFR scaled exec time Add Inter-GPU sync cost (SFR) EMPATHY /I Add Inter-GPU sync cost (AFR) N: Total GPUs I: AFR interleaving Multi-GPU exec time 14

16 Multi-GPU interconnection network CPU PCIe x16 Link CPU PCIe x16 Link FSB FSB SouthBridge NorthBridge MemBus DDR SouthBridge NorthBridge MemBus DDR GPU 0 GPU 1 GPU 0 PCIe bridge GPU 1 GPU 2 PCIe bridge GPU 3 GDDR FB Board 0 GDDR Board 1 FB Display Link GDDR FB Board 0 GDDR FB GDDR FB Board 1 GDDR FB Display Link SFR sync: all-to-all GPU simulatenous transfers. The NorthBridge PCIe ports become the bottleneck. Each GPU sees reduced peak BW as function of the number of GPUs: 15

17 Performance scaling models SFR GPU0 GPU1 Penalization cycles Frame 0 Frame 1 Frame 2 Total estimated multigpu cycles AFR GPU0 Frame 0 GPU1 GPU0 ½ frame initialization interval Frame 1 R S sync cycles U Penalization cycles frame 2 Frame 2 Total estimated multigpu cycles 16

18 Cycles Early copy + concurrent cost (SFR) Penalization cycles Penalization cycles comparison BF2 - Surface 31d delayed copy early copy concurrent update 0 Frames Choose per surface the best sync alternative each frame that incurs in the minimum penalization cycles. 17

19 Scaling Results

20 SFR scaling (early copy + concurrent update) % 50% 40% 30% 20% 10% 0% SFR performance gain for early + concurrent (2 GPUs) delayed copy early copy early + concurrent 3DM06 FEAR COD2c COD2d COH HL2 BF2142 BF2 Avg Avg performance speed-up gain normalized to delayed copy early copy early + concurrent 3DM06 FEAR COD2c COD2d COH HL2 BF2142 BF2 Game syncs RtU sync Gap % Pixel shading bound 3DM % 52.85% FEAR % 65.08% COD2c % 96.28% COD2d % 94.63% COH % 96.80% HL % 21.95% BF % 87.16% BF % 44.44% 19

21 % perfect scaling Combined SFR/AFR modes scaling results Avg multi-gpu efficiency wrt Perfect Scaling (100%) 100% 80% 60% 40% 20% 0% 2-GPUs 3-GPUs 4-GPUs 6-GPUs 8-GPUs 12-GPUs 16-GPUs Pure SFR SFR>AFR SFR = AFR SFR < AFR Pure AFR SFR scaling was tested using Early Copy + Concurrent Update optimization. Low interframe syncs benefits mostly AFR configurations. Game Syncs i2 Interframe Syncs i4 3DM FEAR 0 0 COD2c 0 0 COD2d 0 0 Game Syncs i2 Interframe Syncs i4 COH HL BF BF

22 Conclusion

23 Conclusion Inter-GPU synchronization requirements of render-to-texture surfaces in 3D games impact multi-gpu performance/scaling. This work has evaluated the potential benefits of two proposed sync alternatives based on RTT update anticipation for a set of popular DX9 games. Leverage of the RtU gap and the pixel shading cost increases SFR scaling. This work has shown a simple multi-gpu performance analytic model based on real 3D game execution data, that allows to evaluate SFR, AFR and combined rendering modes. Observed low interframe syncs benefit mostly AFR configurations. 22

24 Thank you!