ACCELEWARE FDTD PERFORMANCE GUIDE

Transcription

1 Eight easy ways to speed up your simulation - January 21 Logan Maxwell, Mike Weldon

2 Copyright Notice All material herein is Acceleware copyright and shall not be reproduced, copied, forwarded, published or shared in any matter without prior written authorization from Acceleware. All rights reserved. Acceleware, the Acceleware logo and wordmark are registered trademarks and /or trademarks of Acceleware Corp. in the United States, Canada and other countries. All other trademarks are the property of their respective owners. Page 2 of 13

3 Table of Contents Overview 4 The Fundamentals of FDTD Performance on GPUs 5 1) Perfectly Matched Layers (PML) 6 2) Reads and Read Regions (Observations, DFT, Convergence, etc.) 7 3) Screen Savers 8 4) Simulation Orientation 9 5) Number of Materials 1 6) Dispersive Materials 11 7) Mesh Density 12 8) Windows Remote Log In 13 Page 3 of 13

4 Overview Introduction Fastest-possible FDTD simulation performance on GPU and multi-core hardware is a key objective for partners and end users alike. The hardware and Acceleware software libraries that make it run are obviously key determinants of the ultimate performance, but partners and end users can still have a large impact on the performance in ways that are not always obvious. This document outlines several key simulation parameters that impact simulation performance. Each case includes a brief description of the parameter, a plot illustrating the performance impact and tips on how to minimize any speed reduction. Improper use of these parameters whether intended or not, can reduce simulation speed by 5% or more. Understanding the suggestions in this document will help you avoid unnecessary reductions in performance and get the most out of all your simulations. Intended Audience Acceleware partners, end users and all those that are interested in running FDTD simulations optimally on GPUs and multi-core hardware. This document should be considered essential reading for engineers and scientists running FDTD simulations and will help make sure that they are always getting the most out of their simulation tools. Page 4 of 13

5 Performance (Mcell/s) The Fundamentals of FDTD Performance on GPUs 8 (Driver ) Knee Simulation: Cubic, 16 Lossy Dielectrics, 4 Layer PML Memory Limit GPU CPU Ramp Up Optimal Range Soft Memory Simulation Size (Mcell) Ramp Up: Knee: Optimal Range: GPU Memory Limit: Soft Memory: In this range the GPU is not using all of its compute resources and memory bandwidth efficiently. Secondly, PML takes up a large portion of the total simulation size and acts to slow the total simulation throughput. The knee is the point at which the performance levels off and the GPU is running optimally. This is the optimal range because the GPU has found a good balance between computation and communication. The goal of any GPU FDTD code is to maximize this region s breadth and magnitude. This is the point at which the GPU runs out of memory and CPU begins to solve the remaining calculations. In this area the CPU is solving the remaining calculations that the GPU does not have memory for. As simulation size goes further into soft memory, the closer the performance will get to 1% CPU performance. How to calculate throughput performance: Note: Simulation Size is not including PML cells Page 5 of 13

6 Performance (Mcells/s) Perfectly Matched Layers (PML) Adding PML (absorbing) boundary layers can reduce simulation performance by as much as 5% which would double run the time. The maximum simulation size the GPU is capable of running will also be partially reduced. PML cells require more memory than non-pml cells. That reduces simulation size. They also are more expensive to compute, which reduces performance. More significantly, we don't include PML cells when calculating capacity or speed. Small simulations are impacted more than larger simulations because PML cells represent the majority of the computational load. 8 (Driver ) PML Performance Simulation: Cubic, 16 Lossy Dielectrics 6 PML Layers Simulation Size (Mcells) - Minimize the number of layers of PML. - Understand how much PML your simulation requires and use no more than that. - Use maximum PML layers only when absolutely necessary. Page 6 of 13

7 Performance (MCells/s) Reads and Read Regions (Observations, DFT, Convergence, etc.) Reading field data during a simulation can dramatically impact performance. Field data is read when observing simulation output, convergence, for DFTs etc. How much of the volume is read and how frequently the volume is read both impact simulation performance. The chart below shows performance for different volumes of reads based on a percentage of the total volume. All six fields are read for each cell. We are sweeping the number of time steps between each read. Read Performance 5 (Driver ) Observations: Ex, Ey, Ez, Hx, Hy, Hz Simulation: 3 Mcells,Cubic, 16 Lossy Dielectrics, 4 Layer PML 4 % of Volume Read 3 2 % 25% 5% 75% 1% All Fields Read Every X Time Steps - Keep the read volume to a minimum; only observe the region (volume) that is of direct interest. - Read only as frequently as is necessary to achieve accurate power, DFT, SAR, optical generation, etc. results. - For optical generation, far field etc. only start to read after a simulation has converged. Page 7 of 13

8 Performance (Mcells/s) Screen Savers Screen savers, especially graphics intensive 3D types can decrease the performance of the simulation. Performance difference between no screen saver and basic screen saver is negligible. Smaller simulations experienced a greater percent decrease in performance. Occasionally, significantly worse performance is observed, and is abnormal. 8 Screen Saver Performance (Driver ) Simulation: Cubic, 16 Lossy Dielectrics, 4 Layer PML 6 Screen Saver 4 None Blank 3D Pipes Simulation Size (Mcells) - Use low detail screen savers, blank, or no screen saver. - Use the management settings to turn off the monitor instead of using a screen saver. - If you must use a screen saver, confirm your performance is not degraded by more than 1-2%. If it is worse, please contact Acceleware Page 8 of 13

9 Performance (Mcells/s) Simulation Orientation Single-GPU simulations where Z is the smallest dimension by a significant margin will experience a decrease in simulation performance and maximum simulation size. This is due to the way in which memory is allocated, this problem is not unique to GPU FDTD solutions - it is also present in CPU-only FDTD solvers. The example below shows an extreme case of smallest dimension, for less extreme cases the decrease in performance and max simulation size is smaller. Partitioning across multiple GPUs will change the effective simulation dimensions on each GPU, and hence the performance. Smallest dimension in the graph is 1% of the other dimensions. x y z a b b x smallest (a, b, b) y smallest (b, a, b) z smallest (b, b, a) 8 (Driver ) Smallest Performance Simulation: 16 Lossy Dielectrics, 4 Layer PML 6 4 Orientation: Cubic X Smallest Y Smallest Z Smallest Simulation Size (Mcells) - Rotate the simulation so that the Z is not the smallest dimension - Avoid extreme differences in dimension, cubic shows the best performance Page 9 of 13

10 Performance (Mcells/s) Number of Materials The number of materials can have a large impact on performance, up to a 2% decrease. The type of material can also have an effect on performance. For simulations with a variety of both E and H materials, the performance drop is more severe. 6 Number of Materials Performance (Driver ) Simulation: 3 Mcells. Cubic, 4 Layer PML E Materials H Materials E and H Materials Unique Materials (#) - If possible keep number of materials below Make sure that all the materials are necessary; some applications add arbitrary complexity by continually varying the number of materials. Page 1 of 13

11 Performance (Mcells/s) Dispersive Materials Dispersive materials have a large impact on simulation performance and maximum simulation size. Both the order (number of poles) of the dispersive materials and the total number of materials present will decrease performance. Higher order dispersive materials show worse performance, and higher numbers of dispersive materials will decrease performance. This applies to all dispersive materials types, Drude, Debye, Lorentz, Drude-Lorentz, etc. 7 Dispersive Performance (Driver ) Simulation: Cubic,, 4 Layer PML, Case 1 Case 2 Case 3 Case 4 Case Simulation Size (Mcells) Case 1 16 non-dispersive materials distributed evenly thought the entire simulation space Case 2 1 single-pole dispersive material occupies 4% of the total volume contiguously. Case 3 1 single-pole dispersive distributed evenly throughout the entire volume, 4% of the total volume is made up of dispersive materials. Case 4 16 Multi-pole dispersive materials distributed contiguously throughout 4% of the total volume. Case 5 16 Multi-pole dispersive materials distributed evenly throughout the entire volume, 4% of the total volume is made up of dispersive materials. - Restrict the total volume of dispersive materials in any simulation. - Use the minimum number of dispersive materials and volume to achieve desired result. Page 11 of 13

12 Simulation Time (h:mm:ss) Mesh Density Increasing the mesh density does not always yield more accurate results, however increasing the mesh density will always increase run time. This is for two reasons, one because there are more cells to compute, and two, because t in the simulation must also decrease to maintain simulation stability which increases the number of time steps required for a given number of periods. The chart below demonstrates the naive linear and actual effect of increasing mesh density on run time with a 1 Mcell simulation. Time to complete 6 periods :2: (Driver ) Simulation: Cubic, 16 Lossy Dielectrics, 4 Layer PML :16: :12: :8: Actual Naïve Linear :4: :: Simulation Size (Mcells) - Only increase mesh density if your simulation accuracy requires it. Page 12 of 13

13 Performance (Mcells/s) Windows Remote Log In Remote desktop software can have a large impact on simulation performance; it can exceed a 5% reduction in speed, give an error or not run at all. This happens because the desktop is virtualized and in some cases access to the GPU is limited or nonexistent. The desktop uses GPU resources which are needed for computation. 8 (Driver ) Ultra VNC Performance Simulation: Cubic, 16 Lossy Dielectrics, 4 Layer PML 6 VNC OFF 4 VNC ON Simulation Size (Mcells/s) - Use a KVM as they have no impact on performance. - Do not use remote desktop tools in general - If absolutely necessary use Ultra VNC, which still has some performance decrease as shown above. Page 13 of 13