Unstructured Grid Numbering Schemes for GPU Coalescing Requirements

Transcription

1 Unstructured Grid Numbering Schemes for GPU Coalescing Requirements Andrew Corrigan 1 and Johann Dahm 2 Laboratories for Computational Physics and Fluid Dynamics Naval Research Laboratory 1 Department of Aerospace Engineering University of Michigan 2

2 Acknowledgements Rainald Löhner (George Mason University) K. Kailasanath Gopal Patnaik Junhui Liu Ravi Ramamurti Douglas Schwer David Kessler Work sponsored through ONR/NRL 6.1

3 Motivation: JENRE JENRE: Jet Engine Noise Reduction code Navy-developed simulation tool for jet engine noise prediction Requires accurate representation of complex nozzle geometry

4 Motivation: JENRE Computationally demanding Hundreds of millions of grid cells Long time duration Increasingly complex physics J. Liu, Numerical Investigation of Advanced Military Aircraft Noise Reduction Concepts DoD HPC Challenge Project for FY2012 Consumes millions of CPU hours per year

5 Motivation: JENRE Physics Euler equations Navier-Stokes equations Numerics Cell-centered finite volume Node-centered finite element Grid Unstructured (Tetrahedral, Hexahedral, Prism, Pyramid) Structured (Hexahedral) Cartesian (Hexahedral) Hybrid Parallelism Distributed memory parallelism via MPI. Shared memory parallelism via Thrust. Single codebase Extensive use of generic programming via C++ templates. Recompile for particular physics, numerics, grid, computational architecture. Individual components can be specialized as required (e.g., grid numbering).

6 Unstructured Grids Accurately represent complex geometry Provide precise control over grid spacing Grid entities can be arbitrarily numbered Lead to an indirect, scattered memory access pattern Potentially lead to a significant reduction of performance

7 Memory Access Pattern Gather-scatter memory access pattern Edge/face-based solvers. Common non-trivial memory access pattern in CFD and other areas. Reflects physical symmetry of conservation laws at discrete level. Sparse solvers are often implemented directly in terms of grid connectivity. For each edge: 1. Gather from points. 2. Compute on edge. 3. Scatter back to points. Edges must be colored for parallel execution

8 Numbering Reorder points to improve memory access pattern as edges are traversed Should be tailored to the unique requirements of each computational architecture 0 1 2

9 Bandwidth Minimization Intended for minimizing CPU cache misses as edges are traversed Examples: (RCM) Reverse Cuthill- McKee, Space-filling curve, Wavefront, Strives to keep data close in physical space close in memory space

10 Coalescing Memory transactions are serviced on a persegment basis Data stored at {L,R} points are accessed simultaneously Adjacency in physical space does NOT imply adjacency in memory space

11 Unstructured Grid Numbering 1. Detect lines 2. Alternate numbering between points on each line 3. Color edges, first along, then between lines 4. Sort edges by its color and point index tuple

12 Detect Lines An external direction field is imposed User input parameter to algorithm. A coordinate direction often works very well. Detect lines along the edges most aligned with this direction Lines may merge or branch

13 Detect Lines 2D triangular grid Detected lines

14 Alternate Numbering Along Lines The numbering is alternated along each line Good coalescence for intra-line edges Requires lines of sufficient length

15 Color Edges Along Lines The first edge colors are constrained to edges along lines The edge groups within these edge colors will typically exhibit good coalescence

16 Color Edges Along Lines The first edge colors are constrained to edges along lines The edge groups within these edge colors will typically exhibit good coalescence

17 Color Edges Between Lines Edges between lines are colored between pairs of lines Avoids large jumps between lines as much as possible. Coalescence will be optimal when these edges are oriented consistently

18 Color Edges Between Lines Edges between lines are colored between pairs of lines Avoids large jumps between lines as much as possible. Coalescence will be optimal when these edges are oriented consistently

19 Color Edges Between Lines Edges between lines are colored between pairs of lines Avoids large jumps between lines as much as possible. Coalescence will be optimal when these edges are oriented consistently

20 Color Edges Between Lines Edges between lines are colored between pairs of lines Avoids large jumps between lines as much as possible. Coalescence will be optimal when these edges are oriented consistently

21 Number Edges Sort edges within each color group according to the point-numbering Consecutive points across edges are numbered with a unit stride

22 Benchmarks JENRE high-speed compressible flow solver Fully-coupled, finite element spatial discretization Flux-corrected transport limiting Explicit Taylor-Galerkin time-stepping Single Geforce GTX 580 Computational throughput measured To compute one time step (averaged over many) Tetrahedral cells / second Includes non-edge-loops FEFLO-GEN3D advancing front grid generator With Cartesian core option

23 Benchmark: Shock Tube 2.5M Tetrahedral cells Based on a fully structured grid. Bandwidth minimization numbering 27M Tetrahedral cells / second Line-based numbering 41M Tetrahedral cells / second

24 Benchmark: Shock Tube 1.9M Tetrahedral cells Advancing front + Cartesian core grid generation. Bandwidth minimization numbering 24M Tetrahedral cells / second Line-based numbering 35M Tetrahedral cells / second

25 Benchmark: NACA M Tetrahedral cells Bandwidth minimization 18M Tetrahedral cells / second Line-based numbering 21M Tetrahedral cells / second

26 Benchmark: Nozzle 5.8M Tetrahedral cells Bandwidth minimization 19M Tetrahedral cells / second Line-based numbering 28M Tetrahedral cells / second

27 Conclusions A significant performance improvement is possible if unstructured grid numbering schemes are tailored for GPU coalescing requirements. Bandwidth minimization numbering Is not directly relevant to achieving coalesced memory access. Does not achieve optimal coalescence in simple cases. Does provide reliable baseline computational performance. Line-based numbering Directly relevant to achieving coalesced memory access. Near-optimal coalescence in simple cases. Dependent on presence of long parallel lines in grid connectivity. Should not be an issue for highly-refined LES grids.