Case Study - Computational Fluid Dynamics (CFD) using Graphics Processing Units

Transcription

1 - Computational Fluid Dynamics (CFD) using Graphics Processing Units Aaron F. Shinn Mechanical Science and Engineering Dept., UIUC Summer School 2009: Many-Core Processors for Science and Engineering Applications, A.F. Shinn CFD using GPUs 1 / 30

2 What is CFD? Introduction Computational Fluid Dynamics: solve governing equations of fluid motion numerically - Conservation of Mass (Continuity Equation) - Conservation of Momentum (Newton s 2nd Law) - Conservation of Energy (1st Law of Thermodynamics) Coupled set of nonlinear Partial Differential Equations (PDEs) Solution time can be very long makes GPUs very attractive A.F. Shinn CFD using GPUs 2 / 30

3 General Governing Equations Conservation of Mass Conservation of Momentum Conservation of Energy ρc p DT Dt ρ t + ρu = 0 ρ Du Dt = p + τ = βt Dp Dt + (k T ) + Φ viscous stress tensor: τ = µ ( ui x j + u j x i ) + δ ij λ( u) substantial derivative: D( ) Dt = ( ) t + u ( ) A.F. Shinn CFD using GPUs 3 / 30

4 of Illustrate CFD implementation issues with real research example CU-FLOW: general-purpose Cartesian-based 3D Navier-Stokes solver written in C/CUDA for GPUs First implementation of fractional-step/multigrid Navier-Stokes solver for Large-Eddy Simulations (LES) of turbulence on GPUs Many different variations of this code were created Countless hours spent on algorithm design, optimizations, and debugging! A.F. Shinn CFD using GPUs 4 / 30

5 Governing Equations for this Study 3D Incompressible Navier-Stokes equations Conservation of Mass Conservation of Momentum u = 0 u t + u u = 1 ρ p + ν 2 u A.F. Shinn CFD using GPUs 5 / 30

6 Numerical Methodology Discretized via Finite-Volume Method on a staggered Cartesian mesh. Smagorinsky SGS model used for turbulence modeling. Solved equations with fractional-step procedure. - Pressure-Poisson equation (PPE) solved using red-black Gauss-Seidel. - Geometric multigrid used for convergence acceleration of PPE solution. - Temporal advancement: explicit 2nd-order Adams-Bashforth scheme. - Spatial derivatives: 2nd-order central differencing. A.F. Shinn CFD using GPUs 6 / 30

7 Geometric Multigrid: V-cycle Figure: Multigrid V-cycle, where S=smooth, R=restrict residual, P=prolongate. Only three mesh levels are shown for simplicity. A.F. Shinn CFD using GPUs 7 / 30

8 Multigrid: How good is it? Consider a unit square 2D domain, solve Laplace equation 2 φ = 0 on that domain Multigrid converges in just a few iterations, whereas using a single grid takes thousands! Figure: Residuals of multigrid and single grid for solution of the Laplace equation on a 256x256 grid, tolerance = A.F. Shinn CFD using GPUs 8 / 30

9 Layout of CU-FLOW code Preprocessing on CPU set I.C. and B.C. generate mesh copy data to GPU Time-stepping loop controlled on CPU for(n=1; n<=nsteps; n++) { Processing solution on GPU (call kernels) advance velocity from u n to u (Adams-Bashforth) advance p n to p n+1 (Multigrid V-cycle) advance u to u n+1 } // end time-stepping loop Postprocessing on CPU copy data from GPU write plot files A.F. Shinn CFD using GPUs 9 / 30

10 Mapping between threads and cells A.F. Shinn CFD using GPUs 10 / 30

11 Multithreading Multigrid Optimal block size may conflict with mesh level dimensions. Example: would like a 4x4x4 mesh as coarsest level, but 32x1x8 is optimal block size. Cannot map one-to-one due to dimensions of block exceeding mesh. Question: how to resolve this conflict? Possible solution: set block size based on mesh level. A.F. Shinn CFD using GPUs 11 / 30

12 Multithreading Multigrid Host code for calling a kernel // define *fine mesh* dimensions of the blocks #define bx_f 32 #define by_f 1 #define bz_f 8 // define *coarse mesh* dimensions of the blocks #define bx_c 4 #define by_c 4 #define bz_c 4... for( n = 1; n<=ngrid; n++) { // use block size for coarse mesh by default bx = bx_c; by = by_c; bz = bz_c; // for finer meshes, use better block size if ( nx[n]%bx_f == 0 && ny[n]%by_f == 0 ) { bx = bx_f; by = by_f; bz = bz_f; } }... dim3 block(bx,by,bz); dim3 grid(nx[n]/bx,ny[n]/by); kernel<<<grid, block>>>(..., n,...); A.F. Shinn CFD using GPUs 12 / 30

13 Multithreading Multigrid Device code for kernel global void kernel(..., n,...) { // i = tx + 2, j = ty + 2 (offset thread indices to mesh indices) i = threadidx.x + blockidx.x * blockdim.x + 2; j = threadidx.y + blockidx.y * blockdim.y + 2; for (slice=0; slice<=nz[n]/blockdim.z-1; slice++) { k = threadidx.z + slice * blockdim.z + 2; m = i + (j-1)*(nx[n]+2) + \ (k-1)*(nx[n]+2)*(ny[n]+2) + begin[n] - 1; } }... kernel computations... A.F. Shinn CFD using GPUs 13 / 30

14 CUDA implementation of Red-Black Gauss-Seidel Color the grid like a checkerboard to enable parallel processing of pressure First update the red pressures, then update the black pressures Figure: 2D example of red-black coloring of a mesh A.F. Shinn CFD using GPUs 14 / 30

15 CUDA implementation of Red-Black Gauss-Seidel Updating pressure: host code for( icyc = 1; icyc<=ncyc; icyc++) { // go through all V-cycles for( n = ngrid; n>=1; n--) { // downleg of V-cycle // use block size for coarse mesh by default bx = bx_c; by = by_c; bz = bz_c; // for finer meshes, use better block size if ( nx[n]%bx_f == 0 && ny[n]%by_f == 0 ) {bx = bx_f; by = by_f; bz = bz_f;} dim3 block(bx,by,bz); dim3 grid(nx[n]/bx,ny[n]/by); for( iswp = 1; iswp<=nswp; iswp++) { red_kernel<<<grid, block>>>(..., n,...); black_kernel<<<grid, block>>>(..., n,...); }... A.F. Shinn CFD using GPUs 15 / 30

16 CUDA implementation of Red-Black Gauss-Seidel red kernel: device code global void red_kernel(... ) { i = threadidx.x + blockidx.x * blockdim.x + 2; j = threadidx.y + blockidx.y * blockdim.y + 2; for (slice=0; slice<=nz_d[n]/blockdim.z-1; slice++) { k = threadidx.z + slab * blockdim.z + 2; if( (i+j+k)%2==0 ) { // test if red cell m = i + (j-1)*(nx[n]+2)+(k-1)*(nx[n]+2)*(ny[n]+2)+begin[n]-1; xm = xm[m]; xp = xp[m]; ym = ym[m]; yp = yp[m]; zm = zm[m]; zp = zp[m]; res = (aw_d[m] * pressure_d[xm] + ae_d[m] * pressure_d[xp] + \ as_d[m] * pressure_d[ym] + an_d[m] * pressure_d[yp] + \ al_d[m] * pressure_d[zm] + ah_d[m] * pressure_d[zp] + \ resc_d[m]) / ap_d[m]; pressure_d[m] = relxp*(res) + (1.0-relxp)*pressure_d[m]; } // end if } //end slice } //end kernel A.F. Shinn CFD using GPUs 16 / 30

17 Profiling of CU-FLOW Red-black Gauss-Seidel kernels consume over 2/3 of GPU time! Must optimize red-black Gauss-Seidel kernels A.F. Shinn CFD using GPUs 17 / 30

18 CUDA implementation of Red-Black Gauss-Seidel Memory management in red-black kernels - Global memory: easiest, but slow - Shared memory: gives marginally better performance, perhaps due to low data reuse or handling of boundary halos for each sub-domain in shared memory. - Texture memory: fetch device memory through textures instead of expensive global memory load. Currently working on this. This is an alternative to avoid uncoalesed memory loads. A.F. Shinn CFD using GPUs 18 / 30

19 Computational Resources GPU verison: CUDA, CPU version: Fortran. Single-precision used for all calculations. Dell Precision 690 Workstation (Linux: Red Hat Enterprise 5) CPU: 3.0 GHz Intel Xeon GPU: NVIDIA Tesla C1060 ( 1 teraflop) A.F. Shinn CFD using GPUs 19 / 30

20 Laminar Flow in 3D Lid-Driven Cube Figure: Computational domain for 3D lid-driven cube. Re L =1000 mesh: 128x128x128, constant mesh spacing. A.F. Shinn CFD using GPUs 20 / 30

21 Laminar Flow in 3D Lid-Driven Cube A.F. Shinn CFD using GPUs 21 / 30

22 Turbulent Flow in 3D square duct Figure: Computational domain for 3D square duct. Re τ =360 mesh: 256x64x64, 3% geometric stretching in y-z plane. A.F. Shinn CFD using GPUs 22 / 30

23 3D square duct (Re τ =360) Figure: Contours and velocity vectors of instantaneous streamwise velocity in cross-flow plane at x = 2.0. A.F. Shinn CFD using GPUs 23 / 30

24 3D square duct (Re τ =360) (a) present GPU simulation (b) Madabhushi and Vanka Figure: Velocity vectors of mean flowfield in cross-flow plane. A.F. Shinn CFD using GPUs 24 / 30

25 Speedup of GPU vs. CPU Performance of GPU versus CPU for first 100 time-steps of simulation, with block size bx=by=bz=4 Table 1: Laminar flow in lid-driven cube. mesh Fortran code (sec) CUDA code (sec) speedup (CPU/GPU) 16x16x x32x x64x x128x Table 2: Turbulent flow in a square duct. mesh Fortran code (sec) CUDA code (sec) speedup (CPU/GPU) 256x64x x64x A.F. Shinn CFD using GPUs 25 / 30

26 Speedup of GPU vs. CPU Performance of GPU versus CPU for first 100 time-steps of simulation, with block size bx=by=bz=4 on coarser meshes and bx=32,by=1,bz=8 on finer meshes. Table 1: Laminar flow in lid-driven cube mesh Fortran code (sec) GPU code (sec) speedup (CPU/GPU) 16x16x x32x x64x x128x Speedup improved by factor of 2.8 for 128x128x128 case Table 2: Turbulent flow in square duct mesh Fortran code (sec) GPU code (sec) speedup (CPU/GPU) 256x64x Speedup improved by factor of 2.4 for 256x64x64 case A.F. Shinn CFD using GPUs 26 / 30

27 Introduction Speedup of GPU scaled with the problem size; largest problem size yielded maximum speedup. Single precision did not appreciably affect the results, even for turbulent flows. Global memory easiest to use, but worst for memory latency. Need global residuals to observe convergence. This requires cudamemcpy between CPU/GPU. Very expensive, so decide when you really need to see the residuals. A.F. Shinn CFD using GPUs 27 / 30

28 Introduction Optimization can be a time drain. Need to decide when code is good enough Two possibilities: - Code is complete, just needs porting to CUDA and tuning. Maybe have more time to optimize - Code is not complete, need to add physics features, write in CUDA, and tune. Maybe need to spend more time on physics algorithm and get what you can get out of minimal time coding in CUDA A.F. Shinn CFD using GPUs 28 / 30

29 Future Work Introduction Model complex geometries in flow using the Immersed Boundary Method (IBM) Multi-GPU capability - collaborating with John Stone, UIUC A.F. Shinn CFD using GPUs 29 / 30

30 References Introduction [1] H. Ku, R. Hirsh, and T. Taylor. A Pseudospectral Method for Solution of the Three-Dimensional Incompressible Navier-Stokes Equations. Journal of Computational Physics, 70: , [2] R.K. Madabhushi and S.P. Vanka. Large eddy simulation of turbulence-driven secondary flow in a square duct. Phys. Fluids, 3(11): , A.F. Shinn CFD using GPUs 30 / 30