OpenFOAM on GPUs. 3rd Northern germany OpenFoam User meeting. Institute of Scientific Computing. September 24th 2015

Transcription

1 OpenFOAM on GPUs 3rd Northern germany OpenFoam User meeting September 24th 2015 Haus der Wissenschaften, Braunschweig

2 Overview HPC on GPGPUs OpenFOAM on GPUs 2013 OpenFOAM on GPUs 2015 BiCGstab/IDR(s) 24. September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 2

3 GPGPUs are perfect for HPC CPU is optimized for serial tasks (single thread) GPU is optimized for massive parallel data handling (multiple thread) GPU does not care if pixel data has to be handled (tessellation, transformation, rendering) or scientific calculation has to be performed. 24. September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 3

4 Massive parallel data throughput Programming model inspired by vector computers (SIMD) Goal: Work of as many threads in parallel as possible Through-put orientated approach Accomplished by: Many Arithmetic Logical Units High clock rate of the data bus Highly suitable for massive parallel computing So could we use GPUs to accelerate OpenFOAM? 24. September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 4

5 So far... several GPGPU-plug-ins for OpenFOAM available Usage: Compile the plug-in Check in the library (controldict) functions { cudagpu { type cudagpu; functionobjectlibs ( " gpu " ); cudadevice 0; } } Declare the solver (fvsolutions) p { solver PCGgpu; preconditioner smoothed_aggregation; tolerance 1e-06; reltol 0.01; } 24. September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 5

6 Situation 2013 My talk from the 1st NOFUN Comparison of different GPU plug-ins Based on realistic scenarios/test cases. Here: Korean research institute Container Ship KCStest case 1.8 M cells 24. September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 6

7 KCS example Test Cases a) Solver: simplefoam (steady state) relative Tolerance reltol = 0.1 absolute Tolerance atol=e-7/e-8 b) Solver: simplefoam (steady state) relative Tolerance reltol = 0.0 absolute Tolerance atol=e-7/e-8 c) Solver: interfoam (transient) relative Tolerance reltol = 0.1 absolute Tolerance atol=1e September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 7

8 Speed-up KCS example Comparison for different plug-ins Hardware a) b) c) remark CPU only ofgpu SinglePrecision cufflink OF 1.6ext. speeditclassic fpe fpe 0.66 CG for pressure SP (tubs) CPUs only CPUs + cufflink September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 8

9 Overhead influence (A. Monakov, V. Platono: Accelerating OpenFOAM with a Parallel GPU, 8th OpenFOAM Workshop 2013) 24. September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 9

10 Consequences What to do do gain the profit from GPU use? For real GPU speed-up: Bring the whole algorithm (i.e. Simple/PISO) to the GPGPU Not only the matrix Better port the whole stuff to the GPUs i.e do it in CUDA OpenCL 24. September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 10

11 What happened in the mean time? We were awarded as NVIDIA education centre (My courses Parallel I & II) Fresh hardware 1 Tesla K ALUs, 12 GB, 288 GB/s, 1.43 Tflops (DP) 5 Tesla Geforce GTX ALUs, 2 GB, 192,3 GB/s, 94,1 Gflops (DP) GPU porting project for OpenFOAM (mid 2015) 24. September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 11

12 Paradigm change OpenFOAM running on GPU All the computation is done entirely on GPU. No need to copy data during calculations between CPU and GPU. i.e. no overhead for GPU-CPU memory copy Open source Multi-GPU support Download from github: September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 12

13 Solvers Status Most of the compressible & incompressible solvers (some with dynamic mesh support) are ported: adjointshapeoptimizationfoam pimplefoam rhosimplefoam buoyantboussinesqpimplefoam pisofoam scalartransportfoam buoyantboussinesqsimplefoam porousinterfoam shallowwaterfoam buoyantpimplefoam poroussimplefoam simplefoam buoyantsimplefoam potentialfoam sonicdymfoam driftfluxfoam rhocentraldymfoam sonicfoam icofoam rhocentralfoam sonicliquidfoam interdymfoam rhopimplecfoam SRFPimpleFoam interfoam rhopimpledymfoam SRFSimpleFoam laplacianfoam rhopimplefoam thermofoam nonnewtonianicofoam rhoporoussimplefoam pimpledymfoam rhosimplecfoam 24. September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 13

14 Equation solvers System-of-equation solvers BICCG GAMG ICCG PBiCG smoothsolver (Jacobi) Lack of good and robust preconditioners (up til now: diagonal, AINV) Mesher & tools not available (use from CPU version) Some schemes/b.c.s are missing 24. September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 14

15 Single GPU/Multi-GPU support Single GPU Single GPU support is directly usable, i.e. uses standard OpenMPI from your operation system (systemmpi) from OpenFOAM ThirdParty directory Multi GPU Multi-GPU support needs a CUDA-aware MPI: OpenMPI or later compiled with with-cuda option Should be located in ThirdParty directory. 24. September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 15

16 GPU device selection Device selection is done using command line arguments For single GPU use -device argument: simplefoam -device 2 Tells OFgpu to use GPU with ID 2. For multi GPU execution you use -devices argument followed by a list of GPU IDs: mpirun -n 2 simplefoam -parallel -devices "(2 3)" Tells OGgpu to use GPU 2 for process 0 and GPU 3 for process September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 16

17 Comparison CPU GPU Case: buoyantcavity solver: buoyantsimplefoam RANS model: komegasst # cells: 3,375,000 Solver Clock time (sec) Speed-up 1 CPU CPU GPU (k40) September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 17

18 Comparison CPU GPU Case: sloshingtank3d solver: interdymfoam RANS model: kepsilon # cells: 2,584,000 Solver Clock time (sec) Speed-up 1 CPU CPU GPU (K40) September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 18

19 Comparison CPU GPU Case: pitzdaily solver: simplefoam RANS model: kepsilon # cells: 1,222,500 Solver Iterations Clock time (sec) Speed-up 1 CPU CPU GPU (K40) September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 19

20 Comparison CPU GPU Case: turbinesiting solver: simplefoam RANS model: kepsilon # cells: Solver Iterations Clock time (sec) Speed-up 1 CPU CPU GPU (K40) (not converged) 24. September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 20

21 Development: BiCGstab to GPU Porting Biorthogonal Conjugated Gradient stabilized 1 Motivation Disadvantages BiCG There are multiplications with A T needed. For a regular matrix, the method could terminate without solution The method has no minimization properties for the iteration vector. This could cause an oscillatory behaviour in the convergence Advantages BiCGstab In general, BiCGstab has smoother convergence properties Multiplication with A T is not necessary. 1 van der Vorst, H. A. BI-CGSTAB: A fast and smoothly converging variant of BI-CG for the solution of nonsymmetric linear systems, SIAM J. Sci. Stat. Comput., 13: , September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 21

22 Development: BiCGstab to GPU Code snippet scalargpufield va(pbcache::va(matrix_.level(),ncells),ncells); // --- Calculate v=a*p and precondition v matrix_.amul(va, pa, interfaceboucoeffs_, interfaces_, cmpt); preconptr->precondition(vp, va, cmpt); // --- Calculate alpha=rpr0/vpr0 scalar alpha2 = gsumprod(vp, r0, matrix().mesh().comm()); scalar alpha = rpr0/alpha2; thrust::transform ( ra.begin(), ra.end(), va.begin(), sa.begin(), raminusalphawafunctor(alpha) ); // --- Precondition s preconptr->precondition(sp, sa, cmpt); // --- Calculate t=a*sp and precondition t matrix_.amul(ta, sp, interfaceboucoeffs_, interfaces_, cmpt); preconptr->precondition(tp, ta, cmpt); 24. September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 22

23 Comparison BiCG/BiCGSTAB Residuals Comparison of the residuals BiCG/BiCGSTAB 24. September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 23

24 Plan: IDR(s) Induced Dimension Reduction method 2 Efficient methods for large nonsymmetric systems Based on the Induced Dimension Reduction (IDR) method proposed by Sonneveld (1980) Competitive with or superior to most Bi-CG-based methods Outperforms BiCGstab for s > 1. 2 Peter Sonneveld and Martin B. van Gijzen, IDR(s): a family of simple and fast algorithms for solving large nonsymmetric linear systems. SIAM J. Sci. Comput. Vol. 31, No. 2, pp , September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 24

25 Conclusions The portation of OpenFOAM to GPU can be a promising approach But: Some work has to be done Solvers (convergence?) Preconditioners Environment/Utilities Examining the possibility to use external GPU equation solver libraries with OpenFOAM. 24. September 2015 Thorsten Grahs OpenFOAM on GPUs Seite 25