Stan Posey NVIDIA, Santa Clara, CA, USA;

Transcription

1 Stan Posey NVIDIA, Santa Clara, CA, USA;

2 Agenda: GPU Progress and Directions for CAE Introduction of GPUs in HPC Progress of CFD on GPUs Review of OpenFOAM on GPUs Discussion on WRF Developments 2

3 CFD Algorithm Suitability for GPUs CFD Speed-Ups Demonstrated in Range of Time Schemes and Spatial Discretization Explicit [usually compressible] ~15x Stencil operations, uniform memory refs ~5x Stencil operations, renumbering schemes ~x Factors Based on Comparisons with Xeon 8-core Sandy Bridge CPU Strategy: Directives Strategy: Directives Implicit [usually incompressible] ~5x ~2-3x Linear algebra solver, uniform memory refs Strategy: Libraries ISVs Linear algebra solver, renumbering schemes Strategy: Libraries Structured Grid Unstructured 3

4 Turbostream: CFD for Turbomachinery Source: Sample Turbostream GPU Simulations Typical Routine Simulation Large-scale Simulation ~19x Speedup 4

5 SD++ and Jameson Aerodynamics Research Stanford University Aerospace Computing Lab Prof. Antony Jameson GPU Application Jameson-developed CFD software SD++ for high order method aerodynamic simulations GPU Benefit Use of 16 x Tesla M2070: 15 hrs vs. 202 hrs for 16 x Xeon X5670 Fast turnaround of complex LES simulations that would otherwise be impractical for CPU-only use 15 hours on 16 x M2070s 202 hours ( > one week) on 16 Xeon x5670 CPUs Transitional flow over SD70053 airfoil, 21M DOF, Ma =.2, Re=60K, AoA=4, 4 th order, 400K RK iters 5

6 Fighter Jet Engine Noise Reduction on GPUs U.S. DoD Naval Research Lab Lab for Computational Physics and Fluid Dynamics GPU Application GPU Benefit NRL-developed CFD software JENRE for simulation of jet engine acoustics Use of Tesla M2070: 3x vs. Hex core Intel (Westmere) CPU More detailed mesh simulations possible for longer durations of jet engine transient conditions 6

7 Commercial Aircraft Wing Design on GPUs COMAC and SJTU Commercial Aircraft Corporation of China GPU Application SJTU-developed CFD software NUS3D for aerodynamic simulations of wing shapes COMAC Wing Candidate GPU Benefit Use of Tesla C2070: 20x 37x vs. single core Intel core i7 CPU Faster simulations for more wing design candidates vs. wind tunnel testing Expanding to multi-gpu and full aircraft ONERA M6 Wing CFD Simulation 7

8 GPU Development Status for CFD Particle CFD (LBM, SPH, etc.) generally better fit vs. continuum Fully deployed explicit solvers generally outperform implicit Explicit i,j,k stencil operations good fit for massively parallel threads Most CFD is distributed parallel across CPU multicores/nodes Fits GPU parallel model well and preserves costly MPI investment Focus on hybrid parallel schemes that utilize all CPU cores + GPU GPU development strategy depends on profile starting point: Legacy explicit scheme: compiler directives such as OpenACC New explicit scheme: CUDA and stencil libraries Legacy implicit scheme: CUDA and libs for solver; OpenACC for rest New implicit scheme: CUDA and libs for solver, matrix assembly, etc. 8

9 FluiDyna and Aerodynamic-Aware Surface Design RTT DeltaGen for photo realistic 3D visualization Integrates FluiDyna LBultra CFD functionality as plug-in Designer only must specify resolution and velocity Simulation data displayed live with GPU performance Courtesy of FluiDyna and Lbultra CFD Software: 9

10 Prometech and Particleworks for Multiphase Flow Oil Flow in HB Gearbox MPS-based method developed at the University of Tokyo [Prof. Koshizuka] Particleworks 3.0 GPU vs. 4 core i7 Courtesy of Prometech Software and Particleworks CFD Software 10

11 Availability of Commercial DSFD-Based Software ISV Software Application Method GPU Status PowerFLOW Aerodynamics LBM Evaluation Lbultra Aerodynamics LBM Available v2.0 XFlow Aerodynamics LBM Evaluation Project Falcon Aerodynamics LBM Evaluation Particleworks Multiphase/FS MPS (~SPH) Available v3.1 BARRACUDA Multiphase/FS MP-PIC In development EDEM Discrete phase DEM In development ANSYS Fluent DDPM Multiphase/FS DEM In development STAR-CCM+ Multiphase/FS DEM Evaluation AFEA High impact SPH Available v2.0 ESI High impact SPH, ALE In development LSTC High impact SPH, ALE Evaluation Altair High impact SPH, ALE Evaluation 11

12 Grid-Based Commercial CFD and GPU Progress ISV Primary Applications (Green color indicates CUDA-ready during 2013) ANSYS ANSYS Mechanical; ANSYS Fluent; ANSYS HFSS DS SIMULIA Abaqus/Standard; Abaqus/Explicit; Abaqus/CFD MSC Software Altair CD-adapco Autodesk ESI Group Siemens LSTC Mentor Metacomp MSC Nastran; Marc; Adams RADIOSS; AcuSolve STAR-CD; STAR-CCM+ AS Mechanical, Moldflow, AS CFD PAM-CRASH imp; CFD-ACE+ NX Nastran LS-DYNA; LS-DYNA CFD FloEFD, FloTherm CFD++ 12

13 Additional Commercial GPU Developments ISV Domain Location Primary Applications FluiDyna CFD Germany Culises for OpenFOAM; LBultra Vratis CFD Poland Speed-IT for OpenFOAM; ARAEL Prometech CFD Japan Particleworks Turbostream CFD England, UK Turbostream IMPETUS Explicit FEA Sweden AFEA AVL CFD Austria FIRE CoreTech CFD (molding) Taiwan Moldex3D Intes Implicit FEA Germany PERMAS Next Limit CFD Spain XFlow CPFD CFD USA BARRACUDA Flow Science CFD USA FLOW-3D 13

14 Status Summary of ISVs and GPU Computing Every primary ISV has products available on GPUs or undergoing evaluation The 4 largest ISVs all have products based on GPUs, some at 3rd generation #1 ANSYS, #2 DS SIMULIA, #3 MSC Software, and #4 Altair The top 4 out of 5 ISV applications are available on GPUs today ANSYS Fluent, ANSYS Mechanical, Abaqus/Standard, MSC Nastran, (LS-DYNA implicit only) Several new ISVs were founded with GPUs as a primary competitive strategy Prometech, FluiDyna, Vratis, IMPETUS, Turbostream Open source CFD OpenFOAM available on GPUs today with many options Commercial options: FluiDyna, Vratis; Open source options: Cufflink, Symscape ofgpu, RAS, etc. 14

15 CFD Algorithm Characterization: Discretization Structured Grid FV Unstructured FV Unstructured FE Finite Volume Finite Element: 15

16 CFD Algorithm Characterization: Time Integration Structured Grid FV Unstructured FV Unstructured FE Explicit Usually Compressible Finite Volume Finite Element: Implicit Usually Incompressible 16

17 CFD Algorithm Characterization: Time Integration Structured Grid FV Unstructured FV Unstructured FE Explicit Numerical operations on I,J,K stencil, no solver [Typically flat profiles: GPU strategy of directives (OpenACC)] Usually Compressible Finite Volume Finite Element: Implicit Usually Incompressible 17

18 GPU Acceleration Relative to Single 8-Core CPU Structured Grid FV Unstructured FV Unstructured FE Explicit Usually Compressible Implicit Usually Incompressible ~15x ~5x Turbostream Veloxi SJTU RANS Finite Volume - SD++ Stanford (Jameson) - FEFLO (Lohner) Finite Element: 18

19 GPU Acceleration Relative to Single 8-Core CPU Structured Grid FV Unstructured FV Unstructured FE Explicit Usually Compressible ~15x ~5x Turbostream Veloxi SJTU RANS Finite Volume - SD++ Stanford (Jameson) - FEFLO (Lohner) Finite Element: Implicit Usually Incompressible Sparse matrix linear algebra iterative solvers [Hot spot ~50%, small % LoC: GPU strategy of CUDA and libs] 19

20 GPU Acceleration Relative to Single 8-Core CPU Structured Grid FV Unstructured FV Unstructured FE Explicit Usually Compressible ~15x ~5x Turbostream Veloxi SJTU RANS - SD++ Stanford (Jameson) - FEFLO (Lohner) Implicit Usually Incompressible Finite Volume - ANSYS Fluent - Culises for OpenFOAM - SpeedIT for OpenFOAM - CFD-ACE+ - FIRE ~2x Finite Element: - Moldflow - AcuSolve - Moldex3D 20

21 Commercial CFD Focus on Sparse Solvers for GPU CFD Application Software Read input, matrix Set-up GPU Implicit Sparse Matrix Operations - Hand-CUDA Parallel - GPU Libraries, CUBLAS 50% - 65% of Profile time, Small % LoC Implicit Sparse Matrix Operations (Investigating OpenACC for more tasks on GPU) CPU - OpenACC Directives Global solution, write output + 21

22 NVIDIA-Developed Library of Linear Solvers Library of nested solvers for large sparse Ax=b Nesting creates a solver hierarchy, e.g. BiCGstab AMG Jacobi Example solvers MC-DILU Jacobi, simple local (neighbor) operations, no/little setup BiCGStab, local and global operations, no setup MC-DILU, graph coloring and factorization at setup AMG, multi-level scheme, on each level: graph coarsening and matrixmatrix products at setup 22

23 ISV Progress with NVIDIA CFD Solver Library Committed: ANSYS ANSYS Fluent and ANSYS CFD : #1 in CFD FluiDyna Culises library use in OpenFOAM: OpenFOAM is #2 in CFD for leveraged hardware Evaluation: Autodesk AS Moldflow: the leader in plastic mold injection simulation Autodesk AS CFD: important to the design engineering market and being hosted on Autodesk cloud Discussion: CD-adapco STAR-CCM+: the # 2 CFD code for software rev, either #2 or #3 for leveraged hardware ESI CFD-ACE+: important CFD code in the semiconductor/electronics industry along with others Cradle SC/Tetra: #3 CFD in Japan (behind ANSYS Fluent and STAR-CCM+) and primary CFD code at Toyota Targets: Altair AcuSolve: GMRES Metacomp CFD++: AMG Mentor FloEFD: AMG SIMULIA Abaqus/CFD: use ML from Petsc LSTC LS-DYNA CFD: AMG AVL FIRE: AMG Convergent Technologies Converge CFD: GMRES 23

24 ANSYS and NVIDIA Technical Collaboration Release ANSYS Mechanical ANSYS Fluent ANSYS EM 13.0 Dec 2010 SMP, Single GPU, Sparse and PCG/JCG Solvers ANSYS Nexxim 14.0 Dec Distributed ANSYS; + Multi-node Support Radiation Heat Transfer (beta) ANSYS Nexxim 14.5 Nov Multi-GPU Support; + Hybrid PCG; + Kepler GPU Support + Radiation HT; + GPU AMG Solver (beta), Single GPU ANSYS Nexxim 15.0 Q CUDA 5 Kepler Tuning + Multi-GPU AMG Solver; + CUDA 5 Kepler Tuning ANSYS Nexxim ANSYS HFSS (Transient) 24

25 ANSYS Fluent 14.5 and Radiation HT on GPU VIEWFAC Utility: Use on CPUs, GPUs or both ~2x speedup Radiation HT Applications: - Underhood cooling - Cabin comfort HVAC - Furnace simulations RAY TRACING Utility: Uses OptiX library from NVIDIA with up to ~15x speedup (Use on GPU only) - Solar loads on buildings - Combustor in turbine - Electronics passive cooling 25

26 ANSYS Fluent CPU Job Profile for Coupled PBNS Non-linear iterations Assemble Linear System of Equations Runtime: ~ 35% Accelerate this first Solve Linear System of Equations: Ax = b ~ 65% No Converged? Yes Stop 26

27 ANSYS Fluent AMG Solver Time per Iteration (Sec) ANSYS Fluent GPU-Based AMG Solver from NVIDIA ANSYS Fluent 14.5 Performance Results by NVIDIA, Nov Airfoil and Aircraft Models with Hexahedral Cells K20X 3930K(6) Lower is Better 2 x Core-i7 3930K, Only 6 Cores Used 6 2.4x Solver settings: x Airfoil (hex 784K) Aircraft (hex 1798K) CPU Fluent solver: F-cycle, agg8, DILU, 0pre, 3post GPU nvamg solver: V-cycle, agg8, MC-DILU, 0pre, 3post NOTE: Times for solver only 27

28 Comparison of AMG Cycles on CPU and GPU 2D Convection Case: F-cycle best for both CPU and GPU Lower is Better CPU-F GPU-F 28

29 GPUs and Distributed Cluster Computing Partition on CPU Geometry decomposed: partitions put on independent cluster nodes; CPU distributed parallel processing Nodes distributed parallel using MPI N1 N1 N2 N3 N4 1 Global Solution 29

30 GPUs and Distributed Cluster Computing Partition on CPU Geometry decomposed: partitions put on independent cluster nodes; CPU distributed parallel processing Nodes distributed parallel using MPI N1 1 Execution on CPU + GPU N1 N2 N3 N4 G1 G2 G3 G4 GPUs shared memory parallel using OpenMP under distributed parallel Global Solution 30

31 ANSYS Fluent Preview for 2 x CPU + 2 x Tesla K20X ANSYS Fluent 15.0 Preview Performance Results by NVIDIA, Feb x K20X E5_2680(16) Lower is Better 2 x E5_2680 SB CPUs, 16 cores total, only 2 cores used with GPUs 1 2.1x Solver settings: x CPU Fluent solver: F-cycle, agg8, DILU, 0pre, 3post 0 Helix (tet 1173K) Airfoil (hex 784K) GPU nvamg solver: V-cycle, agg8, MC-DILU, 0pre, 3post NOTE: Times for solver only 31

32 ANSYS Fluent Scaling Results for 4 x Tesla K20X ANSYS Fluent 15.0 Preview Performance Results by NVIDIA, Mar Helix (Tet 1.2M) Airfoil (Hex.78M) Sedan (Mixed 3.6M) Perfect Scaling Higher is Better Hardware Setup: 2 server nodes 2 GPUs each node Infiniband network GPU Solver Settings: V-cycle, agg8/2, MC-DILU, 0pre, 3post 1 K20X(1) K20X(2) K20X(3) K20X(4) NOTES: Results for solver only Sedan case starts with 2 GPUs 32

33 ANSYS Fluent 15.0 Multi-GPU Demonstration Multi-GPU Acceleration of a 16-Core ANSYS Fluent Simulation of External Aero 2.9X Solver Speedup Click to Launch Movie Xeon E CPUs + Tesla K20X GPUs CPU Configuration CPU + GPU Configuration 16-Core Server Node 8-Cores 8-Cores G1 G2 G3 G4 33

34 Summary: Opportunity for Advanced CFD Problem Statement: CFD demand for increased levels of CFD model resolution for improved simulation accuracy CFD use is 80% steady state RANS today rather a short-cut to faster turn-around Fluid flow is inherently unsteady and in need of better turbulence treatment CPU-based HPC limits advanced CFD Opportunity: CFD ISVs have developed URANS, DES, and LES capabilities which undergo very limited use CPU-based turnaround times are impractical for many product development workflows Large Eddy Simulation (LES) is of most interest and has a high degree of arithmetic intensity GPU computing can offer a practical solution for LES that doesn t exist today with CPUs 34

35 Conclusions For CAE on GPUs Opportunities exist for GPUs to provide significant performance acceleration for solver intensive large jobs Improved product quality Shorten product engineering cycles (Faster Time-to-Market) Better Total Cost of Ownership (TCO) Cut down energy consumption in the CAE process Simulations recently considered intractable are now possible Large Eddy Simulation (LES) with a high degree of arithmetic intensity Parameter optimization with highly increased number of jobs 35

36 Stan Posey NVIDIA, Santa Clara, CA, USA;