A GPU Accelerated Adjoint Solver for Shape Optimization

Transcription

1 A GPU Accelerated Adjoint Solver for Shape Optimization Asitav Mishra 1 Asst. Research Scientist Dylan Jude 1 PhD Candidate James D. Baeder 1 Professor 1 Department of Aerospace Engineering, University of Maryland, College Park, MD Funded by:

2 Introduction CFD Vision 2030 emphasizes HPC and GPUs Modern HPC systems support heterogenous (GPU/CPU) computing; e.g. SUMMIT by Oak Ridge NL GPU allows hi-fidelity (RANS, LES, DNS) CFD for complex vortex dominated flows Need to leverage on GPUs for modern complex aerospace computations Figure: CFD Vision 2030 Figure: SUMMIT Supercomputers (image credit: ORNL) 2 / 48

3 Introduction CFD Vision 2030 emphasizes HPC and GPUs Modern HPC systems support heterogenous (GPU/CPU) computing; e.g. SUMMIT by Oak Ridge NL GPU allows hi-fidelity (RANS, LES, DNS) CFD for complex vortex dominated flows Need to leverage on GPUs for modern complex aerospace computations Figure: CFD Vision 2030 Figure: SUMMIT Supercomputers (image credit: ORNL) 3 / 48

4 Introduction Continued... Adjoint-based optimization popular using CFD Design practically independed of number of design variables Efficient (parallel) adjoint for reduced design computational cost GPUs can provide extreme speed up gains GPU ADjoint platforms can be powerful tools for future aerospace design Figure: UH60 adjoint solution Ref: FUN3D Manual, NASA Figure: HART2 blade optimization Ref: Mishra et. al, AIAAJ, / 48

5 Introduction Continued... Adjoint-based optimization popular using CFD Design practically independed of number of design variables Efficient (parallel) adjoint for reduced design computational cost GPUs can provide extreme speed up gains GPU ADjoint platforms can be powerful tools for future aerospace design Figure: UH60 adjoint solution Ref: FUN3D Manual, NASA Figure: HART2 blade optimization Ref: Mishra et. al, AIAAJ, / 48

6 Introduction Background and Motivation GPU CFD Solver: GARFIELD 2D Adjoint Euler Shape Optimization ADGAR: GPU Accelerated 2-D Adjoint Formulation Conclusions 6 / 48

7 Motivation: GPU Accelerated CFD Elsen: 20-40x, hypersonic NS Chander: 60x, incompressible NS overset (CU++) Kajeh-Saeed: 37x, turbulent DNS (GPU cluster) Walden: 6x per node on FUN3D linear solver (6 GPU vs 2 CPU) GPU accelerated CFD holds great potential Figure: Elsen et. al (JCP 2008) Figure: Chander et. al (J. Supercomputer 2014) 7 / 48

8 Recent GPU CFD RANS at UMD Figure: Jung et. al (AHS 2017) Thomas: GPURANS, 55x helicopter brownout, 2013 Jude: GPURANS 17-28x unsteady grid motion, 2016 Jung: GARFIELD, 7x (16 GPU vs 32 CPU cores, per node), 2017 Sufficient background for future adjoint GPU developments 8 / 48

9 Motivation: GPU-Adjoint CFD for Design Optimization Tsiakas: 50x per-node on U-bend duct optimization, continuous adjoint, 2015 Toit: 2x on Monte Carlo kernels using AD, 2013 Laniewski-Wollk: topology opt w/ Lattice Boltzmann AD, linear weak scaling, 2016 No GPU-enabled discrete adjoint (AD) CFD exist Dr. Baeder s funding as an Associate Langley Professor enables such unique technology development Figure: Baseline (a) vs Optimized (b) u-bend Tsiakas (GRACM, 2015) 9 / 48

10 Motivation: GPU-Adjoint CFD for Design Optimization Tsiakas: 50x per-node on U-bend duct optimization, continuous adjoint, 2015 Toit: 2x on Monte Carlo kernels using AD, 2013 Laniewski-Wollk: topology opt w/ Lattice Boltzmann AD, linear weak scaling, 2016 No GPU-enabled discrete adjoint (AD) CFD exist Dr. Baeder s funding as an Associate Langley Professor enables such unique technology development Figure: Baseline (a) vs Optimized (b) u-bend Tsiakas (GRACM, 2015) 10 / 48

11 Objective Step 1: Develop GARFIELD inspired structured 2D Adjoint Euler optimization platform Verifiy discrete adjoint (hand-derived/ad) with finite difference Step 2: Perform GPU acceleration of the 2D hand-derived adjoint solver Verify GPU adjoint, compare solutions with CPU adjoint Conduct GPU speed up studies Perform reverse design optimization on airfoils Figure: Objective of this work 11 / 48

12 Introduction Background and Motivation GPU CFD Solver: GARFIELD 2D Adjoint Euler Shape Optimization ADGAR: GPU Accelerated 2-D Adjoint Formulation Conclusions 12 / 48

13 GPU CFD Solver: GARFIELD GARFIELD: 3D GPU-Accelerated Rotor flow FIELD solver at UMD Multi-GPU and heterogenous CPU/GPU compute capability, Python wrapped 3D RANS solver for compressible unsteady turbulent flows Implicit operator DADI with up-wind dissipation, MUSCL/WENO reconstruction Minimum CPU-GPU communication; most computations in GPU Variable, point, line parallelism using CUDA kernels Figure: GARFIELD flowchart Speedup GARFIELD Linear Number of GPUs Figure: GARFIELD scalability Ref: Jude et. al, Scitech / 48

14 GARFIELD Results Caradonna-Tung rotor in hover: Untwisted rectangular rotor, AR=6, NAC0012 Rey = , Mach = Mesh points: 9.1M (Background) + 2.2M (blade) Table: GARFIELD vs OVERFLOW (7.2M Blade mesh) Code Cores or GPUs Time (s) Speedup over 32 cores OVERFLOW 32 Core GPU GARFIELD 2 GPU GPU GPU GPU Figure: Caradonna-Tung solutions Ref: Jude et. al, Scitech / 48

15 Introduction Background and Motivation GPU CFD Solver: GARFIELD 2D Adjoint Euler Shape Optimization ADGAR: GPU Accelerated 2-D Adjoint Formulation Conclusions 15 / 48

16 2D Adjoint Euler Shape Optimization Figure: 2D Optimization Flowchart Euler (forward): GARFIELD inspired 2D Euler Adjoint: Hand-derived, auto-differentiated (AD) Meshgen: Poisson equations (Steger & Sorenson sources) Optimization: Python based framework, SciPy library 16 / 48

17 2D Euler/Forward Formulation q = J 1 ρ ρu 1 ρu 2 e q t + f c,i = 0 (1) ξ i, ρv 1 f c,i = J 1 ρu 1 V i + ξ i,1 p ρu 2 V i + ξ i,2 p (e + p)v i (2) J is Jacobian and contravariant velocity V i = u 1 ξ i,1 + u 2 ξ i,2 Implicit discretization: q n+1 q n ( ) = f n+1 f n+1 D n+1 (3) t c,i+ 1 2 c,i 1 2 1st order in space and time Explicit and implicit (DADI) schemes D is artificial dissipation (Roe-flux or scalar dissipation) 17 / 48

18 2D Adjoint Formulation Adjoint: δi q ψ(δr q ) = 0 Residual: where: δr q n j,k = δh j+ 1 2,k δh j 1 2,k + δh j,k+ 1 δh 2 j,k 1 2 δh j+ 1 2,k = δf j+ 1 2,k δd j+ 1 2,k δh j,k+ 1 2 = δf j,k+ 1 2 δd j,k+ 1 2 The term ψ(δr q ) auto-differentiated or hand-derived. Artificial (scalar) dissipation term: d j+ 1 2,k = ɛσ ( ) q j+1,k q j,k with ɛ 0.25, σ = V + c S j+ 1, and c is speed of sound. 2 Discrete adjoint: jmax kmax j k ψ T j,k δr q n j,k 18 / 48

19 Objective Sensitivity: Tangent vs Adjoint ( ) Ic Tangent: = α ( ) T Ic Adjoint: = α ( ) ( ) ( ) Ic Q X Q X α ( ) T ( ) T ( X Q Ic α X Q ) T Finally, Tangent order: α Ẋ Q İ; () = () α Adjoint order: ᾱ X Q Ī; () = () I c I c = I c α = Q T Q = X T Ẋ = ᾱ T α (4) This work computes adjoint X and finite-difference Ẋ 19 / 48

20 Gradient-based Optimization Adjoint sensitivity of objective I: { I T δi = X ψt [ ]} R δx X Term {} using adjoint δx using finite difference, i.e. mesh re-generation Airfoil optimization using SciPy library (SLSQP, CG) Airfoil geometry parameterization using Hicks-Henne bump functions 20 / 48

21 Grid Generation Two-dimensional mesh generation solves Poisson equations: ξ xx + ξ yy = P s η xx + η yy = Q s P s, Q s are Steger and Sorenson (JCP, 1979) source terms Uniform outer boundaries, orthogonality near wall Figure: Example O-grids 21 / 48

22 2D Euler Results NACA0012 2D Euler vs 3D GARFIELD Euler Mach = 0.8, α = Pressure Coefficient Reference Solver 2D Euler Solver Chord Figure: Transonic Euler validation with GARFIELD 22 / 48

23 2D Adjoint Euler Verification NACA0012, Mach = 0.5, AoA = 5 Hicks-Henne surface parametrization Upper/lower D var : bump 50 and 75%c) I c (α) = N i=0 1 2 (P i P d,i ) 2, P d,i = desired surface pressure Brute-Force AD Adjoint Hand Adjoint α I/ α 1 I/ α 2 I/ α 3 I/ α Table: Adjoint and brute-force gradient comparison. 23 / 48

24 Forward vs Adjoint Information Propagation Figure: Forward vs Adjoint x-momemntum vector Direction of information propagation in adjoint is opposite of forward/euler. 24 / 48

25 2D Adjoint Euler Optimization Baseline: NACA0012, target: NACA2312; Mach = 0.5, AoA = 5 SLSQP and CG optimization techniques Figure: 2D Airfoil Optimization 25 / 48

26 Introduction Background and Motivation GPU CFD Solver: GARFIELD 2D Adjoint Euler Shape Optimization ADGAR: GPU Accelerated 2-D Adjoint Formulation Adjoint-based GPU Accelerated Optimization Conclusions 26 / 48

27 ADGAR: GPU Accelerated 2-D Adjoint Solver ADGAR: ADjoint GARfield GPU acceleration using CUDA kernels DADI in forward/adjoint GPU accelerated Parallelism: variable, point and line Verification with serial forward/adjoint solutions Figure: O-grid used by ADGAR (181 60) 27 / 48

28 ADGAR: Forward/Euler Verification CPU (serial) vs GPU ADGAR Euler solution verified Implicit iterative operator DADI is GPU accelerated NACA0012, Mach = 0.5, AoA = 5, O-grid: Figure: CPU vs GPU Euler/Forward solution 28 / 48

29 ADGAR: Adjoint Verification CPU (serial) vs GPU ADGAR Adjoint solution verified Implicit iterative operator Adjoint DADI is GPU accelerated NACA0012, Mach = 0.5, AoA = 5, O-grid: Figure: CPU vs GPU Adjoint convergence and solution 29 / 48

30 GPU Acceleration Paradigms Line parallel: DADI, each line each grid dimension Point parallel: Flux, BC and most routines Variable parallel: Update routines CUDA library, cublasdnrm2: L2-norm computations GPU acceleration tested on 3 mesh sizes Speed up variable upon compiler flags/machines 30 / 48

31 GPU Acceleration of ADjoint DADI ADjoint DADI: ( T 1 η Three steps: ) T [I + hδη Λ η ] T [ ˆN ] S 2 {}}{ T [I + hδξ Λ ξ ] T Tξ T Ψ = R ψ (5) }{{} S 1 1. [I + hδ η Λ η ] T S 1 = T T η R ψ 2. [I + hδ ξ Λ ξ ] T S 2 = [ ˆN ] T S1 3. Ψ = T T ξ S 2 1. [I + hδ η Λ η ] T S 1 = T T η R ψ : i) MVP : T T η R ψ ii) Compute LDU: [I + hδ η Λ η ] T iii) Invert along η : S 1 = [I + hδ η Λ η ] T [ T T η R ψ ] 31 / 48

32 CUDA Kernel: ADjoint DADI Line and Point parallel in Adjoint DADI: // // Eta-inversion // // Custom threads for Tri-Diagonal Inversion thr_line.x = 4; thr_line.y = 4; //h_dim->nvar blk_line.x = (h_dim->kmax-1)/thr_line.x+1; blk_line.y = 1; // 1.MVP (POINT PARALLEL): [T_etaˆT]x[RHS_adj] mvp_eta<<<blk_pt, thr_pt>>>(rhs,...); // 2.Compute LDU of LHS_eta (POINT PARALLEL) LDU<K_DIR><<<blk_pt, thr_pt>>>(); // 3. Tri-diag eta-inversion (LINE PARALLEL) eta_tridiag<<<blk_line, thr_line>>>(rhs,...); 32 / 48

33 CUDA Kernels: Line Parallelism in DADI Figure: Line parallelism in DADI kernel ξ, η tridiagonal inversion in DADI Solves tridiagonal inversion per k-line per variable 33 / 48

34 CUDA Kernels: Variable Parallelism Variable parallel (adjoint vector update): // GPU kernel to update psi: var parallel global void update_psiv(double (*rhs)[4],... ){ //... // update psi d_sol_ad->psi[grid_idx][var] += rhs[grid_idx][var]; } // Update Adj vector kernel (VARIABLE PARALLEL) update_psiv<<<blk_var, thr_var>>>(rhs,...); 34 / 48

35 CUDA Kernels: Race Condition Figure: Race condition in flux computation Race condition: Two threads access same address (cell) in flux computation Solution: split flux computations to two steps 35 / 48

36 GPU Acceleration for Varying Problem Sizes Mesh size CPU Time (s) GPU Time (s) Speed up Forward Adjoint Functional + Jacobian Forward Adjoint Functional + Jacobian Forward Adjoint Functional + Jacobian Table: ADGAR GPU vs CPU speedup (Tesla P100-PCIE-16GB) 36 / 48

37 Speed Up Study: GPU Arch & Flags Table: GPU performance with machines (mesh: ) GPU Arch CPU Time (s) GPU Time (s) Speed up Forward GeForce TitanX (Kepler) Adjoint Total Tesla P100 (Volta) Quadro GV100 (Volta) Forward Adjoint Total Forward Adjoint Total Table: Quadro-GV100 GPU speedup with flags (mesh: ) Compiler Flags CPU Time (s) GPU Time (s) Speed up Debug (-g) Total Optimization (-O3) Total Better performance with newer generation GPU Inappropriate flags misrepresent speed up 37 / 48

38 Introduction Background and Motivation GPU CFD Solver: GARFIELD 2D Adjoint Euler Shape Optimization ADGAR: GPU Accelerated 2-D Adjoint Formulation Adjoint-based GPU Accelerated Optimization Conclusions 38 / 48

39 ADGAR Airfoil Design Optimization Figure: ADGAR Optimization Framework Minimal GPU CPU transfer per time step Major computations done in GPUs 39 / 48

40 ADGAR Airfoil Optimization: Single Variable mod-naca0012, Mach = 0.5, AoA = 5, O-grid: I c (α 1 ) = 1 2 (P 1 P d,1 ) 2 α 1 = location of first Hicks-Henne bump in chords Figure: Comparing α 1 = 0.1 vs α 1 = / 48

41 Single Variable Optimization Continued... Figure: Optimized and target solutions for α 1 = 0.1 CG converges faster than SLSQP for both α 1 = 0.1, 0.7 Both methods yield similar optimization results 41 / 48

42 ADGAR Airfoil Optimization: Multiple Variable I c (α) = N i=0 1 2 (P i P d,i ) 2 α i, i = [1 : 6], upper/lower Hicks-Henne bump 50, 75%) chords Baseline: NACA0012, target: NACA2312, Mach = 0.5 AoA = 5, O-grid: I/ α 1 I/ α 2 I/ α 3 I/ α 4 CPU Adjoint e e e e-02 GPU Adjoint e e e e-02 Table: GPU vs CPU adjoint sensitivities ( α = 10 8 ). 42 / 48

43 Multiple Variable Optimization Continued... Figure: Optimization convergence and solutions 43 / 48

44 Multiple Variable Optimization Continued... Design Iterations Adjoint Calls Function Calls SLSQP CG Table: Optimization function and adjoint calls using SLSQP and CG with six design variables. Summary of ADGAR optimization: GPU sensitivities verified to 14 significant digits wrt CPU SLSQP faster than CG (gradient drop 2 orders) Both methods successfully yield target airfoil and pressure GPU airfoil optimization validated against CPU results 44 / 48

45 Introduction Background and Motivation GPU CFD Solver: GARFIELD 2D Adjoint Euler Shape Optimization ADGAR: GPU Accelerated 2-D Adjoint Formulation Conclusions 45 / 48

46 Conclusions Presented existing capabilities of GARFIELD (multi-gpu, heterogenous CPU/GPU) First step: 2D Adjoint Euler optimizer inspired by GARFIELD developed, verified and tested for inverse design Second step: developed ADGAR - GPU accelerated 2D adjoint Euler shape optimizer DADI for Euler and Adjoint implemented, verified convergence GPU solution verified with CPU to 14 significant digits accuracy Inverse design optimization and speed up studies on NACA-series airfoils Future work: Extend to turbulent flows GPU CUDA kernel optimizations: shared memory, streaming, atomic 46 / 48

47 Acknowledgements Funded by the National Institute of Aerospace (NIA), Hampton, VA through the Langley Professor Program. University of Maryland s DeepThought 2 supercomputing facility ( and other computational resources at the university. 47 / 48

48 Thank you 48 / 48