Predictive Engineering and Computational Sciences. Full System Simulations with Fully Implicit Navier-Stokes

Transcription

1 PECOS Predictive Engineering and Computational Sciences Full System Simulations with Fully Implicit Navier-Stokes Roy Stogner, Benjamin Kirk, Paul Bauman, Todd Oliver, Kemelli Estacio, Marco Panesi, Juan Sanchez The University of Texas at Austin April 27, 2011 R. Stogner Full System Simulations April 27, / 25

2 Introduction Outline 1 Introduction FIN-S 2 Modeling Aerothermochemistry Ablation 3 Software Formulation Robustness Verification Scalability Optimization 4 Results Convergence R. Stogner Full System Simulations April 27, / 25

3 Introduction FIN-S PECOS Goals Verification, Validation, Uncertainty Quantification R. Stogner Full System Simulations April 27, / 25

4 Introduction FIN-S PECOS Goals Verification, Validation, Uncertainty Quantification Numerics Multiphysics Coupling Submodel Testing Adjoint Sensitivities Adaptive Discretization Goal-oriented Refinement QoI Error Estimation Robustness R. Stogner Full System Simulations April 27, / 25

5 Introduction FIN-S PECOS Goals Verification, Validation, Uncertainty Quantification Numerics Multiphysics Coupling Submodel Testing Adjoint Sensitivities Adaptive Discretization Goal-oriented Refinement QoI Error Estimation Robustness Software Modularity Unit testing Physics independence Extensibility Flexible submodeling Operator verification R. Stogner Full System Simulations April 27, / 25

6 Introduction FIN-S PECOS Goals Verification, Validation, Uncertainty Quantification Numerics Multiphysics Coupling Submodel Testing Adjoint Sensitivities Adaptive Discretization Goal-oriented Refinement QoI Error Estimation Robustness Software Modularity Unit testing Physics independence Extensibility Flexible submodeling Operator verification Physics Coupled high fidelity multiphysics Complete model documentation Manufactured benchmarks Exposed model parameters R. Stogner Full System Simulations April 27, / 25

7 Introduction FIN-S PECOS FIN-S Development FIN-S and libmesh Supported: Unstructured meshes Adaptive mesh refinement Adaptive time stepping Preconditioned Newton-Krylov implicit solves R. Stogner Full System Simulations April 27, / 25

8 Introduction FIN-S PECOS FIN-S Development FIN-S and libmesh Supported: Unstructured meshes Adaptive mesh refinement Adaptive time stepping Preconditioned Newton-Krylov implicit solves FIN-S or libmesh Lacked: Distributed meshes Real gas chemistry Radiating surfaces Turbulence modeling Thermal non-equilibrium Ionization R. Stogner Full System Simulations April 27, / 25

9 Introduction FIN-S PECOS FIN-S Development FIN-S and libmesh Supported: Unstructured meshes Adaptive mesh refinement Adaptive time stepping Preconditioned Newton-Krylov implicit solves FIN-S or libmesh Lacked: Distributed meshes Real gas chemistry Radiating surfaces Turbulence modeling Thermal non-equilibrium Ionization All Hypersonics Options Lacked: Ablating surfaces MMS-based verification Adjoint-based sensitivities Adjoint-based goal-oriented refinement R. Stogner Full System Simulations April 27, / 25

10 Introduction FIN-S DPLR++, FIN-S Plans Decision Factors Year 1 results: impossible without DPLR-quality physics R. Stogner Full System Simulations April 27, / 25

11 Introduction FIN-S DPLR++, FIN-S Plans Decision Factors Year 1 results: impossible without DPLR-quality physics Year 5 results: inferior without FIN-S-quality numerics R. Stogner Full System Simulations April 27, / 25

12 Introduction FIN-S DPLR++, FIN-S Plans Decision Factors Year 1 results: impossible without DPLR-quality physics Year 5 results: inferior without FIN-S-quality numerics Planning Initial decision: Demonstrate FIN-S ablating capsule Fall 2011 FIN-S forward uncertainty propagation Spring 2012 R. Stogner Full System Simulations April 27, / 25

13 Introduction FIN-S DPLR++, FIN-S Plans Decision Factors Year 1 results: impossible without DPLR-quality physics Year 5 results: inferior without FIN-S-quality numerics Planning Initial decision: Demonstrate FIN-S ablating capsule Fall 2011 FIN-S forward uncertainty propagation Spring 2012 Review recommendation: Make every effort to perform the Spring 2011 simulation with the FIN-S code R. Stogner Full System Simulations April 27, / 25

14 Introduction FIN-S DPLR++, FIN-S Plans Decision Factors Year 1 results: impossible without DPLR-quality physics Year 5 results: inferior without FIN-S-quality numerics Planning Initial decision: Demonstrate FIN-S ablating capsule Fall 2011 FIN-S forward uncertainty propagation Spring 2012 Review recommendation: Make every effort to perform the Spring 2011 simulation with the FIN-S code Current status: FIN-S ablating capsule results now FIN-S forward uncertainty propagation Spring 2011 R. Stogner Full System Simulations April 27, / 25

15 Modeling Outline 1 Introduction FIN-S 2 Modeling Aerothermochemistry Ablation 3 Software Formulation Robustness Verification Scalability Optimization 4 Results Convergence R. Stogner Full System Simulations April 27, / 25

16 Modeling New Physics Modeling Models Newly Completed Thermal non-equilibrium viscous flows Radiative equilibrium boundary conditions Quasi-steady ablation boundary conditions Spalart-Allmaras turbulence Spalart-Allmaras (Catris and Aupoix form) R. Stogner Full System Simulations April 27, / 25

17 Modeling New Physics Modeling Models Newly Completed Thermal non-equilibrium viscous flows Radiative equilibrium boundary conditions Quasi-steady ablation boundary conditions Spalart-Allmaras turbulence Spalart-Allmaras (Catris and Aupoix form) Models in Progress Ionized flows Improved transport flux models k-ɛ, k-ω turbulence Menter s SST R. Stogner Full System Simulations April 27, / 25

18 Modeling Aerothermochemistry Thermal Nonequilibrium New Development Framework for N-Energy conserved variables 2-Temperature (T tr, T ve ) physics completed Vibrational/electronic energy transport added R. Stogner Full System Simulations April 27, / 25

19 Modeling Aerothermochemistry Thermal Nonequilibrium Verification Initial qualitative behavior correct 1-species laminar MMS testing Convergence testing: turbulence interaction bugfix Code-to-code testing: chemistry interaction bugfix ρ =10-2 kg/m 3 ρ =10-3 kg/m 3 ρ =10-4 kg/m 3 Temperature (K) x (m) R. Stogner Full System Simulations April 27, / solid: T dashed: T V

20 Modeling Aerothermochemistry Ionization Previous FSS Runs ISS return trajectory speeds ( 6.5 km/sec) 13 species air+carbon R. Stogner Full System Simulations April 27, / 25

21 Modeling Aerothermochemistry Ionization Previous FSS Runs ISS return trajectory speeds ( 6.5 km/sec) 13 species air+carbon Final FSS Goal Unified Lunar Mars return trajectory speeds (7.5+ km/sec) 20 species ionized air+carbon chemistry Locally neutral gas (no MHD!) Gupta, Yos transport R. Stogner Full System Simulations April 27, / 25

22 Modeling Ablation Fully Implicit Ablation Coupling Quasi-Steady Ablation Boundary conditions: Nonlinear Robin BC for masses, energies Nonlinear Dirichlet BC for momentum Standard FEM weak source term for Robin BC Penalty formulation for nonlinear Dirichlet BC J i + ρv w C i = Ñi(C i, T ) k T n h i (T )J i +ṁ ch h ch(t ) ρv w h(t ) +α q r σɛt 4 +ρv w h o f,v (T ref ) = 0 ρv w = Ñ i (C i, T w ) = ṁ ch R. Stogner Full System Simulations April 27, / 25

23 Modeling Ablation Fully Implicit Ablation Coupling Quasi-Steady Ablation Boundary conditions: Nonlinear Robin BC for masses, energies Nonlinear Dirichlet BC for momentum Standard FEM weak source term for Robin BC Penalty formulation for nonlinear Dirichlet BC J i + ρv w C i = Ñi(C i, T ) k T n h i (T )J i +ṁ ch h ch(t ) ρv w h(t ) +α q r σɛt 4 +ρv w h o f,v (T ref ) = 0 ρv w = Ñ i (C i, T w ) = ṁ ch Convergence Implications Stability, time step restrictions as good as flow alone Successful two-phase spinup: isothermal ablating R. Stogner Full System Simulations April 27, / 25

24 Software Outline 1 Introduction FIN-S 2 Modeling Aerothermochemistry Ablation 3 Software Formulation Robustness Verification Scalability Optimization 4 Results Convergence R. Stogner Full System Simulations April 27, / 25

25 Software Formulation Stabilization Work Convection Stabilization Consistent τ application Mass diffusion terms in τ New SUPG length scale options DCO disabling for subsonic, verification use Reaction Stabilization Mass lumping Primitive vs Conserved source term interpolation R. Stogner Full System Simulations April 27, / 25

26 Software Robustness Algorithm Developments Application Options Physics-upgrading restarts Ablation chemistry subsets Multiple simulations for QUESO sampling libmesh, PETSc nonlinear solver options Time Integration Smooth adaptive time step control Exceptions thrown on even extreme failure cases Backtracking upon failure R. Stogner Full System Simulations April 27, / 25

27 Software Verification Code Verification MASA Manufactured Solutions Now using MASA 0.30 release solutions for: Euler Laminar Navier-Stokes Turbulent Navier-Stokes Progressing to tests for: Chemical nonequilibrium (dissociating N 2 ) Thermal nonequilibrium (T tr,t ve ) Sublimating carbon boundary Chemay Kinetics Integration Standalone PECOS chemistry library Independent chemistry verification test suite R. Stogner Full System Simulations April 27, / 25

28 Software Verification Solution Verification Refinement Studies Uniform comparison to fine grids AMR comparison to fine uniform T (K) Stagnation Line Temperature Profile Adaptive (x/r) Error Estimation Adjoint-based QoI indicator R. Stogner Full System Simulations April 27, / 25

29 Software Scalability Current Scalability Results Flow-only Scaling Weak scaling: near ideal Strong scaling: sharp plateau 10 3 Ideal Scaled-Size (Weak) Scaling Fixed-Size (Strong) Scaling 10 2 Speedup Number of Processor Cores R. Stogner Full System Simulations April 27, / 25

30 Software Scalability Current Scalability Results Flow-only Scaling 10 2 Weak scaling: near ideal Strong scaling: sharp plateau 10 3 Ideal Scaled-Size (Weak) Scaling Fixed-Size (Strong) Scaling Multiphysics Capsule Scaling Undefined weak scaling: O ( N 2) boundary Strong scaling to 72, 192 cores Fine grid memory-limited? 10 3 Ideal Scaling Coarse Capsule Scaling Medium Capsule Scaling Parallel Scaling of Ablating Capsule Simulations Speedup 10 1 Relative Scaled Speed Number of Processor Cores Number of Processor Cores R. Stogner Full System Simulations April 27, / 25

31 Software Scalability libmesh Scalability SerialMesh vs. ParallelMesh Serial mesh memory usage prohibitive for fine meshes (millions of DoFs) Parallel mesh scaling extends much further (hundreds of millions of DoFs) R. Stogner Full System Simulations April 27, / 25

32 Software Scalability libmesh Scalability SerialMesh vs. ParallelMesh Serial mesh memory usage prohibitive for fine meshes (millions of DoFs) Parallel mesh scaling extends much further (hundreds of millions of DoFs) Hybrid MPI+Threads libmesh support: TBB interface, utility classes, threaded library code PETSc support? HYPRE support R. Stogner Full System Simulations April 27, / 25

33 Software Scalability FIN-S Scalability Hybrid MPI+Threads Assembly loops all multithreaded Thread-safety problems in Cantera Correctable in Chemay interface R. Stogner Full System Simulations April 27, / 25

34 Software Scalability FIN-S Scalability Hybrid MPI+Threads Assembly loops all multithreaded Thread-safety problems in Cantera Correctable in Chemay interface ParallelMesh Usage Computation, restart I/O is parallelized Visualization I/O options require (automatic) temporary serialization Turbulence DistanceFunction isn t ParallelMesh safe! R. Stogner Full System Simulations April 27, / 25

35 Software Scalability FIN-S Scalability Hybrid MPI+Threads Assembly loops all multithreaded Thread-safety problems in Cantera Correctable in Chemay interface ParallelMesh Usage Computation, restart I/O is parallelized Visualization I/O options require (automatic) temporary serialization Turbulence DistanceFunction isn t ParallelMesh safe! Load Balancing PrimitiveVariables inversion cost is variable Partition size should compensate R. Stogner Full System Simulations April 27, / 25

36 Software Optimization Optimization Profiling Callgrind Interpreted code execution Call graph generation, exploration Ideal for high-level optimization R. Stogner Full System Simulations April 27, / 25

37 Software Optimization Optimization Profiling Callgrind Interpreted code execution Call graph generation, exploration Ideal for high-level optimization Oprofile Sampling from CPU interrupts Line-by-line results Ideal for low-level optimization R. Stogner Full System Simulations April 27, / 25

38 Software Optimization Optimization Examples High-Level Optimizations Simultaneous residual/jacobian calculations Use of nodally cached primitive variables Optionally avoid final Newton residual check Avoid redundant assembly in adjoint solve Reuse preconditioner in adjoint solve R. Stogner Full System Simulations April 27, / 25

39 Software Optimization Optimization Examples High-Level Optimizations Simultaneous residual/jacobian calculations Use of nodally cached primitive variables Optionally avoid final Newton residual check Avoid redundant assembly in adjoint solve Reuse preconditioner in adjoint solve Low-Level Optimizations Cached config variables Eigen (now Eigen 3) Dense Linear Algebra Simultaneous property/derivative calculations Formula refactoring MFLOPS eigen3 GOTO2 INTEL_MKL eigen2 ATLAS gmm ublas matrix matrix product matrix size R. Stogner Full System Simulations April 27, / 25

40 Results Outline 1 Introduction FIN-S 2 Modeling Aerothermochemistry Ablation 3 Software Formulation Robustness Verification Scalability Optimization 4 Results Convergence R. Stogner Full System Simulations April 27, / 25

41 Results Convergence Ablating Capsule Example ISS return trajectory, chemical nonequilibrium, ablating boundary Results Dead start isothermal ablating convergence time steps instead of 25, ,000 Order of magnitude wall clock speedup R. Stogner Full System Simulations April 27, / 25

42 Results Convergence Convergence Example ISS return, chemical nonequilibrium, radiative equilibrium boundary Time/Convergence Convergence of Radiative Equilibrium Capsule Simulation Isothermal Time Step Size RadEq Time Step Size Isothermal Simulation Time RadEq Simulation Time Isothermal Transient Residual RadEq Transient Residual Isothermal Wall Clock Time RadEq Wall Clock Time Number of Time Steps Results Constant wall time per time step After shock set up, convergence accelerates Rapid boundary layer redevelopment relative residual reduction Stability = final convergence rate R. Stogner Full System Simulations April 27, / 25

43 Results Convergence Thank you! Questions? R. Stogner Full System Simulations April 27, / 25