The Capsaicin Project

Transcription

1 A status report following the completion of the Q2FY05 LANL L2 milestone CCS-4 Staff Activity Seminar 2005 April 29

2 Who is Capsaicin? The Capsaicin team has 5 team members. Kelly Thompson (project lead) Jim Warsa Kent Budge Jae Chang Alejandro Gonzalez-Aller Capsaicin is a part of a larger deterministic radiation transport team in CCS April 29 Slide 2/30

3 What is Capsaicin? Its what makes chili hot! A colorless irritant phenolic amide, C 18 H 27 NO 3, that is found in various capsicums and that gives hot peppers their hotness [Webster s Dictionary]. A deterministic radiation transport effort in CCS-4 Develop software tools used for rapid production of radiative transfer simulation software. Modern software development techniques (SE, SQA & Verification). Conduct research and development of new algorithms for improved simulation capability. Robust and accurate discretizations. Strategies for improved parallel efficiency April 29 Slide 3/30

4 Product driven development ASC L2 Milestone completed 2005 March 30. Demonstrate a deterministic radiative transfer simulation capability. Solve 1 st order form of transport equation 2D RZ, unstructured triangles, 2 nd order accurate FEM discretization Discrete ordinates, time dependent. Robust and efficient parallel solution algorithm Modern SE and SQA approaches Verification of software must be repeatable and on-demand. Realized in the released software SERRANO. Additionally: provides Multigroup, XY, 3T capabilities. A full report is available as LA-UR Inception ROCOTILLO ANCHO SERRANO (no acceleration) SERRANO (with TSA; IPBJ) 2005 April 29 Slide 4/30 ANAHEIM (Steady State) MACHO (brick meshes)

5 Product driven development Neutrino astrophysics LDRD Provide capability for quadrilateral meshes. Study how to simulate neutrino transport (all 6 flavors!) RadHydro R&D of un-split RadHydro simulation. Provide 1D spherical radiation transport model. Host integration and general code improvements DSA R&D, implementation. Treatments for starting directions and implicit reflection boundary. IPBJ study Use Belos instead of AztecOO as primary GMRES solver. Inception ROCOTILLO ANCHO SERRANO (no acceleration) SERRANO (with TSA; IPBJ) ANAHEIM (Steady State) MACHO (brick meshes) 2005 April 29 Slide 5/30

6 The Physics of SERRANO Solves coupled 3T equations 1 I cρκ + Ω s i I + ρ( r, t) κt ( r, ν ) I( r, Ω, ν, t) = E + ρκab( Te, ν ) + Q r, c t 4π Te ρ( r, t) CV + 4 πρκ ( ) ( ), and e ab Te + αie Te Ti = cρκae + Q e t Ti ρ( r, t) CV = α ( ), where i ie Te Ti + Q i t 1 E( r, t) = dν dωi( r, Ω, ν, t) c 0 4π 2005 April 29 Slide 6/30

7 Assumptions of SERRANO Assumes that the transport equation is valid: Photons are treated as classical, point particles (not wave-like). The spread of photon wave packets assumed to be small. No treatment for: Interference (ok as long as the photon density for a given frequency is low), Reflection, Diffraction, Refraction, Dispersion, Polarization Assumes that there is no preferred direction: Absorption opacity and Planck emission are independent of angle. Possibility of adding moving material (v/c) corrections later. Simplifying assumptions: Neglects ion and electron conduction. Assumes all scattering is isotropic. Assumes thermodynamic equilibrium Linearized Planck emission spectrum. Semi-implicit in time April 29 Slide 7/30

8 Features and Limitations of SERRANO. Additional Features: XY geometry Multigroup capability Uses SweepChunk algorithm Limitations: Fixed t size No restart capabilities Only support analytic opacities, heat capacities, etc (via Draco s cdi package). No electron heat conduction, k T April 29 Slide 8/30

9 Demonstration Problem: Zero Solution Test Start with the simple problem: Correct behavior is necessary but not sufficient for verification. Problem setup: Initial condition: I=0 Boundary conditions I b =0. No sources, vacuum b.c. Let problem cycle to ensure no unphysical introduction of energy. Correct solution after 20 time cycles April 29 Slide 9/30

10 Demonstration Problem: Linear Solution Test Properties: Steady State No scattering Reflective b.c. on left, right (1D solution). Source at bottom, vacuum at top. ( ) z E0 z = Einc H Solution is linear: 8E-16 7E-16 Computed Energy Density (XY) Analytic Energy Density Computed Energy Density (RZ) 6E-16 Energy Density (J/m^3) 5E-16 4E-16 3E-16 2E-16 1E-16 ( ) z E0 z = Einc H z-position (m) 2005 April 29 Slide 10/30

11 Demonstration Problem: Equilibration Properties: Temporal solution, t=10-11 s. Infinite medium, σ s =0, Q=0 ρ=1 g/cm 3, Grey, S 2 Reflective b.c. all sides Initial T e >0, E r =0. Solution: rt 4a Er ( t) = A(1 e ), r = cρκa 1 + β rt βa 4 Te ( t) = A + e, A Te. a β = β + 4a β = a = 1/ 4 / 4, T0 17 K Te Trad Te - Trad T em p erature (K) Absolute Difference (K ) E-11 1E E-10 2E E-11 4E-11 6E-11 8E-11 1E E E E E-10 2E-10 Time (seconds) Time (seconds) 2005 April 29 Slide 11/30

12 Demonstration Problem: Equilibration Temporal Convergence Rate Procedure: Run single time step and vary t. Compare computed to analytic solution. Error dominated by round off at small t. Error dominated by higher order terms at large t Temporal Truncation Error 2 1 Radiation Energy Density Error (erg/cm^3) Ratio of Radiation Energy Density Error Electron Temperature Error (K) Ratio of Electron Temperature Error 0 1E-16 1E-15 1E-14 1E-13 1E-12 1E-11 1E-10 TimeStep Size (sec.) 2005 April 29 Slide 12/30

13 Demonstration Problem: Spatial Truncation Error Procedure: Use the Method of Manufactured Solutions. Compare computed to analytic solution for various mesh resolutions. Problem configuration: steady state periodic solution Assume: I( x, y) = cos(2 π x)cos(2 π y) Source: µ sin(2 π x) cos(2 π y) S σ I = 2π + ξ sin(2 π y) cos(2 π x) 2005 April 29 Slide 13/30

14 Demonstration Problem: Su-Olson Grey, No-scattering Benchmark Benchmark solution: Very accurate 1D solutions are solved by use of similarity solution and numeric integrals. Analytic solution for nonequilibrium problem. 3 Assumes CV = αt e e Allows system to become linear in I and T 4. S April 29 Slide 14/30

15 Demonstration Problem: Su-Olson Grey, No-scattering Benchmark τ = 10 τ = W 1.5 τ = 3.16 V 1 τ = 3.16 diffusion transport Serrano diffusion transport Serrano 1 τ = τ = τ = τ = x τ = x 2005 April 29 Slide 15/30

16 Demonstration Problem: Su-Olson with Scattering / Multigroup With Scattering Same as the purely absorbing problem but with c=0.5 Picket Fence opacity* Requires at least 2 groups We needed 40 groups to resolve the emission spectrum. Solution in report was not fully converged in time, angle or energy u1+u diffusion transport Serrano V diffusion transport Serrano x x 2005 April 29 Slide 16/30

17 Demonstration Problem: Marshak 2B 1D, 2-Temperature wave propagation problem: Assume κ a = 100/T 3, to derive an analytic solution under a similarity transformation in the diffusion approximation. ε=10-7 / 10-9 ; S 6 quad t=2x10-12 s 2005 April 29 Slide 17/30

18 Demonstration Problem: Marshak 2B Strong Scaling Results Strong Scaling for Marshak 2B Strong Scaling for Marshak 2B Outer Iterations FS FSMS 5000 BJ BJMS BJTSA Number of Processors CPU time (seconds) FS 400 FSMS BJ 200 BJMS BJTSA Number of Processors FS iteration count is independent of number of processors. Multi-Step algorithm makes multiple applications of the transport operator per Krylov iteration April 29 Slide 18/30

19 Demonstration Problem: DogLeg Problem What do we see in the movie? Is the propagation speed correct? What happens when the wave front turns the corner? Is that an S N ray? A shadow? 2005 April 29 Slide 19/30

20 Accuracy and Robustness Accuracy Test problems demonstrate solution accuracy via MMS. Spatial discretization is 2 nd order accurate Temporal discretization is 1 st order accurate Angular discretization is 1 st order accurate Robustness Krylov solver allows for efficient solution convergence in optically thick problems where Richardson iteration stalls (FS or IPBJ). Richardson iteration is an efficient algorithm for optically thin problems where a Krylov solver may stagnate (IPBJ only) April 29 Slide 20/30

21 L2 Milestone Significant Findings Our LLD method is a robust spatial discretization in XY and RZ. Krylov iterative methods are effective for solving radiative transfer problems in thick diffusive regions. TSA, though effective, is not an efficient preconditioner for optically thick problems. The IPBJ algorithm proved to be a more efficient parallelization algorithm when compared to full sweeps. Stability issues when the problem is optically thin on a domain. Fixed by preconditioning with TSA or by using Richardson on thin domains April 29 Slide 21/30

22 R&D: lumping on 2D meshes Morel & Warsa: identified lumped equations for quadrilateral meshes: Resist negative solutions and rapidly dampen oscillations. Examined effect of lumping gradient term for non-orthogonal meshes. Lumped scheme maintains the accuracy of the original scheme. Only loose one order of accuracy for high aspect ratio cells (good for resolving boundary layers). Morel, Gonzalez-Aller and Warsa: published an lumped FEM scheme for RZ triangular meshes. Must lump the curvilinear gradient term. 2 nd order accurate in the transport regime. Yields the lumped linear-continuous FEM diffusion discretization in the thick diffusion limit. Provides good results with unresolved spatial boundary layers. Used by SERRANO April 29 Slide 22/30

23 R&D: lumping on tetrahedral meshes Published an S N spatial discretization scheme for thermal radiative transfer on tetrahedral meshes. Resists negative solutions in under-resolved highly absorbing regions. Exhibits 2 nd order accuracy with smooth and non-smooth meshes in both transport and diffusive regimes. Important result: High aspect-ratio tetrahedrons can be used to efficiently and accurately resolve boundary layers. However, limits to 1 st order accurate. Similarly, hexahedral meshes exhibit 2 nd order accuracy when fully lumped but degrade to 1 st order accurate when cells have high aspect ratios April 29 Slide 23/30

24 R&D: Inexact parallel block-jacobi Examine IPBJ preconditioned with DSA. Motivation: The intractable problem of finding optimal sweep schedule. IPBJ is a viable alternative to FS We observe better parallel efficiencies. Stagnation: IPBJ/GMRES algorithm stagnates when sub-domains are thin. Operator becomes indefinite. Mitigation We can use Richardson iteration efficiently if problem is thin. TSA preconditioning of IPBJ/GMRES provides positive definiteness. Effect of DSA preconditioning is unknown at this time April 29 Slide 24/30

25 Modern software development Success of project planning & management Tracking; Oversight; V&S; peer review; training. Product focused development and delivery; schedules Coding practices; document management; release process; code metrics. Risk management. Success of automated on-going and on-demand verification. Unit tests; application tests; Design-by-Contract tests, etc. Success of a levelized, component architecture. Successful use of existing components (Draco, Trilinos, etc.) Finding the right fit for team size/composition is important! 2005 April 29 Slide 25/30

26 Modern software development OO Levelized Component Design 2005 April 29 Slide 26/30

27 Current Development Efforts (Q3FY05) Better time step controller Support for quadrilateral meshes Use conformal triangle sub-zone meshes. Host integration DSA Integration, efficiency and effectiveness when coupled to full sweep or IPBJ algorithm. Issues concerning the differences of preconditioning GMRES or Richardson iteration. Treatments for starting directions and implicit reflection boundary April 29 Slide 27/30

28 Current Development Efforts (Q3FY05) R&D for IPBJ algorithms Optimization Trilinos-5.0 / Belos. Addition of face-connectivity. New algorithm speeds up the generation of large mesh objects April 29 Slide 28/30

29 Future Development Efforts Q4FY05 Performance tuning Enhanced verification suite Finish DSA implementation; Characterize DSA preconditioning for IPBJ and FS algorithms; Characterize DSA+TSA. Provide a restart capability FY06 or later True hybrid mesh support Tetrahedral meshes (3D) Compton Scattering Electron Conduction Quadratic FEM discretization (accelerated with linear FEM) April 29 Slide 29/30

30 Summary The Capsaicin project has demonstrated a robust and accurate 2D-RZ discrete ordinates radiation transport capability embodied in the software SERRANO. Successful simulations of standard benchmark problems and code-to-code comparisons. IPBJ/GMRES is a viable and effective approach for solving the transport equation on parallel architectures. Our implementation of TSA is effective but inefficient. GMRES is effective and efficient for solving optically thick problems April 29 Slide 30/30