HPC Particle Transport Methodologies for Simulation of Nuclear Systems

Transcription

1 HPC Particle Transport Methodologies for Simulation of Nuclear Systems Prof. Alireza Haghighat Virginia Tech Virginia Tech Transport Theory Group (VT 3 G) Director of Nuclear Engineering and Science Lab (NSEL) at Arlington Nuclear Engineering Program, Mechanical Engineering Department VT High Performance Computing Day, Blacksburg, VA, April 11,

2 Objective Particle Transport Theory Determine the expected number of particles in a phase space (d 3 rded) at time t: n( r, 3 E, ˆ, t) d rded z d 3 r dω Ω de r y x 7 independent variables: space (x, y, z), energy (E), direction (,, time (t) 2

3 Simulation Approaches Deterministic Methods Solve the Linear Boltzmann Equation (LBE) to obtain the expected flux in a phase space Statistical Monte Carlo Methods Perform particle transport experiments using random numbers (RN s) on a computer to estimate average properties of a particle in phase space 3

4 Deterministic Linear Boltzmann Equation Integro-differential form ) ˆ,, ( ) ˆ ',, ( ), ( ' ' 4 ) ( ) ˆ ',, ( ) ˆ ' ˆ, ', ( ' ' ) ˆ,, ( ), ( ) ˆ,, (. ˆ E r S E r E r d de E E r E E r d de E r E r E r f s streaming collision scattering fission Independent source 4 Ψ,, Ω,, Ω) Where, angular flux is defined by Particle speed Particle density Source depends on the unknown; requires iteration

5 Integro-differential - Solution Method Spatial variable Integrated over fine meshes using FD or FE methods Angular variable: Discrete Ordinates (Sn) method: A discrete set of directions { ˆ } and associated weights m {w m } are selected ˆ ˆ. (,, ) (, ) (,, ˆ r E r E r E ) q( r, E, ˆ ) m m m, g, A V ijk d 3 r V m, g ijk ( r ) 0 0 m m X V V Energy variable Integrate over energy intervals to prepare multigroup cross sections, σ g 1 1 Example: a small size shield: 100x100x100 (space)x 80 (directions) x 50 (groups) x 8 byte/word = 32 GB 5

6 Monte Carlo (MC) Methods Perform an experiment on a computer; exact simulation of a physical process, e.g., a shield Path length r ln t Type of collision s Scattering angle 2 1 t (isotropic scattering) 0 Fixed source S (r, E, Ω) absorbed absorbed Tally (count) absorbed Issue Precise average values; i.e., small relative uncertainty, x R, require large x computation time x Therefore, Variance Reduction techniques are needed for real world problems! 6

7 Deterministic (Det.) vs. Monte Carlo (MC) Item Det. MC Geometry Discrete/ Exact Exact Energy treatment cross section Direction Discrete Discrete/ Truncated series Exact Exact Input preparation Difficult simple Computer memory Large Small Computer time Small Large Numerical issues Convergence Statistical uncertainty Amount of information Large Limited Variance Reduction ---- Unknown parameters Parallel computing Complex Trivial 7

8 Why not? Hybrid methods? MC Det. (for automated variance reduction) Det. Det. (use of region dependent technique) Parallel processing? Det. memory partitioning, and domain decomposition MC embarrassingly parallel 8

9 VT 3 G Vision Development of accurate particle transport methods for real time simulation of nuclear systems (power, security & medicine) Year Methodology Computer code system Wall clock time 2015 MRT TITAN IR 2014 MRT RAPID 2013 MRT AIMS MRT Hybrid MC det. (AVR) Hybrid det. det. INSPCT s ADIES ( ) TITAN (n, ϒ) 1997 Hybrid MC det. (automated VR AVR) A 3 MCNP (n, ϒ) 1996 Parallel (3 D) PENTRAN (n, ϒ) 1992 Vector & parallel (2 D) 1989 Parallel processing (1 D) 1986 Vector processing (1 D) 9

10 PENTRAN (Parallel Environment Neutral-particle TRANsport) Code System (G. Sjoden and A. Haghighat, 1996) Pre-processing PENMSH-XP (prepares mesh, source, and material distributions) Transport Calculation Domain Decomposition FORTRAN 90 with MPI library Hybrid spatial, angular, energy decomposition algorithms Partition memory; parallel I/O Post-processing PENPRL (3-D linear interpolation for comparison with experimental data) 10 10

11 PENTRAN performance for the Box in the Box test problem water Neutron source Run Scalability Results using PENTRAN on a Constant Size Problem (BP4) All results converged to a 1.0E-5 relative tolerance. PENTRAN Decomposition Strategy # Procs Angle # Procs Group # Procs Space #Total Procs P Sp= Speedup Factor Ep= Efficiency (Sp /P, %) Comm Overhead: % w allclock 1 Angular * Group Spatial Angular Angular-Group Angular Spatial Angular-Group Angular-Spatial Angular Angular-Spatial Angular-Group-Spatial Angular-Group Angular-Spatial Spatial * Average of 2-batch and interactive runs at CNSF 11

12 PENTRAN Performance Amdahl s law Pf =

13 Benchmarking Application of PENTRAN TM Kobayashi 3-D Benchmarks VENUS-3 Benchmark facility in SCK.CEN, Belgium C5G7 criticality benchmark Real-world problems BWR core-shroud and internals Pulsed Gamma Neutron Activation Analysis (PGNAA) device for assaying of waste X-Ray room CT Scan Time-of-Flight (TOF) for cross section measurement Simulation of spent fuel storage CASK 13

14 Venus-3 experimental benchmark (performed in ) ~85,000 cells 26 Groups P3-S8 Discrete Ordinates 32 IBM SP2 Procs, 84 min Parallel Performance on IBM SP2 95% C/E values +/-10%; 5% within +/-15% Case Number of processors Decomposition algorithms (A/G/S) 1 Wall-clock Speedup Efficiency time (min) 2 (%) 1 4 4/1/ /1/ /1/ /1/ (A/G/S) refers to the number of angular, group, and spatial sub-domains. 2 Time is obtained in a BATCH mode. 14

15 PENTRAN BWR Core shroud Modeling (performed in 1999) 67 G, S8/P3; 265,264 mesh 48 Procs (8 A/6 S) on IBM SP2: Within 5-15% A 3 MCNP (continuous energy), Wall-clock time = 12 hours Speedup = 10 (compared to a standard code) Scalability Case No. of Directions No. of Processors Decomposition (A/G/S) 1 Wallclock/iteration (s) /1/ /1/ /1/ /1/ (A/G/S) refers to the number of Angular, Group, and Spatial subdomains. 15

16 PENTRAN Simulation Storage Cask CASK library (22n, 18g) 17 Materials 318,426 fine meshes (1000 coarse meshes) (40 z-levels) P 3, S 12 (168 directions) 1.48 GB per processor 8 processors (~12 GB Total) Model # CPU Dose Ratio Run Time (hrs) PENTRAN Large Model

17 TITAN Hybrid Sn & CM Algorithm (C. Yi, A. Haghighat, 2004) Partition the problem into coarse meshes and allow for the use of different methods in different coarse meshes; Discrete ordinates (Sn) Solver Characteristic Method (CM) solver TITAN is written in Fortran 90 (with some features in Fortran 2003 standard, such as dynamic memory allocation and object oriented23) and MPI library 17

18 TITAN Sn CM Algorithm Sn out gink gink CM in gis Q ink gin gisink ginke (1 e ) Q gi in out gin gink gink s gi ink gi A B E.g., LD Scheme; 2 x o u t x i n 2 y o u t y i n 2 z o u t z i n ( n ) g ijk ( n ) g ijk ( n ) g ijk gin g ink ( Aink sink ) Aink gink Q k gin 1 k ( A s ) ( A s ) k ink ink gi gi ink ink k P B AB A 18

19 TITAN Benchmarking and Application OECD/NEA Benchmarks C5G7 MOX Kobayashi 3 D parameter space VENUS 2 Applications Adjoint calculation for the AIMS active interrogation simulation tool mpower reactor core and external modeling Modeling of a penetration duct in a nuclear reactor Benchmarking the multigroup SDM (subgroup decomposition method) algorithm (developed by Georgia Tech) Medical applications Modeling a CT machine Developed an image reconstruction algorithm, TITAN IR 19

20 TITAN simulation of CT scan 2.500E E-03 Flux (#/cm 2 s) 1.500E E E-04 Case 1: MCNP ref Case 4: Sn Pn-Tn S20 t11.2 t11.2 Case 5: hybrid Pn-Tn S20 t11.2 t E Detector Mesh (cm) CM algorithm z (cm) Method Time Speedup MCNP5 (σ<1%) 3510 sec 1.0 TITAN (P N T N S 100 ) 14 sec 249 y (cm) x (cm) Sn algorithm 20

21 A 3 MCNP (Automated Adjoint Accelerated MCNP) (John Wagner and Alireza Haghighat, 1997) A 3 MCNP performs variance reduction technique based on the CADIS (Consistent Adjoint Driven Importance Sampling) methodology by using an approximate deterministic importance function A 3 MCNP input (MCNP input + A 3 MCNP cards) TORT input A 3 MCNP GIP input Step 1 mesh distribution material composition input files TORT GIP multi-group cross sections S N adjoint function A 3 MCNP Step 2 VR parameters A 3 MCNP Step 3 non-analog MC Calculation 21

22 CADIS (Consistent Adjoint Driven Importance Sampling) methodology (John Wagner and Alireza Haghighat, 1997) Uses a 3-D S N importance function distribution for source biasing transport biasing in a consistent manner*, within the weight-window ( r, E) technique. Source biasing Biased source Transport biasing Splitting/rouletting qˆ( p) ( p) q( p) ( p) q( p) dp p ( p) q( p) R ( p) If ( p') <1, particles are processed through the Russian roulette, Otherwise, particles are split 22

23 PWR Cavity dosimetry Application of A 3 MCNP For determination of neutron interaction rates with dosimetry materials placed at the reactor cavity, and estimation of fluence at the reactor pressure vessel BWR Core Shroud Storage cask Determination of neutron and gamma fields at the reactor pressure vessel Determination of neutron and gamma fields at the cask s outside surface 23

24 A 3 MCNP simulation of Storage Cask Objective: Estimation of Neutron and gamma dose on the cask surface Speedup of A 3 MCNP vs. MCNP Problem Size: 180 x 180 x 840 cm 3 A Deep penetration problem 24

25 Remarks Even Fast particle transport codes, with parallel and hybrid algorithms, are not able to obtain solutions in real time! 25

26 Development of Transport Formulations for Real Time Applications Physics Based transport methodologies are needed: Developed Based on problem physics partition a problem into stages (subproblems), For each stage employ response method, adjoint function methodology, or develop/identify a fast algorithm Pre calculate response function or adjoint function using an accurate and fast transport code Solve a linear system of equations to couple all the stages 26

27 Examples for Nondestructive testing: Optimization of the Westinghouse s PGNNA active interrogation system for detection of RCRA (Resource Conversation and Recovery Act) (e.g., lead, mercury, cadmium) in waste drums (partial implementation of MRT; 1999) Nuclear Safeguards: Monitoring of spent fuel pools for detection of fuel diversion (2009) (funded by LLNL) Nuclear nonproliferation: Active interrogation of cargo containers for simulation of special nuclear materials (SNMs) (2013) (in collaboration with GaTech) Spent fuel safety and security: Real time simulation of spent fuel pools for determination of eigenvalue, subcritical multiplication, and material identification (partly funded by I 2 S project, led by GaTech) (Filing for a patent2014) Image reconstruction for SPECT (Single Photon Emission Computed Tomography): Real time simulation of an SPECT device for generation of project images using an MRT methodology and Maximum Likelihood Estimation Maximization (MLEM) (filed for a patent, 2015) 27

28 INSPCT s (William Walters & Alireza Haghighat, 2009) Inspection of a spent fuel pool : Online Calculation of detector response (R): IAEA inspection of Atucha 1 pool A A R n S n n Neutron source Adjoint (Importance) function Source (S = S intrinsic + S subcritical Multiplication ) Stage 1 Intrinsic Source Spontaneous fission & (, n) from fuel burnup calculation (ORIGEN ARP) (Created a database) Stage 2 Subcritical Multiplication (Hybrid method) Simplified fission matrix (FM) method Use MCNP Monte Carlo to obtain a i,j for each pool type (Created a database for coef. a ij ) Multiplication Source Strength (#/source) F i Multiplication Source by Assembly (Fission Matrix) S4 S5S S1 S2S Assembly Position (x,y) N i1 a i, j ( F j S int. j ) Safeguards Adjoint function Stage 3 Is obtained using the PENTRAN transport code (Created a database for multigroup adjoint for different lattice sizes)

29 INSPCT S (Inspection of Nuclear Spent fuel Pool Computing Tool Spreadsheet) INSPCT S checks for fuel diversion; timing <1 s INPUT OUTPUT src file C:\Users\ali\Documents\haghD\ufttg\LLNL\INSPCT-s\se.dsrc COLUMNS 8 fm file C:\Users\ali\Documents\haghD\ufttg\LLNL\INSPCT-s\seResponse Tolerance Detector Normalization ROWS 6 imp file C:\Users\ali\Documents\haghD\ufttg\LLNL\INSPCT-s\se 15.00% 5.28E-10 run Burnup Independent Source (x,y) (x,y) E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E Cooling time Fission Source (x,y) (x,y) E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E Response (experimental) Response(Calculated) (x,y) (x,y) Response Difference (x,y) % % % % %

30 Active Interrogation for Monitoring Special nuclear materials (AIMS) (K. Royston, W. Walters & A. Haghighat, 2013) Detection of SNM in a cargo container Developed a multi stage responsefunction algorithm Gamma Detector HEU g n (x, y, z) F g n (x x 0 ) 2 (y y 0 ) 2 (z z 0 ) 2 ( r, E, ˆ ) S d g n Detector modeling: IFLEX by Georgia Tech Sample Results Computation Time for 5 Scanning Locations AIMS for all HEU positions (serial) MCNP5* (8 processors) HEU position 0or 5 HEU position 1 HEU position 2, 3, or hours 495 hours 254 hours 63 hours 30

31 RAPID (Real time Analysis of spent nuclear fuel Pool for In Situ Detection) (filing for a patent) (W. Walters, A. Haghighat, N. Roskoff, 2014) RAPID uses a MRT methodology based on the Fission matrix technique; it calculates eigenvalue, subcritical multiplication, and pinwise, axially dependent fission density throughout a pool Sample Cases Assembly wise fission density All Case Results *total pre computation time for all FM : 4600 minutes 31 31

32 RAPID Results: 3 D Fission Density Visualization Y LEVEL ANIMATION Z LEVEL ANIMATION 32

33 TITAN-IR (TITAN deterministic Image Reconstruction for SPECT) (Katherine Royston and Alireza Haghighat, 2015) Single Photon Emission Computed Tomography (SPECT): radiopharmaceutical injected into patient 2 D projection images acquired at different angles around the patient Projection images reconstructed to form 3 D radionuclide distribution Accuracy of flux distribution, back of collimator TITAN IR TITAN includes a hybrid formulation with Sn method with fictitious quadrature and ray tracing for modeling the collimator An iterative technique based on Maximumlikelihood expectation maximization (ML EM) for image reconstruction using TITAN for forward projection step Filed for patent (VTIP disclosure number) Reference Solution Projection Angle Detector Element Sinogram Simulation of two collimators Collimator MCNP* TITAN IR* Speed up LEGP 46.9 hrs 132 sec 1353 LEHS 21.4 hrs 14 sec 5503 *on 8 processors Reconstructed Images: Iteration 0 Iteration 1 Iteration 5 Iteration

34 DEMONSTRATION Jaszczak Phantom Reconstruction with TITAN IR (disclosure#: VTIP ) Reconstruction of noiseless SIMIND data with no collimator blur Computation time for 40 iterations Parallel calculation with 16 processors on dedicated computer cluster: 23.6 sec Serial calculation on MacBook Pro:175.6 sec 34

35 Conclusions The MRT methodology (that is a physics based approach) enables real time simulation: This requires careful understanding of the problem physics, and the use of response methods, adjoint function methodology, or development problem dependent formulations; Multiple MRT calculations can be run in parallel for sensitivity studies and uncertainty quantification using parallel computers For pre calculation of response and adjoint functions can be performed in parallel, Advanced codes such as parallel and hybrid Monte Carlo with automated variance reduction (e.g., A 3 MCNP) and parallel and hybrid deterministic (e.g., PENTRAN, TITAN) should be used 35

36 Thanks! Questions? 36