A methodology for the rigorous verification of plasma simulation codes

Size: px

Start display at page:

Download "A methodology for the rigorous verification of plasma simulation codes"

Phebe Willis
5 years ago
Views:

A methodology for the rigorous verification of plasma simulation codes Fabio Riva P. Ricci, C. Beadle, F.D. Halpern, S. Jolliet, J. Loizu, J. Morales, A. Mosetto, P. Paruta, C.

1 A methodology for the rigorous verification of plasma simulation codes Fabio Riva P. Ricci, C. Beadle, F.D. Halpern, S. Jolliet, J. Loizu, J. Morales, A. Mosetto, P. Paruta, C. Wersal École Polytechnique Fédérale de Lausanne (EPFL), Swiss Plasma Center (SPC), Switzerland 58th Annual Meeting of the APS Division of Plasma Physics, San Jose, California, November 4 th 2016

2 1 Verification Procedures How to rigorously ensure that a simulation code is bug-free? Code Verification How to estimate the numerical uncertainty affecting simulation results? Solution Verification

3 2 1. Simple tests Code Verification Techniques Energy conservation, convergence (without a known exact solution) 2. Code-to-code comparison (benchmarking) NOT RIGOROUS Example: Cyclone test case 3. Convergence tests Do results converge to the exact solution? 4. Order of accuracy tests Do results converge to the exact solution at the expected rate? RIGOROUS, but require analytical solution

4 1. Simple tests Code Verification Techniques Energy conservation, convergence (without a known exact solution) 2. Code-to-code comparison (benchmarking) NOT RIGOROUS Example: Cyclone test case 3. Convergence tests Do results converge to the exact solution? 4. Order of accuracy tests Do results converge to the exact solution at the expected rate? RIGOROUS, but require analytical solution The only procedure ensuring both convergence and correct numerical implementation 2

5 3 Order of accuracy test Model: Solve, with discretization parameter Compute the numerical error

6 3 Order of accuracy test Model: Solve, with discretization parameter Compute the numerical error If, with the order of accuracy, the code is verified

7 4 Method of Manufactured Solutions Solve Model: but unknown

8 4 Method of Manufactured Solutions Solve Model: but unknown MMS developed by CFD community for verifying codes based on finite difference schemes [Roache et al., AIAA J. (1984); Oberkampf et al., AIAA J. (1998)]

9 4 Method of Manufactured Solutions Solve Model: but unknown Method of Manufactured Solutions (MMS): 1. Choose and compute 2. Define 3. Solve 4. Obtain

10 4 Method of Manufactured Solutions Solve Model: but unknown Method of Manufactured Solutions (MMS): 1. Choose and compute 2. Define 3. Solve 4. Obtain If the code is verified

11 Method of Manufactured Solutions Model: unknown Solve but Method of Manufactured Solutions (MMS): 1. Choose and compute 2. Define 3. Solve 4. Obtain While arbitrary, numerical error should excite all terms in equations and ensure no dominating component in 4

12 5 First MMS plasma simulation code verification GBS: 3D fluid code used to simulate SOL plasma turbulence [Ricci et al., PPCF (2012)] Drift-reduced Braginskii equations: Continuity equation: Numerical scheme: RK4 for time integration 2 nd order finite differences for spatial derivatives Arakawa scheme for advection terms

13 6 Verification of GBS [Riva et al., PoP (2014)] Choose Compute Choose Define Obtain for Compute

14 6 Verification of GBS [Riva et al., PoP (2014)] Choose Compute Choose Define Obtain for Compute

15 6 Verification of GBS [Riva et al., PoP (2014)] Choose Compute Choose Define Obtain for Compute Refine the grid Obtain for

16 6 Verification of GBS [Riva et al., PoP (2014)] Choose Compute Choose Define Obtain for Compute Refine the grid Obtain for Verify that

17 6 Verification of GBS [Riva et al., PoP (2014)] Choose Compute Choose Define Obtain for Compute Refine the grid Obtain for Verify that Define the observed order of accuracy

18 Verification of GBS [Riva et al., PoP (2014)] Choose Compute Choose Define Obtain for Compute Refine the grid Obtain for Verify that Define the observed order of accuracy Verify that 6

19 Verification of GBS [Riva et al., PoP (2014)] Choose Compute Choose Define Obtain for Compute Refine the grid Obtain for Verify that Define the observed order of accuracy Verify that 6

20 Choose Compute GBS is verified! Choose Define Obtain for Compute Refine the grid Obtain for Verify that Define the observed order of accuracy Verification of GBS [Riva et al., PoP (2014)] First application of MMS for the verification of a plasma simulation code based on finite difference schemes [Riva et al., PoP (2014)] MMS now routinely used to verify plasma turbulence codes [Tamain et al., JCP (2016); Dudson et al., PoP (2016); ] Verify that 6

21 Particle-In-Cell (PIC) codes [Daughton et al., Nature (2011)] [Fasoli et al., Nature (2016)] [Gordon et al., PRL (2008)] 7

22 8 The PIC algorithm A simple model:

23 8 The PIC algorithm A simple model: Introduce markers (superparticles) and approximate with satisfying equations of motion

24 8 The PIC algorithm A simple model: Introduce markers (superparticles) and approximate with satisfying equations of motion randomly generated numerical results affected by statistical uncertainty

25 9 The modified model: MMS for a PIC simulation code with randomly generated statistical uncertainty on

26 9 The modified model: MMS for a PIC simulation code with Markers randomly generated statistical uncertainty on

27 9 The modified model: MMS for a PIC simulation code How to compare with? with How to account for the statistical uncertainty?

28 Cumulative distribution function 10

29 10 Cumulative distribution function CDF = EDF

30 10 Cumulative distribution function CDF = EDF

31 11 Kolmogorov Smirnov statistic Distance between and defined as EDF CDF

32 Kolmogorov Smirnov statistic 11

33 Kolmogorov Smirnov statistic 11

34 Kolmogorov Smirnov statistic 11

35 Kolmogorov Smirnov statistic [Kolmogorov, G. Ist. Ital. Attuari. (1933); Smirnov, Ann. Math. Stat. (1948)] 11

36 How to generalize to 2D case? 12

37 12 How to generalize to 2D case? CDF = EDF

38 12 How to generalize to 2D case? CDF EDF

39 12 How to generalize to 2D case? CDF EDF

40 12 How to generalize to 2D case? CDF [Peacock, MNRAS (1983)] EDF

41 12 How to generalize to 2D case? CDF How to compute the supremum? EDF

42 13 How to compute? EDF EDF CDF

43 13 How to compute? EDF EDF CDF

44 13 How to compute? EDF EDF CDF Not defined for Extremely demanding - Brute force: - Range counting-tree:

45 14 How we overcome Peakock issues? Generalized to non constant Investigated other definitions of EDF - Consider only marker CDF coordinates, k-d trees: - Distribute sampling points with Monte-Carlo Method - Compute distance only at the boundaries EDF [Riva et al., to be submitted to PoP]

46 15 The PIC simulation code Model equations: Interpolation scheme ( ): first-order weighting (CIC PIC) Poisson solver ( ): second order centered finite differences Time integration ( ): Leapfrog integration scheme

47 15 The PIC simulation code Model equations: Expected numerical error: Interpolation scheme ( ): first-order weighting (CIC PIC) Poisson solver ( ): second order centered finite differences Time integration ( ): Leapfrog integration scheme

48 16 Results: PIC code verification Choose Compute Choose Define Compute for different Verify Notice: is affected by statistical uncertainty Repeat simulations with different random number generator seeds

49 16 Results: PIC code verification Choose Compute Choose Define Compute for different Verify

50 16 Results: PIC code verification Choose Compute Choose Define Compute for different Verify The PIC simulation code is verified! First PIC code verification with MMS [Riva et al., to be submitted to PoP]

51 17 Verification Procedures How to rigorously ensure that a simulation code is bug-free? Code Verification How to estimate the numerical uncertainty affecting simulation results? Solution Verification

52 18 Sources of numerical uncertainties 1. Round-off Finite number of digits 2. Iterative schemes Termination with finite residual 3. Finite statistics E.g. a finite number of markers in representing a distribution function 4. Discretization Grids with finite resolution 5. Post-processing tools Evaluating observables from simulation results

53 18 Sources of numerical uncertainties 1. Round-off Finite number of digits 2. Iterative schemes Termination with finite residual 3. Finite statistics E.g. a finite number of markers in representing a distribution function 4. Discretization Grids with finite resolution 5. Post-processing tools Evaluating observables from simulation results

54 19 Two-stream instability Linear growth rate and its uncertainty?

55 20 Post-processing uncertainty Choose and perform a simulation

56 20 Post-processing uncertainty Choose Get and perform a simulation

57 20 Post-processing uncertainty Choose Get and perform a simulation

58 20 Post-processing uncertainty Choose Get and perform a simulation Post-processing uncertainty

59 21 Statistical uncertainty Choose Repeat with different seeds and perform a simulation

60 21 Statistical uncertainty Choose Repeat with different seeds and perform a simulation Statistical uncertainty

61 22 Discretization uncertainty Choose Repeat with different and perform a simulation

62 22 Discretization uncertainty Choose Repeat with different and perform a simulation Use of high order estimate (Richardson extrapolation)

63 22 Discretization uncertainty Choose Repeat with different and perform a simulation Use of high order estimate (Richardson extrapolation) Discretization uncertainty

64 23 Numerical uncertainty Numerical uncertainty = post-processing + statistical + discretization

65 23 Numerical uncertainty Numerical uncertainty = post-processing + statistical + discretization

66 24 Conclusions We provided rigorous methodologies to verify plasma simulations, a crucial issue in plasma physics MMS is a methodology now routinely used to rigorously verify plasma simulation codes based on finite differences schemes Overcoming the difficulty of comparing distribution functions with markers affected by statistical noise, we now generalized MMS to PIC codes verification We provided a methodology to rigorously estimate the uncertainties affecting simulation results due to finite statistics and discretization

67 25 Open questions MMS with shocks and discontinuities MMS for simulation codes involving adaptive mesh refinements Uncertainty propagation

68 Conclusions We provided rigorous methodologies to verify plasma simulations, a crucial issue in plasma physics MMS is a methodology now routinely used to rigorously verify plasma simulation codes based on finite differences schemes Overcoming the difficulty of comparing distribution functions with markers affected by statistical noise, we now generalized MMS to PIC codes verification We provided a methodology to rigorously estimate the uncertainties affecting simulation results due to finite statistics and discretization

69 Backup slides

70 1 Verification & Validation REALITY OBSERVATIONS MODEL QUALIFICATION EXPERIMENTAL DATA VALIDATION ANALYSIS INTERPRETATION MODEL CODING SIMULATION CODE COMPUTATION NUMERICAL RESULTS CODE VERIFICATION SOLUTION VERIFICATION

71 2 The PIC algorithm A simple model: charge to the grid Equations of motion Poisson equation interpolation to particles

72 3 MMS for a PIC simulation code The modified model: charge to the grid Equations of motion Poisson equation interpolation to particles

73 4 Implementing MMS in PIC codes Manufactured solution: Source terms: Poisson's equation: Weighted function: Initial condition: Equations of motion:

-2 0 2 5 The Kolmogorov Smirnov statistic CDF: Indicator function: 1 CDF EDF EDF:

74 The Kolmogorov Smirnov statistic CDF: Indicator function: 1 CDF EDF EDF: KS statistic: 0 Under null hypothesis: almost surely and where is the Brownian bridge

75 6 A useful property 1D independent of integration direction:

76 2D cumulative distribution functions 7

77 2D empirical distribution functions 8

78 Multidimensional case 9

79 10 Two-dimensional Peacock test Generalization of the KS statistic [J.A. Peacock 1983]:

80 Different approaches 11

81 12 Interpolation scheme: first-order weighting (CIC PIC) Interpolation function: Interpolation: particles grid Numerical scheme Poisson solver: second order centered finite differences Interpolation: grid particles Time integration: Leapfrog integration scheme

82 Results: scan, and small 13

83 Results: scan, small 14

84 Results: scan, small 14

85 Results: scan, small 14

86 Results: scan, small 15

87 Results: scan, small 15

88 Results: scan, small 15

89 16 Results: PIC code verification for

90 17 Results: PIC code verification For we expect

91 18 Monte-Carlo Method Use Monte-Carlo method to estimate

92 19 An alternative approach Decoupling the different dimensions:

93 19 An alternative approach Decoupling the different dimensions:

94 20 Statistical errors, the principles How to estimate the statistical uncertainty affecting a quantity? Assumptions: randomly distributed from Unknown, finite mean Unknown, finite variance Perform observations Law of large numbers Central limit theorem

95 21 Estimate of statistical uncertainties Perform simulations with particles Assume Estimate with particles For one simulation with particles:

96 22 Statistical error affecting a functional Functional with uncorrelated and For

97 Chaotic regimes? 23

Development of a Maxwell Equation Solver for Application to Two Fluid Plasma Models. C. Aberle, A. Hakim, and U. Shumlak

Development of a Maxwell Equation Solver for Application to Two Fluid Plasma Models C. Aberle, A. Hakim, and U. Shumlak Aerospace and Astronautics University of Washington, Seattle American Physical Society