Code Verification: Past Present and Future Chris Roy (cjroy@vt.edu) Professor and Assistant Department Head For Graduate Studies Aerospace and Ocean Engineering Department Virginia Tech Keynote Lecture ASME V&V Symposium May 13 2015
Outline Introduction What is code verification? Why is it hard? Why is it important? Historical perspective Current state-of-the-art Future topics for research Closing remarks 2
What is Code Verification? Code verification is performed to ensure that a scientific computing code is capable of producing the correct result Math Model (PDEs) Numerical Algorithm Software Numerical Solution Discretization Programming Simulation Two primary sources of error code verification can detect: Algorithms deficiencies Software programming mistakes Code verification is achieved by comparing code output to the exact solution to the math model 3
Why is Code Verification Hard? Standard software engineering testing compares software output to the correct result; however in scientific computing the correct answer from a code is never known The solution depends on the following: Numerical algorithm Spatial mesh Time step Floating point precision Iterative tolerance 4
Why is Code Verification Important? The dangers of ignoring code verification include: The code may converge very nicely with mesh refinement but to the wrong solution; this calls into question the results of any subsequent analysis design validation optimization etc. If a code converges to the correct solution but at the wrong rate then the solution cost to achieve a desired numerical error level can grow dramatically e.g. 5 Consider that a formally second-order accurate code is run on a 3D mesh of 1M cells with an error estimate of 20% To achieve a desired error tolerance of 5% one would need: 8M cells if the code is converging at a second-order rate 64M cells if the code is converging at a first-order rate
A (Brief) History of Code Verification 6
Traditional Exact Solutions Exact solutions to well-posed Partial Differential Equations (PDEs) have been around since Newton s time However exact solutions only exist for very simple problems Today we are often concerned with coupled nonlinear systems of partial differential equations with complex boundary conditions applied on general geometries These complex mathematical models do not readily admit traditional exact solutions 7
Manufactured Solutions Roache and Steinberg (1984) had a major breakthrough in the creation of exact solutions for general scientific computing applications; the Method of Manufactured Solutions can be summarized as follows: Step 1: Write PDEs in the form L( u) Step 2: Choose an analytic form of the Manufactured Solution Step 3: Operate the mathematical model onto the Manufactured Solution to obtain the analytic source term û s L(uˆ) Step 4: Obtain the modified form of the mathematical model by including the analytic source term: L( u) s which is solved exactly by û 0 L () û 8 Roache P. J. and S. Steinberg Symbolic manipulation and computational fluid dynamics AIAA Journal Vol. 22 No. 10 1984 pp. 1390 1394.
Order of Accuracy Testing Although Roache and Steinberg (1984) may not have been the first to create exact solutions in such a manner (e.g. see Zadunaisky 1976) they appear to be the first to apply it to code verification using order of accuracy testing The formal order of accuracy is the rate at which meshrelated errors should reduce with systematic mesh refinement The observed order of accuracy is the actual rate at which these errors reduce: rh rh Global ln Norms of ln h Quantities: pˆ h the Solution: pˆ lnr ln r 9 Roache P. J. and S. Steinberg Symbolic manipulation and computational fluid dynamics AIAA Journal Vol. 22 No. 10 1984 pp. 1390 1394.
Code Verification Example Consider the 2D Euler equations that govern frictionless flow Four coupled nonlinear PDEs that governing conservation of mass momentum and energy Manufactured Solutions chosen as sinusoidal functions Systematic mesh refinement to create 5 meshes 1 0.9 1 0.9 4 Euler Equations Norms: y/l 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 0.5 1 x/l p (N/m 2 ) 162500 155000 147500 140000 132500 125000 117500 110000 102500 95000 87500 y/l 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 0.5 1 x/l Source Term Energy Eqn. 9.40015E+08 7.97359E+08 6.54702E+08 5.12045E+08 3.69389E+08 2.26732E+08 8.40757E+07-5.85809E+07-2.01238E+08-3.43894E+08-4.86551E+08 Order of Accuracy p 3 2 1 0 1 L Norm (Premo) L 2 Norm (Premo) L Norm (Wind) L 2 Norm (Wind) h 5 10 15 20 Pressure MS Energy Source Term Order of Accuracy 10 Roy C. J. C. C. Nelson T. M. Smith and C.C. Ober Verification of Euler/Navier Stokes codes using the method of manufactured solutions International Journal for Numerical Methods in Fluids Vol. 44 2004 pp. 599 620.
Verification of Boundary Conditions Bond et al. (2007) developed a novel method for ensuring that Manufactured Solution values and derivatives are satisfied along a general boundary surface Consider a standard Manufactured Solution: One can characterize a bounding surface in 3D by the general function They developed a new Manufactured Solution to satisfy: Dirichlet BCs ( ): Neumann BCs ( ): so that: 11 u u 0 ) ˆ( ) ( ) ( 0 z y x u z y x F u z y x u 0 n u ) ˆ( ) ( ) ( 2 0 z y x u z y x F u z y x u ) ( ˆ ) ( ) ˆ( ) ( 2 2 z y x n u z y x F z y x u n F z y x F n u 0 ) ( z y x F ) ( ˆ z y x u =0 =0 on the boundary Bond R. B. C. C. Ober P. M. Knupp and S. W. Bova Manufactured solution for computational fluid dynamics boundary condition verification AIAA Journal Vol. 45 No. 9 2007 pp. 2224 2236.
Current State-of-the-Art in Code Verification 12
Weak Form of Manufactured Solutions For finite difference methods the Manufactured Solution source term is properly evaluated as a nodal value; however for finite volume and finite element methods it is a volume integral: s ( x y z) dv V Derlaga et al. (2013) employed the divergence theorem to convert the derivative over a cell/element to a surface integral Lowers the dimension of the integration by one; easier to perform quadrature esp. for higher-order methods; e.g. in 2D: A da 13 J. M. Derlaga T. S. Phillips and C. J. Roy SENSEI Computational Fluid Dynamics Code: A Case Study in Modern Fortran Software Development AIAA Paper 2013-2450 21 st AIAA Computational Fluid Dynamics Conference San Diego CA June 24-27 2013.
Verification of Turbulence Models in CFD Turbulence models offer challenges in code verification due to strongly nonlinear source terms and min/max functions Eca et al. (2005) developed physically-realistic Manufactured Solutions for turbulent flows as part a series of workshops held in Lisbon Portugal on CFD Uncertainty E.g. incompressible turbulent boundary layer Manufactured Solutions were developed for six different turbulence models: x-velocity Turbulent Turbulence Viscosity Kinetic Energy 14 L. Eca M. Hoekstra A. Hay and D. Pelletier A Manufactured Solution for a Two-Dimensional Steady Wall- Bounded Incompressible Turbulent Flow IST Report D72-34 EPM Report EMP-RT-2005-08 Nov. 2005.
Code Verification for Solid Mechanics Rigorous code verification in solid mechanics has lagged that in fluid mechanics and heat transfer The tradition in finite elements is to do element or patch tests of the finite element discretization; manufactured solutions and order of accuracy testing generally not used Kamojjala et al. (2015) has recently developed manufactured solutions for solid mechanics Displacement Error Norms 1 2 15 K. Kamojjala R. Brannon A. Sadeghirad and J. Guilkey Verification Tests in Solid Mechanics Engineering with Computers Vol. 31 No. 2 2015 pp. 193-213.
Incompressible Flows For incompressible flows the Manufactured Solutions are not arbitrary and must satisfy the divergence-free condition for velocity i.e. V 0 This is especially true for pressure projection methods where a Poisson equation for pressure is developed which enforces the divergence-free condition Choudhary et al. (2014) used the identity which holds for any vector field H to developed a curlbased MS that satisfies the divergence-free condition H The velocity is thus given by: and the Manufactured Solution is specified for the vector V 0 H H 16 A. Choudhary C. J. Roy J.-F. Dietiker M. Shahnam and R. Garg Code Verification for Multiphase Flows Using the Method of Manufactured Solutions FEDSM2014-21608 Proceedings of the ASME 2014 4 th Joint US- European Fluids Engineering Division Summer Meeting (FEDSM) Chicago IL August 3-7 2014.
Multiphase Flows Until recently there has been very little work published on code verification for multiphase flows Choudhary et al. (2014) verified a continuum two-phase (gassolid) flow model in the MFIX CFD code The two phases are treated as continua with exchange terms Gas Solid Granular Temperature Temperature Energy Order of Accuracy 17 A. Choudhary C. J. Roy J.-F. Dietiker M. Shahnam and R. Garg Code Verification for Multiphase Flows Using the Method of Manufactured Solutions FEDSM2014-21608 Proceedings of the ASME 2014 4 th Joint US- European Fluids Engineering Division Summer Meeting (FEDSM) Chicago IL August 3-7 2014.
Future Challenges for Code Verification 18
Code Verification for Problems with Singularities and Discontinuities Code verification has traditionally been done on smooth problems; however almost all practical problems contain singularities and/or discontinuities Banks et al. (2008) demonstrated that: Nonlinear discontinuities cause first order behavior Linear discontinuities cause order less than one: Grier et al. (2015 2014) have recently developed: Manufactured solutions with discontinuities Numerical integration techniques for discontinuities p smooth p smooth 1 19 J. W. Banks T. Aslam and W. J. Rider On sub-linear convergence for linearly degenerate waves in capturing schemes Journal of Computational Physics Vol. 227 2008 pp. 6985-7002. B. Grier E. Alyanak M. White J. Camberos and R. Figliola Numerical integration techniques for discontinuous manufactured solutions Journal of Computational Physics Vol. 278 2014 pp. 193-203. B. Grier R. Figliola E. Alyanak J. Camberos Discontinuous Solutions Using the Method of Manufactured Solutions on Finite Volume Solvers AIAA Journal pp. 1-10 DOI: 10.2514/1.J053725.
Code Verification for Multiphysics Computations Simulations with multiple physical phenomena are becoming more commonplace (e.g. fluid-structures fluid-thermal thermoelastic) When the coupling occurs at an interface many of the same discontinuity-related issues arise Fully-coupled simulations are expensive and the phenomena often occur at disparate time scales Loosely-coupled simulations often suffer from numerical stability issues that are difficult to analyze Manufactured Solutions are needed to aid in the development and verification of multiphysics codes 20
Numerical Benchmark Solutions Numerical benchmark solutions offer an alternative to code verification using Manufactured Solutions Requirements for a numerical benchmark solution: Must be computed with a rigorously verified code Must have rigorous error estimates including demonstration of the observed order of accuracy Numerical errors must be small especially if used for verifying higher-order codes Example: turbulent flat plate (http://turbmodels.larc.nasa.gov/) Spalart-Allmaras turbulence model Multiple codes used All appear to converge to the same answer with mesh refinement 21
Concluding Remarks 22
Are Your Commercial Scientific Computing Codes Verified? Since code verification is generally done by the code developer should you assume that the commercial (or government) code you use is verified? NO! Code verification is arguably the most mature sub-topic in Verification Validation and Uncertainty Quantification The main code verification techniques have been around for decades However almost no commercial software vendors have "code verification" documents; why? Sacrifice accuracy for robustness Customers have not demanded it 23
Are Your Commercial Scientific Computing Codes Verified? Consider the code verification study of commercial CFD codes by Abanto et al. (2005) using a Manufactured Solution for an incompressible laminar boundary layer Code A Code B Formally second-order accurate finite volume codes Velocity generally converges at less than first order Pressure is non-convergent 24 Abanto J. D. Pelletier A. Garon J.-Y. Trepanier and M. Reggio Verification of some commercial CFD codes on atypical CFD problems 43 rd AIAA Aerospace Sciences Meeting AIAA Paper 2005 0682 2005 Reno NV.
Thank You 25