Testing Algorithms and Software. Software Support for Metrology Good Practice Guide No. 16. R M Barker, P M Harris and L Wright NPL REPORT DEM-ES 003

Transcription

1 NPL REPORT DEM-ES 003 Software Support for Metrology Good Practce Gude No. 6 Testng Algorthms and Software R M Barker, P M Harrs and L Wrght Not Restrcted December 005 Verson.0 Natonal Physcal Laboratory Hampton Road Teddngton Mddlesex Unted Kngdom TW 0LW Swtchboard NPL Helplne Fax

2

3 Software Support for Metrology Good Practce Gude No 6 Testng Algorthms and Software R M Barker, P M Harrs and L Wrght Natonal Physcal Laboratory Teddngton, Mddlesex, Unted Kngdom. TW 0LW December 005 Abstract Snce ther ncepton, the SSfM programmes have worked on testng of software used n metrology: developng methodologes and applyng them to testng our own software and testng packages used n processng measurement data. The success of the testng methodologes s shown by the errors that have been found n wdely-used packages, errors that we have been able to trace to errors n the mplementatons of numercal routnes. Ths gude consoldates and summarses the methodologes for testng software for dscrete and contnuous modellng, and for testng the underlyng algorthms. Ths gude draws on a number of gudes and reports from the prevous SSfM programmes, and uses examples from those documents to llustrate the methodologes. Verson.0

4 Crown copyrght 005 Reproduced wth permsson of the Controller of HMSO and Queens Prnter for Scotland ISSN Extracts from ths gude may be reproduced provded the source s acknowledged and the extract s not taken out of context We gratefully acknowledge the fnancal support of the UK Department of Trade and Industry (Natonal Measurement System Drectorate) Approved on behalf of the Managng Drector, NPL by Jonathan Wllams, Knowledge Leader for the Electrcal and Software team Natonal Physcal Laboratory Teddngton, Mddlesex, Unted Kngdom. TW 0LW

5 Table of Contents. Introducton.... Structure of ths Gude.... Ratonale for the Gude..... Scope..... Effects producng nexactness Mathematcal and statstcal models Ftness for purpose Testng framework Dfferent ssues for dscrete and contnuous modellng Relatonshp to other SSfM documents Best- and Good-Practce Gudes and mportant reports Reports and other SSfM documents Resources Acknowledgements...8. Glossary Overvew of Approach Understandng the testng problem Applyng analyss to the testng problem Applyng reference pars to the testng problem Specfcaton of reference pars Calculaton of reference pars Specfcaton of qualty metrcs Complementary checks Checkng consstency of functons Checkng contnuty of functons Checkng aganst tabulated or other values Checkng soluton charactersatons Smulaton Presentaton and nterpretaton of results Dscrete Modellng Example: Gaussan Peak Fttng Understandng the testng problem Target computatonal am Test computatonal am Implementaton of the test software Applyng analyss to the testng problem Mathematcal analyss of the computatonal ams Numercal analyss of the test software Applyng reference pars to the testng problem Specfcaton of reference pars Calculaton of reference pars Specfcaton of qualty metrcs Smulaton Presentaton and nterpretaton of results...3

6 5. Contnuous Modellng Example: Heat Transfer Understandng the testng problem Target computatonal am Test computatonal am Implementaton of the test software Applyng analyss to the testng problem Mathematcal analyss of the computatonal ams Numercal analyss of the test software Applyng reference pars to the testng problem Specfcaton of reference pars Calculaton of reference pars Specfcaton of qualty metrcs Complementary checks Smulaton Presentaton and nterpretaton of results References... 5

7 Testng Algorthms and Software NPL Report DEM-ES 003. Introducton Ths Gude s produced by the Software Support for Metrology programme, whch s a DTI programme gvng support to measurement scentsts n mathematcs and software. Snce ther ncepton, the SSfM programmes have ncluded work to support the testng of software used n metrology: developng methodologes and applyng them to testng NPL s own software and software packages developed elsewhere for processng measurement data. The success of the testng methodologes s shown by the poor performance that has been dentfed n some wdely-used packages, a problem that we have been able to trace to poor choces of numercal routne [5]. Ths Gude consoldates and summarses the methodologes for testng software for dscrete and contnuous modellng, and for testng the underlyng algorthms. Ths Gude draws on a number of gudes and reports from the prevous SSfM programmes, and uses examples from those documents to llustrate the methodologes.. Structure of ths Gude Ths Gude descrbes a common approach to testng and demonstrates many aspects of the testng approach through two examples. The common technques for testng are descrbed, and the two examples show how these technques are appled n dscrete and contnuous modellng problems. Further detals of the testng approach and technques are referenced n other SSfM documents. The structure of the Gude s as follows: The background and phlosophy to the approach n ths gude A glossary of terms used repeatedly throughout the Gude An overvew of technques common to all problems An example of testng for dscrete modellng problems An example of testng for contnuous modellng problems. Ratonale for the Gude.. Scope The focus of ths document s on gvng gudance on methods for nvestgatng the ftness for purpose of a software mplementaton of an algorthm to solve a specfed, and well-defned, mathematcal model. The software s referred to as the test software, the algorthm mplemented by the test software as the test algorthm, and the (well-defned) specfcaton of the mathematcal model as the computatonal am. The Gude brngs together prevous work on technques for algorthm and numercal software testng, appled to dscrete and contnuous modellng software. The Gude s ntended for both users and developers of software for solvng dscrete and contnuous modellng problems. It addresses the requrements placed on users and developers to demonstrate that software s ft for ts ntended purpose. In addton, the Gude s ntended to assst Qualty Managers by provdng nformaton on how to montor and control the acceptablty of software as part of measurement, producton and nspecton systems. Ths Gude should be consdered n the context of Software Support for Metrology Best- Practce Gude No : Valdaton of software n measurement systems [3]. That gude descrbes an overall approach to valdaton of software, defnng a number of technques that Ths s to be dstngushed from software that may be used to undertake the testng. Page of 53

8 NPL Report DEM-ES 003 Testng Algorthms and Software can be used to ensure the ftness for purpose for a (measurement) system that ncludes software. The partcular technques to be appled are determned by reference to the complexty of the software and the ntended use of the software. Testng s a core element of valdaton and testng of numercal correctness s mportant to all measurement software. Therefore, ths Gude provdes a method of mplementng one of the valdaton technques defned n Best-Practce Gude No... Effects producng nexactness As part of the process of usng software to obtan a soluton to a problem n metrology, there are a number of effects that produce nexactness n the (measurement) result delvered by the software. These nclude: Modellng effects, arsng from lmtatons of the chosen mathematcal model to represent fathfully the physcal problem requred to be solved. Algorthm effects, arsng from lmtatons of a chosen algorthm to delver a soluton to the mathematcal model (above). Numercal effects, arsng from the use of a software mplementaton of the algorthm (above) to compute the soluton. Data effects, arsng from nexactness n data values (ncludng measurement data, reference constants, calbraton nformaton, ntal and boundary data, etc.) that defne the partcular nstance of the physcal problem requred to be solved. We use the term Valdaton for the generc actvty that encompasses consderaton of all these effects wthn ths process. It s an actvty that s mportant to metrology and s drven by the requrement to demonstrate the correctness and traceablty of a measurement result. Traceablty s the documentaton of a calbraton chan, supported by assocated uncertantes, between an nstrument and a prmary standard. For the software used to delver a measurement result, traceablty s the documentaton of all aspects n the process of development, mplementaton and testng of a model n software so that each stage n the process can be understood, checked and reproduced. These are the effects that wll produce nexactness n the numercal calculaton: there are, of course, many other software effect that wll cause numercal software to go wrong. The general approach to testng and valdaton of numercal software n the context of a general (measurement) system s the subject of Software Support for Metrology Best-Practce Gude No [3], referred to above...3 Mathematcal and statstcal models The most dffcult effect to nvestgate s that assocated wth the mathematcal model. Generally, the model s chosen on the bass of the user s understandng and knowledge of the physcal problem. Although there may be establshed tred and tested models avalable for partcular problems, there s often an aspect of subjectvty n choosng the model. Generally, the model can always be expected to ncorporate approxmatons to the physcal problem. Sometmes these approxmatons wll be delberate, and are ntroduced to generate mathematcal models that are tractable or effcent to solve. Examples nclude assumng lnearty, symmetry, homogenety, exactness of data, etc. Investgatng and quantfyng data effects s the subject of uncertanty evaluaton, based on a statstcal model. The measurement result s obtaned by evaluatng the mathematcal model used to descrbe the measurement problem for estmates of the values of the data that defne the partcular nstance of the problem requred to be solved. The uncertanty assocated wth the measurement result s evaluated n terms of the model and the uncertantes assocated wth those estmates (or other nformaton concernng the nexactness of the estmates). Page of 53

9 Testng Algorthms and Software NPL Report DEM-ES Ftness for purpose Valdaton s used to demonstrate that the use of an approxmate mathematcal model, an approxmate algorthm to solve the model, and software as an mplementaton of the approxmate algorthm, are ft for purpose. In ths context, ftness for purpose may be descrbed qualtatvely, e.g., the soluton demonstrates a physcal property such as conservaton of energy. Often valdaton technques are unable to dstngush the contrbutons made by the dfferent effects. For example, f t s found that the soluton delvered by software for solvng a measurement problem fals to satsfy a property expected of a soluton to that problem (examples nclude conservaton of energy, asymptotc behavour, etc.) then t can be dffcult to know (wthout further nformaton) whether the falure s assocated wth the model, the algorthm, the software mplementaton or a combnaton of these. Ftness for purpose can also mean that the effects arsng from the use of the approxmate mathematcal model, approxmate algorthm, etc. are quanttatvely small compared to those effects arsng from the data, the latter descrbed by an uncertanty. In cases that the use of an approxmate mathematcal model, approxmate algorthm, etc. are shown not to be ft for purpose t s necessary ether to correct for these effects, e.g., to use a more sophstcated mathematcal model, algorthm, etc., or to quantfy the effects and nclude them as addtonal contrbutons n the uncertanty evaluaton...5 Testng framework The development of software for solvng a numercal problem s dvded nto the development of an algorthm for solvng the problem and a translaton of the algorthm nto code. Ths subdvson s reflected n testng: f the test algorthm s known, we can perform tests on the algorthm and tests on the software relatve to the algorthm. We can also test algorthms on ther own, so that algorthms can be seen as ft for purpose for a class of problems, wthout snglng out any partcular pece of software. In crcumstances where the test algorthm s known, consderaton may be gven to answerng the questons how good s the test algorthm for solvng the mathematcal model? and how good s the test software for delverng a result for the test algorthm? In ths work, these are the questons addressed by the actvtes of, respectvely, algorthm testng and numercal software testng. The framework for algorthm testng used n ths Gude s based on comparson wthn a matrx of dfferent manfestatons of the problem/calculaton (Fgure.). The vertcal axs s the degree of abstracton of the calculaton from (mathematcal) computatonal am, through algorthm, to software; the other axs s the degree of approxmaton wth whch the calculaton solves the problem. The dfferent technques for algorthm testng can be seen as comparsons along one axs or the other, and some comparsons are dagonal, e.g. comparng an mplementaton of an approxmate algorthm wth the computatonal am. Page 3 of 53

10 NPL Report DEM-ES 003 Testng Algorthms and Software (Target) Computaton am (Test) Computaton am (Reference) Algorthm (Test) Algorthm (Reference) Software (Test) Software Fgure. Framework for algorthm testng. The key technques n algorthm testng are peer revew of the (documented) algorthm and numercal software testng of mplementatons based on the algorthm. When testng an algorthm ndependent of ts mplementaton, numercal software testng may stll be useful. A reference mplementaton of the algorthm may be developed that accurately performs the algorthm (but presumably at some cost n terms of effcency); the reference mplementaton can then be used for comparson wth other mplementatons or for testng. It may also be possble to analyse the results of numercal software testng to determne errors that are contrbuted by a poor choce of algorthm and those that are contrbuted by the mplementaton. It s common to use generc algorthms and generc (commercal off the shelf) software mplementatons of those algorthms to solve a (wde) varety of mathematcal models. Examples nclude usng a lnear least-squares solver to represent expermental data by a lnear calbraton curve, and usng the boundary element method (wth lnear elements) to solve the Helmholtz equaton. If these generc components can be shown to be ft for purpose, the metrologst s left to concentrate on how modellng effects mpact on the soluton to the measurement problem. In crcumstances where detals of the test algorthm are not known (as can be typcal when usng commercal off the shelf software), all that can be done s to nvestgate how good the test software s at delverng a soluton to the problem specfed by the computatonal am. In ths case, the software s regarded as a black box, and ts testng addresses the queston how good s the test software for solvng the mathematcal model? In contrast to black-box testng, testng that uses knowledge of the algorthm or mplementaton to desgn the tests, yet only nteracts wth the software through ts nputs and output, s termed grey-box. Whte-box testng, based on nternal access to the software, s not part of the methodology of ths gude. In the case of commercal off the shelf software users may also want to draw some general conclusons about the software, e.g. how well t solves a class of problems, rather than a specfc test problem(s). Some technques for the testng of algorthms can be appled even when the algorthm employed n software s not known; numercal software testng may reveal whch algorthm s probably used by the software and, f approprate, further testng could be used to test that algorthm. Page 4 of 53

11 Testng Algorthms and Software NPL Report DEM-ES Dfferent ssues for dscrete and contnuous modellng Although the Gude dscusses concepts and technques that are generc to both dscrete and contnuous modellng problems, descrptons of partcular technques for the two types of problem are kept mostly separate. One reason for ths s that the audences for the materal relatng to the two types of problem are generally dfferent. A second reason s that there are often dfferent consderatons for each. For example, some software for solvng contnuous modellng problems (partcularly commercal off the shelf software) s amed at solvng specfc classes of physcal problem (e.g., thermal, acoustc, structural, etc.) rather than general models defned by arbtrary (systems of) dfferental equatons. Consequently, reference problems for testng such software must be constructed to represent these classes of physcal problem, and ndeed t may be dffcult to express the reference problems n terms of (systems of) dfferental equatons. In contrast, software for solvng dscrete modellng problems s nvarably developed to solve general models, e.g., a lnear least-squares solver may be used to determne any lnear calbraton functon from measurement data, and the specfcaton of the model n terms of mathematcal equatons s usually possble. A further reason s that the solutons obtaned from software for solvng contnuous modellng problems are (almost) always subject to algorthm error as a consequence of the dscretsaton of the problem that s a part of the algorthms used by such software. In contrast, there are many dscrete modellng problems for whch algorthms exst for delverng (mathematcally) correct solutons and the concern s wth the numercal propertes of mplementatons of those algorthms. A fnal reason s that challengng reference problems wth known solutons are rare for many types of contnuous modellng. In general, f a contnuous modellng problem has an analytc soluton, the problem s lkely to be ether smple or far removed from real-world problems. Hence t can be dffcult to create reference problems that stretch the software s capabltes and test ts sutablty for applcaton to real metrology problems. The Gude assumes prmarly that the software s avalable as a black box, though consderaton s gven to how knowledge about the test algorthm and ts mplementaton n the test software can contrbute to the testng process. The am of the testng s, predomnantly, to nvestgate the ablty of the software to generate a partcular result rather than to check whether the software mplements a partcular algorthm. Ths s consstent wth the way most propretary (commercal off the shelf) software for dscrete and contnuous modellng s used. Page 5 of 53

12 NPL Report DEM-ES 003 Testng Algorthms and Software.3 Relatonshp to other SSfM documents.3. Best- and Good-Practce Gudes and mportant reports The followng companon best- and good-practce gudes and reports contan relevant materal: SSfM BPG SSfM BPG4 SSfM GPG5 SSfM BPG6 SSfM BPG7 Valdaton of software n measurement systems Testng of software, and determnaton of ftness for purpose of software packages, s a major element of software valdaton. [3] Dscrete modellng and expermental data analyss Software to solve dscrete modellng problems form an mportant class to be tackled by the software testng methodology. The gude also ncludes technques for the valdaton of dscrete models. [5] In the context of testng, BPG 4 covers valdaton of dscrete model effects. Gude to EUROMETROS The EUROMET Repostory of Software (EUROMETROS) contans measurement software that has been tested usng the methodology n ths gude and contans test data sets for testng mplementatons of data fttng software. [4] Uncertanty evaluaton Software to support evaluaton of uncertantes n measurement problems s also a class of software that can be tackled usng the software testng methodology. [9] In the context of testng, BPG6 covers aspects of ftness for purpose and valdaton of data effects. Development and testng of spreadsheet applcatons The testng methodology s applcable to the testng of spreadsheet applcatons; but perhaps more mportantly, the testng methodology has revealed some shortcomngs n popular spreadsheet packages, emphassng the need for developers of spreadsheet applcatons to test ther applcatons, partly to ensure the ftness for purpose of the underlyng spreadsheet package. [] SSfM BPG Numercal analyss for algorthms desgn n metrology Numercal analyss of the ntended computaton s a key component n the methodology for testng algorthms and software. [0] SSfM BPG Gude for test and measurement software Software testng s a key component n the development of measurement software; the methodology n ths gude s equally mportant for determnng the ftness for purpose of mathematcal lbrary software that s used as the bass for partcular measurement software applcatons. [] CMSC 9/03 Model Valdaton n Contnuous Modellng Ths report provdes general advce about the process of valdaton and descrbes methods and technques that can be used for valdatng contnuous models, many of whch have been developed to address problems specfc to contnuous modellng. [3] In the context of testng, CMSC 9/03 covers valdaton of contnuous model effects. Page 6 of 53

13 Testng Algorthms and Software NPL Report DEM-ES Reports and other SSfM documents Ths Gude s based on materal developed n the frst two SSfM programmes: SSfM Testng Algorthms n Standards and METROS, CMSC 8/03 [3] The Comparson of Algorthms for the Assessment of Type A Surface Texture Reference Artefacts, CMSC 33/03 [6] Testng the Numercal Correctness of Software, CMSC 34/04 [] Testng Methods of Java Lbrares, CMSC 35/04 [8] Testng Contnuous Modellng Software, CMSC 4/04 [4] Testng Contnuous Modellng Software: Three Case Studes, CMSC 4/04 [5] Testng Algorthms for Free-Knot Splne Approxmaton, CMSC 48/04 [8] SSfM The Desgn and Use of Reference Data Sets for Testng Scentfc Software (paper) [6] A Methodology for Testng Spreadsheets and Other Packages Used n Metrology, CISE 5/99 [3] Testng Spreadsheets and Other Packages Used n Metrology, a Case Study, CISE 6/99 [4] Testng Spreadsheets and Other Packages Used n Metrology Testng the Intrnsc Functons of Excel, CISE 7/99 [5] Testng Spreadsheets and Other Packages Used n Metrology: Testng the Intrnsc Functons of Mathcad, CMSC 05/00 [6] Testng Spreadsheets and Other Packages Used n Metrology: Testng the Intrnsc Functons of S-Plus, CMSC 06/00 [7] Testng Spreadsheets and Other Packages Used n Metrology: Testng Functons for the Calculaton of the Standard Devaton, CMSC 07/00 [7] Testng Spreadsheets and Other Packages Used n Metrology: Testng Functons for Lnear Regresson, CMSC 08/00 [8] Page 7 of 53

14 NPL Report DEM-ES 003 Testng Algorthms and Software.4 Resources These web-based resources may also provde useful materal for numercal software testng. Tools for Evaluatng Mathematcal and Statstcal Software math.nst.gov/stssf/ NIST statstcal reference data sets Resources for Improvng Accuracy n Statstcal Software A Web-Based Lbrary for Testng the Performance of Numercal Software NAFEMS benchmarks for fnte element calculatons shop/commerce.cg?product=benchmark_tests&cart_d= Tests of the qualty of random number generators: TestU0 (ncludng Small Crush, Crush and Bg Crush) There are also on-lne resources from NPL and SSfM projects. SSfM data generators for calculatons n dscrete modellng SSfM on-lne testng servces for calculatons n dscrete modellng ssfm/ssfm3/theme3/numercal_software_testng/project3_/mlestone3/ EUROMETROS SOFTGAUGES webste that provdes reference software and reference data for testng software for the evaluaton of surface texture parameters Also useful resources are the books Accuracy and Stablty of Numercal Algorthms, by Nck Hgham [30] Statstcs Software Qualfcaton: Reference Data Sets, edted by B.P. Butler, M.G. Cox, S.L.R. Ellson, and W.A. Hardcastle [0] and papers n Qualty Of Numercal Software: Assessment And Enhancement Oxford 996, ncludng: Testng lnear algebra software Nck Hgham [9] A methodology for testng classes of approxmaton and optmsaton software Bernard Butler, Maurce Cox, Alstar Forbes, Smon Hannaby and Peter Harrs [9].5 Acknowledgements Ths Gude was produced under the Software Support for Metrology (SSfM) programme, whch s managed behalf of the DTI by the Natonal Physcal Laboratory. For more nformaton on the SSfM programme vst the webste at or contact the programme manager Mr Bernard Chorley (phone , emal: ssfm@npl.co.uk, Natonal Physcal Laboratory, Hampton Road, Teddngton, Mddlesex, UK TW 0LW). Ths Gude reles heavly on the work that lead to the SSfM documents lsted n secton.3 and we gratefully acknowledge the authors of those documents. Page 8 of 53

15 Testng Algorthms and Software NPL Report DEM-ES 003. Glossary Absolute error Algorthm effect Algorthm testng Black-box testng Computatonal am Data effect Dscretsaton error Doman tests Ft for purpose Grey-box testng Mathematcal equvalence (of algorthms) Modellng effect Numercal equvalence (of algorthms) The magntude of the dfference between test and reference results correspondng to a reference problem. Effect on the correctness of a measurement result arsng from lmtatons of a chosen algorthm to delver a soluton to the problem specfed by a computatonal am. Investgatng the extent to whch usng a (test) algorthm to solve the problem specfed by a test computatonal am delvers a soluton to the target problem (as specfed by the target computatonal am). Testng of software usng solely the nputs and output of the software not based on knowledge of the algorthm, or the manner n whch that algorthm s mplemented. An unambguous specfcaton of the problem solved by software, for example n the form of the nputs to, and outputs from, the software and the mathematcal model relatng the nputs and the outputs. Effect on the correctness of a measurement result arsng from the nexactness of data values (ncludng measurement data, reference constants, calbraton nformaton, ntal and boundary data, etc.). The error caused by approxmatng a contnuous dfferental equaton wth a numercal soluton. Usually consdered as the dfference between an analytc soluton and an dealsed numercal soluton, wthout consderng the effects of fnte precson arthmetc. Tests n whch the mathematcal formulaton of the problem s held fxed, but the doman on whch the problem s solved s vared. Software that satsfes a clam about ts numercal performance, for example expressed n terms of an absolute or relatve error, or a performance measure. Testng of software usng the nputs and output of the software but where the testng strategy s based on a knowledge of the algorthm and how that algorthm s mplemented. The algorthms delver dentcal results for gven data when mplemented correctly usng nfnte precson arthmetc. Effect on the correctness of a measurement result arsng from lmtatons of the chosen mathematcal model to represent fathfully the physcal problem requred to be solved. The algorthms delver results for gven data accurate to wthn a stated bound (dependent on the problem condton, the nput data and the computer arthmetc) when mplemented correctly usng fnte precson arthmetc. Page 9 of 53

16 NPL Report DEM-ES 003 Testng Algorthms and Software Numercal (software) effect Numercal software testng Performance measure Performance parameter Problem condton Qualty metrcs Reference algorthm Reference data Reference data generator Reference par Reference problem Reference result Reference software Relatve error Scalable tests Smulaton Effect on the correctness of a measurement result arsng from the use of a software mplementaton of an algorthm to compute a soluton to the problem specfed by a computatonal am. Investgatng the extent to whch a software mplementaton delvers a soluton to the problem specfed by a computatonal am. A qualty metrc that compares the numercal performance of test software aganst reference software accountng for the condtonng of the problem specfed by the target computatonal am and the computatonal precson of the arthmetc used to obtan test and reference results. A parameter used to descrbe a property of a reference problem, and hence to parameterse the space of admssble nputs to the test software. A measure that descrbes the senstvty of the exact soluton to a problem to changes n the problem. Quanttatve measures of the numercal performance of test software, for example the absolute and relatve errors between test and reference results and a performance measure. An (optmally-stable) algorthm to solve the problem specfed by the target computatonal am. Values of the nput data that defne part of a reference problem Software to generate reference data for whch the soluton to the problem specfed by a computatonal am s specfed a pror. A reference problem together wth correspondng reference results consstent wth a computatonal am. An nstance of a problem specfed by a computatonal am for whch the correspondng reference results are known. The soluton correspondng to a reference problem. Software (developed to a very hgh standard) mplementng a reference algorthm to solve the problem specfed by the target computatonal am. The magntude of the dfference between test and reference results correspondng to a reference problem expressed as a proporton of the magntude of the reference results. A set of reference problems for contnuous modellng software, the results of whch have a known dependence on some functon f of the nputs. The tests are generated by fxng the value of f and varyng the nput values ndvdually. Such tests am to explore the accuracy of the algebrac equaton solvng process and ts senstvty to the nput values. Testng usng reference problems defned by reference data sampled from the probablty dstrbuton descrbng the avalable knowledge about the nexactness of the data. Page 0 of 53

17 Testng Algorthms and Software NPL Report DEM-ES 003 Small-scale tests Target computatonal am Test algorthm Test computatonal am Test result Test software Reference problems for contnuous modellng software that are defned usng only a small mesh. Such tests am to test the algebrac equaton formulaton process. The computatonal problem that s ntended to be solved by test software. The algorthm mplemented by the test software to solve the problem specfed by the test computatonal am. The computatonal problem that s actually solved by test software. The result correspondng to a reference problem delvered by test software. The software subject to testng. Page of 53

18 NPL Report DEM-ES 003 Testng Algorthms and Software 3. Overvew of Approach In ths chapter, some of the general consderatons arsng n algorthm and numercal software testng are dscussed. An overvew of some technques for algorthm and numercal software testng s also gven. In the chapters followng ths one, partcular examples are used to llustrate the applcaton of these technques to testng software for solvng, respectvely, dscrete and contnuous modellng problems. The chapter covers: Understandng the test problem (secton 3.), n terms of nformaton about o The target computatonal am o The test computatonal am o The mplementaton of the test software. Applyng analyss to the testng problem (secton 3.), ncludng: o Mathematcal analyss appled to the target and test computatonal ams o Numercal analyss appled to the test software. Applyng reference pars to the testng problem (secton 3.3), ncludng: o The specfcaton of reference pars o The calculaton of reference pars o The specfcaton of qualty metrcs and decdng ftness for purpose. Checkng soluton charactersatons and other checks (secton 3.4). Smulaton (secton 3.5). The presentaton and nterpretaton of results (secton 3.6). 3. Understandng the testng problem Knowledge about the problem that the test software sets outs to solve s captured by specfyng the target computatonal am. Knowledge about how the test software solves that problem nvolves specfyng the test computatonal am, whch defnes the problem the software actually addresses, as well as detals of the algorthm used to solve the problem specfed by the test computatonal am. Algorthm testng s concerned wth nvestgatng the extent to whch solvng the problem specfed by the test computatonal am delvers a soluton to the problem specfed by the target computatonal am. It s about comparng mathematcs. Numercal software testng s concerned wth nvestgatng the extent to whch software delvers a soluton to the problem specfed by a computatonal am. It s about nvestgatng the correctness and numercal propertes of the software mplementaton. Documentng the target computatonal am s essental to the software testng actvty. The target computatonal am provdes the bass for the testng, e.g., reference data wth correspondng reference results should be generated to be consstent wth the problem defned by the target computatonal am. Knowng about how the test software solves the problem specfed by the target computatonal am can help to dstngush between the effects of ) a test computatonal am that s not mathematcally equvalent to the target computatonal am, and ) a poor or ncorrect mplementaton of software to solve the problem specfed by the test computatonal am. Page of 53

19 Testng Algorthms and Software NPL Report DEM-ES 003 If the specfcaton of the target computatonal am s nconsstent wth the task carred out by the test software, testng the software n accordance wth that specfcaton would yeld the concluson that the software was defcent. But, n fact, the software mght be an acceptable mplementaton of the test computatonal am, but that computatonal am s not a correct approach to the target problem. The specfcaton of a computatonal am may be smple, for example, Ths software calculates the arthmetc mean and standard devaton of a prescrbed set of numercal values. Even n ths smple case, the specfcaton s requred to be unambguous. Hence, t s stated that the arthmetc mean s of concern (rather than, say, the geometrc or harmonc mean). For a contnuous modellng problem, the specfcaton wll typcally nclude: The (system of) dfferental equatons to be solved The geometry of the doman Materal propertes wthn the doman Intal, boundary and loadng condtons Informaton about the way the geometry s meshed. The concern may be to test the software for a partcular choce of mesh, n whch case the specfcaton of the mesh s part of the computatonal am. Alternatvely, f automated meshng tools are avalable, the choce of mesh becomes part of the algorthm mplemented by the test software and wll contrbute an algorthm effect that t s necessary to valdate. An approprate way to provde the specfcaton of a computatonal am s to defne for a software mplementaton: The nputs, represented by quanttes X, ncludng any condtons on the values x of the quanttes The outputs, represented by quanttes Y The mathematcal model relatng the nput and output quanttes. Ths may take the form of a functonal relatonshp between X and Y, for example of the form Y = f(x) or g(x, Y) = 0. For example, for software to calculate the arthmetc mean and standard devaton of a prescrbed set of numercal values : The nputs are the prescrbed numercal values denoted by X, =,, m The outputs are the mean µ Y and standard devaton s Y The mathematcal model relatng the nput and output quanttes s defned by the formulae: m m Y = X, Y = m m = = ( X Y ). For ths sort of problem, the test computatonal am wll often be the same as the target computatonal am. The dfference n the software, wll arse from dfferent algorthms. Page 3 of 53

20 NPL Report DEM-ES 003 Testng Algorthms and Software 3. Applyng analyss to the testng problem In cases where there s knowledge about the test computatonal am, t may be possble to undertake some mathematcal analyss to support algorthm testng. Such analyss may provde qualtatve nformaton, e.g., by dentfyng the crcumstances under whch the two computatonal ams are mathematcally equvalent, and quanttatve nformaton, e.g., n the form of an expresson for the error between the solutons to the problems specfed by them. The approach nvolves workng wth the computatonal ams, and the algorthms for solvng the problems specfed by them, as mathematcal objects ndependent of any software mplementaton. Lkewse, n cases where there s knowledge about the algorthms mplemented by the test software, t may be possble to undertake some numercal analyss to support numercal software testng. Agan, such analyss may provde qualtatve nformaton, e.g., by dentfyng those values of the nputs to the software for whch a formula mplemented n the software s lkely to suffer from subtractve cancellaton, and quanttatve nformaton, e.g., n the form of a bound on the floatng-pont error arsng from the use of the formula. 3.3 Applyng reference pars to the testng problem The applcaton of analyss (secton 3.) to support algorthm testng and numercal software testng has the dsadvantages of generally requrng a hgh degree of techncal skll (and therefore relyng on experts), beng hghly problem-specfc, and consequently beng dffcult (f not mpossble) to automate. Perhaps the most common approach to software testng, however, s to use reference pars to undertake testng of the software. A reference par comprses a reference problem (equvalently, a reference data set) wth correspondng reference results. A reference problem consttutes a partcular nstance of a problem specfed by the computatonal am for the test software, and the reference results are the correspondng soluton to that problem. The test software s appled to the reference problem and the results returned by the software (the test results) are compared wth the reference results suppled as part of the reference par. The approach, whch s essentally one of black-box testng, reles on consderaton beng gven to the specfcaton of reference pars (secton 3.3.), the calculaton of reference pars that are consstent wth the computatonal am (secton 3.3.), and the comparson of test results wth reference results n an objectve manner (secton 3.3.3) Specfcaton of reference pars The specfcaton of a reference par should nclude all aspects of the computatonal am. For a contnuous modellng problem, therefore, the specfcaton ncludes the geometry of the doman, ntal and/or boundary condtons, and possbly nformaton about the way the geometry s meshed. For a computatonal am specfed n terms of nput quanttes X, output quanttes Y and a mathematcal model relatng X and Y (secton 3.), a reference problem s specfed by values x for X, and the reference results correspondng to the reference problem by values y (ref) for Y. Some of the man consderatons n desgnng reference pars for testng software are: The dentfcaton of possble defcences n the test software The constructon of problems wth known propertes, e.g., ther condton or degree of dffculty (Secton ) The need to mmc actual measurement problems,.e., the constructon of problems for whch the solutons are known and whch are close to problems arsng n a partcular applcaton area Page 4 of 53

21 Testng Algorthms and Software NPL Report DEM-ES 003 The need to generate large numbers of problems to ensure adequate coverage of the space of possble nputs to the test software. Nevertheless, the testng of software usng a small number of reference pars can provde valuable nformaton about the software. To be useful, reference pars are chosen to: Be representatve of problems for whch the test software s ntended (as defned by the computatonal am) Be as dsparate as possble n terms of ther locatons n the space of possble data nputs Test partcular paths through the test software f detals of the test algorthm are known (grey-box testng). Such reference pars provde spot checks of the test software, and are useful n the sense that the software s regarded as defcent t s fals to perform adequately on any one of them. They are less effectve for black-box testng for whch lttle can be assumed about the algorthms that have been mplemented. Performance parameters that descrbe the scope of admssble nputs to the test software may be used to specfy reference pars. Sequences of reference pars may be defned for whch, wthn a gven sequence, dfferent pars correspond to dfferent choces of a partcular performance parameter (or parameters). The performance of the test software s nvestgated for such sequences of reference pars. Where a performance parameter can be nterpreted as controllng the condton or degree of dffculty of the problem represented by the reference par, the sequence becomes a graded sequence n the sense that each reference par n the sequence represents a problem that s more dffcult than the prevous member of the sequence. Use of such graded sequences can help to dentfy cases where the test software s based on a poor choce of mathematcal algorthm. In addton, the reference pars may be requred to reflect as far as possble problems that would be obtaned and used n practcal measurement. The specfcaton of the reference pars should dentfy such requrements, for example, by dentfyng performance parameters that capture these requrements and constranng these parameters accordngly Calculaton of reference pars In some cases, reference problems may be avalable for whch the correspondng reference solutons are known. For example, certan problems are amenable to an analytc treatment: a soluton to the ordnary dfferental equaton s of the general form d y + k dx y = F( x) y ( x) = Acos kx + Bsn kx, when F = 0 wth constants A and B determned by ntal and/or boundary condtons. Such reference pars are a useful source of problems for provdng spot checks but may not be representatve of the problems (that do not have analytc solutons, e.g. for other forcng functons, F) to whch the test software s to be appled. Consequently, t s necessary to consder other approaches to provdng reference pars. Page 5 of 53

22 NPL Report DEM-ES 003 Testng Algorthms and Software Reference problem data set Reference software Reference results Test software Test results Comparson Fgure 3.: Procedure for testng software usng reference software. Reference pars may be produced n two ways:. Start wth a reference problem and apply reference software to t to produce correspondng reference results: see fgure 3... Start wth some reference results and apply a data generator to them to produce a correspondng reference problem: see fgure 3.. Reference software s software wrtten to an extremely hgh standard. Such software s akn to a prmary standard n measurement, such as a klogram mass to whch secondary standards are compared for calbraton purposes. It has been stated, however, that reference software s very demandng to provde [0]. Conversely, the effort requred to produce a data generator can be a small fracton of that to produce reference software. The reason s that the mplementaton of a data generator tends to be a much more compact operaton, wth fewer numercal ptfalls, and hence the overhead of testng s much less than that for reference software. For problems wth a unque soluton there s one set of reference results correspondng to a gven reference problem. Conversely, for gven reference results there s n general an nfnte number of correspondng reference problems. Ths latter property can be used to consderable advantage n generatng reference problems, both n terms of formng problems havng selected condton numbers or degrees of dffculty, and n determnng reference problems that mmc actual problems from applcatons. As a smple example, there s a unque sample standard devaton s for a gven sample of two or more numbers, whereas gven s there are nfntely many data sets (reference problems) havng ths value as ther standard devaton. Page 6 of 53

23 Testng Algorthms and Software NPL Report DEM-ES 003 Reference results Reference problem data Test software Data generator Test results Comparson Fgure 3.: Procedure for testng software usng a data generator. To llustrate the desgn of a data generator, consder the problem specfed by the computatonal am calculate the mnmum crcumscrbed crcle for gven ponts n the xy-plane. The mnmum crcumscrbed (MC) crcle s defned as the crcle of mnmum radus that crcumscrbes all the ponts. Necessary and suffcent condtons for a crcle to be the MC crcle for gven ponts are as follows: There are two ponts that le on the crcle and form a dameter or there are three ponts that le on the crcle and form an acute-angled trangle. All other ponts le on or nsde the crcle. These condtons provde a mathematcal (and, n ths case, geometrcal) charactersaton of a soluton to the problem of calculatng the MC crcle. We can use the charactersaton to desgn a data generator for ths problem n the followng way:. Start wth a crcle defned by centre (a, b) and radus R: ths s the reference soluton.. Defne three ponts that le on the crcle and form an acute-angled trangle. For example, the ponts wth co-ordnates and ( a + R cosα, b + Rsnα ) ( a + R cos( α ± 3π / 4), b + Rsn( α ± 3π / 4) ) wll have ths property for any choce of angle α. 3. Defne all other ponts to le on or nsde the crcle. For example, the ponts wth coordnates ( a + ( R δ )cosθ, b + ( R δ ) snθ ) Page 7 of 53

24 NPL Report DEM-ES 003 Testng Algorthms and Software wll have ths property for any choce of angles θ provded 0 δ R. The soluton to the MC crcle problem for the ponts generated n ths way wll be the crcle wth whch we started n step. We note that t s easy to generate many dfferent reference problems havng the same reference soluton. Ths can be done by makng dfferent choces for α, δ and θ reference problems havng partcular propertes. For example, f we assocate the maxmum of the devatons δ wth measurement accuracy,.e., the closeness of the ponts to the soluton crcle, we can generate reference problems wth dfferent measurement accuraces by assgnng the devatons (and ther maxmum) accordngly. Most mportantly, however, the mplementaton of a data generator based on the above steps s much more straghtforward than mplementng reference software for the problem of calculatng the MC crcle for gven ponts. There are many computatonal problems n metrology for whch condtons exst to characterse ther soluton: for such problems t s often possble to desgn a data generator. A further example of a data generator, n the context of least-squares regresson of a model to expermental data, s descrbed n a later chapter Specfcaton of qualty metrcs Qualty metrcs are used to quantfy the performance of the test software for the sequences of reference pars to whch the test software s appled. Furthermore, by specfyng the requrements of the user or developer of the test software n terms of these metrcs, t s possble to assess objectvely whether the test software meets these requrements and s ft for purpose. Generally, the metrcs measure the departure of the test results returned by the test software from the reference results. The departure may be expressed n terms of the (absolute or relatve) dfference between the test and reference results, or a performance measure that accounts for factors ncludng the computatonal precson of the arthmetc used to generate the test and reference results and the condtonng or degree of dffculty of the problem defned by a reference par Problem condton The condton of a problem s a measure that descrbes the senstvty of ts exact soluton to changes n the problem. If small changes n the problem or ts data lead to small changes n the soluton, the problem s sad to be well-condtoned. Otherwse, f small changes n the data lead to large changes n the soluton, the problem s sad to be ll-condtoned for the data. Condtonng s an nherent characterstc of the mathematcal problem, and s ndependent of any algorthm or software for solvng the problem. For the very commonly used floatng-pont arthmetc, the computatonal precson η s the smallest postve representable number u such that the value + u, computed usng the arthmetc, exceeds unty. For the many floatng-pont processors whch today employ IEEE arthmetc, η = 5 0 6, correspondng to approxmately 6-dgt workng. It s unreasonable to expect even software of the hghest qualty to delver results for a problem to the full accuracy ndcated by the computatonal precson η. Ths would only n general be possble for (some) problems that are perfectly condtoned,.e., problems for whch a small change n the data makes a comparably small change n the results. Problems regularly arse n whch the ll-condtonng s sgnfcant and for whch no algorthm, however good, can provde results to the accuracy obtanable for well-condtoned problems. By usng an approprate performance measure (see below) to quantfy the loss of numercal accuracy caused by the test software over and above that whch can be explaned by the Page 8 of 53

25 Testng Algorthms and Software NPL Report DEM-ES 003 problem condton, the type of testng consdered here can mplctly dentfy the use of a poor choce of algorthm or formula from a set of mathematcally (but not numercally) equvalent forms. The calculaton of the performance measure requres knowledge of the relatve condton number κ(x) of the problem specfed by values x, defned by δy κ ( x) = y where δx denotes a small perturbaton of x and δy the correspondng perturbaton of the soluton y to the problem, and δx x, v = n v = for an n-vector v = (v, v,, v n ) T. An analyss of the computatonal am specfyng the problem s necessary to derve the relatve condton number, and such an analyss s often techncally demandng. For ths reason, t may not always be possble to evaluate the performance measure. However, the performance measure s expected to be more useful for the developer of the test software for whom nvestgatng the numercal propertes of the software s mportant, whereas for users of the software consderaton of the (absolute and relatve) qualty metrcs descrbed below s often adequate. Expressons for the relatve condton number for some smple problems, ncludng a dfference calculaton, the calculaton of sample standard devaton and the problem of lnear least-squares regresson, are avalable [3] Qualty metrcs Suppose the reference results and test results correspondng to a reference problem defned by values x of the nputs to the test software are expressed as vectors y (ref) and y (test), respectvely, of floatng-pont numbers. Let Then, y = y (test) y (ref ). d ( x) = RMS( y) () s an absolute measure of departure of the test results from the reference results for the reference problem x. Here, RMS( v ) = denotes the root-mean-square value of an n-vector v = (v, v,, v n ) T. For a scalar-valued result y or when appled to a sngle component of a vector-valued result, the measure reduces to v n (test) (ref ) d( x ) = y y. () In cases where the components of y are not comparably scaled, some care s requred when nterpretng the summary measure d(x) defned by (), and consderaton of the componentwse measure () s recommended. The number N(x) of sgnfcant fgures of agreement between the test results and reference results correspondng to reference data x s calculated from: If d(x) 0, Page 9 of 53

26 NPL Report DEM-ES 003 Testng Algorthms and Software If d(x) = 0, ( RMS( y N( x) = mn M ( x),log0 + d( x) N ( x) = M ( x). ref ) ). Here, M(x) s the number of correct sgnfcant fgures n the reference results correspondng to x, and s ncluded to account for the fact that, due to the condton of the problem, the reference results may themselves have lmted precson. In the same way as for the absolute measures of accuracy descrbed above, N(x) may be appled for the complete vector y or ts components. A performance measure P(x) s calculated, for y (ref) 0, from ( ) log. 0 ( ) ( ) y P x = + ref κ x η y The measure quantfes the performance of the test software and accounts for the followng factors: the dfference between the test and reference results gven by y the condton of the reference problem defned by x gven by the relatve condton number κ(x) the computatonal precson η. The performance measure P(x) ndcates, for the reference data set x, the number of sgnfcant fgures of accuracy lost by the test software over and above the number expected to be lost by an optmally stable algorthm. A value of P(x) close to zero ndcates that, for the reference data set x, the test software returns a test result wth a relatve accuracy that s comparable to that acheved by an optmally stable algorthm. A value of P(x) close to eght, for example, ndcates that, for the reference data set x, the test software returns a test result wth eght fewer sgnfcant fgures of accuracy than that returned by an optmally stable algorthm. A related performance measure s used n testng software for evaluatng specal functons [3] Ftness for purpose If a clam s made about the test software, the clam may be checked for the reference pars used n the testng. The qualty metrcs dscussed n secton are applcable here. For example, f the clam s that the results are accurate to three sgnfcant fgures, the value of N(x) n secton should be three or greater. The clam may orgnate from the developer of the test software or be a requrement mposed by the user of the software. In ether case, testng aganst the clam can be used as the bass of decdng whether the test software s ft for purpose. It has been noted n an earler chapter that the measurement results delvered by software are subject to data effects arsng from nexactness n data values (ncludng measurement data, reference constants, calbraton nformaton, ntal and boundary data, etc.) that defne the partcular nstance of the physcal problem requred to be solved. Suppose, as n secton 3., that X and Y denote the nput and output quanttes, regarded here as random varables, and related by the model Y = f (X), whch together defne the target computatonal am. The nexactness of the nput and output quanttes s descrbed by the (jont) probablty densty functons (PDFs) g(x) and g(y), respectvely. The problem addressed n uncertanty evaluaton s to determne g(y) gven f and g(x). Page 0 of 53

27 Testng Algorthms and Software NPL Report DEM-ES 003 In practce, the computaton of the measurement result s undertaken usng test software that s an mplementaton of an algorthm to solve a problem specfed by the test computatonal am. The output of the test software s denoted by Y (test), also regarded as a random varable, wth (test) Y = f (test) ( X). The nexactness of Y (test) s also descrbed by a PDF, here denoted by g(y (test) ) and determned from f (test) and g(x). The qualty metrcs consdered n secton focus on decdng ftness for purpose usng pont measures,.e., by comparng the measurement result y (test) = f (test) (x) delvered by the test software for the reference problem defned by x wth the reference result y (ref) = f(x) that would be delvered, for example, by reference software. However, a partcular consderaton n metrology s to decde ftness for purpose accountng for the nexactness of the nput data X. If the dfference between the test and reference results s small compared to the possble dsperson of measurement results explaned by data effects, then the test software may be regarded as ft for purpose. Such consderatons lead us to decde ftness for purpose usng statstcal measures based on comparng the PDFs g(y) and g(y (test) ), approxmatons to whch may be obtaned usng smulaton (secton 3.5). Such measures allow us to answer questons ncludng, for example:. What s the probablty of obtanng a soluton to the problem specfed by the target computatonal am that s further from y (ref) than s y (test)?. On average what s the departure of y (test) from y (ref)? For smplcty, let us suppose that the output quantty Y s a sngle scalar quantty. Then, to answer the frst queston, we wsh to evaluate (ref ) (test) (ref ) Pr( Y y y y ), where Pr(A) denotes probablty of A occurrng. Ths corresponds to fndng the area under the curve g(y) to the left of y (ref) y (test) y (ref) and to the rght of y (ref) + y (test) y (ref) (see fgure 3.3), and can therefore be evaluated gven knowledge of g(y). A small probablty suggests that y (test), the result delvered by the test software, s not a lkely soluton to the problem specfed by the target computatonal am accountng for the nexactness of the nput data values. To answer the second queston, we wsh to compare the PDFs g(y ) wth g(y (test) ) (see fgure 3.4). Partcular measures of nterest are the dfferences between the expectaton values of the random varables Y and Y (test) and between the varances of those random varables. The dfference between the expectaton values provdes nformaton about any bas between the results delvered by the test and reference software. In the example llustrated n fgure 3.4we may conclude that (a) there s an apprecable bas between the results, and (b) the dsperson of results delvered by the test software s apprecably greater than that for those delvered by reference software. Page of 53

28 NPL Report DEM-ES 003 Testng Algorthms and Software Probablty densty y (ref) y (test) y (ref) y (ref) y (ref) + y (test) y (ref) Y Fgure 3.3: Measurng the ftness for purpose of test software. Probablty densty y (test) y (ref) Y Fgure 3.4: Probablty densty functons g(y), centred on the result y (ref) delvered by reference software, and g(y (test) ), centred on the result y (test) delvered by test software. Page of 53

29 Testng Algorthms and Software NPL Report DEM-ES Complementary checks Examples of tests that do not requre the avalablty of reference pars are: Checkng consstency of functons (secton 3.4.) Checkng contnuty of functons (secton 3.4.) Checkng aganst tabulated or other values (secton 3.4.3) Checkng soluton charactersatons (secton 3.4.4). All such checks are to be regarded as complementary to and not alternatves to usng reference pars for algorthm and numercal software testng Checkng consstency of functons An example of ths type of test s the use of a forward followed by an nverse calculaton, or vce versa. For example, gven test software for sn x and test software for sn x, the values of sn {sn x} can be compared wth x for a range of values of x. It s essental to observe that ths form of checkng alone s nsuffcent. Two functons could n prncple form a nearperfect nverse par but apply to a dfferent mathematcal operaton Checkng contnuty of functons The algorthms underpnnng much mathematcal software for specal functons such as Bessel functons utlse several mathematcal approxmatons (Chebyshev seres, ratonal functons, etc.), each of whch s vald over sub-ranges of the argument(s) of the functon. By evaluatng the functon over sutable ranges of the argument(s), t s possble to gauge the extent to whch the returned values are contnuous across sub-range boundares. The presence of dscontnutes can have a damagng effect on software that uses these functons as modules, e.g., n adaptve quadrature and optmsaton. The documentaton accompanyng such software may sometmes ndcate the sub-range boundares. Otherwse, they may be nferred from the testng process by graphng the results for a dense set of nput-argument values Checkng aganst tabulated or other values For certan arguments or ranges of the argument(s) t may be possble to compare the results produced by test software wth tabulated values or the results from other software. As an example, functons of a complex varable can be reduced to real functons f the argument s purely real or purely magnary or through the use of mathematcal denttes, e.g., e x = cos x + sn x, and related formulae. In such cases alternatve software may exst whch s already well charactersed Checkng soluton charactersatons In addton to ther use as the bass of data generators, soluton charactersatons can be valuable for checkng drectly the results returned by test software []. For example, we can check whether the crcle returned by test software for calculatng the mnmum crcumscrbed crcle satsfes the (necessary and suffcent) condtons for a soluton to ths problem (secton 3.3.). The approach requres no knowledge of a reference soluton. Instead, we rely on the fact that t s possble to test the propertes the soluton s supposed to possess usng the gven charactersaton. Such tests consttute post-processng tests of the software and can be appled usng smulated test data or data arsng n the real world. Page 3 of 53

30 NPL Report DEM-ES 003 Testng Algorthms and Software 3.5 Smulaton The approaches descrbed n sectons 3.3 and 3.4 are focused on testng software usng partcular reference problems x. It s recommended that the approaches are appled for a range of reference problems typcal of those lkely to be encountered n practce. Ths can mean samplng (perhaps unformly) from the doman of all possble problems that are lkely to be encountered, ncludng those close to the boundares of ths doman. In addton, and n order to assess usng statstcal measures the ftness for purpose of the test software accountng for nexactness n the nput quanttes X (secton ), t s recommended that the approaches are appled usng reference problems sampled from the probablty densty functon g(x) that descrbes the avalable knowledge about the nexactness of the nput data. For example, reference and test software for solvng, respectvely, the problems specfed by the target and test computatonal ams may be combned wth smulaton n the followng manner: Choose a number M of trals.. For r =,, M:. Construct the data set x r as a random sample from the PDF g(x). a. Apply reference software for the target computatonal am to the data set x r to obtan reference results y r (ref). b. Apply test software for the test computatonal am to the data set x r to obtan test results y (test) r. c. Construct an approxmaton g ~ ( Y ) to the PDF g(y) usng the set of reference results y (ref) r, r =,, M. ~ ) (test 3. Construct an approxmaton g ( Y ) to the PDF g(y (test) ) usng the set of test results y (test) r, r =,, M. 4. Compare g (ref) (Y) wth g (test) (Y), for example, n the manner descrbed n secton Presentaton and nterpretaton of results Havng appled the test software to a reference problem to gve a test result, the test result s compared wth reference results correspondng to the reference problem by computng one or more of the qualty metrcs descrbed n secton If a sequence of problems s avalable correspondng to a range of values of a performance parameter (secton 3.3.), the qualty metrcs may be computed as functons of ths parameter. The values of the qualty metrcs may be presented ether n tabular form or as a graph plotted aganst the performance parameter. It s also convenent to gve summary statstcs for the computed qualty metrcs, ncludng ther arthmetc means, sample standard devatons, mnma and maxma n order to gve nsght nto how the performance of the test software depends on the performance parameter. A graph of the values of the performance parameter P(x) plotted aganst the problem condton κ(x) (or some related performance parameter), can provde valuable nformaton about the software. The resultng performance profle can be expressed as a seres of ponts (or the set of straght-lne segments jonng them). By ntroducng varablty nto each data set for each value of the performance parameter, e.g., by usng random numbers n a controlled way to smulate observatonal error, the performance profle wll become a swathe of replcated ponts, whch can be advantageous n the far comparson of rval algorthms. Ftness for purpose mples that the test software meets requrements as specfed usng qualty metrcs or other approprate measures (secton ). The user should verfy Page 4 of 53

31 Testng Algorthms and Software NPL Report DEM-ES 003 whether the reference problem(s) used can be regarded as suffcently representatve of the applcaton n whch the software s to be used. It s necessary to decde whether absolute or relatve or statstcal measures of accuracy are relevant. The performance measure P(x) ndcates the number of fgures of accuracy lost by the test software n the computng the test results compared wth the reference results (secton ). A large value may ndcate the use of an unstable parametersaton of the problem, the use of an unstable algorthm or napproprate formula, or that the test software s defectve. If for graded reference data sets taken n order the correspondng performance measures P(x) have a tendency to ncrease, t s lkely that an unstable parametersaton, formula or numercal method has been employed. Page 5 of 53

32 NPL Report DEM-ES 003 Testng Algorthms and Software 4. Dscrete Modellng Example: Gaussan Peak Fttng 4. Understandng the testng problem The frst task s to document what we know about (a) the problem the test software s ntended to solve, and (b) how the test software solves that problem. For test software to solve the problem of fttng a Gaussan peak model to expermental data, we record below (a) the target computatonal am (secton 4..), (b) the test computatonal am (secton 4..), and (c) nformaton about how the test software solves the problem specfed by the test computatonal am (secton 4..3). 4.. Target computatonal am For the example consdered here, the followng statement may be regarded as suffcent to specfy the problem the test software s ntended to solve: The test software calculates the least-squares best-ft Gaussan peak to data for whch the values of the dependent varable are subject to ndependent and dentcally dstrbuted random nose. Alternatvely, the problem may be defned n terms of the nputs to, and outputs from, the software, and an nput/output model as follows: Inputs: Abscssa values x, =,, m, and correspondng ordnate values y, =,, m. Outputs: Values of the parameters A, x and s n the model ( x x) y ( x) = Aexp () s for a Gaussan peak, and resdual devatons e, =,, m, assocated wth the model, where e = y y x ). () Input/output model: The outputs are determned from the nputs by solvng the followng (non-lnear) least-squares problem: mn m A, x, s = ( ( y y( x )), and evaluatng the resdual devatons () assocated wth the soluton model. 4.. Test computatonal am We suppose that test software s mplemented to solve a problem defned n terms of nputs and outputs as for the target computatonal am but wth a dfferent an nput/output model as follows: Input/output model: The outputs are determned from the nputs by solvng the followng least-squares problem: where I = {: y > 0}. mn A, x, s I ( ln y ln y( x )), Page 6 of 53

33 Testng Algorthms and Software NPL Report DEM-ES Implementaton of the test software A Gaussan peak model may also be wrtten n the form: ( a + a x + a x ), 0. y ( x) = exp a 0 < The values of the parameters A, x and s n the natural parametersaton () of the Gaussan model are then recovered from the values of a 0, a and a usng the followng relatonshps: A = exp a 0 a 4a, a x = a, s = a For reasons of numercal stablty, a better representaton s the centred form: where ( b + b ( x x ) + b ( x x ) ), 0, y ( x) = exp b < (3) b b A = exp b0, x x0, s, 4b = = (4) b b and x 0 s a specfed shft n the varable x. In terms of the parametersaton (3), the test computatonal am reduces to the lnear leastsquares problem: mn b I ( ln y ( b + b ( x x ) + b ( x x ) ). 0 Ths s equvalent to fndng the least-squares soluton to the overdetermned systems of lnear equatons 0 0 K b = t, (5) where t contans the elements t = ln y, I, and K s the matrx wth rows ( ( x x ) ( x x ) ). 0 0 I The test software solves the lnear least-squares problem (5) usng a matrx-factorsaton approach [7], uses (4) to obtan values for A, x and s n the Gaussan peak model (), and () to obtan values for the resdual devatons e, =,, m, assocated wth the soluton model. 4. Applyng analyss to the testng problem For the problem of fttng a Gaussan peak model to expermental data, we record below some results from mathematcal analyss appled to the computatonal ams (secton 4..), and numercal analyss appled to the test software (secton 4..). 4.. Mathematcal analyss of the computatonal ams We seek the condtons,.e., the requrements on the measurement data (x, y ), =,, m, for whch the problems defned by the target and test computatonal ams gve (a) dentcal solutons, and (b) comparable solutons. Of course, even f the problems delver comparable estmates of the values of the parameters for A, x and s, the uncertantes assocated wth the estmates may be qute dfferent. We do not consder here the condtons under whch the problems delver estmates and assocated uncertantes that are comparable.. Page 7 of 53

34 NPL Report DEM-ES 003 Testng Algorthms and Software Defne f = ln y ln y( x ), I. Clearly, f the data s such that y = y( x ),,..., m, = for certan values of A, x and s,.e., there s no error assocated wth the measurements of the dependent varable, then the problems wll delver dentcal solutons, for n ths case I = {,,, m} and e = f = 0, =, K, m, at the soluton. Now, and so by a frst order Taylor seres expanson ln y = ln( y( x ) + e ), ln y ln y( x ) + e. y( x ) Therefore, f e. y( x ) Only f x, =,, m, are such that the values y(x ) are essentally constant can we expect the problems to be approxmately equvalent. Ths s the case, for example, f the measurements are restrcted to a tal of the underlyng Gaussan peak functon. However, n such crcumstances, the data wll not defne the underlyng Gaussan peak functon very well, and the problem of determnng the parameters of ths functon can be expected to be llcondtoned. 4.. Numercal analyss of the test software For gven values of x, =,, m, the matrx K n the overdetermned system of equatons (5) and, n partcular, the relatve condton number κ of the problem defned by those equatons, are functons of the value of the shft x 0. As an example, let m = 0 and x be chosen to be unformly-spaced n the nterval from 4 to +4. Table 4. lsts values (to the nearest nteger) of κ correspondng to dfferent choces of the shft x 0. x κ Table 4.: For gven shft x 0, values (to the nearest nteger) of the relatve condton κ number of the problem defned by the overdetermned system of equatons (5). As x 0 s ncreased, so the problem defned by the equatons (5) becomes ncreasngly llcondtoned. Consequently, the numercal performance of any software to solve the problem defned by the equatons (5) can be expected to be dependent on the choce of x 0. If a matrxfactorsaton approach s used to solve the equatons (5), as t s stated n secton 4..3, an upper bound on the number of decmal dgts of accuracy that can be expected to be lost by software to solve the equatons (5) s log 0 κ If an approach based on formng and solvng the normal equatons was nstead used, an upper bound s gven by log 0 κ. Page 8 of 53

35 Testng Algorthms and Software NPL Report DEM-ES Applyng reference pars to the testng problem The focus of ths example s on applyng reference pars to the testng problem defned n secton 4.. We record below the performance parameters (and ther values) used to specfy reference pars (secton 4.3.), the approach to calculatng reference pars (secton 4.3.), and the qualty metrcs used to compare reference and test results (secton 4.3.3) Specfcaton of reference pars Performance parameters for the Gaussan peak fttng problem nclude () the peak heght A, () the peak locaton x, () the peak wdth s, (v) the data nose σ, (v) the number m of abscssa values x, (v) the shft parameter x 0 n the Gaussan peak model (7), and (v) the data wdth w,.e., the sem-range of the x values, gven by ½(max{x } mn{x }). Performance parameters () () are used to defne the model () n the specfcaton of the target computatonal am (secton 4..), and therefore control the range of solutons returned by the test software. Performance parameter (v) s a parameter of the algorthm mplemented by the test software, and nfluences the numercal performance of the algorthm (secton 4..). The remander of the performance parameters relate to propertes of the nput data provded to the test software. The results descrbed n secton 4.4 relate to reference pars defned by values of the above performance parameters set as follows: m A =, x = 000, s =, 0 σ 0.0, m = 0, x0 = 0 or x, w = 3s, m = wth x, =,, m, unformly-spaced n the nterval from x 3s from x + 3s Calculaton of reference pars A soluton to the problem specfed by the target computatonal am s characterzed by the condton where J s the matrx of partal dervatves e b j J T e = 0,, e = y y( x ), =,..., m, j =,,3, and both J and e are evaluated at the soluton (defned by parameter values b). Based on ths condton, a procedure for generatng reference data y, =,, m, for whch the soluton to the problem specfed by the target computatonal am (secton 4..) s gven a pror takes the followng form [3, 3, 6]. Gven values for A, x, s, x 0, {x : =,, m}:. Form values for parameters b from those for A, x and s.. Form the model values y(x ), =,, m. 3. Form the Jacoban matrx J. 4. Form a set of target resdual devatons e 0,, =,, m. 5. Form the null space N for J T. 6. Form the resdual vector e = Nu, where u = N T e Form the measurement values y = y(x ) + e, =,, m. Page 9 of 53

36 NPL Report DEM-ES 003 Testng Algorthms and Software The resultng data set wll have a known soluton (defned by the gven values for A, x and s) and wll be charactersed by a known vector e of resdual devatons that s close (n a leastsquares sense) to the vector e 0 of target resdual devatons. For example, the target resduals may be chosen to be random samples from a Gaussan (normal) dstrbuton wth mean zero and varance σ, and thus to mmc measurement devatons that are lkely to be encountered n practce. A (smlar) procedure for generatng reference data for whch the soluton to the problem specfed by the test computatonal am (secton 4..) s gven a pror may also be desgned. Detals of such a procedure are avalable [3] Specfcaton of qualty metrcs Let e ref wth elements e ref, =,, m, be reference values for the resdual devatons e, and e test wth elements e test, =,, m, be the correspondng values returned by the test software. The followng qualty metrcs are used for the Gaussan peak fttng problem [3, 3]: d = e ref e m test, e ref e test = m ref test ( e e ), the root-mean-square (r.m.s.) devaton between the reference and test results, and P = log d m 0 +, y = η y = = m a performance measure that ndcates the number of sgnfcant decmal fgures of accuracy lost by the test software compared wth a reference mplementaton of software to solve the problem specfed by the computatonal am. 4.4 Smulaton Gven values for A, x, s, x 0, {x : =,, m}, σ and M, repeat the followng steps for r =,, M:. Construct the data set (x r, y r ), =,, m, by evaluatng y r ( x x) = Aexp s + e r, y, =, K, m, where e r s a random sample from a Gaussan (normal) dstrbuton wth mean zero and varance σ.. Apply the test software (wth shft x 0 set equal to the arthmetc mean of the data abscssa values) to the data set to obtan results A r r, x,s r. The statstcal model chosen n the above procedure for the random samples e r s consstent wth the target computatonal am (secton 4..) that states that the values of the dependent varable are subject to ndependent and dentcally dstrbuted random nose. The results provded by the procedure may be used to construct approxmatons to the PDFs for the values of the model parameters A, x and s, for example dsplayed as hstograms (scaled frequency dstrbutons). The standard uncertanty assocated wth the estmate of the value of the model parameter A provded by the test software s gven by the standard devaton of the values A r, r =,, m (and smlarly for x and s). Results provded by the procedure are presented n secton 4.5 for M = 50,000 trals wth the values of the performance parameters set as follows: Page 30 of 53

37 Testng Algorthms and Software NPL Report DEM-ES 003 m A =, x = 000, s =, σ = 0.0, m = 0, x0 = x, w = 3s, m = wth x, =,, m, unformly-spaced n the nterval from x 3s from x + 3s. We would lke to compare the standard uncertantes wth those that are assocated wth estmates of the values of the model parameters provded by reference software for solvng the problem specfed by the target computatonal am. Ths would allow us to nvestgate whether the qualty of the measurement results s unduly affected by the choce of test computatonal am and software to solve the problem specfed by that computatonal am. If reference software s avalable, we can use smulaton (as above) to provde the standard uncertantes assocated wth the soluton to the problem specfed by the target computatonal am. Otherwse, analyss of the problem specfed by the target computatonal am may be used to provde the necessary nformaton (at least approxmately). For the problem consdered here, the uncertanty (covarance) matrx assocated wth estmates of the soluton to the problem specfed by the target computatonal am s [5, 9] T ( J ), V = σ J where J s the matrx of partal dervatves (Jacoban matrx) defned n secton The dagonal elements of ths matrx contan the squares of the requred standard uncertantes. 4.5 Presentaton and nterpretaton of results Fgure 4. shows how the values of the qualty metrcs d and P (secton 4.3.3) vary wth the performance parameter σ for the case that the shft x 0 s set equal to the arthmetc mean of the data abscssa values. The results shown relate to reference data for whch the soluton to the problem specfed by the test computatonal am (secton 4..) s gven a pror. Fgure 4. shows the same nformaton but for the case that the shft x 0 s set equal to zero. The results ndcate that for the reference data sets consdered here the test software wth x 0 equal to the arthmetc mean of the data abscssa values performs almost as well as a reference mplementaton of software for solvng the problem specfed by the test computatonal am. On the other hand, the test software wth x 0 equal to zero s losng between fve and sx sgnfcant fgures of accuracy compared wth reference software for solvng ths problem. The test software wth x 0 set equal to zero cannot be regarded as a reference mplementaton. In both cases there s no strong dependence of the values of the qualty metrcs d and P on the performance parameter σ. The results shown n fgure 4. and fgure 4., and dscussed above, concern numercal software testng of the test software aganst the test computatonal am. In contrast, the results shown n fgure 4.3 and fgure 4.4 are based on comparng the results returned by the test software wth reference results for the problem specfed by the target computatonal am. The results ndcate that the test software, rrespectve of the way the shft x 0 s set, does not provde accurate solutons to the problem specfed by the target computatonal am. Because the test software solves the problem specfed by the test computatonal am to an accuracy that s apprecably greater than that for the problem specfed by the target computatonal am, we conclude that the results shown n these fgures are predomnantly a consequence of the dfference between the computatonal ams rather than beng attrbutable to the software mplementaton. Ths concluson apples rrespectve of how the shft x 0 s set, and we are therefore able, n ths example, to deduce useful nformaton about the computatonal ams from a poor algorthm for solvng the problem specfed by the test computatonal am. The results shown n fgure 4.3 and fgure 4.4 concern algorthm testng of the test computatonal am aganst the target computatonal am. Fgure 4.5 shows approxmatons to the PDFs for the values of A, x and s constructed usng the results returned by the smulaton procedure descrbed n secton 4.4. Table 4. gves, n Page 3 of 53

38 NPL Report DEM-ES 003 Testng Algorthms and Software ts second and thrd columns, the mean and standard devaton of the values A r, r =,, m (and smlarly for x and s) obtaned from that procedure. The fourth column of the table contans the standard uncertantes assocated wth estmates of the values of the model parameters obtaned as the soluton to the problem specfed by the target computatonal am obtaned by analyss (secton 4.4). The results n the table show that the standard uncertantes assocated wth measurement results obtaned by solvng the problem specfed by the test computatonal am (usng the test software) are apprecably greater (an order of magntude) than those obtaned by solvng the problem specfed by the target computatonal am. There s a hgh probablty of obtanng a soluton to the problem specfed by the test computatonal am that has a low probablty of arsng as a soluton to the problem specfed by the target computatonal am. The mean values, accountng for the assocated standard uncertantes, are not apprecably dfferent from the known values of the model parameters and, consequently, the measurement results obtaned usng the test software would appear not to be based. However, we note that the PDFs for the values of the model parameters A and s are asymmetrc. Parameter Table 4.: Test computatonal am Target computatonal am Mean value Standard uncertanty Standard uncertanty A x s Results of smulaton (based on M = 50,000 trals) appled to the problem specfed by the test computatonal am, and analyss appled to the problem specfed by the target computatonal am. Page 3 of 53

39 Testng Algorthms and Software NPL Report DEM-ES x Root-mean-square devaton Performance measure Measurement nose σ Measurement nose σ Fgure 4.: Testng the numercal correctness of the test software to solve the problem specfed by the test computatonal am. The model parameter x 0 s set equal to the arthmetc mean of the data abscssa values x, =,, m. x Root-mean-square devaton Performance measure Measurement nose σ Measurement nose σ Fgure 4.: As fgure 4., wth the model parameter x 0 set equal to zero. Page 33 of 53

40 NPL Report DEM-ES 003 Testng Algorthms and Software Root-mean-square devaton Performance measure Measurement nose σ Measurement nose σ Fgure 4.3: Performance of the test software to solve the problem specfed by the target computatonal am. The model parameter x 0 s set equal to the arthmetc mean of the data abscssa values x, =,, m Root-mean-square devaton Performance measure Measurement nose σ Measurement nose σ Fgure 4.4: As fgure 4.3, wth the model parameter x 0 set equal to zero. Page 34 of 53

41 Testng Algorthms and Software NPL Report DEM-ES Frequency Value of ampltude Frequency Value of peak poston Frequency Value of peak wdth Fgure 4.5: Approxmatons to the PDFs for the values of A, x and s produced by the test software. Page 35 of 53

42 NPL Report DEM-ES 003 Testng Algorthms and Software 5. Contnuous Modellng Example: Heat Transfer 5. Understandng the testng problem As wth any software testng, the frst stage n testng contnuous modellng software s to defne the computatonal am. The computatonal am of a contnuous modellng software package wll nclude some or all of the followng: The (system of) dfferental equatons to be solved The geometry of the doman Materal propertes wthn the doman Intal, boundary and loadng condtons Informaton about the way the geometry s meshed The frst of these ponts, the specfcaton of the dfferental equatons, must be ncluded. The other four ponts are less mportant snce most software packages can handle a range of geometres, materal propertes, and boundary condtons. In some cases t s more straghtforward to gve a physcal descrpton of the computatonal am, but the descrpton must stll be unambguous. The computatonal am can be splt nto two parts: the target computatonal am and the test computatonal am. For the vast majorty of contnuous modellng software, the target computatonal am s the soluton of some set of ntegral or dfferental equatons, and the test computatonal am specfes the method that wll be used to generate a numercal approxmaton to the soluton of the equatons. The nature of the method s explaned more fully n the secton on the mplementaton of the test software, as a thorough understandng of the software can lead to mproved test desgns. 5.. Target computatonal am A physcal defnton of the target computatonal am of the test software could be The test software solves the transent heat equaton n two dmensons for a prescrbed doman, tme nterval, thermal conductvty, densty, specfc heat capacty, and set of boundary and ntal condtons, gvng values of temperature throughout the doman and tme nterval. A full mathematcal defnton of the same problem could be The test software obtans values of T(r, t) by solvng t α T ( ρ( r,t,t) c ( r,t,t) T ) =. ( λ( r,t,t) T ) + q( r,t,t) p ( r,t) T + β( r,t). T = f ( r,t,t) = T () r, r Ω,t 0, 0 =, r Ω, 0 t t,, r Ω, 0 t t for specfed functons α, β, ρ, c p, λ, q, f, and T 0,a specfed value of t, and a specfed twodmensonal doman Ω. The mathematcal defnton gven above s very general but t can be made more relevant to the underlyng physcal problem by restrctng the boundary condtons: not every choce of functon for α, β, and f corresponds to a physcally meanngful boundary condton. In general, heat transfer boundary condtons model ether fxed temperatures, fxed heat fluxes, convectve transfer, or radatve transfer. Smlarly, realstc assumptons could be made about Page 36 of 53

43 Testng Algorthms and Software NPL Report DEM-ES 003 ρ (densty), c p (specfc heat capacty), and λ (thermal conductvty) as they are physcal propertes of a materal. Wth ths n mnd, the computatonal am could be wrtten as The test software obtans value of T(r, t) by solvng { ρ( r,t,t) c ( r,t,t) T} =. ( λ( r,t,t) T ) + q( r,t,t) t T = T λ λ λ b ( r,t) ( r,t,t) = q ( r,t) ( r,t,t) = h( r,t) ( T T ( r,t) ) ( ), 4 4 ( r,t,t) = σε ( r,t) T T ( r,t) T = T 0, T n r T n r T n r p r Ω () r, r Ω,t = 0 b, fxt, 0 t t, r Ω amb amb fxq,, 0 t t, r Ω conv r Ω,, 0 t t, rad r Ω, 0 t t,, 0 t t, for specfed functons, λ, ρ, c p, q, T b, q b, T amb, h, σ, ε and T 0, a specfed value of t, and a specfed two-dmensonal doman Ω, where { Ω fxt, Ω fxq, Ω conv, Ω rad } s a partton of the boundary Ω. Ths defnton s stll not couched n physcal terms. For nstance, t s not stated that r s poston, that T s temperature, or that t s tme. However, the expressons used n the defnton of the computatonal am are all analogous to physcal aspects of heat transfer. 5.. Test computatonal am The test software used n our examples s fnte element software, and the examples use second-order soparametrc elements. The test computatonal am s then Solve the transent heat equaton n two dmensons for a prescrbed doman, tme nterval, thermal conductvty, densty, specfc heat capacty, and set of boundary and ntal condtons, usng second-order soparametrc fnte elements, to produce temperature values throughout the doman and tme nterval Parameters related to the mesh defnton, whch have a strong effect on the results, wll be defned wthn each of the tests Implementaton of the test software Second-order fnte elements are ether trangles or quadrlaterals whch may have curved sdes. A dscretsed verson of the dfferental equatons an the boundary condtons s created by mappng the elements n the actual doman to the canoncal elements shown n fgure 5., approxmatng the temperature by wrtng T ( ξ, η) T ( ξ )( η)( + ξ + η) + T ( ξ )( η)( + ξ ) T 4 + T + T ( + ξ )( η)( ξ + η) + T ( ξ )( η)( + η) 4 4 ( + ξ )( η)( + η) T ( + ξ η)( + η)( ξ ) ( ξ )( + η)( + ξ ) T ( + ξ )( + η)( ξ η) Page 37 of 53

44 NPL Report DEM-ES 003 Testng Algorthms and Software for quadrlateral elements, or T ( ξ, η) T ( ξ η)( ξ η ) + 4T ( ξ η) ξ + T3ξ ( ξ ) + 4T η( ξ η ) + 4T ξη + T η( η ) for trangular elements, applyng ths approxmaton to the heat equaton and boundary condtons, and combnng all the local approxmatons n a large matrx of lnear equatons. In most cases the temperature s approxmated usng ponts that le wthn the elements, because certan choces of pont provde more accurate approxmatons for ntegral calculatons, but the user generally requres results to be calculated at the nodal ponts, so there may be an nterpolaton stage followng the soluton of the matrx equatons. The approxmatons above have been wrtten n terms of the nodal ponts because t s an easer form to wrte and to understand. Note that the quadrlateral polynomal s hghest order term s of degree 3. Other fnte element formulatons exst that preserve contnuty of frst dervatves at the nodal ponts, but they are not consdered here. η η T 6 T 7 T 8 T 6 T 4 T 5 - T 4 T 5 ξ T T 3 0 ξ 0 T - T T T 3 Fgure 5.: Canoncal second-order fnte elements. The most mportant steps n the software mplementaton are the formulaton of the approxmatons wthn the elements and the assembly and soluton of the matrx of lnear equatons. These steps drve the forms of test that are suggested for use wth contnuous modellng software. 5. Applyng analyss to the testng problem Applyng analyss to the computatonal ams and mplementatons of contnuous modellng software can ndcate forms of test that could be useful. In partcular, t may be possble to fnd analytc solutons to smplfed versons of the computatonal ams, or to explot features of the software to generate problems wth well-understood reference solutons. 5.. Mathematcal analyss of the computatonal ams The test computatonal am and the mplementaton of the software show that the local approxmaton of the temperature s ether a quadratc polynomal or a cubc one, dependng on whether the element s a trangle or a quadrlateral. All quadratc functons of x and y obey Page 38 of 53

45 Testng Algorthms and Software NPL Report DEM-ES 003 the steady-state heat equaton f the materal propertes are unform and constant and the x and y coeffcents are of equal magntude and opposte sgn. Hence f a problem s specfed such that the boundary condtons are the same quadratc functon of poston everywhere on the boundary of the doman, and the materal propertes are constant and unform, then the test computatonal am and the target computatonal am wll be dentcal for any shape of doman. Ths feature can be used to defne a set of doman tests. The most general form for the dfferental equaton used n the target computatonal am of the software s t ( ρ ( r,t,t) c ( r,t,t) T ) =. ( λ( r,t,t) T ) + q( r,t,t), r Ω, p but f the materal propertes (λ, ρ, and c p ) are constant and unform and there s no nternal heat generaton (q = 0), the equaton can be smplfed to T t λ = T, ρc p r Ω. Hence f a problem can be developed wth constant unform propertes and boundary condtons that are ndependent of the materal propertes, then any two problems that have the same boundary condtons, the same doman, and the same value of λ/ρc p wll have the same soluton. Ths s an example of a property of the computatonal am that can be used to defne a set of scalable tests. 5.. Numercal analyss of the test software Varous numercal technques and results are avalable that gve an upper bound on the accuracy of fnte element methods. In general these technques are focussed on the dfferences between a perfectly computed nfnte precson numercal soluton and the analytc soluton (the dscretsaton error ), rather than the dfference between a real-world fnte precson soluton and the analytc soluton. The technques usually produce an upper bound on the dscretsaton error n terms of some property of the mesh. A set of tests could be defned that took a sngle example of the target computatonal am and solved t usng a set of meshes that have known values for the property used n the dscretsaton error estmate, and the dfference between the expected behavour and the actual behavour could be examned. Ths technque s partcularly useful for regular meshes wthn rectlnear geometres, snce bounds are generally tghtest for such meshes. Ths technque was dscussed as a model valdaton technque n an SSfM report on contnuous model valdaton [3]. It was mentoned n secton 5..3 that the two key steps n fnte element soluton of a problem are element formulaton and soluton of the matrx of lnear equatons. Whlst both of these steps wll affect the accuracy of the soluton to all problems, t s lkely that the matrx soluton step wll be more accurate for problems wth a few elements than problems wth a large number of elements. It s often straghtforward to wrte down analytc solutons to the test computatonal am for problems that only feature a few elements, even for non-lnear problems. Hence t can be easy to create sets of reference problems featurng only a small number of elements n order to test element formulaton. 5.3 Applyng reference pars to the testng problem The defnton of a reference problem for contnuous modellng software must contan a defnton of the equatons, doman, materal propertes and boundary or loadng or ntal condtons, as well as the nature of the target results. All of these quanttes are requred to Page 39 of 53

46 NPL Report DEM-ES 003 Testng Algorthms and Software specfy a problem wth a unque soluton. In many cases the problem defnton can be clarfed by ncludng a dagram of the doman and boundary condtons. If the reference problem s to represent the test computatonal am, some nformaton about the mesh or other dscretsaton needs to be suppled as part of the test. In broad terms, more ponts n a dscretsaton produce a more accurate soluton, so ether the mesh needs to be explctly defned, or an approxmate number of ponts to be used n the mesh must be gven. If the software beng tested generates dscretsatons automatcally, then the approxmate number of ponts s the only opton. If the user defnes the mesh ponts, t s preferable to defne the mesh explctly so that the accuracy of the results wll not be affected by the choce of ponts by the user. A dstncton s drawn n ths secton between specfcaton of reference pars and calculaton of reference pars. Specfcaton n ths case s used n the sense of specfyng a famly of tests, by specfyng whch of the nput quanttes are to be vared to generate the famly of tests, and calculaton s the process of defnng the values of those nput quanttes and the correspondng reference soluton. Some tests, partcularly small-scale tests, are not part of a famly of tests and are defned as a stand-alone entty. The reference par specfcaton and calculaton stages are replaced by a sngle reference par defnton. Ths approach s commonly taken when reference software has been used to generate the reference par: rather than specfyng a generc form for the reference problem, a sngle problem ncludng all nput quantty defntons and ts reference soluton are suppled. An example of ths s the frst test that has been appled to the test software. The test llustrates the dea of a small test usng a NAFEMS benchmark that s based on a problem wth an analytc soluton derved n []. The problem as specfed n the benchmark s: Consder a plate of unform thckness, measurng 0.6 m by.0 m. On one short edge the temperature s fxed at 00 C, and on one long edge the plate s perfectly nsulated so that the heat flux s zero along that edge. The other two edges are losng heat va convecton to an ambent temperature of 0 C. The thermal conductvty of the plate s 5.0 Wm - K -, and the convectve heat transfer coeffcent s 750 Wm - K -. There s no nternal generaton of heat. Calculate the temperature 0. m along the un-nsulated long sde, measured from the ntersecton wth the fxed temperature sde. The reference result s 8.5 C. Ths s llustrated n fgure 5., along wth the mesh that the NAFEMS benchmark specfes. Perfect nsulaton: λ T/ x = 0 Fxed temperature: T = 00 C Convecton to ambent: λ T/ x = h(t T 0 ) Reference result s temperature here Fgure 5.: Illustraton of the frst reference problem. Ths problem has a suffcently small number of elements that the matrx soluton step should not affect the calculated results to a sgnfcant degree. Accordng to the NAFEMS gude to ths benchmark [], Ths s a good test for a general steady state heat conducton problem, Page 40 of 53

47 Testng Algorthms and Software NPL Report DEM-ES 003 as t specfes a combnaton of boundary condtons whch results n a hghly varable temperature feld n two dmensons Specfcaton of reference pars The remanng tests used n ths example can have ther defntons separated nto specfcaton and calculaton stages. The second set of tests s an example of a scalable set of tests. The am of these tests s to test the ablty of the software to solve problems wth a wde range of values for the nput parameters. The reference problem s an axsymmetrc transent problem: Consder a unform cylnder of unt radus and unt length. Intally one end of the cylnder s at 0 and the rest of the cylnder s at 0. All sdes of the cylnder are perfectly nsulated throughout. The mesh to be used s an evenly-spaced array of 0 quadratc elements n the radal drecton and 0 quadratc elements n the axal drecton, gvng a total of 34 nodes. Note that unts are not specfed as the problem s ndependent of the choce of unts. If the thermal conductvty s λ, the densty s ρ, and the specfc heat capacty s c p, fnd the temperature at the pont r = 0, z =, at tmes t = {0.0, =,,, 00}. The reference results are gven by T = 3 m= [ t] () m t + { } exp α ( mπ ) where α = λ/(ρc p ) s the thermal dffusvty. Note that the problem requres ntal condtons to be specfed because t s a transent model. The problem s llustrated n fgure 5.3. The set of tests s generated by varyng the ndvdual values of λ, ρ, and c p, whlst holdng the rato λ/(ρc p ) fxed. The propertes λ, ρ, and c p are beng used as performance parameters to parameterse the space of admssble nputs to the test software. Ths problem s related to a laser flash heatng model that smulates a method used to determne thermal dffusvty expermentally. Length = z = 0 z = r = Intal temperature 0 Radus = T nt = 0 T nt = 0 r = 0 Soluton of nterest (a) (b) Fgure 5.3: Illustraton of the second reference problem. Page 4 of 53

48 NPL Report DEM-ES 003 Testng Algorthms and Software The thrd set of tests s based on a smple test expressed n a generc mathematcal form. The problem s to solve T = 0, T r Ω, ( x, y) = a ( x y ) + a xy + a x + a y + a, r = ( x, y) Ω, for Ω a two-dmensonal doman and the a a set of constants. Ths specfcaton leads to a famly of tests generated by varyng Ω, so ntally no doman geometry s specfed explctly. The doman les wthn the square 0.05 x, y.05 and conssts of 00 elements. The method used for creatng the mesh s outlned n the next secton. The requred results are the values of T at all the ponts used to calculate the soluton, and the reference results are gven by T * ( x, y ) = a ( x y ) + a x y + a x + a y + a, x Ω, 3 4 where Ω* s the set of ponts that make up the dscrete approxmaton to the doman used n the calculatons, not ncludng ponts that le on the boundary Calculaton of reference pars The second set of tests as specfed n the prevous secton requres calculaton of three parameters (λ, ρ, and c p ) subject to a constrant (λ/(ρc p ) held fxed). The parameter values used n ths case were based on ntal values. p λ = 0 0, ρ =. 8 0,c = , gvng α 0.45, a value that s lkely to create a well-condtoned problem. Intally λ and α were both held fxed and ρ and c p were vared by factors of 0: ρ was vared between and.8 0 0, and hence c p was vared between and Then λ, ρ, and c p were all vared wth α held fxed, such that λ vared between and.0 0 0, ρ vared between and.8 0 5, and c p vared between and such that the modulus of dfference between the exponents of ρ and c p was less than or equal to at all tmes. A full lst of all parameter sets used n the tests can be found n an earler report [7]. The fnal set of tests were defned by creatng a mesh of 00 second order elements (34 nodes, corner nodes and 0 md-sde nodes) that covered the whole of the square 0 x, y and then allowng the nodes to move by dfferent amounts to perturb the mesh. A total of eleven meshes, ncludng the ntal regular mesh, were used. The procedure for creatng the meshes was:. For the th mesh, set r = ( ), =,,.. Generate the unperturbed corner nodes c j = (x j, y j ), j =,,, such that they defne a unform 0 by 0 mesh on the unt square. 3. Generate random numbers s j, j =,,,, from a unformly dstrbuted random varable on [0, π]. 4. Generate the th perturbed corner nodes from (c j ) = (x j + r cos s j, y j + r sn s j ) and round to fve decmal places. 5. Create md-sde nodes half-way along the straght lnes connectng c j to c j-, c j+, c j-, and c j+. Ths procedure produces elements wth straght sdes, rather than elements that requre quadratc expressons to descrbe them accurately. Ths choce was made so that the element shape could not get too far away from a square (the deal shape for a quadrlateral fnte element). Snce the soluton to the problem vares quadratcally, the second-order elements descrbed n secton 5..3 are stll necessary. 5 Page 4 of 53

49 Testng Algorthms and Software NPL Report DEM-ES 003 Ths mesh generaton procedure wll lead to an ncreasngly dstorted set of meshes. It s expected that the more dstorted the mesh s, the worse wll be the results. The maxmum and mnmum nternal angle of all the elements s a reasonable measure of mesh dstorton: an deal mesh has all angles at 90, and a mesh s generally regarded as acceptable f all ts nternal angles le between 45 and 35. The dstrbuton of nternal angles has been calculated for each mesh, and wll be dscussed further n secton 5.4. For each mesh, sx sets of random values of the a k, k =,,, 5 were chosen by samplng a unformly dstrbuted random varable on [,0], and roundng to two decmal places, and each set of values were used wth each mesh. All boundary condtons and reference results were truncated to eght decmal places because the software gnores all fgures after the eghth. The random values used were: Test number a a a 3 a 4 a Table 5.: Parameters used to defne the reference solutons. All of these values gve rse to results between 0 and 40. The performance parameters n ths set of tests are the coeffcents a, and the radus r j used to perturb the nodal postons Specfcaton of qualty metrcs The choce of qualty metrcs for contnuous models s the same as that for dscrete models. It should be noted that almost all contnuous models produce a large number of results, so the qualty metrcs that can be appled to vectors of results are usually the most useful. Under some crcumstances, t can be worth usng dfferent qualty metrcs for dfferent sets of the results. In partcular, f the model results are close to zero n one regon but not another, t may be helpful to apply an absolute metrc where the results are small and a relatve metrc elsewhere. Condton numbers, as dscussed n secton , can be dffcult to obtan for contnuous models, because the mplct nature of most contnuous models makes obtanng senstvty coeffcents dffcult. Condton of the problem has not been consdered explctly for these problems, but all choces of parameters have been restrcted to produce reasonable values of the results. For nstance, the reference results for all the tests all le between 0 and 00, so there should be lttle problem wth comparatve scalng of results n dfferent regons of the doman. The reference results n all the tests are O(0), and so the qualty metrcs used for these examples s, for T ref the reference results and T test the test results M M = T test ( x,t) T ( x,t), ref where M s the number of nodes of nterest and the summaton could potentally be extended to average over tme as well. Page 43 of 53

50 NPL Report DEM-ES 003 Testng Algorthms and Software The performance measure N ( x,t) mn 8, log = 0 + T ref Tref ( x,t) ( x,t) T ( x,t) 8, test, T T ref ref ( x,t) T ( x,t) test ( x,t) T ( x,t) test > 0, = 0, s also used, where 8 s chosen because the reference results are known to be accurate to 8 fgures. 5.4 Complementary checks It s possble to take the dscrete results produced by the software, use them to construct an approxmaton to the second dervatves of T, and check that ths approxmaton obeys the dfferental equaton. Ths s a consstency check as descrbed n secton However, for non-rectangular meshes ths becomes qute trcky. Hence the check has only been carred out for the frst mesh used n the thrd set of tests as ths s a two-dmensonal rectangular mesh. On average, the calculated values of the dfferental T lay between and These values were obtaned wthout attemptng to reconstruct the detaled quadratc approxmatons used n the model. Instead a smple-mnded approxmaton was constructed usng the nodal values. Gven the approxmaton used, results were about as good as could be expected. The transent tests could have ther conservaton of energy checked to test the soluton charactersaton. Snce the test results were ndependent of the nput parameters (see secton 5.6), ths was only done for a sngle test. Values were found to be constant to seven sgnfcant fgures, whch s a good result. 5.5 Smulaton Smulaton s often avoded for contnuous modellng problems, because the necessary repeated runs of the model can be too computatonally expensve to justfy. However, some software tools now offer automated runnng and processng of multple parametersed jobs n order to explore the desgn space of a model, whch can be used as a tool for carryng out senstvty analyses as descrbed n secton 3.5. A smulaton exercse on a thermal model s beng carred out as part of a dfferent SSfM project, and wll be reported n the forthcomng SSfM Good Practce Gude to contnuous modellng. The am of the smulatons descrbed n secton 3.5 s smlar to the ams of the scalable tests descrbed above: to explore the full ranges of the lkely nput values. Scalable tests am for coverage of the full range rather than generaton of statstcal nformaton, but the ams are stll smlar. 5.6 Presentaton and nterpretaton of results As was mentoned n secton 5.3, t s common for contnuous models to produce results at a large number of ponts. Generally the accuracy of the results s lkely to vary as a functon of poston. Hence t s often useful to present the qualty metrc values as a graph or surface plot. Where possble ths has been done n the examples. As was mentoned n secton 5..3, results are often calculated at nternal ntegraton ponts and then extrapolated to nodes. Outputtng results at the ntegraton ponts and checkng ther accuracy can gve an ndcaton of the accuracy of the extrapolaton algorthm s. Ths has not been done for any of the tests descrbed here, because the extrapolaton algorthm wll be exact for the solutons to the only tests where reference results at doman ponts are avalable (the doman tests). Page 44 of 53

51 Testng Algorthms and Software NPL Report DEM-ES 003 The frst test only has a sngle reference result, and the reference value s only gven to one decmal place. In ths case, t s suffcent to present the reference and test results sde by sde wthout further plottng or calculaton of reference measures. The test result was 7.9, and the reference result was 8.3. The error relatve to the reference result s.%. Further comparsons to the analytc soluton show that the maxmum error over the entre geometry s about 8%. Gven that the soluton s strongly non-lnear and the mesh only has 6 nodes, ths s regarded as acceptable performance. The second set of tests can be presented n several dfferent ways. The frst pont to note s that all the trals (4 n total) produce the same results to 4 sgnfcant fgures. Ths smlarty means a sngle set of test results can be used to represent all the test results. The reproducblty ndcates that the matrx equatons are probably constructed usng the parameter λ/(ρc p ) rather than by constructng one matrx that s dependent on λ and another dependng on ρc p, and nvertng one. Ths behavour s expected of a good mplementaton. A plot of the reference results and a typcal set of test results versus tme s shown n fgure 5.4. It s clear that agreement s good at all tme steps. The dfference between the reference and test results s plotted n fgure 5.5. Unts are not gven on ether graph because the test s dmensonless. The maxmum absolute dfference at a tme s.8 0-3, and the qualty metrc defned n secton 5.3 has a value of Reference results 0.3 Test results 0.5 Temperature Tme Fgure 5.4: Reference results and a typcal set of test results from the second set of tests. Page 45 of 53

52 NPL Report DEM-ES 003 Testng Algorthms and Software Reference results - test results Tme Fgure 5.5: Varaton of T ref (t) T test (t) plotted aganst tme. Plottng the performance measure defned n secton aganst tme gves an ndcaton of how the number of fgures of agreement between the test result and the reference result vares over tme. A plot of N(t) s shown n fgure 5.6. The ntal poor agreement s partly caused by the small values of the reference result. Agreement mproves as tme ncreases because the FE model s convergng to the same steady state as the analytcal soluton. It s possble that the test results could be mproved by usng a fner mesh. Ths possblty has not been nvestgated N (t ) Tme Fgure 5.6: Performance measure N(t) plotted aganst tme. Page 46 of 53

53 Testng Algorthms and Software NPL Report DEM-ES 003 The thrd set of tests generated sx sets of results for eleven dfferent meshes. As was mentoned n secton 5.3., the qualty of the results s lkely to depend on the nternal angles of the elements n the mesh. The angles for all elements n all meshes were calculated, 400 angles altogether, and the statstcs summarsng the results are shown n table 5.. All angles are gven n degrees. Note that the range and the standard devaton ncrease wth mesh number, ndcatng that the mesh s becomng less unform. Also note that the maxmum and mnmum angles n meshes 0 and are beyond the recommended values of 35 and 45. Mesh number Average angle Standard devaton Maxmum angle Mnmum angle Range (max mn) Table 5.: Statstcs descrbng the range of nternal angles of the elements wthn each mesh. There are several useful ways of lookng at the test results. A key pont to note s that because the boundary condtons were specfed by temperature values at the nodes on the boundary, the accuracy of the test results on the boundary s a measure of the accuracy of the nterpolaton algorthms and wll not be affected by any of the other steps n the calculaton. Hence lookng at the error n the results at the nodes on the boundary can gve an ndcaton of the accuracy of the nterpolaton algorthms. As an llustraton, compare the frst, sxth, and eleventh meshes. RMS absolute dfferences between the reference and test results averaged over the full, nternal, and boundary node sets are shown for each test n table 5.3. Page 47 of 53

54 NPL Report DEM-ES 003 Testng Algorthms and Software Test Test Test 3 Test 4 Test 5 Test 6 Mesh, all nodes Mesh, nternal nodes Mesh, boundary nodes Mesh 6, all nodes Mesh 6, nternal nodes Mesh 6, boundary nodes Mesh, all nodes Mesh, nternal nodes Mesh, boundary nodes Table 5.3: Absolute dfference between reference and test results, averaged over dfferent subsets of the nodes, for three dfferent meshes. There are several nterestng features of table 5.3. The frst s that the nternal nodes of mesh have a lower RMS dfference than the boundary nodes for all of the tests. Ths s surprsng, and t mples that for mesh, the nterpolaton and extrapolaton between values at nodes and values at ntegraton ponts wthn each element has as much of an effect on the accuracy of the soluton as the ntermedate calculaton steps. For the more dstorted meshes, the nternal nodes are sgnfcantly less accurate than the boundary nodes, the dfference beng about two orders of magntude by the fnal mesh. Ths llustrates the benefts of usng a regular mesh. Another pont to note s that the RMS dfference averaged across the boundary nodes s more or less constant for all tests and all meshes (ths s true of the meshes that have not been shown as well). Ths means that the error caused by the nterpolaton and extrapolaton s more or less constant. Ths s lkely to be the case for all boundary condtons of ths type. Fgure 5.7 shows an example of the dfference between the reference and test results plotted as a functon of poston. The results n fgure 5.7(a) were generated usng the eleventh mesh, but the results have been plotted at the nodes used n mesh. Mesh s shown n fgure 5.7(b). Each coloured regon n fgure 5.7(a) represents a node n fgure 5.7(b), so for example the maxmum dfference shown n fgure 5.7(a) occurs at the crcled node n fgure 5.7(b). Other tests have smlar plots, but the maxma dd not always occur at the same locatons. Page 48 of 53

55 Testng Algorthms and Software NPL Report DEM-ES 003 (a) (b) Fgure 5.7: (a) Varaton of the absolute dfference between reference and test results for test 4, mesh, and (b) mesh wth the node at whch the maxmum dfference occurs crcled. The rest of the results gven n ths chapter, unless stated otherwse, are averaged over the nternal nodes only. Fgures 5.8 to 5. show the results from all tests on all meshes as bar plots. Fgures 5.8 and 5.9 show the averaged RMS dfference and performance measure (N as defned above) respectvely. Note that both plots have truncated axes, and fgure 5.8 has a logarthmc scale. In general, both plots show a decrease n accuracy wth ncreasng dstorton, as expected, although the results for meshes 9 and 0 are very smlar to one another. It s not clear why ths s the case, snce table 5. shows that mesh 9 s less dstorted than mesh 0. It s nterestng to note that for every mesh except mesh, the frst test produced the smallest average dfferences, the last test produced the largest average dfferences, and tests 3 and 5 produced smlar average dfferences. If d s the RMS dfference for the th test, most meshes have d <d 3 <d 5 <d <d 4 <d 6. Ths behavour s lkely to be related to some functon of the degree of nonlnearty of the test, e.g. a +a (the maxmum dfference n gradent across the doman). Ths quantty may provde a sutable condton number for further problems. In terms of performance measure, the results can be ordered N >N 3 >N >N 5 >N 4 >N 6 for most meshes. Ths orderng s related to the rato of the maxmum temperature to the absolute error, whch works out at approxmately (a +a 3 +a 4 +a 5 )/(a +a ) for the domans used n these tests, and agan ths value could be used as a condton number f other smlar tests are run. Fgures 5.0 and 5. show the maxmum value of the dfference between the reference and test results and the mnmum value of the performance measure respectvely. These results are mportant f the user wants to ensure that all parts of the soluton are accurate. The plots n fgures 5.0 and 5. look broadly smlar to fgures 5.8 and 5.9 n terms of the decreasng accuracy wth ncreasng mesh dstorton, and n terms of the relatve accuraces of the sx tests for each mesh, although the orderng for the mnmum performance measure s N >N 3 >N >N 6 >N 4 >N 5 for most meshes. Page 49 of 53