modeling and simulation, uncertainty quantification and the integration of diverse information sources

Transcription

1 modeling and simulation, uncertainty quantification and the integration of diverse information sources C. Shane Reese 1 DoD/NASA Statistical Engineering Leadership Webinar, May Department of Statistics Brigham Young University 0

2 background

3 current collaborations 2

4 Given inevitable flaws and uncertainties, how should computational results be viewed by those who wish to act on them? The appropriate level of confidence in the results must stem from an understanding of a model s limitations and the uncertainties inherent in its predictions. National Academy Report,

5 motivation Test and evaluation of modern complex systems 3

6 motivation Test and evaluation of modern complex systems Diverse information sources (computer, physical, expert opinion) 3

7 motivation Test and evaluation of modern complex systems Diverse information sources (computer, physical, expert opinion) Diverse information types (functional, continuous, discrete) 3

8 motivation Test and evaluation of modern complex systems Diverse information sources (computer, physical, expert opinion) Diverse information types (functional, continuous, discrete) Complexity of systems increases with new variants, life extension programs, etc. 3

9 motivation Test and evaluation of modern complex systems Diverse information sources (computer, physical, expert opinion) Diverse information types (functional, continuous, discrete) Complexity of systems increases with new variants, life extension programs, etc. Example: Stockpile stewardship 3

10 motivation Test and evaluation of modern complex systems Diverse information sources (computer, physical, expert opinion) Diverse information types (functional, continuous, discrete) Complexity of systems increases with new variants, life extension programs, etc. Example: Stockpile stewardship Modern testing: do more with less 3

11 motivation Test and evaluation of modern complex systems Diverse information sources (computer, physical, expert opinion) Diverse information types (functional, continuous, discrete) Complexity of systems increases with new variants, life extension programs, etc. Example: Stockpile stewardship Modern testing: do more with less A pressing issue: most statistical, mathematical, and engineering programs do not provide sufficient training to tackle these difficult issues. 3

12 background Commonly encountered data sources (not meant to be exhaustive!) 4

13 background Commonly encountered data sources (not meant to be exhaustive!) Developmental testing (lab testing) 4

14 background Commonly encountered data sources (not meant to be exhaustive!) Developmental testing (lab testing) Operational testing (field testing) 4

15 background Commonly encountered data sources (not meant to be exhaustive!) Developmental testing (lab testing) Operational testing (field testing) Modeling and simulation (computer experiments) 4

16 background Commonly encountered data sources (not meant to be exhaustive!) Developmental testing (lab testing) Operational testing (field testing) Modeling and simulation (computer experiments) Engineering judgement 4

17 background Commonly encountered data sources (not meant to be exhaustive!) Developmental testing (lab testing) Operational testing (field testing) Modeling and simulation (computer experiments) Engineering judgement Commonly encountered data types 4

18 background Commonly encountered data sources (not meant to be exhaustive!) Developmental testing (lab testing) Operational testing (field testing) Modeling and simulation (computer experiments) Engineering judgement Commonly encountered data types Continuous 4

19 background Commonly encountered data sources (not meant to be exhaustive!) Developmental testing (lab testing) Operational testing (field testing) Modeling and simulation (computer experiments) Engineering judgement Commonly encountered data types Continuous Discrete 4

20 background Commonly encountered data sources (not meant to be exhaustive!) Developmental testing (lab testing) Operational testing (field testing) Modeling and simulation (computer experiments) Engineering judgement Commonly encountered data types Continuous Discrete Functional 4

21 definition of terms

22 a rose by any other name... (shakespeare) Modeling and Simulation 6

23 a rose by any other name... (shakespeare) Modeling and Simulation Computer Experiments 6

24 a rose by any other name... (shakespeare) Modeling and Simulation Computer Experiments Examples: CFD, FEA, etc. 6

25 a rose by any other name... (shakespeare) Modeling and Simulation Computer Experiments Examples: CFD, FEA, etc. Uncertainty Quantification 6

26 a rose by any other name... (shakespeare) Modeling and Simulation Computer Experiments Examples: CFD, FEA, etc. Uncertainty Quantification Model Validation 6

27 a rose by any other name... (shakespeare) Modeling and Simulation Computer Experiments Examples: CFD, FEA, etc. Uncertainty Quantification Model Validation Model Calibration 6

28 a rose by any other name... (shakespeare) Modeling and Simulation Computer Experiments Examples: CFD, FEA, etc. Uncertainty Quantification Model Validation Model Calibration Design of Experiments 6

29 a rose by any other name... (shakespeare) Modeling and Simulation Computer Experiments Examples: CFD, FEA, etc. Uncertainty Quantification Model Validation Model Calibration Design of Experiments Sensitivity Analysis 6

30 a rose by any other name... (shakespeare) Modeling and Simulation Computer Experiments Examples: CFD, FEA, etc. Uncertainty Quantification Model Validation Model Calibration Design of Experiments Sensitivity Analysis Propagation of Uncertainty 6

31 a rose by any other name... (shakespeare) Modeling and Simulation Computer Experiments Examples: CFD, FEA, etc. Uncertainty Quantification Model Validation Model Calibration Design of Experiments Sensitivity Analysis Propagation of Uncertainty Integrated Analysis of Physical and Computer Experiments 6

32 a rose by any other name... (shakespeare) Modeling and Simulation Computer Experiments Examples: CFD, FEA, etc. Uncertainty Quantification Model Validation Model Calibration Design of Experiments Sensitivity Analysis Propagation of Uncertainty Integrated Analysis of Physical and Computer Experiments Data Assimilation 6

33 a rose by any other name... (shakespeare) Modeling and Simulation Computer Experiments Examples: CFD, FEA, etc. Uncertainty Quantification Model Validation Model Calibration Design of Experiments Sensitivity Analysis Propagation of Uncertainty Integrated Analysis of Physical and Computer Experiments Data Assimilation Resource Allocation Decisions 6

34 modeling and simulation (m&s)

35 modeling and simulation Computer experiments examples 8

36 modeling and simulation Computer experiments examples Computational Fluid Dynamics (CFD) 8

37 modeling and simulation Computer experiments examples Computational Fluid Dynamics (CFD) NW Physics codes 8

38 modeling and simulation Computer experiments examples Computational Fluid Dynamics (CFD) NW Physics codes Finite Element Analysis (FEA, FEM) 8

39 modeling and simulation Computer experiments examples Computational Fluid Dynamics (CFD) NW Physics codes Finite Element Analysis (FEA, FEM) Biomechanics 8

40 modeling and simulation Computer experiments examples Computational Fluid Dynamics (CFD) NW Physics codes Finite Element Analysis (FEA, FEM) Biomechanics Often functional output rather than (y c (x) vs. y c (t, x(t))) 8

41 modeling and simulation Computer experiments examples Computational Fluid Dynamics (CFD) NW Physics codes Finite Element Analysis (FEA, FEM) Biomechanics Often functional output rather than (y c (x) vs. y c (t, x(t))) Ultimate codification and integration of expert opinion, physical theory, and computer model tuning parameters θ. 8

42 modeling and simulation Computer experiments examples Computational Fluid Dynamics (CFD) NW Physics codes Finite Element Analysis (FEA, FEM) Biomechanics Often functional output rather than (y c (x) vs. y c (t, x(t))) Ultimate codification and integration of expert opinion, physical theory, and computer model tuning parameters θ. May be less expensive than field data. 8

43 modeling and simulation Computer experiments examples Computational Fluid Dynamics (CFD) NW Physics codes Finite Element Analysis (FEA, FEM) Biomechanics Often functional output rather than (y c (x) vs. y c (t, x(t))) Ultimate codification and integration of expert opinion, physical theory, and computer model tuning parameters θ. May be less expensive than field data. Tradeoff: inherent bias or discrepancy. 8

44 why computer experiments? Benefits Resource savings 1. Financial Potential for exploration of expanded settings (covariates) Disadvantages 9

45 why computer experiments? Benefits Resource savings 1. Financial 2. Time Potential for exploration of expanded settings (covariates) Disadvantages 9

46 why computer experiments? Benefits Resource savings 1. Financial 2. Time 3. Testing resources Potential for exploration of expanded settings (covariates) Disadvantages 9

47 why computer experiments? Benefits Resource savings 1. Financial 2. Time 3. Testing resources Potential for exploration of expanded settings (covariates) Disadvantages Require different resources 9

48 why computer experiments? Benefits Resource savings 1. Financial 2. Time 3. Testing resources Potential for exploration of expanded settings (covariates) Disadvantages Require different resources May be biased (we use the term discrepancy) 9

49 why computer experiments? Benefits Resource savings 1. Financial 2. Time 3. Testing resources Potential for exploration of expanded settings (covariates) Disadvantages Require different resources May be biased (we use the term discrepancy) Difficult/impossible to validate without physical data 9

50 simple example of computer experimental problem Treat computer experiments as data! x predictor variables (observed) θ computer model parameters η(x, θ) computer model estimate for y given x and θ. y actual outcome at x ϵ statistical error Assume: y = η(x, θ) + ϵ θ unknown. 10

51 bayes rule π(θ y) L(y θ) π(θ) Goal is to understand θ to tune computer model Bayesian approach provides very general approach for inference Required element: prior pdf for θ is required (perhaps noninformative) Issue 1: normalizing π(θ y) is generally difficult, but rarely necessary Issue 2: high dimensional θ can lead to computational challenges 11

52 uncertainty quantification

53 outline Background Definition of Terms Modeling and Simulation (M&S) Uncertainty Quantification Integrated Analysis of Computer and Physical Experiments Model Validation Model Calibration Design of Experiments/Resource Allocation Sensitivity Analysis Examples Summary Questions and Answers 13

54 terminology and nomenclature Field experiments (physical experiments) Traditionally real data Measured without bias/discrepancy Univariate, y(x), multivariate, y(x), or functional, y(t; x) Hereafter, y(x). 14

55 statistical framework i y(x) = ζ(x) + ϵ(x) x: known system inputs y(x): experimental data ζ(x): unobs. system response ϵ(x): statistical error x 15

56 statistical framework ii θ ζ(x) = η(x, θ) + δ(x) θ: unknown calibration inputs η(x, θ): computer model δ(x): model discrepancy x 16

57 statistical framework (output) 0 δ(x) δ^(x) predicted δ(x) Discrepancy = 0 Agreement! feedback to modelers difference surface (data - model) 95/5 uncertainty bounds x 17

58 statistical framework (output) predicted ζ(x) 95/5 uncertainty bounds predicted ζ(x) unobserved truth Discrepancy adjusted! x 18

59 outline Background Definition of Terms Modeling and Simulation (M&S) Uncertainty Quantification Integrated Analysis of Computer and Physical Experiments Model Validation Model Calibration Design of Experiments/Resource Allocation Sensitivity Analysis Examples Summary Questions and Answers 19

60 model validation Does the computer code represent the reality that the code is meant to describe? 20

61 model validation Does the computer code represent the reality that the code is meant to describe? Most statistical techniques for model validation assume that field (physical) experiments are the gold standard (no inherent bias/discrepancy). 20

62 model validation Does the computer code represent the reality that the code is meant to describe? Most statistical techniques for model validation assume that field (physical) experiments are the gold standard (no inherent bias/discrepancy). Only makes sense where physical experiments are possible. 20

63 model validation Does the computer code represent the reality that the code is meant to describe? Most statistical techniques for model validation assume that field (physical) experiments are the gold standard (no inherent bias/discrepancy). Only makes sense where physical experiments are possible. Ambiguous definition: oft discussed, seldom resolved!. 20

64 outline Background Definition of Terms Modeling and Simulation (M&S) Uncertainty Quantification Integrated Analysis of Computer and Physical Experiments Model Validation Model Calibration Design of Experiments/Resource Allocation Sensitivity Analysis Examples Summary Questions and Answers 21

65 model calibration θ (tuning parameters) at the proper value to promote agreement between computer experiments and physical data. 22

66 model calibration θ (tuning parameters) at the proper value to promote agreement between computer experiments and physical data. Requires: 22

67 model calibration θ (tuning parameters) at the proper value to promote agreement between computer experiments and physical data. Requires: 1. True input/output relationship 22

68 model calibration θ (tuning parameters) at the proper value to promote agreement between computer experiments and physical data. Requires: 1. True input/output relationship 2. Computer experiment is a biased version of reality. 22

69 model calibration θ (tuning parameters) at the proper value to promote agreement between computer experiments and physical data. Requires: 1. True input/output relationship 2. Computer experiment is a biased version of reality. 3. Physical experiment is a noisy (statistical error) version of reality. 22

70 model calibration θ (tuning parameters) at the proper value to promote agreement between computer experiments and physical data. Requires: 1. True input/output relationship 2. Computer experiment is a biased version of reality. 3. Physical experiment is a noisy (statistical error) version of reality. 4. Built in to most COTS/publically available software. 22

71 outline Background Definition of Terms Modeling and Simulation (M&S) Uncertainty Quantification Integrated Analysis of Computer and Physical Experiments Model Validation Model Calibration Design of Experiments/Resource Allocation Sensitivity Analysis Examples Summary Questions and Answers 23

72 design of experiments Standard design tools don t apply 24

73 design of experiments Standard design tools don t apply Requires: 24

74 design of experiments Standard design tools don t apply Requires: 1. Usually space filling designs 24

75 design of experiments Standard design tools don t apply Requires: 1. Usually space filling designs 2. Initial small-scale starter design 24

76 design of experiments Standard design tools don t apply Requires: 1. Usually space filling designs 2. Initial small-scale starter design Regime switching with Treed Gaussian Processes (Gramacy & Lee, 2009) 24

77 design of experiments Standard design tools don t apply Requires: 1. Usually space filling designs 2. Initial small-scale starter design Regime switching with Treed Gaussian Processes (Gramacy & Lee, 2009) Modern approaches based on adaptive design procedures that not only make choices of inputs (x), but choose between computer experimental runs and physical experimental runs. 24

78 design of experiments Standard design tools don t apply Requires: 1. Usually space filling designs 2. Initial small-scale starter design Regime switching with Treed Gaussian Processes (Gramacy & Lee, 2009) Modern approaches based on adaptive design procedures that not only make choices of inputs (x), but choose between computer experimental runs and physical experimental runs. Rich literature on this topic. 24

79 outline Background Definition of Terms Modeling and Simulation (M&S) Uncertainty Quantification Integrated Analysis of Computer and Physical Experiments Model Validation Model Calibration Design of Experiments/Resource Allocation Sensitivity Analysis Examples Summary Questions and Answers 25

80 sensitivity analysis Assess the sensitivity of output to individual variables ( main effects ) or combinations of variables ( interactions ). ANOVA-type decomposition gives variability atribution. Useful in determination of important variables/combination of variables. Active area of research. Built in to most COTS/publically available software 26

81 outline Background Definition of Terms Modeling and Simulation (M&S) Uncertainty Quantification Integrated Analysis of Computer and Physical Experiments Model Validation Model Calibration Design of Experiments/Resource Allocation Sensitivity Analysis Examples Summary Questions and Answers 27

82 example 1: microencapsulation food coating Goal: Model uniformity of foot coating application, find best computer model. Physical Experiment: actual food coating process as use 28 runs of food coating line with different (operational testing) settings of temperature, air pressure, density of coating material, etc. Computational Models: 3 different models with different physics at each of the exact same settings. Unique aspects: Possible to run computer experiments at all physical experiments Multiple teams competing to build more accurate experiment All models perfectly tuned (almost never happens!) 28

83 example 1 (output) Model 1 Model 2 Model 3 constant δ(x) δ Discrepancy = 0 Agreement! Computer model 3 is best δ (data - model) Full distribution for each δ location bias 29

84 example 2: stockpile stewardship Goal: Assess safety, security and effectiveness of the stockpile Physical Experiments: underground tests (operational-ish), non nuclear tests Operational physical experiments are desired by nobody. Lab tests expensive, only partially representative Computer Experiments: complex computer codes Months of CPU time on world s fastest supercomputers. Specialized computing equipment required. Computational experiments almost as costly as physical experiments. 30

85 example 3: nasa slosh estimation Goal: Model effect of damping on fluid slosh in booster tanks Physical Experiment: Shaker table (lab testing) Operational physical experiments costly. Computational experiments are a as costly Computer Experiments: Computational Fluid Dynamics (CFD) Expensive, difficult to obtain Computational experiments are perhaps more costly to run (at least in terms of time) Viewed by some as higher fidelity than physical experiments 31

86 example 3: (output) Fill Height: 12.6 in All Wave Heights Force o o o o o predicted ζ(x) 95/5 uncertainty bounds Impressive agreement unobserved truth Discrepancy adjusted! 32

87 summary

88 summary and reflections Statistical framework for integration of computational (M&S) and physical experiments is feasible. 34

89 summary and reflections Statistical framework for integration of computational (M&S) and physical experiments is feasible. Treating computer experiments as biased data allows for 34

90 summary and reflections Statistical framework for integration of computational (M&S) and physical experiments is feasible. Treating computer experiments as biased data allows for Tuning of computer model feedback to modelers 34

91 summary and reflections Statistical framework for integration of computational (M&S) and physical experiments is feasible. Treating computer experiments as biased data allows for Tuning of computer model feedback to modelers Estimation of regions of unbiasedness and regions of bias 34

92 summary and reflections Statistical framework for integration of computational (M&S) and physical experiments is feasible. Treating computer experiments as biased data allows for Tuning of computer model feedback to modelers Estimation of regions of unbiasedness and regions of bias Integration of different sources of information 34

93 summary and reflections Statistical framework for integration of computational (M&S) and physical experiments is feasible. Treating computer experiments as biased data allows for Tuning of computer model feedback to modelers Estimation of regions of unbiasedness and regions of bias Integration of different sources of information Fully quantified uncertainty. 34

94 summary and reflections Statistical framework for integration of computational (M&S) and physical experiments is feasible. Treating computer experiments as biased data allows for Tuning of computer model feedback to modelers Estimation of regions of unbiasedness and regions of bias Integration of different sources of information Fully quantified uncertainty. Each situation has unique elements that make integrated analysis difficult to create and use COTS solutions. 34

95 summary and reflections Statistical framework for integration of computational (M&S) and physical experiments is feasible. Treating computer experiments as biased data allows for Tuning of computer model feedback to modelers Estimation of regions of unbiasedness and regions of bias Integration of different sources of information Fully quantified uncertainty. Each situation has unique elements that make integrated analysis difficult to create and use COTS solutions. Resources are available and research continues to pour out of both statistics and applied math (see below). 34

96 references Santner, T. J., Williams, B. J., & Notz, W. I. (2013). The design and analysis of computer experiments. Springer Science & Business Media. Smith, R. C. (2013). Uncertainty quantification: theory, implementation, and applications (Vol. 12). SIAM. Reese, C. S., Wilson, A. G., Hamada, M., Martz, H. F., & Ryan, K. J. (2004). Integrated analysis of computer and physical experiments. Technometrics, 46(2), Higdon, D., Kennedy, M., Cavendish, J. C., Cafeo, J. A., & Ryne, R. D. (2004). Combining field data and computer simulations for calibration and prediction. SIAM Journal on Scientific Computing, 26(2), Higdon, D., Gattiker, J., Williams, B., & Rightley, M. (2008). Computer model calibration using high-dimensional output. Journal of the American Statistical Association, 103(482), Gramacy, R. B., & Lee, H. K. (2009). Adaptive design and analysis of supercomputer experiments. Technometrics, 51(2),

97 resources for computation LANL Gaussian Spatial Process (GaSP) Code: 36

98 resources for computation LANL Gaussian Spatial Process (GaSP) Code: MATLAB based 36

99 resources for computation LANL Gaussian Spatial Process (GaSP) Code: MATLAB based Examples provides 36

100 resources for computation LANL Gaussian Spatial Process (GaSP) Code: MATLAB based Examples provides Not completely user-friendly 36

101 resources for computation LANL Gaussian Spatial Process (GaSP) Code: MATLAB based Examples provides Not completely user-friendly Free! 36

102 resources for computation LANL Gaussian Spatial Process (GaSP) Code: MATLAB based Examples provides Not completely user-friendly Free! 36

103 resources for computation LANL Gaussian Spatial Process (GaSP) Code: MATLAB based Examples provides Not completely user-friendly Free! SmartUQ 36

104 resources for computation LANL Gaussian Spatial Process (GaSP) Code: MATLAB based Examples provides Not completely user-friendly Free! SmartUQ COTS solution 36

105 resources for computation LANL Gaussian Spatial Process (GaSP) Code: MATLAB based Examples provides Not completely user-friendly Free! SmartUQ COTS solution Fee based 36

106 resources for computation LANL Gaussian Spatial Process (GaSP) Code: MATLAB based Examples provides Not completely user-friendly Free! SmartUQ COTS solution Fee based User-friendly 36

107 resources for new methods and research Conferences: ASA, SIAM, NIPS 37

108 resources for new methods and research Conferences: ASA, SIAM, NIPS Journals: 37

109 questions and answers