Smulaton: Solvng Dynamc Models ABE 5646 Week Chapter 2, Sprng 200 Week Descrpton Readng Materal Mar 5- Mar 9 Evaluatng [Crop] Models Comparng a model wth data - Graphcal, errors - Measures of agreement (bas n mean, varance) - Evaluaton of predctve qualty Cross valdaton Bootstrap estmaton Effects of errors n observatons on MSEP Wallach et al. (2006) Chapter, 2 Also see chapters 2-3 for examples
Weeks -Ch 2 Objectves ) Learn basc methods for evaluatng dynamc bophyscal models
Shftng Focus From: System modelng and smulaton methods To: Workng wth dynamc models (wth crop examples, but methods apply to other bologcal and agrcultural system models) Ths shft means that we take a model that has already been developed and work wth t In our class, we wll focus on model ) evaluaton (comparson wth data from the real system), 2) analyzng uncertanty and senstvty of a model to varous factors, and 3) estmatng parameters and ther uncertanty. We wll not attempt to cover all methods n the book n the remanng tme, but nstead cover 2 or 3 of the most mportant methods and concepts for each topc that we do cover
Evaluatng Dynamc Models Evaluaton vs. valdaton Importance Some steps Objectve of model, crtera for evaluaton Model equatons, mplementaton accuracy Senstvty analyss, parameter estmaton Comparson wth data Methods
Evaluatng Model Agreement wth Measurements Test whether model s a good representaton of the real system In many cases, measurements are used to estmate parameters, smlar to fttng a model to partcular expermental results The man ssue here s whether measurements used n an evaluaton are ndependent When the predctve qualty of a model s to be evaluated, comparsons must be wth data not used to develop or parameterze the model. Ths wll be consdered after examnng measures for comparsons
Model Error Analyss Smple Example Regresson Model How does ths relate to dynamc models?
Comparsons usng Graphcs Predcted vs. Observed Error (resdue) = Predcted Observed
Comparsons usng Graphs vs. Tme Dry Weght (kg/ha) 8000 6000 4000 2000 Smulated and Measured Soybean Ganesvlle, FL 978 Yelds 0 75 200 225 250 275 300 Day of Year Total Crop - IRRIGATED Total Crop - OT IRRIGATED Gran - IRRIGATED Gran - OT IRRIGATED
Graphng Fnal Yeld Predcted vs. Observed Predcted yeld (t/ha) 6 4 2 0 8 6 4 2 Maze Soybean Predcted yeld (t/ha) 6 5 4 3 2 Wheat 0 0 2 4 6 8 0 2 4 6 Observed yeld (t/ha) 0 0 2 3 4 5 6 Observed yeld (t/ha) On farm crop model tests n Pampas Regon, Argentna. Magrn et al.
Smple Measures of Agreement Dfference between observaton and predcton D = Y Yˆ and Bas = = D Mean Square Error = MSE = = ( D ) 2 Root Mean Square Error = RMSE = MSE
Smple Measures of Agreement Mean Absolute Error = = D Relatve Root Mean Square Error = RMSE Y wth Y = = Y
Modelng Effcency EF = = = ( Y ( Y Yˆ Y ) ) 2 2
Correlaton Coeffcent r = ˆ σ ˆ σ Y YYˆ ˆ σ Yˆ [ there was a mstakenbook]
Wllmott D Index
Decomposng Error due to Dfferent Sources Decomposng MSE (Kobayash and Salam(2000): MSE = ( Bas) wth SDSD = ( σ LCS = 2σ Y Y + SDSD + LCS σ ) σ ( r) or Yˆ 2 Yˆ 2 = MSE ( Bas) 2 SDSD and SDSD s due to dfferences between standard devaton of measurements vs. modeled LCS s related to how well the model mmcs observed varaton across stuatons There are other ways to decompose the error (.e., Gauch et al. (2003), Wllmott (98)
Summary of Measures
Evaluatng the Predctve Qualty of a Model When parameters are fxed, a model can be used to predct ndependent observatons MSEP s the Mean Square Error of Predcton MSEP s not equal to MSE MSE may nclude data that were used to develop a model or to estmate ts parameters Data used to compute MSE may not represent the full range of nterest for the model purposes MSE may be a poor estmator of MSEP
MSEP MSEP( ˆ) θ = E{[ Y f ( X ; ˆ)] θ ˆ} θ 2 RMSEP( ˆ) θ = MSEP( ˆ) θ
Components of MSEP
Estmatng MSEP( ) θˆ Data are ndependent from model & parameters MSEP ˆ ( ˆ) θ = MSE = = [ Y f ( X ; ˆ)] θ 2 If dfferent data are used to estmate MSEP, one can also compute an estmate of the varance of MSEP (eq. 23, p.34)
Cross Valdaton Method for estmatng MSEP( ), θˆ uncertanty n predcton Useful when number of observatons s small Assume data are random sample from target populaton Could splt data nto two parts, one for parameter estmaton and one for testng (ndependent) But, lose mportant nformaton for estmatng parameter Cross Valdaton also uses data splttng Frst, usng all but one data pont to estmate parameters Then, calculate error for the one data pont that was left out Repeat ths tmes (# data ponts); end up wth errors Ths approach uses all data to estmate parameters & evaluate predctons
Cross Valdaton MSEP computed by Cross Valdaton s: MSEP ˆ CV ( ˆ) θ = = [ Y f ( X ; ˆ θ )] 2 Where the θˆ s the set of parameters leavng out data pont
Cross Valdaton Example Usng the smple model (5 nputs, 6 parameters): f ˆ) ˆ ˆ ˆ (0) () () (2) (2) (3) (3) (4) (4) (5) (5) 5( X ; θ = θ + θ x + θ x + θ x + θ x + θ x ˆ ˆ ˆ And the data from Table (=8 data ponts)
Cross Valdaton Example What about values of the parameters?
Bootstrap Estmaton Data re-samplng for estmatng MSEP( ), θˆ Useful when number of observatons s small Assume data are the full target populaton Could splt data nto two parts, one for parameter estmaton and one for testng (ndependent) But, lose mportant nformaton for estmatng parameter Bootstrap also uses data splttng Samplng wth replacement =number of data ponts, pck a data pont, replace t, then pck a second data pont, then a 3 rd, etc. to get data ponts Repeat ths b tmes
Bootstrap Estmaton Example Let = 8, usng data from CV example Let B = 3 bootstrap samples Then we have 3 samples of 8 observatons each Estmate parameters for frst sample (b=) Then use those parameters to compute MSEP b for the orgnal 8 samples AD to estmate MSE usng the 8 data ponts n sample b Compute a correcton term for sample b Repeat ths for b=2, then b=3 Average the correcton term and add t to MSE computed from the orgnal dataset
Bootstrap Estmaton of MSEP 2 2 )] ˆ ; ( [ )] ˆ ; ( [ ) ˆ ( ) ˆ ( : b b b b b b b b b b b b X f Y MSE X f Y MSEP MSE and MSEP computng frst by MSE MSEP op Correcton term θ θ θ θ = = = = =
Bootstrap Estmaton of MSEP MSEP ˆ bootstrap ( ˆ) θ = MSE + op ˆ where op ˆ = B B b= op b
Measurement Errors n Y and MSEP If Y has sgnfcant measurement error, then there are really two MSEP values one for the dfference between predcted and measured values the other for the dfference between predcted and the TRUE value Y obs = Y + η η s measurement error, mean 0
MSEP wth Errors n Measurements MSEP obs ( ˆ) θ = σ 2 η + MSEP( ˆ) θ and MSEP ˆ ( ˆ) θ = MSEP ˆ obs ( ˆ) θ ˆ σ f η s ndependent of Y and 2 η X ; η ~ (0, σ 2 η )
Dscusson Random samplng for stochastc models Comparng a model wth data - Graphcal, errors - Measures of agreement (bas n mean, varance) - Evaluaton of predctve qualty Cross valdaton Bootstrap Errors n measurements
Homework Chapter 2 Chapter 2 n Wallach et al. Problem Problem 4 Due on March 25