Key Words: model quality assurance, model evaluation, statistical model evaluation, uncertainty analysis, statistical indices, performance measures

Transcription

1 17: Evaluaton Of Ar Polluton Models Elsa Canepa* and John S. Irwn ** * INFM (Natonal Insttute for the Physcs of Matter) Department of Physcs - Unversty of Genova Va Dodecaneso 33, I Genova (Italy) canepae@fsca.unge.t ** Ar Polcy Support Branch Atmospherc Scences Modelng Dvson (Mal Code D243-01) Natonal Oceanc and Atmospherc Admnstraton 1 Research Trangle Park, North Carolna John.rwn@noaa.gov Abstract: Informaton s gven about model evaluaton, the overall system of procedures desgned to measure model performance, and n partcular the process of statstcal performance evaluatons. Statstcal performance evaluaton s an assessment of model performance based on the comparson of model outputs wth expermental data. Some performance measures, consstng of statstcal ndces and graphcal methodologes currently used are descrbed. Problems related to uncertanty analyss are hghlghted. Key Words: model qualty assurance, model evaluaton, statstcal model evaluaton, uncertanty analyss, statstcal ndces, performance measures 1 Introducton Model qualty assurance s a collecton of actvtes one should perform n order to promote the development and applcaton of good ar qualty smulaton models (dscussed n more detal n Secton 8). One of the elements of model qualty assurance s model evaluaton. Model evaluaton 2 s a collecton of actvtes one should perform n order to understand how a model behaves and how a model compares wth observatons (dscussed n more detal n Secton 6). One of the elements of model evaluaton s statstcal model evaluaton. Statstcal model evaluaton (also called statstcal performance evaluaton ) s an assessment of model performance based on the comparson of model outputs wth expermental data (dscussed n more detal n Secton 7). 1 John Irwn s a NOAA employee, on assgnment to the Offce of Ar Qualty Plannng and Standards, U.S. Envronmental Protecton Agency, Research Trangle Park, NC Readers wll see that we have avoded the use of the term valdaton. Fox (1981) and Olesen (1996) defne valdaton as a concluson resultng from detaled and copous evdence that leads to formal recognton, whch mght nclude several evaluatons.

2 It s our experence and our concluson from a comprehensve revew of past model evaluaton exercses, that t s not proftable provde a cookbook seres of steps that one must accomplsh n order to adequately mplement a statstcal model evaluaton. Models are used n a varety of ways, many of whch were never antcpated when the model was frst developed and made avalable. Models are often used n stuatons that, n prncple, they are ncapable of handlng as they are lackng characterzaton of relevant physcal processes. For example, although most operatonal ar qualty models provde estmates of ensemble average concentratons, they are typcally used to estmate maxmum (peak) concentraton values (whch are extremes values wthn an ensemble). Instead of a seres of steps, we provde a framework (Secton 4), wthn whch one can understand why modelng results and observatons dffer. We beleve that by followng the deas expressed n ths framework, one can develop a successful evaluaton of any model regardless of whether t s beng appled n a manner consstent wth ts desgned physcs and modelng assumptons. In Secton 5 we summarze those performance measures that are n common usage, and then n Secton 7 we dscuss concepts that can be employed n developng a statstcal model evaluaton. 2 Termnology A revew of recent evaluaton exercses reveals 3 that varous characterstcs of atmospherc dsperson models have been tested and often the methods have used applcaton-specfc schemes wth varous performance measures. In fact, n the lterature t s possble to fnd a wde dversty of defntons concernng topcs related to model evaluaton. To avod confuson and msunderstandngs, we beleve that t would be useful to acheve a harmonzaton about termnology and ts use. The etymology of the terms we use s mportant for understandng by both scentsts and decson-makers. Hence, we have tred to not depart too much from the etymologcal word meanngs (e.g., see Schlunzen, 1997). Atmospherc ar qualty model: an dealzaton of atmospherc physcs to calculate the magntude and locaton of pollutant concentratons. Ths may take the form of an equaton, algorthm, or seres of equatons/algorthms used to calculate average or tme-varyng concentraton. They may take the form of a determnstc model or a statstcal model. The model may nvolve process descrptons and numercal methods for soluton. 3 Of the large avalable number, we lst a few for example. Methods/Revew: U.S. Envronmental Protecton Agency (1992), Hanna et al. (1993), Pol and Crllo (1993), Ward (1994), Wel et al. (1997); Long Range Transport: Bellaso et al. (1998), Brandt et al. (1998), Carhart et al. (1989), Mosca et al. (1998); Complex Terran: Cox et al. (1998), Desato (1991), Gronske et al. (1993), Luhar and Rao (1994), Ross and Fox (1991), Thuller (1992); Plume Dsperson: Brusasca et al. (1989), Carruthers et al. (1999), Hanna and Pane (1989), Hanna and Chang (1993), Olesen (1995); Regonal Grd: Davs et al. (2000), Denns (1986), Hanna et al. (1996), Hass et al. (1997), Kumar et al. (1994); Low Wnds/Street Canyons: Kumar Yadav and Sharan (1996), Lanzan and Tampon (1995), Okamoto et al. (1999). 2

3 Calbraton (or model calbraton): a procedure used to make, at the model development stage, estmates of the parameters of model equatons, whch best ft the general model structure to a specfc observed data set. Data assmlaton: a numercal technque, whch makes t possble to combne model results and observatons n one ntegrated system, wth the purpose of mnmzng the dscrepancy between model predctons and observatons Data qualty assessment: the scentfc and statstcal evaluaton of data to determne f data obtaned from envronmental data operatons are of the rght type, qualty and quantty to support ther ntended use. Data qualty objectve: a range of acceptablty for data used n modelng analyses for a specfc applcaton. Determnstc model: a model s determnstc when t s assumed that all possble behavors are determned by the set of equatons comprsng the model. These models are based on fundamental mathematcal descrptons of atmospherc processes, n whch effects (.e., ar polluton) are generated by causes (.e., emssons). Dffuson, absolute: the characterzaton of the spreadng of materal released nto the atmosphere based on a coordnate system fxed n space. Dffuson, relatve: the characterzaton of the spreadng of materal released nto the atmosphere based on a coordnate system that s relatve to some local poston of the dspersng materal (e.g., center of mass). Dsperson: the combned effects of eddy dffuson and advecton (transport). Evaluaton (or model evaluaton): the overall system of procedures desgned to measure the model performance. Evaluaton objectve: a feature or characterstc whch can be defned through an analyss of the observed concentraton pattern (e.g., maxmum centerlne concentraton or lateral extent of the average concentraton pattern as a functon of downwnd dstance) for whch one desres to assess the skll of the models to reproduce. Evaluaton procedure: the analyss steps to be taken to compute the value of the evaluaton objectve from the observed and modeled patterns of concentraton values. Fate: the destny of a chemcal or bologcal pollutant after release nto the atmosphere. 3

4 Model ntercomparson: a process where several models, all presumably approprate for some chosen stuatons (dealzed or real), smultaneously have ther performances assessed can compared. Performance measures (or statstcal comparson metrcs): evaluaton tools (quanttatve and/or qualtatve) lke statstcal ndces and graphcal methodologes, to compare model outputs wth observed values. Process model: an dealzaton of atmospherc physcs envsoned as beng composed of a seres of nter-related processes to calculate the magntude and locaton of pollutant concentratons based on fate, transport, and dffuson n the atmosphere. These models most often are determnstc models, but n prncple, could attempt to characterze the stochastc process effects. Qualty assurance: all those planned and systematc actons necessary to provde adequate confdence that a product or servce wll satsfy gven requrements for qualty. Senstvty analyss: a process for dentfyng the magntude, drecton, and form (e.g., lnear or non-lnear) of the effect of the varaton of one or more model parameters or model nputs on the model result. Statstcal model: a model of a stochastc process that represents the dependence of successve or neghborng events n response to varaton n an external nfluence on the process. These models are parsmonous usng the fewest number of parameters capable of explanng quanttatve varaton n some observed data. They are based upon sem-emprcal statstcal relatons among avalable data and measurements. Statstcal model evaluaton: s the analyss of model performance based on the comparson of model outputs wth expermental data (evaluaton objectves). Statstcal model evaluatons nvolve summarzng model performance n an overall sense (typcally called performance evaluaton), and testng the smulaton of specfc processes wthn a model (typcally called dagnostc evaluatons). Stochastc process: s a contnuous causal process n tme, space, or both, respondng to varaton n an external nfluence, and producng a varyng seres of measured states or events. Uncertanty: a dfference (or dfferences) between what s modeled and what s observed. It s a consequence of a lack of knowledge n model formulaton, and errors (or omssons) n data and observatons. In prncple, uncertanty can be reduced wth ether mproved theory or observatons, however t s generally accepted that there s lmt to how much of the natural varablty can be explctly 4

5 smulated by models. That porton of natural varablty that s beyond the reach of modelng, we refer to as nherent varablty 4. Uncertanty analyss: a process for estmatng model uncertanty. Varablty: s what happens n the natural system; the observable varatons. Verfcaton: s the checkng of the computer code to ensure that t s a true representaton of the conceptual model upon whch t s based. Ths ncludes checkng whether the mathematcal equatons nvolved have been solved correctly, comparng the numercal solutons wth dealzed cases for whch an analytc soluton exsts. 3 Background Ar qualty smulaton models have been used for many decades to characterze the transport and dffuson of materal n the atmosphere (Pasqull, 1961, Randerson, 1984, Hanna et al., 1982). The wder use of atmospherc models n scentfc studes for regulatory purposes and for descrbng ar qualty scenaros requres assessng the degree of relablty of model results. Generally, such an assessment s performed through the comparson of model outputs aganst feld measurements. Tracer experments are partcularly helpful n evaluatng the capablty of these models to properly smulate transport and dffuson. Comparson between model outputs and measurements are performed usng both qualtatve data analyss technques and quanttatve statstcal methods. Up untl the early 1980 s, the comparson of modelng results wth observatons was consdered smple. The outputs of dsperson models were plotted aganst measurements (usng tradtonal scatter plots of the values) and smple performance measures such as the correlaton coeffcent were computed (Clarke, 1964, Martn, 1971, Hanna, 1971). Hgh correlaton values were nterpreted to ndcate that the model was performng well, low correlaton (a not uncommon case) was nterpreted to mean that the model was performng poorly. As ar qualty models came nto more common use, concerns were rased that early statstcal performance evaluaton had been nave. Lttle consderaton had been gven to the consequences and sources of uncertanty and varablty. As dscussed n Sectons 4 and 6, the sources of uncertanty can be envsoned as beng composed of: model formulaton uncertanty, representatveness uncertanty, measurement uncertanty, and nherent varablty 4. Model formulaton uncertanty s composed of theory uncertanty (there may be more than one theory that adequately descrbes avalable data) and 4 What we are labellng nherent varablty s what Fox (1984) and others dscuss as nherent uncertanty, and what Hanna (1993) dscuss as the rreducble scatter caused by stochastc fluctuatons. 5

6 numercal uncertanty (converson of mathematcal algorthms to numercal code may nvolve approxmatons, that f not well-treated, could lead to spurous nose n the solutons); Representatveness uncertanty arses whenever there s a lack of agreement n the data used as model nput, or the data used for comparson wth model output 5, to satsfy the spatal and temporal assumptons of the model; Measurement uncertanty results from errors n measurements, whch can affect model nputs 6 and can affect observed concentratons used for comparson wth model outputs; Inherent varablty arses because models only characterze a porton of the naturally occurrng varatons. In the early comparsons, measurement uncertantes were assumed to be small n comparson to real world fluctuatons, when n fact that was not always a safe assumpton. More mportantly, even hypothetcally error-free measurements possess space and tme lmtatons that prevent them from beng good approxmatons of the tme and space assumptons assumed n the constructon of the model. For nstance, the comparson of measurements taken at an solated receptor wth grd-averaged model outputs s napproprate (Davs et al., 2000). The early statstcal performance evaluatons faled to address the fact that models provde estmates of ensemble means, whereas the observatons are ndvdual realzatons from mperfectly defned ensembles (Lamb from Longhetto, 1980; Venkatram, 1988). Furthermore, relance on lnear regressons and correlaton coeffcent can provde msleadng results (e.g., Zannett and Swtzer, 1979). Lastly, models rely upon emsson and meteorologcal nputs whose uncertantes could justfy dsagreements between predctons and observatons (e. g., Irwn et al., 1987). In the early eghtes, several attempts were made to develop standard methodologes for judgng ar qualty model performance (e.g., Bornsten and Anderson 1979, Venkatram, 1982 and 1983, Wllmot, 1982). The Amercan Meteorologcal Socety sponsored two workshops n an attempt to provde specfc gudelnes on the use of statstcal tools n ar qualty applcatons. A summary of ther recommendatons s provded n two papers by Fox (1981 and 1984). 5 Dfferences from not properly satsfyng the model nput assumptons are referred to by some as data representatveness uncertanty, and by other as nput uncertanty ; dfferences n properly satsfyng the model output assumptons are most often referred to as data representatveness uncertantes. 6 Uncertantes n emsson data may result from measurement uncertantes, or they may result from formulaton uncertantes, n that a wrong methodology mght have been used for emsson estmaton. 6

7 The most nterestng comments and recommendatons from the above workshops were: the concern about the absolute, rather than statstcal nature of U.S. ar qualty standards; the possblty of computng statstcs between measured data values and model predcted values, even when these values are not coupled n tme and/or n space; the dentfcaton of reducble errors and nherent varablty; the recommendatons to decson-makers to educate themselves and accept the challenge of decson makng wth quantfed uncertanty. Followng these two workshops a seres of studes were undertaken to contnue to nvestgate the problem of statstcally evaluatng the performance of ar qualty models. Interestng methods were proposed at the DOE Model Valdaton Workshop, October 23-26, 1984, Charleston, South Carolna, and by Alcamo and Bartnck (1987) and Hanna (1989a). Major operatonal evaluatons of ar qualty models were sponsored by EPRI (e.g., Reynolds et al., 1984; Ruff et al., 1984; Moore et al., 1985; and Reynolds et al., 1985). Further development of the evaluaton methodologes proposed n the early eghtes was needed, as t was found that the rote applcaton of performance measures, such as those lsted n Fox (1981), was ncapable of dscernng dfferences n model performance (Smth, 1984). Whereas f the evaluaton results were sorted by stablty and dstance downwnd, then dfferences n modelng skll could be dscerned (Irwn and Smth, 1984). It was becomng ncreasngly evdent that the models were characterzng only a small porton of the observed varatons n the concentraton values (Hanna, 1988). To better deduce the statstcal sgnfcance of dfferences seen n model performance n the face of small sample szes and unknown uncertantes, nvestgators began to explore the use of bootstrap technques (Hanna, 1989). By the late 1980 s, most of the model evaluatons nvolved the use of bootstrap technques n the comparson of maxmum values of modeled and observed cumulatve frequency dstrbutons of the concentraton values (Cox and Tkvart, 1990). Even though the procedures and measures are stll evolvng to descrbe performance of models that characterze atmospherc fate, transport and dffuson (Wel et al., 1992, Dekker et al., 1990, Cole and Wcks, 1995), there has been a general acceptance for a need to address the large uncertantes nherent n atmospherc processes. There has also been a consensus reached on the phlosophcal reasons that models of earth scence processes can never be verfed (n the sense of clamng that a model s truthfully representng natural processes). No general emprcal proposton about the natural world can be certan, snce there wll always reman the prospect that future observatons may call the theory n queston (Oreskes et al., 1994). It s seen that numercal models of ar polluton 7

8 are a form of a hghly complex scentfc hypothess concernng natural processes that can be confrmed through comparson wth observatons, but never verfed. 4 Framework To set the context for the dscusson to follow (e.g., Irwn, 2000), t s mportant to realze that most of the model evaluaton results currently avalable n the lterature are for appled ar qualty models that use ensemble average characterzatons of the transport and dffuson, the chemcal transformatons, and the physcal removal processes. Thus, these appled ar qualty models only provde a descrpton of the average fate of pollutants to be assocated wth each possble ensemble of condtons (or regme ). Natural varablty not characterzed by the model can result n large devatons n comparsons of ndvdual observatons (whch are ndvdual realzatons from an ensemble of realzatons) wth modelng results (whch are characterzng the ensemble average result). The dfferences seen n comparson of model predctons and observatons of atmospherc ar concentratons may largely reflect nherent varablty. Ths component of varablty s nherent n that t lkely can not be smulated explctly by mprovng the physcs of the ar qualty models. At best, ar qualty models provde an unbased estmate of the average concentraton expected over all realzatons of an ensemble. An estmate of an ensemble can be developed from a set of experments havng fxed external condtons (Lumley and Panofsky, 1964). To accomplsh ths, the avalable concentraton values are sorted nto classes characterzng ensembles. For each of the ensembles thus formed, the dfference between the ensemble average and any observed realzaton (expermental observaton) s then ascrbed to nherent varablty, whose varance, σ n 2, can be expressed as (Venkatram, 1988): o ( C C ) 2 σ = (1) 2 o n where C o s the observed concentraton seen wthn a realzaton; the over-bars refer to an average over all avalable realzatons wthn a gven ensemble, so that o C s the estmated ensemble average. In (1), the ensemble refers to the deal nfnte populaton of all possble realzatons meetng the (fxed) characterstcs of the chosen ensemble. In practce, we wll only have a small sample from ths ensemble. Measurement uncertanty n C o n most tracer experments s typcally a small fracton of the measurement threshold, and when ths s true, ts contrbuton to σ n can usually be deemed neglgble. Defnng the characterstcs of the ensemble n (1) usng the model s nput values, α, one can vew the observed concentratons as: C o o ( α, β ) = C ( α ) + c( c) + c( α, β ) o = C (2) 8

9 where β are the varables needed to descrbe the unresolved transport, fate and dffuson processes, the over-bar represents an average over all possble values of β for the specfed set of model nput parameters α; c( c) represents the effects of concentraton representatveness and measurement uncertanty, and c(α,β) represents gnorance n β, unresolved determnstc processes and stochastc o fluctuatons (Hanna, 1988, Venkatram, 1988). Snce ( α ) o β, t s only a functon of α, and n ths context, ( α ) C s an average over all C represents the ensemble average that the model deally s attemptng to characterze. The modeled concentratons, C s, can be envsoned as: C s o ( α ) = C ( α ) + d( α ) + f ( α ) s = C (3) where d( α) represents the effects of uncertanty n specfyng the model nputs, and f(α) represents the effects of uncertanty n the model theory and numercal mplementaton. The method we propose for performng an evaluaton of modelng skll, s to separately average the observatons and modelng results over a seres of nonoverlappng lmted-ranges of α, whch are called regmes. Averagng the observatons provdes an emprcal estmate of what most of the current models o are attemptng to smulate, C ( α ). A comparson of the respectve observed and modeled averages over a seres of α-groups provdes an emprcal estmate of the combned determnstc error assocated wth nput uncertanty and formulaton errors. Gven ths framework, desgnng a model evaluaton can be envsoned as a two step process. Step one, we analyze the observatons to provde average patterns for comparson wth modeled patterns. Step two, gven the uncertantes n estmatng the average patterns, we test to see whether dfferences seen n a comparson of performance of several models are statstcally sgnfcant. In order to place confdence bounds on conclusons reached n step two, bootstrap resamplng s recommended (see Secton 5.4). Wthn the Amercan Socety for Testng and Materals (ASTM), a standard gude 7 has been developed that outlnes ths strategy for desgnng statstcal evaluatons of dsperson model performance (e.g., a statstcal evaluaton of performance). Ths process s not wthout problems, as groupng data together for analyss requres large data sets, of whch there are few. Sortng the data nto groups requres suffcent knowledge of the expermental condtons to determne that data collected on dfferent days or durng dfferent tme perods should be 7 Standard Gude for the Statstcal Evaluaton of Atmospherc Dsperson Model Performance, D6589, Annual Book of Standards Volume 11.03, Amercan Socety for Testng and Materals, West Conshohocken, PA ( 9

10 grouped together. In realty, the external forcng condtons are mperfectly known, and hence the groups are mperfectly composed. Another problem s that ar qualty models only explan a small porton of the observed varatons, and there are large uncertantes nvolved n any ar qualty modelng assessment. Earler n ths chapter, we mentoned that there are essentally four sources for uncertanty: formulaton uncertanty, representatveness uncertanty, measurement uncertanty, and nherent varablty. We now take a moment to provde some perspectve as to the sze and nature of nherent varablty and model nput uncertanty. From Equaton (2), we see that natural varaton not explaned by the model s the term c(α,β), and we have referred to ths as the nherent varablty. It has been estmated that the porton of natural varablty that s not accounted for by atmospherc transport and dffuson models s of order of the magntude of the regme averages (Wel et al., 1992, Hanna, 1993). Thus, small sample szes n the groups used n the statstcal evaluaton to form pseudo-ensembles could lead to large uncertantes n the estmates of the ensemble averages. Fgure 1: Near-neutral unstable (left) and near-neutral stable (rght) normalzed concentraton values at the 400 meter arc. The neutral-unstable experments are 6, 11, 34, 45, 48 and 57, wth Monn-Obukhov lengths rangng from 263 m to 82 m. The neutral-stable experments are 21,22, 23, 24, 42, and 55, wth Monn-Obukhov lengths rangng from 164 m to 359 m. An llustraton of unexplaned concentraton varablty s presented n Fgure 1. Project Prare Grass (Barad, 1958, and Haugen, 1959) s a classc tracer dsperson experment, where sulfur-doxde (SO 2 ) was released from a small tube placed 46 cm above the ground. Seventy 20-mnutes releases were conducted durng July and August 1956, n a wheat feld near O Nel, Nebraska. Samplng arcs were postoned on semcrcles centered on the release, at downwnd dstances of 50, 100, 200, 400 and 800 m. The samplers were postoned 1.5 m above the ground, and provded 10-mnute concentraton values. For the purpose 10

11 of llustratng concentraton varablty, two small ensembles of sx experments along the 400-m arc have been grouped together n Fgure 1 usng the nverse of Monn-Obukhov length, L, a stablty parameter (as 1/L approaches zero, the surface layer of the atmosphere approaches neutral stablty condtons). Concentraton values from near-surface pont sources are nversely proportonal to the transport wnd speed, U, and drectly proportonal to the emsson rate, Q. To group the results of the sx experments together, the concentraton values have been normalzed by multplyng the concentraton values by U/Q, where U was defned as the value observed at 8 m above the ground. The sold lne shown for each group s a Gaussan ft to the results for the sx experments n the group. The scatter of the normalzed concentraton values about ths Gaussan ft can be statstcally analyzed to provde an estmate of the concentraton varablty not characterzed by the Gaussan ft. From analyses of ths and other tracer studes, the stochastc fluctuatons (nherent varablty) were nvestgated by analyzng o o the dstrbuton of C C for centerlne concentraton values. The dstrbuton was found to be approxmately lognormal, wth a standard geometrc devaton of order 1.5 to 2 (Irwn and Lee, 1997, Irwn, 1999). These results suggest that centerlne concentraton values from ndvdual experments may typcally devate from the ensemble average maxmum by as much as a factor of two. The use of wnd tunnel measurements can be a useful step towards provdng data for a model evaluaton process (e.g., Schatzmann and Letl, 1999). The work of Sten and Wyngaard (2000) nvestgates the relatonshp between nherent varablty n laboratory and atmospherc boundary layer flows. For a gven averagng tme they show that the nherent varablty n laboratory flows s smaller than n the atmospherc boundary layer flows under the same stablty and statstcal condtons. Characterzng the model nput s another source of uncertanty. The varance n modeled concentraton values due to nput uncertanty can be qute large. Usng a Gaussan plume model, Irwn et al. (1987) nvestgated the uncertanty n estmatng the hourly maxmum concentraton from elevated buoyant sources durng unstable atmospherc condtons due to model nput uncertantes. A numercal uncertanty analyss was performed usng the Monte-Carlo technque to propagate the uncertantes assocated wth the model nput. Uncertantes were assumed to exst n four model nput parameters: wnd speed, standard devaton of lateral wnd drecton fluctuatons, standard devaton of vertcal wnd drecton fluctuatons, and plume rse. It was concluded that the uncertanty n the maxmum concentraton estmates s approxmately double the uncertanty assumed n the model nput. For nstance, f half of the nput values are wthn 30% of ther error-free values, then half of the estmated maxmum concentraton values wll be wthn 60% of ther error-free values. Usng a photochemcal grd model, Hanna et al. (1998) nvestgated the uncertanty n estmatng doman-wde hourly maxmum ozone concentraton values near New York Cty for July 7-8, Ffty Monte-Carlo runs were made n whch the emssons, chemcal ntal condtons, meteorologcal nput and chemcal reacton rates were vared wthn 11

12 expected ranges of uncertanty. The amount of uncertanty vared, dependng on the varable. Those varables wth the least assumed uncertanty (most of the meteorologcal nputs) were assumed to be wthn 30% of ther error-free values 95% of the tme. Larger uncertantes were generally assumed for the emssons and reacton rates. They found the doman-wde maxmum hourly averaged ozone ranged from 176 to 331 ppb (almost a factor of two range). These two nvestgatons reveal that the senstvty to model nput uncertantes s qute large, regardless of whether the model s a Gaussan plume model, a photochemcal grd model, or whether the spece beng modeled s nert or chemcally reactve. Fgure 2. Illustraton of dsplacement of observed (sold lnes) and predcted (dashed lnes) ground-level concentraton patterns. Isopleths represent ponts wth the same concentraton. The pont-by-pont correlaton s poor, but the patterns are clearly smlar (adapted from from Hanna, 1988). [Reprnted wth permsson from the Ar Polluton Control Assocaton.] Irwn and Smth (1984) warned that dsagreement between the ndcated wnd drecton and the actual drecton of the path of a plume from an solated pont source s a major cause for dsagreement between model predctons and observatons. As a plume s transported downwnd t typcally expands at an angle of approxmately 10 degrees, and seldom s ths angle larger than 20 degrees. Wth such narrow plumes, even a 2-degree error n estmatng the plume transport drecton can cause very large dsagreement between modeled and observed surface concentraton values. Wel et al. (1992) analyzed nne perods from the EPRI Kncad experments, where each perod was about 4 hours long. They concluded that for short travel tmes (where the growth rate of the plume s wdth s nearly lnear wth travel tme), the uncertanty n the plume transport 12

13 drecton s of the order of 1/4 of the plume s total wdth. Farther downwnd, where the growth rate of the plume s wdth s less rapd, the uncertanty n the plume transport drecton s larger than 1/4 of the plume s total wdth. Fgure 2 llustrates that any pont-to-pont comparson of modeled and observed concentraton values (e.g., correlaton, bas, mean squared error) would suggest poor performance. Whereas t s clearly seen that the basc pattern s modeled well, f the observed pattern s shfted over to better correspond wth that modeled. It s concluded that plume transport drecton uncertantes are substantal, and lkely wll preclude, especally for solated source comparsons, meanngful comparson of modeled and observed concentraton values pared n tme and space. Ths secton has presented a framework, whch provdes a means for understandng why modeled and observed values dffer. The observatons are envsoned as beng composed of an ensemble mean about whch there are devatons ether resultng from representatveness and measurement uncertanty or uncharacterzed natural varablty, c(α,β). Examples were provded that suggest that for maxmum surface concentratons, uncharacterzed varablty, c(α,β), s on the order of the ensemble mean, (.e., could easly account for factor of two devatons from the mean). The model values are envsoned as beng composed of an ensemble mean about whch there are devatons ether resultng from nput (representatveness and measurement) uncertanty or model theory and numercal mplementaton errors. Examples were provded that suggest that the effects of nput uncertanty can be amplfed wthn the model (e.g., doubled), and can lead to varatons on the order of a factor of two. As a pragmatc means for assessng systematc errors n the model formulatons, t was recommended that pseudo-ensembles be formed by groupng evaluaton data nto tme perods where condtons can be assumed to be smlar. It was then recommended that comparsons be made of the group averages, as ths nsulates the comparsons from many of the sources of uncertantes. There are other ssues to be addressed, such as how to cope wth uncertantes n the drecton of transport, as dscussed n Secton 7. Frst, we wll dscuss n the next secton the knds of performance measures one mght choose n developng an evaluaton procedure. 5 Performance Measures The proceedng secton descrbed a phlosophcal framework for understandng why observatons dffer from model smulaton results. Ths secton provdes defntons of the performance measures and methods often employed n current evaluatons of ar qualty models. Proper model evaluaton nvolves the applcaton of a number of both statstcal ndces and graphcal methodologes. The lst of possble performance measures s extensve (e.g., Fox, 1981), but t has been llustrated that a few well-chosen smple-to-understand performance measures can provde adequate characterzaton of a model s performance (e.g., Hanna, 1988). Therefore, the selecton of performance measures to compare model outputs aganst observed values s a fundamental step. Statstcal ndces 13

14 and graphcal methodologes emphasze specfc model characterstcs (e.g., Canepa and Bultjes, 1999); therefore, outlnng the characterstcs of each performance measure s useful. The followng dscusson s not meant to be exhaustve. The key s not n how many performance measures are used, but s n the statstcal desgn used when the performance measures are appled (e.g., Irwn and Smth, 1984). For convenence n the followng dscusson, we dscuss the comparson of the observed and modeled concentraton values. In realty, model evaluaton can nvolve comparsons of observed and modeled plume rse, buldng wake dmensons, etc. Any feature (evaluaton objectve) that can be deduced from an analyss of the concentraton pattern and converted to a numerc value can be substtuted for the word concentratons n the followng dscusson. 5.1 Basc Performance Measures MEAN of both the observed and smulated concentratons s defned as: o s o C s C MEAN observed = C = MEAN = = smulated C (4) N N o s where N s the total number of the values beng averaged, C ( C ) s the -th observed (smulated) concentraton value. A perfect model would gve MEAN observed = MEAN smulated. Note, the values beng averaged may be for the same tme perod (an average over a set of receptors), or the values beng averaged may be at a fxed receptor locaton, ether relatve or absolute (an average over some tme perod). SIGMA (standard devaton) of both the observed and smulated concentratons, t s defned as: SIGMA observed o o 2 ( C C o ) = σ = (5) N SIGMA smulated s s 2 ( C C s ) = σ = (6) N a perfect model would gve SIGMA = SIGMA. observed smulated 5.2 Descrpton of some Pared Performance Measures Often the evaluaton procedure nvolves a comparson that logcally nvolves parng of the observed and modeled values. Ths mght be the maxmum concentraton over a doman seen for an hour or a day; the maxmum concentraton seen on receptor arcs centered on a tracer release locaton; etc. 14

15 It s not possble to assume that uncertanty n both the observatons and the modeled values s small n comparson to the varatons seen n ther mean values (Irwn et al., 1987; Wel et al., 1992; Hanna, 1993; and Hanna et al., 1998). Unless the uncertantes are small n comparson to the varatons n ther mean values, one cannot confdentally make comparsons of raw observatons wth modeled values. However, the pared comparson of group averages s meanngful, especally f the groups are well formulated and provde representatve estmates of the ensemble average concentraton for each group. BIAS s defned as: BIAS s o = C C (7) a perfect model would gve BIAS = 0, whle f BIAS > 0 (< 0) the model on average overestmates (underestmates) the observed concentratons. We have followed here and elsewhere the conventon that a postve BIAS ndcates a model overpredcton. Ths has been found to be better understood by decsonmakers and users of model evaluaton results (whereas, havng to explan a negatve BIAS as a model overpredcton was a constant problem). We menton ths, as you wll see n some lterature that the opposte conventon sometmes used. FB (Fractonal Bas) s defned as: s o C C FB = (8) s o ( C + C ) 2 t ranges between 2 and + 2. For a perfect model, FB = 0, whle f FB > 0 (< 0) the model on average overestmates (underestmates) the observed concentratons values. The MEAN, BIAS and FB characterze the on average model behavor only. One can drectly judge the average model performance by lookng at the MEAN observed and MEAN smulated values smultaneously. From the value of the BIAS, one has an dea of whether the model underestmates (BIAS < 0) or overestmates (BIAS > 0) the observed values. However, the BIAS value does not convey any sense of how large the average dfference s relatve to the average magntude of the values observed. For example, f one s dealng wth two data o o sets, characterzed by C A = 10 and C B = 100 (n approprate unts), and usng a s s model obtans C A = 20 and C B = 110, the BIAS value, n both cases, s 10. However, the on average behavor of the model s better n the case B, because the percentage dfference s less n case B. To address ths ssue, the FB can be 15

16 o s helpful. The FB s the BIAS normalzed by the average value of C and C. As far as the prevous example s concerned, FB A = 0.67 and FB B = Thus, n the case B, the better model performance s evdent. FS (Fractonal Standard devaton) s defned as: s o σ σ FS = (9) s o ( σ + σ ) 2 t ranges between 2 and + 2. For a perfect model, FS = 0, whle f FS > 0 (< 0) the spreadng of the smulated concentraton values s larger (smaller) than the spreadng of the observed concentraton values. The SIGMA and FS provde nformaton about the spread (varance) n the modeled and observed concentraton values. One can drectly judge the model performance lookng at the SIGMA observed and SIGMA smulated values smultaneously. The FS ndex s analogous to the FB ndex, only t s concerned wth the relatve dfference n the varances. COR (lnear CORrelaton coeffcent) s defned as: o o s s ( C C )( C C ) COR = (10) o s σ σ a perfect model would gve COR = + 1, t ranges between 1 and + 1. COR provdes nformaton on the strength of the lnear correlaton between the modeled and observed concentraton values. For a value of + 1, the so-called complete postve correlaton, there s correspondence between all pars of o s modeled and observed concentraton values ( C, C ). If the values were plotted aganst one another n a scatter dagram, all ponts would lay along a straght lne wth postve slope. The complete negatve correlaton corresponds to all the pars on a straght lne wth negatve slope, and COR = 1. A value of COR near to zero ndcates the absence of lnear correlaton between the varables. A model o o s s o s wll have COR = + 1 f C C = C C for any ( C, C ). Because t s o s o s possble that C C the prevous equalty does not mean C = C, for any o s ( C, C ) as we should expect for a perfect model. Furthermore, t should also be ponted out that a hgh correlaton coeffcent does not necessarly ndcate a drect dependence between the varables. Two varables may have no true relatonshp to one another, but may be correlated to a thrd varable ( spurous correlaton ). 16

17 FA2 (fracton wthn a FActor of 2) s defned as: s C fracton of data wth o C (11) a perfect model would gve FA2 = 1. NMSE (Normalzed Mean Square Error) s defned: s o ( C C ) 2 o NMSE = or, f for every, C 0 s o then, C C NMSE = s ( k ) 2 1 s k 2 (12) s o o o where k = C C and s = C C ; a perfect model would gve NMSE = 0, the value of ths ndex s always postve. WNNR (Weghted Normalzed mean square error of the Normalzed Ratos) s defned as: WNNR = 2 s ( 1 kˆ ) where k ˆ = 1 k (f k > 1) and k ˆ = k (f k 1 ); a perfect model would gve WNNR = 0, the value of ths ndex s always postve. s kˆ 2 (13) NNR (Normalzed mean square error of the dstrbuton of Normalzed Ratos) s defned as: ( 1 kˆ ) NNR = kˆ a perfect model would gve NNR = 0, the value of ths ndex s always postve. The FA2, NMSE, WNNR and NNR ndces gve nformaton about the ratos between smulated and measured concentratons. Only the FA2 and NNR ndces, out of all ndces consdered, depend solely on the ratos between smulated and measured concentratons, and not on the data set tself, so they are the only ndces strctly usable to compare smulatons of dfferent experments. NMSE attrbutes more weght to model errors concernng the estmates of the hghest measured concentratons n some cases, of the lowest ones n other cases; WNNR attrbutes more weght to model errors concernng the estmates of the hghest measured concentratons; NNR attrbutes the same weght to model errors ndependently of the poston of the data wthn the concentraton range (Pol and Crllo, 1993; Canepa and Modest, 1997). 2 (14) 17

18 SCATTER DIAGRAM, FOEX, FAα gve agan nformaton about the ratos between smulated and measured concentratons. SCATTER DIAGRAM s a graph where predcted values are plotted versus measured ones (see Fgure 3). The y = x lne represents the perfect agreement between predctons and measured values. A value above (below) the y = x lne ndcates a stuaton of overpredcton (under-predcton). FOEX s defned as FOEX N s o ( ) C = > C N (15) where N s o ( C > C ) s the number of over-predctons,.e. the number of pars where s o C > C. It ranges between 50% and + 50%. If FOEX = 50% all the ponts are below the y = x lne, f FOEX = + 50% all the ponts are above the y = x lne. The best value s 0% meanng that there are half under-predctons and half overpredctons. FOEX does not take nto account the magntude of the overpredctons, t evaluates only the number of events of over-predcton. Representng the scatter dagram on logarthmc paper, the FAα band s the regon between the two lnes of equaton y y0 = ( x x0 ) ± ln( α ) where x 0 and y 0 are the co-ordnates of the orgn of the axes. If α = 2, FAα = FA2 (see above). Fgure 3: Example of SCATTER DIAGRAM. PERCENTILES and BOX PLOT gve nformaton about the cumulatve probablty. The nth percentle of a dstrbuton of values s defned as the cumulatve probablty n percent, that s, the value that bounds the n% of values below and the (100 n)% above t. Lookng at the box plot (see Fgure 4), the general features of the dstrbuton of the consdered values can be dstngushed. 18

19 s o Fgure 4: Example of C C BOX PLOTS stratfed wth respect to the dstance from the source. Fgure 5: Example of FMS [from Grazan et al. (1998), courtesy of EI/JRC] FMS (Fgure of Mert n Space) gves nformaton about the space analyss (see Fgure 5), s defned as 19

20 FMS A A A A 1 2 = 100 (16) FMS s calculated at a fxed tme for a fxed concentraton level (sgnfcant level). FMS s the percentage of overlap between the measured (A 1 ) and predcted (A 2 ) areas. A shft n space of the concentraton patterns can reduce sgnfcantly the FMS (e.g., Fgure 2). FMT (Fgure of Mert n Tme) gves nformaton about the tme analyss, s defned as FMT x 1 2 o s { C ( t j ), C ( t j )} x x o s { C ( t ), C ( t )} mn j = 100 (17) max j FMT s calculated at a fxed locaton x for a tme seres of data. FMT evaluates the overlap of the observed and predcted concentraton patterns n tme. A temporal shft of the tme seres can reduce the FMT sgnfcantly. 5.3 Descrpton of some Unpared Performance Measures Often an evaluaton procedure nvolves a comparson that logcally nvolves unparng of the observed and modeled values. An underlyng assumpton here s that we have two samples, presumably drawn from the same dstrbuton. If the samples are representatve and from the same dstrbuton, then they should both have smlar dstrbutons. We have shown n Equatons (2) and (3) that the observed and modeled concentraton values have dfferent sources of varance, and thus are not from the same dstrbuton. However, f we have groups that are well-formulated and provde representatve samples for a seres of ensembles, then we would antcpate that the observed and modeled group averages are from the same underlyng dstrbuton, and hence should have smlar frequency dstrbutons. The QUANTILE-QUANTILE PLOT s constructed by plottng the ranked concentraton values aganst one another (e.g., hghest concentraton observed versus the hghest concentraton modeled, etc.; see Fgure 6). If the observed and modeled concentraton frequency dstrbutons are smlar, then the plotted values wll le along the 1:1 lne on the plot. By vsual nspecton, one can easly see f the respectve dstrbutons are smlar, and whether the observed and modeled concentraton maxmum values are smlar. x j x j 20

21 Fgure 6: Example of QUANTILE-QUANTILE PLOTS comparng: on the left, observed and modeled centerlne concentraton values (not recommended); observed and modeled regme average centerlne concentraton values (as recommended by the ASTM gude cted n Secton 4). Fgure 7. Example of CUMULATIVE FREQUENCY PLOTS comparng: on the left, observed and modeled centerlne concentraton values (not recommended); on the rght, observed and modeled regme average centerlne concentraton values (as recommended by the ASTM D6589 cted n Secton 4). 21

22 The CUMULATIVE FREQUENCY PLOT (see Fgure 7) s constructed by plottng the ranked concentraton values (lowest to hghest) aganst the plottng poston frequency, f (typcally n percent), where p s the rank (1 = lowest), N s the number of values and f s defned as (Larsen, 1969): ( N p + 0.6) N for p 2 f = 100% 100% > N (18) ( p 0.4) N for p 2 f = 100% < N As wth the QUANTILE-QUANTILE PLOT, a vsual nspecton of the respectve CUMULATIVE FREQUENCY DISTRIBUTION PLOTS (observed and modeled), s usually suffcent to suggest whether the two dstrbutons are smlar, and whether there s a bas n the model to over- or under-estmate the maxmum concentraton values observed. The RHC (Robust Hghest Concentraton) ndex s often used where comparsons are beng made of the maxmum concentraton values, and s envsoned as a more robust test statstc than drect comparson of maxmum values. The RHC s based on an exponental ft to the hghest R - 1 values of the cumulatve frequency dstrbuton, where R s typcally set to be 26 for frequency dstrbutons nvolvng a year s worth of values (averagng tmes of 24 hours or less) (Cox and Tkvart, 1990). The RHC s computed as: 3R 1 RHC = C( R) + Θ ln (19) 2 where Θ s the average of the R-1 largest values, and C(R) s the R th largest value. The value of R may be set to a lower value when there are fewer values n the dstrbuton to work wth; the RHC of the observed and modeled cumulatve frequency dstrbutons are often compared usng a FB ndex, see Cox and Tkvart (1990). 5.4 Bootstrap Resamplng The standard analytcal formulas for confdence ntervals on performance measures from statstcs textbooks may be napproprate (Fox, 1984), snce ar qualty data and model performance measures are not necessarly normallydstrbuted nor can they always be transformed to a normal dstrbuton. The bootstrap resamplng procedure (Hedam, 1987, Hanna, 1989, Cox and Tkvart, 1990, Efron and Tbshran, 1993) was suggested as an alternatve method, snce t dd not depend on the form of the underlyng dstrbuton functon. Followng the descrpton provded by Efron and Tbshran (1993), suppose one s analyzng a data set x1,x2,... xn, whch for convenence s denoted by the vector * * * * x = ( x1,x2,...xn ). A bootstrap sample x = ( x1,x2,...xn ) s obtaned by randomly samplng n tmes, wth replacement, from the orgnal data ponts 22

23 x = ( x1,x2,...xn ). For nstance, wth n = 7 one mght obtan * x = ( x5,x7,x5,x4,x7,x3,x1 ). From each bootstrap sample one can compute some statstc s (say the medan, average, RHC, etc.). By creatng a number of bootstrap samples, m, one can compute the mean, s, and standard devaton, σ s, of the statstc of nterest. For estmaton of standard errors, m s typcally on the order of 50 to 500. The bootstrap resamplng procedure often can be mproved by blockng the data nto two or more blocks or sets, wth each block contanng data havng smlar characterstcs. Ths prevents the possblty of creatng an unrealstc bootstrap sample where all the members are the same value (Hanna, 1989). When performng model evaluatons and model ntercomparsons, for each hour there are not only the observed concentraton values, but also the modelng results from all the models beng tested. In such cases, the ndvdual members, x, n the vector x = ( x1,x2,...xn ) are n themselves vectors, composed of the observed value and ts assocated modelng results (from all models, f there are more than one). Thus the selecton of the bootstrap sample x * also ncludes each model s estmate for ths case. For example, suppose confdence lmts are desred on the NMSE calculated from a set of n couples (C s, C o ), where C s s the model smulaton estmate and C o s the correspondng observed value. In the bootstrap procedure, a new set of n couples (C s, C o ) s randomly drawn from the orgnal set. If a gven (C s, C o ) s drawn, t s replaced before the next draw s made. Thus t s possble (but not very probable) that all n draws consst of the same couple (C s, C o ). For each resample set of sze n, the NMSE s calculated. If m resamples are drawn, the cumulatve dstrbuton functon of the m values of NMSE wll provde estmates of confdence lmts on NMSE. For example, the 95% confdence nterval on NMSE wll range from the 2.5% to the 97.5% ponts on NMSE dstrbuton. For assessng dfferences n model performance, one often wshes to test whether the dfferences seen n a performance measure computed between Model #1 and the observatons (say the NMSE1 ), s sgnfcantly dfferent when compared to that computed for another model (say Model #2, NMSE2 ) usng the same observatons. For testng whether the dfference between performance measures s sgnfcant, the followng procedure s recommended. Let each bootstrap sample be denoted x *b, where * ndcates ths s a bootstrap sample and b ndcates ths s sample b of a seres of bootstrap samples (where the total number of bootstrap samples s B). From each bootstrap sample, x *b, one computes the respectve values for NMSE b 1 and NMSE b 2. The dfference *b *b *b = NMSE 1 - NMSE 2 can then be computed. Once all B samples have been * *1 *2 *B processed, compute from the set of B values of = (,,... ), the average 23

24 and standard devaton, and σ. The null hypothess s that s not equal to zero wth a stated level of confdence, α, and the t-value for use n a Student s t- test s: t = (20) σ For llustraton purposes, assume the level of confdence s 90% (α = 0.1). Then for large values of B, f the t-value from the above equaton s larger than Student s t α /2 equal to 1.645, t can be concluded wth 90% confdence that s not equal to zero, and hence there s a sgnfcant dfference n the NMSE values for the two models beng tested. 6 Model Evaluaton Model evaluaton s one of the elements of model qualty assurance (see Secton 8). Model evaluaton s tself a system of procedures desgned to measure performance (Model Evaluaton Group, 1994a, 1994b; U.S. Envronmental Protecton Agency, 1997). Followng Borrego et al. (2001b), model evaluaton s composed of: model algorthm verfcaton, senstvty analyss, uncertanty analyss, statstcal model evaluaton, and model ntercomparson. Therefore, statstcal model evaluaton s one of the fundamental steps to acheve model evaluaton. Model algorthm verfcaton s the checkng of the computer code to ensure the code s a true representaton of the conceptual model on whch t s based. Ths ncludes: checkng that the mathematcal equatons nvolved have been solved correctly, and comparng numercal solutons wth dealzed cases for whch an analytc soluton exsts ( verfcaton of numercal solutons ) to demonstrate that the two match over the partcular range of condtons under consderaton. Senstvty analyss s a process for characterzng the response of a model to changes n nput and parameter values. The purpose s to dentfy the magntude, drecton, and form (e.g., lnear or non-lnear) of the effect of such varatons. Senstvty tests can be performed wth respect to: 1) uncertanty of physcs/chemstry model parameters, and 2) uncertanty of emsson and meteorologcal model nput data. In ether case, one can use two methods: a) systematcally vary one or more of the model nputs to determne the effect on the modelng results (Hlst, 1970), or b) perform a Monte Carlo study wth random samplng (Irwn et al., 1987). In tradtonal senstvty studes (tem a), each nput would be vared over a reasonable range lkely to be encountered. These studes were routnely performed n the early years of ar polluton modelng, to develop a better understandng of the performance of plume dsperson models smulatng the transport and dffuson of nert pollutants. Monte Carlo studes (tem b) are becomng more common, as they provde a sense of the overall response of the modelng system to known 24