ASTM D6589-05(2010)e1
(Guide)Standard Guide for Statistical Evaluation of Atmospheric Dispersion Model Performance
Standard Guide for Statistical Evaluation of Atmospheric Dispersion Model Performance
SIGNIFICANCE AND USE
Guidance is provided on designing model evaluation performance procedures and on the difficulties that arise in statistical evaluation of model performance caused by the stochastic nature of dispersion in the atmosphere. It is recognized there are examples in the literature where, knowingly or unknowingly, models were evaluated on their ability to describe something which they were never intended to characterize. This guide is attempting to heighten awareness, and thereby, to reduce the number of “unknowing” comparisons. A goal of this guide is to stimulate development and testing of evaluation procedures that accommodate the effects of natural variability. A technique is illustrated to provide information from which subsequent evaluation and standardization can be derived.
SCOPE
1.1 This guide provides techniques that are useful for the comparison of modeled air concentrations with observed field data. Such comparisons provide a means for assessing a model's performance, for example, bias and precision or uncertainty, relative to other candidate models. Methodologies for such comparisons are yet evolving; hence, modifications will occur in the statistical tests and procedures and data analysis as work progresses in this area. Until the interested parties agree upon standard testing protocols, differences in approach will occur. This guide describes a framework, or philosophical context, within which one determines whether a model's performance is significantly different from other candidate models. It is suggested that the first step should be to determine which model's estimates are closest on average to the observations, and the second step would then test whether the differences seen in the performance of the other models are significantly different from the model chosen in the first step. An example procedure is provided in Appendix X1 to illustrate an existing approach for a particular evaluation goal. This example is not intended to inhibit alternative approaches or techniques that will produce equivalent or superior results. As discussed in Section 6, statistical evaluation of model performance is viewed as part of a larger process that collectively is referred to as model evaluation.
1.2 This guide has been designed with flexibility to allow expansion to address various characterizations of atmospheric dispersion, which might involve dose or concentration fluctuations, to allow development of application-specific evaluation schemes, and to allow use of various statistical comparison metrics. No assumptions are made regarding the manner in which the models characterize the dispersion.
1.3 The focus of this guide is on end results, that is, the accuracy of model predictions and the discernment of whether differences seen between models are significant, rather than operational details such as the ease of model implementation or the time required for model calculations to be performed.
1.4 This guide offers an organized collection of information or a series of options and does not recommend a specific course of action. This guide cannot replace education or experience and should be used in conjunction with professional judgment. Not all aspects of this guide may be applicable in all circumstances. This guide is not intended to represent or replace the standard of care by which the adequacy of a given professional service must be judged, nor should it be applied without consideration of a project's many unique aspects. The word “Standard” in the title of this guide means only that the document has been approved through the ASTM consensus process.
1.5 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety and health practices and to determine the applicability of regulatory limitations prior to use.
General Information
Relations
Standards Content (Sample)
NOTICE: This standard has either been superseded and replaced by a new version or withdrawn.
Contact ASTM International (www.astm.org) for the latest information
´1
Designation: D6589 − 05(Reapproved 2010)
Standard Guide for
Statistical Evaluation of Atmospheric Dispersion Model
Performance
This standard is issued under the fixed designation D6589; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
ε NOTE—Reapproved with editorial corrections in April 2010.
1. Scope differences seen between models are significant, rather than
operationaldetailssuchastheeaseofmodelimplementationor
1.1 This guide provides techniques that are useful for the
the time required for model calculations to be performed.
comparison of modeled air concentrations with observed field
data. Such comparisons provide a means for assessing a 1.4 This guide offers an organized collection of information
model’s performance, for example, bias and precision or oraseriesofoptionsanddoesnotrecommendaspecificcourse
uncertainty, relative to other candidate models. Methodologies of action. This guide cannot replace education or experience
for such comparisons are yet evolving; hence, modifications and should be used in conjunction with professional judgment.
will occur in the statistical tests and procedures and data Not all aspects of this guide may be applicable in all circum-
analysis as work progresses in this area. Until the interested stances. This guide is not intended to represent or replace the
parties agree upon standard testing protocols, differences in standard of care by which the adequacy of a given professional
approach will occur. This guide describes a framework, or service must be judged, nor should it be applied without
philosophical context, within which one determines whether a consideration of a project’s many unique aspects. The word
model’s performance is significantly different from other “Standard” in the title of this guide means only that the
candidate models. It is suggested that the first step should be to document has been approved through the ASTM consensus
determine which model’s estimates are closest on average to process.
the observations, and the second step would then test whether
1.5 The values stated in SI units are to be regarded as
the differences seen in the performance of the other models are
standard. No other units of measurement are included in this
significantly different from the model chosen in the first step.
guide.
AnexampleprocedureisprovidedinAppendixX1toillustrate
1.6 This standard does not purport to address all of the
an existing approach for a particular evaluation goal. This
safety concerns, if any, associated with its use. It is the
example is not intended to inhibit alternative approaches or
responsibility of the user of this standard to establish appro-
techniques that will produce equivalent or superior results. As
priate safety and health practices and to determine the
discussed in Section 6, statistical evaluation of model perfor-
applicability of regulatory limitations prior to use.
mance is viewed as part of a larger process that collectively is
referred to as model evaluation.
2. Referenced Documents
1.2 This guide has been designed with flexibility to allow
2.1 ASTM Standards:
expansion to address various characterizations of atmospheric
D1356 Terminology Relating to Sampling and Analysis of
dispersion, which might involve dose or concentration
Atmospheres
fluctuations, to allow development of application-specific
evaluation schemes, and to allow use of various statistical 3. Terminology
comparison metrics. No assumptions are made regarding the
3.1 Definitions—For definitions of terms used in this guide,
manner in which the models characterize the dispersion.
refer to Terminology D1356.
1.3 The focus of this guide is on end results, that is, the
3.2 Definitions of Terms Specific to This Standard:
accuracy of model predictions and the discernment of whether
3.2.1 atmospheric dispersion model, n—an idealization of
atmospheric physics and processes to calculate the magnitude
This guide is under the jurisdiction of ASTM Committee D22 on Air Quality
and is the direct responsibility of Subcommittee D22.11 on Meteorology. For referenced ASTM standards, visit the ASTM website, www.astm.org, or
Current edition approved April 1, 2010. Published July 2010. Originally contact ASTM Customer Service at service@astm.org. For Annual Book of ASTM
approved in 2000. Last previous edition approved in 2005 as D6589 - 05. DOI: Standards volume information, refer to the standard’s Document Summary page on
10.1520/D6589-05R10E01. the ASTM website.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
´1
D6589 − 05 (2010)
and location of pollutant concentrations based on fate, features (evaluation objectives) within the observed and mod-
transport, and dispersion in the atmosphere. This may take the eled concentration patterns to be compared, for example,
form of an equation, algorithm, or series of equations/ maximum surface concentrations, lateral extent of a dispersing
algorithms used to calculate average or time-varying concen- plume. The selection and definition of evaluation objectives
tration. The model may involve numerical methods for solu- typically are tailored to the model’s capabilities and intended
tion. uses. The very nature of the problem of characterizing air
quality and the way models are applied make one single or
3.2.2 dispersion, absolute, n—the characterization of the
absolute evaluation objective impossible to define that is
spreading of material released into the atmosphere based on a
suitable for all purposes. The definition of the evaluation
coordinate system fixed in space.
objectives will be restricted by the limited range conditions
3.2.3 dispersion, relative, n—the characterization of the
experienced in the available comparison data suitable for use.
spreading of material released into the atmosphere based on a
For each evaluation objective, a procedure will need to be
coordinate system that is relative to the local median position
defined that allows definition of the evaluation objective from
of the dispersing material.
the available observations of concentration values.
3.2.4 evaluation objective, n—a feature or characteristic,
4.3 In assessing the performance of air quality models to
which can be defined through an analysis of the observed
characterize a particular evaluation objective, one should
concentration pattern, for example, maximum centerline con-
consider what the models are capable of providing. As dis-
centration or lateral extent of the average concentration pattern
cussed in Section 7, most models attempt to characterize the
as a function of downwind distance, which one desires to
ensemble average concentration pattern. If such models should
assess the skill of the models to reproduce.
provide favorable comparisons with observed concentration
3.2.5 evaluation procedure, n—the analysis steps to be
maxima,thisisresultingfromhappenstance,ratherthanskillin
takentocomputethevalueoftheevaluationobjectivefromthe
the model; therefore, in this discussion, it is suggested a model
observed and modeled patterns of concentration values.
be assessed on its ability to reproduce what it was designed to
3.2.6 fate, n—the destiny of a chemical or biological pol- produce, for at least in these comparisons, one can be assured
lutant after release into the environment.
that zero bias with the least amount of scatter is by definition
good model performance.
3.2.7 model input value, n—characterizations that must be
estimated or provided by the model developer or user before
4.4 As an illustration of the principles espoused in this
model calculations can be performed.
guide, a procedure is provided in Appendix X1 for comparison
of observed and modeled near-centerline concentration values,
3.2.8 regime, n—a repeatable narrow range of conditions,
which accommodates the fact that observed concentration
defined in terms of model input values, which may or may not
values include a large component of stochastic, and possibly
be explicitly employed by all models being tested, needed for
deterministic, variability unaccounted for by current models.
dispersion model calculations. It is envisioned that the disper-
The procedure provides an objective statistical test of whether
sion observed should be similar for all cases having similar
differences seen in model performance are significant.
model input values.
3.2.9 uncertainty, n—refers to a lack of knowledge about
5. Significance and Use
specific factors or parameters. This includes measurement
5.1 Guidance is provided on designing model evaluation
errors, sampling errors, systematic errors, and differences
performance procedures and on the difficulties that arise in
arising from simplification of real-world processes. In
statistical evaluation of model performance caused by the
principle, uncertainty can be reduced with further information
stochastic nature of dispersion in the atmosphere. It is recog-
or knowledge (1) .
nized there are examples in the literature where, knowingly or
3.2.10 variability, n—refers to differences attributable to
unknowingly, models were evaluated on their ability to de-
true heterogeneity or diversity in atmospheric processes that
scribe something which they were never intended to charac-
result in part from natural random processes. Variability
terize. This guide is attempting to heighten awareness, and
usually is not reducible by further increases in knowledge, but
thereby, to reduce the number of “unknowing” comparisons.A
it can in principle be better characterized (1).
goal of this guide is to stimulate development and testing of
evaluation procedures that accommodate the effects of natural
4. Summary of Guide
variability. A technique is illustrated to provide information
4.1 Statistical evaluation of dispersion model performance
from which subsequent evaluation and standardization can be
with field data is viewed as part of a larger process that
derived.
collectively is called model evaluation. Section 6 discusses the
6. Model Evaluation
components of model evaluation.
6.1 Background—Air quality simulation models have been
4.2 To statistically assess model performance, one must
used for many decades to characterize the transport and
define an overall evaluation goal or purpose. This will suggest
dispersion of material in the atmosphere (2-4). Early evalua-
tions of model performance usually relied on linear least-
squares analyses of observed versus modeled values, using
The boldface numbers in parentheses refer to the list of references at the end of
this standard. traditional scatter plots of the values, (5-7). During the 1980s,
´1
D6589 − 05 (2010)
attempts have been made to encourage the standardization of evaluation objectives. In order to conduct a statistical
methods used to judge air quality model performance (8-11). comparison, one will have to define one or more evaluation
Further development of these proposed statistical evaluation objectives for which objective comparisons are desired (Sec-
procedures was needed, as it was found that the rote applica- tion 10).As discussed in 8.4.4, the process of summarizing the
tion of statistical metrics, such as those listed in (8), was overall performance of a model over the range of conditions
incapable of discerning differences in model performance (12), experienced within a field experiment typically involves deter-
whereas if the evaluation results were sorted by stability and mining two points for each of the model evaluation objectives:
distancedownwind,thendifferencesinmodelingskillcouldbe whichofthemodelsbeingassessedhasonaveragethesmallest
discerned (13). It was becoming increasingly evident that the combined bias and scatter in comparisons with observations,
models were characterizing only a small portion of the ob- and whether the differences seen in the comparisons with the
served variations in the concentration values (14). To better other models statistically are significant in light of the uncer-
deduce the statistical significance of differences seen in model tainties in the observations.
performance in the face of large unaccounted for uncertainties
6.5 Other Tasks Supportive to Model Evaluation—As atmo-
and variations, investigators began to explore the use of
spheric dispersion models become more sophisticated, it is not
bootstraptechniques (15).Bythelate1980s,mostofthemodel
easytodetectcodingerrorsintheimplementationofthemodel
performance evaluations involved the use of bootstrap tech-
algorithms. And as models become more complex, discerning
niques in the comparison of maximum values of modeled and
the sensitivity of the modeling results to input parameter
observed cumulative frequency distributions of the concentra-
variations becomes less clear; hence, two important tasks that
tionsvalues (16).Eventhoughtheproceduresandmetricstobe
support model evaluation efforts are verification of software
employed in describing the performance of air quality simula-
and sensitivity and Monte Carlo analyses.
tion models are still evolving (17-19), there has been a general
6.5.1 Verification of Software—Often a set of modeling
acceptance that defining performance of air quality models
algorithms will require numerical solution. An important task
needs to address the large uncertainties inherent in attempting
supportive to a model evaluation is a review in which the
to characterize atmospheric fate, transport and dispersion
mathematics described in the technical description of the
processes. There also has been a consensus reached on the
model are compared with the numerical coding, to ensure that
philosophical reasons that models of earth science processes
the code faithfully implements the physics and mathematics.
can never be validated, in the sense of claiming that a model is
6.5.2 SensitivityandMonteCarloAnalyses—Sensitivityand
truthfully representing natural processes. No general empirical
Monte Carlo analyses provide insight into the response of a
proposition about the natural world can be certain, since there
model to input variation. An example of this technique is to
will always remain the prospect that future observations may
systematically vary one or more of the model inputs to
callthetheoryinquestion (20).Itisseenthatnumericalmodels
determine the effect on the modeling results (22). Each input
of air pollution are a form of a highly complex scientific
should be varied over a reasonable range likely to be encoun-
hypothesis concerning natu
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.