ASTM D6589-23
(Guide)Standard Guide for Statistical Evaluation of Atmospheric Dispersion Model Performance
Standard Guide for Statistical Evaluation of Atmospheric Dispersion Model Performance
SIGNIFICANCE AND USE
5.1 Guidance is provided on designing model evaluation performance procedures and on the difficulties that arise in statistical evaluation of model performance caused by the stochastic nature of dispersion in the atmosphere. It is recognized there are examples in the literature where, knowingly or unknowingly, models were evaluated on their ability to describe something which they were never intended to characterize. This guide is attempting to heighten awareness, and thereby, to reduce the number of “unknowing” comparisons. A goal of this guide is to stimulate development and testing of evaluation procedures that accommodate the effects of natural variability. A technique is illustrated to provide information from which subsequent evaluation and standardization can be derived.
SCOPE
1.1 This guide provides techniques that are useful for the comparison of modeled air concentrations with observed field data. Such comparisons provide a means for assessing a model's performance, for example, bias and precision or uncertainty, relative to other candidate models. Methodologies for such comparisons are yet evolving; hence, modifications will occur in the statistical tests and procedures and data analysis as work progresses in this area. Until the interested parties agree upon standard testing protocols, differences in approach will occur. This guide describes a framework, or philosophical context, within which one determines whether a model's performance is significantly different from other candidate models. It is suggested that the first step should be to determine which model's estimates are closest on average to the observations, and the second step would then test whether the differences seen in the performance of the other models are significantly different from the model chosen in the first step. An example procedure is provided in Appendix X1 to illustrate an existing approach for a particular evaluation goal. This example is not intended to inhibit alternative approaches or techniques that will produce equivalent or superior results. As discussed in Section 6, statistical evaluation of model performance is viewed as part of a larger process that collectively is referred to as model evaluation.
1.2 This guide has been designed with flexibility to allow expansion to address various characterizations of atmospheric dispersion, which might involve dose or concentration fluctuations, to allow development of application-specific evaluation schemes, and to allow use of various statistical comparison metrics. No assumptions are made regarding the manner in which the models characterize the dispersion.
1.3 The focus of this guide is on end results, that is, the accuracy of model predictions and the discernment of whether differences seen between models are significant, rather than operational details such as the ease of model implementation or the time required for model calculations to be performed.
1.4 This guide offers an organized collection of information or a series of options and does not recommend a specific course of action. This guide cannot replace education or experience and should be used in conjunction with professional judgment. Not all aspects of this guide may be applicable in all circumstances. This guide is not intended to represent or replace the standard of care by which the adequacy of a given professional service must be judged, nor should it be applied without consideration of a project's many unique aspects. The word “Standard” in the title of this guide means only that the document has been approved through the ASTM consensus process.
1.5 This standard applies to gaussian plume models; it may not be applicable to non-point sources, heavy gas models from evaporation from pool (for example, liquid spills), as well as near-field receptors.
1.6 The values stated in SI units are to be regarded as standard. No other units of measurement are included in this guide...
General Information
Relations
Buy Standard
Standards Content (Sample)
This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
Designation: D6589 − 23
Standard Guide for
Statistical Evaluation of Atmospheric Dispersion Model
1
Performance
This standard is issued under the fixed designation D6589; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope operational details such as the ease of model implementation or
the time required for model calculations to be performed.
1.1 This guide provides techniques that are useful for the
1.4 This guide offers an organized collection of information
comparison of modeled air concentrations with observed field
or a series of options and does not recommend a specific course
data. Such comparisons provide a means for assessing a
of action. This guide cannot replace education or experience
model’s performance, for example, bias and precision or
and should be used in conjunction with professional judgment.
uncertainty, relative to other candidate models. Methodologies
Not all aspects of this guide may be applicable in all circum-
for such comparisons are yet evolving; hence, modifications
stances. This guide is not intended to represent or replace the
will occur in the statistical tests and procedures and data
standard of care by which the adequacy of a given professional
analysis as work progresses in this area. Until the interested
service must be judged, nor should it be applied without
parties agree upon standard testing protocols, differences in
consideration of a project’s many unique aspects. The word
approach will occur. This guide describes a framework, or
“Standard” in the title of this guide means only that the
philosophical context, within which one determines whether a
document has been approved through the ASTM consensus
model’s performance is significantly different from other
process.
candidate models. It is suggested that the first step should be to
determine which model’s estimates are closest on average to
1.5 This standard applies to gaussian plume models; it may
the observations, and the second step would then test whether
not be applicable to non-point sources, heavy gas models from
the differences seen in the performance of the other models are
evaporation from pool (for example, liquid spills), as well as
significantly different from the model chosen in the first step.
near-field receptors.
An example procedure is provided in Appendix X1 to illustrate
1.6 The values stated in SI units are to be regarded as
an existing approach for a particular evaluation goal. This
standard. No other units of measurement are included in this
example is not intended to inhibit alternative approaches or
guide.
techniques that will produce equivalent or superior results. As
1.7 This standard does not purport to address all of the
discussed in Section 6, statistical evaluation of model perfor-
safety concerns, if any, associated with its use. It is the
mance is viewed as part of a larger process that collectively is
responsibility of the user of this standard to establish appro-
referred to as model evaluation.
priate safety, health, and environmental practices and deter-
1.2 This guide has been designed with flexibility to allow
mine the applicability of regulatory limitations prior to use.
expansion to address various characterizations of atmospheric
1.8 This international standard was developed in accor-
dispersion, which might involve dose or concentration
dance with internationally recognized principles on standard-
fluctuations, to allow development of application-specific
ization established in the Decision on Principles for the
evaluation schemes, and to allow use of various statistical
Development of International Standards, Guides and Recom-
comparison metrics. No assumptions are made regarding the
mendations issued by the World Trade Organization Technical
manner in which the models characterize the dispersion.
Barriers to Trade (TBT) Committee.
1.3 The focus of this guide is on end results, that is, the
2. Referenced Documents
accuracy of model predictions and the discernment of whether
2
2.1 ASTM Standards:
differences seen between models are significant, rather than
D1356 Terminology Relating to Sampling and Analysis of
Atmospheres
1
This guide is under the jurisdiction of ASTM Committee D22 on Air Quality
2
and is the direct responsibility of Subcommittee D22.11 on Meteorology. For referenced ASTM standards, visit the ASTM website, www.astm.org, or
Current edition approved Sept. 1, 2023. Published September 2023. Originally con
...
This document is not an ASTM standard and is intended only to provide the user of an ASTM standard an indication of what changes have been made to the previous version. Because
it may not be technically possible to adequately depict all changes accurately, ASTM recommends that users consult prior editions as appropriate. In all cases only the current version
of the standard as published by ASTM is to be considered the official document.
Designation: D6589 − 05 (Reapproved 2015) D6589 − 23
Standard Guide for
Statistical Evaluation of Atmospheric Dispersion Model
1
Performance
This standard is issued under the fixed designation D6589; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope
1.1 This guide provides techniques that are useful for the comparison of modeled air concentrations with observed field data. Such
comparisons provide a means for assessing a model’s performance, for example, bias and precision or uncertainty, relative to other
candidate models. Methodologies for such comparisons are yet evolving; hence, modifications will occur in the statistical tests and
procedures and data analysis as work progresses in this area. Until the interested parties agree upon standard testing protocols,
differences in approach will occur. This guide describes a framework, or philosophical context, within which one determines
whether a model’s performance is significantly different from other candidate models. It is suggested that the first step should be
to determine which model’s estimates are closest on average to the observations, and the second step would then test whether the
differences seen in the performance of the other models are significantly different from the model chosen in the first step. An
example procedure is provided in Appendix X1 to illustrate an existing approach for a particular evaluation goal. This example
is not intended to inhibit alternative approaches or techniques that will produce equivalent or superior results. As discussed in
Section 6, statistical evaluation of model performance is viewed as part of a larger process that collectively is referred to as model
evaluation.
1.2 This guide has been designed with flexibility to allow expansion to address various characterizations of atmospheric
dispersion, which might involve dose or concentration fluctuations, to allow development of application-specific evaluation
schemes, and to allow use of various statistical comparison metrics. No assumptions are made regarding the manner in which the
models characterize the dispersion.
1.3 The focus of this guide is on end results, that is, the accuracy of model predictions and the discernment of whether differences
seen between models are significant, rather than operational details such as the ease of model implementation or the time required
for model calculations to be performed.
1.4 This guide offers an organized collection of information or a series of options and does not recommend a specific course of
action. This guide cannot replace education or experience and should be used in conjunction with professional judgment. Not all
aspects of this guide may be applicable in all circumstances. This guide is not intended to represent or replace the standard of care
by which the adequacy of a given professional service must be judged, nor should it be applied without consideration of a project’s
many unique aspects. The word “Standard” in the title of this guide means only that the document has been approved through the
ASTM consensus process.
1.5 This standard applies to gaussian plume models; it may not be applicable to non-point sources, heavy gas models from
evaporation from pool (for example, liquid spills), as well as near-field receptors.
1
This guide is under the jurisdiction of ASTM Committee D22 on Air Quality and is the direct responsibility of Subcommittee D22.11 on Meteorology.
Current edition approved April 1, 2015Sept. 1, 2023. Published April 2015September 2023. Originally approved in 2000. Last previous edition approved in 20102015 as
ε1
D6589 – 05 (2010)(2015). . DOI: 10.1520/D6589-05R15.10.1520/D6589-23.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
1
---------------------- Page: 1 ----------------------
D6589 − 23
1.6 The values stated in SI units are to be regarded as standard. No other units of measurement are included in this guide.
1.7 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility
of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of
regulatory limitations prior to use.
1.8 This international standa
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.