ASTM D5157-97
(Guide)Standard Guide for Statistical Evaluation of Indoor Air Quality Models
Standard Guide for Statistical Evaluation of Indoor Air Quality Models
SCOPE
1.1 This guide provides quantitative and qualitative tools for evaluation of indoor air quality (IAQ) models. These tools include methods for assessing overall model performance as well as identifying specific areas of deficiency. Guidance is also provided in choosing data sets for model evaluation and in applying and interpreting the evaluation tools. The focus of the guide is on end results (that is, the accuracy of indoor concentrations predicted by a model), rather than operational details such as the ease of model implementation or the time required for model calculations to be performed.
1.2 Although IAQ models have been used for some time, there is little guidance in the technical literature on the evaluation of such models. Evaluation principles and tools in this guide are drawn from past efforts related to outdoor air quality or meteorological models, which have objectives similar to those for IAQ models and a history of evaluation literature.(1) Some limited experience exists in the use of these tools for evaluation of IAQ models.
General Information
Relations
Standards Content (Sample)
NOTICE: This standard has either been superceded and replaced by a new version or discontinued.
Contact ASTM International (www.astm.org) for the latest information.
Designation: D 5157 – 97
Standard Guide for
Statistical Evaluation of Indoor Air Quality Models
This standard is issued under the fixed designation D 5157; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (e) indicates an editorial change since the last revision or reapproval.
1. Scope 3.2.3 model chamber, n—an indoor airspace of defined
volume used in model calculations; IAQ models can be
1.1 This guide provides quantitative and qualitative tools for
specified for a single chamber or for multiple, interconnected
evaluation of indoor air quality (IAQ) models. These tools
chambers.
include methods for assessing overall model performance as
3.2.4 model evaluation, n—a series of steps through which
well as identifying specific areas of deficiency. Guidance is
a model developer or user assesses a model’s performance for
also provided in choosing data sets for model evaluation and in
selected situations.
applying and interpreting the evaluation tools. The focus of the
3.2.5 model parameter, n—a mathematical term in an IAQ
guide is on end results (that is, the accuracy of indoor
model that must be estimated by the model developer or user
concentrations predicted by a model), rather than operational
before model calculations can be performed.
details such as the ease of model implementation or the time
3.2.6 model residual, n—the difference between an indoor
required for model calculations to be performed.
concentration predicted by an IAQ model and a representative
1.2 Although IAQ models have been used for some time,
measurement of the true indoor concentration; the value should
there is little guidance in the technical literature on the
be stated as positive or negative.
evaluation of such models. Evaluation principles and tools in
3.2.7 model validation, n—a series of evaluations under-
this guide are drawn from past efforts related to outdoor air
taken by an agency or organization to provide a basis for
quality or meteorological models, which have objectives simi-
endorsing a specific model (or models) for a specific applica-
lar to those for IAQ models and a history of evaluation
tion (or applications).
literature.(1) Some limited experience exists in the use of
3.2.8 pollutant concentration, n—the extent of the occur-
these tools for evaluation of IAQ models.
rence of a pollutant or the parameters describing a pollutant in
2. Referenced Documents a defined airspace, expressed in units characteristic to the
3 3 3
pollutant (for example, mg/m , ppm, Bq/m , area/m , or colony
2.1 ASTM Standards:
forming units per cubic metre).
D 1356 Terminology Relating to Sampling and Analysis of
Atmospheres
4. Significance and Use
3. Terminology 4.1 Using the tools described in this guide, an individual
seeking to apply an IAQ model should be able to (1) assess the
3.1 Definitions: For definitions of terms used in this stan-
performance of the model for a specific situation or (2)
dard, refer to Terminology D 1356.
recognize or assess its advantages and limitations.
3.2 Definitions of Terms Specific to This Standard:
4.2 This guide can also be used for identifying specific areas
3.2.1 IAQ model, n—an equation, algorithm, or series of
of model deficiency that require further development or refine-
equations/algorithms used to calculate average or time-varying
ment.
pollutant concentrations in one or more indoor chambers for a
specific situation.
5. Components of Model Evaluation
3.2.2 model bias, n—a systematic difference between model
5.1 The components of model evaluation include the fol-
predictions and measured indoor concentrations (for example,
lowing: (1) stating the purpose(s) or objective(s) of the
the model prediction is generally higher than the measured
evaluation, (2) acquiring a basic understanding of the specifi-
concentration for a specific situation).
cation and underlying principles or assumptions, (3) selecting
data sets as inputs to the evaluation process, and (4) selecting
This guide is under the jurisdiction of ASTM Committee D-22 on Sampling and and using appropriate tools for assessing model performance.
Analysis of Atmospheres and is the direct responsibility of Subcommittee D22.05
Just as model evaluation has multiple components, model
on Indoor Air.
validation consists of one or more evaluations. However,
Current edition approved March 10, 1997. Published May 1997. Originally
model validation is beyond the scope of this document.
published as D 5157 – 91. Last previous edition D 5157 – 91.
The boldface numbers in parentheses refer to the list of references at the end of
5.1.1 Establishing Evaluation Objectives:
this standard.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959, United States.
NOTICE: This standard has either been superceded and replaced by a new version or discontinued.
Contact ASTM International (www.astm.org) for the latest information.
D 5157
5.1.1.1 IAQ models are generally used for the following: (1) ships could be estimated through techniques such as regression
analysis.
to help explain the temporal and spatial variations in the
occurrences of indoor pollutant concentrations, (2) to improve
5.1.2.5 IAQ models may be specified for a particular pol-
the understanding of major influencing factors or underlying
lutant or in general terms; this distinction is important, for
physical/chemical processes, and (3) to predict the temporal/ example, because particle-phase pollutants behave differently
spatial variations in indoor concentrations that can be expected
from gas-phase pollutants. Particulate matter is subject to
to occur in specific types of situations. However, model coagulation, chemical reaction at surfaces, gravitational set-
evaluation relates only to the third type of model use— tling, diffusional deposition, resuspension and interception,
prediction of indoor concentrations. impaction, and diffusional removal by filtration devices;
whereas some gaseous pollutants are subject to sorption and, in
5.1.1.2 The most common evaluation objectives are (1)to
some cases, desorption processes.
compare the performance of two or more models for a specific
5.1.2.6 Dynamic IAQ models predict time-varying indoor
situation or set of situations and (2) to assess the performance
concentrations for time steps that are usually on the order of
of a specific model for different situations. Secondary objec-
seconds, minutes, or hours; whereas integrated models predict
tives include identifying specific areas of model deficiency.
time-averaged indoor concentrations using average values for
Determination of specific objectives will assist in choosing
each input parameter or averaging these parameters during the
appropriate data sets and quantitative or qualitative tools for
course of exercising the model. Models can also differ in the
model evaluation.
extent of partitioning of the indoor airspace, with the simplest
5.1.2 Understanding the Model(s) to be Evaluated:
models treating the entire indoor volume as a single chamber or
5.1.2.1 Although a model user will not necessarily know or
zone assumed to have homogeneous concentrations through-
understand all details of a particular model, some fundamental
out; more complex models can treat the indoor volume as a
understanding of the underlying principles and concepts is
series of interconnected chambers, with a mass balance con-
important to the evaluation process. Thus, before evaluating a
ducted without each chamber and consideration given to
model, the user should develop some understanding of the
communicating airflows among chambers.
basis for the model and its operation. IAQ models can
5.1.2.7 Generally speaking, as the model complexity grows
generally be distinguished by their basis, by the range of
in terms of temporal detail, number of chambers, and types of
pollutants they can address, and by the extent of temporal or
parameters that can be used for calculations, the user’s task of
spatial detail they can accommodate in inputs, calculations, and
supplying appropriate inputs becomes increasingly demanding.
outputs.
Thus users must have a basic understanding of the underlying
5.1.2.2 Theoretical models are generally based on physical
principles, nature and extent of inputs required, inherent
principles such as mass conservation. (2,3) That is, a mass
limitations, and types of outputs provided so that they can
balance is maintained to keep track of material entering and
choose a level of model complexity providing an appropriate
leaving a particular airspace. Within this conceptual frame-
balance between input effort and output detail.
work, pollutant concentrations are increased by emissions
5.1.2.8 A number of assumptions are usually made when
within the defined volume and by transport from other air-
modeling a complex environment such as the indoor airspace.
spaces, including outdoors. Similarly, concentrations are de-
These assumptions, and their potential influence on the mod-
creased by transport exiting the airspace, by removal to
eling results, should be identified in the evaluation process.
chemical/physical sinks within the airspace, or for reactive
One method of gaining insights is by performing sensitivity
species, by conversion to other forms. Relationships are most
analysis. An example of this technique is to systematically vary
often specified through a differential equation quantifying
the values of one input parameter at a time to determine the
factors related to contaminant gain or loss.
effect of each on the modeling results; each parameter should
be varied over a reasonable range of values likely to be
5.1.2.3 Empirical models (3) are generally based on ap-
encountered for the specific situation(s) of interest.
proaches such as least-squares regression analysis, using mea-
surements under different conditions across a variety of struc- 5.1.3 Choosing Data Sets for Model Evaluation:
tures, at different times within the same structure, or both.
5.1.3.1 A fundamental requirement for model evaluation is
Theoretical models will generally be suitable for a wide range
that the data used for the evaluation process should be
of applications, whereas empirical models will generally be
independent of the data used to develop the model. This
applicable only within the range of measurements from which
constraint forces a search for available data pertinent to the
they were developed.
planned application or, if no appropriate data sets can be found,
5.1.2.4 Some combination of theoretical and empirical com- collection of new data to support the evaluation process. Such
data should be collected according to commonly recognized
ponents is also possible. Specific parameters of a theoretical
model may have relationships with other factors that can be and accepted methods, such as those given in the compendium
developed by the U.S. Environmental Protection Agency (4).
more easily quantified than the parameters themselves. For
example, the rate of air infiltration into a structure could
5.1.3.2 The following series of steps should be used in
depend on outdoor windspeed and the indoor-outdoor tempera- choosing data sets for model evaluation: (1) select situations
ture difference, or the emission rate from a cigarette could for applying and testing the model; (2) note the model input
depend on the combustion rate and the constituents of the parameters that require estimation for the situations selected;
particular brand smoked. Given sufficient data, such relation- (3) determine the required levels of temporal detail (for
NOTICE: This standard has either been superceded and replaced by a new version or discontinued.
Contact ASTM International (www.astm.org) for the latest information.
D 5157
¯ ¯
example, minute-by-minute or hour-by-hour) and spatial detail
2a 5 C 2 @~b!~C !# (3)
p o
(that is, number of chambers) for model application as well as
(3) Normalized mean square error (NMSE), a measure of
variations of the contaminants within each chamber; and (4)
the magnitude of prediction error relative to C and C . The
p o
find or collect appropriate data for estimation of the model 2
formula to be used for calculating this measure is as follows:
inputs and comparison with the model outputs.
¯ ¯
NMSE 5 ~C 2 C ! / @~C !~C !# (4)
p o o p
5.1.3.3 Thus, the information required for the evaluation
process includes not only measured indoor concentrations at an
where:
n
appropriate level of temporal detail, but also suitable estimates
2 2
~C 2 C ! 5 ~C 2 C ! /n.
(
for required input parameters. Among the inputs typically p o pi oi
i 5 1
required are outdoor concentrations, indoor emission and sink
The NMSE will have a value of 0 when there is perfect
rates, coagulation coefficients, deposition rates and diffusion
agreement for all pairs of C and C and will tend toward
p o
coefficients for particles, and rates of airflow between indoor
higher values as C and C differ by greater magnitudes. For
p o
and outdoor airspaces (as well as flows among multiple indoor
example, if C and C differ consistently by 50 %, the NMSE
p o
airspaces, if a multichamber model is used). If suitable data to
value will be near 0.2; for differences of 100 %, the NMSE
support the choice of inputs are not available, the alternatives
value will be near 0.5; for differences of one order of
are as follows: (1) to compress the level of temporal detail for
magnitude, the NMSE value will be near 8.0. In addition to
model application to that for which suitable data can be
these quantitative tools, a qualitative tool to be used is a plot of
obtained; (2) to provide best estimates for model inputs,
C and C over time. This plot will indicate not only the general
p o
recognizing the limitations imposed by this particular ap-
extent of agreement between C and C but also the specific
p o
proach; or (3) to collect the additional data required to enable
areas of disagreement. Model residuals can also be plotted over
proper estimation of inputs.
time or against predicted or observed concentrations (after
ordering the concentrations from lowest to highest); such a plot
5.1.4 Tools for Assessing Model Performance:
should indicate no distinct trend or pattern. If a trend or pattern
5.1.4.1 The tools to be used in assessing the performance of
is discerned, possible reasons for the trend should be identified
IAQ models all involve comparisons between indoor concen-
and investigated.
trations predicted by the model, C , and observed concentra-
p
5.1.4.3 The following tools are to be used for assessing bias:
tions, C , compri
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.