ASTM D5157-97(2014)
(Guide)Standard Guide for Statistical Evaluation of Indoor Air Quality Models
Standard Guide for Statistical Evaluation of Indoor Air Quality Models
SIGNIFICANCE AND USE
4.1 Using the tools described in this guide, an individual seeking to apply an IAQ model should be able to (1) assess the performance of the model for a specific situation or (2) recognize or assess its advantages and limitations.
4.2 This guide can also be used for identifying specific areas of model deficiency that require further development or refinement.
SCOPE
1.1 This guide provides quantitative and qualitative tools for evaluation of indoor air quality (IAQ) models. These tools include methods for assessing overall model performance as well as identifying specific areas of deficiency. Guidance is also provided in choosing data sets for model evaluation and in applying and interpreting the evaluation tools. The focus of the guide is on end results (that is, the accuracy of indoor concentrations predicted by a model), rather than operational details such as the ease of model implementation or the time required for model calculations to be performed.
1.2 Although IAQ models have been used for some time, there is little guidance in the technical literature on the evaluation of such models. Evaluation principles and tools in this guide are drawn from past efforts related to outdoor air quality or meteorological models, which have objectives similar to those for IAQ models and a history of evaluation literature.(1)2 Some limited experience exists in the use of these tools for evaluation of IAQ models.
General Information
Relations
Buy Standard
Standards Content (Sample)
NOTICE: This standard has either been superseded and replaced by a new version or withdrawn.
Contact ASTM International (www.astm.org) for the latest information
Designation: D5157 − 97 (Reapproved 2014)
Standard Guide for
Statistical Evaluation of Indoor Air Quality Models
This standard is issued under the fixed designation D5157; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision.Anumber in parentheses indicates the year of last reapproval.A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope 3.2.2 model bias, n—asystematicdifferencebetweenmodel
predictions and measured indoor concentrations (for example,
1.1 Thisguideprovidesquantitativeandqualitativetoolsfor
the model prediction is generally higher than the measured
evaluation of indoor air quality (IAQ) models. These tools
concentration for a specific situation).
include methods for assessing overall model performance as
3.2.3 model chamber, n—an indoor airspace of defined
well as identifying specific areas of deficiency. Guidance is
volume used in model calculations; IAQ models can be
alsoprovidedinchoosingdatasetsformodelevaluationandin
specified for a single chamber or for multiple, interconnected
applyingandinterpretingtheevaluationtools.Thefocusofthe
chambers.
guide is on end results (that is, the accuracy of indoor
concentrations predicted by a model), rather than operational
3.2.4 model evaluation, n—aseriesofstepsthroughwhicha
details such as the ease of model implementation or the time
model developer or user assesses a model’s performance for
required for model calculations to be performed.
selected situations.
1.2 Although IAQ models have been used for some time,
3.2.5 model parameter, n—a mathematical term in an IAQ
there is little guidance in the technical literature on the
model that must be estimated by the model developer or user
evaluation of such models. Evaluation principles and tools in
before model calculations can be performed.
this guide are drawn from past efforts related to outdoor air
3.2.6 model residual, n—the difference between an indoor
quality or meteorological models, which have objectives simi-
concentration predicted by an IAQ model and a representative
lar to those for IAQ models and a history of evaluation
measurementofthetrueindoorconcentration;thevalueshould
literature.(1) Some limited experience exists in the use of
be stated as positive or negative.
these tools for evaluation of IAQ models.
3.2.7 model validation, n—a series of evaluations under-
taken by an agency or organization to provide a basis for
2. Referenced Documents
endorsing a specific model (or models) for a specific applica-
2.1 ASTM Standards:
tion (or applications).
D1356Terminology Relating to Sampling and Analysis of
3.2.8 pollutant concentration, n—the extent of the occur-
Atmospheres
rence of a pollutant or the parameters describing a pollutant in
3. Terminology a defined airspace, expressed in units characteristic to the
3 3 3
pollutant(forexample,mg/m ,ppm,Bq/m ,area/m ,orcolony
3.1 Definitions: For definitions of terms used in this
forming units per cubic metre).
standard, refer to Terminology D1356.
3.2 Definitions of Terms Specific to This Standard:
4. Significance and Use
3.2.1 IAQ model, n—an equation, algorithm, or series of
4.1 Using the tools described in this guide, an individual
equations/algorithmsusedtocalculateaverageortime-varying
seekingtoapplyanIAQmodelshouldbeableto(1)assessthe
pollutant concentrations in one or more indoor chambers for a
performance of the model for a specific situation or (2)
specific situation.
recognize or assess its advantages and limitations.
4.2 Thisguidecanalsobeusedforidentifyingspecificareas
This guide is under the jurisdiction of ASTM Committee D22 on Air Quality
ofmodeldeficiencythatrequirefurtherdevelopmentorrefine-
and is the direct responsibility of Subcommittee D22.05 on Indoor Air.
Current edition approved Sept. 1, 2014. Published September 2014. Originally
ment.
approved in 1991. Last previous edition approved in 2008 as D5157–97 (2008).
DOI: 10.1520/D5157-97R14.
5. Components of Model Evaluation
Theboldfacenumbersinparenthesesrefertothelistofreferencesattheendof
this standard.
5.1 The components of model evaluation include the fol-
For referenced ASTM standards, visit the ASTM website, www.astm.org, or
lowing: (1) stating the purpose(s) or objective(s) of the
contact ASTM Customer Service at service@astm.org. For Annual Book of ASTM
evaluation, (2) acquiring a basic understanding of the specifi-
Standards volume information, refer to the standard’s Document Summary page on
the ASTM website. cation and underlying principles or assumptions, (3) selecting
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
D5157 − 97 (2014)
data sets as inputs to the evaluation process, and (4) selecting example, the rate of air infiltration into a structure could
and using appropriate tools for assessing model performance. dependonoutdoorwindspeedandtheindoor-outdoortempera-
Just as model evaluation has multiple components, model ture difference, or the emission rate from a cigarette could
validation consists of one or more evaluations. However, depend on the combustion rate and the constituents of the
model validation is beyond the scope of this document. particular brand smoked. Given sufficient data, such relation-
shipscouldbeestimatedthroughtechniquessuchasregression
5.1.1 Establishing Evaluation Objectives:
analysis.
5.1.1.1 IAQmodelsaregenerallyusedforthefollowing:(1)
5.1.2.5 IAQ models may be specified for a particular pol-
to help explain the temporal and spatial variations in the
lutant or in general terms; this distinction is important, for
occurrences of indoor pollutant concentrations, (2) to improve
example, because particle-phase pollutants behave differently
the understanding of major influencing factors or underlying
from gas-phase pollutants. Particulate matter is subject to
physical/chemical processes, and (3) to predict the temporal/
coagulation, chemical reaction at surfaces, gravitational
spatialvariationsinindoorconcentrationsthatcanbeexpected
settling, diffusional deposition, resuspension and interception,
to occur in specific types of situations. However, model
impaction, and diffusional removal by filtration devices;
evaluation relates only to the third type of model use—
whereassomegaseouspollutantsaresubjecttosorptionand,in
prediction of indoor concentrations.
some cases, desorption processes.
5.1.1.2 The most common evaluation objectives are (1)to
5.1.2.6 Dynamic IAQ models predict time-varying indoor
compare the performance of two or more models for a specific
concentrations for time steps that are usually on the order of
situation or set of situations and (2) to assess the performance
seconds, minutes, or hours; whereas integrated models predict
of a specific model for different situations. Secondary objec-
time-averaged indoor concentrations using average values for
tives include identifying specific areas of model deficiency.
each input parameter or averaging these parameters during the
Determination of specific objectives will assist in choosing
course of exercising the model. Models can also differ in the
appropriate data sets and quantitative or qualitative tools for
extent of partitioning of the indoor airspace, with the simplest
model evaluation.
modelstreatingtheentireindoorvolumeasasinglechamberor
5.1.2 Understanding the Model(s) to be Evaluated:
zone assumed to have homogeneous concentrations through-
5.1.2.1 Although a model user will not necessarily know or
out; more complex models can treat the indoor volume as a
understand all details of a particular model, some fundamental
series of interconnected chambers, with a mass balance con-
understanding of the underlying principles and concepts is
ducted without each chamber and consideration given to
important to the evaluation process. Thus, before evaluating a
communicating airflows among chambers.
model, the user should develop some understanding of the
5.1.2.7 Generally speaking, as the model complexity grows
basis for the model and its operation. IAQ models can
in terms of temporal detail, number of chambers, and types of
generally be distinguished by their basis, by the range of
parameters that can be used for calculations, the user’s task of
pollutants they can address, and by the extent of temporal or
supplyingappropriateinputsbecomesincreasinglydemanding.
spatialdetailtheycanaccommodateininputs,calculations,and
Thus users must have a basic understanding of the underlying
outputs.
principles, nature and extent of inputs required, inherent
5.1.2.2 Theoretical models are generally based on physical
limitations, and types of outputs provided so that they can
principles such as mass conservation. (2, 3) That is, a mass
choose a level of model complexity providing an appropriate
balance is maintained to keep track of material entering and
balance between input effort and output detail.
leaving a particular airspace. Within this conceptual
5.1.2.8 A number of assumptions are usually made when
framework, pollutant concentrations are increased by emis-
modeling a complex environment such as the indoor airspace.
sions within the defined volume and by transport from other
These assumptions, and their potential influence on the mod-
airspaces, including outdoors. Similarly, concentrations are
eling results, should be identified in the evaluation process.
decreased by transport exiting the airspace, by removal to
One method of gaining insights is by performing sensitivity
chemical/physical sinks within the airspace, or for reactive
analysis.Anexampleofthistechniqueistosystematicallyvary
species, by conversion to other forms. Relationships are most
the values of one input parameter at a time to determine the
often specified through a differential equation quantifying
effect of each on the modeling results; each parameter should
factors related to contaminant gain or loss.
be varied over a reasonable range of values likely to be
5.1.2.3 Empirical models (3) are generally based on ap-
encountered for the specific situation(s) of interest.
proaches such as least-squares regression analysis, using mea-
surements under different conditions across a variety of 5.1.3 Choosing Data Sets for Model Evaluation:
structures,atdifferenttimeswithinthesamestructure,orboth.
5.1.3.1 A fundamental requirement for model evaluation is
Theoretical models will generally be suitable for a wide range
that the data used for the evaluation process should be
of applications, whereas empirical models will generally be
independent of the data used to develop the model. This
applicable only within the range of measurements from which
constraint forces a search for available data pertinent to the
they were developed.
plannedapplicationor,ifnoappropriatedatasetsc
...
This document is not an ASTM standard and is intended only to provide the user of an ASTM standard an indication of what changes have been made to the previous version. Because
it may not be technically possible to adequately depict all changes accurately, ASTM recommends that users consult prior editions as appropriate. In all cases only the current version
of the standard as published by ASTM is to be considered the official document.
Designation: D5157 − 97 (Reapproved 2008) D5157 − 97 (Reapproved 2014)
Standard Guide for
Statistical Evaluation of Indoor Air Quality Models
This standard is issued under the fixed designation D5157; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope
1.1 This guide provides quantitative and qualitative tools for evaluation of indoor air quality (IAQ) models. These tools include
methods for assessing overall model performance as well as identifying specific areas of deficiency. Guidance is also provided in
choosing data sets for model evaluation and in applying and interpreting the evaluation tools. The focus of the guide is on end
results (that is, the accuracy of indoor concentrations predicted by a model), rather than operational details such as the ease of
model implementation or the time required for model calculations to be performed.
1.2 Although IAQ models have been used for some time, there is little guidance in the technical literature on the evaluation of
such models. Evaluation principles and tools in this guide are drawn from past efforts related to outdoor air quality or
meteorological models, which have objectives similar to those for IAQ models and a history of evaluation literature.(1) Some
limited experience exists in the use of these tools for evaluation of IAQ models.
2. Referenced Documents
2.1 ASTM Standards:
D1356 Terminology Relating to Sampling and Analysis of Atmospheres
3. Terminology
3.1 Definitions: For definitions of terms used in this standard, refer to Terminology D1356.
3.2 Definitions of Terms Specific to This Standard:
3.2.1 IAQ model, n—an equation, algorithm, or series of equations/algorithms used to calculate average or time-varying
pollutant concentrations in one or more indoor chambers for a specific situation.
3.2.2 model bias, n—a systematic difference between model predictions and measured indoor concentrations (for example, the
model prediction is generally higher than the measured concentration for a specific situation).
3.2.3 model chamber, n—an indoor airspace of defined volume used in model calculations; IAQ models can be specified for a
single chamber or for multiple, interconnected chambers.
3.2.4 model evaluation, n—a series of steps through which a model developer or user assesses a model’s performance for
selected situations.
3.2.5 model parameter, n—a mathematical term in an IAQ model that must be estimated by the model developer or user before
model calculations can be performed.
3.2.6 model residual, n—the difference between an indoor concentration predicted by an IAQ model and a representative
measurement of the true indoor concentration; the value should be stated as positive or negative.
3.2.7 model validation, n—a series of evaluations undertaken by an agency or organization to provide a basis for endorsing a
specific model (or models) for a specific application (or applications).
3.2.8 pollutant concentration, n—the extent of the occurrence of a pollutant or the parameters describing a pollutant in a defined
3 3 3
airspace, expressed in units characteristic to the pollutant (for example, mg/m , ppm, Bq/m , area/m , or colony forming units per
cubic metre).
This guide is under the jurisdiction of ASTM Committee D22 on Air Quality and is the direct responsibility of Subcommittee D22.05 on Indoor Air.
Current edition approved April 1, 2008Sept. 1, 2014. Published July 2008 September 2014. Originally approved in 1991. Last previous edition approved in 20032008 as
ε1
D5157 – 97 (2008).(2003) . DOI: 10.1520/D5157-97R08.10.1520/D5157-97R14.
The boldface numbers in parentheses refer to the list of references at the end of this standard.
For referenced ASTM standards, visit the ASTM website, www.astm.org, or contact ASTM Customer Service at service@astm.org. For Annual Book of ASTM Standards
volume information, refer to the standard’s Document Summary page on the ASTM website.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
D5157 − 97 (2014)
4. Significance and Use
4.1 Using the tools described in this guide, an individual seeking to apply an IAQ model should be able to (1) assess the
performance of the model for a specific situation or (2) recognize or assess its advantages and limitations.
4.2 This guide can also be used for identifying specific areas of model deficiency that require further development or refinement.
5. Components of Model Evaluation
5.1 The components of model evaluation include the following: (1) stating the purpose(s) or objective(s) of the evaluation, (2)
acquiring a basic understanding of the specification and underlying principles or assumptions, (3) selecting data sets as inputs to
the evaluation process, and (4) selecting and using appropriate tools for assessing model performance. Just as model evaluation
has multiple components, model validation consists of one or more evaluations. However, model validation is beyond the scope
of this document.
5.1.1 Establishing Evaluation Objectives:
5.1.1.1 IAQ models are generally used for the following: (1) to help explain the temporal and spatial variations in the
occurrences of indoor pollutant concentrations, (2) to improve the understanding of major influencing factors or underlying
physical/chemical processes, and (3) to predict the temporal/spatial variations in indoor concentrations that can be expected to
occur in specific types of situations. However, model evaluation relates only to the third type of model use—prediction of indoor
concentrations.
5.1.1.2 The most common evaluation objectives are (1) to compare the performance of two or more models for a specific
situation or set of situations and (2) to assess the performance of a specific model for different situations. Secondary objectives
include identifying specific areas of model deficiency. Determination of specific objectives will assist in choosing appropriate data
sets and quantitative or qualitative tools for model evaluation.
5.1.2 Understanding the Model(s) to be Evaluated:
5.1.2.1 Although a model user will not necessarily know or understand all details of a particular model, some fundamental
understanding of the underlying principles and concepts is important to the evaluation process. Thus, before evaluating a model,
the user should develop some understanding of the basis for the model and its operation. IAQ models can generally be
distinguished by their basis, by the range of pollutants they can address, and by the extent of temporal or spatial detail they can
accommodate in inputs, calculations, and outputs.
5.1.2.2 Theoretical models are generally based on physical principles such as mass conservation. (2, 3) That is, a mass balance
is maintained to keep track of material entering and leaving a particular airspace. Within this conceptual framework, pollutant
concentrations are increased by emissions within the defined volume and by transport from other airspaces, including outdoors.
Similarly, concentrations are decreased by transport exiting the airspace, by removal to chemical/physical sinks within the airspace,
or for reactive species, by conversion to other forms. Relationships are most often specified through a differential equation
quantifying factors related to contaminant gain or loss.
5.1.2.3 Empirical models (3) are generally based on approaches such as least-squares regression analysis, using measurements
under different conditions across a variety of structures, at different times within the same structure, or both. Theoretical models
will generally be suitable for a wide range of applications, whereas empirical models will generally be applicable only within the
range of measurements from which they were developed.
5.1.2.4 Some combination of theoretical and empirical components is also possible. Specific parameters of a theoretical model
may have relationships with other factors that can be more easily quantified than the parameters themselves. For example, the rate
of air infiltration into a structure could depend on outdoor windspeed and the indoor-outdoor temperature difference, or the
emission rate from a cigarette could depend on the combustion rate and the constituents of the particular brand smoked. Given
sufficient data, such relationships could be estimated through techniques such as regression analysis.
5.1.2.5 IAQ models may be specified for a particular pollutant or in general terms; this distinction is important, for example,
because particle-phase pollutants behave differently from gas-phase pollutants. Particulate matter is subject to coagulation,
chemical reaction at surfaces, gravitational settling, diffusional deposition, resuspension and interception, impaction, and
diffusional removal by filtration devices; whereas some gaseous pollutants are subject to sorption and, in some cases, desorption
processes.
5.1.2.6 Dynamic IAQ models predict time-varying indoor concentrations for time steps that are usually on the order of seconds,
minutes, or hours; whereas integrated models predict time-averaged indoor concentrations using average values for each input
parameter or averaging these parameters during the course of exercising the model. Models can also differ in the extent of
partitioning of the indoor airspace, with the simplest models treating the entire indoor volume as a single chamber or zone assumed
to have homogeneous concentrations throughout; more complex models can treat the indoor volume as a series of interconnected
chambers, with a mass balance conducted without each chamber and consideration given to communicating airflows among
chambers.
5.1.2.7 Generally speaking, as the model complexity grows in terms of temporal detail, number of chambers, and types of
parameters that can be used for calculations, the user’s task of supplying appropriate inputs becomes increasingly demanding. Thus
D5157 − 97 (2014)
users must have a basic understanding of the underlying principles, nature and extent of inputs required, inherent limitations, and
types of outputs provided so that they can choose a level of model complexity providing an appropriate balance between input
effort and output detail.
5.1.2.8 A number of assumptions are usually made when modeling a complex environment such as the indoor airspace. These
assumptions, and their potential influence on the modeling results, should be identified in the evaluation process. One method of
gaining insights is by performing sensitivity analysis. An example of this technique is to systematically vary the values of one input
parameter at a time to determine the effect of each on the modeling results; each parameter should be varied over a reasonable range
of values likely to be encountered for the specific situation(s) of interest.
5.1.3 Choosing Data Sets for Model Evaluation:
5.1.3.1 A fundamental requirement for model evaluation is that the data used for the evaluation process should be independent
of the data used to develop the model. This constraint forces a search for available data pertinent to the planned application or,
if no appropriate data sets can be found, collection of new data to support the evaluation process. Such data should be collected
according to commonly recognized and accepted methods, such as those given in the compendium developed by the U.S.
Environmental Protection Agency (4).
5.1.3.2 The following series of steps should be used in choosing data sets for model evaluation: (1) select situations for applying
and testing the model; (2) note the model input parameters that require estimation for the situations selected; (3) determine the
required levels of temporal detail (for example, minute-by-minute or hour-by-hour) and spatial detail (that is, number of chambers)
for model application as well as variations of the contaminants within each chamber; and (4) find or collect appropriate data for
estimation of the model inputs and comparison with the model outputs.
5.1.3.3 Thus, the information required for the evaluation process includes not only measured indoor concentrations at an
appropriate level of temporal detail, but also suitable estimates for required input parameters. Among the inputs typically required
are outdoor concentrations, indoor emission and sink rates, coagulation coefficients, deposition rates and diffusion coefficients for
particles, and rates of airflow between indoor and outdoor airspaces (as well as flows among multiple indoor airspaces, if a
multichamber model is used). If suitable data to support the choice of inputs are not available, the alternatives are as follows: (1)
to compress the level of temporal detail for model application to that for which suitable data can be obtained; (2) to provide best
estimates for model inputs, recognizing the limitations imposed by this particular approach; or (3) to collect the additional data
required to enable proper estimation of inputs.
5.1.4 Tools for Assessing Model Performance:
5.1.4.1 The tools to be used in assessing the performance of IAQ models all involve comparisons between indoor concentrations
predicted by the model, C , and observed concentrations, C , comprising the data set(s) used for evaluation. These tools can be
p o
quantitative, involving various types of statistical indexes, or qualitative, involving plots of C ,C , or differences between the two
p o
(that is, model residuals). The tools presented below are classified by use for (1) assessing the general agreement between predicted
and observed concentrations and (2) assessing bias in the mean or variance of predicted values relative to that for observed values.
5.1.4.2 The following tools are to be used for assessing the general agreement between C and C :
p o
(1) Correlation coefficient, r, ranging from −1 to 1, with 1 indicating a strong, direct relationship between C and C , 0
p o
indicating no relationship, and − 1 indicating a strong but inverse relationship. The formula to be used for calculating this
coefficient (5, 6) is as follows:
n
¯ ¯
r 5 @~C 2 C !~C 2 C !#/ (1)
( oi o pi p
i51
n n
2 2
¯ ¯
Π@~C 2 C ! # ~C 2 C !
F G
( oi o ( pi p
i51 i51
where the summation extends across all C and C pairs and
p o
n
¯
C¯ and C¯ are average
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.