ASTM C1215-92(2012)e1
(Guide)Standard Guide for Preparing and Interpreting Precision and Bias Statements in Test Method Standards Used in the Nuclear Industry
Standard Guide for Preparing and Interpreting Precision and Bias Statements in Test Method Standards Used in the Nuclear Industry
SIGNIFICANCE AND USE
To describe the uncertainties of a standard test method, precision and bias statements are required. The formulation of these statements has been addressed from time to time, and at least two standards practices (Practices E177 and E691) have been issued. The 1986 Compilation of ASTM Standard Definitions (1) devotes several pages to these terms. This guide should not be used in cases where small numbers of test results do not support statistical normality.
ANSI N15.5 attempts to provide “a standard on statistical terminology and notation [that] can benefit communication” among nuclear materials managers. Precision, accuracy, and bias are all discussed. Although these various documents are quite valuable, a simpler document written for analysts appears needed. The intent of this guide is to help analysts prepare and interpret precision and bias statements. It is essential that, when the terms are used, their meaning should be clear and easily understood.
Appendix X1 provides the theoretical foundation for precision and bias concepts and Practice E691 addresses the problem of sources of variation. To illustrate the interplay between sources of variation and formulation of precision and bias statements, a hypothetical data set is analyzed in Appendix X2. This example shows that depending on how the data was collected, different precision and bias statements are possible. Reference to this example will be found throughout this guide.
There has been much debate inside and outside the statistical community on the exact meaning of some statistical terms. Thus, following a number of the terms in Section 3 is a list of several ways in which that term has been used. This listing is not meant to indicate that these meanings are equivalent or equally acceptable. The purpose here is more to encourage clear definition of terms used than to take sides. For example, use of the term systematic error is discouraged by some. If it is to be used, the reader should be told exac...
SCOPE
1.1 This guide covers terminology useful for the preparation and interpretation of precision and bias statements. This guide does not recommend a specific error model or statistical method. It provides awareness of terminology and approaches and options to use for precision and bias statements.
1.2 In formulating precision and bias statements, it is important to understand the statistical concepts involved and to identify the major sources of variation that affect results. Appendix X1 provides a brief summary of these concepts.
1.3 To illustrate the statistical concepts and to demonstrate some sources of variation, a hypothetical data set has been analyzed in Appendix X2. Reference to this example is made throughout this guide.
1.4 It is difficult and at times impossible to ship nuclear materials for interlaboratory testing. Thus, precision statements for test methods relating to nuclear materials will ordinarily reflect only within-laboratory variation.
1.5 No units are used in this statistical analysis.
1.6 This guide does not involve the use of materials, operations, or equipment and does not address any risk associated.
General Information
Relations
Standards Content (Sample)
NOTICE: This standard has either been superseded and replaced by a new version or withdrawn.
Contact ASTM International (www.astm.org) for the latest information
´1
Designation: C1215 − 92 (Reapproved 2012)
Standard Guide for
Preparing and Interpreting Precision and Bias Statements in
Test Method Standards Used in the Nuclear Industry
This standard is issued under the fixed designation C1215; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision.Anumber in parentheses indicates the year of last reapproval.A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
ε NOTE—Changes were made editorially in June 2012.
INTRODUCTION
Test method standards are required to contain precision and bias statements. This guide contains a
glossary that explains various terms that often appear in these statements as well as an example
illustrating such statements for a specific set of data. Precision and bias statements are shown to vary
according to the conditions under which the data were collected.This guide emphasizes that the error
model (an algebraic expression that describes how the various sources of variation affect the
measurement) is an important consideration in the formation of precision and bias statements.
1. Scope 2. Referenced Documents
2.1 ASTM Standards:
1.1 Thisguidecoversterminologyusefulforthepreparation
E177Practice for Use of the Terms Precision and Bias in
and interpretation of precision and bias statements. This guide
ASTM Test Methods
does not recommend a specific error model or statistical
E691Practice for Conducting an Interlaboratory Study to
method. It provides awareness of terminology and approaches
Determine the Precision of a Test Method
and options to use for precision and bias statements.
2.2 ANSI Standard:
1.2 In formulating precision and bias statements, it is
ANSI N15.5 Statistical Terminology and Notation for
importanttounderstandthestatisticalconceptsinvolvedandto
Nuclear Materials Management
identify the major sources of variation that affect results.
Appendix X1 provides a brief summary of these concepts.
3. Terminology for Precision and Bias Statements
1.3 To illustrate the statistical concepts and to demonstrate
3.1 Definitions:
some sources of variation, a hypothetical data set has been 3.1.1 accuracy (seebias) —(1) bias. (2) the closeness of a
analyzed in Appendix X2. Reference to this example is made
measured value to the true value. (3) the closeness of a
throughout this guide. measured value to an accepted reference or standard value.
3.1.1.1 Discussion—For many investigators, accuracy is
1.4 It is difficult and at times impossible to ship nuclear
attained only if a procedure is both precise and unbiased (see
materialsforinterlaboratorytesting.Thus,precisionstatements
bias). Because this blending of precision into accuracy can
for test methods relating to nuclear materials will ordinarily
resultoccasionallyinincorrectanalysesandunclearstatements
reflect only within-laboratory variation.
of results, ASTM requires statement on bias instead of accu-
1.5 No units are used in this statistical analysis. racy.
3.1.2 analysis of variance (ANOVA)—the body of statistical
1.6 This guide does not involve the use of materials,
theory,methods,andpracticesinwhichthevariationinasetof
operations, or equipment and does not address any risk
data is partitioned into identifiable sources of variation.
associated.
For referenced ASTM standards, visit the ASTM website, www.astm.org, or
This guide is under the jurisdiction ofASTM Committee C26 on Nuclear Fuel contact ASTM Customer Service at service@astm.org. For Annual Book of ASTM
Cycle and is the direct responsibility of Subcommittee C26.08 on Quality Standards volume information, refer to the standard’s Document Summary page on
Assurance, Statistical Applications, and Reference Materials. the ASTM website.
Current edition approved June 1, 2012. Published June 2012. Originally Available fromAmerican National Standards Institute (ANSI), 25 W. 43rd St.,
approvedin1992.Lastpreviouseditionapprovedin2006asC1215–92(2006).DOI: 4th Floor, New York, NY 10036, http://www.ansi.org.
10.1520/C1215-92R12E01. Refer to Form and Style for ASTM Standards, 8th Ed., 1989, ASTM.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
´1
C1215 − 92 (2012)
Sources of variation may include analysts, instruments, absence of sample size information detracts from the use-
samples, and laboratories. To use the analysis of variance, the fulness of the confidence interval. If the interval were based
data collection method must be carefully designed based on a on five observations, a second set of five might produce a
modelthatincludesallthesourcesofvariationofinterest.(See very different interval. This would not be the case if 50
Example, X2.1.1) observations were taken.
3.1.3 bias (see accuracy)—a constant positive or negative 3.1.6 confidence level—theprobability,usuallyexpressedas
deviation of the method average from the correct value or a percent, that a confidence interval will contain the parameter
accepted reference value. of interest. (See discussion of confidence interval in Appendix
X3.)
3.1.3.1 Discussion—Bias represents a constant error as op-
posed to a random error.
3.1.7 error model—an algebraic expression that describes
(a)A method bias can be estimated by the difference (or
how a measurement is affected by error and other sources of
relative difference) between a measured average and an ac-
variation. The model may or may not include a sampling error
cepted standard or reference value. The data from which the
term.
estimateisobtainedshouldbestatisticallyanalyzedtoestablish
3.1.7.1 Discussion—Ameasurement error is an error attrib-
bias in the presence of random error. A thorough bias investi-
utable to the measurement process. The error may affect the
gation of a measurement procedure requires a statistically
measurement in many ways and it is important to correctly
designed experiment to repeatedly measure, under essentially
model the effect of the error on the measurement.
thesameconditions,asetofstandardsorreferencematerialsof
(a) Two common models are the additive and the multi-
known value that cover the range of application. Bias often
plicative error models. In the additive model, the errors are
varies with the range of application and should be reported
independent of the value of the item being measured. Thus,
accordingly.
for example, for repeated measurements under identical
(b)In statistical terminology, an estimator is said to be
conditions, the additive error model might be
unbiased if its expected value is equal to the true value of the
X 5 µ1b1ε (1)
i i
parameter being estimated. (See Appendix X1.)
(c)Thebiasofatestmethodisalsocommonlyindicatedby
where:
analytical chemists as percent recovery. A number of repeti- th
X = the result of the i measurement,
i
tions of the test method on a reference material are performed,
µ = the true value of the item,
and an average percent recovery is calculated. This average
b = a bias, and
provides an estimate of the test method bias, which is multi-
ε = a random error usually assumed to have a normal
i
plicative in nature, not additive. (See Appendix X2.)
distribution with mean zero and variance σ .
(d)Use of a single test result to estimate bias is strongly
Inthemultiplicativemodel,theerrorisproportionaltothe
discouraged because, even if there were no bias, random error
true value.Amultiplicative error model for percent recovery
alone would produce a nonzero bias estimate.
(see bias) might be:
3.1.4 coeffıcient of variation—see relative standard devia-
X 5 µbε (2)
i i
tion.
and a multiplicative model for a neutron counter mea-
3.1.5 confidence interval—an interval used to bound the
value of a population parameter with a specified degree of surement might be:
confidence (this is an interval that has different values for
X 5 µ1µb1µ·ε (3)
i i
different random samples).
3.1.5.1 Discussion—When providing a confidence interval, 5µ 11b1ε
~ !
i
analysts should give the number of observations on which the
(b) Clearly, there are many ways in which errors may
interval is based. The specified degree of confidence is usually
affect a final measurement. The additive model is fre-
90, 95, or 99%. The form of a confidence interval depends on
quently assumed and is the basis for many common statis-
underlying assumptions and intentions. Usually, confidence
tical procedures. The form of the model influences how
intervals are taken to be symmetric, but that is not necessarily
the error components will be estimated and is very
so, as in the case of confidence intervals for variances.
important, for example, in the determination of measure-
Construction of a symmetric confidence interval for a popula-
ment uncertainties. Further discussion of models is given
tion mean is discussed in Appendix X3.
in the Example of Appendix X2 and in Appendix X4.
It is important to realize that a given confidence-interval
3.1.8 precision—a generic concept used to describe the
estimate either does or does not contain the population
dispersion of a set of measured values.
parameter. The degree of confidence is actually in the
procedure. For example, if the interval (9, 13) is a 90% 3.1.8.1 Discussion—It is important that some quantitative
confidence interval for the mean, we are confident that the measure be used to specify precision. A statement such as,
procedure(takeasample,constructaninterval)bywhichthe “The precision is 1.54 g” is useless. Measures frequently used
interval (9, 13) was constructed will 90% of the time to express precision are standard deviation, relative standard
produce an interval that does indeed contain the mean. deviation, variance, repeatability, reproducibility, confidence
Likewise,weareconfidentthat10%ofthetimetheinterval interval, and range. In addition to specifying the measure and
estimate obtained will not contain the mean. Note that the the precision, it is important that the number of repeated
´1
C1215 − 92 (2012)
measurementsuponwhichtheprecisionestimatedisbasedalso
1.96=2s, and the reproducibility limit is defined as 1.96=2s ,
r R
be given. (See Example, Appendix X2.)
where s is the estimated standard deviation associated with
r
(a) It is strongly recommended that a statement on
repeatability, and s is the estimated standard deviation asso-
R
precisionofameasurementprocedureincludethefollowing:
ciated with reproducibility.Thus, if normality can be assumed,
(1)A description of the procedure used to obtain the data,
these limits represent 95% limits for the difference between
(2)The number of repetitions, n, of the measurement twomeasurementstakenundertherespectiveconditions.Inthe
procedure,
reproducibility case, this means that “approximately 95% of
(3) The sample mean and standard deviation of the all pairs of test results from laboratories similar to those in the
measurements, study can be expected to differ in absolute value by less than
(4)The measure of precision being reported,
=
1.96 2s .” It is important to realize that if a particular s is a
R R
(5)The computed value of that measure, and
poor estimate of σ , the 95% figure may be substantially in
R
(6) The applicable range or concentration.
error. For this reason, estimates should be based on adequate
The importance of items (3) and (4) lies in the fact that
sample sizes.
with these a reader may calculate a confidence interval or
3.1.9 propagation of variance—a procedure by which the
relative standard deviation as desired.
mean and variance of a function of one or more random
(b) Precision is sometimes measured by repeatability and
variables can be expressed in terms of the mean, variance, and
reproducibility (see Practice E177, and Mandel and Laskof
covariances of the individual random variables themselves
(1)).TheANSI andASTM documents differ slightly in their
(Syn. variance propagation, propagation of error).
usages of these terms.The following is quoted from Kendall
3.1.9.1 Discussion—There are a number of simple exact
and Buckland (2):
formulas and Taylor series approximations which are useful
“In some situations, especially interlaboratory
here (3, 4).
comparisons, precision is defined by employing two addi-
3.1.10 random error—(1) the chance variation encountered
tional concepts: repeatability and reproducibility. The gen-
in all measurement work, characterized by the random occur-
eralsituationgivingrisetothesedistinctionscomesfromthe
rence of deviations from the mean value. (2) an error that
interest in assessing the variability within several groups of
affects each member of a set of data (measurements) in a
measurements and between those groups of measurements.
different manner.
Repeatability, then, refers to the within-group dispersion of
3.1.11 random sample (measurements)—a set of measure-
the measurements, while reproducibility refers to the
ments taken on a single item or on similar items in such a way
between-group dispersion. In interlaboratory comparison
that the measurements are independent and have the same
studies, for example, the investigation seeks to determine
probability distribution.
how well each laboratory can repeat its measurements
3.1.11.1 Discussion—Some authors refer to this as a simple
(repeatability)andhowwellthelaboratoriesagreewitheach
random sample. One must then be careful to distinguish
other (reproducibility). Similar discussions can apply to the
between a simple random sample from a finite population of N
comparison of laboratory technicians’ skills, the study of
items and a simple random sample from an infinite population.
competing types of equipment, and the use of particular
Intheformercase,asimplerandomsampleisasamplechosen
procedures within a laboratory. An essential feature usually
in such a way that all samples of the same size have the same
required, however, is that repeatability and reproducibility
chance of being selected.An example of the latter case occurs
be measured as variances (or standard deviations in certain
when taking measurements. Any value in an interval is
instances), so that both within- and between-group disper-
considered possible and thus the population is conceptually
sions are modeled as a random variable. The statistical tool
infinite. The definition given in 3.1.11 is then the appropriate
usefulfortheanalysisofsuchcomparisonsistheanalysisof
definition. (See representative sample and Appendix X5.)
variance.”
3.1.12 range—the largest minus the smallest of a set of
(c) In Practice E177 it is recommended that the term
numbers.
repeatabilitybereservedfortheintrinsicvariationduesolely
to the measurement procedure, excluding all variation from
3.1.13 relative standard deviation (percent)—the sample
factors
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.