ASTM E141-10(2023)
(Practice)Standard Practice for Acceptance of Evidence Based on the Results of Probability Sampling
Standard Practice for Acceptance of Evidence Based on the Results of Probability Sampling
ABSTRACT
This practice presents rules for accepting or rejecting evidence based on a sample. Statistical evidence for this practice is in the form of an estimate of a proportion, an average, a total, or other numerical characteristic of a finite population or lot. This practice is an estimate of the result which would have been obtained by investigating the entire lot or population under the same rules and with the same care as was used for the sample. One purpose of this practice is to describe straightforward sample selection and data calculation procedures so that courts, commissions, etc. will be able to verify whether such procedures have been applied.
SIGNIFICANCE AND USE
4.1 This practice is designed to permit users of sample survey data to judge the trustworthiness of results from such surveys. Practice E105 provides a statement of principles for guidance of ASTM technical committees and others in the preparation of a sampling plan for a specific material. Guide E1402 describes the principal types of sampling designs. Practice E122 aids in deciding on the required sample size.
4.2 Section 5 gives extended definitions of the concepts basic to survey sampling and the user should verify that such concepts were indeed used and understood by those who conducted the survey. What was the frame? How large (exactly) was the quantity N? How was the parameter θ estimated and its standard error calculated? If replicate subsamples were not used, why not? Adequate answers should be given for all questions. There are many acceptable answers to the last question.
4.3 If the sample design was relatively simple, such as simple random or stratified, then fully valid estimates of sampling variance are easily available. If a more complex design was used then methods such as discussed in Ref (1)3 or in Guide E1402 may be acceptable. Use of replicate subsamples is the most straightforward way to estimate sampling variances when the survey design is complex.
4.4 Once the survey procedures that were used satisfy Section 5, see if any increase in sample size is needed. The calculations for making it objectively are described in Section 6.
4.5 Refer to Section 7 to guide in the interpretation of the uncertainty in the reported value of the parameter estimate, θ^, that is, the value of its standard error, se(θ^). The quantity se(θ^) should be reviewed to verify that the risks it entails are commensurate with the size of the sample.
4.6 When the audit subsample shows that there was reasonable conformity with prescribed procedures and when the known instances of departures from the survey plan can be shown to have no appreciable effect on the est...
SCOPE
1.1 This practice presents rules for accepting or rejecting evidence based on a sample. Statistical evidence for this practice is in the form of an estimate of a proportion, an average, a total, or other numerical characteristic of a finite population or lot. It is an estimate of the result which would have been obtained by investigating the entire lot or population under the same rules and with the same care as was used for the sample.
1.2 One purpose of this practice is to describe straightforward sample selection and data calculation procedures so that courts, commissions, etc. will be able to verify whether such procedures have been applied. The methods may not give least uncertainty at least cost, they should however furnish a reasonable estimate with calculable uncertainty.
1.3 This practice is primarily intended for one-of-a-kind studies. Repetitive surveys allow estimates of sampling uncertainties to be pooled; the emphasis of this practice is on estimation of sampling uncertainty from the sample itself. The parameter of interest for this practice is effectively a constant. Thus, the principal inference is a simple point estimate to be used as if it were the unknown constant, rather than, for example, a forecast or prediction interval or distribution...
General Information
Relations
Standards Content (Sample)
This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
Designation: E141 − 10 (Reapproved 2023) An American National Standard
Standard Practice for
Acceptance of Evidence Based on the Results of Probability
Sampling
This standard is issued under the fixed designation E141; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope mendations issued by the World Trade Organization Technical
Barriers to Trade (TBT) Committee.
1.1 This practice presents rules for accepting or rejecting
evidence based on a sample. Statistical evidence for this
2. Referenced Documents
practice is in the form of an estimate of a proportion, an
2.1 ASTM Standards:
average, a total, or other numerical characteristic of a finite
E105 Guide for Probability Sampling of Materials
population or lot. It is an estimate of the result which would
E122 Practice for Calculating Sample Size to Estimate, With
have been obtained by investigating the entire lot or population
Specified Precision, the Average for a Characteristic of a
under the same rules and with the same care as was used for the
Lot or Process
sample.
E456 Terminology Relating to Quality and Statistics
1.2 One purpose of this practice is to describe straightfor-
E1402 Guide for Sampling Design
ward sample selection and data calculation procedures so that
E2586 Practice for Calculating and Using Basic Statistics
courts, commissions, etc. will be able to verify whether such
procedures have been applied. The methods may not give least
3. Terminology
uncertainty at least cost, they should however furnish a
3.1 Definitions—Refer to Terminology E456 for definitions
reasonable estimate with calculable uncertainty.
of other statistical terms used in this practice.
1.3 This practice is primarily intended for one-of-a-kind
3.1.1 audit subsample, n—a small subsample of a sample
studies. Repetitive surveys allow estimates of sampling uncer-
selected for review of all sample selection and data collection
tainties to be pooled; the emphasis of this practice is on
procedures.
estimation of sampling uncertainty from the sample itself. The
3.1.2 equal complete coverage result, n—the numerical
parameter of interest for this practice is effectively a constant.
characteristic of interest calculated from observations made by
Thus, the principal inference is a simple point estimate to be
drawing randomly from the frame, all of the sampling units
used as if it were the unknown constant, rather than, for
covered by the frame.
example, a forecast or prediction interval or distribution
3.1.2.1 Discussion—Locating the units and evaluating them
devised to match a random quantity of interest.
are supposed to be done in exactly the same way and at the
1.4 A system of units is not specified in this standard.
same time as was done for the sample. The quantity itself is
denoted θ. The equal complete coverage result is never actually
1.5 This standard does not purport to address all of the
calculated. Its purpose is to serve as the objectively defined
safety concerns, if any, associated with its use. It is the
concrete goal of the investigation. The quantity θ may be the
responsibility of the user of this standard to establish appro-
¯
population mean, (Y), total (Y), median (M), the proportion (P),
priate safety, health, and environmental practices and deter-
or any other such quantity.
mine the applicability of regulatory limitations prior to use.
1.6 This international standard was developed in accor-
3.1.3 frame, n—a list, compiled for sampling purposes,
dance with internationally recognized principles on standard-
which designates all of the sampling units (items or groups) of
ization established in the Decision on Principles for the
a population or universe to be considered in a specific study.
Development of International Standards, Guides and Recom-
E1402
3.1.4 probability sample, n—a sample in which the sam-
pling units are selected by a chance process such that a
This practice is under the jurisdiction of ASTM Committee E11 on Quality and
Statistics and is the direct responsibility of Subcommittee E11.10 on Sampling /
Statistics. For referenced ASTM standards, visit the ASTM website, www.astm.org, or
Current edition approved Nov. 1, 2023. Published November 2023. Originally contact ASTM Customer Service at service@astm.org. For Annual Book of ASTM
approved in 1959. Last previous edition approved in 2018 as E141 – 10 (2018). Standards volume information, refer to the standard’s Document Summary page on
DOI: 10.1520/E0141-10R23. the ASTM website.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E141 − 10 (2023)
specified probability of selection can be attached to each 5. Concepts and Procedures of Sampling
possible sample that can be selected. E1402
5.1 Probability sampling is a procedure by which one
3.1.5 replicate subsamples, n—a number of disjoint
obtains a result from a selected set of sampling units that will
samples, each one separately drawn from the frame in accord
agree, within calculable limits of variation, with the equal
with the same probability sampling plan.
complete coverage result. Probability sampling plans include
instructions for using either (1) prepared tables of random
3.1.6 sample, n—a group of observations or test results,
numbers, (2) computer algorithms to generate pseudo-random
taken from a larger collection of observations or test results,
numbers, or (3) certifiably honest physical devices to select the
which serves to provide information that may be used as a basis
sample units so that inferences may be drawn from the test
for making a decision concerning the larger collection. E2586
results and decisions may be made with risks correctly calcu-
3.1.7 sampling unit, n—an item, group of items, or segment
lated by probability theory.
of material that can be selected as part of a probability
5.1.1 Such plans are defined and their relative advantages
sampling plan. E1402
discussed in Guide E1402 and Refs (1-3).
5.2 Procedures must be described in written form. Parties
4. Significance and Use
interested in collecting data should agree on the importance of
4.1 This practice is designed to permit users of sample
knowing θ and its definition including measurement methods.
survey data to judge the trustworthiness of results from such
The frame shall be carefully and explicitly constructed. Every
surveys. Practice E105 provides a statement of principles for
sampling unit in the frame (1) has a unique serial number,
guidance of ASTM technical committees and others in the
which may be preassigned or determined by some definite rule
preparation of a sampling plan for a specific material. Guide
and (2) has an address—a complete and clear instruction (or
E1402 describes the principal types of sampling designs.
rules for its formulation) as to where and when to make the
Practice E122 aids in deciding on the required sample size.
observation or evaluation. Address instructions should refer to
4.2 Section 5 gives extended definitions of the concepts concrete clerical materials such as directories, dials of clocks
basic to survey sampling and the user should verify that such or of meters, ledgers, maps, aerial photographs, etc. Duplicates
concepts were indeed used and understood by those who
in the frame shall be eliminated. N shall be well established.
conducted the survey. What was the frame? How large (ex- Random numbers (or a certifiably honest physical random
actly) was the quantity N? How was the parameter θ estimated
device) shall dictate selection of the sample. There shall be no
and its standard error calculated? If replicate subsamples were substitution of one sampling unit for another. The method of
not used, why not? Adequate answers should be given for all
sample selection shall permit calculation of a standard error of
questions. There are many acceptable answers to the last the estimate. The use of replicate subsamples is recommended
question.
(see 5.4). An audit subsample should be selected and processed
and any departures from prescribed measurement methods and
4.3 If the sample design was relatively simple, such as
ˆ
location instructions noted (see 5.5). A report should list θ and
simple random or stratified, then fully valid estimates of
ˆ
its standard error with the degrees of freedom in the se(θ).
sampling variance are easily available. If a more complex
design was used then methods such as discussed in Ref (1) or
5.3 Parameter Definition—The equal complete coverage
in Guide E1402 may be acceptable. Use of replicate sub- result may or may not be acceptable evidence. Whether it is
samples is the most straightforward way to estimate sampling acceptable depends on many considerations such as definitions,
variances when the survey design is complex. method of test, care exercised in the testing, completeness of
the frame, and on other points not to be settled by statistical
4.4 Once the survey procedures that were used satisfy
theory since these points belong to the subject matter, and are
Section 5, see if any increase in sample size is needed. The
the same whether one uses sampling or not. Mistakes, whether
calculations for making it objectively are described in Section
in testing, counting, or weighing will affect the result of a
6.
complete coverage just as such mistakes will affect the sample
4.5 Refer to Section 7 to guide in the interpretation of the
result. By a more expensive method of measurement or more
ˆ
uncertainty in the reported value of the parameter estimate, θ,
elaborate sampling frame, it may be possible to define a
ˆ ˆ
that is, the value of its standard error, se(θ). The quantity se(θ)
quantity, θ', as a target parameter or ideal goal of an investi-
should be reviewed to verify that the risks it entails are
gation. Criticism that holds θ to be an inappropriate goal should
commensurate with the size of the sample.
demonstrate that the numerical difference between θ and θ' is
substantial. Measurements may be imprecise but so long as
4.6 When the audit subsample shows that there was reason-
measurement errors are not too biased, a large size of the lot or
able conformity with prescribed procedures and when the
population, N, insures that θ and θ' are essentially equal.
known instances of departures from the survey plan can be
shown to have no appreciable effect on the estimate, the value
5.4 Replicate Subsamples—When appropriate, separate
ˆ
of θ is appropriate for use.
laboratories should each work on separate replicate subsamples
and teams of investigators should be assigned to separate
replicate subsamples. This approach insures that the calculated
standard error will not be a systematic underestimate. Such
The boldface numbers in parentheses refer to a list of references at the end of
this standard. subsamples were called interpenetrating in Ref (4) where many
E141 − 10 (2023)
of their basic properties were described. See Ref (5) for further There are n − 1 degrees of freedom in this standard error.
theory and applications.
5.7.1.1 Example—When the observations are:
81.6, 78.7, 79.7, 78.3, 80.9, 79.5, 79.8, 80.3, 79.5, 80.7
5.4.1 For some types of material, a sample selected with
uniform spacing along the frame (systematic sample) has then y¯ = 79.90 and se(y¯) = 0.32.
increased precision over a selection made with randomly 5.7.2 Finite Population Correction (fpc)—Multiplying se(y¯)
varying spacings (simple random sample). Examples include
by =12n/N is always correct when the goal of the survey is to
sampling mineral ore or grain from a conveyor belt or sampling ¯
estimate the finite population mean (θ = Y). If random mea-
from a list of households along a street. If the systematic
surement error exists in the observations, then θ' based on a
sample is obtained by a single random start the plan is then a
reference measurement method may be a more appropriate
probability sampling plan, but it does not permit calculating the
survey goal than θ (see 5.3). If so, then se(y¯) would be further
standard error as required by this practice. After dividing the
adjusted upward by an amount somewhat less than the down-
sample size by an integer k (such as k = 4 or k = 10) and using
ward adjustment of the fpc. Both of these adjustments are often
a random start for each of k replicate subsamples, some of the
numerically so small that these adjustments may be omitted—
increased precision of systematic sampling (and a standard
leaving se(y¯) of Eq 2 as a slight overestimate.
error on k − 1 degrees of freedom) can be achieved.
5.7.2.1 Example—Using the previous data and if N = 50,
5.5 An audit subsample of the survey sample should be
then se(y¯) becomes se(y¯) = 0.28 after applying the fpc.
taken for review of all procedures from use of the random
5.7.3 Proportions and Total Counts—If the quantity of
numbers through locating and measurement, to editing, coding,
interest is (a) a proportion or (b) a total and the sample is
data entry and tabulation. Selection of the audit subsample may
simple random then the above formulas are still applicable. A
be done by putting the n sample observations in order as they
proportion is the mean of zeroes and ones, while the total is a
are collected, calculating the nearest integer to =n, or some
constant times the mean.
other convenient integer, and taking this number to be the
5.7.3.1 When θ is taken to be the population proportion
spacing for systematic selection of the audit subsample. As few
(θ = P) then
as ten observations may be adequate. The review should
ˆ
θ 5 p 5 y /n 5 a/n (3)
i
uncover any gross departures from prescribed practices or any (
conceptual misunderstandings in the definitions. If the audit
where a is the number of units in the sample with the
subsample is large enough (say 30 observations or more) the
attribute, and
regression of audited values on initial observations may be
se p 5 =p 1 2 p / n 2 1 (4)
used to calibrate the estimate. This technique is the method of ~ ! ~ ! ~ !
two-phase sampling as discussed in Ref (1). Helpful discussion
5.7.3.2 When θ = the population total (θ = Y) then
of an audit appears in Ref (2).
ˆ ˆ
θ 5 Np and se~θ! 5 N·se p (5)
~ !
5.6 The estimate is a quantity calculated on the n sample
observations in the same way as the equal complete coverage
5.7.3.3 Example—If a simple random sample of size
result θ would h
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.