ASTM E1488-23

Standard Guide for Statistical Procedures to Use in Developing and Applying Test Methods

ABSTRACT
This guide identifies statistical procedures for use in developing new test methods or revising or evaluating existing test methods, or both. It also cites statistical procedures especially useful in the application of test methods. This standard recommends what approaches may be taken and indicates which standards may be used to perform such assessments.
SIGNIFICANCE AND USE
4.1 The creation of a standardized test method generally follows a series of steps from inception to approval and ongoing use. In all such stages there are questions of how well the test method performs.  
4.1.1 Assessments of a new or existing test method generally involve statistical planning and analysis. This standard recommends what approaches may be taken and indicates which standards may be used to perform such assessments.  
4.2 This standard introduces a series of phases which are recommended to be considered during the life cycle of a test method as depicted in Fig. 1. These begin with a design phase where the standard is initially prepared. A development phase involves a variety of experiments that allow further refinement and understanding of how the test method performs within a laboratory. In an evaluation phase the test method is then examined by way of interlaboratory studies resulting in precision and bias statistics which are published in the standard. Finally, the test method is subject to a monitoring phase.
FIG. 1 Sequence of Steps  
4.3 All ASTM test methods are required to include statements on precision and bias.3  
4.4 Since ASTM began to require all test methods to have precision and bias statements that are based on interlaboratory studies, there has been increased concern regarding what statistical experiments and procedures to use during the development of the test methods. Although there exists a wide range of statistical procedures, there is a small group of generally accepted techniques that are beneficial to follow. This guide is designed to provide a brief overview of these procedures and to suggest an appropriate sequence of conducting these procedures.  
4.5 Statistical procedures often result in interpretations that are not absolutes. Sometimes the information obtained may be inadequate or incomplete, which may lead to additional questions and the need for further experimentation. Information outside the data is also impo...
SCOPE
1.1 This guide identifies statistical procedures for use in developing new test methods or revising or evaluating existing test methods, or both.  
1.2 This guide also cites statistical procedures especially useful in the application of test methods.  
1.3 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.

General Information

• Status: Published
• Publication Date: 31-Oct-2023
• ICS: 07.020 - Mathematics
• ICS: 19.020 - Test conditions and procedures in general
• Technical Committee: E11 - Quality and Statistics
• Drafting Committee: E11.20 - Test Method Evaluation and Quality Control

Relations

• Replaces: ASTM E1488-12(2023) - Standard Guide for Statistical Procedures to Use in Developing and Applying Test Methods
• Effective Date: 01-Nov-2023
• Refers: ASTM E2282-23 - Standard Guide for Defining the Test Result of a Test Method
• Effective Date: 01-Nov-2023
• Refers: ASTM E2282-14(2023) - Standard Guide for Defining the Test Result of a Test Method
• Effective Date: 01-Apr-2023
• Refers: ASTM E456-13a(2022)e1 - Standard Terminology Relating to Quality and Statistics
• Effective Date: 01-Apr-2022
• Referred By: ASTM C1592/C1592M-21 - Standard Guide for Making Quality Nondestructive Assay Measurements
• Effective Date: 01-Nov-2023
• Referred By: ASTM E177-20 - Standard Practice for Use of the Terms Precision and Bias in ASTM Test Methods
• Effective Date: 01-Nov-2023
• Referred By: ASTM D8314-20 - Standard Guide for Performance Testing of Applied Stretch Films and Stretch Wrapping
• Effective Date: 01-Nov-2023
• Referred By: ASTM D8272-19 - Standard Guide for Development and Optimization of D19 Chemical Analysis Methods Intended for EPA Compliance Reporting
• Effective Date: 01-Nov-2023
• Referred By: ASTM E1169-21 - Standard Practice for Conducting Ruggedness Tests
• Effective Date: 01-Nov-2023
• Referred By: ASTM E1326-20 - Standard Guide for Evaluating Non-culture Microbiological Tests
• Effective Date: 01-Nov-2023
• Referred By: ASTM E456-13a(2022) - Standard Terminology Relating to Quality and Statistics
• Effective Date: 01-Nov-2023
• Referred By: ASTM F3207-17 - Standard Guide for <emph type="bdit">in vivo</emph> Evaluation of Rabbit Lumbar Intertransverse Process Spinal Fusion Model
• Effective Date: 01-Nov-2023
• Referred By: ASTM F3295-18 - Standard Guide for Impingement Testing of Total Disc Prostheses
• Effective Date: 01-Nov-2023
• Referred By: ASTM E1325-21 - Standard Terminology Relating to Design of Experiments
• Effective Date: 01-Nov-2023
• Referred By: ASTM E2657-21 - Standard Practice for Determination of Endotoxin Concentrations in Water-Miscible Metalworking Fluids
• Effective Date: 01-Nov-2023

Overview

ASTM E1488-23: Standard Guide for Statistical Procedures to Use in Developing and Applying Test Methods is a critical international standard developed by ASTM Committee E11 on Quality and Statistics. This guide outlines the recommended statistical procedures for developing new test methods or revising and evaluating existing methods. It provides structured approaches for designing, refining, and applying test methods, ensuring reliability, precision, and bias control in laboratory testing. The standard assists organizations in selecting robust statistical techniques and references best practices, thereby supporting accurate, repeatable, and consistent measurement outcomes.

Key Topics

• Lifecycle Phases of Test Methods: The standard recommends considering four main phases during the development and application of test methods - Design, Development, Evaluation, and Monitoring.
• Precision and Bias Statements: All ASTM test methods must include precision and bias statements, informed by interlaboratory studies.
• Sources of Variation: Addresses common sources of test result variability, such as operator differences, equipment tolerances, environmental factors, sampling, and time-related changes.
• Statistical Control: Emphasizes the importance of achieving and demonstrating a state of statistical control through control charts and ongoing monitoring.
• Ruggedness Testing: Advocates for planned experimentation to evaluate the insensitivity of methods to small changes in conditions.
• Sampling and Data Analysis: Highlights the role of proper experimental design, type of data collected (numerical or categorical), and sampling requirements.
• Uncertainty Estimation: Provides guidance for laboratories on estimating and reporting measurement uncertainty, aiding in accreditation and quality assurance processes.
• Interlaboratory Testing: Outlines best practices for conducting and analyzing interlaboratory studies (ILS) to assess precision and reproducibility across different labs.

Applications

ASTM E1488-23 offers practical value for multiple stakeholders in testing and quality assurance:

• Test Method Developers: Ensures new or revised test methods are statistically validated, enhancing credibility in published standards.
• Laboratories: Guides on implementing control procedures, conducting ruggedness tests, and participating in interlaboratory studies to maintain compliance and improve measurement reliability.
• Quality Control Professionals: Supports the use of control charts, monitoring variation and maintaining long-term statistical control.
• Accreditation Bodies: Provides frameworks for assessing test method robustness, uncertainty estimation, and compliance with global trade requirements.
• Industry Sectors: Applicable across materials testing, manufacturing, environmental monitoring, pharmaceuticals, and any field requiring standardized analytical test methods.

By following ASTM E1488-23, organizations improve the repeatability and reproducibility of results, foster transparency in reporting, and enable better comparability between laboratories worldwide.

Related Standards

Several ASTM and international standards are referenced or recommended within ASTM E1488-23, including:

• ASTM E177 - Practice for Use of the Terms Precision and Bias in ASTM Test Methods
• ASTM E178 - Practice for Dealing With Outlying Observations
• ASTM E691 - Practice for Conducting an Interlaboratory Study to Determine the Precision of a Test Method
• ASTM E1169 - Practice for Conducting Ruggedness Tests
• ASTM E1402 - Guide for Sampling Design
• ASTM E2586/E2587 - Practices for Calculating and Using Basic Statistics and Control Charts
• ASTM E2655 - Guide for Reporting Uncertainty of Test Results
• ASTM E2489/E2554 - Practices for Statistical Analysis of Interlaboratory Proficiency Testing and Estimating Uncertainty

These referenced documents provide methodologies and terminology essential for the effective implementation of ASTM E1488-23, supporting organizations in achieving high-quality, statistically valid test results.

Keywords: ASTM E1488-23, statistical procedures, test method development, precision and bias, interlaboratory studies, measurement uncertainty, laboratory quality control, ruggedness testing, quality assurance, reproducibility, standardization.

Standards Content (Sample)

This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the
Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
Designation: E1488 − 23 An American National Standard
Standard Guide for
Statistical Procedures to Use in Developing and Applying
Test Methods
This standard is issued under the fixed designation E1488; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope E2655 Guide for Reporting Uncertainty of Test Results and
Use of the Term Measurement Uncertainty in ASTM Test
1.1 This guide identifies statistical procedures for use in
Methods
developing new test methods or revising or evaluating existing
test methods, or both.
3. Terminology
1.2 This guide also cites statistical procedures especially
3.1 Definitions—Unless otherwise noted in this standard, all
useful in the application of test methods.
terms relating to quality and statistics are defined in Terminol-
1.3 This international standard was developed in accor-
ogy E456.
dance with internationally recognized principles on standard-
3.1.1 bias, n—the difference between the expectation of the
ization established in the Decision on Principles for the
test results and an accepted reference value. E177
Development of International Standards, Guides and Recom-
3.1.1.1 Discussion—Statistical procedures include the sam-
mendations issued by the World Trade Organization Technical
pling considerations or the experiment design for the collection
Barriers to Trade (TBT) Committee.
of data, or both, and the numerical and graphical approaches to
summarize and analyze the collected data.
2. Referenced Documents
3.1.2 coeffıcient of variation, CV, n—for a nonnegative
2.1 ASTM Standards:
characteristic, the ratio of the standard deviation to the mean
E177 Practice for Use of the Terms Precision and Bias in
for a population or sample. E2586
ASTM Test Methods
3.1.3 component of variance, n—a part of a total variance
E178 Practice for Dealing With Outlying Observations
identified with a specified source of variability.
E456 Terminology Relating to Quality and Statistics
E691 Practice for Conducting an Interlaboratory Study to
3.1.4 control chart, n—chart on which are plotted a statis-
Determine the Precision of a Test Method
tical measure of a subgroup versus time of sampling along with
E1169 Practice for Conducting Ruggedness Tests
limits based on the statistical distribution of that measure so as
E1402 Guide for Sampling Design
to indicate how much common, or chance, cause variation is
E2282 Guide for Defining the Test Result of a Test Method
inherent in the process or product. E2587
E2489 Practice for Statistical Analysis of One-Sample and
3.1.5 observation, n—the process of obtaining information
Two-Sample Interlaboratory Proficiency Testing Programs
regarding the presence or absence of an attribute of a test
E2554 Practice for Estimating and Monitoring the Uncer-
specimen, or of making a reading on a characteristic or
tainty of Test Results of a Test Method Using Control
dimension of a test specimen. E2282
Chart Techniques
3.1.6 observed value, n—the value obtained by making an
E2586 Practice for Calculating and Using Basic Statistics
observation. E2282
E2587 Practice for Use of Control Charts in Statistical
Process Control 3.1.7 precision, n—the closeness of agreement between
independent test results obtained under stipulated conditions.
E177
This guide is under the jurisdiction of ASTM Committee E11 on Quality and
3.1.8 proficiency testing, n—determination of laboratory
Statistics and is the direct responsibility of Subcommittee E11.20 on Test Method
testing performance by means of interlaboratory comparisons.
Evaluation and Quality Control.
Current edition approved Nov. 1, 2023. Published November 2023. Originally
E2489
approved in 1992. Last previous edition approved in 2023 as E1488 – 12 (2023).
3.1.9 repeatability, n—precision under repeatability condi-
DOI: 10.1520/E1488-23.
For referenced ASTM standards, visit the ASTM website, www.astm.org, or
tions. E177
contact ASTM Customer Service at [e-mail]. For Annual Book of ASTM
3.1.10 repeatability conditions, n—conditions where inde-
Standards volume information, refer to the standard’s Document Summary page on
the ASTM website. pendent test results are obtained with the same method on
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E1488 − 23
identical test items in the same laboratory by the same operator 4.1.1 Assessments of a new or existing test method gener-
using the same equipment within short intervals of time. E177 ally involve statistical planning and analysis. This standard
recommends what approaches may be taken and indicates
3.1.11 repeatability limit r, n—the value below which the
which standards may be used to perform such assessments.
absolute difference between two individual test results obtained
under repeatability conditions may be expected to occur with a
4.2 This standard introduces a series of phases which are
probability of approximately 0.95 (95 %). E177
recommended to be considered during the life cycle of a test
method as depicted in Fig. 1. These begin with a design phase
3.1.12 repeatability standard deviation, s , n—the standard
r
where the standard is initially prepared. A development phase
deviation of test results obtained under repeatability condi-
involves a variety of experiments that allow further refinement
tions. E177
and understanding of how the test method performs within a
3.1.13 reproducibility, n—precision under reproducibility
laboratory. In an evaluation phase the test method is then
conditions. E177
examined by way of interlaboratory studies resulting in preci-
3.1.14 reproducibility conditions, n—conditions where test
sion and bias statistics which are published in the standard.
results are obtained with the same method on identical test
Finally, the test method is subject to a monitoring phase.
items in different laboratories with different operators using
4.3 All ASTM test methods are required to include state-
different equipment. E177
ments on precision and bias.
3.1.15 reproducibility limit, R, n—the value below which
4.4 Since ASTM began to require all test methods to have
the absolute difference between two test results obtained under
precision and bias statements that are based on interlaboratory
reproducibility conditions may be expected to occur with a
studies, there has been increased concern regarding what
probability of approximately 0.95 (95 %). E177
statistical experiments and procedures to use during the devel-
3.1.16 reproducibility standard deviation, s , n—the stan-
R
opment of the test methods. Although there exists a wide range
dard deviation of test results obtained under reproducibility
of statistical procedures, there is a small group of generally
conditions. E177
accepted techniques that are beneficial to follow. This guide is
3.1.17 ruggedness, n—insensitivity of a test method to
designed to provide a brief overview of these procedures and to
departures from specified test or environmental conditions.
suggest an appropriate sequence of conducting these proce-
E1169
dures.
3.1.18 ruggedness test, n—a planned experiment in which
4.5 Statistical procedures often result in interpretations that
environmental factors or test conditions are deliberately varied
are not absolutes. Sometimes the information obtained may be
in order to evaluate the effects of such variation. E1169
inadequate or incomplete, which may lead to additional ques-
3.1.19 standard deviation, n—of a population, σ, the square tions and the need for further experimentation. Information
root of the average or expected value of the squared deviation
outside the data is also important in establishing standards and
of a variable from its mean – of a sample x¯, the square root of in the interpretation of numerical results.
the sum of the squared deviations of the observed values in the
5. Summary of Guide
sample divided by the sample size minus 1. E2586
5.1 Outlined here is a suggested sequence of four phases
3.1.20 state of statistical control, n—process condition
useful in the development of a test method. A flowchart is
when only common causes are operating on the process. E2587
provided in Fig. 1. Such a sequence of analyses may require
3.1.21 statistical procedures, n—the organized techniques
modification in specific situations. The assistance of a qualified
and methods used to collect, analyze, and interpret data.
statistician is recommended at each review phase.
3.1.21.1 Discussion—Statistical procedures include the
sampling considerations or the experiment design for the 5.2 Design Phase:
5.2.1 This phase includes the formalization of the scope and
collection of data, or both, and the numerical and graphical
approaches to summarize and analyze the collected data. the significance and use sections. It may include determining
the purpose and describing a general approach to the test
3.1.22 test determination, n—the value of a characteristic or
method but usually does not involve statistical studies.
dimension of a single test specimen derived from one or more
5.2.2 Studies may be conducted to evaluate the basic
observed values. E2282
performance of the method. The draft test method is prepared
3.1.23 test method, n—a definitive procedure that produces
and sampling requirements and the test result (see Guide
a test result. E2282
E2282) are clearly defined.
3.1.24 test observation, n—see observation. E2282
5.2.3 A flow chart is extremely valuable to identify the
sequence of operations involved in a test method, for example,
3.1.25 test result, n—the value of a characteristic obtained
the sampling steps required to obtain the test specimens,
by carrying out a specified test method. E2282
definition of the test determination, how a test result is to be
4. Significance and Use
computed, and running the tests on the specimens.
4.1 The creation of a standardized test method generally
follows a series of steps from inception to approval and
See the Form and Style Manual for ASTM Standards that specifies, when
ongoing use. In all such stages there are questions of how well
possible, precision statements shall be estimated based on the results of an
the test method performs. interlaboratory test program.
E1488 − 23
FIG. 1 Sequence of Steps
5.3 Development Phase: estimates of repeatability standard deviations should be devel-
5.3.1 The test method is examined for such concerns as its
oped and published in this test method. Until an interlaboratory
stability, ruggedness, statistical control and the contributions to study is performed, these estimates generally are considered to
variability. The completion of this phase should result in
be provisional. Information on how a laboratory should de-
preliminary estimates of precision and the identification and
velop uncertainty estimates should also be provided.
suggested ways to estimate potential contributors to uncer-
5.3.6 Statistical Control—A test method must show capa-
tainty.
bility of performing consistently over time. The use of control
5.3.2 Evaluation of Short Term Control of Test Method—A
charts (see Guide E2655) to monitor a proposed, or existing,
test method must exhibit an ability to provide consistent results
test method over time is one recommended way to examine the
at least over short time periods. Preliminary studies or a pilot
controllability or stability of a test method. This statistical
test should be conducted to evaluate the short term stability of
control should be demonstrated in one or two laboratories
the test method. A small series of repeated tests should be
using homogeneous material (test specimen).
conducted.
5.3.3 Analysis of Variability—Statistically designed experi-
5.4 Evaluation Phase:
ments conducted in one or two laboratories can be used to
5.4.1 The test method is subjected to interlaboratory studies
assess the relative magnitudes of different sources or potential
to provide estimates of within-laboratory repeatability and
contributors to variability of the test results. Such studies can
between-laboratory reproducibility. Additional information is
provide estimates of intermediate measures of precision.
supplied from proficiency studies when conducted.
5.3.4 Ruggedness Test—A ruggedness test (see Practice
5.4.2 Interlaboratory Study (ILS)—In accordance with
E1169) is a statistically designed experiment that helps identify
ASTM Form and Style Manual, whenever feasible, an interla-
problems in running the test method, clarifies errors, and points
boratory study must be conducted. This procedure will provide
out possible environmental conditions, which may adversely
specific estimates of variation anticipated when using the test
affect the test method or point out need for tightening require-
method.
ments. The ruggedness test can assist in locating ways of
reducing variability in the test method. 5.4.3 Protocol for the ILS, Practice E691, provides a guide
for developing the ILS for the test method. A first step is the
5.3.5 Preliminary Estimates of Precision—From the various
studies conducted in accordance with 5.3.2 – 5.3.4, preliminary writing of an ILS Protocol, which will define what must be
E1488 − 23
done before the test specimens (or test materials) are distrib- been conducted. Such tests may provide better information
uted to the participating laboratories. about variation and necessary tolerances than has previously
been available.
5.4.4 Precision Statements—Using the estimates of varia-
tion obtained in the ILS, one may prepare precision statements 6.3.2 If precision estimates have not been established
through an actual interlaboratory test program, then such a
using Practices E691 and E177 or equivalent procedures.
program should be initiated.
5.5 Monitoring Phase:
5.5.1 After a test method is approved and in use it is
7. Data and Sampling
important to ensure that the published precision and bias
7.1 Sample Determination:
statistics for the test method remain achievable and consistent
7.1.1 The sampling section of a standard should indicate
over time or amongst different groups conducting the tests.
clearly what constitutes the primary sampling unit, how that
5.5.2 Monitoring Within a Single Location—It is important
sampling unit is further subdivided, and how multiple test
for any laboratory or organization that will use a particular test
values are designated. (See Guide E1402.)
method over time that a means of monitoring to ensure the
7.1.2 In considering the implication of test results as they
method results using quality control samples are stable and in
relate to the material, the test method should be clear whether
control. Regular evaluation of the uncertainty (Practice E2554)
the sampling method or the test is destructive or nondestruc-
or use of a control charting method (Practice E2587) are two
tive.
ways to monitor the test method.
7.1.3 The user of the test method should be aware of
5.5.3 Between Laboratory Comparisons—Proficiency test-
whether the standard calls for a random sample. In some
ing programs measure the typical variation amongst various
standards, as for example in sampling from coils or rolls of
laboratories. The specific laboratories involved also obtain
material, samples may be taken only from certain portions of
information about how well they perform compared to other
the material.
laboratories.
7.2 Test Result Determination—The procedure for determin-
ing a test result must be clear and unambiguous.
6. Development of Test Method
7.2.1 An observation leads to an observed value.
6.1 Proposed standards that are under development should
7.2.2 Several observed values may lead to a test determina-
be treated in a formal manner following as many of the
tion. The observed values need not be the same type of
suggested procedures as possible. Standards that are already in
measurements (for example, they may consist of three readings
existence as approved test methods or in general practice
such as length, width, and mass).
require periodic review that would include selected procedures.
7.2.3 Several test determinations may lead to a test result, as
6.2 Under Development—The development stage involves
by averaging three test determinations.
test methods that are in the preliminary stages during which
7.2.4 A test result is the consequence of a single execution
equipment may not have been fully tested, practices are not
of the entire test method.
agreed upon, and operators have yet to be adequately trained.
7.3 Type of Data—The kind of data that results from the
Often this stage also applies to standards that have not yet been
application of the test method determines the types of statistical
approved.
analyses to be performed.
6.2.1 It is essential that tests for statistical control,
7.3.1 Numerical versus Categorical/Attribute Data—Most
ruggedness, and variability analyses be conducted prior to any
of the statistical procedures referred to in this guide deal with
interlaboratory test programs.
numerical data. Control charts are available for all types of
6.2.2 After all major environmental contributors have been
data, but all ILS procedures currently in use depend on
identified, controlled, and incorporated into the test method,
numerical data.
and after adequate standardized equipment is available, an ILS
7.3.2 “Normally” Distributed Data—Most of the statistical
can be conducted. The ILS must be completed prior to the first
procedures referred to in this guide consider that the unknown
five-year review. The committee should strive to have ILS
distribution of the test results can be modeled by a normal
results as soon as possible.
distribution.
6.2.3 After evaluating data from ruggedness tests, variabil-
ity analysis, or an ILS, changes to the test method may be
8. Sources of Variability
suggested.
8.1 Experimental Realization of a Test Method:
6.2.4 If major changes are made to the test method, a repeat
8.1.1 A realization of a test method refers to an actual
of the various steps is usually necessary. Precision and bias
application of the test method to produce a test result as
statements should reflect the most current version of the test
specified by the test method. The realization involves an
method.
interpretation of the written document by an operator, who uses
6.3 Existing Standards—These standards comprise test
a specific unit and version of the specified test apparatus, in the
methods that are in common use for which standard equipment particular environment of a testing laboratory, to evaluate a
may exist and for which experienced operators have been
specified number of test specimens of the material to be tested.
trained and are available. Another realization of the test method may involve a change in
6.3.1 Control charting, ruggedness tests, and variability one or more of the experimental factors listed above. The test
analyses will be useful, especially if they have not previously result obtained by another realization of the test method will
E1488 − 23
usually differ from the test result obtained from the first variability that inevitably occurs even though measurement and
realization. Even when none of the experimental factors is adjustment for the environmental variation have been used to
intentionally changed, small changes usually occur. The out- obtain control. Thus, the method must be both robust to the
come of these changes may be seen as variability among the differences between laboratories and require a sufficient num-
test results. ber of test determinations to minimize the effect of within-
laboratory variability.
8.1.2 Each of the listed experimental factors and all others,
known and unknown, that can change the realization of a test
8.5 Sample (Test Specimens):
method, are potential sources of variability in test results. Some
8.5.1 A lot (or shipment) of material must be sampled. Since
of the more common factors are discussed in 8.2 – 8.6.
it is unlikely that the material is perfectly uniform, sampling
8.2 Operator:
variability is another source of variability among test results. In
8.2.1 Clarity of Test Method—Every effort must be made in some applications, useful interpretation of test results may
preparing an ASTM standard test method to eliminate the require the measurement of the sampling error.
possibility of serious differences in interpretation. One way to
8.5.2 In interlaboratory evaluation of test methods to
check clarity is to observe, without comment, a competent
determine testing variability, special attention is required in the
laboratory technician, not previously familiar with the method,
selection of the material sample) to obtain test specimens that
apply the draft test method. If the technician has any difficulty,
are as similar as possible. A small residual amount of material
the draft most likely needs revision.
variability is almost always an inseparable component of any
8.2.2 Completeness of Test Method—It is necessary that estimate of testing variability.
technicians, who are generally familiar with the test method or
8.6 Time:
similar methods, not read anything into the instructions that is
8.6
...

This document is not an ASTM standard and is intended only to provide the user of an ASTM standard an indication of what changes have been made to the previous version. Because
it may not be technically possible to adequately depict all changes accurately, ASTM recommends that users consult prior editions as appropriate. In all cases only the current version
of the standard as published by ASTM is to be considered the official document.
Designation: E1488 − 12 (Reapproved 2023) E1488 − 23 An American National Standard
Standard Guide for
Statistical Procedures to Use in Developing and Applying
Test Methods
This standard is issued under the fixed designation E1488; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope
1.1 This guide identifies statistical procedures for use in developing new test methods or revising or evaluating existing test
methods, or both.
1.2 This guide also cites statistical procedures especially useful in the application of test methods.
1.3 This international standard was developed in accordance with internationally recognized principles on standardization
established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued
by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
2. Referenced Documents
2.1 ASTM Standards:
E177 Practice for Use of the Terms Precision and Bias in ASTM Test Methods
E178 Practice for Dealing With Outlying Observations
E456 Terminology Relating to Quality and Statistics
E691 Practice for Conducting an Interlaboratory Study to Determine the Precision of a Test Method
E1169 Practice for Conducting Ruggedness Tests
E1402 Guide for Sampling Design
E2282 Guide for Defining the Test Result of a Test Method
E2489 Practice for Statistical Analysis of One-Sample and Two-Sample Interlaboratory Proficiency Testing Programs
E2554 Practice for Estimating and Monitoring the Uncertainty of Test Results of a Test Method Using Control Chart Techniques
E2586 Practice for Calculating and Using Basic Statistics
E2587 Practice for Use of Control Charts in Statistical Process Control
E2655 Guide for Reporting Uncertainty of Test Results and Use of the Term Measurement Uncertainty in ASTM Test Methods
3. Terminology
3.1 Definitions—For a more extensive list of terms in E11 standards, see —Unless otherwise noted in this standard, all terms
relating to quality and statistics are defined in Terminology E456.
3.1.1 bias, n—the difference between the expectation of the test results and an accepted reference value. E177
This guide is under the jurisdiction of ASTM Committee E11 on Quality and Statistics and is the direct responsibility of Subcommittee E11.20 on Test Method Evaluation
and Quality Control.
Current edition approved April 1, 2023Nov. 1, 2023. Published April 2023November 2023. Originally approved in 1992. Last previous edition approved in 20182023 as
E1488 – 12 (2018).(2023). DOI: 10.1520/E1488-12R23.10.1520/E1488-23.
For referenced ASTM standards, visit the ASTM website, www.astm.org, or contact ASTM Customer Service at [e-mail]. For Annual Book of ASTM Standards
volume information, refer to the standard’s Document Summary page on the ASTM website.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States
E1488 − 23
3.1.1.1 Discussion—
Statistical procedures include the sampling considerations or the experiment design for the collection of data, or both, and the
numerical and graphical approaches to summarize and analyze the collected data.
3.1.2 coeffıcient of variation, CV, n—for a nonnegative characteristic, the ratio of the standard deviation to the mean for a
population or sample. E2586
3.1.3 component of variance, n—a part of a total variance identified with a specified source of variability.
3.1.4 control chart, n—chart on which are plotted a statistical measure of a subgroup versus time of sampling along with limits
based on the statistical distribution of that measure so as to indicate how much common, or chance, cause variation is inherent in
the process or product. E2587
3.1.5 observation, n—the process of obtaining information regarding the presence or absence of an attribute of a test specimen,
or of making a reading on a characteristic or dimension of a test specimen. E2282
3.1.6 observed value, n—the value obtained by making an observation. E2282
3.1.7 precision, n—the closeness of agreement between independent test results obtained under stipulated conditions. E177
3.1.8 proficiency testing, n—determination of laboratory testing performance by means of interlaboratory comparisons. E2489
3.1.9 repeatability, n—precision under repeatability conditions. E177
3.1.10 repeatability conditions, n—conditions where independent test results are obtained with the same method on identical test
items in the same laboratory by the same operator using the same equipment within short intervals of time. E177
3.1.11 repeatability limit r, n—the value below which the absolute difference between two individual test results obtained under
repeatability conditions may be expected to occur with a probability of approximately 0.95 (95 %). E177
3.1.12 repeatability standard deviation, s , n—the standard deviation of test results obtained under repeatability conditions. E177
r
3.1.13 reproducibility, n—precision under reproducibility conditions. E177
3.1.14 reproducibility conditions, n—conditions where test results are obtained with the same method on identical test items in
different laboratories with different operators using different equipment. E177
3.1.15 reproducibility limit, R, n—the value below which the absolute difference between two test results obtained under
reproducibility conditions may be expected to occur with a probability of approximately 0.95 (95 %). E177
3.1.16 reproducibility standard deviation, s , n—the standard deviation of test results obtained under reproducibility conditions.
R
E177
3.1.17 ruggedness, n—insensitivity of a test method to departures from specified test or environmental conditions. E1169
3.1.18 ruggedness test, n—a planned experiment in which environmental factors or test conditions are deliberately varied in order
to evaluate the effects of such variation. E1169
3.1.19 standard deviation, n—of a population, σ, the square root of the average or expected value of the squared deviation of a
variable from its mean – of a sample x¯, the square root of the sum of the squared deviations of the observed values in the sample
divided by the sample size minus 1. E2586
3.1.20 state of statistical control, n—process condition when only common causes are operating on the process. E2587
E1488 − 23
3.1.21 statistical procedures, n—the organized techniques and methods used to collect, analyze, and interpret data.
3.1.21.1 Discussion—
Statistical procedures include the sampling considerations or the experiment design for the collection of data, or both, and the
numerical and graphical approaches to summarize and analyze the collected data.
3.1.22 test determination, n—the value of a characteristic or dimension of a single test specimen derived from one or more
observed values. E2282
3.1.23 test method, n—a definitive procedure that produces a test result. E2282
3.1.24 test observation, n—see observation. E2282
3.1.25 test result, n—the value of a characteristic obtained by carrying out a specified test method. E2282
4. Significance and Use
4.1 The creation of a standardized test method generally follows a series of steps from inception to approval and ongoing use. In
all such stages there are questions of how well the test method performs.
4.1.1 Assessments of a new or existing test method generally involve statistical planning and analysis. This standard recommends
what approaches may be taken and indicates which standards may be used to perform such assessments.
4.2 This standard introduces a series of phases which are recommended to be considered during the life cycle of a test method
as depicted in Fig. 1. These begin with a design phase where the standard is initially prepared. A development phase involves a
FIG. 1 Sequence of Steps
E1488 − 23
variety of experiments that allow further refinement and understanding of how the test method performs within a laboratory. In
an evaluation phase the test method is then examined by way of interlaboratory studies resulting in precision and bias
statistics which are published in the standard. Finally, the test method is subject to a monitoring phase.
4.3 All ASTM test methods are required to include statements on precision and bias.
4.4 Since ASTM began to require all test methods to have precision and bias statements that are based on interlaboratory test
methods, studies, there has been increased concern regarding what statistical experiments and procedures to use during the
development of the test methods. Although there exists a wide range of statistical procedures, there is a small group of generally
accepted techniques that are beneficial to follow. This guide is designed to provide a brief overview of these procedures and to
suggest an appropriate sequence of carrying out conducting these procedures.
4.5 Statistical procedures often result in interpretations that are not absolutes. Sometimes the information obtained may be
inadequate or incomplete, which may lead to additional questions and the need for further experimentation. Information outside
the data is also important in establishing standards and in the interpretation of numerical results.
5. Summary of Guide
5.1 Outlined belowhere is a suggested sequence of four phases useful in the development of a test method. A flowchart is provided
in Fig. 1. Such a sequence of analyses may need to be modified require modification in specific situations. The assistance of a
qualified statistician is recommended at each review phase.
5.2 Design Phase:
5.2.1 This phase includes the formalization of the scope and the significance and use sections. It may include determining the
purpose and describing a general approach to the test method but usually does not involve statistical studies.
5.2.2 Studies may be conducted to evaluate the basic performance of the method. The draft test method is prepared and sampling
requirements and the test result (see Guide E2282) are clearly defined.
5.2.3 A flow chart is extremely valuable to identify the sequence of operations involved in a test method, for example, the
sampling steps required to obtain the test specimens, definition of the test determination, how a test result is to be computed, and
running the tests on the specimens.
5.3 Development Phase:
5.3.1 The test method is examined for such concerns as its stability, ruggedness, statistical control and the contributions to
variability. The completion of this phase should result in preliminary estimates of precision and the identification and suggested
ways to estimate potential contributors to uncertainty.
5.3.2 Evaluation of Short Term Control of Test Method—A test method must exhibit an ability to provide consistent results at least
over short time periods. Preliminary studies or a pilot test should be conducted to evaluate the short term stability of the test
method. A small series of repeated tests should be conducted.
5.3.3 Analysis of Variability—Statistically designed experiments conducted in one or two laboratories can be used to assess the
relative magnitudes of different sources or potential contributors to variability of the test results. Such studies can provide estimates
of intermediate measures of precision.
5.3.4 Ruggedness Test—A ruggedness test (see Practice E1169) is a statistically designed experiment that helps identify problems
in running the test method, clarifies errors, and points out possible environmental conditions, which may adversely affect the test
method or point out need for tightening requirements. The ruggedness test can assist in locating ways of reducing variability in
the test method.
5.3.5 Preliminary Estimates of Precision—From the various studies conducted in accordance with 5.3.2 – 5.3.4, preliminary
See the Form and Style Manual for ASTM Standards that specifies, when possible, precision statements shall be estimated based on the results of an interlaboratory test
program.
E1488 − 23
estimates of repeatability standard deviations should be developed and published in this test method. Until an interlaboratory study
is performed, these estimates generally are considered to be provisional. Information on how a lablaboratory should develop
uncertainty estimates should also be provided.
5.3.6 Statistical Control—A test method must show capability of performing in a consistent way consistently over time. The use
of control charts (see Guide E2655) to monitor a proposed, or existing, test method over time is one recommended way to examine
the controllability or stability of a test method. This statistical control should be demonstrated in one or two laboratories using
homogeneous material (test specimen).
5.4 Evaluation Phase:
5.4.1 The test method is subjected to interlaboratory studies to provide estimates of within-laboratory repeatability and
between-laboratory reproducibility. Additional information is supplied from proficiency studies when conducted.
5.4.2 Interlaboratory Study (ILS)—In accordance with ASTM Form and Style Manual, whenever feasible, an interlaboratory study
must be conducted. This procedure will provide specific estimates of variation anticipated when using the test method.
5.4.3 Protocol for the ILS, Practice E691, provides a guide for developing the ILS for the test method. A first step is the writing
of an ILS Protocol, which will set out what needs to define what must be done before the test specimens (or test materials) are
distributed to the participating laboratories.
5.4.4 Precision Statements—Using the estimates of variation obtained in the interlaboratory test, ILS, one may prepare precision
statements using Practices E691 and E177 or equivalent procedures.
5.5 Monitoring Phase:
5.5.1 After a test method is approved and in use it is important to ensure that the published precision and bias statistics for the
test method remain achievable and consistent over time or amongst different groups conducting the tests.
5.5.2 Monitoring Within a Single Location—It is important for any laboratory or organization that will use a particular test method
over time that a means of monitoring to ensure the method results using quality control samples are stable and in control. Regular
evaluation of the uncertainty (Practice E2554) or use of a control charting method (Practice E2587) are two ways to monitor the
test method.
5.5.3 Between Laboratory Comparisons—Proficiency testing programs measure the typical variation amongst ordinaryvarious
laboratories. The specific laboratories involved also obtain information about how well they perform compared to other
laboratories.
6. Development of Test Method
6.1 Proposed standards that are under development should be treated in a formal manner following as many of the suggested
procedures as possible. Standards that are already in existence as approved test methods or in general practice require periodic
review that would include selected procedures.
6.2 Under Development—The development stage involves test methods that are in the preliminary stages during which equipment
may not have been fully tested, practices are not agreed upon, and operators have yet to be adequately trained. Often this stage
also applies to standards that have not yet been approved.
6.2.1 It is essential that tests for statistical control, ruggedness, and variability analyses be conducted prior to any interlaboratory
test programs.
6.2.2 After all major environmental contributors have been identified, controlled, and incorporated into the test method, and after
adequate standardized equipment is available, an interlaboratory test ILS can be conducted. The interlaboratory test program ILS
must be completed prior to the first five-year review. The committee should strive to have interlaboratoryILS results as soon as
possible.
E1488 − 23
6.2.3 After evaluating data from ruggedness tests, variability analysis, or an interlaboratory test program, ILS, changes to the test
method may be suggested.
6.2.4 If major changes are made to the test method, a repeat of the various steps is usually necessary. Precision and bias statements
should reflect the most current version of the test method.
6.3 Existing Standards—These standards comprise test methods that are in common use for which standard equipment may exist
and for which experienced operators have been trained and are available.
6.3.1 Control charting, ruggedness tests, and variability analyses will be useful, especially if they have not previously been
conducted. Such tests may provide better information about variation and necessary tolerances than has previously been available.
6.3.2 If precision estimates have not been established through an actual interlaboratory test program, then such a program should
be initiated.
7. Data and Sampling
7.1 Sample Determination:
7.1.1 The sampling section of a standard should indicate clearly what constitutes the primary sampling unit, how that sampling
unit is further subdivided, and how multiple test values are designated. (See Guide E1402.)
7.1.2 In considering the implication of test results as they relate to the material, the test method should be clear as to whether the
sampling method or the test is destructive or nondestructive.
7.1.3 The user of the test method should be aware of whether the standard calls for a random sample. In some standards, as for
example in sampling from coils or rolls of material, samples may be taken only from certain portions of the material.
7.2 Test Result Determination—The procedure for determining a test result must be clear and unambiguous.
7.2.1 An observation leads to an observed value.
7.2.2 Several observed values may lead to a test determination. The observed values need not be the same type of measurements
(for example, they may consist of three readings such as length, width, and mass).
7.2.3 Several Testtest determinations may lead to a test result, as by averaging three test determinations.
7.2.4 A test result is the consequence of a single execution of the entire test method.
7.3 Type of Data—The kind of data that results from the application of the test method determines the types of statistical analyses
to be performed.
7.3.1 Numerical versus Categorical/Attribute Data—Most of the statistical procedures referred to in this guide deal with numerical
data. Control charts are available for all types of data, but all interlaboratory test ILS procedures currently in use depend on
numerical data.
7.3.2 “Normally” Distributed Data—Most of the statistical procedures referred to in this guide consider that the unknown
distribution of the test results can be modeled by a normal distribution.
8. Sources of Variability
8.1 Experimental Realization of a Test Method:
8.1.1 A realization of a test method refers to an actual application of the test method to produce a test result as specified by the
test method. The realization involves an interpretation of the written document by a specific test operator , an operator, who uses
a specific unit and version of the specified test apparatus, in the particular environment of hisa testing laboratory, to evaluate a
specified number of test specimens of the material to be tested. Another realization of the test method may involve a change in
E1488 − 23
one or more of the above emphasized experimental factors.experimental factors listed above. The test result obtained by another
realization of the test method will usually differ from the test result obtained from the first realization. Even when none of the
experimental factors is intentionally changed, small changes usually occur. The outcome of these changes may be seen as
variability among the test results.
8.1.2 Each of the abovelisted experimental factors and all others, known and unknown, that can change the realization of a test
method, are potential sources of variability in test results. Some of the more common factors are discussed in 8.2 – 8.6.
8.2 Operator:
8.2.1 Clarity of Test Method—Every effort must be made in preparing an ASTM standard test method to eliminate the possibility
of serious differences in interpretation. One way to check clarity is to observe, without comment, a competent laboratory
technician, not previously familiar with the method, apply the draft test method. If the technician has any difficulty, the draft most
likely needs revision.
8.2.2 Completeness of Test Method—It is necessary that technicians, who are generally familiar with the test method or similar
methods, not read anything into the instructions that is not explicitly stated therein. Therefore, to ensure minimum variability due
to interpretation, procedural requirements must be complete.
8.2.3 Differences in Operator Technique—Even when operators have been trained by the same teacher or supervisor to give
practically identical interpretations to the various steps of the test method, different operators (or even the same operator at different
times) may still differ in such things as dexterity, reaction time, color sensitivity, interpolation in scale reading, and so forth.
Unavoidable operator differences are thus one source of variability between test results. The test method should be designed and
described to minimize the effects of these operator sources of variability.
8.3 Apparatus:
8.3.1 Tolerances—In order to avoid prohibitive costs, only necessary and reasonable manufacturing and maintenance tolerances
can be specified. The variations allowed by these reasonable specification tolerances can be one source of variability between test
results from different sets o
...